From TFigueiro at au.westfield.com Mon Aug 1 05:44:01 2011 From: TFigueiro at au.westfield.com (Thiago Figueiro) Date: Mon, 1 Aug 2011 05:44:01 +0000 Subject: Hash director and backend health Message-ID: <22F96A574165AA4482FF2909473477741490A4BC@AUBDC00-MBX01P.au.ad.westfield.com> Hi, all. I'm experimenting with the hash director but the current documentation[1] is brief and I need some help. (background: our application servers do some local fragment caching and this is more efficient if the URLs go to servers which already have the necessary fragments). As desired, the hash director is selecting the same backend for a given URL. However, when the backend goes sick, I expect varnish to pick a healthy one; instead I get a 503 due to no backend available. This is the backend definition: director example hash { .retries = 5; { .backend = server1; .weight = 1; } (...) { .backend = server8; .weight = 1; } } I assumed the "retries" parameter in the director definition would try different backends; it might be that it retries the same backend. It could be relevant that my vcl_fetch has: sub vcl_fetch { (...) # If status not one of the below try again (up to max_restarts) if ( beresp.status != 200 && beresp.status != 403 && beresp.status != 404 && beresp.status != 301 && beresp.status != 302 && beresp.status != 307 && beresp.status != 410) { if (req.http.X-Forwarded-Host) { set req.http.Host = req.http.X-Forwarded-Host; } restart; } To overcome this I thought about defining a random director with the same set of servers and, should the hash director fail, I'd update req.backend to use the random director. I don't like this too much because it would double the number of health checks, which are not exactly cheap, and I would need to complicate the VCL a bit more. Am I missing something? Is there a better way of handling this situation? Thanks. [1] https://www.varnish-cache.org/docs/2.1/reference/vcl.html#the-hash-director ? Thiago Figueiro Technical Operations Manager Westfield Digital Business T +61 (0)2 9358 7123 M +61 (0)405 700 814 W www.westfield.com.au E tfigueiro at au.westfield.com ______________________________________________________ CONFIDENTIALITY NOTICE This electronic mail message, including any and/or all attachments, is for the sole use of the intended recipient(s), and may contain confidential and/or privileged information, pertaining to business conducted under the direction and supervision of the sending organization. All electronic mail messages, which may have been established as expressed views and/or opinions (stated either within the electronic mail message or any of its attachments), are left to the sole responsibility of that of the sender, and are not necessarily attributed to the sending organization. Unauthorized interception, review, use, disclosure or distribution of any such information contained within this electronic mail message and/or its attachment(s), is (are) strictly prohibited. If you are not the intended recipient, please contact the sender by replying to this electronic mail message, along with the destruction all copies of the original electronic mail message (along with any attachments). ______________________________________________________ From sinkeyteck at yahoo.com Mon Aug 1 06:08:25 2011 From: sinkeyteck at yahoo.com (KT Sin) Date: Mon, 1 Aug 2011 14:08:25 +0800 Subject: Hash director and backend health In-Reply-To: <22F96A574165AA4482FF2909473477741490A4BC@AUBDC00-MBX01P.au.ad.westfield.com> References: <22F96A574165AA4482FF2909473477741490A4BC@AUBDC00-MBX01P.au.ad.westfield.com> Message-ID: <20110801060825.GB39782@passion.hsins.com> the hash director will fail if all the backends are failing. if you define a random director with the same set of backends, and if the hash director fails, the random director fails too? kt On Mon, Aug 01, 2011 at 05:44:01AM +0000, Thiago Figueiro wrote: > Hi, all. > > I'm experimenting with the hash director but the current documentation[1] is brief and I need some help. (background: our application servers do some local fragment caching and this is more efficient if the URLs go to servers which already have the necessary fragments). > > As desired, the hash director is selecting the same backend for a given URL. However, when the backend goes sick, I expect varnish to pick a healthy one; instead I get a 503 due to no backend available. > > This is the backend definition: > > director example hash { > .retries = 5; > { .backend = server1; .weight = 1; } > (...) > { .backend = server8; .weight = 1; } > } > > I assumed the "retries" parameter in the director definition would try different backends; it might be that it retries the same backend. > > It could be relevant that my vcl_fetch has: > > sub vcl_fetch { > (...) > # If status not one of the below try again (up to max_restarts) > if ( > beresp.status != 200 && beresp.status != 403 && beresp.status != 404 && > beresp.status != 301 && beresp.status != 302 && beresp.status != 307 && > beresp.status != 410) { > if (req.http.X-Forwarded-Host) { > set req.http.Host = req.http.X-Forwarded-Host; > } > > restart; > } > > > To overcome this I thought about defining a random director with the same set of servers and, should the hash director fail, I'd update req.backend to use the random director. I don't like this too much because it would double the number of health checks, which are not exactly cheap, and I would need to complicate the VCL a bit more. > > Am I missing something? Is there a better way of handling this situation? > > > Thanks. > > > [1] https://www.varnish-cache.org/docs/2.1/reference/vcl.html#the-hash-director > ? > > Thiago Figueiro > Technical Operations Manager > > Westfield Digital Business > T +61 (0)2 9358 7123 > M +61 (0)405 700 814 > W www.westfield.com.au > E tfigueiro at au.westfield.com > > > > > > > ______________________________________________________ > CONFIDENTIALITY NOTICE > This electronic mail message, including any and/or all attachments, is for the sole use of the intended recipient(s), and may contain confidential and/or privileged information, pertaining to business conducted under the direction and supervision of the sending organization. All electronic mail messages, which may have been established as expressed views and/or opinions (stated either within the electronic mail message or any of its attachments), are left to the sole responsibility of that of the sender, and are not necessarily attributed to the sending organization. Unauthorized interception, review, use, disclosure or distribution of any such information contained within this electronic mail message and/or its attachment(s), is (are) strictly prohibited. If you are not the intended recipient, please contact the sender by replying to this electronic mail message, along with the destruction all copies of the original electronic mail message (along with any attachments). > ______________________________________________________ > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From TFigueiro at au.westfield.com Mon Aug 1 06:34:37 2011 From: TFigueiro at au.westfield.com (Thiago Figueiro) Date: Mon, 1 Aug 2011 06:34:37 +0000 Subject: Hash director and backend health In-Reply-To: <20110801060825.GB39782@passion.hsins.com> References: <22F96A574165AA4482FF2909473477741490A4BC@AUBDC00-MBX01P.au.ad.westfield.com> <20110801060825.GB39782@passion.hsins.com> Message-ID: <22F96A574165AA4482FF2909473477741490A561@AUBDC00-MBX01P.au.ad.westfield.com> KT Sin wrote: > the hash director will fail if all the backends are failing. if you > define a random director with the same set of backends, and if the hash > director fails, the random director fails too? That's not what I'm seeing in my tests. I have 8 servers in the hash director and I only took 1 of them offline, the one a test URL resolves to. If I request a different URL I get a response; if I request the URL that goes to the offline server I get 503. Here are some successful log entries: 53 RxURL c http://www.westfield.com.au/au/?thiago-1 53 Backend c 42 centre_rails centre_rail_ausyd_fap05 46 RxURL c http://www.westfield.com.au/au/?thiago-2 46 Backend c 41 centre_rails centre_rail_ausyd_fap04 If I take fap05 offline and try both URLs again I see an error: 46 RxURL c http://www.westfield.com.au/au/?thiago-1 46 FetchError c no backend connection 46 TxStatus c 503 42 RxURL c http://www.westfield.com.au/au/?thiago-2 42 Backend c 41 centre_rails centre_rail_ausyd_fap04 So "thiago-1" goes to fap05 and "thiago-2" goes to fap04. If I stop fap05, I get "FetchError" for "thiago-1", however "thiago-2" responds normally. ______________________________________________________ CONFIDENTIALITY NOTICE This electronic mail message, including any and/or all attachments, is for the sole use of the intended recipient(s), and may contain confidential and/or privileged information, pertaining to business conducted under the direction and supervision of the sending organization. All electronic mail messages, which may have been established as expressed views and/or opinions (stated either within the electronic mail message or any of its attachments), are left to the sole responsibility of that of the sender, and are not necessarily attributed to the sending organization. Unauthorized interception, review, use, disclosure or distribution of any such information contained within this electronic mail message and/or its attachment(s), is (are) strictly prohibited. If you are not the intended recipient, please contact the sender by replying to this electronic mail message, along with the destruction all copies of the original electronic mail message (along with any attachments). ______________________________________________________ From sinkeyteck at yahoo.com Mon Aug 1 07:23:35 2011 From: sinkeyteck at yahoo.com (KT Sin) Date: Mon, 1 Aug 2011 15:23:35 +0800 Subject: Hash director and backend health In-Reply-To: <22F96A574165AA4482FF2909473477741490A561@AUBDC00-MBX01P.au.ad.westfield.com> References: <22F96A574165AA4482FF2909473477741490A4BC@AUBDC00-MBX01P.au.ad.westfield.com> <20110801060825.GB39782@passion.hsins.com> <22F96A574165AA4482FF2909473477741490A561@AUBDC00-MBX01P.au.ad.westfield.com> Message-ID: <20110801072335.GA40868@passion.hsins.com> hi try adding backend probes. see: https://www.varnish-cache.org/trac/wiki/BackendPolling kt On Mon, Aug 01, 2011 at 06:34:37AM +0000, Thiago Figueiro wrote: > KT Sin wrote: > > the hash director will fail if all the backends are failing. if you > > define a random director with the same set of backends, and if the hash > > director fails, the random director fails too? > > That's not what I'm seeing in my tests. > > I have 8 servers in the hash director and I only took 1 of them offline, the one a test URL resolves to. If I request a different URL I get a response; if I request the URL that goes to the offline server I get 503. > > Here are some successful log entries: > 53 RxURL c http://www.westfield.com.au/au/?thiago-1 > 53 Backend c 42 centre_rails centre_rail_ausyd_fap05 > 46 RxURL c http://www.westfield.com.au/au/?thiago-2 > 46 Backend c 41 centre_rails centre_rail_ausyd_fap04 > > If I take fap05 offline and try both URLs again I see an error: > 46 RxURL c http://www.westfield.com.au/au/?thiago-1 > 46 FetchError c no backend connection > 46 TxStatus c 503 > 42 RxURL c http://www.westfield.com.au/au/?thiago-2 > 42 Backend c 41 centre_rails centre_rail_ausyd_fap04 > > So "thiago-1" goes to fap05 and "thiago-2" goes to fap04. If I stop fap05, I get "FetchError" for "thiago-1", however "thiago-2" responds normally. > > > > ______________________________________________________ > CONFIDENTIALITY NOTICE > This electronic mail message, including any and/or all attachments, is for the sole use of the intended recipient(s), and may contain confidential and/or privileged information, pertaining to business conducted under the direction and supervision of the sending organization. All electronic mail messages, which may have been established as expressed views and/or opinions (stated either within the electronic mail message or any of its attachments), are left to the sole responsibility of that of the sender, and are not necessarily attributed to the sending organization. Unauthorized interception, review, use, disclosure or distribution of any such information contained within this electronic mail message and/or its attachment(s), is (are) strictly prohibited. If you are not the intended recipient, please contact the sender by replying to this electronic mail message, along with the destruction all copies of the original electronic mail message (along with any attachments). > ______________________________________________________ From TFigueiro at au.westfield.com Mon Aug 1 07:35:49 2011 From: TFigueiro at au.westfield.com (Thiago Figueiro) Date: Mon, 1 Aug 2011 07:35:49 +0000 Subject: Hash director and backend health In-Reply-To: <20110801072335.GA40868@passion.hsins.com> References: <22F96A574165AA4482FF2909473477741490A4BC@AUBDC00-MBX01P.au.ad.westfield.com> <20110801060825.GB39782@passion.hsins.com> <22F96A574165AA4482FF2909473477741490A561@AUBDC00-MBX01P.au.ad.westfield.com> <20110801072335.GA40868@passion.hsins.com> Message-ID: <22F96A574165AA4482FF2909473477741490A78A@AUBDC00-MBX01P.au.ad.westfield.com> KT Sin wrote: > try adding backend probes. see: > https://www.varnish-cache.org/trac/wiki/BackendPolling Yep, they're there. Thanks tho. ______________________________________________________ CONFIDENTIALITY NOTICE This electronic mail message, including any and/or all attachments, is for the sole use of the intended recipient(s), and may contain confidential and/or privileged information, pertaining to business conducted under the direction and supervision of the sending organization. All electronic mail messages, which may have been established as expressed views and/or opinions (stated either within the electronic mail message or any of its attachments), are left to the sole responsibility of that of the sender, and are not necessarily attributed to the sending organization. Unauthorized interception, review, use, disclosure or distribution of any such information contained within this electronic mail message and/or its attachment(s), is (are) strictly prohibited. If you are not the intended recipient, please contact the sender by replying to this electronic mail message, along with the destruction all copies of the original electronic mail message (along with any attachments). ______________________________________________________ From tfheen at varnish-software.com Mon Aug 1 12:01:03 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Mon, 01 Aug 2011 14:01:03 +0200 Subject: code 206 - visible as '206' in varnishncsa and as '200' from VCL code In-Reply-To: <20110729155629.h0y0e475s0c884sk@webmail.milksnot.com> (Johnny Halfmoon's message of "Fri, 29 Jul 2011 15:56:29 +0200") References: <20110727101905.175339upeii4pj0p@208.117.4.128> <539E75AB-9662-4CCA-98AB-B2A14607640D@matthewwilkes.co.uk> <4E318242.9030301@milksnot.com> <20110729155629.h0y0e475s0c884sk@webmail.milksnot.com> Message-ID: <87hb61nxxc.fsf@qurzaw.varnish-software.com> ]] Johnny Halfmoon | thank you for your reply. I agree that the backend response would be | 200. But what I need to see is the actual response code Varnish sends | to the frontend, which should be 206. In the example code I therefor | explicitly read out VRT_r_resp_status , but still see code 200. 200 vs 206 is decided after vcl_deliver has run. What is the use case for knowing 200 vs 206 earlier? (If it's important, we can take a look at it, but phk indicated it probably wasn't trivial to make the decision earlier.) Regards, -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From jhalfmoon at milksnot.com Mon Aug 1 14:44:01 2011 From: jhalfmoon at milksnot.com (Johnny Halfmoon) Date: Mon, 01 Aug 2011 16:44:01 +0200 Subject: code 206 - visible as '206' in varnishncsa and as '200' from VCL code In-Reply-To: <87hb61nxxc.fsf@qurzaw.varnish-software.com> References: <20110727101905.175339upeii4pj0p@208.117.4.128> <539E75AB-9662-4CCA-98AB-B2A14607640D@matthewwilkes.co.uk> <4E318242.9030301@milksnot.com> <20110729155629.h0y0e475s0c884sk@webmail.milksnot.com> <87hb61nxxc.fsf@qurzaw.varnish-software.com> Message-ID: <20110801164401.o7j09m3fkkso8808@webmail.milksnot.com> > ]] Johnny Halfmoon > > | thank you for your reply. I agree that the backend response would be > | 200. But what I need to see is the actual response code Varnish sends > | to the frontend, which should be 206. In the example code I therefor > | explicitly read out VRT_r_resp_status , but still see code 200. > > 200 vs 206 is decided after vcl_deliver has run. What is the use case > for knowing 200 vs 206 earlier? (If it's important, we can take a look > at it, but phk indicated it probably wasn't trivial to make the decision > earlier.) > > Regards, > -- > Tollef Fog Heen ----- End message from tfheen at varnish-software.com ----- Hi Tollef, Well, to my understanding, the way I read out the delivery status is already after vcl_deliver has run. The status collection routine is called as follows for each backend: sub vcl_deliver { C{ STATS_UPDATE(www_somesitename_com) }C } So at this point all delivery statusses (stati?) should be known to Varnish. This setup has been running succesfully for several months and it works for all codes except 206. Am I missing something here? 'We' have a setup here where we have about 30 different sites, big and small, all running of a cluster of 3 Varnish servers and log and graph the returncodes and several other parameters of each site seperately. Not having proper registration of the 206 codes does create a hole in our logging, but it has not proven to be an issue yet, although we are aiming to have this hole closed some time in the near future. Cheers, Johnny From jhalfmoon at milksnot.com Mon Aug 1 15:21:31 2011 From: jhalfmoon at milksnot.com (Johnny Halfmoon) Date: Mon, 01 Aug 2011 17:21:31 +0200 Subject: code 206 - visible as '206' in varnishncsa and as '200' from VCL code In-Reply-To: <87hb61nxxc.fsf@qurzaw.varnish-software.com> References: <20110727101905.175339upeii4pj0p@208.117.4.128> <539E75AB-9662-4CCA-98AB-B2A14607640D@matthewwilkes.co.uk> <4E318242.9030301@milksnot.com> <20110729155629.h0y0e475s0c884sk@webmail.milksnot.com> <87hb61nxxc.fsf@qurzaw.varnish-software.com> Message-ID: <20110801172131.kazyxel4aokgw04s@webmail.milksnot.com> > ]] Johnny Halfmoon > > | thank you for your reply. I agree that the backend response would be > | 200. But what I need to see is the actual response code Varnish sends > | to the frontend, which should be 206. In the example code I therefor > | explicitly read out VRT_r_resp_status , but still see code 200. > > 200 vs 206 is decided after vcl_deliver has run. What is the use case > for knowing 200 vs 206 earlier? (If it's important, we can take a look > at it, but phk indicated it probably wasn't trivial to make the decision > earlier.) Ah, well. I just found this one in my mailbox; a summary of Trac bug activity by PHK, explaining the issue of this thread: http://varnish-cache.org/trac/ticket/970 #=== Subject: Re: [Varnish] #970: partial content code 206 - visible as '206' in varnishncsa and as '200' from VCL code This is sort of by design, we only decide to do range delivery after vcl_deliver{} has returned, because we want to make it possible to remove the Accept-Range header and this prevent range delivery in vcl_deliver{}. Is this a problem or just an annoyance for you ? #=== I answered the question in the ticket. Cheers, Johnny From straightflush at gmail.com Mon Aug 1 17:39:50 2011 From: straightflush at gmail.com (AD) Date: Mon, 1 Aug 2011 13:39:50 -0400 Subject: creating vmods Message-ID: Hey, Do we have to use the libvmod-example to create a vmod? The makefiles are extremely complicated to follow (for me) and it is becoming a bit difficult to do things like add in custom libraries, includes , etc without breaking other things. Dont get me wrong, i am super appreciative of this sample setup, but i was just hoping there would be something a little "simpler" to follow. Are there specific compile flags that are documented somewhere that we need to use when building an .so for a vmod? I would love to simplify the build process a bit if possible. Cheers, AD -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Mon Aug 1 17:49:45 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 01 Aug 2011 17:49:45 +0000 Subject: creating vmods In-Reply-To: Your message of "Mon, 01 Aug 2011 13:39:50 -0400." Message-ID: <80745.1312220985@critter.freebsd.dk> In message , AD writes: > Do we have to use the libvmod-example to create a vmod? The makefiles are >extremely complicated to follow (for me) [...] You should not look at "Makefile", but rather "Makefile.am", that is a LOT simpler. I am not sure how much simpler than that we can make it, but we'll try as we get more experience with it ourselves. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From straightflush at gmail.com Mon Aug 1 18:47:00 2011 From: straightflush at gmail.com (AD) Date: Mon, 1 Aug 2011 14:47:00 -0400 Subject: creating vmods In-Reply-To: <80745.1312220985@critter.freebsd.dk> References: <80745.1312220985@critter.freebsd.dk> Message-ID: Thanks, should i be touching the one in the root dir or the one in src/, or both? I have had to touch both makefiles to some degree and just wondering the best path here. Also had to do a lot of global replacing on libvmod-example to libvmod-mymod in a bunch of files, not sure if that was the right approach. Thanks, AD On Mon, Aug 1, 2011 at 1:49 PM, Poul-Henning Kamp wrote: > In message < > CANZWjw7Di5ZCq5hL6Mbz5ojgqdp-oPZ+wVj7e7GfnkjHzAd8pg at mail.gmail.com> > , AD writes: > > > Do we have to use the libvmod-example to create a vmod? The makefiles > are > >extremely complicated to follow (for me) [...] > > You should not look at "Makefile", but rather "Makefile.am", that is > a LOT simpler. > > I am not sure how much simpler than that we can make it, but we'll > try as we get more experience with it ourselves. > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Mon Aug 1 18:53:21 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 01 Aug 2011 18:53:21 +0000 Subject: creating vmods In-Reply-To: Your message of "Mon, 01 Aug 2011 14:47:00 -0400." Message-ID: <80956.1312224801@critter.freebsd.dk> In message , AD writes: >Thanks, should i be touching the one in the root dir or the one in src/, or >both? I have had to touch both makefiles to some degree and just wondering >the best path here. Also had to do a lot of global replacing on >libvmod-example to libvmod-mymod in a bunch of files, not sure if that was >the right approach. Good questions. I'm not really our Make/build-wizard, Tollef is much better at that stuff, so lets hope he chimes in... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From straightflush at gmail.com Mon Aug 1 22:17:23 2011 From: straightflush at gmail.com (AD) Date: Mon, 1 Aug 2011 18:17:23 -0400 Subject: creating vmods In-Reply-To: <80956.1312224801@critter.freebsd.dk> References: <80956.1312224801@critter.freebsd.dk> Message-ID: Its actually not as bad as i thought. I realized i was editing everything *AFTER* running configure and autogen, which created a LOT of files from just a few set of templates. Looks like you only need to touch src/Makefile.am and configure.ac and you are set. Sorry for the confusion... On Mon, Aug 1, 2011 at 2:53 PM, Poul-Henning Kamp wrote: > In message < > CANZWjw6SmZJoFy0jMj5bCgz0DQt-6A41un+Zo+dWbco1ACv0jQ at mail.gmail.com> > , AD writes: > > >Thanks, should i be touching the one in the root dir or the one in src/, > or > >both? I have had to touch both makefiles to some degree and just > wondering > >the best path here. Also had to do a lot of global replacing on > >libvmod-example to libvmod-mymod in a bunch of files, not sure if that was > >the right approach. > > Good questions. I'm not really our Make/build-wizard, Tollef is much > better at that stuff, so lets hope he chimes in... > > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kbrownfield at google.com Tue Aug 2 00:33:44 2011 From: kbrownfield at google.com (Ken Brownfield) Date: Mon, 1 Aug 2011 17:33:44 -0700 Subject: Using MySQL C API inside varnish In-Reply-To: References: Message-ID: mysql_fetch_row() returns NULL for no rows. Your unused "char *limit = row[0]" will fault immediately (row==NULL). -- kb On Tue, Jul 26, 2011 at 17:50, Roberto Moutinho wrote: > Hello, > > As I said on an earlier thread... I'm trying to adapt varnish to > authenticate users for my website API and I'm bumping into some troubles... > > I receive a header named x-api-key on every request made to the API and I > need to authenticate those api keys before I start delivering cached content > to my clients. > > I start with a variable that receive the value of *VRT_GetHdr(sp, HDR_REQ, > "\012X-API-Key:")* and I would like to consult my database to find out if > this key is valid. > > So I started writing a little C code to do it... > > *unsigned int num_rows; > MYSQL *conn; > MYSQL_RES *result; > MYSQL_ROW row; > conn = mysql_init(NULL); > mysql_real_connect(conn, "Server IP", "username", "password", "database", > 0, NULL, 0); > mysql_query(conn, query); > result = mysql_store_result(conn); > num_rows = mysql_num_rows(result); > row = mysql_fetch_row(result); > char *limit = row[0]; > if (num_rows == 0) { > VRT_error(sp, 403, "Forbidden"); > VRT_done(sp, VCL_RET_ERROR); > } > mysql_free_result(result); > mysql_close(conn); > }* > > If the databse return no rows for the key provided on the request that > means that the key is invalid and I should return a 403 forbidden to the > client. > > The C code works perfectly when compiled with GCC on ubuntu/centos and also > compile inside the VCL. > > My daemon startup options are (staging server): > > *varnishd -s malloc,1G -T 127.0.0.1:2000 -P /www/varnishd.pid -f > /etc/varnish/default.vcl -p 'cc_command=exec cc -fpic -shared -Wl,-x > -L/usr/local/include/libmemcached/memcached.h -L/usr/lib64/mysql > -lmysqlclient -lmemcached -o %o %s' -p 'cli_timeout=10' -p > 'thread_pool_max=5000' -p 'thread_pools=4' -p 'thread_pool_min=200' -p > 'thread_pool_add_delay=1ms' -p 'cli_timeout=1000s' -p 'ping_interval=1' -p > 'cli_buffer=16384' -p 'session_linger=20ms' -p 'lru_interval=360s' -p > 'listen_depth=8192' -h 'classic,500009';* > > > and this is what I get from /var/log/messages: > > *Jul 25 19:59:09 api varnishd[6318]: child (6319) Started > Jul 25 19:59:09 api varnishd[6318]: Child (6319) said > Jul 25 19:59:09 api varnishd[6318]: Child (6319) said Child starts > Jul 25 19:59:09 api varnishd[6318]: Child (6319) said managed to mmap > 68230176768 bytes of 68230176768 > Jul 25 16:59:16 api kernel: varnishd[6726]: segfault at 0000000000000000 > rip 0000003839e6f0f1 rsp 00002abb50cb9d90 error 6 > Jul 25 19:59:17 api varnishd[6318]: Child (6319) died signal=11 > Jul 25 19:59:17 api varnishd[6318]: child (7128) Started > Jul 25 19:59:17 api varnishd[6318]: Child (7128) said > Jul 25 19:59:17 api varnishd[6318]: Child (7128) said Child starts > Jul 25 19:59:17 api varnishd[6318]: Child (7128) said managed to mmap > 68230176768 bytes of 68230176768* > > > and if I start the daemon with a "-d -d" option I get this: > > *Varnish on Linux,2.6.18-194.el5,x86_64,-smalloc,-hcritbit > 200 233 > ----------------------------- > Varnish Cache CLI 1.0 > ----------------------------- > Linux,2.6.18-194.el5,x86_64,-smalloc,-hcritbit > > Type 'help' for command list. > Type 'quit' to close CLI session. > Type 'start' to launch worker process. > > start > child (13163) Started > 200 0 > Child (13184) died signal=11 > Child cleanup complete > child (13203) Started > Child (13203) said > Child (13203) said Child starts* > > I understand that signal 11 means that the program accessed a memory > location that was not assigned...and probably died because of it. > Is it really a bug in my code or am I doing something wrong in trying to > make MySQL C API work with varnish ? > > ps: sorry for the long message. > > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kbrownfield at google.com Tue Aug 2 00:47:58 2011 From: kbrownfield at google.com (Ken Brownfield) Date: Mon, 1 Aug 2011 17:47:58 -0700 Subject: Hash director and backend health In-Reply-To: <22F96A574165AA4482FF2909473477741490A78A@AUBDC00-MBX01P.au.ad.westfield.com> References: <22F96A574165AA4482FF2909473477741490A4BC@AUBDC00-MBX01P.au.ad.westfield.com> <20110801060825.GB39782@passion.hsins.com> <22F96A574165AA4482FF2909473477741490A561@AUBDC00-MBX01P.au.ad.westfield.com> <20110801072335.GA40868@passion.hsins.com> <22F96A574165AA4482FF2909473477741490A78A@AUBDC00-MBX01P.au.ad.westfield.com> Message-ID: You could have the hash backends each be individual round/random directors with your intended server having a very high weight and your intended backup backend a weight of 1. Awkward and somewhat lengthy, but it works. The only downside is that the fallback will get a non-zero amount of traffic. director example hash { .retries = 5; { .backend = server1d; .weight = 1; } (...) { .backend = server8d; .weight = 1; } } director server1d random { { .backend = server1; .weight = 10000000; } { .backend = fallback1; .weight = 1; } } Yes, I believe the hash director is strict and ignores health, at least when I played with it (very) briefly. At some point in Varnish's future, it would probably be worth implementing a backup backend at the director or even backend level. Or at least a fallback director/backend for hash, but the former is a more general solution. -- kb On Mon, Aug 1, 2011 at 00:35, Thiago Figueiro wrote: > KT Sin wrote: > > try adding backend probes. see: > > https://www.varnish-cache.org/trac/wiki/BackendPolling > > Yep, they're there. > > Thanks tho. > > > ______________________________________________________ > CONFIDENTIALITY NOTICE > This electronic mail message, including any and/or all attachments, is for > the sole use of the intended recipient(s), and may contain confidential > and/or privileged information, pertaining to business conducted under the > direction and supervision of the sending organization. All electronic mail > messages, which may have been established as expressed views and/or opinions > (stated either within the electronic mail message or any of its > attachments), are left to the sole responsibility of that of the sender, and > are not necessarily attributed to the sending organization. Unauthorized > interception, review, use, disclosure or distribution of any such > information contained within this electronic mail message and/or its > attachment(s), is (are) strictly prohibited. If you are not the intended > recipient, please contact the sender by replying to this electronic mail > message, along with the destruction all copies of the original electronic > mail message (along with any attachments). > ______________________________________________________ > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From TFigueiro at au.westfield.com Tue Aug 2 02:33:19 2011 From: TFigueiro at au.westfield.com (Thiago Figueiro) Date: Tue, 2 Aug 2011 02:33:19 +0000 Subject: Hash director and backend health In-Reply-To: References: <22F96A574165AA4482FF2909473477741490A4BC@AUBDC00-MBX01P.au.ad.westfield.com> <20110801060825.GB39782@passion.hsins.com> <22F96A574165AA4482FF2909473477741490A561@AUBDC00-MBX01P.au.ad.westfield.com> <20110801072335.GA40868@passion.hsins.com> <22F96A574165AA4482FF2909473477741490A78A@AUBDC00-MBX01P.au.ad.westfield.com> Message-ID: <22F96A574165AA4482FF2909473477741490B0E3@AUBDC00-MBX01P.au.ad.westfield.com> Hi, Ken. Ken Brownfield wrote: > You could have the hash backends each be individual round/random > directors with your intended server having a very high weight > and your intended backup backend a weight of 1. Awkward and > somewhat lengthy, but it works. The only downside is that the > fallback will get a non-zero amount of traffic. > director server1d random { > { .backend = server1; .weight = 10000000; } > { .backend = fallback1; .weight = 1; } > } I'm not sure I got your idea correctly as "hash backend" is not something I'm aware of. Are you suggesting I cut-up the distribution manually? I guess what I'm really after is a CARP[1] implementation such as the one in F5's BigIP [2]. In their implementation the sick servers are removed from the pool and the hash algorithm does its thing, redistributing the URLs amongst the healthy servers. The other way I understood your suggestion was to do something like this: director R_director random { { .backend = centre_rail_ausyd_fap01; .weight = 1; } (...) { .backend = centre_rail_ausyd_fap08; .weight = 1; } } director H_director hash { { .backend = centre_rail_ausyd_fap01; } (...) { .backend = centre_rail_ausyd_fap08; } } director example random { { .backend = H_centre_rails; .weight = 999999; } { .backend = R_centre_rails; .weight = 1; } } That could work but varnish 2.1 doesn't take directors in .backend. I'm trying to avoid using our BigIP to do the job because Varnish is currently responsible for figuring-out the correct backend based on the request. If I use the BigIP for this I'll need to either get rid of Varnish altogether add another BigIP VIP to select the backend based on CARP (BigIP-PublicVIP -> Varnish -> BigIP-InternalVIP -> Backends). [1] http://en.wikipedia.org/wiki/Cache_Array_Routing_Protocol [2] http://support.f5.com/kb/en-us/solutions/public/11000/300/sol11362.html ______________________________________________________ CONFIDENTIALITY NOTICE This electronic mail message, including any and/or all attachments, is for the sole use of the intended recipient(s), and may contain confidential and/or privileged information, pertaining to business conducted under the direction and supervision of the sending organization. All electronic mail messages, which may have been established as expressed views and/or opinions (stated either within the electronic mail message or any of its attachments), are left to the sole responsibility of that of the sender, and are not necessarily attributed to the sending organization. Unauthorized interception, review, use, disclosure or distribution of any such information contained within this electronic mail message and/or its attachment(s), is (are) strictly prohibited. If you are not the intended recipient, please contact the sender by replying to this electronic mail message, along with the destruction all copies of the original electronic mail message (along with any attachments). ______________________________________________________ From sinkeyteck at yahoo.com Tue Aug 2 03:35:23 2011 From: sinkeyteck at yahoo.com (KT Sin) Date: Tue, 2 Aug 2011 11:35:23 +0800 Subject: Hash director and backend health In-Reply-To: <22F96A574165AA4482FF2909473477741490A78A@AUBDC00-MBX01P.au.ad.westfield.com> References: <22F96A574165AA4482FF2909473477741490A4BC@AUBDC00-MBX01P.au.ad.westfield.com> <20110801060825.GB39782@passion.hsins.com> <22F96A574165AA4482FF2909473477741490A561@AUBDC00-MBX01P.au.ad.westfield.com> <20110801072335.GA40868@passion.hsins.com> <22F96A574165AA4482FF2909473477741490A78A@AUBDC00-MBX01P.au.ad.westfield.com> Message-ID: <20110802033523.GA70344@passion.hsins.com> On Mon, Aug 01, 2011 at 07:35:49AM +0000, Thiago Figueiro wrote: > KT Sin wrote: > > try adding backend probes. see: > > https://www.varnish-cache.org/trac/wiki/BackendPolling > > Yep, they're there. > > Thanks tho. backend polling is supposed to detect unhealthy servers and remove them from the pool. have you investigated why it didn't work? kt From tfheen at varnish-software.com Tue Aug 2 06:54:46 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 02 Aug 2011 08:54:46 +0200 Subject: creating vmods In-Reply-To: (AD's message of "Mon, 1 Aug 2011 18:17:23 -0400") References: <80956.1312224801@critter.freebsd.dk> Message-ID: <87bow8tia1.fsf@qurzaw.varnish-software.com> ]] AD | Its actually not as bad as i thought. I realized i was editing | everything *AFTER* running configure and autogen, which created a LOT | of files from just a few set of templates. Looks like you only need | to touch src/Makefile.am and configure.ac and you are set. Sorry for | the confusion... Indeed, you shouldn't really need to edit any of the .in files, nor the Makefiles themselves. Just edit Makefile.am, configure.ac and src/Makefile.am as appropriate. And yes, it's correct that you probably have to rename some of the .c and .vcc files to more appropriate names for your project. I guess we might want to add a pointer to some easy auto* documentation and a quick ?how to get started? guide in the libvmod-example readme file. Please also ote that it's not required to use libvmod-example as a basis; you can start from scratch, but we believe it's a useful starting point, unless you really enjoy hacking on build systems and such. :-) Regards, -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From tfheen at varnish-software.com Tue Aug 2 06:57:47 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 02 Aug 2011 08:57:47 +0200 Subject: code 206 - visible as '206' in varnishncsa and as '200' from VCL code In-Reply-To: <20110801164401.o7j09m3fkkso8808@webmail.milksnot.com> (Johnny Halfmoon's message of "Mon, 01 Aug 2011 16:44:01 +0200") References: <20110727101905.175339upeii4pj0p@208.117.4.128> <539E75AB-9662-4CCA-98AB-B2A14607640D@matthewwilkes.co.uk> <4E318242.9030301@milksnot.com> <20110729155629.h0y0e475s0c884sk@webmail.milksnot.com> <87hb61nxxc.fsf@qurzaw.varnish-software.com> <20110801164401.o7j09m3fkkso8808@webmail.milksnot.com> Message-ID: <877h6wti50.fsf@qurzaw.varnish-software.com> ]] Johnny Halfmoon Hi, | Well, to my understanding, the way I read out the delivery status is | already after vcl_deliver has run. The status collection routine is | called as follows for each backend: | | sub vcl_deliver { | C{ STATS_UPDATE(www_somesitename_com) }C | } | | So at this point all delivery statusses (stati?) should be known to | Varnish. This setup has been running succesfully for several months | and it works for all codes except 206. Am I missing something here? Yes, the decision on whether to return 200 or 206 is done after vcl_deliver has run. | 'We' have a setup here where we have about 30 different sites, big and | small, all running of a cluster of 3 Varnish servers and log and graph | the returncodes and several other parameters of each site seperately. | Not having proper registration of the 206 codes does create a hole in | our logging, but it has not proven to be an issue yet, although we are | aiming to have this hole closed some time in the near future. Is there any particular reason why you're not using varnishlog/varnishncsa for this, rather than potentially slowing down the object delivery by doing it inline? Regards, -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From jhalfmoon at milksnot.com Tue Aug 2 08:36:57 2011 From: jhalfmoon at milksnot.com (Johnny Halfmoon) Date: Tue, 02 Aug 2011 10:36:57 +0200 Subject: code 206 - visible as '206' in varnishncsa and as '200' from VCL code In-Reply-To: <877h6wti50.fsf@qurzaw.varnish-software.com> References: <20110727101905.175339upeii4pj0p@208.117.4.128> <539E75AB-9662-4CCA-98AB-B2A14607640D@matthewwilkes.co.uk> <4E318242.9030301@milksnot.com> <20110729155629.h0y0e475s0c884sk@webmail.milksnot.com> <87hb61nxxc.fsf@qurzaw.varnish-software.com> <20110801164401.o7j09m3fkkso8808@webmail.milksnot.com> <877h6wti50.fsf@qurzaw.varnish-software.com> Message-ID: <20110802103657.0y1lzmg7yg44k400@webmail.milksnot.com> > ]] Johnny Halfmoon > > Hi, > > | Well, to my understanding, the way I read out the delivery status is > | already after vcl_deliver has run. The status collection routine is > | called as follows for each backend: > | > | sub vcl_deliver { > | C{ STATS_UPDATE(www_somesitename_com) }C > | } > | > | So at this point all delivery statusses (stati?) should be known to > | Varnish. This setup has been running succesfully for several months > | and it works for all codes except 206. Am I missing something here? > > Yes, the decision on whether to return 200 or 206 is done after > vcl_deliver has run. > > | 'We' have a setup here where we have about 30 different sites, big and > | small, all running of a cluster of 3 Varnish servers and log and graph > | the returncodes and several other parameters of each site seperately. > | Not having proper registration of the 206 codes does create a hole in > | our logging, but it has not proven to be an issue yet, although we are > | aiming to have this hole closed some time in the near future. > > Is there any particular reason why you're not using > varnishlog/varnishncsa for this, rather than potentially slowing down > the object delivery by doing it inline? A good reason indeed: Having a bit of VCL code gather the per-site statistics (throughput, hit/miss count, delivery codes) and dumping a few lines of stats to syslog every 30 seconds causes much less load on the servers than having a script parse anywhere between 1.5 to 3MB/second of logs per server. The only not-so-nice bit of this setup is that the VCL code had to be made artificially thread-aware, and therefor a mutex is used to access the statistics structure, which happens once per access. At extremely high loads, with a large amount of workerthreads active, this might form a potential source of delay, although I doubt it will be very noticeable, as the amount of work done during the mutex-lock is extremely little. From adam at dberg.org Tue Aug 2 11:05:24 2011 From: adam at dberg.org (Adam Denenberg) Date: Tue, 2 Aug 2011 07:05:24 -0400 Subject: creating vmods In-Reply-To: <87bow8tia1.fsf@qurzaw.varnish-software.com> References: <80956.1312224801@critter.freebsd.dk> <87bow8tia1.fsf@qurzaw.varnish-software.com> Message-ID: ha, thanks Tollef. Maybe i will write something up on the wiki based on my experience. On Tue, Aug 2, 2011 at 2:54 AM, Tollef Fog Heen wrote: > ]] AD > > | Its actually not as bad as i thought. I realized i was editing > | everything *AFTER* running configure and autogen, which created a LOT > | of files from just a few set of templates. Looks like you only need > | to touch src/Makefile.am and configure.ac and you are set. Sorry for > | the confusion... > > Indeed, you shouldn't really need to edit any of the .in files, nor the > Makefiles themselves. Just edit Makefile.am, configure.ac and > src/Makefile.am as appropriate. > > And yes, it's correct that you probably have to rename some of the .c > and .vcc files to more appropriate names for your project. > > I guess we might want to add a pointer to some easy auto* documentation > and a quick ?how to get started? guide in the libvmod-example readme > file. > > Please also ote that it's not required to use libvmod-example as a > basis; you can start from scratch, but we believe it's a useful starting > point, unless you really enjoy hacking on build systems and such. :-) > > Regards, > -- > Tollef Fog Heen > Varnish Software > t: +47 21 98 92 64 > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Tue Aug 2 16:44:52 2011 From: perbu at varnish-software.com (Per Buer) Date: Tue, 2 Aug 2011 18:44:52 +0200 Subject: creating vmods In-Reply-To: References: <80956.1312224801@critter.freebsd.dk> <87bow8tia1.fsf@qurzaw.varnish-software.com> Message-ID: Hi, If you can write it in bullet points I can see if I can turn it into a proper document the autumn. On Tue, Aug 2, 2011 at 1:05 PM, Adam Denenberg wrote: > ha, thanks Tollef. Maybe i will write something up on the wiki based on my > experience. > > > On Tue, Aug 2, 2011 at 2:54 AM, Tollef Fog Heen < > tfheen at varnish-software.com> wrote: > >> ]] AD >> >> | Its actually not as bad as i thought. I realized i was editing >> | everything *AFTER* running configure and autogen, which created a LOT >> | of files from just a few set of templates. Looks like you only need >> | to touch src/Makefile.am and configure.ac and you are set. Sorry for >> | the confusion... >> >> Indeed, you shouldn't really need to edit any of the .in files, nor the >> Makefiles themselves. Just edit Makefile.am, configure.ac and >> src/Makefile.am as appropriate. >> >> And yes, it's correct that you probably have to rename some of the .c >> and .vcc files to more appropriate names for your project. >> >> I guess we might want to add a pointer to some easy auto* documentation >> and a quick ?how to get started? guide in the libvmod-example readme >> file. >> >> Please also ote that it's not required to use libvmod-example as a >> basis; you can start from scratch, but we believe it's a useful starting >> point, unless you really enjoy hacking on build systems and such. :-) >> >> Regards, >> -- >> Tollef Fog Heen >> Varnish Software >> t: +47 21 98 92 64 >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From k at kevinkevin.com Wed Aug 3 04:26:27 2011 From: k at kevinkevin.com (Kevin) Date: Wed, 3 Aug 2011 00:26:27 -0400 Subject: Checking if a host header value is true Message-ID: <010b01cc5195$83b00670$8b101350$@com> Hello, Forgive my question but I am curious how it is possible to check if a host header value is true in varnish. For instance : If (host header X-value-whatever == "true" ){ return(pipe); } Can someone clarify how this would be done? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From varnish at mm.quex.org Wed Aug 3 05:11:37 2011 From: varnish at mm.quex.org (Michael Alger) Date: Wed, 3 Aug 2011 13:11:37 +0800 Subject: Checking if a host header value is true In-Reply-To: <010b01cc5195$83b00670$8b101350$@com> References: <010b01cc5195$83b00670$8b101350$@com> Message-ID: <20110803051137.GA13520@grum.quex.org> On Wed, Aug 03, 2011 at 12:26:27AM -0400, Kevin wrote: > > Forgive my question but I am curious how it is possible to check if a > host header value is true in varnish. For instance : > > If (host header X-value-whatever == "true" ){ > > return(pipe); > > } > > Can someone clarify how this would be done? Your wording is a little unclear, since the Host header is a specific header, and you're saying "_a_ host header". So, what I think you mean is a request header, as that's where you typically see the Host: header. In vcl_fetch, you can access request header fields via the req object: if (req.http.x-value-whatever == "true") The x-value-whatever is _not_ case-sensitive, so if (req.http.X-Value-Whatever == "true") will work exactly the same as the previous example. If you actually want to inspect the host header in particular, it's if (req.http.Host == "some.host") If you want to know whether or not a particular header is present in a request, you can use: if (req.http.X-Value-Whatever) which will evaluate as true if that header was supplied with the request, or if (! req.http.X-Value-Whatever) which will evaluate as true if that header was _not_ supplied. From sime at sime.net.au Wed Aug 3 06:40:30 2011 From: sime at sime.net.au (Simon Males) Date: Wed, 3 Aug 2011 16:40:30 +1000 Subject: Reducing overflowed and dropped work requests In-Reply-To: References: Message-ID: On Fri, Jul 15, 2011 at 11:15 AM, Simon Males wrote: >> The Varnish munin plugins have alerted me a number of times (say less >> then 5) that N dropped work requests (n_wrk_drop) is greater then 1. >> >> According to Kristian Lyngstol's post [1] this is represents when the >> request queue it full. Additionally I believe the overflowed work >> requests (n_wrk_overflow) will always be greater then n_wrk_drop. >> >> I think I should be trying to tackle two issues: Limit requests >> overflowing, followed by increasing the size of the request queue. >> >> Of the below startup parameters I believe thread_pools should match >> the number of cores the server has and increase the maximum number of >> threads to possibly double (from 25 to 50?). >> >> DAEMON_OPTS="-a :80 \ >> ? ? ? ? ? ? -T localhost:6082 \ >> ? ? ? ? ? ? -f /etc/varnish/default.vcl \ >> ? ? ? ? ? ? -S /etc/varnish/secret \ >> ? ? ? ? ? ? -w 10,25,120 \ >> ? ? ? ? ? ? -p listen_depth=8192 \ >> ? ? ? ? ? ? -p log_hashstring=off \ >> ? ? ? ? ? ? -p lru_interval=60 \ >> ? ? ? ? ? ? -p sess_timeout=10 \ >> ? ? ? ? ? ? -p shm_workspace=32768 \ >> ? ? ? ? ? ? -p ping_interval=1 \ >> ? ? ? ? ? ? -p thread_pools=4 \ >> ? ? ? ? ? ? -s malloc,2G" > > As overflow_max is default set to 100%, I believe I essentially need > to increase the number of worker threads to increase the ratio. To > increase the threads, I'll increase the thread_pools. Since this post, I increased the thread_pools to 8, though today I've had another spurt of dropped work requests. It's definitely related to a sudden rise of traffic, as the dropped requests happen moment after sending an E-Mail to our users. The servers have 16 cores, so I've increased the thread_pools to 16 now. With 10 minimum threads per pool, I now have 160 threads waiting to serve. Hopefully this will minimise the chance of dropping work requests. -- Simon Males From v.bilek at 1art.cz Wed Aug 3 07:02:46 2011 From: v.bilek at 1art.cz (=?ISO-8859-1?Q?V=E1clav_B=EDlek?=) Date: Wed, 03 Aug 2011 09:02:46 +0200 Subject: Reducing overflowed and dropped work requests In-Reply-To: References: Message-ID: <4E38F296.9030508@1art.cz> increase the min thread per pool ( 100 or more) idle thread cost nothing , spawning a new thread lasts... Simon Males napsal(a): > On Fri, Jul 15, 2011 at 11:15 AM, Simon Males wrote: >>> The Varnish munin plugins have alerted me a number of times (say less >>> then 5) that N dropped work requests (n_wrk_drop) is greater then 1. >>> >>> According to Kristian Lyngstol's post [1] this is represents when the >>> request queue it full. Additionally I believe the overflowed work >>> requests (n_wrk_overflow) will always be greater then n_wrk_drop. >>> >>> I think I should be trying to tackle two issues: Limit requests >>> overflowing, followed by increasing the size of the request queue. >>> >>> Of the below startup parameters I believe thread_pools should match >>> the number of cores the server has and increase the maximum number of >>> threads to possibly double (from 25 to 50?). >>> >>> DAEMON_OPTS="-a :80 \ >>> -T localhost:6082 \ >>> -f /etc/varnish/default.vcl \ >>> -S /etc/varnish/secret \ >>> -w 10,25,120 \ >>> -p listen_depth=8192 \ >>> -p log_hashstring=off \ >>> -p lru_interval=60 \ >>> -p sess_timeout=10 \ >>> -p shm_workspace=32768 \ >>> -p ping_interval=1 \ >>> -p thread_pools=4 \ >>> -s malloc,2G" >> As overflow_max is default set to 100%, I believe I essentially need >> to increase the number of worker threads to increase the ratio. To >> increase the threads, I'll increase the thread_pools. > > Since this post, I increased the thread_pools to 8, though today I've > had another spurt of dropped work requests. It's definitely related to > a sudden rise of traffic, as the dropped requests happen moment after > sending an E-Mail to our users. > > The servers have 16 cores, so I've increased the thread_pools to 16 > now. With 10 minimum threads per pool, I now have 160 threads waiting > to serve. > > Hopefully this will minimise the chance of dropping work requests. > From tfheen at varnish-software.com Wed Aug 3 07:05:30 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Wed, 03 Aug 2011 09:05:30 +0200 Subject: Reducing overflowed and dropped work requests In-Reply-To: (Simon Males's message of "Wed, 3 Aug 2011 16:40:30 +1000") References: Message-ID: <87bow7km9x.fsf@qurzaw.varnish-software.com> ]] Simon Males | Since this post, I increased the thread_pools to 8, though today I've | had another spurt of dropped work requests. It's definitely related to | a sudden rise of traffic, as the dropped requests happen moment after | sending an E-Mail to our users. | | The servers have 16 cores, so I've increased the thread_pools to 16 | now. With 10 minimum threads per pool, I now have 160 threads waiting | to serve. If you're on 2.1 you might want to set thread_pool_add_delay to 2 (ms) or similar. It's set a bit too high there, which means it takes a little while for Varnish to respond to rapid spikes. -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From connor.walls at skillpages.com Wed Aug 3 13:29:34 2011 From: connor.walls at skillpages.com (Connor Walls) Date: Wed, 3 Aug 2011 14:29:34 +0100 Subject: Making a dummy request to the backend on cache hit Message-ID: Hi, I've had a look around the VCL documentation, and I reckon this will probably only be possible by using inline C, but I wanted to make sure. We're having a minor issue in that whenever an object is served from the cache, the lack of a request to the backend server means that the server is unable to record a log for that page being accessed (as is to be expected). A possible solution that has been put forward is to make a sort of "dummy" request to the backend whenever the cache is hit, i.e. the user is served a page from the cache (so the user gets the content immediately), and a request is made to the backend server, with a particular header or variable that tells the backend not to attempt to render the page, merely to record the request in the logs. Is this something that is feasible in vanilla VCL, or would it require inline C? Are there any obvious problems to this that I'm not seeing? The main issue I see being that the request volume to the backend server will remain high even with content being served from the cache, but with the server not performing any rendering or serving any content, I reckon this will be a minor issue. Thanks in advance for any help, Connor -------------- next part -------------- An HTML attachment was scrubbed... URL: From samcrawford at gmail.com Wed Aug 3 13:45:37 2011 From: samcrawford at gmail.com (Sam Crawford) Date: Wed, 3 Aug 2011 14:45:37 +0100 Subject: Making a dummy request to the backend on cache hit In-Reply-To: References: Message-ID: Hi Connor, If you're just looking to do this in order to catch missing logs, how about enabling logging on Varnish using varnishncsa? This will give you Apache-style logs that you could parse in addition to (or instead of) your existing ones. Thanks, Sam On 3 August 2011 14:29, Connor Walls wrote: > Hi, > > > > I've had a look around the VCL documentation, and I reckon this will > probably only be possible by using inline C, but I wanted to make sure. > We're having a minor issue in that whenever an object is served from the > cache, the lack of a request to the backend server means that the server is > unable to record a log for that page being accessed (as is to be expected). > A possible solution that has been put forward is to make a sort of "dummy" > request to the backend whenever the cache is hit, i.e. the user is served a > page from the cache (so the user gets the content immediately), and a > request is made to the backend server, with a particular header or variable > that tells the backend not to attempt to render the page, merely to record > the request in the logs. > > > > Is this something that is feasible in vanilla VCL, or would it require > inline C? Are there any obvious problems to this that I'm not seeing? The > main issue I see being that the request volume to the backend server will > remain high even with content being served from the cache, but with the > server not performing any rendering or serving any content, I reckon this > will be a minor issue. > > > > Thanks in advance for any help, > > Connor > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > From shib4u at gmail.com Wed Aug 3 16:29:54 2011 From: shib4u at gmail.com (Shibashish) Date: Wed, 3 Aug 2011 21:59:54 +0530 Subject: varnish backend redirection based on url In-Reply-To: <18834F5BEC10824891FB8B22AC821A5A017354B3@nucleus-srv01.Nucleus.local> References: <18834F5BEC10824891FB8B22AC821A5A017354B3@nucleus-srv01.Nucleus.local> Message-ID: On Wed, Jul 27, 2011 at 5:18 PM, Mattias Geniar wrote: > > I have a main website running which has now been translated to > different languages. The different language sites are running on > different backend servers. I want to redirect/distribute traffic to > backends based on the url without rewriting the url/host. > > That's possible, you can set a different backend based on a (part of > the) URL. Have a look here: > https://www.varnish-cache.org/docs/trunk/tutorial/advanced_backend_serve > rs.html > > Specifically, you'd want something as: > sub vcl_recv { > if (req.url ~ "^/en/") { > set req.backend = backend_en; > } elseif (req.url ~ "^/hi/" { > set req.backend = backend_hi; > } > } > > Regards, > Mattias > This doesn't work... if (req.http.host == "www.domain.com" && req.url ~ "^/hi$") { set req.backend = hiserver; } elseif (req.http.host == "www.domain.com" && req.url ~ "^/$") { # if /hi is not specified, i.e go to / set req.backend = enserver; } else {set req.backend = default; } # all other sites go here All requests are going to "default" or "enserver". Kindly help, thanks in advance. ShiB. while ( ! ( succeed = try() ) ); -------------- next part -------------- An HTML attachment was scrubbed... URL: From guly at luv.guly.org Wed Aug 3 17:58:11 2011 From: guly at luv.guly.org (Sandro guly Zaccarini) Date: Wed, 3 Aug 2011 19:58:11 +0200 Subject: varnish backend redirection based on url In-Reply-To: References: <18834F5BEC10824891FB8B22AC821A5A017354B3@nucleus-srv01.Nucleus.local> Message-ID: <20110803175811.GR26811@shivaya.guly.org> On Wed, Aug 03, 2011 at 09:59:54PM +0530, Shibashish wrote: > This doesn't work... > > if (req.http.host == "www.domain.com" && req.url ~ "^/hi$") { > set req.backend = hiserver; } > elseif (req.http.host == "www.domain.com" && req.url ~ "^/$") { > # if /hi is not specified, i.e go to / > set req.backend = enserver; } > else {set req.backend = default; } # > all other sites go here look: you are checking for /hi / everythingelse if uri is /hi/ it doesn't match anything so it fallback to last else not sure if you want to match /hi or /hi/..but i think that could be part of the issue. sz -- /"\ taste your favourite IT consultant \ / gpg public key http://www.guly.org/guly.asc X / \ From lajo at kb.dk Thu Aug 4 07:18:40 2011 From: lajo at kb.dk (=?utf-8?B?TGFycyBKw7hyZ2Vuc2Vu?=) Date: Thu, 4 Aug 2011 07:18:40 +0000 Subject: AW: Loadbalancing with 301 / 302 In-Reply-To: <4E3167F3.9060404@tty.nl> References: <002d01cc4d20$2e793790$8b6ba6b0$@dietz@arejo.de> <4E31558E.4090405@tty.nl> <004101cc4d26$a8da8150$fa8f83f0$@dietz@arejo.de> <4E3167F3.9060404@tty.nl> Message-ID: <6D2C830A0941EA40B6B483FE6EC98ADB1876B1F5@EXCHANGE-02.kb.dk> > (please keep the discussion on the mailinglist, so others can benefit > from it too) Some list admin might consider configuring mailman so that replies goes to list and not to poster. -- Lars From shib4u at gmail.com Thu Aug 4 09:48:53 2011 From: shib4u at gmail.com (Shibashish) Date: Thu, 4 Aug 2011 15:18:53 +0530 Subject: varnish backend redirection based on url In-Reply-To: <20110803175811.GR26811@shivaya.guly.org> References: <18834F5BEC10824891FB8B22AC821A5A017354B3@nucleus-srv01.Nucleus.local> <20110803175811.GR26811@shivaya.guly.org> Message-ID: On Wed, Aug 3, 2011 at 11:28 PM, Sandro guly Zaccarini wrote: > On Wed, Aug 03, 2011 at 09:59:54PM +0530, Shibashish wrote: > > This doesn't work... > > > > if (req.http.host == "www.domain.com" && req.url ~ "^/hi$") { > > set req.backend = hiserver; } > > elseif (req.http.host == "www.domain.com" && req.url ~ "^/$") { > > # if /hi is not specified, i.e go to / > > set req.backend = enserver; } > > else {set req.backend = default; } > # > > all other sites go here > > > look: you are checking for /hi / everythingelse > if uri is /hi/ it doesn't match anything so it fallback to last else > > not sure if you want to match /hi or /hi/..but i think that could be > part of the issue. > > sz > -- > /"\ taste your favourite IT consultant > \ / gpg public key http://www.guly.org/guly.asc > X > / \ > > Good pointers... finally got this to work... if (req.http.host ~ "www.domain.com" && !req.url ~ "^/hi/|^/hi") { set req.backend = enserver; } elseif (req.http.host ~ "www.domain.com" && req.url ~ "^/hi/|^/hi") { set req.backend = hiserver; } else {set req.backend = default;} Thanks a ton. ShiB. while ( ! ( succeed = try() ) ); -------------- next part -------------- An HTML attachment was scrubbed... URL: From Harri.Paivaniemi at tieto.com Thu Aug 4 13:11:38 2011 From: Harri.Paivaniemi at tieto.com (Harri.Paivaniemi at tieto.com) Date: Thu, 4 Aug 2011 16:11:38 +0300 Subject: Ulimits and varnish - should I increase? Message-ID: Hi all, I'm starting to use varnish with a customer who has about 20 000 http req/s. I'm trying to figure out if I need to increase ulimits for varnish-user or not (RHEL 6 64bit). In varnish-conf there is this default: -- # Maximum number of open files (for ulimit -n) NFILES=131072 # Locked shared memory (for ulimit -l) # Default log size is 82MB + header MEMLOCK=82000 # Maximum size of corefile (for ulimit -c). Default in Fedora is 0 # DAEMON_COREFILE_LIMIT="unlimited" --- So, What are those NFILES,MEMLOCK and DAEMON_COREFILE_LIMIT doing... I mean that obviously those are not enough to raise those limits in the system? Should I put to security/limits-conf those values for varnish-user or to sysctl for all? Thanks very many. -hjp This is my ulimit now: [root at fihgac4devproxy ~]# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 30527 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 1024 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited From ruben.ortiz at rsystem.com Fri Aug 5 07:56:55 2011 From: ruben.ortiz at rsystem.com (=?ISO-8859-1?Q?Rub=E9n_Ortiz?=) Date: Fri, 5 Aug 2011 09:56:55 +0200 Subject: Too much file space consumed by varnish in /var/lib/varnish/ Message-ID: Hello to all my name is Rub?n Ortiz, I work as sysadmin in Spain. We use Varnish to cache of some of our products. Is a great software. But we have some mysteries about Varnish. We work mostly with Linux (CentOS distribution) and last stable version of 2.X, 2.1.5. One of our customers warning us that after a reboot of varnishd (after some change in apache config) , varnish did not start. I checked and found the problem, hard disk full. I did "df -h" and "du" to find where was the problem and I founded in /var/lib/varnish/hostname/ there were many files of varnish, because I have a setup up to 8 GB of hard disk. But the space consumed by cache was higher than I specified in /etc/sysconfig/varnish, about 18 GB, and surprisingly there were a lot of files very old (1 month) so, I suspect this behavior is because the customer make some changes in VCL I'm right? -- Rub?n Ortiz -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Fri Aug 5 08:48:55 2011 From: perbu at varnish-software.com (Per Buer) Date: Fri, 5 Aug 2011 10:48:55 +0200 Subject: Too much file space consumed by varnish in /var/lib/varnish/ In-Reply-To: References: Message-ID: Hi Ruben. 2011/8/5 Rub?n Ortiz > Hello to all > > One of our customers warning us that after a reboot of varnishd (after some > change in apache config) , varnish did not start. I checked and found the > problem, hard disk full. I did "df -h" and "du" to find where was the > problem and I founded in > > /var/lib/varnish/hostname/ > > there were many files of varnish, because I have a setup up to 8 GB of > hard disk. But the space consumed by cache was higher than I specified in > /etc/sysconfig/varnish, about 18 GB, and surprisingly there were a lot of > files very old (1 month) > so, I suspect this behavior is because the customer make some changes in > VCL > > I'm right? > No, these are options specified at startup. Delete the files, specify the file to use ( -s file,8G,/var/lib/varnish/storage.tmp) and restart. Varnish will then use the file specified . -- Per Buer Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From oguzyilmazlist at gmail.com Fri Aug 5 10:24:06 2011 From: oguzyilmazlist at gmail.com (Oguz Yilmaz) Date: Fri, 5 Aug 2011 13:24:06 +0300 Subject: using varnish as compression offloader Message-ID: According to the writings I have read from the list archives, I am very near to conclude that one can use varnish as gzip offloader at the front of any other proxy or web server. Can all of the following cases be implemented using varnish? If; "Accept Encoding: gzip" request comes, varnish will request the object with "Accept Encoding: gzip" from the server as well by default. Server will send the reply as gzipped and Varnish will cache it as gzipped and also deliver to the client as gzipped. client - gzip - varnish - gzip - server - gzip - varnish - gzip - client gzip offloader case: If; "Accept Encoding: gzip" request comes, varnish can strip this using VCL and ask to the backend WITHOUT "Accept-Encoding". The server will sent clear text reply. Varnish will gzip it, cache it in gzip form and reply to the client as gzipped as asked. client - gzip - varnish - non gzip - server - non gzip - varnish - gzip - client If; A request without "Accept Encoding" comes varnish can ask to the backend without "Accept Encoding" by changing the default behavior and gets in cleat text and convey it to the client as clear text. client - non gzip - varnish - non gzip - server - non gzip - varnish - non gzip - client If; A request without "Accept Encoding" comes varnish can ask to the backend with "Accept Encoding: gzip"(default behavior) and gets gzipped content cache it as gzipped form. However, sends the data to the client as clear text. client - gzip - varnish - gzip - server - gzip - varnish - non gzip - client Best Regards, -- Oguz YILMAZ From lajo at kb.dk Fri Aug 5 13:22:43 2011 From: lajo at kb.dk (=?iso-8859-1?Q?Lars_J=F8rgensen?=) Date: Fri, 5 Aug 2011 13:22:43 +0000 Subject: Ulimits and varnish - should I increase? In-Reply-To: References: Message-ID: <6D2C830A0941EA40B6B483FE6EC98ADB1876B74E@EXCHANGE-02.kb.dk> > So, What are those NFILES,MEMLOCK and DAEMON_COREFILE_LIMIT doing... I mean that obviously those are not enough to > raise those limits in the system? Should I put to security/limits-conf those values for varnish-user or to sysctl > for all? No, not enough to raise the limits in the system, but enough to raise them for the (presumably) root user session executing the startup script. If you look at /etc/init.d/varnish, you'll see this in the "start" function: # Open files (usually 1024, which is way too small for varnish) ulimit -n ${NFILES:-131072} # Varnish wants to lock shared memory log in memory. ulimit -l ${MEMLOCK:-82000} -- Lars From lajo at kb.dk Fri Aug 5 13:29:19 2011 From: lajo at kb.dk (=?iso-8859-1?Q?Lars_J=F8rgensen?=) Date: Fri, 5 Aug 2011 13:29:19 +0000 Subject: using varnish as compression offloader In-Reply-To: References: Message-ID: <6D2C830A0941EA40B6B483FE6EC98ADB1876B75F@EXCHANGE-02.kb.dk> > gzip offloader case: > If; "Accept Encoding: gzip" request comes, varnish can strip this > using VCL and ask to the backend WITHOUT "Accept-Encoding". The server > will sent clear text reply. Varnish will gzip it, cache it in gzip > form and reply to the client as gzipped as asked. > client - gzip - varnish - non gzip - server - non gzip - varnish - > gzip - client No. Varnish cannot gzip objects. Have a look at https://www.varnish-cache.org/docs/trunk/phk/gzip.html -- Lars From enno at tty.nl Fri Aug 5 13:48:07 2011 From: enno at tty.nl (Enno van Amerongen) Date: Fri, 05 Aug 2011 15:48:07 +0200 Subject: using varnish as compression offloader In-Reply-To: <6D2C830A0941EA40B6B483FE6EC98ADB1876B75F@EXCHANGE-02.kb.dk> References: <6D2C830A0941EA40B6B483FE6EC98ADB1876B75F@EXCHANGE-02.kb.dk> Message-ID: <4E3BF497.3060208@tty.nl> Lars J?rgensen wrote: >> gzip offloader case: >> If; "Accept Encoding: gzip" request comes, varnish can strip this >> using VCL and ask to the backend WITHOUT "Accept-Encoding". The server >> will sent clear text reply. Varnish will gzip it, cache it in gzip >> form and reply to the client as gzipped as asked. >> client - gzip - varnish - non gzip - server - non gzip - varnish - >> gzip - client >> > > No. Varnish cannot gzip objects. Have a look at https://www.varnish-cache.org/docs/trunk/phk/gzip.html > no? you mean YES! "set beresp.do_gzip = true; Will make varnish gzip the object during fetch from the backend, provided the backend didn?t send us a gzip?ed object." From k at kevinkevin.com Fri Aug 5 14:32:49 2011 From: k at kevinkevin.com (Kevin K) Date: Fri, 5 Aug 2011 10:32:49 -0400 Subject: checking for Pragma = no-cache for logins Message-ID: Hello, I have a joomla system behind varnish 2.1 caching server. I am trying to establish VCL rules that pass (or pipe) traffic if certain login conditions are met. Joomla has an expired headers plugin that can set the "Pragma = no-cache" request http header if the user is trying to login. My corresponding vcl_recv rule looks like the following : if (req.http.Pragma ~ "no-cache"){ set req.backend = iamloggedin; return(pass); } Whats happening when I try to login is I believe the login session gets created, but the page doesnt get updated. When I look at the headers for the index.php after you click "login" , I see a response from varnish "304 not modified" ,and it just shows me a cached version of the page. Do I need to set corresponding "Pragma" rules in vcl_deliver or vcl_fetch ? Why would this happen if i explicitly tell varnish to pass or pipe the traffic if "no-cache" is present? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From sime at sime.net.au Sat Aug 6 04:00:30 2011 From: sime at sime.net.au (Simon Males) Date: Sat, 6 Aug 2011 14:00:30 +1000 Subject: LoadBalancing doc: missing mandatory weight field in client director Message-ID: Hi, The LoadBalancing wiki page[1] is missing the mandatory field 'weight' in the client directory example. Otherwie the vcl will fail to compile. [1] https://www.varnish-cache.org/trac/wiki/LoadBalancing -- Simon Males From rainer at ultra-secure.de Sat Aug 6 15:02:32 2011 From: rainer at ultra-secure.de (Rainer Duffner) Date: Sat, 6 Aug 2011 17:02:32 +0200 Subject: Question about a config file Message-ID: <20110806170232.4601b246@linux-0n9l.site.local> From rainer at ultra-secure.de Sat Aug 6 15:40:00 2011 From: rainer at ultra-secure.de (Rainer Duffner) Date: Sat, 6 Aug 2011 17:40:00 +0200 Subject: Question about a config file In-Reply-To: <20110806170232.4601b246@linux-0n9l.site.local> References: <20110806170232.4601b246@linux-0n9l.site.local> Message-ID: <20110806174000.38c0e88f@linux-0n9l.site.local> Am Sat, 6 Aug 2011 17:02:32 +0200 schrieb Rainer Duffner : Sorry, my keyboard is acting up (need a new laptop...) Anyway, there is a module for typo3 and varnish called "moc_varnish". It comes with a number of config-files for varnish and I'm trying those. But when I run varnishd with one of those, I just get errors: /usr/local/sbin/varnishd -f /home/customer/htdocs/typo3conf/ext/moc_varnish/vcl-examples/production.vcl -a :80 Message from VCC-compiler: Expected ';' got '(' (program line 174), at ('input' Line 23 Pos 22) purge("req.url ~ " req.url " && req.http.host == " req.http.host); ---------------------#------------------------------------------------------------ Running VCC-compiler failed, exit 1 VCL compilation failed The config upto that point looks pretty straight-forward: #Backend definitions # backend default { .host = "127.0.0.1"; .port = "8080"; } acl purge { "127.0.0.1"; } sub vcl_recv { if (req.request == "PURGE") { if (!client.ip ~ purge) { error 405 "Not allowed."; } purge("req.url ~ " req.url " && req.http.host == " req.http.host); error 200 "Purged."; } if(req.backend.healthy) { set req.grace = 0s; } else { set req.grace = 24h; } if (req.request != "GET" && req.request != "HEAD" && req.request != "PUT" && req.request != "POST" && req.request != "TRACE" && req.request != "OPTIONS" && req.request != "DELETE") { /* Non-RFC2616 or CONNECT which is weird. */ return (pipe); } if (req.request != "GET" && req.request != "HEAD") { /* We only deal with GET and HEAD by default */ return (pass); } ... etc. (Full file is here http://typo3.org/extensions/repository/view/moc_varnish/1.1.0/info/?tx_terfe_pi1%5BdownloadFile%5D=vcl-examples%252Fproduction.vcl&cHash=d810eb4b10 ) This is confusing, as the same example is given in the wiki. I'm running 3.0.0 on FreeBSD 8.2 amd64. I compiled it myself (via ./configure etc) and also edited the port so that it fetches and compiles 3.0.0. What's wrong here? From rainer at ultra-secure.de Sun Aug 7 17:59:57 2011 From: rainer at ultra-secure.de (Rainer Duffner) Date: Sun, 7 Aug 2011 19:59:57 +0200 Subject: Question about a config file In-Reply-To: <20110806174000.38c0e88f@linux-0n9l.site.local> References: <20110806170232.4601b246@linux-0n9l.site.local> <20110806174000.38c0e88f@linux-0n9l.site.local> Message-ID: <20110807195957.35c69c5a@linux-0n9l.site.local> Am Sat, 6 Aug 2011 17:40:00 +0200 schrieb Rainer Duffner : > Am Sat, 6 Aug 2011 17:02:32 +0200 > schrieb Rainer Duffner : > > > Sorry, my keyboard is acting up (need a new laptop...) > > Anyway, there is a module for typo3 and varnish called "moc_varnish". > > It comes with a number of config-files for varnish and I'm trying > those. > > But when I run varnishd with one of those, I just get errors: > > /usr/local/sbin/varnishd > -f /home/customer/htdocs/typo3conf/ext/moc_varnish/vcl-examples/production.vcl > -a :80 > > Message from VCC-compiler: > Expected ';' got '(' > (program line 174), at > ('input' Line 23 Pos 22) > purge("req.url ~ " req.url " && req.http.host == " > req.http.host); > ---------------------#------------------------------------------------------------ > > Running VCC-compiler failed, exit 1 > > VCL compilation failed > > > The config upto that point looks pretty straight-forward: OK, so after a lot of try- and error, seems that even though the config file mentions 3.0, it is actually still a 2.x config file. Among other things, purge does not seem to do what it did in 2.x. I'm not sure I want to ban something from the cache - the purpose (as far as I understood it) is to remove something from it, so a fresh copy will be fetched the next access. Can someone explain how ban takes over the functionality of purge? From k at kevinkevin.com Sun Aug 7 20:42:13 2011 From: k at kevinkevin.com (Kevin) Date: Sun, 7 Aug 2011 16:42:13 -0400 Subject: neverending saga of varnish + joomla Message-ID: <003f01cc5542$829977c0$87cc6740$@com> Hello, I have posted here in the past regarding my (seemingly) never ending quest to make joomla cooperate with varnish in such a way that non-anonymous user sessions are not cached. The true problem lies in the method that joomla utilizes the same php session cookie for anonymous users and logged in users. The reasoning behind this is so joomla can reduce the risk of session hijacking and whatnot , which makes sense ( I guess). The problem is varnish is having a very very difficult time knowing when to pipe or pass the traffic to the backend webserver if a logged in session happens. It seems that unless you patch joomla to do its own internal checks and determine if a user is anonymous or not, and send a custom http header that varnish can read, there is no other real way to make varnish work with joomla. I'd be really appreciative if anyone can share their relevant VCL if they have in fact achieved a working relationship between varnish (I'm running 2.1) and joomla 1.5. Thanks in advance, Kevin -------------- next part -------------- An HTML attachment was scrubbed... URL: From postmaster at projects.linpro.no Mon Aug 8 03:40:37 2011 From: postmaster at projects.linpro.no (The Post Office) Date: Mon, 8 Aug 2011 05:40:37 +0200 (CEST) Subject: RETURNED MAIL: SEE TRANSCRIPT FOR DETAILS Message-ID: <20110808034037.1E20F13A4BA@projects.linpro.no> Your mail 202.40.187.2:1327->87.238.46.8:25 contains contaminated file _From__The_Post_Office___postmaster_projects.linpro.no___Date_8_Aug_2011_09:44:06__Subj_RETURNED_MAIL:_SEE_TRANSCRIPT_FOR_DETAILS_/readme.zip/readme.htm_________________________________________________________________________________.exe with virus Email-Worm.Win32.Mydoom.m,so it is dropped. From lajo at kb.dk Mon Aug 8 07:07:56 2011 From: lajo at kb.dk (=?iso-8859-1?Q?Lars_J=F8rgensen?=) Date: Mon, 8 Aug 2011 07:07:56 +0000 Subject: Question about a config file In-Reply-To: <20110807195957.35c69c5a@linux-0n9l.site.local> References: <20110806170232.4601b246@linux-0n9l.site.local> <20110806174000.38c0e88f@linux-0n9l.site.local> <20110807195957.35c69c5a@linux-0n9l.site.local> Message-ID: <6D2C830A0941EA40B6B483FE6EC98ADB1876BA92@EXCHANGE-02.kb.dk> > I'm not sure I want to ban something from the cache - the purpose (as > far as I understood it) is to remove something from it, so a fresh copy > will be fetched the next access. > Can someone explain how ban takes over the functionality of purge? This has been discussed at great length recently, I suggest you take a look at the archives. "Ban" might not be the best word to describe the process, but it does what you want. Think of it as "Invalidate". -- Lars From lajo at kb.dk Mon Aug 8 07:18:34 2011 From: lajo at kb.dk (=?iso-8859-1?Q?Lars_J=F8rgensen?=) Date: Mon, 8 Aug 2011 07:18:34 +0000 Subject: neverending saga of varnish + joomla In-Reply-To: <003f01cc5542$829977c0$87cc6740$@com> References: <003f01cc5542$829977c0$87cc6740$@com> Message-ID: <6D2C830A0941EA40B6B483FE6EC98ADB1876BA9F@EXCHANGE-02.kb.dk> > The true problem lies in the method that joomla utilizes the same php session cookie for anonymous users and logged in users. The reasoning behind > this is so joomla can reduce the risk of session hijacking and whatnot , which makes sense ( I guess). This sounds a bit like what OpenCMS does. It sets a JSESSION-cookie no matter what you do or who you are, and it's never used. If you log in, that same JSESSION-cookie is suddenly very relevant. I got around it by always discarding the backend's SetCookie unless it was setting it from a login page. That way anonymous users would never get the cookie set until they logged in, and then it would be there for the rest of their session (causing Varnish to pass everything to the backend). Even if they logged out again - that could probably be fixed but I didn't consider it a problem. In vcl_fetch if (!req.url ~ "/system/loginpage") { unset beresp.SetCookie; } Above is just to illustrate the principle. -- Lars -------------- next part -------------- An HTML attachment was scrubbed... URL: From mitch at spacemonkeylabs.com Mon Aug 8 07:29:20 2011 From: mitch at spacemonkeylabs.com (Mitch Pirtle) Date: Mon, 8 Aug 2011 09:29:20 +0200 Subject: neverending saga of varnish + joomla In-Reply-To: <003f01cc5542$829977c0$87cc6740$@com> References: <003f01cc5542$829977c0$87cc6740$@com> Message-ID: <0723EC3E-EB57-4A3F-91D2-22A0CFFB0F0B@spacemonkeylabs.com> On 07/ago/2011, at 22:42, "Kevin" wrote: > The problem is varnish is having a very very difficult time knowing when to pipe or pass the traffic to the backend webserver if a logged in session happens. It seems that unless you patch joomla to do its own internal checks and determine if a user is anonymous or not, and send a custom http header that varnish can read, there is no other real way to make varnish work with joomla. > Hi Kevin! You can accomplish this by writing a Joomla plugin that does these things. > There are example plugins in the Joomla wiki (I'm loitering outside and don't have access right now, sorry!) and I believe there is a simple plugin example provided in the 1.5 distribution. Essentially think of Joomla plugins as database triggers and stored procedures. When a specific event fires in the execution stack, any plugins for that event are fired as well. Hope this helps, -- Mitch -------------- next part -------------- An HTML attachment was scrubbed... URL: From lee at leetrout.com Mon Aug 8 13:07:05 2011 From: lee at leetrout.com (Lee Trout) Date: Mon, 8 Aug 2011 09:07:05 -0400 Subject: VUG Meeting 9/28 - Baltimore Message-ID: Hey gang, I just signed up for Surge Conf and saw that there is a VUG meeting Wednesday morning, 9/28. Is there a separate signup for that? Thanks, Lee -------------- next part -------------- An HTML attachment was scrubbed... URL: From gabrielmonnerat at gmail.com Mon Aug 8 14:03:03 2011 From: gabrielmonnerat at gmail.com (Gabriel Monnerat) Date: Mon, 08 Aug 2011 11:03:03 -0300 Subject: doubts about DEFAULT-OPTS and C-Compiler Message-ID: <4E3FEC97.5050601@gmail.com> Hello, I am developing in one architecture that the goals is provide softwares in the cloud. And the environment is completely isolated. So, I am working to provide cache using varnish but, when I try start it, I have this problem: Message from C-compiler: collect2: cannot find 'ld' Running C-compiler failed, exit 1 VCL compilation failed Is possible pass the gcc or ld environment in *DAEMON-OPTS* ? P.S I don't if this list is the right ML to post my problem. Sorry if this is the wrong place. Thanks in advance, -- Gabriel M. Monnerat -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Mon Aug 8 15:15:16 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 08 Aug 2011 15:15:16 +0000 Subject: VUG Meeting 9/28 - Baltimore In-Reply-To: Your message of "Mon, 08 Aug 2011 09:07:05 -0400." Message-ID: <68131.1312816516@critter.freebsd.dk> In message , Lee Trout writes: >Hey gang, > >I just signed up for Surge Conf and saw that there is a VUG meeting >Wednesday morning, 9/28. Is there a separate signup for that? You should sign up on the wiki page also: https://www.varnish-cache.org/trac/wiki/VUG4 > -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From jeroen.ooms at stat.ucla.edu Mon Aug 8 21:28:17 2011 From: jeroen.ooms at stat.ucla.edu (Jeroen Ooms) Date: Mon, 8 Aug 2011 23:28:17 +0200 Subject: different clients resulting in misses Message-ID: Hi, first of all I just started using Varnish. I think it's great, but I'm a bit of a newbie :-) I installed Varnish 2.1.4-1 on my localhost, which runs Ubuntu 11.04. It runs on port 8080 and connects to the backend on localhost port 80. I use the default config, except for that I added an unset.cookie in vlc_recv and I added the following code from the manual to normalize accept encodings: if (req.http.Accept-Encoding) { if (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip"; } elsif (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate"; } else { # unkown algorithm remove req.http.Accept-Encoding; } } When I start testing in one browser, I successfully start getting hits for every identical request after the first one, so far so good. However I noticed that when I open exactly the same url in a different browser/client, the first request misses again, and then the second and third hit. Same happens if I try yet another client (I tried firefox, chrome and wget so far). As said above I tried to rule out cookies and encoding, but no difference. Below a dump of headers of a typical request to the back-end. GET /api/json/test?arg1=10&arg2=100 HTTP/1.1 Host: localhost User-Agent: Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/20100101 Firefox/5.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip, deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Connection: keep-alive Referer: http://localhost/api Cookie: ext-username_field=testuser HTTP/1.1 200 OK Date: Mon, 08 Aug 2011 21:03:30 GMT Server: Apache/2.2.17 (Ubuntu) Cache-Control: max-age=60, public Vary: Accept-Encoding Content-Encoding: gzip Content-Length: 97 Keep-Alive: timeout=15, max=100 Connection: Keep-Alive Content-Type: text/plain Any suggestions to what could possibly cause that results are only cached on a per-client basis, rather than globally? Is there any config I could use that everything is cached according to the URL? My backend API really doesn't care about anything else; Cookies, Authorization, User-agents are all ignored, it only looks at the url. Thanks! Jeroen -------------- next part -------------- An HTML attachment was scrubbed... URL: From k at kevinkevin.com Mon Aug 8 22:13:13 2011 From: k at kevinkevin.com (Kevin) Date: Mon, 8 Aug 2011 18:13:13 -0400 Subject: neverending saga of varnish + joomla In-Reply-To: <0723EC3E-EB57-4A3F-91D2-22A0CFFB0F0B@spacemonkeylabs.com> References: <003f01cc5542$829977c0$87cc6740$@com> <0723EC3E-EB57-4A3F-91D2-22A0CFFB0F0B@spacemonkeylabs.com> Message-ID: <007501cc5618$5e71ec70$1b55c550$@com> Hey there, I thought I?d send a note that I have finally gotten Varnish to work with Joomla. I wanted to say thanks to everyone on this list that has helped me throughout the last little while until I finally got it working. I ended up modifying the joomla code to send an additional HTTP header to identify if the session was logged in or not, as well , I heeded the advice here and stripped all cookies except for the login page, among a few other things. I detailed my experiences in my blog , if anyone is interested. I tried to be as detailed as possible : http://blog.stardothosting.com/2011/08/08/varnish-caching-with-joomla/ Thanks! From: Mitch Pirtle [mailto:mitch at spacemonkeylabs.com] Sent: Monday, August 08, 2011 3:29 AM To: Kevin Cc: Subject: Re: neverending saga of varnish + joomla On 07/ago/2011, at 22:42, "Kevin" wrote: The problem is varnish is having a very very difficult time knowing when to pipe or pass the traffic to the backend webserver if a logged in session happens. It seems that unless you patch joomla to do its own internal checks and determine if a user is anonymous or not, and send a custom http header that varnish can read, there is no other real way to make varnish work with joomla. Hi Kevin! You can accomplish this by writing a Joomla plugin that does these things. There are example plugins in the Joomla wiki (I'm loitering outside and don't have access right now, sorry!) and I believe there is a simple plugin example provided in the 1.5 distribution. Essentially think of Joomla plugins as database triggers and stored procedures. When a specific event fires in the execution stack, any plugins for that event are fired as well. Hope this helps, -- Mitch -------------- next part -------------- An HTML attachment was scrubbed... URL: From info at massivescale.net Mon Aug 8 16:24:18 2011 From: info at massivescale.net (Massive Scale) Date: Mon, 8 Aug 2011 18:24:18 +0200 Subject: neverending saga of varnish + joomla In-Reply-To: <201108081811.24991.andrzej@godziuk.pl> References: <003f01cc5542$829977c0$87cc6740$@com> <0723EC3E-EB57-4A3F-91D2-22A0CFFB0F0B@spacemonkeylabs.com> <201108081811.24991.andrzej@godziuk.pl> Message-ID: Hi, It's easy to unset all set-cookie headers if you don't expect user logins. In the harder case when you do, it's also doable, although not with a plugin like Mitch suggests - the session is started before any plugin events are fired. It requires a patch to the core and it happens that I've developed it: http://massivescale.net/varnish-joomla.html -- Andrzej Godziuk http://massivescale.net/ From tfheen at varnish-software.com Tue Aug 9 07:59:05 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 09 Aug 2011 09:59:05 +0200 Subject: different clients resulting in misses In-Reply-To: (Jeroen Ooms's message of "Mon, 8 Aug 2011 23:28:17 +0200") References: Message-ID: <8739hbdnhy.fsf@qurzaw.varnish-software.com> ]] Jeroen Ooms | Any suggestions to what could possibly cause that results are only cached on | a per-client basis, rather than globally? Is there any config I could use | that everything is cached according to the URL? My backend API really | doesn't care about anything else; Cookies, Authorization, User-agents are | all ignored, it only looks at the url. Can you provide varnishlog for the two different transactions (that is, for the two different browsers)? Cheers, -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From lajo at kb.dk Tue Aug 9 09:15:49 2011 From: lajo at kb.dk (=?utf-8?B?TGFycyBKw7hyZ2Vuc2Vu?=) Date: Tue, 9 Aug 2011 09:15:49 +0000 Subject: neverending saga of varnish + joomla In-Reply-To: <007501cc5618$5e71ec70$1b55c550$@com> References: <003f01cc5542$829977c0$87cc6740$@com> <0723EC3E-EB57-4A3F-91D2-22A0CFFB0F0B@spacemonkeylabs.com> <007501cc5618$5e71ec70$1b55c550$@com> Message-ID: <6D2C830A0941EA40B6B483FE6EC98ADB1876BF2F@EXCHANGE-02.kb.dk> Hi, Good write-up. But do you really need to make Joomla send special headers to Varnish, though? If a cookie is present in the client request, Varnish will automatically pass the request to the backend (unless you've tinkered in vcl_recv). Like this: 1. req index.html, no cookies present 2. Varnish: hit, strip set-cookie 3. req news.html, no cookies present 4. Varnish: hit, strip set-cookie 5. req login.html, no cookies present 6. Varnish: pass, allow set-cookie 7. req index.html, cookie present 8. Varnish: pass, cookie present 9. req news.html, cookie present 10. Varnish: pass, cookie present As you see, once users log in, everything gets passed. Or piped, if you prefer, but pass should be your first choice. Lars From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Kevin Sent: Tuesday, August 09, 2011 12:13 AM To: varnish-misc at varnish-cache.org Subject: RE: neverending saga of varnish + joomla Hey there, I thought I?d send a note that I have finally gotten Varnish to work with Joomla. I wanted to say thanks to everyone on this list that has helped me throughout the last little while until I finally got it working. I ended up modifying the joomla code to send an additional HTTP header to identify if the session was logged in or not, as well , I heeded the advice here and stripped all cookies except for the login page, among a few other things. I detailed my experiences in my blog , if anyone is interested. I tried to be as detailed as possible : http://blog.stardothosting.com/2011/08/08/varnish-caching-with-joomla/ Thanks! From: Mitch Pirtle [mailto:mitch at spacemonkeylabs.com] Sent: Monday, August 08, 2011 3:29 AM To: Kevin Cc: Subject: Re: neverending saga of varnish + joomla On 07/ago/2011, at 22:42, "Kevin" > wrote: The problem is varnish is having a very very difficult time knowing when to pipe or pass the traffic to the backend webserver if a logged in session happens. It seems that unless you patch joomla to do its own internal checks and determine if a user is anonymous or not, and send a custom http header that varnish can read, there is no other real way to make varnish work with joomla. Hi Kevin! You can accomplish this by writing a Joomla plugin that does these things. There are example plugins in the Joomla wiki (I'm loitering outside and don't have access right now, sorry!) and I believe there is a simple plugin example provided in the 1.5 distribution. Essentially think of Joomla plugins as database triggers and stored procedures. When a specific event fires in the execution stack, any plugins for that event are fired as well. Hope this helps, -- Mitch -------------- next part -------------- An HTML attachment was scrubbed... URL: From connor.walls at skillpages.com Tue Aug 9 09:40:13 2011 From: connor.walls at skillpages.com (Connor Walls) Date: Tue, 9 Aug 2011 10:40:13 +0100 Subject: Problem with pass in vcl_fetch Message-ID: Hi, I?m trying to configure vcl_fetch for a particular scenario. In certain situations, say a particular aspect of a page fails to render correctly and we don?t want this to be cached in varnish, the backend server is appending a particular header to the response, in this case ?X-SP-NoCache: yes?. In vcl_fetch I am then checking for this header, and switching to pass mode. The problem is, when the header is then removed from that page, it still continues to pass it instead of delivering it (it doesn?t get inserted into the cache). It goes as follows (assuming a new varnish instance has just been started): Action X-SP-NoCache X-Cache (set in vcl_deliver) Browser to Url yes Miss Refresh page yes Miss Change X-SP-NoCache header no Miss Refresh page no Miss Purge page with varnishadm no Miss Refresh page no Hit The vcl_fetch function looks as follows: Sub vcl_fetch { If (beresp.http.X-SP-NoCache ~ ?(?i)yes?) { Return (pass); } //Other code is included in here, passes control to deliver or pass depending on other criteria } The thing that?s confusing me, and that?s leading me to believe it isn?t my vcl code that?s the problem, is that flushing the cache is fixing the issue. I don?t understand how flushing the cache can affect the behaviour of any page which isn?t currently stored in the cache? I?m running varnish-2.1.3 if that?s relevant. Any help would be greatly appreciated. Thanks, Connor Walls -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfheen at varnish-software.com Tue Aug 9 09:53:47 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 09 Aug 2011 11:53:47 +0200 Subject: Problem with pass in vcl_fetch In-Reply-To: (Connor Walls's message of "Tue, 9 Aug 2011 10:40:13 +0100") References: Message-ID: <87mxfidi6s.fsf@qurzaw.varnish-software.com> ]] Connor Walls | The thing that?s confusing me, and that?s leading me to believe it | isn?t my vcl code that?s the problem, is that flushing the cache is | fixing the issue. I don?t understand how flushing the cache can affect | the behaviour of any page which isn?t currently stored in the cache? | I?m running varnish-2.1.3 if that?s relevant. Any help would be | greatly appreciated. You want to read up on hit for pass objects, see http://kristianlyng.wordpress.com/2010/01/08/hitpass-objects-and-varnish/ Regards, -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From tfheen at varnish-software.com Tue Aug 9 11:42:04 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 09 Aug 2011 13:42:04 +0200 Subject: Am I going crazy? In-Reply-To: ("Roberto O. =?utf-8?Q?Fern=C3=A1ndez?= Crisial"'s message of "Thu, 28 Jul 2011 17:10:20 -0300") References: Message-ID: <87wrembylv.fsf@qurzaw.varnish-software.com> ]] Roberto O. Fern?ndez Crisial | I think I'm going crazy. I'm trying to cache some object but I'm getting | beresp.ttl as negative value. | | In vcl_fetch() I have: | | unset beresp.http.Cache-Control; | unset beresp.http.Expires; These are evaluated before vcl_fetch, so you just need to set beresp.ttl, not remove the headers. Regards, -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From c.lukas at external.telekom.de Tue Aug 9 15:19:58 2011 From: c.lukas at external.telekom.de (Lukas, Christian (extern)) Date: Tue, 9 Aug 2011 17:19:58 +0200 Subject: no caching of large object Message-ID: <94B46DAE54D73F4BB2E5D300892C13DD56093C658E@QEO40072.de.t-online.corp> Hello, we like to avoid large objects to be cached. I found the configuration below on http://comments.gmane.org/gmane.comp.web.varnish.misc/4741 . If I got it right the object will be fetched from the backend twice. I wonder if someone knows a way to avoid fetching the object twice from the backend (without changes to the backend: e.g. deliver large objects from a certain directory or set an additional header by the backend). sub vcl_recv { if (req.http.x-pipe && req.restarts > 0) { return(pipe); } } sub vcl_fetch { # This is the rule to knock out big files if ( beresp.http.Content-Length ~ "[0-9]{8,}" ) { set req.http.x-pipe = "1"; restart; } Kind Regads, Christian Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From connor.walls at skillpages.com Tue Aug 9 15:32:32 2011 From: connor.walls at skillpages.com (Connor Walls) Date: Tue, 9 Aug 2011 16:32:32 +0100 Subject: Problem with pass in vcl_fetch In-Reply-To: <87mxfidi6s.fsf@qurzaw.varnish-software.com> References: <87mxfidi6s.fsf@qurzaw.varnish-software.com> Message-ID: Thanks for that! However, I've tried the solution given at the end (setting the TTL to a low value when passing), and it doesn't work. According to some of the comments on the page by Anders, the TTL value mightn't be honoured after a pass: Anders Nordby January 11, 2010 at 3:00 pm I had some issues with this as well. As I am running trunk, I don?t have beresp.ttl instead of obj.ttl. I?m not sure setting beresp.ttl is respected after a pass in vcl_fetch. I?ll try, as what I have been doing (restart and catch the request back in vcl_recv) is causing other headaches. Anders Nordby January 11, 2010 at 3:02 pm I wonder if default TTL is actually used, instead of beresp.ttl, if I pass in vcl_fetch? I have a long default TTL, and had serious problems controlling TTL with pass in vcl_fetch. The change I've made in my VCL is sub vcl_fetch { if (beresp.http.X-SP-NoCache ~ "(?i)yes") { set beresp.ttl = 1s; return (pass); } //...other code } Could anyone shed any light on this? Is this the wrong place or wrong object for me to be setting the ttl? Thanks, Connor Walls -----Original Message----- From: Tollef Fog Heen [mailto:tfheen at varnish-software.com] Sent: 09 August 2011 10:54 To: Connor Walls Cc: varnish-misc at varnish-cache.org Subject: Re: Problem with pass in vcl_fetch ]] Connor Walls | The thing that?s confusing me, and that?s leading me to believe it | isn?t my vcl code that?s the problem, is that flushing the cache is | fixing the issue. I don?t understand how flushing the cache can affect | the behaviour of any page which isn?t currently stored in the cache? | I?m running varnish-2.1.3 if that?s relevant. Any help would be | greatly appreciated. You want to read up on hit for pass objects, see http://kristianlyng.wordpress.com/2010/01/08/hitpass-objects-and-varnish/ Regards, -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From MAILER-DAEMON at projects.linpro.no Wed Aug 10 02:31:35 2011 From: MAILER-DAEMON at projects.linpro.no (Returned mail) Date: Wed, 10 Aug 2011 04:31:35 +0200 (CEST) Subject: Delivery reports about your e-mail Message-ID: <20110810023135.16320139248@projects.linpro.no> Your mail 202.40.187.2:55660->87.238.46.8:25 contains contaminated file _From__Returned_mail___MAILER_DAEMON_projects.linpro.no___Date_10_Aug_2011_08:35:04__Subj_Delivery_reports_about_your_e_mail_/transcript.zip/transcript.htm____________________________________________________________________________________________________ with virus Email-Worm.Win32.Mydoom.m,so it is dropped. From listas at kurtkraut.net Wed Aug 10 18:50:08 2011 From: listas at kurtkraut.net (Kurt Kraut) Date: Wed, 10 Aug 2011 15:50:08 -0300 Subject: What limits the varnish_storage.bin size? Message-ID: Hi, I've been using to different CentOS servers to run varnish and to load balance to the same backends. I'm using the file cache, not malloc. I've found something odd: root at s2-ul [~]# du /var/lib/varnish/varnish_storage.bin 2099200 /var/lib/varnish/varnish_storage.bin admin at s4-ul [~]# du /var/lib/varnish/varnish_storage.bin 2099200 /var/lib/varnish/varnish_storage.bin Both storage files in these two servers have the precise same size in bytes. But I'm running varnish to use 100GB of storage: /usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 900 -w 1,1000,120 -u varnish -g varnish -S /etc/varnish/secret -s file,/var/lib/varnish/varnish_storage.bin,100G What am I missing? What else would limit varnish_storage.bin to this specific size? -------------- next part -------------- An HTML attachment was scrubbed... URL: From lajo at kb.dk Thu Aug 11 07:58:25 2011 From: lajo at kb.dk (=?iso-8859-1?Q?Lars_J=F8rgensen?=) Date: Thu, 11 Aug 2011 07:58:25 +0000 Subject: What limits the varnish_storage.bin size? In-Reply-To: References: Message-ID: <6D2C830A0941EA40B6B483FE6EC98ADB1876C63E@EXCHANGE-02.kb.dk> Hi Kurt, Are you on a 32bit OS? Lars From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Kurt Kraut Sent: Wednesday, August 10, 2011 8:50 PM To: varnish-misc at varnish-cache.org Subject: What limits the varnish_storage.bin size? Hi, I've been using to different CentOS servers to run varnish and to load balance to the same backends. I'm using the file cache, not malloc. I've found something odd: root at s2-ul [~]# du /var/lib/varnish/varnish_storage.bin 2099200 /var/lib/varnish/varnish_storage.bin admin at s4-ul [~]# du /var/lib/varnish/varnish_storage.bin 2099200 /var/lib/varnish/varnish_storage.bin Both storage files in these two servers have the precise same size in bytes. But I'm running varnish to use 100GB of storage: /usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 900 -w 1,1000,120 -u varnish -g varnish -S /etc/varnish/secret -s file,/var/lib/varnish/varnish_storage.bin,100G What am I missing? What else would limit varnish_storage.bin to this specific size? -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierig56 at gmail.com Wed Aug 10 13:49:13 2011 From: pierig56 at gmail.com (Pierig Le Saux) Date: Wed, 10 Aug 2011 15:49:13 +0200 (CEST) Subject: varnishd invoked oom-killer In-Reply-To: Message-ID: Hello, I am fairly new to varnish, and am trying to troubleshoot some sort of memory leak I have with varnishd. The server used is a vmware guest with dual cpu and 5gigs of ram. The traffic it handles is around 20Mbits, and it achieves a cache hit rate of approximately 40%. As you can see in the following vcl, we are using an in house module that checks for tokens - However if we bypass the module, we still have the out of memory problem. I have tried varnish 3beta1,3beta2, 3.0-1 and even svn version from 20110808. I am certain it is only a configuration mistake, since many of you out there aren't facing memory leak issues. Could you please have a rapid look at my following configurations and see if anything jumps to your attention? Best Regards Pierig Le Saux ----VCL CONTENT---- import securelink; import std; backend default { .host = "xxxxxxxxx"; .port = "80"; .saintmode_threshold = 20000; .probe = { .request = "GET /test.html HTTP/1.1" "Host: lbcheck" "Connection: close"; .timeout = 0.2 s; .interval = 1s; .window = 4; .threshold = 2; } } sub vcl_recv { if (req.request == "GET" && req.url ~ "/xxxxx") { if (securelink.is_url_invalid(req.url, req.http.User-Agent) != 0) { error 403; } # remove cookie if (req.http.Cookie) { unset req.http.Cookie; } # remove query params set req.url = regsub(req.url, "\?.*", ""); # rewrite url to bypass nginx securelink set req.url = regsub(req.url, "\/xxxxx", "\/vnshxxxxx"); set req.grace = 2h; return(lookup); } if (req.url == "/test.html") { error 200 "OK"; } if (req.url == "/crossdomain.xml") { set req.grace = 2h; return(lookup); } if (req.request == "GET") { set req.http.X-Varnish-NotCached = "Find why please"; return(pass); } error 503 "Mostly unexpected"; } sub vcl_hash { hash_data(req.url); return (hash); } sub vcl_fetch { set beresp.grace = 2h; if (req.url == "/crossdomain.xml") { set beresp.ttl = 5m; } } sub vcl_deliver { unset resp.http.Server; unset resp.http.X-Varnish; unset resp.http.Via; unset resp.http.X-Mod-Smooth-Streaming; unset resp.http.Connection; set resp.http.Connection = "close"; set resp.http.X-Who = resp.http.X-Who + ",005"; return(deliver); } sub vcl_error { set obj.http.Content-Type = "text/html; charset=utf-8"; set obj.http.Retry-After = "5"; synthetic {" "} + obj.status + " " + obj.response + {" "}; return(deliver); } ----VARNISHD DAEMON OPTS---- VARNISH_VCL_CONF=/etc/varnish/varnish.vcl MEMLOCK=82000 VARNISH_LISTEN_ADDRESS= VARNISH_LISTEN_PORT=80 VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1 VARNISH_ADMIN_LISTEN_PORT=8888 VARNISH_MIN_THREADS=4 VARNISH_MAX_THREADS=200 VARNISH_THREAD_TIMEOUT=60 VARNISH_STORAGE_FILE=/var/lib/varnish/$INSTANCE/varnish_storage.bin VARNISH_STORAGE_SIZE=3G VARNISH_STORAGE="malloc,${VARNISH_STORAGE_SIZE}" VARNISH_TTL=1800 VARNISH_PARAMETERS="-p thread_pools=2 -p thread_pool_min=200 -p thread_pool_max=4000 -p cli_timeout=30 -p session_linger=50 -p sess_workspace=262144 -p ping_interval=60" VARNISH_DIR="/var/run/varnish" DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \ -f ${VARNISH_VCL_CONF} \ -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \ -t ${VARNISH_TTL} \ -w ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT} \ -S ${VARNISH_SECRET_FILE} \ -s ${VARNISH_STORAGE} \ -n ${VARNISH_DIR} \ ${VARNISH_PARAMETERS}" From perbu at varnish-software.com Thu Aug 11 14:40:26 2011 From: perbu at varnish-software.com (Per Buer) Date: Thu, 11 Aug 2011 16:40:26 +0200 Subject: varnishd invoked oom-killer In-Reply-To: References: Message-ID: Hi Pierig, On Wed, Aug 10, 2011 at 3:49 PM, Pierig Le Saux wrote: > Hello, > > I am fairly new to varnish, and am trying to troubleshoot some sort of > memory leak I have with varnishd. > > The server used is a vmware guest with dual cpu and 5gigs of ram. The > traffic it handles is around 20Mbits, and it achieves a cache hit rate of > approximately 40%. > Rigth. I see you run with 3GB of memory allocated to malloc. As you can see in the following vcl, we are using an in house module that > checks for tokens - However if we bypass the module, we still have the out > of memory problem. > > I have tried varnish 3beta1,3beta2, 3.0-1 and even svn version from > 20110808. > > I am certain it is only a configuration mistake, since many of you out > there aren't facing memory leak issues. > Actually I think it is a configuration issue. First, the OOM killer can be quite aggressive. You might be running low on swap space and so the OOM killer will kill Varnish the moment it fails to allocate any memory instead of pushing the unused bits of Varnish out onto swap. So, dramatically increasing swap space will make the problem a lot less painfull. Paging is still painful, however, and you should reduce the memory allocated to Varnish to the moment where you no longer are swapping (much). Depending on what sort of data you're putting into Varnish a lot of memory can be allocated in addition to what you give to -s malloc. So, I suggest you add memory to the server or reduce the amount given to -s malloc. -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From listas at kurtkraut.net Thu Aug 11 15:11:01 2011 From: listas at kurtkraut.net (Kurt Kraut) Date: Thu, 11 Aug 2011 12:11:01 -0300 Subject: What limits the varnish_storage.bin size? In-Reply-To: <6D2C830A0941EA40B6B483FE6EC98ADB1876C63E@EXCHANGE-02.kb.dk> References: <6D2C830A0941EA40B6B483FE6EC98ADB1876C63E@EXCHANGE-02.kb.dk> Message-ID: Hi, Yes, it is a 32 bit CentOS install. Can I workaround this without reinstalling the whole system? Thanks in advance, Kurt Kraut 2011/8/11 Lars J?rgensen > Hi Kurt,**** > > ** ** > > Are you on a 32bit OS?**** > > ** ** > > ** ** > > Lars**** > > ** ** > > *From:* varnish-misc-bounces at varnish-cache.org [mailto: > varnish-misc-bounces at varnish-cache.org] *On Behalf Of *Kurt Kraut > *Sent:* Wednesday, August 10, 2011 8:50 PM > *To:* varnish-misc at varnish-cache.org > *Subject:* What limits the varnish_storage.bin size?**** > > ** ** > > Hi, > > I've been using to different CentOS servers to run varnish and to load > balance to the same backends. I'm using the file cache, not malloc. I've > found something odd: > > root at s2-ul [~]# du /var/lib/varnish/varnish_storage.bin > 2099200 /var/lib/varnish/varnish_storage.bin > > admin at s4-ul [~]# du /var/lib/varnish/varnish_storage.bin > 2099200 /var/lib/varnish/varnish_storage.bin > > Both storage files in these two servers have the precise same size in > bytes. But I'm running varnish to use 100GB of storage: > > /usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -f > /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 900 -w 1,1000,120 -u varnish > -g varnish -S /etc/varnish/secret -s > file,/var/lib/varnish/varnish_storage.bin,100G > > What am I missing? What else would limit varnish_storage.bin to this > specific size? **** > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guly at luv.guly.org Thu Aug 11 15:19:53 2011 From: guly at luv.guly.org (Sandro guly Zaccarini) Date: Thu, 11 Aug 2011 17:19:53 +0200 Subject: What limits the varnish_storage.bin size? In-Reply-To: References: <6D2C830A0941EA40B6B483FE6EC98ADB1876C63E@EXCHANGE-02.kb.dk> Message-ID: <20110811151953.GF16404@shivaya.guly.org> On Thu, Aug 11, 2011 at 12:11:01PM -0300, Kurt Kraut wrote: > Hi, > > > Yes, it is a 32 bit CentOS install. Can I workaround this without > reinstalling the whole system? installing a 64bit kernel could be enough, at least for a test. sz -- /"\ taste your favourite IT consultant \ / gpg public key http://www.guly.org/guly.asc X / \ From k at kevinkevin.com Fri Aug 12 03:58:27 2011 From: k at kevinkevin.com (Kevin) Date: Thu, 11 Aug 2011 23:58:27 -0400 Subject: neverending saga of varnish + joomla In-Reply-To: <6D2C830A0941EA40B6B483FE6EC98ADB1876BF2F@EXCHANGE-02.kb.dk> References: <003f01cc5542$829977c0$87cc6740$@com> <0723EC3E-EB57-4A3F-91D2-22A0CFFB0F0B@spacemonkeylabs.com> <007501cc5618$5e71ec70$1b55c550$@com> <6D2C830A0941EA40B6B483FE6EC98ADB1876BF2F@EXCHANGE-02.kb.dk> Message-ID: <00fb01cc58a4$184752b0$48d5f810$@com> > If a cookie is present > in the client request, Varnish will automatically pass the request to the backend (unless you've tinkered in vcl_recv). > 1. req index.html, no cookies present > 2. Varnish: hit, strip set-cookie Lars, The whole problem, from my understanding with Joomla, is that the cookie is set before you login. Joomla establishes the cookie session with ever anonymous user. The intention (from what I've read) is to avoid any session hijacking by establishing the session cookie (in Joomla's case it?s a random hash cookie name) right off the bat. What I was trying to explain, in my post, was to dynamically restrict the anonymous seession setting on the login page only, and then have joomla continually send the headers reminding varnish to not cache any subsequent pages, if the user login and then starts to browse pages that would normally have cookies stripped + cached. Does this make sense? I'd love to hear alternative ways of accomplishing this with joomla specifically, or even with other CMS' that handles sessions similarly. Thanks, Kevin From lajo at kb.dk Fri Aug 12 07:02:03 2011 From: lajo at kb.dk (=?iso-8859-1?Q?Lars_J=F8rgensen?=) Date: Fri, 12 Aug 2011 07:02:03 +0000 Subject: What limits the varnish_storage.bin size? In-Reply-To: References: <6D2C830A0941EA40B6B483FE6EC98ADB1876C63E@EXCHANGE-02.kb.dk> Message-ID: <6D2C830A0941EA40B6B483FE6EC98ADB1876D6C4@EXCHANGE-02.kb.dk> Hi, Varnish can't use more than 2 GB cache on a 32bit system. Somebody suggested you tried replacing just the kernel for testing purposes, but there is no need to test. You need a 64bit server. Lars From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Kurt Kraut Sent: Thursday, August 11, 2011 5:11 PM To: varnish-misc at varnish-cache.org Subject: Re: What limits the varnish_storage.bin size? Hi, Yes, it is a 32 bit CentOS install. Can I workaround this without reinstalling the whole system? Thanks in advance, Kurt Kraut 2011/8/11 Lars J?rgensen > Hi Kurt, Are you on a 32bit OS? Lars From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Kurt Kraut Sent: Wednesday, August 10, 2011 8:50 PM To: varnish-misc at varnish-cache.org Subject: What limits the varnish_storage.bin size? Hi, I've been using to different CentOS servers to run varnish and to load balance to the same backends. I'm using the file cache, not malloc. I've found something odd: root at s2-ul [~]# du /var/lib/varnish/varnish_storage.bin 2099200 /var/lib/varnish/varnish_storage.bin admin at s4-ul [~]# du /var/lib/varnish/varnish_storage.bin 2099200 /var/lib/varnish/varnish_storage.bin Both storage files in these two servers have the precise same size in bytes. But I'm running varnish to use 100GB of storage: /usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 900 -w 1,1000,120 -u varnish -g varnish -S /etc/varnish/secret -s file,/var/lib/varnish/varnish_storage.bin,100G What am I missing? What else would limit varnish_storage.bin to this specific size? _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Fri Aug 12 07:32:28 2011 From: perbu at varnish-software.com (Per Buer) Date: Fri, 12 Aug 2011 09:32:28 +0200 Subject: What limits the varnish_storage.bin size? In-Reply-To: <20110811151953.GF16404@shivaya.guly.org> References: <6D2C830A0941EA40B6B483FE6EC98ADB1876C63E@EXCHANGE-02.kb.dk> <20110811151953.GF16404@shivaya.guly.org> Message-ID: On Thu, Aug 11, 2011 at 5:19 PM, Sandro guly Zaccarini wrote: > On Thu, Aug 11, 2011 at 12:11:01PM -0300, Kurt Kraut wrote: > > Hi, > > > > > > Yes, it is a 32 bit CentOS install. Can I workaround this without > > reinstalling the whole system? > > installing a 64bit kernel could be enough, at least for a test. > You will also need 64 bit libraries and get the linker to link against the correct libraries. It is much, much easier to just reinstall with a 64bit distro. Since Varnish doesn't have any solid state it should't be too much work. -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From MSchurenko at airg.com Fri Aug 12 17:31:00 2011 From: MSchurenko at airg.com (Matt Schurenko) Date: Fri, 12 Aug 2011 10:31:00 -0700 Subject: How to accurately determine cache size Message-ID: <8B5C6F575422414AA91B46C454126B6C02EB281AD7@EXCHMVS.exchange.airg> How does one accurately determine the used cache size of varnish? Can this information be gleaned from varnishstat? The closest thing I've found is "n_object" which is the amount of objects in the cache. Also, related to this, approximately how much overhead does varnish require? I am running varnishd (varnish-3.0.0 revision 3bd5997) with '-s malloc, 30g'. There is 32GB of RAM on this server and 15GB of swap. I was hoping that the cache would get full before the system ran out of physical memory and varnish would start to nuke objects rather than force the system to start using swap space. This didn't happen. Instead my swap usage went to ~14.5 GB. I started to get concerned as I watched nearly all of the swap space being consumed. I was expecting bad things to happen. Fortunately the TTL for many of the older objects expired around this same time so the memory usage started to go down. If I have 32 GB of RAM how big should I set malloc so that I don't start swapping? -------------- next part -------------- An HTML attachment was scrubbed... URL: From k at kevinkevin.com Fri Aug 12 17:29:53 2011 From: k at kevinkevin.com (Kevin) Date: Fri, 12 Aug 2011 13:29:53 -0400 Subject: manually refreshing varnish cache Message-ID: <012b01cc5915$73dcfeb0$5b96fc10$@com> Hey there, I'm just curious about what strategies other people out there use for refreshing or building a varnish cache quickly. I've obviously considered wget as a strong contender, but I'd love to hear if anyone else has favourite tools for manually refreshing a stale cache. I thought I'd give a scenario where this might be useful, before people start asking why. I plan on having a "standby" varnish server that is geographically in a different data center than the primary varnish servers. I'd like the grace, and cache times to be significantly different than the live varnish servers. In the event of a disaster scenario in the live data center, I'd like to flip the switch to point to the geographically different varnish server, and serve the stale cache for however long it takes the live data center to come back. The challenge (not really a hard one mind you) would be to regularly update the cache in said standby varnish server at scheduled intervals so that the stale cache is not so far out of date when / if I do need to flip that DNS switch. If anyone has dealt with this scenario specifically ,or just with crawling varnish in general, I'd like to hear anyone's thoughts! Thanks, Kevin http://blog.stardothosting.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From Harri.Paivaniemi at tieto.com Sat Aug 13 05:11:56 2011 From: Harri.Paivaniemi at tieto.com (Harri.Paivaniemi at tieto.com) Date: Sat, 13 Aug 2011 08:11:56 +0300 Subject: How to maintain virtualhost name in backend requests correctly Message-ID: Hi, How can I tell varnish to do the backend request using the correct virtual host so that in backend, requests made by varnish are logged to the right virtual host log? In my backend there is 15 namevirtualhosts and now every req varnish makes comes to default apache-log, obviously because the req is not coming using name but a backend IP... I tought "set req.http.Host = "dev.foo.fi";" would do the trick but not... Thanks. -hjp --------------------------------------------------- if (req.http.Host ~ "dev\.foo\.fi") { set req.http.Host = "dev.foo.fi"; .................. set req.backend = xxxxx_http; } } backend xxxxx_http { .host = "xx.xx.xx.xx"; .port = "80"; .probe = { .request = "GET /crossdomain.xml HTTP/1.1" "Host: dev.foo.fi" "Connection: Close" "X-SE-CustomAuth: fffdfdfefrerefdfddfggfsul"; .interval = 5 s; .timeout = 30 s; .window = 3; .threshold = 2; } .connect_timeout = 15s; .first_byte_timeout = 60s; .between_bytes_timeout = 60s; } From mattias at nucleus.be Sun Aug 14 11:28:50 2011 From: mattias at nucleus.be (Mattias Geniar) Date: Sun, 14 Aug 2011 13:28:50 +0200 Subject: manually refreshing varnish cache In-Reply-To: <012b01cc5915$73dcfeb0$5b96fc10$@com> References: <012b01cc5915$73dcfeb0$5b96fc10$@com> Message-ID: <18834F5BEC10824891FB8B22AC821A5A019831BE@nucleus-srv01.Nucleus.local> > The challenge (not really a hard one mind you) would be to regularly update the > cache in said standby varnish server at scheduled intervals so that the stale cache > is not so far out of date when / if I do need to flip that DNS switch. > If anyone has dealt with this scenario specifically ,or just with crawling varnish in > general, I'd like to hear anyone's thoughts! Hi Kevin, You may want to look into the "varnishreplay" command: https://www.varnish-cache.org/docs/2.1/reference/varnishreplay.html You could restart your Varnish on the standby side, run varnishreplay to cache all the URLs you want with a TTL of > 1 day to keep it in cache. Regards, Mattias From mattias at nucleus.be Sun Aug 14 11:32:48 2011 From: mattias at nucleus.be (Mattias Geniar) Date: Sun, 14 Aug 2011 13:32:48 +0200 Subject: How to maintain virtualhost name in backend requests correctly In-Reply-To: References: Message-ID: <18834F5BEC10824891FB8B22AC821A5A019831BF@nucleus-srv01.Nucleus.local> > How can I tell varnish to do the backend request using the correct virtual host > so that in backend, requests made by varnish are logged to the right virtual host > log? In my backend there is 15 namevirtualhosts and now every req varnish makes > comes to default apache-log, obviously because the req is not coming using name > but a backend IP... > > I tought "set req.http.Host = "dev.foo.fi";" would do the trick but not... This doesn't make sense. Varnish will keep the original host-header intact, so your backend should see the request coming correctly. It may be that your Virtual Host is bound on the wrong IP and thus loading its configuration from the wrong virtual host? To debug, place a simple PHP script in (one of) your virtual host(s) that simply does this: That will tell you the host name, server port, ... that your backend actually gets to see (assuming PHP is enabled on your system). Regards, Mattias From jaka at celtra.com Sun Aug 14 15:29:42 2011 From: jaka at celtra.com (=?utf-8?Q?Jaka_Jan=C4=8Dar?=) Date: Sun, 14 Aug 2011 17:29:42 +0200 Subject: Increase delay for "deduplicating" requests to backend Message-ID: Hi, If varnish receives 2 requests for the same url, the second one will wait for the first one to finish (which makes sense). However, if the first one doesn't finish in 60s, the second one will hit the backend server either way. Is there a way to increase this time? Thanks, Jaka -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfheen at varnish-software.com Mon Aug 15 05:20:10 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Mon, 15 Aug 2011 07:20:10 +0200 Subject: no caching of large object In-Reply-To: <94B46DAE54D73F4BB2E5D300892C13DD56093C658E@QEO40072.de.t-online.corp> (Christian Lukas's message of "Tue, 9 Aug 2011 17:19:58 +0200") References: <94B46DAE54D73F4BB2E5D300892C13DD56093C658E@QEO40072.de.t-online.corp> Message-ID: <877h6fmet1.fsf@qurzaw.varnish-software.com> ]] "Lukas, Christian (extern)" | we like to avoid large objects to be cached. | I found the configuration below on http://comments.gmane.org/gmane.comp.web.varnish.misc/4741 . | If I got it right the object will be fetched from the backend twice. It depends on your Varnish version. In newer versions, it won't be fetched twice. Cheers, -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From straightflush at gmail.com Mon Aug 15 11:02:20 2011 From: straightflush at gmail.com (AD) Date: Mon, 15 Aug 2011 07:02:20 -0400 Subject: no caching of large object In-Reply-To: <877h6fmet1.fsf@qurzaw.varnish-software.com> References: <94B46DAE54D73F4BB2E5D300892C13DD56093C658E@QEO40072.de.t-online.corp> <877h6fmet1.fsf@qurzaw.varnish-software.com> Message-ID: So does that mean the same recipe holds for 3.0 just with the improvement that the restart does not cause a second fetch ? Or is there a better recipe now for piping large files ? On Mon, Aug 15, 2011 at 1:20 AM, Tollef Fog Heen < tfheen at varnish-software.com> wrote: > ]] "Lukas, Christian (extern)" > > | we like to avoid large objects to be cached. > | I found the configuration below on > http://comments.gmane.org/gmane.comp.web.varnish.misc/4741 . > | If I got it right the object will be fetched from the backend twice. > > It depends on your Varnish version. In newer versions, it won't be > fetched twice. > > Cheers, > -- > Tollef Fog Heen > Varnish Software > t: +47 21 98 92 64 > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dami.onajole at largeblue.com Mon Aug 15 13:37:12 2011 From: dami.onajole at largeblue.com (Dami Onajole) Date: Mon, 15 Aug 2011 14:37:12 +0100 Subject: Varnish Caching ESI Fragment In-Reply-To: References: Message-ID: Hi All, I am having problem using ESI with Varnish 3.0 successfully. I have an ESI fragment that displays a notice for logged in users, included via an ESI tag. However, varnish caches the included ESI fragment, so changes made to the fragment, either manually, or as a result of a session does not reflect in the including (and cached) web page. ESI tag as used: VCL configuration: sub vcl_recv { if (req.url ~ "[A-Za-z0-9_-]*.esi$") { return (pass); } if (req.http.cookie) { set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", ""); if (req.http.cookie ~ "^ *$") { remove req.http.cookie; } } remove req.http.cookie; remove req.http.etag; } sub vcl_fetch { remove beresp.http.Set-Cookie; remove beresp.http.ETag; #do esi processing set beresp.do_esi = true; if (bereq.url ~ "[A-Za-z0-9_-]*.esi$") { set beresp.ttl = 0s; } else { set beresp.ttl = 24h; } } My assumptions are: 1) Varnish will make requests to "path/to/fragment.esi" and the re-assemble cache web page every time it fetches from its cache store, especially since beresp.ttl is set to 0s for every .esi fragments 2) Varnish does not store the ESI fragments together with a web page in its cache store -- //////////////////////////////////////////////////////////////////// *Damilare Onajole* Python Developer *Large Blue* 120 Long Acre, Covent Garden, London. WC2E 9PA. www.largeblue.com +44 (0)20 7240 0330 //////////////////////////////////////////////////////////////////// Large Blue is a committed Carbon Neutral company. Please consider the environment before printing this email -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexus at gmail.com Mon Aug 15 17:20:11 2011 From: alexus at gmail.com (alexus) Date: Mon, 15 Aug 2011 13:20:11 -0400 Subject: 2.1.x -> 3.x Message-ID: I'm trying to upgrade our varnish from 2.x to 3.x Mar 22 06:10:04 Installed: varnish-libs-2.1.5-1.x86_64 Mar 22 06:10:09 Installed: varnish-2.1.5-1.x86_64 May 04 08:47:17 Updated: varnish-libs-2.1.5-1.el6.x86_64 May 04 08:47:50 Updated: varnish-2.1.5-1.el6.x86_64 Aug 10 12:08:18 Installed: varnish-libs-3.0.0-2.el5.x86_64 Aug 10 12:08:20 Installed: varnish-3.0.0-2.el5.x86_64 installed packages [root at 6prx01 ~]# service varnish start Starting Varnish Cache: [FAILED] [root at 6prx01 ~]# it used to work fine w/ 2.x branch [root at 6prx01 ~]# sh -x /etc/init.d/varnish start + . /etc/init.d/functions ++ TEXTDOMAIN=initscripts ++ umask 022 ++ PATH=/sbin:/usr/sbin:/bin:/usr/bin ++ export PATH ++ '[' -z '' ']' ++ COLUMNS=80 ++ '[' -z '' ']' +++ /sbin/consoletype ++ CONSOLETYPE=pty ++ '[' -f /etc/sysconfig/i18n -a -z '' -a -z '' ']' ++ . /etc/profile.d/lang.sh ++ unset LANGSH_SOURCED ++ '[' -z '' ']' ++ '[' -f /etc/sysconfig/init ']' ++ . /etc/sysconfig/init +++ BOOTUP=color +++ RES_COL=60 +++ MOVE_TO_COL='echo -en \033[60G' +++ SETCOLOR_SUCCESS='echo -en \033[0;32m' +++ SETCOLOR_FAILURE='echo -en \033[0;31m' +++ SETCOLOR_WARNING='echo -en \033[0;33m' +++ SETCOLOR_NORMAL='echo -en \033[0;39m' +++ PROMPT=yes +++ AUTOSWAP=no +++ ACTIVE_CONSOLES='/dev/tty[1-6]' +++ SINGLE=/sbin/sushell ++ '[' pty = serial ']' ++ __sed_discard_ignored_files='/\(~\|\.bak\|\.orig\|\.rpmnew\|\.rpmorig\|\.rpmsave\)$/d' + retval=0 + pidfile=/var/run/varnish.pid + exec=/usr/sbin/varnishd + reload_exec=/usr/bin/varnish_reload_vcl + prog=varnishd + config=/etc/sysconfig/varnish + lockfile=/var/lock/subsys/varnish + '[' -e /etc/sysconfig/varnish ']' + . /etc/sysconfig/varnish ++ NFILES=131072 ++ MEMLOCK=82000 ++ RELOAD_VCL=1 ++ VARNISH_VCL_CONF=/etc/varnish/default.vcl ++ VARNISH_LISTEN_PORT=6081 ++ VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1 ++ VARNISH_ADMIN_LISTEN_PORT=6082 ++ VARNISH_SECRET_FILE=/etc/varnish/secret ++ VARNISH_MIN_THREADS=1 ++ VARNISH_MAX_THREADS=1000 ++ VARNISH_THREAD_TIMEOUT=120 ++ VARNISH_STORAGE_FILE=/var/lib/varnish/varnish_storage.bin ++ VARNISH_STORAGE_SIZE=1G ++ VARNISH_STORAGE=file,/var/lib/varnish/varnish_storage.bin,1G ++ VARNISH_TTL=120 ++ DAEMON_OPTS='-a :6081 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 120 -w 1,1000,120 -u varnish -g varnish -S /etc/varnish/secret -s file,/var/lib/varnish/varnish_storage.bin,1G' + case "$1" in + rh_status_q + rh_status + start + '[' '!' -x /usr/sbin/varnishd ']' + '[' '!' -f /etc/sysconfig/varnish ']' + echo -n 'Starting Varnish Cache: ' Starting Varnish Cache: + ulimit -n 131072 + ulimit -l 82000 + '[' '-a :6081 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 120 -w 1,1000,120 -u varnish -g varnish -S /etc/varnish/secret -s file,/var/lib/varnish/varnish_storage.bin,1G' = '' ']' + daemon /usr/sbin/varnishd -P /var/run/varnish.pid '-a :6081 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 120 -w 1,1000,120 -u varnish -g varnish -S /etc/varnish/secret -s file,/var/lib/varnish/varnish_storage.bin,1G' + retval=1 + '[' 1 -eq 0 ']' + echo_failure + '[' color = color ']' + echo -en '\033[60G' + echo -n '[' [+ '[' color = color ']' + echo -en '\033[0;31m' + echo -n FAILED FAILED+ '[' color = color ']' + echo -en '\033[0;39m' + echo -n ']' ]+ echo -ne '\r' + return 1 + echo + return 1 + exit 1 [root at 6prx01 ~]# not really sure where/how to look where the problem is... -- http://alexus.org/ From dami.onajole at largeblue.com Mon Aug 15 13:34:27 2011 From: dami.onajole at largeblue.com (Dami Onajole) Date: Mon, 15 Aug 2011 14:34:27 +0100 Subject: Varnish Caching ESI Fragment Message-ID: Hi All, I am having problem using ESI with Varnish 3.0 successfully. I have an ESI fragment that displays a notice for logged in users, included via an ESI tag. However, varnish caches the included ESI fragment, so changes made to the fragment, either manually, or as a result of a session does not reflect in the including (and cached) web page. ESI tag as used: VCL configuration: sub vcl_recv { if (req.url ~ "[A-Za-z0-9_-]*.esi$") { return (pass); } if (req.http.cookie) { set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", ""); if (req.http.cookie ~ "^ *$") { remove req.http.cookie; } } remove req.http.cookie; remove req.http.etag; } sub vcl_fetch { remove beresp.http.Set-Cookie; remove beresp.http.ETag; #do esi processing set beresp.do_esi = true; if (bereq.url ~ "[A-Za-z0-9_-]*.esi$") { set beresp.ttl = 0s; } else { set beresp.ttl = 24h; } } My assumptions are: 1) Varnish will make requests to "path/to/fragment.esi" and the re-assemble cache web page every time it fetches from its cache store, especially since beresp.ttl is set to 0s for every .esi fragments 2) Varnish does not store the ESI fragments together with a web page in its cache store -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfheen at varnish-software.com Tue Aug 16 05:33:01 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 16 Aug 2011 07:33:01 +0200 Subject: 2.1.x -> 3.x In-Reply-To: (alexus@gmail.com's message of "Mon, 15 Aug 2011 13:20:11 -0400") References: Message-ID: <87liutly42.fsf@qurzaw.varnish-software.com> ]] alexus | I'm trying to upgrade our varnish from 2.x to 3.x [...] | it used to work fine w/ 2.x branch Have you seen https://www.varnish-cache.org/docs/trunk/installation/upgrade.html and changed your configuration accordingly? -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From cosimo at streppone.it Tue Aug 16 07:57:16 2011 From: cosimo at streppone.it (Cosimo Streppone) Date: Tue, 16 Aug 2011 09:57:16 +0200 Subject: [PATCH] String concatenation operator in doc/sphinx/installation/upgrade.rst Message-ID: Here's a trivial documentation patch against current master that makes the string concatenation operator change more evident, at least to me. I had missed it in a quick scan of the document. Cheers, --------------------8<-------------------------- diff --git a/doc/sphinx/installation/upgrade.rst b/doc/sphinx/installation/upgrade.rst index 1a31733..aef903f 100644 --- a/doc/sphinx/installation/upgrade.rst +++ b/doc/sphinx/installation/upgrade.rst @@ -9,6 +9,9 @@ Changes to VCL In most cases you need to update your VCL since there has been some changes to the syntax. +string concatenation operator +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + String concatenation did not have an operator previously, but this has now been changed to ``+``. ``log`` moved to the std vmod --------------------8<-------------------------- -- Cosimo -------------- next part -------------- A non-text attachment was scrubbed... Name: string-concat-clarification.patch Type: application/octet-stream Size: 524 bytes Desc: not available URL: From brad at teambrad.net Tue Aug 16 09:03:37 2011 From: brad at teambrad.net (Bradley Falzon) Date: Tue, 16 Aug 2011 18:33:37 +0930 Subject: 503 - Resource temporarily unavailable - When Varnish Connects to Itself Message-ID: Hi, We're having a bizarre issues with Varnish and 503 errors, when Varnish calls a Nginx when then calls its own Varnish. This may all sound like a bad setup, but it came from the fact during development we need to soap to a web service which currently resides on the same development server. The script we're calling is called php2.php, the file that is calling is a different file: brad.css, so it's not looping on itself. Connection sequence: 1. HTTP GET to Varnish 192.168.1.9:80 2. Connection to Nginx:8080 3. Connection to PHP FastCGI (php2.php does a file_get_contents(" http://192.168.1.9/members/css/brad.css") - http://php.net/file-get-contents ) 4. Connection to Varnish:80 5. Connection to Nginx:8080 Relevant part of the log (rest of the logs, success: http://dl.dropbox.com/u/3014365/varnishLogSuccess.txt fail: http://dl.dropbox.com/u/3014365/varnishLogFail.txt): 12 RxHeader c Cache-Control: max-age=0 12 VCL_call c recv lookup 12 VCL_call c hash 12 Hash c /retail/php2.php 12 Hash c 192.168.1.9 12 VCL_return c hash 12 VCL_call c miss fetch 12 Backend c 14 default default 12 FetchError c http first read error: -1 11 (Resource temporarily unavailable) 12 VCL_call c error deliver 12 VCL_call c deliver deliver 12 TxProtocol c HTTP/1.1 12 TxStatus c 503 12 TxResponse c Service Unavailable 12 TxHeader c Server: Varnish Varnish configuration fails in the default.vcl supplied (only changes were removing comments from all lines). Server Varnish 3.0.0 (Port 80), Nginx 1.0.5 (Port 8080), PHP 5.3.2 (FastCGI), CentOS 6.0, all running on VMWare Fusion. Client 192.168.1.14 (Firefox 5.0.1) to server 192.168.1.9. Nginx Logs: 127.0.0.1 - - [16/Aug/2011:17:47:02 +0930] "GET /retail/php2.php HTTP/1.1" 499 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:5.0.1) Gecko/20100101 Firefox/5.0.1" 127.0.0.1 - - [16/Aug/2011:17:47:02 +0930] "GET /members/css/brad.css HTTP/1.1" 200 3 "-" "-" Notice the GET responds with a 200 OK, according to Nginx it responded fine to the GET, BUT it doesn't log that until Varnish finally ends with a 503 (when Varnish finally closed the connection?). The 499 response is Nginx's way of saying "Client pre-maturely closed connection" Other notes: * Disabled keep-alive on nginx * Disabled gzip on nginx * Varnish default.vcl: http://dl.dropbox.com/u/3014365/varnishDefault.vcl.txt * Varnish sysconfig: http://dl.dropbox.com/u/3014365/varnishSysconfig.txt * Issue seems to be intermittent, it continues to fail give it a restart and it's ok again, restart again and it beings to fail all over again. Restart sometimes it might work again or continue to fail etc etc. No noticeable pattern. * I haven't 'actually' done a 'make install' yet, I'm just running it out of /usr/local/varnish/bin/varnishd/.libs/lt-varnishd * The fault doesn't happen when the PHP file_get_contents connects to a remove Nginx server * The fault doesn't happen when I connect directly to Nginx (ie, skip previously noted step 1 and connect to step 2 directly). * Varnish logs in different orders between fails and successes (of 6 tests - 3 success 3 fail). ** During a fail, the first log is the TxURL b /retail/php2.php ** During a success, the first log is TxURL b /members/css/brad.css Any thoughts ? -- Bradley Falzon brad at teambrad.net -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexus at gmail.com Tue Aug 16 13:52:39 2011 From: alexus at gmail.com (alexus) Date: Tue, 16 Aug 2011 09:52:39 -0400 Subject: 2.1.x -> 3.x In-Reply-To: <87liutly42.fsf@qurzaw.varnish-software.com> References: <87liutly42.fsf@qurzaw.varnish-software.com> Message-ID: yes, I did found this documents and I've made a lot of changes to my config and now varnish starts, yet I'm getting "The service is not available. Please try again later." whenever I tried to access my website... through "varnishlog" I keep seeing this (no matter what I do) 0 CLI - Rd ping 0 CLI - Wr 200 19 PONG 1313488237 1.0 from that environment I've tried to reach out to my backends, no problem but varnish somehow i guess isn't able or something else is wrong and I can't figure out what it is... On Tue, Aug 16, 2011 at 1:33 AM, Tollef Fog Heen wrote: > ]] alexus > > | I'm trying to upgrade our varnish from 2.x to 3.x > > [...] > > | it used to work fine w/ 2.x branch > > Have you seen > https://www.varnish-cache.org/docs/trunk/installation/upgrade.html and > changed your configuration accordingly? > > -- > Tollef Fog Heen > Varnish Software > t: +47 21 98 92 64 > -- http://alexus.org/ From alexus at gmail.com Tue Aug 16 14:13:00 2011 From: alexus at gmail.com (alexus) Date: Tue, 16 Aug 2011 10:13:00 -0400 Subject: 2.1.x -> 3.x In-Reply-To: References: <87liutly42.fsf@qurzaw.varnish-software.com> Message-ID: sorry guys since it was fresh install of varnish on new VM it by default bounded to another port other then I was hitting... but now I rebounded it to right port and it's all working :) On Tue, Aug 16, 2011 at 9:52 AM, alexus wrote: > yes, I did found this documents and I've made a lot of changes to my > config and now varnish starts, yet I'm getting > > "The service is not available. Please try again later." > > whenever I tried to access my website... > > through "varnishlog" I keep seeing this (no matter what I do) > > ? ?0 CLI ? ? ? ? ?- Rd ping > ? ?0 CLI ? ? ? ? ?- Wr 200 19 PONG 1313488237 1.0 > > from that environment I've tried to reach out to my backends, no problem > but varnish somehow i guess isn't able or something else is wrong and > I can't figure out what it is... > > > > On Tue, Aug 16, 2011 at 1:33 AM, Tollef Fog Heen > wrote: >> ]] alexus >> >> | I'm trying to upgrade our varnish from 2.x to 3.x >> >> [...] >> >> | it used to work fine w/ 2.x branch >> >> Have you seen >> https://www.varnish-cache.org/docs/trunk/installation/upgrade.html and >> changed your configuration accordingly? >> >> -- >> Tollef Fog Heen >> Varnish Software >> t: +47 21 98 92 64 >> > > > > -- > http://alexus.org/ > -- http://alexus.org/ From brad at teambrad.net Wed Aug 17 07:44:22 2011 From: brad at teambrad.net (Bradley Falzon) Date: Wed, 17 Aug 2011 17:14:22 +0930 Subject: 503 - Resource temporarily unavailable - When Varnish Connects to Itself In-Reply-To: References: Message-ID: It appears this was simply a configuration issue. The default setting of min threads was 1, changing this to a higher setting (2 was enough) appears to have resolved the issue. The maximum was set to 1000, I never tested this configuration before because I guessed Varnish would simply start an extra thread. In other words, the command line parameter from "-w 1,1000,120" to "-w 2,1000,120" was enough to stop the 'Resource temporarily unavailable messages'. We'll obviously adjust this configuration further as required. Thanks -- Bradley Falzon brad at teambrad.net From pax at businessinsider.com Wed Aug 17 18:38:16 2011 From: pax at businessinsider.com (Pax Dickinson) Date: Wed, 17 Aug 2011 14:38:16 -0400 Subject: Using Varnish as a backend for another Varnish server, with ESIs Message-ID: Hi, I'm attempting to use two varnish servers as the preferred caching backend for each other, falling back to my apache backends if neither varnish server has the content in cache. I'm doing it to reduce my backend traffic and keep my caches hot more efficiently than having each varnish server totally separate, which is what I do now. Also this will scale better if the day comes that I need a third server. I match against an ACL and if a request comes from anywhere other than my other varnish server, I set the varnish backend. This is working fine, except my edge side includes need to be processed only by the server that received the original request. If the secondary server were to process and include my ESIs at that point they would be returned to the primary server and become part of its cached copy, and that defeats the purpose of having ESIs. It seems that it should be as simple as: sub vcl_recv { if (!client.ip ~ othervarnish) { set req.backend = varnish; } } sub vcl_fetch { if (beresp.http.X-Meta-Has-Esi && !client.ip ~ othervarnish) { esi; } } BUT.. If I call it as above, ESIs work and are processed correctly by whichever server received the request first. However, when hitting the secondary server and getting a cached result the ESIs are not processed, vcl_fetch is never called, and the esi tags remain in the result. Also, it seems that the esi function can only be called in vcl_fetch. In vcl_deliver it does nothing. So for example: The initial request works: Request ---> VarnishA MISS --> VarnishB MISS --> Backend (success with esi parsed) However subsequent requests look like this: New Request ---> VarnishA HIT (success with esi parsed) New Request ---> VarnishB HIT (fail, esi will not be parsed) Is there a way to make this work? Or is using varnish as a backend for another varnish incompatible with edge side includes? I'm using varnish 2.1.5, is this possible in 3.0? Thanks.. -- Pax Dickinson Chief Architect Business Insider 257 Park Avenue South, 13th floor New York, NY 10010 pax at businessinsider.com From akolychev at marthastewart.com Wed Aug 17 19:51:42 2011 From: akolychev at marthastewart.com (Kolychev, Aleksey) Date: Wed, 17 Aug 2011 15:51:42 -0400 Subject: Using Varnish as a backend for another Varnish server, with ESIs In-Reply-To: References: Message-ID: Are you load-balancing between the two varnish instances? If you are, you should check if you're hashing the cache keys properly, e.g., not sending the same response to the other varnish instance that you'd send to all other clients, so that your outer varnish instance doesn't serve cached pages with unparsed ESI tags to clients. You could try doing something along the lines of - sub vcl_recv { if(client.ip ~ other-varnish) { set req.http.X-Via-Varnish = 1; } } sub vcl_hash { set req.hash += req.http.X-Via-Varnish; } sub vcl_fetch { if (beresp.http.X-Meta-Has-Esi) { unset beresp.http.X-Meta-Has-Esi; } else { esi; } } (Disclaimer - I didn't test this, and cannot be held responsible for syntax errors) -- Lex Kolychev Application and Systems Architect Internet Technology Dept. Martha Stewart Living Omnimedia Ph: 212-827-8948 From: Pax Dickinson > Date: August 17, 2011 2:38:16 PM EDT To: "varnish-misc at varnish-cache.org" > Subject: Using Varnish as a backend for another Varnish server, with ESIs Hi, I'm attempting to use two varnish servers as the preferred caching backend for each other, falling back to my apache backends if neither varnish server has the content in cache. I'm doing it to reduce my backend traffic and keep my caches hot more efficiently than having each varnish server totally separate, which is what I do now. Also this will scale better if the day comes that I need a third server. I match against an ACL and if a request comes from anywhere other than my other varnish server, I set the varnish backend. This is working fine, except my edge side includes need to be processed only by the server that received the original request. If the secondary server were to process and include my ESIs at that point they would be returned to the primary server and become part of its cached copy, and that defeats the purpose of having ESIs. It seems that it should be as simple as: sub vcl_recv { if (!client.ip ~ othervarnish) { set req.backend = varnish; } } sub vcl_fetch { if (beresp.http.X-Meta-Has-Esi && !client.ip ~ othervarnish) { esi; } } BUT.. If I call it as above, ESIs work and are processed correctly by whichever server received the request first. However, when hitting the secondary server and getting a cached result the ESIs are not processed, vcl_fetch is never called, and the esi tags remain in the result. Also, it seems that the esi function can only be called in vcl_fetch. In vcl_deliver it does nothing. So for example: The initial request works: Request ---> VarnishA MISS --> VarnishB MISS --> Backend (success with esi parsed) However subsequent requests look like this: New Request ---> VarnishA HIT (success with esi parsed) New Request ---> VarnishB HIT (fail, esi will not be parsed) Is there a way to make this work? Or is using varnish as a backend for another varnish incompatible with edge side includes? I'm using varnish 2.1.5, is this possible in 3.0? Thanks.. -- Pax Dickinson Chief Architect Business Insider 257 Park Avenue South, 13th floor New York, NY 10010 pax at businessinsider.com _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc ________________________________ Don't miss a brand new season of the Emmy award-winning "Martha Stewart Show", followed by the all-new series "Emeril's Table"...Beginning September 26, only on Hallmark Channel. Find Martha Stewart on Facebook: http://facebook.com/MarthaStewartLiving Follow Martha Stewart on Twitter: http://twitter.com/MarthaStewart -------------- next part -------------- An HTML attachment was scrubbed... URL: From straightflush at gmail.com Wed Aug 17 21:29:52 2011 From: straightflush at gmail.com (AD) Date: Wed, 17 Aug 2011 17:29:52 -0400 Subject: Using Varnish as a backend for another Varnish server, with ESIs In-Reply-To: References: Message-ID: I would personally solve this another way as I dont think its worth all this pain just to save 1 extra hit. What i would suggest is you use a "hash" algorithm on your load balancer that sends the same URL to the same varnish server, might be easier. On Wed, Aug 17, 2011 at 3:51 PM, Kolychev, Aleksey < akolychev at marthastewart.com> wrote: > Are you load-balancing between the two varnish instances? If you are, you > should check if you?re hashing the cache keys properly, e.g., not sending > the same response to the other varnish instance that you?d send to all other > clients, so that your outer varnish instance doesn?t serve cached pages with > unparsed ESI tags to clients.**** > > ** ** > > You could try doing something along the lines of ?**** > > ** ** > > sub vcl_recv {**** > > if(client.ip ~ other-varnish) {**** > > set req.http.X-Via-Varnish = 1;**** > > }**** > > }**** > > ** ** > > sub vcl_hash {**** > > set req.hash += req.http.X-Via-Varnish;**** > > }**** > > ** ** > > sub vcl_fetch { > if (beresp.http.X-Meta-Has-Esi) { > unset beresp.http.X-Meta-Has-Esi; > } else {**** > > esi;**** > > } > }**** > > ** ** > > (Disclaimer ? I didn?t test this, and cannot be held responsible for syntax > errors)**** > > ** ** > > --**** > > Lex Kolychev**** > > Application and Systems Architect**** > > Internet Technology Dept.**** > > Martha Stewart Living Omnimedia > Ph: 212-827-8948**** > > > > **** > > *From: *Pax Dickinson **** > > *Date: *August 17, 2011 2:38:16 PM EDT**** > > *To: *"varnish-misc at varnish-cache.org" *** > * > > *Subject: Using Varnish as a backend for another Varnish server, with ESIs > ***** > > ** ** > > Hi, > > I'm attempting to use two varnish servers as the preferred caching > backend for each other, falling back to my apache backends if neither > varnish server has the content in cache. I'm doing it to reduce my > backend traffic and keep my caches hot more efficiently than having > each varnish server totally separate, which is what I do now. Also > this will scale better if the day comes that I need a third server. > > I match against an ACL and if a request comes from anywhere other than > my other varnish server, I set the varnish backend. This is working > fine, except my edge side includes need to be processed only by the > server that received the original request. If the secondary server > were to process and include my ESIs at that point they would be > returned to the primary server and become part of its cached copy, and > that defeats the purpose of having ESIs. > > It seems that it should be as simple as: > > sub vcl_recv { > if (!client.ip ~ othervarnish) { > set req.backend = varnish; > } > } > > sub vcl_fetch { > if (beresp.http.X-Meta-Has-Esi && !client.ip ~ othervarnish) { > esi; > } > } > > BUT.. If I call it as above, ESIs work and are processed correctly by > whichever server received the request first. However, when hitting the > secondary server and getting a cached result the ESIs are not > processed, vcl_fetch is never called, and the esi tags remain in the > result. Also, it seems that the esi function can only be called in > vcl_fetch. In vcl_deliver it does nothing. > > So for example: > > The initial request works: > > Request ---> VarnishA MISS --> VarnishB MISS --> Backend (success with > esi parsed) > > However subsequent requests look like this: > > New Request ---> VarnishA HIT (success with esi parsed) > New Request ---> VarnishB HIT (fail, esi will not be parsed) > > Is there a way to make this work? Or is using varnish as a backend for > another varnish incompatible with edge side includes? I'm using > varnish 2.1.5, is this possible in 3.0? > > Thanks.. > > -- > Pax Dickinson > Chief Architect > Business Insider > 257 Park Avenue South, 13th floor > New York, NY 10010 > pax at businessinsider.com > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc**** > > ** ** > > ------------------------------ > Don't miss a brand new season of the Emmy award-winning "Martha Stewart > Show", followed by the all-new series "Emeril's Table"...Beginning September > 26, only on Hallmark Channel. > > Find Martha Stewart on Facebook: http://facebook.com/MarthaStewartLiving > > Follow Martha Stewart on Twitter: http://twitter.com/MarthaStewart > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sime at sime.net.au Wed Aug 17 23:31:09 2011 From: sime at sime.net.au (Simon Males) Date: Thu, 18 Aug 2011 09:31:09 +1000 Subject: Reducing overflowed and dropped work requests In-Reply-To: <87bow7km9x.fsf@qurzaw.varnish-software.com> References: <87bow7km9x.fsf@qurzaw.varnish-software.com> Message-ID: On Wed, Aug 3, 2011 at 5:05 PM, Tollef Fog Heen wrote: > ]] Simon Males > > | Since this post, I increased the thread_pools to 8, though today I've > | had another spurt of dropped work requests. It's definitely related to > | a sudden rise of traffic, as the dropped requests happen moment after > | sending an E-Mail to our users. > | > | The servers have 16 cores, so I've increased the thread_pools to 16 > | now. With 10 minimum threads per pool, I now have 160 threads waiting > | to serve. > > If you're on 2.1 you might want to set thread_pool_add_delay to 2 (ms) > or similar. ?It's set a bit too high there, which means it takes a > little while for Varnish to respond to rapid spikes. Just wanted to say thanks to the contributors to this thread V?clav B?lek and Tollef Fog Heen. I actually held back on the advice and wanted to see how far 160 minimum threads would get me (16 pools x 10 minimum). The next spike produced 320k limited worker threads (n_wrk_max) per server. I set the following parameters on the fly hoping to manage the spike. But the real test will be the next spike. thread_pool_add_delay 2 thread_pool_max 250 thread_pool_min 100 Though now my committed memory usage (according to Munin) is 16G. The server has a total of 4G and 2G malloc cache. Is there an equation I can use to determine how much memory I need with my current parameters? Additionally can I deduce from varnishstats how many threads are being used? e.g. How many of the minimum 1600 are being used. -- Simon Males From MSchurenko at airg.com Thu Aug 18 00:36:44 2011 From: MSchurenko at airg.com (Matt Schurenko) Date: Wed, 17 Aug 2011 17:36:44 -0700 Subject: file stoarage type Message-ID: <8B5C6F575422414AA91B46C454126B6C02EB281C4B@EXCHMVS.exchange.airg> I've read all the documentation I can get my hands on and still don't understand how the file Storage Type works. What does "Storage for each object is allocated from an arena backed by a file." really mean? For example, if I specify a size of 40GB how much physical RAM will varnish attempt to use? Will it try to use all available RAM and, if there's no more available, then use the file? Matt Schurenko -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfheen at varnish-software.com Thu Aug 18 06:18:56 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Thu, 18 Aug 2011 08:18:56 +0200 Subject: file stoarage type In-Reply-To: <8B5C6F575422414AA91B46C454126B6C02EB281C4B@EXCHMVS.exchange.airg> (Matt Schurenko's message of "Wed, 17 Aug 2011 17:36:44 -0700") References: <8B5C6F575422414AA91B46C454126B6C02EB281C4B@EXCHMVS.exchange.airg> Message-ID: <87bovnckdr.fsf@qurzaw.varnish-software.com> ]] Matt Schurenko | I've read all the documentation I can get my hands on and still don't | understand how the file Storage Type works. What does "Storage for | each object is allocated from an arena backed by a file." really mean? | For example, if I specify a size of 40GB how much physical RAM will | varnish attempt to use? Will it try to use all available RAM and, if | there's no more available, then use the file? The kernel manages the amount of physical memory to allocate as cache for the file, Varnish does not try to manage that explicitly. So, the answer is ?it depends?. (Varnish will use memory for threads and various other data structures, but the cache contents itself will be balanced between memory and disk by the kernel automatically.) Cheers, -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From humansky at Princeton.EDU Thu Aug 18 14:40:52 2011 From: humansky at Princeton.EDU (Henry M. Umansky) Date: Thu, 18 Aug 2011 14:40:52 +0000 Subject: HTTP 503 Service Unavailable errors Message-ID: <9BDF32C3-868E-4D2A-999B-BBE958C3FDAB@exchange.princeton.edu> Hello, I recently installed Varnish from the Redhat EPEL repository (Varnish 2.1.5-1.el6.x86_64) and I'm having a slight problem with load balancing. I currently have two backends: director default round-robin { { .backend = { .host = "webhead01.example.com"; .port = "http"; .connect_timeout = 1s; .first_byte_timeout = 5s; .between_bytes_timeout = 2s; } } { .backend = { .host = "webhead02.example.com"; .port = "http"; .connect_timeout = 1s; .first_byte_timeout = 5s; .between_bytes_timeout = 2s; } } } and when I shut down Apache on one of the backends and restart Varnish, I get "503 Service Unavailable" errors on random files (usually just one or two files). I never get this error when both backends are running. How does the Varnish probes work, do they ping the backend server to see if they are alive or actually attempt to connect to port 80 to see if Apache is alive? I'm relatively new to Varnish, so I want to apologize if I don't have all the information or missing a configuration somewhere. Henry Umansky Web Development Services Princeton University humansky at princeton.edu 609-258-1674 From MSchurenko at airg.com Thu Aug 18 16:33:39 2011 From: MSchurenko at airg.com (Matt Schurenko) Date: Thu, 18 Aug 2011 09:33:39 -0700 Subject: file stoarage type In-Reply-To: <87bovnckdr.fsf@qurzaw.varnish-software.com> References: <8B5C6F575422414AA91B46C454126B6C02EB281C4B@EXCHMVS.exchange.airg> <87bovnckdr.fsf@qurzaw.varnish-software.com> Message-ID: <8B5C6F575422414AA91B46C454126B6C02EB281C5C@EXCHMVS.exchange.airg> Thanks for answering my question Tollef. I was reading about the mmap system call in Linux last night so your answer makes some sense to me. :) Could you be so kind as to answer another question? I am wondering if I should use malloc or file for the storage type. I have read that if your cache is small enough to remain mostly in memory it is best to use malloc, but if it will exceed the amount of physical RAM you have available that mmap performs better than relying on your system's use of swap space. Is this accurate? Matt Schurenko Systems Administrator airG share your world Suite 710, 1133 Melville Street Vancouver, BC V6E 4E5 P: +1.604.408.2228 F: +1.866.874.8136 E: MSchurenko at airg.com W: www.airg.com -----Original Message----- From: Tollef Fog Heen [mailto:tfheen at varnish-software.com] Sent: August-17-11 11:19 PM To: Matt Schurenko Cc: varnish-misc at varnish-cache.org Subject: Re: file stoarage type ]] Matt Schurenko | I've read all the documentation I can get my hands on and still don't | understand how the file Storage Type works. What does "Storage for | each object is allocated from an arena backed by a file." really mean? | For example, if I specify a size of 40GB how much physical RAM will | varnish attempt to use? Will it try to use all available RAM and, if | there's no more available, then use the file? The kernel manages the amount of physical memory to allocate as cache for the file, Varnish does not try to manage that explicitly. So, the answer is ?it depends?. (Varnish will use memory for threads and various other data structures, but the cache contents itself will be balanced between memory and disk by the kernel automatically.) Cheers, -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From jcpetit at syspark.com Thu Aug 18 22:01:29 2011 From: jcpetit at syspark.com (Jean-Christophe Petit) Date: Thu, 18 Aug 2011 18:01:29 -0400 Subject: sending old content while fetching Message-ID: <4E4D8BB9.2030802@syspark.com> Hello, is there a way in varnish-3.x to serve old content while fetching new one ? The goal is to avoid slowdown because of a slow backend. Thank you, -- Jean-Christophe Petit GROUPE SYSPARK From perbu at varnish-software.com Fri Aug 19 07:03:10 2011 From: perbu at varnish-software.com (Per Buer) Date: Fri, 19 Aug 2011 09:03:10 +0200 Subject: sending old content while fetching In-Reply-To: <4E4D8BB9.2030802@syspark.com> References: <4E4D8BB9.2030802@syspark.com> Message-ID: Hi, On Fri, Aug 19, 2011 at 12:01 AM, Jean-Christophe Petit wrote: > Hello, > > is there a way in varnish-3.x to serve old content while fetching new one ? > The goal is to avoid slowdown because of a slow backend. > Yes. This functionality is called "grace". The one limitation that exists is that the client requesting the new content will be kept waiting. While that client is waiting all the others will get the new content. You can have a look at "man vcl" or the varnish tutorial ( https://www.varnish-cache.org/docs/trunk/tutorial/handling_misbehaving_servers.html) for some docs on how to do it. If you find any of the docs unclear, please let me know. -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfheen at varnish-software.com Fri Aug 19 07:30:44 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Fri, 19 Aug 2011 09:30:44 +0200 Subject: file stoarage type In-Reply-To: <8B5C6F575422414AA91B46C454126B6C02EB281C5C@EXCHMVS.exchange.airg> (Matt Schurenko's message of "Thu, 18 Aug 2011 09:33:39 -0700") References: <8B5C6F575422414AA91B46C454126B6C02EB281C4B@EXCHMVS.exchange.airg> <87bovnckdr.fsf@qurzaw.varnish-software.com> <8B5C6F575422414AA91B46C454126B6C02EB281C5C@EXCHMVS.exchange.airg> Message-ID: <878vqpc0yj.fsf@qurzaw.varnish-software.com> ]] Matt Schurenko | Could you be so kind as to answer another question? I am wondering if | I should use malloc or file for the storage type. I have read that if | your cache is small enough to remain mostly in memory it is best to | use malloc, but if it will exceed the amount of physical RAM you have | available that mmap performs better than relying on your system's use | of swap space. Is this accurate? Yes, it's generally accurate. Regards, -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From lajo at kb.dk Fri Aug 19 10:34:28 2011 From: lajo at kb.dk (=?iso-8859-1?Q?Lars_J=F8rgensen?=) Date: Fri, 19 Aug 2011 10:34:28 +0000 Subject: HTTP 503 Service Unavailable errors In-Reply-To: <9BDF32C3-868E-4D2A-999B-BBE958C3FDAB@exchange.princeton.edu> References: <9BDF32C3-868E-4D2A-999B-BBE958C3FDAB@exchange.princeton.edu> Message-ID: <6D2C830A0941EA40B6B483FE6EC98ADB1876F879@EXCHANGE-02.kb.dk> Hi Henry, Please have a look at https://www.varnish-cache.org/docs/2.1/tutorial/advanced_backend_servers.html#health-checks. Here you can explicitly configure how Varnish will check the health of each backend. Lars -----Original Message----- From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Henry M. Umansky Sent: Thursday, August 18, 2011 4:41 PM To: varnish-misc at varnish-cache.org Subject: HTTP 503 Service Unavailable errors Hello, I recently installed Varnish from the Redhat EPEL repository (Varnish 2.1.5-1.el6.x86_64) and I'm having a slight problem with load balancing. I currently have two backends: director default round-robin { { .backend = { .host = "webhead01.example.com"; .port = "http"; .connect_timeout = 1s; .first_byte_timeout = 5s; .between_bytes_timeout = 2s; } } { .backend = { .host = "webhead02.example.com"; .port = "http"; .connect_timeout = 1s; .first_byte_timeout = 5s; .between_bytes_timeout = 2s; } } } and when I shut down Apache on one of the backends and restart Varnish, I get "503 Service Unavailable" errors on random files (usually just one or two files). I never get this error when both backends are running. How does the Varnish probes work, do they ping the backend server to see if they are alive or actually attempt to connect to port 80 to see if Apache is alive? I'm relatively new to Varnish, so I want to apologize if I don't have all the information or missing a configuration somewhere. Henry Umansky Web Development Services Princeton University humansky at princeton.edu 609-258-1674 _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From k at kevinkevin.com Fri Aug 19 16:06:46 2011 From: k at kevinkevin.com (Kevin) Date: Fri, 19 Aug 2011 12:06:46 -0400 Subject: cookie matching + regex Message-ID: <00a801cc5e8a$004ac660$00e05320$@com> Hello, This might be more of a regular expression question , but I am trying to strip all cookies except ones with a prefix that matches. The cookie name would be something like this : Auth_cookie_asldkfjasdfkasdjflaksdjkjflj Where the last part of the cookie name would be a random MD5 hash that is generated differently for each session. If I wanted to match all cookies with "auth_cookie_" in the name (like auth_cookie_(.*) in a query such as this : set req.http.Cookie = regsuball(req.http.Cookie, ";(auth_cookie)=", "; \1="); How would I set it to match only the first part of the cookie that is non-random? Thanks in advance! -------------- next part -------------- An HTML attachment was scrubbed... URL: From whshub at gmail.com Fri Aug 19 23:33:18 2011 From: whshub at gmail.com (Jacques) Date: Fri, 19 Aug 2011 16:33:18 -0700 Subject: Tuncated Log Items Message-ID: Using Varnish 3.0 and varnishncsa I'm seeing truncated urls at 255 characters. I see this message from 4 years ago: http://www.mail-archive.com/varnish-misc at projects.linpro.no/msg00866.html Does Varnish now support longer values for the header/etc fields? If so, is there something I need to do to enable it? Thanks, Jacques -------------- next part -------------- An HTML attachment was scrubbed... URL: From bstillwell at photobucket.com Mon Aug 22 02:58:21 2011 From: bstillwell at photobucket.com (Bryan Stillwell) Date: Sun, 21 Aug 2011 20:58:21 -0600 Subject: Increasing transient memory usage Message-ID: I believe I'm running into the problem mentioned in this bug report: https://www.varnish-cache.org/trac/ticket/953 I've tried the following fix, but it doesn't appear to fix the problem: https://www.varnish-cache.org/trac/changeset/beb0c5b1f4f49d711822e90ca73d69bbed683a71 The behavior I'm seeing is that once SMA.s0.nbytes hits the number specified in '-s malloc,?G' the value of SMA.Transient.nbytes starts going up (it's at zero the whole time before hitting the '-s malloc,?G' number). I have two machines I'm testing with that both have 32G of memory (varnish 3.0.0 on centos 6.0), but for troubleshooting this problem I set one to use 4G and the other to 8G. The values after running for around an hour are: 4G test config: SMA.s0.nbytes 4294213238 . Outstanding bytes SMA.Transient.nbytes 3137682125 . Outstanding bytes 8G test config: SMA.s0.nbytes 8589339844 . Outstanding bytes SMA.Transient.nbytes 1260306015 . Outstanding bytes So the 4G config is using less memory after an hour, but still quite a bit more than the 20% that is recommended to leave for overhead (plus it continues to climb). Is anyone besides the people on that bug seeing this problem? Are there any other patches or workarounds? Thanks, Bryan From phk at phk.freebsd.dk Mon Aug 22 06:22:10 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 22 Aug 2011 06:22:10 +0000 Subject: Increasing transient memory usage In-Reply-To: Your message of "Sun, 21 Aug 2011 20:58:21 CST." Message-ID: <28078.1313994130@critter.freebsd.dk> In message , Bryan Stillwell writes: >I believe I'm running into the problem mentioned in this bug report: > >https://www.varnish-cache.org/trac/ticket/953 > >I've tried the following fix, but it doesn't appear to fix the problem: > >https://www.varnish-cache.org/trac/changeset/beb0c5b1f4f49d711822e90ca73d69bbed683a71 I'm still trying to find the root cause of this issue. The current theory is that the objects in the transient storage do not get expired, but I have not found out why. Any information you can get me in that direction would be helpful. In particular, please try out the -trunk version, which has a number of fixes which might bear on this issue. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Mon Aug 22 08:24:37 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 22 Aug 2011 08:24:37 +0000 Subject: Increasing transient memory usage In-Reply-To: Your message of "Mon, 22 Aug 2011 06:22:10 GMT." <28078.1313994130@critter.freebsd.dk> Message-ID: <99357.1314001477@critter.freebsd.dk> In message <28078.1313994130 at critter.freebsd.dk>, "Poul-Henning Kamp" writes: >In message >, Bryan Stillwell writes: > >>I believe I'm running into the problem mentioned in this bug report: >> >>https://www.varnish-cache.org/trac/ticket/953 >> >>I've tried the following fix, but it doesn't appear to fix the problem: >> >>https://www.varnish-cache.org/trac/changeset/beb0c5b1f4f49d711822e90ca73d69bbed683a71 > >I'm still trying to find the root cause of this issue. Update: I think I just nailed it, please try -trunk. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From tfheen at varnish-software.com Mon Aug 22 10:25:00 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Mon, 22 Aug 2011 12:25:00 +0200 Subject: Tuncated Log Items In-Reply-To: (Jacques's message of "Fri, 19 Aug 2011 16:33:18 -0700") References: Message-ID: <87k4a5g2v7.fsf@qurzaw.varnish-software.com> ]] Jacques | Does Varnish now support longer values for the header/etc fields? If so, is | there something I need to do to enable it? You need to change the shm_reclen parameter. -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From alexus at gmail.com Mon Aug 22 15:28:24 2011 From: alexus at gmail.com (alexus) Date: Mon, 22 Aug 2011 11:28:24 -0400 Subject: varnish saint-mode Message-ID: I'm trying to implement "saint mode" into my default.vcl, I copy and paste that little snippet that came with that documentation but now my varnish wont even start. Misbehaving servers ? Varnish version 3.0.0 documentation https://www.varnish-cache.org/docs/3.0/tutorial/handling_misbehaving_servers.html#saint-mode either I'm putting that into a wrong place or something else I dont know, all I know as soon as I comment it out, varnish starts Ok. any ideas how to troubleshoot it? -- http://alexus.org/ From bstillwell at photobucket.com Mon Aug 22 20:34:17 2011 From: bstillwell at photobucket.com (Bryan Stillwell) Date: Mon, 22 Aug 2011 14:34:17 -0600 Subject: Increasing transient memory usage In-Reply-To: <99357.1314001477@critter.freebsd.dk> References: <28078.1313994130@critter.freebsd.dk> <99357.1314001477@critter.freebsd.dk> Message-ID: That seems to have done the trick! I've only tested for about half an hour so far, but the memory usage is quite stable so far. Thanks! Bryan On Mon, Aug 22, 2011 at 2:24 AM, Poul-Henning Kamp wrote: > In message <28078.1313994130 at critter.freebsd.dk>, "Poul-Henning Kamp" writes: >>In message >>, Bryan Stillwell writes: >> >>>I believe I'm running into the problem mentioned in this bug report: >>> >>>https://www.varnish-cache.org/trac/ticket/953 >>> >>>I've tried the following fix, but it doesn't appear to fix the problem: >>> >>>https://www.varnish-cache.org/trac/changeset/beb0c5b1f4f49d711822e90ca73d69bbed683a71 >> >>I'm still trying to find the root cause of this issue. > > Update: I think I just nailed it, please try -trunk. > > -- > Poul-Henning Kamp ? ? ? | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG ? ? ? ? | TCP/IP since RFC 956 > FreeBSD committer ? ? ? | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > From hugo.cisneiros at gmail.com Mon Aug 22 22:27:53 2011 From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch)) Date: Mon, 22 Aug 2011 19:27:53 -0300 Subject: Increases in memory usage and dropped connections Message-ID: Hi, I use Varnish for at least one year and in the past days began to get really strange behavior (currently using 2.1.3). Sometimes I get some "Connection reset by peer" requesting pages. When this happens, I look at the stats and see something like these, for example: client_conn 1565694 187.17 Client connections accepted client_drop 271 0.03 Connection dropped, no sess/wrk client_req 2790588 333.60 Client requests received cache_hit 1813984 216.85 Cache hits cache_hitpass 0 0.00 Cache hits for pass cache_miss 546088 65.28 Cache misses backend_conn 32042 3.83 Backend conn. success backend_unhealthy 1351 0.16 Backend conn. not attempted backend_busy 0 0.00 Backend conn. too many backend_fail 1 0.00 Backend conn. failures backend_reuse 889261 106.31 Backend conn. reuses backend_toolate 157 0.02 Backend conn. was closed backend_recycle 889425 106.33 Backend conn. recycles backend_unused 0 0.00 Backend conn. unused fetch_head 759 0.09 Fetch head fetch_length 920288 110.02 Fetch with Length fetch_chunked 0 0.00 Fetch chunked fetch_eof 0 0.00 Fetch EOF fetch_bad 0 0.00 Fetch had bad headers fetch_close 0 0.00 Fetch wanted close fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed fetch_zero 0 0.00 Fetch zero len fetch_failed 0 0.00 Fetch failed n_sess_mem 65783 . N struct sess_mem n_sess 54012 . N struct sess n_object 217320 . N struct object n_vampireobject 0 . N unresurrected objects n_objectcore 218999 . N struct objectcore n_objecthead 57686 . N struct objecthead n_smf 0 . N struct smf n_smf_frag 0 . N small free smf n_smf_large 0 . N large free smf n_vbe_conn 28 . N struct vbe_conn n_wrk 2283 . N worker threads n_wrk_create 2326 0.28 N worker threads created n_wrk_failed 0 0.00 N worker threads not created n_wrk_max 0 0.00 N worker threads limited n_wrk_queue 0 0.00 N queued work requests n_wrk_overflow 25533 3.05 N overflowed work requests n_wrk_drop 286 0.03 N dropped work requests n_backend 14 . N backends n_expired 0 . N expired objects n_lru_nuked 157040 . N LRU nuked objects n_lru_saved 0 . N LRU saved objects n_lru_moved 74969 . N LRU moved objects n_deathrow 0 . N objects on deathrow losthdr 0 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 2494603 298.22 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 1565320 187.13 Total Sessions s_req 2790588 333.60 Total Requests s_pipe 0 0.00 Total pipe s_pass 376511 45.01 Total pass s_fetch 921039 110.11 Total fetch s_hdrbytes 844190338 100919.35 Total header bytes s_bodybytes 40983737008 4899430.60 Total body bytes sess_closed 125532 15.01 Session Closed sess_pipeline 788 0.09 Session Pipeline sess_readahead 810 0.10 Session Read Ahead sess_linger 2656690 317.60 Session Linger sess_herd 2874808 343.67 Session herd shm_records 146433038 17505.44 SHM records shm_writes 12159673 1453.64 SHM writes shm_flushes 0 0.00 SHM flushes due to overflow shm_cont 51273 6.13 SHM MTX contention shm_cycles 69 0.01 SHM cycles through buffer sm_nreq 0 0.00 allocator requests sm_nobj 0 . outstanding allocations sm_balloc 0 . bytes allocated sm_bfree 0 . bytes free sma_nreq 1440966 172.26 SMA allocator requests sma_nobj 433212 . SMA outstanding allocations sma_nbytes 5368687547 . SMA outstanding bytes sma_balloc 10635681699 . SMA bytes allocated sma_bfree 5266994152 . SMA bytes free sms_nreq 1206 0.14 SMS allocator requests sms_nobj 0 . SMS outstanding allocations sms_nbytes 0 . SMS outstanding bytes sms_balloc 472992 . SMS bytes allocated sms_bfree 472992 . SMS bytes freed backend_req 921115 110.12 Backend requests made n_vcl 1 0.00 N vcl total n_vcl_avail 1 0.00 N vcl available n_vcl_discard 0 0.00 N vcl discarded n_purge 1 . N total active purges n_purge_add 1 0.00 N new purges added n_purge_retire 0 0.00 N old purges deleted n_purge_obj_test 0 0.00 N objects tested n_purge_re_test 0 0.00 N regexps tested against n_purge_dups 0 0.00 N duplicate purges removed hcb_nolock 0 0.00 HCB Lookups without lock hcb_lock 0 0.00 HCB Lookups with lock hcb_insert 0 0.00 HCB Inserts esi_parse 0 0.00 Objects ESI parsed (unlock) esi_errors 0 0.00 ESI parse errors (unlock) accept_fail 0 0.00 Accept failures client_drop_late 15 0.00 Connection dropped late uptime 8365 1.00 Client uptime There are lots of worrisome stats like n_wrk (2283), n_wrk_overflow (25533), n_wrk_drop (286), client_drop (271) ... The number of active threads (2283) is very high for no apparent reason since the normal usage is like 300 threads or so. The server has 7.5GB RAM and is using malloc with 5GB size, running only a Varnish instance on an Ubuntu Server 10.10. Tried going to 3GB malloc size, increasing thread_pool_min to 500 with 2 pools (1000 idle threads total), while max is 5000 (from 3000), session_linger to 50, 100 and 150ms, connection_timeout to 5s, between_bytes_timeout to 5s, but sometimes the workers go way up crazy and the entire ram plus 3gb of swap get in use. Any ideas? Maybe something hanging up the sessions (network latency between varnish and backend? or someone doing this on purpose?) Here are my settings: acceptor_sleep_decay 0.900000 [] acceptor_sleep_incr 0.001000 [s] acceptor_sleep_max 0.050000 [s] auto_restart on [bool] ban_lurker_sleep 30.000000 [s] between_bytes_timeout 50.000000 [s] cache_vbe_conns off [bool] cc_command "exec cc -fpic -shared -Wl,-x -o %o %s" cli_buffer 8192 [bytes] cli_timeout 25 [seconds] clock_skew 10 [s] connect_timeout 50.000000 [s] critbit_cooloff 180.000000 [s] default_grace 10 [seconds] default_ttl 120 [seconds] diag_bitmap 0x0 [bitmap] err_ttl 0 [seconds] esi_syntax 0 [bitmap] fetch_chunksize 128 [kilobytes] first_byte_timeout 25.000000 [s] group varnish (112) http_headers 64 [header lines] http_range_support off [bool] listen_address :80 listen_depth 8192 [connections] log_hashstring off [bool] log_local_address off [bool] lru_interval 60 [seconds] max_esi_includes 5 [includes] max_restarts 4 [restarts] overflow_max 100 [%] ping_interval 3 [seconds] pipe_timeout 60 [seconds] prefer_ipv6 off [bool] purge_dups on [bool] rush_exponent 3 [requests per request] saintmode_threshold 10 [objects] send_timeout 6000 [seconds] sess_timeout 10 [seconds] sess_workspace 524288 [bytes] session_linger 50 [ms] session_max 100000 [sessions] shm_reclen 255 [bytes] shm_workspace 262144 [bytes] syslog_cli_traffic on [bool] thread_pool_add_delay 2 [milliseconds] thread_pool_add_threshold 2 [requests] thread_pool_fail_delay 200 [milliseconds] thread_pool_max 5000 [threads] thread_pool_min 500 [threads] thread_pool_purge_delay 1000 [milliseconds] thread_pool_stack unlimited [bytes] thread_pool_timeout 300 [seconds] thread_pools 2 [pools] thread_stats_rate 10 [requests] user varnish (106) vcl_trace off [bool] waiter default (epoll, poll) -- []'s Hugo From info at bizhat.com Mon Aug 22 23:03:17 2011 From: info at bizhat.com (BizHat.com Support) Date: Tue, 23 Aug 2011 04:33:17 +0530 Subject: Make error make[3]: *** [vmod_std.3] Error 1 Message-ID: Hi, I am install varnish from source (git) on CentOS, i have installed the dependencies. When i run make, i get following error. ======================================== You need rst2man installed to make dist ======================================== make[3]: *** [vmod_std.3] Error 1 make[3]: Leaving directory `/usr/local/src/varnish-cache/lib/libvmod_std' make[2]: *** [all-recursive] Error 1 make[2]: Leaving directory `/usr/local/src/varnish-cache/lib' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/usr/local/src/varnish-cache' make: *** [all] Error 2 Anyone know how to fix this ? Thanks, Yujin -------------- next part -------------- An HTML attachment was scrubbed... URL: From hugo.cisneiros at gmail.com Mon Aug 22 23:47:48 2011 From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch)) Date: Mon, 22 Aug 2011 20:47:48 -0300 Subject: Increases in memory usage and dropped connections In-Reply-To: References: Message-ID: On Mon, Aug 22, 2011 at 7:27 PM, Hugo Cisneiros (Eitch) wrote: > Any ideas? Maybe something hanging up the sessions (network latency > between varnish and backend? or someone doing this on purpose?) Also, I got some errors in syslog regarding max memory maps per process, found in this ticket: https://www.varnish-cache.org/trac/ticket/784 Since I raised the value from vm.max_map_count I don't get this error anymore, but I'm seeing very high values up to 100k+... Is this normal? -- []'s Hugo From sime at sime.net.au Tue Aug 23 01:15:50 2011 From: sime at sime.net.au (Simon Males) Date: Tue, 23 Aug 2011 11:15:50 +1000 Subject: varnish saint-mode In-Reply-To: References: Message-ID: On Tue, Aug 23, 2011 at 1:28 AM, alexus wrote: > I'm trying to implement "saint mode" into my default.vcl, I copy and > paste that little snippet that came with that documentation but now my > varnish wont even start. > > Misbehaving servers ? Varnish version 3.0.0 documentation > https://www.varnish-cache.org/docs/3.0/tutorial/handling_misbehaving_servers.html#saint-mode > > either I'm putting that into a wrong place or something else I dont > know, all I know as soon as I comment it out, varnish starts Ok. > any ideas how to troubleshoot it? You may have some compiler output in /tmp/*.varnish. Otherwise you can compile it on demand with a running varnishd: $ varnishadm -S /etc/varnish/secrect -T localhost:6082 vcl.load name_the_vcl /etc/varnish/default.vcl (I'm using 2.1, so varnishadm in 3.0 may be different). Failure to compile will produce 'classic' compiler type output. Moons ago I had a problem when compiler would fail when I copied and pasted something some the web, it felt like it was a funny line ending character. -- Simon Males From perbu at varnish-software.com Tue Aug 23 07:40:31 2011 From: perbu at varnish-software.com (Per Buer) Date: Tue, 23 Aug 2011 09:40:31 +0200 Subject: Make error make[3]: *** [vmod_std.3] Error 1 In-Reply-To: References: Message-ID: Hi?Yujin, On Tue, Aug 23, 2011 at 1:03 AM, BizHat.com Support wrote: > > Hi, > > I am install varnish from source (git) on CentOS, i have installed the dependencies. > > When i run make, i get following error. > > ======================================== > You need rst2man installed to make dist > ======================================== You need rst2man installed to build the source from Git as the man pages are written in RST. ?It's in the python-docutils package. If you build from one of our official tarballs it should not be needed. Or you could use an RPM. -- Per Buer Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer Varnish makes websites fly! Whitepapers?| Video?| Twitter From tfheen at varnish-software.com Tue Aug 23 08:42:40 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 23 Aug 2011 10:42:40 +0200 Subject: Make error make[3]: *** [vmod_std.3] Error 1 In-Reply-To: (BizHat com Support's message of "Tue, 23 Aug 2011 04:33:17 +0530") References: Message-ID: <87aab0a58f.fsf@qurzaw.varnish-software.com> ]] "BizHat.com Support" | ======================================== | You need rst2man installed to make dist | ======================================== [...] | Anyone know how to fix this ? Install rst2man? -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From alexus at gmail.com Tue Aug 23 13:52:31 2011 From: alexus at gmail.com (alexus) Date: Tue, 23 Aug 2011 09:52:31 -0400 Subject: varnish saint-mode In-Reply-To: References: Message-ID: I'm not sure what am I suppose to do with that and how would that help me... [root at 6prx01 ~]# varnishadm help CLI connected to 127.0.0.1 6082 help [command] ping [timestamp] auth response quit banner status start stop vcl.load vcl.inline vcl.use vcl.discard vcl.list vcl.show param.show [-l] [] param.set panic.show panic.clear storage.list ban.url ban [&& ]... ban.list [root at 6prx01 ~]# On Mon, Aug 22, 2011 at 9:15 PM, Simon Males wrote: > On Tue, Aug 23, 2011 at 1:28 AM, alexus wrote: >> I'm trying to implement "saint mode" into my default.vcl, I copy and >> paste that little snippet that came with that documentation but now my >> varnish wont even start. >> >> Misbehaving servers ? Varnish version 3.0.0 documentation >> https://www.varnish-cache.org/docs/3.0/tutorial/handling_misbehaving_servers.html#saint-mode >> >> either I'm putting that into a wrong place or something else I dont >> know, all I know as soon as I comment it out, varnish starts Ok. >> any ideas how to troubleshoot it? > > You may have some compiler output in /tmp/*.varnish. > > Otherwise you can compile it on demand with a running varnishd: > > $ varnishadm -S /etc/varnish/secrect -T localhost:6082 vcl.load > name_the_vcl /etc/varnish/default.vcl > > (I'm using 2.1, so varnishadm in 3.0 may be different). > > Failure to compile will produce 'classic' compiler type output. > > Moons ago I had a problem when compiler would fail when I copied and > pasted something some the web, it felt like it was a funny line ending > character. > > -- > Simon Males > -- http://alexus.org/ From mark at simplercomputing.net Tue Aug 23 15:55:04 2011 From: mark at simplercomputing.net (Mark E) Date: Tue, 23 Aug 2011 09:55:04 -0600 Subject: 503 errors In-Reply-To: References: Message-ID: <4E53CD58.3070600@simplercomputing.net> Hi, Anyone know why Varnish would throw a "503 Guru Mediation" error when landing on site that uses cookies? For example, we have a WordPress site that of course requires login to reach the admin interface. Every so often anyone who has ever logged in sees the 503 error. The only way around it that I know of so far is to clear the cookie by either deleting it or going to the WordPress login screen which runs a bit of code to clear the cookie. Or at least that's what I think is happening. The VCL isn't caching pages with cookies. How do I remedy this issue? Mark From audun at ytterdal.net Tue Aug 23 20:57:21 2011 From: audun at ytterdal.net (Audun Ytterdal) Date: Tue, 23 Aug 2011 22:57:21 +0200 Subject: sequental squid-like purge hangs. Message-ID: I have this in my cache that worked perfectly in Varnish 2.1 sub vcl_hit { if (req.request == "PURGE") { set obj.ttl = 0s; error 200 "Purged."; } } sub vcl_miss { if (req.request == "PURGE") { error 404 "Not in cache."; } } But after upgraded to 3.0 i see something very strange when purging the same url several times.. for i in `seq 1 10`;do time echo -e "PURGE / HTTP/1.0\r\nHost:www.vg.no\r\n\r\n" |nc -w 80 quinn.int.vgnett.no 80 | head -1;done HTTP/1.1 200 Purged. real 0m0.005s user 0m0.002s sys 0m0.001s HTTP/1.1 404 Not in cache. real 0m0.005s user 0m0.000s sys 0m0.004s real 1m20.006s user 0m0.000s sys 0m0.003s HTTP/1.1 200 Purged. real 0m0.005s user 0m0.001s sys 0m0.002s HTTP/1.1 404 Not in cache. real 0m0.005s user 0m0.002s sys 0m0.001s real 1m20.006s user 0m0.001s sys 0m0.002s HTTP/1.1 200 Purged. real 0m0.004s user 0m0.000s sys 0m0.002s HTTP/1.1 404 Not in cache. real 0m0.004s user 0m0.001s sys 0m0.000s real 1m20.005s user 0m0.002s sys 0m0.001s HTTP/1.1 200 Purged. real 0m0.006s user 0m0.003s sys 0m0.001s So every 3de time (or purging a already purged object) seems to hang the connection nc is timed out after 80 seconds. If I have no timeout it just hangs seemingly forever. From sime at sime.net.au Wed Aug 24 01:31:27 2011 From: sime at sime.net.au (Simon Males) Date: Wed, 24 Aug 2011 11:31:27 +1000 Subject: varnish saint-mode In-Reply-To: References: Message-ID: Back to your original problem, it's sounds like the VCL is failing to compile. Using varnishadm, you get see the compiler error's on screen. vcl.list List all loaded configuration. In your case, you will only one, likely called 'boot', as your restart varnish to load a vcl. vcl.load Compile and load the VCL file under the name provided. Here is an example of trying to compile a bogus VCL. # varnishadm -S /etc/varnish/secret -T localhost:6082 vcl.load fail /etc/varnish/default.vcl Message from VCC-compiler: Expected '.' got 'crap' (program line 101), at (input Line 3 Pos 9) crap code compiler fail --------####------------------- On Tue, Aug 23, 2011 at 11:52 PM, alexus wrote: > I'm not sure what am I suppose to do with that and how would that help me... > > [root at 6prx01 ~]# varnishadm help > CLI connected to 127.0.0.1 6082 > help [command] > ping [timestamp] > auth response > quit > banner > status > start > stop > vcl.load > vcl.inline > vcl.use > vcl.discard > vcl.list > vcl.show > param.show [-l] [] > param.set > panic.show > panic.clear > storage.list > ban.url > ban [&& ]... > ban.list > > [root at 6prx01 ~]# > > > On Mon, Aug 22, 2011 at 9:15 PM, Simon Males wrote: >> On Tue, Aug 23, 2011 at 1:28 AM, alexus wrote: >>> I'm trying to implement "saint mode" into my default.vcl, I copy and >>> paste that little snippet that came with that documentation but now my >>> varnish wont even start. >>> >>> Misbehaving servers ? Varnish version 3.0.0 documentation >>> https://www.varnish-cache.org/docs/3.0/tutorial/handling_misbehaving_servers.html#saint-mode >>> >>> either I'm putting that into a wrong place or something else I dont >>> know, all I know as soon as I comment it out, varnish starts Ok. >>> any ideas how to troubleshoot it? >> >> You may have some compiler output in /tmp/*.varnish. >> >> Otherwise you can compile it on demand with a running varnishd: >> >> $ varnishadm -S /etc/varnish/secrect -T localhost:6082 vcl.load >> name_the_vcl /etc/varnish/default.vcl >> >> (I'm using 2.1, so varnishadm in 3.0 may be different). >> >> Failure to compile will produce 'classic' compiler type output. >> >> Moons ago I had a problem when compiler would fail when I copied and >> pasted something some the web, it felt like it was a funny line ending >> character. -- Simon Males From tfheen at varnish-software.com Wed Aug 24 08:36:46 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Wed, 24 Aug 2011 10:36:46 +0200 Subject: 3.0.1 rc 1 Message-ID: <8739grtdcx.fsf@qurzaw.varnish-software.com> Hi, I've just tagged and released 3.0.1 rc 1. If people can please test that and give feedback, it would be most appreciated. It's available from http://repo.varnish-cache.org/source/varnish-3.0.1-rc1.tar.gz with packages available from http://repo.varnish-cache.org/test/rpms and http://repo.varnish-cache.org/test/squeeze Regards, -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From jordi.salvat at salir.com Wed Aug 24 16:09:01 2011 From: jordi.salvat at salir.com (Jordi Salvat) Date: Wed, 24 Aug 2011 18:09:01 +0200 Subject: Need help: Varnish unexpectedly flushing cache weekly Message-ID: Hi, we restarted our Varnish instance on Sunday 7th around 19:00. On Wednesday 10th 9:45 the child process restarted (causes unknown) -- loosing the existing cache. ... the cache slowly grew again .... On Wednesday 17th around 2:00, the cache was somehow flushed, loosing about 75% of its objects. ... the cache slowly grew again .... On Wednesday 24th at 2:00, the cache was flushed again, loosing about 85% of its objects. On each of these occasions our cache hit rate went down from ~90% to under 50%. Needless to say this is highly undesirable. What might be causing this behaviour? Any hints and directions helping research welcome. Thanks. More details: Varnish version: 2.1.5 (Debian lenny package 2.1.5-1~lenny4) O.S.: Linux s229-161 2.6.38 #1 SMP Sat Mar 26 09:51:44 CET 2011 x86_64 GNU/Linux MRTG daily graphs for cache size and objects in cache: [image: varnish_cache-day.png] [image: varnish_objects-day.png] Note how the two graphs are slightly different: size goes down between 2:00 and 15:30, object count goes down suddenly at 15:30. Main VCL file attached (long to read -- feel free to ask questions). -- Jordi Salvat i Alabart CTO & Project Manager Salir.com +34 676 49 86 00 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: varnish_objects-day.png Type: image/png Size: 2161 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: varnish_cache-day.png Type: image/png Size: 2400 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: salir.vcl Type: application/octet-stream Size: 8048 bytes Desc: not available URL: From alexus at gmail.com Wed Aug 24 18:04:35 2011 From: alexus at gmail.com (alexus) Date: Wed, 24 Aug 2011 14:04:35 -0400 Subject: varnish saint-mode In-Reply-To: References: Message-ID: turns out had another issue going and as soon as i fixed it saint mode start working :)) On Tue, Aug 23, 2011 at 9:31 PM, Simon Males wrote: > Back to your original problem, it's sounds like the VCL is failing to compile. > > Using varnishadm, you get see the compiler error's on screen. > > vcl.list > ? ? ? ?List all loaded configuration. > > In your case, you will only one, likely called 'boot', as your restart > varnish to load a vcl. > > vcl.load > ? ? ? ?Compile and load the VCL file under the name provided. > > Here is an example of trying to compile a bogus VCL. > > # varnishadm -S /etc/varnish/secret -T localhost:6082 vcl.load fail > /etc/varnish/default.vcl > Message from VCC-compiler: > Expected '.' got 'crap' > (program line 101), at > (input Line 3 Pos 9) > ? ? ? ?crap code compiler fail > --------####------------------- > > > > On Tue, Aug 23, 2011 at 11:52 PM, alexus wrote: >> I'm not sure what am I suppose to do with that and how would that help me... >> >> [root at 6prx01 ~]# varnishadm help >> CLI connected to 127.0.0.1 6082 >> help [command] >> ping [timestamp] >> auth response >> quit >> banner >> status >> start >> stop >> vcl.load >> vcl.inline >> vcl.use >> vcl.discard >> vcl.list >> vcl.show >> param.show [-l] [] >> param.set >> panic.show >> panic.clear >> storage.list >> ban.url >> ban [&& ]... >> ban.list >> >> [root at 6prx01 ~]# >> >> >> On Mon, Aug 22, 2011 at 9:15 PM, Simon Males wrote: >>> On Tue, Aug 23, 2011 at 1:28 AM, alexus wrote: >>>> I'm trying to implement "saint mode" into my default.vcl, I copy and >>>> paste that little snippet that came with that documentation but now my >>>> varnish wont even start. >>>> >>>> Misbehaving servers ? Varnish version 3.0.0 documentation >>>> https://www.varnish-cache.org/docs/3.0/tutorial/handling_misbehaving_servers.html#saint-mode >>>> >>>> either I'm putting that into a wrong place or something else I dont >>>> know, all I know as soon as I comment it out, varnish starts Ok. >>>> any ideas how to troubleshoot it? >>> >>> You may have some compiler output in /tmp/*.varnish. >>> >>> Otherwise you can compile it on demand with a running varnishd: >>> >>> $ varnishadm -S /etc/varnish/secrect -T localhost:6082 vcl.load >>> name_the_vcl /etc/varnish/default.vcl >>> >>> (I'm using 2.1, so varnishadm in 3.0 may be different). >>> >>> Failure to compile will produce 'classic' compiler type output. >>> >>> Moons ago I had a problem when compiler would fail when I copied and >>> pasted something some the web, it felt like it was a funny line ending >>> character. > > > -- > Simon Males > -- http://alexus.org/ From jeroen.ooms at stat.ucla.edu Wed Aug 24 18:59:50 2011 From: jeroen.ooms at stat.ucla.edu (Jeroen Ooms) Date: Wed, 24 Aug 2011 20:59:50 +0200 Subject: Caching non 200 results Message-ID: By default, varnish only cashes results if the backend returned a http 200 OK. How can I set varnish to cashe HTTP 400 results as well as HTTP 200 results? The HTTP 400 is quite frequent in my app and a request that resulted in HTTP 400 will always do so if it is re-requested, so I would like to cashe these results as well. Thanks, Jeroen -------------- next part -------------- An HTML attachment was scrubbed... URL: From kbrownfield at google.com Wed Aug 24 19:51:40 2011 From: kbrownfield at google.com (Ken Brownfield) Date: Wed, 24 Aug 2011 12:51:40 -0700 Subject: Caching non 200 results In-Reply-To: References: Message-ID: sub vcl_fetch { [...] if ( beresp.status == 400 ) { set beresp.ttl = 300s; } [...] } Typically done with 404s, but you can do any comparison you want with beresp.status (>=400, etc) -- kb On Wed, Aug 24, 2011 at 11:59, Jeroen Ooms wrote: > By default, varnish only cashes results if the backend returned a http 200 > OK. How can I set varnish to cashe HTTP 400 results as well as HTTP 200 > results? The HTTP 400 is quite frequent in my app and a request that > resulted in HTTP 400 will always do so if it is re-requested, so I would > like to cashe these results as well. > > Thanks, > > Jeroen > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeroen.ooms at stat.ucla.edu Wed Aug 24 22:06:14 2011 From: jeroen.ooms at stat.ucla.edu (Jeroen Ooms) Date: Thu, 25 Aug 2011 00:06:14 +0200 Subject: max simultaneous connections per client IP Message-ID: I am using the .max_connections config to limit the total maximum simultaneous connections to the backend server. Is it also possible to have a maximum on the number of simultaneous connections per client IP? So e.g. have .max_connections = 10, but only 2 per client IP. Thanks, Jeroen -------------- next part -------------- An HTML attachment was scrubbed... URL: From jlevine at iwin.com Thu Aug 25 06:33:50 2011 From: jlevine at iwin.com (Joshua Levine) Date: Wed, 24 Aug 2011 23:33:50 -0700 Subject: Need help: Varnish unexpectedly flushing cache weekly In-Reply-To: References: Message-ID: <1445AFBA-6CD8-451A-8AC8-6DEE463B4CDD@iwin.com> Hi Jordi! Been a while? :) We have definitely seen the issue, though I am happy to say it has been months since the last time. I recommend that you start with a look at SESS_WORKSPACE, default is 8192, but we set this to 32768 as we were seeing crashes: # http://varnish.projects.linpro.no/ticket/364 # http://varnish.projects.linpro.no/ticket/391 The 32768 matches what we have set as the max on our load balancers as well as in our http profile. Cheers, Joshua On Aug 24, 2011, at 9:09 AM, Jordi Salvat wrote: > Hi, > > we restarted our Varnish instance on Sunday 7th around 19:00. > > On Wednesday 10th 9:45 the child process restarted (causes unknown) -- loosing the existing cache. > > ... the cache slowly grew again .... > > On Wednesday 17th around 2:00, the cache was somehow flushed, loosing about 75% of its objects. > > ... the cache slowly grew again .... > > On Wednesday 24th at 2:00, the cache was flushed again, loosing about 85% of its objects. > > On each of these occasions our cache hit rate went down from ~90% to under 50%. Needless to say this is highly undesirable. > > What might be causing this behaviour? Any hints and directions helping research welcome. > > Thanks. > > More details: > > Varnish version: 2.1.5 (Debian lenny package 2.1.5-1~lenny4) > O.S.: Linux s229-161 2.6.38 #1 SMP Sat Mar 26 09:51:44 CET 2011 x86_64 GNU/Linux > > MRTG daily graphs for cache size and objects in cache: > > > > > > Note how the two graphs are slightly different: size goes down between 2:00 and 15:30, object count goes down suddenly at 15:30. > > Main VCL file attached (long to read -- feel free to ask questions). > > -- > Jordi Salvat i Alabart > CTO & Project Manager > Salir.com > +34 676 49 86 00 > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From jlevine at iwin.com Thu Aug 25 06:40:18 2011 From: jlevine at iwin.com (Joshua Levine) Date: Wed, 24 Aug 2011 23:40:18 -0700 Subject: Need help: Varnish unexpectedly flushing cache weekly In-Reply-To: References: Message-ID: <1C863CAC-5143-4689-97A5-200DCFAFDE05@iwin.com> I sent that reply explicit to crashes, crashes that actually appear addressed in your version, so I should also have asked, based on your references to a somewhat regular schedule, if you are using MALLOC or FILE, and also what you are allowing for NFILES? perhaps you are exceeding those resource allocations? Cheers, Joshua On Aug 24, 2011, at 9:09 AM, Jordi Salvat wrote: > Hi, > > we restarted our Varnish instance on Sunday 7th around 19:00. > > On Wednesday 10th 9:45 the child process restarted (causes unknown) -- loosing the existing cache. > > ... the cache slowly grew again .... > > On Wednesday 17th around 2:00, the cache was somehow flushed, loosing about 75% of its objects. > > ... the cache slowly grew again .... > > On Wednesday 24th at 2:00, the cache was flushed again, loosing about 85% of its objects. > > On each of these occasions our cache hit rate went down from ~90% to under 50%. Needless to say this is highly undesirable. > > What might be causing this behaviour? Any hints and directions helping research welcome. > > Thanks. > > More details: > > Varnish version: 2.1.5 (Debian lenny package 2.1.5-1~lenny4) > O.S.: Linux s229-161 2.6.38 #1 SMP Sat Mar 26 09:51:44 CET 2011 x86_64 GNU/Linux > > MRTG daily graphs for cache size and objects in cache: > > > > > > Note how the two graphs are slightly different: size goes down between 2:00 and 15:30, object count goes down suddenly at 15:30. > > Main VCL file attached (long to read -- feel free to ask questions). > > -- > Jordi Salvat i Alabart > CTO & Project Manager > Salir.com > +34 676 49 86 00 > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From tfheen at varnish-software.com Thu Aug 25 07:12:22 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Thu, 25 Aug 2011 09:12:22 +0200 Subject: Reducing overflowed and dropped work requests In-Reply-To: (Simon Males's message of "Thu, 18 Aug 2011 09:31:09 +1000") References: <87bow7km9x.fsf@qurzaw.varnish-software.com> Message-ID: <871uwat161.fsf@qurzaw.varnish-software.com> ]] Simon Males | Though now my committed memory usage (according to Munin) is 16G. The | server has a total of 4G and 2G malloc cache. Is there an equation I | can use to determine how much memory I need with my current | parameters? You'll generally see overhead in the range of 1k per object in addition to the -s parameters. There's some per-thread overhead, but I'm not sure what you'll be looking at there. | Additionally can I deduce from varnishstats how many threads are being | used? e.g. How many of the minimum 1600 are being used. Unfortunately, no. This is something we'd like to fix, but it's not there yet. -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From tfheen at varnish-software.com Thu Aug 25 07:17:52 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Thu, 25 Aug 2011 09:17:52 +0200 Subject: [PATCH] String concatenation operator in doc/sphinx/installation/upgrade.rst In-Reply-To: (Cosimo Streppone's message of "Tue, 16 Aug 2011 09:57:16 +0200") References: Message-ID: <87wre2rmcf.fsf@qurzaw.varnish-software.com> ]] "Cosimo Streppone" | Here's a trivial documentation patch against current | master that makes the string concatenation operator | change more evident, at least to me. Cheers, added. -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From tvannuland at mirabeau.nl Thu Aug 25 07:29:50 2011 From: tvannuland at mirabeau.nl (Thijs van Nuland) Date: Thu, 25 Aug 2011 07:29:50 +0000 Subject: Varnish on top of CAStor Message-ID: Hi, I am using Varnish in front of the Caringo CAStor storage system. CAStor requires Varnish to perform the actions described below, is this possible? (basically I would need some kind of 'God mode' for the backends, or I need to bypass the director - or make my own director) ( the redirects occur on PUT-POST-GET-HEAD) Thanks in advance. Client applications must be able to accept and properly interpret two different kinds of redirect responses. If CAStor returns a 301 Moved Permanently response, this means the Primary Access Note (PAN) is asking the client to resend the current request to the location supplied in the response headers (see Section 10.3.2 of the HTTP/1.1 spec) and to direct all future requests to that new node until further notice. An authorization parameter will be supplied in the location header. This parameter should be included in the request to the new PAN. It is possible that a future Moved Permanently will be received redirecting to yet another PAN (in that respect, the response is slightly misnamed). The second type of redirect response CAStor might send is the 307 Temporary Redirect. This response asks the client to resend the current request to the location supplied in the response headers, but to continue using the original PAN for the next request until further notice. [cid:blank4a37.gif] Thijs van Nuland Senior Applicatiebeheerder [cid:blank2447.gif] [cid:blank4bdf.gif] Mirabeau | Managed Services Emmasingel 29-31, 5611 AZ Eindhoven +31(0)20-5950550 - www.mirabeau.nl [Mirabeau] [cid:leaf4a68.gif] Please consider the environment before printing this email -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: blank4a37.gif Type: image/gif Size: 43 bytes Desc: blank4a37.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: blank2447.gif Type: image/gif Size: 43 bytes Desc: blank2447.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: blank2222.gif Type: image/gif Size: 43 bytes Desc: blank2222.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: blank4bdf.gif Type: image/gif Size: 43 bytes Desc: blank4bdf.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: miralogo4e6c.png Type: image/png Size: 1445 bytes Desc: miralogo4e6c.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: leaf4a68.gif Type: image/gif Size: 665 bytes Desc: leaf4a68.gif URL: From jordi.salvat at salir.com Thu Aug 25 08:42:01 2011 From: jordi.salvat at salir.com (Jordi Salvat) Date: Thu, 25 Aug 2011 10:42:01 +0200 Subject: Need help: Varnish unexpectedly flushing cache weekly In-Reply-To: <1C863CAC-5143-4689-97A5-200DCFAFDE05@iwin.com> References: <1C863CAC-5143-4689-97A5-200DCFAFDE05@iwin.com> Message-ID: Hey Josh! Long while indeed! Thanks for your help. We're using FILE (20GB) , ulimt -n is 1024 (is that what you're referring to?) -- right now the process only has 122 files open, so that doesn't seem to be the issue... but I'll watch out for next Wednesday. I'll also look at SESS_WORKSPACE. -- Jordi Salvat i Alabart CTO & Project Manager Salir.com +34 676 49 86 00 2011/8/25 Joshua Levine > > > I sent that reply explicit to crashes, crashes that actually appear > addressed in your version, so I should also have asked, based on your > references to a somewhat regular schedule, if you are using MALLOC or FILE, > and also what you are allowing for NFILES? perhaps you are exceeding those > resource allocations? > > Cheers, > Joshua > > On Aug 24, 2011, at 9:09 AM, Jordi Salvat wrote: > > > Hi, > > > > we restarted our Varnish instance on Sunday 7th around 19:00. > > > > On Wednesday 10th 9:45 the child process restarted (causes unknown) -- > loosing the existing cache. > > > > ... the cache slowly grew again .... > > > > On Wednesday 17th around 2:00, the cache was somehow flushed, loosing > about 75% of its objects. > > > > ... the cache slowly grew again .... > > > > On Wednesday 24th at 2:00, the cache was flushed again, loosing about 85% > of its objects. > > > > On each of these occasions our cache hit rate went down from ~90% to > under 50%. Needless to say this is highly undesirable. > > > > What might be causing this behaviour? Any hints and directions helping > research welcome. > > > > Thanks. > > > > More details: > > > > Varnish version: 2.1.5 (Debian lenny package 2.1.5-1~lenny4) > > O.S.: Linux s229-161 2.6.38 #1 SMP Sat Mar 26 09:51:44 CET 2011 x86_64 > GNU/Linux > > > > MRTG daily graphs for cache size and objects in cache: > > > > > > > > > > > > Note how the two graphs are slightly different: size goes down between > 2:00 and 15:30, object count goes down suddenly at 15:30. > > > > Main VCL file attached (long to read -- feel free to ask questions). > > > > -- > > Jordi Salvat i Alabart > > CTO & Project Manager > > Salir.com > > +34 676 49 86 00 > > _______________________________________________ > > varnish-misc mailing list > > varnish-misc at varnish-cache.org > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From l at lrowe.co.uk Thu Aug 25 11:28:48 2011 From: l at lrowe.co.uk (Laurence Rowe) Date: Thu, 25 Aug 2011 12:28:48 +0100 Subject: Varnish on top of CAStor In-Reply-To: References: Message-ID: On 25 August 2011 08:29, Thijs van Nuland wrote: > > Hi, I am using Varnish in front of the Caringo CAStor storage system. > > CAStor requires Varnish to perform the actions described below, is this possible? (basically I would need some kind of ?God mode? for the backends, or I need to bypass the director ? or make my own director) > > > > ( the redirects occur on PUT-POST-GET-HEAD) > > > > Thanks in advance. > > > > Client applications must be able to accept and properly interpret two different > kinds of redirect responses. If CAStor returns a 301 Moved Permanently response, this means the Primary Access Note > (PAN) is asking the client to resend the current request to the location supplied in the response headers (see Section 10.3.2 of the HTTP/1.1 spec) and to direct all future requests to that new node until further > notice. > > An authorization parameter will be supplied in the location header. This > parameter should be included in the request to the new PAN. It is possible that a > future Moved Permanently will be received > redirecting to yet another PAN (in that respect, the response is slightly > misnamed). > > The second type of redirect response CAStor might send is the 307 Temporary Redirect. This response asks the client > to resend the current request to the location > supplied in the response headers, but to continue using the original PAN for the > next request until further notice. This will not be possible to implement as Varnish does not store the body of the request so it is not possible to restart a POST or PUT request. Laurence From listas at kurtkraut.net Thu Aug 25 15:03:09 2011 From: listas at kurtkraut.net (Kurt Kraut) Date: Thu, 25 Aug 2011 12:03:09 -0300 Subject: 3.0.1 rc 1 In-Reply-To: <8739grtdcx.fsf@qurzaw.varnish-software.com> References: <8739grtdcx.fsf@qurzaw.varnish-software.com> Message-ID: Hi, I've found an issue with Varnish 3.0: downloading files larger than 1mb the download is constantly interrupted and reestabilished by HTTP 206. Rolling back to varnish 2.x immediatly fixed the issue. My question is: how can I know if this issue was covered in 3.0.1 rc 1? Is there a changelog for release candidates? Thanks, Kurt Kraut 2011/8/24 Tollef Fog Heen > > Hi, > > I've just tagged and released 3.0.1 rc 1. If people can please test > that and give feedback, it would be most appreciated. > > It's available from > http://repo.varnish-cache.org/source/varnish-3.0.1-rc1.tar.gz with > packages available from http://repo.varnish-cache.org/test/rpms and > http://repo.varnish-cache.org/test/squeeze > > Regards, > -- > Tollef Fog Heen > Varnish Software > t: +47 21 98 92 64 > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfheen at varnish-software.com Fri Aug 26 05:48:54 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Fri, 26 Aug 2011 07:48:54 +0200 Subject: 3.0.1 rc 1 In-Reply-To: (Kurt Kraut's message of "Thu, 25 Aug 2011 12:03:09 -0300") References: <8739grtdcx.fsf@qurzaw.varnish-software.com> Message-ID: <87hb54soxl.fsf@qurzaw.varnish-software.com> ]] Kurt Kraut | I've found an issue with Varnish 3.0: downloading files larger than 1mb the | download is constantly interrupted and reestabilished by HTTP 206. Rolling | back to varnish 2.x immediatly fixed the issue. Can you see if this happens with 3.0.0 as well as 3.0.1 rc 1? Any chance you can provide varnishlog from when it happens? | My question is: how can I know if this issue was covered in 3.0.1 rc 1? Is | there a changelog for release candidates? https://www.varnish-cache.org/trac/browser/doc/changes.rst has a high level list of changes. Cheers, -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From audun at ytterdal.net Fri Aug 26 07:12:36 2011 From: audun at ytterdal.net (Audun Ytterdal) Date: Fri, 26 Aug 2011 09:12:36 +0200 Subject: 3.0.1 rc 1 In-Reply-To: <87hb54soxl.fsf@qurzaw.varnish-software.com> References: <8739grtdcx.fsf@qurzaw.varnish-software.com> <87hb54soxl.fsf@qurzaw.varnish-software.com> Message-ID: On Fri, Aug 26, 2011 at 7:48 AM, Tollef Fog Heen wrote: > ]] Kurt Kraut > > | I've found an issue with Varnish 3.0: downloading files larger than 1mb the > | download is constantly interrupted and reestabilished by HTTP 206. Rolling > | back to varnish 2.x immediatly fixed the issue. > > Can you see if this happens with 3.0.0 as well as 3.0.1 rc 1? ?Any > chance you can provide varnishlog from when it happens? > > | My question is: how can I know if this issue was covered in 3.0.1 rc 1? Is > | there a changelog for release candidates? > > https://www.varnish-cache.org/trac/browser/doc/changes.rst > > has a high level list of changes. Are you sure the limit is 1MB and not the time it takes to download the file? On low bandwith 1MB could take more than 60 seconds to retrive. Varnish 3.0 has a send_timeout of 60 seconds while Varnish 2.1 had 600 seconds. I had to adjust send_timeout to 600 again on my 3.0 installation for pdf downloads and such. -- Audun From noreply at badoo.com Fri Aug 26 11:41:31 2011 From: noreply at badoo.com (Badoo) Date: Fri, 26 Aug 2011 11:41:31 +0000 Subject: You have a new message on Badoo! Check it out now... Message-ID: Shejuti left a message for you... Its sender and content will be shown only to you and you can delete it at any time. You can instantly reply to it, using the message exchange system. To find out what's in the message, just follow this link: http://eu1.badoo.com/0252649018/in/1k9Su3I23kM/?lang_id=3&m=21&mid=4e57865f0000000000030000d728714c Some more patiently waiting folks: ???? (Moscow, Russia) Lisamunny (Dhaka, Bangladesh) Ataz (Dhaka, Bangladesh) http://eu1.badoo.com/0252649018/in/1k9Su3I23kM/?lang_id=3&m=21&mid=4e57865f0000000000030000d728714c If the link in this message does not work, try copying and pasting it into your browser. This email is part of our delivery procedure for the message sent by Shejuti. If you have received this email by mistake, please ignore it. The message will be deleted soon. Have fun! The Badoo Team You have received this email, because a Badoo member has left a message for you on Badoo. This is a post-only mailing. Replies to this message are not monitored or answered. If you don't want to receive any more messages from Badoo, please notify us: http://eu1.badoo.com/impersonation.phtml?lang_id=3&mail_code=21&email=varnish-misc%40varnish-cache.org&secret=&block_code=e2d509&m=21&mid=4e57865f0000000000030000d728714c -------------- next part -------------- An HTML attachment was scrubbed... URL: From mhettwer at team.mobile.de Fri Aug 26 14:22:12 2011 From: mhettwer at team.mobile.de (Hettwer, Marian) Date: Fri, 26 Aug 2011 15:22:12 +0100 Subject: 3.0.1 rc 1 In-Reply-To: Message-ID: Hi All, On 26.08.11 09:12, "Audun Ytterdal" wrote: >On low bandwith 1MB could take more than 60 seconds to retrive. >Varnish 3.0 has a send_timeout of 60 seconds while Varnish 2.1 had 600 >seconds. >I had to adjust send_timeout to 600 again on my 3.0 installation for >pdf downloads and such. Just out of curiosity, why does varnish has a send_timeout in the first place? As long as data is going over the wire, a connection shouldn't get closed. I can understand a first_byte_timeout and an in_between_bytes_timeout. But a pure send_timeout doesn't make sense to me. Or am I understanding the use of send_timeout wrong? Cheers, Marian From phk at phk.freebsd.dk Fri Aug 26 18:46:11 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 26 Aug 2011 18:46:11 +0000 Subject: phk's Travel plans Message-ID: <17916.1314384371@critter.freebsd.dk> Here are my preliminary travel plans for the VUG4+SURGE/2011 trip next month. If anybody wants to hang out along the way, either in in New York or Washington DC, drop me an email. Poul-Henning sep 24 Arrive CPH->New York Staying with FreeBSD friends sep 25 Sightseeing, general hanging out Staying with FreeBSD friends sep 26 Train NY -> Washington DC Sightseeing along the National Mall (Smithsonian ?) Need place to stay, ideas welcome sep 27 Give talk at USNO @ 10:30 Train Washington DC -> Baltimore SURGE conference hotel sep 28 VUG4 (Sign up! https://www.varnish-cache.org/trac/wiki/VUG4) SURGE conference hotel sep 29 SURGE/2011 Give Varnish talk at 14:30 SURGE conference hotel sep 30 SURGE/2011 Fly home XXX -> CPH -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From j.vanarragon at lukkien.com Mon Aug 29 12:17:07 2011 From: j.vanarragon at lukkien.com (Jaap van Arragon) Date: Mon, 29 Aug 2011 14:17:07 +0200 Subject: ESI processing external URL Message-ID: Hello, I?m trying to implement an ESI external url within our website. After turning off the deflate module within apache I?m seeing an error message like: 0 Debug - "VCL_error(404, Unkown Virtual host)" The ESI include looks like this: The vcl is simply: sub vcl_fetch { esi; } Can somebody tell me why it is looking inside his own varnish and not just include the esi url? Thank you. Regards, Jaap -------------- next part -------------- An HTML attachment was scrubbed... URL: From apj at mutt.dk Mon Aug 29 12:34:51 2011 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Mon, 29 Aug 2011 14:34:51 +0200 Subject: ESI processing external URL In-Reply-To: References: Message-ID: <20110829123451.GI1944@nerd.dk> On Mon, Aug 29, 2011 at 02:17:07PM +0200, Jaap van Arragon wrote: > > onerror="continue"/> > > The vcl is simply: > > sub vcl_fetch { > esi; > } > > Can somebody tell me why it is looking inside his own varnish and not just > include the esi url? ESI is handled like any other request, and thus goes to the backend specified by your vcl. You need to add VCL to handle that particular hostname. -- Andreas From roberto.fernandezcrisial at gmail.com Mon Aug 29 14:00:14 2011 From: roberto.fernandezcrisial at gmail.com (=?utf-8?B?Um9iZXJ0byBPLiBGZXJuw6FuZGV6IENyaXNpYWw=?=) Date: Mon, 29 Aug 2011 14:00:14 +0000 Subject: ESI processing external URL In-Reply-To: <20110829123451.GI1944@nerd.dk> References: <20110829123451.GI1944@nerd.dk> Message-ID: <1072143392-1314626423-cardhu_decombobulator_blackberry.rim.net-1700356685-@b16.c27.bise6.blackberry> Have you tried to create a particular backend for "example.com" hostname and process request as ESI forwarded to that backend? I use (no ESI handle) a subdomain for images and send the requeste to original webserver (outside Varnish environment Roberto O. Fern?ndez Crisial @rofc -----Original Message----- From: Andreas Plesner Jacobsen Sender: varnish-misc-bounces at varnish-cache.org Date: Mon, 29 Aug 2011 14:34:51 To: Subject: Re: ESI processing external URL On Mon, Aug 29, 2011 at 02:17:07PM +0200, Jaap van Arragon wrote: > > onerror="continue"/> > > The vcl is simply: > > sub vcl_fetch { > esi; > } > > Can somebody tell me why it is looking inside his own varnish and not just > include the esi url? ESI is handled like any other request, and thus goes to the backend specified by your vcl. You need to add VCL to handle that particular hostname. -- Andreas _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From j.vanarragon at lukkien.com Mon Aug 29 14:27:24 2011 From: j.vanarragon at lukkien.com (Jaap van Arragon) Date: Mon, 29 Aug 2011 16:27:24 +0200 Subject: ESI processing external URL In-Reply-To: <1072143392-1314626423-cardhu_decombobulator_blackberry.rim.net-1700356685-@b16.c27.bise6.blackberry> Message-ID: I've got it working now. I've just included the url example.com in my vcl_receive. Thanks for the help. Regards Jaap van Arragon On 8/29/11 4:00 PM, "Roberto O. Fern?ndez Crisial" wrote: > Have you tried to create a particular backend for "example.com" hostname and > process request as ESI forwarded to that backend? > > I use (no ESI handle) a subdomain for images and send the requeste to original > webserver (outside Varnish environment > > Roberto O. Fern?ndez Crisial > @rofc > -----Original Message----- > From: Andreas Plesner Jacobsen > Sender: varnish-misc-bounces at varnish-cache.org > Date: Mon, 29 Aug 2011 14:34:51 > To: > Subject: Re: ESI processing external URL > > On Mon, Aug 29, 2011 at 02:17:07PM +0200, Jaap van Arragon wrote: >> >> > onerror="continue"/> >> >> The vcl is simply: >> >> sub vcl_fetch { >> esi; >> } >> >> Can somebody tell me why it is looking inside his own varnish and not just >> include the esi url? > > ESI is handled like any other request, and thus goes to the backend specified > by your vcl. > > You need to add VCL to handle that particular hostname. From alexus at gmail.com Mon Aug 29 15:18:02 2011 From: alexus at gmail.com (alexus) Date: Mon, 29 Aug 2011 11:18:02 -0400 Subject: /var/lib/varnish/varnish.* Message-ID: I dont know if something is wrong or if it suppose to be like this, but if I leave varnish by itself for a long time I get something like this: [root at svprx01 ~]# cd /var/lib/varnish/svprx01.uftmasterad.org/ [root at svprx01 svprx01.uftmasterad.org]# ll total 1095444 -rw------- 1 root root 3525955584 May 14 12:28 varnish.1N8wZV -rw------- 1 root root 3114553344 Aug 29 10:09 varnish.6Q9iEt -rw------- 1 root root 3553878016 May 26 15:53 varnish.9zlasm -rw------- 1 root root 3634511872 May 12 10:02 varnish.hIxzdx -rw------- 1 root root 3227156480 Aug 4 17:07 varnish.IecRUa -rw------- 1 root root 3392823296 Jul 1 12:40 varnish.J0TlGV -rw------- 1 root root 3525922816 May 17 13:59 varnish.JyDhF9 -rw------- 1 root root 3481899008 Jun 29 18:31 varnish.p4XhuS -rw------- 1 root root 3227156480 Aug 4 17:07 varnish.p9BGnV -rw------- 1 root root 3319324672 Jul 29 06:12 varnish.qDNSHS -rw------- 1 root root 3387392000 Jul 22 09:53 varnish.sZtkKx -rw------- 1 root root 3627753472 May 14 10:06 varnish.TnxuFD -rw------- 1 root root 3227148288 Aug 27 17:21 varnish.tvyaZt -rw------- 1 root root 3279007744 Aug 4 17:03 varnish.wfKTj5 -rw------- 1 root root 3504787456 May 24 17:25 varnish.yN7NLF -rw------- 1 root root 3555561472 Jun 11 10:58 varnish.zSKcCg -rwxr-xr-x 1 root root 34150 Aug 29 09:31 vcl.1P9zoqAU.so -rw-r--r-- 1 root root 83952744 Aug 29 11:14 _.vsl [root at svprx01 svprx01.uftmasterad.org]# and then my partition fills up and Nagios sends me a warning and I gotta go do something like this [root at svprx01 svprx01.uftmasterad.org]# service varnish stop && rm -f * && service varnish start Stopping varnish HTTP accelerator: [ OK ] Starting varnish HTTP accelerator: [ OK ] [root at svprx01 svprx01.uftmasterad.org]# ll total 108 -rw------- 1 root root 3675017216 Aug 29 11:16 varnish.L4U1kl -rwxr-xr-x 1 root root 34150 Aug 29 11:16 vcl.1P9zoqAU.so -rw-r--r-- 1 root root 83952744 Aug 29 11:16 _.vsl [root at svprx01 svprx01.uftmasterad.org]# and then I'm good until I see another alert from Nagios is this a bug or a feature? -- http://alexus.org/ From whshub at gmail.com Mon Aug 29 16:33:09 2011 From: whshub at gmail.com (Jacques) Date: Mon, 29 Aug 2011 09:33:09 -0700 Subject: Segfault in varnishncsa when exclude regex matches Message-ID: I'm trying to use varnish on rhel6.1. I get a segfault within the varnishncsa when I use -X and the regex matches a call. For example, varnishncsa -X loadbalancer works until the request /test?loadbalancer ping occurs. varnishncsa -X foobar works indefinitely (as there are no requests which match foobar) Any thoughts? Thanks, Jacques // Program terminated with signal 11, Segmentation fault. #0 0x0000000000403ae7 in __dso_handle () (gdb) bt #0 0x0000000000403ae7 in __dso_handle () #1 0x00000000004049de in ?? () #2 0x00007fffd4ec8e70 in ?? () #3 0x00000000004049de in ?? () #4 0x0000000000402b69 in collect_client (priv=0x8f8270, tag=SLT_Debug, fd=, len=1615485128, spec=, ptr=0x56
, bitmap=140736765660960) at varnishncsa.c:406 #5 h_ncsa (priv=0x8f8270, tag=SLT_Debug, fd=, len=1615485128, spec=, ptr=0x56
, bitmap=140736765660960) at varnishncsa.c:517 #6 0x00007fffd4ec8f18 in ?? () #7 0x00007fffd4ec8f20 in ?? () #8 0xaf2734be055b6ceb in ?? () #9 0x0000000000000000 in ?? () -------------- next part -------------- An HTML attachment was scrubbed... URL: From whshub at gmail.com Mon Aug 29 16:38:42 2011 From: whshub at gmail.com (Jacques) Date: Mon, 29 Aug 2011 09:38:42 -0700 Subject: Segfault in varnishncsa when exclude regex matches In-Reply-To: References: Message-ID: Sorry, I completely forgot to include version information. I'm seeing this problem on 3.0.1rc1. The same was happening on 3.0ga. Thanks, Jacques On Mon, Aug 29, 2011 at 9:33 AM, Jacques wrote: > I'm trying to use varnish on rhel6.1. I get a segfault within the > varnishncsa when I use -X and the regex matches a call. > > For example, > > varnishncsa -X loadbalancer works until the request /test?loadbalancer ping > occurs. > varnishncsa -X foobar works indefinitely (as there are no requests which > match foobar) > > > Any thoughts? > > Thanks, > Jacques > > // > Program terminated with signal 11, Segmentation fault. > #0 0x0000000000403ae7 in __dso_handle () > (gdb) bt > #0 0x0000000000403ae7 in __dso_handle () > #1 0x00000000004049de in ?? () > #2 0x00007fffd4ec8e70 in ?? () > #3 0x00000000004049de in ?? () > #4 0x0000000000402b69 in collect_client (priv=0x8f8270, tag=SLT_Debug, > fd=, len=1615485128, spec=, > ptr=0x56
, bitmap=140736765660960) at > varnishncsa.c:406 > #5 h_ncsa (priv=0x8f8270, tag=SLT_Debug, fd=, > len=1615485128, spec=, ptr=0x56
bounds>, > bitmap=140736765660960) at varnishncsa.c:517 > #6 0x00007fffd4ec8f18 in ?? () > #7 0x00007fffd4ec8f20 in ?? () > #8 0xaf2734be055b6ceb in ?? () > #9 0x0000000000000000 in ?? () > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From varnishlist at realvideosite.com Mon Aug 29 16:52:05 2011 From: varnishlist at realvideosite.com (Drew) Date: Mon, 29 Aug 2011 12:52:05 -0400 Subject: Speed up Varnish miss Message-ID: <4E5BC3B5.7020702@realvideosite.com> A Varnish fetch to the back-end takes longer than not having varnish at all. Is this normal and if so is there a way to speed up cache misses? Thanks, Drew From mhettwer at team.mobile.de Mon Aug 29 17:10:12 2011 From: mhettwer at team.mobile.de (Hettwer, Marian) Date: Mon, 29 Aug 2011 18:10:12 +0100 Subject: Accept-Enconding stripped Message-ID: Hi All, I'm having trouble to understand my varnish (that's kinda rare). I'm trying to cache a url with and without gzip'ed. Apparently, I'm doing something wrong, because usually varnish would not strip away headers, unless I told him so. This is a request to varnish, running on localhost: curl -v -s -o /dev/null -H "Accept-Encoding: gzip" -H "Host:kleinanzeigen.ebay.de" "http://localhost/anzeigen/s-beliebte-angebote.html?catId=241&letter=z" * About to connect() to localhost port 80 (#0) * Trying 127.0.0.1... connected * Connected to localhost (127.0.0.1) port 80 (#0) > GET /anzeigen/s-beliebte-angebote.html?catId=241&letter=z HTTP/1.1 > User-Agent: curl/7.18.2 (x86_64-pc-linux-gnu) libcurl/7.18.2 >OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.8 libssh2/0.18 > Accept: */* > Accept-Encoding: gzip > Host:kleinanzeigen.ebay.de > < HTTP/1.1 200 OK < Server: Apache < Cache-Control: public, max-age=86400 < Content-Language: de-DE < Vary: Accept-Encoding < Content-Type: text/html;charset=UTF-8 < Content-Length: 15828 < Date: Mon, 29 Aug 2011 17:01:27 GMT < X-Varnish: 800174289 800173404 < Age: 2 < Via: 1.1 varnish < Connection: keep-alive < { [data not shown] * Connection #0 to host localhost left intact * Closing connection #0 I'm missing the "Content-Encoding: gzip" response header. If I'm sending the request against the apache backend, it looks like what I would expect: curl -v -s -o /dev/null -H "Accept-Encoding: gzip" -H "Host:kleinanzeigen.ebay.de" "http://apache/anzeigen/s-beliebte-angebote.html?catId=241&letter=z" * About to connect() to apache port 80 (#0) * Trying 10.xx.xx.xx... connected * Connected to febayk46-1 (10.xx.xx.xx) port 80 (#0)> GET /anzeigen/s-beliebte-angebote.html?catId=241&letter=z HTTP/1.1 > User-Agent: curl/7.18.2 (x86_64-pc-linux-gnu) libcurl/7.18.2 >OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.8 libssh2/0.18 > Accept: */* > Accept-Encoding: gzip > Host:kleinanzeigen.ebay.de > < HTTP/1.1 200 OK < Date: Mon, 29 Aug 2011 17:03:59 GMT < Server: Apache < Cache-Control: public, max-age=86400 < Content-Language: de-DE < Vary: Accept-Encoding < Content-Encoding: gzip < Content-Length: 3284 < Content-Type: text/html;charset=UTF-8 < { [data not shown] * Connection #0 to host febayk46-1 left intact * Closing connection #0 Look, there it is. It's encoded in gzip. So the backend would actually gzip. But if I'm going through my varnish instance, it's not gzip'ed anymore. Whereas, if I use firefox, I get a gzipped answer via varnish. Firebug answer section: Server: Apache Content-Language: de-DE Vary: Accept-Encoding Content-Type: text/html;charset=UTF-8 ntCoent-Length: 15828 Date: Mon, 29 Aug 2011 17:06:58 GMT X-Varnish: 800298936 800173404 Age: 333 Via: 1.1 varnish Connection: keep-alive Cache-Control: private Content-Encoding: gzip Content-Length: 3254 This is really strange. Am I missing something obvious here?! Thanks for any hints / pointers. BeSD regards, Marian From hugo.cisneiros at gmail.com Mon Aug 29 18:31:05 2011 From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch)) Date: Mon, 29 Aug 2011 15:31:05 -0300 Subject: Increases in memory usage and dropped connections In-Reply-To: References: Message-ID: On Mon, Aug 22, 2011 at 8:47 PM, Hugo Cisneiros (Eitch) wrote: > On Mon, Aug 22, 2011 at 7:27 PM, Hugo Cisneiros (Eitch) > wrote: >> Any ideas? Maybe something hanging up the sessions (network latency >> between varnish and backend? or someone doing this on purpose?) > > Also, I got some errors in syslog regarding max memory maps per > process, found in this ticket: > > https://www.varnish-cache.org/trac/ticket/784 > > Since I raised the value from vm.max_map_count I don't get this error > anymore, but I'm seeing very high values up to 100k+... Is this > normal? I upgraded the servers to Varnish 3.0 (with same configuration) and stopped getting Connection Reset errors. The memory consumption is still high, but not as high to give me errors. It's working fine now with high traffic for days along =) memory maps per proccess is a lot less to. 100k+ to 12k stable. The only strange thing is sometimes I'm still getting many stats on 'n_wrk_max - N worker threads limited', but it seems that the worker queue is working just fine. Thanks =) -- []'s Hugo www.devin.com.br From tfheen at varnish-software.com Tue Aug 30 06:52:34 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 30 Aug 2011 08:52:34 +0200 Subject: Speed up Varnish miss In-Reply-To: <4E5BC3B5.7020702@realvideosite.com> (Drew's message of "Mon, 29 Aug 2011 12:52:05 -0400") References: <4E5BC3B5.7020702@realvideosite.com> Message-ID: <87r543s85p.fsf@qurzaw.varnish-software.com> ]] Drew | A Varnish fetch to the back-end takes longer than not having varnish | at all. Naturally, there is more processing involved. | Is this normal and if so is there a way to speed up cache | misses? Apart from obvious and snarky answers like ?get a faster backend?, not really. Varnish is handling it as quickly as it can. :-) Cheers, -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From tfheen at varnish-software.com Tue Aug 30 07:05:52 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 30 Aug 2011 09:05:52 +0200 Subject: Segfault in varnishncsa when exclude regex matches In-Reply-To: (Jacques's message of "Mon, 29 Aug 2011 09:33:09 -0700") References: Message-ID: <87mxers7jj.fsf@qurzaw.varnish-software.com> ]] Jacques | varnishncsa -X loadbalancer works until the request /test?loadbalancer ping | occurs. | varnishncsa -X foobar works indefinitely (as there are no requests which | match foobar) Cheers, fixed in master, will be in 3.0.1. -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From apj at mutt.dk Tue Aug 30 07:42:21 2011 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Tue, 30 Aug 2011 09:42:21 +0200 Subject: Speed up Varnish miss In-Reply-To: <4E5BC3B5.7020702@realvideosite.com> References: <4E5BC3B5.7020702@realvideosite.com> Message-ID: <20110830074221.GJ1944@nerd.dk> On Mon, Aug 29, 2011 at 12:52:05PM -0400, Drew wrote: > A Varnish fetch to the back-end takes longer than not having varnish > at all. Is this normal and if so is there a way to speed up cache > misses? How much longer? If we're talking ms, it's just an artifact of introducing another element in the delivery chain. -- Andreas From kristian at varnish-software.com Tue Aug 30 08:24:42 2011 From: kristian at varnish-software.com (Kristian Lyngstol) Date: Tue, 30 Aug 2011 10:24:42 +0200 Subject: Speed up Varnish miss In-Reply-To: <4E5BC3B5.7020702@realvideosite.com> References: <4E5BC3B5.7020702@realvideosite.com> Message-ID: <20110830082442.GB2752@freud.kly.no> On Mon, Aug 29, 2011 at 12:52:05PM -0400, Drew wrote: > A Varnish fetch to the back-end takes longer than not having varnish > at all. Is this normal and if so is there a way to speed up cache > misses? Like the others have pointed out, a bit of overhead is natural. However, there is one big factor here so far left unmentioned. Varnish, by default, will fetch an entire object before it starts sending it to the client. For a slow object (a large object fetched from a slow web server, for instance), this will add a significant delay. However, with Varnish 3.0.0, Varnish can stream these objects when there is a cache miss or a pass. That means Varnish will start sending the data to the client as the data arrives from the web server. This is not enabled by default, and requires beresp.do_stream = true;. Note that this doesn't apply to busy objects. In other words: if a client is already fetching an object, and an other client requests the same object, the first client will get the data on-the-fly, while the second client has to wait. Support for full streaming is work in progress and a working implementation exists, though it is not yet merged with the master branch. - Kristian From straightflush at gmail.com Tue Aug 30 11:56:33 2011 From: straightflush at gmail.com (AD) Date: Tue, 30 Aug 2011 07:56:33 -0400 Subject: Speed up Varnish miss In-Reply-To: <20110830082442.GB2752@freud.kly.no> References: <4E5BC3B5.7020702@realvideosite.com> <20110830082442.GB2752@freud.kly.no> Message-ID: Does the browser need to support this explicitly? On Tue, Aug 30, 2011 at 4:24 AM, Kristian Lyngstol < kristian at varnish-software.com> wrote: > On Mon, Aug 29, 2011 at 12:52:05PM -0400, Drew wrote: > > A Varnish fetch to the back-end takes longer than not having varnish > > at all. Is this normal and if so is there a way to speed up cache > > misses? > > Like the others have pointed out, a bit of overhead is natural. > > However, there is one big factor here so far left unmentioned. > > Varnish, by default, will fetch an entire object before it starts > sending it to the client. For a slow object (a large object fetched from > a slow web server, for instance), this will add a significant delay. > > However, with Varnish 3.0.0, Varnish can stream these objects when there > is a cache miss or a pass. That means Varnish will start sending the > data to the client as the data arrives from the web server. This is not > enabled by default, and requires beresp.do_stream = true;. > > Note that this doesn't apply to busy objects. In other words: if a > client is already fetching an object, and an other client requests the > same object, the first client will get the data on-the-fly, while the > second client has to wait. > > Support for full streaming is work in progress and a working > implementation exists, though it is not yet merged with the master > branch. > > - Kristian > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kristian at varnish-software.com Tue Aug 30 12:06:46 2011 From: kristian at varnish-software.com (Kristian Lyngstol) Date: Tue, 30 Aug 2011 14:06:46 +0200 Subject: Speed up Varnish miss In-Reply-To: References: <4E5BC3B5.7020702@realvideosite.com> <20110830082442.GB2752@freud.kly.no> Message-ID: <20110830120646.GC2752@freud.kly.no> On Tue, Aug 30, 2011 at 07:56:33AM -0400, AD wrote: > On Tue, Aug 30, 2011 at 4:24 AM, Kristian Lyngstol < > kristian at varnish-software.com> wrote: > > However, with Varnish 3.0.0, Varnish can stream these objects when there > > is a cache miss or a pass. That means Varnish will start sending the > > data to the client as the data arrives from the web server. > > Does the browser need to support this explicitly? No, this is not affected by the client-support. It's just getting the same data faster. - Kristian From saiaman01 at gmail.com Tue Aug 30 13:26:59 2011 From: saiaman01 at gmail.com (Albuquerque Rui) Date: Tue, 30 Aug 2011 15:26:59 +0200 Subject: Updating from 3.0.0 to 3.0.1 on CentOS Message-ID: Hello, i've tried yum update varnish on CentOS 5 but .... :( Resolving Dependencies --> Running transaction check ---> Package varnish.x86_64 0:3.0.1-1.el5 set to be updated --> Processing Dependency: varnish-libs = 3.0.1-1.el5 for package: varnish --> Processing Dependency: libedit.so.0()(64bit) for package: varnish --> Running transaction check ---> Package varnish.x86_64 0:3.0.1-1.el5 set to be updated --> Processing Dependency: libedit.so.0()(64bit) for package: varnish ---> Package varnish-libs.x86_64 0:3.0.1-1.el5 set to be updated --> Finished Dependency Resolution varnish-3.0.1-1.el5.x86_64 from varnish-3.0 has depsolving problems --> Missing Dependency: libedit.so.0()(64bit) is needed by package varnish-3.0.1-1.el5.x86_64 (varnish-3.0) Error: Missing Dependency: libedit.so.0()(64bit) is needed by package varnish-3.0.1-1.el5.x86_64 (varnish-3.0) You could try using --skip-broken to work around the problem You could try running: package-cleanup --problems package-cleanup --dupes rpm -Va --nofiles --nodigest Any solution ??? -------------- next part -------------- An HTML attachment was scrubbed... URL: From whshub at gmail.com Tue Aug 30 14:55:24 2011 From: whshub at gmail.com (Jacques) Date: Tue, 30 Aug 2011 07:55:24 -0700 Subject: Segfault in varnishncsa when exclude regex matches In-Reply-To: <87mxers7jj.fsf@qurzaw.varnish-software.com> References: <87mxers7jj.fsf@qurzaw.varnish-software.com> Message-ID: I've also noticed the following 3.0.1: # varnishncsa -F '%{Varnish:handling}x' Unknown format starting at: %{Varnish:handling}x # varnishncsa -F '%{Varnish:time_firstbyte}x' Unknown format starting at: }x # varnishncsa -F '%{Varnish:hitmiss}x' - - - - (Varnish:hitmiss throws no error but also appears to give no info.) On Tue, Aug 30, 2011 at 12:05 AM, Tollef Fog Heen < tfheen at varnish-software.com> wrote: > ]] Jacques > > | varnishncsa -X loadbalancer works until the request /test?loadbalancer > ping > | occurs. > | varnishncsa -X foobar works indefinitely (as there are no requests which > | match foobar) > > Cheers, fixed in master, will be in 3.0.1. > > -- > Tollef Fog Heen > Varnish Software > t: +47 21 98 92 64 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From graham.lyons at bbc.co.uk Tue Aug 30 16:19:17 2011 From: graham.lyons at bbc.co.uk (Graham Lyons) Date: Tue, 30 Aug 2011 17:19:17 +0100 Subject: req.grace default Message-ID: Version 2.1.3 I'm trying to set Varnish up to use Saint mode and I've noticed that if I set beresp.grace=30s and beresp.saintmode=10s, cache a response and then make the backend fail I'll get stale content served for 10 seconds and then get 503 error responses from Varnish. The backend is still only contacted every 10 seconds. If req.grace is set to 30s then I get stale content for 30 seconds. Does req.grace have a default value which is causing this behaviour? Thanks, Graham. http://www.bbc.co.uk/ This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated. If you have received it in error, please delete it from your system. Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately. Please note that the BBC monitors e-mails sent or received. Further communication will signify your consent to this. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2179 bytes Desc: not available URL: From paulo at aliancaproject.com Tue Aug 30 17:32:14 2011 From: paulo at aliancaproject.com (Paulo Paracatu) Date: Tue, 30 Aug 2011 14:32:14 -0300 Subject: Updating from 3.0.0 to 3.0.1 on CentOS In-Reply-To: References: Message-ID: libedit is available at epel repository http://fedoraproject.org/wiki/EPEL 2011/8/30 Albuquerque Rui > Hello, > i've tried yum update varnish on CentOS 5 but .... :( > > Resolving Dependencies > --> Running transaction check > ---> Package varnish.x86_64 0:3.0.1-1.el5 set to be updated > --> Processing Dependency: varnish-libs = 3.0.1-1.el5 for package: varnish > --> Processing Dependency: libedit.so.0()(64bit) for package: varnish > --> Running transaction check > ---> Package varnish.x86_64 0:3.0.1-1.el5 set to be updated > --> Processing Dependency: libedit.so.0()(64bit) for package: varnish > ---> Package varnish-libs.x86_64 0:3.0.1-1.el5 set to be updated > --> Finished Dependency Resolution > varnish-3.0.1-1.el5.x86_64 from varnish-3.0 has depsolving problems > --> Missing Dependency: libedit.so.0()(64bit) is needed by package > varnish-3.0.1-1.el5.x86_64 (varnish-3.0) > Error: Missing Dependency: libedit.so.0()(64bit) is needed by package > varnish-3.0.1-1.el5.x86_64 (varnish-3.0) > You could try using --skip-broken to work around the problem > You could try running: package-cleanup --problems > package-cleanup --dupes > rpm -Va --nofiles --nodigest > > > > Any solution ??? > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: