From forest at tmswiki.net Mon Jan 2 19:30:27 2012 From: forest at tmswiki.net (Forest) Date: Mon, 2 Jan 2012 11:30:27 -0800 (PST) Subject: Default VCL for MediaWiki Message-ID: <1325532627.84942.YahooMailNeo@web113803.mail.gq1.yahoo.com> Hello. I recently added Varnish to my MediaWiki wiki and was amazed by Varnish's amazing performance. Many thanks to everyone who helps make this incredible software available to the community. Mediawiki, the software that powers Wikipedia, has a manual page about using MediaWiki with Varnish, but it needs to be updated to reflect both the changes to VCL in version 3 and some updates to Mediawiki itself. I'm planning on writing this update and am requesting a review of the example VCL code I'll be including in the MediaWiki manual. I am in no way a Varnish or HTTP expert (in fact, I'm just a graduate student in economics who is passionate about supporting his wiki and free culture in general), so I would be very grateful for any feedback on the example code. My hope is to spur adoption of Varnish within the Mediawiki community. Since many Mediawiki installations are done by people with little or no experience with HTTP, so I think that adoption requires a good manual page. Some notes: * When Mediawiki's built in support for http accelerators is enabled, if a page or image is changed on the wiki, MediaWiki will send notification to every registered Varnish or Squid server, telling it to discard the outdated stored page. The specific notification protocol was designed for Squid and involves HTTP requests with a PURGE method. Also, with support enabled, changes made to the wiki by anonymous users will be attributed to their IP address, found in 'x-forwarded-for,' rather than to the requesting cache's IP address. * The current manual page, designed for Varnish 2.x is http://www.mediawiki.org/wiki/Manual:Varnish_caching? . I've brought the issue with the Vector skin up on the Mediawiki-L list, and Mediawiki's Erik Moeller pointed me to Wikia's VCL script, from which I adapted the vcl_recv code for dealing with cookies. * Finally, the example code is still a bit rough and I plan on cleaning it up a bit when writing the documentation. Knowing whether I'm on the right track would be very helpful, though, as my own wiki will be going live with the code in less than a week. :) Thank you. Forest # set default backend if no server cluster specified backend default { ??? .host = "localhost"; ??? .port = "8080"; ??? # .port = "80"; also works well, but using 8080 allows direct access to Apache for debugging purposes. } # access control list for "purge": open to only localhost and other local nodes acl purge { ??? "localhost"; } # The default code for vcl_recv is incorporated into the following subroutine to make it easier to specify the proper order of execution. sub vcl_recv { ??? set req.backend = default; ??? # Serve objects up to 2 minutes past their expiry if the backend is slow to respond. ??? # Not relevant to low traffic wikis. ??? set req.grace = 120s; ??? if (req.restarts == 0) { ??? ??? if (req.http.x-forwarded-for) { ??? ??? ??? set req.http.X-Forwarded-For = ??? ??? ??? req.http.X-Forwarded-For + ", " + client.ip; ??? ??? } else { ??? ??? ??? set req.http.X-Forwarded-For = client.ip; ??? ??? } ??? } ??? # This uses the ACL action called "purge". Basically if a request to ??? # PURGE the cache comes from anywhere other than localhost, ignore it. ??? if (req.request == "PURGE") { ??? ??? if (!client.ip ~ purge) { ??? ??? ??? error 405 "Not allowed."; ??? ??? } ??? ??? return(lookup); ??? } ??? if (req.request != "GET" && ??? ??? req.request != "HEAD" && ??? ??? req.request != "PUT" && ??? ??? req.request != "POST" && ??? ??? req.request != "TRACE" && ??? ??? req.request != "OPTIONS" && ??? ??? req.request != "DELETE") { ??? ??? # Non-RFC2616 or CONNECT which is weird. ??? ??? ??? return (pipe); ??? } ??? if (req.request != "GET" && req.request != "HEAD") { ??? ??? # We only deal with GET and HEAD by default ??? ??? return (pass); ??? } ??? # Replace "/wiki/ with the path to your MediaWiki installation. ??? # CONCERN: is the following line robust? ??? if(req.url ~ "^/wiki/"){ ??? ??? if(req.http.Cookie ~ "(session|UserID|UserName|Token|LoggedOut)") { ??? ??? ??? # dont do anything, the user is logged in ??? ??? } else { ??? ??? ??? # dont care about any other cookies ??? ??? ??? unset req.http.Cookie; ??? ??? } ??? } ??? if (req.http.Authorization || req.http.Cookie) { ???????? /* Not cacheable by default */ ???????? return (pass); ??? } ??? # Legacy: (I've marked some commented out code as Legacy. This is code that was found in the existing manual page, but which I'm planning on dropping in the new version. If you think it's worth keeping, please let me know.) #??? if (req.http.If-None-Match) #??? ??? {return(pass);} ??? # Force lookup if the request is a no-cache request from the client. ??? if (req.http.Cache-Control ~ "no-cache") { ??? ??? set req.hash_always_miss = true;??? ??? ??? # https://www.varnish-cache.org/trac/wiki/VCLExampleEnableForceRefresh #??? ??? ban_url(req.url); ??? } ??? return (lookup); ?} sub vcl_pipe { ??? # This is otherwise not necessary if you do not do any request rewriting. ??? set bereq.http.connection = "close"; } sub vcl_hit { ??? if (req.request == "PURGE") { ??? ??? purge; ??? ??? error 200 "Purged"; ??? } ??? # Legacy: ??? # if (!obj.cacheable) { ??? # ??? return(pass); ??? # } } sub vcl_miss { ??? if (req.request == "PURGE") { ??? ??? error 200 "Not in cache"; ??? } } sub vcl_fetch { ??? # For debugging only. Varnish's internal Time To Live for cached object ??? set beresp.http.X-orig-ttl = beresp.ttl; ??? # I think the following is redundant because caches aren't allowed to change Cache Control headers ??? set beresp.http.X-Orig-Cache-Control = beresp.http.Cache-Control; ??? # set minimum timeouts to auto-discard stored objects #??? set beresp.prefetch = -30s; ??? set beresp.grace = 120s; ??? # Legacy: ??? # if (beresp.http.Cache-Control ~ "(private|no-cache|no-store)") { ??? #??? return(hit_for_pass); ??? # } ??? # Legacy: ??? # if (req.http.Authorization && !beresp.http.Cache-Control ~ "public") { ??? #??? return(hit_for_pass); ??? # } } sub vcl_deliver { ??? # For debugging only. ??? # The approximate number of times the object has been delivered. A value of 0 indicates a cache miss. ??? set resp.http.X-obj-hits = obj.hits; } -------------- next part -------------- An HTML attachment was scrubbed... URL: From tochi2k at hotmail.com Tue Jan 3 02:11:47 2012 From: tochi2k at hotmail.com (charles ilogu) Date: Mon, 2 Jan 2012 18:11:47 -0800 Subject: configuring Varnish Message-ID: Does anyone know how i can configure varnish vcl file so that it does not ban .css and .js files when i issue a BAN? -------------- next part -------------- An HTML attachment was scrubbed... URL: From hugo.cisneiros at gmail.com Tue Jan 3 13:40:51 2012 From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch)) Date: Tue, 3 Jan 2012 11:40:51 -0200 Subject: configuring Varnish In-Reply-To: References: Message-ID: On Tue, Jan 3, 2012 at 12:11 AM, charles ilogu wrote: > Does anyone know how i can configure varnish vcl file so that it does not > ban .css and .js files when i issue a BAN? Before the ban() function, something like this: if (req.request == "BAN") { # Deny ban on css and js extensions if (req.url ~ "\.(css|js)$") { error 405 "Ban not allowed."; } ban(...) } Source: https://www.varnish-cache.org/docs/trunk/tutorial/purging.html -- []'s Hugo www.devin.com.br From cianmcgovern91 at gmail.com Sun Jan 1 01:22:43 2012 From: cianmcgovern91 at gmail.com (Cian Mc Govern) Date: Sun, 1 Jan 2012 01:22:43 +0000 Subject: Rewriting/enforcing SSL behing an SSL termination point Message-ID: Hi, I'm having an issue when I try to implement the redirect and reload varnish: Message from VCC-compiler: Expected variable, string or semicolon (input Line 17 Pos 68) set req.http.x-redir-url = "https://www.cianmcgovern.com/" + req.url; ---------------------------------------------------------------------------#--------- Can't find a solution to this but I suspect it might be due to the version, 2.1.5, I'm using?? Thanks for any help! -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Tue Jan 3 14:22:26 2012 From: perbu at varnish-software.com (Per Buer) Date: Tue, 3 Jan 2012 15:22:26 +0100 Subject: generate a list of objects In-Reply-To: References: Message-ID: Hi Steve. On Fri, Dec 16, 2011 at 2:16 PM, Steve A wrote: > Hi, > > Is there any way to generate a list of object keys (host + url) that are > present in a running varnish cache? > No. There is no such way at the moment. -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Tue Jan 3 14:24:13 2012 From: perbu at varnish-software.com (Per Buer) Date: Tue, 3 Jan 2012 15:24:13 +0100 Subject: Rewriting/enforcing SSL behing an SSL termination point In-Reply-To: References: Message-ID: Hi Cian, On Sun, Jan 1, 2012 at 2:22 AM, Cian Mc Govern wrote: I'm having an issue when I try to implement the redirect and reload varnish: > > Message from VCC-compiler: > Expected variable, string or semicolon > (input Line 17 Pos 68) > set req.http.x-redir-url = "https://www.cianmcgovern.com/" + > req.url; > > ---------------------------------------------------------------------------#--------- > > Can't find a solution to this but I suspect it might be due to the > version, 2.1.5, I'm using?? > In 2.1 string concatenation looks like this: set req.http.x-redir-url = "https://www.cianmcgovern.com/" req.url; In 3.0 this was done explicit using "+". > Thanks for any help! > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto.fernandezcrisial at gmail.com Tue Jan 3 14:53:09 2012 From: roberto.fernandezcrisial at gmail.com (=?ISO-8859-1?Q?Roberto_O=2E_Fern=E1ndez_Crisial?=) Date: Tue, 3 Jan 2012 11:53:09 -0300 Subject: Problem with Virtual Host In-Reply-To: <4EF24FBB.3040507@terra.cl> References: <4EF24FBB.3040507@terra.cl> Message-ID: Daniel, Which version of Varnish are you running? Have you tried "telnet www.omegasystems.cl 80" from Varnish server? Which was the response? -- Roberto O. Fern?ndez Crisial @rofc On Wed, Dec 21, 2011 at 6:29 PM, Daniel Benavides wrote: > mrs. > > I'm cashing server, if backend page not is virtual host, the cashing > working. > > but the backend page is a virtual host, the cashing not working > > I can not modify de webservers. > > example > ---working > > backend default { > .host = "www.sgi.com"; > .port = "80"; > } > > > ---- Not working > > backend default { > .host = "www.omegasystems.cl"; > .port = "80"; > } > > Please help me > > > ______________________________**_________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/**lists/mailman/listinfo/**varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jhalfmoon at milksnot.com Thu Jan 5 12:40:38 2012 From: jhalfmoon at milksnot.com (Johnny Halfmoon) Date: Thu, 05 Jan 2012 13:40:38 +0100 Subject: Possible bug: Missing X-Forwarded-For and X-Varnish headers (Varnish 2.1.5 64bit Linux) In-Reply-To: References: <4EE7DCB4.3050107@milksnot.com> Message-ID: <4F059A46.5040600@milksnot.com> On 12/14/2011 09:36 AM, Per Buer wrote: > Hi Johnny, > > On Wed, Dec 14, 2011 at 12:16 AM, Johnny Halfmoon > wrote: > > Hi, > > I have a problem with Varnish (2.1.5) not adding 'X-Forwarded-For' and an 'X-Varnish' to a large portion of the backend requests it makes. > > > Typically this happens when people are piping requests and forget to set a Connection: close. So then the TCP socket stays open and the browsers keeps on talking to the backend. Is this what is going on with you? > > Per Buer, CEO Hi, A bit late, but I would like to update this thread by closing it correctly with a summary of the solution. This issue has been resolved very quickly and correctly by the Varnish support team, on the same day that the issue was reported. Thanks guys! The problem was not caused by a bug in Varnish at all, but by 2 errors in my VCL files, which indeed, as Per suggested, led to pipe connections remaining open. This is what happened: 1. POST requests were accidentally being handled by pipe(). I completely missed this in the config, and was convinced pipe was not being used. This was my config: if ( req.request != "GET"&& req.request != "HEAD"&& req.request != "PURGE") { return(pipe); } 2. To add to this error, another config errror was causing whole groups of requests to be piped because piped requests were not being 'closed'. But I thought they were, so I was looking elsewhere for the solution. I happened to be a silly typo. I was doing this: vcl_pipe() { set req.http.connection = "close"; } which is wrong and instead I should have been doing this: vcl_pipe() { set bereq.http.connection = "close"; } So in the end, closing the pipe connections properly fixed the problem. Again, thanks for the great support you guys! Cheers, Johnny From cyberroadie at gmail.com Thu Jan 5 13:50:18 2012 From: cyberroadie at gmail.com (Olivier Van Acker) Date: Thu, 5 Jan 2012 13:50:18 +0000 Subject: Reordering query parameters Message-ID: I've written a small piece of C code which can be embedded in the varnish configuration. It puts the query parameters of an URL in order, so equal URLs with their parameters in different order become the same blog post explaining it: http://cyberroadie.wordpress.com/2012/01/05/varnish-reordering-query-string/ source code: https://github.com/cyberroadie/varnish-urlsort I haven't used this in a large scale live environment (yet), so please be careful and test first! Hope its useful, Olivier -------------- next part -------------- An HTML attachment was scrubbed... URL: From straightflush at gmail.com Thu Jan 5 15:28:59 2012 From: straightflush at gmail.com (AD) Date: Thu, 5 Jan 2012 10:28:59 -0500 Subject: Reordering query parameters In-Reply-To: References: Message-ID: very cool, want some help turning this into a vmod? inline C is sooo 2.0 :-) This is actually very helpful to have a consistent cache key when you get the same URL but different order of query params. Cheers, AD On Thu, Jan 5, 2012 at 8:50 AM, Olivier Van Acker wrote: > I've written a small piece of C code which can be embedded in the varnish > configuration. It puts the query parameters of an URL in order, so equal > URLs with their parameters in different order become the same > blog post explaining it: > > http://cyberroadie.wordpress.com/2012/01/05/varnish-reordering-query-string/ > > source code: > https://github.com/cyberroadie/varnish-urlsort > > I haven't used this in a large scale live environment (yet), so please be > careful and test first! > > Hope its useful, > > Olivier > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto.fernandezcrisial at gmail.com Thu Jan 5 15:30:39 2012 From: roberto.fernandezcrisial at gmail.com (=?ISO-8859-1?Q?Roberto_O=2E_Fern=E1ndez_Crisial?=) Date: Thu, 5 Jan 2012 12:30:39 -0300 Subject: Reordering query parameters In-Reply-To: References: Message-ID: Any extra help? I think this is very interesting. -- Roberto O. Fern?ndez Crisial @rofc On Thu, Jan 5, 2012 at 12:28 PM, AD wrote: > very cool, want some help turning this into a vmod? inline C is sooo 2.0 > :-) > > This is actually very helpful to have a consistent cache key when you get > the same URL but different order of query params. > > Cheers, > AD > > On Thu, Jan 5, 2012 at 8:50 AM, Olivier Van Acker wrote: > >> I've written a small piece of C code which can be embedded in the varnish >> configuration. It puts the query parameters of an URL in order, so equal >> URLs with their parameters in different order become the same >> blog post explaining it: >> >> http://cyberroadie.wordpress.com/2012/01/05/varnish-reordering-query-string/ >> >> source code: >> https://github.com/cyberroadie/varnish-urlsort >> >> I haven't used this in a large scale live environment (yet), so please be >> careful and test first! >> >> Hope its useful, >> >> Olivier >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cyberroadie at gmail.com Thu Jan 5 15:36:05 2012 From: cyberroadie at gmail.com (Olivier Van Acker) Date: Thu, 5 Jan 2012 15:36:05 +0000 Subject: Reordering query parameters In-Reply-To: References: Message-ID: very cool, want some help turning this into a vmod? inline C is sooo 2.0 > :-) > > Haha, I feel like an old fart now ;-) Yes I could use some help turning this in a vmod, reading the doc now but it's kinda limited Olivier > This is actually very helpful to have a consistent cache key when you get > the same URL but different order of query params. > > Cheers, > AD > > On Thu, Jan 5, 2012 at 8:50 AM, Olivier Van Acker wrote: > >> I've written a small piece of C code which can be embedded in the varnish >> configuration. It puts the query parameters of an URL in order, so equal >> URLs with their parameters in different order become the same >> blog post explaining it: >> >> http://cyberroadie.wordpress.com/2012/01/05/varnish-reordering-query-string/ >> >> source code: >> https://github.com/cyberroadie/varnish-urlsort >> >> I haven't used this in a large scale live environment (yet), so please be >> careful and test first! >> >> Hope its useful, >> >> Olivier >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From straightflush at gmail.com Thu Jan 5 15:48:49 2012 From: straightflush at gmail.com (AD) Date: Thu, 5 Jan 2012 10:48:49 -0500 Subject: Reordering query parameters In-Reply-To: References: Message-ID: i will create branch of your github repo and stick it in a "vmod" subdir if thats OK ? On Thu, Jan 5, 2012 at 10:36 AM, Olivier Van Acker wrote: > > > very cool, want some help turning this into a vmod? inline C is sooo 2.0 >> :-) >> >> > Haha, I feel like an old fart now ;-) > Yes I could use some help turning this in a vmod, reading the doc now but > it's kinda limited > > Olivier > > > >> This is actually very helpful to have a consistent cache key when you get >> the same URL but different order of query params. >> >> Cheers, >> AD >> >> On Thu, Jan 5, 2012 at 8:50 AM, Olivier Van Acker wrote: >> >>> I've written a small piece of C code which can be embedded in the >>> varnish configuration. It puts the query parameters of an URL in order, so >>> equal URLs with their parameters in different order become the same >>> blog post explaining it: >>> >>> http://cyberroadie.wordpress.com/2012/01/05/varnish-reordering-query-string/ >>> >>> source code: >>> https://github.com/cyberroadie/varnish-urlsort >>> >>> I haven't used this in a large scale live environment (yet), so please >>> be careful and test first! >>> >>> Hope its useful, >>> >>> Olivier >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>> >> >> > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cyberroadie at gmail.com Thu Jan 5 15:52:47 2012 From: cyberroadie at gmail.com (Olivier Van Acker) Date: Thu, 5 Jan 2012 15:52:47 +0000 Subject: Reordering query parameters In-Reply-To: References: Message-ID: i will create branch of your github repo and stick it in a "vmod" subdir if > thats OK ? > > Yes, go for it :-) Or I can also add you as a collaborator to this repo, what's your github name? -------------- next part -------------- An HTML attachment was scrubbed... URL: From straightflush at gmail.com Thu Jan 5 16:00:45 2012 From: straightflush at gmail.com (AD) Date: Thu, 5 Jan 2012 11:00:45 -0500 Subject: Reordering query parameters In-Reply-To: References: Message-ID: I just forked it, will let you know when committed (denen99) On Thu, Jan 5, 2012 at 10:52 AM, Olivier Van Acker wrote: > > > > i will create branch of your github repo and stick it in a "vmod" subdir >> if thats OK ? >> >> > Yes, go for it :-) Or I can also add you as a collaborator to this repo, > what's your github name? > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ruben at varnish-software.com Thu Jan 5 17:45:04 2012 From: ruben at varnish-software.com (=?UTF-8?Q?Rub=C3=A9n_Romero?=) Date: Thu, 5 Jan 2012 18:45:04 +0100 Subject: Welcome to VUG5 in Paris, France - March 22-23, 2012 Message-ID: Hello to you all, The year has started and we need to plan ahead. So it is my pleasure to announce that our Fifth Varnish User Group meeting, VUG5, will happen in Paris, France. The dates are March 22nd (User Day) and 23rd (Dev Day), 2012. For all the gory details and registration visit the following page on the Varnish Cache community website: As we have a rather limited number of seats we need to know the number of assistants in advance, so register even if you are not 100% sure about coming to Paris. We will send you a confirmation reminder later. We need YOU! -> Do not forget to add your favorite discussion item (or your own Varnish use case) to the agenda. This goes both for the User and Dev days. We want to know how you use the software, which problems you solve with it and what challenges you have so we can together make an even better Varnish Cache! If you have *any* questions related to this meeting, do not hesitate in contacting me directly and I will get back to you. Last but not least: Hope to see you all in Paris! Best wishes, -- Rub?n Romero, Self-appointed <3 Non-Official VUG Coordinator & Cheerleader Phone: +47 21 98 92 62 / Mobile: +47 959 64 088 / Skype: ruben_varnish / GTalk: *Varnish makes websites fly!* Whitepapers | Video | Twitter | LinkedIn -------------- next part -------------- An HTML attachment was scrubbed... URL: From straightflush at gmail.com Thu Jan 5 18:09:45 2012 From: straightflush at gmail.com (AD) Date: Thu, 5 Jan 2012 13:09:45 -0500 Subject: Reordering query parameters In-Reply-To: References: Message-ID: Roberto/Oliver, I forked and committed the vmod to https://github.com/denen99/libvmod-urlsort. I just copied your source and updated the vmod_urlsort.c file to include your source and create the necessary vcc file. To compile i just did ./autogen.sh VARNISHSRC=/Downloads/varnish-3.0.2 VMODDIR=/tmp/vmod ./configure make make install then copy /tmp/vmod/libvmod_urlsort.* to your lib/vmod directory of varnish. I havent tested the vcl yet, so if someone can do "import urlsort;" and try using it that would be great... i will try to test a bit later. AD On Thu, Jan 5, 2012 at 11:00 AM, AD wrote: > I just forked it, will let you know when committed (denen99) > > > > On Thu, Jan 5, 2012 at 10:52 AM, Olivier Van Acker wrote: > >> >> >> >> i will create branch of your github repo and stick it in a "vmod" subdir >>> if thats OK ? >>> >>> >> Yes, go for it :-) Or I can also add you as a collaborator to this repo, >> what's your github name? >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bedis9 at gmail.com Thu Jan 5 20:55:09 2012 From: bedis9 at gmail.com (Baptiste) Date: Thu, 5 Jan 2012 21:55:09 +0100 Subject: Welcome to VUG5 in Paris, France - March 22-23, 2012 In-Reply-To: References: Message-ID: Hey Ruben, Just registred! I'm looking forward to meet you again. cheers 2012/1/5 Rub?n Romero > Hello to you all, > > The year has started and we need to plan ahead. So it is my pleasure to > announce that our Fifth Varnish User Group meeting, VUG5, will happen in > Paris, France. The dates are March 22nd (User Day) and 23rd (Dev Day), > 2012. For all the gory details and registration visit the following page on > the Varnish Cache community website: > > > > As we have a rather limited number of seats we need to know the number of > assistants in advance, so register even if you are not 100% sure about > coming to Paris. We will send you a confirmation reminder later. > > We need YOU! -> Do not forget to add your favorite discussion item (or > your own Varnish use case) to the agenda. This goes both for the User and > Dev days. We want to know how you use the software, which problems you > solve with it and what challenges you have so we can together make an even > better Varnish Cache! > > If you have *any* questions related to this meeting, do not hesitate in > contacting me directly and I will get back to you. > > Last but not least: Hope to see you all in Paris! > > > Best wishes, > -- > Rub?n Romero, Self-appointed <3 Non-Official VUG Coordinator & Cheerleader > Phone: +47 21 98 92 62 / Mobile: +47 959 64 088 / Skype: ruben_varnish / > GTalk: > *Varnish makes websites fly!* > Whitepapers | Video > | Twitter | LinkedIn > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfheen at varnish-software.com Fri Jan 6 10:09:29 2012 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Fri, 06 Jan 2012 11:09:29 +0100 Subject: RHEL5 build issue In-Reply-To: (Marc Ochs's message of "Thu, 29 Dec 2011 15:15:52 -0500") References: Message-ID: <87vcopdtwm.fsf@qurzaw.varnish-software.com> ]] Marc Ochs > I'm having some issue building from souce where when I try to run make I > get a bunch of errors like this: [...] > Something with my echo or sed but I'm not sure... any help greatly > appreciated. Sorry in advance if I'm missing something super obvious. Not sure which version you're trying to build, but there's some oddness in RHEL5 wrt libtool, so I'd recommend just grabbing the tarballs and building those (or using our provided binary packages instead). -- Tollef Fog Heen Technical lead, Varnish Software t: +47 21 98 92 64 From tfheen at varnish-software.com Fri Jan 6 10:10:39 2012 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Fri, 06 Jan 2012 11:10:39 +0100 Subject: Varnish 3.0.1 Backend conditional requests In-Reply-To: (=?utf-8?B?ItCh0LXRgNCz0LXQuSDQr9C80YnQuNC60L7QsiIncw==?= message of "Mon, 19 Dec 2011 18:10:48 +0300") References: Message-ID: <87r4zddtuo.fsf@qurzaw.varnish-software.com> ]] ?????? ??????? > Hello. How to enable Backend conditional requests (If-None-Match and > If-Modified-Since) for the Varnish 3.0.1. Now it doesn't send to me > headers (If-None-Match and If-Modified-Since) when max-age=0 or > expired, on first response i set ETag and Last-Modified headers. Subs > and stale_obj from > https://www.varnish-cache.org/trac/wiki/BackendConditionalRequests > throw exception in "*.vcl" file when compile. As it says on that wiki page: it's a proposed feature, it's not yet incorporated. -- Tollef Fog Heen Technical lead, Varnish Software t: +47 21 98 92 64 From ochs at marcochs.com Fri Jan 6 14:15:09 2012 From: ochs at marcochs.com (Marc Ochs) Date: Fri, 6 Jan 2012 09:15:09 -0500 Subject: RHEL5 build issue In-Reply-To: <87vcopdtwm.fsf@qurzaw.varnish-software.com> References: <87vcopdtwm.fsf@qurzaw.varnish-software.com> Message-ID: Thanks, I meant to reply that I actually got past this (after some googling) to build 3.0.2 by doing this before autogen, which seemed to fix the parsing issue: cp /usr/share/aclocal/libtool.m4 ./acinclude.m4 These are my versions: Installed Packages autoconf.noarch 2.59-12 installed libtool.x86_64 1.5.22-7.el5_4 installed $ cat /etc/redhat-release Red Hat Enterprise Linux Server release 5.5 (Tikanga) I'm not clear on whether this is an issue with my version of libtool or autoconf. On Fri, Jan 6, 2012 at 5:09 AM, Tollef Fog Heen wrote: > ]] Marc Ochs > > > I'm having some issue building from souce where when I try to run make I > > get a bunch of errors like this: > > [...] > > > Something with my echo or sed but I'm not sure... any help greatly > > appreciated. Sorry in advance if I'm missing something super obvious. > > Not sure which version you're trying to build, but there's some oddness > in RHEL5 wrt libtool, so I'd recommend just grabbing the tarballs and > building those (or using our provided binary packages instead). > > -- > Tollef Fog Heen > Technical lead, Varnish Software > t: +47 21 98 92 64 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hugo.cisneiros at gmail.com Fri Jan 6 14:52:19 2012 From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch)) Date: Fri, 6 Jan 2012 12:52:19 -0200 Subject: Varnish 3.0.1 Backend conditional requests In-Reply-To: <87r4zddtuo.fsf@qurzaw.varnish-software.com> References: <87r4zddtuo.fsf@qurzaw.varnish-software.com> Message-ID: 2012/1/6 Tollef Fog Heen : >> Hello. How to enable Backend conditional requests (If-None-Match and >> If-Modified-Since) for the Varnish 3.0.1. Now it doesn't send to me >> headers (If-None-Match and If-Modified-Since) when max-age=0 or >> expired, on first response i set ETag and Last-Modified headers. Subs >> and stale_obj from >> https://www.varnish-cache.org/trac/wiki/BackendConditionalRequests >> throw exception in "*.vcl" file when compile. > > As it says on that wiki page: it's a proposed feature, it's not yet > incorporated. By the way, I have tested the experimental-ims branch which contains the feature and it's working nice! :) -- []'s Hugo www.devin.com.br From cyberroadie at gmail.com Fri Jan 6 16:59:37 2012 From: cyberroadie at gmail.com (Olivier Van Acker) Date: Fri, 6 Jan 2012 16:59:37 +0000 Subject: Reordering query parameters In-Reply-To: References: Message-ID: Merged, a bit of a cleanup and written a man page, code can be found here: https://github.com/cyberroadie/varnish-urlsort Thanks AD for the vmod help :-) Olivier On 5 January 2012 18:09, AD wrote: > Roberto/Oliver, > > I forked and committed the vmod to > https://github.com/denen99/libvmod-urlsort. > > I just copied your source and updated the vmod_urlsort.c file to include > your source and create the necessary vcc file. > > To compile i just did > > ./autogen.sh > VARNISHSRC=/Downloads/varnish-3.0.2 VMODDIR=/tmp/vmod ./configure > make > make install > > then copy /tmp/vmod/libvmod_urlsort.* to your lib/vmod directory of > varnish. > > I havent tested the vcl yet, so if someone can do "import urlsort;" and > try using it that would be great... i will try to test a bit later. > > AD > > > On Thu, Jan 5, 2012 at 11:00 AM, AD wrote: > >> I just forked it, will let you know when committed (denen99) >> >> >> >> On Thu, Jan 5, 2012 at 10:52 AM, Olivier Van Acker > > wrote: >> >>> >>> >>> >>> i will create branch of your github repo and stick it in a "vmod" subdir >>>> if thats OK ? >>>> >>>> >>> Yes, go for it :-) Or I can also add you as a collaborator to this repo, >>> what's your github name? >>> >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jocelyn.delarosa at smartjog.com Mon Jan 9 10:55:09 2012 From: jocelyn.delarosa at smartjog.com (Jocelyn De La Rosa) Date: Mon, 09 Jan 2012 11:55:09 +0100 Subject: Varnish 3.0.1 Backend conditional requests In-Reply-To: References: <87r4zddtuo.fsf@qurzaw.varnish-software.com> Message-ID: <4F0AC78D.8010506@smartjog.com> On 01/06/2012 03:52 PM, Hugo Cisneiros (Eitch) wrote: > 2012/1/6 Tollef Fog Heen: > >>> Hello. How to enable Backend conditional requests (If-None-Match and >>> If-Modified-Since) for the Varnish 3.0.1. Now it doesn't send to me >>> headers (If-None-Match and If-Modified-Since) when max-age=0 or >>> expired, on first response i set ETag and Last-Modified headers. Subs >>> and stale_obj from >>> https://www.varnish-cache.org/trac/wiki/BackendConditionalRequests >>> throw exception in "*.vcl" file when compile. >>> >> As it says on that wiki page: it's a proposed feature, it's not yet >> incorporated. >> > By the way, I have tested the experimental-ims branch which contains > the feature and it's working nice! :) > > Hi,I have also tested the experimental-ims branch and it's working great! Are there any (hidden) bugs that prevent its use in production mode? We are very interested by this feature, so if some work is needed we could do it. -- Jocelyn De La Rosa Smartjog - Developer - Research& Engineering From perbu at varnish-software.com Mon Jan 9 11:20:56 2012 From: perbu at varnish-software.com (Per Buer) Date: Mon, 9 Jan 2012 12:20:56 +0100 Subject: Varnish 3.0.1 Backend conditional requests In-Reply-To: <4F0AC78D.8010506@smartjog.com> References: <87r4zddtuo.fsf@qurzaw.varnish-software.com> <4F0AC78D.8010506@smartjog.com> Message-ID: On Mon, Jan 9, 2012 at 11:55 AM, Jocelyn De La Rosa < jocelyn.delarosa at smartjog.com> wrote: Hi,I have also tested the experimental-ims branch and it's working great! > Are there any (hidden) bugs that prevent its use in production mode? > There are currently 5 bugs which we have not yet discovered. :-) On a serious note, if you _really_ need it you could try it carefully in production. Try it with varnishreplay, then try giving it a small fraction of your production load before giving it all of your traffic. After all if it crashes it is not the end of the world - you'd loose the content of the cache but hopefully not more. -- Per Buer Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at uplex.de Mon Jan 9 12:23:27 2012 From: geoff at uplex.de (Geoff Simmons) Date: Mon, 09 Jan 2012 13:23:27 +0100 Subject: Varnish 3.0.1 Backend conditional requests In-Reply-To: <4F0AC78D.8010506@smartjog.com> References: <87r4zddtuo.fsf@qurzaw.varnish-software.com> <4F0AC78D.8010506@smartjog.com> Message-ID: <4F0ADC3F.7000707@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 01/ 9/12 11:55 AM, Jocelyn De La Rosa wrote: >> > Hi,I have also tested the experimental-ims branch and it's working great! > Are there any (hidden) bugs that prevent its use in production mode? > > We are very interested by this feature, so if some work is needed we could > do it. Actually, there may be some hidden bugs. Kristian Lyngstol set up some automated stress tests for the various branches a few months ago, and the experimental-ims branch was not passing all of them. (The branch has been out of the automation for a while, I presume because there haven't been any updates.) Also the branch is not currently up to date with the changes that have been done on the master branch, especially the changes in memory management (workspaces). These matters are on my to do list. %^) Let's please get in touch offline if you plan to make contributions. phk recently announced plans on the dev list for an upcoming new release, including adding the IMS feature. These issues will need to be resolved by then, of course. IMS actually almost made it into 3.0, but there was some unresolved discussion then about whether the configuration in VCL as currently implemented is the best way to go. There was an idea under consideration that, rather than have stale_obj and the two different keep and grace intervals after TTL elapses, there could be a new vcl sub (say, vcl_stale()) to cover both the grace and keep cases. I've had some feedback from people who've tried the experimental-ims branch that they're happy with the configuration the way it is. At any rate, a consensus about that will of course be necessary. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Schwanenwik 24 22087 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (SunOS) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJPCtw/AAoJEOUwvh9pJNURKPsP+gLCVhgYhk4YamuxoMKMf1rD Mgm9ho62ih9s5mqShcAvJfhgj6asnUi8RbfYCX1uHsdxa4ohyZjXFvVyrBLZ140i +6erB9WT+tkTdsAUJgRrKfThhL29P+CpliTqtettmYAdjxVvEeDcDuDGWu20W6AF EtkO8NdtocE+ZiYF1PwSHz2hiks81hjdBrjB3Dc1YNtdEVX6wE2tiTS3LvGBSQBP b17G7+sXVg160e7EYZJoY/tJpm8Z08ab6uS7IixRZPohRPHz/78/G8RF+T7eI1a8 ilThPViSYZsczhNGu//LmDxI//+CCz/mEXkAU6jMWI6I/wy3kYWX6yDuxdcA39e/ t2qNo25bbr/VZH9eCPeLn2l07kGSXQNdZEpe2vitjAj6TZkHKaWdIDl361/VogK3 ZFu1zyLR8pDVUHySzZD0Vfq9MYKK9d3auzIwsA98ouL0ld+yEv8fn6163biZu+3c UY/ZH6HupT2z6vC0xhAeo+GU36JMVIDMOmGnkGcmonIzIOrFXzqU+yZzQtFDm3iW F3yyy1g4aQV1Clr75QayhPTrlZPu008CjCN71+eGg8a9QXTHSJjCT5QKolEJCjUn aYBjLXOVxXFsJIxGEIhnxx1OeB9BQAfKoqPt855Bzv/AIW93LJM723Xtlpw+qMpk RjbCEIrSjDgGYHBDazil =Qmg7 -----END PGP SIGNATURE----- From roland at mohrbacher.eu Tue Jan 10 10:11:27 2012 From: roland at mohrbacher.eu (Roland Mohrbacher) Date: Tue, 10 Jan 2012 11:11:27 +0100 Subject: Persistent Varnish crashes since using bans and lurker Message-ID: <4F0C0ECF.6030005@mohrbacher.eu> Hello all, we use a farm with three persistent Varnishes (-s persistent,/cms/varnish_cache/persistent/varnish_storage.bin,204800M"). This Varnishes runs since 3 months without any crashes (in the moment not in production, but stressed with several stress tests). Since some days, we use bans and the lurker process (lurker-friendly bans via: ban("obj.http.x-url ~ " + req.url); We have about 250 bans/hour. Now we have the big problem, that the varnishes crashes after some hours. Curios: all three Varnishes crashes in the same moment. And they runs on three different Servers! The follow part from syslog suggest, that there is an problem with an invalid ban: Jan 9 19:40:32 ece-fe1 /var/lib/varnish/persistent[19622]: Child (19623) said CHK(0x7f91ffd261a0 BAN 2 0x7f34724f4000 ) = 1 Jan 9 19:40:33 ece-fe1 /var/lib/varnish/persistent[19622]: Child (19623) died signal=6 Jan 9 19:40:33 ece-fe1 /var/lib/varnish/persistent[19622]: Child (19623) Panic message: Missing errorhandling code in smp_append_sign(), storage_persistent_subr.c line 128:#012 Condition((smp_chk_sign(ctx)) == 0) not true.thread = (cache-worker)#012ident = Linux,2.6.32-131.2.1.el6.x86_64,x86_64,-spersistent,-smalloc,-hcritbit,epoll#012Backtrace:#012 0x42c7a6: /usr/sbin/varnishd() [0x42c7a6]#012 0x44a346: /usr/sbin/varnishd(smp_append_sign+0x126) [0x44a346]#012 0x447b6d: /usr/sbin/varnishd(SMP_NewBan+0x3d) [0x447b6d]#012 0x4125c7: /usr/sbin/varnishd(BAN_Insert+0x1a7) [0x4125c7]#012 0x433bd5: /usr/sbin/varnishd(VRT_ban_string+0xc5) [0x433bd5]#012 0x7f91f39fa4be: ./vcl.PNU3fGhs.so(+0x24be) [0x7f91f39fa4be]#012 0x433863: /usr/sbin/varnishd(VCL_recv_method+0x43) [0x433863]#012 0x417c22: /usr/sbin/varnishd(CNT_Session+0xb62) [0x417c22]#012 0x42efb8: /usr/sbin/varnishd() [0x42efb8]#012 0x42e19b: /usr/sbin/varnishd() [0x42e19b]#012sp = 0x7f91ed4ab008 {#012 fd = 15, id = 15, xid = 683670119,#012 client = 172.27.70.103 36115,#012 step = STP_RECV,#012 handling = deliver,#012 restarts = 0, esi_level = 0#012 flags = #012 bodystatus = 4#012 ws = 0x7f91ed4ab080 { #012 id = "sess",#012 {s,f,r,e} = {0x7f91ed4abc90,+56,(nil),+65536},#012 },#012 http[req] = {#012 ws = 0x7f91ed4ab080[sess]#012 "PURGE",#012 "105867846",#012 "HTTP/1.0",#012 },#012 worker = 0x7f91ef1faa80 {#012 ws = 0x7f91ef1facc0 { #012 id = "wrk",#012 {s,f,r,e} = {0x7f91ef1e8a30,+32,(nil),+65536},#012 },#012 },#012 vcl = {#012 srcname = {#012 "input",#012 "Default",#012 },#012 },#012},#012 Jan 9 19:40:33 ece-fe1 /var/lib/varnish/persistent[19622]: child (6907) Started Jan 9 19:40:33 ece-fe1 /var/lib/varnish/persistent[19622]: Pushing vcls failed:#012CLI communication error (hdr) Jan 9 19:40:33 ece-fe1 /var/lib/varnish/persistent[19622]: Child (6907) died signal=6 Jan 9 19:40:33 ece-fe1 /var/lib/varnish/persistent[19622]: Child (6907) Panic message: Assert error in smp_open(), storage_persistent.c line 320:#012 Condition((smp_valid_silo(sc)) == 0) not true.#012thread = (cache-main)#012ident = Linux,2.6.32-131.2.1.el6.x86_64,x86_64,-spersistent,-smalloc,-hcritbit,no_waiter#012Backtrace:#012 0x42c7a6: /usr/sbin/varnishd() [0x42c7a6]#012 0x44756a: /usr/sbin/varnishd() [0x44756a]#012 0x444d57: /usr/sbin/varnishd(STV_open+0x27) [0x444d57]#012 0x42b525: /usr/sbin/varnishd(child_main+0xc5) [0x42b525]#012 0x43d5ec: /usr/sbin/varnishd() [0x43d5ec]#012 0x43de7c: /usr/sbin/varnishd() [0x43de7c]#012 0x7f92015684c7: /usr/lib64/varnish/libvarnish.so(+0x94c7) [0x7f92015684c7]#012 0x7f9201568b58: /usr/lib64/varnish/libvarnish.so(vev_schedule+0x88) [0x7f9201568b58]#012 0x43d7c2: /usr/sbin/varnishd(MGT_Run+0x132) [0x43d7c2]#012 0x44cacb: /usr/sbin/varnishd(main+0xd1b) [0x44cacb]#012 Jan 9 19:40:33 ece-fe1 /var/lib/varnish/persistent[19622]: Child (-1) said Child starts Jan 9 19:40:33 ece-fe1 /var/lib/varnish/persistent[19622]: Child (-1) said CHK(0x7f91ffd26120 BAN 1 0x7f34723f4000 BAN 1) = 4 Jan 9 19:40:33 ece-fe1 /var/lib/varnish/persistent[19622]: Child (-1) said CHK(0x7f91ffd261a0 BAN 2 0x7f34724f4000 ) = 1 Is this an known problem? Are there work a rounds to use persistent Varnish together with lurkers? Best regards Roland From phk at phk.freebsd.dk Tue Jan 10 10:16:43 2012 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 10 Jan 2012 10:16:43 +0000 Subject: Persistent Varnish crashes since using bans and lurker In-Reply-To: Your message of "Tue, 10 Jan 2012 11:11:27 +0100." <4F0C0ECF.6030005@mohrbacher.eu> Message-ID: <60165.1326190603@critter.freebsd.dk> In message <4F0C0ECF.6030005 at mohrbacher.eu>, Roland Mohrbacher writes: Please open a ticket with this data. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From jan at architechs.eu Wed Jan 11 21:01:37 2012 From: jan at architechs.eu (=?UTF-8?Q?Jan=2DAage_Frydenb=C3=B8=2DBruvoll?=) Date: Wed, 11 Jan 2012 21:01:37 +0000 Subject: Cannot allocate memory Message-ID: Dear list, Lately we have had problems loading up new VCLs to our varnish machines, with the following message: Command failed with error code 106 VCL compiled.dlopen(./vcl.K_vwYyM9.so): ./vcl.K_vwYyM9.so: failed to map segment from shared object: Cannot allocate memory The list of VCLs loaded at the time of failure was: $ varnishadm vcl.list CLI connected to 10.140.106.18 8081 available 0 boot available 0 jafb-test available 0 jafb-tes2 available 0 cap_release_1326203294 available 0 cap_release_1326203474 available 0 cap_release_1326204927 available 0 cap_release_1326210562 available 0 cap_release_1326275426 active 127 cap_release_1326275758 The machine in question should have plenty of memory free and available. The Varnish instance has an uptime of 48-50 hours. The Varnish version is varnish-3.0.1 revision 6152bf7 and the command line parameters used are -u nobody -f /etc/varnish/default.vcl -T :8081 -s malloc,14G -p thread_pools=4 -p thread_pool_max=1500 -p listen_depth=2048 -p lru_interval=1800 -h classic,169313 -p connect_timeout=600 Would anyone have any ideas on how we can pinpoint the source of the problem and/or whether there are any obvious tuning knobs we should have twiddled? Thanks in advance for your kind assistance. Best regards Jan From sufehmi at gmail.com Thu Jan 12 21:37:27 2012 From: sufehmi at gmail.com (Harry Sufehmi) Date: Fri, 13 Jan 2012 04:37:27 +0700 Subject: "not cacheable" even on static files Message-ID: Hi folks, for the past few days I've been puzzled by this problem - Varnish have refused to cache even static files. While it was fine before, and will cache even dynamic HTML files (with proper cache headers, of course) The problem begun when my client reported that his website was constantly going up & down. When I checked, it was under some sort of DDoS attack. This is not news, because his website was pretty much under 24x7 syn flood attack. However, this attack is now able to bypass HAproxy & Varnish - and hit Apache directly, right where it hurts most. To cut the long story short - basically now Varnish refuses to cache almost everything. The X-Cacheable header that I enabled contains the dreaded "NO:Not Cacheable" status. And I've not been able to find out why. This is where I stuck, if I can find out the cause, the I'd be able to rectify it. Anyway, the DDoS attack may not be related at all to the current problem (varnish not caching), I included the story just for the sake of completeness. Let's proceed to some facts that I've gathered : Same GIF file - but different sizes (note the "Received" column) : http://minus.com/mbawzSZUxJ#3 The URLs with port 8181 are direct requests to Varnish, bypassing HAproxy. HTTP Headers produced by Varnish : http://minus.com/mbawzSZUxJ#4 HTTP Headers produced by HAproxy : http://minus.com/mbawzSZUxJ#2 List of cookies sent by browser & received from Varnish : http://minus.com/mbawzSZUxJ#1 (stripped clean by varnish, basically) Please find the result of varnishstat -1 & content of /etc/varnish/default.vcl attached to the end of this email. Varnish is running with the following parameters : $ ps aux|grep varnish myuser 3327 0.0 0.0 3324 796 pts/6 S+ 04:36 0:00 grep varnish root 8441 0.0 1.0 86024 83532 pts/2 SL+ Jan12 0:01 /usr/sbin/varnishd -P /var/run/varnishd.pid -a 0.0.0.0:8181 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 180 -w 2,1000,30 -s malloc,2G -d Hopefully someone will be able to point / give me a hint to the right direction. Thanks, Harry =================== $ telnet localhost 6082 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. 200 199 ----------------------------- Varnish Cache CLI 1.0 ----------------------------- Linux,2.6.32-25-generic-pae,i686,-smalloc,-hcritbit Type 'help' for command list. Type 'quit' to close CLI session. stats 200 1978 16564302 Client connections accepted 16514647 Client requests received 2378064 Cache hits 988836 Cache misses 592750 Backend conn. success 462880 Backend conn. failures 13096234 Backend conn. reuses 63305 Backend conn. was closed 13160380 Backend conn. recycles 82 Fetch head 9937334 Fetch with Length 1856 Fetch chunked 245 Fetch wanted close 66 Fetch failed 26436 N struct sess_mem 26091 N struct sess 18773 N struct object 18862 N struct objectcore 6827 N struct objecthead 3 N struct vbe_conn 107 N worker threads 26496 N worker threads created 144188 N overflowed work requests 1 N backends 932755 N expired objects 1851315 N LRU moved objects 12437230 Objects sent with write 16564302 Total Sessions 16514647 Total Requests 870 Total pipe 13146877 Total pass 13630039 Total fetch 6464791313 Total header bytes 122615308544 Total body bytes 16564289 Session Closed 11 Session Linger 17 Session herd 1364697125 SHM records 93867958 SHM writes 2574 SHM flushes due to overflow 446883 SHM MTX contention 540 SHM cycles through buffer 10815252 SMA allocator requests 37554 SMA outstanding allocations 45497812 SMA outstanding bytes 114575871831 SMA bytes allocated 114530374019 SMA bytes free 505674 SMS allocator requests 211877406 SMS bytes allocated 211877406 SMS bytes freed 13688813 Backend requests made 1 N vcl total 1 N vcl available 1 N total active purges 1 N new purges added 3365986 HCB Lookups without lock 122890 HCB Lookups with lock 122890 HCB Inserts 99485 Client uptime 16426 Backend conn. retry 3690588 Fetch no body (304) =================== default.vcl =============== $ cat /etc/varnish/default.vcl # Default backend definition. Set this to point to your content # server. backend default { .host = "127.0.0.1"; .port = "81"; } sub vcl_deliver { if (obj.hits > 0) { set resp.http.X-Cache = "HIT"; } else { set resp.http.X-Cache = "MISS"; } } # Below is a commented-out copy of the default VCL logic. If you # redefine any of these subroutines, the built-in logic will be # appended to your code. # sub vcl_recv { if (req.backend.healthy) { set req.grace = 180s; } else { set req.grace = 1h; } } sub vcl_fetch { set beresp.grace = 1h; # Varnish determined the object was not cacheable if (!beresp.cacheable) { set beresp.http.X-Cacheable = "NO:Not Cacheable"; # You don't wish to cache content for logged in users } elsif (req.http.Cookie ~ "(UserID|_session)") { set beresp.http.X-Cacheable = "NO:Got Session"; return(pass); # You are respecting the Cache-Control=private header from the backend } elsif (beresp.http.Cache-Control ~ "private") { set beresp.http.X-Cacheable = "NO:Cache-Control=private"; return(pass); # You are extending the lifetime of the object artificially } elsif (beresp.ttl < 1s) { set beresp.ttl = 5s; set beresp.grace = 5s; set beresp.http.X-Cacheable = "YES:FORCED"; # Varnish determined the object was cacheable } else { set beresp.http.X-Cacheable = "YES"; } if (req.url ~ "\.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|pdf|txt|tar|wav|bmp|rtf|js|flv|swf|html|htm)$") { unset beresp.http.set-cookie; set beresp.ttl = 24h; } return(deliver); # if (!beresp.cacheable) { # return (pass); # } # if (beresp.http.Set-Cookie) { # return (pass); # } # return (deliver); } From drtaber at northcarolina.edu Thu Jan 12 22:48:11 2012 From: drtaber at northcarolina.edu (Douglas R Taber) Date: Thu, 12 Jan 2012 17:48:11 -0500 Subject: "Large" Binary Files triggering 503 Response Message-ID: Hi all- I'm relatively new to varnish, but i haven't seen/stumbled upon a similar situation in the last few months of mailing list archives. I'm having a weird issue where below approximately 200k, varnish returns binary files just fine, but above that it returns a 503 page. According to the log, apache is returning a 200 status code from the backend. I'm not really sure what to make of it, but it looks like the exact error being thrown back is: 11 FetchError c straight read_error: 0 0 (Remote closed connection) I've included my configs and the associated log segment. Hopefully someone can shed some light on it. The server version information is: Varnish Server: RHEL6.2 Varnish 3.0.2 (from the EL5 RPM) Apache Server: RHEL6.2 Apache 2.2.x The varnish command is(from /etc/sysconfig/varnish): DAEMON_OPTS="-a 80\ -f /etc/varnish/default.vcl \ -T 127.0.0.1:6082 \ -t 120 \ -w 10,5000,120 -u varnish -g varnish \ -S /etc/varnish/secret \ -p thread_pool_add_delay=2 \ -p thread_pools=4 \ -p thread_pool_min=200 \ -p thread_pool_max=4000 \ -p cli_timeout=25 \ -p session_linger=60 \ -p sess_timeout=60 \ -s malloc,10G \ -s $file,/var/lib/varnish/varnish_storage.bin,15G" The default config is: backend default { .host = "10.11.12.1"; .port = "80"; } include "/etc/varnish/federation.vcl"; include "/etc/varnish/mailwatch.vcl"; include "/etc/varnish/devcollab.vcl"; include "/etc/varnish/www.vcl"; include "/etc/varnish/collab.vcl"; include "/etc/varnish/uncdm.vcl"; include "/etc/varnish/services.vcl"; include "/etc/varnish/devfederation.vcl"; include "/etc/varnish/devservices.vcl"; include "/etc/varnish/devuncdm.vcl"; include "/etc/varnish/devwww.vcl"; sub vcl_recv { # if (req.url ~ "(?i)\.(png|gif|jpeg|jpg|ico|css|js|gz|tgz|bz2|tbz|mp3|ogg|mp4|flv|f4v|pdf|swf)(\?[a-z0-9]+)?$") { # unset req.http.Cookie; # return(lookup); # } if (req.restarts == 0) { if (req.http.x-forwarded-for) { set req.http.X-Forwarded-For = req.http.X-Forwarded-For; } else { set req.http.X-Forwarded-For = client.ip; } } if (req.request != "GET" && req.request != "HEAD" && req.request != "PUT" && req.request != "POST" && req.request != "TRACE" && req.request != "OPTIONS" && req.request != "DELETE") { /* Non-RFC2616 or CONNECT which is weird. */ return (pipe); } if (req.request != "GET" && req.request != "HEAD") { /* We only deal with GET and HEAD by default */ return (pass); } if (req.http.Authorization || req.http.Cookie) { /* Not cacheable by default */ return (pass); } return (lookup); } sub vcl_error { set obj.http.Content-Type = "text/html; charset=utf-8"; synthetic {"

The University of North Carolina General Administration

Our websites are currently down for maintenance.
We apologize for the inconvenience.

"}; return (deliver); } The error log is: 11 SessionOpen c 71.77.0.73 49448 :80 11 ReqStart c 71.77.0.73 49448 570029255 11 RxRequest c GET 11 RxURL c /QA/exams/standards_and_guidelines.pdf 11 RxProtocol c HTTP/1.1 11 RxHeader c Host: 152.4.21.240 11 RxHeader c User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/534.52.7 (KHTML, like Gecko) Version/5.1.2 Safari/534.52.7 11 RxHeader c Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 11 RxHeader c Accept-Language: en-us 11 RxHeader c Accept-Encoding: gzip, deflate 11 RxHeader c Cookie: __utma=67627729.194036926.1325818231.1325818231.1326406657.2; __utmb=67627729.5.10.1326406657; __utmc=67627729; __utmz=67627729.1325818231.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); PHPSESSID=4mvdfvuv1gd9h59m8nqg5gsgs1 11 RxHeader c Connection: keep-alive 11 VCL_call c recv lookup 11 VCL_call c hash 11 Hash c /QA/exams/standards_and_guidelines.pdf 11 Hash c 152.4.21.240 11 VCL_return c hash 11 VCL_call c miss fetch 11 Backend c 4 devservices devservices2 11 TTL c 570029255 RFC 0 -1 -1 1326407133 0 1326407118 375007920 0 11 VCL_call c fetch 11 TTL c 570029255 VCL 302 -1 -1 1326407131 -2 11 VCL_return c deliver 11 ObjProtocol c HTTP/1.1 11 ObjResponse c OK 11 ObjHeader c Date: Thu, 12 Jan 2012 22:25:18 GMT 11 ObjHeader c Server: Apache/2.2.15 (Red Hat) 11 ObjHeader c X-Powered-By: PHP/5.3.3 11 ObjHeader c Set-Cookie: PHPSESSID=af0ivec0rh864s6rifiirs0gu4; path=/ 11 ObjHeader c Expires: Thu, 19 Nov 1981 08:52:00 GMT 11 ObjHeader c Cache-Control: public, must-revalidate 11 ObjHeader c Pragma: hack 11 ObjHeader c X-Pad: avoid browser bug 11 ObjHeader c Content-Encoding: gzip 11 ObjHeader c Vary: Accept-Encoding 11 ObjHeader c Content-Type: application/pdf 11 FetchError c straight read_error: 0 0 (Remote closed connection) 11 Gzip c u F - 90795 96895 80 726280 726290 11 VCL_call c error deliver 11 VCL_call c deliver deliver 11 TxProtocol c HTTP/1.1 11 TxStatus c 503 11 TxResponse c Service Unavailable 11 TxHeader c Server: Varnish 11 TxHeader c Content-Type: text/html; charset=utf-8 11 TxHeader c Content-Length: 690 11 TxHeader c Accept-Ranges: bytes 11 TxHeader c Date: Thu, 12 Jan 2012 22:25:33 GMT 11 TxHeader c X-Varnish: 570029255 11 TxHeader c Age: 2 11 TxHeader c Via: 1.1 varnish 11 TxHeader c Connection: close 11 Length c 690 11 ReqEnd c 570029255 1326407130.792037964 1326407133.193643808 0.000042439 2.401573896 0.000031948 11 SessionClose c error 11 StatSess c 71.77.0.73 49448 2 1 1 0 0 0 240 690 Thanks in advance if anyone has any insight into this. From bedis9 at gmail.com Fri Jan 13 08:07:13 2012 From: bedis9 at gmail.com (Baptiste) Date: Fri, 13 Jan 2012 09:07:13 +0100 Subject: "not cacheable" even on static files In-Reply-To: References: Message-ID: Hi Harry, Is HAProxy in front of your Varnish servers or between varnish and Apache? You could use HAProxy to mitigate the attack, if you know the pattern, I can help on this point. cheers On Thu, Jan 12, 2012 at 10:37 PM, Harry Sufehmi wrote: > Hi folks, for the past few days I've been puzzled by this problem - > Varnish have refused to cache even static files. While it was fine > before, and will cache even dynamic HTML files (with proper cache > headers, of course) > > The problem begun when my client reported that his website was > constantly going up & down. When I checked, it was under some sort of > DDoS attack. This is not news, because his website was pretty much > under 24x7 syn flood attack. > However, this attack is now able to bypass HAproxy & Varnish - and hit > Apache directly, right where it hurts most. > > To cut the long story short - basically now Varnish refuses to cache > almost everything. > The X-Cacheable header that I enabled contains the dreaded "NO:Not > Cacheable" status. And I've not been able to find out why. > > This is where I stuck, if I can find out the cause, the I'd be able to > rectify it. > > Anyway, the DDoS attack may not be related at all to the current > problem (varnish not caching), I included the story just for the sake > of completeness. > > Let's proceed to some facts that I've gathered : > > Same GIF file - but different sizes (note the "Received" column) : > http://minus.com/mbawzSZUxJ#3 > > The URLs with port 8181 are direct requests to Varnish, bypassing HAproxy. > > HTTP Headers produced by Varnish : http://minus.com/mbawzSZUxJ#4 > > HTTP Headers produced by HAproxy : http://minus.com/mbawzSZUxJ#2 > > List of cookies sent by browser & received from Varnish : > http://minus.com/mbawzSZUxJ#1 > (stripped clean by varnish, basically) > > Please find the result of varnishstat -1 & content of > /etc/varnish/default.vcl attached to the end of this email. > > Varnish is running with the following parameters : > > $ ps aux|grep varnish > myuser ? 3327 ?0.0 ?0.0 ? 3324 ? 796 pts/6 ? ?S+ ? 04:36 ? 0:00 grep varnish > root ? ? ?8441 ?0.0 ?1.0 ?86024 83532 pts/2 ? ?SL+ ?Jan12 ? 0:01 > /usr/sbin/varnishd -P /var/run/varnishd.pid -a 0.0.0.0:8181 -f > /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 180 -w 2,1000,30 -s > malloc,2G -d > > > Hopefully someone will be able to point / give me a hint to the right direction. > > > Thanks, > Harry > > =================== > > $ telnet localhost 6082 > Trying 127.0.0.1... > Connected to localhost. > Escape character is '^]'. > 200 199 > ----------------------------- > Varnish Cache CLI 1.0 > ----------------------------- > Linux,2.6.32-25-generic-pae,i686,-smalloc,-hcritbit > > Type 'help' for command list. > Type 'quit' to close CLI session. > > stats > 200 1978 > ? ?16564302 ?Client connections accepted > ? ?16514647 ?Client requests received > ? ? 2378064 ?Cache hits > ? ? ?988836 ?Cache misses > ? ? ?592750 ?Backend conn. success > ? ? ?462880 ?Backend conn. failures > ? ?13096234 ?Backend conn. reuses > ? ? ? 63305 ?Backend conn. was closed > ? ?13160380 ?Backend conn. recycles > ? ? ? ? ?82 ?Fetch head > ? ? 9937334 ?Fetch with Length > ? ? ? ?1856 ?Fetch chunked > ? ? ? ? 245 ?Fetch wanted close > ? ? ? ? ?66 ?Fetch failed > ? ? ? 26436 ?N struct sess_mem > ? ? ? 26091 ?N struct sess > ? ? ? 18773 ?N struct object > ? ? ? 18862 ?N struct objectcore > ? ? ? ?6827 ?N struct objecthead > ? ? ? ? ? 3 ?N struct vbe_conn > ? ? ? ? 107 ?N worker threads > ? ? ? 26496 ?N worker threads created > ? ? ?144188 ?N overflowed work requests > ? ? ? ? ? 1 ?N backends > ? ? ?932755 ?N expired objects > ? ? 1851315 ?N LRU moved objects > ? ?12437230 ?Objects sent with write > ? ?16564302 ?Total Sessions > ? ?16514647 ?Total Requests > ? ? ? ? 870 ?Total pipe > ? ?13146877 ?Total pass > ? ?13630039 ?Total fetch > ?6464791313 ?Total header bytes > 122615308544 ?Total body bytes > ? ?16564289 ?Session Closed > ? ? ? ? ?11 ?Session Linger > ? ? ? ? ?17 ?Session herd > ?1364697125 ?SHM records > ? ?93867958 ?SHM writes > ? ? ? ?2574 ?SHM flushes due to overflow > ? ? ?446883 ?SHM MTX contention > ? ? ? ? 540 ?SHM cycles through buffer > ? ?10815252 ?SMA allocator requests > ? ? ? 37554 ?SMA outstanding allocations > ? ?45497812 ?SMA outstanding bytes > 114575871831 ?SMA bytes allocated > 114530374019 ?SMA bytes free > ? ? ?505674 ?SMS allocator requests > ? 211877406 ?SMS bytes allocated > ? 211877406 ?SMS bytes freed > ? ?13688813 ?Backend requests made > ? ? ? ? ? 1 ?N vcl total > ? ? ? ? ? 1 ?N vcl available > ? ? ? ? ? 1 ?N total active purges > ? ? ? ? ? 1 ?N new purges added > ? ? 3365986 ?HCB Lookups without lock > ? ? ?122890 ?HCB Lookups with lock > ? ? ?122890 ?HCB Inserts > ? ? ? 99485 ?Client uptime > ? ? ? 16426 ?Backend conn. retry > ? ? 3690588 ?Fetch no body (304) > > =================== default.vcl =============== > > $ cat /etc/varnish/default.vcl > > # Default backend definition. ?Set this to point to your content > # server. > ?backend default { > ? ? .host = "127.0.0.1"; > ? ? .port = "81"; > ?} > > > sub vcl_deliver { > ? ? ? ?if (obj.hits > 0) { > ? ? ? ? ? ? ? ?set resp.http.X-Cache = "HIT"; > ? ? ? ?} else { > ? ? ? ? ? ? ? ?set resp.http.X-Cache = "MISS"; > ? ? ? ?} > } > > # Below is a commented-out copy of the default VCL logic. ?If you > # redefine any of these subroutines, the built-in logic will be > # appended to your code. > # > ?sub vcl_recv { > > ?if (req.backend.healthy) { > ? ?set req.grace = 180s; > ?} else { > ? ?set req.grace = 1h; > ?} > ?} > > > > ?sub vcl_fetch { > > ? set beresp.grace = 1h; > > ? ?# Varnish determined the object was not cacheable > ? ?if (!beresp.cacheable) { > ? ? ? ?set beresp.http.X-Cacheable = "NO:Not Cacheable"; > > ? ?# You don't wish to cache content for logged in users > ? ?} elsif (req.http.Cookie ~ "(UserID|_session)") { > ? ? ? ?set beresp.http.X-Cacheable = "NO:Got Session"; > ? ? ? ?return(pass); > > ? ?# You are respecting the Cache-Control=private header from the backend > ? ?} elsif (beresp.http.Cache-Control ~ "private") { > ? ? ? ?set beresp.http.X-Cacheable = "NO:Cache-Control=private"; > ? ? ? ?return(pass); > > ? ?# You are extending the lifetime of the object artificially > ? ?} elsif (beresp.ttl < 1s) { > ? ? ? ?set beresp.ttl ? = 5s; > ? ? ? ?set beresp.grace = 5s; > ? ? ? ?set beresp.http.X-Cacheable = "YES:FORCED"; > > ? ?# Varnish determined the object was cacheable > ? ?} else { > ? ? ? ?set beresp.http.X-Cacheable = "YES"; > ? ?} > > ? ?if (req.url ~ > "\.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|pdf|txt|tar|wav|bmp|rtf|js|flv|swf|html|htm)$") > { > ? ? ? ? ? ?unset beresp.http.set-cookie; > ? ? ? ? ? ?set beresp.ttl ? = 24h; > ? ?} > > ? ?return(deliver); > > # ? ? if (!beresp.cacheable) { > # ? ? ? ? return (pass); > # ? ? } > # ? ? if (beresp.http.Set-Cookie) { > # ? ? ? ? return (pass); > # ? ? } > # ? ? return (deliver); > ?} > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From enrico.sartorello at gmail.com Fri Jan 13 11:17:41 2012 From: enrico.sartorello at gmail.com (Enrico Sartorello) Date: Fri, 13 Jan 2012 12:17:41 +0100 Subject: Parsing beresp.response in vcl_fetch Message-ID: Hello, For the caching solution I am considering, we need to decide whether an object is cacheable or not based on the content of the HTTP response (actually, a JSON object). Therefore, my question is the following: is it possible from vcl_fetch to parse\evaluate somehow the beresp.response object? If so, which is the correct way to do so? Thanks for the attention, -- Enrico Sartorello http://enrico.sartorello.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From teemu.antti-poika at reaktor.fi Mon Jan 16 11:03:55 2012 From: teemu.antti-poika at reaktor.fi (Teemu Antti-Poika) Date: Mon, 16 Jan 2012 13:03:55 +0200 Subject: Varnish and Streaming questions Message-ID: <4F14041B.1090501@reaktor.fi> Hi all, I'm interested in upgrading our Varnish installation to the latest and greatest. We serve large files and the streaming support is of interest: previously we've had to disable Varnish caching for huge files in order for our servers to be responsive. I'm testing the release announced in November: https://www.varnish-cache.org/lists/pipermail/varnish-announce/2011-November/000677.html Is there anything I should be aware of about the release? Any known issues? The reason I'm asking is partly because the streaming support does not seem to have made it's way to the official release versions, yet. Thanks for any observations, Teemu Antti-Poika From straightflush at gmail.com Mon Jan 16 14:02:25 2012 From: straightflush at gmail.com (AD) Date: Mon, 16 Jan 2012 09:02:25 -0500 Subject: Varnish and Streaming questions In-Reply-To: <4F14041B.1090501@reaktor.fi> References: <4F14041B.1090501@reaktor.fi> Message-ID: I believe you are looking for this fork of Varnish https://github.com/mbgrydeland/varnish-cache-streaming It has enhanced streaming support. It doesnt seem at this point there are plans to merge this into the official codebase yet, although from reading planet varnish it seems they are working on building a 3.0.2s release with this formally baked in. @Tollef/@Per any thoughts on formal integration based on the post here https://www.varnish-software.com/blog/http-streaming-varnish Regards, AD On Mon, Jan 16, 2012 at 6:03 AM, Teemu Antti-Poika < teemu.antti-poika at reaktor.fi> wrote: > Hi all, > > I'm interested in upgrading our Varnish installation to the latest and > greatest. We serve large files and the streaming support is of interest: > previously we've had to disable Varnish caching for huge files in order for > our servers to be responsive. > > I'm testing the release announced in November: > > https://www.varnish-cache.org/**lists/pipermail/varnish-** > announce/2011-November/000677.**html > > Is there anything I should be aware of about the release? Any known > issues? The reason I'm asking is partly because the streaming support does > not seem to have made it's way to the official release versions, yet. > > Thanks for any observations, > Teemu Antti-Poika > > ______________________________**_________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/**lists/mailman/listinfo/**varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Mon Jan 16 14:34:39 2012 From: perbu at varnish-software.com (Per Buer) Date: Mon, 16 Jan 2012 15:34:39 +0100 Subject: Varnish and Streaming questions In-Reply-To: References: <4F14041B.1090501@reaktor.fi> Message-ID: Hi. I'll keep top-posting since mixing quoting styles suck. We did release a 3.0.2s which can be found here: http://repo.varnish-cache.org/test/3.0.2+streaming/ It has received a fair amount of testing and is used in production in several installations. Martin is working with PHK on re-factoring the code for inclusion in git-master. This feature will be part of the next major release of Varnish and if our customers want us to we'll continue to maintain the 3.0.Xs variant of Varnish Cache. Best regards, Per. On Mon, Jan 16, 2012 at 3:02 PM, AD wrote: > I believe you are looking for this fork of Varnish > https://github.com/mbgrydeland/varnish-cache-streaming > > It has enhanced streaming support. It doesnt seem at this point there are > plans to merge this into the official codebase yet, although from reading > planet varnish it seems they are working on building a 3.0.2s release with > this formally baked in. > > @Tollef/@Per any thoughts on formal integration based on the post here > https://www.varnish-software.com/blog/http-streaming-varnish > > Regards, > AD > > > On Mon, Jan 16, 2012 at 6:03 AM, Teemu Antti-Poika < > teemu.antti-poika at reaktor.fi> wrote: > >> Hi all, >> >> I'm interested in upgrading our Varnish installation to the latest and >> greatest. We serve large files and the streaming support is of interest: >> previously we've had to disable Varnish caching for huge files in order for >> our servers to be responsive. >> >> I'm testing the release announced in November: >> >> https://www.varnish-cache.org/**lists/pipermail/varnish-** >> announce/2011-November/000677.**html >> >> Is there anything I should be aware of about the release? Any known >> issues? The reason I'm asking is partly because the streaming support does >> not seem to have made it's way to the official release versions, yet. >> >> Thanks for any observations, >> Teemu Antti-Poika >> >> ______________________________**_________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/**lists/mailman/listinfo/**varnish-misc >> > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From tcpipqa at gmail.com Tue Jan 17 10:46:06 2012 From: tcpipqa at gmail.com (satish M G) Date: Tue, 17 Jan 2012 16:16:06 +0530 Subject: Dropped connections with tcp_tw_recycle=1 Message-ID: Hi I am new to this list. In continuation to the thread "Dropped connections with tcp_tw_recycle=1" I have a query - we faced the same issue of dropping the connections with tcp_tw_recycle set and NAT. we then tried with tcp_tw_reuse and it worked fine with the NAT and large number of incoming connections. however the documentation says that tcp_tw_reuse - tcp_tw_reuse - BOOLEAN465 Allow to reuse TIME-WAIT sockets for new connections when it is466 safe from protocol viewpoint. Default value is 0.467 It should not be changed without advice/request of technical468 experts. The 'advice/request' part is to explain that: - Connections that go through TCP state aware nodes, such as firewalls, NAT devices or load balancers may see dropped frames. The more connections there are, the more likely you will see this issue. My query is - 1) does tcp_tw_reuse work well with large number of incoming connections and NAT device between the client and server? 2) if tcp_tw_reuse does not work well, which part of the linux code is causing the dropped frames? 3) generally what is the difference between tcp_tw_reuse and tcp_tw_recycle? regards satish -------------- next part -------------- An HTML attachment was scrubbed... URL: From teemu.antti-poika at reaktor.fi Wed Jan 18 08:37:14 2012 From: teemu.antti-poika at reaktor.fi (Teemu Antti-Poika) Date: Wed, 18 Jan 2012 10:37:14 +0200 Subject: Varnish and Streaming questions In-Reply-To: References: <4F14041B.1090501@reaktor.fi> Message-ID: <4F1684BA.8030702@reaktor.fi> Thanks for the information, folks! Teemu From jan at architechs.eu Wed Jan 18 15:34:40 2012 From: jan at architechs.eu (=?UTF-8?Q?Jan=2DAage_Frydenb=C3=B8=2DBruvoll?=) Date: Wed, 18 Jan 2012 15:34:40 +0000 Subject: Varnish memory usage Message-ID: Dear list, I am currently in a situation where I need to restart Varnish every day due to memory usage spinning out of control. Would anybody on the list be able to shed any light on what variables come into play and how I can estimate the actual memory usage on that basis? At the moment I have Varnish 3.0.2 running with command line parameters /usr/sbin/varnishd -P /var/run/varnishd.pid -u nobody -f /etc/varnish/default.vcl -s malloc,6.5G -p thread_pools 4 -p thread_pool_max 1500 -p listen_depth 2048 -p lru_interval 1800 -h classic,169313 -p connect_timeout 600 After a few hours of operation this leads to the following resource usage: proxy03 jan # ps axu | grep varnish ... nobody 25653 4.3 12.8 24646344 12734204 ? Sl Jan15 185:04 /usr/sbin/varnishd -P /var/run/varnishd.pid ... So far, RES will continue to rise well above this number - the worst I have had before I had to restart in order to avoid problems on the machine was 48GB (RES, not VIRT) Thank you in advance for any pointers on this. Best regards Jan From jan at architechs.eu Wed Jan 18 16:09:09 2012 From: jan at architechs.eu (=?UTF-8?Q?Jan=2DAage_Frydenb=C3=B8=2DBruvoll?=) Date: Wed, 18 Jan 2012 16:09:09 +0000 Subject: Varnish memory usage In-Reply-To: <20120118160530.GV3214@nerd.dk> References: <20120118160530.GV3214@nerd.dk> Message-ID: Hi, On Wed, Jan 18, 2012 at 16:05, Andreas Plesner Jacobsen wrote: > We'd like to see varnishstat -1 and your vcl too. proxy03 varnish # varnishstat -1 client_conn 11139628 42.95 Client connections accepted client_drop 0 0.00 Connection dropped, no sess/wrk client_req 48047460 185.27 Client requests received cache_hit 42313341 163.16 Cache hits cache_hitpass 59583 0.23 Cache hits for pass cache_miss 4988042 19.23 Cache misses backend_conn 3294448 12.70 Backend conn. success backend_unhealthy 36426 0.14 Backend conn. not attempted backend_busy 0 0.00 Backend conn. too many backend_fail 211 0.00 Backend conn. failures backend_reuse 2402445 9.26 Backend conn. reuses backend_toolate 2748 0.01 Backend conn. was closed backend_recycle 2405196 9.27 Backend conn. recycles backend_retry 144 0.00 Backend conn. retry fetch_head 478 0.00 Fetch head fetch_length 5138161 19.81 Fetch with Length fetch_chunked 475194 1.83 Fetch chunked fetch_eof 0 0.00 Fetch EOF fetch_bad 0 0.00 Fetch had bad headers fetch_close 2985 0.01 Fetch wanted close fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed fetch_zero 0 0.00 Fetch zero len fetch_failed 45 0.00 Fetch failed fetch_1xx 0 0.00 Fetch no body (1xx) fetch_204 0 0.00 Fetch no body (204) fetch_304 329 0.00 Fetch no body (304) n_sess_mem 2580 . N struct sess_mem n_sess 1315 . N struct sess n_object 761556 . N struct object n_vampireobject 0 . N unresurrected objects n_objectcore 761679 . N struct objectcore n_objecthead 523573 . N struct objecthead n_waitinglist 4299 . N struct waitinglist n_vbc 12 . N struct vbc n_wrk 136 . N worker threads n_wrk_create 1520 0.01 N worker threads created n_wrk_failed 0 0.00 N worker threads not created n_wrk_max 0 0.00 N worker threads limited n_wrk_lqueue 0 0.00 work request queue length n_wrk_queued 26664 0.10 N queued work requests n_wrk_drop 0 0.00 N dropped work requests n_backend 19 . N backends n_expired 902775 . N expired objects n_lru_nuked 3222521 . N LRU nuked objects n_lru_moved 395588 . N LRU moved objects losthdr 37 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 40679068 156.86 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 11139613 42.95 Total Sessions s_req 48047460 185.27 Total Requests s_pipe 204 0.00 Total pipe s_pass 747967 2.88 Total pass s_fetch 5617078 21.66 Total fetch s_hdrbytes 18002004233 69414.68 Total header bytes s_bodybytes 496206310294 1913342.76 Total body bytes sess_closed 1772616 6.84 Session Closed sess_pipeline 198838 0.77 Session Pipeline sess_readahead 48712 0.19 Session Read Ahead sess_linger 47017485 181.30 Session Linger sess_herd 42381685 163.42 Session herd shm_records 2342457931 9032.38 SHM records shm_writes 138479662 533.97 SHM writes shm_flushes 14 0.00 SHM flushes due to overflow shm_cont 3181498 12.27 SHM MTX contention shm_cycles 920 0.00 SHM cycles through buffer sms_nreq 89029 0.34 SMS allocator requests sms_nobj 0 . SMS outstanding allocations sms_nbytes 0 . SMS outstanding bytes sms_balloc 30624944 . SMS bytes allocated sms_bfree 30624944 . SMS bytes freed backend_req 5696167 21.96 Backend requests made n_vcl 2 0.00 N vcl total n_vcl_avail 2 0.00 N vcl available n_vcl_discard 0 0.00 N vcl discarded n_ban 709 . N total active bans n_ban_add 5198 0.02 N new bans added n_ban_retire 4489 0.02 N old bans deleted n_ban_obj_test 80926784 312.05 N objects tested n_ban_re_test 225982892 871.38 N regexps tested against n_ban_dups 3000 0.01 N duplicate bans removed hcb_nolock 0 0.00 HCB Lookups without lock hcb_lock 0 0.00 HCB Lookups with lock hcb_insert 0 0.00 HCB Inserts esi_errors 0 0.00 ESI parse errors (unlock) esi_warnings 0 0.00 ESI parse warnings (unlock) accept_fail 0 0.00 Accept failures client_drop_late 0 0.00 Connection dropped late uptime 259340 1.00 Client uptime dir_dns_lookups 0 0.00 DNS director lookups dir_dns_failed 0 0.00 DNS director failed lookups dir_dns_hit 0 0.00 DNS director cached lookups hit dir_dns_cache_full 0 0.00 DNS director full dnscache vmods 0 . Loaded VMODs n_gzip 0 0.00 Gzip operations n_gunzip 3849737 14.84 Gunzip operations LCK.sms.creat 1 0.00 Created locks LCK.sms.destroy 0 0.00 Destroyed locks LCK.sms.locks 267087 1.03 Lock Operations LCK.sms.colls 0 0.00 Collisions LCK.smp.creat 0 0.00 Created locks LCK.smp.destroy 0 0.00 Destroyed locks LCK.smp.locks 0 0.00 Lock Operations LCK.smp.colls 0 0.00 Collisions LCK.sma.creat 2 0.00 Created locks LCK.sma.destroy 0 0.00 Destroyed locks LCK.sma.locks 27800385 107.20 Lock Operations LCK.sma.colls 0 0.00 Collisions LCK.smf.creat 0 0.00 Created locks LCK.smf.destroy 0 0.00 Destroyed locks LCK.smf.locks 0 0.00 Lock Operations LCK.smf.colls 0 0.00 Collisions LCK.hsl.creat 0 0.00 Created locks LCK.hsl.destroy 0 0.00 Destroyed locks LCK.hsl.locks 0 0.00 Lock Operations LCK.hsl.colls 0 0.00 Collisions LCK.hcb.creat 0 0.00 Created locks LCK.hcb.destroy 0 0.00 Destroyed locks LCK.hcb.locks 0 0.00 Lock Operations LCK.hcb.colls 0 0.00 Collisions LCK.hcl.creat 169313 0.65 Created locks LCK.hcl.destroy 0 0.00 Destroyed locks LCK.hcl.locks 93960449 362.31 Lock Operations LCK.hcl.colls 0 0.00 Collisions LCK.vcl.creat 1 0.00 Created locks LCK.vcl.destroy 0 0.00 Destroyed locks LCK.vcl.locks 8668 0.03 Lock Operations LCK.vcl.colls 0 0.00 Collisions LCK.stat.creat 1 0.00 Created locks LCK.stat.destroy 0 0.00 Destroyed locks LCK.stat.locks 2580 0.01 Lock Operations LCK.stat.colls 0 0.00 Collisions LCK.sessmem.creat 1 0.00 Created locks LCK.sessmem.destroy 0 0.00 Destroyed locks LCK.sessmem.locks 11156227 43.02 Lock Operations LCK.sessmem.colls 0 0.00 Collisions LCK.wstat.creat 1 0.00 Created locks LCK.wstat.destroy 0 0.00 Destroyed locks LCK.wstat.locks 2617990 10.09 Lock Operations LCK.wstat.colls 0 0.00 Collisions LCK.herder.creat 1 0.00 Created locks LCK.herder.destroy 0 0.00 Destroyed locks LCK.herder.locks 23393 0.09 Lock Operations LCK.herder.colls 0 0.00 Collisions LCK.wq.creat 4 0.00 Created locks LCK.wq.destroy 0 0.00 Destroyed locks LCK.wq.locks 89338248 344.48 Lock Operations LCK.wq.colls 0 0.00 Collisions LCK.objhdr.creat 3861706 14.89 Created locks LCK.objhdr.destroy 3338136 12.87 Destroyed locks LCK.objhdr.locks 108880937 419.84 Lock Operations LCK.objhdr.colls 0 0.00 Collisions LCK.exp.creat 1 0.00 Created locks LCK.exp.destroy 0 0.00 Destroyed locks LCK.exp.locks 9303270 35.87 Lock Operations LCK.exp.colls 0 0.00 Collisions LCK.lru.creat 2 0.00 Created locks LCK.lru.destroy 0 0.00 Destroyed locks LCK.lru.locks 8143794 31.40 Lock Operations LCK.lru.colls 0 0.00 Collisions LCK.cli.creat 1 0.00 Created locks LCK.cli.destroy 0 0.00 Destroyed locks LCK.cli.locks 31371 0.12 Lock Operations LCK.cli.colls 0 0.00 Collisions LCK.ban.creat 1 0.00 Created locks LCK.ban.destroy 0 0.00 Destroyed locks LCK.ban.locks 90171669 347.70 Lock Operations LCK.ban.colls 0 0.00 Collisions LCK.vbp.creat 1 0.00 Created locks LCK.vbp.destroy 0 0.00 Destroyed locks LCK.vbp.locks 2838069 10.94 Lock Operations LCK.vbp.colls 0 0.00 Collisions LCK.vbe.creat 1 0.00 Created locks LCK.vbe.destroy 0 0.00 Destroyed locks LCK.vbe.locks 6589306 25.41 Lock Operations LCK.vbe.colls 0 0.00 Collisions LCK.backend.creat 19 0.00 Created locks LCK.backend.destroy 0 0.00 Destroyed locks LCK.backend.locks 22723286 87.62 Lock Operations LCK.backend.colls 0 0.00 Collisions SMA.s0.c_req 11969376 46.15 Allocator requests SMA.s0.c_fail 412116470473 1589097.21 Allocator failures SMA.s0.c_bytes 277501708539 1070030.49 Bytes allocated SMA.s0.c_freed 270522525345 1043119.17 Bytes freed SMA.s0.g_alloc 1096776 . Allocations outstanding SMA.s0.g_bytes 6979183194 . Bytes outstanding SMA.s0.g_space 138662 . Bytes available SMA.Transient.c_req 2763044 10.65 Allocator requests SMA.Transient.c_fail 0 0.00 Allocator failures SMA.Transient.c_bytes 179601302623 692532.21 Bytes allocated SMA.Transient.c_freed 175191674974 675528.94 Bytes freed SMA.Transient.g_alloc 429519 . Allocations outstanding SMA.Transient.g_bytes 4409627649 . Bytes outstanding SMA.Transient.g_space 0 . Bytes available VBE.fake(127.0.0.1,,80).vcls 2 . VCL references VBE.fake(127.0.0.1,,80).happy 0 . Happy health probes VBE.app01(10.140.106.10,,80).vcls 2 . VCL references VBE.app01(10.140.106.10,,80).happy18446744073709551615 . Happy health probes VBE.app02(10.140.106.11,,80).vcls 2 . VCL references VBE.app02(10.140.106.11,,80).happy18446744073709551615 . Happy health probes VBE.app03(10.140.106.12,,80).vcls 2 . VCL references VBE.app03(10.140.106.12,,80).happy18446744073709551615 . Happy health probes VBE.app04(10.140.106.13,,80).vcls 2 . VCL references VBE.app04(10.140.106.13,,80).happy18446744073709551615 . Happy health probes VBE.app05(10.140.106.14,,80).vcls 2 . VCL references VBE.app05(10.140.106.14,,80).happy18446744073709551615 . Happy health probes VBE.app06(10.140.106.15,,80).vcls 2 . VCL references VBE.app06(10.140.106.15,,80).happy18446744073709551615 . Happy health probes VBE.app07(10.140.106.16,,80).vcls 2 . VCL references VBE.app07(10.140.106.16,,80).happy18446744073709551615 . Happy health probes VBE.app08(10.140.106.17,,80).vcls 2 . VCL references VBE.app08(10.140.106.17,,80).happy18446744073709551615 . Happy health probes VBE.app09(10.140.106.39,,80).vcls 2 . VCL references VBE.app09(10.140.106.39,,80).happy18446744073709551615 . Happy health probes VBE.app10(10.140.106.40,,80).vcls 2 . VCL references VBE.app10(10.140.106.40,,80).happy18446744073709551615 . Happy health probes VBE.app11(10.140.106.41,,80).vcls 2 . VCL references VBE.app11(10.140.106.41,,80).happy18446744073709551615 . Happy health probes VBE.app12(10.140.106.42,,80).vcls 2 . VCL references VBE.app12(10.140.106.42,,80).happy18446744073709551615 . Happy health probes VBE.app13(10.140.106.43,,80).vcls 2 . VCL references VBE.app13(10.140.106.43,,80).happy18446744073709551615 . Happy health probes VBE.app14(10.140.106.46,,80).vcls 2 . VCL references VBE.app14(10.140.106.46,,80).happy18446744073709551615 . Happy health probes VBE.static01(10.140.106.18,,8000).vcls 2 . VCL references VBE.static01(10.140.106.18,,8000).happy18446744073709551615 . Happy health probes VBE.static02(10.140.106.19,,8000).vcls 2 . VCL references VBE.static02(10.140.106.19,,8000).happy18446744073709551615 . Happy health probes VBE.static03(10.140.106.20,,8000).vcls 2 . VCL references VBE.static03(10.140.106.20,,8000).happy18446744073709551615 . Happy health probes VBE.static04(10.140.106.21,,8000).vcls 2 . VCL references VBE.static04(10.140.106.21,,8000).happy18446744073709551615 . Happy health probes I'd rather not post the whole VCL - is it possible to assess which parts would be relevant here? > My first suspicion is that objects end up in the transient storage. How would I know? Thanks Jan From miguel.chavez at inventmx.com Wed Jan 18 16:13:28 2012 From: miguel.chavez at inventmx.com (=?ISO-8859-1?Q?Miguel_Ch=E1vez?=) Date: Wed, 18 Jan 2012 10:13:28 -0600 Subject: Newbie questions Message-ID: Good day to all Today I've been given a site and this board administrator with varnish, initially had an excess of meditation gurus, I base a fewadjustments can reduce. Now I have the following problem after stopping and starting the varnish and run a ps aux | grep varnish see 2 instances running. root 6524 0.0 0.0 105716 864? Ss Jan17 0:00 / usr / sbin / varnishd-P / var / run / varnish.pid-a *: 80-T 127.0.0.1:8181-f / etc /varnish / varnish.vcl-p 4-p thread_pools thread_pool_min 200-p 4000-p thread_pool_max thread_pool_add_delay 2-p 600-pfirst_byte_timeout between_bytes_timeout 600- 200-p p session_linger listen_depth 2048-p 1800-h lru_interval classic, 169 313-600-p p connect_timeout max_restarts 6-s malloc, 3500M nobody 6525 96.5 51.6 13371952 4149812? Sl Jan17 589:41 / usr / sbin / varnishd-P / var / run / varnish.pid-a *: 80-T127.0.0.1:8181-f / etc / varnish / varnish.vcl-p 4-p thread_pools thread_pool_min 200-p 4000-p thread_pool_maxthread_pool_add_delay 2-p 600-p first_byte_timeout between_bytes_timeout 600-200-p p session_linger listen_depth 2048-p1800-h lru_interval classic, 169 313-600-p p connect_timeout max_restarts 6-s malloc, 3500M 61 160 23 483 0.0 0.0 root 756 pts / 1 S + 10:04 0:00 grep varnish On the other side before he could give-I varnishlog varnishlog RxStatus and showed me values ??are now no longer responding. -- -------------- next part -------------- An HTML attachment was scrubbed... URL: From apj at mutt.dk Wed Jan 18 16:52:04 2012 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Wed, 18 Jan 2012 17:52:04 +0100 Subject: Varnish memory usage In-Reply-To: References: <20120118160530.GV3214@nerd.dk> Message-ID: <20120118165204.GW3214@nerd.dk> On Wed, Jan 18, 2012 at 04:09:09PM +0000, Jan-Aage Frydenb?-Bruvoll wrote: > > I'd rather not post the whole VCL - is it possible to assess which > parts would be relevant here? Any part that sets ttl. >From man vcl: shortlived ? Units: s ? Default: 10.0 Objects created with TTL shorter than this are always put in transient storage. > > My first suspicion is that objects end up in the transient storage. > > How would I know? >From varnishstat: > SMA.Transient.g_bytes 4409627649 . Bytes outstanding -- Andreas From hugo.cisneiros at gmail.com Thu Jan 19 16:20:58 2012 From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch)) Date: Thu, 19 Jan 2012 14:20:58 -0200 Subject: High SMA.s0.c_fail in varnishstat Message-ID: Hi! Looking at my statistics, I saw something running varnishstat -1: SMA.s0.c_req 4400778 73.80 Allocator requests SMA.s0.c_fail 136937945724 2296268.06 Allocator failures SMA.s0.c_bytes 85269901510 1429863.36 Bytes allocated SMA.s0.c_freed 79902088046 1339852.24 Bytes freed SMA.s0.g_alloc 504320 . Allocations outstanding SMA.s0.g_bytes 5367813464 . Bytes outstanding SMA.s0.g_space 895656 . Bytes available I'm not getting any errors and varnish is working just fine, but the 'SMA.s0.c_fail' number is too high. I couldn't find anything detailed about this statistic, but my intuition says that the best thing is to get a zero on this :) Anyone have some details on this item? What does it mean and what can I do to not get the failures? Transient storage is fine, with 0 fails. Relevant command-line options: -h classic,500009 -s malloc,5G -p thread_pools=3 -p thread_pool_min=128 -p thread_pool_max=3840 -p lru_interval=60 -p sess_workspace=524288 -p shm_workspace=262144 -- []'s Hugo www.devin.com.br From perbu at varnish-software.com Sat Jan 21 08:04:26 2012 From: perbu at varnish-software.com (Per Buer) Date: Sat, 21 Jan 2012 09:04:26 +0100 Subject: Stuff has been restored from backup Message-ID: Hi. We lost a server yesterday. Our Dell R610 threw a E1422 CPU1 MACHINE CHK which means that is some sort of dead. I can't remember if it was because I was ~broke or by mistake, but the server didn't have 24/7 hardware support (Stupid! Stupid! Stupid! Arrrrrgh!). We decided to move the disks to a server we were allowed to borrow from Redpill Linpro (Thanks!). That sort of worked, except the raid controller wrote weird crap over the md superblocks and we lost the content. So we restored from backup. Tollef is the hero of the day and has gotten the basics up and running. The timing here is a bit awkward as Tollef will be leaving for Asia in couple of hours. Anyway, if anybody sees something that looks broken and unless it requires rocket science I'll see if I can fix it. Send me an email, please. Thanks for the patience. At some point in the not to distant future we're planning on having a somewhat more robust setup. Personally I have some experience with DRBD and together with manual fail-over it was rather robust. I have no idea if that is still the smart way to do stuff. If you have opinions feel free to send them to me directly (the topic is a bit off topic on this list). Oh, and if you're still reading this and live in Oslo - we're looking for Unix hackers to hire. -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From isp at daviesinc.com Sat Jan 21 19:37:22 2012 From: isp at daviesinc.com (Chris Davies) Date: Sat, 21 Jan 2012 14:37:22 -0500 Subject: Varnish Example VCL configs Message-ID: <112FB5F6-8CCB-45EA-9A4A-DD3B45A732FC@daviesinc.com> Since releasing a Wordpress plugin, I've been bombarded with requests for 'the definitive' WordPress VCL. I know that Dr. Carter and I have published some VCL that is fairly decent for Wordpress but has been missing a few features. I've added a section to the Wiki including VCL Example Templates. While they may not be absolutely perfect, I haven't had any major issues with these configs, influenced by a number of VCL config files floating around the Internet. I would like to collect base templates for other applications, i.e. Magento, Typepad, Drupal, Joomla, etc and have them on this page for people to refer to that are just getting started with Varnish. If you would like to submit the VCL to me, or, post them on the wiki, I think a number of people may benefit from it. https://www.varnish-cache.org/trac/wiki/VCLExamples#VCLExampleTemplates Thank you. I'll be adding comments to the VCL to better explain the choices made in the VCL, but, now we have a place to direct people. From jan at architechs.eu Sun Jan 22 15:32:29 2012 From: jan at architechs.eu (=?UTF-8?Q?Jan=2DAage_Frydenb=C3=B8=2DBruvoll?=) Date: Sun, 22 Jan 2012 15:32:29 +0000 Subject: Varnish memory usage In-Reply-To: <20120118165204.GW3214@nerd.dk> References: <20120118160530.GV3214@nerd.dk> <20120118165204.GW3214@nerd.dk> Message-ID: Hi, On Wed, Jan 18, 2012 at 16:52, Andreas Plesner Jacobsen wrote: > > SMA.Transient.g_bytes ? 4409627649 ? ? ? ? ?. ? Bytes outstanding Thanks Andreas, this is incredibly useful information. Now, since last I posted, the process in question has grown further, to 25653 nobody 20 0 59.7g 21g 81m S 22 22.4 1092:15 varnishd and varnishstat says proxy03 jan # varnishstat -1 | grep -i trans SMA.Transient.c_req 14802020 24.56 Allocator requests SMA.Transient.c_fail 0 0.00 Allocator failures SMA.Transient.c_bytes 963605906690 1598878.84 Bytes allocated SMA.Transient.c_freed 951833551349 1579345.37 Bytes freed SMA.Transient.g_alloc 1239195 . Allocations outstanding SMA.Transient.g_bytes 11772355341 . Bytes outstanding SMA.Transient.g_space 0 . Bytes available When would transient objects actually leave the storage again? Can I force/trigger/monitor this at all? The manual says "By default Varnish would use an unlimited malloc backend for this", which in my case is rather inconvenient since this varnish process sits in an LXC container, where it apparently sees the host's memory allocation and not the container's - that in turn leads to amusing situations where Varnish runs head first into the LXC resource limitations and gets OOM-killed. Best regards Jan From apj at mutt.dk Sun Jan 22 15:44:45 2012 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Sun, 22 Jan 2012 16:44:45 +0100 Subject: Varnish memory usage In-Reply-To: References: <20120118160530.GV3214@nerd.dk> <20120118165204.GW3214@nerd.dk> Message-ID: <20120122154444.GY3214@nerd.dk> On Sun, Jan 22, 2012 at 03:32:29PM +0000, Jan-Aage Frydenb?-Bruvoll wrote: > > When would transient objects actually leave the storage again? Can I > force/trigger/monitor this at all? Same rules apply for transient as for any other malloc store: TTL, LRU or ban/purge. > The manual says "By default Varnish would use an unlimited malloc > backend for this", which in my case is rather inconvenient since this > varnish process sits in an LXC container, where it apparently sees the > host's memory allocation and not the container's - that in turn leads > to amusing situations where Varnish runs head first into the LXC > resource limitations and gets OOM-killed. You can just create a storage backend called "Transient" to override the default. However, I think it's much more appropriate to investigate how they end up in Transient in the first place. -- Andreas From jan at architechs.eu Sun Jan 22 15:50:56 2012 From: jan at architechs.eu (=?UTF-8?Q?Jan=2DAage_Frydenb=C3=B8=2DBruvoll?=) Date: Sun, 22 Jan 2012 15:50:56 +0000 Subject: Varnish memory usage In-Reply-To: <20120122154444.GY3214@nerd.dk> References: <20120118160530.GV3214@nerd.dk> <20120118165204.GW3214@nerd.dk> <20120122154444.GY3214@nerd.dk> Message-ID: Hi, On Sun, Jan 22, 2012 at 15:44, Andreas Plesner Jacobsen wrote: > > Same rules apply for transient as for any other malloc store: TTL, LRU or > ban/purge. Will Varnish keep multiple generations of objects that expire, and potentially wait with LRU eviction until it gets closer to the known sensible maximum memory allocation (the host has 128GB RAM)? We do have a limited number of (smallish) objects cached for 5 seconds as a somewhat desperate means of shielding our application servers, but the amount of pages we cache in this way in no way rhymes with the enormous amount of data kept by Varnish in transient storage. All other objects are stored for a substantially longer time (i.e. weeks and above) and should not come into play here. Putting things into perspective - the transient storage is currently almost as big as the full on-disk storage of 6 years of data, and given that this is a newspaper, 95% of the on-disk data would be long-tail and hardly ever accessed. Best regards Jan From apj at mutt.dk Sun Jan 22 15:54:05 2012 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Sun, 22 Jan 2012 16:54:05 +0100 Subject: Varnish memory usage In-Reply-To: References: <20120118160530.GV3214@nerd.dk> <20120118165204.GW3214@nerd.dk> <20120122154444.GY3214@nerd.dk> Message-ID: <20120122155405.GZ3214@nerd.dk> On Sun, Jan 22, 2012 at 03:50:56PM +0000, Jan-Aage Frydenb?-Bruvoll wrote: > > > > Same rules apply for transient as for any other malloc store: TTL, LRU or > > ban/purge. > > Will Varnish keep multiple generations of objects that expire, and > potentially wait with LRU eviction until it gets closer to the known > sensible maximum memory allocation (the host has 128GB RAM)? We do > have a limited number of (smallish) objects cached for 5 seconds as a > somewhat desperate means of shielding our application servers, but the > amount of pages we cache in this way in no way rhymes with the > enormous amount of data kept by Varnish in transient storage. All > other objects are stored for a substantially longer time (i.e. weeks > and above) and should not come into play here. Are you sure that these objects are really hashed correctly, and do not have a Vary header that will blot the cache? Do you see hits on these objects? -- Andreas From mattias at nucleus.be Sun Jan 22 15:55:52 2012 From: mattias at nucleus.be (Mattias Geniar) Date: Sun, 22 Jan 2012 16:55:52 +0100 Subject: Varnish Example VCL configs In-Reply-To: <112FB5F6-8CCB-45EA-9A4A-DD3B45A732FC@daviesinc.com> References: <112FB5F6-8CCB-45EA-9A4A-DD3B45A732FC@daviesinc.com> Message-ID: <18834F5BEC10824891FB8B22AC821A5A01FDB029@nucleus-srv01.Nucleus.local> > If you would like to submit the VCL to me, or, post them on the wiki, I think a number of people may benefit from it. Hi Chris, I have some templates for popular CMS systems posted here: https://github.com/mattiasgeniar/varnish-3.0-configuration-templates While most require a few dirty hacks (to avoid all the phpsessid cookies etc.), I've found them to be fairly stable in production use. Regards, Mattias From apj at mutt.dk Sun Jan 22 15:58:58 2012 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Sun, 22 Jan 2012 16:58:58 +0100 Subject: Varnish memory usage In-Reply-To: References: <20120118160530.GV3214@nerd.dk> <20120118165204.GW3214@nerd.dk> <20120122154444.GY3214@nerd.dk> Message-ID: <20120122155858.GA3214@nerd.dk> On Sun, Jan 22, 2012 at 03:50:56PM +0000, Jan-Aage Frydenb?-Bruvoll wrote: > > > > Same rules apply for transient as for any other malloc store: TTL, LRU or > > ban/purge. > > Will Varnish keep multiple generations of objects that expire, and > potentially wait with LRU eviction until it gets closer to the known > sensible maximum memory allocation (the host has 128GB RAM)? Sorry, forgot this part: No, Transient is unbounded unless you configure it otherwise. It should really only be used for short-lived objects. In your case, I would stick these small objects in a store I controlled the parameters of (beresp.storage) -- Andreas From jan at architechs.eu Sun Jan 22 16:03:03 2012 From: jan at architechs.eu (=?UTF-8?Q?Jan=2DAage_Frydenb=C3=B8=2DBruvoll?=) Date: Sun, 22 Jan 2012 16:03:03 +0000 Subject: Varnish memory usage In-Reply-To: <20120122155405.GZ3214@nerd.dk> References: <20120118160530.GV3214@nerd.dk> <20120118165204.GW3214@nerd.dk> <20120122154444.GY3214@nerd.dk> <20120122155405.GZ3214@nerd.dk> Message-ID: Hi, On Sun, Jan 22, 2012 at 15:54, Andreas Plesner Jacobsen wrote: > Are you sure that these objects are really hashed correctly, and do not have a > Vary header that will blot the cache? Do you see hits on these objects? I'm never sure of anything, but we've tried to do a good job on forcing headers down on these objects. However it will certainly not hurt to go through this again. Also thank you very much for the hint regarding unbounded transient storage - that's definitely something I need to look at immediately. Thanks Jan From apj at mutt.dk Sun Jan 22 20:33:37 2012 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Sun, 22 Jan 2012 21:33:37 +0100 Subject: Varnish memory usage In-Reply-To: <20120122155858.GA3214@nerd.dk> References: <20120118160530.GV3214@nerd.dk> <20120118165204.GW3214@nerd.dk> <20120122154444.GY3214@nerd.dk> <20120122155858.GA3214@nerd.dk> Message-ID: <20120122203337.GB3214@nerd.dk> On Sun, Jan 22, 2012 at 04:58:58PM +0100, Andreas Plesner Jacobsen wrote: > > > > > > Same rules apply for transient as for any other malloc store: TTL, LRU or > > > ban/purge. > > > > Will Varnish keep multiple generations of objects that expire, and > > potentially wait with LRU eviction until it gets closer to the known > > sensible maximum memory allocation (the host has 128GB RAM)? > > Sorry, forgot this part: No, Transient is unbounded unless you configure it > otherwise. It should really only be used for short-lived objects. After looking over the code with help from DocWilco, it looks like Transient is used in three circumstances: 1. Short-lived objects 2. vcl_error 3. When allocations in all other stores fail It looks like you're hitting 3: SMA.s0.c_fail 412116470473 1589097.21 Allocator failures My guess is that you're stressing LRU and are hitting some race conditions where one request LRUs and makes space for another object, while not actually making space for itself. I think your only option right now is to be more aggressive with your TTL and ensure that your main storage has a little space free. -- Andreas From yavor.k.atanasov at gmail.com Tue Jan 24 10:59:34 2012 From: yavor.k.atanasov at gmail.com (Yavor Atanasov) Date: Tue, 24 Jan 2012 10:59:34 +0000 Subject: What errors does "varnishlog -i Error" actually log? Message-ID: Gey guys, I have a couple of questions regarding varnishlog tags. We would like use varnishlog to record only certain logs from varnish, mostly errors. My first question is - what types of errors does actually " varnishlog -i Error " display? I am asking because I can't seem to be able to trigger one that would be recorded with this tag. The vcl_error subroutine doesn't seem to fall into that category, and also running out of session workspace doesn't either. What errors does this tag actually log? My other question is regarding the HttpError tag - " varnishlog -i HttpError ". That is documented as an available tag in the man pages for varnishlog, but when I try to run varnishlog with that option it says this tag does not exist: Could not match "HttpError" to any tag usage: varnishlog [-bCcd] [-i tag] [-I regexp] [-k keep] [-r file] [-s skip] [-X regexp] [-x tag] [-aDoV] [-n varnish_name] [-P file] [-w file] Any thoughts very welcome. Thank you very much! Kind regards, Yavor -- *Yavor Atanasov* MSc Digital Media, BA Busines Administration, Computer Science http://yavkata.co.uk -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at uplex.de Tue Jan 24 13:05:50 2012 From: geoff at uplex.de (Geoff Simmons) Date: Tue, 24 Jan 2012 14:05:50 +0100 Subject: What errors does "varnishlog -i Error" actually log? In-Reply-To: References: Message-ID: <4F1EACAE.6070306@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 01/24/12 11:59 AM, Yavor Atanasov wrote: > > I have a couple of questions regarding varnishlog tags. We would like use > varnishlog to record only certain logs from varnish, mostly errors. > > My first question is - what types of errors does actually " varnishlog -i > Error " display? I am asking because I can't seem to be able to trigger one > that would be recorded with this tag. The vcl_error subroutine doesn't seem > to fall into that category, and also running out of session workspace > doesn't either. What errors does this tag actually log? Searching in the current development trunk: $ find . -name *.c | xargs grep SLT_Error ./bin/varnishd/cache/cache_vary.c: WSP(sp, SLT_Error, "Vary header had extra ':', fix backend"); The extra ':' in Vary is evidently the only case. > My other question is regarding the HttpError tag - " varnishlog -i > HttpError ". That is documented as an available tag in the man pages for > varnishlog, but when I try to run varnishlog with that option it says this > tag does not exist: > > Could not match "HttpError" to any tag > usage: varnishlog [-bCcd] [-i tag] [-I regexp] [-k keep] [-r file] [-s > skip] [-X regexp] [-x tag] [-aDoV] [-n varnish_name] [-P file] [-w file] The same grep for HttpError doesn't yield any hits, so it looks like the documentation is out of date. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Schwanenwik 24 22087 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (SunOS) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJPHqyuAAoJEOUwvh9pJNUR2CgQAJNtFC9alnKd3Wk7bMXLYygS VhVj5meJbar7DETYNxftIXnIW/v45tuLIDNhk2I0HBUSc+xD2iwtyidWflioxhdh 5BNLTOsLRWGxMt5Qx/IbQvEGXycWle3mPLyyy1hgwBuPW9lT430++lW/bCJGU0Kz ufdbWBrtYEfXp1O8vkce26QNYRhE6AOIyV/r+hWp5q6PcZC1tPeDpzEz/8gmT0A9 9OnWZqoo8gwLoTThADFLlrnJPeDzzet5UD9wCDWp+TnsJx4E9iWZU6VmzrkQ3RFe MkBKFDSjsoL9aXLNlpBz872O6stfOy65C4jK2pfJUC5RpZiu/lMEbZw2Cxjufch0 7FzmJ/EN02mBSttxAWoLh5ae8aKlcOMmgcf7rqYLzW8hi6rUFrJajDSQItQraHde gklSYm4Fr7bNvVJxGwdPw9ZbjgjLZjV94cF+AL41qncXbBYjtet98qWQiJir9U4B tEVaectqfsZHrx9gbvEQoknL7SyWF8P+8NilQVooq18ruIVuhhYSF7DU8eZOuEOQ y/d3EYZMHfSxAQkqKCwCQ65Y+axC2v3R1C0IaCcrfbKKT9Ji7jtHoZinaCEjgVaz uUMno0S4a3rZdPxDKz5g78jq386TI6pKt5Hs5jj6EpIXzswHjcm6NVnrbXq9iioz 9kUn/N/3/eMDSwT8l2gR =+QoB -----END PGP SIGNATURE----- From meetheus at gmail.com Wed Jan 25 17:19:51 2012 From: meetheus at gmail.com (David Miller) Date: Wed, 25 Jan 2012 12:19:51 -0500 Subject: VCL question - where is cacheability determined? Message-ID: I'm converting from 2.1 to 3.0. I got bit by the beresp.cacheable variable going away, and replacing it with beresp.ttl > 0s doesn't work because ttl hasn't been determined yet; it's a -1.000. This is the block of code in question (in vcl_fetch) if (!beresp.cacheable && beresp.status != 404) { set beresp.http.X-Cacheable = "NO: !beresp.cacheable"; return (pass); } else { # From http://varnish-cache.org/wiki/VCLExampleLongerCaching /* Remove Expires from backend, it's not long enough */ unset beresp.http.expires; } If beresp.ttl hasn't been determined yet, what's the appropriate replacement? On a related note, I was looking at the source code for 2.1. From my quick reading, it looks like beresp.cacheable is determined by request type and return status. I don't see where apache/drupal/web-server cache headers are consulted at all. Am I missing something there? Are back-end cache headers to be parsed in VCL rather than being automagically determined? TIA, --- David -------------- next part -------------- An HTML attachment was scrubbed... URL: From meetheus at gmail.com Wed Jan 25 21:15:20 2012 From: meetheus at gmail.com (David Miller) Date: Wed, 25 Jan 2012 16:15:20 -0500 Subject: Caching while the whole back end is in maintenance mode Message-ID: We have periodic code updates on our drupal based back end. The back-end servers have a maintenance mode that will return a 503 for HTTP requests without cookies. The code updates sometimes take several hours - enough for the cached copy to expire. Grace mode appears (sorry; single data point; this is what the developers told me) to time out after an hour no matter what we set it to in vcl_recv/vcl_fetch. Is there a better approach to have everything that's cacheable available for a few hours? What are others doing? TIA, --- David -------------- next part -------------- An HTML attachment was scrubbed... URL: From j.vanarragon at lukkien.com Mon Jan 30 14:23:15 2012 From: j.vanarragon at lukkien.com (Jaap van Arragon) Date: Mon, 30 Jan 2012 15:23:15 +0100 Subject: varnish serve old object during refresh Message-ID: Hello, How can I serve an old object during the fetch of an expired object or max out connection? I?ve configured the grace period in both the vcl_recv as the vcl_fetch. I even tried to use the saint mode to give me back an ?old? cached object. Can somebody give me help in this matter? sub vcl_recv { # Purge through http if (req.request == "PURGE") { if (!client.ip ~ purge) { error 405 "Not allowed."; } ban("req.url ~ " + req.url ); error 200 "Purged."; } # Unset all cookies available if (req.http.cookie) { unset req.http.cookie; } if (req.request != "GET" && req.request != "HEAD") { /* We only deal with GET and HEAD by default */ return (pass); } # Adding Grace period in case backend lags set req.grace = 3d; return (lookup); } .... sub vcl_fetch { set beresp.grace = 4d; set beresp.saintmode = 50s; set beresp.ttl = 30s; return (deliver); } -------------- next part -------------- An HTML attachment was scrubbed... URL: From zengzhimin628 at 163.com Thu Jan 12 03:24:21 2012 From: zengzhimin628 at 163.com (=?GBK?B?1PjWx8P0?=) Date: Thu, 12 Jan 2012 03:24:21 -0000 Subject: varnishlog doesnot work sometimes, can you give some help, thank you very much! Message-ID: <5529b95e.1e3ce.134cff1bab7.Coremail.zengzhimin628@163.com> varnish version: varnish-2.1.3 varnishlog doesnot work sometimes, when we implement the varnishlog command,it prints nothing on the screen. we deploy a varnish on our web service , it works normally , but the varnishlog does not work sometimes. The varnish is ok ,it can also reply the request of client when the varnishlog does not work . Our application scene is that the varnish accepts 50 thounsand batch requests within 2 minutes five times everyday and the most requests can hit the cache. The varnish accepts batch requests on these time :00:00 ,04:00,08:0,12:00,16:00 and 20:00 . The problem usually happends on 20:00 , when the varinsh accepts 50 thounsand requests within 2 minutes ,the varnishlog does not work .but when it can restore work some time later , it usually restore work on next 00:00 we have traced the problem when the varnishlog does not work, some problem happends on these method below: mgt_cli_askchild child_poker mgt_cli_cb_before cls_vlu The code is so complicated ,we can not understand completely , can you give us some help to resolve the problem , because the varnishlog can help us locate some web problems, it is very useful to us we will look forward to your response. thank you very much ! Best regards! Yours sincerely, zhimin Zeng 2012.1.12 -------------- next part -------------- An HTML attachment was scrubbed... URL: