From Umesh.Telang at bbc.co.uk Thu Apr 2 12:28:49 2015 From: Umesh.Telang at bbc.co.uk (Umesh Telang) Date: Thu, 2 Apr 2015 12:28:49 +0000 Subject: Determining a HIT or a MISS in Varnish 4 VCL Message-ID: Hello, It's unclear if the way to detect whether a particular response was the result of a cache hit or miss in VCL, has changed from Varnish 3 to Varnish 4. At this page (not an official Varnish page...), it suggests using the value of resp.http.X-Varnish (whether it has a single or multiple values) to determine this: http://foshttpcache.readthedocs.org/en/latest/varnish-configuration.html# Alternatively I've also seen suggestions for continuing to use obj.hits in vcl_deliver. e.g. : sub vcl_deliver { if (obj.hits > 0) { # Add debug header to see if it's a HIT/MISS and the number of hits, disable when not needed set resp.http.X-Cache = "HIT"; } else { set resp.http.X-Cache = "MISS"; } } We've observed that the value of %{Varnish:handling}x in the varnish ncsa log has been accurate in reporting HITs vs MISSes. Could we use a VCL equivalent of this in vcl_deliver to determine whether a particular response is the result of a HIT or a MISS? Any advice on the recommended way in VCL to determine whether a response is the result of a HIT or MISS would be greatly appreciated! Thanks, Umesh -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.jennings at nytimes.com Thu Apr 2 12:38:48 2015 From: raymond.jennings at nytimes.com (Jennings III, Raymond) Date: Thu, 2 Apr 2015 08:38:48 -0400 Subject: Determining a HIT or a MISS in Varnish 4 VCL In-Reply-To: References: Message-ID: I think that's the way - exactly as you have in vcl_deliver. I've always used that in V3 and now in V4 as well. I'm suspect of that page telling you to not use obj in vcl_deliver but use that header value instead. Raymond Jennings III *nytimes.com * *Office: 212.556.7786 <212-556-7786>* *iPhone: 914.330.5074 <914-330-5074>E-mail: Raymond.Jennings at nytimes.com FaceTime: Raymond.Jennings at nytimes.com * On Thu, Apr 2, 2015 at 8:28 AM, Umesh Telang wrote: > Hello, > > It's unclear if the way to detect whether a particular response was the > result of a cache hit or miss in VCL, has changed from Varnish 3 to Varnish > 4. > > At this page (not an official Varnish page...), it suggests using the > value of resp.http.X-Varnish (whether it has a single or multiple values) > to determine this: > http://foshttpcache.readthedocs.org/en/latest/varnish-configuration.html# > > Alternatively I've also seen suggestions for continuing to use obj.hits > in vcl_deliver. e.g. : > sub vcl_deliver { > if (obj.hits > 0) { # Add debug header to see if it's a HIT/MISS and > the number of hits, disable when not needed set resp.http.X-Cache = > "HIT"; } else { set resp.http.X-Cache = "MISS"; } > } > > We've observed that the value of %{Varnish:handling}x in the varnish > ncsa log has been accurate in reporting HITs vs MISSes. Could we use a VCL > equivalent of this in vcl_deliver to determine whether a particular > response is the result of a HIT or a MISS? > > Any advice on the recommended way in VCL to determine whether a response > is the result of a HIT or MISS would be greatly appreciated! > > Thanks, > Umesh > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hugo.cisneiros at gmail.com Mon Apr 6 21:08:58 2015 From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch)) Date: Mon, 6 Apr 2015 18:08:58 -0300 Subject: Caching requests with Authorization Header Message-ID: Hi, >From documentation: https://www.varnish-cache.org/docs/trunk/users-guide/increasing-your-hitrate.html#authorization "Authorization If Varnish sees an 'Authorization' header it will pass the request. If this is not what you want you can unset the header." I have a scenario that works on varnish3 but won't work on varnish4. As the subject says, I want to cache requests that were made with the Authorization header. Before you start thinking "what a crazy and insecure thing to do", let me explain: I have an internal varnish cache that receives requests from applications and serve content from an external CDN that is authenticated (thus the Authorization header). Only internal servers use it and I do this to save bandwidth. Every possible client knows and uses the same Authorization header, so caching the requests are not an issue. I'm also trying to use the Authorization header in the hash_data too, since it's always the same and if someone requests without knowing the proper pass, varnish won't serve the right cache entry. When I use the Authorization header, varnish4 does not cache at all... And when I remove the Authorization header, it caches but I get 401 Forbidden from the CDN. Is there a way to solve this? -- []'s Hugo www.devin.com.br From raymond.jennings at nytimes.com Mon Apr 6 21:19:18 2015 From: raymond.jennings at nytimes.com (Jennings III, Raymond) Date: Mon, 6 Apr 2015 17:19:18 -0400 Subject: Caching requests with Authorization Header In-Reply-To: References: Message-ID: What about doing something like renaming that header so that the default vcl subroutine won't see it? Or maybe just always return from your sub-routine so that the default vcl never runs. I use a couple of techniques like what you are describing - where I add GET params on the URL based on a particular header value as well as the inverse (take GET parameters off of the URL so that they are not hashed) and stick those values into a "X-" type header. Maybe you can describe your technique in V3 so that we can see how we might do the same in V4. I'm not an expert on V4 but I have reworked some of the techniques that I have been using in V3 to V4 successfully. Raymond Jennings III *nytimes.com * *Office: 212.556.7786 <212-556-7786>* *iPhone: 914.330.5074 <914-330-5074>E-mail: Raymond.Jennings at nytimes.com FaceTime: Raymond.Jennings at nytimes.com * On Mon, Apr 6, 2015 at 5:08 PM, Hugo Cisneiros (Eitch) < hugo.cisneiros at gmail.com> wrote: > Hi, > > From documentation: > > https://www.varnish-cache.org/docs/trunk/users-guide/increasing-your-hitrate.html#authorization > > "Authorization > If Varnish sees an 'Authorization' header it will pass the request. > If this is not what you want you can unset the header." > > I have a scenario that works on varnish3 but won't work on varnish4. > > As the subject says, I want to cache requests that were made with the > Authorization header. Before you start thinking "what a crazy and > insecure thing to do", let me explain: > > I have an internal varnish cache that receives requests from > applications and serve content from an external CDN that is > authenticated (thus the Authorization header). Only internal servers > use it and I do this to save bandwidth. Every possible client knows > and uses the same Authorization header, so caching the requests are > not an issue. > > I'm also trying to use the Authorization header in the hash_data too, > since it's always the same and if someone requests without knowing the > proper pass, varnish won't serve the right cache entry. > > When I use the Authorization header, varnish4 does not cache at all... > And when I remove the Authorization header, it caches but I get 401 > Forbidden from the CDN. > > Is there a way to solve this? > > -- > []'s > Hugo > www.devin.com.br > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hugo.cisneiros at gmail.com Mon Apr 6 21:28:44 2015 From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch)) Date: Mon, 6 Apr 2015 18:28:44 -0300 Subject: Caching requests with Authorization Header In-Reply-To: References: Message-ID: On Mon, Apr 6, 2015 at 6:19 PM, Jennings III, Raymond wrote: > What about doing something like renaming that header so that the default vcl > subroutine won't see it? Or maybe just always return from your sub-routine > so that the default vcl never runs. Hi! I did that and it didn't work :( What I tried: sub vcl_recv { [...] set req.http.X-Authorization = req.http.Authorization; unset req.http.Authorization; [...] return (hash); } sub vcl_hash { [...] hash_data(req.http.X-Authorization); [...] return (lookup) } sub vcl_backend_fetch { set bereq.http.Authorization = bereq.http.X-Authorization; } I think varnish4 do the validation on between backend fetch and response, and decides that it won't store the reply on the cache since the backend request contained the Authorization header... Since it never stores on the cache, I'll always get a miss no matter what I do with the request o vcl_recv. > I use a couple of techniques like what you are describing - where I add GET > params on the URL based on a particular header value as well as the inverse > (take GET parameters off of the URL so that they are not hashed) and stick > those values into a "X-" type header. > > Maybe you can describe your technique in V3 so that we can see how we might > do the same in V4. I'm not an expert on V4 but I have reworked some of the > techniques that I have been using in V3 to V4 successfully. In V3 I really didn't need to do anything unusual. Just a simple caching VCL and it's working: sub vcl_recv { [...] unset req.http.cookie; return (lookup); } sub vcl_fetch { [...] set beresp.ttl = 1m; return (deliver); } Thanks, -- []'s Hugo www.devin.com.br From numard at gmail.com Mon Apr 6 21:44:17 2015 From: numard at gmail.com (Norberto Meijome) Date: Tue, 7 Apr 2015 07:44:17 +1000 Subject: Caching requests with Authorization Header In-Reply-To: References: Message-ID: I think you want to return (hash) from vcl_hash and lookup from vcl_recv... On 07/04/2015 7:30 am, "Hugo Cisneiros (Eitch)" wrote: > On Mon, Apr 6, 2015 at 6:19 PM, Jennings III, Raymond > wrote: > > What about doing something like renaming that header so that the default > vcl > > subroutine won't see it? Or maybe just always return from your > sub-routine > > so that the default vcl never runs. > > Hi! > > I did that and it didn't work :( > > What I tried: > > sub vcl_recv { > [...] > set req.http.X-Authorization = req.http.Authorization; > unset req.http.Authorization; > [...] > return (hash); > } > > sub vcl_hash { > [...] > hash_data(req.http.X-Authorization); > [...] > return (lookup) > } > > sub vcl_backend_fetch { > set bereq.http.Authorization = bereq.http.X-Authorization; > } > > I think varnish4 do the validation on between backend fetch and > response, and decides that it won't store the reply on the cache since > the backend request contained the Authorization header... Since it > never stores on the cache, I'll always get a miss no matter what I do > with the request o vcl_recv. > > > I use a couple of techniques like what you are describing - where I add > GET > > params on the URL based on a particular header value as well as the > inverse > > (take GET parameters off of the URL so that they are not hashed) and > stick > > those values into a "X-" type header. > > > > Maybe you can describe your technique in V3 so that we can see how we > might > > do the same in V4. I'm not an expert on V4 but I have reworked some of > the > > techniques that I have been using in V3 to V4 successfully. > > In V3 I really didn't need to do anything unusual. Just a simple > caching VCL and it's working: > > sub vcl_recv { > [...] > unset req.http.cookie; > return (lookup); > } > > sub vcl_fetch { > [...] > set beresp.ttl = 1m; > return (deliver); > } > > Thanks, > > -- > []'s > Hugo > www.devin.com.br > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hugo.cisneiros at gmail.com Mon Apr 6 21:55:30 2015 From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch)) Date: Mon, 6 Apr 2015 18:55:30 -0300 Subject: Caching requests with Authorization Header In-Reply-To: References: Message-ID: On Mon, Apr 6, 2015 at 6:44 PM, Norberto Meijome wrote: > I think you want to return (hash) from vcl_hash and lookup from vcl_recv... There is no need. This changed on varnish4. For reference, see these pages: https://www.varnish-cache.org/docs/trunk/users-guide/vcl-built-in-subs.html https://www.varnish-cache.org/docs/trunk/whats-new/upgrading.html -- []'s Hugo www.devin.com.br From vnxsmail at gmail.com Tue Apr 7 21:14:50 2015 From: vnxsmail at gmail.com (VNX de Vries) Date: Tue, 7 Apr 2015 23:14:50 +0200 Subject: Ban lurker doesn't seem to remove bans Message-ID: Hi, We use Varnish 4.0.3 for out website and serve a lot of pages with semi-unlimited (i.e. 1 year) TTL. When content gets changed, we ban te cache objects through special http headers. Please note that there can be 250.000+ new bans per day. Our problem is that page load times turn out to be slower and slower the longer Varnish runs. When checking with "varnishstat -1" it seems that band never get deleted. MAIN.bans_deleted is always 0. When i login to varnishadm and type the ban.list command I see a long list of bans with status "C". Varnishstat learns me that there can be hundreds of thousands of these bans with C status active. I don't have a clue what this C means, I can't find it anywhere in the documentation. --- Here are the important part from our VCL: sub vcl_recv { if (req.method == "BAN") { if (!client.ip ~ purgers) { return (synth(405, "Not allowed")); } if (req.http.X-Cache-Tags) { ban("obj.http.X-Cache-Tags ~ " + req.http.X-Cache-Tags); return (synth(200, "Banned")); } if (req.http.X-Url) { ban("obj.http.x-origurl ~ " + req.http.X-Url); return (synth(200, "Banned")); } return (synth(405, "No cache tags provided")); } ... } sub vcl_backend_response { set beresp.http.x-origurl = bereq.url; ... } --- Here's an example of the ban.list output from varnishadm: 1428440119.975700 20 C 1428440119.875178 21 C 1428440119.775048 0 C 1428440119.675068 1 C 1428440119.575059 0 C 1428440119.474236 0 C 1428440119.374664 1 C 1428440119.275055 0 C 1428440119.173203 1 C 1428440119.070568 1 C 1428440118.968622 0 C 1428440118.867101 0 C 1428440118.767099 2 C 1428440118.665274 3 C 1428440118.565995 2 C 1428440118.463959 4 C 1428440118.360379 3 C 1428440118.259975 4 C 1428440118.157665 4 C 1428440118.054391 8 C 1428440117.953973 3 C 1428440117.851816 16 C 1428440117.749820 7 C 1428440117.647959 8 C 1428440117.546431 2 C 1428440117.446593 2 C 1428440117.345096 1 C 1428440117.245096 0 C 1428440117.143993 0 C 1428440117.! --- Here are my two questions: 1) What does this C status mean? 2) Why aren't bans ever deleted? -------------- next part -------------- An HTML attachment was scrubbed... URL: From devel at jasonwoods.me.uk Wed Apr 8 06:02:06 2015 From: devel at jasonwoods.me.uk (Jason Woods) Date: Wed, 8 Apr 2015 07:02:06 +0100 Subject: Ban lurker doesn't seem to remove bans In-Reply-To: References: Message-ID: > On 7 Apr 2015, at 22:14, VNX de Vries wrote: > Here are my two questions: > 1) What does this C status mean? > I believe it means completed. > 2) Why aren't bans ever deleted? > I wonder if it's this bug: https://www.varnish-cache.org/trac/ticket/1635 Jason -------------- next part -------------- An HTML attachment was scrubbed... URL: From Umesh.Telang at bbc.co.uk Wed Apr 8 08:50:30 2015 From: Umesh.Telang at bbc.co.uk (Umesh Telang) Date: Wed, 8 Apr 2015 08:50:30 +0000 Subject: Determining a HIT or a MISS in Varnish 4 VCL In-Reply-To: References: Message-ID: Thanks Raymond. If we don't hear any views to the contrary from Varnish themselves, we'll go with the standard approach of using obj.hits Regards, Umesh From: "", Raymond > Date: Thursday, 2 April 2015 13:38 To: Umesh Telang > Cc: "varnish-misc at varnish-cache.org" > Subject: Re: Determining a HIT or a MISS in Varnish 4 VCL I think that's the way - exactly as you have in vcl_deliver. I've always used that in V3 and now in V4 as well. I'm suspect of that page telling you to not use obj in vcl_deliver but use that header value instead. Raymond Jennings III nytimes.com Office: 212.556.7786 iPhone: 914.330.5074 E-mail: Raymond.Jennings at nytimes.com FaceTime: Raymond.Jennings at nytimes.com On Thu, Apr 2, 2015 at 8:28 AM, Umesh Telang > wrote: Hello, It's unclear if the way to detect whether a particular response was the result of a cache hit or miss in VCL, has changed from Varnish 3 to Varnish 4. At this page (not an official Varnish page...), it suggests using the value of resp.http.X-Varnish (whether it has a single or multiple values) to determine this: http://foshttpcache.readthedocs.org/en/latest/varnish-configuration.html# Alternatively I've also seen suggestions for continuing to use obj.hits in vcl_deliver. e.g. : sub vcl_deliver { if (obj.hits > 0) { # Add debug header to see if it's a HIT/MISS and the number of hits, disable when not needed set resp.http.X-Cache = "HIT"; } else { set resp.http.X-Cache = "MISS"; } } We've observed that the value of %{Varnish:handling}x in the varnish ncsa log has been accurate in reporting HITs vs MISSes. Could we use a VCL equivalent of this in vcl_deliver to determine whether a particular response is the result of a HIT or a MISS? Any advice on the recommended way in VCL to determine whether a response is the result of a HIT or MISS would be greatly appreciated! Thanks, Umesh _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From hugo.cisneiros at gmail.com Tue Apr 14 13:32:04 2015 From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch)) Date: Tue, 14 Apr 2015 10:32:04 -0300 Subject: Caching requests with Authorization Header In-Reply-To: References: Message-ID: On Mon, Apr 6, 2015 at 6:08 PM, Hugo Cisneiros (Eitch) wrote: > "Authorization > If Varnish sees an 'Authorization' header it will pass the request. > If this is not what you want you can unset the header." > > I have a scenario that works on varnish3 but won't work on varnish4. > > As the subject says, I want to cache requests that were made with the > Authorization header. Before you start thinking "what a crazy and > insecure thing to do", let me explain: [...] > When I use the Authorization header, varnish4 does not cache at all... > And when I remove the Authorization header, it caches but I get 401 > Forbidden from the CDN. > > Is there a way to solve this? Hi! It's been a week and a didn't find any solutions for this yet :( I've been using Varnish 4 with great success in other installs and I really want to use on this one too, specially when Varnish 3 EOL is near :P Is this a feature that will always be like this, or this behavior can change? Is there anything I could do to help? :) Thanks! -- []'s Hugo www.devin.com.br From bertrand.caplet at chunkz.net Tue Apr 14 18:48:11 2015 From: bertrand.caplet at chunkz.net (Bertrand Caplet) Date: Tue, 14 Apr 2015 20:48:11 +0200 Subject: Varnish purge behind nginX proxy Message-ID: <552D60EB.30507@chunkz.net> Hey guys, I set varnish behind a nginX proxy few days ago. I want to be able to purge varnish only from the machine where it's hosted but because there is a nginx reverse proxy in front of it, varnish thinks it's coming from 127.0.0.1 so basically whatever the client is, anyone can purge the cache. By the way I set some rules for varnish to know x-forwarded-for = client.ip. Do you know how can I change that? Thanks in advance, -- CHUNKZ.NET - script kiddie and computer technician Bertrand Caplet, Flers (FR) Feel free to send encrypted/signed messages Key ID: 37F70C30 GPG FP: 134A 4027 518B 5F4D D409 558D BA9B 7BF0 37F7 0C30 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 473 bytes Desc: OpenPGP digital signature URL: From andrew.langhorn at digital.cabinet-office.gov.uk Tue Apr 14 19:10:56 2015 From: andrew.langhorn at digital.cabinet-office.gov.uk (Andrew Langhorn) Date: Tue, 14 Apr 2015 20:10:56 +0100 Subject: Varnish purge behind nginX proxy In-Reply-To: <552D60EB.30507@chunkz.net> References: <552D60EB.30507@chunkz.net> Message-ID: Can you post your VCL? There's a vmod to pass True-Client-Ip through. On Tuesday, 14 April 2015, Bertrand Caplet wrote: > Hey guys, > I set varnish behind a nginX proxy few days ago. > I want to be able to purge varnish only from the machine where it's > hosted but because there is a nginx reverse proxy in front of it, > varnish thinks it's coming from 127.0.0.1 so basically whatever the > client is, anyone can purge the cache. > By the way I set some rules for varnish to know x-forwarded-for = > client.ip. > > Do you know how can I change that? > > Thanks in advance, > -- > CHUNKZ.NET - script kiddie and computer technician > Bertrand Caplet, Flers (FR) > Feel free to send encrypted/signed messages > Key ID: 37F70C30 > GPG FP: 134A 4027 518B 5F4D D409 558D BA9B 7BF0 37F7 0C30 > > > -- -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Tue Apr 14 20:07:27 2015 From: perbu at varnish-software.com (Per Buer) Date: Tue, 14 Apr 2015 22:07:27 +0200 Subject: Varnish purge behind nginX proxy In-Reply-To: <552D60EB.30507@chunkz.net> References: <552D60EB.30507@chunkz.net> Message-ID: On Tue, Apr 14, 2015 at 8:48 PM, Bertrand Caplet wrote: > Hey guys, > I set varnish behind a nginX proxy few days ago. > I want to be able to purge varnish only from the machine where it's > hosted but because there is a nginx reverse proxy in front of it, > varnish thinks it's coming from 127.0.0.1 so basically whatever the > client is, anyone can purge the cache. > By the way I set some rules for varnish to know x-forwarded-for = > client.ip. > Do you know how can I change that? > See man vmod_std, look for the function "ip". It allows you to cast a string in a ip address to match it against a acl. Oh. There an online version; https://www.varnish-cache.org/docs/4.0/reference/vmod_std.generated.html#ip-ip-string-ip -- *Per Buer* CTO | Varnish Software AS Cell: +47 95839117 We Make Websites Fly! www.varnish-software.com [image: Register now] -------------- next part -------------- An HTML attachment was scrubbed... URL: From bedis9 at gmail.com Tue Apr 14 21:28:18 2015 From: bedis9 at gmail.com (Baptiste) Date: Tue, 14 Apr 2015 23:28:18 +0200 Subject: Varnish purge behind nginX proxy In-Reply-To: <552D60EB.30507@chunkz.net> References: <552D60EB.30507@chunkz.net> Message-ID: On Tue, Apr 14, 2015 at 8:48 PM, Bertrand Caplet wrote: > Hey guys, > I set varnish behind a nginX proxy few days ago. > I want to be able to purge varnish only from the machine where it's > hosted but because there is a nginx reverse proxy in front of it, > varnish thinks it's coming from 127.0.0.1 so basically whatever the > client is, anyone can purge the cache. > By the way I set some rules for varnish to know x-forwarded-for = client.ip. > > Do you know how can I change that? > > Thanks in advance, > -- > CHUNKZ.NET - script kiddie and computer technician > Bertrand Caplet, Flers (FR) > Feel free to send encrypted/signed messages > Key ID: 37F70C30 > GPG FP: 134A 4027 518B 5F4D D409 558D BA9B 7BF0 37F7 0C30 > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc Hey, This is why the proxy-protocol is made :) Unfortunately, nginx doesn't support it on the backend side. Baptiste From bertrand.caplet at chunkz.net Tue Apr 14 22:35:39 2015 From: bertrand.caplet at chunkz.net (Bertrand Caplet) Date: Wed, 15 Apr 2015 00:35:39 +0200 Subject: Varnish purge behind nginX proxy In-Reply-To: References: <552D60EB.30507@chunkz.net> Message-ID: <552D963B.5050907@chunkz.net> > Can you post your VCL? There's a vmod to pass True-Client-Ip through. Here you go: vcl 4.0; # Default backend definition. Set this to point to your content server. backend default { .host = "127.0.0.1"; .port = "8082"; } acl purge { "127.0.0.1"; } # Drop any cookies sent to Wordpress. sub vcl_recv { # Set the X-Forwarded-For header so the backend can see the original # IP address. If one is already set by an upstream proxy, we'll just re-use that. if (req.http.X-Forwarded-For) { # set or append the client.ip to X-Forwarded-For header set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip; } else { set req.http.X-Forwarded-For = client.ip; } if (req.url ~ "wp-admin|wp-login") { return (pass); } set req.http.cookie = regsuball(req.http.cookie, "wp-settings-\d+=[^;]+(; )?", ""); set req.http.cookie = regsuball(req.http.cookie, "wp-settings-time-\d+=[^;]+(; )?", ""); set req.http.cookie = regsuball(req.http.cookie, "wordpress_test_cookie=[^;]+(; )?", ""); set req.http.cookie = regsuball(req.http.cookie, "quick_chat[^;]", ""); set req.http.Cookie = regsuball(req.http.Cookie, "(^|(?<=; )) *__quick_chat_alias.=[^;]+;? *", "\1"); if (req.http.cookie == "") { unset req.http.cookie; } if (req.method == "PURGE") { if (!client.ip ~ purge) { return (synth(405,"Not allowed.")); } return (purge); } } I already tried vmod_std with this string : if (std.ip(req.http.X-forwarded-for, "0.0.0.0") !~ purge) but I got the following error : Message from VCC-compiler: Symbol not found: 'std.ip' (expected type BOOL) Do you have a clue ? -- CHUNKZ.NET - script kiddie and computer technician Bertrand Caplet, Flers (FR) Feel free to send encrypted/signed messages Key ID: 37F70C30 GPG FP: 134A 4027 518B 5F4D D409 558D BA9B 7BF0 37F7 0C30 From info at massivescale.net Wed Apr 15 08:23:52 2015 From: info at massivescale.net (Andrzej Godziuk) Date: Wed, 15 Apr 2015 10:23:52 +0200 Subject: Varnish 3 EOL Message-ID: <20150415102352.1c9ae202@gdr-desktop.gdr.name> Hello, Regarding the Varnish 3 EOL announcement [1], I understand that Varnish Software will not release security patches to Varnish 3 any more. Is that correct? Do you plan on cooperating with LTS Linux distributions who shipped Varnish 3? For example, Ubuntu 12.04 is supported until April 2017 and I wonder how urgent the upgrade to Varnish 4 is on systems running this OS. [1] https://www.varnish-cache.org/lists/pipermail/varnish-announce/2015-April/000702.html -- Andrzej Godziuk From dridi at varni.sh Wed Apr 15 08:25:00 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 15 Apr 2015 10:25:00 +0200 Subject: Varnish purge behind nginX proxy In-Reply-To: <552D963B.5050907@chunkz.net> References: <552D60EB.30507@chunkz.net> <552D963B.5050907@chunkz.net> Message-ID: On Wed, Apr 15, 2015 at 12:35 AM, Bertrand Caplet wrote: > Do you have a clue ? If your purger is on the same host, why bother going through nginx? Is your nginx proxy doing anything special required for both regular users and in-house tools? Dridi From feld at feld.me Wed Apr 15 13:29:14 2015 From: feld at feld.me (Mark Felder) Date: Wed, 15 Apr 2015 08:29:14 -0500 Subject: Varnish 3 EOL In-Reply-To: <20150415102352.1c9ae202@gdr-desktop.gdr.name> References: <20150415102352.1c9ae202@gdr-desktop.gdr.name> Message-ID: <1429104554.348073.254100401.318AF415@webmail.messagingengine.com> On Wed, Apr 15, 2015, at 03:23, Andrzej Godziuk wrote: > Hello, > > Regarding the Varnish 3 EOL announcement [1], I understand that Varnish > Software will not release security patches to Varnish 3 any more. Is > that correct? > > Do you plan on cooperating with LTS Linux distributions who shipped > Varnish 3? For example, Ubuntu 12.04 is supported until April 2017 and > I wonder how urgent the upgrade to Varnish 4 is on systems running this > OS. > > [1] > https://www.varnish-cache.org/lists/pipermail/varnish-announce/2015-April/000702.html > > I'm not aware of any LTS Linux distro that has upstream "cooperate" with them when issues arise in versions they dropped support for. It's up to the package maintainers to be competent enough to backport the security fixes themselves. Sadly, there is a disconnection in the way open source software is developed and the way Linux distros deliver it to end users. Mistakes are made all too regularly and you end up with situations like this: "The fix that was included in Debian for CVE-2012-1836 is incomplete, and does not solve the original remote code execution problem." So if you're worried about vulnerabilities in Varnish 3.x on LTS Linux distros I would advise you to not use their Varnish 3.x packages and to build Varnish yourself or find a trustworthy 3rd-party package repository that supplies packages for your favorite LTS Linux distro. The only other solution I can think of is to use a rolling-release Linus distro or one of the BSDs which use a rolling-release model for their non-base system software (ports/packages). From bertrand.caplet at chunkz.net Wed Apr 15 14:44:20 2015 From: bertrand.caplet at chunkz.net (Bertrand Caplet) Date: Wed, 15 Apr 2015 16:44:20 +0200 Subject: Varnish purge behind nginX proxy In-Reply-To: References: <552D60EB.30507@chunkz.net> <552D963B.5050907@chunkz.net> Message-ID: <7e4f8089d5fb5f3c946c22e49069e3e6@chunkz.net> Le 2015-04-15 10:25, Dridi Boukelmoune a ?crit?: > On Wed, Apr 15, 2015 at 12:35 AM, Bertrand Caplet > wrote: >> Do you have a clue ? > > If your purger is on the same host, why bother going through nginx? > > Is your nginx proxy doing anything special required for both regular > users and in-house tools? > > Dridi Because of SSL termination. Purging through commandline would be fine to me too. But I couldn't find a working command for 4.0 that flush all site. Regards, -- CHUNKZ.NET - dodgy DIYer and computer technician Bertrand Caplet, Flers (FR) Feel free to send encrypted/signed messages Key ID: FF395BD9 GPG FP: DE10 73FD 17EB 5544 A491 B385 1EDA 35DC FF39 5BD9 From info at massivescale.net Wed Apr 15 14:48:32 2015 From: info at massivescale.net (Andrzej Godziuk) Date: Wed, 15 Apr 2015 16:48:32 +0200 Subject: Varnish 3 EOL In-Reply-To: <1429104554.348073.254100401.318AF415@webmail.messagingengine.com> References: <20150415102352.1c9ae202@gdr-desktop.gdr.name> <1429104554.348073.254100401.318AF415@webmail.messagingengine.com> Message-ID: <20150415164832.2cd856f2@gdr-desktop.gdr.name> Thanks, this pretty much answers my question. From guillaume.quintard at smartjog.com Wed Apr 15 14:49:59 2015 From: guillaume.quintard at smartjog.com (Guillaume Quintard) Date: Wed, 15 Apr 2015 16:49:59 +0200 Subject: Varnish purge behind nginX proxy In-Reply-To: <7e4f8089d5fb5f3c946c22e49069e3e6@chunkz.net> References: <552D60EB.30507@chunkz.net> <552D963B.5050907@chunkz.net> <7e4f8089d5fb5f3c946c22e49069e3e6@chunkz.net> Message-ID: <552E7A97.8040205@smartjog.com> On 04/15/2015 04:44 PM, Bertrand Caplet wrote: > Because of SSL termination. > Purging through commandline would be fine to me too. But I couldn't > find a working command for 4.0 that flush all site. > > Regards, You'll probably have to fill a header (X-Forwarded-for is an obvious candidate) with nginx and check against that in varnish. -- Guillaume Quintard From dridi at varni.sh Wed Apr 15 15:59:14 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 15 Apr 2015 17:59:14 +0200 Subject: Varnish purge behind nginX proxy In-Reply-To: <7e4f8089d5fb5f3c946c22e49069e3e6@chunkz.net> References: <552D60EB.30507@chunkz.net> <552D963B.5050907@chunkz.net> <7e4f8089d5fb5f3c946c22e49069e3e6@chunkz.net> Message-ID: On Wed, Apr 15, 2015 at 4:44 PM, Bertrand Caplet wrote: > > > Because of SSL termination. You probably don't need SSL for an HTTP request on the same host. > Purging through commandline would be fine to me too. But I couldn't find a > working command for 4.0 that flush all site. Whatever request you'd send through nginx, you can send it with curl or any similar tool directly to your varnish instance. Regards, Dridi From dridi at varni.sh Wed Apr 15 16:07:27 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 15 Apr 2015 18:07:27 +0200 Subject: Caching requests with Authorization Header In-Reply-To: References: Message-ID: On Tue, Apr 14, 2015 at 3:32 PM, Hugo Cisneiros (Eitch) wrote: > Hi! > > It's been a week and a didn't find any solutions for this yet :( I've > been using Varnish 4 with great success in other installs and I really > want to use on this one too, specially when Varnish 3 EOL is near :P > > Is this a feature that will always be like this, or this behavior can > change? Is there anything I could do to help? :) Hi Hugo, What does varnishlog show you? Showing logs might help us figure what's going on. Regards, Dridi > Thanks! > > -- > []'s > Hugo > www.devin.com.br From miguel_3_gonzalez at yahoo.es Wed Apr 15 17:08:21 2015 From: miguel_3_gonzalez at yahoo.es (=?UTF-8?B?TWlndWVsIEdvbnrDoWxleg==?=) Date: Wed, 15 Apr 2015 19:08:21 +0200 Subject: overflow_max setting in varnish 4 Message-ID: <552E9B05.7020200@yahoo.es> Dear all, I have found this guide that is a bit outdated: https://blog.engineyard.com/2010/varnish-its-not-just-for-wood-anymore that mentions the overflow_max setting which is predecated. It was replaced by queue_max in version 3, but I can?t figure out if this setting has been removed in version 4. Regards, Miguel From miguel_3_gonzalez at yahoo.es Thu Apr 16 14:44:22 2015 From: miguel_3_gonzalez at yahoo.es (Miguel Gonzalez) Date: Thu, 16 Apr 2015 14:44:22 +0000 (UTC) Subject: increase hitrate Message-ID: <1962428629.5273348.1429195462345.JavaMail.yahoo@mail.yahoo.com> Dear all, ?? I have been testing my varnish configuration using tools.pingdom.com, webpagetest.org and loadimpact.com and apache benchmark. ? I? have Varnish 4.0 in front of an Apache installation from Cpanel in a VPS with 6 Gb with 100 mbps of bandwith. ?I'm realizing that i can only get a hitrate around 0.6 and the throughput is around 43 request per second.? What am I doing wrong? How can I troubleshoot what's going on? ?Thanks ?Miguel This is my /etc/sysconfig/varnish daemon configuration: /usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 120 -p thread_pools=2 -p thread_pool_add_delay=1 -p thread_pool_min=500 -p thread_pool_max=2000 -p thread_pool_timeout=120 -u varnish -g varnish -S /etc/varnish/secret -s malloc,3G This is my default.vcl configuration: ? vcl 4.0; import std; # Default backend definition. Set this to point to your content server. backend default { ??? .host = "192.168.1.10"; ??? .port = "82"; } acl purge { ??????? "localhost"; } # This function is used when a request is send by a HTTP client (Browser) sub vcl_recv { ??????? # remove ?ver=xxxxx strings from urls so css and js files are cached. ??????? # Watch out when upgrading WordPress, need to restart Varnish or flush cache. ??????? set req.url = regsub(req.url, "\?ver=.*$", ""); ? ??????? # Remove "replytocom" from requests to make caching better. ??????? set req.url = regsub(req.url, "\?replytocom=.*$", ""); ??????? # We pass real IP to the backend ??????? if (req.restarts == 0) { ??????????? if (req.http.X-Forwarded-For) { ?????????????? set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip; ?????????? } else { ??????????? set req.http.X-Forwarded-For = client.ip; ?????????? } ??????? } ??? ??? # Normalize the header, remove the port (in case you're testing this on various TCP ports) ??????? set req.http.Host = regsub(req.http.Host, ":[0-9]+", ""); ??? # Remove has_js and CloudFlare/Google Analytics __* cookies. ??? set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(_[_a-z]+|has_js)=[^;]*", ""); ??? # Remove a ";" prefix, if present. ??? set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", ""); ??? # Allow purging from ACL ??? if (req.method == "PURGE") { ??? ??? # If not allowed then a error 405 is returned ??? ??? if (!client.ip ~ purge) { ??? ??? ??? return(synth(405, "This IP is not allowed to send PURGE requests.")); ??? ??? } ??? ??? # If allowed, do a cache_lookup -> vlc_hit() or vlc_miss() ??? ??? return (purge); ??? } ??? # Post requests will not be cached ??? #if (req.http.Authorization || req.method == "POST") { ??? #??? return (pass); ??? #} ??????? # Pass anything other than GET and HEAD directly. ??????? if (req.method != "GET" && req.method != "HEAD") { ??????????????? return( pass ); ??????? }????? /* We only deal with GET and HEAD by default */ ??? # --- WordPress specific configuration ??? # Did not cache the admin and login pages ??? if (req.url ~ "wp-(login|admin)" || req.url ~ "preview=true") { ??? return (pass); ??? } ??? if (req.url ~ "(ajax|dynamic|custom)") { ???????????? return(pass); ??????? } ??? # Remove the "has_js" cookie ??? set req.http.Cookie = regsuball(req.http.Cookie, "has_js=[^;]+(; )?", ""); ??? # Remove any Google Analytics based cookies ??? set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", ""); ??? # Remove the Quant Capital cookies (added by some plugin, all __qca) ??? set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(; )?", ""); ??? # Remove the wp-settings-1 cookie ??? set req.http.Cookie = regsuball(req.http.Cookie, "wp-settings-1=[^;]+(; )?", ""); ??? # Remove the wp-settings-time-1 cookie ??? set req.http.Cookie = regsuball(req.http.Cookie, "wp-settings-time-1=[^;]+(; )?", ""); ??? # Remove the wp test cookie ??? set req.http.Cookie = regsuball(req.http.Cookie, "wordpress_test_cookie=[^;]+(; )?", ""); ??? # Are there cookies left with only spaces or that are empty? ??? if (req.http.cookie ~ "^ *$") { ??? ??? ??? unset req.http.cookie; ??? } ??? # Cache the following files extensions ??? if (req.url ~ "\.(css|js|png|gif|jp(e)?g|swf|ico)") { ??? ??? unset req.http.cookie; ??? } ??? # Normalize Accept-Encoding header and compression ??? # https://www.varnish-cache.org/docs/3.0/tutorial/vary.html ??? if (req.http.Accept-Encoding) { ??? ??? # Do no compress compressed files... ??? ??? if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") { ??? ??? ??? ?? ??? unset req.http.Accept-Encoding; ??? ??? } elsif (req.http.Accept-Encoding ~ "gzip") { ??? ??? ??? ??? set req.http.Accept-Encoding = "gzip"; ??? ??? } elsif (req.http.Accept-Encoding ~ "deflate") { ??? ??? ??? ??? set req.http.Accept-Encoding = "deflate"; ??? ??? } else { ??? ??? ??? unset req.http.Accept-Encoding; ??? ??? } ??? } ??? # Check the cookies for wordpress-specific items ??? if (req.http.Cookie ~ "wordpress_" || req.http.Cookie ~ "comment_") { ??? ??? return (pass); ??? } ??? if (!req.http.cookie) { ??? ??? unset req.http.cookie; ??? } ??? # --- End of WordPress specific configuration ??? # Did not cache HTTP authentication and HTTP Cookie ??? if (req.http.Authorization || req.http.Cookie) { ??? ??? # Not cacheable by default ??? ??? return (pass); ??? } ??? # Cache all others requests ??? return (hash); } sub vcl_pipe { ??? return (pipe); } sub vcl_pass { ??? return (fetch); } # The data on which the hashing will take place sub vcl_hash { ???? hash_data(req.url); ???? if (req.http.host) { ???? ??? hash_data(req.http.host); ???? } else { ???? ??? hash_data(server.ip); ???? } ??? # If the client supports compression, keep that in a different cache ??? ??? if (req.http.Accept-Encoding) { ??????? ??? hash_data(req.http.Accept-Encoding); ??? } ??? return (lookup); } # This function is used when a request is sent by our backend (Nginx server) sub vcl_backend_response { ??? # Remove some headers we never want to see ??? unset beresp.http.Server; ??? unset beresp.http.X-Powered-By; ??? # For static content strip all backend cookies ??? if (bereq.url ~ "\.(css|js|png|gif|jp(e?)g)|swf|ico") { ??? ??? unset beresp.http.cookie; ??? } ??? # Don't store backend ??? if (bereq.url ~ "wp-(login|admin)" || bereq.url ~ "preview=true") { ??? ??? set beresp.uncacheable = true; ??? ??? set beresp.ttl = 30s; ??? ??? return (deliver); ??? } ??? # Only allow cookies to be set if we're in admin area ??? ??? if (!(bereq.url ~ "(wp-login|wp-admin|preview=true)")) { ??????? ??? unset beresp.http.set-cookie; ??? } ??? # don't cache response to posted requests or those with basic auth ??? if ( bereq.method == "POST" || bereq.http.Authorization ) { ??????? ??? set beresp.uncacheable = true; ??? ??? set beresp.ttl = 120s; ??? ??? return (deliver); ??? ??? } ??? ??? # don't cache search results ??? if ( bereq.url ~ "\?s=" ){ ??? ??? set beresp.uncacheable = true; ??????????????? set beresp.ttl = 120s; ??????????????? return (deliver); ??? } ??? # only cache status ok ??? if ( beresp.status != 200 ) { ??? ??? set beresp.uncacheable = true; ??????????????? set beresp.ttl = 120s; ??????????????? return (deliver); ??? } ??? # A TTL of 2h ??? set beresp.ttl = 2h; ??? # Define the default grace period to serve cached content ??? set beresp.grace = 30s; ??? return (deliver); } # The routine when we deliver the HTTP request to the user # Last chance to modify headers that are sent to the client sub vcl_deliver { ??? if (obj.hits > 0) { ??? ??? set resp.http.X-Cache = "cached"; ??? } else { ??? ??? set resp.http.x-Cache = "uncached"; ??? } ??? # Remove some headers: PHP version ??? unset resp.http.X-Powered-By; ??? # Remove some headers: Apache version & OS ??? unset resp.http.Server; ??? # Remove some heanders: Varnish ??? unset resp.http.Via; ??? unset resp.http.X-Varnish; ??? return (deliver); } sub vcl_init { ???? return (ok); } sub vcl_fini { ???? return (ok); } -------------- next part -------------- An HTML attachment was scrubbed... URL: From r at roze.lv Thu Apr 16 16:04:39 2015 From: r at roze.lv (Reinis Rozitis) Date: Thu, 16 Apr 2015 19:04:39 +0300 Subject: increase hitrate In-Reply-To: <1962428629.5273348.1429195462345.JavaMail.yahoo@mail.yahoo.com> References: <1962428629.5273348.1429195462345.JavaMail.yahoo@mail.yahoo.com> Message-ID: > I'm realizing that i can only get a hitrate around 0.6 and the throughput > is around 43 request per second. What am I doing wrong? How can I > troubleshoot what's going on? > > > sub vcl_deliver { > if (obj.hits > 0) { > set resp.http.X-Cache = "cached"; > } else { > set resp.http.x-Cache = "uncached"; > } Without going too much into the details, do you get those 43 req/s with the cached header? Eg is Varnish even caching anything or just piping the requests back to your backend (since in the configuration there are quite a lot conditions where the request is not cached)? p.s. for testing purposes just to see the varnish raw throughput take a static file. rr From jdh132 at psu.edu Thu Apr 16 16:05:33 2015 From: jdh132 at psu.edu (Jason Heffner) Date: Thu, 16 Apr 2015 12:05:33 -0400 Subject: Don't cache large files Message-ID: I?m working on updating our Varnish 3 config to Varnish 4 and I was able to figure out how to have the same functionality for almost all of the config. I wanted to find opinions on how to handle large files. We don?t want to cache any large files and had this in place for varnish 3. I?ve read streaming instead of piping was a solution but found nothing to determine this based solely on file size from the backend. # Varnish 3 config added to vcl_recv: /* Bypass cache for large files. The x-pipe header is set in vcl_fetch when a too large file is detected. */ if (req.http.x-pipe && req.restarts > 0) { remove req.http.x-pipe; return (pipe); } added to vcl_fetch: # don't cache files larger than 10MB /* Don't try to cache too large files. It appears Varnish just crashes if we don't filter them. */ if (beresp.http.Content-Length ~ "[0-9]{8,}" ) { set req.http.x-pipe = "1"; return (restart); } Since req.* is no longer available in vcl_recv this code doesn?t function anymore. Thanks, Jason From miguel_3_gonzalez at yahoo.es Thu Apr 16 17:27:11 2015 From: miguel_3_gonzalez at yahoo.es (Miguel Gonzalez) Date: Thu, 16 Apr 2015 17:27:11 +0000 (UTC) Subject: increase hitrate In-Reply-To: References: Message-ID: <1444195914.5485066.1429205231470.JavaMail.yahoo@mail.yahoo.com> Running a curl -I against a txt file,? I get this: HTTP/1.1 200 OK Date: Thu, 16 Apr 2015 17:20:02 GMT Vary: Accept-Encoding,User-Agent Last-Modified: Thu, 23 Sep 2010 14:56:01 GMT Cache-Control: max-age=3600, public Expires: Thu, 16 Apr 2015 18:20:02 GMT Pragma: public Content-Type: text/plain Age: 0 ETag: W/"460f-490ee75dee240-gzip" x-Cache: uncached Connection: keep-alive so it seems they are not getting cached and getting an User-Agent in Vary. I'm hosting Wordpress sites with the W3 Total cache plugin running and mod_pagespeed activated. Should I disable some of them? Thanks! Miguel De: Reinis Rozitis Para: varnish-misc at varnish-cache.org Enviado: Jueves 16 de abril de 2015 18:04 Asunto: Re: increase hitrate >? I'm realizing that i can only get a hitrate around 0.6 and the throughput > is around 43 request per second.? What am I doing wrong? How can I > troubleshoot what's going on? > > > sub vcl_deliver { >? ? if (obj.hits > 0) { >? ? ? ? set resp.http.X-Cache = "cached"; >? ? } else { >? ? ? set resp.http.x-Cache = "uncached"; >? } Without going too much into the details, do you get those 43 req/s with the cached header? Eg is Varnish even caching anything or just piping the requests back to your backend (since in the configuration there are quite a lot conditions where the request is not cached)? p.s. for testing purposes just to see the varnish raw throughput take a static file. rr _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From bertrand.caplet at chunkz.net Thu Apr 16 17:42:36 2015 From: bertrand.caplet at chunkz.net (Bertrand Caplet) Date: Thu, 16 Apr 2015 19:42:36 +0200 Subject: Varnish purge behind nginX proxy In-Reply-To: <552E8E7F.6020902@chunkz.net> References: <552D60EB.30507@chunkz.net> <552D963B.5050907@chunkz.net> <7e4f8089d5fb5f3c946c22e49069e3e6@chunkz.net> <552E8E7F.6020902@chunkz.net> Message-ID: <552FF48C.4040605@chunkz.net> To make it clear for you here's my config: Client ---> Nginx Reverse SSL ---> Varnish ---> Nginx + PHP-FPM X-Forwarded-for is already completed by nginx (and working ofc) and I tried as Per Buer told me with this rule : if (std.ip(req.http.X-forwarded-for, "0.0.0.0") !~ purge) but I got the following error : Message from VCC-compiler: Symbol not found: 'std.ip' (expected type BOOL) -- CHUNKZ.NET - script kiddie and computer technician Bertrand Caplet, Flers (FR) Feel free to send encrypted/signed messages Key ID: 37F70C30 GPG FP: 134A 4027 518B 5F4D D409 558D BA9B 7BF0 37F7 0C30 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 473 bytes Desc: OpenPGP digital signature URL: From bertrand.caplet at chunkz.net Thu Apr 16 17:43:58 2015 From: bertrand.caplet at chunkz.net (Bertrand Caplet) Date: Thu, 16 Apr 2015 19:43:58 +0200 Subject: Varnish purge behind nginX proxy In-Reply-To: References: <552D60EB.30507@chunkz.net> <552D963B.5050907@chunkz.net> <7e4f8089d5fb5f3c946c22e49069e3e6@chunkz.net> <552E8E7F.6020902@chunkz.net> Message-ID: <552FF4DE.3020604@chunkz.net> > Did you import the standard module at the beginning of your VCL? > > import std; Yes thanks !! I forgot to import it. The only rule to make it work is : if (!std.ip(regsub(req.http.X-Forwarded-For, "[, ].*$", ""), client.ip) ~ purge) Thanks again, have a good evening. (by the way sorry for replying to you, there was no 'reply to list' on my mail client.) -- CHUNKZ.NET - script kiddie and computer technician Bertrand Caplet, Flers (FR) Feel free to send encrypted/signed messages Key ID: 37F70C30 GPG FP: 134A 4027 518B 5F4D D409 558D BA9B 7BF0 37F7 0C30 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 473 bytes Desc: OpenPGP digital signature URL: From miguel_3_gonzalez at yahoo.es Thu Apr 16 18:36:28 2015 From: miguel_3_gonzalez at yahoo.es (Miguel Gonzalez) Date: Thu, 16 Apr 2015 18:36:28 +0000 (UTC) Subject: increase hitrate In-Reply-To: <1962428629.5273348.1429195462345.JavaMail.yahoo@mail.yahoo.com> References: <1962428629.5273348.1429195462345.JavaMail.yahoo@mail.yahoo.com> Message-ID: <1806867220.5536486.1429209388467.JavaMail.yahoo@mail.yahoo.com> Ok, I have just disabled w3 total cache, all deflate, expiresbytype and modpagespeed in .htaccess and now the hitrate is close to 1 and the load time is less than a second (before it was 1.5 seconds). But still the number of requests per second reported by Apache Benchmark is low. Regards, Miguel De: Miguel Gonzalez Para: "varnish-misc at varnish-cache.org" Enviado: Jueves 16 de abril de 2015 16:44 Asunto: increase hitrate Dear all, ?? I have been testing my varnish configuration using tools.pingdom.com, webpagetest.org and loadimpact.com and apache benchmark. ? I? have Varnish 4.0 in front of an Apache installation from Cpanel in a VPS with 6 Gb with 100 mbps of bandwith. ?I'm realizing that i can only get a hitrate around 0.6 and the throughput is around 43 request per second.? What am I doing wrong? How can I troubleshoot what's going on? ?Thanks ?Miguel This is my /etc/sysconfig/varnish daemon configuration: /usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 120 -p thread_pools=2 -p thread_pool_add_delay=1 -p thread_pool_min=500 -p thread_pool_max=2000 -p thread_pool_timeout=120 -u varnish -g varnish -S /etc/varnish/secret -s malloc,3G This is my default.vcl configuration: ? vcl 4.0; import std; # Default backend definition. Set this to point to your content server. backend default { ??? .host = "192.168.1.10"; ??? .port = "82"; } acl purge { ??????? "localhost"; } # This function is used when a request is send by a HTTP client (Browser) sub vcl_recv { ??????? # remove ?ver=xxxxx strings from urls so css and js files are cached. ??????? # Watch out when upgrading WordPress, need to restart Varnish or flush cache. ??????? set req.url = regsub(req.url, "\?ver=.*$", ""); ? ??????? # Remove "replytocom" from requests to make caching better. ??????? set req.url = regsub(req.url, "\?replytocom=.*$", ""); ??????? # We pass real IP to the backend ??????? if (req.restarts == 0) { ??????????? if (req.http.X-Forwarded-For) { ?????????????? set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip; ?????????? } else { ??????????? set req.http.X-Forwarded-For = client.ip; ?????????? } ??????? } ??? ??? # Normalize the header, remove the port (in case you're testing this on various TCP ports) ??????? set req.http.Host = regsub(req.http.Host, ":[0-9]+", ""); ??? # Remove has_js and CloudFlare/Google Analytics __* cookies. ??? set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(_[_a-z]+|has_js)=[^;]*", ""); ??? # Remove a ";" prefix, if present. ??? set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", ""); ??? # Allow purging from ACL ??? if (req.method == "PURGE") { ??? ??? # If not allowed then a error 405 is returned ??? ??? if (!client.ip ~ purge) { ??? ??? ??? return(synth(405, "This IP is not allowed to send PURGE requests.")); ??? ??? } ??? ??? # If allowed, do a cache_lookup -> vlc_hit() or vlc_miss() ??? ??? return (purge); ??? } ??? # Post requests will not be cached ??? #if (req.http.Authorization || req.method == "POST") { ??? #??? return (pass); ??? #} ??????? # Pass anything other than GET and HEAD directly. ??????? if (req.method != "GET" && req.method != "HEAD") { ??????????????? return( pass ); ??????? }????? /* We only deal with GET and HEAD by default */ ??? # --- WordPress specific configuration ??? # Did not cache the admin and login pages ??? if (req.url ~ "wp-(login|admin)" || req.url ~ "preview=true") { ??? return (pass); ??? } ??? if (req.url ~ "(ajax|dynamic|custom)") { ???????????? return(pass); ??????? } ??? # Remove the "has_js" cookie ??? set req.http.Cookie = regsuball(req.http.Cookie, "has_js=[^;]+(; )?", ""); ??? # Remove any Google Analytics based cookies ??? set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", ""); ??? # Remove the Quant Capital cookies (added by some plugin, all __qca) ??? set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(; )?", ""); ??? # Remove the wp-settings-1 cookie ??? set req.http.Cookie = regsuball(req.http.Cookie, "wp-settings-1=[^;]+(; )?", ""); ??? # Remove the wp-settings-time-1 cookie ??? set req.http.Cookie = regsuball(req.http.Cookie, "wp-settings-time-1=[^;]+(; )?", ""); ??? # Remove the wp test cookie ??? set req.http.Cookie = regsuball(req.http.Cookie, "wordpress_test_cookie=[^;]+(; )?", ""); ??? # Are there cookies left with only spaces or that are empty? ??? if (req.http.cookie ~ "^ *$") { ??? ??? ??? unset req.http.cookie; ??? } ??? # Cache the following files extensions ??? if (req.url ~ "\.(css|js|png|gif|jp(e)?g|swf|ico)") { ??? ??? unset req.http.cookie; ??? } ??? # Normalize Accept-Encoding header and compression ??? # https://www.varnish-cache.org/docs/3.0/tutorial/vary.html ??? if (req.http.Accept-Encoding) { ??? ??? # Do no compress compressed files... ??? ??? if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") { ??? ??? ??? ?? ??? unset req.http.Accept-Encoding; ??? ??? } elsif (req.http.Accept-Encoding ~ "gzip") { ??? ??? ??? ??? set req.http.Accept-Encoding = "gzip"; ??? ??? } elsif (req.http.Accept-Encoding ~ "deflate") { ??? ??? ??? ??? set req.http.Accept-Encoding = "deflate"; ??? ??? } else { ??? ??? ??? unset req.http.Accept-Encoding; ??? ??? } ??? } ??? # Check the cookies for wordpress-specific items ??? if (req.http.Cookie ~ "wordpress_" || req.http.Cookie ~ "comment_") { ??? ??? return (pass); ??? } ??? if (!req.http.cookie) { ??? ??? unset req.http.cookie; ??? } ??? # --- End of WordPress specific configuration ??? # Did not cache HTTP authentication and HTTP Cookie ??? if (req.http.Authorization || req.http.Cookie) { ??? ??? # Not cacheable by default ??? ??? return (pass); ??? } ??? # Cache all others requests ??? return (hash); } sub vcl_pipe { ??? return (pipe); } sub vcl_pass { ??? return (fetch); } # The data on which the hashing will take place sub vcl_hash { ???? hash_data(req.url); ???? if (req.http.host) { ???? ??? hash_data(req.http.host); ???? } else { ???? ??? hash_data(server.ip); ???? } ??? # If the client supports compression, keep that in a different cache ??? ??? if (req.http.Accept-Encoding) { ??????? ??? hash_data(req.http.Accept-Encoding); ??? } ??? return (lookup); } # This function is used when a request is sent by our backend (Nginx server) sub vcl_backend_response { ??? # Remove some headers we never want to see ??? unset beresp.http.Server; ??? unset beresp.http.X-Powered-By; ??? # For static content strip all backend cookies ??? if (bereq.url ~ "\.(css|js|png|gif|jp(e?)g)|swf|ico") { ??? ??? unset beresp.http.cookie; ??? } ??? # Don't store backend ??? if (bereq.url ~ "wp-(login|admin)" || bereq.url ~ "preview=true") { ??? ??? set beresp.uncacheable = true; ??? ??? set beresp.ttl = 30s; ??? ??? return (deliver); ??? } ??? # Only allow cookies to be set if we're in admin area ??? ??? if (!(bereq.url ~ "(wp-login|wp-admin|preview=true)")) { ??????? ??? unset beresp.http.set-cookie; ??? } ??? # don't cache response to posted requests or those with basic auth ??? if ( bereq.method == "POST" || bereq.http.Authorization ) { ??????? ??? set beresp.uncacheable = true; ??? ??? set beresp.ttl = 120s; ??? ??? return (deliver); ??? ??? } ??? ??? # don't cache search results ??? if ( bereq.url ~ "\?s=" ){ ??? ??? set beresp.uncacheable = true; ??????????????? set beresp.ttl = 120s; ??????????????? return (deliver); ??? } ??? # only cache status ok ??? if ( beresp.status != 200 ) { ??? ??? set beresp.uncacheable = true; ??????????????? set beresp.ttl = 120s; ??????????????? return (deliver); ??? } ??? # A TTL of 2h ??? set beresp.ttl = 2h; ??? # Define the default grace period to serve cached content ??? set beresp.grace = 30s; ??? return (deliver); } # The routine when we deliver the HTTP request to the user # Last chance to modify headers that are sent to the client sub vcl_deliver { ??? if (obj.hits > 0) { ??? ??? set resp.http.X-Cache = "cached"; ??? } else { ??? ??? set resp.http.x-Cache = "uncached"; ??? } ??? # Remove some headers: PHP version ??? unset resp.http.X-Powered-By; ??? # Remove some headers: Apache version & OS ??? unset resp.http.Server; ??? # Remove some heanders: Varnish ??? unset resp.http.Via; ??? unset resp.http.X-Varnish; ??? return (deliver); } sub vcl_init { ???? return (ok); } sub vcl_fini { ???? return (ok); } _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Fri Apr 17 06:58:30 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Fri, 17 Apr 2015 08:58:30 +0200 Subject: Don't cache large files In-Reply-To: References: Message-ID: Why pipe when you could simply have a hit-for-pass object? In vcl_backend_response you could simply set beresp.uncacheable to true (return(pass) in vcl_fetch for v3). With such a config you'd hit your backend twice for all responses of more than 10MB. Dridi On Thu, Apr 16, 2015 at 6:05 PM, Jason Heffner wrote: > I?m working on updating our Varnish 3 config to Varnish 4 and I was able to figure out how to have the same functionality for almost all of the config. I wanted to find opinions on how to handle large files. We don?t want to cache any large files and had this in place for varnish 3. I?ve read streaming instead of piping was a solution but found nothing to determine this based solely on file size from the backend. > > # Varnish 3 config > added to vcl_recv: > /* Bypass cache for large files. The x-pipe header is > set in vcl_fetch when a too large file is detected. */ > if (req.http.x-pipe && req.restarts > 0) { > remove req.http.x-pipe; > return (pipe); > } > > added to vcl_fetch: > # don't cache files larger than 10MB > /* Don't try to cache too large files. It appears > Varnish just crashes if we don't filter them. */ > if (beresp.http.Content-Length ~ "[0-9]{8,}" ) { > set req.http.x-pipe = "1"; > return (restart); > } > > Since req.* is no longer available in vcl_recv this code doesn?t function anymore. > > Thanks, > Jason > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From guillaume.quintard at smartjog.com Fri Apr 17 08:19:33 2015 From: guillaume.quintard at smartjog.com (Guillaume Quintard) Date: Fri, 17 Apr 2015 10:19:33 +0200 Subject: Varnish purge behind nginX proxy In-Reply-To: <552FF48C.4040605@chunkz.net> References: <552D60EB.30507@chunkz.net> <552D963B.5050907@chunkz.net> <7e4f8089d5fb5f3c946c22e49069e3e6@chunkz.net> <552E8E7F.6020902@chunkz.net> <552FF48C.4040605@chunkz.net> Message-ID: <5530C215.7000007@smartjog.com> On 04/16/2015 07:42 PM, Bertrand Caplet wrote: > To make it clear for you here's my config: > > > Client ---> Nginx Reverse SSL ---> Varnish ---> Nginx + PHP-FPM > > X-Forwarded-for is already completed by nginx (and working ofc) and I > tried as Per Buer told me with this rule : > > if (std.ip(req.http.X-forwarded-for, "0.0.0.0") !~ purge) > > but I got the following error : > > Message from VCC-compiler: > Symbol not found: 'std.ip' (expected type BOOL) > could you be missing a "import std;" ? -- Guillaume Quintard From bertrand.caplet at chunkz.net Fri Apr 17 10:15:39 2015 From: bertrand.caplet at chunkz.net (Bertrand Caplet) Date: Fri, 17 Apr 2015 12:15:39 +0200 Subject: Varnish purge behind nginX proxy In-Reply-To: <5530C215.7000007@smartjog.com> References: <552D60EB.30507@chunkz.net> <552D963B.5050907@chunkz.net> <7e4f8089d5fb5f3c946c22e49069e3e6@chunkz.net> <552E8E7F.6020902@chunkz.net> <552FF48C.4040605@chunkz.net> <5530C215.7000007@smartjog.com> Message-ID: <4d2799fc25786169119c51e353574041@chunkz.net> > could you be missing a "import std;" ? Yes I was, thanks :-) But the if string here : https://www.varnish-cache.org/docs/4.0/reference/vmod_std.generated.html#ip-ip-string-ip didn't work for me. And for information this one works really well : if (!std.ip(regsub(req.http.X-Forwarded-For, "[, ].*$", ""), client.ip) ~ purge) -- CHUNKZ.NET - dodgy DIYer and computer technician Bertrand Caplet, Flers (FR) Feel free to send encrypted/signed messages Key ID: FF395BD9 GPG FP: DE10 73FD 17EB 5544 A491 B385 1EDA 35DC FF39 5BD9 From jdh132 at psu.edu Fri Apr 17 12:38:47 2015 From: jdh132 at psu.edu (Jason Heffner) Date: Fri, 17 Apr 2015 08:38:47 -0400 Subject: Don't cache large files In-Reply-To: References: Message-ID: <5CF29039-FABC-4571-AE9A-3556FCA59B52@psu.edu> Thanks Dridi. I had switched to just marking the content uncacheable and using return(deliver), per the new v4 method, temporarily. I wasn?t sure if there was any advantage to using pipe anymore for files ranging from ~10MB - 500MB in size on the server? Jason > On Apr 17, 2015, at 2:58 AM, Dridi Boukelmoune wrote: > > Why pipe when you could simply have a hit-for-pass object? > > In vcl_backend_response you could simply set beresp.uncacheable to > true (return(pass) in vcl_fetch for v3). With such a config you'd hit > your backend twice for all responses of more than 10MB. > > Dridi > > On Thu, Apr 16, 2015 at 6:05 PM, Jason Heffner wrote: >> I?m working on updating our Varnish 3 config to Varnish 4 and I was able to figure out how to have the same functionality for almost all of the config. I wanted to find opinions on how to handle large files. We don?t want to cache any large files and had this in place for varnish 3. I?ve read streaming instead of piping was a solution but found nothing to determine this based solely on file size from the backend. >> >> # Varnish 3 config >> added to vcl_recv: >> /* Bypass cache for large files. The x-pipe header is >> set in vcl_fetch when a too large file is detected. */ >> if (req.http.x-pipe && req.restarts > 0) { >> remove req.http.x-pipe; >> return (pipe); >> } >> >> added to vcl_fetch: >> # don't cache files larger than 10MB >> /* Don't try to cache too large files. It appears >> Varnish just crashes if we don't filter them. */ >> if (beresp.http.Content-Length ~ "[0-9]{8,}" ) { >> set req.http.x-pipe = "1"; >> return (restart); >> } >> >> Since req.* is no longer available in vcl_recv this code doesn?t function anymore. >> >> Thanks, >> Jason >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From raymond.jennings at nytimes.com Wed Apr 22 13:40:11 2015 From: raymond.jennings at nytimes.com (Jennings III, Raymond) Date: Wed, 22 Apr 2015 09:40:11 -0400 Subject: Trying to make sense of varnishstat Message-ID: I have the following output (Varnish 4) MAIN.client_req 2098626 39.11 Good client requests received MAIN.cache_hit 1122150 20.91 Cache hits MAIN.cache_hitpass 450 0.01 Cache hits for pass MAIN.cache_miss 246939 4.60 Cache misses Varnish 3 used to have "client_conn" and that was essentially the sum of the hit, pass, hitpass. What am I missing here? Raymond Jennings III *nytimes.com * *Office: 212.556.7786 <212-556-7786>* *iPhone: 914.330.5074 <914-330-5074>E-mail: Raymond.Jennings at nytimes.com FaceTime: Raymond.Jennings at nytimes.com * -------------- next part -------------- An HTML attachment was scrubbed... URL: From japrice at gmail.com Fri Apr 24 01:57:47 2015 From: japrice at gmail.com (Jason Price) Date: Thu, 23 Apr 2015 21:57:47 -0400 Subject: Varnish 3 EOL In-Reply-To: <20150415102352.1c9ae202@gdr-desktop.gdr.name> References: <20150415102352.1c9ae202@gdr-desktop.gdr.name> Message-ID: Well... this is quite a challenging statement. My challenge is this: The vmod community is awaiting features in 4.1 (not released yet) before they can begin porting over to varnish 4.x. What is the correct approach to handle this situation? Varnish 4 is 'incomplete' to the needs of these users, yet varnish3 is EOL. -Jason (if the required features to support vmod_redis and similar have made it to 4.x, please correct my understanding) On Wed, Apr 15, 2015 at 4:23 AM, Andrzej Godziuk wrote: > Hello, > > Regarding the Varnish 3 EOL announcement [1], I understand that Varnish > Software will not release security patches to Varnish 3 any more. Is > that correct? > > Do you plan on cooperating with LTS Linux distributions who shipped > Varnish 3? For example, Ubuntu 12.04 is supported until April 2017 and > I wonder how urgent the upgrade to Varnish 4 is on systems running this > OS. > > [1] > > https://www.varnish-cache.org/lists/pipermail/varnish-announce/2015-April/000702.html > > -- > Andrzej Godziuk > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tobias.eichelbroenner at lamp-solutions.de Fri Apr 24 11:32:37 2015 From: tobias.eichelbroenner at lamp-solutions.de (=?windows-1252?Q?Tobias_Eichelbr=F6nner?=) Date: Fri, 24 Apr 2015 13:32:37 +0200 Subject: Varnish 3 EOL In-Reply-To: References: <20150415102352.1c9ae202@gdr-desktop.gdr.name> Message-ID: <553A29D5.9050801@lamp-solutions.de> Hi Jason, > -Jason (if the required features to support vmod_redis and similar have > made it to 4.x, please correct my understanding) about vmod_redis. Do you know https://www.varnish-cache.org/vmod/redis-0 We have it running with varnish4 without any problems. Sincerely, Tobias -- LAMP solutions GmbH Gostenhofer Hauptstrasse 35 90443 Nuernberg Amtsgericht Nuernberg: HRB 22366 Geschaeftsfuehrer: Heiko Schubert Es gelten unsere allgemeinen Geschaeftsbedingungen. http://www.lamp-solutions.de/agbs/ Telefon : 0911 / 376 516 0 Fax : 0911 / 376 516 11 E-Mail : support at lamp-solutions.de Web : www.lamp-solutions.de Facebook : http://www.facebook.com/LAMPsolutions Twitter : http://twitter.com/#!/lampsolutions From perbu at varnish-software.com Fri Apr 24 11:35:33 2015 From: perbu at varnish-software.com (Per Buer) Date: Fri, 24 Apr 2015 13:35:33 +0200 Subject: Varnish 3 EOL In-Reply-To: References: <20150415102352.1c9ae202@gdr-desktop.gdr.name> Message-ID: Jason, It would be up to the maintainer of the Redis VMOD to port it to Varnish 4.0. It should be very straightforward to port the redis vmod forward unless it does something really weird. I've ported VMODs to 4 and if I can do it then most other people should manage. Of course the problem might be that the VMOD is abandoned and if that is the case, well, then you might want to do something about that. Besides, Varnish 3.0 being EOL doesn't instantly make it unusable. On Fri, Apr 24, 2015 at 3:57 AM, Jason Price wrote: > Well... this is quite a challenging statement. > > My challenge is this: The vmod community is awaiting features in 4.1 (not > released yet) before they can begin porting over to varnish 4.x. > > What is the correct approach to handle this situation? Varnish 4 is > 'incomplete' to the needs of these users, yet varnish3 is EOL. > > -Jason (if the required features to support vmod_redis and similar have > made it to 4.x, please correct my understanding) > > On Wed, Apr 15, 2015 at 4:23 AM, Andrzej Godziuk > wrote: > >> Hello, >> >> Regarding the Varnish 3 EOL announcement [1], I understand that Varnish >> Software will not release security patches to Varnish 3 any more. Is >> that correct? >> >> Do you plan on cooperating with LTS Linux distributions who shipped >> Varnish 3? For example, Ubuntu 12.04 is supported until April 2017 and >> I wonder how urgent the upgrade to Varnish 4 is on systems running this >> OS. >> >> [1] >> >> https://www.varnish-cache.org/lists/pipermail/varnish-announce/2015-April/000702.html >> >> -- >> Andrzej Godziuk >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- *Per Buer* CTO | Varnish Software AS Cell: +47 95839117 We Make Websites Fly! www.varnish-software.com [image: Register now] -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlos.abalde at gmail.com Fri Apr 24 12:44:19 2015 From: carlos.abalde at gmail.com (Carlos Abalde) Date: Fri, 24 Apr 2015 14:44:19 +0200 Subject: Varnish 3 EOL In-Reply-To: References: <20150415102352.1c9ae202@gdr-desktop.gdr.name> Message-ID: > On 24 Apr 2015, at 03:57, Jason Price wrote: > > Well... this is quite a challenging statement. > > My challenge is this: The vmod community is awaiting features in 4.1 (not released yet) before they can begin porting over to varnish 4.x. > > What is the correct approach to handle this situation? Varnish 4 is 'incomplete' to the needs of these users, yet varnish3 is EOL. > > -Jason (if the required features to support vmod_redis and similar have made it to 4.x, please correct my understanding) Hi Jason, I ported to V4 the Redis VMOD (https://github.com/carlosabalde/libvmod-redis) some months ago. That VMOD is similar to other existing options (e.g. https://github.com/zephirworks/libvmod-redis, https://github.com/brandonwamboldt/libvmod-redis, https://github.com/julp/libvmod-keystore, etc.) but fixing many small bugs, including a more flexible API and also including support for multiple servers, Redis Cluster, LUA scripting, connection pools, etc. Personally, I don?t think there are so many VMODs waiting for 4.1 to be migrated. In most cases the migration from V3 is pretty simple. Best, -- Carlos Abalde From japrice at gmail.com Mon Apr 27 15:50:23 2015 From: japrice at gmail.com (Jason Price) Date: Mon, 27 Apr 2015 11:50:23 -0400 Subject: Varnish using malloc, but still writing to disk? Message-ID: Varnish invocation: /usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -n vorigin -f /etc/varnish/vorigin.vcl -T 127.0.0.1:6082 -t 120 -w 2000,8000,300 -p thread_pools=2 -p thread_pool_workspace=100000 -p sess_workspace=1000000 -u varnish -g varnish -S /etc/varnish/secret -s malloc,90G We've had a production issue and I've been asserting that varnish doesn't write anything to disk, so the root volume IO spike we're seeing CAN'T be varnish. Of course, I look into iotop (specifically 'iotop -obPk') I see periodic times where the output is as follows: PID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND 6 be/4 root 0.00 K/s 0.00 K/s 0.00 % 33.26 % [kworker/u128:0] 4185 be/4 varnish 0.00 K/s 6673.66 K/s 0.00 % 0.00 % varnishd -P /var/run/varnish.pid -a :80 -n vorigin -f /etc/varnish/vorigin.vcl -T 127.0.0.1:6082 -t 120 -w 2000,8000,300 -p thread_pools=2 -p thread_pool_workspace=100000 -p sess_workspace=1000000 -u varnish -g varnish -S /etc/varnish/secret -s malloc,90G Total DISK READ: 0.00 K/s | Total DISK WRITE: 0.00 K/s PID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND 4185 be/4 varnish 0.00 K/s 12659.41 K/s 0.00 % 0.00 % varnishd -P /var/run/varnish.pid -a :80 -n vorigin -f /etc/varnish/vorigin.vcl -T 127.0.0.1:6082 -t 120 -w 2000,8000,300 -p thread_pools=2 -p thread_pool_workspace=100000 -p sess_workspace=1000000 -u varnish -g varnish -S /etc/varnish/secret -s malloc,90G I've dug into it with LSOF and sundry, and I can't figure out what/where varnish is doing the writing. It's not constant... 10-20 seconds will pass before I see it again for several seconds (2-8ish). NOTE: normal load for these boxes is 3-10k req/sec. Any thoughts? Why is varnish writing to disk (more to the point: writing tens megabytes at a time)? varnish version 3.0.5 Amazon Linux Kernel 3.14.35-28 Thank you for any help; -Jason From japrice at gmail.com Mon Apr 27 18:18:32 2015 From: japrice at gmail.com (Jason Price) Date: Mon, 27 Apr 2015 14:18:32 -0400 Subject: Varnish using malloc, but still writing to disk? In-Reply-To: References: Message-ID: Self reply: Does varnish rewrite the /var/lib/varnish//_.vsm file periodically? http://www.gossamer-threads.com/lists/varnish/misc/24678 Looks like the big control-knob for this is vm.dirty_expire_centisecs for modern linux kernels. Every interval, an async copy of the shared memory log goes to disk. I'll be evaluating just tweaking this, or moving it to a tmpfs volume. yay. -Jason On Mon, Apr 27, 2015 at 11:50 AM, Jason Price wrote: > Varnish invocation: > > /usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -n vorigin -f > /etc/varnish/vorigin.vcl -T 127.0.0.1:6082 -t 120 -w 2000,8000,300 -p > thread_pools=2 -p thread_pool_workspace=100000 -p > sess_workspace=1000000 -u varnish -g varnish -S /etc/varnish/secret -s > malloc,90G > > We've had a production issue and I've been asserting that varnish > doesn't write anything to disk, so the root volume IO spike we're > seeing CAN'T be varnish. > > Of course, I look into iotop (specifically 'iotop -obPk') I see > periodic times where the output is as follows: > > PID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND > 6 be/4 root 0.00 K/s 0.00 K/s 0.00 % 33.26 % [kworker/u128:0] > 4185 be/4 varnish 0.00 K/s 6673.66 K/s 0.00 % 0.00 % varnishd > -P /var/run/varnish.pid -a :80 -n vorigin -f /etc/varnish/vorigin.vcl > -T 127.0.0.1:6082 -t 120 -w 2000,8000,300 -p thread_pools=2 -p > thread_pool_workspace=100000 -p sess_workspace=1000000 -u varnish -g > varnish -S /etc/varnish/secret -s malloc,90G > Total DISK READ: 0.00 K/s | Total DISK WRITE: 0.00 K/s > PID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND > 4185 be/4 varnish 0.00 K/s 12659.41 K/s 0.00 % 0.00 % varnishd > -P /var/run/varnish.pid -a :80 -n vorigin -f /etc/varnish/vorigin.vcl > -T 127.0.0.1:6082 -t 120 -w 2000,8000,300 -p thread_pools=2 -p > thread_pool_workspace=100000 -p sess_workspace=1000000 -u varnish -g > varnish -S /etc/varnish/secret -s malloc,90G > > I've dug into it with LSOF and sundry, and I can't figure out > what/where varnish is doing the writing. It's not constant... 10-20 > seconds will pass before I see it again for several seconds (2-8ish). > > NOTE: normal load for these boxes is 3-10k req/sec. > > Any thoughts? Why is varnish writing to disk (more to the point: > writing tens megabytes at a time)? > > varnish version 3.0.5 > Amazon Linux > Kernel 3.14.35-28 > > Thank you for any help; > -Jason From jnerin at gmail.com Mon Apr 27 19:19:20 2015 From: jnerin at gmail.com (Jorge) Date: Mon, 27 Apr 2015 20:19:20 +0100 Subject: Varnish using malloc, but still writing to disk? In-Reply-To: References: Message-ID: Yes, if you want to avoid writes to disk, mount tmpfs over this directory, that's what I did. Cheers. On Mon, Apr 27, 2015 at 7:18 PM, Jason Price wrote: > Self reply: Does varnish rewrite the > /var/lib/varnish//_.vsm file periodically? > > > > http://www.gossamer-threads.com/lists/varnish/misc/24678 > > > > Looks like the big control-knob for this is vm.dirty_expire_centisecs > for modern linux kernels. Every interval, an async copy of the shared > memory log goes to disk. I'll be evaluating just tweaking this, or > moving it to a tmpfs volume. yay. > > -Jason > > On Mon, Apr 27, 2015 at 11:50 AM, Jason Price wrote: > > Varnish invocation: > > > > /usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -n vorigin -f > > /etc/varnish/vorigin.vcl -T 127.0.0.1:6082 -t 120 -w 2000,8000,300 -p > > thread_pools=2 -p thread_pool_workspace=100000 -p > > sess_workspace=1000000 -u varnish -g varnish -S /etc/varnish/secret -s > > malloc,90G > > > > We've had a production issue and I've been asserting that varnish > > doesn't write anything to disk, so the root volume IO spike we're > > seeing CAN'T be varnish. > > > > Of course, I look into iotop (specifically 'iotop -obPk') I see > > periodic times where the output is as follows: > > > > PID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND > > 6 be/4 root 0.00 K/s 0.00 K/s 0.00 % 33.26 % > [kworker/u128:0] > > 4185 be/4 varnish 0.00 K/s 6673.66 K/s 0.00 % 0.00 % varnishd > > -P /var/run/varnish.pid -a :80 -n vorigin -f /etc/varnish/vorigin.vcl > > -T 127.0.0.1:6082 -t 120 -w 2000,8000,300 -p thread_pools=2 -p > > thread_pool_workspace=100000 -p sess_workspace=1000000 -u varnish -g > > varnish -S /etc/varnish/secret -s malloc,90G > > Total DISK READ: 0.00 K/s | Total DISK WRITE: 0.00 K/s > > PID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND > > 4185 be/4 varnish 0.00 K/s 12659.41 K/s 0.00 % 0.00 % varnishd > > -P /var/run/varnish.pid -a :80 -n vorigin -f /etc/varnish/vorigin.vcl > > -T 127.0.0.1:6082 -t 120 -w 2000,8000,300 -p thread_pools=2 -p > > thread_pool_workspace=100000 -p sess_workspace=1000000 -u varnish -g > > varnish -S /etc/varnish/secret -s malloc,90G > > > > I've dug into it with LSOF and sundry, and I can't figure out > > what/where varnish is doing the writing. It's not constant... 10-20 > > seconds will pass before I see it again for several seconds (2-8ish). > > > > NOTE: normal load for these boxes is 3-10k req/sec. > > > > Any thoughts? Why is varnish writing to disk (more to the point: > > writing tens megabytes at a time)? > > > > varnish version 3.0.5 > > Amazon Linux > > Kernel 3.14.35-28 > > > > Thank you for any help; > > -Jason > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- Jorge Ner?n -------------- next part -------------- An HTML attachment was scrubbed... URL: