From bluethundr at gmail.com Mon Dec 1 04:44:30 2014 From: bluethundr at gmail.com (Tim Dunphy) Date: Sun, 30 Nov 2014 23:44:30 -0500 Subject: hit rate on a staging website Message-ID: Hey guys, We've setup some monitoring on a staging version of a production web site at work. It looks like we're getting about a .50 (or 50%) hit rate on the staging site. We haven't monitored the production site for varnish hit rates yet. We're using a Nagios check written in perl for that. But if it's possible to think of this in general terms, what I'd like to know is if you think that .50 is a respectable hit rate for a php/drupal site that's basically not in production, and the traffic it encounters is synthetic. Basically it's tested out by the developers and load tests are run against it using load generators as you might expect. And of course, some of the site content is dynamic and not meant to be cached. Here's how it's broken down: Cache_hit_percent=50.22 cache_hit 222628 cache_miss 220674 Here's we how calculate cache hit percent cache_hit_percent = ( cache_hit / ( cache_hit + cache_miss ) ) * 100 So all I want to know from the more experienced varnish guys is, would you consider this an acceptable cache hit rate given my situation? Thanks Tim -- GPG me!! gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.langhorn at digital.cabinet-office.gov.uk Mon Dec 1 07:19:06 2014 From: andrew.langhorn at digital.cabinet-office.gov.uk (Andrew Langhorn) Date: Mon, 1 Dec 2014 07:19:06 +0000 Subject: hit rate on a staging website In-Reply-To: References: Message-ID: I think that depends on all sorts of factors, especially including what your VCL says to Varnish to consider a hit, miss or pass. Let's assume you used defaults, though. Half of the content on your staging site is being returned by Varnish with no backend calls required. That seems a good starting point to me. If it's a development site, then things will be changing a lot all the time I assume and you won't want some stuff (like auth) cached. If it's an ecommerce site, you'll want even less cached - eg checkout etc. Instead of caring about your hit rate, I'd care more about your miss rate. These are the requests being sent to origin to respond via Varnish. And these are the requests that take up resources on your origin. Try to keep getting that lower, and you should be fine. By the way - have you tried varnishstat? It's a good way to get all these stats out of Varnish. On Monday, 1 December 2014, Tim Dunphy wrote: > Hey guys, > > We've setup some monitoring on a staging version of a production web site > at work. It looks like we're getting about a .50 (or 50%) hit rate on the > staging site. We haven't monitored the production site for varnish hit > rates yet. We're using a Nagios check written in perl for that. > > But if it's possible to think of this in general terms, what I'd like to > know is if you think that .50 is a respectable hit rate for a php/drupal > site that's basically not in production, and the traffic it encounters is > synthetic. Basically it's tested out by the developers and load tests are > run against it using load generators as you might expect. > > And of course, some of the site content is dynamic and not meant to be > cached. > > Here's how it's broken down: > > Cache_hit_percent=50.22 > > > cache_hit 222628 > > cache_miss 220674 > > > Here's we how calculate cache hit percent > > > cache_hit_percent = ( cache_hit / ( cache_hit + cache_miss ) ) * 100 > > > So all I want to know from the more experienced varnish guys is, would you > consider this an acceptable cache hit rate given my situation? > > > Thanks > > Tim > > -- > GPG me!! > > gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B > > -- -------------- next part -------------- An HTML attachment was scrubbed... URL: From marco at nucleus.it Mon Dec 1 10:08:48 2014 From: marco at nucleus.it (marco at nucleus.it) Date: Mon, 1 Dec 2014 11:08:48 +0100 Subject: Could not get storage Message-ID: <20141201110848.5517d976@lobo.lobo.dom> With varnish-3.0.5 i setup some files (like pdf) to not be saved to cache with hit_for_pass and serve them directly to the client. Sometimes i have "FetchError? c Could not get storage" error when Varnish can't free enought space from storage backend. The question is : Why Varnish try to save the object in the cache even if i set it with hit_for_pass to don't do that ? 17 Hash c /test.pdf 17 Hash c www.mytest.it 17 VCL_return c hash 17 VCL_call c pass pass 17 Backend c 30 mytest mytest 17 TTL c 830726718 RFC 0 -1 -1 1417425694 0 1417425693 280299600 0 17 VCL_call c fetch 17 TTL c 830726718 VCL 604800 -1 -1 1417425694 -0 17 TTL c 830726718 VCL 604800 86400 -1 1417425694 -0 17 VCL_return c hit_for_pass 17 ObjProtocol c HTTP/1.1 17 ObjResponse c OK 17 ObjHeader c Date: Mon, 01 Dec 2014 09:21:33 GMT 17 ObjHeader c Server: Apache/2.2.22 (Linux/SUSE) 17 ObjHeader c X-Powered-By: PHP/5.3.15 17 ObjHeader c Last-Modified: Mon, 01 Dec 2014 09:21:33 GMT 17 ObjHeader c ETag: "1417425693" 17 ObjHeader c Content-Length: 5866600 17 ObjHeader c Content-Type: application/pdf 17 FetchError c Could not get storage 17 VCL_call c error -- ---- Cordiali saluti Marco Felettigh Nucleus S.r.l. Tel. 0432526336 From jnerin at gmail.com Mon Dec 1 12:37:31 2014 From: jnerin at gmail.com (Jorge) Date: Mon, 1 Dec 2014 13:37:31 +0100 Subject: hit rate on a staging website In-Reply-To: References:

Message-ID: Just one minor nitpick, you have more "states" other than cache_hit & cache_miss, look at varnishstat and sum your numbers, for me it's: MAIN.client_req = MAIN.cache_hit + MAIN.cache_hitpass + MAIN.cache_miss + MAIN.s_synth I don't have s_pipe, so I'm not sure how/if they are counted as client_req/s_req. This should't alter your stats much (if any), but it's an easy thing to correct. Regards. El Mon Dec 01 2014 at 8:23:17, Andrew Langhorn (< andrew.langhorn at digital.cabinet-office.gov.uk>) escribi?: > I think that depends on all sorts of factors, especially including what > your VCL says to Varnish to consider a hit, miss or pass. > > Let's assume you used defaults, though. Half of the content on your > staging site is being returned by Varnish with no backend calls required. > That seems a good starting point to me. If it's a development site, then > things will be changing a lot all the time I assume and you won't want some > stuff (like auth) cached. If it's an ecommerce site, you'll want even less > cached - eg checkout etc. > > Instead of caring about your hit rate, I'd care more about your miss rate. > These are the requests being sent to origin to respond via Varnish. And > these are the requests that take up resources on your origin. Try to keep > getting that lower, and you should be fine. > > By the way - have you tried varnishstat? It's a good way to get all these > stats out of Varnish. > > > On Monday, 1 December 2014, Tim Dunphy wrote: > >> Hey guys, >> >> We've setup some monitoring on a staging version of a production web >> site at work. It looks like we're getting about a .50 (or 50%) hit rate on >> the staging site. We haven't monitored the production site for varnish hit >> rates yet. We're using a Nagios check written in perl for that. >> >> But if it's possible to think of this in general terms, what I'd like to >> know is if you think that .50 is a respectable hit rate for a php/drupal >> site that's basically not in production, and the traffic it encounters is >> synthetic. Basically it's tested out by the developers and load tests are >> run against it using load generators as you might expect. >> >> And of course, some of the site content is dynamic and not meant to be >> cached. >> >> Here's how it's broken down: >> >> Cache_hit_percent=50.22 >> >> >> cache_hit 222628 >> >> cache_miss 220674 >> >> >> Here's we how calculate cache hit percent >> >> >> cache_hit_percent = ( cache_hit / ( cache_hit + cache_miss ) ) * 100 >> >> >> So all I want to know from the more experienced varnish guys is, would >> you consider this an acceptable cache hit rate given my situation? >> >> >> Thanks >> >> Tim >> >> -- >> GPG me!! >> >> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B >> >> > > -- > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at jpluscplusm.com Thu Dec 4 23:23:08 2014 From: contact at jpluscplusm.com (Jonathan Matthews) Date: Thu, 4 Dec 2014 23:23:08 +0000 Subject: hit-for-pass ignores Vary header Message-ID: Hello all - I'm running with an (almost) vanilla default.vcl. My backends are returning cachable responses, with a single Vary: header - let's call it X-Auth. When a client requests an object which we already have cached, but with a different value in its X-Auth header, I quite rightly see a cache miss and a backend response. If this response is a 401 with no cache-control header (as it entirely valid in my setup), then this appears to force the previously cached response to be inaccessible - even to requests with its original X-Auth header set correctly. What I believe is happening is that the resulting hit-for-pass object that gets generated and cached seems to ignore the Vary header's presence. All subsequent requests for that Host & URI combination get hit-for-pass'ed, regardless of the previously valid object sitting in the cache for the Host & URI & Vary-header-pointer combination. More procedurally, I seem to be seeing this: ------------------------------------------- Client A: GET /resource Host: api X-Auth: Key1 Backend: 200 Here is some content #123 Vary: X-Auth Varnish: I'll cache "#123" with key(path:/resource & host:api & X-Auth:Key1) Client B Get /resource Host: api X-Auth: BadKey Backend 401 Wrong key, sorry Vary: X-Auth Varnish: 401 and non-cachable content observed I'll create a hit-for-pass for key path:/resource & host:api Client A: GET /resource Host: api X-Auth: Key1 Varnish: I have a hit-for-pass for path:/resource & host:api I won't check the cache for any matching content, but go straight to the backend ------------------------------------------- I'm wondering if there's a trick I'm missing to fix this behaviour? I've generally found the stock default.vcl to do very sane things, so I'm a little confused as to why this is the default behaviour. I (believe) I can fix this by hard-coding the X-Auth header as a component of the cache lookup key in vcl_hash(), but that would seem to negate the entire point of using the Vary header! [ I'm running 3.0.5 as it's the version shipped with the current Ubuntu LTS, and am not looking to upgrade as part of fixing this. Unless it turns out that my fix, above, doesn't work and there's no other way to do so :-) ] Many thanks for any help, Jonathan From lcruzero at gmail.com Fri Dec 5 17:09:09 2014 From: lcruzero at gmail.com (L Cruzero) Date: Fri, 5 Dec 2014 12:09:09 -0500 Subject: Backend response ESI Error handling Message-ID: HI, for starters i'm using this piece VLC code within my config to serve blank html non 200 or 304's. however, the intended goal is so that it is only applied to an ESI backend response. the names that we use on our ESI's URLS vary a lot. so i can't just use a RegEx to match a unique if conditional against obj.response ~ or req.url ~ respectively, i've inserted some bad ESI src tags into a few pages in hopes of finding a tag within varnishlog on which i could operate. though so far, no luck. sub vcl_fetch { set beresp.do_esi = true; if (beresp.status != 200 && beresp.status != 304 ) { error 200 "Not Found"; } } sub vcl_error { synthetic ""; set obj.status = 200; return (deliver); } any ideas and or thoughts would be appreciated. -LC From bluethundr at gmail.com Sat Dec 6 01:37:19 2014 From: bluethundr at gmail.com (Tim Dunphy) Date: Fri, 5 Dec 2014 20:37:19 -0500 Subject: just a simple thank you Message-ID: Guys, I've been on this list for a while now. And I've always been impressed with how eager, friendly and competent the help has been. I haven't always been the best at replying to advice I've been given here. And for that I do apologize. But I seem to be making some progress in learning how to use Varnish. And I absolutely could not have been able to get that start without the help I got from this list. So thank you! Especially to Thierry who actually taught me how to overcome my cache-busting problem of the site I was trying to cache producing cookie headers. He showed me the magic forumla to get past that. And now look at the cache hits I'm getting from the site: Hitrate ratio: 10 100 440 *Hitrate avg: 0.9999 0.9121 0.8825* Pretty nice, huh? If I read that correctly (and I'm not completely sure I am) I seem to be getting a 99% cache hit rate on this site? This is just for a demo site I put up with a couple of friends. But I do use Varnish very regularly at a major web company and the knowledge I gain there does translate directly to my work. So this is all great stuff. Thanks again varnish guys. You rock!! Tim -- GPG me!! gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B -------------- next part -------------- An HTML attachment was scrubbed... URL: From gkaragiannidis at dolnet.gr Tue Dec 9 15:33:33 2014 From: gkaragiannidis at dolnet.gr (=?iso-8859-7?B?w+nc7e3n8iDK4fHh4+nh7e3f5Ofy?=) Date: Tue, 9 Dec 2014 15:33:33 +0000 Subject: varnish crash Message-ID: Dear Varnish List, Today, we experienced a Varnish crash. Just look at the syslog file. We did, and we found the following error message: Dec 9 13:14:45 server1 varnishd[4347]: Child (22446) not responding to CLI, killing it. Dec 9 13:14:45 server1 varnishd[4347]: Child (22446) not responding to CLI, killing it. Dec 9 13:14:45 server1 varnishd[4347]: Child (22446) died signal=6 Dec 9 13:14:45 server1 varnishd[4347]: Child (22446) Panic message: Assert error in VRT_r_req_xid(), cache_vrt_var.c line 430:#012 Condition((p = WS_Alloc(sp->http->ws, size)) != 0) not true.#012thread = (cache-worker)#012ident = Linux,3.2.0-4-amd64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012 0x432215: pan_ic+0xd5#012 0x43f4f9: VRT_r_req_xid+0x109#012 0x7fe285462e1c: ./vcl.6EDYfe7E.so(VGC_function_vcl_error+0x813) [0x7fe285462e1c]#012 0x43986b: VCL_error_method+0x4b#012 0x41a260: CNT_Session+0x470#012 0x433f89: wrk_thread_real+0x489#012 0x7fe714a2ab50: /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7fe714a2ab50]#012 0x7fe7147750ed: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fe7147750ed]#012sp = 0x7fdc8f282008 {#012 fd = 4385, id = 4385, xid = 1916227992,#012 client = 66.249.78.214 64989,#012 step = STP_ERROR,#012 handling = deliver,#012 err_code = 503, err_reason = (null),#012 restarts = 4, esi_level = 0#012 flags = #012 bodystatus = 4#012 ws = 0x7fdc8f282080 { overflow#012 id = "sess",#012 {s,f,r,e} = {0x7fdc8f282c78,+65536,(nil),+65536},#012 },#012 http[req] = {#012 ws = 0x7fdc8f282080[sess]#012 "GET",#012 "/%CE%88%CF%84%CE%BF%CE%B9%CE%BC%CE%BF%CF%82%20%CE%BD%CE%B1%20%CF%83%CF%85%CE%BD%CE%B1%CE%BD%CF%84%CE%B7%CE%B8%CE%B5%CE%AF%20%CE%BC%CE%B5%20%CF%84%CE%BF%CE%BD%20%CF%80%CF%81%CF%8C%CE%B5%CE%B4%CF%81%CE%BF%20%CF%84%CE%BF%CF%85%20%CE%A3%CE%A5%CE%A1%CE%99%CE%96%CE%91,%20%CE%B1%CE%BB%CE%BB%CE%AC%20%CE%BA%CE%B1%CE%B9%20%CE%BC%CE%B5%20%CF%84%CE%BF%CF%85%CF%82%20%CE%B5%CF%80%CE%B9%CE%BA%CE%B5%CF%86%CE%B1%CE%BB%CE%AE%CF%82%20%CF%8C%CE%BB%CF%89%CE%BD%20%CF%84%CF%89%CE%BD%20%CF%83%CF%85%CE%BB%CE%BB%CE%BF%CE%B3%CE%B9%CE%BA%CE%BF%CF%84%CE%AE%CF%84%CF%89%CE%BD%20%CE%BC%CE%B5%20%CF%84%CE%B9%CF%82%20%CE%BF%CF%80%CE%BF%CE%AF%CE%B5%CF%82%20%CE%B7%20%CE%94%CE%97%CE%9C%CE%91%CE%A1%20%CE%B5%CE%AF%CE%BD%CE%B1%CE%B9%20%CF%83%CE%B5%20%CF%83%CF%85%CE%B6%CE%B7%CF%84%CE%AE%CF%83%CE%B5%CE%B9%CF%82%20%CE%B4%CE%B7%CE%BB%CF%8E%CE%BD%CE%B5%CE%B9%20%CE%BF%20%CE%A6%CF%8E%CF%84%CE% Dec 9 13:14:45 server1 varnishd[4347]: Child cleanup complete Dec 9 13:14:45 server1 varnishd[4347]: child (13526) Started Dec 9 13:14:45 server1 varnishd[4347]: Child (13526) said Child starts We are running varnishd (varnish-3.0.6 revision 1899836) Varnish is started with the following settings: #!/bin/sh echo starting varnish daemon ulimit -n 131072 ulimit -l unlimited /path/to/varnishd \ -s malloc,48G \ -a 0.0.0.0:80 \ -T 127.0.0.1:7777 \ -p thread_pools=8 \ -p thread_pool_min=100 \ -p thread_pool_max=4000 \ -f /path/to/server.vcl echo done Any ideas and or thoughts would be appreciated. Regards Yiannis -------------- next part -------------- An HTML attachment was scrubbed... URL: From thierry.magnien at sfr.com Tue Dec 9 15:44:06 2014 From: thierry.magnien at sfr.com (MAGNIEN, Thierry) Date: Tue, 9 Dec 2014 15:44:06 +0000 Subject: varnish crash In-Reply-To: References: Message-ID: <5D103CE839D50E4CBC62C9FD7B83287CC9CAE4E8@EXCN015.encara.local.ads> Hi, It seems your workspace got exhausted due to the length of the URL your received. Best regards, Thierry De : varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org [mailto:varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org] De la part de G?????? ?a?a??a???d?? Envoy? : mardi 9 d?cembre 2014 16:34 ? : varnish-misc at varnish-cache.org Objet : varnish crash Dear Varnish List, Today, we experienced a Varnish crash. Just look at the syslog file. We did, and we found the following error message: Dec 9 13:14:45 server1 varnishd[4347]: Child (22446) not responding to CLI, killing it. Dec 9 13:14:45 server1 varnishd[4347]: Child (22446) not responding to CLI, killing it. Dec 9 13:14:45 server1 varnishd[4347]: Child (22446) died signal=6 Dec 9 13:14:45 server1 varnishd[4347]: Child (22446) Panic message: Assert error in VRT_r_req_xid(), cache_vrt_var.c line 430:#012 Condition((p = WS_Alloc(sp->http->ws, size)) != 0) not true.#012thread = (cache-worker)#012ident = Linux,3.2.0-4-amd64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012 0x432215: pan_ic+0xd5#012 0x43f4f9: VRT_r_req_xid+0x109#012 0x7fe285462e1c: ./vcl.6EDYfe7E.so(VGC_function_vcl_error+0x813) [0x7fe285462e1c]#012 0x43986b: VCL_error_method+0x4b#012 0x41a260: CNT_Session+0x470#012 0x433f89: wrk_thread_real+0x489#012 0x7fe714a2ab50: /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7fe714a2ab50]#012 0x7fe7147750ed: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fe7147750ed]#012sp = 0x7fdc8f282008 {#012 fd = 4385, id = 4385, xid = 1916227992,#012 client = 66.249.78.214 64989,#012 step = STP_ERROR,#012 handling = deliver,#012 err_code = 503, err_reason = (null),#012 restarts = 4, esi_level = 0#012 flags = #012 bodystatus = 4#012 ws = 0x7fdc8f282080 { overflow#012 id = "sess",#012 {s,f,r,e} = {0x7fdc8f282c78,+65536,(nil),+65536},#012 },#012 http[req] = {#012 ws = 0x7fdc8f282080[sess]#012 "GET",#012 "/%CE%88%CF%84%CE%BF%CE%B9%CE%BC%CE%BF%CF%82%20%CE%BD%CE%B1%20%CF%83%CF%85%CE%BD%CE%B1%CE%BD%CF%84%CE%B7%CE%B8%CE%B5%CE%AF%20%CE%BC%CE%B5%20%CF%84%CE%BF%CE%BD%20%CF%80%CF%81%CF%8C%CE%B5%CE%B4%CF%81%CE%BF%20%CF%84%CE%BF%CF%85%20%CE%A3%CE%A5%CE%A1%CE%99%CE%96%CE%91,%20%CE%B1%CE%BB%CE%BB%CE%AC%20%CE%BA%CE%B1%CE%B9%20%CE%BC%CE%B5%20%CF%84%CE%BF%CF%85%CF%82%20%CE%B5%CF%80%CE%B9%CE%BA%CE%B5%CF%86%CE%B1%CE%BB%CE%AE%CF%82%20%CF%8C%CE%BB%CF%89%CE%BD%20%CF%84%CF%89%CE%BD%20%CF%83%CF%85%CE%BB%CE%BB%CE%BF%CE%B3%CE%B9%CE%BA%CE%BF%CF%84%CE%AE%CF%84%CF%89%CE%BD%20%CE%BC%CE%B5%20%CF%84%CE%B9%CF%82%20%CE%BF%CF%80%CE%BF%CE%AF%CE%B5%CF%82%20%CE%B7%20%CE%94%CE%97%CE%9C%CE%91%CE%A1%20%CE%B5%CE%AF%CE%BD%CE%B1%CE%B9%20%CF%83%CE%B5%20%CF%83%CF%85%CE%B6%CE%B7%CF%84%CE%AE%CF%83%CE%B5%CE%B9%CF%82%20%CE%B4%CE%B7%CE%BB%CF%8E%CE%BD%CE%B5%CE%B9%20%CE%BF%20%CE%A6%CF%8E%CF%84%CE% Dec 9 13:14:45 server1 varnishd[4347]: Child cleanup complete Dec 9 13:14:45 server1 varnishd[4347]: child (13526) Started Dec 9 13:14:45 server1 varnishd[4347]: Child (13526) said Child starts We are running varnishd (varnish-3.0.6 revision 1899836) Varnish is started with the following settings: #!/bin/sh echo starting varnish daemon ulimit -n 131072 ulimit -l unlimited /path/to/varnishd \ -s malloc,48G \ -a 0.0.0.0:80 \ -T 127.0.0.1:7777 \ -p thread_pools=8 \ -p thread_pool_min=100 \ -p thread_pool_max=4000 \ -f /path/to/server.vcl echo done Any ideas and or thoughts would be appreciated. Regards Yiannis -------------- next part -------------- An HTML attachment was scrubbed... URL: From mattias at nucleus.be Tue Dec 9 16:00:00 2014 From: mattias at nucleus.be (Mattias Geniar) Date: Tue, 9 Dec 2014 16:00:00 +0000 Subject: varnish crash In-Reply-To: <5D103CE839D50E4CBC62C9FD7B83287CC9CAE4E8@EXCN015.encara.local.ads> References: <5D103CE839D50E4CBC62C9FD7B83287CC9CAE4E8@EXCN015.encara.local.ads> Message-ID: > It seems your workspace got exhausted due to the length of the URL your >received. To fix this, add the "http_req_hdr_len" parameter to your Varnish startup; -p http_req_hdr_len=8192 Increase if needed. Mattias From nesta at iceburg.net Tue Dec 9 19:11:27 2014 From: nesta at iceburg.net (Brice Burgess) Date: Tue, 09 Dec 2014 14:11:27 -0500 Subject: varnish crash In-Reply-To: References: <5D103CE839D50E4CBC62C9FD7B83287CC9CAE4E8@EXCN015.encara.local.ads> Message-ID: <5487495F.4050807@iceburg.net> On 12/09/2014 11:00 AM, Mattias Geniar wrote: >> It seems your workspace got exhausted due to the length of the URL your >> received. > > > To fix this, add the "http_req_hdr_len" parameter to your Varnish startup; > > -p http_req_hdr_len=8192 > > Increase if needed. Yowser! So you can crash Varnish by sending extremely large request URLs? I hope this is not the case in 4.0. Failing gracefully would be nice. From perbu at varnish-software.com Wed Dec 10 07:45:56 2014 From: perbu at varnish-software.com (Per Buer) Date: Wed, 10 Dec 2014 08:45:56 +0100 Subject: varnish crash In-Reply-To: <5487495F.4050807@iceburg.net> References: <5D103CE839D50E4CBC62C9FD7B83287CC9CAE4E8@EXCN015.encara.local.ads> <5487495F.4050807@iceburg.net> Message-ID: Hi, On Tue, Dec 9, 2014 at 8:11 PM, Brice Burgess wrote: > On 12/09/2014 11:00 AM, Mattias Geniar wrote: > >> It seems your workspace got exhausted due to the length of the URL your >>> received. >>> >> >> To fix this, add the "http_req_hdr_len" parameter to your Varnish startup; >> >> -p http_req_hdr_len=8192 >> >> Increase if needed. >> > > Yowser! So you can crash Varnish by sending extremely large request URLs? > I hope this is not the case in 4.0. Failing gracefully would be nice. Of course not. That would have been a rather obvious DoS and would have been posted to the various sec-lists in a few minutes. However, you might exhaust your workspace if you start copying the long URL around a lot or otherwise process it. That is what it seems is happening here. -- *Per Buer* CTO | Varnish Software AS Cell: +47 95839117 We Make Websites Fly! www.varnish-software.com [image: Register now] -------------- next part -------------- An HTML attachment was scrubbed... URL: From grantksupport at operamail.com Wed Dec 10 22:17:56 2014 From: grantksupport at operamail.com (grantksupport at operamail.com) Date: Wed, 10 Dec 2014 14:17:56 -0800 Subject: new Varnish4 install -- seeing MISSes instead of HITs ? vcl & log included Message-ID: <1418249876.530624.201411121.3C685A07@webmail.messagingengine.com> I have Varnish4 in-between an nginx SSL-terminator, and an apache backend, nginx -> varnish -> apache Loading a simple image https://DEV.mydomain.com/test.png I expect to (a) see it, and (b) have it cached/served by Varnish. The image IS served through the chain, but I see a MISS in 'varnishlog'; question is -- why not a HIT? Here's the varnishlog output, * << BeReq >> 753667 - Begin bereq 753666 pass - Timestamp Start: 1418248243.885700 0.000000 0.000000 - BereqMethod GET - BereqURL /test.png - BereqProtocol HTTP/1.0 - BereqHeader Host: DEV.mydomain.com - BereqHeader X-Real-IP: xxxx:xxx:xxx:xxx::7 - BereqHeader X-Client-Verify: NONE - BereqHeader X-SSL-Client-Verify: NONE - BereqHeader X-Forwarded-Proto: https - BereqHeader HTTPS: on - BereqHeader user-agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:24.0) Gecko/20100101 Firefox/24.0 - BereqHeader accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 - BereqHeader accept-language: en-US,en;q=0.5 - BereqHeader dnt: 1 - BereqHeader X-Forwarded-For: xxxx:xxx:xxx:xxx::7, 127.0.0.1, 127.0.0.1 - BereqHeader X-Varnish: 753667 - VCL_call BACKEND_FETCH - VCL_return fetch - BackendOpen 22 img_DEV(127.0.0.1,,9003) 127.0.0.1 53820 - Backend 22 img_DEV img_DEV(127.0.0.1,,9003) - Timestamp Bereq: 1418248243.885876 0.000176 0.000176 - Timestamp Beresp: 1418248243.886222 0.000522 0.000346 - BerespProtocol HTTP/1.1 - BerespStatus 200 - BerespReason OK - BerespHeader Date: Wed, 10 Dec 2014 21:50:43 GMT - BerespHeader Server: Apache - BerespHeader Last-Modified: Fri, 20 Mar 2009 02:05:12 GMT - BerespHeader ETag: "f80f71-1179-4658358d02600" - BerespHeader Accept-Ranges: bytes - BerespHeader Content-Length: 4473 - BerespHeader Cache-Control: max-age=1209600 - BerespHeader Expires: Wed, 24 Dec 2014 21:50:43 GMT - BerespHeader Connection: close - BerespHeader Content-Type: image/png - TTL RFC 1209600 -1 -1 1418248244 1418248244 1418248243 1419457843 1209600 - VCL_call BACKEND_RESPONSE - TTL VCL -1 3600 0 1418248244 - BerespHeader x-host: DEV.mydomain.com - BerespHeader x-url: /test.png - TTL VCL 120 3600 0 1418248244 - VCL_return deliver - Storage malloc Transient - ObjProtocol HTTP/1.1 - ObjStatus 200 - ObjReason OK - ObjHeader Date: Wed, 10 Dec 2014 21:50:43 GMT - ObjHeader Server: Apache - ObjHeader Last-Modified: Fri, 20 Mar 2009 02:05:12 GMT - ObjHeader ETag: "f80f71-1179-4658358d02600" - ObjHeader Accept-Ranges: bytes - ObjHeader Content-Length: 4473 - ObjHeader Cache-Control: max-age=1209600 - ObjHeader Expires: Wed, 24 Dec 2014 21:50:43 GMT - ObjHeader Content-Type: image/png - ObjHeader x-host: DEV.mydomain.com - ObjHeader x-url: /test.png - Fetch_Body 3 length stream - Timestamp BerespBody: 1418248243.886310 0.000609 0.000088 - BackendClose 22 img_DEV(127.0.0.1,,9003) - Length 4473 - BereqAcct 455 0 455 313 4473 4786 - End * << Request >> 753666 - Begin req 753665 rxreq - Timestamp Start: 1418248243.885608 0.000000 0.000000 - Timestamp Req: 1418248243.885608 0.000000 0.000000 - ReqStart 127.0.0.1 34117 - ReqMethod GET - ReqURL /test.png - ReqProtocol HTTP/1.0 - ReqHeader Host: DEV.mydomain.com - ReqHeader X-Real-IP: xxxx:xxx:xxx:xxx::7 - ReqHeader X-Forwarded-For: xxxx:xxx:xxx:xxx::7 - ReqHeader X-Client-Verify: NONE - ReqHeader X-SSL-Client-Verify: NONE - ReqHeader X-Forwarded-Proto: https - ReqHeader HTTPS: on - ReqHeader Connection: close - ReqHeader user-agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:24.0) Gecko/20100101 Firefox/24.0 - ReqHeader accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 - ReqHeader accept-language: en-US,en;q=0.5 - ReqHeader dnt: 1 - ReqUnset X-Forwarded-For: xxxx:xxx:xxx:xxx::7 - ReqHeader X-Forwarded-For: xxxx:xxx:xxx:xxx::7, 127.0.0.1 - VCL_call RECV - ReqUnset X-Forwarded-For: xxxx:xxx:xxx:xxx::7, 127.0.0.1 - ReqHeader X-Forwarded-For: xxxx:xxx:xxx:xxx::7, 127.0.0.1, 127.0.0.1 - VCL_return hash - VCL_call HASH - VCL_return lookup - Hit 2147483651 - VCL_call HIT - VCL_return fetch - VCL_Error vcl_hit{} returns fetch without busy object. Doing pass. - VCL_call PASS - VCL_return fetch - Link bereq 753667 pass - Timestamp Fetch: 1418248243.886311 0.000703 0.000703 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Date: Wed, 10 Dec 2014 21:50:43 GMT - RespHeader Server: Apache - RespHeader Last-Modified: Fri, 20 Mar 2009 02:05:12 GMT - RespHeader ETag: "f80f71-1179-4658358d02600" - RespHeader Accept-Ranges: bytes - RespHeader Content-Length: 4473 - RespHeader Cache-Control: max-age=1209600 - RespHeader Expires: Wed, 24 Dec 2014 21:50:43 GMT - RespHeader Content-Type: image/png - RespHeader x-host: DEV.mydomain.com - RespHeader x-url: /test.png - RespHeader X-Varnish: 753666 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish-v4 - VCL_call DELIVER - RespHeader X-Varnish-Cache: MISS - RespUnset Server: Apache - RespUnset x-host: DEV.mydomain.com - RespUnset x-url: /test.png - VCL_return deliver - Timestamp Process: 1418248243.886335 0.000727 0.000024 - Debug "RES_MODE 2" - RespHeader Connection: close - RespHeader Accept-Ranges: bytes - Timestamp Resp: 1418248243.886391 0.000784 0.000056 - Debug "XXX REF 1" - ReqAcct 433 0 433 390 4473 4863 - End * << Session >> 753665 - Begin sess 0 HTTP/1 - SessOpen 127.0.0.1 34117 127.0.0.1:6081 127.0.0.1 6081 1418248243.885560 16 - Link req 753666 rxreq - SessClose RESP_CLOSE 0.001 - End * << BeReq >> 786435 - Begin bereq 786434 pass - Timestamp Start: 1418248244.021572 0.000000 0.000000 - BereqMethod GET - BereqURL /test.png - BereqProtocol HTTP/1.0 - BereqHeader Host: DEV.mydomain.com - BereqHeader X-Real-IP: xxxx:xxx:xxx:xxx::7 - BereqHeader X-Client-Verify: NONE - BereqHeader X-SSL-Client-Verify: NONE - BereqHeader X-Forwarded-Proto: https - BereqHeader HTTPS: on - BereqHeader user-agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:24.0) Gecko/20100101 Firefox/24.0 - BereqHeader accept: image/png,image/*;q=0.8,*/*;q=0.5 - BereqHeader accept-language: en-US,en;q=0.5 - BereqHeader dnt: 1 - BereqHeader X-Forwarded-For: xxxx:xxx:xxx:xxx::7, 127.0.0.1, 127.0.0.1 - BereqHeader X-Varnish: 786435 - VCL_call BACKEND_FETCH - VCL_return fetch - BackendOpen 22 img_DEV(127.0.0.1,,9003) 127.0.0.1 53822 - Backend 22 img_DEV img_DEV(127.0.0.1,,9003) - Timestamp Bereq: 1418248244.021719 0.000147 0.000147 - Timestamp Beresp: 1418248244.021968 0.000396 0.000249 - BerespProtocol HTTP/1.1 - BerespStatus 200 - BerespReason OK - BerespHeader Date: Wed, 10 Dec 2014 21:50:44 GMT - BerespHeader Server: Apache - BerespHeader Last-Modified: Fri, 20 Mar 2009 02:05:12 GMT - BerespHeader ETag: "f80f71-1179-4658358d02600" - BerespHeader Accept-Ranges: bytes - BerespHeader Content-Length: 4473 - BerespHeader Cache-Control: max-age=1209600 - BerespHeader Expires: Wed, 24 Dec 2014 21:50:44 GMT - BerespHeader Connection: close - BerespHeader Content-Type: image/png - TTL RFC 1209600 -1 -1 1418248244 1418248244 1418248244 1419457844 1209600 - VCL_call BACKEND_RESPONSE - TTL VCL -1 3600 0 1418248244 - BerespHeader x-host: DEV.mydomain.com - BerespHeader x-url: /test.png - TTL VCL 120 3600 0 1418248244 - VCL_return deliver - Storage malloc Transient - ObjProtocol HTTP/1.1 - ObjStatus 200 - ObjReason OK - ObjHeader Date: Wed, 10 Dec 2014 21:50:44 GMT - ObjHeader Server: Apache - ObjHeader Last-Modified: Fri, 20 Mar 2009 02:05:12 GMT - ObjHeader ETag: "f80f71-1179-4658358d02600" - ObjHeader Accept-Ranges: bytes - ObjHeader Content-Length: 4473 - ObjHeader Cache-Control: max-age=1209600 - ObjHeader Expires: Wed, 24 Dec 2014 21:50:44 GMT - ObjHeader Content-Type: image/png - ObjHeader x-host: DEV.mydomain.com - ObjHeader x-url: /test.png - Fetch_Body 3 length stream - Timestamp BerespBody: 1418248244.022039 0.000467 0.000070 - BackendClose 22 img_DEV(127.0.0.1,,9003) - Length 4473 - BereqAcct 425 0 425 313 4473 4786 - End * << Request >> 786434 - Begin req 786433 rxreq - Timestamp Start: 1418248244.021493 0.000000 0.000000 - Timestamp Req: 1418248244.021493 0.000000 0.000000 - ReqStart 127.0.0.1 34119 - ReqMethod GET - ReqURL /test.png - ReqProtocol HTTP/1.0 - ReqHeader Host: DEV.mydomain.com - ReqHeader X-Real-IP: xxxx:xxx:xxx:xxx::7 - ReqHeader X-Forwarded-For: xxxx:xxx:xxx:xxx::7 - ReqHeader X-Client-Verify: NONE - ReqHeader X-SSL-Client-Verify: NONE - ReqHeader X-Forwarded-Proto: https - ReqHeader HTTPS: on - ReqHeader Connection: close - ReqHeader user-agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:24.0) Gecko/20100101 Firefox/24.0 - ReqHeader accept: image/png,image/*;q=0.8,*/*;q=0.5 - ReqHeader accept-language: en-US,en;q=0.5 - ReqHeader dnt: 1 - ReqUnset X-Forwarded-For: xxxx:xxx:xxx:xxx::7 - ReqHeader X-Forwarded-For: xxxx:xxx:xxx:xxx::7, 127.0.0.1 - VCL_call RECV - ReqUnset X-Forwarded-For: xxxx:xxx:xxx:xxx::7, 127.0.0.1 - ReqHeader X-Forwarded-For: xxxx:xxx:xxx:xxx::7, 127.0.0.1, 127.0.0.1 - VCL_return hash - VCL_call HASH - VCL_return lookup - Hit 2147483651 - VCL_call HIT - VCL_return fetch - VCL_Error vcl_hit{} returns fetch without busy object. Doing pass. - VCL_call PASS - VCL_return fetch - Link bereq 786435 pass - Timestamp Fetch: 1418248244.022041 0.000549 0.000549 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Date: Wed, 10 Dec 2014 21:50:44 GMT - RespHeader Server: Apache - RespHeader Last-Modified: Fri, 20 Mar 2009 02:05:12 GMT - RespHeader ETag: "f80f71-1179-4658358d02600" - RespHeader Accept-Ranges: bytes - RespHeader Content-Length: 4473 - RespHeader Cache-Control: max-age=1209600 - RespHeader Expires: Wed, 24 Dec 2014 21:50:44 GMT - RespHeader Content-Type: image/png - RespHeader x-host: DEV.mydomain.com - RespHeader x-url: /test.png - RespHeader X-Varnish: 786434 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish-v4 - VCL_call DELIVER - RespHeader X-Varnish-Cache: MISS - RespUnset Server: Apache - RespUnset x-host: DEV.mydomain.com - RespUnset x-url: /test.png - VCL_return deliver - Timestamp Process: 1418248244.022060 0.000568 0.000019 - Debug "RES_MODE 2" - RespHeader Connection: close - RespHeader Accept-Ranges: bytes - Timestamp Resp: 1418248244.022119 0.000626 0.000058 - Debug "XXX REF 1" - ReqAcct 403 0 403 390 4473 4863 - End * << Session >> 786433 - Begin sess 0 HTTP/1 - SessOpen 127.0.0.1 34119 127.0.0.1:6081 127.0.0.1 6081 1418248244.021453 21 - Link req 786434 rxreq - SessClose RESP_CLOSE 0.001 - End here's the VCL in use, ========================================================= vcl 4.0; import std; acl internal { "127.0.0.1"/32; "::1"/128; "10.15.13.0"/24; "xxxx:xxx:xxx:xxx::7"/64; } acl purge { "localhost"; "127.0.0.1"/24; "::1"/128; "10.15.13.0"/24; "xxxx:xxx:xxx:xxx::7"/64; } backend def_DEV { .port = "9000"; .host = "127.0.0.1"; } backend css_DEV { .port = "9001"; .host = "127.0.0.1"; } backend js_DEV { .port = "9002"; .host = "127.0.0.1"; } backend img_DEV { .port = "9003"; .host = "127.0.0.1"; } backend def_FALLBACK { .port = "9999"; .host = "127.0.0.1"; } sub hostNorm { if (req.http.host == "www.dev.mydomain.com" ) { set req.http.host = "dev.mydomain.com"; } } sub backAssign { if (req.http.host == "dev.mydomain.com") { if (req.url ~ "\.(css|css?.*|woff|eot|svg|ttf|otf)$") { set req.backend_hint = css_DEV; } elseif (req.url ~ "\.(js|js?.*)$" ) { set req.backend_hint = js_DEV; } elseif (req.url ~ "\.(jp(e?)g|gif|png|ti(f?)f)$" ) { set req.backend_hint = img_DEV; } else { set req.backend_hint = def_DEV; } } else { set req.backend_hint = def_FALLBACK; } } sub vcl_recv { call hostNorm; call backAssign; if (req.method == "PURGE") { if (!client.ip ~ purge) { return (synth(405, "Not allowed.")); } ban("obj.http.x-url ~ " + req.url); return (purge); } if (req.restarts == 0) { if (req.http.x-forwarded-for) { set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + regsub(client.ip, ":.*", ""); } else { set req.http.X-Forwarded-For = regsub(client.ip, ":.*", ""); } } if (req.http.Accept-Encoding) { if (req.url ~ "\.(gif|jpg|jpeg|swf|flv|mp3|mp4|ogg|pdf|ico|png|gz|tgz|bz2|xz)(\?.*|)$") { unset req.http.Accept-Encoding; } else if (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip"; } else if (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate"; } else { unset req.http.Accept-Encoding; } } if (req.url ~ "(?i)\.(pdf|asc|dat|txt|doc|xls|ppt|tgz|csv|png|gif|jpeg|jpg|ico|swf|css|js)(\?[a-z0-9]+)?$") { unset req.http.Cookie; } if (req.http.Cookie) { set req.http.Cookie = ";" + req.http.Cookie; set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";"); set req.http.Cookie = regsuball(req.http.Cookie, ";(SESS[a-z0-9]+|NO_CACHE)=", "; \1="); set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", ""); if (req.http.Cookie == "") { unset req.http.Cookie; } else { return (pass); } } } sub vcl_hit { return(fetch); } sub vcl_miss { return(fetch); } sub vcl_deliver { if (obj.hits > 0) { set resp.http.X-Varnish-Cache = "HIT"; set resp.http.X-Cache-Hits = obj.hits; } else { set resp.http.X-Varnish-Cache = "MISS"; } unset resp.http.x-host; unset resp.http.x-url; } ========================================================= From contact at jpluscplusm.com Thu Dec 11 21:00:40 2014 From: contact at jpluscplusm.com (Jonathan Matthews) Date: Thu, 11 Dec 2014 21:00:40 +0000 Subject: hit-for-pass ignores Vary header In-Reply-To: References: Message-ID: On 4 Dec 2014 23:23, "Jonathan Matthews" wrote: [snip] Any ideas anyone? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi.boukelmoune at zenika.com Thu Dec 11 22:36:56 2014 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Thu, 11 Dec 2014 23:36:56 +0100 Subject: new Varnish4 install -- seeing MISSes instead of HITs ? vcl & log included In-Reply-To: <1418249876.530624.201411121.3C685A07@webmail.messagingengine.com> References: <1418249876.530624.201411121.3C685A07@webmail.messagingengine.com> Message-ID: Hi, My guess is both a mistake from you and a bug in varnish. In vcl_hit you should "return(deliver)" instead of "return(fetch)". Looking at the source code, I don't understand what the hit->fetch transition means, and according to the documentation it shouldn't even be possible: https://www.varnish-cache.org/docs/4.0/users-guide/vcl-built-in-subs.html#vcl-hit And yet this transition is explicitly mentioned in your logs and in the VCL state machine code. Unfortunately I don't have time to look more into it. Dridi From grantksupport at operamail.com Thu Dec 11 22:48:12 2014 From: grantksupport at operamail.com (grantksupport at operamail.com) Date: Thu, 11 Dec 2014 14:48:12 -0800 Subject: new Varnish4 install -- seeing MISSes instead of HITs ? vcl & log included In-Reply-To: References: <1418249876.530624.201411121.3C685A07@webmail.messagingengine.com> Message-ID: <1418338092.965752.201879805.77882E6C@webmail.messagingengine.com> On Thu, Dec 11, 2014, at 02:36 PM, Dridi Boukelmoune wrote: > My guess is both a mistake from you and a bug in varnish. > > In vcl_hit you should "return(deliver)" instead of "return(fetch)". i'm working with shameless cribbing from supsected-to-work-well vcl's this one https://github.com/mattiasgeniar/varnish-4.0-configuration-templates/blob/master/default.vcl does include the vcl_hit() { ... - return (deliver) + return (fetch) } Making JUST that change in MY, current vcl still makes no difference; I still see the MISS. BUT, switching completely to that vcl DOES do the trick -- I see HIT on the image. Now, a matter of piecemeal substitution to find & understand the actual problem. From dridi.boukelmoune at zenika.com Thu Dec 11 23:12:17 2014 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Fri, 12 Dec 2014 00:12:17 +0100 Subject: new Varnish4 install -- seeing MISSes instead of HITs ? vcl & log included In-Reply-To: <1418338092.965752.201879805.77882E6C@webmail.messagingengine.com> References: <1418249876.530624.201411121.3C685A07@webmail.messagingengine.com> <1418338092.965752.201879805.77882E6C@webmail.messagingengine.com> Message-ID: Ok, You just got me curious in the middle of the night, and after I looked at your link I conveniently remembered seeing something about this before. https://www.varnish-cache.org/docs/4.0/users-guide/vcl-grace.html#grace-mode Looking again at your logs after reading this, it makes sense that you'd always miss. What happens if you remove vcl_hit from your VCL and entirely leave it to the built-in vcl_hit? Dridi On Thu, Dec 11, 2014 at 11:48 PM, wrote: > > > On Thu, Dec 11, 2014, at 02:36 PM, Dridi Boukelmoune wrote: >> My guess is both a mistake from you and a bug in varnish. >> >> In vcl_hit you should "return(deliver)" instead of "return(fetch)". > > i'm working with shameless cribbing from supsected-to-work-well vcl's > > this one > > https://github.com/mattiasgeniar/varnish-4.0-configuration-templates/blob/master/default.vcl > > does include the > > vcl_hit() { > ... > - return (deliver) > + return (fetch) > } > > Making JUST that change in MY, current vcl still makes no difference; I still see the MISS. > > BUT, switching completely to that vcl DOES do the trick -- I see HIT on the image. > > Now, a matter of piecemeal substitution to find & understand the actual problem. > From grantksupport at operamail.com Thu Dec 11 23:26:14 2014 From: grantksupport at operamail.com (grantksupport at operamail.com) Date: Thu, 11 Dec 2014 15:26:14 -0800 Subject: new Varnish4 install -- seeing MISSes instead of HITs ? vcl & log included In-Reply-To: References: <1418249876.530624.201411121.3C685A07@webmail.messagingengine.com> <1418338092.965752.201879805.77882E6C@webmail.messagingengine.com> Message-ID: <1418340374.975373.201892129.14B4005C@webmail.messagingengine.com> On Thu, Dec 11, 2014, at 03:12 PM, Dridi Boukelmoune wrote: > https://www.varnish-cache.org/docs/4.0/users-guide/vcl-grace.html#grace-mode > > Looking again at your logs after reading this, it makes sense that > you'd always miss. hm. I need to read that a couple of more times; for now I'll take your word for it > What happens if you remove vcl_hit from your VCL > and entirely leave it to the built-in vcl_hit? it HITs. as does the vlc_hit from https://github.com/mattiasgeniar/varnish-4.0-configuration-templates/blob/master/default.vcl which appears to use the recommendations from your reference, above good starting point -- at least, far better than my own attempt. now I've an idea what the problem is, and a working basis; re-reading will fix the rest. thanks! From numard at gmail.com Fri Dec 12 07:46:28 2014 From: numard at gmail.com (Norberto Meijome) Date: Fri, 12 Dec 2014 18:46:28 +1100 Subject: new Varnish4 install -- seeing MISSes instead of HITs ? vcl & log included In-Reply-To: <1418340374.975373.201892129.14B4005C@webmail.messagingengine.com> References: <1418249876.530624.201411121.3C685A07@webmail.messagingengine.com> <1418338092.965752.201879805.77882E6C@webmail.messagingengine.com> <1418340374.975373.201892129.14B4005C@webmail.messagingengine.com> Message-ID: Not near a computer atm...but this is interesting in your log - VCL_return hash - VCL_call HASH - VCL_return lookup - Hit 2147483651 - VCL_call HIT - VCL_return fetch - VCL_Error vcl_hit{} returns fetch without busy object. Doing pass. - VCL_call PASS - VCL_return fetch On 12/12/2014 10:27 am, wrote: > > > On Thu, Dec 11, 2014, at 03:12 PM, Dridi Boukelmoune wrote: > > > https://www.varnish-cache.org/docs/4.0/users-guide/vcl-grace.html#grace-mode > > > > Looking again at your logs after reading this, it makes sense that > > you'd always miss. > > hm. I need to read that a couple of more times; for now I'll take your > word for it > > > What happens if you remove vcl_hit from your VCL > > and entirely leave it to the built-in vcl_hit? > > it HITs. > > as does the vlc_hit from > > > https://github.com/mattiasgeniar/varnish-4.0-configuration-templates/blob/master/default.vcl > > which appears to use the recommendations from your reference, above > > good starting point -- at least, far better than my own attempt. > > now I've an idea what the problem is, and a working basis; re-reading will > fix the rest. > > thanks! > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From YARIV-H at yit.co.il Fri Dec 12 11:27:43 2014 From: YARIV-H at yit.co.il (=?windows-1255?B?6fjp4SDk+ODs?=) Date: Fri, 12 Dec 2014 11:27:43 +0000 Subject: Varnish panics Message-ID: <1DC6764DE551494993B3C6C09A6045F301EF6963@mailbox10.yedioth.co.il> Hi , Varnish version 3.0.3-1 Once in a few minutes varnish panics with : varnishd[25412]: Child (4070) Panic message: Assert error in http_Write(), cache_http.c line 1082:#012 Condition(( hp->hd[HTTP_HDR_STATUS].b) != 0) not true.#012thread = (cache-worker)#012ident = Linux,2.6.32-431.29.2.el6.x86_64,x86_64,-sfile,-smalloc,-hcritbi t,epoll#012Backtrace:#012 0x42dc76: /usr/sbin/varnishd() [0x42dc76]#012 0x428ac3: /usr/sbin/varnishd(http_Write+0x293) [0x428ac3]#012 0x430e0d : /usr/sbin/varnishd(RES_WriteObj+0x33d) [0x430e0d]#012 0x41673b: /usr/sbin/varnishd() [0x41673b]#012 0x418837: /usr/sbin/varnishd(CNT_Session+ 0x7a7) [0x418837]#012 0x42f706: /usr/sbin/varnishd() [0x42f706]#012 0x7f2e4027b9d1: /lib64/libpthread.so.0(+0x79d1) [0x7f2e4027b9d1]#012 0x7f2 e3ffc886d: /lib64/libc.so.6(clone+0x6d) [0x7f2e3ffc886d]#012sp = 0x7f2cae26b008 {#012 fd = 434, id = 434, xid = 161457694,#012 client = 2.22.50 .93 62809,#012 step = STP_DELIVER,#012 handling = deliver,#012 err_code = 200, err_reason = (null),#012 restarts = 0, esi_level = 0#012 flag s = is_gunzip#012 bodystatus = 4#012 ws = 0x7f2cae26b080 { #012 id = "sess",#012 {s,f,r,e} = {0x7f2cae26d5f8,+712,(nil),+131072},#012 } ,#012 http[req] = {#012 ws = 0x7f2cae26b080[sess]#012 "GET",#012 "/articles/1,7340,L-3092999,00.html",#012 "HTTP/1.1",#012 "Pragma: no-cache",#012 "Accept: */*",#012 "From: bingbot(at)microsoft.com",#012 "User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)",#012 "X-Akamai-CONFIG-LOG-DETAIL: true",#012 "TE: chunked;q=1.0",#012 "Connection: TE",#012 "Akamai-Origin-Hop: 2",#012 "Via: 1.1 v1-akamaitech.net(ghost) (AkamaiGHost), 1.1 akamai.net(ghost) (AkamaiGHost)",#012 "Host: www.ynet.co.il",#012 "Cache-Control: no-cache, max-age=106",#012 "Connection: keep-alive",#012 "Cookie: akamai=true;akamai=true",#012 "X-Forwarded-For: 157.55.39.10, 184.27.178.151, 2.22.50.93",#012 },#012 worker = 0x7f2cf19fca90 {#012 ws = 0x7f2cf19fccc8 { overflow#012 id = "wrk",#012 Thanks in advance [Banner] Powered by U-BTech XTRABANNER -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Banner1-06b8.jpg Type: image/jpeg Size: 52763 bytes Desc: Banner1-06b8.jpg URL: From jnerin at gmail.com Fri Dec 12 12:00:31 2014 From: jnerin at gmail.com (Jorge) Date: Fri, 12 Dec 2014 13:00:31 +0100 Subject: new Varnish4 install -- seeing MISSes instead of HITs ? vcl & log included In-Reply-To: <1418338092.965752.201879805.77882E6C@webmail.messagingengine.com> References: <1418249876.530624.201411121.3C685A07@webmail.messagingengine.com> <1418338092.965752.201879805.77882E6C@webmail.messagingengine.com> Message-ID: Uhm, doing the change so your vcl_hit looks like: sub vcl_hit { return(deliver); } Works for me (tm). Note that in your first email your vcl_hit unconditionally returned fetch: sub vcl_hit { return(fetch); } P.D. I was the one doing the last update to this github template, and the idea of returning fetch in vcl_hit was exactly for grace support. On Thu, Dec 11, 2014 at 11:48 PM, wrote: > > > > On Thu, Dec 11, 2014, at 02:36 PM, Dridi Boukelmoune wrote: > > My guess is both a mistake from you and a bug in varnish. > > > > In vcl_hit you should "return(deliver)" instead of "return(fetch)". > > i'm working with shameless cribbing from supsected-to-work-well vcl's > > this one > > > https://github.com/mattiasgeniar/varnish-4.0-configuration-templates/blob/master/default.vcl > > does include the > > vcl_hit() { > ... > - return (deliver) > + return (fetch) > } > > Making JUST that change in MY, current vcl still makes no difference; I > still see the MISS. > > BUT, switching completely to that vcl DOES do the trick -- I see HIT on > the image. > > Now, a matter of piecemeal substitution to find & understand the actual > problem. > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- Jorge Ner?n -------------- next part -------------- An HTML attachment was scrubbed... URL: From YARIV-H at yit.co.il Fri Dec 12 12:03:10 2014 From: YARIV-H at yit.co.il (=?utf-8?B?15nXqNeZ15Eg15TXqNeQ15w=?=) Date: Fri, 12 Dec 2014 12:03:10 +0000 Subject: Varnish panics In-Reply-To: <1DC6764DE551494993B3C6C09A6045F301EF6963@mailbox10.yedioth.co.il> References: <1DC6764DE551494993B3C6C09A6045F301EF6963@mailbox10.yedioth.co.il> Message-ID: <1DC6764DE551494993B3C6C09A6045F301EF6DFF@mailbox10.yedioth.co.il> Start up params : DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \ -f ${VARNISH_VCL_CONF} \ -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \ -t ${VARNISH_TTL} \ -w ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT} \ -u varnish -g varnish \ -S ${VARNISH_SECRET_FILE} \ -s ${VARNISH_STORAGE} \ -p sess_timeout=305\ -p http_req_hdr_len=16384 \ -p http_req_size=65536 \ -p http_resp_hdr_len=16384 \ -p http_resp_size=65536 \ -p http_gzip_support=off \ -p pipe_timeout=120 \ -p sess_workspace=131072 \ -p http_max_hdr=256" The strange thing is that this is only happening on 1 varnish server of 3 Maybe some related to sess_workspace and http headers ? the web site has some vary unusual cookies and other headers regarding size Thanks From: varnish-misc-bounces+yariv-h=yit.co.il at varnish-cache.org [mailto:varnish-misc-bounces+yariv-h=yit.co.il at varnish-cache.org] On Behalf Of ???? ???? Sent: Friday, December 12, 2014 1:28 PM To: varnish-misc at varnish-cache.org Subject: Varnish panics Hi , Varnish version 3.0.3-1 Once in a few minutes varnish panics with : varnishd[25412]: Child (4070) Panic message: Assert error in http_Write(), cache_http.c line 1082:#012 Condition(( hp->hd[HTTP_HDR_STATUS].b) != 0) not true.#012thread = (cache-worker)#012ident = Linux,2.6.32-431.29.2.el6.x86_64,x86_64,-sfile,-smalloc,-hcritbi t,epoll#012Backtrace:#012 0x42dc76: /usr/sbin/varnishd() [0x42dc76]#012 0x428ac3: /usr/sbin/varnishd(http_Write+0x293) [0x428ac3]#012 0x430e0d : /usr/sbin/varnishd(RES_WriteObj+0x33d) [0x430e0d]#012 0x41673b: /usr/sbin/varnishd() [0x41673b]#012 0x418837: /usr/sbin/varnishd(CNT_Session+ 0x7a7) [0x418837]#012 0x42f706: /usr/sbin/varnishd() [0x42f706]#012 0x7f2e4027b9d1: /lib64/libpthread.so.0(+0x79d1) [0x7f2e4027b9d1]#012 0x7f2 e3ffc886d: /lib64/libc.so.6(clone+0x6d) [0x7f2e3ffc886d]#012sp = 0x7f2cae26b008 {#012 fd = 434, id = 434, xid = 161457694,#012 client = 2.22.50 .93 62809,#012 step = STP_DELIVER,#012 handling = deliver,#012 err_code = 200, err_reason = (null),#012 restarts = 0, esi_level = 0#012 flag s = is_gunzip#012 bodystatus = 4#012 ws = 0x7f2cae26b080 { #012 id = "sess",#012 {s,f,r,e} = {0x7f2cae26d5f8,+712,(nil),+131072},#012 } ,#012 http[req] = {#012 ws = 0x7f2cae26b080[sess]#012 "GET",#012 "/articles/1,7340,L-3092999,00.html",#012 "HTTP/1.1",#012 "Pragma: no-cache",#012 "Accept: */*",#012 "From: bingbot(at)microsoft.com",#012 "User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)",#012 "X-Akamai-CONFIG-LOG-DETAIL: true",#012 "TE: chunked;q=1.0",#012 "Connection: TE",#012 "Akamai-Origin-Hop: 2",#012 "Via: 1.1 v1-akamaitech.net(ghost) (AkamaiGHost), 1.1 akamai.net(ghost) (AkamaiGHost)",#012 "Host: www.ynet.co.il",#012 "Cache-Control: no-cache, max-age=106",#012 "Connection: keep-alive",#012 "Cookie: akamai=true;akamai=true",#012 "X-Forwarded-For: 157.55.39.10, 184.27.178.151, 2.22.50.93",#012 },#012 worker = 0x7f2cf19fca90 {#012 ws = 0x7f2cf19fccc8 { overflow#012 id = "wrk",#012 Thanks in advance [Banner] Powered by U?BTech XTRABANNER -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 52763 bytes Desc: image001.jpg URL: From jbrown at uber.com Sat Dec 13 20:13:08 2014 From: jbrown at uber.com (James Brown) Date: Sat, 13 Dec 2014 12:13:08 -0800 Subject: binding to a domain socket? Message-ID: We have varnish used for on-box caching in some systems and are running out of ephemeral sockets having it listen on a TCP/IP socket ; it would be super-nice if Varnish supported binding do an AF_UNIX/SOCK_STREAM socket. Does anyone know if there are any patches out there to allow VSS_parse() to pull out a domain socket from some form of config? If I were to write one, do y'all think it would be of interest to anyone else? -- James Brown Staff Infrastructure, Security, and Database Engineer At Large Uber Technologies, Inc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi.boukelmoune at zenika.com Sun Dec 14 12:40:14 2014 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Sun, 14 Dec 2014 13:40:14 +0100 Subject: binding to a domain socket? In-Reply-To: References: Message-ID: Hi James, It has already been discussed and it *is* of interest to others! https://www.varnish-cache.org/trac/wiki/Future_Feature#UseUNIXsocketsforbackend Cheers, Dridi On Sat, Dec 13, 2014 at 9:13 PM, James Brown wrote: > We have varnish used for on-box caching in some systems and are running out > of ephemeral sockets having it listen on a TCP/IP socket ; it would be > super-nice if Varnish supported binding do an AF_UNIX/SOCK_STREAM socket. > Does anyone know if there are any patches out there to allow VSS_parse() to > pull out a domain socket from some form of config? If I were to write one, > do y'all think it would be of interest to anyone else? > > -- > James Brown > Staff Infrastructure, Security, and Database Engineer At Large > Uber Technologies, Inc. > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From stephan008 at gmx.com Mon Dec 15 10:48:15 2014 From: stephan008 at gmx.com (Stephan Alz) Date: Mon, 15 Dec 2014 11:48:15 +0100 Subject: Varnish4 + Symfony2 is not caching Message-ID: Hello Folks I have a 4.0.2-1~wheezy + Symfony2 on Apache setup on 2 different linux hosts. Could anybody tell me what is wrong with the following configuration: http://pastebin.com/DzFrQijG When I try the url with curl I get: < HTTP/1.1 200 OK < Date: Mon, 15 Dec 2014 10:46:31 GMT < Server: Apache/2.2.22 (Debian) < Vary: Host,Accept-Encoding < Set-Cookie: PHPSESSID=0hel5b796j5eddr0qumq6ie344; path=/ < Cache-Control: no-cache < Set-Cookie: hl=fr; expires=Tue, 15-Dec-2015 10:46:31 GMT; path=/ < Content-Type: text/html; charset=UTF-8 < X-Varnish: 32776 < Age: 0 < Via: 1.1 varnish-v4 < X-Cache: MISS < X-Cache-Hits: 0 < Transfer-Encoding: chunked < Accept-Ranges: bytes X-Cache is always miss for the pages, which means Varnish does not do it's job. Thank you, Stephan From thierry.magnien at sfr.com Mon Dec 15 11:04:57 2014 From: thierry.magnien at sfr.com (MAGNIEN, Thierry) Date: Mon, 15 Dec 2014 11:04:57 +0000 Subject: Varnish4 + Symfony2 is not caching In-Reply-To: References: Message-ID: <5D103CE839D50E4CBC62C9FD7B83287CC9CB449B@EXCN015.encara.local.ads> Hi Stephan, With such headers, Varnish will only do what it has to : < Set-Cookie: PHPSESSID=0hel5b796j5eddr0qumq6ie344; path=/ < Cache-Control: no-cache < Set-Cookie: hl=fr; expires=Tue, 15-Dec-2015 10:46:31 GMT; path=/ Not only Varnish won't cache responses with set-cookie headers, unless explicitly told to in VCL, but the Cache-Control header tells Varnish not to cache... Regards, Thierry -----Message d'origine----- De?: varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org [mailto:varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org] De la part de Stephan Alz Envoy??: lundi 15 d?cembre 2014 11:48 ??: varnish-misc at varnish-cache.org Objet?: Varnish4 + Symfony2 is not caching Hello Folks I have a 4.0.2-1~wheezy + Symfony2 on Apache setup on 2 different linux hosts. Could anybody tell me what is wrong with the following configuration: http://pastebin.com/DzFrQijG When I try the url with curl I get: < HTTP/1.1 200 OK < Date: Mon, 15 Dec 2014 10:46:31 GMT < Server: Apache/2.2.22 (Debian) < Vary: Host,Accept-Encoding < Set-Cookie: PHPSESSID=0hel5b796j5eddr0qumq6ie344; path=/ < Cache-Control: no-cache < Set-Cookie: hl=fr; expires=Tue, 15-Dec-2015 10:46:31 GMT; path=/ < Content-Type: text/html; charset=UTF-8 < X-Varnish: 32776 < Age: 0 < Via: 1.1 varnish-v4 < X-Cache: MISS < X-Cache-Hits: 0 < Transfer-Encoding: chunked < Accept-Ranges: bytes X-Cache is always miss for the pages, which means Varnish does not do it's job. Thank you, Stephan _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From dridi.boukelmoune at zenika.com Mon Dec 15 15:08:52 2014 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Mon, 15 Dec 2014 16:08:52 +0100 Subject: Varnish4 + Symfony2 is not caching In-Reply-To: <5D103CE839D50E4CBC62C9FD7B83287CC9CB449B@EXCN015.encara.local.ads> References: <5D103CE839D50E4CBC62C9FD7B83287CC9CB449B@EXCN015.encara.local.ads> Message-ID: Does Varnish 4 honor the Cache-Control's "no-cache"? IIRC Varnish 3 will cache such responses unless you choose not to. I'm not including Set-Cookie headers when I say "such responses". Dridi On Mon, Dec 15, 2014 at 12:04 PM, MAGNIEN, Thierry wrote: > Hi Stephan, > > With such headers, Varnish will only do what it has to : > > < Set-Cookie: PHPSESSID=0hel5b796j5eddr0qumq6ie344; path=/ > < Cache-Control: no-cache > < Set-Cookie: hl=fr; expires=Tue, 15-Dec-2015 10:46:31 GMT; path=/ > > Not only Varnish won't cache responses with set-cookie headers, unless explicitly told to in VCL, but the Cache-Control header tells Varnish not to cache... > > Regards, > Thierry > > -----Message d'origine----- > De : varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org [mailto:varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org] De la part de Stephan Alz > Envoy? : lundi 15 d?cembre 2014 11:48 > ? : varnish-misc at varnish-cache.org > Objet : Varnish4 + Symfony2 is not caching > > Hello Folks > > I have a 4.0.2-1~wheezy + Symfony2 on Apache setup on 2 different linux hosts. > > Could anybody tell me what is wrong with the following configuration: > > http://pastebin.com/DzFrQijG > > When I try the url with curl I get: > > < HTTP/1.1 200 OK > < Date: Mon, 15 Dec 2014 10:46:31 GMT > < Server: Apache/2.2.22 (Debian) > < Vary: Host,Accept-Encoding > < Set-Cookie: PHPSESSID=0hel5b796j5eddr0qumq6ie344; path=/ > < Cache-Control: no-cache > < Set-Cookie: hl=fr; expires=Tue, 15-Dec-2015 10:46:31 GMT; path=/ > < Content-Type: text/html; charset=UTF-8 > < X-Varnish: 32776 > < Age: 0 > < Via: 1.1 varnish-v4 > < X-Cache: MISS > < X-Cache-Hits: 0 > < Transfer-Encoding: chunked > < Accept-Ranges: bytes > > X-Cache is always miss for the pages, which means Varnish does not do it's job. > > Thank you, > > Stephan > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From grantksupport at operamail.com Tue Dec 16 03:01:16 2014 From: grantksupport at operamail.com (grantksupport at operamail.com) Date: Mon, 15 Dec 2014 19:01:16 -0800 Subject: optimizing a multi-tier webserver arch: improving on nginx/SSL -> varnish -> apache/php-fpm CDN ? Message-ID: <1418698876.3081237.203338629.7F735F4C@webmail.messagingengine.com> I'm consolidating servers. I have yest another server with Varnish4 in-between an nginx SSL-terminator, and an apache/php-fpm backend. I'm looking for some experienced comments on reorganizing them -- still around Varnish -- for best performance, and lowest resource. Digging through the server, the backend was (supposedly) setup for use as a self-hosted CDN (to facilitate parallel request processing) / reverse proxy cache, specifically in Drupal. I'm moving away from Drupal to Symfony/Laravel sites. Still PHP, but, obviously, nothing Drupal-specific. Making some efforts to reevaluate the server stack for performance, and simple common sense. Frankly, I'm not convinced that what follows is optimal, is at all functional. My goal is to migrate if/as necessary to a well-justified, robust stack. Currently, in the VCL, backend X_def { .port = "10000"; ... } backend X_css { .port = "10001"; ... } backend X_js { .port = "10002"; ... } backend X_img { .port = "10003"; ... } sub backend_assign { if (req.url ~ "\.(css|css?.*)$") { set req.backend_hint = X_css; } elseif (req.url ~ "\.(js|js?.*)$" ) { set req.backend_hint = X_js; } elseif (req.url ~ "\.(jp(e?)g|gif|png|ti(f?)f)$" ) { set req.backend_hint = X_img; } else { set req.backend_hint = X_def; } } sub vcl_recv { call backend_assign; ... Which, IIUC, results in nginx:443 -> varnish:8000 -- default --> httpd/php-fpm:10000 |--------- css --> httpd/php-fpm:10001 |--------- js --> httpd/php-fpm:10002 |--------- img --> httpd/php-fpm:10003 This allows *varnish* to make parallel requests to the backend server. But the 'net <--> nginx <--> varnish stream is not parallel. There are a variety of possible options for streamlining the above -- - swapping nginx in for the httpd instances - putting some parallelization in-FRONT of varnish - service some static assets from the front-side nginx+cache only (no varnish + httpd) - etc The current arch seems to work well enough, but my gut tells me it's unncessarily heavy/fat. As yet, I've not compared to any of the aforementioned alternatives. We definitely want to continue to make best use of varnish's scaling/flexibility in the mix. To that end, I'd appreciate any insight/comments on the best arch to plug 'around' varnish. Thanks, Grant From viktor.villafuerte at optusnet.com.au Tue Dec 16 06:56:45 2014 From: viktor.villafuerte at optusnet.com.au (Viktor Villafuerte) Date: Tue, 16 Dec 2014 17:56:45 +1100 Subject: Content-Length vs Transfer-Encoding Message-ID: <20141216065645.GB1828@optusnet.com.au> Hi all, I'd like to ask about bug report https://www.varnish-cache.org/trac/ticket/1506 I can see that the patch is not included in the latest 4.0.2 Varnish release. I assume it will be included in next (4.0.3?) but would I be able to get any info when would that be? Also in my case doing set beresp.do_esi = true; seems to fix the problem. Is that a good workaround considering that I'm not doing any ESI stuff? Is there anything that should be included with this, or avoided? Thanks for any answers.. :) v From grantksupport at operamail.com Tue Dec 16 17:58:52 2014 From: grantksupport at operamail.com (grantksupport at operamail.com) Date: Tue, 16 Dec 2014 09:58:52 -0800 Subject: caching anon users', & auth users' ESI, content specifically in Laravel PHP framework apps? Message-ID: <1418752732.3380593.203620329.6E0D755D@webmail.messagingengine.com> I'm trying to get varnish to cooperate with (a new-installed) laravel php framework's sesion caching (http://laravel.io/forum/12-16-2014-laravel-redis-session-storage-varnish-reverse-proxy-cache-how-to-preventstrip-cookies-to-enable-varnish-caching) Currently, when using redis for session storage, laravel sets a COOKIE -- for both authenticated & anonymous users Cookie: laravel_session=eyJ...UifQ%3D%3D For anon users, and for ESI blocks with auth'd users, Varnish should cache. What's the correct method for ensuring Varnish caches in these cases, when using a Laravel app? Can/should it be handled only in Varnish VCL -- stripping the cookie? If so, how's the ID of auth vs anon cookie best handled? or, must in be done in concert with, or solely within, the Laravel app? From spaceman at antispaceman.com Tue Dec 16 20:08:16 2014 From: spaceman at antispaceman.com (spaceman) Date: Tue, 16 Dec 2014 20:08:16 +0000 Subject: caching anon users', & auth users' ESI, content specifically in Laravel PHP framework apps? In-Reply-To: <1418752732.3380593.203620329.6E0D755D@webmail.messagingengine.com> References: <1418752732.3380593.203620329.6E0D755D@webmail.messagingengine.com> Message-ID: <20141216200816.GA23484@home> If I were you I would put a filter on the ESI routes that need to be cached that prevents the cookie like this: Route::filter('nocookie', function() { Config::set('session.driver', 'array'); }); The session driver array doesn't set cookies. However if there are client side cookies then you will need to strip them using Varnish. I don't know if this is correct way but I do it like: if ( req.http.host ~ "^(.*\.)?hostname(\..*)?$" ) { if ( req.url ~ "^/path/" ) { unset req.http.Cookie; } } in vcl_recv. This should make the pages cachable by Varnish. You should also set the cache control headers using this: $response->setCache(array('last_modified' => new \DateTime(), 'max_age' => 300 )); This can be done in a filter or on the controller, it will need to be a proper laravel response object. On 2014/12/16 09:12:52 , grantksupport at operamail.com wrote: >I'm trying to get varnish to cooperate with (a new-installed) laravel php framework's sesion caching (http://laravel.io/forum/12-16-2014-laravel-redis-session-storage-varnish-reverse-proxy-cache-how-to-preventstrip-cookies-to-enable-varnish-caching) > >Currently, when using redis for session storage, laravel sets a COOKIE -- for both authenticated & anonymous users > > Cookie: laravel_session=eyJ...UifQ%3D%3D > >For anon users, and for ESI blocks with auth'd users, Varnish should cache. > >What's the correct method for ensuring Varnish caches in these cases, when using a Laravel app? > >Can/should it be handled only in Varnish VCL -- stripping the cookie? If so, how's the ID of auth vs anon cookie best handled? > >or, must in be done in concert with, or solely within, the Laravel app? > >_______________________________________________ >varnish-misc mailing list >varnish-misc at varnish-cache.org >https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From grantksupport at operamail.com Tue Dec 16 20:25:15 2014 From: grantksupport at operamail.com (grantksupport at operamail.com) Date: Tue, 16 Dec 2014 12:25:15 -0800 Subject: caching anon users', & auth users' ESI, content specifically in Laravel PHP framework apps? In-Reply-To: <20141216200816.GA23484@home> References: <1418752732.3380593.203620329.6E0D755D@webmail.messagingengine.com> <20141216200816.GA23484@home> Message-ID: <1418761515.3416445.203683669.13223A4F@webmail.messagingengine.com> I didn't get to ESI yet. but, what appears to be a clean fix is in laravel/app/config/app.php 'driver' => 'redis', ( in my case; other options are available ) @ laravel site top-dir, install & enable the "Session Monster" service provider http://packalyst.com/packages/package/haifanghui/session-monster That will set a X-No-Session: yeah header when NOT authenticated Then, in Varnish VCL strip cookies if the header's detected ... sub vcl_backend_response { ... if (beresp.http.X-No-Session ~ "yeah") { unset beresp.http.set-cookie; } ... ... sub vcl_deliver { ... + unset resp.X-No-Session; // not required, but cleaner ... Now on serving anon pages, there's no cookie, and Varnish caches properly From hernan at cmsmedios.com Wed Dec 17 21:07:49 2014 From: hernan at cmsmedios.com (=?UTF-8?Q?Hern=C3=A1n_Marsili?=) Date: Wed, 17 Dec 2014 18:07:49 -0300 Subject: session id problem Message-ID: Hi, We have a simple problem, every time we ask for a page, the sessionid change. Is this a normal behavior with Varnish? and how can we solved it? Any help will be much appreciate! Saludos, Hern?n. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pprocacci at datapipe.com Wed Dec 17 21:11:34 2014 From: pprocacci at datapipe.com (Paul Procacci) Date: Wed, 17 Dec 2014 16:11:34 -0500 Subject: session id problem In-Reply-To: References: Message-ID: <5491F186.20207@datapipe.com> On 12/17/2014 4:07 PM, Hern?n Marsili wrote: > Hi, > > We have a simple problem, every time we ask for a page, the sessionid > change. Is this a normal behavior with Varnish? and how can we solved it? > Stop stripping cookies on pages that require dynamic content? -------------- next part -------------- An HTML attachment was scrubbed... URL: From infos at opendoc.net Wed Dec 31 07:29:05 2014 From: infos at opendoc.net (Alexandre) Date: Wed, 31 Dec 2014 08:29:05 +0100 Subject: How stop DDOS with Varnish ? Message-ID: <54A3A5C1.7000905@opendoc.net> Hello everyone, I search a documentation to stop HTTP flood DDoS attacks with varnish. Do you have any recommendation ? Thank you. Alex. From behzadaltaf at gmail.com Wed Dec 31 07:48:54 2014 From: behzadaltaf at gmail.com (Behzad Altaf) Date: Wed, 31 Dec 2014 13:18:54 +0530 Subject: Varnish Clustering Message-ID: Hi All, Would anyone please help me with the exact steps that are needed for putting varnish boxes into clustering and self replication mode. I have two boxes on which Varnish is running with a backend webserver. I looked at the following but it does not have the steps https://www.varnish-software.com/blog/introducing-varnish-high-availability-2 Thanks and Regards, BA -------------- next part -------------- An HTML attachment was scrubbed... URL: From tobias.eichelbroenner at lamp-solutions.de Wed Dec 31 08:12:32 2014 From: tobias.eichelbroenner at lamp-solutions.de (=?windows-1252?Q?Tobias_Eichelbr=F6nner?=) Date: Wed, 31 Dec 2014 09:12:32 +0100 Subject: How stop DDOS with Varnish ? In-Reply-To: <54A3A5C1.7000905@opendoc.net> References: <54A3A5C1.7000905@opendoc.net> Message-ID: <54A3AFF0.40006@lamp-solutions.de> Hi Alex, > I search a documentation to stop HTTP flood DDoS attacks with varnish. > Do you have any recommendation ? you can take a look at: https://www.varnish-cache.org/vmod/throttle I don?t think you can find a general documentation dealing with DDos. It heavily depends on how the attack looks like and what you want to achieve. We had some very simple but large DDos attacks we simply could deal with using some regexes in VCL sending out an error page instead of directing to the backend. In other cases you might be able to force a 100% hitrate in varnish, protecting your backend and then set up a large number of varnishes to throw out gigibytes of http traffic per second just to sit out the attack. Sincerely, Tobias -- LAMP solutions GmbH Gostenhofer Hauptstrasse 35 90443 Nuernberg Amtsgericht Nuernberg: HRB 22366 Geschaeftsfuehrer: Heiko Schubert Es gelten unsere allgemeinen Geschaeftsbedingungen. http://www.lamp-solutions.de/agbs/ Telefon : 0911 / 376 516 0 Fax : 0911 / 376 516 11 E-Mail : support at lamp-solutions.de Web : www.lamp-solutions.de Facebook : http://www.facebook.com/LAMPsolutions Twitter : http://twitter.com/#!/lampsolutions From infos at opendoc.net Wed Dec 31 08:17:55 2014 From: infos at opendoc.net (Alexandre) Date: Wed, 31 Dec 2014 09:17:55 +0100 Subject: How stop DDOS with Varnish ? In-Reply-To: <54A3AFF0.40006@lamp-solutions.de> References: <54A3A5C1.7000905@opendoc.net> <54A3AFF0.40006@lamp-solutions.de> Message-ID: <54A3B133.1080105@opendoc.net> Thank you very much Tobias, I'll try that now. On 31/12/14 09:12, Tobias Eichelbr?nner wrote: > Hi Alex, > >> I search a documentation to stop HTTP flood DDoS attacks with varnish. >> Do you have any recommendation ? > > you can take a look at: > https://www.varnish-cache.org/vmod/throttle > > I don?t think you can find a general documentation dealing with DDos. > It heavily depends on how the attack looks like and what you want to > achieve. > > We had some very simple but large DDos attacks we simply could deal with > using some regexes in VCL sending out an error page instead of directing > to the backend. > > In other cases you might be able to force a 100% hitrate in varnish, > protecting your backend and then set up a large number of varnishes to > throw out gigibytes of http traffic per second just to sit out the attack. > > Sincerely, > > Tobias > > From rainer at ultra-secure.de Wed Dec 31 16:58:57 2014 From: rainer at ultra-secure.de (Rainer Duffner) Date: Wed, 31 Dec 2014 17:58:57 +0100 Subject: Varnish Clustering In-Reply-To: References: Message-ID: <3EE228D3-4022-4F4D-B642-7F614015F924@ultra-secure.de> > Am 31.12.2014 um 08:48 schrieb Behzad Altaf : > > Hi All, > > Would anyone please help me with the exact steps that are needed for putting varnish boxes into clustering and self replication mode. > > I have two boxes on which Varnish is running with a backend webserver. > > I looked at the following but it does not have the steps > > https://www.varnish-software.com/blog/introducing-varnish-high-availability-2 > This is for the commercial, subscription-only varnish plus (and then you probably need the Gold Subscription). OSS varnish does not have a cache-replication feature. Can?t you just load-balance between the two varnish-instances? Both will warm-up slower, but at least you?d have two warm caches. Of course, your application should then do cache-invalidation on both instances - which is usually the trickier part and probably one reason why people fork-over the money for the subscription... -------------- next part -------------- An HTML attachment was scrubbed... URL: