From travis.crowder at spechal.com Tue May 1 23:46:50 2012 From: travis.crowder at spechal.com (Travis Crowder) Date: Tue, 01 May 2012 18:46:50 -0500 Subject: Cache page with dynamic query string Message-ID: <4FA075EA.4000008@spechal.com> I am pretty sure the answer to this is: It's not possible; but I am hoping to learn something new. There is a page, let's call it promo.php. We tell a bunch of people about it and append their email address to the page for tracking ... i.e promo.php?foo at bar.baz I believe in other to cache the page, I need to strip the query string (since it will be unique). Is there a way for Varnish to cache this page, yet still track the hits? i.e. Strip the query string for the lookup and add it back on delivery. Thanks, Travis From hugues.alary at gmail.com Tue May 1 23:50:30 2012 From: hugues.alary at gmail.com (Hugues Alary) Date: Tue, 1 May 2012 16:50:30 -0700 Subject: Cache page with dynamic query string In-Reply-To: <4FA075EA.4000008@spechal.com> References: <4FA075EA.4000008@spechal.com> Message-ID: If the tracking is done via Javascript (like google analytics) stripping the query string won't prevent the tracking from happening as it will happen on the client browser. If the tracking happens on the backend, you're out of luck. On Tue, May 1, 2012 at 4:46 PM, Travis Crowder wrote: > I am pretty sure the answer to this is: It's not possible; but I am hoping > to learn something new. > > There is a page, let's call it promo.php. We tell a bunch of people about > it and append their email address to the page for tracking ... i.e > promo.php?foo at bar.baz > > I believe in other to cache the page, I need to strip the query string > (since it will be unique). Is there a way for Varnish to cache this page, > yet still track the hits? i.e. Strip the query string for the lookup and > add it back on delivery. > > Thanks, > Travis > > ______________________________**_________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/**lists/mailman/listinfo/**varnish-misc > -- Hugues ALARY -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis.crowder at spechal.com Tue May 1 23:54:29 2012 From: travis.crowder at spechal.com (Travis Crowder) Date: Tue, 01 May 2012 18:54:29 -0500 Subject: Cache page with dynamic query string In-Reply-To: References: <4FA075EA.4000008@spechal.com> Message-ID: <4FA077B5.3080405@spechal.com> Thanks for your quick reply. I appreciate it. The tracking does happen on the front end (omniture javascript), which is why I am weary of just stripping it out. Do you believe the approach of stripping and adding back in the query string is appropriate? Thanks again, Travis On 5/1/2012 6:50 PM, Hugues Alary wrote: > If the tracking is done via Javascript (like google analytics) > stripping the query string won't prevent the tracking from happening > as it will happen on the client browser. > > If the tracking happens on the backend, you're out of luck. > > On Tue, May 1, 2012 at 4:46 PM, Travis Crowder > > wrote: > > I am pretty sure the answer to this is: It's not possible; but I > am hoping to learn something new. > > There is a page, let's call it promo.php. We tell a bunch of > people about it and append their email address to the page for > tracking ... i.e promo.php?foo at bar.baz > > I believe in other to cache the page, I need to strip the query > string (since it will be unique). Is there a way for Varnish to > cache this page, yet still track the hits? i.e. Strip the query > string for the lookup and add it back on delivery. > > Thanks, > Travis > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > > > -- > Hugues ALARY -------------- next part -------------- An HTML attachment was scrubbed... URL: From hugues.alary at gmail.com Wed May 2 00:17:26 2012 From: hugues.alary at gmail.com (Hugues Alary) Date: Tue, 1 May 2012 17:17:26 -0700 Subject: Cache page with dynamic query string In-Reply-To: <4FA077B5.3080405@spechal.com> References: <4FA075EA.4000008@spechal.com> <4FA077B5.3080405@spechal.com> Message-ID: You don't need to put the query string back and even if you do so, it won't change anything. GET /promo.php?foo at baz.com GET /promo.php client -----------------------------------------------> Varnish -------------------------> backend /promo.php?foo at baz.com /promo.php client <---------------------------------------------- Varnish <-------------------------- backend On the Varnish side, the url cached is promo.php. On the client side, the url in the browser bar will stay promo.php? foo at baz.com and you can access the query string, in javascript. On Tue, May 1, 2012 at 4:54 PM, Travis Crowder wrote: > Thanks for your quick reply. I appreciate it. > > The tracking does happen on the front end (omniture javascript), which is > why I am weary of just stripping it out. Do you believe the approach of > stripping and adding back in the query string is appropriate? > > Thanks again, > Travis > > > On 5/1/2012 6:50 PM, Hugues Alary wrote: > > If the tracking is done via Javascript (like google analytics) stripping > the query string won't prevent the tracking from happening as it will > happen on the client browser. > > If the tracking happens on the backend, you're out of luck. > > On Tue, May 1, 2012 at 4:46 PM, Travis Crowder > wrote: > >> I am pretty sure the answer to this is: It's not possible; but I am >> hoping to learn something new. >> >> There is a page, let's call it promo.php. We tell a bunch of people >> about it and append their email address to the page for tracking ... i.e >> promo.php?foo at bar.baz >> >> I believe in other to cache the page, I need to strip the query string >> (since it will be unique). Is there a way for Varnish to cache this page, >> yet still track the hits? i.e. Strip the query string for the lookup and >> add it back on delivery. >> >> Thanks, >> Travis >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > > > -- > Hugues ALARY > > -- Hugues ALARY -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nicholas_Maesepp at scee.net Wed May 2 00:22:12 2012 From: Nicholas_Maesepp at scee.net (Nicholas_Maesepp at scee.net) Date: Wed, 2 May 2012 11:22:12 +1100 Subject: Strange 302 In-Reply-To: <4FA077B5.3080405@spechal.com> References: <4FA075EA.4000008@spechal.com> <4FA077B5.3080405@spechal.com> Message-ID: Hi All Been running varnish-3.0.2 revision 55e70a4 for a couple weeks using -smalloc and 6G. Got several of them and they run fine for a couple weeks and for the 2nd time now the first cache has stopped serving a key css file and returning a 302. I don't cache the css and I also don't implement any 302 redirects via varnish. GET http://www.example.com/r09_d71/static/includes/style/portal_combined_min.css HTTP/1.1" 302 210 "http:/www.example.com/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20100101 Firefox/12.0" After a restart of varnish only I get GET http://www.example.com/r09_d71/static/includes/style/portal_combined_min.css HTTP/1.1" 200 44805 "http://www.example.com/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20100101 Firefox/12.0" I believe a few other image objects may also not load properly but it is hard to tell with a broken css and the interesting part is I don't cache the css whilst I do the images. I have made some tweaks(thanks internet), which I may not fully understand and worked fine in limited testing whilst giving performance improvements. -p log_hashstring=off \ -p shm_workspace=32768 \ -p thread_pool_workspace=262144 \ -p http_gzip_support=off \ -p http_range_support=on \ -p http_req_hdr_len=9072 \ Any ideas why I would get a 302? I track varnish memory using cloudwatch (in AWS) and at the time of restart it was at 65% and had been on the decline for a couple hours from 88%. The total Ram on the box is 7.5G so 90% is about the limit varnish will hit. Have I not allowed enough RAM for worker threads? # # The minimum number of worker threads to start VARNISH_MIN_THREADS=30 # # # The Maximum number of worker threads to start VARNISH_MAX_THREADS=1000 # # # Idle timeout for worker threads VARNISH_THREAD_TIMEOUT=120 Cheers Nick Maesepp Sony Computer Entertainment Australia Pty Ltd Level 1, 63-73 Ann Street Surry Hills NSW 2010 P.O. Box 5023 Darlinghurst NSW 2010 ph: +61 (0)2 9324 9500 fax: +61 (0)2 9324 9558 http://au.playstation.com http://www.facebook.com/PlayStationAU THE WORLD IS IN PLAY. PlayStation?Vita has arrived. http://www.psvita.com ********************************************************************** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify postmaster at scee.net This footnote also confirms that this email message has been checked for all known viruses. Sony Computer Entertainment Australia Pty. Limited Registered Office: Level 1, 63-73 Ann Street, Surry Hills, NSW 2010 Australia Registered in Australia: 077 583 183 ********************************************************************** Please consider the environment before printing this e-mail -------------- next part -------------- An HTML attachment was scrubbed... URL: From hugues.alary at gmail.com Wed May 2 00:27:21 2012 From: hugues.alary at gmail.com (Hugues Alary) Date: Tue, 1 May 2012 17:27:21 -0700 Subject: Strange 302 In-Reply-To: References: <4FA075EA.4000008@spechal.com> <4FA077B5.3080405@spechal.com> Message-ID: Hi, Have you tried bypassing varnish by either removing it or doing a return(pipe); as the 1st instruction in your vcl_recv() and see if you still get the 302? If not, it *could* be varnish, if yes, it is your backend application that send a 302. -Hugues On Tue, May 1, 2012 at 5:22 PM, wrote: > Hi All > > Been running varnish-3.0.2 revision 55e70a4 for a couple weeks using > -smalloc and 6G. Got several of them and they run fine for a couple weeks > and for the 2nd time now the first cache has stopped serving a key css file > and returning a 302. I don't cache the css and I also don't implement any > 302 redirects via varnish. > > GET > http://www.example.com/r09_d71/static/includes/style/portal_combined_min.cssHTTP/1.1" 302 210 "http:/ > www.example.com/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) > Gecko/20100101 Firefox/12.0" > > After a restart of varnish only I get > > GET > http://www.example.com/r09_d71/static/includes/style/portal_combined_min.cssHTTP/1.1" 200 44805 " > http://www.example.com/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) > Gecko/20100101 Firefox/12.0" > > I believe a few other image objects may also not load properly but it is > hard to tell with a broken css and the interesting part is I don't cache > the css whilst I do the images. > > I have made some tweaks(thanks internet), which I may not fully understand > and worked fine in limited testing whilst giving performance improvements. > > -p log_hashstring=off \ > -p shm_workspace=32768 \ > -p thread_pool_workspace=262144 \ > -p http_gzip_support=off \ > -p http_range_support=on \ > -p http_req_hdr_len=9072 \ > > Any ideas why I would get a 302? I track varnish memory using cloudwatch > (in AWS) and at the time of restart it was at 65% and had been on the > decline for a couple hours from 88%. The total Ram on the box is 7.5G so > 90% is about the limit varnish will hit. Have I not allowed enough RAM for > worker threads? > > # # The minimum number of worker threads to start > VARNISH_MIN_THREADS=30 > # > # # The Maximum number of worker threads to start > VARNISH_MAX_THREADS=1000 > # > # # Idle timeout for worker threads > VARNISH_THREAD_TIMEOUT=120 > > > > Cheers > > Nick Maesepp > > > > > > > > * > * > > * > Sony Computer Entertainment Australia Pty Ltd* > Level 1, 63-73 Ann Street Surry Hills NSW 2010 > P.O. Box 5023 Darlinghurst NSW 2010 > ph: +61 (0)2 9324 9500 fax: +61 (0)2 9324 9558 > > *http://au.playstation.com* * > **http://www.facebook.com/PlayStationAU* > > > * > THE WORLD IS IN PLAY.* > PlayStation?Vita has arrived. > http://www.psvita.com > > * > *[image: PSMOVE] > > ********************************************************************** > This email and any files transmitted with it are confidential and intended > solely for the use of the individual or entity to whom they are addressed. > If you have received this email in error please notify postmaster at scee.net > This footnote also confirms that this email message has been checked for > all known viruses. > Sony Computer Entertainment Australia Pty. Limited > Registered Office: Level 1, 63-73 Ann Street, Surry Hills, NSW 2010 > Australia > Registered in Australia: 077 583 183 > ********************************************************************** > * > Please consider the environment before printing this e-mail* > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- Hugues ALARY -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nicholas_Maesepp at scee.net Wed May 2 00:37:09 2012 From: Nicholas_Maesepp at scee.net (Nicholas_Maesepp at scee.net) Date: Wed, 2 May 2012 11:37:09 +1100 Subject: Strange 302 In-Reply-To: References: <4FA075EA.4000008@spechal.com> <4FA077B5.3080405@spechal.com> Message-ID: If I go direct to the backend I do not have the error. I imagine I could use varnishadm console to load a different VCL, thought I've never tried, as restarting varnish fixes the issues straight away. I find it unlikely as my other caches didn't have the problem at the time and they were hitting the same backend. Also should mention I do use grace and saintmode where I can. Nick From: Hugues Alary To: Nicholas_Maesepp at scee.net Cc: varnish-misc at varnish-cache.org Date: 05/02/2012 10:28 AM Subject: Re: Strange 302 Hi, Have you tried bypassing varnish by either removing it or doing a return(pipe); as the 1st instruction in your vcl_recv() and see if you still get the 302? If not, it *could* be varnish, if yes, it is your backend application that send a 302. -Hugues On Tue, May 1, 2012 at 5:22 PM, wrote: Hi All Been running varnish-3.0.2 revision 55e70a4 for a couple weeks using -smalloc and 6G. Got several of them and they run fine for a couple weeks and for the 2nd time now the first cache has stopped serving a key css file and returning a 302. I don't cache the css and I also don't implement any 302 redirects via varnish. GET http://www.example.com/r09_d71/static/includes/style/portal_combined_min.css HTTP/1.1" 302 210 "http:/www.example.com/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20100101 Firefox/12.0" After a restart of varnish only I get GET http://www.example.com/r09_d71/static/includes/style/portal_combined_min.css HTTP/1.1" 200 44805 "http://www.example.com/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20100101 Firefox/12.0" I believe a few other image objects may also not load properly but it is hard to tell with a broken css and the interesting part is I don't cache the css whilst I do the images. I have made some tweaks(thanks internet), which I may not fully understand and worked fine in limited testing whilst giving performance improvements. -p log_hashstring=off \ -p shm_workspace=32768 \ -p thread_pool_workspace=262144 \ -p http_gzip_support=off \ -p http_range_support=on \ -p http_req_hdr_len=9072 \ Any ideas why I would get a 302? I track varnish memory using cloudwatch (in AWS) and at the time of restart it was at 65% and had been on the decline for a couple hours from 88%. The total Ram on the box is 7.5G so 90% is about the limit varnish will hit. Have I not allowed enough RAM for worker threads? # # The minimum number of worker threads to start VARNISH_MIN_THREADS=30 # # # The Maximum number of worker threads to start VARNISH_MAX_THREADS=1000 # # # Idle timeout for worker threads VARNISH_THREAD_TIMEOUT=120 Cheers Nick Maesepp -- Hugues ALARY Sony Computer Entertainment Australia Pty Ltd Level 1, 63-73 Ann Street Surry Hills NSW 2010 P.O. Box 5023 Darlinghurst NSW 2010 ph: +61 (0)2 9324 9500 fax: +61 (0)2 9324 9558 http://au.playstation.com http://www.facebook.com/PlayStationAU THE WORLD IS IN PLAY. PlayStation?Vita has arrived. http://www.psvita.com ********************************************************************** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify postmaster at scee.net This footnote also confirms that this email message has been checked for all known viruses. Sony Computer Entertainment Australia Pty. Limited Registered Office: Level 1, 63-73 Ann Street, Surry Hills, NSW 2010 Australia Registered in Australia: 077 583 183 ********************************************************************** Please consider the environment before printing this e-mail -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfheen at varnish-software.com Wed May 2 10:31:41 2012 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Wed, 2 May 2012 12:31:41 +0200 Subject: varnish and ram size In-Reply-To: References: Message-ID: <20120502103141.GA3897@err.no> ]] shahab bakhtiyari Hi, > what am I doing wrong? is there a way prohibit that or at least tell > varnish not to use the whole cache? Please provide the output of varnishstat -1 when it's using more than the allocated amount of resources and we might be able to figure out what's wrong. -- Tollef Fog Heen Technical lead, Varnish Software t: +47 21 98 92 64 From shahab371 at gmail.com Wed May 2 14:37:30 2012 From: shahab371 at gmail.com (shahab bakhtiyari) Date: Wed, 2 May 2012 16:37:30 +0200 Subject: varnish and ram size In-Reply-To: References: <20120502103141.GA3897@err.no> Message-ID: On 2 May 2012 15:54, shahab bakhtiyari wrote: > Hi > > here you can see outputs from top, and varnishstat -1 > > top: > 12792 nobody 20 0 5639m 3.6g 3.4g S 13 94.3 2:01.63 varnishd > > > > > > root at varnish:~# varnishstat -1 > client_conn 259312 158.89 Client connections accepted > client_drop 0 0.00 Connection dropped, no sess/wrk > client_req 259212 158.83 Client requests received > cache_hit 33921 20.78 Cache hits > cache_hitpass 0 0.00 Cache hits for pass > cache_miss 225291 138.05 Cache misses > backend_conn 43774 26.82 Backend conn. success > backend_unhealthy 0 0.00 Backend conn. not attempted > backend_busy 0 0.00 Backend conn. too many > backend_fail 0 0.00 Backend conn. failures > backend_reuse 181617 111.28 Backend conn. reuses > backend_toolate 0 0.00 Backend conn. was closed > backend_recycle 181617 111.28 Backend conn. recycles > backend_unused 0 0.00 Backend conn. unused > fetch_head 0 0.00 Fetch head > fetch_length 225291 138.05 Fetch with Length > fetch_chunked 0 0.00 Fetch chunked > fetch_eof 0 0.00 Fetch EOF > fetch_bad 0 0.00 Fetch had bad headers > fetch_close 0 0.00 Fetch wanted close > fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed > fetch_zero 0 0.00 Fetch zero len > fetch_failed 0 0.00 Fetch failed > n_sess_mem 351 . N struct sess_mem > n_sess 102 . N struct sess > n_object 220554 . N struct object > n_vampireobject 0 . N unresurrected objects > n_objectcore 220557 . N struct objectcore > n_objecthead 216397 . N struct objecthead > n_smf 441172 . N struct smf > n_smf_frag 0 . N small free smf > n_smf_large 2 . N large free smf > n_vbe_conn 100 . N struct vbe_conn > n_wrk 121 . N worker threads > n_wrk_create 169 0.10 N worker threads created > n_wrk_failed 0 0.00 N worker threads not created > n_wrk_max 951160 582.82 N worker threads limited > n_wrk_queue 0 0.00 N queued work requests > n_wrk_overflow 2082 1.28 N overflowed work requests > n_wrk_drop 0 0.00 N dropped work requests > n_backend 1 . N backends > n_expired 4737 . N expired objects > n_lru_nuked 0 . N LRU nuked objects > n_lru_saved 0 . N LRU saved objects > n_lru_moved 26276 . N LRU moved objects > n_deathrow 0 . N objects on deathrow > losthdr 0 0.00 HTTP header overflows > n_objsendfile 0 0.00 Objects sent with sendfile > n_objwrite 259207 158.83 Objects sent with write > n_objoverflow 0 0.00 Objects overflowing workspace > s_sess 259212 158.83 Total Sessions > s_req 259212 158.83 Total Requests > s_pipe 0 0.00 Total pipe > s_pass 0 0.00 Total pass > s_fetch 225291 138.05 Total fetch > s_hdrbytes 119525246 73238.51 Total header bytes > s_bodybytes 2748370979 1684050.84 Total body bytes > sess_closed 259212 158.83 Session Closed > sess_pipeline 0 0.00 Session Pipeline > sess_readahead 0 0.00 Session Read Ahead > sess_linger 0 0.00 Session Linger > sess_herd 0 0.00 Session herd > shm_records 21397311 13111.10 SHM records > shm_writes 1445766 885.89 SHM writes > shm_flushes 27 0.02 SHM flushes due to overflow > shm_cont 9812 6.01 SHM MTX contention > shm_cycles 9 0.01 SHM cycles through buffer > sm_nreq 450644 276.13 allocator requests > sm_nobj 441170 . outstanding allocations > sm_balloc 3777261568 . bytes allocated > sm_bfree 743198720 . bytes free > sma_nreq 0 0.00 SMA allocator requests > sma_nobj 0 . SMA outstanding allocations > sma_nbytes 0 . SMA outstanding bytes > sma_balloc 0 . SMA bytes allocated > sma_bfree 0 . SMA bytes free > sms_nreq 0 0.00 SMS allocator requests > sms_nobj 0 . SMS outstanding allocations > sms_nbytes 0 . SMS outstanding bytes > sms_balloc 0 . SMS bytes allocated > sms_bfree 0 . SMS bytes freed > backend_req 225390 138.11 Backend requests made > n_vcl 1 0.00 N vcl total > n_vcl_avail 1 0.00 N vcl available > n_vcl_discard 0 0.00 N vcl discarded > n_purge 1 . N total active purges > n_purge_add 2 0.00 N new purges added > n_purge_retire 1 0.00 N old purges deleted > n_purge_obj_test 0 0.00 N objects tested > n_purge_re_test 0 0.00 N regexps tested against > n_purge_dups 0 0.00 N duplicate purges removed > hcb_nolock 259312 158.89 HCB Lookups without lock > hcb_lock 217430 133.23 HCB Lookups with lock > hcb_insert 217428 133.23 HCB Inserts > esi_parse 0 0.00 Objects ESI parsed (unlock) > esi_errors 0 0.00 ESI parse errors (unlock) > accept_fail 0 0.00 Accept failures > client_drop_late 0 0.00 Connection dropped late > uptime 1632 1.00 Client uptime > > > > > On 2 May 2012 12:31, Tollef Fog Heen wrote: > >> ]] shahab bakhtiyari >> >> Hi, >> >> > what am I doing wrong? is there a way prohibit that or at least tell >> > varnish not to use the whole cache? >> >> Please provide the output of varnishstat -1 when it's using more than >> the allocated amount of resources and we might be able to figure out >> what's wrong. >> >> -- >> Tollef Fog Heen >> Technical lead, Varnish Software >> t: +47 21 98 92 64 >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shahab371 at gmail.com Wed May 2 14:45:18 2012 From: shahab371 at gmail.com (shahab bakhtiyari) Date: Wed, 2 May 2012 16:45:18 +0200 Subject: varnish and ram size In-Reply-To: <20120502103141.GA3897@err.no> References: <20120502103141.GA3897@err.no> Message-ID: Hi here you can see outputs from top, and varnishstat -1 top: 12792 nobody 20 0 5639m 3.6g 3.4g S 13 94.3 2:01.63 varnishd root at varnish:~# varnishstat -1 client_conn 259312 158.89 Client connections accepted client_drop 0 0.00 Connection dropped, no sess/wrk client_req 259212 158.83 Client requests received cache_hit 33921 20.78 Cache hits cache_hitpass 0 0.00 Cache hits for pass cache_miss 225291 138.05 Cache misses backend_conn 43774 26.82 Backend conn. success backend_unhealthy 0 0.00 Backend conn. not attempted backend_busy 0 0.00 Backend conn. too many backend_fail 0 0.00 Backend conn. failures backend_reuse 181617 111.28 Backend conn. reuses backend_toolate 0 0.00 Backend conn. was closed backend_recycle 181617 111.28 Backend conn. recycles backend_unused 0 0.00 Backend conn. unused fetch_head 0 0.00 Fetch head fetch_length 225291 138.05 Fetch with Length fetch_chunked 0 0.00 Fetch chunked fetch_eof 0 0.00 Fetch EOF fetch_bad 0 0.00 Fetch had bad headers fetch_close 0 0.00 Fetch wanted close fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed fetch_zero 0 0.00 Fetch zero len fetch_failed 0 0.00 Fetch failed n_sess_mem 351 . N struct sess_mem n_sess 102 . N struct sess n_object 220554 . N struct object n_vampireobject 0 . N unresurrected objects n_objectcore 220557 . N struct objectcore n_objecthead 216397 . N struct objecthead n_smf 441172 . N struct smf n_smf_frag 0 . N small free smf n_smf_large 2 . N large free smf n_vbe_conn 100 . N struct vbe_conn n_wrk 121 . N worker threads n_wrk_create 169 0.10 N worker threads created n_wrk_failed 0 0.00 N worker threads not created n_wrk_max 951160 582.82 N worker threads limited n_wrk_queue 0 0.00 N queued work requests n_wrk_overflow 2082 1.28 N overflowed work requests n_wrk_drop 0 0.00 N dropped work requests n_backend 1 . N backends n_expired 4737 . N expired objects n_lru_nuked 0 . N LRU nuked objects n_lru_saved 0 . N LRU saved objects n_lru_moved 26276 . N LRU moved objects n_deathrow 0 . N objects on deathrow losthdr 0 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 259207 158.83 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 259212 158.83 Total Sessions s_req 259212 158.83 Total Requests s_pipe 0 0.00 Total pipe s_pass 0 0.00 Total pass s_fetch 225291 138.05 Total fetch s_hdrbytes 119525246 73238.51 Total header bytes s_bodybytes 2748370979 1684050.84 Total body bytes sess_closed 259212 158.83 Session Closed sess_pipeline 0 0.00 Session Pipeline sess_readahead 0 0.00 Session Read Ahead sess_linger 0 0.00 Session Linger sess_herd 0 0.00 Session herd shm_records 21397311 13111.10 SHM records shm_writes 1445766 885.89 SHM writes shm_flushes 27 0.02 SHM flushes due to overflow shm_cont 9812 6.01 SHM MTX contention shm_cycles 9 0.01 SHM cycles through buffer sm_nreq 450644 276.13 allocator requests sm_nobj 441170 . outstanding allocations sm_balloc 3777261568 . bytes allocated sm_bfree 743198720 . bytes free sma_nreq 0 0.00 SMA allocator requests sma_nobj 0 . SMA outstanding allocations sma_nbytes 0 . SMA outstanding bytes sma_balloc 0 . SMA bytes allocated sma_bfree 0 . SMA bytes free sms_nreq 0 0.00 SMS allocator requests sms_nobj 0 . SMS outstanding allocations sms_nbytes 0 . SMS outstanding bytes sms_balloc 0 . SMS bytes allocated sms_bfree 0 . SMS bytes freed backend_req 225390 138.11 Backend requests made n_vcl 1 0.00 N vcl total n_vcl_avail 1 0.00 N vcl available n_vcl_discard 0 0.00 N vcl discarded n_purge 1 . N total active purges n_purge_add 2 0.00 N new purges added n_purge_retire 1 0.00 N old purges deleted n_purge_obj_test 0 0.00 N objects tested n_purge_re_test 0 0.00 N regexps tested against n_purge_dups 0 0.00 N duplicate purges removed hcb_nolock 259312 158.89 HCB Lookups without lock hcb_lock 217430 133.23 HCB Lookups with lock hcb_insert 217428 133.23 HCB Inserts esi_parse 0 0.00 Objects ESI parsed (unlock) esi_errors 0 0.00 ESI parse errors (unlock) accept_fail 0 0.00 Accept failures client_drop_late 0 0.00 Connection dropped late uptime 1632 1.00 Client uptime On 2 May 2012 12:31, Tollef Fog Heen wrote: > ]] shahab bakhtiyari > > Hi, > > > what am I doing wrong? is there a way prohibit that or at least tell > > varnish not to use the whole cache? > > Please provide the output of varnishstat -1 when it's using more than > the allocated amount of resources and we might be able to figure out > what's wrong. > > -- > Tollef Fog Heen > Technical lead, Varnish Software > t: +47 21 98 92 64 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shib4u at gmail.com Thu May 3 10:07:52 2012 From: shib4u at gmail.com (Shibashish) Date: Thu, 3 May 2012 15:37:52 +0530 Subject: Cache page with dynamic query string In-Reply-To: References: <4FA075EA.4000008@spechal.com> <4FA077B5.3080405@spechal.com> Message-ID: #add in vcl_recv set req.url = regsub(req.url, "(\?|&)$", ""); -------------- next part -------------- An HTML attachment was scrubbed... URL: From nchriss at gmail.com Thu May 3 15:48:42 2012 From: nchriss at gmail.com (Neha Chriss) Date: Thu, 3 May 2012 17:48:42 +0200 Subject: Security VCL Connection Tracking Message-ID: Hello I am wondering if any one can recommend a method of identifying repeated POST attempts to a single URI with Security VCL or through some native varnish mechanism. I am currenlty using security vcl as a WAF with the modsecurity CRS. We occasionally have malicious users who will attempt to bruteforce promotions codes, or, alternative, attempt to scan our web application for vulnerabilities. I am looking for a way to mitigate these risks at the WAF-layer.. any suggestions? cheers, Neha -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Thu May 3 17:34:19 2012 From: perbu at varnish-software.com (Per Buer) Date: Thu, 3 May 2012 19:34:19 +0200 Subject: Security VCL Connection Tracking In-Reply-To: References: Message-ID: Hi, On Thu, May 3, 2012 at 5:48 PM, Neha Chriss wrote: > Hello > > I am wondering if any one can recommend a method of identifying repeated > POST attempts to a single URI with Security VCL or through some native > varnish mechanism. I am currenlty using security vcl as a WAF with the > modsecurity CRS. We occasionally have malicious users who will attempt to > bruteforce promotions codes, or, alternative, attempt to scan our web > application for vulnerabilities. I am looking for a way to mitigate these > risks at the WAF-layer.. any suggestions? > You could build something on top of the variable vmod. It probably needs a data structure that scales better, a hash or a tree. Then you can store IP-adress+URL somewhere and count the occurrences and blacklist clients whenever they pass a threshold. Or something. -- Per Buer Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From fred at clift.org Thu May 3 21:07:32 2012 From: fred at clift.org (Fred Clift) Date: Thu, 3 May 2012 15:07:32 -0600 Subject: dns director efficiency on large number of hosts? Message-ID: So I'm setting up a test varnish server, and have configured configured a director block that looks something like: director www dns { .list = { .port = "80"; "10.1.55.165/32"; "10.1.54.126/32"; "10.1.222.1/32"; } .ttl = 5m; .suffix = ""; } (IP addressed changed to protect the innocent) There is a valid DNS entry for each of 3 domains, each pointing at one of those addresses. When testing, this appears to do what I would like - which is cache for 3 unrelated sites. For now, I change my local resolver to, for my test client, point the domains at the varnish server, while having a .suffix of "" means they'll look up the real public IPs of the semi-live sites and it will cache those sites for just me. Eventually I will point public DNS for those sites at the varnish server and maintain a domain-to-IP mapping in my DNS server (hm /etc/hosts on varnish server? nsswitch works right?) Ideally I'd like to just load a config file with public.dns.name./ backend.dns.name. mappings or public.dns.name./IP mappings but I've found several archived discussions of this (e.g. dns resolved at vcl compilation time...) that doesn't show promise for now. I guess I'll just have to update the config (scripted) if a domain changes back-end IP address. My real question is: How efficient is this for a large number of hosts? say 1K? 10K? Is the hostname to IP lookup efficient? I know that having many sites all cached separately will not be great for cache hit rate.... I believe that if it becomes a problem I can throw more ram and SSD swap at that problem, to some extent anyway. I guess an alternative for me would be to write my own director code that included some file with hostname/backend mappings and compiled it into the VCL along with a simple hash. or maybe just some inline C that would allow: sub vcl_recv { req.backend = myDictionaryLookup(req.http.host) ... } rather than having to do 1000 if/elseif/elseif blocks.... Is there some other easier way to do this? would the DNS director perform adequately? Thanks. Fred Clift -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfheen at varnish-software.com Fri May 4 08:32:19 2012 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Fri, 4 May 2012 10:32:19 +0200 Subject: varnish and ram size In-Reply-To: References: <20120502103141.GA3897@err.no> Message-ID: <20120504083219.GF29959@err.no> ]] shahab bakhtiyari > here you can see outputs from top, and varnishstat -1 > > top: > 12792 nobody 20 0 5639m 3.6g 3.4g S 13 94.3 2:01.63 varnishd > sm_balloc 3777261568 . bytes allocated > sm_bfree 743198720 . bytes free These numbers look consistent. What -s argument are you giving varnishd, and does the storage file already exist on disk with a different size? If so, removing it and letting Varnish recreate it might be helpful. Cheers, -- Tollef Fog Heen Technical lead, Varnish Software t: +47 21 98 92 64 From shahab371 at gmail.com Fri May 4 08:58:11 2012 From: shahab371 at gmail.com (shahab bakhtiyari) Date: Fri, 4 May 2012 10:58:11 +0200 Subject: varnish and ram size In-Reply-To: <20120504083219.GF29959@err.no> References: <20120502103141.GA3897@err.no> <20120504083219.GF29959@err.no> Message-ID: On 4 May 2012 10:32, Tollef Fog Heen wrote: > ]] shahab bakhtiyari > > > here you can see outputs from top, and varnishstat -1 > > > > top: > > 12792 nobody 20 0 5639m 3.6g 3.4g S 13 94.3 2:01.63 varnishd > > > sm_balloc 3777261568 . bytes allocated > > sm_bfree 743198720 . bytes free > > These numbers look consistent. What -s argument are you giving > varnishd, -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G" > and does the storage file already exist on disk with a > different size? no it does not exist. I upgraded to version 3.0.2 yesterday. now I notice that when I start varnish, it creates a vcl.somechars.so file instead of varnish_storage.bin . Does it have something to say? If so, removing it and letting Varnish recreate it > might be helpful. > > generally, isn't there any way to tell varnish how much ram it should use? something like malloc option, but malloc is well, ment to be used when varnish is supposed to use ONLY ram, right? > Cheers, > -- > Tollef Fog Heen > Technical lead, Varnish Software > t: +47 21 98 92 64 > thanks in advance Shahab -------------- next part -------------- An HTML attachment was scrubbed... URL: From nitinpawar432 at gmail.com Tue May 8 06:59:54 2012 From: nitinpawar432 at gmail.com (Nitin Pawar) Date: Tue, 8 May 2012 12:29:54 +0530 Subject: hashing based on cookie combinations Message-ID: Hi, I am trying to put varnish cache server infront of my apache server to accelerate the performance. I want to cache the data based on few cookies I have tested with single cookie and it does work but I have an use case where i need to cache the data based on combination of cookies use case if( req.http.Cookie ~ "Cookie1" && (! req.http.Cookie ~ "Cookie2")) { set req.http.X-Varnish-Hashed-On = regsub( req.http.Cookie, "Cookie1=([^;]*);*.*$", "\1" ); } if( req.http.Cookie ~ "Cookie1" && req.http.Cookie ~ "Cookie2") { set req.http.X-Varnish-Hashed-On = regsub( req.http.Cookie, "Cookie1=([^;]*);*.*$", "\1" )+regsub( req.http.Cookie, "Cookie2=([^;]*);*.*$", "\1" ); } hash_data(req.http.X-Varnish-Hashed-On); return(hash); On the compilation phase it works fine but when I send a web request with these user cases ( first send with cookie1 and then both cookie1 and cookie2) , it always returns the data cached with cookie1 and does not respect the cookie2 combination. Please do let me know if I am doing something wrong. I could not find anywhere how do put multiple cookies in place for hashing the data. -- Nitin Pawar -------------- next part -------------- An HTML attachment was scrubbed... URL: From torbped at gmail.com Tue May 8 11:00:26 2012 From: torbped at gmail.com (Torben Pedersen) Date: Tue, 8 May 2012 13:00:26 +0200 Subject: Cookie set on beresp not present on next request Message-ID: <381EE28D-6B55-4183-A097-AD738FEF314F@gmail.com> Hi, I'm trying to use Varnish + a cookie to control access from mobile devices. I've seen other people doing similar, but I can't get it to work. I'm accessing my solution by https://?mobile_redirect=0 rutine to check user agent sub detect_device { if ((req.http.User-Agent ~ "iP(hone|od|ad)" || req.http.User-Agent ~ "Android" || req.http.User-Agent ~ "SymbianOS" || req.http.User-Agent ~ "HTC" || req.http.User-Agent ~ "BlackBerry" || req.http.User-Agent ~ "SonyEricsson" || req.http.User-Agent ~ "Nokia" || req.http.User-Agent ~ "NOKIA" || req.http.User-Agent ~"IEMobile" || req.http.User-Agent ~ "SAMSUNG" || req.http.User-Agent ~ "LG")) { set req.http.X-Device = "smart"; } } in vcl_recv I have if (req.http.X-Device == "smart") { if ((req.url ~ "^/client/$") || (req.url ~ "^/client/index.html") || (req.url ~ "^/$") || (req.url ~ "^/")) { if ((req.url !~ "(.*)(mobile_redirect=0)") && (req.http.Cookie !~ "nomobile=1")) { error 707 "Redirected to mobile site"; } } } in vcl_fetch I have if (req.http.X-Device == "smart") { if (req.url ~ "(.*)(mobile_redirect=0)") { set beresp.http.Set-Cookie = "nomobile=1; expires: Session; path=/; domain=."; } } in vcl_error I have #Mobile User-Agents are send to Mobilesolution if (obj.status == 707) { set obj.status = 302; set obj.http.Location = "https:// return(deliver); } So on first request to https://?mobile_redirect=0 , I can se in Chrome (Response headers) - I've set Chrome to act as an Android (Mozilla/5.0 (Linux; U; Android 2.3.6; en-us; Nexus S Build/GRK39F) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1) Set-Cookie:nomobile=1; expires: Session; path=/; domain=. This seems to be correct. So now I call https:// And I expect not to get redirected since the Set-Cookie is present. But the nomobile=1 is not present in request / Cookie. I don't quite get why ? Any ideas ? kind regards Torben -------------- next part -------------- An HTML attachment was scrubbed... URL: From sparshgupta at gmail.com Tue May 8 18:02:15 2012 From: sparshgupta at gmail.com (Sparsh Gupta) Date: Tue, 8 May 2012 23:32:15 +0530 Subject: Varnish is restarting frequently Message-ID: I have a bunch of Varnish servers (Exactly same configurations, caching different keys on Intel 2650 series Computer + 16GB RAM + 32GB SSD each). I am enclosing my configurations and settings along with my current varnishstats. I am looking for more debug information and how to find the route cause of my varnish getting restarting every 6-8 hours (all varnish servers) Please do let me know in case you need any more information. I am loosing all my cache (I tried 20G rather than 32G but still it restarts flushing all the cache and becoming unavailable for split seconds). I am using them behind nginx load balancers (which also distribute traffic among different varnish servers based on some keys). The varnish are fueled by 4 powerful LAMP servers. I have high miss rate because I keep loosing all my cache Daemon Options: START=yes NFILES=131072 MEMLOCK=82000 INSTANCE=$(uname -n) VARNISH_VCL_CONF=/etc/varnish/default.vcl VARNISH_LISTEN_ADDRESS=10.56.140.12 VARNISH_LISTEN_PORT=80 VARNISH_ADMIN_LISTEN_ADDRESS=10.56.140.12 VARNISH_ADMIN_LISTEN_PORT=6082 VARNISH_MIN_THREADS=200 VARNISH_MAX_THREADS=4000 VARNISH_THREAD_TIMEOUT=60 VARNISH_STORAGE_SIZE=32G VARNISH_STORAGE_MALLOC="malloc" VARNISH_USER="varnish" VARNISH_GROUP="varnish" DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \ -f ${VARNISH_VCL_CONF} \ -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \ -w ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT} \ -s ${VARNISH_STORAGE_MALLOC},${VARNISH_STORAGE_SIZE} \ -u ${VARNISH_USER} -g ${VARNISH_GROUP} \ -i ${INSTANCE} \ -p thread_pool_add_delay=2 \ -p thread_pools=32 \ -p thread_pool_min=25 \ -p thread_pool_max=4000 \ -p sess_timeout=3 \ -p session_max=1000000 \ -p session_linger=2000" Varnishstat -1: client_conn 49815 26.78 Client connections accepted client_drop 0 0.00 Connection dropped, no sess/wrk client_req 1822206 979.68 Client requests received cache_hit 253196 136.13 Cache hits cache_hitpass 1379 0.74 Cache hits for pass cache_miss 1567631 842.81 Cache misses backend_conn 16163 8.69 Backend conn. success backend_unhealthy 0 0.00 Backend conn. not attempted backend_busy 0 0.00 Backend conn. too many backend_fail 1264 0.68 Backend conn. failures backend_reuse 1552269 834.55 Backend conn. reuses backend_toolate 3 0.00 Backend conn. was closed backend_recycle 1553574 835.25 Backend conn. recycles backend_retry 0 0.00 Backend conn. retry fetch_head 0 0.00 Fetch head fetch_length 1567981 843.00 Fetch with Length fetch_chunked 394 0.21 Fetch chunked fetch_eof 0 0.00 Fetch EOF fetch_bad 0 0.00 Fetch had bad headers fetch_close 0 0.00 Fetch wanted close fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed fetch_zero 0 0.00 Fetch zero len fetch_failed 0 0.00 Fetch failed fetch_1xx 0 0.00 Fetch no body (1xx) fetch_204 0 0.00 Fetch no body (204) fetch_304 0 0.00 Fetch no body (304) n_sess_mem 968 . N struct sess_mem n_sess 209 . N struct sess n_object 1565359 . N struct object n_vampireobject 0 . N unresurrected objects n_objectcore 1565551 . N struct objectcore n_objecthead 1565798 . N struct objecthead n_waitinglist 965 . N struct waitinglist n_vbc 19 . N struct vbc n_wrk 800 . N worker threads n_wrk_create 836 0.45 N worker threads created n_wrk_failed 0 0.00 N worker threads not created n_wrk_max 0 0.00 N worker threads limited n_wrk_lqueue 0 0.00 work request queue length n_wrk_queued 210 0.11 N queued work requests n_wrk_drop 0 0.00 N dropped work requests n_backend 4 . N backends n_expired 1637 . N expired objects n_lru_nuked 0 . N LRU nuked objects n_lru_moved 177419 . N LRU moved objects losthdr 0 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 1817561 977.18 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 49763 26.75 Total Sessions s_req 1822206 979.68 Total Requests s_pipe 0 0.00 Total pipe s_pass 1379 0.74 Total pass s_fetch 1568375 843.21 Total fetch s_hdrbytes 543007555 291939.55 Total header bytes s_bodybytes 366746858 197175.73 Total body bytes sess_closed 1973 1.06 Session Closed sess_pipeline 0 0.00 Session Pipeline sess_readahead 0 0.00 Session Read Ahead sess_linger 1821411 979.25 Session Linger sess_herd 50288 27.04 Session herd shm_records 132731919 71361.25 SHM records shm_writes 5168655 2778.85 SHM writes shm_flushes 0 0.00 SHM flushes due to overflow shm_cont 31135 16.74 SHM MTX contention shm_cycles 63 0.03 SHM cycles through buffer sms_nreq 635 0.34 SMS allocator requests sms_nobj 0 . SMS outstanding allocations sms_nbytes 0 . SMS outstanding bytes sms_balloc 265430 . SMS bytes allocated sms_bfree 265430 . SMS bytes freed backend_req 1569626 843.88 Backend requests made n_vcl 2 0.00 N vcl total n_vcl_avail 2 0.00 N vcl available n_vcl_discard 0 0.00 N vcl discarded n_ban 48 . N total active bans n_ban_add 49 0.03 N new bans added n_ban_retire 1 0.00 N old bans deleted n_ban_obj_test 82431 44.32 N objects tested n_ban_re_test 211534 113.73 N regexps tested against n_ban_dups 35 0.02 N duplicate bans removed hcb_nolock 1822862 980.03 HCB Lookups without lock hcb_lock 1568249 843.14 HCB Lookups with lock hcb_insert 1568249 843.14 HCB Inserts esi_errors 0 0.00 ESI parse errors (unlock) esi_warnings 0 0.00 ESI parse warnings (unlock) accept_fail 0 0.00 Accept failures client_drop_late 0 0.00 Connection dropped late uptime 1860 1.00 Client uptime dir_dns_lookups 0 0.00 DNS director lookups dir_dns_failed 0 0.00 DNS director failed lookups dir_dns_hit 0 0.00 DNS director cached lookups hit dir_dns_cache_full 0 0.00 DNS director full dnscache vmods 0 . Loaded VMODs n_gzip 0 0.00 Gzip operations n_gunzip 3377737 1815.99 Gunzip operations LCK.sms.creat 4 0.00 Created locks LCK.sms.destroy 0 0.00 Destroyed locks LCK.sms.locks 360546 193.84 Lock Operations LCK.sms.colls 0 0.00 Collisions LCK.smp.creat 0 0.00 Created locks LCK.smp.destroy 0 0.00 Destroyed locks LCK.smp.locks 0 0.00 Lock Operations LCK.smp.colls 0 0.00 Collisions LCK.sma.creat 8 0.00 Created locks LCK.sma.destroy 0 0.00 Destroyed locks LCK.sma.locks 223989728 120424.58 Lock Operations LCK.sma.colls 0 0.00 Collisions LCK.smf.creat 0 0.00 Created locks LCK.smf.destroy 0 0.00 Destroyed locks LCK.smf.locks 0 0.00 Lock Operations LCK.smf.colls 0 0.00 Collisions LCK.hsl.creat 0 0.00 Created locks LCK.hsl.destroy 0 0.00 Destroyed locks LCK.hsl.locks 0 0.00 Lock Operations LCK.hsl.colls 0 0.00 Collisions LCK.hcb.creat 4 0.00 Created locks LCK.hcb.destroy 0 0.00 Destroyed locks LCK.hcb.locks 74720192 40172.15 Lock Operations LCK.hcb.colls 0 0.00 Collisions LCK.hcl.creat 0 0.00 Created locks LCK.hcl.destroy 0 0.00 Destroyed locks LCK.hcl.locks 0 0.00 Lock Operations LCK.hcl.colls 0 0.00 Collisions LCK.vcl.creat 4 0.00 Created locks LCK.vcl.destroy 0 0.00 Destroyed locks LCK.vcl.locks 19211 10.33 Lock Operations LCK.vcl.colls 0 0.00 Collisions LCK.stat.creat 4 0.00 Created locks LCK.stat.destroy 0 0.00 Destroyed locks LCK.stat.locks 6926 3.72 Lock Operations LCK.stat.colls 0 0.00 Collisions LCK.sessmem.creat 4 0.00 Created locks LCK.sessmem.destroy 0 0.00 Destroyed locks LCK.sessmem.locks 2970656 1597.13 Lock Operations LCK.sessmem.colls 0 0.00 Collisions LCK.wstat.creat 4 0.00 Created locks LCK.wstat.destroy 0 0.00 Destroyed locks LCK.wstat.locks 7972177 4286.12 Lock Operations LCK.wstat.colls 0 0.00 Collisions LCK.herder.creat 4 0.00 Created locks LCK.herder.destroy 0 0.00 Destroyed locks LCK.herder.locks 6214 3.34 Lock Operations LCK.herder.colls 0 0.00 Collisions LCK.wq.creat 128 0.07 Created locks LCK.wq.destroy 0 0.00 Destroyed locks LCK.wq.locks 9782145 5259.22 Lock Operations LCK.wq.colls 0 0.00 Collisions LCK.objhdr.creat 74421535 40011.58 Created locks LCK.objhdr.destroy 271865 146.16 Destroyed locks LCK.objhdr.locks 349625261 187970.57 Lock Operations LCK.objhdr.colls 0 0.00 Collisions LCK.exp.creat 4 0.00 Created locks LCK.exp.destroy 0 0.00 Destroyed locks LCK.exp.locks 74639057 40128.53 Lock Operations LCK.exp.colls 0 0.00 Collisions LCK.lru.creat 8 0.00 Created locks LCK.lru.destroy 0 0.00 Destroyed locks LCK.lru.locks 74353412 39974.95 Lock Operations LCK.lru.colls 0 0.00 Collisions LCK.cli.creat 4 0.00 Created locks LCK.cli.destroy 0 0.00 Destroyed locks LCK.cli.locks 32390 17.41 Lock Operations LCK.cli.colls 0 0.00 Collisions LCK.ban.creat 4 0.00 Created locks LCK.ban.destroy 0 0.00 Destroyed locks LCK.ban.locks 78392714 42146.62 Lock Operations LCK.ban.colls 0 0.00 Collisions LCK.vbp.creat 4 0.00 Created locks LCK.vbp.destroy 0 0.00 Destroyed locks LCK.vbp.locks 92827 49.91 Lock Operations LCK.vbp.colls 0 0.00 Collisions LCK.vbe.creat 4 0.00 Created locks LCK.vbe.destroy 0 0.00 Destroyed locks LCK.vbe.locks 4095777 2202.03 Lock Operations LCK.vbe.colls 0 0.00 Collisions LCK.backend.creat 16 0.01 Created locks LCK.backend.destroy 0 0.00 Destroyed locks LCK.backend.locks 227718873 122429.50 Lock Operations LCK.backend.colls 0 0.00 Collisions SMA.s0.c_req 3136117 1686.08 Allocator requests SMA.s0.c_fail 0 0.00 Allocator failures SMA.s0.c_bytes 206790032960 111177437.08 Bytes allocated SMA.s0.c_freed 205374002652 110416130.46 Bytes freed SMA.s0.g_alloc 3132208 . Allocations outstanding SMA.s0.g_bytes 1416030308 . Bytes outstanding SMA.s0.g_space 32943708060 . Bytes available SMA.Transient.c_req 2758 1.48 Allocator requests SMA.Transient.c_fail 0 0.00 Allocator failures SMA.Transient.c_bytes 56648986 30456.44 Bytes allocated SMA.Transient.c_freed 56648986 30456.44 Bytes freed SMA.Transient.g_alloc 0 . Allocations outstanding SMA.Transient.g_bytes 0 . Bytes outstanding SMA.Transient.g_space 0 . Bytes available VBE.dev1(10.56.140.8,,80).vcls 8 . VCL references VBE.dev1(10.56.140.8,,80).happy18446744073709551615 . Happy health probes VBE.dev2(10.56.140.2,,80).vcls 8 . VCL references VBE.dev2(10.56.140.2,,80).happy18446744073709551615 . Happy health probes VBE.dev3(10.56.140.4,,80).vcls 8 . VCL references VBE.dev3(10.56.140.4,,80).happy18446744073709551615 . Happy health probes VBE.dev4(10.56.140.6,,80).vcls 8 . VCL references VBE.dev4(10.56.140.6,,80).happy18446744073709551615 . Happy health probes Thanks Sparsh Gupta -------------- next part -------------- An HTML attachment was scrubbed... URL: From emilio at antispam.it Tue May 8 20:41:23 2012 From: emilio at antispam.it (emilio brambilla) Date: Tue, 08 May 2012 22:41:23 +0200 Subject: Varnish is restarting frequently In-Reply-To: References: Message-ID: <4FA984F3.6050500@antispam.it> hello, On 2012/05/08 8:02 PM, Sparsh Gupta wrote: > I have a bunch of Varnish servers (Exactly same configurations, > caching different keys on Intel 2650 series Computer + 16GB RAM + 32GB > SSD each). I am enclosing my configurations and settings along with my > current varnishstats. > > I am looking for more debug information and how to find the route > cause of my varnish getting restarting every 6-8 hours (all varnish > servers) > > VARNISH_STORAGE_SIZE=32G > VARNISH_STORAGE_MALLOC="malloc" if you have 16 Gb of phisical ram you should not malloc 32 Gb, but something less than 16 Gb (usually 20% less than the phisical) how many swap did you set on these boxes? maybe the varnish is restarting because the kernel cannot allocate the ram it want. if you want to use more than space you may use 'file' instead of 'malloc' (but I don't know if it's a good idea to use SSD for the cache.... how many rewrite does your ssd allow?) -- bye, emilio From Nicholas_Maesepp at scee.net Wed May 9 02:11:16 2012 From: Nicholas_Maesepp at scee.net (Nicholas_Maesepp at scee.net) Date: Wed, 9 May 2012 13:11:16 +1100 Subject: Varnish 500's now not 302's Message-ID: Hi Continuing on with my last problem of the varnish cache returning a 302 for certain objects without contacting the backend I am now getting 500 errors from the cache. Object is available from the backend so definitely not a 500 from the backend and logs + tcpdump show it trying to deliver from cache only. Oddly enough its on mainly gif and png images that I notice anyhow, and it happened to multiple caches within 1 hour of each other on the same objects. A restart of varnish stopped the 500 errors. Previously to attempt to resolve my 302 issue I lowered the amount of RAM malloc'ed to 5 gig. Varnish now uses up 73% of the system's RAM. I have no swap and there was at least a gig free on all caches before they started serving 500's. Here is a copy of the varnishlog. 16 ReqStart c 10.130.XX.XX 3259 1429474950 16 RxRequest c GET 16 RxURL c /static/furn/portal/common/pixel.gif 16 RxProtocol c HTTP/1.1 16 RxHeader c host: www.example.com 16 RxHeader c Accept: text/html, application/xhtml+xml, */* 16 RxHeader c Accept-Encoding: gzip, deflate 16 RxHeader c Accept-Language: en-US 16 RxHeader c Cache-Control: max-age=259200 16 RxHeader c Cookie: portalLocaleCookie=en_AU; WT_FPC=id=XX.XX.XX.XX-249718560.30138265:lv=1336488750511:ss=1336488730455; PDC4_COOKIE=Vk6CPp9WGbfxrZMnGxQlhFfWW0cJD0NGB4mNFcvQJ6nFwmRD1kxS!-925861289 16 RxHeader c User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0) 16 RxHeader c Via: 1.1 xx.xx.xx.xx:3128 (squid/2.6.STABLE21) 16 RxHeader c Xroxy-Connection: keep-alive 16 RxHeader c Z-Forwarded-For: AAAAAAAAAAAA 16 RxHeader c X-Forwarded-For: XX.XX.XX.XX 16 RxHeader c X-Forwarded-Port: 80 16 RxHeader c X-Forwarded-Proto: http 16 RxHeader c Connection: keep-alive 16 VCL_call c recv lookup 16 VCL_call c hash hash 16 Hit c 1429284571 16 VCL_call c hit deliver 16 VCL_call c deliver deliver 16 TxProtocol c HTTP/1.1 16 TxStatus c 500 16 TxResponse c Internal Server Error 16 TxHeader c cache-control: max-age=900 16 TxHeader c Content-Length: 0 16 TxHeader c Accept-Ranges: bytes 16 TxHeader c Date: Wed, 09 May 2012 00:54:41 GMT 16 TxHeader c X-Varnish: 1429474950 1429284571 16 TxHeader c Age: 33236 16 TxHeader c Via: 1.1 varnish 16 TxHeader c Connection: keep-alive 16 Length c 0 16 ReqEnd c 1429474950 1336524881.724712133 How I understand this is my server does the recv call, determines its a lookup and then hashes as per my vcl, which is req,url and then server.ip and then it does hit which is just to return deliver. Which should deliver the object from cache, instead it gives me a 500. I've decided to turn off range_support as I understand this is considered experimental ? I'd like to avoid trial and error of my configuration if possible. I'm just guessing at current really. Any more knowledgeable ideas are welcomed. However I'd like to not have to go to disk for the cache, I don't see why malloc should not be working as intended if this is my issue. Sony Computer Entertainment Australia Pty Ltd Level 1, 63-73 Ann Street Surry Hills NSW 2010 P.O. Box 5023 Darlinghurst NSW 2010 ph: +61 (0)2 9324 9500 fax: +61 (0)2 9324 9558 http://au.playstation.com http://www.facebook.com/PlayStationAU THE WORLD IS IN PLAY. PlayStation?Vita has arrived. http://www.psvita.com ********************************************************************** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify postmaster at scee.net This footnote also confirms that this email message has been checked for all known viruses. Sony Computer Entertainment Australia Pty. Limited Registered Office: Level 1, 63-73 Ann Street, Surry Hills, NSW 2010 Australia Registered in Australia: 077 583 183 ********************************************************************** Please consider the environment before printing this e-mail -------------- next part -------------- An HTML attachment was scrubbed... URL: From apj at mutt.dk Wed May 9 07:45:25 2012 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Wed, 9 May 2012 09:45:25 +0200 Subject: Varnish 500's now not 302's In-Reply-To: References: Message-ID: <20120509074525.GL12685@nerd.dk> On Wed, May 09, 2012 at 01:11:16PM +1100, Nicholas_Maesepp at scee.net wrote: > > How I understand this is my server does the recv call, determines its a > lookup and then hashes as per my vcl, which is req,url and then server.ip > and then it does hit which is just to return deliver. Which should deliver > the object from cache, instead it gives me a 500. It does deliver from cache. The 500 is the object in the cache. -- Andreas From sparshgupta at gmail.com Wed May 9 12:40:41 2012 From: sparshgupta at gmail.com (Sparsh Gupta) Date: Wed, 9 May 2012 18:10:41 +0530 Subject: Varnish is restarting frequently In-Reply-To: <4FA984F3.6050500@antispam.it> References: <4FA984F3.6050500@antispam.it> Message-ID: Sorry for not including information on my swap before: On 9 May 2012 02:11, emilio brambilla wrote: > hello, > > > On 2012/05/08 8:02 PM, Sparsh Gupta wrote: > >> I have a bunch of Varnish servers (Exactly same configurations, caching >> different keys on Intel 2650 series Computer + 16GB RAM + 32GB SSD each). I >> am enclosing my configurations and settings along with my current >> varnishstats. >> >> I am looking for more debug information and how to find the route cause >> of my varnish getting restarting every 6-8 hours (all varnish servers) >> >> VARNISH_STORAGE_SIZE=32G >> VARNISH_STORAGE_MALLOC="**malloc" >> > if you have 16 Gb of phisical ram you should not malloc 32 Gb, but > something less than 16 Gb (usually 20% less than the phisical) > > Each box is configured with 24GB of swap. they have 16GB physical making them 40GB of available memory. I am using only 80% of it in Varnish as adviced. > how many swap did you set on these boxes? maybe the varnish is restarting > because the kernel cannot allocate the ram it want. > It seems Kernel is indeed killing the varnish daemon due to no memory available. May 9 12:27:51 localhost kernel: [168419.861130] Node 0 DMA: 2*4kB 1*8kB 1*16kB 2*32kB 2*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15840kB May 9 12:27:51 localhost kernel: [168419.926467] Node 0 DMA32: 119*4kB 218*8kB 62*16kB 19*32kB 5*64kB 5*128kB 2*256kB 2*512kB 1*1024kB 3*2048kB 3*4096kB = 25772kB May 9 12:27:51 localhost kernel: [168419.995910] Node 0 Normal: 910*4kB 0*8kB 0*16kB 0*32kB 0*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 4792kB May 9 12:27:51 localhost kernel: [168420.063296] Node 1 Normal: 494*4kB 0*8kB 3*16kB 0*32kB 1*64kB 5*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 7336kB May 9 12:27:51 localhost kernel: [168420.130742] 35438 total pagecache pages May 9 12:27:51 localhost kernel: [168420.154201] 32999 pages in swap cache May 9 12:27:51 localhost kernel: [168420.176624] Swap cache stats: add 31746386, delete 31713387, find 4761486/8157803 May 9 12:27:51 localhost kernel: [168420.221926] Free swap = 0kB May 9 12:27:51 localhost kernel: [168420.239681] Total swap = 25999352kB May 9 12:27:51 localhost kernel: [168420.328578] 4194288 pages RAM May 9 12:27:51 localhost kernel: [168420.346865] 83407 pages reserved May 9 12:27:51 localhost kernel: [168420.366696] 9039 pages shared May 9 12:27:51 localhost kernel: [168420.385006] 4091500 pages non-shared May 9 12:27:51 localhost kernel: [168420.406909] Out of memory: kill process 2984 (varnishd) score 103268864 or a child May 9 12:27:51 localhost kernel: [168420.452754] Killed process 2985 (varnishd) I am unable to figure out a lot of sense out of these numbers and will appreciate any help in fine tuning my varnish instance. I can try with 30G or 28G but what is an idea number if we can conclude something based on this info I found in my syslog > > if you want to use more than space you may use 'file' instead of 'malloc' > (but I don't know if it's a good idea to use SSD for the cache.... how many > rewrite does your ssd allow?) > > Before buying the boxes, I read that fast SSDs are good enough for Varnish. I got fast read write SSDs available these days on the servers > -- > bye, > emilio > > Thanks Sparsh > ______________________________**_________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/**lists/mailman/listinfo/**varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From varnish at ds.schledermann.net Wed May 9 13:07:09 2012 From: varnish at ds.schledermann.net (Daniel Schledermann) Date: Wed, 09 May 2012 15:07:09 +0200 Subject: Varnish is restarting frequently In-Reply-To: References: <4FA984F3.6050500@antispam.it> Message-ID: On Wed, 9 May 2012 18:10:41 +0530, Sparsh Gupta wrote: > >>> VARNISH_STORAGE_SIZE=32G >>> VARNISH_STORAGE_MALLOC="malloc" >> if you have 16 Gb of phisical ram you should not malloc 32 Gb, but >> something less than 16 Gb (usually 20% less than the phisical) > > Each box is configured with 24GB of swap. they have 16GB physical > making them 40GB of available memory. I am using only 80% of it in > Varnish as adviced. It would be more effective for you to use file-storage instead of malloc + swap. It does not matter wether you are using SSD or normal harddrives. > > ? > I am unable to figure out a lot of sense out of these numbers and > will > appreciate any help in fine tuning my varnish instance. I can try > with > 30G or 28G but what is an idea number if we can conclude something > based on this info I found in my syslog > if you want to use more than space you may use file instead of ma If you have 32G SSD you should cap the memory at around that, but use file storage. Also you had this in your DAEMON_OPTS: -p thread_pools=32 \ -p thread_pool_min=25 \ -p thread_pool_max=4000 \ This is waaaay to high. 2 or 4 threadpools is sufficient, and threadpool_max is also very high. It makes for a total of 128000 total maximum threads, which I very much doubt that you need. From sparshgupta at gmail.com Wed May 9 13:11:37 2012 From: sparshgupta at gmail.com (Sparsh Gupta) Date: Wed, 9 May 2012 18:41:37 +0530 Subject: Varnish is restarting frequently In-Reply-To: References: <4FA984F3.6050500@antispam.it> Message-ID: Sparsh Gupta On 9 May 2012 18:37, Daniel Schledermann wrote: > On Wed, 9 May 2012 18:10:41 +0530, Sparsh Gupta wrote: > >> >> VARNISH_STORAGE_SIZE=32G >>>> VARNISH_STORAGE_MALLOC="**malloc" >>>> >>> if you have 16 Gb of phisical ram you should not malloc 32 Gb, but >>> something less than 16 Gb (usually 20% less than the phisical) >>> >> >> Each box is configured with 24GB of swap. they have 16GB physical >> making them 40GB of available memory. I am using only 80% of it in >> Varnish as adviced. >> > > It would be more effective for you to use file-storage instead of malloc + > swap. It does not matter wether you are using SSD or normal harddrives. > By efficient do you also mean fast? Unfortunately the only thing I care is response times out of these boxes. I am happy to get more RAM if needed but got to have best possible performance/response times out of them (both both Hits, misses and passed queries). Are you sure file will be better than malloc + swap as far as speed / response times are concerned > > >> >> I am unable to figure out a lot of sense out of these numbers and will >> appreciate any help in fine tuning my varnish instance. I can try with >> 30G or 28G but what is an idea number if we can conclude something >> based on this info I found in my syslog >> if you want to use more than space you may use file instead of ma >> > > If you have 32G SSD you should cap the memory at around that, but use file > storage. > > Also you had this in your DAEMON_OPTS: > > > > -p thread_pools=32 \ > -p thread_pool_min=25 \ > -p thread_pool_max=4000 \ > > > This is waaaay to high. 2 or 4 threadpools is sufficient, and > threadpool_max is also very high. It makes for a total of 128000 total > maximum threads, which I very much doubt that you need. Thanks for the comments. I will try with lower values. What are the downsides of high values? My instances do around 4000req/second (max) and at times, a bunch of them reaches backend due to poor hit rate currently. I added extra threads to ensure they never choke the server, but if there is a downside, I will relook. Any way I can find out the best numbers for my instance (in my live environment, what should I look to find how to further tweak things) > > > > > ______________________________**_________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/**lists/mailman/listinfo/**varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From varnish at ds.schledermann.net Wed May 9 13:28:28 2012 From: varnish at ds.schledermann.net (Daniel Schledermann) Date: Wed, 09 May 2012 15:28:28 +0200 Subject: Varnish is restarting frequently In-Reply-To: References: <4FA984F3.6050500@antispam.it> Message-ID: On Wed, 9 May 2012 18:41:37 +0530, Sparsh Gupta wrote: >> It would be more effective for you to use file-storage instead of >> malloc + swap. It does not matter wether you are using SSD or normal >> harddrives. > > By efficient do you also mean fast? Unfortunately the only thing I > care is response times out of these boxes. I am happy to get more RAM > if needed but got to have best possible performance/response times > out > of them (both both Hits, misses and passed queries). Are you sure > file > will be better than malloc + swap as far as speed / response times > are > concerned Well.. there is a small performance gain by using malloc over file, provided that your cache will fit in RAM. Malloc+swap is slower than file. I'm quite sure of this. The precise effiency will vary from kernel to kernel. >> ? ? ? ? ? ? -p thread_pools=32 >> ? ? ? ? ? ? -p thread_pool_min=25 >> ? ? ? ? ? ? -p thread_pool_max=4000 >> >> This is waaaay to high. 2 or 4 threadpools is sufficient, and >> threadpool_max is also very high. It makes for a total of 128000 >> total maximum threads, which I very much doubt that you need. > > Thanks for the comments. I will try with lower values. What are the > downsides of high values? My instances do around 4000req/second (max) > and at times, a bunch of them reaches backend due to poor hit rate > currently. I added extra threads to ensure they never choke the > server, but if there is a downside, I will relook. Any way I can find > out the best numbers for my instance (in my live environment, what > should I look to find how to further tweak things) The old way of configuring threadpools was one per CPU-core. I believe however that it has been proven not to matter. 2 or 4 thread pools will be sufficient, regardless of the number of cores. Too many threads can chew up some RAM if you are doing expensive operations such as ESI. You can also have problems with exhausting your filedescriptors. Thats the main downside AFAIK. From glennerik at glennerik.com Wed May 9 13:36:05 2012 From: glennerik at glennerik.com (Glenn-Erik Sandbakken) Date: Wed, 9 May 2012 15:36:05 +0200 Subject: varnishd restarting randomly Message-ID: I am updating this thread to inform you that after I upgraded to varnish-3.0.2 revision 55e70a4, about two weeks ago, it's been running flawless. Thanks for a great product varnish-guys, keep it up =) - Glenn-Erik http://www.mp3lyrics.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From emilio at antispam.it Wed May 9 13:46:52 2012 From: emilio at antispam.it (emilio brambilla) Date: Wed, 09 May 2012 15:46:52 +0200 Subject: Varnish is restarting frequently In-Reply-To: References: <4FA984F3.6050500@antispam.it> Message-ID: <4FAA754C.3060705@antispam.it> hello, On 2012/05/09 2:40 PM, Sparsh Gupta wrote: > > Before buying the boxes, I read that fast SSDs are good enough for > Varnish. I got fast read write SSDs available these days on the servers I'm not concerned about performance of the SSD, but about the reliability: SSD has a limited number of rewrite cycle and you may hit this limit with very dynamic cache. note: I'm not saying you will hit the rewrite limit, I'm just asking if it may be a problem (I'm using malloc all-in-phisical-ram for all my varnish installation) -- bye, emilio From emilio at antispam.it Wed May 9 13:52:33 2012 From: emilio at antispam.it (emilio brambilla) Date: Wed, 09 May 2012 15:52:33 +0200 Subject: Varnish is restarting frequently In-Reply-To: References: <4FA984F3.6050500@antispam.it> Message-ID: <4FAA76A1.7090006@antispam.it> hello, On 2012/05/09 3:11 PM, Sparsh Gupta wrote: > By efficient do you also mean fast? Unfortunately the only thing I > care is response times out of these boxes. I am happy to get more RAM > if needed but got to have best possible performance/response times out > of them (both both Hits, misses and passed queries). Are you sure file > will be better than malloc + swap as far as speed / response times are > concerned with the same hw, if the cache does not fit in the physical memory, 'file' is better than malloc+swap. of course with 'file' your operating system will also use the physical ram as cache of the disk. if you can buy memory enough to fit all the cache in ram, than it's the best performance configuration (using malloc, of course for the storage), as ram is better than any disk (even ssd) for quick response time and transfer rate. -- bye, emilio From shahab371 at gmail.com Thu May 10 08:27:58 2012 From: shahab371 at gmail.com (shahab bakhtiyari) Date: Thu, 10 May 2012 10:27:58 +0200 Subject: policy Message-ID: Hi guys I have a question which is not exactly technical but more theoretical. I couldnt find it in documentations though. The question is, what kind of replacement algorithm (policy) the traffic server is using ? I mean is that recency based like LRU or frequency based like LFU or ... ? Would appreciate also links to documents if any ? thank you Shahab -------------- next part -------------- An HTML attachment was scrubbed... URL: From enno at tty.nl Thu May 10 09:04:07 2012 From: enno at tty.nl (Enno van Amerongen) Date: Thu, 10 May 2012 11:04:07 +0200 Subject: policy In-Reply-To: References: Message-ID: <4FAB8487.7040500@tty.nl> shahab bakhtiyari wrote: > Hi guys > > I have a question which is not exactly technical but more theoretical. > I couldnt find it in documentations though. > The question is, what kind of replacement algorithm (policy) the > traffic server is using ? I mean is that recency based like LRU or > frequency based like LFU or ... ? > > Would appreciate also links to documents if any ? > > thank you > Shahab > I think you want this link: https://www.varnish-cache.org/trac/wiki/ArchitectureLRU regards, Enno From lasse.karstensen at gmail.com Thu May 10 19:25:55 2012 From: lasse.karstensen at gmail.com (Lasse Karstensen) Date: Thu, 10 May 2012 21:25:55 +0200 Subject: Cookie set on beresp not present on next request In-Reply-To: <381EE28D-6B55-4183-A097-AD738FEF314F@gmail.com> References: <381EE28D-6B55-4183-A097-AD738FEF314F@gmail.com> Message-ID: <20120510192553.GA4120@yankee.samfundet.no> Torben Pedersen: > Hi, > I'm trying to use Varnish + a cookie to control access from mobile devices. I've seen other people doing similar, but I can't get it to work. [..] > Set-Cookie:nomobile=1; expires: Session; path=/; domain=. > This seems to be correct. So now I call > https:// > And I expect not to get redirected since the Set-Cookie is present. But the nomobile=1 is not present in request / Cookie. I don't quite get why ? > Any ideas ? The "domain=." part stands out a bit. Are you sure this is accepted by the browser? Is the Cookie included on the next request? As a reference you can check out the documentation for device detection in Varnish: https://www.varnish-cache.org/docs/trunk/tutorial/devicedetection.html https://github.com/varnish/varnish-devicedetect/ -- Lasse Karstensen From hugues.alary at gmail.com Thu May 10 19:31:59 2012 From: hugues.alary at gmail.com (Hugues Alary) Date: Thu, 10 May 2012 12:31:59 -0700 Subject: Cookie set on beresp not present on next request In-Reply-To: <381EE28D-6B55-4183-A097-AD738FEF314F@gmail.com> References: <381EE28D-6B55-4183-A097-AD738FEF314F@gmail.com> Message-ID: On Tue, May 8, 2012 at 4:00 AM, Torben Pedersen wrote: > Hi, > Hi, > > I'm trying to use Varnish + a cookie to control access from mobile > devices. I've seen other people doing similar, but I can't get it to work. > > I'm accessing my solution by > > https://?mobile_redirect=0 > You're talking about accessing your server via https. Is it a mistake or are you really accessing it via HTTPS? If so, that's your problem. Varnish does not support HTTPS. Your request should not even reach varnish. -Hugues -------------- next part -------------- An HTML attachment was scrubbed... URL: From torbped at gmail.com Thu May 10 19:53:47 2012 From: torbped at gmail.com (Torben Pedersen) Date: Thu, 10 May 2012 21:53:47 +0200 Subject: Cookie set on beresp not present on next request In-Reply-To: <20120510192553.GA4120@yankee.samfundet.no> References: <381EE28D-6B55-4183-A097-AD738FEF314F@gmail.com> <20120510192553.GA4120@yankee.samfundet.no> Message-ID: <4D1AF5BF-8966-4AEC-8639-4EC33D7A36C4@gmail.com> On 10/05/2012, at 21.25, Lasse Karstensen wrote: > Torben Pedersen: >> Hi, >> I'm trying to use Varnish + a cookie to control access from mobile devices. I've seen other people doing similar, but I can't get it to work. > [..] >> Set-Cookie:nomobile=1; expires: Session; path=/; domain=. >> This seems to be correct. So now I call >> https:// >> And I expect not to get redirected since the Set-Cookie is present. But the nomobile=1 is not present in request / Cookie. I don't quite get why ? >> Any ideas ? > > The "domain=." part stands out a bit. Are you sure this is accepted by the > browser? Is the Cookie included on the next request? > > As a reference you can check out the documentation for device detection > in Varnish: > https://www.varnish-cache.org/docs/trunk/tutorial/devicedetection.html > https://github.com/varnish/varnish-devicedetect/ > > -- > Lasse Karstensen > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc Hi, You're most likely correct Lasse. Now I've changed the solution a bit - because it wouldn't work since I was trying to delete the cookie. Now I'm changing the value of the cookie instead - for controlling access (thanks to Daniel S) (I'm using nginx in front of Varnish in this solution for the https) Thanks for all the replies. kind regards Torben From tobias.eichelbroenner at lamp-solutions.de Thu May 10 19:54:15 2012 From: tobias.eichelbroenner at lamp-solutions.de (=?ISO-8859-15?Q?Tobias_Eichelbr=F6nner?=) Date: Thu, 10 May 2012 21:54:15 +0200 Subject: purge content after POST request Message-ID: <4FAC1CE7.3040005@lamp-solutions.de> Hi, on a webpage behind varnish, visitors can leave comments. New comments are sent to the server via htmlform using POST. The Problem is, that new comments are not visible until I clear the cache. I tried to purge or ban some content from cache, after a post request is sent. For example like this: sub vcl_recv { if (req.request == "POST") { ban("req.http.host == " + req.http.host + "&& req.url == " + req.url); return (pass); } } or this: sub vcl_fetch { if (req.request == "POST") { ban("req.http.host == " + req.http.host + "&& req.url == " + req.url); set beresp.ttl = 0h; } } or even this: sub vcl_hit { if (req.request == "POST") { purge; } return (deliver); } Does anyone solved a problem similar to mine? Sincerly, Tobias From Nicholas_Maesepp at scee.net Thu May 10 22:30:52 2012 From: Nicholas_Maesepp at scee.net (Nicholas_Maesepp at scee.net) Date: Fri, 11 May 2012 09:30:52 +1100 Subject: Varnish 500's now not 302's In-Reply-To: <20120509074525.GL12685@nerd.dk> References: <20120509074525.GL12685@nerd.dk> Message-ID: I guess technically I realize this is from the cache, my question is more along the lines of what could be causing my objects in the cache to become stop being delivered as is and having a 500 error returned. Is my cache being corrupted in some way when varnish starts to max out the malloc'ed RAM? Seems like a bug to me but can't pinpoint exactly what causes it, just that it happens after X amount of time to any of my instances. Cheers Nick From: Andreas Plesner Jacobsen To: varnish-misc at varnish-cache.org Date: 05/09/2012 05:47 PM Subject: Re: Varnish 500's now not 302's Sent by: varnish-misc-bounces at varnish-cache.org On Wed, May 09, 2012 at 01:11:16PM +1100, Nicholas_Maesepp at scee.net wrote: > > How I understand this is my server does the recv call, determines its a > lookup and then hashes as per my vcl, which is req,url and then server.ip > and then it does hit which is just to return deliver. Which should deliver > the object from cache, instead it gives me a 500. It does deliver from cache. The 500 is the object in the cache. -- Andreas _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc Sony Computer Entertainment Australia Pty Ltd Level 1, 63-73 Ann Street Surry Hills NSW 2010 P.O. Box 5023 Darlinghurst NSW 2010 ph: +61 (0)2 9324 9500 fax: +61 (0)2 9324 9558 http://au.playstation.com http://www.facebook.com/PlayStationAU THE WORLD IS IN PLAY. PlayStation?Vita has arrived. http://www.psvita.com ********************************************************************** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify postmaster at scee.net This footnote also confirms that this email message has been checked for all known viruses. Sony Computer Entertainment Australia Pty. Limited Registered Office: Level 1, 63-73 Ann Street, Surry Hills, NSW 2010 Australia Registered in Australia: 077 583 183 ********************************************************************** Please consider the environment before printing this e-mail -------------- next part -------------- An HTML attachment was scrubbed... URL: From tungcnw at gmail.com Fri May 11 05:06:11 2012 From: tungcnw at gmail.com (Dang Tung) Date: Fri, 11 May 2012 13:06:11 +0800 Subject: Varnish return 503 Message-ID: Dear all, We get 503 page when using varnish 2.1.4, when I check log I see this line: FetchError - http first read error: -1 11 (Resource temporarily unavailable) Do you have any ideas, please have. I also attach my configure. -- Nguyen Dang Tung -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: default.vcl Type: application/octet-stream Size: 4100 bytes Desc: not available URL: From apj at mutt.dk Fri May 11 08:40:52 2012 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Fri, 11 May 2012 10:40:52 +0200 Subject: Varnish 500's now not 302's In-Reply-To: References: <20120509074525.GL12685@nerd.dk> Message-ID: <20120511084052.GM12685@nerd.dk> On Fri, May 11, 2012 at 09:30:52AM +1100, Nicholas_Maesepp at scee.net wrote: > I guess technically I realize this is from the cache, my question is more > along the lines of what could be causing my objects in the cache to become > stop being delivered as is and having a 500 error returned. Is my cache > being corrupted in some way when varnish starts to max out the malloc'ed > RAM? Seems like a bug to me but can't pinpoint exactly what causes it, > just that it happens after X amount of time to any of my instances. Then you misunderstood me. The 500 is a cached object that your backend served at the time. You can avoid caching such objects by inspecting the HTTP response in vcl_fetch and setting TTL and/or using hit for pass and/or saint mode. It's almost certainly not a bug. -- Andreas From hugues.alary at gmail.com Fri May 11 17:27:27 2012 From: hugues.alary at gmail.com (Hugues Alary) Date: Fri, 11 May 2012 10:27:27 -0700 Subject: Varnish 500's now not 302's In-Reply-To: <20120511084052.GM12685@nerd.dk> References: <20120509074525.GL12685@nerd.dk> <20120511084052.GM12685@nerd.dk> Message-ID: At this point, sending your configuration file would maybe be a good idea. On May 11, 2012 1:41 AM, "Andreas Plesner Jacobsen" wrote: > On Fri, May 11, 2012 at 09:30:52AM +1100, Nicholas_Maesepp at scee.net wrote: > > > I guess technically I realize this is from the cache, my question is more > > along the lines of what could be causing my objects in the cache to > become > > stop being delivered as is and having a 500 error returned. Is my cache > > being corrupted in some way when varnish starts to max out the malloc'ed > > RAM? Seems like a bug to me but can't pinpoint exactly what causes it, > > just that it happens after X amount of time to any of my instances. > > Then you misunderstood me. The 500 is a cached object that your backend > served > at the time. You can avoid caching such objects by inspecting the HTTP > response > in vcl_fetch and setting TTL and/or using hit for pass and/or saint mode. > > It's almost certainly not a bug. > > -- > Andreas > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim.hayter at gmail.com Fri May 11 18:47:51 2012 From: jim.hayter at gmail.com (Jim Hayter) Date: Fri, 11 May 2012 14:47:51 -0400 Subject: Cache reload - is this the right way to do this? Message-ID: Varnish 3.0.0 on Ubuntu 11.10. I am trying to modify my vcl to provide a way to load/update cached pages. My idea is to check for a specific IP and User-Agent in vcl_recv and return pass for a match. This should by-pass the cache and go to the back end to obtain the object. This seems to work but then my object is not ending up in the cache. If a previous object for the request is in cache, it remains in cache and is returned for subsequent requests. If the object is not in cache, the above will not add it to the cache. Am I missing something or asking varnish to do something it won't by design? Is there another way to accomplish a forced retrieval of a cached object and updating the cache with the new object without banning the object? Comments and/or pointers to appropriate doc welcomed. Thanks, Jim -------------- next part -------------- An HTML attachment was scrubbed... URL: From bedis9 at gmail.com Fri May 11 20:48:59 2012 From: bedis9 at gmail.com (Baptiste) Date: Fri, 11 May 2012 22:48:59 +0200 Subject: Varnish return 503 In-Reply-To: References: Message-ID: Hi, Your server may be down and you may not have a probe in your configuration. cheers On Fri, May 11, 2012 at 7:06 AM, Dang Tung wrote: > Dear all, > > We get 503 page when using varnish 2.1.4, when I check log I see this line: > FetchError?? - http first read error: -1 11 (Resource temporarily > unavailable) > > Do you have any ideas, please have. > > I also attach my configure. > > -- > Nguyen Dang Tung > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From tungcnw at gmail.com Sat May 12 07:28:05 2012 From: tungcnw at gmail.com (Dang Tung) Date: Sat, 12 May 2012 14:28:05 +0700 Subject: Varnish return 503 In-Reply-To: References: Message-ID: My backend server it's ok, I think this issue come from varnish. I will add probe to check again, but any ideas about this case? On Sat, May 12, 2012 at 3:48 AM, Baptiste wrote: > Hi, > > Your server may be down and you may not have a probe in your configuration. > > cheers > > On Fri, May 11, 2012 at 7:06 AM, Dang Tung wrote: > > Dear all, > > > > We get 503 page when using varnish 2.1.4, when I check log I see this > line: > > FetchError - http first read error: -1 11 (Resource temporarily > > unavailable) > > > > Do you have any ideas, please have. > > > > I also attach my configure. > > > > -- > > Nguyen Dang Tung > > > > _______________________________________________ > > varnish-misc mailing list > > varnish-misc at varnish-cache.org > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- Nguyen Dang Tung -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon.hauge at relationhouse.com Sun May 13 18:30:11 2012 From: jon.hauge at relationhouse.com (Jon Hauge) Date: Sun, 13 May 2012 20:30:11 +0200 Subject: Multiple DAEMON_OPTS -p paramter Message-ID: Can somebody help me debug this ?p parameter: DAEMON_OPTS="-a :80 \ -f /etc/varnish/viasat_default.vcl \ -S /etc/varnish/secret \ -s malloc,4G \ -p send_timeout=600, max_restarts=9" I am not sure if it?s the comma that causes varnish not to start. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cosimo at streppone.it Sun May 13 18:34:53 2012 From: cosimo at streppone.it (Cosimo Streppone) Date: Sun, 13 May 2012 20:34:53 +0200 Subject: Multiple DAEMON_OPTS -p paramter In-Reply-To: References: Message-ID: On Sun, 13 May 2012 20:30:11 +0200, Jon Hauge wrote: > Can somebody help me debug this ?p parameter: > > DAEMON_OPTS="-a :80 \ > -f /etc/varnish/viasat_default.vcl \ > -S /etc/varnish/secret \ > -s malloc,4G \ > -p send_timeout=600, max_restarts=9" Try this: DAEMON_OPTS="-a :80 \ -f /etc/varnish/viasat_default.vcl \ -S /etc/varnish/secret \ -s malloc,4G \ -p send_timeout=600 \ -p max_restarts=9" -- Cosimo From javier at casares.org Sun May 13 18:34:46 2012 From: javier at casares.org (Javier Casares) Date: Sun, 13 May 2012 20:34:46 +0200 Subject: Multiple DAEMON_OPTS -p paramter In-Reply-To: References: Message-ID: My sysconfig file lately is like this: NFILES=102400 MEMLOCK=131072 DAEMON_COREFILE_LIMIT="unlimited" DAEMON_OPTS="-a 000.000.000.000:80 \ -T 000.000.000.000:6082 \ -f /etc/varnish/default.vcl \ -S /etc/varnish/secret \ -u varnish -g varnish \ -t 300 \ -w 100,4000,60 \ -p between_bytes_timeout=60 \ -p connect_timeout=5 \ -p default_grace=1800 \ -p first_byte_timeout=30 \ -p session_max=10240 \ -s malloc,1G \ -s file,/var/lib/varnish/varnish_storage.bin,20G" Javier Casares http://javiercasares.com/ 2012/5/13 Jon Hauge : > Can somebody help me debug this ?p parameter: > > > > DAEMON_OPTS="-a :80? \ > > ???????????? -f /etc/varnish/viasat_default.vcl \ > > ???????????? -S /etc/varnish/secret \ > > ???????????? -s malloc,4G \ > > -p send_timeout=600, max_restarts=9" > > > > I am not sure if it?s the comma that causes varnish not to start. > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From Nicholas_Maesepp at scee.net Mon May 14 00:20:23 2012 From: Nicholas_Maesepp at scee.net (Nicholas_Maesepp at scee.net) Date: Mon, 14 May 2012 11:20:23 +1100 Subject: Varnish 500's now not 302's In-Reply-To: References: <20120509074525.GL12685@nerd.dk> <20120511084052.GM12685@nerd.dk> Message-ID: Nothing special about my config file apart from the hashing of objects, I do have the following already in my vcl_fetch: set beresp.grace = 3m; if (beresp.status == 500) { set beresp.saintmode = 5s; } ====== vcl_recv { set req.grace = 40s; if (req.request != "GET" && req.request != "HEAD") { /* We only deal with GET and HEAD by default */ return (pass); } if (req.http.Accept-Encoding && req.url ~ "\.(png|gif|jpg|swf|mov|flv)$") { remove req.http.Accept-Encoding; } return (lookup); I'm not convinced that I'm caching 500 responses on all these objects, I do wonder if its because I remove accept-encoding on those objects, those are the ones that end up with 500's. But also maybe my saintmode and grace numbers are too low if the back-end is infact serving 500's. But at least it's an avenue to explore and get some evidence to prove or disprove it. Will also try using hit for pass instead and see if that gives me more resilience. Cheers Nick From: Hugues Alary To: Andreas Plesner Jacobsen Cc: varnish-misc at varnish-cache.org Date: 05/12/2012 03:29 AM Subject: Re: Varnish 500's now not 302's Sent by: varnish-misc-bounces at varnish-cache.org At this point, sending your configuration file would maybe be a good idea. On May 11, 2012 1:41 AM, "Andreas Plesner Jacobsen" wrote: On Fri, May 11, 2012 at 09:30:52AM +1100, Nicholas_Maesepp at scee.net wrote: > I guess technically I realize this is from the cache, my question is more > along the lines of what could be causing my objects in the cache to become > stop being delivered as is and having a 500 error returned. Is my cache > being corrupted in some way when varnish starts to max out the malloc'ed > RAM? Seems like a bug to me but can't pinpoint exactly what causes it, > just that it happens after X amount of time to any of my instances. Then you misunderstood me. The 500 is a cached object that your backend served at the time. You can avoid caching such objects by inspecting the HTTP response in vcl_fetch and setting TTL and/or using hit for pass and/or saint mode. It's almost certainly not a bug. -- Andreas Sony Computer Entertainment Australia Pty Ltd Level 1, 63-73 Ann Street Surry Hills NSW 2010 P.O. Box 5023 Darlinghurst NSW 2010 ph: +61 (0)2 9324 9500 fax: +61 (0)2 9324 9558 http://au.playstation.com http://www.facebook.com/PlayStationAU THE WORLD IS IN PLAY. PlayStation?Vita has arrived. http://www.psvita.com ********************************************************************** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify postmaster at scee.net This footnote also confirms that this email message has been checked for all known viruses. Sony Computer Entertainment Australia Pty. Limited Registered Office: Level 1, 63-73 Ann Street, Surry Hills, NSW 2010 Australia Registered in Australia: 077 583 183 ********************************************************************** Please consider the environment before printing this e-mail -------------- next part -------------- An HTML attachment was scrubbed... URL: From shahab371 at gmail.com Mon May 14 00:29:30 2012 From: shahab371 at gmail.com (shahab bakhtiyari) Date: Mon, 14 May 2012 02:29:30 +0200 Subject: policy In-Reply-To: <4FAB8487.7040500@tty.nl> References: <4FAB8487.7040500@tty.nl> Message-ID: Hi again The link does not say clearly what policies varnish uses rather discusses about problems implementing LRU .. does it mean that varnish is using LRU and only LRU for both Ram and disk object replacements? --Shahab On 10 May 2012 11:04, Enno van Amerongen wrote: > shahab bakhtiyari wrote: > > Hi guys > > > > I have a question which is not exactly technical but more theoretical. > > I couldnt find it in documentations though. > > The question is, what kind of replacement algorithm (policy) the > > traffic server is using ? I mean is that recency based like LRU or > > frequency based like LFU or ... ? > > > > Would appreciate also links to documents if any ? > > > > thank you > > Shahab > > > I think you want this link: > > https://www.varnish-cache.org/trac/wiki/ArchitectureLRU > > regards, > > Enno > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hugues.alary at gmail.com Mon May 14 01:11:22 2012 From: hugues.alary at gmail.com (Hugues Alary) Date: Sun, 13 May 2012 18:11:22 -0700 Subject: Varnish 500's now not 302's In-Reply-To: References: <20120509074525.GL12685@nerd.dk> <20120511084052.GM12685@nerd.dk> Message-ID: Are you specifically not caching 500 responses in your config file? Something like: if(beresp.status == 500) { return (hit_for_pass); } If you're not doing that, then you *are* caching 500 responses. On Sun, May 13, 2012 at 5:20 PM, wrote: > Nothing special about my config file apart from the hashing of objects, I > do have the following already in my vcl_fetch: > > set beresp.grace = 3m; > if (beresp.status == 500) { > set beresp.saintmode = 5s; > } > > ====== > vcl_recv { > set req.grace = 40s; > if (req.request != "GET" && req.request != "HEAD") { > /* We only deal with GET and HEAD by default */ > return (pass); > } > if (req.http.Accept-Encoding && req.url ~ "\.(png|gif|jpg|swf|mov|flv)$") { > remove req.http.Accept-Encoding; > } > return (lookup); > > I'm not convinced that I'm caching 500 responses on all these objects, I > do wonder if its because I remove accept-encoding on those objects, those > are the ones that end up with 500's. But also maybe my saintmode and grace > numbers are too low if the back-end is infact serving 500's. But at least > it's an avenue to explore and get some evidence to prove or disprove it. > Will also try using hit for pass instead and see if that gives me more > resilience. > > > Cheers > > Nick > > > > > From: Hugues Alary > To: Andreas Plesner Jacobsen > Cc: varnish-misc at varnish-cache.org > Date: 05/12/2012 03:29 AM > Subject: Re: Varnish 500's now not 302's > Sent by: varnish-misc-bounces at varnish-cache.org > ------------------------------ > > > > At this point, sending your configuration file would maybe be a good idea. > > On May 11, 2012 1:41 AM, "Andreas Plesner Jacobsen" <*apj at mutt.dk*> > wrote: > On Fri, May 11, 2012 at 09:30:52AM +1100, *Nicholas_Maesepp at scee.net*wrote: > > > I guess technically I realize this is from the cache, my question is more > > along the lines of what could be causing my objects in the cache to > become > > stop being delivered as is and having a 500 error returned. Is my cache > > being corrupted in some way when varnish starts to max out the malloc'ed > > RAM? Seems like a bug to me but can't pinpoint exactly what causes it, > > just that it happens after X amount of time to any of my instances. > > Then you misunderstood me. The 500 is a cached object that your backend > served > at the time. You can avoid caching such objects by inspecting the HTTP > response > in vcl_fetch and setting TTL and/or using hit for pass and/or saint mode. > > It's almost certainly not a bug. > > -- > Andreas > > > * > * > > * > Sony Computer Entertainment Australia Pty Ltd* > Level 1, 63-73 Ann Street Surry Hills NSW 2010 > P.O. Box 5023 Darlinghurst NSW 2010 > ph: +61 (0)2 9324 9500 fax: +61 (0)2 9324 9558 > > *http://au.playstation.com* * > **http://www.facebook.com/PlayStationAU* > > > * > THE WORLD IS IN PLAY.* > PlayStation?Vita has arrived. > http://www.psvita.com > > * > *[image: PSMOVE] > > ********************************************************************** > This email and any files transmitted with it are confidential and intended > solely for the use of the individual or entity to whom they are addressed. > If you have received this email in error please notify postmaster at scee.net > This footnote also confirms that this email message has been checked for > all known viruses. > Sony Computer Entertainment Australia Pty. Limited > Registered Office: Level 1, 63-73 Ann Street, Surry Hills, NSW 2010 > Australia > Registered in Australia: 077 583 183 > ********************************************************************** > * > Please consider the environment before printing this e-mail* > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- Hugues ALARY -------------- next part -------------- An HTML attachment was scrubbed... URL: From apj at mutt.dk Mon May 14 06:13:42 2012 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Mon, 14 May 2012 08:13:42 +0200 Subject: Varnish 500's now not 302's In-Reply-To: References: <20120509074525.GL12685@nerd.dk> <20120511084052.GM12685@nerd.dk> Message-ID: <20120514061342.GN12685@nerd.dk> On Mon, May 14, 2012 at 11:20:23AM +1100, Nicholas_Maesepp at scee.net wrote: > Nothing special about my config file apart from the hashing of objects, I > do have the following already in my vcl_fetch: > > set beresp.grace = 3m; So you're at least caching everything for 3 minutes. > if (beresp.status == 500) { > set beresp.saintmode = 5s; > } And at least using the cached version of 500s for 5 seconds (assuming a single backend). It shouldn't be too hard to use varnishlog to figure out what happens to these objects. Try something like varnishlog -cm TxStatus:500 -- Andreas From apj at mutt.dk Mon May 14 07:14:34 2012 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Mon, 14 May 2012 09:14:34 +0200 Subject: Varnish 500's now not 302's In-Reply-To: <20120514061342.GN12685@nerd.dk> References: <20120509074525.GL12685@nerd.dk> <20120511084052.GM12685@nerd.dk> <20120514061342.GN12685@nerd.dk> Message-ID: <20120514071434.GA14047@nerd.dk> On Mon, May 14, 2012 at 08:13:42AM +0200, Andreas Plesner Jacobsen wrote: > > > if (beresp.status == 500) { > > set beresp.saintmode = 5s; > > } > > And at least using the cached version of 500s for 5 seconds (assuming a single backend). > > It shouldn't be too hard to use varnishlog to figure out what happens to these > objects. Try something like varnishlog -cm TxStatus:500 I just found your original mail, and in that you have varnishlog of a hit to the cache. What is interesting is finding the original request that put the page in the cache, but from your original post, it's evident, that you've set a high TTL on the 500 object (Age: 33236). In other words: You probably have an unconditional set beresp.ttl = somewhere, and that's why varnish will cache your 500. You should also consider restarting the request after setting saint mode. If you don't, the request will just be added to the backend's saint mode list, but the response will still be served to the client. -- Andreas From nbubingo at gmail.com Mon May 14 07:15:18 2012 From: nbubingo at gmail.com (=?GB2312?B?0qbOsLHz?=) Date: Mon, 14 May 2012 15:15:18 +0800 Subject: Varnishd and the disk IO Message-ID: Hello, folks, I have a question. Does the varnish often sync the shared memory file named "_.vsm" with the disk? I saw many disk IOs with varnish: [wenjing.ywb at i4i-013050.cm6 dstat]$ ./dstat -c --top-cpu -d --top-bio ----total-cpu-usage---- -most-expensive- -dsk/total- ----most-expensive---- usr sys idl wai hiq siq| cpu process | read writ| block i/o process 9 2 89 0 0 1|ksoftirqd/1 0.0|3181k 6879k|bash 0 0 19 8 66 2 0 5|varnishd 27| 0 14M| 18 7 71 0 0 5|varnishd 23| 0 24k| 19 8 67 0 0 5|varnishd 26| 0 13M| 16 7 72 0 0 5|varnishd 23| 0 19M| 19 7 69 0 0 5|varnishd 25| 0 5288k| 18 7 70 0 0 5|varnishd 24| 0 2684k| My server processed 1K-2K requests per second. I use the malloc memory cache method. It seems varnish writes 10M-20M data to disk per three second. Varnish start command: /opt/com/install/varnish/sbin/varnishd -P /home/admin/varnish/pid/varnish.pid -a :8888 -f /home/admin/varnish/conf/varnish.vcl -n /home/admin/varnish/cache -T 0.0.0.0:2000 -t 120 -w 5,1000,120 -u admin -g admin -s malloc,50G My OS: Linux i4i-013050.cm6 2.6.32-131.12.1.tb332.el6.x86_64 #1 SMP Mon Oct 31 14:46:57 CST 2011 x86_64 x86_64 x86_64 GNU/Linux Thanks. From perbu at varnish-software.com Mon May 14 08:17:21 2012 From: perbu at varnish-software.com (Per Buer) Date: Mon, 14 May 2012 10:17:21 +0200 Subject: Varnishd and the disk IO In-Reply-To: References: Message-ID: Hi On Mon, May 14, 2012 at 9:15 AM, ??? wrote: > Hello, folks, > > I have a question. Does the varnish often sync the shared memory file > named "_.vsm" with the disk? I saw many disk IOs with varnish: > No. But Linux does sync the shmlog to disk as Linux hasn't properly implemented the flag that keeps stuff off the disk. You can try to either move the shmlog to a memory based filesystem or tune /proc/sys/vm/*dirty* and friends. Modern Linux distros seem to prefer syncing stuff to disk very aggresivly. -- Per Buer, Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Mon May 14 08:20:57 2012 From: perbu at varnish-software.com (Per Buer) Date: Mon, 14 May 2012 10:20:57 +0200 Subject: policy In-Reply-To: References: <4FAB8487.7040500@tty.nl> Message-ID: Hi. On Mon, May 14, 2012 at 2:29 AM, shahab bakhtiyari wrote: > Hi again > > The link does not say clearly what policies varnish uses rather discusses > about problems implementing LRU .. does it mean that varnish is using LRU > and only LRU for both Ram and disk object replacements? > You're quoting wrong. Varnish doesn't know what is on disk and what is in memory. Varnish maintains one global LRU list. Actually I think it keeps one LRU list for each stevedore. So you could tweak stuff somewhat by sending objects to various stevedores. You can set beresp.storage in vcl_fetch to pick the backend. -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From arne.mohr at s24.com Mon May 14 15:38:56 2012 From: arne.mohr at s24.com (Arne Mohr) Date: Mon, 14 May 2012 17:38:56 +0200 Subject: Not able to run multiple varnish instances on single maschine - Pushing vcls failed:#012CLI communication error (hdr) Message-ID: Hi folks, i have a problem when i try to run multiple varnish instances on a single maschine. I have 3 varnishes running on Port 80,81,82 the starting parameters are: /usr/sbin/varnishd -P /var/run/varnish-www.portal-a.de.pid -a 10.10.145.41:80 -T localhost:6080 -f /etc/varnish/www.portal-a.de.vcl -S /etc/varnish/secret -u varnish -g varnish -t 28800 -w 10,1000 -n www.portal-a.de -i frontend -p thread_pools 4 -p thread_pool_max 1024 -p thread_pool_min 250 -p listen_depth 2048 -p thread_pool_add_delay 2 -s malloc,1G /usr/sbin/varnishd -P /var/run/varnish-www.portal-b.de.pid -a 10.10.145.41:81 -T localhost:6081 -f /etc/varnish/www.portal-b.de.vcl -S /etc/varnish/secret -u varnish -g varnish -t 28800 -w 10,1000 -n www.portal-b.de -i frontend -p thread_pools 4 -p thread_pool_max 1024 -p thread_pool_min 250 -p listen_depth 2048 -p thread_pool_add_delay 2 -s malloc,1G /usr/sbin/varnishd -P /var/run/varnish-www.portal-c.de.pid -a 10.10.145.41:82 -T localhost:6082 -f /etc/varnish/www.portal-c.de.vcl -S /etc/varnish/secret -u varnish -g varnish -t 28800 -w 10,1000 -n www.portal-c.de -i frontend -p thread_pools 4 -p thread_pool_max 1024 -p thread_pool_min 250 -p listen_depth 2048 -p thread_pool_add_delay 2 -s malloc,1G The first varnish ist starting up correctly and is also able to process the incoming requests. Also it's child process is forked in a correct way. The problem I have occurs when I try to start the second varnish. I found following log entries in my syslog May 14 14:12:51 webcache01 www.portal-b.de[10021]: Platform: Linux,2.6.32-220.4.2.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit May 14 14:12:51 webcache01 www.portal-b.de[10021]: child (10022) Started May 14 14:12:51 webcache01 www.portal-b.de[10021]: Pushing vcls failed:#012CLI communication error (hdr) May 14 14:12:51 webcache01 www.portal-b.de[10021]: Child (10022) died signal=6 May 14 14:12:51 webcache01 www.portal-b.de[10021]: Child (-1) said Missing errorhandling code in mgt_sandbox(), mgt_sandbox.c line 71: May 14 14:12:51 webcache01 www.portal-b.de[10021]: Child (-1) said Condition((setuid(params->uid)) == 0) not true. May 14 14:12:51 webcache01 www.portal-b.de[10021]: Child (-1) said errno = 11 (Resource temporarily unavailable) At the moment I'm stuck with my problem and I'm not able to find whether a solution nor a starting point. Maybe anyone had a similiar problem and has some tips for me? With best regards! -- Arne Mohr ? Software-Entwickler Backend Development shopping24 internet group Wandsbeker Stra?e 3-7 ? 22172 Hamburg Telefon: +49 (0) 40 6461 8986 ? Fax: +49 (0) 40 6461 7879 arne.mohr at s24.com ? www.s24.com AG Hamburg HRB 63371 vertreten durch Dr. Bj?rn Sch?fers und Martin Mildner -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim.hayter at gmail.com Mon May 14 15:42:39 2012 From: jim.hayter at gmail.com (Jim Hayter) Date: Mon, 14 May 2012 11:42:39 -0400 Subject: Cache reload - is this the right way to do this? Message-ID: On Sat, May 12, 2012 at 6:00 AM, I wrote: > Varnish 3.0.0 on Ubuntu 11.10. > > I am trying to modify my vcl to provide a way to load/update cached pages. > > My idea is to check for a specific IP and User-Agent in vcl_recv and return > pass for a match. ?This should by-pass the cache and go to the back end to > obtain the object. ?This seems to work but then my object is not ending up > in the cache. > > If a previous object for the request is in cache, it remains in cache and > is returned for subsequent requests. ?If the object is not in cache, the > above will not add it to the cache. > > Am I missing something or asking varnish to do something it won't by > design? ?Is there another way to accomplish a forced retrieval of a cached > object and updating the cache with the new object without banning the > object? > > Comments and/or pointers to appropriate doc welcomed. > Just to update this, I found one answer. ?It is to set req.hash_always_miss to true. ?See https://www.varnish-cache.org/trac/wiki/VCLExampleEnableForceRefresh. After some testing, this seems to be exactly what I was looking for. Jim From richard at zuidhof.nl Tue May 15 14:14:38 2012 From: richard at zuidhof.nl (Richard Zuidhof) Date: Tue, 15 May 2012 16:14:38 +0200 Subject: More objects in malloc storage? Message-ID: Due to a bug in 3.0.0 or 3.0.1 we had to switch back to file storage instead of malloc last summer. Until now I had not bothered to change it back after upgrading to 3.0.2 because I thought the increase in efficiency would not matter too much for a relatively small server. To my surprise there was an unexpected side effect. After I switched the config from 2G file to 1500M malloc the number of objects in the Varnish cache increased from 58000 to 115000. Less memory, more objects.Is this by design? Can anyone explain this? Kind regards, Richard Zuidhof -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Tue May 15 18:53:30 2012 From: perbu at varnish-software.com (Per Buer) Date: Tue, 15 May 2012 20:53:30 +0200 Subject: More objects in malloc storage? In-Reply-To: References: Message-ID: Hi On Tue, May 15, 2012 at 4:14 PM, Richard Zuidhof wrote: > Due to a bug in 3.0.0 or 3.0.1 we had to switch back to file storage > instead of malloc last summer. Until now I had not bothered to change it > back after upgrading to 3.0.2 because I thought the increase in efficiency > would not matter too much for a relatively small server. To my surprise > there was an unexpected side effect. After I switched the config from 2G > file to 1500M malloc the number of objects in the Varnish cache increased > from 58000 to 115000. Less memory, more objects.Is this by design? Can > anyone explain this? > -sfile put objects at 4k boundaries if I recall correctly. If you have many small objects the overhead is significant. -- Per Buer Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From rickytato at r2consulting.it Tue May 15 20:41:25 2012 From: rickytato at r2consulting.it (rickytato rickytato) Date: Tue, 15 May 2012 22:41:25 +0200 Subject: Varnishd and the disk IO In-Reply-To: References: Message-ID: I use a simple tmpfs and work well: # Varnish tmpfs /usr/local/var/varnish tmpfs rw,size=256M 0 0 2012/5/14 Per Buer > Hi > > On Mon, May 14, 2012 at 9:15 AM, ??? wrote: > >> Hello, folks, >> >> I have a question. Does the varnish often sync the shared memory file >> named "_.vsm" with the disk? I saw many disk IOs with varnish: >> > > No. But Linux does sync the shmlog to disk as Linux hasn't properly > implemented the flag that keeps stuff off the disk. You can try to either > move the shmlog to a memory based filesystem or tune /proc/sys/vm/*dirty* > and friends. Modern Linux distros seem to prefer syncing stuff to disk very > aggresivly. > > -- > Per Buer, > Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer > *Varnish makes websites fly!* > Whitepapers | Video | > Twitter > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From n.j.saunders at gmail.com Wed May 16 14:58:22 2012 From: n.j.saunders at gmail.com (Neil Saunders) Date: Wed, 16 May 2012 15:58:22 +0100 Subject: Changing the Varnish Date header to UTC Message-ID: Hi all - Hopefully a quick one - Is there any way to get varnish to provide the date header in UTC instead of GMT? neil at Neils-MacBook-Pro:~$ curl -v http://www.zeebox.com < HTTP/1.1 301 Moved Permanently < Accept-Ranges: bytes < Age: 117 < Content-Type: text/html < Date: Wed, 16 May 2012 14:55:38 GMT <<<<<<<<<<<<<<<< < Server: nginx/0.7.65 < Via: 1.1 varnish < X-Varnish: 1342116745 1342115968 < X-Zeebox-Backend: live_ew1b_web4 < Content-Length: 185 < Connection: keep-alive Thanks! - Neil -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Wed May 16 18:04:23 2012 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 16 May 2012 18:04:23 +0000 Subject: Changing the Varnish Date header to UTC In-Reply-To: Your message of "Wed, 16 May 2012 15:58:22 +0100." Message-ID: <1666.1337191463@critter.freebsd.dk> In message , Neil Saunders writes: >Hopefully a quick one - Is there any way to get varnish to provide the date >header in UTC instead of GMT? That's OS being retarded, Varnish sets its TZ variable to "UTC". -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From arundobriyaliitkgp at gmail.com Thu May 17 13:28:39 2012 From: arundobriyaliitkgp at gmail.com (Arun Dobriyal) Date: Thu, 17 May 2012 18:58:39 +0530 Subject: varnishlog absent Message-ID: Hi, I have a very naive question. Why the varnish 3.0.1 doesnt have varnishlog (Its there is 2.1.5). I compiled the source code, but couldnt find this command. Thanks Arun -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Fri May 18 08:37:18 2012 From: perbu at varnish-software.com (Per Buer) Date: Fri, 18 May 2012 10:37:18 +0200 Subject: varnishlog absent In-Reply-To: References: Message-ID: Hi, On Thu, May 17, 2012 at 3:28 PM, Arun Dobriyal wrote: > Hi, > > I have a very naive question. Why the varnish 3.0.1 doesnt have varnishlog > (Its there is 2.1.5). > I compiled the source code, but couldnt find this command. > Some of the commands have dependencies on the curses library. If it isn't there it wont compile it. Check the output of "configure" and install the curses stuff (ncurses-dev og libncurses-dev or something). -- Per Buer Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From javier at casares.org Fri May 18 10:27:41 2012 From: javier at casares.org (Javier Casares) Date: Fri, 18 May 2012 12:27:41 +0200 Subject: Multiple storage with multiple backends Message-ID: Hi! I'm readin this: -s [name=]type[,options] Use the specified storage backend. See Storage Types for a list of supported storage types. This option can be used multiple times to specify multiple storage files. You can name the different backends. Varnish will then reference that backend with the given name in logs, statistics, etc. In my sysconfig file I have this (it's a demo try): -s malloc,128M \ -s file,/var/lib/varnish/varnish_storage_1.bin,128M \ -s file,/var/lib/varnish/varnish_storage_2.bin,512M" Is possible to asign in the VCL backends what kind of storage use in each backend, or by host or something similar? How can I do it? Javier Casares http://javiercasares.com/ From perbu at varnish-software.com Fri May 18 11:01:36 2012 From: perbu at varnish-software.com (Per Buer) Date: Fri, 18 May 2012 13:01:36 +0200 Subject: Multiple storage with multiple backends In-Reply-To: References: Message-ID: Hi, On Fri, May 18, 2012 at 12:27 PM, Javier Casares wrote: > I'm readin this: > > -s [name=]type[,options] > Use the specified storage backend. See Storage Types for a list of > supported storage types. This option can be used multiple times to > specify multiple storage files. You can name the different backends. > Varnish will then reference that backend with the given name in logs, > statistics, etc. > > In my sysconfig file I have this (it's a demo try): > > -s malloc,128M \ > -s file,/var/lib/varnish/varnish_storage_1.bin,128M \ > -s file,/var/lib/varnish/varnish_storage_2.bin,512M" > > Is possible to asign in the VCL backends what kind of storage use in > each backend, or by host or something similar? How can I do it? > Yes. Just give it a name (-s foo=file,...) and set beresp.storage in vcl_fetch. sub vcl_fetch { if (beresp.http.storage-hind ~ "bar") { beresp.storage = "bar"; } else { beresp.storage = "foo"; } } I'm not 100% sure about the quoting. This is untested code. -- Per Buer Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From javier at casares.org Fri May 18 11:21:53 2012 From: javier at casares.org (Javier Casares) Date: Fri, 18 May 2012 13:21:53 +0200 Subject: Multiple storage with multiple backends In-Reply-To: References: Message-ID: I'll try it! If works i'll publish a sample code :) Javier Casares http://javiercasares.com/ 2012/5/18 Per Buer > > Hi, > > On Fri, May 18, 2012 at 12:27 PM, Javier Casares wrote: >> >> I'm readin this: >> >> -s [name=]type[,options] >> ? ?Use the specified storage backend. See Storage Types for a list of >> supported storage types. This option can be used multiple times to >> specify multiple storage files. You can name the different backends. >> Varnish will then reference that backend with the given name in logs, >> statistics, etc. >> >> In my sysconfig file I have this (it's a demo try): >> >> -s malloc,128M \ >> -s file,/var/lib/varnish/varnish_storage_1.bin,128M \ >> -s file,/var/lib/varnish/varnish_storage_2.bin,512M" >> >> Is possible to asign in the VCL backends what kind of storage use in >> each backend, or by host or something similar? How can I do it? > > > Yes. Just give it a name (-s foo=file,...) and set?beresp.storage in vcl_fetch. > > sub vcl_fetch { > ? ?if (beresp.http.storage-hind ~ "bar") { > ? ? ?beresp.storage = "bar"; > ? ?} else { > ? ? ?beresp.storage = "foo"; > ? } > } > > I'm not 100% sure about the quoting. This is untested code. > > -- > Per Buer > Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer > Varnish makes websites fly! > Whitepapers?| Video?| Twitter > > From arundobriyaliitkgp at gmail.com Mon May 21 12:42:09 2012 From: arundobriyaliitkgp at gmail.com (Arun Dobriyal) Date: Mon, 21 May 2012 18:12:09 +0530 Subject: Modification to url lookup Message-ID: Hi, Is it possible to smehow fire multiple queries to the varnish cache depending on the output of the cache hit or miss of a url. So, if a particular url is not present in the cache, I want to modify this url to create a new url. Then I want to do a lookup using the new url, to see if this new url is in the cache or not. Thanks Arun -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Mon May 21 16:37:42 2012 From: perbu at varnish-software.com (Per Buer) Date: Mon, 21 May 2012 18:37:42 +0200 Subject: Modification to url lookup In-Reply-To: References: Message-ID: Hi Arun, On Mon, May 21, 2012 at 2:42 PM, Arun Dobriyal wrote: > Hi, > Is it possible to smehow fire multiple queries to the varnish cache > depending on the output of the cache hit or miss of a > url. So, if a particular url is not present in the cache, I want to modify > this url to create a new url. > Then I want to do a lookup using the new url, to see if this new url is in > the cache or not. > That is doable. Take https://www.varnish-cache.org/trac/wiki/VCLExampleRestarts and modify it. You probably want to trigger the restarts in vcl_miss instead. If you reach it you've missed the cache, then just alter the URL and restart. -- Per Buer Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From arundobriyaliitkgp at gmail.com Tue May 22 08:15:41 2012 From: arundobriyaliitkgp at gmail.com (Arun Dobriyal) Date: Tue, 22 May 2012 13:45:41 +0530 Subject: Modification to url lookup In-Reply-To: References: Message-ID: Hi , Take https://www.varnish-cache.org/trac/wiki/VCLExampleRestarts and modify > it. You probably want to trigger the restarts in vcl_miss instead. If you > reach it you've missed the cache, then just alter the URL and restart. > Thanks @Per , I was looking for this only. I modified the vcl file and the content of my vcl_miss file is * sub vcl_miss { if(req.restarts==0){ std.log("I came here "+req.http.host+" and the url is "+req.url); return(restart); } return (fetch); }* Clearly the above code should also work (It will enter the if block once whenever it gets a cache miss.. but then subsequently the req.restarts will increase and varnish wont enter the block) . But when I tested this on my machine.. I get these strange messages on the varnishlog, and varnish cant load any page. * 0 Debug - "INCOMPLETE AT: cnt_miss(1213)" 0 WorkThread - 0xb15fd0e0 start 0 CLI - Rd vcl.load "boot" ./vcl.ndNJIZ0m.so 0 CLI - Wr 200 36 Loaded "./vcl.ndNJIZ0m.so" as "boot" 0 CLI - Rd vcl.use "boot" 0 CLI - Wr 200 0 0 CLI - Rd start 0 Debug - "Acceptor is epoll" 0 CLI - Wr 200 0 0 WorkThread - 0xb78a90e0 start 0 WorkThread - 0xb78980e0 start 0 WorkThread - 0xb78870e0 start 0 WorkThread - 0xb78760e0 start 0 WorkThread - 0xb78650e0 start 0 WorkThread - 0xb78540e0 start 0 WorkThread - 0xb78430e0 start 0 WorkThread - 0xb78320e0 start 0 WorkThread - 0xb78210e0 start 0 CLI - Rd ping 0 CLI - Wr 200 19 PONG 1337673620 1.0 0 CLI - Rd ping 0 CLI - Wr 200 19 PONG 1337673623 1.0 0 CLI - Rd ping* What is this error about ? Thanks Arun Dobriyal -------------- next part -------------- An HTML attachment was scrubbed... URL: From shib4u at gmail.com Tue May 22 12:31:59 2012 From: shib4u at gmail.com (Shibashish) Date: Tue, 22 May 2012 18:01:59 +0530 Subject: Varnish sending 304 not modified - unwanted Message-ID: Hi, I have a site which has a FB login in it, login happens, but while logout, varnish sends a 304 not modified, hence the page is not refreshed and logout is not shown on the page. The cookies do get deleted as it should happen. If i check from backend, everything seems ok. How can I rectify this? in vcl_recv.... if (req.http.host ~ "^host.domain.com$" { return(pass); } varnishlog... 296 RxRequest c GET 296 RxURL c / 296 RxProtocol c HTTP/1.1 296 RxHeader c Host: host.domain.com 296 RxHeader c User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20100101 Firefox/12.0 296 RxHeader c Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 296 RxHeader c Accept-Language: en-us,en;q=0.5 296 RxHeader c Accept-Encoding: gzip, deflate 296 RxHeader c Connection: keep-alive 296 RxHeader c Referer: http://host.domain.com/ 296 RxHeader c Cookie: __utma=198839274.365445528.1337689518.1337689518.1337689518.1; __utmb=198839274.5.10.1337689518; __utmc=198839274; __utmz=198839274.1337689518.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __qca=P0-1956808353-1337689518459; fbsr_2770104656637 296 RxHeader c If-Modified-Since: Tue, 22 May 2012 12:24:34 GMT 296 VCL_call c recv pass 296 VCL_call c hash hash 296 VCL_call c pass pass 296 Backend c 267 video video[0] 296 TTL c 313634256 RFC 60 1337689482 0 0 0 0 296 VCL_call c fetch 296 TTL c 313634256 VCL 30 1337689483 296 VCL_return c deliver 296 ObjProtocol c HTTP/1.1 296 ObjStatus c 200 296 ObjResponse c OK 296 ObjHeader c Date: Tue, 22 May 2012 12:24:42 GMT 296 ObjHeader c Last-Modified: Tue, 22 May 2012 12:23:12 GMT 296 ObjHeader c Vary: Cookie,Accept-Encoding 296 ObjHeader c X-Pingback: http://host.domain.com/xmlrpc.php 296 ObjHeader c Content-Encoding: gzip 296 ObjHeader c Content-Length: 18062 296 ObjHeader c Content-Type: text/html; charset=UTF-8 296 ObjHeader c Server: Web/10.1.10 296 VCL_call c deliver deliver 296 TxProtocol c HTTP/1.1 296 TxStatus c 304 296 TxResponse c Not Modified 296 TxHeader c Date: Tue, 22 May 2012 12:24:42 GMT 296 TxHeader c Last-Modified: Tue, 22 May 2012 12:23:12 GMT 296 TxHeader c Connection: keep-alive 296 TxHeader c X-Cache: MISS 296 TxHeader c Vary: Accept-Encoding 296 TxHeader c X-Country-Code: Unknown 296 Length c 0 296 ReqEnd c 313634256 1337689482.766963959 1337689482.779349089 0.051923990 0.012341022 0.000044107 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jpotter-varnish at codepuppy.com Tue May 22 15:05:32 2012 From: jpotter-varnish at codepuppy.com (Jeff Potter) Date: Tue, 22 May 2012 08:05:32 -0700 Subject: max_connections, backend_busy, and queueing requests Message-ID: <569D2680-1B6A-4FF0-BFBE-F31627A5F56B@codepuppy.com> Hi List, I'm having a hard time finding documentation or other users who've done what I'm trying to do, although it seems pretty straight-forward and I must be missing something (or have something misconfigured?). Let's say that I have a simple setup of 1 varnish server and 5 backend nodes defined in a single director; nothing fancy other than each backend's max_connections being set to 100. This setup would allow for 500 concurrent requests when everything is healthy. As soon as there are 500 concurrent requests, varnish kicks any additional requests to vcl_error and we send out a 5xx error for the requests. And varnishstats increments the backend_busy/"Backend conn. too many" counter. How do I get Varnish to queue up the requests instead, so that when a backend worker connection becomes available it gets sent to a backend? (Currently running Varnish 2; planning to migrate to Varnish 3 shortly. If the answer is "upgrade to varnish 3", that's useful to know too!) It may help to know that, in our case, we expect most of the requests to go to the backend -- i.e. while we do use cacheing, most cacheable requests have been pushed off to other hosts. If the only way to do this is configure varnish such that it only handles 500 frontend requests at a time, that would be okay (but not ideal). Thanks! -Jeff P.S. Thanks for a great piece of software -- Varnish is pretty amazing. From perbu at varnish-software.com Tue May 22 16:14:18 2012 From: perbu at varnish-software.com (Per Buer) Date: Tue, 22 May 2012 18:14:18 +0200 Subject: max_connections, backend_busy, and queueing requests In-Reply-To: <569D2680-1B6A-4FF0-BFBE-F31627A5F56B@codepuppy.com> References: <569D2680-1B6A-4FF0-BFBE-F31627A5F56B@codepuppy.com> Message-ID: Hi Jeff. On Tue, May 22, 2012 at 5:05 PM, Jeff Potter wrote: > > As soon as there are 500 concurrent requests, varnish kicks any additional > requests to vcl_error and we send out a 5xx error for the requests. And > varnishstats increments the backend_busy/"Backend conn. too many" counter. > > How do I get Varnish to queue up the requests instead, so that when a > backend worker connection becomes available it gets sent to a backend? Varnish, in it's current state doesn't do this, I'm afraid. Since Varnish uses one thread, and thus has some overhead, for each connection it wouldn't be ideal to have them idly wait. The current trunk code uses separate threads to fetch content from the backend so having some sort of queue there shouldn't be to hard. I don't know of any plans to implement this, however. -- Per Buer Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From barrycie at aol.com Wed May 23 11:18:13 2012 From: barrycie at aol.com (Barry Conroy) Date: Wed, 23 May 2012 12:18:13 +0100 Subject: Varnish - high session_linger and session herd Message-ID: <4FBCC775.8030006@aol.com> Hi all, Ive been testing Varnish over the past while and have a few questions with which i hope you can help. We have deployed Varnish 3.0.2 in front of our XML api. During my load testing Ive noticed that the session_linger and session_herd stats are quite high. Does this signal an issue? Also, during my load testing Im getting an error for 0.04% of requests to Varnish - "readResponse: java.net.SocketException: Connection reset ". Has anyone seen this before? Are these connections getting closed on the Varnish side? I had been using the default setting for session_linger but changed it to 0. Do I need session linger considering I am not caching a web page but am instead caching XML api responses? I was surprised how little memory Varnish is using to cache its objects - this is reflect in the line SMA.s0.g_bytes 2109235 . Bytes outstanding The following are the output from varnishstat -1 and also the param.show. The VCL in play is very simple and passes through 1 request type and caches everything else. Thanks for you help in advance, Barry client_conn 5990408 14.10 Client connections accepted client_drop 0 0.00 Connection dropped, no sess/wrk client_req 2351555655 5534.53 Client requests received cache_hit 2346417108 5522.44 Cache hits cache_hitpass 0 0.00 Cache hits for pass cache_miss 4836412 11.38 Cache misses backend_conn 11271 0.03 Backend conn. success backend_unhealthy 0 0.00 Backend conn. not attempted backend_busy 0 0.00 Backend conn. too many backend_fail 2 0.00 Backend conn. failures backend_reuse 5127357 12.07 Backend conn. reuses backend_toolate 1044 0.00 Backend conn. was closed backend_recycle 5128407 12.07 Backend conn. recycles backend_retry 99 0.00 Backend conn. retry fetch_head 295036 0.69 Fetch head fetch_length 4836477 11.38 Fetch with Length fetch_chunked 7012 0.02 Fetch chunked fetch_eof 0 0.00 Fetch EOF fetch_bad 0 0.00 Fetch had bad headers fetch_close 0 0.00 Fetch wanted close fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed fetch_zero 0 0.00 Fetch zero len fetch_failed 0 0.00 Fetch failed fetch_1xx 0 0.00 Fetch no body (1xx) fetch_204 0 0.00 Fetch no body (204) fetch_304 0 0.00 Fetch no body (304) n_sess_mem 1251 . N struct sess_mem n_sess 789 . N struct sess n_object 453 . N struct object n_vampireobject 0 . N unresurrected objects n_objectcore 627 . N struct objectcore n_objecthead 1859 . N struct objecthead n_waitinglist 15213 . N struct waitinglist n_vbc 5 . N struct vbc n_wrk 174 . N worker threads n_wrk_create 10484 0.02 N worker threads created n_wrk_failed 0 0.00 N worker threads not created n_wrk_max 0 0.00 N worker threads limited n_wrk_lqueue 0 0.00 work request queue length n_wrk_queued 514194 1.21 N queued work requests n_wrk_drop 0 0.00 N dropped work requests n_backend 1 . N backends n_expired 4835952 . N expired objects n_lru_nuked 0 . N LRU nuked objects n_lru_moved 33485426 . N LRU moved objects losthdr 0 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 2351109391 5533.48 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 5990506 14.10 Total Sessions s_req 2351555664 5534.53 Total Requests s_pipe 0 0.00 Total pipe s_pass 302120 0.71 Total pass s_fetch 5138525 12.09 Total fetch s_hdrbytes 544265355452 1280961.94 Total header bytes s_bodybytes 3089476577758 7271272.85 Total body bytes sess_closed 227804 0.54 Session Closed sess_pipeline 0 0.00 Session Pipeline sess_readahead 0 0.00 Session Read Ahead sess_linger 2209279166 5199.67 Session Linger sess_herd 323937467 762.41 Session herd shm_records 89745845415 211222.36 SHM records shm_writes 2562338698 6030.62 SHM writes shm_flushes 62 0.00 SHM flushes due to overflow shm_cont 27674729 65.13 SHM MTX contention shm_cycles 43605 0.10 SHM cycles through buffer sms_nreq 22 0.00 SMS allocator requests sms_nobj 0 . SMS outstanding allocations sms_nbytes 0 . SMS outstanding bytes sms_balloc 8856 . SMS bytes allocated sms_bfree 8856 . SMS bytes freed backend_req 5138625 12.09 Backend requests made n_vcl 1 0.00 N vcl total n_vcl_avail 1 0.00 N vcl available n_vcl_discard 0 0.00 N vcl discarded n_ban 1 . N total active bans n_ban_add 1 0.00 N new bans added n_ban_retire 0 0.00 N old bans deleted n_ban_obj_test 0 0.00 N objects tested n_ban_re_test 0 0.00 N regexps tested against n_ban_dups 0 0.00 N duplicate bans removed hcb_nolock 2351224626 5533.75 HCB Lookups without lock hcb_lock 1666108 3.92 HCB Lookups with lock hcb_insert 1666107 3.92 HCB Inserts esi_errors 0 0.00 ESI parse errors (unlock) esi_warnings 0 0.00 ESI parse warnings (unlock) accept_fail 0 0.00 Accept failures client_drop_late 0 0.00 Connection dropped late uptime 424888 1.00 Client uptime dir_dns_lookups 0 0.00 DNS director lookups dir_dns_failed 0 0.00 DNS director failed lookups dir_dns_hit 0 0.00 DNS director cached lookups hit dir_dns_cache_full 0 0.00 DNS director full dnscache vmods 0 . Loaded VMODs n_gzip 0 0.00 Gzip operations n_gunzip 2355834985 5544.60 Gunzip operations LCK.sms.creat 1 0.00 Created locks LCK.sms.destroy 0 0.00 Destroyed locks LCK.sms.locks 66 0.00 Lock Operations LCK.sms.colls 0 0.00 Collisions LCK.smp.creat 0 0.00 Created locks LCK.smp.destroy 0 0.00 Destroyed locks LCK.smp.locks 0 0.00 Lock Operations LCK.smp.colls 0 0.00 Collisions LCK.sma.creat 2 0.00 Created locks LCK.sma.destroy 0 0.00 Destroyed locks LCK.sma.locks 24799574 58.37 Lock Operations LCK.sma.colls 0 0.00 Collisions LCK.smf.creat 0 0.00 Created locks LCK.smf.destroy 0 0.00 Destroyed locks LCK.smf.locks 0 0.00 Lock Operations LCK.smf.colls 0 0.00 Collisions LCK.hsl.creat 0 0.00 Created locks LCK.hsl.destroy 0 0.00 Destroyed locks LCK.hsl.locks 0 0.00 Lock Operations LCK.hsl.colls 0 0.00 Collisions LCK.hcb.creat 1 0.00 Created locks LCK.hcb.destroy 0 0.00 Destroyed locks LCK.hcb.locks 3334164 7.85 Lock Operations LCK.hcb.colls 0 0.00 Collisions LCK.hcl.creat 0 0.00 Created locks LCK.hcl.destroy 0 0.00 Destroyed locks LCK.hcl.locks 0 0.00 Lock Operations LCK.hcl.colls 0 0.00 Collisions LCK.vcl.creat 1 0.00 Created locks LCK.vcl.destroy 0 0.00 Destroyed locks LCK.vcl.locks 32230 0.08 Lock Operations LCK.vcl.colls 0 0.00 Collisions LCK.stat.creat 1 0.00 Created locks LCK.stat.destroy 0 0.00 Destroyed locks LCK.stat.locks 1251 0.00 Lock Operations LCK.stat.colls 0 0.00 Collisions LCK.sessmem.creat 1 0.00 Created locks LCK.sessmem.destroy 0 0.00 Destroyed locks LCK.sessmem.locks 5999643 14.12 Lock Operations LCK.sessmem.colls 0 0.00 Collisions LCK.wstat.creat 1 0.00 Created locks LCK.wstat.destroy 0 0.00 Destroyed locks LCK.wstat.locks 155943114 367.02 Lock Operations LCK.wstat.colls 0 0.00 Collisions LCK.herder.creat 1 0.00 Created locks LCK.herder.destroy 0 0.00 Destroyed locks LCK.herder.locks 25245 0.06 Lock Operations LCK.herder.colls 0 0.00 Collisions LCK.wq.creat 2 0.00 Created locks LCK.wq.destroy 0 0.00 Destroyed locks LCK.wq.locks 649162540 1527.84 Lock Operations LCK.wq.colls 0 0.00 Collisions LCK.objhdr.creat 1676572 3.95 Created locks LCK.objhdr.destroy 1674713 3.94 Destroyed locks LCK.objhdr.locks 9417295559 22164.18 Lock Operations LCK.objhdr.colls 0 0.00 Collisions LCK.exp.creat 1 0.00 Created locks LCK.exp.destroy 0 0.00 Destroyed locks LCK.exp.locks 10095406 23.76 Lock Operations LCK.exp.colls 0 0.00 Collisions LCK.lru.creat 2 0.00 Created locks LCK.lru.destroy 0 0.00 Destroyed locks LCK.lru.locks 4836405 11.38 Lock Operations LCK.lru.colls 0 0.00 Collisions LCK.cli.creat 1 0.00 Created locks LCK.cli.destroy 0 0.00 Destroyed locks LCK.cli.locks 141594 0.33 Lock Operations LCK.cli.colls 0 0.00 Collisions LCK.ban.creat 1 0.00 Created locks LCK.ban.destroy 0 0.00 Destroyed locks LCK.ban.locks 10095484 23.76 Lock Operations LCK.ban.colls 0 0.00 Collisions LCK.vbp.creat 1 0.00 Created locks LCK.vbp.destroy 0 0.00 Destroyed locks LCK.vbp.locks 0 0.00 Lock Operations LCK.vbp.colls 0 0.00 Collisions LCK.vbe.creat 1 0.00 Created locks LCK.vbe.destroy 0 0.00 Destroyed locks LCK.vbe.locks 22541 0.05 Lock Operations LCK.vbe.colls 0 0.00 Collisions LCK.backend.creat 1 0.00 Created locks LCK.backend.destroy 0 0.00 Destroyed locks LCK.backend.locks 13459516 31.68 Lock Operations LCK.backend.colls 0 0.00 Collisions SMA.s0.c_req 9672832 22.77 Allocator requests SMA.s0.c_fail 0 0.00 Allocator failures SMA.s0.c_bytes 637361546992 1500069.54 Bytes allocated SMA.s0.c_freed 637360393374 1500066.83 Bytes freed SMA.s0.g_alloc 906 . Allocations outstanding SMA.s0.g_bytes 1153618 . Bytes outstanding SMA.s0.g_space 1072588206 . Bytes available SMA.Transient.c_req 309204 0.73 Allocator requests SMA.Transient.c_fail 0 0.00 Allocator failures SMA.Transient.c_bytes 212330558 499.73 Bytes allocated SMA.Transient.c_freed 212330558 499.73 Bytes freed SMA.Transient.g_alloc 0 . Allocations outstanding SMA.Transient.g_bytes 0 . Bytes outstanding SMA.Transient.g_space 0 . Bytes available VBE.default(172.29.101.121,,8080).vcls 1 . VCL references VBE.default(172.29.101.121,,8080).happy 0 . Happy health probes ------------------------------ acceptor_sleep_decay 0.900000 [] acceptor_sleep_incr 0.001000 [s] acceptor_sleep_max 0.050000 [s] auto_restart on [bool] ban_dups on [bool] ban_lurker_sleep 0.010000 [s] between_bytes_timeout 60.000000 [s] cc_command "exec gcc -std=gnu99 -pthread -fpic -shared -Wl,-x -o %o %s" cli_buffer 8192 [bytes] cli_timeout 10 [seconds] clock_skew 10 [s] connect_timeout 0.700000 [s] critbit_cooloff 180.000000 [s] default_grace 10.000000 [seconds] default_keep 0.000000 [seconds] default_ttl 30.000000 [seconds] diag_bitmap 0x0 [bitmap] esi_syntax 0 [bitmap] expiry_sleep 1.000000 [seconds] fetch_chunksize 128 [kilobytes] fetch_maxchunksize 262144 [kilobytes] first_byte_timeout 60.000000 [s] gzip_level 6 [] gzip_memlevel 8 [] gzip_stack_buffer 32768 [Bytes] gzip_tmp_space 0 [] gzip_window 15 [] http_gzip_support on [bool] http_max_hdr 64 [header lines] http_range_support on [bool] http_req_hdr_len 8192 [bytes] http_req_size 32768 [bytes] http_resp_hdr_len 8192 [bytes] http_resp_size 32768 [bytes] listen_address 0.0.0.0:10300 listen_depth 1024 [connections] log_hashstring on [bool] log_local_address off [bool] lru_interval 2 [seconds] max_esi_depth 5 [levels] max_restarts 4 [restarts] nuke_limit 50 [allocations] ping_interval 3 [seconds] pipe_timeout 60 [seconds] prefer_ipv6 off [bool] queue_max 100 [%] rush_exponent 3 [requests per request] saintmode_threshold 10 [objects] send_timeout 60 [seconds] sess_timeout 5 [seconds] sess_workspace 65536 [bytes] session_linger 0 [ms] session_max 100000 [sessions] shm_reclen 255 [bytes] shm_workspace 8192 [bytes] shortlived 10.000000 [s] syslog_cli_traffic on [bool] thread_pool_add_delay 2 [milliseconds] thread_pool_add_threshold 2 [requests] thread_pool_fail_delay 200 [milliseconds] thread_pool_max 1000 [threads] thread_pool_min 60 [threads] thread_pool_purge_delay 1000 [milliseconds] thread_pool_stack unlimited [bytes] thread_pool_timeout 120 [seconds] thread_pool_workspace 65536 [bytes] thread_pools 2 [pools] thread_stats_rate 10 [requests] vcc_err_unref on [bool] vcl_trace off [bool] waiter default (epoll, poll) -------------- next part -------------- An HTML attachment was scrubbed... URL: From arundobriyaliitkgp at gmail.com Wed May 23 12:42:09 2012 From: arundobriyaliitkgp at gmail.com (Arun Dobriyal) Date: Wed, 23 May 2012 18:12:09 +0530 Subject: how to create variables Message-ID: "How can I create string variables in the default.vcl file... Is it possible to create variables and then use them.." alternatively, " Is it possible to use the vcl variables in the inline c code and vice-versa ? " Thanks, Arun -------------- next part -------------- An HTML attachment was scrubbed... URL: From tobias.eichelbroenner at lamp-solutions.de Wed May 23 12:52:58 2012 From: tobias.eichelbroenner at lamp-solutions.de (=?ISO-8859-1?Q?Tobias_Eichelbr=F6nner?=) Date: Wed, 23 May 2012 14:52:58 +0200 Subject: how to create variables In-Reply-To: References: Message-ID: <4FBCDDAA.6000402@lamp-solutions.de> Hi, > "How can I create string variables in the default.vcl file... Is it > possible to create variables and then use them.." you can store informationen in the http-header in vcl: req.http.X-Variable = "value"; and in inline c VRT_GetHdr(sp, HDR_REQ, "\013X-Variable:") VRT_SetHdr(sp, HDR_REQ, "\013X-Variable:", "value", vrt_magic_string_end); Sincerely From pwlazy at gmail.com Wed May 23 15:42:37 2012 From: pwlazy at gmail.com (=?GB2312?B?xe3OsA==?=) Date: Wed, 23 May 2012 23:42:37 +0800 Subject: how to debug varnish without compiler optimization Message-ID: hi when i debug varnish source code with gdb, i found some variables showing " *value optimized out"* * * i think it should compiler optimization,and i found CFlags = " -g -O2 " in Makefile. so i recompiled the source code: export CFLAGS=" -g -O0 " && ./configure && make && make install i check the CFlags in Makefile and found it was CFlags = " -g -O0 ", it seems the compilation should be success then i debug the source code again and found those variables still showing " *value optimized out"* * * it seems strange, how to solve it? thanks for you answer! -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Wed May 23 19:02:52 2012 From: perbu at varnish-software.com (Per Buer) Date: Wed, 23 May 2012 21:02:52 +0200 Subject: how to create variables In-Reply-To: References: Message-ID: Hi Arun, On Wed, May 23, 2012 at 2:42 PM, Arun Dobriyal wrote: > "How can I create string variables in the default.vcl file... Is it > possible to create variables and then use them.." > Have a look at Tollefs excellent Variable VMOD. https://www.varnish-cache.org/vmod/variable-support Also, as Tobias mentioned. The old school way is to stuff things into HTTP headers. :-) -- Per Buer Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From jpotter-varnish at codepuppy.com Wed May 23 21:39:26 2012 From: jpotter-varnish at codepuppy.com (Jeff Potter) Date: Wed, 23 May 2012 14:39:26 -0700 Subject: max_connections, backend_busy, and queueing requests In-Reply-To: References: <569D2680-1B6A-4FF0-BFBE-F31627A5F56B@codepuppy.com> Message-ID: Hi Per, >> As soon as there are 500 concurrent requests, varnish kicks any additional requests to vcl_error and we send out a 5xx error for the requests. And varnishstats increments the backend_busy/"Backend conn. too many" counter. How do I get Varnish to queue up the requests instead, so that when a backend worker connection becomes available it gets sent to a backend? > > Varnish, in it's current state doesn't do this, I'm afraid. Since Varnish uses one thread, and thus has some overhead, for each connection it wouldn't be ideal to have them idly wait. The current trunk code uses separate threads to fetch content from the backend so having some sort of queue there shouldn't be to hard. I don't know of any plans to implement this, however. Thanks for the reply. If I were to implement this and submit it, would Varnish be willing to accept the patch into the mainline? And, presumably an implementation would want two config variables: the max number of requests to allow on the queue and the max time allowed on queue until vcl_error is triggered? Best, Jeff From arundobriyaliitkgp at gmail.com Thu May 24 04:06:26 2012 From: arundobriyaliitkgp at gmail.com (Arun Dobriyal) Date: Thu, 24 May 2012 09:36:26 +0530 Subject: how to create variables In-Reply-To: References: Message-ID: Thanks Per and Tobias.. http headers is a good way :) On Thu, May 24, 2012 at 12:32 AM, Per Buer wrote: > Hi Arun, > > On Wed, May 23, 2012 at 2:42 PM, Arun Dobriyal < > arundobriyaliitkgp at gmail.com> wrote: > >> "How can I create string variables in the default.vcl file... Is it >> possible to create variables and then use them.." >> > > Have a look at Tollefs excellent Variable VMOD. > > https://www.varnish-cache.org/vmod/variable-support > > Also, as Tobias mentioned. The old school way is to stuff things into HTTP > headers. :-) > -- > Per Buer > Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer > *Varnish makes websites fly!* > Whitepapers | Video | > Twitter > > > -- Arun Dobriyal, Fifth year Dual Degree Student, Department of Computer Science and Engineering, Indian Institute of Technology, Kharagpur, India. -------------- next part -------------- An HTML attachment was scrubbed... URL: From t.winkelmann at ffh.de Thu May 24 06:03:45 2012 From: t.winkelmann at ffh.de (Winkelmann, Thomas (Radio/Tele FFH - Online)) Date: Thu, 24 May 2012 08:03:45 +0200 Subject: Problems with varnish backend health Message-ID: <8A76B2DB46634944BEEBB218DAE22BC20CCEF0DA@exchange-srv2.office.radioteleffh.de> Hello, we have a setup with two varnish servers (version 3.0.2) and two backend servers. This setup works fine, if we split the requests with an loadbalancer to the two varnish servers. Around 500req/s 20MBit/s bandwidth on each server. In Addition we have defined a backend probe which will tests the backend each 10s. This also works fine. Now, if we switch all requests to one varnish server, after some time (around 20 seconds) the backends will be marked as sick, but they are both up and answering all requests. Also without any backend probe defined in VCL, varnish gets in trouble serving content. Some months ago, we had no problems with this and served more than 400Mbit/s from one single server. What could be the bottleneck? Kind regards Thomas Winkelmann Web-Entwickler ___________________________________ FFH Digital Radio/Tele FFH GmbH & Co. Betriebs-KG FFH-Platz 1, 61111 Bad Vilbel Tel.: 06101 / 988 - 346 Fax: 06101 / 988 - 555 Email: t.winkelmann at FFH.de FFH Digital - Die Webradios www.FFH-Digital.de HIT RADIO FFH www.FFH.de planet more music radio www.planetradio.de harmony.fm - Hessens Greatest Hits www.harmonyfm.de ________________________________ RADIO / TELE FFH GmbH & Co. Betriebs-KG FFH-Platz 1, 61111 Bad Vilbel HRA - Nr. 26092 Frankfurt/Main USt.IdNr. DE 112152620 Gesch?ftsf?hrer / Programmdirektor: Hans-Dieter Hillmoth -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Thu May 24 07:36:01 2012 From: perbu at varnish-software.com (Per Buer) Date: Thu, 24 May 2012 09:36:01 +0200 Subject: max_connections, backend_busy, and queueing requests In-Reply-To: References: <569D2680-1B6A-4FF0-BFBE-F31627A5F56B@codepuppy.com> Message-ID: On Wed, May 23, 2012 at 11:39 PM, Jeff Potter wrote: > > > Thanks for the reply. If I were to implement this and submit it, would > Varnish be willing to accept the patch into the mainline? And, presumably > an implementation would want two config variables: the max number of > requests to allow on the queue and the max time allowed on queue until > vcl_error is triggered? > For me, at least, this sounds feasible as long as the implementation is clean and fits into trunk. It must pass General Muster, aka PHK. See https://www.varnish-cache.org/trac/wiki/Contributing for details. -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From arundobriyaliitkgp at gmail.com Thu May 24 12:29:01 2012 From: arundobriyaliitkgp at gmail.com (Arun Dobriyal) Date: Thu, 24 May 2012 17:59:01 +0530 Subject: how to create variables In-Reply-To: References: Message-ID: I myself was having trouble with vmods, for the people who are looking at the vmods for the first time, probably the better way to understand them is by writing the vcl file and running "varnishd" with "-C" option, this option shows the c code generated for the vcl logic written in the default.vcl file.. by this method, one can easily check how the vcl code is converted into its corresponding c code, and it would serve as a big help so as how to use/introduce variables in vcl logic. -Arun -------------- next part -------------- An HTML attachment was scrubbed... URL: From randallk867 at 123mail.org Thu May 24 21:56:31 2012 From: randallk867 at 123mail.org (randallk867 at 123mail.org) Date: Thu, 24 May 2012 14:56:31 -0700 Subject: Need to add/modify Expires headers for external CDN-provided scripts. Varnish is in the middle of my webstack. Do I do this in Varnish? Message-ID: <1337896591.14303.140661080327993.222C4012@webmail.messagingengine.com> My web site is configured with nginx/ssl -> varnish3 (reverse proxy) -> drupal7+apache I've added using Typekit webfonts to my Drupal site. I learned that in order to prevent FOUT, it's css/js need to load asap. I did that, so I have ... test ... It works well. When I run an optimization scan with YSlow on my site it reports: Grade D on Add Expires headers There are 2 static components without a far-future expiration date. (2012/5/24) https://use.typekit.com/######-d.css?... (2012/5/24) https://use.typekit.com/######.js I want to set/modify the Expire headers specifically for those two scripts. Given the stack, and that I already modify other headers in varnish vcl for this config, do I do THIS in the varnish config as well? Does the fact that these script are loading from an external CDN affect this at all? I did find an old comment @ typekit's support, https://getsatisfaction.com/typekit/topics/use_far_future_expires_headers_for_better_caching that suggests that there's a setting that needs to be 'in' the Typekit itself. I'm confused about where this needs to be handled. So I thought I'd ask. RandyK From stewsnooze at gmail.com Fri May 25 05:52:50 2012 From: stewsnooze at gmail.com (Stewart Robinson) Date: Fri, 25 May 2012 06:52:50 +0100 Subject: Need to add/modify Expires headers for external CDN-provided scripts. Varnish is in the middle of my webstack. Do I do this in Varnish? In-Reply-To: <1337896591.14303.140661080327993.222C4012@webmail.messagingengine.com> References: <1337896591.14303.140661080327993.222C4012@webmail.messagingengine.com> Message-ID: <21583471246143588@unknownmsgid> On 24 May 2012, at 22:57, "randallk867 at 123mail.org" wrote: > > My web site is configured with > > nginx/ssl -> varnish3 (reverse proxy) -> drupal7+apache > > I've added using Typekit webfonts to my Drupal site. I learned that in > order to prevent FOUT, it's css/js need to load asap. > > I did that, so I have > > > > > ... > test > > > > ... > > It works well. When I run an optimization scan with YSlow on my site it > reports: > > Grade D on Add Expires headers > There are 2 static components without a far-future expiration > date. > (2012/5/24) https://use.typekit.com/######-d.css?... > (2012/5/24) https://use.typekit.com/######.js > > > I want to set/modify the Expire headers specifically for those two > scripts. > > Given the stack, and that I already modify other headers in varnish vcl > for this config, do I do THIS in the varnish config as well? > > Does the fact that these script are loading from an external CDN affect > this at all? > > I did find an old comment @ typekit's support, > > https://getsatisfaction.com/typekit/topics/use_far_future_expires_headers_for_better_caching > > that suggests that there's a setting that needs to be 'in' the Typekit > itself. > > I'm confused about where this needs to be handled. So I thought I'd > ask. > > RandyK > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc Hi, Varnish config can only affect content that is being served by the domains pointing at your Varnish server. Stewart Robinson @stewsnooze FullFatThings.com From nick.tailor at gmail.com Fri May 25 19:26:13 2012 From: nick.tailor at gmail.com (nick tailor) Date: Fri, 25 May 2012 12:26:13 -0700 Subject: Question Message-ID: Does anyone know. If you can have Varnish send traffic to multiple VIPS? I dont want to have to provision a separate pair of varnish servers for each HA cluster I have. I have my services going through a LB which the varnish servers will sit in front of. I want to be able to define separate VIP's in the default.vcl so that traffic on specific vips go to specific backends, is that possible? Cheers Nick Tailor -------------- next part -------------- An HTML attachment was scrubbed... URL: From footplus at gmail.com Fri May 25 20:01:36 2012 From: footplus at gmail.com (=?UTF-8?B?QXVyw6lsaWVu?=) Date: Fri, 25 May 2012 22:01:36 +0200 Subject: Question In-Reply-To: References: Message-ID: On Fri, May 25, 2012 at 9:26 PM, nick tailor wrote: > Does anyone know. If you can have Varnish send traffic to multiple VIPS? I > dont want to have to provision a separate pair of varnish servers for each > HA cluster I have. > > I have my services going through a LB which the varnish servers will sit > in front of. > > I want to be able to define separate VIP's in the default.vcl so that > traffic on specific vips go to specific backends, is that possible? > > Hello, Unless i'm mistaken, you just want to implement several backends or directors, then you can use server.ip and server.port ( https://www.varnish-cache.org/docs/3.0/reference/vcl.html#varnish-configuration-language) in VCL to make the requests go to one specific backend (you need to make Varnish listen on several IP/Ports on the command-line, though). You can then use your LB or even let the Varnish director of your choice handle the load-balancing - depending on your needs. If you have a single IP per cluster, you can use a single backend. Regards, -- Aur?lien Guillaume * * -------------- next part -------------- An HTML attachment was scrubbed... URL: From mailing.lists.wam at gmail.com Sat May 26 13:18:05 2012 From: mailing.lists.wam at gmail.com (Mailing Lists) Date: Sat, 26 May 2012 15:18:05 +0200 Subject: pass and cache at the same time Message-ID: <5EA740A2-6DF7-4D95-830C-2ACA4610A8BF@gmail.com> Hi, In my vcl_fetch i return(pass) for specifics url, but i want to create a cached version too, in case my backend fails i would be able to serve grace version, how can i achieve this ? I thought of using restart, to first create a cached version and then do the pass, using intermediary http-header. Regards, From footplus at gmail.com Sat May 26 17:30:30 2012 From: footplus at gmail.com (=?utf-8?Q?Aur=C3=A9lien_Guillaume?=) Date: Sat, 26 May 2012 19:30:30 +0200 Subject: pass and cache at the same time In-Reply-To: <5EA740A2-6DF7-4D95-830C-2ACA4610A8BF@gmail.com> References: <5EA740A2-6DF7-4D95-830C-2ACA4610A8BF@gmail.com> Message-ID: <8223FEE9-D0A3-487C-89EE-27E86F4A0FE9@gmail.com> Hello, You may perhaps do a lookup, but set the ttl to 0s for these particular URLs. This way you may perhaps be able to never serve expired objects unless your backend is dead... Regards, -- Aur?lien Sent from my phone Le 26 mai 2012 ? 15:18, Mailing Lists a ?crit : > Hi, > > In my vcl_fetch i return(pass) for specifics url, but i want to create a cached version too, in case my backend fails i would be able to serve grace version, how can i achieve this ? I thought of using restart, to first create a cached version and then do the pass, using intermediary http-header. > > Regards, > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From rowan at jetboy.co.uk Sat May 26 21:24:41 2012 From: rowan at jetboy.co.uk (Rowan @ Jetboy) Date: Sat, 26 May 2012 22:24:41 +0100 Subject: varnishncsa -F flag custom log format Message-ID: <000301cd3b85$f865e950$e931bbf0$@co.uk> Running Varnish 3.02 rev. cbf1284 on Ubuntu Server 12.04 LTS x64: I can get varnishncsa writing to logs successfully with: VARNISHNCSA_ENABLED=1 DAEMON_OPTS="-a \ -c \ -D \ -P ${PIDFILE} \ -w /var/log/varnish/hollywood.varnishncsa.log" in /etc/default/varnishncsa but I'm really struggling with the -F flag for custom log formats. I'm trying to achieve an Apache combined log format prefixed by the virtual host, which'd look something like this: %{Host}i %h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\" but all I'm managing to do is capture the %{Host}i bit in the logfile. There's an old thread on here about issues with spaces and escaping quotes, but the responses are all about setting the format on the command line, which I'm really trying to avoid. I've got to: VARNISHNCSA_ENABLED=1 FORMAT="%{Host}i %h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\"" DAEMON_OPTS="-a \ -c \ -D \ -F ${FORMAT} \ -P ${PIDFILE} \ -w /var/log/varnish/hollywood.varnishncsa.log" I've tried various permutations of single and double quotes, and the format both in and out of a variable. No luck so far. Can anyone help me out? -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.carrillo at gmail.com Sun May 27 10:29:14 2012 From: daniel.carrillo at gmail.com (Daniel Carrillo) Date: Sun, 27 May 2012 12:29:14 +0200 Subject: varnishncsa -F flag custom log format In-Reply-To: <000301cd3b85$f865e950$e931bbf0$@co.uk> References: <000301cd3b85$f865e950$e931bbf0$@co.uk> Message-ID: 2012/5/26 Rowan @ Jetboy : > Running Varnish 3.02 rev. cbf1284 on Ubuntu Server 12.04 LTS x64: > > > > I can get varnishncsa writing to logs successfully with: > > > > VARNISHNCSA_ENABLED=1 > > DAEMON_OPTS="-a \ > > ???????????? -c \ > > ???????????? -D \ > > ???????????? -P ${PIDFILE} \ > > ???????????? -w /var/log/varnish/hollywood.varnishncsa.log" > > > > in /etc/default/varnishncsa > > > > but I'm really struggling with the -F flag for custom log formats. I'm > trying to achieve an Apache combined log format prefixed by the virtual > host, which'd look something like this: > > > > %{Host}i %h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\" > > > > but all I'm managing to do is capture the %{Host}i bit in the logfile. > > > > There's an old thread on here about issues with spaces and escaping quotes, > but the responses are all about setting the format on the command line, > which I'm really trying to avoid. I've got to: > > > > VARNISHNCSA_ENABLED=1 > > FORMAT="%{Host}i %h %l %u %t \"%r\" %s %b \"%{Referer}i\" > \"%{User-agent}i\"" > > DAEMON_OPTS="-a \ > > ???????????? -c \ > > ???????????? -D \ > > ???????????? -F ${FORMAT} \ > > ???????????? -P ${PIDFILE} \ > > ???????????? -w /var/log/varnish/hollywood.varnishncsa.log" > > > > I've tried various permutations of single and double quotes, and the format > both in and out of a variable. No luck so far. Can anyone help me out? This one works for me: LOG_FORMAT="%h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\" %{Host}i %{Varnish:time_firstbyte}x" DAEMON_OPTS="-c -a -w $logfile -D -P $pidfile -F '${LOG_FORMAT}'" Please, note the single quotes in -F Hope this help. From rowan at jetboy.co.uk Sun May 27 14:09:55 2012 From: rowan at jetboy.co.uk (Rowan @ Jetboy) Date: Sun, 27 May 2012 15:09:55 +0100 Subject: varnishncsa -F flag custom log format In-Reply-To: References: <000301cd3b85$f865e950$e931bbf0$@co.uk> Message-ID: <001e01cd3c12$64ad2880$2e077980$@co.uk> Hi Daniel, Thanks for the reply. I've tried entirely replacing my code with yours (other than the capitalisation of $PIDFILE, and putting $logfile inline), but I get similar results. It's only recording the first log parameter in the log (in this case, the host IP). The single quotes result in the first single quote being recorded in the log. E.g.: '66.249.72.57 '157.55.17.194 Are you running Varnish on Linux with bash, or some other OS flavour? This one works for me: LOG_FORMAT="%h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\" %{Host}i %{Varnish:time_firstbyte}x" DAEMON_OPTS="-c -a -w $logfile -D -P $pidfile -F '${LOG_FORMAT}'" Please, note the single quotes in -F Hope this help. From jon.hauge at relationhouse.com Sun May 27 16:46:47 2012 From: jon.hauge at relationhouse.com (Jon Hauge) Date: Sun, 27 May 2012 18:46:47 +0200 Subject: guro Message-ID: <22629abc65c8b84673389a658a2df293@mail.gmail.com> In my VCL I have: set req.grace = 12h; and set beresp.grace = 15h; But I still get a lot of guros e.g. on a site frontpage that has a ttl of 2 mins Why don?t varnish not just serve the cached version? I know for sure that the frontpage is served at least once without problem every 12 hours from the backend. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rekcah78 at gmail.com Sun May 27 20:20:21 2012 From: rekcah78 at gmail.com (Christophe Gasmi) Date: Sun, 27 May 2012 22:20:21 +0200 Subject: varnishncsa -F flag custom log format In-Reply-To: <001e01cd3c12$64ad2880$2e077980$@co.uk> References: <000301cd3b85$f865e950$e931bbf0$@co.uk> <001e01cd3c12$64ad2880$2e077980$@co.uk> Message-ID: Hi, in the init file (Ubuntu): I set: + LOG_FORMAT="%{Host}i %h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\"" - DAEMON_OPTS="-a -w ${LOGFILE} -D -P ${PIDFILE}" + DAEMON_OPTS="-a -w ${LOGFILE} -D -P ${PIDFILE} -F" - --chuid $USER --exec ${DAEMON} -- ${DAEMON_OPTS} \ + --chuid $USER --exec ${DAEMON} -- ${DAEMON_OPTS} "${LOG_FORMAT}" \ Now I have a correct log line: www.XXXXXX.com 2a01:e35:2f16:....:21b:..ff:....:1c0f - - [27/May/2012:22:07:05 +0200] "GET http://www.XXXXXX.com/favicon.ico HTTP/1.1" 404 188 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.52 Safari/536.5" But now I want to change the value %r not contain http://www.XXXXXX.com :) Christophe 2012/5/27 Rowan @ Jetboy : > Hi Daniel, > > Thanks for the reply. I've tried entirely replacing my code with yours > (other than the capitalisation of $PIDFILE, and putting $logfile inline), > but I get similar results. It's only recording the first log parameter in > the log (in this case, the host IP). The single quotes result in the first > single quote being recorded in the log. E.g.: > > '66.249.72.57 > '157.55.17.194 > > Are you running Varnish on Linux with bash, or some other OS flavour? > > > > > This one works for me: > > LOG_FORMAT="%h %l %u %t \"%r\" %s %b \"%{Referer}i\" > \"%{User-agent}i\" %{Host}i %{Varnish:time_firstbyte}x" > DAEMON_OPTS="-c -a -w $logfile -D -P $pidfile -F '${LOG_FORMAT}'" > > Please, note the single quotes in -F > > Hope this help. > > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From arundobriyaliitkgp at gmail.com Mon May 28 12:39:32 2012 From: arundobriyaliitkgp at gmail.com (Arun Dobriyal) Date: Mon, 28 May 2012 18:09:32 +0530 Subject: Including library to compile default.vcl Message-ID: Hi, I am extending the default.vcl file, by including some library and doing the function calls to this library.. Now while generating the c code by varnishd command, varnish reports the missing library path (because I didnt know where to include the library) Had it been a normal c file, I would have compiled using -L option, how to compile the default.vcl in this case and where to include libraries ? -Arun -------------- next part -------------- An HTML attachment was scrubbed... URL: From hugo.cisneiros at gmail.com Mon May 28 16:54:39 2012 From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch)) Date: Mon, 28 May 2012 13:54:39 -0300 Subject: Problems with varnish backend health In-Reply-To: <8A76B2DB46634944BEEBB218DAE22BC20CCEF0DA@exchange-srv2.office.radioteleffh.de> References: <8A76B2DB46634944BEEBB218DAE22BC20CCEF0DA@exchange-srv2.office.radioteleffh.de> Message-ID: On Thu, May 24, 2012 at 3:03 AM, Winkelmann, Thomas (Radio/Tele FFH - Online) wrote: > Hello, > > we have a setup with two varnish servers (version 3.0.2) and two backend > servers. This setup works fine, if we split the requests with an > loadbalancer to the two varnish servers. Around 500req/s 20MBit/s bandwidth > on each server. > > In Addition we have defined a backend probe which will tests the backend > each 10s. This also works fine. > > Now, if we switch all requests to one varnish server, after some time > (around 20 seconds) the backends will be marked as sick, but they are both > up and answering all requests. Also without any backend probe defined in > VCL, varnish gets in trouble serving content. Some months ago, we had no > problems with this and served more than 400Mbit/s from one single server. I don't really know exactly what could be, but sometimes when I get backend errors (missconfiguration or overload on servers), and begin debugging with varnishlog: varnishlog -i Backend_health ( | "grep Backend_health" to show only the probes) This usually gives me the return code from the probes and any problems with it. If the server begins to return a different code than HTTP 200, you can see exactly when this happens... It's a start :) -- []'s Hugo www.devin.com.br From hugo.cisneiros at gmail.com Mon May 28 23:02:08 2012 From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch)) Date: Mon, 28 May 2012 20:02:08 -0300 Subject: Including library to compile default.vcl In-Reply-To: References: Message-ID: On Mon, May 28, 2012 at 9:39 AM, Arun Dobriyal wrote: > I am extending the default.vcl file, by including some library and doing the > function calls to this library.. > > Now while generating the c code by varnishd command, varnish reports the > missing library path (because I didnt know where to include the library) > Had it been a normal c file, I would have compiled using -L option, how to > compile the default.vcl in this case and where to include libraries ? Hello, Use the cc_command command line option (-p) to pass the extra options to the VCL compiler. The command 'param.show' will show you the run time parameters used. You can set this option on /etc/sysconfig/varnish (rh/centos/fedora) or /etc/default/varnish (debian/ubuntu). If you're using Debian/Ubuntu, you might get into trouble because the init script isn't accepting spaces on the -p options. This posts shows how to fix this: http://my.opera.com/cstrep/blog/2011/11/30/how-to-start-up-varnish-with-a-custom-cc-command-on-debian -- []'s Hugo www.devin.com.br From nick.tailor at gmail.com Tue May 29 17:10:29 2012 From: nick.tailor at gmail.com (nick tailor) Date: Tue, 29 May 2012 10:10:29 -0700 Subject: Question Message-ID: I have a bunch of drupal sites. However, even with varnish they take a really long time to load and the varnish log is showing that its caching. However, if I clear my browser cache and reload the site it takes like 10 seconds before it loads. Any ideas? not sure how to fix this? Cheers Nick Tailor -------------- next part -------------- An HTML attachment was scrubbed... URL: From ketil at froyn.name Tue May 29 17:38:37 2012 From: ketil at froyn.name (Ketil Froyn) Date: Tue, 29 May 2012 19:38:37 +0200 Subject: Question In-Reply-To: References: Message-ID: Perhaps you should also have a look at the pagespeed tools, and also the best practices. https://developers.google.com/speed/pagespeed/ -Ketil On May 29, 2012 7:13 PM, "nick tailor" wrote: > I have a bunch of drupal sites. However, even with varnish they take a > really long time to load and the varnish log is showing that its caching. > > However, if I clear my browser cache and reload the site it takes like 10 > seconds before it loads. Any ideas? not sure how to fix this? > > Cheers > > Nick Tailor > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From scott.stewart at nbtsolutions.com Wed May 30 19:50:47 2012 From: scott.stewart at nbtsolutions.com (Scott Stewart) Date: Wed, 30 May 2012 15:50:47 -0400 Subject: Basic Auth Message-ID: Hi I have a newly minted varnish (and a newly minted varnish user myself) install that has two backends, as spec'ed in this vcl (below). The "wms1" has no auth on it. The "default" backend is a Apache server with basic auth, but no one is getting asked to authenticate. My understanding was the out of the box config for varnish would not cache those authentication required pages, but no one is being asked for a username/password What am I doing wrong here? The app works perfectly otherwise. # This is a basic VCL configuration file for varnish. See the vcl(7) # man page for details on VCL syntax and semantics. # # Default backend definition. Set this to point to your content # server. # backend default { .host = "127.0.0.1"; .port = "8081"; } backend wms1 { .host = "example.com"; .port = "80"; } sub vcl_recv { if (req.http.host ~ "^(mapsdev\.)example\.com" && req.url~ "^/wms") { set req.http.host = "maps.example.com"; set req.url = regsub(req.url, "^/wms", "/wms"); set req.backend = wms1; if (req.request == "GET" && req.http.cookie) { unset req.http.cookie; } } if (req.http.host ~ "^(mapsdev\.)?example\.com" && req.url~ "^/app/") { set req.url = regsub(req.url, "^/app/", "/flol/admin_2/public/"); remove req.http.X-Forwarded-For; set req.http.X-Forwarded-For = client.ip; } sub vcl_fetch { set beresp.ttl = 72h; // default ttl 72 hours if (req.http.Authorization) { return(hit_for_pass); } } # # Below is a commented-out copy of the default VCL logic. If you # redefine any of these subroutines, the built-in logic will be # appended to your code. # sub vcl_recv { # if (req.restarts == 0) { # if (req.http.x-forwarded-for) { # set req.http.X-Forwarded-For = # req.http.X-Forwarded-For + ", " + client.ip; # } else { # set req.http.X-Forwarded-For = client.ip; # } # } # if (req.request != "GET" && # req.request != "HEAD" && # req.request != "PUT" && # req.request != "POST" && # req.request != "TRACE" && # req.request != "OPTIONS" && # req.request != "DELETE") { # /* Non-RFC2616 or CONNECT which is weird. */ # return (pipe); # } # if (req.request != "GET" && req.request != "HEAD") { # /* We only deal with GET and HEAD by default */ # return (pass); # } # if (req.http.Authorization || req.http.Cookie) { # /* Not cacheable by default */ # return (pass); # } # return (lookup); # } # # sub vcl_pipe { # # Note that only the first request to the backend will have # # X-Forwarded-For set. If you use X-Forwarded-For and want to # # have it set for all requests, make sure to have: # # set bereq.http.connection = "close"; # # here. It is not set by default as it might break some broken web # # applications, like IIS with NTLM authentication. # return (pipe); # } # # sub vcl_pass { # return (pass); # } # # sub vcl_hash { # hash_data(req.url); # if (req.http.host) { # hash_data(req.http.host); # } else { # hash_data(server.ip); # } # return (hash); # } # # sub vcl_hit { # return (deliver); # } # # sub vcl_miss { # return (fetch); # } # # sub vcl_fetch { # if (beresp.ttl <= 0s || # beresp.http.Set-Cookie || # beresp.http.Vary == "*") { # /* # * Mark as "Hit-For-Pass" for the next 2 minutes # */ # set beresp.ttl = 120 s; # return (hit_for_pass); # } # return (deliver); # } # # sub vcl_deliver { # return (deliver); # } # # sub vcl_error { # set obj.http.Content-Type = "text/html; charset=utf-8"; # set obj.http.Retry-After = "5"; # synthetic {" # # # # # "} + obj.status + " " + obj.response + {" # # #

Error "} + obj.status + " " + obj.response + {"

#

"} + obj.response + {"

#

Guru Meditation:

#

XID: "} + req.xid + {"

#
#

Varnish cache server

# # # "}; # return (deliver); # } # # sub vcl_init { # return (ok); # } # # sub vcl_fini { # return (ok); # } -------------- next part -------------- An HTML attachment was scrubbed... URL: From hugo.cisneiros at gmail.com Thu May 31 17:56:32 2012 From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch)) Date: Thu, 31 May 2012 14:56:32 -0300 Subject: Basic Auth In-Reply-To: References: Message-ID: On Wed, May 30, 2012 at 4:50 PM, Scott Stewart wrote: > I have a newly minted varnish (and a newly minted?varnish?user myself) > install that has two backends, ?as spec'ed in this vcl (below). > > The "wms1" has no auth on it. > > The "default" backend is a Apache server with basic auth, but no one is > getting asked to authenticate. My understanding was the out of the box > config for varnish would not cache those authentication required pages, but > no one is being asked for a username/password > > What am I doing wrong here? The app works perfectly otherwise. The default varnish config doesn't cache when Authentication is required. But since you redefined the vcl_recv, it isn't using the default rules. Commented out, are the lines that do this: # ? ? if (req.http.Authorization || req.http.Cookie) { # ? ? ? ? /* Not cacheable by default */ # ? ? ? ? return (pass); # ? ? } -- []'s Hugo www.devin.com.br From simon.shepard at googlemail.com Wed May 9 14:12:22 2012 From: simon.shepard at googlemail.com (Simon Kenyon Shepard) Date: Wed, 09 May 2012 14:12:22 -0000 Subject: using varnish cache behind a http proxy Message-ID: Hi list, I have a question, I have googled it but unfortunately the question lends itself to exactly what Varnish is used for, so it was not very helpful. I would like to run Varnish on my servers as a caching proxy, the only catch is that to access the outside internet I have to go through another proxy that is not controlled by me, that is set up with no caching. Is there anyway to set a http proxy for Varnish to route it's requests to the outside world through another proxy? Thanks for your help! Simon -- "We are the music makers, And we are the dreamers of dreams" -------------- next part -------------- An HTML attachment was scrubbed... URL: From kofreestyler at gmail.com Wed May 16 09:58:19 2012 From: kofreestyler at gmail.com (Zhiping Deng) Date: Wed, 16 May 2012 09:58:19 -0000 Subject: varnish serving a wrong file Message-ID: Hi, I have been using varnish for about 2years. Recently I discovered that sometimes it served a wrong file. for example: The url is /foo/a.jpg, but the response content is actually from another file (/foo/b.jpg) I upgrade to varnish-3.0.2, and it still happens sometimes. What's the possible reasons of this problem? Thanks, Zhiping Deng From barrycie at aol.com Mon May 21 21:12:06 2012 From: barrycie at aol.com (Barry Conroy) Date: Mon, 21 May 2012 22:12:06 +0100 Subject: Varnish - high session_linger and session herd Message-ID: <4FBAAFA6.2030606@aol.com> Hi all, Ive been testing Varnish over the past while and have a few questions with which i hope you can help. We have deployed Varnish 3.0.2 in front of our XML api. During my load testing Ive noticed that the session_linger and session_herd stats are quite high. Does this signal an issue? Also, during my load testing Im getting an error for 0.04% of requests to Varnish - "readResponse: java.net.SocketException: Connection reset ". Has anyone seen this before? Are these connections getting closed on the Varnish side? I had been using the default setting for session_linger but changed it to 0. Do I need session linger considering I am not caching a web page but am instead caching XML api responses? I was surprised how little memory Varnish is using to cache its objects - this is reflect in the line SMA.s0.g_bytes 2109235 . Bytes outstanding The following are the output from varnishstat -1 and also the param.show. The VCL in play is very simple and passes through 1 request type and caches everything else. Thanks for you help in advance, Barry client_conn 5990408 14.10 Client connections accepted client_drop 0 0.00 Connection dropped, no sess/wrk client_req 2351555655 5534.53 Client requests received cache_hit 2346417108 5522.44 Cache hits cache_hitpass 0 0.00 Cache hits for pass cache_miss 4836412 11.38 Cache misses backend_conn 11271 0.03 Backend conn. success backend_unhealthy 0 0.00 Backend conn. not attempted backend_busy 0 0.00 Backend conn. too many backend_fail 2 0.00 Backend conn. failures backend_reuse 5127357 12.07 Backend conn. reuses backend_toolate 1044 0.00 Backend conn. was closed backend_recycle 5128407 12.07 Backend conn. recycles backend_retry 99 0.00 Backend conn. retry fetch_head 295036 0.69 Fetch head fetch_length 4836477 11.38 Fetch with Length fetch_chunked 7012 0.02 Fetch chunked fetch_eof 0 0.00 Fetch EOF fetch_bad 0 0.00 Fetch had bad headers fetch_close 0 0.00 Fetch wanted close fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed fetch_zero 0 0.00 Fetch zero len fetch_failed 0 0.00 Fetch failed fetch_1xx 0 0.00 Fetch no body (1xx) fetch_204 0 0.00 Fetch no body (204) fetch_304 0 0.00 Fetch no body (304) n_sess_mem 1251 . N struct sess_mem n_sess 789 . N struct sess n_object 453 . N struct object n_vampireobject 0 . N unresurrected objects n_objectcore 627 . N struct objectcore n_objecthead 1859 . N struct objecthead n_waitinglist 15213 . N struct waitinglist n_vbc 5 . N struct vbc n_wrk 174 . N worker threads n_wrk_create 10484 0.02 N worker threads created n_wrk_failed 0 0.00 N worker threads not created n_wrk_max 0 0.00 N worker threads limited n_wrk_lqueue 0 0.00 work request queue length n_wrk_queued 514194 1.21 N queued work requests n_wrk_drop 0 0.00 N dropped work requests n_backend 1 . N backends n_expired 4835952 . N expired objects n_lru_nuked 0 . N LRU nuked objects n_lru_moved 33485426 . N LRU moved objects losthdr 0 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 2351109391 5533.48 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 5990506 14.10 Total Sessions s_req 2351555664 5534.53 Total Requests s_pipe 0 0.00 Total pipe s_pass 302120 0.71 Total pass s_fetch 5138525 12.09 Total fetch s_hdrbytes 544265355452 1280961.94 Total header bytes s_bodybytes 3089476577758 7271272.85 Total body bytes sess_closed 227804 0.54 Session Closed sess_pipeline 0 0.00 Session Pipeline sess_readahead 0 0.00 Session Read Ahead sess_linger 2209279166 5199.67 Session Linger sess_herd 323937467 762.41 Session herd shm_records 89745845415 211222.36 SHM records shm_writes 2562338698 6030.62 SHM writes shm_flushes 62 0.00 SHM flushes due to overflow shm_cont 27674729 65.13 SHM MTX contention shm_cycles 43605 0.10 SHM cycles through buffer sms_nreq 22 0.00 SMS allocator requests sms_nobj 0 . SMS outstanding allocations sms_nbytes 0 . SMS outstanding bytes sms_balloc 8856 . SMS bytes allocated sms_bfree 8856 . SMS bytes freed backend_req 5138625 12.09 Backend requests made n_vcl 1 0.00 N vcl total n_vcl_avail 1 0.00 N vcl available n_vcl_discard 0 0.00 N vcl discarded n_ban 1 . N total active bans n_ban_add 1 0.00 N new bans added n_ban_retire 0 0.00 N old bans deleted n_ban_obj_test 0 0.00 N objects tested n_ban_re_test 0 0.00 N regexps tested against n_ban_dups 0 0.00 N duplicate bans removed hcb_nolock 2351224626 5533.75 HCB Lookups without lock hcb_lock 1666108 3.92 HCB Lookups with lock hcb_insert 1666107 3.92 HCB Inserts esi_errors 0 0.00 ESI parse errors (unlock) esi_warnings 0 0.00 ESI parse warnings (unlock) accept_fail 0 0.00 Accept failures client_drop_late 0 0.00 Connection dropped late uptime 424888 1.00 Client uptime dir_dns_lookups 0 0.00 DNS director lookups dir_dns_failed 0 0.00 DNS director failed lookups dir_dns_hit 0 0.00 DNS director cached lookups hit dir_dns_cache_full 0 0.00 DNS director full dnscache vmods 0 . Loaded VMODs n_gzip 0 0.00 Gzip operations n_gunzip 2355834985 5544.60 Gunzip operations LCK.sms.creat 1 0.00 Created locks LCK.sms.destroy 0 0.00 Destroyed locks LCK.sms.locks 66 0.00 Lock Operations LCK.sms.colls 0 0.00 Collisions LCK.smp.creat 0 0.00 Created locks LCK.smp.destroy 0 0.00 Destroyed locks LCK.smp.locks 0 0.00 Lock Operations LCK.smp.colls 0 0.00 Collisions LCK.sma.creat 2 0.00 Created locks LCK.sma.destroy 0 0.00 Destroyed locks LCK.sma.locks 24799574 58.37 Lock Operations LCK.sma.colls 0 0.00 Collisions LCK.smf.creat 0 0.00 Created locks LCK.smf.destroy 0 0.00 Destroyed locks LCK.smf.locks 0 0.00 Lock Operations LCK.smf.colls 0 0.00 Collisions LCK.hsl.creat 0 0.00 Created locks LCK.hsl.destroy 0 0.00 Destroyed locks LCK.hsl.locks 0 0.00 Lock Operations LCK.hsl.colls 0 0.00 Collisions LCK.hcb.creat 1 0.00 Created locks LCK.hcb.destroy 0 0.00 Destroyed locks LCK.hcb.locks 3334164 7.85 Lock Operations LCK.hcb.colls 0 0.00 Collisions LCK.hcl.creat 0 0.00 Created locks LCK.hcl.destroy 0 0.00 Destroyed locks LCK.hcl.locks 0 0.00 Lock Operations LCK.hcl.colls 0 0.00 Collisions LCK.vcl.creat 1 0.00 Created locks LCK.vcl.destroy 0 0.00 Destroyed locks LCK.vcl.locks 32230 0.08 Lock Operations LCK.vcl.colls 0 0.00 Collisions LCK.stat.creat 1 0.00 Created locks LCK.stat.destroy 0 0.00 Destroyed locks LCK.stat.locks 1251 0.00 Lock Operations LCK.stat.colls 0 0.00 Collisions LCK.sessmem.creat 1 0.00 Created locks LCK.sessmem.destroy 0 0.00 Destroyed locks LCK.sessmem.locks 5999643 14.12 Lock Operations LCK.sessmem.colls 0 0.00 Collisions LCK.wstat.creat 1 0.00 Created locks LCK.wstat.destroy 0 0.00 Destroyed locks LCK.wstat.locks 155943114 367.02 Lock Operations LCK.wstat.colls 0 0.00 Collisions LCK.herder.creat 1 0.00 Created locks LCK.herder.destroy 0 0.00 Destroyed locks LCK.herder.locks 25245 0.06 Lock Operations LCK.herder.colls 0 0.00 Collisions LCK.wq.creat 2 0.00 Created locks LCK.wq.destroy 0 0.00 Destroyed locks LCK.wq.locks 649162540 1527.84 Lock Operations LCK.wq.colls 0 0.00 Collisions LCK.objhdr.creat 1676572 3.95 Created locks LCK.objhdr.destroy 1674713 3.94 Destroyed locks LCK.objhdr.locks 9417295559 22164.18 Lock Operations LCK.objhdr.colls 0 0.00 Collisions LCK.exp.creat 1 0.00 Created locks LCK.exp.destroy 0 0.00 Destroyed locks LCK.exp.locks 10095406 23.76 Lock Operations LCK.exp.colls 0 0.00 Collisions LCK.lru.creat 2 0.00 Created locks LCK.lru.destroy 0 0.00 Destroyed locks LCK.lru.locks 4836405 11.38 Lock Operations LCK.lru.colls 0 0.00 Collisions LCK.cli.creat 1 0.00 Created locks LCK.cli.destroy 0 0.00 Destroyed locks LCK.cli.locks 141594 0.33 Lock Operations LCK.cli.colls 0 0.00 Collisions LCK.ban.creat 1 0.00 Created locks LCK.ban.destroy 0 0.00 Destroyed locks LCK.ban.locks 10095484 23.76 Lock Operations LCK.ban.colls 0 0.00 Collisions LCK.vbp.creat 1 0.00 Created locks LCK.vbp.destroy 0 0.00 Destroyed locks LCK.vbp.locks 0 0.00 Lock Operations LCK.vbp.colls 0 0.00 Collisions LCK.vbe.creat 1 0.00 Created locks LCK.vbe.destroy 0 0.00 Destroyed locks LCK.vbe.locks 22541 0.05 Lock Operations LCK.vbe.colls 0 0.00 Collisions LCK.backend.creat 1 0.00 Created locks LCK.backend.destroy 0 0.00 Destroyed locks LCK.backend.locks 13459516 31.68 Lock Operations LCK.backend.colls 0 0.00 Collisions SMA.s0.c_req 9672832 22.77 Allocator requests SMA.s0.c_fail 0 0.00 Allocator failures SMA.s0.c_bytes 637361546992 1500069.54 Bytes allocated SMA.s0.c_freed 637360393374 1500066.83 Bytes freed SMA.s0.g_alloc 906 . Allocations outstanding SMA.s0.g_bytes 1153618 . Bytes outstanding SMA.s0.g_space 1072588206 . Bytes available SMA.Transient.c_req 309204 0.73 Allocator requests SMA.Transient.c_fail 0 0.00 Allocator failures SMA.Transient.c_bytes 212330558 499.73 Bytes allocated SMA.Transient.c_freed 212330558 499.73 Bytes freed SMA.Transient.g_alloc 0 . Allocations outstanding SMA.Transient.g_bytes 0 . Bytes outstanding SMA.Transient.g_space 0 . Bytes available VBE.default(172.29.101.121,,8080).vcls 1 . VCL references VBE.default(172.29.101.121,,8080).happy 0 . Happy health probes ------------------------------ acceptor_sleep_decay 0.900000 [] acceptor_sleep_incr 0.001000 [s] acceptor_sleep_max 0.050000 [s] auto_restart on [bool] ban_dups on [bool] ban_lurker_sleep 0.010000 [s] between_bytes_timeout 60.000000 [s] cc_command "exec gcc -std=gnu99 -pthread -fpic -shared -Wl,-x -o %o %s" cli_buffer 8192 [bytes] cli_timeout 10 [seconds] clock_skew 10 [s] connect_timeout 0.700000 [s] critbit_cooloff 180.000000 [s] default_grace 10.000000 [seconds] default_keep 0.000000 [seconds] default_ttl 30.000000 [seconds] diag_bitmap 0x0 [bitmap] esi_syntax 0 [bitmap] expiry_sleep 1.000000 [seconds] fetch_chunksize 128 [kilobytes] fetch_maxchunksize 262144 [kilobytes] first_byte_timeout 60.000000 [s] gzip_level 6 [] gzip_memlevel 8 [] gzip_stack_buffer 32768 [Bytes] gzip_tmp_space 0 [] gzip_window 15 [] http_gzip_support on [bool] http_max_hdr 64 [header lines] http_range_support on [bool] http_req_hdr_len 8192 [bytes] http_req_size 32768 [bytes] http_resp_hdr_len 8192 [bytes] http_resp_size 32768 [bytes] listen_address 0.0.0.0:10300 listen_depth 1024 [connections] log_hashstring on [bool] log_local_address off [bool] lru_interval 2 [seconds] max_esi_depth 5 [levels] max_restarts 4 [restarts] nuke_limit 50 [allocations] ping_interval 3 [seconds] pipe_timeout 60 [seconds] prefer_ipv6 off [bool] queue_max 100 [%] rush_exponent 3 [requests per request] saintmode_threshold 10 [objects] send_timeout 60 [seconds] sess_timeout 5 [seconds] sess_workspace 65536 [bytes] session_linger 0 [ms] session_max 100000 [sessions] shm_reclen 255 [bytes] shm_workspace 8192 [bytes] shortlived 10.000000 [s] syslog_cli_traffic on [bool] thread_pool_add_delay 2 [milliseconds] thread_pool_add_threshold 2 [requests] thread_pool_fail_delay 200 [milliseconds] thread_pool_max 1000 [threads] thread_pool_min 60 [threads] thread_pool_purge_delay 1000 [milliseconds] thread_pool_stack unlimited [bytes] thread_pool_timeout 120 [seconds] thread_pool_workspace 65536 [bytes] thread_pools 2 [pools] thread_stats_rate 10 [requests] vcc_err_unref on [bool] vcl_trace off [bool] waiter default (epoll, poll) -------------- next part -------------- An HTML attachment was scrubbed... URL: From silentneedle at Safe-mail.net Wed May 23 09:41:05 2012 From: silentneedle at Safe-mail.net (silentneedle at Safe-mail.net) Date: Wed, 23 May 2012 05:41:05 -0400 Subject: Multi-language (cookie) site configuration Message-ID: Hey list, I'm trying to setup varnish for a multi-language vbulletin forum. The language is set through a cookie (bblanguageid). My current configuration is: http://pastebin.com/wCKZUcTX Now I've tried to cache a separate version if the bblanguageid=2 cookie is present: sub vcl_hash { if (req.http.cookie ~ "bblanguageid=2") { set req.hash += "deDE"; } } But it doesn't seem to work, it still always show the language at cache generation. Is there anything else that I need to do? Thanks! From ganesh at corp.india.com Tue May 29 09:51:35 2012 From: ganesh at corp.india.com (Ganesh Raut) Date: Tue, 29 May 2012 15:21:35 +0530 Subject: Using varnish to cache remote content Message-ID: How do i cache the external content in varnish...e.g js,css,images etc.... can you please send us example of varnish files. -- Best Regards, Ganesh Raut -------------- next part -------------- An HTML attachment was scrubbed... URL: