From angie.tawfik at gmail.com Tue Jun 1 09:45:37 2010 From: angie.tawfik at gmail.com (Angie T. Muhammad) Date: Tue, 1 Jun 2010 12:45:37 +0300 Subject: Parsing non-English URLs In-Reply-To: <877hmk61k6.fsf@qurzaw.linpro.no> References: <877hmk61k6.fsf@qurzaw.linpro.no> Message-ID: Thank you Tollef. I create the log using: # varnishlog -a -w /var/log/varnish/varnish.log & And now I read it using: # varnishlog -r /var/log/varnish/varnish.log I get some lines like: 14 TxHeader b Referer: http://myserver.example.com/news/%D8%A7%D9%84%D9%85%D9%84%D9%8A%D8%A7%D8%B1%D8%AF%D9%8A%D8%B1-%D8%A7%D9%84%D9%85%D8%B5%D8%B1%D9%8A-%C2%AB%D9%85%D8%AD%D9%85%D8%AF-%D8%A7%D9%84%D9%81%D8%A7%D9%8A%D8%AF%C2%BB-%D9%8A%D9%82%D8%B1%D8%B1-%D8%A8%D9%8A%D The point is that I need to make varnish accept such percentages or any other syntax in VCL file. So that some pages are cached longer based on their Arabic address (Like the one in the referer above). Any ideas? Thank you in advance :) On Mon, May 31, 2010 at 9:13 AM, Tollef Fog Heen < tfheen at varnish-software.com> wrote: > ]] "Angie T. Muhammad" > > | I tried opening the log file with less, vim, and tail but all what am > | getting is either binary (less) or stuff like above (tail). > > Use varnishlog(1) to read the varnishlog. > > tail and less won't do the right thing as the log file is a circular > buffer and they have no idea where varnish is currently writing. > > -- > Tollef Fog Heen > Varnish Software > t: +47 21 54 41 73 > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > -- All the best, Angie T. Muhammad Linux Systems Engineer Al Masry Al Youm http://www.almasryalyoum.com/en -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at mgoldman.com Tue Jun 1 12:07:26 2010 From: me at mgoldman.com (Martin Goldman) Date: Tue, 1 Jun 2010 08:07:26 -0400 Subject: Changing req.hash Message-ID: Hi folks, We're adding a mobile version of our site. We don't want to redirect mobile users to a different URL, we want to just present the mobile site to visitors who access our site with a mobile-like useragent. So I got the idea from this site (http://groups.drupal.org/node/63203) to change req.hash for mobile requests so that they'll get cached separately using logic like this: sub vcl_hash { /* Have a separate object cache for mobile site based on User-Agent */ if (req.http.host == "www.domain.org" && req.http.User-Agent ~ "(iPhone|iPod)") { set req.hash += "mobile"; } } This basically works as expected, but I'm having one issue. We programmatically purges pages from Varnish when they're updated. Obviously to do so, we need to know the hash of the page. The hash syntax we've always used, which works fine for the non-mobile requests, is: ^/this/is/the/uri#mysite.com#$ But I'm not totally clear on how appending "mobile" to the hash changes the syntax. I remember reading once that custom VCL runs *before* the default VCL, so I thought the hash would be something like this: ^mobile/this/is/the/uri#mysite.com#$ However, purging that doesn't work. Any ideas? Apologies if the answer to this is out there somewhere, I just haven't had any luck finding it. Thanks and regards, Martin -------------- next part -------------- An HTML attachment was scrubbed... URL: From vburshteyn at gmail.com Tue Jun 1 16:39:38 2010 From: vburshteyn at gmail.com (vitaly burshteyn) Date: Tue, 1 Jun 2010 12:39:38 -0400 Subject: wont server from Cache Message-ID: Hi folks, So for last few days i have been trying to get my varnish box to server files from cache, and not having much luck. Even though its telling me it is hitting the cache its not delivering files from it, since when i shut down the appache i get 503 errors. 232 Client requests received 130 Cache hits 5 Cache hits for pass 97 Cache misses 99 Backend conn. success 3 Backend conn. failures 99 Fetch with Length 15 N struct sess_mem Here is the copy of my vcl file with out the backend info.. Any and all suggestion would be appreciated. sub vcl_recv { #set req.backend = bw; set req.grace = 60m; if (req.request == "PURGE") {if (!client.ip ~ purge) {error 405 "Not allowed.";} return(lookup);} if (req.request != "GET" && req.request != "HEAD") {return(lookup);} if (req.http.Cache-Control ~ "no-cache") {purge_url(req.url);} return(lookup); } sub vcl_hit { if (!obj.cacheable) { return(pass); }} sub vcl_fetch { if (beresp.status == 500 || beresp.status == 503 || beresp.status == 504) { restart; } set beresp.ttl = 600s; set beresp.grace = 600s; if (beresp.ttl < 300s) { set beresp.ttl = 300s;} if (!beresp.cacheable) {return(pass);} if (beresp.http.Set-Cookie) {return(pass);} } -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfheen at varnish-software.com Wed Jun 2 07:59:18 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Wed, 02 Jun 2010 09:59:18 +0200 Subject: Parsing non-English URLs In-Reply-To: (Angie T. Muhammad's message of "Tue, 1 Jun 2010 12:45:37 +0300") References: <877hmk61k6.fsf@qurzaw.linpro.no> Message-ID: <871vcpoofd.fsf@qurzaw.linpro.no> ]] "Angie T. Muhammad" | I get some lines like: | 14 TxHeader b Referer: | http://myserver.example.com/news/%D8%A7%D9%84%D9%85%D9%84%D9%8A%D8%A7%D8%B1%D8%AF%D9%8A%D8%B1-%D8%A7%D9%84%D9%85%D8%B5%D8%B1%D9%8A-%C2%AB%D9%85%D8%AD%D9%85%D8%AF-%D8%A7%D9%84%D9%81%D8%A7%D9%8A%D8%AF%C2%BB-%D9%8A%D9%82%D8%B1%D8%B1-%D8%A8%D9%8A%D | | The point is that I need to make varnish accept such percentages or any | other syntax in VCL file. So that some pages are cached longer based on | their Arabic address (Like the one in the referer above). We use %-escaped strings in vcl, so referring to that string above would be to match it to ?http://myserver.example.com/news/%25D8%25A7%25D9%2584%25D9%2585%25D9%2584%25D9%258A%25D8%25A7%25D8%25B1%25D8%25AF%25D9%258A%25D8%25B1-%25D8%25A7%25D9%2584%25D9%2585%25D8%25B5%25D8%25B1%25D9%258A-%25C2%25AB%25D9%2585%25D8%25AD%25D9%2585%25D8%25AF-%25D8%25A7%25D9%2584%25D9%2581%25D8%25A7%25D9%258A%25D8%25AF%25C2%25BB-%25D9%258A%25D9%2582%25D8%25B1%25D8%25B1-%25D8%25A8%25D9%258A%25D? or more easily {"http://myserver.example.com/news/%D8%A7%D9%84%D9%85%D9%84%D9%8A%D8%A7%D8%B1%D8%AF%D9%8A%D8%B1-%D8%A7%D9%84%D9%85%D8%B5%D8%B1%D9%8A-%C2%AB%D9%85%D8%AD%D9%85%D8%AF-%D8%A7%D9%84%D9%81%D8%A7%D9%8A%D8%AF%C2%BB-%D9%8A%D9%82%D8%B1%D8%B1-%D8%A8%D9%8A%D"} -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From tfheen at varnish-software.com Wed Jun 2 08:01:37 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Wed, 02 Jun 2010 10:01:37 +0200 Subject: Changing req.hash In-Reply-To: (Martin Goldman's message of "Tue, 1 Jun 2010 08:07:26 -0400") References: Message-ID: <87wruhn9r2.fsf@qurzaw.linpro.no> ]] Martin Goldman | But I'm not totally clear on how appending "mobile" to the hash changes the | syntax. I remember reading once that custom VCL runs *before* the default | VCL, so I thought the hash would be something like this: | | ^mobile/this/is/the/uri#mysite.com#$ | | However, purging that doesn't work. Any ideas? It's more likely to be ^mobile#/this/is/the/uri#mysite.com#$, but why don't you just purge by host and url instead? purge("req.url == " req.url " && req.http.host == " req.http.host); at the right place in your VCL. -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From angie.tawfik at gmail.com Wed Jun 2 12:56:29 2010 From: angie.tawfik at gmail.com (Angie T. Muhammad) Date: Wed, 2 Jun 2010 15:56:29 +0300 Subject: Parsing non-English URLs In-Reply-To: <871vcpoofd.fsf@qurzaw.linpro.no> References: <877hmk61k6.fsf@qurzaw.linpro.no> <871vcpoofd.fsf@qurzaw.linpro.no> Message-ID: I tried the second syntax many times before. Varnish refuses to start. But the good news it accepted the first syntax. I will be checking if it applies the rule as required and update you. Thank you Tollef :) On 6/2/10, Tollef Fog Heen wrote: > ]] "Angie T. Muhammad" > > | I get some lines like: > | 14 TxHeader b Referer: > | > http://myserver.example.com/news/%D8%A7%D9%84%D9%85%D9%84%D9%8A%D8%A7%D8%B1%D8%AF%D9%8A%D8%B1-%D8%A7%D9%84%D9%85%D8%B5%D8%B1%D9%8A-%C2%AB%D9%85%D8%AD%D9%85%D8%AF-%D8%A7%D9%84%D9%81%D8%A7%D9%8A%D8%AF%C2%BB-%D9%8A%D9%82%D8%B1%D8%B1-%D8%A8%D9%8A%D > | > | The point is that I need to make varnish accept such percentages or any > | other syntax in VCL file. So that some pages are cached longer based on > | their Arabic address (Like the one in the referer above). > > We use %-escaped strings in vcl, so referring to that string above would > be to match it to > > ?http://myserver.example.com/news/%25D8%25A7%25D9%2584%25D9%2585%25D9%2584%25D9%258A%25D8%25A7%25D8%25B1%25D8%25AF%25D9%258A%25D8%25B1-%25D8%25A7%25D9%2584%25D9%2585%25D8%25B5%25D8%25B1%25D9%258A-%25C2%25AB%25D9%2585%25D8%25AD%25D9%2585%25D8%25AF-%25D8%25A7%25D9%2584%25D9%2581%25D8%25A7%25D9%258A%25D8%25AF%25C2%25BB-%25D9%258A%25D9%2582%25D8%25B1%25D8%25B1-%25D8%25A8%25D9%258A%25D? > > or more easily > > {"http://myserver.example.com/news/%D8%A7%D9%84%D9%85%D9%84%D9%8A%D8%A7%D8%B1%D8%AF%D9%8A%D8%B1-%D8%A7%D9%84%D9%85%D8%B5%D8%B1%D9%8A-%C2%AB%D9%85%D8%AD%D9%85%D8%AF-%D8%A7%D9%84%D9%81%D8%A7%D9%8A%D8%AF%C2%BB-%D9%8A%D9%82%D8%B1%D8%B1-%D8%A8%D9%8A%D"} > > -- > Tollef Fog Heen > Varnish Software > t: +47 21 54 41 73 > -- All the best, Angie T. Muhammad Linux Systems Engineer Al Masry Al Youm http://www.almasryalyoum.com/en From me at mgoldman.com Wed Jun 2 13:15:03 2010 From: me at mgoldman.com (Martin Goldman) Date: Wed, 2 Jun 2010 09:15:03 -0400 Subject: Changing req.hash In-Reply-To: <87wruhn9r2.fsf@qurzaw.linpro.no> References: <87wruhn9r2.fsf@qurzaw.linpro.no> Message-ID: Thanks Tollef, it looks like that worked. The reason I'm trying to do this is that we don't have a separate URL for our mobile site -- users on both mobile and regular platforms access the URL, and then the application renders the mobile view of the site if the useragent matches an appropriate regex, and the regular view otherwise. By messing with the hash, we can cache the two versions separately. Thanks again, Martin On Wed, Jun 2, 2010 at 4:01 AM, Tollef Fog Heen wrote: > ]] Martin Goldman > > | But I'm not totally clear on how appending "mobile" to the hash changes > the > | syntax. I remember reading once that custom VCL runs *before* the default > | VCL, so I thought the hash would be something like this: > | > | ^mobile/this/is/the/uri#mysite.com#$ > | > | However, purging that doesn't work. Any ideas? > > It's more likely to be ^mobile#/this/is/the/uri#mysite.com#$, but why > don't you just purge by host and url instead? > > purge("req.url == " req.url " && req.http.host == " req.http.host); at > the right place in your VCL. > > -- > Tollef Fog Heen > Varnish Software > t: +47 21 54 41 73 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vburshteyn at gmail.com Wed Jun 2 13:24:20 2010 From: vburshteyn at gmail.com (vitaly burshteyn) Date: Wed, 2 Jun 2010 09:24:20 -0400 Subject: serving from cache Message-ID: Hi folks, So on my path to getting the website to server from cache instead of the back end i realized that i had a cookie issue (sorry i am new to Varnish) Cookie: affi=old; __ngt_cli=C8D6A206C800FDA28F4A7F07; __utma=32297913.1583573073.1275401467.1275478692.1275481795.6; So i changed my vcl_recv to strip out all the utma and other google analytics cookies. Yet for some reason no matter what i try i cant get it to strip affi=old; So i just want to confirm that untill this is empty: Set-Cookie: affi=old; Path=/^@^[^@^V^@^@^@ I wont be able to server from cache? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From vburshteyn at gmail.com Wed Jun 2 18:32:15 2010 From: vburshteyn at gmail.com (vitaly burshteyn) Date: Wed, 2 Jun 2010 14:32:15 -0400 Subject: cache/503 Message-ID: Hi folks, Just curious if you can point me in the right direction. I have 2.0.6 varnish running. So it seems as if the cache is working and i am getting cache hits. When i shut down the back end i can still get to the cached pages. On the other hand when i try a new browser i keep getting 503 pages. Any ideas? backend default { .host = "10.249.129.171"; .port = "80"; } sub vcl_recv { unset req.http.cookie; if (req.url ~ "^/admin/"){ set req.url = regsub(req.url, "^/admin/", "/oops/"); } if (req.url ~ "/affiliates/visa-signature/") { return(pass); } if (req.http.cookie) { set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", ""); if (req.http.cookie ~ "^ *$") { remove req.http.cookie; } } if (req.url ~ "^/r/\d+/") { return(pass); } # Don't cache if specific HTTP headers exist if (req.http.Expect) { return(pass); } if (req.http.Authorization) { return(pass); } return(lookup);} sub vcl_deliver { return (deliver);} sub vcl_fetch { set obj.ttl = 120m; unset obj.http.set-cookie; if (req.url ~ "^/$") { set obj.ttl = 1800s;} # Don't cache error pages if (obj.status == 503) { return(pass); } if (obj.http.Vary == "Cookie") { remove obj.http.Vary; } if (obj.http.Vary == "Cookie,Accept-Encoding") { set obj.http.Vary = "Accept-Encoding"; } # Anti-dogpiling feature - serve a stale request until a new one # can be fetched, for up to 30s if (!obj.cacheable) { return(pass); } return(deliver);} sub vcl_hit { remove obj.http.Set-Cookie;} -------------- next part -------------- An HTML attachment was scrubbed... URL: From rtshilston at gmail.com Wed Jun 2 18:40:46 2010 From: rtshilston at gmail.com (Rob S) Date: Wed, 02 Jun 2010 19:40:46 +0100 Subject: cache/503 In-Reply-To: References: Message-ID: <4C06A5AE.9050308@gmail.com> If you send the output of varnishlog from one working and one non-working browser, with the back end shutdown, then we might be able to spot the difference. Rob vitaly burshteyn wrote: > Hi folks, > > Just curious if you can point me in the right direction. > > I have 2.0.6 varnish running. So it seems as if the cache is working > and i am getting cache hits. > > When i shut down the back end i can still get to the cached pages. > > On the other hand when i try a new browser i keep getting 503 pages. > > Any ideas? > > > > backend default { > .host = "10.249.129.171"; > .port = "80"; } > > sub vcl_recv { > unset req.http.cookie; > > if (req.url ~ "^/admin/"){ > set req.url = regsub(req.url, "^/admin/", "/oops/"); } > > if (req.url ~ "/affiliates/visa-signature/") { > return(pass); } > > if (req.http.cookie) { > set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; > )?", ""); > if (req.http.cookie ~ "^ *$") { > remove req.http.cookie; } } > > if (req.url ~ "^/r/\d+/") { > return(pass); } > > # Don't cache if specific HTTP headers exist > if (req.http.Expect) { > return(pass); } > > if (req.http.Authorization) { > return(pass); } > return(lookup);} > > sub vcl_deliver { > return (deliver);} > > sub vcl_fetch { > set obj.ttl = 120m; > > unset obj.http.set-cookie; > if (req.url ~ "^/$") { > set obj.ttl = 1800s;} > > # Don't cache error pages > if (obj.status == 503) { > return(pass); } > > if (obj.http.Vary == "Cookie") { > remove obj.http.Vary; > } > if (obj.http.Vary == "Cookie,Accept-Encoding") { > set obj.http.Vary = "Accept-Encoding"; } > > # Anti-dogpiling feature - serve a stale request until a new one > # can be fetched, for up to 30s > > if (!obj.cacheable) { > return(pass); } > > return(deliver);} > > sub vcl_hit { > remove obj.http.Set-Cookie;} > > ------------------------------------------------------------------------ > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From perbu at varnish-software.com Wed Jun 2 19:03:11 2010 From: perbu at varnish-software.com (Per Buer) Date: Wed, 2 Jun 2010 21:03:11 +0200 Subject: cache/503 In-Reply-To: References: Message-ID: On Wed, Jun 2, 2010 at 8:32 PM, vitaly burshteyn wrote: > Hi folks, > Just curious if you can point me in the right direction. > I have 2.0.6 varnish running. ?So it seems as if the cache is working and i > am getting cache hits. > When i shut down the back end i can still get to the cached pages. > On the other hand when i try a new browser i keep getting 503 pages. > Any ideas? I guess your backend server sends a "Vary: User-Agent" along with its content. I tried describing the problem here: http://varnish-cache.org/static/perbu/dirhtml/tutorial/increasing_your_hitrate/#vary -- Per Buer, Varnish Software Phone: +47 21 54 41 21 / Mobile: +47 958 39 117 / skype: per.buer From vburshteyn at gmail.com Wed Jun 2 19:13:13 2010 From: vburshteyn at gmail.com (vitaly burshteyn) Date: Wed, 2 Jun 2010 15:13:13 -0400 Subject: cache/503 In-Reply-To: <4C06A5AE.9050308@gmail.com> References: <4C06A5AE.9050308@gmail.com> Message-ID: So on this one the site comes up 13 SessionOpen c 4.58.0.2 23450 *:80 13 ReqStart c 4.58.0.2 23450 831017267 13 RxRequest c GET 13 RxURL c / 13 RxProtocol c HTTP/1.1 13 RxHeader c Host: varnish2.broadway.com 13 RxHeader c User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/532.5 (KHTML, like Gecko) Chrome/4.1.249.1064 Safari/532.5 13 RxHeader c Referer: http://varnish2.broadway.com/ 13 RxHeader c Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5 13 RxHeader c Accept-Encoding: gzip,deflate,sdch 13 RxHeader c Accept-Language: en-US,en;q=0.8 13 RxHeader c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3 13 RxHeader c Cookie: OAX=BDoAAktdupEAAym8; __qca=P0-1578676227-1264519209511; __utmz=105990736.1264519210.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utma=105990736.1647079563.1264519210.1269538667.1270578927.6; __ngt_cli=9AD2EBDB8489C157C3867ADB; __utmz=7868 13 RxHeader c Cache-Control: max-stale=0 13 RxHeader c Connection: Keep-Alive 13 RxHeader c X-BlueCoat-Via: C92BD551DF341D76 13 VCL_call c recv lookup 13 VCL_call c hash hash 13 Hit c 831017141 13 VCL_call c hit deliver 13 Length c 9057 13 VCL_call c deliver deliver 13 TxProtocol c HTTP/1.1 13 TxStatus c 200 13 TxResponse c OK 13 TxHeader c Server: Apache/2.2.9 (Fedora) 13 TxHeader c Vary: User-Agent,Cookie,Accept-Encoding 13 TxHeader c Content-Encoding: gzip 13 TxHeader c Content-Type: text/html; charset=utf-8 13 TxHeader c Content-Length: 9057 13 TxHeader c Date: Wed, 02 Jun 2010 19:11:45 GMT 13 TxHeader c X-Varnish: 831017267 831017141 13 TxHeader c Age: 72 13 TxHeader c Via: 1.1 varnish 13 TxHeader c Connection: keep-alive 13 ReqEnd c 831017267 1275505905.109919071 1275505905.110054970 0.000252008 0.000067949 0.000067949 and this one is a 503 13 SessionOpen c 4.58.0.2 23716 *:80 13 ReqStart c 4.58.0.2 23716 831017268 13 RxRequest c GET 13 RxURL c / 13 RxProtocol c HTTP/1.1 13 RxHeader c Accept: */* 13 RxHeader c Accept-Language: en-us 13 RxHeader c UA-CPU: x86 13 RxHeader c Accept-Encoding: gzip, deflate 13 RxHeader c User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; WOW64; SV1; .NET CLR 2.0.50727) 13 RxHeader c Host: varnish2.broadway.com 13 RxHeader c Cookie: s_cc=true; s_sq=%5B%5BB%5D%5D 13 RxHeader c Cache-Control: max-stale=0 13 RxHeader c Connection: Keep-Alive 13 RxHeader c X-BlueCoat-Via: C92BD551DF341D76 13 VCL_call c recv lookup 13 VCL_call c hash hash 13 VCL_call c miss fetch 13 FetchError c no backend connection 13 VCL_call c error deliver 13 Length c 487 13 VCL_call c deliver deliver 13 TxProtocol c HTTP/1.1 13 TxStatus c 503 13 TxResponse c Service Unavailable 13 TxHeader c Server: Varnish 13 TxHeader c Retry-After: 0 13 TxHeader c Content-Type: text/html; charset=utf-8 13 TxHeader c Content-Length: 487 13 TxHeader c Date: Wed, 02 Jun 2010 19:12:16 GMT 13 TxHeader c X-Varnish: 831017268 13 TxHeader c Age: 0 13 TxHeader c Via: 1.1 varnish 13 TxHeader c Connection: close 13 ReqEnd c 831017268 1275505936.200603962 1275505936.203883886 0.000077009 0.003217936 0.000061989 On Wed, Jun 2, 2010 at 2:40 PM, Rob S wrote: > If you send the output of varnishlog from one working and one non-working > browser, with the back end shutdown, then we might be able to spot the > difference. > > Rob > > vitaly burshteyn wrote: > >> Hi folks, >> >> Just curious if you can point me in the right direction. >> >> I have 2.0.6 varnish running. So it seems as if the cache is working and >> i am getting cache hits. >> >> When i shut down the back end i can still get to the cached pages. >> >> On the other hand when i try a new browser i keep getting 503 pages. >> >> Any ideas? >> >> >> >> backend default { >> .host = "10.249.129.171"; >> .port = "80"; } >> >> sub vcl_recv { >> unset req.http.cookie; >> >> if (req.url ~ "^/admin/"){ >> set req.url = regsub(req.url, "^/admin/", "/oops/"); } >> >> if (req.url ~ "/affiliates/visa-signature/") { >> return(pass); } >> >> if (req.http.cookie) { >> set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", >> ""); >> if (req.http.cookie ~ "^ *$") { >> remove req.http.cookie; } } >> >> if (req.url ~ "^/r/\d+/") { >> return(pass); } >> >> # Don't cache if specific HTTP headers exist >> if (req.http.Expect) { >> return(pass); } >> >> if (req.http.Authorization) { >> return(pass); } >> return(lookup);} >> >> sub vcl_deliver { >> return (deliver);} >> >> sub vcl_fetch { >> set obj.ttl = 120m; >> >> unset obj.http.set-cookie; >> if (req.url ~ "^/$") { >> set obj.ttl = 1800s;} >> >> # Don't cache error pages >> if (obj.status == 503) { >> return(pass); } >> >> if (obj.http.Vary == "Cookie") { >> remove obj.http.Vary; >> } >> if (obj.http.Vary == "Cookie,Accept-Encoding") { >> set obj.http.Vary = "Accept-Encoding"; } >> >> # Anti-dogpiling feature - serve a stale request until a new one >> # can be fetched, for up to 30s >> >> if (!obj.cacheable) { >> return(pass); } >> >> return(deliver);} >> >> sub vcl_hit { >> remove obj.http.Set-Cookie;} >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From l at lrowe.co.uk Wed Jun 2 19:26:14 2010 From: l at lrowe.co.uk (Laurence Rowe) Date: Wed, 2 Jun 2010 20:26:14 +0100 Subject: serving from cache In-Reply-To: References: Message-ID: If you know cookies do not affect the backend, you can just customise ``vcl_recv`` to ``return (lookup)`` when you have a cookie. http://varnish-cache.org/browser/branches/2.1/varnish-cache/bin/varnishd/default.vcl#L63 If a page always sets the same cookie value, you can cache the response with the Set-Cookie header (or strip it as you see fit) by customising ``vcl_fetch`` to ``return (deliver)``. Obviously that's not suitable for session cookies. Laurence On 2 June 2010 14:24, vitaly burshteyn wrote: > > Hi folks, > So on my path to getting the website to server from cache instead of the > back end i realized that i had a cookie issue (sorry i am new to Varnish) > Cookie: affi=old; __ngt_cli=C8D6A206C800FDA28F4A7F07; > __utma=32297913.1583573073.1275401467.1275478692.1275481795.6; > So i changed my vcl_recv to strip out all the utma and other google > analytics cookies. > Yet for some reason no matter what i try i cant get it to strip??affi=old; > So i just want to confirm that untill this is empty: > Set-Cookie: affi=old; Path=/^@^[^@^V^@^@^@ > I wont be able to server from cache? > > > Thanks > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From l at lrowe.co.uk Wed Jun 2 19:31:14 2010 From: l at lrowe.co.uk (Laurence Rowe) Date: Wed, 2 Jun 2010 20:31:14 +0100 Subject: cache/503 In-Reply-To: References: Message-ID: Different browsers have different Accept-Encoding headers (sometimes just the difference between a comma followed by a space as the separator rather than just a comma), so it's best to normalise this header in ``vcl_recv``. I use something similar to the example at http://varnish-cache.org/wiki/FAQ/Compression to normalise the header to either ``gzip`` or remove it completely. Laurence On 2 June 2010 19:32, vitaly burshteyn wrote: > Hi folks, > Just curious if you can point me in the right direction. > I have 2.0.6 varnish running. ?So it seems as if the cache is working and i > am getting cache hits. > When i shut down the back end i can still get to the cached pages. > On the other hand when i try a new browser i keep getting 503 pages. > Any ideas? > > > backend default { > ??.host = "10.249.129.171"; > ?? ? ? ?.port = "80"; } > ?sub vcl_recv { > unset req.http.cookie; > ?? ?if (req.url ~ "^/admin/"){ > ?? ?set req.url = regsub(req.url, "^/admin/", "/oops/"); ? } > ??if (req.url ~ "/affiliates/visa-signature/") { > ?? ? ? ?return(pass); ? ?} > if (req.http.cookie) { > ?? ?set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", > ""); > ?? ?if (req.http.cookie ~ "^ *$") { > ?? ? ? ?remove req.http.cookie; ?} ?} > ?? ?if (req.url ~ "^/r/\d+/") { > ?? ? ? ?return(pass); ? ?} > ?? ?# Don't cache if specific HTTP headers exist > ?? ?if (req.http.Expect) { > ?? ? ? ?return(pass); ? ?} > ?? ?if (req.http.Authorization) { > ?? ? ? ?return(pass); ? } > return(lookup);} > sub vcl_deliver { > ?? ?return (deliver);} > sub vcl_fetch { > ?? ?set obj.ttl = 120m; > unset obj.http.set-cookie; > ?if (req.url ~ ?"^/$") { > ?set obj.ttl = 1800s;} > ?? ?# Don't cache error pages > ?? ?if (obj.status == 503) { > ?? ? ? ?return(pass); ? ?} > ?? ?if (obj.http.Vary == "Cookie") { > ?? ? ? ?remove obj.http.Vary; > ?? ?} > ?if (obj.http.Vary == "Cookie,Accept-Encoding") { > ?? ? ? ?set obj.http.Vary = "Accept-Encoding"; ? ?} > ?? ?# Anti-dogpiling feature - serve a stale request until a new one > ?? ?# can be fetched, for up to 30s > ?? ?if (!obj.cacheable) { > ?? ? ? return(pass); ? ? ? } > ?? ?return(deliver);} > sub vcl_hit { > ??remove obj.http.Set-Cookie;} > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From vburshteyn at gmail.com Wed Jun 2 22:05:42 2010 From: vburshteyn at gmail.com (vitaly burshteyn) Date: Wed, 2 Jun 2010 18:05:42 -0400 Subject: cache/503 In-Reply-To: References: Message-ID: On Jun 2, 2010 3:31 PM, "Laurence Rowe" wrote: Different browsers have different Accept-Encoding headers (sometimes just the difference between a comma followed by a space as the separator rather than just a comma), so it's best to normalise this header in ``vcl_recv``. I use something similar to the example at http://varnish-cache.org/wiki/FAQ/Compression to normalise the header to either ``gzip`` or remove it completely. Laurence On 2 June 2010 19:32, vitaly burshteyn wrote: > Hi folks, > Just curious if ... > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish... -------------- next part -------------- An HTML attachment was scrubbed... URL: From vburshteyn at gmail.com Wed Jun 2 23:01:17 2010 From: vburshteyn at gmail.com (vitaly burshteyn) Date: Wed, 2 Jun 2010 19:01:17 -0400 Subject: cache/503 In-Reply-To: References: Message-ID: Thanks for responce but still no dice... Just curious if I am getting Vary: User-Agent, Cookie, Accept-Encoding do i need to account for all 3 of those items? Also, how do you recommend to normalize User-Agent? Thanks Sorry about these questions but i am brand new to varnish. On Wed, Jun 2, 2010 at 6:05 PM, vitaly burshteyn wrote: > On Jun 2, 2010 3:31 PM, "Laurence Rowe" wrote: > > Different browsers have different Accept-Encoding headers (sometimes > just the difference between a comma followed by a space as the > separator rather than just a comma), so it's best to normalise this > header in ``vcl_recv``. I use something similar to the example at > http://varnish-cache.org/wiki/FAQ/Compression to normalise the header > to either ``gzip`` or remove it completely. > > Laurence > > > On 2 June 2010 19:32, vitaly burshteyn wrote: > > Hi folks, > > Just curious if ... > > > _______________________________________________ > > varnish-misc mailing list > > varnish-misc at varnish... > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From l at lrowe.co.uk Wed Jun 2 23:32:14 2010 From: l at lrowe.co.uk (Laurence Rowe) Date: Thu, 3 Jun 2010 00:32:14 +0100 Subject: cache/503 In-Reply-To: References: Message-ID: You are almost guaranteed to get unique cached objects for each user with your Vary: Vary: User-Agent, Cookie, Accept-Encoding Varying by User-Agent is never a good idea, there are thousands of different user agent strings, as it includes both the browser version number and operating system version number (including any patch levels...). You will have to know how your backend application uses this information in order to sensibly normalise the value. Excepting the google analytics __utm. cookies which you strip, three others are left from your first trace: Cookie: OAX=BDoAAktdupEAAym8; __qca=P0-1578676227-1264519209511; __utmz=105990736.1264519210.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utma=105990736.1647079563.1264519210.1269538667.1270578927.6; __ngt_cli=9AD2EBDB8489C157C3867ADB; __utmz=7868 In the second trace you seem to have other cookies set as well. Cookie: s_cc=true; s_sq=%5B%5BB%5D%5D If you need to vary on any information stored in a cookie, write some VCL to extract that cookie, place it into another header, and then Vary on your custom header. For instance, on a site where a user can pick their language, I extract the value from the relevant cookie and overwrite the Accept-Language header with the it, and then vary on Accept-Language. Laurence On 3 June 2010 00:01, vitaly burshteyn wrote: > Thanks for responce but still no dice... > Just curious if I am getting?Vary: User-Agent, Cookie, Accept-Encoding do i > need to account for all 3 of those items? > Also, how do you recommend to normalize User-Agent? > Thanks > > > Sorry about these questions but i am brand new to varnish. > > > On Wed, Jun 2, 2010 at 6:05 PM, vitaly burshteyn > wrote: >> >> On Jun 2, 2010 3:31 PM, "Laurence Rowe" wrote: >> >> Different browsers have different Accept-Encoding headers (sometimes >> just the difference between a comma followed by a space as the >> separator rather than just a comma), so it's best to normalise this >> header in ``vcl_recv``. I use something similar to the example at >> http://varnish-cache.org/wiki/FAQ/Compression to normalise the header >> to either ``gzip`` or remove it completely. >> >> Laurence >> >> On 2 June 2010 19:32, vitaly burshteyn wrote: >> > Hi folks, >> > Just curious if ... >> >> > _______________________________________________ >> > varnish-misc mailing list >> > varnish-misc at varnish... > From angie.tawfik at gmail.com Thu Jun 3 11:45:05 2010 From: angie.tawfik at gmail.com (Angie T. Muhammad) Date: Thu, 3 Jun 2010 14:45:05 +0300 Subject: Parsing non-English URLs In-Reply-To: References: <877hmk61k6.fsf@qurzaw.linpro.no> <871vcpoofd.fsf@qurzaw.linpro.no> Message-ID: Hello group, I tried using the %-escaped syntax but it does not work for me. Here is the rule that varnish totally ignores as if it doesn't exist: =========================================================================================== # cache AR taxonomy differently else if (req.url == "/%25D9%2588%25D8%25B3%25D9%2588%25D9%2585%25D8%25A7%25D8%25AA") { set beresp.http.X-Cacheable = "Yes"; set beresp.ttl = 180m; return(deliver); } ========================================================================================== The link exists, and even copying it in this syntax to the browser retrieves the page, but the caching for 180 minutes does not work. Other rules with only English URLS are working like a charm. Please help. Thanks. On Wed, Jun 2, 2010 at 3:56 PM, Angie T. Muhammad wrote: > I tried the second syntax many times before. Varnish refuses to start. > But the good news it accepted the first syntax. I will be checking if > it applies the rule as required and update you. > > Thank you Tollef :) > > On 6/2/10, Tollef Fog Heen wrote: > > ]] "Angie T. Muhammad" > > > > | I get some lines like: > > | 14 TxHeader b Referer: > > | > > > http://myserver.example.com/news/%D8%A7%D9%84%D9%85%D9%84%D9%8A%D8%A7%D8%B1%D8%AF%D9%8A%D8%B1-%D8%A7%D9%84%D9%85%D8%B5%D8%B1%D9%8A-%C2%AB%D9%85%D8%AD%D9%85%D8%AF-%D8%A7%D9%84%D9%81%D8%A7%D9%8A%D8%AF%C2%BB-%D9%8A%D9%82%D8%B1%D8%B1-%D8%A8%D9%8A%D > > | > > | The point is that I need to make varnish accept such percentages or any > > | other syntax in VCL file. So that some pages are cached longer based on > > | their Arabic address (Like the one in the referer above). > > > > We use %-escaped strings in vcl, so referring to that string above would > > be to match it to > > > > ? > http://myserver.example.com/news/%25D8%25A7%25D9%2584%25D9%2585%25D9%2584%25D9%258A%25D8%25A7%25D8%25B1%25D8%25AF%25D9%258A%25D8%25B1-%25D8%25A7%25D9%2584%25D9%2585%25D8%25B5%25D8%25B1%25D9%258A-%25C2%25AB%25D9%2585%25D8%25AD%25D9%2585%25D8%25AF-%25D8%25A7%25D9%2584%25D9%2581%25D8%25A7%25D9%258A%25D8%25AF%25C2%25BB-%25D9%258A%25D9%2582%25D8%25B1%25D8%25B1-%25D8%25A8%25D9%258A%25D > ? > > > > or more easily > > > > {" > http://myserver.example.com/news/%D8%A7%D9%84%D9%85%D9%84%D9%8A%D8%A7%D8%B1%D8%AF%D9%8A%D8%B1-%D8%A7%D9%84%D9%85%D8%B5%D8%B1%D9%8A-%C2%AB%D9%85%D8%AD%D9%85%D8%AF-%D8%A7%D9%84%D9%81%D8%A7%D9%8A%D8%AF%C2%BB-%D9%8A%D9%82%D8%B1%D8%B1-%D8%A8%D9%8A%D > "} > > > > -- > > Tollef Fog Heen > > Varnish Software > > t: +47 21 54 41 73 > > > > > -- > All the best, > Angie T. Muhammad > Linux Systems Engineer > Al Masry Al Youm > http://www.almasryalyoum.com/en > -- All the best, Angie T. Muhammad Linux Systems Engineer Al Masry Al Youm http://www.almasryalyoum.com/en -------------- next part -------------- An HTML attachment was scrubbed... URL: From dolzenko at gmail.com Fri Jun 4 12:34:40 2010 From: dolzenko at gmail.com (Evgeniy Dolzhenko) Date: Fri, 4 Jun 2010 16:34:40 +0400 Subject: Configuration to avoid dog-pile when cache is empty Message-ID: I've been having that problem for awhile and haven't yet saw the clean and simple solution to it. Also I'm really new to Varnish so please excuse my ignorance if there is any. The scenario is as follows: 1. There is the page on Application Server which takes forever to load (/takes_forever_to_load.html) and which is not user-specific and can be always cached for a known period of time. 2. Varnish cache is empty 3. Multiple users request /takes_forever_to_load.html page. If my understanding is correct with the default configuration all requests will hit Application Server. Now what I would like ideally to achieve with Varnish is that only first request hits Application Server and every other request is "joined" to that request, i.e. will be served as soon as Application Server responds to the first request and with the content of the response. I.e. Varnish will have to hold the connections open and then use response cached by first request Is that possible? Or is it just a too crazy thing to do? TIA, Evgeniy From perbu at varnish-software.com Sat Jun 5 07:18:08 2010 From: perbu at varnish-software.com (Per Buer) Date: Sat, 5 Jun 2010 09:18:08 +0200 Subject: Configuration to avoid dog-pile when cache is empty In-Reply-To: References: Message-ID: On Fri, Jun 4, 2010 at 2:34 PM, Evgeniy Dolzhenko wrote: > (..) > 3. Multiple users request /takes_forever_to_load.html page. > If my understanding is correct with the default configuration all requests > will hit Application Server. No. Varnish will only send one request to the App Server unless you have indicated that it shall pass. > Now what I would like ideally to achieve with Varnish is that only > first request hits Application Server and every other request is > "joined" to that request, i.e. will be served as soon as > Application Server responds to the first request and with > the content of the response. I.e. Varnish will have to hold > the connections open and then use response cached by first request > > Is that possible? Or is it just a too crazy thing to do? No. It's quite sensible. It's also the default. :-) -- Per Buer, Varnish Software Phone: +47 21 54 41 21 / Mobile: +47 958 39 117 / skype: per.buer From cosimo at streppone.it Sat Jun 5 11:52:37 2010 From: cosimo at streppone.it (Cosimo Streppone) Date: Sat, 05 Jun 2010 13:52:37 +0200 Subject: Configuration to avoid dog-pile when cache is empty In-Reply-To: References: Message-ID: On Sat, 05 Jun 2010 09:18:08 +0200, Per Buer wrote: > On Fri, Jun 4, 2010 at 2:34 PM, Evgeniy Dolzhenko > wrote: >> Now what I would like ideally to achieve with Varnish is that only >> first request hits Application Server and every other request is >> "joined" to that request, i.e. will be served as soon as >> Application Server responds to the first request and with >> the content of the response. I.e. Varnish will have to hold >> the connections open and then use response cached by first request >> >> Is that possible? Or is it just a too crazy thing to do? > > No. It's quite sensible. It's also the default. :-) That sounds surprising to me. Can you explain? Say that I restart varnish. Then 10 clients hit the "/takes_long_time_to_load.html" page at the same time. What I understand from your reply is that varnish will hold the 9 remaining connections until the first one sees the backend reply. At that point, the 9 remaining clients will get served the cached object. Is that really how it works? I'm specifically referring to the first time after restart, so grace doesn't apply here. -- Cosimo From schmidt at ze.tum.de Mon Jun 7 08:34:16 2010 From: schmidt at ze.tum.de (Gerhard Schmidt) Date: Mon, 07 Jun 2010 10:34:16 +0200 Subject: CLI communication error (hdr) on vcl.discard Message-ID: <4C0CAF08.8090702@ze.tum.de> Hi, I'm in the process of migrating auch website from squid to varnish as reverse proxy. I use a system to dynamicaly add and remove backends. This system creates new configs via vcl.inline. Works fine so far. but new configs are created some times in 10 sec intervals. So the config list gets quite crowded quite soon. When i try to delete old configs via vcl.discard I get an "CLI communication error (hdr)" here is an example cli session telnet localhost 6082 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. 200 154 ----------------------------- Varnish HTTP accelerator CLI. ----------------------------- Type 'help' for command list. Type 'quit' to close CLI session. vcl.list 200 228 available 0 boot available 0 dynconf_20100607101631 available 0 dynconf_20100607101856 available 0 dynconf_20100607103126 available 2 dynconf_20100607103142 active 1 dynconf_20100607103209 vcl.discard dynconf_20100607101631 400 29 CLI communication error (hdr) I'm using varnish 2.1.2 on FreeBSD Regards Estartu -- ------------------------------------------------- Gerhard Schmidt | E-Mail: schmidt at ze.tum.de TU-M?nchen | WWW & Online Services | Tel: 089/289-25270 | Fax: 089/289-25257 | PGP-Publickey auf Anfrage -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 544 bytes Desc: OpenPGP digital signature URL: From alex at acasa.ro Mon Jun 7 08:43:46 2010 From: alex at acasa.ro (Alex F) Date: Mon, 07 Jun 2010 11:43:46 +0300 Subject: VCL changes from 2.0 to 2.1 Message-ID: <20100607084347.B44712F76EE@smtp1.acasa.ro> Hello, I just upgraded from varnish 2.0.5 to 2.1.2 and realized that the old VCL is not working anymore. My VCL looks like this: backend default { .host = "0.0.0.0"; .port = "80"; } acl purge { "localhost"; "192.168.100.0"/24; } sub vcl_recv { if (req.request == "PURGE") { if(!client.ip ~ purge) { error 405 "Not Allowed"; } purge("req.url == " req.url); } if (req.request != "GET" && req.request != "HEAD" && req.request != "PUT" && req.request != "POST" && req.request != "TRACE" && req.request != "OPTIONS" && req.request != "DELETE") { return(pipe); } if (req.http.Expect) { return(pipe); } if (req.request != "GET" && req.request != "HEAD") { return(pipe); } if (req.http.Authorization) { return(pass); } if (req.http.Accept-Encoding) { if (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip"; } elsif (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate"; } else { remove req.http.Accept-Encoding; } } set req.grace = 30s; if (req.url ~ "^/(uploads|images|img|css|js)/" || req.url ~ "\.(txt|ico|png|jpeg|jpg|gif|tiff|js|css)$") { return(lookup); } if (req.http.Cookie) { return(pass); } return(lookup); } sub vcl_hit { if (!obj.cacheable) { return(pass); } return(deliver); } sub vcl_miss { return (fetch); } sub vcl_fetch { if (!obj.cacheable) { return(pass); } if (obj.http.Set-Cookie) { return(pass); } if (obj.http.Pragma ~ "no-cache" || obj.http.Cache-Control ~ "no-cache" || obj.http.Cache-Control ~ "private") { return(pass); } if (obj.status !=200) { return (pass); } set obj.grace=10080m; if (obj.ttl < 180s) { set obj.ttl = 1d; } } sub vcl_deliver { if (obj.hits > 0) { set resp.http.X-Cache = "HIT"; set resp.http.X-Cache-Hits = obj.hits; } else { set resp.http.X-Cache = "MISS"; } } And upon compiling this VCL I receive: Message from VCC-compiler: Variable 'obj.cacheable' not accessible in method 'vcl_fetch'. At: (input Line 92 Pos 14) if (!obj.cacheable) { -------------#############--- Running VCC-compiler failed, exit 1 VCL compilation failed Can you please point me in the right direction? Also, any advices for the current VCL are welcome. Thank you. From samcrawford at gmail.com Mon Jun 7 08:54:59 2010 From: samcrawford at gmail.com (Sam Crawford) Date: Mon, 7 Jun 2010 09:54:59 +0100 Subject: VCL changes from 2.0 to 2.1 In-Reply-To: <20100607084347.B44712F76EE@smtp1.acasa.ro> References: <20100607084347.B44712F76EE@smtp1.acasa.ro> Message-ID: See http://varnish-cache.org/wiki/changelog_2.0.6-2.1.0 obj.* should now be beresp.* Thanks, Sam On 7 June 2010 09:43, Alex F wrote: > Hello, > I just upgraded from varnish 2.0.5 to 2.1.2 and realized that the old VCL is > not working anymore. > My VCL looks like this: > > ? ?backend default { > ? ? ? ?.host = "0.0.0.0"; > ? ? ? ?.port = "80"; > } > ? ?acl purge { > ? ? ? ?"localhost"; > ? ? ? ?"192.168.100.0"/24; > } > ? ?sub vcl_recv { > ? ? ? ?if (req.request == "PURGE") { > ? ? ? ? ? ? ? ?if(!client.ip ~ purge) { > ? ? ? ? ? ? ? ? ? ? ? ?error 405 "Not Allowed"; > ? ? ? ? ? ? ? ?} > ? ? ? ? ? ? ? ?purge("req.url == " req.url); > } > ? ?if (req.request != "GET" && req.request != "HEAD" && req.request != "PUT" > && req.request != "POST" && req.request != "TRACE" && req.request != > "OPTIONS" && req.request != "DELETE") { > ? ? ? ? ? ? ? ?return(pipe); > } > ? ?if (req.http.Expect) { > ? ? ? ? ? ? ? ?return(pipe); > } > ? ?if (req.request != "GET" && req.request != "HEAD") { > ? ? ? ? ? ? ? ?return(pipe); > } > ? ?if (req.http.Authorization) { > ? ? ? ? ? ? ? ?return(pass); > } > ? ?if (req.http.Accept-Encoding) { > ? ? ? ? ? ? ? ?if (req.http.Accept-Encoding ~ "gzip") { > ? ? ? ? ? ? ? ?set req.http.Accept-Encoding = "gzip"; > ? ? ? ? ? ? ? ?} elsif (req.http.Accept-Encoding ~ "deflate") { > ? ? ? ? ? ? ? ?set req.http.Accept-Encoding = "deflate"; > ? ? ? ? ? ? ? ?} else { > ? ? ? ? ? ? ? ?remove req.http.Accept-Encoding; > } > } > ? ?set req.grace = 30s; > ? ?if (req.url ~ "^/(uploads|images|img|css|js)/" || req.url ~ > "\.(txt|ico|png|jpeg|jpg|gif|tiff|js|css)$") { > ? ? ? ?return(lookup); > } > ? ? ? ?if (req.http.Cookie) { > ? ? ? ?return(pass); > } > ? ? ? ?return(lookup); > } > ? ?sub vcl_hit { > ? ? ? ?if (!obj.cacheable) { > ? ? ? ? ? ? ? ?return(pass); > } > ? ? ? ?return(deliver); > } > ? ?sub vcl_miss { > ? ? ? ? ? ? ? ?return (fetch); > } > ? ?sub vcl_fetch { > ? ?if (!obj.cacheable) { > ? ? ? ? ? ? ? ?return(pass); > } > ? ?if (obj.http.Set-Cookie) { > ? ? ? ? ? ? ? ?return(pass); > } > ? ?if (obj.http.Pragma ~ "no-cache" || obj.http.Cache-Control ~ "no-cache" > || obj.http.Cache-Control ~ "private") { > ? ? ? ? ? ? ? ?return(pass); > } > ? ?if (obj.status !=200) { > ? ? ? ?return (pass); > } > ? ?set obj.grace=10080m; > ? ?if (obj.ttl < 180s) { > ? ?set obj.ttl = 1d; > } > } > ? ?sub vcl_deliver { > ? ?if (obj.hits > 0) { > ? ? ? ? ? ? ? ?set resp.http.X-Cache = "HIT"; > ? ? ? ? ? ? ? ?set resp.http.X-Cache-Hits = obj.hits; > ? ? ? ?} else { > ? ? ? ? ? ? ? ?set resp.http.X-Cache = "MISS"; > ? ? ? ?} > } > > And upon compiling this VCL I receive: > ? ?Message from VCC-compiler: > ? ?Variable 'obj.cacheable' not accessible in method 'vcl_fetch'. > ? ?At: (input Line 92 Pos 14) > ? ? ? ?if (!obj.cacheable) { > ? ?-------------#############--- > ? ?Running VCC-compiler failed, exit 1 > ? ?VCL compilation failed > Can you please point me in the right direction? Also, any advices for the > current VCL are welcome. > Thank you. > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From kofreestyler at gmail.com Mon Jun 7 08:55:01 2010 From: kofreestyler at gmail.com (freestyler) Date: Mon, 7 Jun 2010 16:55:01 +0800 Subject: VCL changes from 2.0 to 2.1 In-Reply-To: <20100607084347.B44712F76EE@smtp1.acasa.ro> References: <20100607084347.B44712F76EE@smtp1.acasa.ro> Message-ID: obj.* is now called beresp.* in vcl_fetch, and obj.* is now read-only. see http://varnish-cache.org/wiki/changelog_2.0.6-2.1.0 On Mon, Jun 7, 2010 at 4:43 PM, Alex F wrote: > Hello, > I just upgraded from varnish 2.0.5 to 2.1.2 and realized that the old VCL is > not working anymore. > My VCL looks like this: > > ? ?backend default { > ? ? ? ?.host = "0.0.0.0"; > ? ? ? ?.port = "80"; > } > ? ?acl purge { > ? ? ? ?"localhost"; > ? ? ? ?"192.168.100.0"/24; > } > ? ?sub vcl_recv { > ? ? ? ?if (req.request == "PURGE") { > ? ? ? ? ? ? ? ?if(!client.ip ~ purge) { > ? ? ? ? ? ? ? ? ? ? ? ?error 405 "Not Allowed"; > ? ? ? ? ? ? ? ?} > ? ? ? ? ? ? ? ?purge("req.url == " req.url); > } > ? ?if (req.request != "GET" && req.request != "HEAD" && req.request != "PUT" > && req.request != "POST" && req.request != "TRACE" && req.request != > "OPTIONS" && req.request != "DELETE") { > ? ? ? ? ? ? ? ?return(pipe); > } > ? ?if (req.http.Expect) { > ? ? ? ? ? ? ? ?return(pipe); > } > ? ?if (req.request != "GET" && req.request != "HEAD") { > ? ? ? ? ? ? ? ?return(pipe); > } > ? ?if (req.http.Authorization) { > ? ? ? ? ? ? ? ?return(pass); > } > ? ?if (req.http.Accept-Encoding) { > ? ? ? ? ? ? ? ?if (req.http.Accept-Encoding ~ "gzip") { > ? ? ? ? ? ? ? ?set req.http.Accept-Encoding = "gzip"; > ? ? ? ? ? ? ? ?} elsif (req.http.Accept-Encoding ~ "deflate") { > ? ? ? ? ? ? ? ?set req.http.Accept-Encoding = "deflate"; > ? ? ? ? ? ? ? ?} else { > ? ? ? ? ? ? ? ?remove req.http.Accept-Encoding; > } > } > ? ?set req.grace = 30s; > ? ?if (req.url ~ "^/(uploads|images|img|css|js)/" || req.url ~ > "\.(txt|ico|png|jpeg|jpg|gif|tiff|js|css)$") { > ? ? ? ?return(lookup); > } > ? ? ? ?if (req.http.Cookie) { > ? ? ? ?return(pass); > } > ? ? ? ?return(lookup); > } > ? ?sub vcl_hit { > ? ? ? ?if (!obj.cacheable) { > ? ? ? ? ? ? ? ?return(pass); > } > ? ? ? ?return(deliver); > } > ? ?sub vcl_miss { > ? ? ? ? ? ? ? ?return (fetch); > } > ? ?sub vcl_fetch { > ? ?if (!obj.cacheable) { > ? ? ? ? ? ? ? ?return(pass); > } > ? ?if (obj.http.Set-Cookie) { > ? ? ? ? ? ? ? ?return(pass); > } > ? ?if (obj.http.Pragma ~ "no-cache" || obj.http.Cache-Control ~ "no-cache" > || obj.http.Cache-Control ~ "private") { > ? ? ? ? ? ? ? ?return(pass); > } > ? ?if (obj.status !=200) { > ? ? ? ?return (pass); > } > ? ?set obj.grace=10080m; > ? ?if (obj.ttl < 180s) { > ? ?set obj.ttl = 1d; > } > } > ? ?sub vcl_deliver { > ? ?if (obj.hits > 0) { > ? ? ? ? ? ? ? ?set resp.http.X-Cache = "HIT"; > ? ? ? ? ? ? ? ?set resp.http.X-Cache-Hits = obj.hits; > ? ? ? ?} else { > ? ? ? ? ? ? ? ?set resp.http.X-Cache = "MISS"; > ? ? ? ?} > } > > And upon compiling this VCL I receive: > ? ?Message from VCC-compiler: > ? ?Variable 'obj.cacheable' not accessible in method 'vcl_fetch'. > ? ?At: (input Line 92 Pos 14) > ? ? ? ?if (!obj.cacheable) { > ? ?-------------#############--- > ? ?Running VCC-compiler failed, exit 1 > ? ?VCL compilation failed > Can you please point me in the right direction? Also, any advices for the > current VCL are welcome. > Thank you. > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From alex at acasa.ro Mon Jun 7 09:34:26 2010 From: alex at acasa.ro (Alex F) Date: Mon, 07 Jun 2010 12:34:26 +0300 Subject: VCL changes from 2.0 to 2.1 In-Reply-To: References: <20100607084347.B44712F76EE@smtp1.acasa.ro> Message-ID: <20100607093426.CDE912F77BD@smtp1.acasa.ro> Success! Thank you. On 7.6.2010 11:54, Sam Crawford wrote: > See http://varnish-cache.org/wiki/changelog_2.0.6-2.1.0 > > obj.* should now be beresp.* > > Thanks, > > Sam > On 7.6.2010 11:55, freestyler wrote: > obj.* is now called beresp.* in vcl_fetch, and obj.* is now read-only. > > see http://varnish-cache.org/wiki/changelog_2.0.6-2.1.0 > From dolzenko at gmail.com Mon Jun 7 10:56:34 2010 From: dolzenko at gmail.com (Evgeniy Dolzhenko) Date: Mon, 7 Jun 2010 14:56:34 +0400 Subject: Configuration to avoid dog-pile when cache is empty In-Reply-To: References: Message-ID: Ok, actually I found this behavior surprising since I was thinking of every request being processed in complete isolation, but here is the FAQ entry which indirectly documents the feature I believe http://varnish-cache.org/wiki/FAQ#WhyamIgettingacachehitbutarequestisstillgoingtomybackend On Sat, Jun 5, 2010 at 3:52 PM, Cosimo Streppone wrote: > On Sat, 05 Jun 2010 09:18:08 +0200, Per Buer > wrote: > >> On Fri, Jun 4, 2010 at 2:34 PM, Evgeniy Dolzhenko >> wrote: > >>> Now what I would like ideally to achieve with Varnish is that only >>> first request hits Application Server and every other request is >>> "joined" to that request, i.e. will be served as soon as >>> Application Server responds to the first request and with >>> the content of the response. I.e. Varnish will have to hold >>> the connections open and then use response cached by first request >>> >>> Is that possible? Or is it just a too crazy thing to do? >> >> No. It's quite sensible. It's also the default. :-) > > That sounds surprising to me. Can you explain? > Say that I restart varnish. > Then 10 clients hit the "/takes_long_time_to_load.html" page at > the same time. > > What I understand from your reply is that varnish will hold the > 9 remaining connections until the first one sees the backend > reply. At that point, the 9 remaining clients will get served > the cached object. > > Is that really how it works? > I'm specifically referring to the first time after restart, > so grace doesn't apply here. > > -- > Cosimo > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From perbu at varnish-software.com Mon Jun 7 11:15:32 2010 From: perbu at varnish-software.com (Per Buer) Date: Mon, 7 Jun 2010 13:15:32 +0200 Subject: Configuration to avoid dog-pile when cache is empty In-Reply-To: References: Message-ID: On Sat, Jun 5, 2010 at 1:52 PM, Cosimo Streppone wrote: > That sounds surprising to me. Can you explain? > Say that I restart varnish. > Then 10 clients hit the "/takes_long_time_to_load.html" page at > the same time. > > What I understand from your reply is that varnish will hold the > 9 remaining connections until the first one sees the backend > reply. At that point, the 9 remaining clients will get served > the cached object. Yes. There is a whole lot of work spent on the code that releases the 9 remaining clients. At some point (1.x) we had massive problems when thousands of clients where released at the same time. > Is that really how it works? If it where to work any other way it would be a bit silly, wouldn't you think? Load spikes could really kill the backend. > I'm specifically referring to the first time after restart, > so grace doesn't apply here. Yes. Grace is something else. Grace is a deliberate, this is not. You can't turn this behavior off. Evgeniy: I saw your reference to hit-for-pass. Hit-for-pass objects are created especially for the situation where we find that it doesn't make sense to queue up objects. A "hit for pass" object is created to cache this finding so incomming reqeusts are passed directly when there is a hit-for-pass object. -- Per Buer, Varnish Software Phone: +47 21 54 41 21 / Mobile: +47 958 39 117 / skype: per.buer From phk at phk.freebsd.dk Mon Jun 7 11:54:27 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 07 Jun 2010 11:54:27 +0000 Subject: CLI communication error (hdr) on vcl.discard In-Reply-To: Your message of "Mon, 07 Jun 2010 10:34:16 +0200." <4C0CAF08.8090702@ze.tum.de> Message-ID: <7545.1275911667@critter.freebsd.dk> Please open a ticket. Poul-Henning In message <4C0CAF08.8090702 at ze.tum.de>, Gerhard Schmidt writes: > > telnet localhost 6082 >Trying 127.0.0.1... >Connected to localhost. >Escape character is '^]'. >200 154 >----------------------------- >Varnish HTTP accelerator CLI. >----------------------------- >Type 'help' for command list. >Type 'quit' to close CLI session. > >vcl.list >200 228 >available 0 boot >available 0 dynconf_20100607101631 >available 0 dynconf_20100607101856 >available 0 dynconf_20100607103126 >available 2 dynconf_20100607103142 >active 1 dynconf_20100607103209 > >vcl.discard dynconf_20100607101631 >400 29 >CLI communication error (hdr) > >I'm using varnish 2.1.2 on FreeBSD > >Regards > Estartu > >--=20 >------------------------------------------------- >Gerhard Schmidt | E-Mail: schmidt at ze.tum.de >TU-M=FCnchen | >WWW & Online Services | >Tel: 089/289-25270 | >Fax: 089/289-25257 | PGP-Publickey auf Anfrage > > > >--------------enigF7DCCA83A33B1AE457B25626 >Content-Type: application/pgp-signature; name="signature.asc" >Content-Description: OpenPGP digital signature >Content-Disposition: attachment; filename="signature.asc" > >-----BEGIN PGP SIGNATURE----- >Version: GnuPG v1.4.10 (FreeBSD) >Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > >iQEVAwUBTAyvDdl1K6RAAKkVAQrOcQgAm5Z86CJlAWK56mJUBggC5ogqxoJ8uRD/ >8anRj9SwvYVV0FMZV19X2/XOuLhd6dQRzUa2p4BRMNAHQciBzeSEtDyRvQGmE/7V >0obz0ezU0O2YG1Ut1XvabKzBPqbgoZhAHbb62jSPE6t+mY60Y5DeHU0po4BDV26F >wXhLaful3sZbGHr5heMTx1WXEFPPJm79kmK/147tP2jqzIFZ2o4EliqbZOaoWVmk >QBaWmWUrRKqZMKYvH3KH7KwhaZWj1tAaLYghF9M1zPGBqnoxyubw9+JiMQmQWQAV >qTWwLj9esJB9xAD4F5SpwNagr/qC6cP0F7/FJe7gHBMQRru9w26seQ== >=Sg7d >-----END PGP SIGNATURE----- > >--------------enigF7DCCA83A33B1AE457B25626-- > > >--===============3607559826138576911== >Content-Type: text/plain; charset="us-ascii" >MIME-Version: 1.0 >Content-Transfer-Encoding: 7bit >Content-Disposition: inline > >_______________________________________________ >varnish-misc mailing list >varnish-misc at varnish-cache.org >http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >--===============3607559826138576911==-- > -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From l.monsees at atlantismedia.de Mon Jun 7 14:48:25 2010 From: l.monsees at atlantismedia.de (Lars Monsees) Date: Mon, 7 Jun 2010 16:48:25 +0200 Subject: Varnish returns default host instead of virtual host Message-ID: Hi, I am very new to Varnish and have only configured my backend as I guess Varnish will use default values that should work for the beginning. The problem is that Varnish returns the content of the server?s default domain (the standard "it work?s" text") instead of the content of the wanted domain. backend default { .host = "127.0.0.1"; .port = "80"; } I have searched a lot for a solution but only found these tickets that don?t help me: http://varnish-cache.org/ticket/399 http://varnish-cache.org/ticket/248 So, it seems like Varnish doesn?t pass the hostname to the backend. How can I fix this or what am I doing wrong here? Varnish 2.1.2 (compiled from source) on Debian Lenny with Apache 2.2.8 Lars From marcussmith at britarch.ac.uk Mon Jun 7 15:20:52 2010 From: marcussmith at britarch.ac.uk (Marcus Smith) Date: Mon, 07 Jun 2010 16:20:52 +0100 Subject: Varnish returns default host instead of virtual host In-Reply-To: References: Message-ID: <4C0D0E54.4070202@britarch.ac.uk> Lars Monsees wrote: > Hi, > > I am very new to Varnish and have only configured my backend as I > guess Varnish will use default values that should work for the > beginning. The problem is that Varnish returns the content of the > server?s default domain (the standard "it work?s" text") instead of > the content of the wanted domain. > backend default { > .host = "127.0.0.1"; > .port = "80"; > } It looks like the backend is on the same machine that varnish is running on. In that case, it needs to be listening on a port other than 80 as you have above, as varnish will be bound to port 80 by default. best wishes, Marcus From stewsnooze at gmail.com Mon Jun 7 15:13:13 2010 From: stewsnooze at gmail.com (Stewart Robinson) Date: Mon, 7 Jun 2010 16:13:13 +0100 Subject: Varnish returns default host instead of virtual host In-Reply-To: References: Message-ID: Hi, This is typically an Apache config set up. I had this and it was because my Listen directive in VirtualHost was listening on the real IP address, rather than * or 127.0.0.1 Stew On 7 June 2010 15:48, Lars Monsees wrote: > Hi, > > I am very new to Varnish and have only configured my backend as I guess Varnish will use default values that should work for the beginning. The problem is that Varnish returns the content of the server?s default domain (the standard "it work?s" text") instead of the content of the wanted domain. > > backend default { > ? ?.host = "127.0.0.1"; > ? ?.port = "80"; > } > > > I have searched a lot for a solution but only found these tickets that don?t help me: > ? ?http://varnish-cache.org/ticket/399 > ? ?http://varnish-cache.org/ticket/248 > > > So, it seems like Varnish doesn?t pass the hostname to the backend. How can I fix this or what am I doing wrong here? > > > Varnish 2.1.2 (compiled from source) on Debian Lenny with Apache 2.2.8 > > > Lars > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From scaunter at topscms.com Mon Jun 7 15:35:52 2010 From: scaunter at topscms.com (Caunter, Stefan) Date: Mon, 7 Jun 2010 11:35:52 -0400 Subject: Varnish returns default host instead of virtual host In-Reply-To: <4C0D0E54.4070202@britarch.ac.uk> References: <4C0D0E54.4070202@britarch.ac.uk> Message-ID: <064FF286FD17EC418BFB74629578BED119ACF03C@tmg-mail4.torstar.net> -----Original Message----- From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Marcus Smith Sent: June-07-10 11:21 AM To: varnish-misc at varnish-cache.org Subject: Re: Varnish returns default host instead of virtual host Lars Monsees wrote: > Hi, > > I am very new to Varnish and have only configured my backend as I > guess Varnish will use default values that should work for the > beginning. The problem is that Varnish returns the content of the > server?s default domain (the standard "it work?s" text") instead of > the content of the wanted domain. > backend default { > .host = "127.0.0.1"; > .port = "80"; > } It looks like the backend is on the same machine that varnish is running on. In that case, it needs to be listening on a port other than 80 as you have above, as varnish will be bound to port 80 by default. best wishes, Marcus Hi Lars and Marcus: Varnish would not start if the port was not available, and it's returning the default page because it doesn't know about a host header (yet). Add this for your domain name. if (req.http.host ~ "^(www.)?example.com") { set req.http.host = "www.example.com"; } Stefan From l.monsees at atlantismedia.de Mon Jun 7 16:02:12 2010 From: l.monsees at atlantismedia.de (Lars Monsees) Date: Mon, 7 Jun 2010 18:02:12 +0200 Subject: Varnish returns default host instead of virtual host In-Reply-To: References: Message-ID: Hi Stew, > This is typically an Apache config set up. I had this and it was > because my Listen directive in VirtualHost was listening on the real > IP address, rather than * or 127.0.0.1 exactly the case here. I changed it to * and now it works. Thx =) From l.monsees at atlantismedia.de Mon Jun 7 16:06:04 2010 From: l.monsees at atlantismedia.de (Lars Monsees) Date: Mon, 7 Jun 2010 18:06:04 +0200 Subject: Experience with Varnish and Plesk Message-ID: Hi, I had to change my vhost configuration so that Apache is listening not only on one specific IP but rather on * (127.0.0.1 would probably work, too). Now, as we have a server with Plesk running, Plesk will surely overwrite the configuration the next time something is changed or updated. Has anyone experience in how to get Varnish working on a server with Plesk? Lars From kacperw at gmail.com Mon Jun 7 22:08:49 2010 From: kacperw at gmail.com (Kacper Wysocki) Date: Tue, 8 Jun 2010 00:08:49 +0200 Subject: CLI communication error (hdr) on vcl.discard In-Reply-To: <4C0CAF08.8090702@ze.tum.de> References: <4C0CAF08.8090702@ze.tum.de> Message-ID: On Mon, Jun 7, 2010 at 10:34 AM, Gerhard Schmidt wrote: > Hi, > > I'm in the process of migrating auch website from squid to varnish as reverse > proxy. I use a system to dynamicaly add and remove backends. This system > creates new configs via vcl.inline. Works fine so far. but new configs are > created some times in 10 sec intervals. So the config list gets quite crowded > quite soon. I'm curious. Would you mind posting the workings of your system? -Kacper > When i try to delete old configs via vcl.discard I get an "CLI communication > error (hdr)" > > here is an example cli session > > ?telnet localhost 6082 > Trying 127.0.0.1... > Connected to localhost. > Escape character is '^]'. > 200 154 > ----------------------------- > Varnish HTTP accelerator CLI. > ----------------------------- > Type 'help' for command list. > Type 'quit' to close CLI session. > > vcl.list > 200 228 > available ? ? ? 0 boot > available ? ? ? 0 dynconf_20100607101631 > available ? ? ? 0 dynconf_20100607101856 > available ? ? ? 0 dynconf_20100607103126 > available ? ? ? 2 dynconf_20100607103142 > active ? ? ? ? ?1 dynconf_20100607103209 > > vcl.discard dynconf_20100607101631 > 400 29 > CLI communication error (hdr) > > I'm using varnish 2.1.2 on FreeBSD > > Regards > ? ?Estartu > > -- > ------------------------------------------------- > Gerhard Schmidt ? ? ? | E-Mail: schmidt at ze.tum.de > TU-M?nchen ? ? ? ? ? ?| > WWW & Online Services | > Tel: 089/289-25270 ? ?| > Fax: 089/289-25257 ? ?| PGP-Publickey auf Anfrage > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > -- http://kacper.doesntexist.org http://windows.dontexist.com Employ no technique to gain supreme enlightment. - Mar pa Chos kyi blos gros From alex at acasa.ro Tue Jun 8 13:51:37 2010 From: alex at acasa.ro (Alex F) Date: Tue, 08 Jun 2010 16:51:37 +0300 Subject: varnish constant hdd write Message-ID: <20100608135134.CC3982F7671@smtp1.acasa.ro> Hi, Yesterday I compiled and installed varnish 2.1.2 instead of my old 2.0.5 as it was randomly failing. Now, the new varnish is constantly writing with 1-2megs to my HDD, which in turn causes 100% iowait, constant load of 10-20, random timeouts. My VCL was unchanged. The thing is that I do not know if the old varnish was constantly writing to my HDD, I started being aware of this problem because of the high load and timeouts, which I did not have until today. Varnish has 1.6GB of virtual memory allocated, 3500 hits, and 119.000 cached objects. I have started it using "-s file,/var/lib/varnish/varnish_storage.bin,2G". Is there any better alternative? Also, my machine has ran out of ram, should I decrease the allocated memory? free -m total used free shared buffers cached Mem: 3950 3891 58 0 29 2155 -/+ buffers/cache: 1706 2243 Swap: 3999 22 3977 From perbu at varnish-software.com Tue Jun 8 14:15:08 2010 From: perbu at varnish-software.com (Per Buer) Date: Tue, 8 Jun 2010 16:15:08 +0200 Subject: varnish constant hdd write In-Reply-To: <20100608135134.CC3982F7671@smtp1.acasa.ro> References: <20100608135134.CC3982F7671@smtp1.acasa.ro> Message-ID: Hi. There have been changes in both the FreeBSD and Linux vm in recent years and the file backend doesn't perform as good on rotating hard drives as it used to. Modern virtual machines seem to be much more aggressive when writing. I would recommend you try using malloc or getting a SSD. Or try going back to RHEL4 or FreeBSD 6. :-) Per. On Tue, Jun 8, 2010 at 3:51 PM, Alex F wrote: > Hi, > > Yesterday I compiled and installed varnish 2.1.2 instead of my old 2.0.5 as > it was randomly failing. > Now, the new varnish is constantly writing with 1-2megs to my HDD, which in > turn causes 100% iowait, constant load of 10-20, random timeouts. > My VCL was unchanged. > The thing is that I do not know if the old varnish was constantly writing to > my HDD, I started being aware of this problem because of the high load and > timeouts, which I did not have until today. > > Varnish has 1.6GB of virtual memory allocated, 3500 hits, and 119.000 cached > objects. I have started it using "-s > file,/var/lib/varnish/varnish_storage.bin,2G". Is there any better > alternative? > Also, my machine has ran out of ram, should I decrease the allocated memory? > free -m > ? ? ? ? ? ? ? total ? ? ? ?used ? ? free ? ?shared ? ? buffers ? ?cached > Mem: ? ? ? ? ?3950 ? 3891 ? ? 58 ? ? ? ? ? 0 ? ? ? ? 29 ? ? ? ? 2155 > -/+ buffers/cache: ? 1706 ? ?2243 > Swap: ? ? ? 3999 ? ? ? 22 ? ? ?3977 > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > -- Per Buer, Varnish Software Phone: +47 21 54 41 21 / Mobile: +47 958 39 117 / skype: per.buer From angie.tawfik at gmail.com Tue Jun 8 15:34:26 2010 From: angie.tawfik at gmail.com (Angie T. Muhammad) Date: Tue, 8 Jun 2010 18:34:26 +0300 Subject: Parsing non-English URLs In-Reply-To: References: <877hmk61k6.fsf@qurzaw.linpro.no> <871vcpoofd.fsf@qurzaw.linpro.no> Message-ID: My bad! I was missing something the last time. Eventually the syntax below worked :D .. woot woot !! Thank you Tollef and everybody for helping. On Thu, Jun 3, 2010 at 2:45 PM, Angie T. Muhammad wrote: > Hello group, > > I tried using the %-escaped syntax but it does not work for me. Here is the > rule that varnish totally ignores as if it doesn't exist: > > > =========================================================================================== > # cache AR taxonomy differently > else if (req.url == > "/%25D9%2588%25D8%25B3%25D9%2588%25D9%2585%25D8%25A7%25D8%25AA") { > > set beresp.http.X-Cacheable = "Yes"; > set beresp.ttl = 180m; > return(deliver); > } > > ========================================================================================== > > The link exists, and even copying it in this syntax to the browser > retrieves the page, but the caching for 180 minutes does not work. Other > rules with only English URLS are working like a charm. Please help. > > Thanks. > > > > On Wed, Jun 2, 2010 at 3:56 PM, Angie T. Muhammad wrote: > >> I tried the second syntax many times before. Varnish refuses to start. >> But the good news it accepted the first syntax. I will be checking if >> it applies the rule as required and update you. >> >> Thank you Tollef :) >> >> On 6/2/10, Tollef Fog Heen wrote: >> > ]] "Angie T. Muhammad" >> > >> > | I get some lines like: >> > | 14 TxHeader b Referer: >> > | >> > >> http://myserver.example.com/news/%D8%A7%D9%84%D9%85%D9%84%D9%8A%D8%A7%D8%B1%D8%AF%D9%8A%D8%B1-%D8%A7%D9%84%D9%85%D8%B5%D8%B1%D9%8A-%C2%AB%D9%85%D8%AD%D9%85%D8%AF-%D8%A7%D9%84%D9%81%D8%A7%D9%8A%D8%AF%C2%BB-%D9%8A%D9%82%D8%B1%D8%B1-%D8%A8%D9%8A%D >> > | >> > | The point is that I need to make varnish accept such percentages or >> any >> > | other syntax in VCL file. So that some pages are cached longer based >> on >> > | their Arabic address (Like the one in the referer above). >> > >> > We use %-escaped strings in vcl, so referring to that string above would >> > be to match it to >> > >> > ? >> http://myserver.example.com/news/%25D8%25A7%25D9%2584%25D9%2585%25D9%2584%25D9%258A%25D8%25A7%25D8%25B1%25D8%25AF%25D9%258A%25D8%25B1-%25D8%25A7%25D9%2584%25D9%2585%25D8%25B5%25D8%25B1%25D9%258A-%25C2%25AB%25D9%2585%25D8%25AD%25D9%2585%25D8%25AF-%25D8%25A7%25D9%2584%25D9%2581%25D8%25A7%25D9%258A%25D8%25AF%25C2%25BB-%25D9%258A%25D9%2582%25D8%25B1%25D8%25B1-%25D8%25A8%25D9%258A%25D >> ? >> > >> > or more easily >> > >> > {" >> http://myserver.example.com/news/%D8%A7%D9%84%D9%85%D9%84%D9%8A%D8%A7%D8%B1%D8%AF%D9%8A%D8%B1-%D8%A7%D9%84%D9%85%D8%B5%D8%B1%D9%8A-%C2%AB%D9%85%D8%AD%D9%85%D8%AF-%D8%A7%D9%84%D9%81%D8%A7%D9%8A%D8%AF%C2%BB-%D9%8A%D9%82%D8%B1%D8%B1-%D8%A8%D9%8A%D >> "} >> > >> > -- >> > Tollef Fog Heen >> > Varnish Software >> > t: +47 21 54 41 73 >> > >> >> >> -- >> All the best, >> Angie T. Muhammad >> Linux Systems Engineer >> Al Masry Al Youm >> http://www.almasryalyoum.com/en >> > > > > -- > All the best, > Angie T. Muhammad > Linux Systems Engineer > Al Masry Al Youm > http://www.almasryalyoum.com/en > -- All the best, Angie T. Muhammad Linux Systems Engineer Al Masry Al Youm http://www.almasryalyoum.com/en -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex at acasa.ro Wed Jun 9 07:53:48 2010 From: alex at acasa.ro (Alex F) Date: Wed, 09 Jun 2010 10:53:48 +0300 Subject: varnish constant hdd write In-Reply-To: References: <20100608135134.CC3982F7671@smtp1.acasa.ro> Message-ID: <20100609075349.435BC2F7626@smtp1.acasa.ro> SSD would be a good choice indeed. But not possible right now. I searched about malloc and see that it is used mostly when one has much available RAM. My current system only has 4GB and there is no hope on upgrading. My highest peak on cached objects was 126.000 (with 1 day TTL) and highest Virtual Mem allocated 2.1GB. Available RAM right now is at 1.6GB, but I'd say 1GB given the fact that apache+resin+lighttpd also reside on this machine. Now if I configure Varnish to use malloc 1GB, it only has 1GB to store objects, right? On 8.6.2010 17:15, Per Buer wrote: > Hi. > > There have been changes in both the FreeBSD and Linux vm in recent > years and the file backend doesn't perform as good on rotating hard > drives as it used to. Modern virtual machines seem to be much more > aggressive when writing. > > I would recommend you try using malloc or getting a SSD. Or try going > back to RHEL4 or FreeBSD 6. :-) > > Per. From perbu at varnish-software.com Wed Jun 9 08:18:50 2010 From: perbu at varnish-software.com (Per Buer) Date: Wed, 9 Jun 2010 10:18:50 +0200 Subject: varnish constant hdd write In-Reply-To: <20100609075349.435BC2F7626@smtp1.acasa.ro> References: <20100608135134.CC3982F7671@smtp1.acasa.ro> <20100609075349.435BC2F7626@smtp1.acasa.ro> Message-ID: You're right. Varnish would only have 1GB of memory to store objects. Linux doesn't really do paging very well so your wise to stay within the boundaries of physical memory. This might not be so bad if your backend if somewhat snappy. If your web site is news or portal like most of the 'hot' content will be the content linked from the front page + related content. In most cases a web site won't have more than 100MB of 'hot' content and such 1GB of cache will go a really long way. Per. On Wed, Jun 9, 2010 at 9:53 AM, Alex F wrote: > SSD would be a good choice indeed. But not possible right now. > I searched about malloc and see that it is used mostly when one has much > available RAM. My current system only has 4GB and there is no hope on > upgrading. > My highest peak on cached objects was 126.000 (with 1 day TTL) and highest > Virtual Mem allocated 2.1GB. > Available RAM right now is at 1.6GB, but I'd say 1GB given the fact that > apache+resin+lighttpd also reside on this machine. > > Now if I configure Varnish to use malloc 1GB, it only has 1GB to store > objects, right? > > On 8.6.2010 17:15, Per Buer wrote: >> >> Hi. >> >> There have been changes in both the FreeBSD and Linux vm in recent >> years and the file backend doesn't perform as good on rotating hard >> drives as it used to. Modern virtual machines seem to be much more >> aggressive when writing. >> >> I would recommend you try using malloc or getting a SSD. Or try going >> back to RHEL4 or FreeBSD 6. :-) >> >> Per. > -- Per Buer, Varnish Software Phone: +47 21 54 41 21 / Mobile: +47 958 39 117 / skype: per.buer From tfheen at varnish-software.com Wed Jun 9 08:28:07 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Wed, 09 Jun 2010 10:28:07 +0200 Subject: varnish constant hdd write In-Reply-To: <20100608135134.CC3982F7671@smtp1.acasa.ro> (Alex F.'s message of "Tue, 08 Jun 2010 16:51:37 +0300") References: <20100608135134.CC3982F7671@smtp1.acasa.ro> Message-ID: <87631shap4.fsf@qurzaw.linpro.no> ]] Alex F | Varnish has 1.6GB of virtual memory allocated, 3500 hits, and 119.000 | cached objects. I have started it using "-s | file,/var/lib/varnish/varnish_storage.bin,2G". Is there any better | alternative? | Also, my machine has ran out of ram, should I decrease the allocated memory? | free -m | total used free shared buffers cached | Mem: 3950 3891 58 0 29 2155 | -/+ buffers/cache: 1706 2243 | Swap: 3999 22 3977 You have about 2G of buffer cache, so no, you're not nearly out of memory. In general, if you're seeing I/O problems, you should start by putting the varnish log file (which usually lives in /var/lib/varnish) on a tmpfs, to prevent Linux from writing that to disk. Given you're using a 2G storage file on a 4G machine, you probably also want to use -s malloc rather than -s file. -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From alex at acasa.ro Wed Jun 9 09:13:43 2010 From: alex at acasa.ro (Alex F) Date: Wed, 09 Jun 2010 12:13:43 +0300 Subject: varnish constant hdd write In-Reply-To: <87631shap4.fsf@qurzaw.linpro.no> References: <20100608135134.CC3982F7671@smtp1.acasa.ro> <87631shap4.fsf@qurzaw.linpro.no> Message-ID: <20100609091336.3112F2F7691@smtp1.acasa.ro> I issued a fresh free -m so that's why I said 1.6 free in my last mail: total used free shared buffers cached Mem: 3950 3916 33 0 175 1450 -/+ buffers/cache: 2290 1659 Swap: 3999 0 3999 By "varnish log file" do you mean the .bin? I now see that the most active partition with many HDD writes is /var. So I think it has something to do with /var/lib/varnish given the fact that my websites are located in /usr. Also, can anyone explain to me if there is any relation between virtual memory allocated to varnish and the amount it uses for caching objects? As a side note, I use munin for monitoring, and since I installed the latest version yesterday, the Memory Usage graphic shows that swap is not being used at all, compared to constant 1GB of swap used by the 2.0.5 varnish version. Screenshot: http://i48.tinypic.com/66f7us.png On 9.6.2010 11:28, Tollef Fog Heen wrote: > You have about 2G of buffer cache, so no, you're not nearly out of > memory. In general, if you're seeing I/O problems, you should start by > putting the varnish log file (which usually lives in /var/lib/varnish) > on a tmpfs, to prevent Linux from writing that to disk. > > Given you're using a 2G storage file on a 4G machine, you probably also > want to use -s malloc rather than -s file Indeed, it is a web portal. I cache only (txt|ico|png|jpeg|jpg|gif|tiff|js|css). On 9.6.2010 11:18, Per Buer wrote: > You're right. Varnish would only have 1GB of memory to store objects. > Linux doesn't really do paging very well so your wise to stay within > the boundaries of physical memory. > > This might not be so bad if your backend if somewhat snappy. If your > web site is news or portal like most of the 'hot' content will be the > content linked from the front page + related content. In most cases a > web site won't have more than 100MB of 'hot' content and such 1GB of > cache will go a really long way. > > Per. From hap at spamproof.nospammail.net Wed Jun 9 17:34:21 2010 From: hap at spamproof.nospammail.net (Hank A. Paulson) Date: Wed, 09 Jun 2010 10:34:21 -0700 Subject: varnish constant hdd write In-Reply-To: <20100609091336.3112F2F7691@smtp1.acasa.ro> References: <20100608135134.CC3982F7671@smtp1.acasa.ro> <87631shap4.fsf@qurzaw.linpro.no> <20100609091336.3112F2F7691@smtp1.acasa.ro> Message-ID: <4C0FD09D.8050707@spamproof.nospammail.net> You can cut it pretty close on the memory if you are using the box just for varnish - 400,178,190 connections with just 735 MB of 11996 MB memory free :) [root at varn198 ~]# varnishstat -1 | egrep "n_object|client_conn" client_conn 400178190 68.82 Client connections accepted n_object 345931 . N struct object n_objecthead 345933 . N struct objecthead [root at varn198 ~]# fgrep malloc /etc/sysconfig/varnish -s malloc,11100M \ [root at varn198 ~]# free -m total used free shared buffers cached Mem: 11996 11727 269 0 243 223 -/+ buffers/cache: 11260 735 Swap: 0 0 0 [root at varn198 ~]# uname -a Linux varn198 2.6.32.10-90.fc12.x86_64 #1 SMP Tue Mar 23 09:47:08 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux [root at varn198 ~]# w 00:20:33 up 67 days, 7:23, 1 user, load average: 0.00, 0.00, 0.00 On 6/9/10 2:13 AM, Alex F wrote: > I issued a fresh free -m so that's why I said 1.6 free in my last mail: > total used free shared buffers cached > Mem: 3950 3916 33 0 175 1450 > -/+ buffers/cache: 2290 1659 > Swap: 3999 0 3999 > > By "varnish log file" do you mean the .bin? > I now see that the most active partition with many HDD writes is /var. > So I think it has something to do with /var/lib/varnish given the fact > that my websites are located in /usr. > > Also, can anyone explain to me if there is any relation between virtual > memory allocated to varnish and the amount it uses for caching objects? > > As a side note, I use munin for monitoring, and since I installed the > latest version yesterday, the Memory Usage graphic shows that swap is > not being used at all, compared to constant 1GB of swap used by the > 2.0.5 varnish version. Screenshot: http://i48.tinypic.com/66f7us.png > > On 9.6.2010 11:28, Tollef Fog Heen wrote: >> You have about 2G of buffer cache, so no, you're not nearly out of >> memory. In general, if you're seeing I/O problems, you should start by >> putting the varnish log file (which usually lives in /var/lib/varnish) >> on a tmpfs, to prevent Linux from writing that to disk. >> >> Given you're using a 2G storage file on a 4G machine, you probably also >> want to use -s malloc rather than -s file > > Indeed, it is a web portal. I cache only > (txt|ico|png|jpeg|jpg|gif|tiff|js|css). > > On 9.6.2010 11:18, Per Buer wrote: >> You're right. Varnish would only have 1GB of memory to store objects. >> Linux doesn't really do paging very well so your wise to stay within >> the boundaries of physical memory. >> >> This might not be so bad if your backend if somewhat snappy. If your >> web site is news or portal like most of the 'hot' content will be the >> content linked from the front page + related content. In most cases a >> web site won't have more than 100MB of 'hot' content and such 1GB of >> cache will go a really long way. >> >> Per. > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From michaell at dazzit.com Thu Jun 10 00:24:40 2010 From: michaell at dazzit.com (Michael Lenaghan) Date: Wed, 9 Jun 2010 20:24:40 -0400 Subject: Varnish memory usage Message-ID: I'm running Varnish 2.1 on Ubuntu 10.04. We ran a test to purposely fill Varnish's cache. "top" shows: 1486m 769 nobody 20 0 1486m 1.3g 81m S 0.0 78.6 0:03.71 varnishd From michaell at dazzit.com Thu Jun 10 00:29:24 2010 From: michaell at dazzit.com (Michael Lenaghan) Date: Wed, 9 Jun 2010 20:29:24 -0400 Subject: Varnish memory usage In-Reply-To: References: Message-ID: What a way to start! Obviously, I accidentally hit "send." Let's try this again... I'm running Varnish 2.1 on Ubuntu 10.04. We ran a test to purposely fill Varnish's cache. We've defined that cache as "malloc,1G". After running our test, "top" shows: 1486m VIRT memory 1.3g RES memory 81m SHR memory Either I've mis-configured something or mis-understood something. Either way, help is appreciated. In case it's of use the output of varnishstat follows. 0+05:22:49 domU-12-31-38-00-9D-27 Hitrate ratio: 10 100 1000 Hitrate avg: 0.0381 0.0813 0.0705 3889 0.00 0.20 Client connections accepted 227735 0.99 11.76 Client requests received 10916 0.00 0.56 Cache hits 215592 0.00 11.13 Cache misses 211160 0.00 10.90 Backend conn. success 12 0.00 0.00 Backend conn. failures 5646 0.99 0.29 Backend conn. reuses 211144 0.00 10.90 Backend conn. was closed 216791 0.99 11.19 Backend conn. recycles 4575 0.99 0.24 Fetch with Length 212231 0.00 10.96 Fetch chunked 31 . . N struct sess_mem 2 . . N struct sess 205011 . . N struct object 205014 . . N struct objectcore 205014 . . N struct objecthead 1 . . N struct vbe_conn 500 . . N worker threads 500 0.00 0.03 N worker threads created 1259710 0.00 65.04 N worker threads limited 2 . . N backends 10569 . . N expired objects 4425 . . N LRU moved objects 216589 0.99 11.18 Objects sent with write 3889 0.00 0.20 Total Sessions 227735 0.99 11.76 Total Requests 1226 0.99 0.06 Total pass 216806 0.99 11.19 Total fetch 97606086 221.79 5039.29 Total header bytes 833650072 2.97 43040.43 Total body bytes 13 0.00 0.00 Session Closed 231611 0.99 11.96 Session herd 21199614 59.41 1094.51 SHM records 697327 2.97 36.00 SHM writes 38 0.00 0.00 SHM MTX contention 8 0.00 0.00 SHM cycles through buffer 432162 0.99 22.31 SMA allocator requests 410022 . . SMA outstanding allocations 966403669 . . SMA outstanding bytes 27988956353 . . SMA bytes allocated 27022552684 . . SMA bytes free 13 0.00 0.00 SMS allocator requests 3670 . . SMS bytes allocated 3670 . . SMS bytes freed 216806 0.99 11.19 Backend requests made 1 0.00 0.00 N vcl total 1 0.00 0.00 N vcl available 1 . . N total active purges 1 0.00 0.00 N new purges added 19369 1.98 1.00 Client uptime From kb+varnish at slide.com Thu Jun 10 01:31:56 2010 From: kb+varnish at slide.com (Ken Brownfield) Date: Wed, 9 Jun 2010 18:31:56 -0700 Subject: Varnish memory usage In-Reply-To: References: Message-ID: <07D0A945-FA47-4061-B3E1-455320C3CA0C@slide.com> > 966403669 . . SMA outstanding bytes There's your 1GB cache buffer, which is what you're setting. Obviously, Varnish needs memory to do other things, so you'll see a fixed couple of hundred extra megs. Also, each thread (e.g., worker) will allocate its own memory, including a stack. This memory usage will scale and contract as load spawns and reaps threads. This is true of any server, especially examples such as Squid and MySQLd. Linux (including 10.04) sets a default stack size of 8MB: -s: stack size (kbytes) 8192 With 500 threads, you could in theory see up to 4GB of VIRT. Run "pmap -x PID" on the child process (or grandchildren), and you will see where the memory goes. It should reflect the above -- you'll see 500+ 8MB stacks, plus a lot of other 1MB allocations, which is the block size used by jemalloc IIRC. If you really only want to use 1G, specify something like "malloc,800M". Hope it helps, -- Ken On Jun 9, 2010, at 5:29 PM, Michael Lenaghan wrote: > What a way to start! Obviously, I accidentally hit "send." Let's try > this again... > > I'm running Varnish 2.1 on Ubuntu 10.04. We ran a test to purposely > fill Varnish's cache. We've defined that cache as "malloc,1G". After > running our test, "top" shows: > > 1486m VIRT memory > 1.3g RES memory > 81m SHR memory > > Either I've mis-configured something or mis-understood something. > Either way, help is appreciated. > > In case it's of use the output of varnishstat follows. > > 0+05:22:49 domU-12-31-38-00-9D-27 > Hitrate ratio: 10 100 1000 > Hitrate avg: 0.0381 0.0813 0.0705 > > 3889 0.00 0.20 Client connections accepted > 227735 0.99 11.76 Client requests received > 10916 0.00 0.56 Cache hits > 215592 0.00 11.13 Cache misses > 211160 0.00 10.90 Backend conn. success > 12 0.00 0.00 Backend conn. failures > 5646 0.99 0.29 Backend conn. reuses > 211144 0.00 10.90 Backend conn. was closed > 216791 0.99 11.19 Backend conn. recycles > 4575 0.99 0.24 Fetch with Length > 212231 0.00 10.96 Fetch chunked > 31 . . N struct sess_mem > 2 . . N struct sess > 205011 . . N struct object > 205014 . . N struct objectcore > 205014 . . N struct objecthead > 1 . . N struct vbe_conn > 500 . . N worker threads > 500 0.00 0.03 N worker threads created > 1259710 0.00 65.04 N worker threads limited > 2 . . N backends > 10569 . . N expired objects > 4425 . . N LRU moved objects > 216589 0.99 11.18 Objects sent with write > 3889 0.00 0.20 Total Sessions > 227735 0.99 11.76 Total Requests > 1226 0.99 0.06 Total pass > 216806 0.99 11.19 Total fetch > 97606086 221.79 5039.29 Total header bytes > 833650072 2.97 43040.43 Total body bytes > 13 0.00 0.00 Session Closed > 231611 0.99 11.96 Session herd > 21199614 59.41 1094.51 SHM records > 697327 2.97 36.00 SHM writes > 38 0.00 0.00 SHM MTX contention > 8 0.00 0.00 SHM cycles through buffer > 432162 0.99 22.31 SMA allocator requests > 410022 . . SMA outstanding allocations > 966403669 . . SMA outstanding bytes > 27988956353 . . SMA bytes allocated > 27022552684 . . SMA bytes free > 13 0.00 0.00 SMS allocator requests > 3670 . . SMS bytes allocated > 3670 . . SMS bytes freed > 216806 0.99 11.19 Backend requests made > 1 0.00 0.00 N vcl total > 1 0.00 0.00 N vcl available > 1 . . N total active purges > 1 0.00 0.00 N new purges added > 19369 1.98 1.00 Client uptime > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From michaell at dazzit.com Thu Jun 10 02:16:10 2010 From: michaell at dazzit.com (Michael Lenaghan) Date: Wed, 9 Jun 2010 22:16:10 -0400 Subject: Varnish memory usage In-Reply-To: <07D0A945-FA47-4061-B3E1-455320C3CA0C@slide.com> References: <07D0A945-FA47-4061-B3E1-455320C3CA0C@slide.com> Message-ID: Thanks for the reply, Ken. >> ? 966403669 ? ? ? ? ?. ? ? ? ? ? ?. ? SMA outstanding bytes > > There's your 1GB cache buffer, which is what you're setting. OK, I see that "SMA bytes allocated" - "SMA bytes free" = "SMA outstanding bytes". Perhaps, in the process of allocating and freeing, memory has become fragmented--leaving unusable holes...? > Linux (including 10.04) sets a default stack size of 8MB: > > -s: stack size (kbytes) ? ? ? ?8192 > > With 500 threads, you could in theory see up to 4GB of VIRT. Varnish sets the stack size to 65K on this system by default: thread_pool_stack 65536 [bytes] Default is -1 Worker thread stack size. That works out to 32 MB for 500 threads. And I force the creation of all threads up front in order to avoid surprise memory allocations under load. > Run "pmap -x PID" on the child process (or grandchildren), and you will see where the memory goes. > It should reflect the above -- you'll see 500+ 8MB stacks, plus a lot of other 1MB allocations, which > is the block size used by jemalloc IIRC. Using pmap I can confirm the 65 KB thread stacks. After pushing Varnish a little further--it's at 1.6 GB virtual memory now--I also see one line like this: 52f00000 1432576 - - - rw--- [ anon ] That's a single 1.4 GB allocation? The next biggest allocations (in address order) are: ac3f6000 10240 - - - rw--- [ anon ] acdf7000 10240 - - - rw--- [ anon ] ad7f8000 10240 - - - rw--- [ anon ] ae200000 12288 - - - rw--- [ anon ] afaf0000 10304 - - - rw--- [ anon ] b05ed000 10240 - - - rw--- [ anon ] b0fee000 10240 - - - rw--- [ anon ] b19ef000 10240 - - - rw--- [ anon ] b23ef000 81988 - - - rw-s- _.vsl Those ten allocations account for nearly all of the memory used by Varnish. (The first line alone is just under 90%.) > If you really only want to use 1G, specify something like "malloc,800M". Yes, I can try that. And in production I think I can set things up so that I guarantee Varnish never fills its cache. But I wanted to see how Varnish performed at the limit. And I'd like to get as close to the limit as I safely can. From alex at acasa.ro Thu Jun 10 08:34:27 2010 From: alex at acasa.ro (Alex F) Date: Thu, 10 Jun 2010 11:34:27 +0300 Subject: varnish constant hdd write In-Reply-To: <87631shap4.fsf@qurzaw.linpro.no> References: <20100608135134.CC3982F7671@smtp1.acasa.ro> <87631shap4.fsf@qurzaw.linpro.no> Message-ID: <20100610083432.A7D332F76DE@smtp1.acasa.ro> Hello again, As of yesterday the problem seems to have stopped. And I think it was related to varnish constantly caching objects. It has reached the limit now, 160.000 objects (wohoo), 2.06gb VM, no swapping, no more iowait, disk usage has dropped to normal figures. Compared to the old version, I now have a lot more hits in varnish, and in munin graphics are looking nice and constant. In the past, things were kind of chaotic, the Objects graphic would drop/rise randomly before hitting the cache TTL. Even more, the max cached objects it reached was at 130.000, compared to 160.000 with the new version. And I still have 1.9GB of free memory, so my fears of running out of ram were unconfirmed. I guess I could switch over to malloc because I'm curious about performance gain. So yeah, I'm liking v2.1.2! Will wait a couple of days and install it on my second cluster aswell. My regards to the varnish team. From michaell at dazzit.com Thu Jun 10 16:03:17 2010 From: michaell at dazzit.com (Michael Lenaghan) Date: Thu, 10 Jun 2010 12:03:17 -0400 Subject: Varnish memory usage In-Reply-To: References: <07D0A945-FA47-4061-B3E1-455320C3CA0C@slide.com> Message-ID: Hello, all. Just thought I'd share our results: * With "malloc,512M" Varnish seemed to stabilize at ~935 MB virtual memory. * With "malloc,850M" Varnish seemed to stabilize at ~1,330 MB virtual memory. * With "malloc,1G" Varnish seemed to stabilize at ~1,630 MB virtual memory. No doubt all of this is sensitive to our test patterns, etc. Still, might be useful. Thanks again, Ken. From richard.chiswell at mangahigh.com Thu Jun 10 17:20:22 2010 From: richard.chiswell at mangahigh.com (Richard Chiswell) Date: Thu, 10 Jun 2010 18:20:22 +0100 Subject: Varnish memory usage In-Reply-To: References: <07D0A945-FA47-4061-B3E1-455320C3CA0C@slide.com> Message-ID: <4C111ED6.6050008@mangahigh.com> Hi Ken, That's useful - how many threads and thread pools are you using? Rich On 10/06/2010 17:03, Michael Lenaghan wrote: > Hello, all. Just thought I'd share our results: > > * With "malloc,512M" Varnish seemed to stabilize at ~935 MB virtual memory. > > * With "malloc,850M" Varnish seemed to stabilize at ~1,330 MB virtual memory. > > * With "malloc,1G" Varnish seemed to stabilize at ~1,630 MB virtual memory. > > No doubt all of this is sensitive to our test patterns, etc. Still, > might be useful. > > Thanks again, Ken. > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From michaell at dazzit.com Thu Jun 10 18:08:07 2010 From: michaell at dazzit.com (Michael Lenaghan) Date: Thu, 10 Jun 2010 14:08:07 -0400 Subject: Varnish memory usage In-Reply-To: <4C111ED6.6050008@mangahigh.com> References: <07D0A945-FA47-4061-B3E1-455320C3CA0C@slide.com> <4C111ED6.6050008@mangahigh.com> Message-ID: > That's useful - how many threads and thread pools are you using? Hello, Rich. I think the question was actually aimed at me... I'm using: -h classic,333333 -p listen_depth=2048 -p thread_pools=1 -p thread_pool_min=500 -p thread_pool_max=500 -p sess_timeout=600 -p session_linger=0 (The hash table is bigger than I need at this point, but that's what the test used.) In other news: I ran the test on an Amazon m1.small instance type ("equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor" with 1.7 GB of memory). In a separate performance test that box could achieve ~6,000 RPS* on a tiny document (~128 B), or ~10,000 RPS if requests were pipelined. Large documents (> ~32 KB) could saturate the network pipe (~80,000 KB/s) at a significantly lower RPS. CPU utilization never exceeded ~38%, though there's a significant discrepancy between the numbers reported by top and the numbers reported by Amazon's CloudWatch. === *RPS means "requests per second," not "sessions per second." For what we're doing I wasn't interested in measuring TCP connection overhead. From alex at acasa.ro Fri Jun 11 09:12:44 2010 From: alex at acasa.ro (Alex F) Date: Fri, 11 Jun 2010 12:12:44 +0300 Subject: automatically compile default.vcl at startup In-Reply-To: References: Message-ID: <20100611091236.E4F7E2F7663@smtp1.acasa.ro> Hello, One little question: Shouldn't varnish automatically compile my vcl when I start it up? DAEMON_OPTS="-a 127.0.0.1:3880 \ -T 192.168.100.1:6082 \ -f /etc/varnish/default.vcl \ -u varnish -g varnish \ -s malloc,2G" service varnish start Starting varnish HTTP accelerator: storage_malloc: max size 2048 MB. Using old SHMFILE [ OK ] telnet 192.168.100.1 6082 vcl.list 200 23 active 1 boot and I have to manually load it: vcl.load default.vcl /etc/varnish/default.vcl 200 13 VCL compiled. vcl.use default.vcl 200 0 vcl.list 200 53 available 0 boot active 1 default.vcl It used to compile it automatically. From phk at phk.freebsd.dk Fri Jun 11 09:22:20 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 11 Jun 2010 09:22:20 +0000 Subject: automatically compile default.vcl at startup In-Reply-To: Your message of "Fri, 11 Jun 2010 12:12:44 +0300." <20100611091236.E4F7E2F7663@smtp1.acasa.ro> Message-ID: <18774.1276248140@critter.freebsd.dk> In message <20100611091236.E4F7E2F7663 at smtp1.acasa.ro>, Alex F writes: >Hello, >One little question: >Shouldn't varnish automatically compile my vcl when I start it up? > >telnet 192.168.100.1 6082 >vcl.list >200 23 >active 1 boot > That means it did compile it. It wouldn't start up, if it couldn't. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From alex at acasa.ro Fri Jun 11 09:36:31 2010 From: alex at acasa.ro (Alex F) Date: Fri, 11 Jun 2010 12:36:31 +0300 Subject: automatically compile default.vcl at startup In-Reply-To: <18774.1276248140@critter.freebsd.dk> References: <18774.1276248140@critter.freebsd.dk> Message-ID: <20100611093632.DC4892F75D9@smtp1.acasa.ro> I see. I had the impression that it didn't. False alarm. Thank you! On 11.6.2010 12:22, Poul-Henning Kamp wrote: > > That means it did compile it. > > It wouldn't start up, if it couldn't. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marc.fournier at camptocamp.com Fri Jun 11 12:55:48 2010 From: marc.fournier at camptocamp.com (Marc Fournier) Date: Fri, 11 Jun 2010 14:55:48 +0200 Subject: a couple of questions about -spersistent Message-ID: <20100611145548.0a7fe866@lonquimay.wrk.lsn.camptocamp.com> Hello, I've found a couple of threads on the list where people were trying to request objects from the cache after a restart, but none of them seem to have been successful, and the threads ended without any solution. My question is: what am I missing to be able to: - fetch an object from a backend to prime the cache - fetch this object a 2nd time to check I get a HIT - restart varnish (as in /etc/init.d/varnish restart) - fetch the object and get a HIT. Has someone actually managed to have data persist in the cache across restarts ? Another thing I noticed is that when running "varnishd -spersistent,...", varnishstat always reports: sm_balloc 0 . bytes allocated sm_bfree 0 . bytes free even when some objects are present in the cache (requests return cache hits). Are these values supposed be called something else in persistent mode ? With "varnishd -sfile,/path/to/file,N", sm_balloc + sm_bfree = N, which makes sense. Thanks ! Marc From harald at webmeisterei.com Mon Jun 14 12:53:14 2010 From: harald at webmeisterei.com (Harald Friessnegger) Date: Mon, 14 Jun 2010 14:53:14 +0200 Subject: munin plugin varnish_ - feature request: support multiple varnish instances Message-ID: <201006141453.14787.harald@webmeisterei.com> This is a feature request for the munin plugin varnish_ Although it's rather trivial to implement i'm not enough into perl to contribute a patch. We?re running multiple varnish instances (different varnishd/varnishstats binaries) on our servers. Currently varnish_ only allows to monitor one of them. (http://varnish-cache.org/browser/trunk/varnish-tools/munin/varnish_?rev=4439) As a workaround i?m having something like varnish1_ varnish2_ scripts with some modifications: line 759: $self =~ s/^.*\/varnish_//; becomes line 759: $self =~ s/^.*\/varnish1_//; and line 731: print ?graph_category Varnish\n?; becomes line 731: print ?graph_category Varnish Project 1\n?; Would be great if varnish_ was a wildcard plugin such as haproxy (http://munin.projects.linpro.no/browser/trunk/plugins/node.d/haproxy_.in) as the ending ?_? already suggests. This way one needn?t copy/paste/modify varnish_ but simly use symlinks like varnish_instance1_uptime and varnish_instance2_uptime Thanks for your help/attention fRiSi -- Webmeisterei GmbH - B?ro f?r Netzfragen Tel: +43 5572 908877, Fax: +43 5572 908877-66 Steinebach 18, A-6850 Dornbirn http://www.webmeisterei.com From harald at webmeisterei.com Mon Jun 14 13:00:58 2010 From: harald at webmeisterei.com (Harald Friessnegger) Date: Mon, 14 Jun 2010 15:00:58 +0200 Subject: munin plugin varnish_ - feature request: support multiple varnish instances In-Reply-To: <201006141453.14787.harald@webmeisterei.com> References: <201006141453.14787.harald@webmeisterei.com> Message-ID: <201006141500.58996.harald@webmeisterei.com> i forgot to add that the most complicated part in this might be to stay backward compatible with existing installations that symlink their plugins with just one parameter (the aspect) varnish_ -> varnish_expunge instead of the new way (instance + aspect) varnish_ -> varnish_project1_expunge i don't have any idea on how to solve that atm.. :-( regards, fRiSi Am Montag, 14. Juni 2010, um 14:53:14 schrieb Harald Friessnegger: > This is a feature request for the munin plugin varnish_ > Although it's rather trivial to implement i'm not enough into perl to > contribute a patch. > > > We?re running multiple varnish instances (different varnishd/varnishstats > binaries) on our servers. > > Currently varnish_ only allows to monitor one of them. > (http://varnish-cache.org/browser/trunk/varnish-tools/munin/varnish_?rev=44 > 39) > > As a workaround i?m having something like varnish1_ varnish2_ scripts with > some modifications: > > line 759: $self =~ s/^.*\/varnish_//; > becomes > line 759: $self =~ s/^.*\/varnish1_//; > > and > line 731: print ?graph_category Varnish\n?; > becomes > line 731: print ?graph_category Varnish Project 1\n?; > > > Would be great if varnish_ was a wildcard plugin such as haproxy > (http://munin.projects.linpro.no/browser/trunk/plugins/node.d/haproxy_.in) > as the ending ?_? already suggests. > > This way one needn?t copy/paste/modify varnish_ but simly use symlinks like > varnish_instance1_uptime and varnish_instance2_uptime > > > Thanks for your help/attention > fRiSi -- Webmeisterei GmbH - B?ro f?r Netzfragen Tel: +43 5572 908877, Fax: +43 5572 908877-66 Steinebach 18, A-6850 Dornbirn http://www.webmeisterei.com From bjorn at ruberg.no Mon Jun 14 13:33:36 2010 From: bjorn at ruberg.no (=?UTF-8?Q?Bj=C3=B8rn_Ruberg?=) Date: Mon, 14 Jun 2010 15:33:36 +0200 Subject: munin plugin =?UTF-8?Q?varnish=5F=20-=20feature=20request=3A=20suppor?= =?UTF-8?Q?t=20multiple=20varnish=20instances?= In-Reply-To: <201006141500.58996.harald@webmeisterei.com> References: <201006141453.14787.harald@webmeisterei.com> <201006141500.58996.harald@webmeisterei.com> Message-ID: <142b8b61fda41397a359f29913bbc1e0@doffen.hjemme.ruberg.no> > i forgot to add that the most complicated part in this might be to stay > backward compatible with existing installations that symlink their plugins > with just one parameter (the aspect) > > varnish_ -> varnish_expunge > > instead of the new way (instance + aspect) > > varnish_ -> varnish_project1_expunge > > > > i don't have any idea on how to solve that atm.. :-( While breaking old links if you're not renaming them manually (which is no big deal really), the best way to solve the OP's challenge is rewriting the plugin into a multigraph plugin (compatible with Munin from 1.4 and above). That way the symlinks will only point to the instance (i.e. varnish_ -> varnish_project1, varnish_ -> project2 etc), and all aspects will be managed internally in the plugin. -- Bj?rn From jdixon at omniti.com Tue Jun 15 03:53:47 2010 From: jdixon at omniti.com (Jason Dixon) Date: Mon, 14 Jun 2010 23:53:47 -0400 Subject: CFP for Surge Scalability Conference 2010 Message-ID: <20100615035347.GR29381@omniti.com> We're excited to announce Surge, the Scalability and Performance Conference, to be held in Baltimore on Sept 30 and Oct 1, 2010. The event focuses on case studies that demonstrate successes (and failures) in Web applications and Internet architectures. Our Keynote speakers include John Allspaw and Theo Schlossnagle. We are currently accepting submissions for the Call For Papers through July 9th. You can find more information, including our current list of speakers, online: http://omniti.com/surge/2010 If you've been to Velocity, or wanted to but couldn't afford it, then Surge is just what you've been waiting for. For more information, including CFP, sponsorship of the event, or participating as an exhibitor, please contact us at surge at omniti.com. Thanks, -- Jason Dixon OmniTI Computer Consulting, Inc. jdixon at omniti.com 443.325.1357 x.241 From holger.rauch at empic.de Tue Jun 15 08:12:04 2010 From: holger.rauch at empic.de (Holger Rauch) Date: Tue, 15 Jun 2010 10:12:04 +0200 Subject: Varnish & ClamAV on FreeBSD 8 STABLE Message-ID: <20100615081204.GA16478@heitec.de> Hi, has anybody tried setting up Varnish in conjunction with ClamAV on FreeBSD 8 STABLE or, more generally asked, does Varnish support some kind of "interface" to virus scanners? I haven't come accross anything in the Varnish wiki. Thanks in advance & kind regards, Holger -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: Digital signature URL: From anders at fupp.net Tue Jun 15 10:07:51 2010 From: anders at fupp.net (Anders Nordby) Date: Tue, 15 Jun 2010 12:07:51 +0200 Subject: B-heap performance and stability in practice Message-ID: <20100615100751.GA86372@fupp.net> Hi, I had a discussion with PHK regarding the performance and stability of r4646, aka the introduction of the B-heap. What I noticed after upgrading from Varnish trunk version 4516 to 4755 which includes this change: - CPU usage/load peaks went up generating warnings in Nagios. - I get occasional hangs on HTTP, from 3 to 10 seconds or more, also generating alerts. - Varnish has space for more objects, but the total number of objects fluctuates more. - Disk wait time peaks are up by a couple of milliseconds (FYI I use swap (malloc) backing on SSD in FreeBSD with max size set to 60 GB). - Number of VM faults seems identical. To back up my claims, I made a page with a graph comparison between cache11 (which ran 4755 until june 8th when I downgraded to 4516 again) and cache12 (which has been running 4516 all the time): http://anders.fupp.net/Varnish-b-heap/ My conclusion so far after downgrading from 4755 to 4516 is that in my setup, Varnish trunk 4516 runs with less hiccups and load than 4755 which has the B-heap change. I wonder if anyone else has been testing and measuring this change with setups that has a large number of objects/large dataset. Any feedback welcome! Regards, -- Anders. From phk at phk.freebsd.dk Tue Jun 15 18:53:39 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 15 Jun 2010 18:53:39 +0000 Subject: B-heap performance and stability in practice In-Reply-To: Your message of "Tue, 15 Jun 2010 12:07:51 +0200." <20100615100751.GA86372@fupp.net> Message-ID: <36986.1276628019@critter.freebsd.dk> In message <20100615100751.GA86372 at fupp.net>, Anders Nordby writes: >Hi, > >I had a discussion with PHK regarding the performance and stability of >r4646, aka the introduction of the B-heap. What I noticed after >upgrading from Varnish trunk version 4516 to 4755 which includes this >change: There is a significant number of changes between those two deltas, and I havn't seen any indication that the B-heap is involved. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From lists at zopyx.com Wed Jun 16 07:38:12 2010 From: lists at zopyx.com (Andreas Jung) Date: Wed, 16 Jun 2010 09:38:12 +0200 Subject: Configuring a fallback server for a director Message-ID: Is it possible to configure a fallback server (serving a static maintenance page) for a director with some backend servers in case all backend servers are down? -aj -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Wed Jun 16 08:35:44 2010 From: perbu at varnish-software.com (Per Buer) Date: Wed, 16 Jun 2010 10:35:44 +0200 Subject: Configuring a fallback server for a director In-Reply-To: References: Message-ID: On Wed, Jun 16, 2010 at 9:38 AM, Andreas Jung wrote: > Is it possible to configure a fallback server (serving a static maintenance > page) for a director with some backend > servers in case all backend servers are down? At the moment I don't think it is possible. But you could serve a synthetic response in vcl_error. You can even have inline images, I believe. -- Per Buer, Varnish Software Phone: +47 21 54 41 21 / Mobile: +47 958 39 117 / skype: per.buer From marc.fournier at camptocamp.com Wed Jun 16 08:56:12 2010 From: marc.fournier at camptocamp.com (Marc Fournier) Date: Wed, 16 Jun 2010 10:56:12 +0200 Subject: Configuring a fallback server for a director References: Message-ID: <20100616105612.7e433242@lonquimay.wrk.lsn.camptocamp.com> Hello, > Is it possible to configure a fallback server (serving a static > maintenance page) for a director with some backend > servers in case all backend servers are down? I had the same need some time ago, and solved it by simply switching backends when the main ones are in an "unhealthy" state: director cluster round-robin { { .backend = { ... .probe = { # required to let req.backend.healthy work ... } } } ... } backend sorryserver { ... } sub vcl_recv { set req.backend = cluster; if (!req.backend.healthy) { set req.backend = sorryserver; pass; } } I hope this helps ! Marc From marc.fournier at camptocamp.com Wed Jun 16 09:06:33 2010 From: marc.fournier at camptocamp.com (Marc Fournier) Date: Wed, 16 Jun 2010 11:06:33 +0200 Subject: munin plugin varnish_ - feature request: support multiple varnish instances References: <201006141453.14787.harald@webmeisterei.com> Message-ID: <20100616110633.77ab4944@lonquimay.wrk.lsn.camptocamp.com> Hello, > This is a feature request for the munin plugin varnish_ > Although it's rather trivial to implement i'm not enough into perl to > contribute a patch. > > > We?re running multiple varnish instances (different > varnishd/varnishstats binaries) on our servers. > > Currently varnish_ only allows to monitor one of them. > (http://varnish-cache.org/browser/trunk/varnish-tools/munin/varnish_?rev=4439) In case you're just using munin to collect varnish stats in RRD files, you could use collectd instead, which has a plugin for varnish which supports multiple instances. It's currently in development and testing would be welcome. -> http://collectd.org/wiki/index.php/Plugin:Varnish If you have a whole munin infrastructure, this workaround will probably not be the best solution. Marc From lists at zopyx.com Wed Jun 16 16:23:51 2010 From: lists at zopyx.com (Andreas Jung) Date: Wed, 16 Jun 2010 18:23:51 +0200 Subject: Configuring a fallback server for a director In-Reply-To: <20100616105612.7e433242@lonquimay.wrk.lsn.camptocamp.com> References: <20100616105612.7e433242@lonquimay.wrk.lsn.camptocamp.com> Message-ID: Excellent hint - works perfectly! Andreas 2010/6/16 Marc Fournier > Hello, > > > Is it possible to configure a fallback server (serving a static > > maintenance page) for a director with some backend > > servers in case all backend servers are down? > > I had the same need some time ago, and solved it by simply switching > backends when the main ones are in an "unhealthy" state: > > director cluster round-robin { > { .backend = { > ... > .probe = { # required to let req.backend.healthy work > ... > } > } > } > ... > } > > backend sorryserver { > ... > } > > sub vcl_recv { > > set req.backend = cluster; > > if (!req.backend.healthy) { > set req.backend = sorryserver; > pass; > } > } > > I hope this helps ! > Marc > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marc.fournier at camptocamp.com Thu Jun 17 12:17:54 2010 From: marc.fournier at camptocamp.com (Marc Fournier) Date: Thu, 17 Jun 2010 14:17:54 +0200 Subject: a couple of questions about -spersistent References: <20100611145548.0a7fe866@lonquimay.wrk.lsn.camptocamp.com> Message-ID: <20100617141754.7c062ee9@lonquimay.wrk.lsn.camptocamp.com> Hello, > I've found a couple of threads on the list where people were trying to > request objects from the cache after a restart, but none of them seem > to have been successful, and the threads ended without any solution. > > My question is: what am I missing to be able to: > > - fetch an object from a backend to prime the cache > - fetch this object a 2nd time to check I get a HIT > - restart varnish (as in /etc/init.d/varnish restart) > - fetch the object and get a HIT. > > Has someone actually managed to have data persist in the cache across > restarts ? Hum, must I interpret the lack of answers as "no, no one never got persistence working" ? And therefore, should I open a bug report ? Thanks for any hints :-) Marc From jean-francois.laurens at rts.ch Thu Jun 17 13:00:16 2010 From: jean-francois.laurens at rts.ch (Jean-Francois Laurens) Date: Thu, 17 Jun 2010 15:00:16 +0200 Subject: Byte range and varnish Message-ID: <33C23F91-FFD0-4E14-8ACD-FF07F694DB93@rts.ch> Hi, I'm running varnish 2.0.6 under centos 5.4: varnish-libs-2.0.6-2.el5 varnish-2.0.6-2.el5 I'm setting up a media server and would like to use varnish for caching some of the heavily requested medias (mp3, mp4 ... ). I've a few questions regarding media broadcasting with varnish. mp3: progressive download is used in order to read mp3 from the webpages, is it foreseen to have a function to would allow providing the media before it gets completely retrieved by varnish from the backend ? mp4: how is handled Accept-range field in the header, does varnish supports it ? Trying to retrieve a mp4 file from apache gives me the following response: HTTP/1.1 200 OK Date: Tue, 15 Jun 2010 14:24:46 GMT Server: Apache Last-Modified: Mon, 14 Jun 2010 18:01:41 GMT ETag: "2d4d6-1c7d167-4890147e4eb40" Accept-Ranges: bytes Content-Length: 29872487 Content-Type: text/plain; charset=UTF-8 with varnish in the middle: HTTP/1.1 200 OK Server: Apache Last-Modified: Mon, 14 Jun 2010 18:01:41 GMT ETag: "2d4d6-1c7d167-4890147e4eb40" Content-Type: text/plain; charset=UTF-8 Content-Length: 29872487 X-ece-cache: server-01-03.arteria.tsr Cache-Control: max-age=43200 X-ece-was-cached: media: 12h and max-age=43200, age 0 Date: Tue, 15 Jun 2010 14:26:15 GMT X-Varnish: 321078687 321076768 Via: 1.1 varnish Connection: keep-alive Age: 0 The Accept-Ranges field is lost, do you know why ? Jean-Francois Laurens pavionove at hotmail.com Jean-Fran?ois Laurens Ing?nieur Syst?me Unix RTS - radio t?l?vision suisse Quai Ernest-Ansermet 20 Case Postale 234 CH-1211 Gen?ve 8 T +41 (0)22 708 8163 -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin.boer at netclever.nl Thu Jun 17 13:54:47 2010 From: martin.boer at netclever.nl (Martin Boer) Date: Thu, 17 Jun 2010 15:54:47 +0200 Subject: using parallel varnishes Message-ID: <4C1A2927.1010104@netclever.nl> Hello all, I want to have 2 servers running varnish in parallel so that if one fails the other still contains all cacheable data and the backend servers won't be overloaded. Could someone check to see if I'm on the right track ? This is how I figure it should be working. I don't know how large 'weight' can be, but with varnish caching > 90% that impact would be affordable. Regards, Martin Boer director via_other_varnish random { .retries = 5; { .backend = other_server; .weight = 9; } # use the regular backends if the other varnish instance fails. { .backend = backend_1; .weight = 1; } { .backend = backend_2; .weight = 1; } { .backend = backend_3; .weight = 1; } } director via_backends random { { .backend = backend_1; .weight = 1; } { .backend = backend_2; .weight = 1; } { .backend = backend_3; .weight = 1; } } sub vcl_recv { if ( resp.http.X-through-varnish > 0 ) { # other varnish forwarded the request already # so forward to backends set req.backend = via_backends; remove resp.http.X-through-varnish; } else { # try the other varnish resp.http.X-through-varnish = 1; set req.backend = via_other_varnish; } .. From rtshilston at gmail.com Thu Jun 17 13:44:00 2010 From: rtshilston at gmail.com (Rob S) Date: Thu, 17 Jun 2010 14:44:00 +0100 Subject: a couple of questions about -spersistent In-Reply-To: <20100617141754.7c062ee9@lonquimay.wrk.lsn.camptocamp.com> References: <20100611145548.0a7fe866@lonquimay.wrk.lsn.camptocamp.com> <20100617141754.7c062ee9@lonquimay.wrk.lsn.camptocamp.com> Message-ID: <4C1A26A0.2090001@gmail.com> Marc Fournier wrote: > Hello, > > >> I've found a couple of threads on the list where people were trying to >> request objects from the cache after a restart, but none of them seem >> to have been successful, and the threads ended without any solution. >> >> My question is: what am I missing to be able to: >> >> - fetch an object from a backend to prime the cache >> - fetch this object a 2nd time to check I get a HIT >> - restart varnish (as in /etc/init.d/varnish restart) >> - fetch the object and get a HIT. >> >> Has someone actually managed to have data persist in the cache across >> restarts ? >> > > Hum, must I interpret the lack of answers as "no, no one never got > persistence working" ? And therefore, should I open a bug report ? > > Thanks for any hints :-) > > Marc > Marc, My understanding is that -spersistent enables the cache to remain full whilst a child varnish process restarts, but not for the parent process (that you're restarting via /etc/init.d) to restart. But, there's definitely a lack of documentation at http://varnish-cache.org/docs/reference/varnishd/#storage-types, which just describes the persistent storage option as "New, shiny, better.". Can someone who knows more about this feature add some comment here? Rob From phk at phk.freebsd.dk Thu Jun 17 18:22:11 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 17 Jun 2010 18:22:11 +0000 Subject: a couple of questions about -spersistent In-Reply-To: Your message of "Thu, 17 Jun 2010 14:44:00 +0100." <4C1A26A0.2090001@gmail.com> Message-ID: <2416.1276798931@critter.freebsd.dk> In message <4C1A26A0.2090001 at gmail.com>, Rob S writes: >Can someone who knows more about this feature add some comment here? Sorry about being somewhat offline-ish. -spersistent is still experimental. It should work and keep objects *in closed segments* across both child and parent process restarts. The first requirement for this to work, is that it actuall finds the same file on the disk again when you start. The second is that the segments must be closed, that means that either the segment is full or a orderly shutdown of the child process must happen. If you start varnish, do one requst, kill -9' it, it will not work, as the segment is not closed. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From kb+varnish at slide.com Thu Jun 17 18:50:14 2010 From: kb+varnish at slide.com (Ken Brownfield) Date: Thu, 17 Jun 2010 11:50:14 -0700 Subject: using parallel varnishes In-Reply-To: <4C1A2927.1010104@netclever.nl> References: <4C1A2927.1010104@netclever.nl> Message-ID: Seems like that will do the job. You might also want to look into the consistent hash of haproxy, which should provide cache "distribution" over an arbitrary pool. Doing it in varnish would get pretty complicated as you add more varnishes, and the infinite loop potential is a little unnerving (to me anyway :) We wanted redundant caches in a similar way (but for boxes with ~1T of cache) and set up a test config with haproxy that seems to work, but we haven't put real-world load on it yet. -- Ken On Jun 17, 2010, at 6:54 AM, Martin Boer wrote: > Hello all, > > I want to have 2 servers running varnish in parallel so that if one fails the other still contains all cacheable data and the backend servers won't be overloaded. > Could someone check to see if I'm on the right track ? > > This is how I figure it should be working. > I don't know how large 'weight' can be, but with varnish caching > 90% that impact would be affordable. > Regards, > Martin Boer > > > director via_other_varnish random { > .retries = 5; > { > .backend = other_server; > .weight = 9; > } > # use the regular backends if the other varnish instance fails. > { > .backend = backend_1; > .weight = 1; > } > { > .backend = backend_2; > .weight = 1; > } > { > .backend = backend_3; > .weight = 1; > } > } > > director via_backends random { > { > .backend = backend_1; > .weight = 1; > } > { > .backend = backend_2; > .weight = 1; > } > { > .backend = backend_3; > .weight = 1; > } > } > > > sub vcl_recv { > if ( resp.http.X-through-varnish > 0 ) { > # other varnish forwarded the request already > # so forward to backends > set req.backend = via_backends; > remove resp.http.X-through-varnish; > } else { > # try the other varnish > resp.http.X-through-varnish = 1; > set req.backend = via_other_varnish; > } > .. > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From kb+varnish at slide.com Thu Jun 17 18:56:46 2010 From: kb+varnish at slide.com (Ken Brownfield) Date: Thu, 17 Jun 2010 11:56:46 -0700 Subject: a couple of questions about -spersistent In-Reply-To: <2416.1276798931@critter.freebsd.dk> References: <2416.1276798931@critter.freebsd.dk> Message-ID: <1CA68FE7-95D5-4AE2-A66E-C107D7233B34@slide.com> Even with a graceful stop via the CLI, I've never had any data retained after a parent restart. And even after a day of operation (which should result in many closed segments, though I don't know what the "max segment size" is) and a graceful shut, nothing is preserved. Every restart of the parent shows: CHK(0x7f520fa2b080 SILO 0x7f51cf3e0000 SILO) = 3 Which means the silo is reinitialized from scratch. From my cursory scan of storage_persistent.c (long ago) I didn't see code to do a reload of the relevant data structures. Is it possible to create a unit test for this, where the child is shut down in a way that properly flushes non-full segments? I'd be willing to write one as a starting point. -- Ken On Jun 17, 2010, at 11:22 AM, Poul-Henning Kamp wrote: > In message <4C1A26A0.2090001 at gmail.com>, Rob S writes: > >> Can someone who knows more about this feature add some comment here? > > Sorry about being somewhat offline-ish. > > -spersistent is still experimental. > > It should work and keep objects *in closed segments* across both > child and parent process restarts. > > The first requirement for this to work, is that it actuall finds > the same file on the disk again when you start. > > The second is that the segments must be closed, that means that > either the segment is full or a orderly shutdown of the child > process must happen. > > If you start varnish, do one requst, kill -9' it, it will not > work, as the segment is not closed. > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From phk at phk.freebsd.dk Thu Jun 17 19:55:59 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 17 Jun 2010 19:55:59 +0000 Subject: a couple of questions about -spersistent In-Reply-To: Your message of "Thu, 17 Jun 2010 11:56:46 MST." <1CA68FE7-95D5-4AE2-A66E-C107D7233B34@slide.com> Message-ID: <16036.1276804559@critter.freebsd.dk> In message <1CA68FE7-95D5-4AE2-A66E-C107D7233B34 at slide.com>, Ken Brownfield wri tes: >CHK(0x7f520fa2b080 SILO 0x7f51cf3e0000 SILO) = 3 What does your -spersistent option look like ? >Is it possible to create a unit test for this, where the child is shut >down in a way that properly flushes non-full segments? I'd be willing >to write one as a starting point. We have that, check the varnishtest/tests/p*.vtc files. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From kb+varnish at slide.com Thu Jun 17 20:17:19 2010 From: kb+varnish at slide.com (Ken Brownfield) Date: Thu, 17 Jun 2010 13:17:19 -0700 Subject: a couple of questions about -spersistent In-Reply-To: <16036.1276804559@critter.freebsd.dk> References: <16036.1276804559@critter.freebsd.dk> Message-ID: <395D9311-EABB-4B19-9ED7-51DBBB15EBFC@slide.com> We use five 300GB SSD volumes: -h classic,524287 -p between_bytes_timeout 15s -p cache_vbe_conns on -p cli_buffer 32768 -p default_grace 0s -p default_ttl 0s -p err_ttl 0s -p first_byte_timeout 15s -p listen_depth 8192 -p lru_interval 3600 -p max_restarts 2 -p pipe_timeout 15s -p purge_dups on -p rush_exponent 32 -p sess_timeout 15s -p thread_pools 8 -p thread_pool_min 32 -p thread_pool_max 8192 -p thread_pool_stack 262144 -s persistent,/cache/sdb/varnish,240G -s persistent,/cache/sdc/varnish,240G -s persistent,/cache/sdd/varnish,240G -s persistent,/cache/sde/varnish,240G -s persistent,/cache/sdf/varnish,240G I'm /pretty/ sure I also tested with a single -s option, but I (embarrassingly) may not have. Yeah, those tests look solid. I'll run the tests when I'm back at my build machine. Thanks, -- Ken On Jun 17, 2010, at 12:55 PM, Poul-Henning Kamp wrote: > In message <1CA68FE7-95D5-4AE2-A66E-C107D7233B34 at slide.com>, Ken Brownfield wri > tes: > >> CHK(0x7f520fa2b080 SILO 0x7f51cf3e0000 SILO) = 3 > > What does your -spersistent option look like ? > >> Is it possible to create a unit test for this, where the child is shut >> down in a way that properly flushes non-full segments? I'd be willing >> to write one as a starting point. > > We have that, check the varnishtest/tests/p*.vtc files. > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. From dfaulkner at pobox.com Thu Jun 17 22:31:10 2010 From: dfaulkner at pobox.com (Don Faulkner) Date: Thu, 17 Jun 2010 17:31:10 -0500 Subject: using parallel varnishes In-Reply-To: References: <4C1A2927.1010104@netclever.nl> Message-ID: I would like to hear more about how you're combining varnish and haproxy, and what you're trying to achieve. I'm just getting started with varnish, but I've used haproxy before. I'm trying to construct a cluster of caching, load balancing, and ssl termination to sit in front of my web infrastructure. In thinking about this, I seem to be caught in an infinite loop. I've seen several threads suggesting that the "right" way to build the web pipeline is this: web server -> cache -> load balancer -> ssl endpoint -> (internet & clients) But, in this case, all I have the load balancer doing is balancing between the various caches. On the other hand, if I reverse this and put the cache in front, then I'm caching the output of the load balancers, and there's no load balancing for the caches. I obviously haven't thought this through enough. Could someone pry me out of my loop? -- Don Faulkner, KB5WPM All that is gold does not glitter. Not all those who wander are lost. On Jun 17, 2010, at 1:50 PM, Ken Brownfield wrote: > Seems like that will do the job. > > You might also want to look into the consistent hash of haproxy, which should provide cache "distribution" over an arbitrary pool. Doing it in varnish would get pretty complicated as you add more varnishes, and the infinite loop potential is a little unnerving (to me anyway :) > > We wanted redundant caches in a similar way (but for boxes with ~1T of cache) and set up a test config with haproxy that seems to work, but we haven't put real-world load on it yet. > -- > Ken > > On Jun 17, 2010, at 6:54 AM, Martin Boer wrote: > >> Hello all, >> >> I want to have 2 servers running varnish in parallel so that if one fails the other still contains all cacheable data and the backend servers won't be overloaded. >> Could someone check to see if I'm on the right track ? >> >> This is how I figure it should be working. >> I don't know how large 'weight' can be, but with varnish caching > 90% that impact would be affordable. >> Regards, >> Martin Boer >> >> >> director via_other_varnish random { >> .retries = 5; >> { >> .backend = other_server; >> .weight = 9; >> } >> # use the regular backends if the other varnish instance fails. >> { >> .backend = backend_1; >> .weight = 1; >> } >> { >> .backend = backend_2; >> .weight = 1; >> } >> { >> .backend = backend_3; >> .weight = 1; >> } >> } >> >> director via_backends random { >> { >> .backend = backend_1; >> .weight = 1; >> } >> { >> .backend = backend_2; >> .weight = 1; >> } >> { >> .backend = backend_3; >> .weight = 1; >> } >> } >> >> >> sub vcl_recv { >> if ( resp.http.X-through-varnish > 0 ) { >> # other varnish forwarded the request already >> # so forward to backends >> set req.backend = via_backends; >> remove resp.http.X-through-varnish; >> } else { >> # try the other varnish >> resp.http.X-through-varnish = 1; >> set req.backend = via_other_varnish; >> } >> .. >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From david.birdsong at gmail.com Thu Jun 17 22:43:17 2010 From: david.birdsong at gmail.com (David Birdsong) Date: Thu, 17 Jun 2010 15:43:17 -0700 Subject: using parallel varnishes In-Reply-To: References: <4C1A2927.1010104@netclever.nl> Message-ID: On Thu, Jun 17, 2010 at 3:31 PM, Don Faulkner wrote: > I would like to hear more about how you're combining varnish and haproxy, and what you're trying to achieve. > > I'm just getting started with varnish, but I've used haproxy before. > > I'm trying to construct a cluster of caching, load balancing, and ssl termination to sit in front of my web infrastructure. In thinking about this, I seem to be caught in an infinite loop. > > I've seen several threads suggesting that the "right" way to build the web pipeline is this: > > web server -> cache -> load balancer -> ssl endpoint -> (internet & clients) > > But, in this case, all I have the load balancer doing is balancing between the various caches. Is there a something that you don't like about this setup? > On the other hand, if I reverse this and put the cache in front, then I'm caching the output of the load balancers, and there's no load balancing for the caches. > > I obviously haven't thought this through enough. Could someone pry me out of my loop? > -- > Don Faulkner, KB5WPM > All that is gold does not glitter. Not all those who wander are lost. > > On Jun 17, 2010, at 1:50 PM, Ken Brownfield wrote: > >> Seems like that will do the job. >> >> You might also want to look into the consistent hash of haproxy, which should provide cache "distribution" over an arbitrary pool. ?Doing it in varnish would get pretty complicated as you add more varnishes, and the infinite loop potential is a little unnerving (to me anyway :) >> >> We wanted redundant caches in a similar way (but for boxes with ~1T of cache) and set up a test config with haproxy that seems to work, but we haven't put real-world load on it yet. >> -- >> Ken >> >> On Jun 17, 2010, at 6:54 AM, Martin Boer wrote: >> >>> Hello all, >>> >>> I want to have 2 servers running varnish in parallel so that if one fails the other still contains all cacheable data and the backend servers won't be overloaded. >>> Could someone check to see if I'm on the right track ? >>> >>> This is how I figure it should be working. >>> I don't know how large 'weight' can be, but with varnish caching > 90% that impact would be affordable. >>> Regards, >>> Martin Boer >>> >>> >>> director via_other_varnish random { >>> .retries = 5; >>> { >>> ? ?.backend = other_server; >>> ? ?.weight = 9; >>> } >>> # use the regular backends if the other varnish instance fails. >>> { >>> ? ?.backend = backend_1; >>> ? ?.weight = 1; >>> ?} >>> { >>> ? ?.backend = backend_2; >>> ? ?.weight = 1; >>> ?} >>> { >>> ? ?.backend = backend_3; >>> ? ?.weight = 1; >>> ?} >>> } >>> >>> director via_backends random { >>> ?{ >>> ? ?.backend = backend_1; >>> ? ?.weight = 1; >>> ?} >>> { >>> ? ?.backend = backend_2; >>> ? ?.weight = 1; >>> ?} >>> { >>> ? ?.backend = backend_3; >>> ? ?.weight = 1; >>> ?} >>> } >>> >>> >>> sub vcl_recv { >>> if ( resp.http.X-through-varnish > 0 ) { >>> ? ?# other varnish forwarded the request already >>> ? ?# so forward to backends >>> ? ?set req.backend = via_backends; >>> ? ?remove resp.http.X-through-varnish; >>> } else { >>> ? ?# try the other varnish >>> ? ?resp.http.X-through-varnish = 1; >>> ? ?set req.backend = via_other_varnish; >>> } >>> .. >>> >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From david.birdsong at gmail.com Fri Jun 18 05:09:41 2010 From: david.birdsong at gmail.com (David Birdsong) Date: Thu, 17 Jun 2010 22:09:41 -0700 Subject: using parallel varnishes In-Reply-To: <933261EA-1F7E-4337-B79A-26E437670A45@pobox.com> References: <4C1A2927.1010104@netclever.nl> <933261EA-1F7E-4337-B79A-26E437670A45@pobox.com> Message-ID: On Thu, Jun 17, 2010 at 10:02 PM, Don Faulkner wrote: > I like the setup. But for some reason I think it needs to be: > > web server -> load balancer -> cache -> load balancer -> ssl endpoint > ok, put haproxy behind the cache again, or nginx...or let varnish do the load balancing. it depends how your web servers need to be balanced. > Becuase the caches aren't load balancers and so aren't really balancing between the various servers that might serve a vhost. > > I guess my question is "which comes first, the load balancer or the cache?" and of course, why? if you have multiple cache's then definitely a load balancer. otherwise your cache/hit ratio is prone to be poor and out of your control. use a lb algorithm that hash's on some key. most hardware load balancers let you balance on uri. haproxy provides consistent hashing (search those two words if you're not familiar) which will protect key distribution as best as is possible between cache server failures. ...then load balance behind the cache again using whatever mechanism makes the most sense for your specific 'backends.' for me, that often means simply a forward proxy to *help* find the content somewhere using dns lookups etc. other times it means balancing on least connections. varnish offers some of these balancing mechanisms internally. > -- > Don Faulkner, KB5WPM > All that is gold does not glitter. Not all those who wander are lost. > > On Jun 17, 2010, at 5:43 PM, David Birdsong wrote: > >> On Thu, Jun 17, 2010 at 3:31 PM, Don Faulkner wrote: >>> I would like to hear more about how you're combining varnish and haproxy, and what you're trying to achieve. >>> >>> I'm just getting started with varnish, but I've used haproxy before. >>> >>> I'm trying to construct a cluster of caching, load balancing, and ssl termination to sit in front of my web infrastructure. In thinking about this, I seem to be caught in an infinite loop. >>> >>> I've seen several threads suggesting that the "right" way to build the web pipeline is this: >>> >>> web server -> cache -> load balancer -> ssl endpoint -> (internet & clients) >>> >>> But, in this case, all I have the load balancer doing is balancing between the various caches. >> Is there a something that you don't like about this setup? >> >>> On the other hand, if I reverse this and put the cache in front, then I'm caching the output of the load balancers, and there's no load balancing for the caches. >>> >>> I obviously haven't thought this through enough. Could someone pry me out of my loop? >>> -- >>> Don Faulkner, KB5WPM >>> All that is gold does not glitter. Not all those who wander are lost. >>> >>> On Jun 17, 2010, at 1:50 PM, Ken Brownfield wrote: >>> >>>> Seems like that will do the job. >>>> >>>> You might also want to look into the consistent hash of haproxy, which should provide cache "distribution" over an arbitrary pool. ?Doing it in varnish would get pretty complicated as you add more varnishes, and the infinite loop potential is a little unnerving (to me anyway :) >>>> >>>> We wanted redundant caches in a similar way (but for boxes with ~1T of cache) and set up a test config with haproxy that seems to work, but we haven't put real-world load on it yet. >>>> -- >>>> Ken >>>> >>>> On Jun 17, 2010, at 6:54 AM, Martin Boer wrote: >>>> >>>>> Hello all, >>>>> >>>>> I want to have 2 servers running varnish in parallel so that if one fails the other still contains all cacheable data and the backend servers won't be overloaded. >>>>> Could someone check to see if I'm on the right track ? >>>>> >>>>> This is how I figure it should be working. >>>>> I don't know how large 'weight' can be, but with varnish caching > 90% that impact would be affordable. >>>>> Regards, >>>>> Martin Boer >>>>> >>>>> >>>>> director via_other_varnish random { >>>>> .retries = 5; >>>>> { >>>>> ? ?.backend = other_server; >>>>> ? ?.weight = 9; >>>>> } >>>>> # use the regular backends if the other varnish instance fails. >>>>> { >>>>> ? ?.backend = backend_1; >>>>> ? ?.weight = 1; >>>>> ?} >>>>> { >>>>> ? ?.backend = backend_2; >>>>> ? ?.weight = 1; >>>>> ?} >>>>> { >>>>> ? ?.backend = backend_3; >>>>> ? ?.weight = 1; >>>>> ?} >>>>> } >>>>> >>>>> director via_backends random { >>>>> ?{ >>>>> ? ?.backend = backend_1; >>>>> ? ?.weight = 1; >>>>> ?} >>>>> { >>>>> ? ?.backend = backend_2; >>>>> ? ?.weight = 1; >>>>> ?} >>>>> { >>>>> ? ?.backend = backend_3; >>>>> ? ?.weight = 1; >>>>> ?} >>>>> } >>>>> >>>>> >>>>> sub vcl_recv { >>>>> if ( resp.http.X-through-varnish > 0 ) { >>>>> ? ?# other varnish forwarded the request already >>>>> ? ?# so forward to backends >>>>> ? ?set req.backend = via_backends; >>>>> ? ?remove resp.http.X-through-varnish; >>>>> } else { >>>>> ? ?# try the other varnish >>>>> ? ?resp.http.X-through-varnish = 1; >>>>> ? ?set req.backend = via_other_varnish; >>>>> } >>>>> .. >>>>> >>>>> >>>>> _______________________________________________ >>>>> varnish-misc mailing list >>>>> varnish-misc at varnish-cache.org >>>>> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >>>> >>>> >>>> _______________________________________________ >>>> varnish-misc mailing list >>>> varnish-misc at varnish-cache.org >>>> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >>> >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >>> > > From dfaulkner at pobox.com Fri Jun 18 05:02:21 2010 From: dfaulkner at pobox.com (Don Faulkner) Date: Fri, 18 Jun 2010 00:02:21 -0500 Subject: using parallel varnishes In-Reply-To: References: <4C1A2927.1010104@netclever.nl> Message-ID: <933261EA-1F7E-4337-B79A-26E437670A45@pobox.com> I like the setup. But for some reason I think it needs to be: web server -> load balancer -> cache -> load balancer -> ssl endpoint Becuase the caches aren't load balancers and so aren't really balancing between the various servers that might serve a vhost. I guess my question is "which comes first, the load balancer or the cache?" and of course, why? -- Don Faulkner, KB5WPM All that is gold does not glitter. Not all those who wander are lost. On Jun 17, 2010, at 5:43 PM, David Birdsong wrote: > On Thu, Jun 17, 2010 at 3:31 PM, Don Faulkner wrote: >> I would like to hear more about how you're combining varnish and haproxy, and what you're trying to achieve. >> >> I'm just getting started with varnish, but I've used haproxy before. >> >> I'm trying to construct a cluster of caching, load balancing, and ssl termination to sit in front of my web infrastructure. In thinking about this, I seem to be caught in an infinite loop. >> >> I've seen several threads suggesting that the "right" way to build the web pipeline is this: >> >> web server -> cache -> load balancer -> ssl endpoint -> (internet & clients) >> >> But, in this case, all I have the load balancer doing is balancing between the various caches. > Is there a something that you don't like about this setup? > >> On the other hand, if I reverse this and put the cache in front, then I'm caching the output of the load balancers, and there's no load balancing for the caches. >> >> I obviously haven't thought this through enough. Could someone pry me out of my loop? >> -- >> Don Faulkner, KB5WPM >> All that is gold does not glitter. Not all those who wander are lost. >> >> On Jun 17, 2010, at 1:50 PM, Ken Brownfield wrote: >> >>> Seems like that will do the job. >>> >>> You might also want to look into the consistent hash of haproxy, which should provide cache "distribution" over an arbitrary pool. Doing it in varnish would get pretty complicated as you add more varnishes, and the infinite loop potential is a little unnerving (to me anyway :) >>> >>> We wanted redundant caches in a similar way (but for boxes with ~1T of cache) and set up a test config with haproxy that seems to work, but we haven't put real-world load on it yet. >>> -- >>> Ken >>> >>> On Jun 17, 2010, at 6:54 AM, Martin Boer wrote: >>> >>>> Hello all, >>>> >>>> I want to have 2 servers running varnish in parallel so that if one fails the other still contains all cacheable data and the backend servers won't be overloaded. >>>> Could someone check to see if I'm on the right track ? >>>> >>>> This is how I figure it should be working. >>>> I don't know how large 'weight' can be, but with varnish caching > 90% that impact would be affordable. >>>> Regards, >>>> Martin Boer >>>> >>>> >>>> director via_other_varnish random { >>>> .retries = 5; >>>> { >>>> .backend = other_server; >>>> .weight = 9; >>>> } >>>> # use the regular backends if the other varnish instance fails. >>>> { >>>> .backend = backend_1; >>>> .weight = 1; >>>> } >>>> { >>>> .backend = backend_2; >>>> .weight = 1; >>>> } >>>> { >>>> .backend = backend_3; >>>> .weight = 1; >>>> } >>>> } >>>> >>>> director via_backends random { >>>> { >>>> .backend = backend_1; >>>> .weight = 1; >>>> } >>>> { >>>> .backend = backend_2; >>>> .weight = 1; >>>> } >>>> { >>>> .backend = backend_3; >>>> .weight = 1; >>>> } >>>> } >>>> >>>> >>>> sub vcl_recv { >>>> if ( resp.http.X-through-varnish > 0 ) { >>>> # other varnish forwarded the request already >>>> # so forward to backends >>>> set req.backend = via_backends; >>>> remove resp.http.X-through-varnish; >>>> } else { >>>> # try the other varnish >>>> resp.http.X-through-varnish = 1; >>>> set req.backend = via_other_varnish; >>>> } >>>> .. >>>> >>>> >>>> _______________________________________________ >>>> varnish-misc mailing list >>>> varnish-misc at varnish-cache.org >>>> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >>> >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> From perbu at varnish-software.com Fri Jun 18 08:22:01 2010 From: perbu at varnish-software.com (Per Buer) Date: Fri, 18 Jun 2010 10:22:01 +0200 Subject: using parallel varnishes In-Reply-To: <933261EA-1F7E-4337-B79A-26E437670A45@pobox.com> References: <4C1A2927.1010104@netclever.nl> <933261EA-1F7E-4337-B79A-26E437670A45@pobox.com> Message-ID: On Fri, Jun 18, 2010 at 7:02 AM, Don Faulkner wrote: > I like the setup. But for some reason I think it needs to be: > > web server -> load balancer -> cache -> load balancer -> ssl endpoint One thing to consider; almost any server that is still within warranty can deliver at least 1Gbps of traffic through Varnish, on new hardware reaching 10Gbps shouldn't be that big a deal (is there someone out there with 10Gbps hardware that would like to help us test? :-). So, you should ask yourself - do you really need a load balancer in front of Varnish? Having more Varnish server than you need will decrease your hit rate (unless you're hashing on the URL) and will increase your response time. It will also add to the complexity of the setup. Relying on a simple cluster of just two servere where just the IP address moves in case of failure will in a lot of scenarios lead to better performance and better uptime. -- Per Buer, Varnish Software Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer From a.hongens at netmatch.nl Fri Jun 18 09:32:52 2010 From: a.hongens at netmatch.nl (=?ISO-8859-1?Q?Angelo_H=F6ngens?=) Date: Fri, 18 Jun 2010 11:32:52 +0200 Subject: using parallel varnishes In-Reply-To: <4C1A2927.1010104@netclever.nl> References: <4C1A2927.1010104@netclever.nl> Message-ID: <4C1B3D44.7000800@netmatch.nl> Hoi Martin, See attached document, it's a diagram explaining how we do things. Perhaps it will give you some inspiration. On the balancers we have varnish running on port 80, and squid on port 443. They both do almost exactly the same thing. (Although Varnish is waaay faster, and Squid does SSL). -- With kind regards, Angelo H?ngens systems administrator MCSE on Windows 2003 MCSE on Windows 2000 MS Small Business Specialist ------------------------------------------ NetMatch tourism internet software solutions Ringbaan Oost 2b 5013 CA Tilburg +31 (0)13 5811088 +31 (0)13 5821239 A.Hongens at netmatch.nl www.netmatch.nl ------------------------------------------ -------------- next part -------------- A non-text attachment was scrubbed... Name: 2010_balancers_diagram.pdf Type: application/pdf Size: 192727 bytes Desc: not available URL: From martin.boer at netclever.nl Fri Jun 18 09:39:18 2010 From: martin.boer at netclever.nl (Martin Boer) Date: Fri, 18 Jun 2010 11:39:18 +0200 Subject: using parallel varnishes In-Reply-To: References: <4C1A2927.1010104@netclever.nl> <933261EA-1F7E-4337-B79A-26E437670A45@pobox.com> Message-ID: <4C1B3EC6.3010504@netclever.nl> Per Buer wrote: > On Fri, Jun 18, 2010 at 7:02 AM, Don Faulkner wrote: > >> I like the setup. But for some reason I think it needs to be: >> >> web server -> load balancer -> cache -> load balancer -> ssl endpoint >> > > > One thing to consider; almost any server that is still within warranty > can deliver at least 1Gbps of traffic through Varnish, on new hardware > reaching 10Gbps shouldn't be that big a deal (is there someone out > there with 10Gbps hardware that would like to help us test? :-). So, > you should ask yourself - do you really need a load balancer in front > of Varnish? Having more Varnish server than you need will decrease > your hit rate (unless you're hashing on the URL) and will increase > your response time. It will also add to the complexity of the setup. > Relying on a simple cluster of just two servere where just the IP > address moves in case of failure will in a lot of scenarios lead to > better performance and better uptime. > > The reason we want the double loadbalancer setup is that with 2 active varnishes one can fail without having 90+ % increase of hits to the backend servers for as long as it takes to get the backup-varnish loaded up. We don't have a lot of traffic but during peakhours having a slow website will scare away potential paying customers instantly. The setup we plan is simple enough protocol/application wise. The complexity is in the number of SSL entrances we need for all of our brands, not in the network setup. I can understand what it should do which automatically means it's quite simple. :) Regards, Martin From phk at phk.freebsd.dk Fri Jun 18 12:50:54 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 18 Jun 2010 12:50:54 +0000 Subject: Status, plans, bugs and vacations... Message-ID: <54353.1276865454@critter.freebsd.dk> So, 2.1.2 is out and it has some bugs, I already tore -trunk to pieces with 3.0 work and btw. everybody is on vacation on and off for the next 8 weeks... ouch. Here is the status: ------------------- The critbit/cpu-usage issue is diagnosed and likely fixed in -trunk, it needs testing and beating up to see that it is solid and actually does fix the problem. The ESI panic (#719) I just figured out, now I need to fix it. I fixed a panic issue with director destruction on vcl.discard yesterday, that needs to go in also. Any other stuff/fixes from -trunk that can, should be included. Unfortunately some of the work already done in -trunk makes it very hard to do plain merges of these changes to 2.1 branch, and to make life even more interesting, our SVN-meister, Tollef, is busy with having a life and whats not. Here is what we will do: ------------------------ I will try to create a patch file relative to 2.1.2 for people to test, it will undoubtedly be updated a couple of times along the way. I will announce it in an email when it is ready for testing. When we are happy with the patch, and Tollef pops back from his extra-curricular activities, we will do the necessary SVN magic to get those patches sensibly into the 2.1 branch, some by merges, some by direct commit. All that will, hopefully, happen over the next 6-8 weeks, resulting in a 2.1.3 release, but obviously, we have families and sunshine that needs our attention too. (Let me point out, that it is not in your best interest to pray for rain in Denmark: I sunburn easily and must stay inside in high sun. Rain would allow me to do other things than work on Varnish and read books[1] :-) Once the patch is out and tested/approved, I will dive back into 3.0 mode and hopefully we can do a 3.0.0 preview sort of release in the last months of 2010. And with that, I wish all of you on the top half a nice summer, and all on the bottom half a nice winter. Poul-Henning [1] See also: http://www.amazon.com/exec/obidos/wishlist/1XSZNDWXKIAS0 -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From sheldond at gmail.com Sun Jun 20 12:40:36 2010 From: sheldond at gmail.com (Sheldon Dsouza) Date: Sun, 20 Jun 2010 18:10:36 +0530 Subject: Varnish and Subdomains Message-ID: I have installed varnish on my test and production server. I have many subdomains that are used tastykhana.in - for users of our site. client.tastykhana.in - for our clients which requires authentication. admin.tastykhana.in - for our call center which requires authentication. After setting up varnish the user site tastykhana.in worked perfectly but the subdomains started redirecting to the parent domain tastykhana.in. How do i prevent the subdomains from caching? as they require authentication and i have also added the check for Cookies which passes the request directly to backend server apache. I have also attached a copy of my configuration. Regards, Sheldon -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: tastykhana.vcl Type: application/octet-stream Size: 7049 bytes Desc: not available URL: From marc.fournier at camptocamp.com Sun Jun 20 15:02:13 2010 From: marc.fournier at camptocamp.com (Marc Fournier) Date: Sun, 20 Jun 2010 17:02:13 +0200 Subject: Varnish and Subdomains References: Message-ID: <20100620170213.3eb2d785@lonquimay.wrk.lsn.camptocamp.com> Hello, > tastykhana.in - for users of our site. > client.tastykhana.in - for our clients which requires authentication. > admin.tastykhana.in - for our call center which requires > authentication. > > After setting up varnish the user site tastykhana.in worked perfectly > but the subdomains started redirecting to the parent domain > tastykhana.in. As your 3 websites seem to be hosted on the same backend, the only way for your webserver to differentiate them is the "Host" HTTP header. Try commenting out this line: set req.http.host = "tastykhana.in"; This will let the "Host" header from user's browser get passed to the backend without getting altered by varnish. > How do i prevent the subdomains from caching? as they require > authentication and i have also added the check for Cookies which > passes the request directly to backend server apache. The default behaviour is to not cache documents with HTTP authentication, so you shouldn't need to do anything. But if prefer, you can always do something like this: if (req.http.host == "client.tastykhana.in" || req.http.host == "admin.tastykhana.in") { return (pass); } I hope this will help ! Marc From harald at webmeisterei.com Mon Jun 21 08:21:04 2010 From: harald at webmeisterei.com (Harald Friessnegger) Date: Mon, 21 Jun 2010 10:21:04 +0200 Subject: munin plugin varnish_ - feature request: support multiple varnish instances References: <201006141453.14787.harald@webmeisterei.com> <201006141500.58996.harald@webmeisterei.com> <142b8b61fda41397a359f29913bbc1e0@doffen.hjemme.ruberg.no> Message-ID: hi Bj?rn thanks for you feedback. if i uderstand correctly multigraph plugins allow me to draw multiple graphs with a single script/symlink in /etc/munin/plugins. however we've still got a problem with backward compatibility since users of the varnish_ plugin need to change their symlinks: from varnish_expunge to a) varnish_project1_expunge (using the solution proposed initially in this thread) or b) varnish_project1 (after making varnish_ a multigraph plugin) i agree that changing symlinks "is no big deal" but i'm wondering what Kristian Lyngstol - the author of the script - thinks about this change. The varnish-tools repository is the canonical upstream for this plugin. Please send patches to Kristian Lyngstol and/or varnish-misc at projects.linpro.no for inclusion before adding to munin trunk. regards, harald Bj?rn Ruberg wrote: > While breaking old links if you're not renaming them manually (which is no > big deal really), the best way to solve the OP's challenge is rewriting > the plugin into a multigraph plugin (compatible with Munin from 1.4 and > above). > > That way the symlinks will only point to the instance (i.e. varnish_ -> > varnish_project1, varnish_ -> project2 etc), and all aspects will be > managed internally in the plugin. > > Bj?rn -- Webmeisterei GmbH - B?ro f?r Netzfragen Tel: +43 5572 908877, Fax: +43 5572 908877-66 Steinebach 18, A-6850 Dornbirn http://www.webmeisterei.com From sheldond at gmail.com Mon Jun 21 10:36:55 2010 From: sheldond at gmail.com (Sheldon Dsouza) Date: Mon, 21 Jun 2010 16:06:55 +0530 Subject: Varnish and Subdomains In-Reply-To: <20100620170213.3eb2d785@lonquimay.wrk.lsn.camptocamp.com> References: <20100620170213.3eb2d785@lonquimay.wrk.lsn.camptocamp.com> Message-ID: Hi Marc, Thanks for your reply, i have made the necessary changes in my configuration and things seem to be working fine now. Regards, Sheldon On Sun, Jun 20, 2010 at 8:32 PM, Marc Fournier wrote: > Hello, > > > tastykhana.in - for users of our site. > > client.tastykhana.in - for our clients which requires authentication. > > admin.tastykhana.in - for our call center which requires > > authentication. > > > > After setting up varnish the user site tastykhana.in worked perfectly > > but the subdomains started redirecting to the parent domain > > tastykhana.in. > > As your 3 websites seem to be hosted on the same backend, the only way > for your webserver to differentiate them is the "Host" HTTP header. Try > commenting out this line: > > set req.http.host = "tastykhana.in"; > > This will let the "Host" header from user's browser get passed to the > backend without getting altered by varnish. > > > How do i prevent the subdomains from caching? as they require > > authentication and i have also added the check for Cookies which > > passes the request directly to backend server apache. > > The default behaviour is to not cache documents with HTTP > authentication, so you shouldn't need to do anything. But if prefer, > you can always do something like this: > > if (req.http.host == "client.tastykhana.in" || > req.http.host == "admin.tastykhana.in") { > return (pass); > } > > I hope this will help ! > Marc > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex at acasa.ro Mon Jun 21 13:29:27 2010 From: alex at acasa.ro (Alex F) Date: Mon, 21 Jun 2010 16:29:27 +0300 Subject: purge files from cache Message-ID: <20100621132930.C5D162F7573@smtp1.acasa.ro> Hello, Prior to version 2.1.2 there used to be hash.purge to which I would pass the file and the url to be purged. Now I see that it has been removed. How can I achieve the same functionality? eg: hash.purge style.css example.com Thanks. From checker at d6.com Mon Jun 21 23:34:28 2010 From: checker at d6.com (Chris Hecker) Date: Mon, 21 Jun 2010 16:34:28 -0700 Subject: best configuration for varnish and multiple IPs and vhosts? Message-ID: <4C1FF704.1030004@d6.com> I searched around for an answer to this, but wasn't able to find anything comprehensive, so I'll ask here. I'm a total newbie at this stuff, so please let me know if I missed some obvious link. I have a single webserver with 5 IP addresses. Each IP has at least one apache-based site on it, and sometimes there are multiple on a single IP via NameVirtualHost. None of the sites share any content. What is the best varnish setup for this? It seems there are multiple options: 1. Have a single varnishd listen on all the IPs and run them all through a single apache port with name virtual hosts, so it's one big cache? 2. Multiple varnish daemons, one per IP, apache listening on multiple ports, one per IP for the back end? 3. Something else entirely? I don't have a ton of memory, if that matters. I need to use a cache mostly to handle spikes from being reddited and whatnot, the normal daily load is fine for the server. Thanks, Chris From scaunter at topscms.com Mon Jun 21 23:45:37 2010 From: scaunter at topscms.com (Caunter, Stefan) Date: Mon, 21 Jun 2010 19:45:37 -0400 Subject: best configuration for varnish and multiple IPs and vhosts? Message-ID: <064FF286FD17EC418BFB74629578BED1159E25AB@tmg-mail4.torstar.net> One varnish on one IP, that knows about all of the apache vhosts. You can run apache on 127.0.0.1 if you like. Varnish must answer externally for these websites. -------------------------- Sent using BlackBerry 416 561 4871 ----- Original Message ----- From: varnish-misc-bounces at varnish-cache.org To: varnish-misc at varnish-cache.org Sent: Mon Jun 21 19:34:28 2010 Subject: best configuration for varnish and multiple IPs and vhosts? I searched around for an answer to this, but wasn't able to find anything comprehensive, so I'll ask here. I'm a total newbie at this stuff, so please let me know if I missed some obvious link. I have a single webserver with 5 IP addresses. Each IP has at least one apache-based site on it, and sometimes there are multiple on a single IP via NameVirtualHost. None of the sites share any content. What is the best varnish setup for this? It seems there are multiple options: 1. Have a single varnishd listen on all the IPs and run them all through a single apache port with name virtual hosts, so it's one big cache? 2. Multiple varnish daemons, one per IP, apache listening on multiple ports, one per IP for the back end? 3. Something else entirely? I don't have a ton of memory, if that matters. I need to use a cache mostly to handle spikes from being reddited and whatnot, the normal daily load is fine for the server. Thanks, Chris _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org http://lists.varnish-cache.org/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From checker at d6.com Mon Jun 21 23:55:17 2010 From: checker at d6.com (Chris Hecker) Date: Mon, 21 Jun 2010 16:55:17 -0700 Subject: best configuration for varnish and multiple IPs and vhosts? In-Reply-To: <064FF286FD17EC418BFB74629578BED1159E25AB@tmg-mail4.torstar.net> References: <064FF286FD17EC418BFB74629578BED1159E25AB@tmg-mail4.torstar.net> Message-ID: <4C1FFBE5.6000504@d6.com> Hmm, by "one IP", do you mean I should change the DNS entries on the sites and have them all pointing at a single IP (in other words, if there was no varnish, have them all running as NameVirtualHosts)? If so, why not just have varnish listen on all the IPs instead of pushing it all through one IP? Thanks, Chris Caunter, Stefan wrote: > One varnish on one IP, that knows about all of the apache vhosts. You > can run apache on 127.0.0.1 if you like. Varnish must answer externally > for these websites. > > -------------------------- > Sent using BlackBerry > 416 561 4871 > > ----- Original Message ----- > From: varnish-misc-bounces at varnish-cache.org > > To: varnish-misc at varnish-cache.org > Sent: Mon Jun 21 19:34:28 2010 > Subject: best configuration for varnish and multiple IPs and vhosts? > > > I searched around for an answer to this, but wasn't able to find > anything comprehensive, so I'll ask here. I'm a total newbie at this > stuff, so please let me know if I missed some obvious link. > > I have a single webserver with 5 IP addresses. Each IP has at least one > apache-based site on it, and sometimes there are multiple on a single IP > via NameVirtualHost. None of the sites share any content. > > What is the best varnish setup for this? It seems there are multiple > options: > > 1. Have a single varnishd listen on all the IPs and run them all > through a single apache port with name virtual hosts, so it's one big cache? > > 2. Multiple varnish daemons, one per IP, apache listening on multiple > ports, one per IP for the back end? > > 3. Something else entirely? > > I don't have a ton of memory, if that matters. I need to use a cache > mostly to handle spikes from being reddited and whatnot, the normal > daily load is fine for the server. > > Thanks, > Chris > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From martin.boer at netclever.nl Tue Jun 22 06:36:26 2010 From: martin.boer at netclever.nl (Martin Boer) Date: Tue, 22 Jun 2010 08:36:26 +0200 Subject: best configuration for varnish and multiple IPs and vhosts? In-Reply-To: <064FF286FD17EC418BFB74629578BED1159E25AB@tmg-mail4.torstar.net> References: <064FF286FD17EC418BFB74629578BED1159E25AB@tmg-mail4.torstar.net> Message-ID: <4C2059EA.5010501@netclever.nl> We've been using option nr.1 for over half a year now. It works fine. The reason we do it this way is that for all of our HTTP sites we have a HTTPS one as well. Pound handles the HTTPS sites of things and Varnish does the HTTP part. Regards, Martin Caunter, Stefan wrote: > > One varnish on one IP, that knows about all of the apache vhosts. You > can run apache on 127.0.0.1 if you like. Varnish must answer > externally for these websites. > > -------------------------- > Sent using BlackBerry > 416 561 4871 > > ----- Original Message ----- > From: varnish-misc-bounces at varnish-cache.org > > To: varnish-misc at varnish-cache.org > Sent: Mon Jun 21 19:34:28 2010 > Subject: best configuration for varnish and multiple IPs and vhosts? > > > I searched around for an answer to this, but wasn't able to find > anything comprehensive, so I'll ask here. I'm a total newbie at this > stuff, so please let me know if I missed some obvious link. > > I have a single webserver with 5 IP addresses. Each IP has at least one > apache-based site on it, and sometimes there are multiple on a single IP > via NameVirtualHost. None of the sites share any content. > > What is the best varnish setup for this? It seems there are multiple > options: > > 1. Have a single varnishd listen on all the IPs and run them all > through a single apache port with name virtual hosts, so it's one big > cache? > > 2. Multiple varnish daemons, one per IP, apache listening on multiple > ports, one per IP for the back end? > > 3. Something else entirely? > > I don't have a ton of memory, if that matters. I need to use a cache > mostly to handle spikes from being reddited and whatnot, the normal > daily load is fine for the server. > > Thanks, > Chris > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > ------------------------------------------------------------------------ > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From checker at d6.com Tue Jun 22 06:45:30 2010 From: checker at d6.com (Chris Hecker) Date: Mon, 21 Jun 2010 23:45:30 -0700 Subject: best configuration for varnish and multiple IPs and vhosts? In-Reply-To: <4C2059EA.5010501@netclever.nl> References: <064FF286FD17EC418BFB74629578BED1159E25AB@tmg-mail4.torstar.net> <4C2059EA.5010501@netclever.nl> Message-ID: <4C205C0A.7080603@d6.com> Cool, thanks. Is there any issue with getting good logs per-site? Chris Martin Boer wrote: > We've been using option nr.1 for over half a year now. > It works fine. > > The reason we do it this way is that for all of our HTTP sites we have a > HTTPS one as well. > Pound handles the HTTPS sites of things and Varnish does the HTTP part. > > Regards, > Martin > > > > Caunter, Stefan wrote: >> >> One varnish on one IP, that knows about all of the apache vhosts. You >> can run apache on 127.0.0.1 if you like. Varnish must answer >> externally for these websites. >> >> -------------------------- >> Sent using BlackBerry >> 416 561 4871 >> >> ----- Original Message ----- >> From: varnish-misc-bounces at varnish-cache.org >> >> To: varnish-misc at varnish-cache.org >> Sent: Mon Jun 21 19:34:28 2010 >> Subject: best configuration for varnish and multiple IPs and vhosts? >> >> >> I searched around for an answer to this, but wasn't able to find >> anything comprehensive, so I'll ask here. I'm a total newbie at this >> stuff, so please let me know if I missed some obvious link. >> >> I have a single webserver with 5 IP addresses. Each IP has at least one >> apache-based site on it, and sometimes there are multiple on a single IP >> via NameVirtualHost. None of the sites share any content. >> >> What is the best varnish setup for this? It seems there are multiple >> options: >> >> 1. Have a single varnishd listen on all the IPs and run them all >> through a single apache port with name virtual hosts, so it's one big >> cache? >> >> 2. Multiple varnish daemons, one per IP, apache listening on multiple >> ports, one per IP for the back end? >> >> 3. Something else entirely? >> >> I don't have a ton of memory, if that matters. I need to use a cache >> mostly to handle spikes from being reddited and whatnot, the normal >> daily load is fine for the server. >> >> Thanks, >> Chris >> >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > From sheldond at gmail.com Tue Jun 22 09:21:01 2010 From: sheldond at gmail.com (Sheldon Dsouza) Date: Tue, 22 Jun 2010 14:51:01 +0530 Subject: varnishhist stats Message-ID: I have installed Varnish on my production server, the configuration is attached. I tried to get a few stats using the varnishhist command, according to the stats a pipe "|" indicates cache hits I dont see any pipes in my display, i have attached the out of the stats in the varnish.txt file. What am i doing wrong? Regards, Sheldon -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- 1:2, n = 313 Tastykhana-new # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # ## ## # ## # ## # ## # ## ## ## ## ##### ##### ##### ##### ## ####### ## ######## # | ## #### # ######## # # | ##### ###### ## # # ############# ## +------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------ |1e-6 |1e-5 |1e-4 |1e-3 |1e-2 |1e-1 |1e0 |1e1 |1e2 -------------- next part -------------- A non-text attachment was scrubbed... Name: tastykhana.vcl Type: application/octet-stream Size: 7059 bytes Desc: not available URL: From fh-varnish at fholzhauer.de Tue Jun 22 10:07:04 2010 From: fh-varnish at fholzhauer.de (Florian Holzhauer) Date: Tue, 22 Jun 2010 12:07:04 +0200 Subject: varnishhist stats In-Reply-To: References: Message-ID: <4C208B48.105@fholzhauer.de> Am 22.06.10 11:21, schrieb Sheldon Dsouza: > I have installed Varnish on my production server, the configuration is > attached. I tried to get a few stats using the varnishhist command, > according to the stats a pipe "|" indicates cache hits > > I dont see any pipes in my display, i have attached the out of the stats > in the varnish.txt file. What am i doing wrong? On a first brief look: If I do not recall it completely wrong, the error is the return(pass) as the default in vcl_fetch. "In this mode, the request is passed on to the backend, and the backend's response is passed on to the client, but is not entered into the cache. Subsequent requests submitted over the same client connection are handled normally." - so basically, you are bypassing the cache. The correct should be return (deliver); Cheers, Florian. From sheldond at gmail.com Tue Jun 22 10:18:31 2010 From: sheldond at gmail.com (Sheldon Dsouza) Date: Tue, 22 Jun 2010 15:48:31 +0530 Subject: varnishhist stats In-Reply-To: <4C208B48.105@fholzhauer.de> References: <4C208B48.105@fholzhauer.de> Message-ID: Hi Florian, Thanks for the reply, its works now. I have across another problem, our backend portal requires a login and password and tracks users sessions. How do you prevent these authenticated sessions from getting cached? Regards, Sheldon On Tue, Jun 22, 2010 at 3:37 PM, Florian Holzhauer wrote: > Am 22.06.10 11:21, schrieb Sheldon Dsouza: > > I have installed Varnish on my production server, the configuration is >> attached. I tried to get a few stats using the varnishhist command, >> according to the stats a pipe "|" indicates cache hits >> >> I dont see any pipes in my display, i have attached the out of the stats >> in the varnish.txt file. What am i doing wrong? >> > > On a first brief look: > If I do not recall it completely wrong, the error is the return(pass) as > the default in vcl_fetch. > > "In this mode, the request is passed on to the backend, and the backend's > response is passed on to the client, but is not entered into the cache. > Subsequent requests submitted over the same client connection are handled > normally." - so basically, you are bypassing the cache. > > The correct should be return (deliver); > > Cheers, > Florian. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jhayter at manta.com Tue Jun 22 14:29:00 2010 From: jhayter at manta.com (Jim Hayter) Date: Tue, 22 Jun 2010 10:29:00 -0400 Subject: best configuration for varnish and multiple IPs and vhosts? In-Reply-To: <4C1FF704.1030004@d6.com> References: <4C1FF704.1030004@d6.com> Message-ID: Similar to Martin, we've been using option 1 for about four months now. One thing I did differently, I put nginx in front of varnish to handle both http and https connections and proxy them back to varnish. Varnish sits in front of two sets of web servers and selects the backend director based on the hostname. HTH, Jim -----Original Message----- From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Chris Hecker Sent: Monday, June 21, 2010 7:34 PM To: varnish-misc at varnish-cache.org Subject: best configuration for varnish and multiple IPs and vhosts? I searched around for an answer to this, but wasn't able to find anything comprehensive, so I'll ask here. I'm a total newbie at this stuff, so please let me know if I missed some obvious link. I have a single webserver with 5 IP addresses. Each IP has at least one apache-based site on it, and sometimes there are multiple on a single IP via NameVirtualHost. None of the sites share any content. What is the best varnish setup for this? It seems there are multiple options: 1. Have a single varnishd listen on all the IPs and run them all through a single apache port with name virtual hosts, so it's one big cache? 2. Multiple varnish daemons, one per IP, apache listening on multiple ports, one per IP for the back end? 3. Something else entirely? I don't have a ton of memory, if that matters. I need to use a cache mostly to handle spikes from being reddited and whatnot, the normal daily load is fine for the server. Thanks, Chris _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From samcrawford at gmail.com Tue Jun 22 18:12:38 2010 From: samcrawford at gmail.com (Sam Crawford) Date: Tue, 22 Jun 2010 19:12:38 +0100 Subject: varnishhist stats In-Reply-To: References: <4C208B48.105@fholzhauer.de> Message-ID: By default Varnish will not cache requests with cookies set, or with authentication provided. Looking at your VCL config, it looks like you've already accounted for this. What makes you think that authentication sessions (i.e. cookies) are being cached? Thanks, Sam On 22 June 2010 11:18, Sheldon Dsouza wrote: > Hi Florian, > > Thanks for the reply, its works now. I have across another problem, our > backend portal requires a login and password and tracks users sessions. > > How do you prevent these authenticated sessions from getting cached? > > Regards, > Sheldon > > On Tue, Jun 22, 2010 at 3:37 PM, Florian Holzhauer > wrote: >> >> Am 22.06.10 11:21, schrieb Sheldon Dsouza: >>> >>> I have installed Varnish on my production server, the configuration is >>> attached. I tried to get a few stats using the varnishhist command, >>> according to the stats a pipe "|" indicates cache hits >>> >>> I dont see any pipes in my display, i have attached the out of the stats >>> in the varnish.txt file. What am i doing wrong? >> >> On a first brief look: >> If I do not recall it completely wrong, the error is the return(pass) as >> the default in vcl_fetch. >> >> "In this mode, the request is passed on to the backend, and the backend's >> response is passed on to the client, but is not entered into the cache. >> Subsequent requests submitted over the same client connection are handled >> normally." - so basically, you are bypassing the cache. >> >> The correct should be return (deliver); >> >> Cheers, >> Florian. > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From sheldond at gmail.com Wed Jun 23 09:48:46 2010 From: sheldond at gmail.com (Sheldon Dsouza) Date: Wed, 23 Jun 2010 15:18:46 +0530 Subject: varnishhist stats In-Reply-To: References: <4C208B48.105@fholzhauer.de> Message-ID: Actually i made a slight change in the configuration for the cookie check, it now checks the exact cookie name that is set from my application and things are working fine now. Regards, Sheldon On Tue, Jun 22, 2010 at 11:42 PM, Sam Crawford wrote: > By default Varnish will not cache requests with cookies set, or with > authentication provided. Looking at your VCL config, it looks like > you've already accounted for this. > > What makes you think that authentication sessions (i.e. cookies) are > being cached? > > Thanks, > > Sam > > > On 22 June 2010 11:18, Sheldon Dsouza wrote: > > Hi Florian, > > > > Thanks for the reply, its works now. I have across another problem, our > > backend portal requires a login and password and tracks users sessions. > > > > How do you prevent these authenticated sessions from getting cached? > > > > Regards, > > Sheldon > > > > On Tue, Jun 22, 2010 at 3:37 PM, Florian Holzhauer > > wrote: > >> > >> Am 22.06.10 11:21, schrieb Sheldon Dsouza: > >>> > >>> I have installed Varnish on my production server, the configuration is > >>> attached. I tried to get a few stats using the varnishhist command, > >>> according to the stats a pipe "|" indicates cache hits > >>> > >>> I dont see any pipes in my display, i have attached the out of the > stats > >>> in the varnish.txt file. What am i doing wrong? > >> > >> On a first brief look: > >> If I do not recall it completely wrong, the error is the return(pass) as > >> the default in vcl_fetch. > >> > >> "In this mode, the request is passed on to the backend, and the > backend's > >> response is passed on to the client, but is not entered into the cache. > >> Subsequent requests submitted over the same client connection are > handled > >> normally." - so basically, you are bypassing the cache. > >> > >> The correct should be return (deliver); > >> > >> Cheers, > >> Florian. > > > > > > _______________________________________________ > > varnish-misc mailing list > > varnish-misc at varnish-cache.org > > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex at acasa.ro Thu Jun 24 08:45:00 2010 From: alex at acasa.ro (Alex F) Date: Thu, 24 Jun 2010 11:45:00 +0300 Subject: POST x-forwarded-for Message-ID: <20100624084500.292262F7A61@smtp1.acasa.ro> Hello, I'm trying to make varnish forward the IP of the request to the backend. While I have succeeded in doing so with GET requests, I cannot do the same with POST requests. I have found the following link ( http://varnish-cache.org/ticket/203 ) which says: "post mode is trickier. Assuming the request is an HTTP request, it is of course possible to add / modify X-Forwarded-For before entering post mode, but it won't be possible to modify subsequent requests on the same connection. This can be mitigated by also adding Connection: close to the request." There is also a piece of VCL code: if (bereq.http.x-forwarded-for) { set bereq.http.X-forwarded-for = bereq.http.X-forwarded-for ", " regsub(client.ip, ":.*", ""); } else { set bereq.http.X-forwarded-for = regsub(client.ip, ":.*", ""); } But I cannot seem to convince varnish. Please advise. From bedis9 at gmail.com Thu Jun 24 11:24:26 2010 From: bedis9 at gmail.com (Bedis 9) Date: Thu, 24 Jun 2010 13:24:26 +0200 Subject: POST x-forwarded-for In-Reply-To: <20100624084500.292262F7A61@smtp1.acasa.ro> References: <20100624084500.292262F7A61@smtp1.acasa.ro> Message-ID: Hi, In which hook have you put your peace of code? cheers On Thu, Jun 24, 2010 at 10:45 AM, Alex F wrote: > Hello, > > I'm trying to make varnish forward the IP of the request to the backend. > While I have succeeded in doing so with GET requests, I cannot do the same > with POST requests. > I have found the following link ( http://varnish-cache.org/ticket/203 ) > which says: > > "post mode is trickier. Assuming the request is an HTTP request, it is of > course possible to add / modify X-Forwarded-For before entering post mode, > but it won't be possible to modify subsequent requests on the same > connection. This can be mitigated by also adding Connection: close to the > request." > > There is also a piece of VCL code: > if (bereq.http.x-forwarded-for) { > ? ? set bereq.http.X-forwarded-for = > ? ? bereq.http.X-forwarded-for ", " > ? ? regsub(client.ip, ":.*", ""); > ? ? } else { > ? ? set bereq.http.X-forwarded-for = > ? ? regsub(client.ip, ":.*", ""); > ?} > > But I cannot seem to convince varnish. > Please advise. > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From alex at acasa.ro Thu Jun 24 13:41:34 2010 From: alex at acasa.ro (Alex F) Date: Thu, 24 Jun 2010 16:41:34 +0300 Subject: POST x-forwarded-for In-Reply-To: References: <20100624084500.292262F7A61@smtp1.acasa.ro> Message-ID: <20100624134135.458E32F7608@smtp1.acasa.ro> I think I got it: sub vcl_pipe { set bereq.http.connection = "close"; if (bereq.http.x-forwarded-for) { set bereq.http.X-forwarded-for = bereq.http.X-forwarded-for ", "regsub(client.ip, ":.*", ""); } else { set bereq.http.X-forwarded-for = regsub(client.ip, ":.*", ""); } This should be all of it, no? On 24.6.2010 14:24, Bedis 9 wrote: > Hi, > > In which hook have you put your peace of code? > > cheers > > > From michaell at dazzit.com Thu Jun 24 15:34:07 2010 From: michaell at dazzit.com (Michael Lenaghan) Date: Thu, 24 Jun 2010 11:34:07 -0400 Subject: Discarding old VCL Message-ID: We're using Varnish 2.1 on Ubuntu 10.04. We load new VCL configs on a daily basis. After "loading" and "using" the new config we look to see if the ref counts on any old ("available") configs have dropped to zero. If they have, we discard the old configs. The problem is that the ref counts never seem to make it to zero: available 2 boot available 1 2010_06_22T05_32_01Z available 1 2010_06_23T14_08_01Z available 1 2010_06_23T14_19_01Z available 2 2010_06_24T11_00_16Z active 0 2010_06_24T11_03_01Z Any idea where that ref count is coming from? From perbu at varnish-software.com Thu Jun 24 16:08:03 2010 From: perbu at varnish-software.com (Per Buer) Date: Thu, 24 Jun 2010 18:08:03 +0200 Subject: Maximum Object Size to be Cached In-Reply-To: References: Message-ID: On Thu, Jun 24, 2010 at 5:46 PM, Anderson Brown wrote: > Thanks. ?Not sure which part of the varnishlogs you need: > (..) > > ? 13 RxRequest ? ?c GET > ? 13 RxURL ? ? ? ?c /somefile > ? 13 RxProtocol ? c HTTP/1.1 The client request. > (..) > ? 13 VCL_call ? ? c miss Which isn't in the cache. > (..) > ? 14 TxRequest ? ?b GET > ? 14 TxURL ? ? ? ?b /somefile > ? 14 TxProtocol ? b HTTP/1.1 Request it from the backend. > (..) > ? 13 ObjHeader ? ?c ETag: "2da8848-12cbec-48531772a5f80" > ? 13 ObjHeader ? ?c Content-Type: text/plain; charset=UTF-8 > ? 13 VCL_call ? ? c discard > ? 13 VCL_return ? c discard > ? ?0 ExpKill ? ? ?- 660189645 LRU > ? 13 VCL_call ? ? c discard > ? 13 VCL_return ? c discard > ? ?0 ExpKill ? ? ?- 660189646 LRU LRU kicks inn. You're running low on cache. Try adding more cache to Varnish. -- Per Buer, Varnish Software Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer From bnowacky at competitorgroup.com Thu Jun 24 17:28:48 2010 From: bnowacky at competitorgroup.com (Ben Nowacky) Date: Thu, 24 Jun 2010 10:28:48 -0700 Subject: Varnish restarting sporadically... losing entire cache... Message-ID: <5A399E8D-1C9B-40E7-8AC9-F21CC60473E2@competitorgroup.com> Hoping someone can help... We're just trying to roll out varnish to a somewhat large production site (30million monthly pageviews). Things seem to be working, hitting about a 90% hitrate so far. Using Wordpress for a CMS and standard LAMP stack on the back-end. - Deployed varnish on 1 of our servers yesterday for final testing on the live site. Runs along well, seems to be doing great. Run varnishstat to keep an eye on it, and randomly all the stats reset back to 0, and there's a full cache purge, which means i'm guessing varnish has restarted? I can't, for the life of me, figure out why it's been restarting. I've tweeked settings that I've seen recommendations for, upped cache/thread/etc an still seems to be happening. Not sure if anyone else has seen this behavior or not, but if anyone has any suggestions for troubleshooting, or figuring out what's going on, would appreciate any help ! I've got varnish running with persistence now, so it's helping a bit when there's a restart since it's not completely unprimed, but need to figure out the cause of the instability. Any help would be awesome... Thanks ! From fla_torres at yahoo.com.br Thu Jun 24 17:51:22 2010 From: fla_torres at yahoo.com.br (Flavio Torres) Date: Thu, 24 Jun 2010 14:51:22 -0300 Subject: Varnish restarting sporadically... losing entire cache... In-Reply-To: <5A399E8D-1C9B-40E7-8AC9-F21CC60473E2@competitorgroup.com> References: <5A399E8D-1C9B-40E7-8AC9-F21CC60473E2@competitorgroup.com> Message-ID: <4C239B1A.5030602@yahoo.com.br> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/24/2010 02:28 PM, Ben Nowacky wrote: > - Deployed varnish on 1 of our servers yesterday for final testing on the > live site. Runs along well, seems to be doing great. Run varnishstat to keep > an eye on it, and randomly all the stats reset back to 0, and there's a full > cache purge, which means i'm guessing varnish has restarted? Hi Ben, Check your /var/log/messages, I had this problem with a "cli time out" message. If you too, just increment your varnishd with -p cli_timeout 20 -p ping_interval 5 option. hope this helps -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkwjmxcACgkQNRQApncg295beACfVOKpxYcqDehDYe/7Mp6o28+F LHoAn0PuwzfCgbehD9siMQVYZsR5ww0F =Egpi -----END PGP SIGNATURE----- From bnowacky at competitorgroup.com Thu Jun 24 17:50:58 2010 From: bnowacky at competitorgroup.com (Ben Nowacky) Date: Thu, 24 Jun 2010 10:50:58 -0700 Subject: Varnish restarting sporadically... losing entire cache... In-Reply-To: <4C239B1A.5030602@yahoo.com.br> References: <5A399E8D-1C9B-40E7-8AC9-F21CC60473E2@competitorgroup.com> <4C239B1A.5030602@yahoo.com.br> Message-ID: <2F1860CE-56C7-405B-A3C8-55AC033AC145@competitorgroup.com> Thanks Flavio! Here's the errors that I see in the /var/log/messages... Is this what you were seeing? Jun 24 17:38:23 srv860 varnishd[15625]: Child (22165) Panic message: Assert error in SMP_FreeObj(), storage_persistent.c line 802: Condition(sg->nfixed > 0) not true. thread = (cache-timeout) ident = Linux,2.6.18-128.4.1.el5PAE,i686,-spersistent,-hclassic,epoll Backtrace: 0x806ca7c: pan_ic+cc 0x808851e: SMP_FreeObj+13e 0x8064b5f: HSH_Deref+21f 0x80618d1: exp_timer+321 0x806f1fd: wrk_bgthread+cd 0x44249b: /lib/libpthread.so.0 [0x44249b] 0x39942e: /lib/libc.so.6(clone+0x5e) [0x39942e] Jun 24 17:38:23 srv860 varnishd[15625]: child (22984) Started Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Child starts Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Dropped 0 segments to make free_reserve Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Silo completely loaded On Jun 24, 2010, at 10:51 AM, Flavio Torres wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 06/24/2010 02:28 PM, Ben Nowacky wrote: >> - Deployed varnish on 1 of our servers yesterday for final testing on the >> live site. Runs along well, seems to be doing great. Run varnishstat to > keep >> an eye on it, and randomly all the stats reset back to 0, and there's a > full >> cache purge, which means i'm guessing varnish has restarted? > > Hi Ben, > > Check your /var/log/messages, I had this problem with a "cli time out" > message. > > If you too, just increment your varnishd with -p cli_timeout 20 -p > ping_interval 5 option. > > hope this helps > > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.10 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAkwjmxcACgkQNRQApncg295beACfVOKpxYcqDehDYe/7Mp6o28+F > LHoAn0PuwzfCgbehD9siMQVYZsR5ww0F > =Egpi > -----END PGP SIGNATURE----- > > From scaunter at topscms.com Thu Jun 24 17:58:25 2010 From: scaunter at topscms.com (Caunter, Stefan) Date: Thu, 24 Jun 2010 13:58:25 -0400 Subject: Varnish restarting sporadically... losing entire cache... In-Reply-To: <2F1860CE-56C7-405B-A3C8-55AC033AC145@competitorgroup.com> References: <5A399E8D-1C9B-40E7-8AC9-F21CC60473E2@competitorgroup.com><4C239B1A.5030602@yahoo.com.br> <2F1860CE-56C7-405B-A3C8-55AC033AC145@competitorgroup.com> Message-ID: <064FF286FD17EC418BFB74629578BED11A28A74F@tmg-mail4.torstar.net> Check dmesg too, child is probably dying. Problem with persistent I found, I had to go back to file. Stefan Caunter :: Senior Systems Administrator :: TOPS e: scaunter at topscms.com :: m: (416) 561-4871 www.thestar.com www.topscms.com -----Original Message----- From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Ben Nowacky Sent: June-24-10 1:51 PM To: Flavio Torres Cc: varnish-misc at varnish-cache.org Subject: Re: Varnish restarting sporadically... losing entire cache... Thanks Flavio! Here's the errors that I see in the /var/log/messages... Is this what you were seeing? Jun 24 17:38:23 srv860 varnishd[15625]: Child (22165) Panic message: Assert error in SMP_FreeObj(), storage_persistent.c line 802: Condition(sg->nfixed > 0) not true. thread = (cache-timeout) ident = Linux,2.6.18-128.4.1.el5PAE,i686,-spersistent,-hclassic,epoll Backtrace: 0x806ca7c: pan_ic+cc 0x808851e: SMP_FreeObj+13e 0x8064b5f: HSH_Deref+21f 0x80618d1: exp_timer+321 0x806f1fd: wrk_bgthread+cd 0x44249b: /lib/libpthread.so.0 [0x44249b] 0x39942e: /lib/libc.so.6(clone+0x5e) [0x39942e] Jun 24 17:38:23 srv860 varnishd[15625]: child (22984) Started Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Child starts Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Dropped 0 segments to make free_reserve Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Silo completely loaded On Jun 24, 2010, at 10:51 AM, Flavio Torres wrote: From bnowacky at competitorgroup.com Thu Jun 24 18:13:38 2010 From: bnowacky at competitorgroup.com (Ben Nowacky) Date: Thu, 24 Jun 2010 11:13:38 -0700 Subject: Varnish restarting sporadically... losing entire cache... In-Reply-To: <064FF286FD17EC418BFB74629578BED11A28A74F@tmg-mail4.torstar.net> References: <5A399E8D-1C9B-40E7-8AC9-F21CC60473E2@competitorgroup.com><4C239B1A.5030602@yahoo.com.br> <2F1860CE-56C7-405B-A3C8-55AC033AC145@competitorgroup.com> <064FF286FD17EC418BFB74629578BED11A28A74F@tmg-mail4.torstar.net> Message-ID: <9461D638-AE40-4567-B8DB-862A68096CD0@competitorgroup.com> Looks like the child processes are dying regularly, causing the restart.. I increased the ping_interval as flavio recommended ,and the cli timeout. Took longer to die, (12 minutes this time), but finally pooped out.. This was in the /var/log/messages... Jun 24 18:13:23 srv860 varnishd[25646]: Child (26549) died signal=11 Jun 24 18:13:23 srv860 varnishd[25646]: child (27359) Started Jun 24 18:13:23 srv860 varnishd[25646]: Child (27359) said Jun 24 18:13:23 srv860 varnishd[25646]: Child (27359) said Child starts Any other suggestions? On Jun 24, 2010, at 10:58 AM, Caunter, Stefan wrote: > Check dmesg too, child is probably dying. Problem with persistent I > found, I had to go back to file. > > Stefan Caunter :: Senior Systems Administrator :: TOPS > e: scaunter at topscms.com :: m: (416) 561-4871 > www.thestar.com www.topscms.com > > > -----Original Message----- > From: varnish-misc-bounces at varnish-cache.org > [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Ben Nowacky > Sent: June-24-10 1:51 PM > To: Flavio Torres > Cc: varnish-misc at varnish-cache.org > Subject: Re: Varnish restarting sporadically... losing entire cache... > > Thanks Flavio! Here's the errors that I see in the /var/log/messages... > Is this what you were seeing? > > Jun 24 17:38:23 srv860 varnishd[15625]: Child (22165) Panic message: > Assert error in SMP_FreeObj(), storage_persistent.c line 802: > Condition(sg->nfixed > 0) not true. thread = (cache-timeout) ident = > Linux,2.6.18-128.4.1.el5PAE,i686,-spersistent,-hclassic,epoll Backtrace: > 0x806ca7c: pan_ic+cc 0x808851e: SMP_FreeObj+13e 0x8064b5f: > HSH_Deref+21f 0x80618d1: exp_timer+321 0x806f1fd: wrk_bgthread+cd > 0x44249b: /lib/libpthread.so.0 [0x44249b] 0x39942e: > /lib/libc.so.6(clone+0x5e) [0x39942e] > Jun 24 17:38:23 srv860 varnishd[15625]: child (22984) Started > Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said > Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Child starts > Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Dropped 0 > segments to make free_reserve > Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Silo > completely loaded > On Jun 24, 2010, at 10:51 AM, Flavio Torres wrote: > From perbu at varnish-software.com Thu Jun 24 21:52:12 2010 From: perbu at varnish-software.com (Per Buer) Date: Thu, 24 Jun 2010 23:52:12 +0200 Subject: Varnish restarting sporadically... losing entire cache... In-Reply-To: <2F1860CE-56C7-405B-A3C8-55AC033AC145@competitorgroup.com> References: <5A399E8D-1C9B-40E7-8AC9-F21CC60473E2@competitorgroup.com> <4C239B1A.5030602@yahoo.com.br> <2F1860CE-56C7-405B-A3C8-55AC033AC145@competitorgroup.com> Message-ID: Hi. Note that you are running on a 32 bit platform. You could easily be running out of virtual memory. We recommend you only run on 64 bit at least for production. I guess that if you switch platform the problem will automatically go away. Per. On Thu, Jun 24, 2010 at 7:50 PM, Ben Nowacky wrote: > Thanks Flavio! Here's the errors that I see in the /var/log/messages... Is this what you were seeing? > > Jun 24 17:38:23 srv860 varnishd[15625]: Child (22165) Panic message: Assert error in SMP_FreeObj(), storage_persistent.c line 802: ? Condition(sg->nfixed > 0) not true. thread = (cache-timeout) ident = Linux,2.6.18-128.4.1.el5PAE,i686,-spersistent,-hclassic,epoll Backtrace: ? 0x806ca7c: pan_ic+cc ? 0x808851e: SMP_FreeObj+13e ? 0x8064b5f: HSH_Deref+21f ? 0x80618d1: exp_timer+321 ? 0x806f1fd: wrk_bgthread+cd ? 0x44249b: /lib/libpthread.so.0 [0x44249b] ? 0x39942e: /lib/libc.so.6(clone+0x5e) [0x39942e] > Jun 24 17:38:23 srv860 varnishd[15625]: child (22984) Started > Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said > Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Child starts > Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Dropped 0 segments to make free_reserve > Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Silo completely loaded > On Jun 24, 2010, at 10:51 AM, Flavio Torres wrote: > >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> On 06/24/2010 02:28 PM, Ben Nowacky wrote: >>> - Deployed varnish on 1 of our servers yesterday for final testing on the >>> live site. Runs along well, seems to be doing great. Run varnishstat to >> keep >>> an eye on it, and randomly all the stats reset back to 0, and there's a >> full >>> cache purge, which means i'm guessing varnish has restarted? >> >> Hi Ben, >> >> Check your /var/log/messages, I had this problem with a "cli time out" >> message. >> >> If you too, just increment your varnishd with -p cli_timeout 20 -p >> ping_interval 5 option. >> >> hope this helps >> >> >> >> -----BEGIN PGP SIGNATURE----- >> Version: GnuPG v1.4.10 (GNU/Linux) >> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ >> >> iEYEARECAAYFAkwjmxcACgkQNRQApncg295beACfVOKpxYcqDehDYe/7Mp6o28+F >> LHoAn0PuwzfCgbehD9siMQVYZsR5ww0F >> =Egpi >> -----END PGP SIGNATURE----- >> >> > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > -- Per Buer, Varnish Software Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer From bnowacky at competitorgroup.com Thu Jun 24 23:36:23 2010 From: bnowacky at competitorgroup.com (Ben Nowacky) Date: Thu, 24 Jun 2010 16:36:23 -0700 Subject: Varnish restarting sporadically... losing entire cache... In-Reply-To: <064FF286FD17EC418BFB74629578BED11A28A74F@tmg-mail4.torstar.net> References: <5A399E8D-1C9B-40E7-8AC9-F21CC60473E2@competitorgroup.com><4C239B1A.5030602@yahoo.com.br> <2F1860CE-56C7-405B-A3C8-55AC033AC145@competitorgroup.com> <064FF286FD17EC418BFB74629578BED11A28A74F@tmg-mail4.torstar.net> Message-ID: <0AFDC162-D135-42A7-9D0F-3C7AADFF4E81@competitorgroup.com> Here's the error I get consistently: Jun 24 23:35:31 srv860 varnishd[20605]: Child (21427) died signal=11 Jun 24 23:35:31 srv860 varnishd[20605]: child (21660) Started Jun 24 23:35:31 srv860 varnishd[20605]: Child (21660) said Jun 24 23:35:31 srv860 varnishd[20605]: Child (21660) said Child starts Here's my config: "-f /usr/local/varnish-2.1.2/etc/default.vcl \ -s malloc,1G \ -p thread_pool_max=5000 \ -p thread_pools=4 \ -p thread_pool_min=200 \ -p thread_pool_add_delay=1ms \ -p cli_timeout=1000s \ -p ping_interval=1 \ -p cli_buffer=16384 \ -p session_linger=20ms \ -p lru_interval=360s \ -p listen_depth=8192 \ -h classic,500009 \ -T localhost:2000 " Am I doing anything in here atrocious that would be causing the random resets? I've tried file and malloc storage to no avail.. Neither one fixed the issue. I've tried adjusting sess_timeout, sess_workspace, etc... also nothing.. Changed the hash from classic to critbit also, with no success. Bashing head against the wall, if anyone has any advice could really use it ! ! On Jun 24, 2010, at 10:58 AM, Caunter, Stefan wrote: > Check dmesg too, child is probably dying. Problem with persistent I > found, I had to go back to file. > > Stefan Caunter :: Senior Systems Administrator :: TOPS > e: scaunter at topscms.com :: m: (416) 561-4871 > www.thestar.com www.topscms.com > > > -----Original Message----- > From: varnish-misc-bounces at varnish-cache.org > [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Ben Nowacky > Sent: June-24-10 1:51 PM > To: Flavio Torres > Cc: varnish-misc at varnish-cache.org > Subject: Re: Varnish restarting sporadically... losing entire cache... > > Thanks Flavio! Here's the errors that I see in the /var/log/messages... > Is this what you were seeing? > > Jun 24 17:38:23 srv860 varnishd[15625]: Child (22165) Panic message: > Assert error in SMP_FreeObj(), storage_persistent.c line 802: > Condition(sg->nfixed > 0) not true. thread = (cache-timeout) ident = > Linux,2.6.18-128.4.1.el5PAE,i686,-spersistent,-hclassic,epoll Backtrace: > 0x806ca7c: pan_ic+cc 0x808851e: SMP_FreeObj+13e 0x8064b5f: > HSH_Deref+21f 0x80618d1: exp_timer+321 0x806f1fd: wrk_bgthread+cd > 0x44249b: /lib/libpthread.so.0 [0x44249b] 0x39942e: > /lib/libc.so.6(clone+0x5e) [0x39942e] > Jun 24 17:38:23 srv860 varnishd[15625]: child (22984) Started > Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said > Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Child starts > Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Dropped 0 > segments to make free_reserve > Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Silo > completely loaded > On Jun 24, 2010, at 10:51 AM, Flavio Torres wrote: > From kristian at varnish-software.com Fri Jun 25 02:00:43 2010 From: kristian at varnish-software.com (=?UTF-8?Q?Kristian_Lyngst=C3=B8l?=) Date: Fri, 25 Jun 2010 04:00:43 +0200 Subject: Varnish restarting sporadically... losing entire cache... In-Reply-To: <0AFDC162-D135-42A7-9D0F-3C7AADFF4E81@competitorgroup.com> References: <5A399E8D-1C9B-40E7-8AC9-F21CC60473E2@competitorgroup.com> <4C239B1A.5030602@yahoo.com.br> <2F1860CE-56C7-405B-A3C8-55AC033AC145@competitorgroup.com> <064FF286FD17EC418BFB74629578BED11A28A74F@tmg-mail4.torstar.net> <0AFDC162-D135-42A7-9D0F-3C7AADFF4E81@competitorgroup.com> Message-ID: As Per says, it's likely you run out of vm space. You are also specifying a great deal of parameters which I suspect are not actually adjusted to your site. I would not recommend half of them unless you actually know why. It looks like your log entries are from /var/log/messages. You will likely find more in /var/log/syslog on Ubuntu. Also: 5000 threads is going to be far too many on a 32-bit system. Using 64-bit is by far the simplest way to avoid hassel. If you insist on 32-bit, you will need to reduce the maximum amount of threads, and possibly adjust the stack size, though newer varnish packages might try to do the latter. At any rate, closely monitor vm-usage. Also, signal 11 is a segfault. This means invalid or illegal memory access, which could match the symptoms of a 32-bit varnish-installation running out of virtual memory address space. - Kristian 2010/6/25, Ben Nowacky : > Here's the error I get consistently: > Jun 24 23:35:31 srv860 varnishd[20605]: Child (21427) died signal=11 > Jun 24 23:35:31 srv860 varnishd[20605]: child (21660) Started > Jun 24 23:35:31 srv860 varnishd[20605]: Child (21660) said > Jun 24 23:35:31 srv860 varnishd[20605]: Child (21660) said Child starts > > Here's my config: > "-f /usr/local/varnish-2.1.2/etc/default.vcl \ > -s malloc,1G \ > -p thread_pool_max=5000 \ > -p thread_pools=4 \ > -p thread_pool_min=200 \ > -p thread_pool_add_delay=1ms \ > -p cli_timeout=1000s \ > -p ping_interval=1 \ > -p cli_buffer=16384 \ > -p session_linger=20ms \ > -p lru_interval=360s \ > -p listen_depth=8192 \ > -h classic,500009 \ > -T localhost:2000 " > > Am I doing anything in here atrocious that would be causing the random > resets? I've tried file and malloc storage to no avail.. Neither one fixed > the issue. I've tried adjusting sess_timeout, sess_workspace, etc... also > nothing.. Changed the hash from classic to critbit also, with no success. > Bashing head against the wall, if anyone has any advice could really use it > ! ! > > > On Jun 24, 2010, at 10:58 AM, Caunter, Stefan wrote: > >> Check dmesg too, child is probably dying. Problem with persistent I >> found, I had to go back to file. >> >> Stefan Caunter :: Senior Systems Administrator :: TOPS >> e: scaunter at topscms.com :: m: (416) 561-4871 >> www.thestar.com www.topscms.com >> >> >> -----Original Message----- >> From: varnish-misc-bounces at varnish-cache.org >> [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Ben Nowacky >> Sent: June-24-10 1:51 PM >> To: Flavio Torres >> Cc: varnish-misc at varnish-cache.org >> Subject: Re: Varnish restarting sporadically... losing entire cache... >> >> Thanks Flavio! Here's the errors that I see in the /var/log/messages... >> Is this what you were seeing? >> >> Jun 24 17:38:23 srv860 varnishd[15625]: Child (22165) Panic message: >> Assert error in SMP_FreeObj(), storage_persistent.c line 802: >> Condition(sg->nfixed > 0) not true. thread = (cache-timeout) ident = >> Linux,2.6.18-128.4.1.el5PAE,i686,-spersistent,-hclassic,epoll Backtrace: >> 0x806ca7c: pan_ic+cc 0x808851e: SMP_FreeObj+13e 0x8064b5f: >> HSH_Deref+21f 0x80618d1: exp_timer+321 0x806f1fd: wrk_bgthread+cd >> 0x44249b: /lib/libpthread.so.0 [0x44249b] 0x39942e: >> /lib/libc.so.6(clone+0x5e) [0x39942e] >> Jun 24 17:38:23 srv860 varnishd[15625]: child (22984) Started >> Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said >> Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Child starts >> Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Dropped 0 >> segments to make free_reserve >> Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Silo >> completely loaded >> On Jun 24, 2010, at 10:51 AM, Flavio Torres wrote: >> > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From kristian at varnish-software.com Fri Jun 25 02:35:23 2010 From: kristian at varnish-software.com (=?UTF-8?Q?Kristian_Lyngst=C3=B8l?=) Date: Fri, 25 Jun 2010 04:35:23 +0200 Subject: Varnish restarting sporadically... losing entire cache... In-Reply-To: <88DF82A8-91C1-49B8-B9D2-F00C16E3F000@competitorgroup.com> References: <5A399E8D-1C9B-40E7-8AC9-F21CC60473E2@competitorgroup.com> <4C239B1A.5030602@yahoo.com.br> <2F1860CE-56C7-405B-A3C8-55AC033AC145@competitorgroup.com> <064FF286FD17EC418BFB74629578BED11A28A74F@tmg-mail4.torstar.net> <0AFDC162-D135-42A7-9D0F-3C7AADFF4E81@competitorgroup.com> <88DF82A8-91C1-49B8-B9D2-F00C16E3F000@competitorgroup.com> Message-ID: If it's not the vm you will have to turn on core dumps to figure it out. That involves setting ulimit -c unlimited in the startup script (or running it manually on the shell you start varnish from). You also likely want to set /proc/sys/vm/core_pattern to a path where you can both fit the core dump and actually find it. If you're unfamiliar with analyzing core dumps, you can gzip it and send it to me along with your varnish binaries, if you want to. As for logging, I suppose it might have changed in Ubuntu. I'll have to check that. You got the assert error though, so it's all there. Just out of curiosity though: why 32-bit? Is it by any chance a virtual machine, or similar? -Kristian PS: I'm not on a computer right now, so you will want to verify the ulimit argument-name and core_pattern path. 2010/6/25, Ben Nowacky : > Thanks Kristian! Been reading your blog, and got some of these from your > site... Guess I went overboard with some of them... > > - Ther is no /var/log/syslog so nothing else is being logged. This is the > only location i've been able to get any debug info out of varnish. We're not > tapping out VM or anything else it appears though.. Everything looks okay on > that front, but I'm going to lower the max threads and see how that takes > us.. maybe it'll be a simple solution. > > Appreciate the help! > On Jun 24, 2010, at 7:00 PM, Kristian Lyngst?l wrote: > >> As Per says, it's likely you run out of vm space. You are also >> specifying a great deal of parameters which I suspect are not actually >> adjusted to your site. I would not recommend half of them unless you >> actually know why. >> >> It looks like your log entries are from /var/log/messages. You will >> likely find more in /var/log/syslog on Ubuntu. >> >> Also: 5000 threads is going to be far too many on a 32-bit system. >> Using 64-bit is by far the simplest way to avoid hassel. If you insist >> on 32-bit, you will need to reduce the maximum amount of threads, and >> possibly adjust the stack size, though newer varnish packages might >> try to do the latter. At any rate, closely monitor vm-usage. >> >> Also, signal 11 is a segfault. This means invalid or illegal memory >> access, which could match the symptoms of a 32-bit >> varnish-installation running out of virtual memory address space. >> >> - Kristian >> >> 2010/6/25, Ben Nowacky : >>> Here's the error I get consistently: >>> Jun 24 23:35:31 srv860 varnishd[20605]: Child (21427) died signal=11 >>> Jun 24 23:35:31 srv860 varnishd[20605]: child (21660) Started >>> Jun 24 23:35:31 srv860 varnishd[20605]: Child (21660) said >>> Jun 24 23:35:31 srv860 varnishd[20605]: Child (21660) said Child starts >>> >>> Here's my config: >>> "-f /usr/local/varnish-2.1.2/etc/default.vcl \ >>> -s malloc,1G \ >>> -p thread_pool_max=5000 \ >>> -p thread_pools=4 \ >>> -p thread_pool_min=200 \ >>> -p thread_pool_add_delay=1ms \ >>> -p cli_timeout=1000s \ >>> -p ping_interval=1 \ >>> -p cli_buffer=16384 \ >>> -p session_linger=20ms \ >>> -p lru_interval=360s \ >>> -p listen_depth=8192 \ >>> -h classic,500009 \ >>> -T localhost:2000 " >>> >>> Am I doing anything in here atrocious that would be causing the random >>> resets? I've tried file and malloc storage to no avail.. Neither one >>> fixed >>> the issue. I've tried adjusting sess_timeout, sess_workspace, etc... also >>> nothing.. Changed the hash from classic to critbit also, with no >>> success. >>> Bashing head against the wall, if anyone has any advice could really use >>> it >>> ! ! >>> >>> >>> On Jun 24, 2010, at 10:58 AM, Caunter, Stefan wrote: >>> >>>> Check dmesg too, child is probably dying. Problem with persistent I >>>> found, I had to go back to file. >>>> >>>> Stefan Caunter :: Senior Systems Administrator :: TOPS >>>> e: scaunter at topscms.com :: m: (416) 561-4871 >>>> www.thestar.com www.topscms.com >>>> >>>> >>>> -----Original Message----- >>>> From: varnish-misc-bounces at varnish-cache.org >>>> [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Ben Nowacky >>>> Sent: June-24-10 1:51 PM >>>> To: Flavio Torres >>>> Cc: varnish-misc at varnish-cache.org >>>> Subject: Re: Varnish restarting sporadically... losing entire cache... >>>> >>>> Thanks Flavio! Here's the errors that I see in the /var/log/messages... >>>> Is this what you were seeing? >>>> >>>> Jun 24 17:38:23 srv860 varnishd[15625]: Child (22165) Panic message: >>>> Assert error in SMP_FreeObj(), storage_persistent.c line 802: >>>> Condition(sg->nfixed > 0) not true. thread = (cache-timeout) ident = >>>> Linux,2.6.18-128.4.1.el5PAE,i686,-spersistent,-hclassic,epoll Backtrace: >>>> 0x806ca7c: pan_ic+cc 0x808851e: SMP_FreeObj+13e 0x8064b5f: >>>> HSH_Deref+21f 0x80618d1: exp_timer+321 0x806f1fd: wrk_bgthread+cd >>>> 0x44249b: /lib/libpthread.so.0 [0x44249b] 0x39942e: >>>> /lib/libc.so.6(clone+0x5e) [0x39942e] >>>> Jun 24 17:38:23 srv860 varnishd[15625]: child (22984) Started >>>> Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said >>>> Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Child starts >>>> Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Dropped 0 >>>> segments to make free_reserve >>>> Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Silo >>>> completely loaded >>>> On Jun 24, 2010, at 10:51 AM, Flavio Torres wrote: >>>> >>> >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >>> > > From bnowacky at competitorgroup.com Fri Jun 25 02:39:22 2010 From: bnowacky at competitorgroup.com (Ben Nowacky) Date: Thu, 24 Jun 2010 19:39:22 -0700 Subject: Varnish restarting sporadically... losing entire cache... In-Reply-To: References: <5A399E8D-1C9B-40E7-8AC9-F21CC60473E2@competitorgroup.com> <4C239B1A.5030602@yahoo.com.br> <2F1860CE-56C7-405B-A3C8-55AC033AC145@competitorgroup.com> <064FF286FD17EC418BFB74629578BED11A28A74F@tmg-mail4.torstar.net> <0AFDC162-D135-42A7-9D0F-3C7AADFF4E81@competitorgroup.com> <88DF82A8-91C1-49B8-B9D2-F00C16E3F000@competitorgroup.com> Message-ID: <1A736DA4-F9BE-43A2-A030-0C7663F772B2@competitorgroup.com> NOpe, this is a dedicated server.. We're running CentOs... How do you know we're running 32-bit version? I had to compile from source on CentOS, so just grabbed the binaries from the site and did a build from them. How are you guessing it's 32-bit? Definitely not familiar with analyzing core-dumps or even getting them to run... I'm not a sys-admin, just the guy stuck trying to get our servers ready for an onslaught of traffic coming next week that I know we can not handle right now.... On Jun 24, 2010, at 7:35 PM, Kristian Lyngst?l wrote: > If it's not the vm you will have to turn on core dumps to figure it > out. That involves setting ulimit -c unlimited in the startup script > (or running it manually on the shell you start varnish from). You also > likely want to set /proc/sys/vm/core_pattern to a path where you can > both fit the core dump and actually find it. If you're unfamiliar with > analyzing core dumps, you can gzip it and send it to me along with > your varnish binaries, if you want to. > > As for logging, I suppose it might have changed in Ubuntu. I'll have > to check that. You got the assert error though, so it's all there. > > Just out of curiosity though: why 32-bit? Is it by any chance a > virtual machine, or similar? > > -Kristian > PS: I'm not on a computer right now, so you will want to verify the > ulimit argument-name and core_pattern path. > > 2010/6/25, Ben Nowacky : >> Thanks Kristian! Been reading your blog, and got some of these from your >> site... Guess I went overboard with some of them... >> >> - Ther is no /var/log/syslog so nothing else is being logged. This is the >> only location i've been able to get any debug info out of varnish. We're not >> tapping out VM or anything else it appears though.. Everything looks okay on >> that front, but I'm going to lower the max threads and see how that takes >> us.. maybe it'll be a simple solution. >> >> Appreciate the help! >> On Jun 24, 2010, at 7:00 PM, Kristian Lyngst?l wrote: >> >>> As Per says, it's likely you run out of vm space. You are also >>> specifying a great deal of parameters which I suspect are not actually >>> adjusted to your site. I would not recommend half of them unless you >>> actually know why. >>> >>> It looks like your log entries are from /var/log/messages. You will >>> likely find more in /var/log/syslog on Ubuntu. >>> >>> Also: 5000 threads is going to be far too many on a 32-bit system. >>> Using 64-bit is by far the simplest way to avoid hassel. If you insist >>> on 32-bit, you will need to reduce the maximum amount of threads, and >>> possibly adjust the stack size, though newer varnish packages might >>> try to do the latter. At any rate, closely monitor vm-usage. >>> >>> Also, signal 11 is a segfault. This means invalid or illegal memory >>> access, which could match the symptoms of a 32-bit >>> varnish-installation running out of virtual memory address space. >>> >>> - Kristian >>> >>> 2010/6/25, Ben Nowacky : >>>> Here's the error I get consistently: >>>> Jun 24 23:35:31 srv860 varnishd[20605]: Child (21427) died signal=11 >>>> Jun 24 23:35:31 srv860 varnishd[20605]: child (21660) Started >>>> Jun 24 23:35:31 srv860 varnishd[20605]: Child (21660) said >>>> Jun 24 23:35:31 srv860 varnishd[20605]: Child (21660) said Child starts >>>> >>>> Here's my config: >>>> "-f /usr/local/varnish-2.1.2/etc/default.vcl \ >>>> -s malloc,1G \ >>>> -p thread_pool_max=5000 \ >>>> -p thread_pools=4 \ >>>> -p thread_pool_min=200 \ >>>> -p thread_pool_add_delay=1ms \ >>>> -p cli_timeout=1000s \ >>>> -p ping_interval=1 \ >>>> -p cli_buffer=16384 \ >>>> -p session_linger=20ms \ >>>> -p lru_interval=360s \ >>>> -p listen_depth=8192 \ >>>> -h classic,500009 \ >>>> -T localhost:2000 " >>>> >>>> Am I doing anything in here atrocious that would be causing the random >>>> resets? I've tried file and malloc storage to no avail.. Neither one >>>> fixed >>>> the issue. I've tried adjusting sess_timeout, sess_workspace, etc... also >>>> nothing.. Changed the hash from classic to critbit also, with no >>>> success. >>>> Bashing head against the wall, if anyone has any advice could really use >>>> it >>>> ! ! >>>> >>>> >>>> On Jun 24, 2010, at 10:58 AM, Caunter, Stefan wrote: >>>> >>>>> Check dmesg too, child is probably dying. Problem with persistent I >>>>> found, I had to go back to file. >>>>> >>>>> Stefan Caunter :: Senior Systems Administrator :: TOPS >>>>> e: scaunter at topscms.com :: m: (416) 561-4871 >>>>> www.thestar.com www.topscms.com >>>>> >>>>> >>>>> -----Original Message----- >>>>> From: varnish-misc-bounces at varnish-cache.org >>>>> [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Ben Nowacky >>>>> Sent: June-24-10 1:51 PM >>>>> To: Flavio Torres >>>>> Cc: varnish-misc at varnish-cache.org >>>>> Subject: Re: Varnish restarting sporadically... losing entire cache... >>>>> >>>>> Thanks Flavio! Here's the errors that I see in the /var/log/messages... >>>>> Is this what you were seeing? >>>>> >>>>> Jun 24 17:38:23 srv860 varnishd[15625]: Child (22165) Panic message: >>>>> Assert error in SMP_FreeObj(), storage_persistent.c line 802: >>>>> Condition(sg->nfixed > 0) not true. thread = (cache-timeout) ident = >>>>> Linux,2.6.18-128.4.1.el5PAE,i686,-spersistent,-hclassic,epoll Backtrace: >>>>> 0x806ca7c: pan_ic+cc 0x808851e: SMP_FreeObj+13e 0x8064b5f: >>>>> HSH_Deref+21f 0x80618d1: exp_timer+321 0x806f1fd: wrk_bgthread+cd >>>>> 0x44249b: /lib/libpthread.so.0 [0x44249b] 0x39942e: >>>>> /lib/libc.so.6(clone+0x5e) [0x39942e] >>>>> Jun 24 17:38:23 srv860 varnishd[15625]: child (22984) Started >>>>> Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said >>>>> Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Child starts >>>>> Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Dropped 0 >>>>> segments to make free_reserve >>>>> Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Silo >>>>> completely loaded >>>>> On Jun 24, 2010, at 10:51 AM, Flavio Torres wrote: >>>>> >>>> >>>> >>>> _______________________________________________ >>>> varnish-misc mailing list >>>> varnish-misc at varnish-cache.org >>>> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >>>> >> >> From checker at d6.com Fri Jun 25 02:42:14 2010 From: checker at d6.com (Chris Hecker) Date: Thu, 24 Jun 2010 19:42:14 -0700 Subject: Varnish restarting sporadically... losing entire cache... In-Reply-To: <1A736DA4-F9BE-43A2-A030-0C7663F772B2@competitorgroup.com> References: <5A399E8D-1C9B-40E7-8AC9-F21CC60473E2@competitorgroup.com> <4C239B1A.5030602@yahoo.com.br> <2F1860CE-56C7-405B-A3C8-55AC033AC145@competitorgroup.com> <064FF286FD17EC418BFB74629578BED11A28A74F@tmg-mail4.torstar.net> <0AFDC162-D135-42A7-9D0F-3C7AADFF4E81@competitorgroup.com> <88DF82A8-91C1-49B8-B9D2-F00C16E3F000@competitorgroup.com> <1A736DA4-F9BE-43A2-A030-0C7663F772B2@competitorgroup.com> Message-ID: <4C241786.6010202@d6.com> uname -m i386 or i686 then it is 32 bit x86_64 is 64bit from http://www.linuxforums.org/forum/linux-newbie/72026-how-tell-if-32-64-bit-linux-machine-i-am-accessing.html I'm also on 32 bit centos, and hoping varnish will work. Chris Ben Nowacky wrote: > NOpe, this is a dedicated server.. We're running CentOs... How do you know we're running 32-bit version? I had to compile from source on CentOS, so just grabbed the binaries from the site and did a build from them. How are you guessing it's 32-bit? > > Definitely not familiar with analyzing core-dumps or even getting them to run... I'm not a sys-admin, just the guy stuck trying to get our servers ready for an onslaught of traffic coming next week that I know we can not handle right now.... > On Jun 24, 2010, at 7:35 PM, Kristian Lyngst?l wrote: > >> If it's not the vm you will have to turn on core dumps to figure it >> out. That involves setting ulimit -c unlimited in the startup script >> (or running it manually on the shell you start varnish from). You also >> likely want to set /proc/sys/vm/core_pattern to a path where you can >> both fit the core dump and actually find it. If you're unfamiliar with >> analyzing core dumps, you can gzip it and send it to me along with >> your varnish binaries, if you want to. >> >> As for logging, I suppose it might have changed in Ubuntu. I'll have >> to check that. You got the assert error though, so it's all there. >> >> Just out of curiosity though: why 32-bit? Is it by any chance a >> virtual machine, or similar? >> >> -Kristian >> PS: I'm not on a computer right now, so you will want to verify the >> ulimit argument-name and core_pattern path. >> >> 2010/6/25, Ben Nowacky : >>> Thanks Kristian! Been reading your blog, and got some of these from your >>> site... Guess I went overboard with some of them... >>> >>> - Ther is no /var/log/syslog so nothing else is being logged. This is the >>> only location i've been able to get any debug info out of varnish. We're not >>> tapping out VM or anything else it appears though.. Everything looks okay on >>> that front, but I'm going to lower the max threads and see how that takes >>> us.. maybe it'll be a simple solution. >>> >>> Appreciate the help! >>> On Jun 24, 2010, at 7:00 PM, Kristian Lyngst?l wrote: >>> >>>> As Per says, it's likely you run out of vm space. You are also >>>> specifying a great deal of parameters which I suspect are not actually >>>> adjusted to your site. I would not recommend half of them unless you >>>> actually know why. >>>> >>>> It looks like your log entries are from /var/log/messages. You will >>>> likely find more in /var/log/syslog on Ubuntu. >>>> >>>> Also: 5000 threads is going to be far too many on a 32-bit system. >>>> Using 64-bit is by far the simplest way to avoid hassel. If you insist >>>> on 32-bit, you will need to reduce the maximum amount of threads, and >>>> possibly adjust the stack size, though newer varnish packages might >>>> try to do the latter. At any rate, closely monitor vm-usage. >>>> >>>> Also, signal 11 is a segfault. This means invalid or illegal memory >>>> access, which could match the symptoms of a 32-bit >>>> varnish-installation running out of virtual memory address space. >>>> >>>> - Kristian >>>> >>>> 2010/6/25, Ben Nowacky : >>>>> Here's the error I get consistently: >>>>> Jun 24 23:35:31 srv860 varnishd[20605]: Child (21427) died signal=11 >>>>> Jun 24 23:35:31 srv860 varnishd[20605]: child (21660) Started >>>>> Jun 24 23:35:31 srv860 varnishd[20605]: Child (21660) said >>>>> Jun 24 23:35:31 srv860 varnishd[20605]: Child (21660) said Child starts >>>>> >>>>> Here's my config: >>>>> "-f /usr/local/varnish-2.1.2/etc/default.vcl \ >>>>> -s malloc,1G \ >>>>> -p thread_pool_max=5000 \ >>>>> -p thread_pools=4 \ >>>>> -p thread_pool_min=200 \ >>>>> -p thread_pool_add_delay=1ms \ >>>>> -p cli_timeout=1000s \ >>>>> -p ping_interval=1 \ >>>>> -p cli_buffer=16384 \ >>>>> -p session_linger=20ms \ >>>>> -p lru_interval=360s \ >>>>> -p listen_depth=8192 \ >>>>> -h classic,500009 \ >>>>> -T localhost:2000 " >>>>> >>>>> Am I doing anything in here atrocious that would be causing the random >>>>> resets? I've tried file and malloc storage to no avail.. Neither one >>>>> fixed >>>>> the issue. I've tried adjusting sess_timeout, sess_workspace, etc... also >>>>> nothing.. Changed the hash from classic to critbit also, with no >>>>> success. >>>>> Bashing head against the wall, if anyone has any advice could really use >>>>> it >>>>> ! ! >>>>> >>>>> >>>>> On Jun 24, 2010, at 10:58 AM, Caunter, Stefan wrote: >>>>> >>>>>> Check dmesg too, child is probably dying. Problem with persistent I >>>>>> found, I had to go back to file. >>>>>> >>>>>> Stefan Caunter :: Senior Systems Administrator :: TOPS >>>>>> e: scaunter at topscms.com :: m: (416) 561-4871 >>>>>> www.thestar.com www.topscms.com >>>>>> >>>>>> >>>>>> -----Original Message----- >>>>>> From: varnish-misc-bounces at varnish-cache.org >>>>>> [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Ben Nowacky >>>>>> Sent: June-24-10 1:51 PM >>>>>> To: Flavio Torres >>>>>> Cc: varnish-misc at varnish-cache.org >>>>>> Subject: Re: Varnish restarting sporadically... losing entire cache... >>>>>> >>>>>> Thanks Flavio! Here's the errors that I see in the /var/log/messages... >>>>>> Is this what you were seeing? >>>>>> >>>>>> Jun 24 17:38:23 srv860 varnishd[15625]: Child (22165) Panic message: >>>>>> Assert error in SMP_FreeObj(), storage_persistent.c line 802: >>>>>> Condition(sg->nfixed > 0) not true. thread = (cache-timeout) ident = >>>>>> Linux,2.6.18-128.4.1.el5PAE,i686,-spersistent,-hclassic,epoll Backtrace: >>>>>> 0x806ca7c: pan_ic+cc 0x808851e: SMP_FreeObj+13e 0x8064b5f: >>>>>> HSH_Deref+21f 0x80618d1: exp_timer+321 0x806f1fd: wrk_bgthread+cd >>>>>> 0x44249b: /lib/libpthread.so.0 [0x44249b] 0x39942e: >>>>>> /lib/libc.so.6(clone+0x5e) [0x39942e] >>>>>> Jun 24 17:38:23 srv860 varnishd[15625]: child (22984) Started >>>>>> Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said >>>>>> Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Child starts >>>>>> Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Dropped 0 >>>>>> segments to make free_reserve >>>>>> Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Silo >>>>>> completely loaded >>>>>> On Jun 24, 2010, at 10:51 AM, Flavio Torres wrote: >>>>>> >>>>> >>>>> _______________________________________________ >>>>> varnish-misc mailing list >>>>> varnish-misc at varnish-cache.org >>>>> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >>>>> >>> > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From kristian at varnish-software.com Fri Jun 25 02:51:27 2010 From: kristian at varnish-software.com (=?UTF-8?Q?Kristian_Lyngst=C3=B8l?=) Date: Fri, 25 Jun 2010 04:51:27 +0200 Subject: Varnish restarting sporadically... losing entire cache... In-Reply-To: <1A736DA4-F9BE-43A2-A030-0C7663F772B2@competitorgroup.com> References: <5A399E8D-1C9B-40E7-8AC9-F21CC60473E2@competitorgroup.com> <4C239B1A.5030602@yahoo.com.br> <2F1860CE-56C7-405B-A3C8-55AC033AC145@competitorgroup.com> <064FF286FD17EC418BFB74629578BED11A28A74F@tmg-mail4.torstar.net> <0AFDC162-D135-42A7-9D0F-3C7AADFF4E81@competitorgroup.com> <88DF82A8-91C1-49B8-B9D2-F00C16E3F000@competitorgroup.com> <1A736DA4-F9BE-43A2-A030-0C7663F772B2@competitorgroup.com> Message-ID: The log message you posted earlier with the assert error contains a identity string that - amoung other things - reads i686 (as opposed to amd64/x86_64 or similar). That's how I can tell. And this is exactly why it was added to begin with :) As for centos, that explains the logs. I must've mixed you up with someone else because I could've sworn you said you were on Ubuntu. Oh well. We've used centos as a test platform though, so nothing fundamentally wrong with it as far as Varnish is concerned. Let me know when/if you get a core dump. - Kristian 2010/6/25, Ben Nowacky : > NOpe, this is a dedicated server.. We're running CentOs... How do you know > we're running 32-bit version? I had to compile from source on CentOS, so > just grabbed the binaries from the site and did a build from them. How are > you guessing it's 32-bit? > > Definitely not familiar with analyzing core-dumps or even getting them to > run... I'm not a sys-admin, just the guy stuck trying to get our servers > ready for an onslaught of traffic coming next week that I know we can not > handle right now.... > On Jun 24, 2010, at 7:35 PM, Kristian Lyngst?l wrote: > >> If it's not the vm you will have to turn on core dumps to figure it >> out. That involves setting ulimit -c unlimited in the startup script >> (or running it manually on the shell you start varnish from). You also >> likely want to set /proc/sys/vm/core_pattern to a path where you can >> both fit the core dump and actually find it. If you're unfamiliar with >> analyzing core dumps, you can gzip it and send it to me along with >> your varnish binaries, if you want to. >> >> As for logging, I suppose it might have changed in Ubuntu. I'll have >> to check that. You got the assert error though, so it's all there. >> >> Just out of curiosity though: why 32-bit? Is it by any chance a >> virtual machine, or similar? >> >> -Kristian >> PS: I'm not on a computer right now, so you will want to verify the >> ulimit argument-name and core_pattern path. >> >> 2010/6/25, Ben Nowacky : >>> Thanks Kristian! Been reading your blog, and got some of these from your >>> site... Guess I went overboard with some of them... >>> >>> - Ther is no /var/log/syslog so nothing else is being logged. This is the >>> only location i've been able to get any debug info out of varnish. We're >>> not >>> tapping out VM or anything else it appears though.. Everything looks okay >>> on >>> that front, but I'm going to lower the max threads and see how that takes >>> us.. maybe it'll be a simple solution. >>> >>> Appreciate the help! >>> On Jun 24, 2010, at 7:00 PM, Kristian Lyngst?l wrote: >>> >>>> As Per says, it's likely you run out of vm space. You are also >>>> specifying a great deal of parameters which I suspect are not actually >>>> adjusted to your site. I would not recommend half of them unless you >>>> actually know why. >>>> >>>> It looks like your log entries are from /var/log/messages. You will >>>> likely find more in /var/log/syslog on Ubuntu. >>>> >>>> Also: 5000 threads is going to be far too many on a 32-bit system. >>>> Using 64-bit is by far the simplest way to avoid hassel. If you insist >>>> on 32-bit, you will need to reduce the maximum amount of threads, and >>>> possibly adjust the stack size, though newer varnish packages might >>>> try to do the latter. At any rate, closely monitor vm-usage. >>>> >>>> Also, signal 11 is a segfault. This means invalid or illegal memory >>>> access, which could match the symptoms of a 32-bit >>>> varnish-installation running out of virtual memory address space. >>>> >>>> - Kristian >>>> >>>> 2010/6/25, Ben Nowacky : >>>>> Here's the error I get consistently: >>>>> Jun 24 23:35:31 srv860 varnishd[20605]: Child (21427) died signal=11 >>>>> Jun 24 23:35:31 srv860 varnishd[20605]: child (21660) Started >>>>> Jun 24 23:35:31 srv860 varnishd[20605]: Child (21660) said >>>>> Jun 24 23:35:31 srv860 varnishd[20605]: Child (21660) said Child starts >>>>> >>>>> Here's my config: >>>>> "-f /usr/local/varnish-2.1.2/etc/default.vcl \ >>>>> -s malloc,1G \ >>>>> -p thread_pool_max=5000 \ >>>>> -p thread_pools=4 \ >>>>> -p thread_pool_min=200 \ >>>>> -p thread_pool_add_delay=1ms \ >>>>> -p cli_timeout=1000s \ >>>>> -p ping_interval=1 \ >>>>> -p cli_buffer=16384 \ >>>>> -p session_linger=20ms \ >>>>> -p lru_interval=360s \ >>>>> -p listen_depth=8192 \ >>>>> -h classic,500009 \ >>>>> -T localhost:2000 " >>>>> >>>>> Am I doing anything in here atrocious that would be causing the random >>>>> resets? I've tried file and malloc storage to no avail.. Neither one >>>>> fixed >>>>> the issue. I've tried adjusting sess_timeout, sess_workspace, etc... >>>>> also >>>>> nothing.. Changed the hash from classic to critbit also, with no >>>>> success. >>>>> Bashing head against the wall, if anyone has any advice could really >>>>> use >>>>> it >>>>> ! ! >>>>> >>>>> >>>>> On Jun 24, 2010, at 10:58 AM, Caunter, Stefan wrote: >>>>> >>>>>> Check dmesg too, child is probably dying. Problem with persistent I >>>>>> found, I had to go back to file. >>>>>> >>>>>> Stefan Caunter :: Senior Systems Administrator :: TOPS >>>>>> e: scaunter at topscms.com :: m: (416) 561-4871 >>>>>> www.thestar.com www.topscms.com >>>>>> >>>>>> >>>>>> -----Original Message----- >>>>>> From: varnish-misc-bounces at varnish-cache.org >>>>>> [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Ben >>>>>> Nowacky >>>>>> Sent: June-24-10 1:51 PM >>>>>> To: Flavio Torres >>>>>> Cc: varnish-misc at varnish-cache.org >>>>>> Subject: Re: Varnish restarting sporadically... losing entire cache... >>>>>> >>>>>> Thanks Flavio! Here's the errors that I see in the >>>>>> /var/log/messages... >>>>>> Is this what you were seeing? >>>>>> >>>>>> Jun 24 17:38:23 srv860 varnishd[15625]: Child (22165) Panic message: >>>>>> Assert error in SMP_FreeObj(), storage_persistent.c line 802: >>>>>> Condition(sg->nfixed > 0) not true. thread = (cache-timeout) ident = >>>>>> Linux,2.6.18-128.4.1.el5PAE,i686,-spersistent,-hclassic,epoll >>>>>> Backtrace: >>>>>> 0x806ca7c: pan_ic+cc 0x808851e: SMP_FreeObj+13e 0x8064b5f: >>>>>> HSH_Deref+21f 0x80618d1: exp_timer+321 0x806f1fd: wrk_bgthread+cd >>>>>> 0x44249b: /lib/libpthread.so.0 [0x44249b] 0x39942e: >>>>>> /lib/libc.so.6(clone+0x5e) [0x39942e] >>>>>> Jun 24 17:38:23 srv860 varnishd[15625]: child (22984) Started >>>>>> Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said >>>>>> Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Child >>>>>> starts >>>>>> Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Dropped 0 >>>>>> segments to make free_reserve >>>>>> Jun 24 17:38:23 srv860 varnishd[15625]: Child (22984) said Silo >>>>>> completely loaded >>>>>> On Jun 24, 2010, at 10:51 AM, Flavio Torres wrote: >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> varnish-misc mailing list >>>>> varnish-misc at varnish-cache.org >>>>> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >>>>> >>> >>> > > From michaell at dazzit.com Fri Jun 25 03:23:12 2010 From: michaell at dazzit.com (Michael Lenaghan) Date: Thu, 24 Jun 2010 23:23:12 -0400 Subject: Varnish restarting sporadically... losing entire cache... Message-ID: Hello, Ben. Sorry for coming late to the party; I'm on digest mode for the mailing list. Running Varnish 2.1 on a 32-bit Ubuntu 10.04 I had sporadic crashes, by which I mean the child process would die and get re-started. The crashes were easiest to reproduce under very heavy load--very many requests for small documents, etc. I was able to fix the problem by adding "session_linger=0" to my config. I spent time narrowing it to that, and then verifying that exactly that setting was what I needed. At least in my case. To the best of my knowledge Varnish hasn't crashed since. Of course, I don't know if my problem is or was the same as yours... From perbu at varnish-software.com Sat Jun 26 19:44:49 2010 From: perbu at varnish-software.com (Per Buer) Date: Sat, 26 Jun 2010 21:44:49 +0200 Subject: Varnish restarting sporadically... losing entire cache... In-Reply-To: <1A736DA4-F9BE-43A2-A030-0C7663F772B2@competitorgroup.com> References: <5A399E8D-1C9B-40E7-8AC9-F21CC60473E2@competitorgroup.com> <4C239B1A.5030602@yahoo.com.br> <2F1860CE-56C7-405B-A3C8-55AC033AC145@competitorgroup.com> <064FF286FD17EC418BFB74629578BED11A28A74F@tmg-mail4.torstar.net> <0AFDC162-D135-42A7-9D0F-3C7AADFF4E81@competitorgroup.com> <88DF82A8-91C1-49B8-B9D2-F00C16E3F000@competitorgroup.com> <1A736DA4-F9BE-43A2-A030-0C7663F772B2@competitorgroup.com> Message-ID: On Fri, Jun 25, 2010 at 4:39 AM, Ben Nowacky wrote: > NOpe, this is a dedicated server.. We're running CentOs... How do you know we're running 32-bit version? I had to compile from source on CentOS, so just grabbed the binaries from the site and did a build from them. ?How are you guessing it's 32-bit? > > Definitely not familiar with analyzing core-dumps or even getting them to run... I'm not a sys-admin, just the guy stuck trying to get our servers ready for an onslaught of traffic coming next week that I know we can not handle right now.... Then seriously, reinstall that server with a 64 bit OS. I haven't heard of anyone thats been running 32bit Varnish in production for the last two years and having it run stable. The CC-people did for some time but I believe they migrated due some issues. If you can't do that, reduce the number of threads to an absolute minimum, reduce the cache to max 1G, and reduce the stack size until it's stable. -- Per Buer, Varnish Software Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer From pprocacci at datapipe.com Sat Jun 26 20:29:05 2010 From: pprocacci at datapipe.com (Paul Procacci) Date: Sat, 26 Jun 2010 15:29:05 -0500 Subject: Varnish restarting sporadically... losing entire cache... In-Reply-To: References: <5A399E8D-1C9B-40E7-8AC9-F21CC60473E2@competitorgroup.com> <4C239B1A.5030602@yahoo.com.br> <2F1860CE-56C7-405B-A3C8-55AC033AC145@competitorgroup.com> <064FF286FD17EC418BFB74629578BED11A28A74F@tmg-mail4.torstar.net> <0AFDC162-D135-42A7-9D0F-3C7AADFF4E81@competitorgroup.com> <88DF82A8-91C1-49B8-B9D2-F00C16E3F000@competitorgroup.com> <1A736DA4-F9BE-43A2-A030-0C7663F772B2@competitorgroup.com> Message-ID: <4C266311.5040700@datapipe.com> On 6/26/2010 2:44 PM, Per Buer wrote: > On Fri, Jun 25, 2010 at 4:39 AM, Ben Nowacky > wrote: > >> NOpe, this is a dedicated server.. We're running CentOs... How do you know we're running 32-bit version? I had to compile from source on CentOS, so just grabbed the binaries from the site and did a build from them. How are you guessing it's 32-bit? >> >> Definitely not familiar with analyzing core-dumps or even getting them to run... I'm not a sys-admin, just the guy stuck trying to get our servers ready for an onslaught of traffic coming next week that I know we can not handle right now.... >> > Then seriously, reinstall that server with a 64 bit OS. I haven't > heard of anyone thats been running 32bit Varnish in production for the > last two years and having it run stable. The CC-people did for some > time but I believe they migrated due some issues. > > That's a pretty bold statement. Among the 1000's of machines that I administer daily, I have several running varnish on 32-bit archs. You may not of heard of this in the past two years, but now you can say, you've heard of _two_ people running varnish on 32-bit archs, even if I don't exclusively run it on 32-bit archs. ;P > If you can't do that, reduce the number of threads to an absolute > minimum, reduce the cache to max 1G, and reduce the stack size until > it's stable. > > > I concur. Cheers! This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/about-us-legal-email-disclaimer.htm for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. From phk at phk.freebsd.dk Mon Jun 28 07:24:26 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 28 Jun 2010 07:24:26 +0000 Subject: Discarding old VCL In-Reply-To: Your message of "Thu, 24 Jun 2010 11:34:07 -0400." Message-ID: <29113.1277709866@critter.freebsd.dk> In message , Mich ael Lenaghan writes: >available 2 boot >available 1 2010_06_22T05_32_01Z >available 1 2010_06_23T14_08_01Z >available 1 2010_06_23T14_19_01Z >available 2 2010_06_24T11_00_16Z >active 0 2010_06_24T11_03_01Z > >Any idea where that ref count is coming from? Some worker thread, which must be stuck somewhere. What is the oldest non-zero refcount you have seen ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From michaell at dazzit.com Mon Jun 28 11:42:22 2010 From: michaell at dazzit.com (Michael Lenaghan) Date: Mon, 28 Jun 2010 07:42:22 -0400 Subject: Discarding old VCL In-Reply-To: <29113.1277709866@critter.freebsd.dk> References: <29113.1277709866@critter.freebsd.dk> Message-ID: >>Any idea where that ref count is coming from? > > Some worker thread, which must be stuck somewhere. > > What is the oldest non-zero refcount you have seen ? Hello. It looks like the ref counts do clear eventually--but it can take several days. On 6/26, for example, there was a config from 6/22 that still had a ref count of 1. By 6/27 it dropped to zero. As of today, 6/28, the oldest config is from 6/25. So perhaps it's strange--or at the very least unexpected--but not really a problem. Michael. From phk at phk.freebsd.dk Mon Jun 28 12:22:23 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 28 Jun 2010 12:22:23 +0000 Subject: Discarding old VCL In-Reply-To: Your message of "Mon, 28 Jun 2010 07:42:22 -0400." Message-ID: <30376.1277727743@critter.freebsd.dk> In message , Mich ael Lenaghan writes: >>>Any idea where that ref count is coming from? >> >> Some worker thread, which must be stuck somewhere. >> >> What is the oldest non-zero refcount you have seen ? > >Hello. > >It looks like the ref counts do clear eventually--but it can take >several days. On 6/26, for example, there was a config from 6/22 that >still had a ref count of 1. By 6/27 it dropped to zero. As of today, >6/28, the oldest config is from 6/25. So perhaps it's strange--or at >the very least unexpected--but not really a problem. I don't think there is a problem per se. The idle worker threads hold a reference to the last VCL they used when they sleep, so until you have exercised all your threads during peak-load, you may have old VCL's linger around. Not optimal obviously, but you would have to load a lot of VCL's for it to become a problem. Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From michaell at dazzit.com Mon Jun 28 12:32:52 2010 From: michaell at dazzit.com (Michael Lenaghan) Date: Mon, 28 Jun 2010 08:32:52 -0400 Subject: Discarding old VCL In-Reply-To: <30376.1277727743@critter.freebsd.dk> References: <30376.1277727743@critter.freebsd.dk> Message-ID: > The idle worker threads hold a reference to the last VCL they used > when they sleep, so until you have exercised all your threads during > peak-load, you may have old VCL's linger around. > > Not optimal obviously, but you would have to load a lot of VCL's > for it to become a problem. Ah, thanks for the explanation. It helps to understand the model. And the model is certainly good enough for us; my only concern was a very slow resource leak. From bnowacky at competitorgroup.com Mon Jun 28 14:11:08 2010 From: bnowacky at competitorgroup.com (Ben Nowacky) Date: Mon, 28 Jun 2010 07:11:08 -0700 Subject: varnish-misc Digest, Vol 51, Issue 34 In-Reply-To: References: Message-ID: <06EF5F02-DE75-466C-B658-801D202E1CD0@competitorgroup.com> How do you change the stack size? I've tried to find the -p flag for startup, but can't find any documentation on it anywhere... Don't think upgrading is going to be possible for a little while, so have to work within the parameters of a 32 bit OS unfortunately. I've got storage down to 600 megs, and 210 threads. 1 box is running stable, but the other 4 are a bit flakey. How can I adjust the thread stack? On Jun 27, 2010, at 3:00 AM, > > wrote: On Fri, Jun 25, 2010 at 4:39 AM, Ben Nowacky > wrote: NOpe, this is a dedicated server.. We're running CentOs... How do you know we're running 32-bit version? I had to compile from source on CentOS, so just grabbed the binaries from the site and did a build from them. ?How are you guessing it's 32-bit? Definitely not familiar with analyzing core-dumps or even getting them to run... I'm not a sys-admin, just the guy stuck trying to get our servers ready for an onslaught of traffic coming next week that I know we can not handle right now.... Then seriously, reinstall that server with a 64 bit OS. I haven't heard of anyone thats been running 32bit Varnish in production for the last two years and having it run stable. The CC-people did for some time but I believe they migrated due some issues. If you can't do that, reduce the number of threads to an absolute minimum, reduce the cache to max 1G, and reduce the stack size until it's stable. -- Per Buer, Varnish Software Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer -------------- next part -------------- An HTML attachment was scrubbed... URL: From bnowacky at competitorgroup.com Mon Jun 28 14:34:44 2010 From: bnowacky at competitorgroup.com (Ben Nowacky) Date: Mon, 28 Jun 2010 07:34:44 -0700 Subject: varnish-misc Digest, Vol 51, Issue 34 In-Reply-To: References: Message-ID: <7A341253-4A1D-4468-90E5-AB0ABEB432AF@competitorgroup.com> Yes, I have tried Ben, thread__pool_stacksize with no luck. varnish will not start up with me with that flag in my startup string. > > Then seriously, reinstall that server with a 64 bit OS. I haven't > heard of anyone thats been running 32bit Varnish in production for the > last two years and having it run stable. The CC-people did for some > time but I believe they migrated due some issues. > > If you can't do that, reduce the number of threads to an absolute > minimum, reduce the cache to max 1G, and reduce the stack size until > it's stable. > > > -- > Per Buer, Varnish Software > Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer > > From checker at d6.com Tue Jun 29 04:42:24 2010 From: checker at d6.com (Chris Hecker) Date: Mon, 28 Jun 2010 21:42:24 -0700 Subject: varnish and wordpress Message-ID: <4C2979B0.1000708@d6.com> This page is rather old, is it still accurate? http://varnish-cache.org/wiki/VarnishAndWordpress Chris From mjdavies at glam.ac.uk Tue Jun 29 09:56:18 2010 From: mjdavies at glam.ac.uk (Davies Matt J A (LCSS)) Date: Tue, 29 Jun 2010 09:56:18 +0000 Subject: Using varnish as a cache for more than one website Message-ID: <282B7DC0-A852-4666-BE1F-B413BC6CF989@glam.ac.uk> Hello everyone What is the best practice way of setting up varnish to act as a cache for more than one web site? I can see I can make multiple backend, and I've read that I can run multiple instances of varnish, I'm just wondering what is the best practice way? One thing I can't quite get my head around is how do I tell varnish which incoming URL to serve with which backend, or instance. Scenario I've got our DNS boys to point a.com and b.com at our varnish server, which is going to act as a cache for both sites, both sites being very different and needing different rules. Rules that may collide also. If I use the multiple instance route, how do associate an instance with an incoming URL? Also, the same applies for multiple backends in the same instance. I'm brand new to Varnish so this is probably a remarkably stupid question, apologies for my greeness. Matt From martin.boer at netclever.nl Tue Jun 29 10:11:34 2010 From: martin.boer at netclever.nl (Martin Boer) Date: Tue, 29 Jun 2010 12:11:34 +0200 Subject: Using varnish as a cache for more than one website In-Reply-To: <282B7DC0-A852-4666-BE1F-B413BC6CF989@glam.ac.uk> References: <282B7DC0-A852-4666-BE1F-B413BC6CF989@glam.ac.uk> Message-ID: <4C29C6D6.1000702@netclever.nl> Matt, It isn't a remarkably stupid question but there are a remarkably lot of options and reasons why to choose one path over another. Do you have separate backends for a.com and b.com ? Is there no common content ? Are you planning to use 1 server for varnish ? How bad is it for business when it crashes ? Regards, Martin Davies Matt J A (LCSS) wrote: > Hello everyone > > What is the best practice way of setting up varnish to act as a cache for more than one web site? > > I can see I can make multiple backend, and I've read that I can run multiple instances of varnish, I'm just wondering what is the best practice way? > > One thing I can't quite get my head around is how do I tell varnish which incoming URL to serve with which backend, or instance. > > Scenario > > I've got our DNS boys to point a.com and b.com at our varnish server, which is going to act as a cache for both sites, both sites being very different and needing different rules. Rules that may collide also. > > If I use the multiple instance route, how do associate an instance with an incoming URL? > > Also, the same applies for multiple backends in the same instance. > > I'm brand new to Varnish so this is probably a remarkably stupid question, apologies for my greeness. > > Matt > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > > From marcussmith at britarch.ac.uk Tue Jun 29 10:23:22 2010 From: marcussmith at britarch.ac.uk (Marcus Smith) Date: Tue, 29 Jun 2010 11:23:22 +0100 Subject: Using varnish as a cache for more than one website In-Reply-To: <282B7DC0-A852-4666-BE1F-B413BC6CF989@glam.ac.uk> References: <282B7DC0-A852-4666-BE1F-B413BC6CF989@glam.ac.uk> Message-ID: <4C29C99A.4030605@britarch.ac.uk> Davies Matt J A (LCSS) wrote: > Hello everyone > > What is the best practice way of setting up varnish to act as a cache > for more than one web site? > > I can see I can make multiple backend, and I've read that I can run > multiple instances of varnish, I'm just wondering what is the best > practice way? > > One thing I can't quite get my head around is how do I tell varnish > which incoming URL to serve with which backend, or instance. > > Scenario > > I've got our DNS boys to point a.com and b.com at our varnish server, > which is going to act as a cache for both sites, both sites being > very different and needing different rules. Rules that may collide > also. > > If I use the multiple instance route, how do associate an instance > with an incoming URL? With a single instance of varnish, having defined your back ends, you might for example do something like this: sub vcl_recv { if (req.http.host == "a.com") { set req.backend = backend-A; # More site-specific rules here } elsif (req.http.host ~ "(www\.)?b\.com") { set req.backend = backend-B; # More site-specific rules here } # More sites here else { error 404 "Unknown host!"; } } best wishes, Marcus From mjdavies at glam.ac.uk Tue Jun 29 10:32:56 2010 From: mjdavies at glam.ac.uk (Davies Matt J A (LCSS)) Date: Tue, 29 Jun 2010 10:32:56 +0000 Subject: Using varnish as a cache for more than one website In-Reply-To: <4C29C99A.4030605@britarch.ac.uk> References: <282B7DC0-A852-4666-BE1F-B413BC6CF989@glam.ac.uk> <4C29C99A.4030605@britarch.ac.uk> Message-ID: <572848AD-597B-46DB-8C92-C08E5682A734@glam.ac.uk> Thanks Marcus I got there in the end with the documentation, wonderful stuff :-) backend a { .host = "machinename"; .port = "80"; } backend b { .host = "machinename"; .port = "8082"; } # #Below is a commented-out copy of the default VCL logic. If you #redefine any of these subroutines, the built-in logic will be #appended to your code. # sub vcl_recv { if (req.http.host == "a.com") { set req.backend = a; } if (req.http.host == "b.com") { set req.backend = b; } if (req.url ~ "^/") { unset req.http.cookie; } } sub vcl_fetch { if (req.url ~ "^/") { unset obj.http.set-cookie; set obj.ttl = 600s; } } BTW, is it good form to reply to the list or the person who replied? Told you I was green :-) Matt On 29 Jun 2010, at 11:23, Marcus Smith wrote: > Davies Matt J A (LCSS) wrote: >> Hello everyone >> >> What is the best practice way of setting up varnish to act as a cache >> for more than one web site? >> >> I can see I can make multiple backend, and I've read that I can run >> multiple instances of varnish, I'm just wondering what is the best >> practice way? >> >> One thing I can't quite get my head around is how do I tell varnish >> which incoming URL to serve with which backend, or instance. >> >> Scenario >> >> I've got our DNS boys to point a.com and b.com at our varnish server, >> which is going to act as a cache for both sites, both sites being >> very different and needing different rules. Rules that may collide >> also. >> >> If I use the multiple instance route, how do associate an instance >> with an incoming URL? > > With a single instance of varnish, having defined your back ends, you > might for example do something like this: > > > sub vcl_recv { > if (req.http.host == "a.com") { > set req.backend = backend-A; > # More site-specific rules here > } > > elsif (req.http.host ~ "(www\.)?b\.com") { > set req.backend = backend-B; > # More site-specific rules here > } > > # More sites here > > else { > error 404 "Unknown host!"; > } > } > > > best wishes, > Marcus > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From mjdavies at glam.ac.uk Tue Jun 29 10:49:50 2010 From: mjdavies at glam.ac.uk (Davies Matt J A (LCSS)) Date: Tue, 29 Jun 2010 10:49:50 +0000 Subject: Using varnish as a cache for more than one website In-Reply-To: <4C29C6D6.1000702@netclever.nl> References: <282B7DC0-A852-4666-BE1F-B413BC6CF989@glam.ac.uk> <4C29C6D6.1000702@netclever.nl> Message-ID: <5FB849E2-3F12-462D-B98D-478F6D45A41D@glam.ac.uk> Hi Martin > Do you have separate backends for a.com and b.com Yes mate > Is there no common content ? Yes, there is no common content > Are you planning to use 1 server for varnish ? Yes > How bad is it for business when it crashes ? If the varnish server crashes then it'll be super duper duper bad. It won't mean losing money though or jobs to be honest, we're a college. That is a good point though, does anyone use God or another monitoring process to make sure varnish does not go down? Or it will restart it if it does go down. I've posted up what I've got so far in reply to Marcus Both of you, thanks for you help, it's really appreciated. Currently, 1978 requests per second with varnish on a medium load, 25 without varnish, same load :-) Matt On 29 Jun 2010, at 11:11, Martin Boer wrote: > Matt, > > It isn't a remarkably stupid question but there are a remarkably lot of options and reasons why to choose one path over another. > > Do you have separate backends for a.com and b.com ? Is there no common content ? > Are you planning to use 1 server for varnish ? How bad is it for business when it crashes ? > > Regards, > Martin > > > Davies Matt J A (LCSS) wrote: >> Hello everyone >> >> What is the best practice way of setting up varnish to act as a cache for more than one web site? >> >> I can see I can make multiple backend, and I've read that I can run multiple instances of varnish, I'm just wondering what is the best practice way? >> >> One thing I can't quite get my head around is how do I tell varnish which incoming URL to serve with which backend, or instance. >> >> Scenario >> >> I've got our DNS boys to point a.com and b.com at our varnish server, which is going to act as a cache for both sites, both sites being very different and needing different rules. Rules that may collide also. >> >> If I use the multiple instance route, how do associate an instance with an incoming URL? >> >> Also, the same applies for multiple backends in the same instance. >> >> I'm brand new to Varnish so this is probably a remarkably stupid question, apologies for my greeness. >> >> Matt >> >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> >> >> > From perbu at varnish-software.com Tue Jun 29 11:04:18 2010 From: perbu at varnish-software.com (Per Buer) Date: Tue, 29 Jun 2010 13:04:18 +0200 Subject: Maximum Object Size to be Cached In-Reply-To: References: Message-ID: Right. You only sent the logs for the first request. It would be better if we saw the second one as well (the one that failed). Second, you should be aware that your backend sends no caching headers. So, unless you override TTL in VCL you'll be left with a mere 120 seconds caching by the default policy. Maybe that is what is bugging you? Per. On Mon, Jun 28, 2010 at 11:15 PM, Anderson Brown wrote: > Thanks. ?I deleted cache.db, hosts.db and restarted varnish. > Unfortunately, I still see it querying the origin server for those > files all the time. ?Also, I saw these from traffic_line -r after the > tests. ?I queried 1000 urls at least 2-3 times of size between > 1mb-2mb. > > Seems like it didn't use the cache at all after the curls: > proxy.process.cache.bytes_used ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?: 0 > proxy.process.cache.bytes_total ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? : 536308736 > > Thoughts? > > Thanks! > > Logs: > >> 13 StatSess ? ? c 10.121.42.43 44438 0 1 1 0 0 1 310 1095295 >> 13 SessionOpen ?c 10.121.42.43 44440 :6081 >> 13 ReqStart ? ? c 10.121.42.43 44440 660194525 >> 13 RxRequest ? ?c GET >> 13 RxURL ? ? ? ?c /somefile >> 13 RxProtocol ? c HTTP/1.1 >> 13 RxHeader ? ? c User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) >> libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5 >> 13 RxHeader ? ? c Host: jk.com:6081 >> 13 RxHeader ? ? c Accept: */* >> 13 VCL_call ? ? c recv >> 13 VCL_return ? c lookup >> 13 VCL_call ? ? c hash >> 13 VCL_return ? c hash >> 13 VCL_call ? ? c miss >> 13 VCL_return ? c fetch >> 14 BackendOpen ?b default 127.0.0.1 24044 127.0.0.1 8080 >> 13 Backend ? ? ?c 14 default default >> 14 TxRequest ? ?b GET >> 14 TxURL ? ? ? ?b /somefile >> 14 TxProtocol ? b HTTP/1.1 >> 14 TxHeader ? ? b User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) >> libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5 >> 14 TxHeader ? ? b Host: jk.com:6081 >> 14 TxHeader ? ? b Accept: */* >> 14 TxHeader ? ? b X-Varnish: 660194525 >> 14 TxHeader ? ? b X-Forwarded-For: 10.121.42.43 >> 14 RxProtocol ? b HTTP/1.1 >> 14 RxStatus ? ? b 200 >> 14 RxResponse ? b OK >> 14 RxHeader ? ? b Date: Fri, 25 Jun 2010 13:25:19 GMT >> 14 RxHeader ? ? b Server: Apache/2.2.3 (Red Hat) >> 14 RxHeader ? ? b Last-Modified: Tue, 27 Apr 2010 00:32:52 GMT >> 14 RxHeader ? ? b ETag: "1e9032c-1131fd-4852d08a69500" >> 14 RxHeader ? ? b Accept-Ranges: bytes >> 14 RxHeader ? ? b Content-Length: 1126909 >> 14 RxHeader ? ? b Connection: close >> 14 RxHeader ? ? b Content-Type: text/plain; charset=UTF-8 >> 13 ObjProtocol ?c HTTP/1.1 >> 13 ObjStatus ? ?c 200 >> 13 ObjResponse ?c OK >> 13 ObjHeader ? ?c Date: Fri, 25 Jun 2010 13:25:19 GMT >> 13 ObjHeader ? ?c Server: Apache/2.2.3 (Red Hat) >> 13 ObjHeader ? ?c Last-Modified: Tue, 27 Apr 2010 00:32:52 GMT >> 13 ObjHeader ? ?c ETag: "1e9032c-1131fd-4852d08a69500" >> 13 ObjHeader ? ?c Content-Type: text/plain; charset=UTF-8 >> 13 VCL_call ? ? c discard >> 13 VCL_return ? c discard >> ?0 ExpKill ? ? ?- 660193643 LRU >> 14 BackendClose b default >> 13 TTL ? ? ? ? ?c 660194525 RFC 120 1277472319 0 0 0 0 >> 13 VCL_call ? ? c fetch >> 13 VCL_return ? c deliver >> 13 Length ? ? ? c 1126909 >> 13 VCL_call ? ? c deliver >> 13 VCL_return ? c deliver >> 13 TxProtocol ? c HTTP/1.1 >> 13 TxStatus ? ? c 200 >> 13 TxResponse ? c OK >> 13 TxHeader ? ? c Server: Apache/2.2.3 (Red Hat) >> 13 TxHeader ? ? c Last-Modified: Tue, 27 Apr 2010 00:32:52 GMT >> 13 TxHeader ? ? c ETag: "1e9032c-1131fd-4852d08a69500" >> 13 TxHeader ? ? c Content-Type: text/plain; charset=UTF-8 >> 13 TxHeader ? ? c Content-Length: 1126909 >> 13 TxHeader ? ? c Date: Fri, 25 Jun 2010 13:25:19 GMT >> 13 TxHeader ? ? c X-Varnish: 660194525 >> 13 TxHeader ? ? c Age: 0 >> 13 TxHeader ? ? c Via: 1.1 varnish >> 13 TxHeader ? ? c Connection: keep-alive >> 13 ReqEnd ? ? ? c 660194525 1277472319.461021900 1277472319.464087963 >> 0.000044823 0.001679182 0.001386881 >> 13 SessionClose c EOF >> 13 ReqEnd ? ? ? c 0 1277472319.464224100 1277472319.464224100 >> 0.000136137 0.000000000 0.000000000 >> 13 StatSess ? ? c 10.121.42.43 44440 0 1 1 0 0 1 310 1126909 >> ?0 CLI ? ? ? ? ?- Rd ping >> ?0 CLI ? ? ? ? ?- Wr 0 200 PONG 1277472321 1.0 >> >> >> On Thu, Jun 24, 2010 at 12:08 PM, Per Buer wrote: >>> On Thu, Jun 24, 2010 at 5:46 PM, Anderson Brown >>> wrote: >>>> Thanks. ?Not sure which part of the varnishlogs you need: >>>> (..) >>>> >>>> ? 13 RxRequest ? ?c GET >>>> ? 13 RxURL ? ? ? ?c /somefile >>>> ? 13 RxProtocol ? c HTTP/1.1 >>> >>> The client request. >>> >>>> (..) >>>> ? 13 VCL_call ? ? c miss >>> >>> Which isn't in the cache. >>> >>>> (..) >>>> ? 14 TxRequest ? ?b GET >>>> ? 14 TxURL ? ? ? ?b /somefile >>>> ? 14 TxProtocol ? b HTTP/1.1 >>> >>> Request it from the backend. >>> >>>> (..) >>>> ? 13 ObjHeader ? ?c ETag: "2da8848-12cbec-48531772a5f80" >>>> ? 13 ObjHeader ? ?c Content-Type: text/plain; charset=UTF-8 >>>> ? 13 VCL_call ? ? c discard >>>> ? 13 VCL_return ? c discard >>>> ? ?0 ExpKill ? ? ?- 660189645 LRU >>>> ? 13 VCL_call ? ? c discard >>>> ? 13 VCL_return ? c discard >>>> ? ?0 ExpKill ? ? ?- 660189646 LRU >>> >>> LRU kicks inn. You're running low on cache. Try adding more cache to Varnish. >>> >>> >>> -- >>> Per Buer, ?Varnish Software >>> Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer >>> >> > -- Per Buer, Varnish Software Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer From fh-varnish at fholzhauer.de Tue Jun 29 11:15:13 2010 From: fh-varnish at fholzhauer.de (Florian Holzhauer) Date: Tue, 29 Jun 2010 13:15:13 +0200 Subject: Using varnish as a cache for more than one website In-Reply-To: <5FB849E2-3F12-462D-B98D-478F6D45A41D@glam.ac.uk> References: <282B7DC0-A852-4666-BE1F-B413BC6CF989@glam.ac.uk> <4C29C6D6.1000702@netclever.nl> <5FB849E2-3F12-462D-B98D-478F6D45A41D@glam.ac.uk> Message-ID: <4C29D5C1.9060405@fholzhauer.de> Dear Davies, Am 29.06.10 12:49, schrieb Davies Matt J A (LCSS): > That is a good point though, does anyone use God or another > monitoring process to make sure varnish does not go down? > Or it will restart it if it does go down. No, it does not restart automatically. I use Monit http://mmonit.com/monit/ for such a purpose, daemontools or God might fit here, too. This is my config: check process varnish with pidfile /var/run/varnishd.pid start program = "/etc/init.d/varnish start" stop program = "/etc/init.d/varnish stop" if loadavg(5min) greater than 10 for 8 cycles then stop if 3 restarts within 5 cycles then timeout Please note that the loadavg-statement might not be useful in all scenarios. :-) Cheers, Florian. From perbu at varnish-software.com Tue Jun 29 11:24:11 2010 From: perbu at varnish-software.com (Per Buer) Date: Tue, 29 Jun 2010 13:24:11 +0200 Subject: Using varnish as a cache for more than one website In-Reply-To: <5FB849E2-3F12-462D-B98D-478F6D45A41D@glam.ac.uk> References: <282B7DC0-A852-4666-BE1F-B413BC6CF989@glam.ac.uk> <4C29C6D6.1000702@netclever.nl> <5FB849E2-3F12-462D-B98D-478F6D45A41D@glam.ac.uk> Message-ID: On Tue, Jun 29, 2010 at 12:49 PM, Davies Matt J A (LCSS) wrote: > That is a good point though, does anyone use God or another monitoring process to make > sure varnish does not go down? ?Or it will restart it if it does go down. Varnish mgmt process will restart the cache if it crashes or stops responding. See http://varnish-cache.org/docs/phk/barriers/#the-really-important-barrier for some background. -- Per Buer, Varnish Software Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer From mjdavies at glam.ac.uk Tue Jun 29 11:32:14 2010 From: mjdavies at glam.ac.uk (Davies Matt J A (LCSS)) Date: Tue, 29 Jun 2010 11:32:14 +0000 Subject: Using varnish as a cache for more than one website In-Reply-To: <4C29D5C1.9060405@fholzhauer.de> References: <282B7DC0-A852-4666-BE1F-B413BC6CF989@glam.ac.uk> <4C29C6D6.1000702@netclever.nl> <5FB849E2-3F12-462D-B98D-478F6D45A41D@glam.ac.uk> <4C29D5C1.9060405@fholzhauer.de> Message-ID: <3CA4AE2B-C18B-4B4B-AC04-604404D0366A@glam.ac.uk> Thanks Florian I'll use God, I'm familiar with it so it'll be easier. Matt On 29 Jun 2010, at 12:15, Florian Holzhauer wrote: > Dear Davies, > > Am 29.06.10 12:49, schrieb Davies Matt J A (LCSS): > >> That is a good point though, does anyone use God or another > > monitoring process to make sure varnish does not go down? > > Or it will restart it if it does go down. > > No, it does not restart automatically. I use Monit > http://mmonit.com/monit/ for such a purpose, daemontools or God might fit here, too. This is my config: > > check process varnish with pidfile /var/run/varnishd.pid > start program = "/etc/init.d/varnish start" > stop program = "/etc/init.d/varnish stop" > if loadavg(5min) greater than 10 for 8 cycles then stop > if 3 restarts within 5 cycles then timeout > > Please note that the loadavg-statement might not be useful in all scenarios. :-) > > Cheers, > Florian. > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From felix at seconddrawer.com.au Tue Jun 29 12:02:22 2010 From: felix at seconddrawer.com.au (felix) Date: Tue, 29 Jun 2010 22:02:22 +1000 Subject: Using varnish as a cache for more than one website In-Reply-To: <5FB849E2-3F12-462D-B98D-478F6D45A41D@glam.ac.uk> References: <282B7DC0-A852-4666-BE1F-B413BC6CF989@glam.ac.uk> <4C29C6D6.1000702@netclever.nl> <5FB849E2-3F12-462D-B98D-478F6D45A41D@glam.ac.uk> Message-ID: <20100629120222.GA29975@thinkpad> Regarding a supervisor process, I have used varnish with daemontools for some time now. I used to manage a couple of internal sites that got similar traffic to yours (the varnish instance did, not the individual sites) on daemontools and it saved itself many times. I recommend it. -felix On Tue, Jun 29, 2010 at 10:49:50AM +0000, Davies Matt J A (LCSS) wrote: > > That is a good point though, does anyone use God or another monitoring process to make sure varnish does not go down? Or it will restart it if it does go down. > -- email: felix at seconddrawer.com.au web: http://seconddrawer.com.au/ gpg: E6FC 5BC6 268D B874 E546 8F6F A2BB 220B D5F6 92E3 Please don't send me Word or PowerPoint attachments. See http://www.gnu.org/philosophy/no-word-attachments.html -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From hitendra at itp.com Tue Jun 29 12:54:45 2010 From: hitendra at itp.com (Hitendra Molleti) Date: Tue, 29 Jun 2010 16:54:45 +0400 Subject: Varnish: Initial lag before dumping output to the browser Message-ID: <00c401cb178a$402bf760$c083e620$@com> Hi, We are using a CMS called Escenic which uses the below technologies. 1) Tomcat (Web Server) 2) J2EE & JSP (Coding platform) 3) Varnish (Caching server) 4) Mysql (Database) We have setup Varnish to cache the pages, a page which is put together using various widgets, seems to load all at once instead of gradual loading. Hence, there is an initial 3 to 4sec wait where the browser does not get any output, after that the whole rendered page is dumped on the browser. This makes the site seem much slower than it actually is. Can someone please let us know if you have experienced this issue and what are other ways to speed up Varnish to deliver pages faster? Thanks Hitendra -------------- next part -------------- An HTML attachment was scrubbed... URL: From bedis9 at gmail.com Tue Jun 29 20:58:28 2010 From: bedis9 at gmail.com (Bedis 9) Date: Tue, 29 Jun 2010 22:58:28 +0200 Subject: Varnish: Initial lag before dumping output to the browser In-Reply-To: <00c401cb178a$402bf760$c083e620$@com> References: <00c401cb178a$402bf760$c083e620$@com> Message-ID: Hi, If you run a tcpdump on your client, do you see any packet exchanged or the traffic is on hold for the 3-4s? Do you have this behavior when browsing the backend directly? cheers On Tue, Jun 29, 2010 at 2:54 PM, Hitendra Molleti wrote: > Hi, > > > > We are using a CMS called Escenic which uses the below technologies. > > > > 1)???? Tomcat (Web Server) > > 2)???? J2EE & JSP (Coding platform) > > 3)???? Varnish (Caching server) > > 4)???? Mysql (Database) > > > > We have setup Varnish to cache the pages, a page which is put together using > various widgets, seems to load all at once instead of gradual loading. > Hence, there is an initial 3 to 4sec wait where the browser does not get any > output, after that the whole rendered page is dumped on the browser. This > makes the site seem much slower than it actually is. > > > > Can someone please let us know if you have experienced this issue and what > are other ways to speed up Varnish to deliver pages faster? > > > > Thanks > > > > Hitendra > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From scaunter at topscms.com Tue Jun 29 21:13:23 2010 From: scaunter at topscms.com (Caunter, Stefan) Date: Tue, 29 Jun 2010 17:13:23 -0400 Subject: Varnish: Initial lag before dumping output to the browser Message-ID: <064FF286FD17EC418BFB74629578BED1159E25C8@tmg-mail4.torstar.net> You likely need to check with yslow for elements that are synchronous and therefore blocking speedy browser rendering. Follow those rules especially the parallel downloads. Varnish cannot help bad page design. -------------------------- Sent using BlackBerry 416 561 4871 ----- Original Message ----- From: varnish-misc-bounces at varnish-cache.org To: Hitendra Molleti Cc: varnish-misc at varnish-cache.org Sent: Tue Jun 29 16:58:28 2010 Subject: Re: Varnish: Initial lag before dumping output to the browser Hi, If you run a tcpdump on your client, do you see any packet exchanged or the traffic is on hold for the 3-4s? Do you have this behavior when browsing the backend directly? cheers On Tue, Jun 29, 2010 at 2:54 PM, Hitendra Molleti wrote: > Hi, > > > > We are using a CMS called Escenic which uses the below technologies. > > > > 1)???? Tomcat (Web Server) > > 2)???? J2EE & JSP (Coding platform) > > 3)???? Varnish (Caching server) > > 4)???? Mysql (Database) > > > > We have setup Varnish to cache the pages, a page which is put together using > various widgets, seems to load all at once instead of gradual loading. > Hence, there is an initial 3 to 4sec wait where the browser does not get any > output, after that the whole rendered page is dumped on the browser. This > makes the site seem much slower than it actually is. > > > > Can someone please let us know if you have experienced this issue and what > are other ways to speed up Varnish to deliver pages faster? > > > > Thanks > > > > Hitendra > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org http://lists.varnish-cache.org/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjdavies at glam.ac.uk Wed Jun 30 07:05:15 2010 From: mjdavies at glam.ac.uk (Matt Davies) Date: Wed, 30 Jun 2010 08:05:15 +0100 Subject: Starting varnish with a PID file In-Reply-To: <3CA4AE2B-C18B-4B4B-AC04-604404D0366A@glam.ac.uk> References: <282B7DC0-A852-4666-BE1F-B413BC6CF989@glam.ac.uk> <4C29C6D6.1000702@netclever.nl> <5FB849E2-3F12-462D-B98D-478F6D45A41D@glam.ac.uk> <4C29D5C1.9060405@fholzhauer.de> <3CA4AE2B-C18B-4B4B-AC04-604404D0366A@glam.ac.uk> Message-ID: <5EB37A97-C274-4E6A-BA9B-C403016A79D6@glam.ac.uk> Hello again I'm going to be using God to monitor our varnish installation. Here's how I plan on starting God with a PID file sudo /usr/local/sbin/varnishd -f /usr/local/etc/varnish/default.vcl -s malloc,4G -T 127.0.0.1:2000 -a 0.0.0.0:80 -P /usr/local/etc/varnish/ varnish.pid Then when I want to restart varnish programatically if the PID in that file is not running I guess I'll have to clean out all varnishd processes, including the child one, in case that is still running when the MGR is not, and then start Varnish again. Or if the MGR process is not running will the child one stop too? As always, any tips or assistance is greatly appreciated. I'll share the God script once I've worked it out. Matt From tfheen at varnish-software.com Wed Jun 30 07:49:49 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Wed, 30 Jun 2010 09:49:49 +0200 Subject: varnish-misc Digest, Vol 51, Issue 34 In-Reply-To: <7A341253-4A1D-4468-90E5-AB0ABEB432AF@competitorgroup.com> (Ben Nowacky's message of "Mon, 28 Jun 2010 07:34:44 -0700") References: <7A341253-4A1D-4468-90E5-AB0ABEB432AF@competitorgroup.com> Message-ID: <87y6dx2c4i.fsf@qurzaw.linpro.no> ]] Ben Nowacky | Yes, I have tried Ben, thread__pool_stacksize with no luck. varnish | will not start up with me with that flag in my startup string. use ulimit -s in your startup script. -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From vburshteyn at broadway.com Tue Jun 1 16:28:22 2010 From: vburshteyn at broadway.com (Vitaly Burshteyn) Date: Tue, 01 Jun 2010 16:28:22 -0000 Subject: wont server from Cache Message-ID: <0F69574F9901D4459C6B75C9FF64FBC404B4DB9B@mxfl01.hollywoodmedia.corp> Hi folks, So I sent this post few days ago, I have made some changes but still cant get my varnish box to server content from cache even though it looks like it is: 232 Client requests received 130 Cache hits 5 Cache hits for pass 97 Cache misses 99 Backend conn. success 3 Backend conn. failures 99 Fetch with Length 15 N struct sess_mem When I shut down the web server it gives me a 503 error. Here is the copy of my vcl file with out the backend info.. Any and all suggestion would be appreciated. sub vcl_recv { #set req.backend = bw; set req.grace = 60m; if (req.request == "PURGE") {if (!client.ip ~ purge) {error 405 "Not allowed.";} return(lookup);} if (req.request != "GET" && req.request != "HEAD") {return(lookup);} if (req.http.Cache-Control ~ "no-cache") {purge_url(req.url);} return(lookup); } sub vcl_hit { if (!obj.cacheable) { return(pass); }} sub vcl_fetch { if (beresp.status == 500 || beresp.status == 503 || beresp.status == 504) { restart; } set beresp.ttl = 600s; set beresp.grace = 600s; if (beresp.ttl < 300s) { set beresp.ttl = 300s;} if (!beresp.cacheable) {return(pass);} if (beresp.http.Set-Cookie) {return(pass);} } Vitaly Burshteyn Senior Network Engineer Broadway.com, Theatre Direct International 729 7th Avenue New York, New York 10019 Phone: 212.817.9117 Cell# 917-701-5732 ____________________________________ The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.florescu at acasa.ro Mon Jun 7 09:32:43 2010 From: alex.florescu at acasa.ro (Alex Florescu) Date: Mon, 07 Jun 2010 09:32:43 -0000 Subject: VCL changes from 2.0 to 2.1 In-Reply-To: References: <20100607084347.B44712F76EE@smtp1.acasa.ro> Message-ID: <4C0CBCB2.2050802@acasa.ro> Success! Thank you. On 7.6.2010 11:54, Sam Crawford wrote: > See http://varnish-cache.org/wiki/changelog_2.0.6-2.1.0 > > obj.* should now be beresp.* > > Thanks, > > Sam > > On 7.6.2010 11:55, freestyler wrote: > obj.* is now called beresp.* in vcl_fetch, and obj.* is now read-only. > > see http://varnish-cache.org/wiki/changelog_2.0.6-2.1.0 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pavionove at hotmail.com Tue Jun 15 14:38:00 2010 From: pavionove at hotmail.com (Jean-Francois Laurens) Date: Tue, 15 Jun 2010 14:38:00 -0000 Subject: Byte range and varnish Message-ID: Hi, I'm running varnish 2.0.6 under centos 5.4: varnish-libs-2.0.6-2.el5 varnish-2.0.6-2.el5 I'm setting up a media server and would like to use varnish for caching some of the heavily requested medias (mp3, mp4 ... ). I've a few questions regarding media broadcasting with varnish. mp3: progressive download is used in order to read mp3 from the webpages, is it foreseen to have a function to would allow providing the media before it gets completely retrieved by varnish from the backend ? mp4: how is handled Accept-range field in the header, does varnish supports it ? Trying to retrieve a mp4 file from apache gives me the following response: HTTP/1.1 200 OK Date: Tue, 15 Jun 2010 14:24:46 GMT Server: Apache Last-Modified: Mon, 14 Jun 2010 18:01:41 GMT ETag: "2d4d6-1c7d167-4890147e4eb40" Accept-Ranges: bytes Content-Length: 29872487 Content-Type: text/plain; charset=UTF-8 with varnish in the middle: HTTP/1.1 200 OK Server: Apache Last-Modified: Mon, 14 Jun 2010 18:01:41 GMT ETag: "2d4d6-1c7d167-4890147e4eb40" Content-Type: text/plain; charset=UTF-8 Content-Length: 29872487 X-ece-cache: server-01-03.arteria.tsr Cache-Control: max-age=43200 X-ece-was-cached: media: 12h and max-age=43200, age 0 Date: Tue, 15 Jun 2010 14:26:15 GMT X-Varnish: 321078687 321076768 Via: 1.1 varnish Connection: keep-alive Age: 0 The Accept-Ranges field is lost, do you know why ? Jean-Francois Laurens pavionove at hotmail.com From pavionove at hotmail.com Thu Jun 17 12:53:15 2010 From: pavionove at hotmail.com (Jean-Francois Laurens) Date: Thu, 17 Jun 2010 12:53:15 -0000 Subject: Byte range and varnish Message-ID: Hi, I'm running varnish 2.0.6 under centos 5.4: varnish-libs-2.0.6-2.el5 varnish-2.0.6-2.el5 I'm setting up a media server and would like to use varnish for caching some of the heavily requested medias (mp3, mp4 ... ). I've a few questions regarding media broadcasting with varnish. mp3: progressive download is used in order to read mp3 from the webpages, is it foreseen to have a function to would allow providing the media before it gets completely retrieved by varnish from the backend ? mp4: how is handled Accept-range field in the header, does varnish supports it ? Trying to retrieve a mp4 file from apache gives me the following response: HTTP/1.1 200 OK Date: Tue, 15 Jun 2010 14:24:46 GMT Server: Apache Last-Modified: Mon, 14 Jun 2010 18:01:41 GMT ETag: "2d4d6-1c7d167-4890147e4eb40" Accept-Ranges: bytes Content-Length: 29872487 Content-Type: text/plain; charset=UTF-8 with varnish in the middle: HTTP/1.1 200 OK Server: Apache Last-Modified: Mon, 14 Jun 2010 18:01:41 GMT ETag: "2d4d6-1c7d167-4890147e4eb40" Content-Type: text/plain; charset=UTF-8 Content-Length: 29872487 X-ece-cache: server-01-03.arteria.tsr Cache-Control: max-age=43200 X-ece-was-cached: media: 12h and max-age=43200, age 0 Date: Tue, 15 Jun 2010 14:26:15 GMT X-Varnish: 321078687 321076768 Via: 1.1 varnish Connection: keep-alive Age: 0 The Accept-Ranges field is lost, do you know why ? Jean-Francois Laurens pavionove at hotmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ereboos at googlemail.com Fri Jun 18 10:59:37 2010 From: ereboos at googlemail.com (Ereboos) Date: Fri, 18 Jun 2010 10:59:37 -0000 Subject: deliver graced content instead of waiting for backend response Message-ID: <4C1B5196.7070803@gmail.com> Hello, In our setup we have some web content that takes a longer time to be built (around 10seconds). We have set up a varnish cache to cache this request for 20 seconds. But we dont want our users who access this service after the 20 seconds to wait for 10 seconds after the backend has finished. Instead we would like a old document to be delivered and the cache to be updated (if age is still under 40 seconds) So I setup a grace time of 20 seconds. But unfortunately the first request coming after the 20 seconds of ttl is delayed for the 10 seconds of generation time. all other requests which follow immediatelly will receive the old content (grace). How can I achieve that the first request will also receive old content and varnish internally updates his cache then? thank you in advance, Markus From ereboos at googlemail.com Fri Jun 18 11:01:36 2010 From: ereboos at googlemail.com (Ereboos) Date: Fri, 18 Jun 2010 11:01:36 -0000 Subject: saintmode with multiple backends Message-ID: <4C1B509D.9010609@gmail.com> Hello, In my configuration I have multiple backends which are loadbalanced with a director. I now want to enable saintmode in such a way that if a request fails with >=500 error code this url is blocked for the next X minutes _for_all_ backends in this director, so the infrastructure gets a chance to recover [1]. During that recovery time old content should be delivered instead of the 50x error message (I am able to achieve this with the grace mode) In my current configuration I use this vcl_fetch: sub vcl_fetch { set beresp.grace = 1h; if ( beresp.status == 503 ) { set beresp.saintmode = 20s; restart; } } the problem is that restart will lead to the same request sent to the next backend and only if all backends failed the grace is done. Do you have any ideas how I can solve this? thank you in advance, Markus [1] the http backends rely on some replicated database servers which often cause lock problems. So if a request fails on one http server no other http server will be able to handle it. From jimmy at jetheis.com Mon Jun 21 01:49:00 2010 From: jimmy at jetheis.com (Jimmy Theis) Date: Mon, 21 Jun 2010 01:49:00 -0000 Subject: Problems loading directory URLs Message-ID: <4C1EC501.4090201@jetheis.com> I'm new to Varnish and am attempting to use the stock settings for the FreeBSD ports distribution of the program. So far I'm very happy with the performance increase I've seen, even without digging down and tuning, but there is one bit of strange behavior I'm struggling with: When the url of a directory is requested without a / following it, the request fails and nothing is returned. However, in all other situations, including appending the final / to the url, Varnish handles it correctly. I'm running Apache for Subversion and PHP handling, with nginx on top of that as a proxy and for hosting static files. Nginx serves on port 8000 and Varnish uses it as its backend. The URL weirdness occurs both when accessing a directory that nginx serves up and directories inside Subversion repositories that Apache serves. This behavior is not present when nginx is listening directly to port 80. I'm happy to provide extra information if need be. This is a server for personal projects, so I'm fine tinkering with it if it helps solve the problem. Thanks in advance, Jimmy From bnowacky at competitorgroup.com Thu Jun 24 17:28:15 2010 From: bnowacky at competitorgroup.com (Ben Nowacky) Date: Thu, 24 Jun 2010 17:28:15 -0000 Subject: varnish restarting sporadically... cache being lost. Message-ID: Hoping someone can help... We're just trying to roll out varnish to a somewhat large production site (30million monthly pageviews). Things seem to be working, hitting about a 90% hitrate so far. Using Wordpress for a CMS and standard LAMP stack on the back-end. - Deployed varnish on 1 of our servers yesterday for final testing on the live site. Runs along well, seems to be doing great. Run varnishstat to keep an eye on it, and randomly all the stats reset back to 0, and there's a full cache purge, which means i'm guessing varnish has restarted? I can't, for the life of me, figure out why it's been restarting. I've tweeked settings that I've seen recommendations for, upped cache/thread/etc an still seems to be happening. Not sure if anyone else has seen this behavior or not, but if anyone has any suggestions for troubleshooting, or figuring out what's going on, would appreciate any help ! I've got varnish running with persistence now, so it's helping a bit when there's a restart since it's not completely unprimed, but need to figure out the cause of the instability. Any help would be awesome... Thanks ! From andersonbrown8 at gmail.com Mon Jun 28 21:15:50 2010 From: andersonbrown8 at gmail.com (Anderson Brown) Date: Mon, 28 Jun 2010 21:15:50 -0000 Subject: Maximum Object Size to be Cached In-Reply-To: References: Message-ID: Thanks. ?I deleted cache.db, hosts.db and restarted varnish. Unfortunately, I still see it querying the origin server for those files all the time. ?Also, I saw these from traffic_line -r after the tests. ?I queried 1000 urls at least 2-3 times of size between 1mb-2mb. Seems like it didn't use the cache at all after the curls: proxy.process.cache.bytes_used ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?: 0 proxy.process.cache.bytes_total ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? : 536308736 Thoughts? Thanks! Logs: > 13 StatSess ? ? c 10.121.42.43 44438 0 1 1 0 0 1 310 1095295 > 13 SessionOpen ?c 10.121.42.43 44440 :6081 > 13 ReqStart ? ? c 10.121.42.43 44440 660194525 > 13 RxRequest ? ?c GET > 13 RxURL ? ? ? ?c /somefile > 13 RxProtocol ? c HTTP/1.1 > 13 RxHeader ? ? c User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) > libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5 > 13 RxHeader ? ? c Host: jk.com:6081 > 13 RxHeader ? ? c Accept: */* > 13 VCL_call ? ? c recv > 13 VCL_return ? c lookup > 13 VCL_call ? ? c hash > 13 VCL_return ? c hash > 13 VCL_call ? ? c miss > 13 VCL_return ? c fetch > 14 BackendOpen ?b default 127.0.0.1 24044 127.0.0.1 8080 > 13 Backend ? ? ?c 14 default default > 14 TxRequest ? ?b GET > 14 TxURL ? ? ? ?b /somefile > 14 TxProtocol ? b HTTP/1.1 > 14 TxHeader ? ? b User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) > libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5 > 14 TxHeader ? ? b Host: jk.com:6081 > 14 TxHeader ? ? b Accept: */* > 14 TxHeader ? ? b X-Varnish: 660194525 > 14 TxHeader ? ? b X-Forwarded-For: 10.121.42.43 > 14 RxProtocol ? b HTTP/1.1 > 14 RxStatus ? ? b 200 > 14 RxResponse ? b OK > 14 RxHeader ? ? b Date: Fri, 25 Jun 2010 13:25:19 GMT > 14 RxHeader ? ? b Server: Apache/2.2.3 (Red Hat) > 14 RxHeader ? ? b Last-Modified: Tue, 27 Apr 2010 00:32:52 GMT > 14 RxHeader ? ? b ETag: "1e9032c-1131fd-4852d08a69500" > 14 RxHeader ? ? b Accept-Ranges: bytes > 14 RxHeader ? ? b Content-Length: 1126909 > 14 RxHeader ? ? b Connection: close > 14 RxHeader ? ? b Content-Type: text/plain; charset=UTF-8 > 13 ObjProtocol ?c HTTP/1.1 > 13 ObjStatus ? ?c 200 > 13 ObjResponse ?c OK > 13 ObjHeader ? ?c Date: Fri, 25 Jun 2010 13:25:19 GMT > 13 ObjHeader ? ?c Server: Apache/2.2.3 (Red Hat) > 13 ObjHeader ? ?c Last-Modified: Tue, 27 Apr 2010 00:32:52 GMT > 13 ObjHeader ? ?c ETag: "1e9032c-1131fd-4852d08a69500" > 13 ObjHeader ? ?c Content-Type: text/plain; charset=UTF-8 > 13 VCL_call ? ? c discard > 13 VCL_return ? c discard > ?0 ExpKill ? ? ?- 660193643 LRU > 14 BackendClose b default > 13 TTL ? ? ? ? ?c 660194525 RFC 120 1277472319 0 0 0 0 > 13 VCL_call ? ? c fetch > 13 VCL_return ? c deliver > 13 Length ? ? ? c 1126909 > 13 VCL_call ? ? c deliver > 13 VCL_return ? c deliver > 13 TxProtocol ? c HTTP/1.1 > 13 TxStatus ? ? c 200 > 13 TxResponse ? c OK > 13 TxHeader ? ? c Server: Apache/2.2.3 (Red Hat) > 13 TxHeader ? ? c Last-Modified: Tue, 27 Apr 2010 00:32:52 GMT > 13 TxHeader ? ? c ETag: "1e9032c-1131fd-4852d08a69500" > 13 TxHeader ? ? c Content-Type: text/plain; charset=UTF-8 > 13 TxHeader ? ? c Content-Length: 1126909 > 13 TxHeader ? ? c Date: Fri, 25 Jun 2010 13:25:19 GMT > 13 TxHeader ? ? c X-Varnish: 660194525 > 13 TxHeader ? ? c Age: 0 > 13 TxHeader ? ? c Via: 1.1 varnish > 13 TxHeader ? ? c Connection: keep-alive > 13 ReqEnd ? ? ? c 660194525 1277472319.461021900 1277472319.464087963 > 0.000044823 0.001679182 0.001386881 > 13 SessionClose c EOF > 13 ReqEnd ? ? ? c 0 1277472319.464224100 1277472319.464224100 > 0.000136137 0.000000000 0.000000000 > 13 StatSess ? ? c 10.121.42.43 44440 0 1 1 0 0 1 310 1126909 > ?0 CLI ? ? ? ? ?- Rd ping > ?0 CLI ? ? ? ? ?- Wr 0 200 PONG 1277472321 1.0 > > > On Thu, Jun 24, 2010 at 12:08 PM, Per Buer wrote: >> On Thu, Jun 24, 2010 at 5:46 PM, Anderson Brown >> wrote: >>> Thanks. ?Not sure which part of the varnishlogs you need: >>> (..) >>> >>> ? 13 RxRequest ? ?c GET >>> ? 13 RxURL ? ? ? ?c /somefile >>> ? 13 RxProtocol ? c HTTP/1.1 >> >> The client request. >> >>> (..) >>> ? 13 VCL_call ? ? c miss >> >> Which isn't in the cache. >> >>> (..) >>> ? 14 TxRequest ? ?b GET >>> ? 14 TxURL ? ? ? ?b /somefile >>> ? 14 TxProtocol ? b HTTP/1.1 >> >> Request it from the backend. >> >>> (..) >>> ? 13 ObjHeader ? ?c ETag: "2da8848-12cbec-48531772a5f80" >>> ? 13 ObjHeader ? ?c Content-Type: text/plain; charset=UTF-8 >>> ? 13 VCL_call ? ? c discard >>> ? 13 VCL_return ? c discard >>> ? ?0 ExpKill ? ? ?- 660189645 LRU >>> ? 13 VCL_call ? ? c discard >>> ? 13 VCL_return ? c discard >>> ? ?0 ExpKill ? ? ?- 660189646 LRU >> >> LRU kicks inn. You're running low on cache. Try adding more cache to Varnish. >> >> >> -- >> Per Buer, ?Varnish Software >> Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer >> > From hitendra.molleti at itp.com Tue Jun 29 12:39:15 2010 From: hitendra.molleti at itp.com (Hitendra Molleti) Date: Tue, 29 Jun 2010 12:39:15 -0000 Subject: Varnish: Initial lag before dumping output to the browser Message-ID: <00aa01cb1788$38ca3c40$aa5eb4c0$@molleti@itp.com> Hi, We are using a CMS called Escenic which uses the below technologies. 1) Tomcat (Web Server) 2) J2EE & JSP (Coding platform) 3) Varnish (Caching server) 4) Mysql (Database) We have setup Varnish to cache the pages, a page which is put together using various widgets, seems to load all at once instead of gradual loading. Hence, there is an initial 3 to 4sec wait where the browser does not get any output, after that the whole rendered page is dumped on the browser. This makes the site seem much slower than it actually is. Can someone please let us know if you have experienced this issue and what are other ways to speed up Varnish to deliver pages faster? Thanks Hitendra -------------- next part -------------- An HTML attachment was scrubbed... URL: From hitendra.molleti at itp.com Tue Jun 29 12:43:21 2010 From: hitendra.molleti at itp.com (Hitendra Molleti) Date: Tue, 29 Jun 2010 12:43:21 -0000 Subject: Varnish: Initial lag before dumping output to the browser Message-ID: <00b201cb1788$c949a8f0$5bdcfad0$@molleti@itp.com> Hi, We are using a CMS called Escenic which uses the below technologies. 1) Tomcat (Web Server) 2) J2EE & JSP (Coding platform) 3) Varnish (Caching server) 4) Mysql (Database) We have setup Varnish to cache the pages, a page which is put together using various widgets, seems to load all at once instead of gradual loading. Hence, there is an initial 3 to 4sec wait where the browser does not get any output, after that the whole rendered page is dumped on the browser. This makes the site seem much slower than it actually is. Can someone please let us know if you have experienced this issue and what are other ways to speed up Varnish to deliver pages faster? Thanks Hitendra -------------- next part -------------- An HTML attachment was scrubbed... URL: