From erik at erikwickstrom.com Thu Sep 1 06:01:43 2011 From: erik at erikwickstrom.com (Erik Wickstrom) Date: Wed, 31 Aug 2011 23:01:43 -0700 Subject: Long Grace setting for huge cache Message-ID: I have a web application where the objects are expensive to generate (about 30 seconds), yet each individual object is accessed infrequently (a few times a week). The dataset in total is larger than RAM. Can Varnish be configured to cache these items for a short amount of time (say 20 minutes), but also have a relatively long (i.e. 7 days) grace setting so clients will always get a fast response (but at the same time will trigger an update of the cached item). I would warm the cache by spidering my sitemap upon starting the Varnish server so the first visitors won't have to wait for the object to be generated. Basically I'm trying to keep my cache warm, without overwhelming my app servers, and keep access quick for my visitors. Is this doable? Also -- since my dataset is larger than RAM, which backend is the best choice? I was thinking the file backend would make the most sense -- but just wanted to make sure. Thanks! Erik From ljorg6 at gmail.com Thu Sep 1 07:26:01 2011 From: ljorg6 at gmail.com (=?ISO-8859-1?Q?Lars_J=F8rgensen?=) Date: Thu, 01 Sep 2011 09:26:01 +0200 Subject: Long Grace setting for huge cache In-Reply-To: References: Message-ID: <4E5F3389.4000402@gmail.com> On 01-09-2011 08:01, Erik Wickstrom wrote: > I have a web application where the objects are expensive to generate > (about 30 seconds), > yet each individual object is accessed infrequently (a few times a > week). The dataset in total is larger than RAM. How often do these objects change? If they change infrequently, you might consider setting your cache for a very long ttl (like a year) and then have your backend purge objects from the cache as they are changed in the backend. That way you end up having everything in the cache (sooner or later), and everything in the cache always being the latest version. This works well if the backend is pretty static. Depending on how much you can control from your backend, you can purge a changed object from the cache and then request it to get a new version in there immediately and not inconvenience the first requesting user with a 30 second wait. -- Lars From mls at pooteeweet.org Fri Sep 2 07:52:06 2011 From: mls at pooteeweet.org (Lukas Kahwe Smith) Date: Fri, 2 Sep 2011 09:52:06 +0200 Subject: Is LCI on the radar? In-Reply-To: <087AF654-7980-4AE2-A8DE-34D6937FB081@pooteeweet.org> References: <49097.1309250575@critter.freebsd.dk> <087AF654-7980-4AE2-A8DE-34D6937FB081@pooteeweet.org> Message-ID: <472F667B-5761-42D7-8F16-4612A6E8106A@pooteeweet.org> On Jun 28, 2011, at 10:49 , Lukas Kahwe Smith wrote: > > On 28.06.2011, at 10:42, Poul-Henning Kamp wrote: > >> In message <4B5D88B4-F620-43C0-AB8A-DE19565B22DF at pooteeweet.org>, Lukas Kahwe S >> mith writes: >> >>>> I assume some of you have stumbled over LCI by now: >>>> http://www.ietf.org/id/draft-nottingham-linked-cache-inv-00.txt >> >> It's not a proposal I particular is happy about because it is very >> expensive to implement one way or the other. >> >> But I made a quick attempt and you can implement this using VCL and bans >> in Varnish already, so I don't expect to do more about it. > > > I would appreciate it a lot if your tests and Laurence's solution [1] could somehow we documented on the wiki. We will happily beta test it :) > > regards, > Lukas Kahwe Smith > mls at pooteeweet.org > > [1] http://dev.plone.org/collective/browser/experimental.depends/trunk/varnish.vcl ok .. it looks like there is a good chance we will work on something based on the above for a large news site which is drives multiple separate frontends (each with their own caching) from a single backend. here is my game plan: http://pooteeweet.org/blog/1979 regards, Lukas Kahwe Smith mls at pooteeweet.org From saiaman01 at gmail.com Fri Sep 2 08:16:40 2011 From: saiaman01 at gmail.com (Albuquerque Rui) Date: Fri, 2 Sep 2011 10:16:40 +0200 Subject: Listing cached urls Message-ID: Hello, i'm trying to find out how to list ( varnishlog ) or stat ( varnishtop ) the urls that are cached or not .... Any idea ? Thanks a lot -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjy at geizhals.at Fri Sep 2 08:23:18 2011 From: mjy at geizhals.at (Marinos Yannikos) Date: Fri, 02 Sep 2011 10:23:18 +0200 Subject: Cannot stop Varnish from caching 403 responses Message-ID: <4E609276.6010200@geizhals.at> Hi, I cannot figure this out: varnishd (varnish-3.0.0 revision 3bd5997) host1 is not allowed to access due to IP address blacklisting: host1:~# HEAD -S HEAD --> 403 denied Cache-Control: no-cache, no-store, private Connection: close Date: Fri, 02 Sep 2011 08:05:15 GMT Pragma: no-cache Vary: Accept-Encoding Content-Type: text/html; charset=ISO-8859-1 Expires: Thu, 01 Sep 2011 08:05:15 GMT Client-Date: Fri, 02 Sep 2011 08:05:15 GMT Client-Peer: <...> Client-Response-Num: 1 X-Varnish: 127539184 (from another IP address, which is allowed to access that URL, a few seconds later) host2:~# HEAD -S HEAD --> 403 denied Cache-Control: no-cache, no-store, private Connection: close Date: Fri, 02 Sep 2011 08:05:17 GMT Pragma: no-cache Vary: Accept-Encoding Content-Type: text/html; charset=ISO-8859-1 Expires: Thu, 01 Sep 2011 08:05:15 GMT Client-Date: Fri, 02 Sep 2011 08:05:17 GMT Client-Peer: <...> Client-Response-Num: 1 X-Varnish: 127539465 127539184 (a while later) host2:~# HEAD -S ... HEAD --> 200 OK So basically 403 reponses that are dependant on client IP address in our case, are getting cached despite all cache-defeating headers. varnishlog reports in the second request: 37 VCL_call c recv lookup 37 VCL_call c hash 37 Hash c 37 Hash c 37 VCL_return c hash 37 Hit c 127539184 37 VCL_call c hit deliver 37 VCL_call c deliver deliver 37 TxProtocol c HTTP/1.1 37 TxStatus c 403 37 TxResponse c denied 37 TxHeader c Expires: Thu, 01 Sep 2011 08:05:15 GMT 37 TxHeader c Pragma: no-cache So the object is getting stored in the cache despite all those headers - how can I prevent it (and why are 403 pages even cached)? default.vcl does nothing interesting, it only sets the backend based on server.ip and vcl_deliver clears a few headers: sub vcl_deliver { # remove resp.http.X-Varnish; remove resp.http.Via; remove resp.http.Age; unset resp.http.Server; set resp.http.Server = "GH"; } Regards, Marinos From shib4u at gmail.com Fri Sep 2 08:25:32 2011 From: shib4u at gmail.com (Shibashish) Date: Fri, 2 Sep 2011 13:55:32 +0530 Subject: Listing cached urls In-Reply-To: References: Message-ID: On Fri, Sep 2, 2011 at 1:46 PM, Albuquerque Rui wrote: > Hello, > i'm trying to find out how to list ( varnishlog ) or stat ( varnishtop ) > the urls that are cached or not .... Any idea ? > > Thanks a lot > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > varnishtop -b -i TxURL ... shows real time top-urls being sent to backend servers. varnishlog -b -o Only list backend traffic. varnishlog -b -o -i TxURL List URLS going to a backend. ShiB. while ( ! ( succeed = try() ) ); -------------- next part -------------- An HTML attachment was scrubbed... URL: From mit at stagename.com Fri Sep 2 16:22:54 2011 From: mit at stagename.com (Mit Rowe) Date: Fri, 2 Sep 2011 12:22:54 -0400 Subject: varnishncsa -F question Message-ID: Hello! I am using varnish 3.0.1 and have a question about varnishncsalog I would like to spit out extra parameters in my logs, so i'm attempting to use the -F command line flag. However, my need for spaces seems to be causing some problems. for example... -F %h %t (which i don't expect to work) prints 127.0.0.1 (and that's it) -F '%h %t' prints '127.0.0.1 -F "%h %t" prints "127.0.0.1 and -F %h\ %t prints 127.0.0.1\ Has anyone had any success specifying a log format containing spaces? -- Will 'Mit' Rowe Stagename* *1-866-326-3098 mit at stagename.com www.stagename.com Twitter: @stagename *The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of this information by persons or entities other than the intended recipient is prohibited. If you received this transmission in error, please contact the sender and delete all material contained herein from your computer.* -------------- next part -------------- An HTML attachment was scrubbed... URL: From whshub at gmail.com Fri Sep 2 18:26:09 2011 From: whshub at gmail.com (Jacques) Date: Fri, 2 Sep 2011 11:26:09 -0700 Subject: varnishncsa -F question In-Reply-To: References: Message-ID: I think I'm missing something. We use the following without problem on 3.0.1 varnishncsa -F '%{X-Forwarded-For}i %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i" "%{Cookie}i" "%{Varnish:hitmiss}x"' -X 'pingdom$|\.png$|\.ico$|\.jpg$|\.gif$' On Fri, Sep 2, 2011 at 9:22 AM, Mit Rowe wrote: > Hello! > > I am using varnish 3.0.1 and have a question about varnishncsalog > > I would like to spit out extra parameters in my logs, so i'm attempting to > use the -F command line flag. > > However, my need for spaces seems to be causing some problems. > > for example... > > -F %h %t > > (which i don't expect to work) > > prints > > 127.0.0.1 > > (and that's it) > > -F '%h %t' > > prints > > '127.0.0.1 > > -F "%h %t" > > prints > > "127.0.0.1 > > and > > -F %h\ %t > > prints > > 127.0.0.1\ > > Has anyone had any success specifying a log format containing spaces? > > > > > -- > Will 'Mit' Rowe > Stagename* > *1-866-326-3098 > mit at stagename.com > www.stagename.com > Twitter: @stagename > > *The information transmitted is intended only for the person or entity to > which it is addressed and may contain confidential and/or privileged > material. Any review, retransmission, dissemination or other use of this > information by persons or entities other than the intended recipient is > prohibited. If you received this transmission in error, please contact the > sender and delete all material contained herein from your computer.* > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ruben.ortiz at rsystem.com Mon Sep 5 10:58:53 2011 From: ruben.ortiz at rsystem.com (=?ISO-8859-1?Q?Rub=E9n_Ortiz?=) Date: Mon, 5 Sep 2011 12:58:53 +0200 Subject: Varnish shows 503 meanwhile Apache returns CSV correctly Message-ID: Hi all I need some support here :) I have configured a Varnish service in front of Apache (backend), the software which runs in backend is PHP (Magento). The problem is without Varnish, when we login in magento panel and we click on Clients/Export, Apache returns OK a CSV to download. No problem. But if we activate Varnish listening on :80 and Apache on :81, and we do the same process, at final moment Varnish returns a 503. We have deployed Varnish VLC to work with Magento and all works fine. Here it is a brief extract of code relevant about Magento admin # Don't cache pages for Magento Admin # FIXME: change this rule if you use custom url in admin if (req.url ~ "^/(index.php/)?pc_deporv") { return(pass); } # Don't cache checkout/customer pages, product compare if (req.url ~ "^/(index.php/)?(checkout|customer|catalog/product_compare|wishlist)") { return(pass); } I have added some other lines to try to avoid cache of .csv files, without success. Every time we execute code to expor CSV, we get a 503. Can somebody give us some help? thanks! -- Rub?n Ortiz -------------- next part -------------- An HTML attachment was scrubbed... URL: From ljorg6 at gmail.com Mon Sep 5 13:44:08 2011 From: ljorg6 at gmail.com (=?ISO-8859-1?Q?Lars_J=F8rgensen?=) Date: Mon, 05 Sep 2011 15:44:08 +0200 Subject: Listing cached urls In-Reply-To: References: Message-ID: <4E64D228.70203@gmail.com> On 02-09-2011 10:16, Albuquerque Rui wrote: > Hello, > i'm trying to find out how to list ( varnishlog ) or stat ( varnishtop ) > the urls that are cached or not .... Any idea ? Check out the manpage for varnishncsa. There's a parameter that will print "hit" or "miss" on each line accordingly. Be aware that it might be broken at the moment, but I'm sure a fix is on the way. -- Lars From ljorg6 at gmail.com Mon Sep 5 13:48:10 2011 From: ljorg6 at gmail.com (=?ISO-8859-1?Q?Lars_J=F8rgensen?=) Date: Mon, 05 Sep 2011 15:48:10 +0200 Subject: Cannot stop Varnish from caching 403 responses In-Reply-To: <4E609276.6010200@geizhals.at> References: <4E609276.6010200@geizhals.at> Message-ID: <4E64D31A.9080804@gmail.com> On 02-09-2011 10:23, Marinos Yannikos wrote: > So the object is getting stored in the cache despite all those headers - > how can I prevent it (and why are 403 pages even cached)? default.vcl > does nothing interesting, it only sets the backend based on server.ip > and vcl_deliver clears a few headers: AFAIK Varnish is not supposed to cache error documents, but we had the same problem (on 3.0.0). I added this bit to vcl_fetch: if (beresp.status >= 400) { return (hit_for_pass); } You can go lower than that, but I still want to cache redirects (3xx). -- Lars From ljorg6 at gmail.com Mon Sep 5 13:51:54 2011 From: ljorg6 at gmail.com (=?ISO-8859-1?Q?Lars_J=F8rgensen?=) Date: Mon, 05 Sep 2011 15:51:54 +0200 Subject: Varnish shows 503 meanwhile Apache returns CSV correctly In-Reply-To: References: Message-ID: <4E64D3FA.8000209@gmail.com> On 05-09-2011 12:58, Rub?n Ortiz wrote: > No problem. But if we activate Varnish listening on :80 and > Apache on :81, and we do the same process, at final moment Varnish > returns a 503. Quick "where's the problem" check: Have you tried accessing your backend on port 81 from a browser? If that works, the problem is Varnish. If it doesn't, the problem is the backend. -- Lars From ruben.ortiz at rsystem.com Mon Sep 5 13:55:45 2011 From: ruben.ortiz at rsystem.com (=?ISO-8859-1?Q?Rub=E9n_Ortiz?=) Date: Mon, 5 Sep 2011 15:55:45 +0200 Subject: Varnish shows 503 meanwhile Apache returns CSV correctly In-Reply-To: <4E64D3FA.8000209@gmail.com> References: <4E64D3FA.8000209@gmail.com> Message-ID: Hello Lars yes, I have tried it but different way. I stopped Varnish service and then I started Apache to listen in 80. And yes, I was able then to download CSV file. So Apache/PHP is working fine, it returns csv file. And don't know why Varnish returns a Guru Meditation. Do you have any idea? Bye :) On Mon, Sep 5, 2011 at 3:51 PM, Lars J?rgensen wrote: > On 05-09-2011 12:58, Rub?n Ortiz wrote: > >> No problem. But if we activate Varnish listening on :80 and >> Apache on :81, and we do the same process, at final moment Varnish >> returns a 503. >> > > Quick "where's the problem" check: Have you tried accessing your backend on > port 81 from a browser? If that works, the problem is Varnish. If it > doesn't, the problem is the backend. > > > -- > Lars > > ______________________________**_________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/**lists/mailman/listinfo/**varnish-misc > -- Rub?n Ortiz -------------- next part -------------- An HTML attachment was scrubbed... URL: From apj at mutt.dk Mon Sep 5 14:01:08 2011 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Mon, 5 Sep 2011 16:01:08 +0200 Subject: Varnish shows 503 meanwhile Apache returns CSV correctly In-Reply-To: References: <4E64D3FA.8000209@gmail.com> Message-ID: <20110905140108.GM1944@nerd.dk> On Mon, Sep 05, 2011 at 03:55:45PM +0200, Rub?n Ortiz wrote: > > yes, I have tried it but different way. I stopped Varnish service and then I > started Apache to listen in 80. And yes, I was able then to download CSV > file. So Apache/PHP is working fine, it returns csv file. And don't know why > Varnish returns a Guru Meditation. Post a varnishlog (varnishlog -o) of a failing request. -- Andreas From hensel.tobias at googlemail.com Mon Sep 5 14:12:03 2011 From: hensel.tobias at googlemail.com (Tobias Hensel) Date: Mon, 5 Sep 2011 16:12:03 +0200 Subject: delete hitpass objects Message-ID: Hello, I use varnish 2.1.5 in front of a java based webapp served by jboss. The webapp is a bit... special... I browse the following url the first time via webbapp.local/a I get a 302 to webapp.local?site=a I browse the url webbapp.local/a a second time. Now I do not get a redirect but the content I expect is show directly under the url webbapp.local/a I want to use this webapp behind varnish. When I first browse to varnish.local/a varnish generates a hitpass object for this url When I browse the url the second time varnish uses the hitpass object for not storing the site in the cache (I guess it works that way). Is it possible, that varnish does not cache the site when the response is a 302 and does not creates a hitpass object but caches the site when the response is a normal 200? Greetings Tobias From ruben.ortiz at rsystem.com Mon Sep 5 14:27:12 2011 From: ruben.ortiz at rsystem.com (=?ISO-8859-1?Q?Rub=E9n_Ortiz?=) Date: Mon, 5 Sep 2011 16:27:12 +0200 Subject: Varnish shows 503 meanwhile Apache returns CSV correctly In-Reply-To: <20110905140108.GM1944@nerd.dk> References: <4E64D3FA.8000209@gmail.com> <20110905140108.GM1944@nerd.dk> Message-ID: Hi all again, and thanks by your interest! I have executed this command (I don't have very hugh experience on varnishlog) 15 SessionOpen c 217.113.245.102 51356 :80 15 ReqStart c 217.113.245.102 51356 461182411 15 RxRequest c GET 15 RxURL c /index.php/pc_deporv/customer/exportCsv/ 15 RxProtocol c HTTP/1.1 15 RxHeader c Host: www.deporvillage.com 15 RxHeader c User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; es-ES; rv:1.9.2.21) Gecko/20110830 Firefox/3.6.21 15 RxHeader c Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 15 RxHeader c Accept-Language: es-es,es;q=0.8,en-us;q=0.5,en;q=0.3 15 RxHeader c Accept-Encoding: gzip,deflate 15 RxHeader c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 15 RxHeader c Keep-Alive: 115 15 RxHeader c Connection: keep-alive 15 RxHeader c Referer: http://www.deporvillage.com/index.php/pc_deporv/customer/ 15 RxHeader c Cookie: __utma=266457499.736506096.1313500965.1315222103.1315229776.44; __utmz=266457499.1313500965.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); _bkuid=5996; _bkaui=BK-212245503780-2-459702727612; __utmc=266457499; adminhtml=jtrc6bdutlclu5vhnks684n4 15 VCL_call c recv pass 15 VCL_call c hash hash 15 VCL_call c pass pass 15 Backend c 16 default default 15 TTL c 461182411 RFC 0 1315231825 1315231804 375007920 0 0 15 VCL_call c fetch deliver 15 ObjProtocol c HTTP/1.1 15 ObjStatus c 200 15 ObjResponse c OK 15 ObjHeader c Date: Mon, 05 Sep 2011 14:10:04 GMT 15 ObjHeader c Server: Apache/2.2.3 (CentOS) 15 ObjHeader c X-Powered-By: PHP/5.2.10 15 ObjHeader c Expires: Thu, 19 Nov 1981 08:52:00 GMT 15 ObjHeader c Cache-Control: must-revalidate, post-check=0, pre-check=0 15 ObjHeader c Pragma: public 15 ObjHeader c Content-Length: 728498 15 ObjHeader c Content-Disposition: attachment; filename=customers.csv 15 ObjHeader c Last-Modified: Mon, 05 Sep 2011 14:10:25 +0000 15 ObjHeader c Content-Encoding: gzip 15 ObjHeader c Vary: Accept-Encoding,User-Agent 15 ObjHeader c Content-Type: application/octet-stream 15 FetchError c straight read_error: 0 15 VCL_call c error deliver 15 VCL_call c deliver deliver 15 TxProtocol c HTTP/1.1 15 TxStatus c 503 15 TxResponse c Service Unavailable 15 TxHeader c Server: Varnish 15 TxHeader c Retry-After: 0 15 TxHeader c Content-Type: text/html; charset=utf-8 15 TxHeader c Content-Length: 418 15 TxHeader c Date: Mon, 05 Sep 2011 14:10:30 GMT 15 TxHeader c X-Varnish: 461182411 15 TxHeader c Age: 27 15 TxHeader c Via: 1.1 varnish 15 TxHeader c Connection: close 15 TxHeader c X-Cache: MISS 15 Length c 418 15 ReqEnd c 461182411 1315231804.141705036 1315231830.895998955 0.000108004 26.754230976 0.000062943 There are some more entries but I'm not sure if they are important. I will glad if we will be able to make varnish pass this url always to BACKEND. I understand this a little bit tricky :-/ Thanks guys! On Mon, Sep 5, 2011 at 4:01 PM, Andreas Plesner Jacobsen wrote: > On Mon, Sep 05, 2011 at 03:55:45PM +0200, Rub?n Ortiz wrote: > > > > yes, I have tried it but different way. I stopped Varnish service and > then I > > started Apache to listen in 80. And yes, I was able then to download CSV > > file. So Apache/PHP is working fine, it returns csv file. And don't know > why > > Varnish returns a Guru Meditation. > > Post a varnishlog (varnishlog -o) of a failing request. > > -- > Andreas > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- Rub?n Ortiz -------------- next part -------------- An HTML attachment was scrubbed... URL: From mhettwer at team.mobile.de Mon Sep 5 16:01:49 2011 From: mhettwer at team.mobile.de (Hettwer, Marian) Date: Mon, 5 Sep 2011 17:01:49 +0100 Subject: Varnish shows 503 meanwhile Apache returns CSV correctly In-Reply-To: Message-ID: Hi There, On 05.09.11 15:55, "Rub?n Ortiz" wrote: >Hello Lars > >yes, I have tried it but different way. I stopped Varnish service and >then I started Apache to listen in 80. And yes, I was able then to >download CSV file. So Apache/PHP is working fine, it returns csv file. >And don't know why Varnish returns a Guru Meditation. How long does the generation of this CSV file takes on the backend? IIRC varnish has a default first_byte_timeout of 60 seconds. If your backend takes longer than 60 seconds to start sending data, varnish will have closed the connection and your client will geht a 503. Cheers, Marian From kristian at varnish-software.com Tue Sep 6 08:13:53 2011 From: kristian at varnish-software.com (Kristian Lyngstol) Date: Tue, 6 Sep 2011 10:13:53 +0200 Subject: Bug with VRT_GetHDR on vcl_fetch In-Reply-To: References: Message-ID: <20110906081353.GE3515@freud.kly.no> On Mon, Sep 05, 2011 at 10:05:49PM +0200, Jean Praloran wrote: > I'm currently using varnish 3.0.0 on debian Squeeze 64bits, installed from > the devbian varnish repository. I'm trying to follow the idea of this script > : https://www.varnish-cache.org/trac/wiki/VCLExampleExtendingCacheControl > > I can't manage to make it work since varnish child crashes : [...] > Condition(l == strlen(hdr + 1)) not true.#012thread = > > while using this C embedded : > > ttl = VRT_GetHdr(sp, HDR_OBJ, "\016Varnish-Control:"); [...] \016 is octal. Your length thus ends up being incorrect. I suggest \020. Also, you should use either a bug report or the -misc or -dev list. The -bug list is mainly intended for automatic mails from trac. - Kristian From connor.walls at skillpages.com Tue Sep 6 14:01:28 2011 From: connor.walls at skillpages.com (Connor Walls) Date: Tue, 6 Sep 2011 15:01:28 +0100 Subject: Accept, Accept-Encoding and similar pitfalls Message-ID: Hi, I know this isn't specifically a varnish issue, but I imagine it's a problem that many people here have faced, and might be able to give some help with. Recently we had an issue where our varnish server was serving up mobile content to desktop users, which was causing a "Save As" dialog whenever a user tried to browse to the homepage. We realised that our backend server was responding with a "Vary: Accept-Encoding" header, but no vary for "Accept", and we've since fixed this problem. I'm just curious to find out if there are any other headers that anyone has experienced similar problems with? We're now explicitly dealing with Accept, Accept-Encoding, and deliberately ignoring User-Agent, but is there anything else we should be looking out for? Thanks, Connor Walls From robertomoutinho at gmail.com Tue Sep 6 17:37:16 2011 From: robertomoutinho at gmail.com (Roberto Moutinho) Date: Tue, 6 Sep 2011 14:37:16 -0300 Subject: Using varnish to authenticate http basic API Keys Message-ID: Hello. I'm posting this little inline C code that I'm using to authenticate api keys (http basic) to deliver cached content with varnish cache. A big thank you to the varnish misc mailing list that helped me thru the process. (this is a work in progress...if you have any suggestions to the code I'm here) http://robertomoutinho.com.br/blog/using-memcached-mysql-inline-c-varnish-cache-to-authenticate-basic-keys-restfull-api/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeanpralo at gmail.com Tue Sep 6 08:30:16 2011 From: jeanpralo at gmail.com (Jean Praloran) Date: Tue, 6 Sep 2011 10:30:16 +0200 Subject: Bug with VRT_GetHDR on vcl_fetch In-Reply-To: <20110906081353.GE3515@freud.kly.no> References: <20110906081353.GE3515@freud.kly.no> Message-ID: Ok sure ! Thanks for your help should have search more on the internet. Found few doc about the embedded c thought. Wouldn't it be better just checking the lenght first and returns a null pointer instead of crashing the process ? On Tue, Sep 6, 2011 at 10:13 AM, Kristian Lyngstol < kristian at varnish-software.com> wrote: > On Mon, Sep 05, 2011 at 10:05:49PM +0200, Jean Praloran wrote: > > I'm currently using varnish 3.0.0 on debian Squeeze 64bits, installed > from > > the devbian varnish repository. I'm trying to follow the idea of this > script > > : > https://www.varnish-cache.org/trac/wiki/VCLExampleExtendingCacheControl > > > > I can't manage to make it work since varnish child crashes : > > [...] > > > Condition(l == strlen(hdr + 1)) not true.#012thread = > > > > while using this C embedded : > > > > ttl = VRT_GetHdr(sp, HDR_OBJ, "\016Varnish-Control:"); > > [...] > > \016 is octal. Your length thus ends up being incorrect. I suggest \020. > > Also, you should use either a bug report or the -misc or -dev list. The > -bug list is mainly intended for automatic mails from trac. > > - Kristian > -- Praloran Jean -------------- next part -------------- An HTML attachment was scrubbed... URL: From Jeff.Stroomer at disney.com Wed Sep 7 19:41:01 2011 From: Jeff.Stroomer at disney.com (Stroomer, Jeff) Date: Wed, 7 Sep 2011 12:41:01 -0700 Subject: caching and hiding 302 redirects Message-ID: <4F86EEFDF44D2D41951D491B61B7D4332486B9C732@SM-CALA-VXMB06B.swna.wdpr.disney.com> Varnish folks, I'm trying to combine Varnish with a Tomcat servlet to make a fancy reverse proxy. The idea is that Varnish forwards a URL to the servlet, which uses a 302 redirect to tell Varnish what the "real" URL should be. For this to work well I need Varnish to do two things: 1) cache the 302 redirects from the servlet, and also 2) hide the redirect from the client. It seems like I can get Varnish to do one or the other, but not both simultaneously. I'm using the following in default.vcl (which is more or less the approach described in http://www.gossamer-threads.com/lists/varnish/dev/15409): sub vcl_fetch { if (beresp.status == 302) { set req.url = beresp.http.Location; restart; } } This works, and hides the redirect from the client, but doesn't cache the 302 response. I also tried this: sub vcl_fetch { if (beresp.status == 302) { set req.url = beresp.http.Location; return(deliver); } } This also works, and caches the 302 response, but does not hide the 302 from the client. I am guessing that vcl_deliver is the subroutine in which back-end responses get pushed into the cache, which means that if I hide the redirect by doing a restart, then I lose my chance to cache the 302 response. Is this right? Any suggestions as to how I can cache the 302 response and also hide the redirect from the client? Jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: From rtshilston at gmail.com Wed Sep 7 19:51:14 2011 From: rtshilston at gmail.com (Rob S) Date: Wed, 07 Sep 2011 20:51:14 +0100 Subject: caching and hiding 302 redirects In-Reply-To: <4F86EEFDF44D2D41951D491B61B7D4332486B9C732@SM-CALA-VXMB06B.swna.wdpr.disney.com> References: <4F86EEFDF44D2D41951D491B61B7D4332486B9C732@SM-CALA-VXMB06B.swna.wdpr.disney.com> Message-ID: <4E67CB32.80109@gmail.com> On 07/09/2011 20:41, Stroomer, Jeff wrote: > > Varnish folks, > > I?m trying to combine Varnish with a Tomcat servlet to make a fancy > reverse proxy. The idea is that Varnish forwards a URL to the > servlet, which uses a 302 redirect to tell Varnish what the ?real? URL > should be. For this to work well I need Varnish to do two things: 1) > cache the 302 redirects from the servlet, and also 2) hide the > redirect from the client. It seems like I can get Varnish to do one > or the other, but not both simultaneously. > ... > Any suggestions as to how I can cache the 302 response and also hide > the redirect from the client? > > Jeff > Jeff, Take a look at https://www.varnish-cache.org/trac/ticket/411 - this suggests putting a restart in vcl_deliver. Rob -------------- next part -------------- An HTML attachment was scrubbed... URL: From Jeff.Stroomer at disney.com Wed Sep 7 20:57:13 2011 From: Jeff.Stroomer at disney.com (Stroomer, Jeff) Date: Wed, 7 Sep 2011 13:57:13 -0700 Subject: caching and hiding 302 redirects In-Reply-To: <4E67CB32.80109@gmail.com> References: <4F86EEFDF44D2D41951D491B61B7D4332486B9C732@SM-CALA-VXMB06B.swna.wdpr.disney.com> <4E67CB32.80109@gmail.com> Message-ID: <4F86EEFDF44D2D41951D491B61B7D4332486B9C847@SM-CALA-VXMB06B.swna.wdpr.disney.com> From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Rob S Sent: Wednesday, September 07, 2011 1:51 PM To: varnish-misc at varnish-cache.org Subject: Re: caching and hiding 302 redirects On 07/09/2011 20:41, Stroomer, Jeff wrote: Varnish folks, I?m trying to combine Varnish with a Tomcat servlet to make a fancy reverse proxy. The idea is that Varnish forwards a URL to the servlet, which uses a 302 redirect to tell Varnish what the ?real? URL should be. For this to work well I need Varnish to do two things: 1) cache the 302 redirects from the servlet, and also 2) hide the redirect from the client. It seems like I can get Varnish to do one or the other, but not both simultaneously. ... Any suggestions as to how I can cache the 302 response and also hide the redirect from the client? Jeff Jeff, Take a look at https://www.varnish-cache.org/trac/ticket/411 - this suggests putting a restart in vcl_deliver. Rob ________________________________ Rob, Thanks, this seems like exactly what I need ? but I tried it, and my 302 redirects are still not getting cached. Here?s what I have in my default.vcl: sub vcl_hit { if (obj.http.X-Cache-Redirect == "1") { set req.url = obj.http.Location; restart; } } sub vcl_fetch { if (beresp.status == 302) { set req.url = beresp.http.Location; set beresp.http.X-Cache-Redirect = "1"; set beresp.ttl = 600s; set beresp.cacheable = true; return(deliver); } } sub vcl_deliver { if (resp.http.X-Cache-Redirect == "1") { unset resp.http.X-Cache-Redirect; restart; } return(deliver); } I think this is a faithful copy of what?s in the trac ticket, but updated to the current VCL syntax. If it makes a difference, I?m running the 2.1.3 version of Varnish. Jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: From Jeff.Stroomer at disney.com Wed Sep 7 22:49:17 2011 From: Jeff.Stroomer at disney.com (Stroomer, Jeff) Date: Wed, 7 Sep 2011 15:49:17 -0700 Subject: caching and hiding 302 redirects In-Reply-To: <4F86EEFDF44D2D41951D491B61B7D43324849876F3@SM-CALA-VXMB06B.swna.wdpr.disney.com> References: <4F86EEFDF44D2D41951D491B61B7D4332486B9C732@SM-CALA-VXMB06B.swna.wdpr.disney.com> <4E67CB32.80109@gmail.com> <4F86EEFDF44D2D41951D491B61B7D43324849876F3@SM-CALA-VXMB06B.swna.wdpr.disney.com> Message-ID: <4F86EEFDF44D2D41951D491B61B7D4332486B9C97E@SM-CALA-VXMB06B.swna.wdpr.disney.com> From: Stroomer, Jeff Sent: Wednesday, September 07, 2011 2:57 PM To: 'Rob S'; 'varnish-misc at varnish-cache.org' Subject: RE: caching and hiding 302 redirects From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Rob S Sent: Wednesday, September 07, 2011 1:51 PM To: varnish-misc at varnish-cache.org Subject: Re: caching and hiding 302 redirects On 07/09/2011 20:41, Stroomer, Jeff wrote: Varnish folks, I?m trying to combine Varnish with a Tomcat servlet to make a fancy reverse proxy. The idea is that Varnish forwards a URL to the servlet, which uses a 302 redirect to tell Varnish what the ?real? URL should be. For this to work well I need Varnish to do two things: 1) cache the 302 redirects from the servlet, and also 2) hide the redirect from the client. It seems like I can get Varnish to do one or the other, but not both simultaneously. ... Any suggestions as to how I can cache the 302 response and also hide the redirect from the client? Jeff Jeff, Take a look at https://www.varnish-cache.org/trac/ticket/411 - this suggests putting a restart in vcl_deliver. Rob ________________________________ Rob, Thanks, this seems like exactly what I need ? but I tried it, and my 302 redirects are still not getting cached. Here?s what I have in my default.vcl: ? Jeff ________________________________ Rob, et al., Correction ? I must have made a mistake earlier. When I try the solution outlined in https://www.varnish-cache.org/trac/ticket/411, it looks like things fall apart (browser gets empty response). The problem seems to be doing the deliver at the end of my vcl_fetch. Is it possible that my copy of Varnish is too old to include the fix for ticket #411? I?m running the 2.1.3 version of varnish (which is what I got as the default when I did an apt-get in Ubuntu 10.10). I see that the fix for #411 dates back to October of 2010, so now I?m thinking maybe I need to upgrade to a newer copy of varnish. Does that sound plausible? If I do upgrade, what?s the best way ? build from source? Jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: From damon at huddler-inc.com Wed Sep 7 18:46:34 2011 From: damon at huddler-inc.com (Damon Snyder) Date: Wed, 7 Sep 2011 11:46:34 -0700 Subject: Proper cache allocation (nuked objects but sm_bfree) Message-ID: Hello, I'm pretty sure we need to allocate a bigger cache for our varnish instance, but I'm trying to understand what metrics we should be looking at to determine how severely under allocated we are (and confirm that we need more space). I don't want to just throw more memory at it without under standing more about what is being allocated. I would also be interested to know if we can't reclaim some object space to prevent nuking by doing some runtime parameter tuning. Here are some basic stats about the system: 4 core Centos box with kernel 2.6.18, 8GB of RAM, and a 64GB Intel X-25E SSD. And the varnishstat -1 output and startup options are here: http://pastebin.com/XmM0pMKv. We are running varnishd (varnish-2.1.3 SVN 5049:5055). According to the docs, the key statistic to look at is n_lru_nuked. This value is constantly increasing. Every time you run 'varnishstat -1 -f n_lru_nuked' the value changes. However, the value of sm_bfree seems to always show some space available: varnishstat -1 -f n_lru_nuked,sm_bfree,sm_balloc n_lru_nuked 135193763 . N LRU nuked objects sm_balloc 5468946432 . bytes allocated sm_bfree 2047246336 . bytes free Does sm_bfree mean that we aren't filling the allocated cache? But if so, why do we see the value of n_lru_nuked going up? The n_lru_nuked going up also seems to be confirmed by the IO that we are seeing on the system. I assume this is probably the OS shuffling objects to and from disk. Any suggestions would be appreciated. Thanks, Damon -------------- next part -------------- An HTML attachment was scrubbed... URL: From kristian at varnish-software.com Thu Sep 8 07:40:13 2011 From: kristian at varnish-software.com (Kristian Lyngstol) Date: Thu, 8 Sep 2011 09:40:13 +0200 Subject: Proper cache allocation (nuked objects but sm_bfree) In-Reply-To: References: Message-ID: <20110908074013.GJ3515@freud.kly.no> On Wed, Sep 07, 2011 at 11:46:34AM -0700, Damon Snyder wrote: > According to the docs, the key statistic to look at is n_lru_nuked. This > value is constantly increasing. Every time you run 'varnishstat -1 -f > n_lru_nuked' the value changes. However, the value of sm_bfree seems to > always show some space available: > > varnishstat -1 -f n_lru_nuked,sm_bfree,sm_balloc > n_lru_nuked 135193763 . N LRU nuked objects > sm_balloc 5468946432 . bytes allocated > sm_bfree 2047246336 . bytes free sm_bfree is a counter of how much memory has been freed, not how much is available. Every time an object is removed, expired, etc, this will increase, and it is never reduced. _balloc is the counter part of that, and is increased every time something is allocated, and never reduced. >From these numbers you can calculate how much memory is currently used: 5468946432 - 2047246336 = 3421700096 (a little over 3GB) However, you don't have to actually do that yourself, as _nbytes is doing exactly that. An important detail is that the size specified in the -s arguments is /not/ the total memory footprint varnish will have. It is only the total cache size for actual data, not counting overhead. For Varnish 2.1, we know that there's an additional overhead of slightly over 1kB for each object stored, assuming 64-bit systems (your millage may vary, but this gives you an idea). On top of that is a bit of data for things like threads and sessions, but I rarely take that into consideration myself as it will be a practically constant size measured in MB. Assuming the memory-footprint for each thread is 10kB (it's unlikely that it's that large, specially considering copy-on-write and whatnot), 1000 threads will give a constant overhead of 10MB, so not something to consider. However, 10 million objects will give 10GB of overhead, so that should be accounted for when you decide how much memory the cache will use with -s. Examples I've run with is -s malloc,24GB and -s malloc,28GB on two different sites, both running on a 32GB-system but with different average object-size. Hopefully this helps you figure out what's going on. - Kristian From mjy at geizhals.at Thu Sep 8 07:57:23 2011 From: mjy at geizhals.at (Marinos Yannikos) Date: Thu, 08 Sep 2011 09:57:23 +0200 Subject: Cannot stop Varnish from caching 403 responses In-Reply-To: <4E64D31A.9080804@gmail.com> References: <4E609276.6010200@geizhals.at> <4E64D31A.9080804@gmail.com> Message-ID: <4E687563.7080502@geizhals.at> On 05.09.2011 15:48, Lars J?rgensen wrote: > if (beresp.status >= 400) { > return (hit_for_pass); > } > > You can go lower than that, but I still want to cache redirects (3xx). That worked great, thanks. We also don't want to cache redirects in many cases, so we are using 300 now. We're still considering an exemption for 404 though. Regards, Marinos From Jeff.Stroomer at disney.com Thu Sep 8 17:20:21 2011 From: Jeff.Stroomer at disney.com (Stroomer, Jeff) Date: Thu, 8 Sep 2011 10:20:21 -0700 Subject: caching/hiding 302 redirects Message-ID: <4F86EEFDF44D2D41951D491B61B7D4332486B9CD2B@SM-CALA-VXMB06B.swna.wdpr.disney.com> Varnish folks, Apologies for the repost, but I'm still stuck, and close to running out of ideas. I'm trying to combine Varnish with a Tomcat servlet to make a fancy reverse proxy. The idea is that Varnish forwards a URL to the servlet, which uses a 302 redirect to tell Varnish what the "real" URL should be. For this to work I need Varnish to do two things: 1) cache the 302 redirects from the servlet, and also 2) hide the redirect from the client. Rob S pointed me to the following trac ticket, which addresses exactly this problem: https://www.varnish-cache.org/trac/ticket/411 But when I use the solution described in the ticket, Varnish errors out. (To be precise, it gets into my vcl_deliver, and then dies with an INCOMPLETE AT error message.) Here is the relevant part of my default.vcl: sub vcl_hit { if (obj.http.X-Magic-Redirect == "1") { set req.url = obj.http.Location; restart; } } sub vcl_fetch { if (beresp.status == 302) { set beresp.http.X-Magic-Redirect = "1"; return(deliver); } } sub vcl_deliver { if (resp.http.X-Magic-Redirect == "1") { unset resp.http.X-Magic-Redirect; restart; } return(deliver); } So far as I can tell, I have duplicated the solution from the ticket exactly (except for vcl syntax updates corresponding to the evolution of varnish since the ticket was filed). Here's what I see in varnishlog: 0 CLI - Rd ping 0 CLI - Wr 200 PONG 1315497816 1.0 12 SessionOpen c ::1 57092 :6081 12 ReqStart c ::1 57092 611991685 12 RxRequest c GET 12 RxURL c /create 12 RxProtocol c HTTP/1.1 12 RxHeader c Host: localhost:6081 12 RxHeader c User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.22) Gecko/20110905 Ubuntu/10.10 (maverick) Firefox/3.6.22 12 RxHeader c Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 12 RxHeader c Accept-Language: en-us,en;q=0.5 12 RxHeader c Accept-Encoding: gzip,deflate 12 RxHeader c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 12 RxHeader c Keep-Alive: 115 12 RxHeader c Connection: keep-alive 12 RxHeader c Cache-Control: max-age=0 12 VCL_call c recv 12 VCL_return c lookup 12 VCL_call c hash 12 VCL_return c hash 12 VCL_call c miss 12 VCL_return c fetch 14 BackendOpen b default 127.0.0.1 37812 127.0.0.1 8080 12 Backend c 14 default default 14 TxRequest b GET 14 TxURL b /urlredirector/#a02/create 14 TxProtocol b HTTP/1.1 14 TxHeader b Host: localhost:6081 14 TxHeader b User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.22) Gecko/20110905 Ubuntu/10.10 (maverick) Firefox/3.6.22 14 TxHeader b Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 14 TxHeader b Accept-Language: en-us,en;q=0.5 14 TxHeader b Accept-Encoding: gzip,deflate 14 TxHeader b Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 14 TxHeader b X-cip: ::1 14 TxHeader b X-Bucket: a02 14 TxHeader b X-Varnish: 611991685 14 RxProtocol b HTTP/1.1 14 RxStatus b 302 14 RxResponse b Moved Temporarily 14 RxHeader b Server: Apache-Coyote/1.1 14 RxHeader b Location: http://redirected-localhost:6081/production_webapp/production_env/create 14 RxHeader b Max-Age: 300 14 RxHeader b Content-Type: text/html 14 RxHeader b Content-Length: 86 14 RxHeader b Date: Thu, 08 Sep 2011 16:03:38 GMT 12 TTL c 611991685 RFC 120 1315497818 0 0 0 0 12 VCL_call c fetch 12 VCL_return c deliver 12 ObjProtocol c HTTP/1.1 12 ObjStatus c 302 12 ObjResponse c Moved Temporarily 12 ObjHeader c Server: Apache-Coyote/1.1 12 ObjHeader c Location: http://redirected-localhost:6081/production_webapp/production_env/create 12 ObjHeader c Max-Age: 300 12 ObjHeader c Content-Type: text/html 12 ObjHeader c Date: Thu, 08 Sep 2011 16:03:38 GMT 12 ObjHeader c X-Magic-Redirect: 1 14 Length b 86 14 BackendReuse b default 0 Debug - "INCOMPLETE AT: cnt_deliver(196)" 0 CLI - Rd vcl.load "boot" ./vcl.1P9zoqAU.so 0 CLI - Wr 200 Loaded "./vcl.1P9zoqAU.so" as "boot" 0 CLI - Rd vcl.use "boot" 0 CLI - Wr 200 0 CLI - Rd start 0 Debug - "Acceptor is epoll" 0 CLI - Wr 200 0 WorkThread - 0x7f587a7f1be0 start 0 WorkThread - 0x7f5879ff0be0 start 0 WorkThread - 0x7f58797efbe0 start 0 WorkThread - 0x7f5878feebe0 start 0 WorkThread - 0x7f58787edbe0 start 0 WorkThread - 0x7f5877fecbe0 start 0 WorkThread - 0x7f58777ebbe0 start 0 WorkThread - 0x7f5876feabe0 start 0 WorkThread - 0x7f58767e9be0 start 0 WorkThread - 0x7f5875fe8be0 start So far as I can tell, my servlet issues the 302 redirect, which gets me into my vcl_fetch subroutine. vcl_fetch gets me into vcl_deliver, which is where I try to do a restart. If, in my vcl_fetch, I do a restart instead of a return(deliver), then everything works beautifully (but, of course, then I don't get my 302 redirect cached). I'm running Varnish 2.1.3 under Ubuntu 10.10, but see the same problem under Varnish 2.1.4 on Ubuntu 11.04. Thanks in advance for the help. Jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: From Jeff.Stroomer at disney.com Thu Sep 8 20:38:40 2011 From: Jeff.Stroomer at disney.com (Stroomer, Jeff) Date: Thu, 8 Sep 2011 13:38:40 -0700 Subject: problem installing Varnish 3.0 on Ubuntu In-Reply-To: <4F86EEFDF44D2D41951D491B61B7D43324849876FC@SM-CALA-VXMB06B.swna.wdpr.disney.com> References: <4F86EEFDF44D2D41951D491B61B7D43324849876FC@SM-CALA-VXMB06B.swna.wdpr.disney.com> Message-ID: <4F86EEFDF44D2D41951D491B61B7D4332486B9CF69@SM-CALA-VXMB06B.swna.wdpr.disney.com> Varnish folks, I figured out that I need to pick up the 3.0 version of Varnish to pick up the fix for trac bug #411. But I'm running into a problem when I try the procedure described here: https://www.varnish-cache.org/installation/ubuntu I've tried this on the 10.10 and 11.04 versions of Ubuntu (64 and 32 bit respectively), but run into this problem when I do the apt-get update: Hit http://us.archive.ubuntu.com maverick-updates/restricted Sources Hit http://us.archive.ubuntu.com maverick-updates/universe Sources Err http://repo.varnish-cache.org maverick/varnish-3.0 amd64 Packages 404 Not Found Hit http://us.archive.ubuntu.com maverick-updates/multiverse Sources Hit http://us.archive.ubuntu.com maverick-updates/main amd64 Packages Hit http://us.archive.ubuntu.com maverick-updates/restricted amd64 Packages Hit http://us.archive.ubuntu.com maverick-updates/universe amd64 Packages Hit http://us.archive.ubuntu.com maverick-updates/multiverse amd64 Packages Err http://repo.varnish-cache.org maverick/varnish-3.0 amd64 Packages 404 Not Found Ign http://dl.google.com/linux/chrome/deb/ stable/main Translation-en_US Get:2 http://dl.google.com stable Release [1,347B] Get:3 http://dl.google.com stable/main amd64 Packages [1,201B] Fetched 2,746B in 22s (122B/s) W: Failed to fetch http://repo.varnish-cache.org/ubuntu/dists/maverick/varnish-3.0/binary-amd64/Packages.gz 404 Not Found W: Failed to fetch http://repo.varnish-cache.org/debian/dists/maverick/varnish-3.0/binary-amd64/Packages.gz 404 Not Found Any ideas? Thanks for the help, Jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: From insyte at gmail.com Thu Sep 8 20:46:17 2011 From: insyte at gmail.com (Ben Beuchler) Date: Thu, 8 Sep 2011 15:46:17 -0500 Subject: problem installing Varnish 3.0 on Ubuntu In-Reply-To: <4F86EEFDF44D2D41951D491B61B7D4332486B9CF69@SM-CALA-VXMB06B.swna.wdpr.disney.com> References: <4F86EEFDF44D2D41951D491B61B7D43324849876FC@SM-CALA-VXMB06B.swna.wdpr.disney.com> <4F86EEFDF44D2D41951D491B61B7D4332486B9CF69@SM-CALA-VXMB06B.swna.wdpr.disney.com> Message-ID: > I figured out that I need to pick up the 3.0 version of Varnish to pick up > the fix for trac bug #411.? But I?m running into a problem when I try the > procedure described here: > > https://www.varnish-cache.org/installation/ubuntu As indicated on that page, Varnish only provides deb packages for the LTS releases of Ubuntu. Maverick is not an LTS release. -Ben From Jeff.Stroomer at disney.com Thu Sep 8 21:05:11 2011 From: Jeff.Stroomer at disney.com (Stroomer, Jeff) Date: Thu, 8 Sep 2011 14:05:11 -0700 Subject: problem installing Varnish 3.0 on Ubuntu Message-ID: <4F86EEFDF44D2D41951D491B61B7D4332486B9CFA9@SM-CALA-VXMB06B.swna.wdpr.disney.com> > I figured out that I need to pick up the 3.0 version of Varnish to pick up > the fix for trac bug #411. But I'm running into a problem when I try the > procedure described here: > > https://www.varnish-cache.org/installation/ubuntu As indicated on that page, Varnish only provides deb packages for the LTS releases of Ubuntu. Maverick is not an LTS release. ________________________________ Ben, Thanks - now I know what an LTS release is. I also tried building from source but ran into a problem. I followed the instructions here https://www.varnish-cache.org/docs/trunk/installation/install.html#compiling-varnish-from-source Made sure I had all the dependencies satisfied, then ran autogen.sh, configure, and make. But the make fails with the following: make[4]: Entering directory `/home/jstroomer/downloads/varnish/varnish-3.0.0/lib/libvarnish' /bin/bash ../../libtool --tag=CC --mode=compile gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../.. -I../../include -DVARNISH_STATE_DIR='"/usr/local/var/varnish"' -g -O2 -pthread -Wextra -Wno-missing-field-initializers -Wno-sign-compare -MT libvarnish_la-version.lo -MD -MP -MF .deps/libvarnish_la-version.Tpo -c -o libvarnish_la-version.lo `test -f 'version.c' || echo './'`version.c libtool: compile: gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../.. -I../../include -DVARNISH_STATE_DIR=\"/usr/local/var/varnish\" -g -O2 -pthread -Wextra -Wno-missing-field-initializers -Wno-sign-compare -MT libvarnish_la-version.lo -MD -MP -MF .deps/libvarnish_la-version.Tpo -c version.c -fPIC -DPIC -o .libs/libvarnish_la-version.o version.c:37: fatal error: vcs_version.h: No such file or directory compilation terminated. make[4]: *** [libvarnish_la-version.lo] Error 1 make[4]: Leaving directory `/home/jstroomer/downloads/varnish/varnish-3.0.0/lib/libvarnish' make[3]: *** [all] Error 2 make[3]: Leaving directory `/home/jstroomer/downloads/varnish/varnish-3.0.0/lib/libvarnish' make[2]: *** [all-recursive] Error 1 make[2]: Leaving directory `/home/jstroomer/downloads/varnish/varnish-3.0.0/lib' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/home/jstroomer/downloads/varnish/varnish-3.0.0' make: *** [all] Error 2 Any thoughts? Jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: From Jeff.Stroomer at disney.com Thu Sep 8 21:44:53 2011 From: Jeff.Stroomer at disney.com (Stroomer, Jeff) Date: Thu, 8 Sep 2011 14:44:53 -0700 Subject: problem installing Varnish 3.0 on Ubuntu In-Reply-To: <4F86EEFDF44D2D41951D491B61B7D4332484987702@SM-CALA-VXMB06B.swna.wdpr.disney.com> References: <4F86EEFDF44D2D41951D491B61B7D4332484987702@SM-CALA-VXMB06B.swna.wdpr.disney.com> Message-ID: <4F86EEFDF44D2D41951D491B61B7D4332486B9D01F@SM-CALA-VXMB06B.swna.wdpr.disney.com> Update: I was able to build from source using the 3.0.1 tarball. Thanks for the help. Jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: From MSchurenko at airg.com Thu Sep 8 22:08:59 2011 From: MSchurenko at airg.com (Matt Schurenko) Date: Thu, 8 Sep 2011 15:08:59 -0700 Subject: Proper cache allocation (nuked objects but sm_bfree) In-Reply-To: <20110908074013.GJ3515@freud.kly.no> References: <20110908074013.GJ3515@freud.kly.no> Message-ID: <8B5C6F575422414AA91B46C454126B6C02EB282065@EXCHMVS.exchange.airg> I've been trying to understand these stats myself. A couple of questions regarding your post Kristian: 1) Where can I find this _nbytes statistic you speak of? This is all I get from running varnishstat on varnish 2.1.5: [root at varnish-test ~]# varnishstat -1 | grep _nbytes sma_nbytes 0 . SMA outstanding bytes sms_nbytes 0 . SMS outstanding bytes 2) If sm_bfree and sm_balloc are indeed counters wouldn't they go up indefinitely? I find that if I add those two values together the sum equals the size of the cache I specified (in my case 40GB): [root at varnish-test ~]# varnishstat -1 | grep ^sm_bfree sm_bfree 32433844224 . bytes free [root at varnish-test ~]# varnishstat -1 | grep ^sm_balloc sm_balloc 10515988480 . bytes allocated [root at varnish-test ~]# echo '(32433844224 + 10515988480) / 1024 / 1024 / 1024' | bc 40 [root at varnish-test ~]# ps -ef | grep varnish root 3574 1 0 Aug24 ? 00:00:00 /usr/local/sbin/varnishd -s file,/tmp/varnish-cache,40G -T 127.0.0.1:2000 -a 0.0.0.0:80 -t 604800 -f /usr/local/etc/varnish/default.vcl Thanks, Matt Schurenko Systems Administrator airG share your world Suite 710, 1133 Melville Street Vancouver, BC V6E 4E5 P: +1.604.408.2228 F: +1.866.874.8136 E: MSchurenko at airg.com W: www.airg.com -----Original Message----- From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Kristian Lyngstol Sent: September-08-11 12:40 AM To: Damon Snyder Cc: varnish-misc at varnish-cache.org Subject: Re: Proper cache allocation (nuked objects but sm_bfree) On Wed, Sep 07, 2011 at 11:46:34AM -0700, Damon Snyder wrote: > According to the docs, the key statistic to look at is n_lru_nuked. This > value is constantly increasing. Every time you run 'varnishstat -1 -f > n_lru_nuked' the value changes. However, the value of sm_bfree seems to > always show some space available: > > varnishstat -1 -f n_lru_nuked,sm_bfree,sm_balloc > n_lru_nuked 135193763 . N LRU nuked objects > sm_balloc 5468946432 . bytes allocated > sm_bfree 2047246336 . bytes free sm_bfree is a counter of how much memory has been freed, not how much is available. Every time an object is removed, expired, etc, this will increase, and it is never reduced. _balloc is the counter part of that, and is increased every time something is allocated, and never reduced. >From these numbers you can calculate how much memory is currently used: 5468946432 - 2047246336 = 3421700096 (a little over 3GB) However, you don't have to actually do that yourself, as _nbytes is doing exactly that. An important detail is that the size specified in the -s arguments is /not/ the total memory footprint varnish will have. It is only the total cache size for actual data, not counting overhead. For Varnish 2.1, we know that there's an additional overhead of slightly over 1kB for each object stored, assuming 64-bit systems (your millage may vary, but this gives you an idea). On top of that is a bit of data for things like threads and sessions, but I rarely take that into consideration myself as it will be a practically constant size measured in MB. Assuming the memory-footprint for each thread is 10kB (it's unlikely that it's that large, specially considering copy-on-write and whatnot), 1000 threads will give a constant overhead of 10MB, so not something to consider. However, 10 million objects will give 10GB of overhead, so that should be accounted for when you decide how much memory the cache will use with -s. Examples I've run with is -s malloc,24GB and -s malloc,28GB on two different sites, both running on a 32GB-system but with different average object-size. Hopefully this helps you figure out what's going on. - Kristian _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From tfheen at varnish-software.com Fri Sep 9 06:05:06 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Fri, 09 Sep 2011 08:05:06 +0200 Subject: problem installing Varnish 3.0 on Ubuntu In-Reply-To: (Ben Beuchler's message of "Thu, 8 Sep 2011 15:46:17 -0500") References: <4F86EEFDF44D2D41951D491B61B7D43324849876FC@SM-CALA-VXMB06B.swna.wdpr.disney.com> <4F86EEFDF44D2D41951D491B61B7D4332486B9CF69@SM-CALA-VXMB06B.swna.wdpr.disney.com> Message-ID: <87aaae9rot.fsf@qurzaw.varnish-software.com> ]] Ben Beuchler | > I figured out that I need to pick up the 3.0 version of Varnish to pick up | > the fix for trac bug #411.? But I?m running into a problem when I try the | > procedure described here: | > | > https://www.varnish-cache.org/installation/ubuntu | | As indicated on that page, Varnish only provides deb packages for the | LTS releases of Ubuntu. Maverick is not an LTS release. That said, I believe the lucid packages should work just fine on maverick. -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From damon at huddler-inc.com Thu Sep 8 22:42:25 2011 From: damon at huddler-inc.com (Damon Snyder) Date: Thu, 8 Sep 2011 15:42:25 -0700 Subject: Proper cache allocation (nuked objects but sm_bfree) In-Reply-To: <20110908074013.GJ3515@freud.kly.no> References: <20110908074013.GJ3515@freud.kly.no> Message-ID: Hi Kristian, Thanks for the explanation. However, I'm still struggling with how to make sense of the first part of my post. Specifically, how do I know how much of my cache has been used and how severely under allocated is it? If there is some memory free, why is n_lru_nuked increasing at a rate of ~50/s? I have another varnish box with 24GB of RAM and a 20GB cache file with the following stats: varnishstat -1 -f n_lru_nuked,sm_bfree,sm_balloc n_lru_nuked 11350022 . N LRU nuked objects sm_balloc 4124241920 . bytes allocated sm_bfree 17350594560 . bytes free According to the calculation above then the used memory would be negative. Am I misunderstanding something? Thanks, Damon On Thu, Sep 8, 2011 at 12:40 AM, Kristian Lyngstol < kristian at varnish-software.com> wrote: > On Wed, Sep 07, 2011 at 11:46:34AM -0700, Damon Snyder wrote: > > According to the docs, the key statistic to look at is n_lru_nuked. This > > value is constantly increasing. Every time you run 'varnishstat -1 -f > > n_lru_nuked' the value changes. However, the value of sm_bfree seems to > > always show some space available: > > > > varnishstat -1 -f n_lru_nuked,sm_bfree,sm_balloc > > n_lru_nuked 135193763 . N LRU nuked objects > > sm_balloc 5468946432 . bytes allocated > > sm_bfree 2047246336 . bytes free > > sm_bfree is a counter of how much memory has been freed, not how much is > available. Every time an object is removed, expired, etc, this will > increase, and it is never reduced. _balloc is the counter part of that, > and is increased every time something is allocated, and never reduced. > From these numbers you can calculate how much memory is currently used: > > 5468946432 - 2047246336 = 3421700096 (a little over 3GB) > > However, you don't have to actually do that yourself, as _nbytes is > doing exactly that. > > An important detail is that the size specified in the -s arguments is > /not/ the total memory footprint varnish will have. It is only the total > cache size for actual data, not counting overhead. For Varnish 2.1, we > know that there's an additional overhead of slightly over 1kB for each > object stored, assuming 64-bit systems (your millage may vary, but this > gives you an idea). > > On top of that is a bit of data for things like threads and sessions, > but I rarely take that into consideration myself as it will be a > practically constant size measured in MB. Assuming the memory-footprint > for each thread is 10kB (it's unlikely that it's that large, specially > considering copy-on-write and whatnot), 1000 threads will give a > constant overhead of 10MB, so not something to consider. > > However, 10 million objects will give 10GB of overhead, so that should > be accounted for when you decide how much memory the cache will use with > -s. Examples I've run with is -s malloc,24GB and -s malloc,28GB on two > different sites, both running on a 32GB-system but with different > average object-size. > > Hopefully this helps you figure out what's going on. > > - Kristian > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kristian at varnish-software.com Fri Sep 9 13:50:18 2011 From: kristian at varnish-software.com (Kristian Lyngstol) Date: Fri, 9 Sep 2011 15:50:18 +0200 Subject: Proper cache allocation (nuked objects but sm_bfree) In-Reply-To: <20110908074013.GJ3515@freud.kly.no> References: <20110908074013.GJ3515@freud.kly.no> Message-ID: <20110909135018.GA8315@luke.kly.no> Ok, I'll reply to my own mail, since I've gotten two, distinct, replies, both which point out a few mistakes I made. On Thu, Sep 08, 2011 at 09:40:13AM +0200, Kristian Lyngstol wrote: > On Wed, Sep 07, 2011 at 11:46:34AM -0700, Damon Snyder wrote: > > varnishstat -1 -f n_lru_nuked,sm_bfree,sm_balloc > > n_lru_nuked 135193763 . N LRU nuked objects > > sm_balloc 5468946432 . bytes allocated > > sm_bfree 2047246336 . bytes free > > sm_bfree is a counter of how much memory has been freed, not how much is > available. Every time an object is removed, expired, etc, this will > increase, and it is never reduced. _balloc is the counter part of that, > and is increased every time something is allocated, and never reduced. > From these numbers you can calculate how much memory is currently used: > > 5468946432 - 2047246336 = 3421700096 (a little over 3GB) Ok, for Varnish 3 and -s malloc in Varnish 2, this is true. For -s file, which you are using, it is NOT true. Terribly sorry for the confusion. In Varnish 2, sma_balloc and sma_bfree are counters that never are reduced, and sma_nbytes is the value of sma_balloc-sma_bfree. In Varnish 2, sm_balloc is the size of allocated data, while sm_bfree is the size of data you can allocate. They both go up and down, and there is no sm_nbytes. So, to correct myself: You are using roughly 5GB of space and have roughly 2GB of space available. That does not necessarily map directly to real memory. If you are concerned with disk I/O, I would use -s malloc on 2.1. And regardless, I would but /var/lib/varnish/ on a tmpfs (but make sure you don't also use -s file,/var/lib/varnish/..., or you'll have a bit of a problem). The tmpfs for /var/lib/varnish/ is meant for the 80MB shmlog, to ensure it's never flushed to disk. As for why you have lru nukes, I really can't say. If you have a few very large files, that could explain it, or if your size usage goes up and down. I haven't done the math for 8GB systems on 2.1 recently, but given that you're using -s file,7GB and the number of objects you have indicate a little less than 200MB of overhead, + whatever else (let's say 100MB for threads, sessions and just misc), you're cutting it a bit close. But then again, this could just be the other reasons stated above. The rest of my mail (about the 1kB+/object oberhead) holds true. - Kristian From jeroen.ooms at stat.ucla.edu Fri Sep 9 18:04:31 2011 From: jeroen.ooms at stat.ucla.edu (Jeroen Ooms) Date: Fri, 9 Sep 2011 11:04:31 -0700 Subject: max simultaneous connections per client IP In-Reply-To: References: Message-ID: I posted the question below a couple of weeks ago, but unfortunately there wasn't a single answer. I would still be very interested to hear if this would be possible, or if there is an alternative approach in which varnish can to some extend help me prevent a dos from a single user. Thank you very much. On Wed, Aug 24, 2011 at 3:06 PM, Jeroen Ooms wrote: I am using the .max_connections config to limit the total maximum > simultaneous connections to the backend server. Is it also possible to have > a maximum on the number of simultaneous connections per client IP? So e.g. > have .max_connections = 10, but only 2 per client IP. > > Thanks, > > Jeroen > -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Fri Sep 9 18:17:52 2011 From: perbu at varnish-software.com (Per Buer) Date: Fri, 9 Sep 2011 20:17:52 +0200 Subject: max simultaneous connections per client IP In-Reply-To: References: Message-ID: Hi, On Thu, Aug 25, 2011 at 12:06 AM, Jeroen Ooms wrote: > I am using the .max_connections config to limit the total maximum > simultaneous connections to the backend server. Is it also possible to have > a maximum on the number of simultaneous connections per client IP? So e.g. > have .max_connections = 10, but only 2 per client IP. No, this is not possible. -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From damon at huddler-inc.com Fri Sep 9 21:21:43 2011 From: damon at huddler-inc.com (Damon Snyder) Date: Fri, 9 Sep 2011 14:21:43 -0700 Subject: Proper cache allocation (nuked objects but sm_bfree) In-Reply-To: <20110909135018.GA8315@luke.kly.no> References: <20110908074013.GJ3515@freud.kly.no> <20110909135018.GA8315@luke.kly.no> Message-ID: Hi Kristian, Thanks again for the explanation. It forced me to look at the code to try and get a better sense for what is going on. Can you explain the meaning of n_smf_frag and n_smf_large? They are missing explanations here https://www.varnish-cache.org/trac/wiki/StatsExplained. >From the code, it appears that they are just counters. On one of our boxes n_smf_large is > 10k and there is no nuking. On the other n_smf_large is close to 0 and there is a lot of nuking. I was trying to understand the relationship. Maybe we should re-evaluate our storage type. From a quick look at the malloc storage type, it looks like objects of arbitrary size are being allocated as opposed to rounding up to the next nearest page size (more space efficient) and the large objects aren't confined to the last bucket. Thanks, Damon On Fri, Sep 9, 2011 at 6:50 AM, Kristian Lyngstol < kristian at varnish-software.com> wrote: > Ok, I'll reply to my own mail, since I've gotten two, distinct, replies, > both which point out a few mistakes I made. > > On Thu, Sep 08, 2011 at 09:40:13AM +0200, Kristian Lyngstol wrote: > > On Wed, Sep 07, 2011 at 11:46:34AM -0700, Damon Snyder wrote: > > > varnishstat -1 -f n_lru_nuked,sm_bfree,sm_balloc > > > n_lru_nuked 135193763 . N LRU nuked objects > > > sm_balloc 5468946432 . bytes allocated > > > sm_bfree 2047246336 . bytes free > > > > sm_bfree is a counter of how much memory has been freed, not how much is > > available. Every time an object is removed, expired, etc, this will > > increase, and it is never reduced. _balloc is the counter part of that, > > and is increased every time something is allocated, and never reduced. > > From these numbers you can calculate how much memory is currently used: > > > > 5468946432 - 2047246336 = 3421700096 (a little over 3GB) > > Ok, for Varnish 3 and -s malloc in Varnish 2, this is true. For -s file, > which you are using, it is NOT true. Terribly sorry for the confusion. > > In Varnish 2, sma_balloc and sma_bfree are counters that never are > reduced, and sma_nbytes is the value of sma_balloc-sma_bfree. > > In Varnish 2, sm_balloc is the size of allocated data, while sm_bfree is > the size of data you can allocate. They both go up and down, and there > is no sm_nbytes. > > So, to correct myself: > > You are using roughly 5GB of space and have roughly 2GB of space > available. That does not necessarily map directly to real memory. > > If you are concerned with disk I/O, I would use -s malloc on 2.1. And > regardless, I would but /var/lib/varnish/ on a tmpfs (but make sure you > don't also use -s file,/var/lib/varnish/..., or you'll have a bit of a > problem). The tmpfs for /var/lib/varnish/ is meant for the 80MB shmlog, > to ensure it's never flushed to disk. > > As for why you have lru nukes, I really can't say. If you have a few > very large files, that could explain it, or if your size usage goes up > and down. > > I haven't done the math for 8GB systems on 2.1 recently, but given that > you're using -s file,7GB and the number of objects you have indicate a > little less than 200MB of overhead, + whatever else (let's say 100MB for > threads, sessions and just misc), you're cutting it a bit close. But > then again, this could just be the other reasons stated above. > > The rest of my mail (about the 1kB+/object oberhead) holds true. > > - Kristian > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon.gilbert at net-entwicklung.de Mon Sep 12 08:07:17 2011 From: jon.gilbert at net-entwicklung.de (jon.gilbert at net-entwicklung.de) Date: Mon, 12 Sep 2011 10:07:17 +0200 (CEST) Subject: How to pass request body using Varnish fetch? Message-ID: <928479624.16275.1315814837976.JavaMail.open-xchange@oxgw09.schlund.de> Hi!? I'm afraid I am fairly new to varnish but I have a problem whch I cannot find a solution to anywhere (yet): Varnish is set up to cache GET requests. We have some requests which have so many parameters that we decided to pass them in the body of the request. This works fine when we bypass Varnish but when we go through Varnish (for caching), the request is passed on without the body, so the service behind Varnish fails. I know we could use POST, but we want to GET data. I also know that Varnish CAN pass the request body on if we use pass mode but as far as I can see, requests made in pass mode aren't cached. I've already put a hash into the url so that when things work, we will actually get the correct data from cache (as far as the url goes the calls would otherwise all look to be the same). The problem now is "just" how to rewrite vcl_fetch to pass on the request body to the webserver? Any hints and tips welcome! ? Thanks in advance Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: From mls at pooteeweet.org Mon Sep 12 11:17:28 2011 From: mls at pooteeweet.org (Lukas Kahwe Smith) Date: Mon, 12 Sep 2011 13:17:28 +0200 Subject: efficient strategy to cache warm multiple varnish servers In-Reply-To: References: Message-ID: <21A54F6D-9BDE-4B14-BDAB-07E98FC807AB@pooteeweet.org> Hi, Reposting this once more in the hopes that maybe now someone has a reply .. On Jul 13, 2011, at 08:28 , Lukas Kahwe Smith wrote: > Hi, > > If we ever have to restart our setup from scratch because of some big crash or because we are rolling out a new version of the site, I am wondering how to most efficiently warm all the varnish servers. Do I just warm one server and then rsync? Searching for "varnish cache warm" and didnt find anything very detailed. > > This link seems useful in the general context of cache warming scripts: > https://www.varnish-cache.org/trac/wiki/VCLExampleHashAlwaysMiss > > This one however seems to sound like I can chain my varnish caches which could be what I need to do: > https://www.varnish-cache.org/trac/wiki/VCLExampleHashIgnoreBusy > > Any other RTFM links much appreciated :) > > regards, > Lukas Kahwe Smith > mls at pooteeweet.org > > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc Lukas Kahwe Smith mls at pooteeweet.org From mhettwer at team.mobile.de Mon Sep 12 11:44:28 2011 From: mhettwer at team.mobile.de (Hettwer, Marian) Date: Mon, 12 Sep 2011 12:44:28 +0100 Subject: efficient strategy to cache warm multiple varnish servers In-Reply-To: Message-ID: Hi there, On 13.07.11 08:28, "Lukas Kahwe Smith" wrote: >Hi, > >If we ever have to restart our setup from scratch because of some big >crash or because we are rolling out a new version of the site, I am >wondering how to most efficiently warm all the varnish servers. Do I just >warm one server and then rsync? Searching for "varnish cache warm" and >didnt find anything very detailed. > >This link seems useful in the general context of cache warming scripts: >https://www.varnish-cache.org/trac/wiki/VCLExampleHashAlwaysMiss Since I know which url's I need to cache, I'm using a script based on the example above. Basically I'm running all the time against varnish and compare the Max-Age header and the Age header. If Max-Age - Age less than 600 seconds, my script fires a second request against varnish with the preconfigured special X- header. Therefor I always have a warm cache. However, this only works since I can ask my application "hej, gimme a list of url's we want to cache". And additionally I mess up my cache hit / miss ratio since my script will very often generate cache hits. HTH, Marian Example: cat $TEMP_FILE | while read _url; do for i in "$HEADER_1" "$HEADER_2"; do HEADER_CACHE=$(wget -O /dev/null -S --header "$i" --header "Host:xxxxxx" "http://localhost/${_url}" 2>&1) MAX_AGE=$(echo "${HEADER_CACHE}"|awk -F\= '/max-age/{print $2}') AGE=$(echo "${HEADER_CACHE}"|awk -F\ '/Age/{print $2}') AGE_DIFF=$(($MAX_AGE-$AGE)) [ ! -z $DEBUG ] && echo $_url $MAX_AGE $AGE $AGE_DIFF [ $AGE -eq 0 ] || [ $AGE_DIFF -le $TIME_AFTER ] && \ ( [ ! -z $DEBUG ] && echo -e $RED "^ cache is being updated" $END wget -O /dev/null -S --header "${MAGIC}" --header "$i" --header "Host:xxxxx" "http://localhost/${_url}" 2>/dev/nulli ) done Done From mls at pooteeweet.org Mon Sep 12 11:58:06 2011 From: mls at pooteeweet.org (Lukas Kahwe Smith) Date: Mon, 12 Sep 2011 13:58:06 +0200 Subject: efficient strategy to cache warm multiple varnish servers In-Reply-To: References: Message-ID: <9DF0F67E-A7D3-4B9B-8EAE-F0E507C7ABA1@pooteeweet.org> On Sep 12, 2011, at 13:44 , Hettwer, Marian wrote: > Hi there, > > > > On 13.07.11 08:28, "Lukas Kahwe Smith" wrote: > >> Hi, >> >> If we ever have to restart our setup from scratch because of some big >> crash or because we are rolling out a new version of the site, I am >> wondering how to most efficiently warm all the varnish servers. Do I just >> warm one server and then rsync? Searching for "varnish cache warm" and >> didnt find anything very detailed. >> >> This link seems useful in the general context of cache warming scripts: >> https://www.varnish-cache.org/trac/wiki/VCLExampleHashAlwaysMiss > > Since I know which url's I need to cache, I'm using a script based on the > example above. ok thx. what i am however wondering about is how i can prevent having to call my app once per url for each varnish to get them all warmed up. is there some way to warm multiple varnish servers but execute just one frontend request per url? regards, Lukas Kahwe Smith mls at pooteeweet.org From perbu at varnish-software.com Mon Sep 12 12:13:15 2011 From: perbu at varnish-software.com (Per Buer) Date: Mon, 12 Sep 2011 14:13:15 +0200 Subject: efficient strategy to cache warm multiple varnish servers In-Reply-To: <9DF0F67E-A7D3-4B9B-8EAE-F0E507C7ABA1@pooteeweet.org> References: <9DF0F67E-A7D3-4B9B-8EAE-F0E507C7ABA1@pooteeweet.org> Message-ID: On Mon, Sep 12, 2011 at 1:58 PM, Lukas Kahwe Smith wrote: > > ok thx. > > what i am however wondering about is how i can prevent having to call my > app once per url for each varnish to get them all warmed up. is there some > way to warm multiple varnish servers but execute just one frontend request > per url? > You can probably chain them, but seriously, why would you? You would of course script these actions, and executing them one after the other. -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From mls at pooteeweet.org Mon Sep 12 12:25:01 2011 From: mls at pooteeweet.org (Lukas Kahwe Smith) Date: Mon, 12 Sep 2011 14:25:01 +0200 Subject: efficient strategy to cache warm multiple varnish servers In-Reply-To: References: <9DF0F67E-A7D3-4B9B-8EAE-F0E507C7ABA1@pooteeweet.org> Message-ID: <3904FA0A-5F95-4F03-A918-C79AE9633ACA@pooteeweet.org> On Sep 12, 2011, at 14:13 , Per Buer wrote: > On Mon, Sep 12, 2011 at 1:58 PM, Lukas Kahwe Smith wrote: > > ok thx. > > what i am however wondering about is how i can prevent having to call my app once per url for each varnish to get them all warmed up. is there some way to warm multiple varnish servers but execute just one frontend request per url? > > You can probably chain them, but seriously, why would you? You would of course script these actions, and executing them one after the other. well the use case is deploying a new version of the app or some horrible issue causing the varnishes to need a full reset and so quickly needing to get all the varnish instances uptodate. the problem with just "chaining" them is that varnish is configured to strip some of the headers etc .. regards, Lukas Kahwe Smith mls at pooteeweet.org From straightflush at gmail.com Mon Sep 12 15:07:16 2011 From: straightflush at gmail.com (AD) Date: Mon, 12 Sep 2011 11:07:16 -0400 Subject: PRIV_CALL issue in vmod Message-ID: hello, I am trying to use the PRIV_CALL capability of the vmods in order to cache an expensive operation used later in the vcl itself. I am running into an issue where a PRIV_CALL is being re-used for each request acting more like PRIV_VCL. Not sure if i am missing a "free" call somewhere but the below snippet reproduces the issue from my vmod. When hitting varnish with multiple different URLs the very first call logs "str is NULL" but every other request logs "str is VALID" so priv->priv never gets unset for future calls. vcc file: Init init_function Function STRING get_str1(PRIV_CALL) Function STRING get_str2(PRIV_CALL) source file: int init_function(struct vmod_priv *priv, const struct VCL_conf *conf) { } char * str_init(struct sess *sp, struct vmod_priv *priv) { char * str ; str = priv->priv; if (str == NULL) { syslog(LOG_INFO, "str is NULL, setting value\n"); priv->priv = str = malloc(256); priv->priv = strdup("MY VALUE"); priv->free = free ; } else { syslog(LOG_INFO, "str is VALID, returning value\n"); return str; } } const char * vmod_get_str1(struct sess *sp, struct vmod_priv *priv) { return str_init(sp,priv); } const char * vmod_get_str2(struct sess *sp, struct vmod_priv *priv) { return str_init(sp,priv); } -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon.gilbert at net-entwicklung.de Mon Sep 12 17:27:12 2011 From: jon.gilbert at net-entwicklung.de (jon.gilbert at net-entwicklung.de) Date: Mon, 12 Sep 2011 19:27:12 +0200 (CEST) Subject: How to pass request body using Varnish fetch? In-Reply-To: <928479624.16275.1315814837976.JavaMail.open-xchange@oxgw09.schlund.de> References: <928479624.16275.1315814837976.JavaMail.open-xchange@oxgw09.schlund.de> Message-ID: <455267210.21864.1315848433004.JavaMail.open-xchange@oxgw09.schlund.de> Is it possible that this is currently *not* possible? (ANY feedback welcome ;) In the FAQs it says that "In 2.0 and later, pass will handle the request body correctly". So someone has implemented passing the request body - is there anyone on the list that can put in a feature request, that this also be done for fetch mode? Didn't find it in trac :(? ? ? Possible alternative: is there any way to cache calls made in pass mode? ? Thx. Jon ? ? ? "jon.gilbert at net-entwicklung.de" hat am 12. September 2011 um 10:07 geschrieben: > > Hi!? > I'm afraid I am fairly new to varnish but I have a problem whch I cannot find a solution to anywhere (yet): Varnish is set up to cache GET requests. We have some requests which have so many parameters that we decided to pass them in the body of the request. This works fine when we bypass Varnish but when we go through Varnish (for caching), the request is passed on without the body, so the service behind Varnish fails. > I know we could use POST, but we want to GET data. I also know that Varnish CAN pass the request body on if we use pass mode but as far as I can see, requests made in pass mode aren't cached. I've already put a hash into the url so that when things work, we will actually get the correct data from cache (as far as the url goes the calls would otherwise all look to be the same). > The problem now is "just" how to rewrite vcl_fetch to pass on the request body to the webserver? Any hints and tips welcome! > ? > Thanks in advance > Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: From footplus at gmail.com Mon Sep 12 18:10:09 2011 From: footplus at gmail.com (=?UTF-8?B?QXVyw6lsaWVu?=) Date: Mon, 12 Sep 2011 20:10:09 +0200 Subject: Persistent storage usage Message-ID: Hello, I'm using persistent storage on several caches, and i'm wondering how to monitor the cache usage (in bytes). There doesn't seem to be a stat about that in varnishstat. Is it implemented ? Can it be simply added ? Thanks, Best regards, -- Aur?lien Guillaume -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Mon Sep 12 21:48:09 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 12 Sep 2011 21:48:09 +0000 Subject: PRIV_CALL issue in vmod In-Reply-To: Your message of "Mon, 12 Sep 2011 11:07:16 -0400." Message-ID: <2632.1315864089@critter.freebsd.dk> In message , AD writes: > I am trying to use the PRIV_CALL capability of the vmods in order to cache >an expensive operation used later in the vcl itself. That is not how it works. PRIV_CALL is for caching things for the next time this call is executed. The canonical example is a caching a compiled regexp. What you need is a "PRIV_SESS" or "PRIV_REQ" variable, and I'm beating my head against the wall trying to find the best way to implement that as we're speaking... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From straightflush at gmail.com Mon Sep 12 22:09:36 2011 From: straightflush at gmail.com (AD) Date: Mon, 12 Sep 2011 18:09:36 -0400 Subject: PRIV_CALL issue in vmod In-Reply-To: <2632.1315864089@critter.freebsd.dk> References: <2632.1315864089@critter.freebsd.dk> Message-ID: Ah i see. So basically if we have an expensive operation, like a DB call (as an example), we would have to re-do it every time we use the Vmod function inside the VCL. The issue i am having is i have a vmod, where i would like to return a complex struct but i cant seem to be able to do that due to limitations on the return types. So i was planning on writing accessor methods (one for each element of the struct) but for each accessor, i would need to re-query the database to get the value which was trying to avoid. Example: typedef struct _mydata { int val1; int val2; } mydata ; i would love to return mydata from the vmod, but instead i need to write vmod_get_int1, vmod_get_int2 to get at these. The issue is to populate this struct is "expensive". Is there a better way around this issue ? Cheers. On Mon, Sep 12, 2011 at 5:48 PM, Poul-Henning Kamp wrote: > In message < > CANZWjw5t+Q0iEa-5rN6xriuy6Kw7i8Auq-z_vGj5uBKw9_gpog at mail.gmail.com> > , AD writes: > > > I am trying to use the PRIV_CALL capability of the vmods in order to > cache > >an expensive operation used later in the vcl itself. > > That is not how it works. > > PRIV_CALL is for caching things for the next time this call is executed. > > The canonical example is a caching a compiled regexp. > > What you need is a "PRIV_SESS" or "PRIV_REQ" variable, and I'm beating > my head against the wall trying to find the best way to implement > that as we're speaking... > > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From straightflush at gmail.com Tue Sep 13 03:37:21 2011 From: straightflush at gmail.com (AD) Date: Mon, 12 Sep 2011 23:37:21 -0400 Subject: invalid ban url syntax Message-ID: hello, when trying to ban a req.url from the CLI i get the following error, any ideas? ban req.http.host == "www.domain.com" && req.url ~ "\.png" 100 85 Unknown request. Type 'help' for more info. Syntax Error: Invalid backslash sequence Removing backslash ban req.http.host == "www.domain.com && req.url ~ ".png" 106 25 Wrong number of arguments Cheers. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bedis9 at gmail.com Tue Sep 13 05:43:39 2011 From: bedis9 at gmail.com (Baptiste) Date: Tue, 13 Sep 2011 07:43:39 +0200 Subject: invalid ban url syntax In-Reply-To: References: Message-ID: Hi, You also removed the " after your domain name. cheers On Tue, Sep 13, 2011 at 5:37 AM, AD wrote: > hello, when trying to ban a req.url from the CLI i get the following error, > any ideas? > ban req.http.host == "www.domain.com" && req.url ~ "\.png" > 100 85 > Unknown request. > Type 'help' for more info. > Syntax Error: Invalid backslash sequence > Removing backslash > ban req.http.host == "www.domain.com && req.url ~ ".png" > 106 25 > Wrong number of arguments > Cheers. > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > From tfheen at varnish-software.com Tue Sep 13 06:52:33 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 13 Sep 2011 08:52:33 +0200 Subject: How to pass request body using Varnish fetch? In-Reply-To: <928479624.16275.1315814837976.JavaMail.open-xchange@oxgw09.schlund.de> (jon's message of "Mon, 12 Sep 2011 10:07:17 +0200 (CEST)") References: <928479624.16275.1315814837976.JavaMail.open-xchange@oxgw09.schlund.de> Message-ID: <87d3f5x7bi.fsf@qurzaw.varnish-software.com> ]] | I'm afraid I am fairly new to varnish but I have a problem whch I | cannot find a solution to anywhere (yet): Varnish is set up to cache | GET requests. We have some requests which have so many parameters that | we decided to pass them in the body of the request. This makes no sense, you should just add the parameters to the URL instead of putting them in the body. Cheers, -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From phk at phk.freebsd.dk Tue Sep 13 07:26:37 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 13 Sep 2011 07:26:37 +0000 Subject: PRIV_CALL issue in vmod In-Reply-To: Your message of "Mon, 12 Sep 2011 18:09:36 -0400." Message-ID: <2910.1315898797@critter.freebsd.dk> In message , AD writes: >Ah i see. So basically if we have an expensive operation, like a DB call (as >an example), we would have to re-do it every time we use the Vmod function >inside the VCL. Until I get the PRIV_SESS stuff working, yes. And yes, accessor functions is the way to go. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From lampe at hauke-lampe.de Tue Sep 13 12:43:24 2011 From: lampe at hauke-lampe.de (Hauke Lampe) Date: Tue, 13 Sep 2011 14:43:24 +0200 Subject: invalid ban url syntax In-Reply-To: References: Message-ID: <4E6F4FEC.1070307@hauke-lampe.de> On 09/13/2011 05:37 AM, AD wrote: > hello, when trying to ban a req.url from the CLI i get the following error, > any ideas? > Syntax Error: Invalid backslash sequence Which version of varnish do you use? You might have to use double backslashes. See: https://www.varnish-cache.org/trac/ticket/755 Hauke. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: OpenPGP digital signature URL: From paulo at aliancaproject.com Tue Sep 13 12:49:45 2011 From: paulo at aliancaproject.com (Paulo Paracatu) Date: Tue, 13 Sep 2011 09:49:45 -0300 Subject: varnishd panic message Message-ID: Hello, I was monitoring my messages log and varnish is producing a lot of "panic messages" like this one: Sep 13 12:43:18 cache5 varnishd[970965]: Child (987439) died signal=6 Sep 13 12:43:18 cache5 varnishd[970965]: Child (987439) Panic message: Assert error in Tcheck(), cache.h line 977: Condition((t.b) != 0) not true. thread = (cache-worker) ident = Linux,2.6.18-238.9.1.el5.centos.plus,x86_64,-smalloc,-smalloc,-hcritbit,epoll Backtrace: 0x42cea6: /usr/sbin/varnishd [0x42cea6] 0x4284ef: /usr/sbin/varnishd [0x4284ef] 0x4304d5: /usr/sbin/varnishd(RES_BuildHttp+0x85) [0x4304d5] 0x416f3b: /usr/sbin/varnishd [0x416f3b] 0x417e0b: /usr/sbin/varnishd(CNT_Session+0x7fb) [0x417e0b] 0x42f698: /usr/sbin/varnishd [0x42f698] 0x42e87b: /usr/sbin/varnishd [0x42e87b] 0x34aa40673d: /lib64/libpthread.so.0 [0x34aa40673d] 0x34a90d44bd: /lib64/libc.so.6(clone+0x6d) [0x34a90d44bd] sp = 0x2aaae2d02008 { fd = 1296, id = 1296, xid = 910121180, client = 72.232.146.20 58870, step = STP_PREPRESP, handling = deliver, restarts = 0, esi_level = 0 flags = bodystatus = 4 ws = 0x2aaae2d02080 { id = "sess", {s,f,r,e} = {0x2aaae2d02cd0,+1248,(nil),+65536}, }, http[re Sep 13 12:43:18 cache5 varnishd[970965]: child (987541) Started Sep 13 12:43:18 cache5 varnishd[970965]: Child (987541) said Child starts Does anyone know why is this happening? I'm using: [root at cache5 tmp]# varnishd -V varnishd (varnish-trunk revision 71ee192) Copyright (c) 2006 Verdens Gang AS Copyright (c) 2006-2011 Varnish Software AS With IMS experimental branch Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From straightflush at gmail.com Tue Sep 13 15:09:37 2011 From: straightflush at gmail.com (AD) Date: Tue, 13 Sep 2011 11:09:37 -0400 Subject: invalid ban url syntax In-Reply-To: <4E6F4FEC.1070307@hauke-lampe.de> References: <4E6F4FEC.1070307@hauke-lampe.de> Message-ID: Thanks both of these worked: ban req.http.host == "www.domain.com" && req.url ~ "\\.png" 200 0 ban req.http.host == "www.domain.com" && req.url ~ .png 200 0 On Tue, Sep 13, 2011 at 8:43 AM, Hauke Lampe wrote: > On 09/13/2011 05:37 AM, AD wrote: > > > hello, when trying to ban a req.url from the CLI i get the following > error, > > any ideas? > > > Syntax Error: Invalid backslash sequence > > Which version of varnish do you use? > > You might have to use double backslashes. See: > https://www.varnish-cache.org/trac/ticket/755 > > > Hauke. > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From straightflush at gmail.com Tue Sep 13 16:59:02 2011 From: straightflush at gmail.com (AD) Date: Tue, 13 Sep 2011 12:59:02 -0400 Subject: pass body to a VMOD Message-ID: is there a way to request pass the body to a vmod? -------------- next part -------------- An HTML attachment was scrubbed... URL: From kpettijohn at tarot.com Wed Sep 14 20:10:09 2011 From: kpettijohn at tarot.com (Kevin Pettijohn) Date: Wed, 14 Sep 2011 20:10:09 +0000 Subject: Varnish VCL changes from 2.1.5 to 3.0.1 Message-ID: Hello everyone, I am having some issues with the VCL in Varnish 3.0.1. Below is the VCL configuration I used to remove unwanted cookies from our sites but after upgrading to Varnish 3.0.1 this will no longer compile. // Remove all cookies except the Drupal session cookie if (req.http.Cookie) { set req.http.Cookie = ";" req.http.Cookie; set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";"); set req.http.Cookie = regsuball(req.http.Cookie, ";(PHPSESSID)=", "; \1="); set req.http.Cookie = regsuball(req.http.Cookie, ";(SESS[0-9a-f]{32,32})=", "; \1="); set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", ""); set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", ""); if (req.http.Cookie == "") { remove req.http.Cookie; } } Here is the error from the compile: Message from VCC-compiler: Expected ';' got 'req.http.Cookie' (program line 174), at ('input' Line 24 Pos 31) set req.http.Cookie = ";" req.http.Cookie; ------------------------------###############- Running VCC-compiler failed, exit 1 VCL compilation failed Is there a fix for this configuration? Any insight would be much appreciated. Thanks, Kevin __________________ Kevin Pettijohn Operations & IT -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto.fernandezcrisial at gmail.com Wed Sep 14 20:13:10 2011 From: roberto.fernandezcrisial at gmail.com (=?ISO-8859-1?Q?Roberto_O=2E_Fern=E1ndez_Crisial?=) Date: Wed, 14 Sep 2011 17:13:10 -0300 Subject: Varnish VCL changes from 2.1.5 to 3.0.1 In-Reply-To: References: Message-ID: Look at https://www.varnish-cache.org/trac/browser/doc/changes.rst, and then look for "Change string concatenation to be done using + rather than implicitly" ;) Roberto @rofc On Wed, Sep 14, 2011 at 5:10 PM, Kevin Pettijohn wrote: > Hello everyone, > > I am having some issues with the VCL in Varnish 3.0.1. Below is the VCL > configuration I used to remove unwanted cookies from our sites but after > upgrading to Varnish 3.0.1 this will no longer compile. > > > // Remove all cookies except the Drupal session cookie > if (req.http.Cookie) { > * set req.http.Cookie = ";" req.http.Cookie;* > set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";"); > set req.http.Cookie = regsuball(req.http.Cookie, ";(PHPSESSID)=", "; > \1="); > set req.http.Cookie = regsuball(req.http.Cookie, > ";(SESS[0-9a-f]{32,32})=", "; \1="); > set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", ""); > set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", ""); > > if (req.http.Cookie == "") { > remove req.http.Cookie; > } > } > > > Here is the error from the compile: > > Message from VCC-compiler: > Expected ';' got 'req.http.Cookie' > (program line 174), at > ('input' Line 24 Pos 31) > set req.http.Cookie = ";" req.http.Cookie; > ------------------------------###############- > > Running VCC-compiler failed, exit 1 > VCL compilation failed > > > Is there a fix for this configuration? > > Any insight would be much appreciated. > > Thanks, > Kevin > > > __________________ > Kevin Pettijohn > Operations & IT > > > > > > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From apj at mutt.dk Wed Sep 14 20:14:51 2011 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Wed, 14 Sep 2011 22:14:51 +0200 Subject: Varnish VCL changes from 2.1.5 to 3.0.1 In-Reply-To: References: Message-ID: <20110914201451.GR1944@nerd.dk> On Wed, Sep 14, 2011 at 08:10:09PM +0000, Kevin Pettijohn wrote: > > I am having some issues with the VCL in Varnish 3.0.1. Below is the VCL > configuration I used to remove unwanted cookies from our sites but after > upgrading to Varnish 3.0.1 this will no longer compile. https://www.varnish-cache.org/docs/trunk/installation/upgrade.html has instructions for what you need to change. -- Andreas From kpettijohn at tarot.com Wed Sep 14 20:25:38 2011 From: kpettijohn at tarot.com (Kevin Pettijohn) Date: Wed, 14 Sep 2011 20:25:38 +0000 Subject: Varnish VCL changes from 2.1.5 to 3.0.1 In-Reply-To: <20110914201451.GR1944@nerd.dk> References: <20110914201451.GR1944@nerd.dk> Message-ID: I apologizes as I should have seen that in the docs but thanks for the quick responses. After adding the + all is well. Thanks again, Kevin __________________ Kevin Pettijohn Operations & IT On Sep 14, 2011, at 1:14 PM, Andreas Plesner Jacobsen wrote: > On Wed, Sep 14, 2011 at 08:10:09PM +0000, Kevin Pettijohn wrote: >> >> I am having some issues with the VCL in Varnish 3.0.1. Below is the VCL >> configuration I used to remove unwanted cookies from our sites but after >> upgrading to Varnish 3.0.1 this will no longer compile. > > https://www.varnish-cache.org/docs/trunk/installation/upgrade.html has > instructions for what you need to change. > > -- > Andreas > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From viapanda at gmail.com Thu Sep 15 02:13:20 2011 From: viapanda at gmail.com (Olivier G) Date: Thu, 15 Sep 2011 04:13:20 +0200 Subject: Problem with error and (large) POST requests Message-ID: <7BE95A05-3065-402C-94C8-5CB14113AE99@gmail.com> Hi there. I have a problem with calling "error" and POST requests. Reproducing is rather simple - use a default varnish install, then just do: sub vcl_recv{ error 413 "Whatever"; } Now, curl your server by posting a (small) payload. -> (might) work, and the client (might) get a 413 answer Now, curl your server by posting a (somewhat huge ~ 1MB) payload. -> curl will end-up with a "broken pipe" without getting any answer. Browsers (at least Firefox and Chrome) doing XHR in the same setup will end-up in the same state (readyState 4, status 0) (without being able to read the answer). On the other hand, Varnishlog demonstrates that the answer clearly is sent, and everything is normal from Varnish POV. Interestingly, even Varnish as a client (if pointing a backend to the Varnish sending the error in recv), exhibits: 5 FetchError c backend write error: 32 (Broken pipe) and returns a 503 no matter the actual code sent by the "backend". ? which makes me think that varnish (server) behavior in that case is wrong. Also: If you do the "error" in "vcl_pass", you get the same problem. If you "error" in "vcl_fetch", though? things work as expected. So, it looks to me like: 1- varnish doesn't read the request body before it actually queries the backend (in which case it passes it), or before it gets out of hit (in which case the body is dropped - though, if the request has a body, it's quite likely we will call "pass"...) 2- calling "error" means Varnish closes the tcp connection immediately 3- a client that finds itself in a position where the connection is closed *before it has time to write the entire request* just stops here and fail 4- calling "error" means Varnish doesn't wait for the entire request body to be available/consumed (if it hasn't been already) Assuming I'm correct in these assumptions, then it means that "error" when handling requests with a body is NOT usable at all before having queried a backend. I'm not sure this behavior is intended - if it is, then maybe the documentation should be updated? (I couldn't find a reference to that problem in here: https://www.varnish-cache.org/docs/3.0/reference/vcl.html), and error is clearly mentioned as being usable in recv, etc. If it's not, then possibly the solution would be that the error routine actually makes sure that the entire request body has been sent. Right now, I'm workarounding by calling "error" inside vcl_fetch after having queried a dumb backend which only purpose is to force varnish to consume the body - though this is quite convoluted and suboptimal. What do you guys think? Thanks a lot! - Olivier -------------- next part -------------- An HTML attachment was scrubbed... URL: From viapanda at gmail.com Thu Sep 15 03:37:02 2011 From: viapanda at gmail.com (Olivier G) Date: Thu, 15 Sep 2011 05:37:02 +0200 Subject: Problem with error and (large) POST requests In-Reply-To: <7BE95A05-3065-402C-94C8-5CB14113AE99@gmail.com> References: <7BE95A05-3065-402C-94C8-5CB14113AE99@gmail.com> Message-ID: I've done additional tests, and I'm probably misunderstanding some stuff? :/ First, it seems to me that Varnish do consume the entire payload *before* sending anything to the backend (naively, I was thinking that varnish would start the backend request "sooner"). So, I set-up ngynx as a "dumb" backend to absorbe the post payloads. Additionally, I told ngynx to have a very small POST size limit, and send 413. If curl-ing ngynx with a (too) big payload, ngynx immediately validates the content-length and sends the 413 answer. The upload stops gracefully without much content being sent, and curl happily reads the 413. Interestingly, if I use that setup as a backend to Varnish to workaround my problem (as I described in the previous mail), Varnish ends-up again with 5 FetchError c backend write error: 32 (Broken pipe) and does 503. Meaning to me that it couldn't handle ngynx answer? while curl can? - Olivier Le 15 sept. 2011 ? 04:13, Olivier G a ?crit : > Hi there. > > I have a problem with calling "error" and POST requests. > > Reproducing is rather simple - use a default varnish install, then just do: > > sub vcl_recv{ > error 413 "Whatever"; > } > > Now, curl your server by posting a (small) payload. > -> (might) work, and the client (might) get a 413 answer > > Now, curl your server by posting a (somewhat huge ~ 1MB) payload. > -> curl will end-up with a "broken pipe" without getting any answer. > > Browsers (at least Firefox and Chrome) doing XHR in the same setup will end-up in the same state (readyState 4, status 0) (without being able to read the answer). > > On the other hand, Varnishlog demonstrates that the answer clearly is sent, and everything is normal from Varnish POV. > > > Interestingly, even Varnish as a client (if pointing a backend to the Varnish sending the error in recv), exhibits: > 5 FetchError c backend write error: 32 (Broken pipe) > and returns a 503 no matter the actual code sent by the "backend". > ? which makes me think that varnish (server) behavior in that case is wrong. > > > Also: > If you do the "error" in "vcl_pass", you get the same problem. > If you "error" in "vcl_fetch", though? things work as expected. > > > > So, it looks to me like: > 1- varnish doesn't read the request body before it actually queries the backend (in which case it passes it), or before it gets out of hit (in which case the body is dropped - though, if the request has a body, it's quite likely we will call "pass"...) > 2- calling "error" means Varnish closes the tcp connection immediately > 3- a client that finds itself in a position where the connection is closed *before it has time to write the entire request* just stops here and fail > 4- calling "error" means Varnish doesn't wait for the entire request body to be available/consumed (if it hasn't been already) > > > Assuming I'm correct in these assumptions, then it means that "error" when handling requests with a body is NOT usable at all before having queried a backend. > > > I'm not sure this behavior is intended - if it is, then maybe the documentation should be updated? (I couldn't find a reference to that problem in here: https://www.varnish-cache.org/docs/3.0/reference/vcl.html), and error is clearly mentioned as being usable in recv, etc. > If it's not, then possibly the solution would be that the error routine actually makes sure that the entire request body has been sent. > > > > Right now, I'm workarounding by calling "error" inside vcl_fetch after having queried a dumb backend which only purpose is to force varnish to consume the body - though this is quite convoluted and suboptimal. > > > What do you guys think? > > > Thanks a lot! > > - Olivier -------------- next part -------------- An HTML attachment was scrubbed... URL: From paulo at aliancaproject.com Thu Sep 15 12:42:27 2011 From: paulo at aliancaproject.com (Paulo Paracatu) Date: Thu, 15 Sep 2011 09:42:27 -0300 Subject: varnishd panic message In-Reply-To: References: Message-ID: Hi, I tried to reinstall my Varnish using the official RPM available at the site, but I'm still getting this error: Sep 15 12:29:52 cache5 varnishd[6430]: Child (18352) died signal=6 Sep 15 12:29:52 cache5 varnishd[6430]: Child (18352) Panic message: Assert error in Tcheck(), cache.h line 969: Condition((t.b) != 0) not true. thread = (cache-worker) ident = Linux,2.6.18-238.9.1.el5.centos.plus,x86_64,-smalloc,-smalloc,-hcritbit,epoll Backtrace: 0x42bc96: /usr/sbin/varnishd [0x42bc96] 0x42799f: /usr/sbin/varnishd [0x42799f] 0x42f2b5: /usr/sbin/varnishd(RES_BuildHttp+0x85) [0x42f2b5] 0x41669b: /usr/sbin/varnishd [0x41669b] 0x41756e: /usr/sbin/varnishd(CNT_Session+0x7fe) [0x41756e] 0x42e478: /usr/sbin/varnishd [0x42e478] 0x42d65b: /usr/sbin/varnishd [0x42d65b] 0x34aa40673d: /lib64/libpthread.so.0 [0x34aa40673d] 0x34a90d44bd: /lib64/libc.so.6(clone+0x6d) [0x34a90d44bd] sp = 0x2aaade417008 { fd = 1000, id = 1000, xid = 2124510457, client = 72.232.146.21 61405, step = STP_PREPRESP, handling = deliver, restarts = 0, esi_level = 0 flags = bodystatus = 4 ws = 0x2aaade417080 { id = "sess", {s,f,r,e} = {0x2aaade417cc8,+1456,(nil),+65536}, }, http[req] Sep 15 12:29:52 cache5 varnishd[6430]: child (18656) Started Sep 15 12:29:52 cache5 varnishd[6430]: Child (18656) said Child starts Any ideas? 2011/9/13 Paulo Paracatu > Hello, > > I was monitoring my messages log and varnish is producing a lot of "panic > messages" like this one: > Sep 13 12:43:18 cache5 varnishd[970965]: Child (987439) died signal=6 > Sep 13 12:43:18 cache5 varnishd[970965]: Child (987439) Panic message: > Assert error in Tcheck(), cache.h line 977: Condition((t.b) != 0) not > true. thread = (cache-worker) ident = > Linux,2.6.18-238.9.1.el5.centos.plus,x86_64,-smalloc,-smalloc,-hcritbit,epoll > Backtrace: 0x42cea6: /usr/sbin/varnishd [0x42cea6] 0x4284ef: > /usr/sbin/varnishd [0x4284ef] 0x4304d5: > /usr/sbin/varnishd(RES_BuildHttp+0x85) [0x4304d5] 0x416f3b: > /usr/sbin/varnishd [0x416f3b] 0x417e0b: > /usr/sbin/varnishd(CNT_Session+0x7fb) [0x417e0b] 0x42f698: > /usr/sbin/varnishd [0x42f698] 0x42e87b: /usr/sbin/varnishd [0x42e87b] > 0x34aa40673d: /lib64/libpthread.so.0 [0x34aa40673d] 0x34a90d44bd: > /lib64/libc.so.6(clone+0x6d) [0x34a90d44bd] sp = 0x2aaae2d02008 { fd = > 1296, id = 1296, xid = 910121180, client = 72.232.146.20 58870, step = > STP_PREPRESP, handling = deliver, restarts = 0, esi_level = 0 flags = > bodystatus = 4 ws = 0x2aaae2d02080 { id = "sess", {s,f,r,e} = > {0x2aaae2d02cd0,+1248,(nil),+65536}, }, http[re > Sep 13 12:43:18 cache5 varnishd[970965]: child (987541) Started > Sep 13 12:43:18 cache5 varnishd[970965]: Child (987541) said Child starts > > Does anyone know why is this happening? > > I'm using: > [root at cache5 tmp]# varnishd -V > varnishd (varnish-trunk revision 71ee192) > Copyright (c) 2006 Verdens Gang AS > Copyright (c) 2006-2011 Varnish Software AS > > With IMS experimental branch > > Thanks > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nsrivastava2 at gmail.com Thu Sep 15 12:46:47 2011 From: nsrivastava2 at gmail.com (Nitin Srivastava) Date: Thu, 15 Sep 2011 18:16:47 +0530 Subject: My Varnish goes beyond the specified RAM limit Message-ID: Hi, I am executing this command - */usr/sbin/varnishd -f /etc/varnish/default.vcl -s malloc,3G -T 127.0.0.1:2000* which I believe specifies Varnish to use 3GB of RAM only while my Varnish going beyond it and taking some 9GB of RAM -- Thanks and regards, Nitin Srivastava -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Thu Sep 15 12:51:23 2011 From: perbu at varnish-software.com (Per Buer) Date: Thu, 15 Sep 2011 14:51:23 +0200 Subject: My Varnish goes beyond the specified RAM limit In-Reply-To: References: Message-ID: Hi Nitin, On Thu, Sep 15, 2011 at 2:46 PM, Nitin Srivastava wrote: > Hi, > > I am executing this command - > */usr/sbin/varnishd -f /etc/varnish/default.vcl -s malloc,3G -T > 127.0.0.1:2000* > > which I believe specifies Varnish to use 3GB of RAM Not quite. It specifies that Varnish should use 3GB of memory for storing objects. In addition there are certain data structures that are allocated in memory. Those are typically 1kB per object. > only while my Varnish going beyond it and taking some 9GB of RAM > Varnish 3.0.0 can leak memory in certain conditions. You failed to specify what Varnish version you're running, but if you're on 3.0 I would suggest an upgrade. Per. -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From barak.yaish at gmail.com Thu Sep 15 13:19:36 2011 From: barak.yaish at gmail.com (Barak Yaish) Date: Thu, 15 Sep 2011 16:19:36 +0300 Subject: Caching based on response header Message-ID: Hello, I'm new to Varnish, and I didn't find in the docs how to do the following - cache the response based on customized HTTP header (which the backend may or may not set), for example cache the response in case the one of the response headers is "SomeHeader". Thanks, Barak. -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Thu Sep 15 13:25:51 2011 From: perbu at varnish-software.com (Per Buer) Date: Thu, 15 Sep 2011 15:25:51 +0200 Subject: Caching based on response header In-Reply-To: References: Message-ID: Hi, On Thu, Sep 15, 2011 at 3:19 PM, Barak Yaish wrote: > > > I'm new to Varnish, and I didn't find in the docs how to do the following - > cache the response based on customized HTTP header (which the backend may or > may not set), for example cache the response in case the one of the response > headers is "SomeHeader". > How will Varnish know the value of the response header before it has talked to the backend? Cache lookups happend after we get the request, and at that time the response headers are unknown. You need to figure out what content to deliver purely based on the request. -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto.fernandezcrisial at gmail.com Thu Sep 15 13:43:32 2011 From: roberto.fernandezcrisial at gmail.com (=?ISO-8859-1?Q?Roberto_O=2E_Fern=E1ndez_Crisial?=) Date: Thu, 15 Sep 2011 10:43:32 -0300 Subject: Varnish + Kernel Tunning Message-ID: Hi guys, A few days ago I went on a pike of traffic on my Varnish server. The Varnish works fine until OS' Kernel parameters were exhausted, specifically "kern.openfiles" and "kern.maxfilesperproc". I'm currently using FreeBSD as base OS because it was the best environment I could find for my infrastructure. Now I have a tunned server with Performance recommendations ( https://www.varnish-cache.org/trac/wiki/Performance). But I need to know which are the best practices for these kind of configuration/tunning values. Does anybody have a high-traffic Varnish on FreeBSD server? Regards, Roberto O. Fern?ndez Crisial @rofc -------------- next part -------------- An HTML attachment was scrubbed... URL: From barak.yaish at gmail.com Thu Sep 15 13:51:24 2011 From: barak.yaish at gmail.com (Barak Yaish) Date: Thu, 15 Sep 2011 16:51:24 +0300 Subject: Caching based on response header In-Reply-To: References: Message-ID: Wow, that was quick! :-) Ok, may I describe my backend and then can you suggest whether or not Varnish will suite here? The server (Tomcat) basically acts as a cache, where the key is page url and the value is some info on the page. Clients accessing the service pass the url in the request, and if the data exist in the cache, it will be returned to client, otherwise a default response will be returned to client, and a background process will start process the page, in its end the page data will be placed in the cache. So, since Tomcat (java) do not handle network very well in high scales, I was thinking I could use Varnish to cache backend responses in case found in the service cache. Isn't it possible to configure Varnish to fetch from cache based on some header in the request (in my case it will be the page url that client is querying)? Hope I was clear enough, Barak. On Thu, Sep 15, 2011 at 4:25 PM, Per Buer wrote: > Hi, > > > On Thu, Sep 15, 2011 at 3:19 PM, Barak Yaish wrote: >> >> >> I'm new to Varnish, and I didn't find in the docs how to do the following >> - cache the response based on customized HTTP header (which the backend may >> or may not set), for example cache the response in case the one of the >> response headers is "SomeHeader". >> > > How will Varnish know the value of the response header before it has talked > to the backend? > > Cache lookups happend after we get the request, and at that time the > response headers are unknown. You need to figure out what content to deliver > purely based on the request. > > -- > Per Buer, CEO > Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer > *Varnish makes websites fly!* > Whitepapers | Video | > Twitter > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kristian at varnish-software.com Thu Sep 15 14:02:17 2011 From: kristian at varnish-software.com (Kristian Lyngstol) Date: Thu, 15 Sep 2011 16:02:17 +0200 Subject: Caching based on response header In-Reply-To: References: Message-ID: <20110915140217.GA19330@freud.kly.no> Greetings, On Thu, Sep 15, 2011 at 04:51:24PM +0300, Barak Yaish wrote: > The server (Tomcat) basically acts as a cache, where the key is page url and > the value is some info on the page. Clients accessing the service pass the > url in the request, and if the data exist in the cache, it will be returned > to client, otherwise a default response will be returned to client, and a > background process will start process the page, in its end the page data > will be placed in the cache. So, since Tomcat (java) do not handle network > very well in high scales, I was thinking I could use Varnish to cache > backend responses in case found in the service cache. Isn't it possible to > configure Varnish to fetch from cache based on some header in the request > (in my case it will be the page url that client is querying)? You two seem to be talking about slightly different things. You can use response headers to determine if something is to be cached. This is done in VCL, in the vcl_fetch method, and is fairly common. If you set a Cache-Control header on your response with either a s-maxage or max-age variable (s-maxage takes priority), Varnish will, by default cache the content for that period of time. In addition, if neither variables are present and no Expires header is present, there is a default cache period of 120 seconds (configurable). There are some other factors here too, like cookies, response codes etc, but that's better left to the documentation. And while you can't ask the web server if you should fetch something from cache (well, you can, but I doubt that's what you really want, and it's not trivial), I suspect you did not mean "decide to fetch from cache" so much as "decide to cache"? As for documentation, https://www.varnish-cache.org/docs/3.0/ is the place to go. Particularly the part about Using Varnish, which should have ample examples. Hope this cleared things up! - Kristian From barak.yaish at gmail.com Thu Sep 15 14:20:09 2011 From: barak.yaish at gmail.com (Barak Yaish) Date: Thu, 15 Sep 2011 17:20:09 +0300 Subject: Caching based on response header In-Reply-To: <803226781.53161.1316095680139.JavaMail.open-xchange@oxgw02.schlund.de> References: <803226781.53161.1316095680139.JavaMail.open-xchange@oxgw02.schlund.de> Message-ID: I meant exactly that, doesn't the answer of Kristian Lyngstol address this logic? On Thu, Sep 15, 2011 at 5:07 PM, jon.gilbert at net-entwicklung.de < jon.gilbert at net-entwicklung.de> wrote: > ** > > I think you mean, you want Varnish to pass the call through to tomcat and > if the response has header XYZ then cache it for the next call. If header > XYZ is not there, do not cache the response, so the next call goes directly > to Tomcat too. As soon as Tomcat has the correct page it returns the > response with the header XYZ and Varnish caches the response. All subsequent > calls to Varnish then get a cache hit. > > > > I think I get what you want to do, but I'm not sure that Varnish will > support that. Sorry > > > > Jon > > > > > Barak Yaish hat am 15. September 2011 um 15:51 > geschrieben: > > > Wow, that was quick! :-) > > > > Ok, may I describe my backend and then can you suggest whether or not > > Varnish will suite here? > > > > The server (Tomcat) basically acts as a cache, where the key is page url > and > > the value is some info on the page. Clients accessing the service pass > the > > url in the request, and if the data exist in the cache, it will be > returned > > to client, otherwise a default response will be returned to client, and a > > > background process will start process the page, in its end the page data > > will be placed in the cache. So, since Tomcat (java) do not handle > network > > very well in high scales, I was thinking I could use Varnish to cache > > backend responses in case found in the service cache. Isn't it possible > to > > configure Varnish to fetch from cache based on some header in the request > > > (in my case it will be the page url that client is querying)? > > > > Hope I was clear enough, > > > > Barak. > > > > On Thu, Sep 15, 2011 at 4:25 PM, Per Buer wrote: > > > > > > Hi, > > > > > > > > > On Thu, Sep 15, 2011 at 3:19 PM, Barak Yaish wrote: > > > >> > > >> > > >> I'm new to Varnish, and I didn't find in the docs how to do the > following > > >> - cache the response based on customized HTTP header (which the > backend may > > >> or may not set), for example cache the response in case the one of the > > > >> response headers is "SomeHeader". > > >> > > > > > > How will Varnish know the value of the response header before it has > talked > > > to the backend? > > > > > > Cache lookups happend after we get the request, and at that time the > > > response headers are unknown. You need to figure out what content to > deliver > > > purely based on the request. > > > > > > -- > > > Per Buer, CEO > > > Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer > > > *Varnish makes websites fly!* > > > Whitepapers | Video< > http://www.youtube.com/watch?v=x7t2Sp174eI> | > > > Twitter > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew at matthewwilkes.co.uk Thu Sep 15 14:51:27 2011 From: matthew at matthewwilkes.co.uk (Matthew Wilkes) Date: Thu, 15 Sep 2011 15:51:27 +0100 Subject: Caching based on response header In-Reply-To: References: <803226781.53161.1316095680139.JavaMail.open-xchange@oxgw02.schlund.de> Message-ID: On 2011-09-15, at 1520, Barak Yaish wrote: > I meant exactly that, doesn't the answer of Kristian Lyngstol address this logic? You just want to pass if there isn't that header in the beresp. You'll then get a HitPass object cached, so it will retry all requests to that URL for the next few minutes, after which if the header is set it will be cached. That said, you should probably be using more sensible error codes, like 503 or 404 and only returning a 200 OK when the content is available. Matt From aurelien.lemaire at smile.fr Thu Sep 15 09:29:36 2011 From: aurelien.lemaire at smile.fr (=?ISO-8859-1?Q?Aur=E9lien_Lemaire?=) Date: Thu, 15 Sep 2011 11:29:36 +0200 Subject: varnish nuking agressively all my objects. Message-ID: <4E71C580.9010403@smile.fr> Good day folks, First of all, varnish is an outstanding piece of software that my company and i are addicted to. So thanks to all the coders. Here is my problem : I allocated varnish 1G of RAM on a website that can have more than 2 Go of possible cacheable objects . Not to worry though as any proxy-cache system should smartly nuke old objects to make place to new one to live peacefully within its allocated RAM. And that's where Varnish behave unexpectedly : each time it need to nuke SOME objects : it nukes ALMOST ALL of them (often ~80% of my 35k objects) which is quite aggressive ; thus i lost almost all my cache....IRK ! 3 Munin graphs attached to see the problem clearly : big drop each time a nuking happens. To make sure my pbr is about varnish nuking system : i increased from 1G to 3G(more than the max possible 2G cacheable objects) on another varnish of this platefom (this website is delivered by multiple front/varnish server all stricly similar and independant) and this issue disappeared (no more nuking : no lost of ~80%of my objects) Here is my env : Debian 5.0.8 64 bits on 2.6.32-5-openvz-amd64 kernel Varnish 2.1.3 SVN 5049:5055(debian package 2.1.3-8) 200 varnish 's worker threads running constantly (no issue on workers) 30req/s average with 60/s in peak Daemon run as such : /usr/sbin/varnishd -P /var/run/varnishd.pid -a :80 -T localhost:6082 -S /etc/varnish/secret -f /etc/varnish/serverx.vcl -w 100,1024 -s file,/var/lib/varnish/serverx/varnish_storage.bin,3G Here a quick varnishstat -1 : client_conn 17177518 9.26 Client connections accepted client_drop 0 0.00 Connection dropped, no sess/wrk client_req 31277524 16.86 Client requests received cache_hit 20909485 11.27 Cache hits cache_hitpass 2558712 1.38 Cache hits for pass cache_miss 5422557 2.92 Cache misses backend_conn 175175 0.09 Backend conn. success backend_unhealthy 0 0.00 Backend conn. not attempted backend_busy 0 0.00 Backend conn. too many backend_fail 0 0.00 Backend conn. failures backend_reuse 10192538 5.49 Backend conn. reuses backend_toolate 98629 0.05 Backend conn. was closed backend_recycle 10291261 5.55 Backend conn. recycles backend_unused 0 0.00 Backend conn. unused fetch_head 459553 0.25 Fetch head fetch_length 9907522 5.34 Fetch with Length fetch_chunked 33 0.00 Fetch chunked fetch_eof 0 0.00 Fetch EOF fetch_bad 0 0.00 Fetch had bad headers fetch_close 7 0.00 Fetch wanted close fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed fetch_zero 638 0.00 Fetch zero len fetch_failed 0 0.00 Fetch failed n_sess_mem 1189 . N struct sess_mem n_sess 890 . N struct sess n_object 32678 . N struct object n_vampireobject 0 . N unresurrected objects n_objectcore 32794 . N struct objectcore n_objecthead 30698 . N struct objecthead n_smf 68810 . N struct smf n_smf_frag 3002 . N small free smf n_smf_large 490 . N large free smf n_vbe_conn 8 . N struct vbe_conn n_wrk 200 . N worker threads n_wrk_create 200 0.00 N worker threads created n_wrk_failed 0 0.00 N worker threads not created n_wrk_max 4133 0.00 N worker threads limited n_wrk_queue 0 0.00 N queued work requests n_wrk_overflow 5 0.00 N overflowed work requests n_wrk_drop 0 0.00 N dropped work requests n_backend 1 . N backends n_expired 2291302 . N expired objects *n_lru_nuked 3098052 . N LRU nuked objects* n_lru_saved 0 . N LRU saved objects n_lru_moved 19894617 . N LRU moved objects n_deathrow 0 . N objects on deathrow losthdr 0 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 23874235 12.87 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 17177512 9.26 Total Sessions s_req 31277524 16.86 Total Requests s_pipe 0 0.00 Total pipe s_pass 4945241 2.67 Total pass s_fetch 10367753 5.59 Total fetch s_hdrbytes 10732679592 5784.04 Total header bytes s_bodybytes 316828226025 170744.51 Total body bytes sess_closed 7071576 3.81 Session Closed sess_pipeline 13905 0.01 Session Pipeline sess_readahead 8085 0.00 Session Read Ahead sess_linger 24485521 13.20 Session Linger sess_herd 23385907 12.60 Session herd shm_records 1766166234 951.82 SHM records shm_writes 128900505 69.47 SHM writes shm_flushes 13215 0.01 SHM flushes due to overflow shm_cont 78095 0.04 SHM MTX contention shm_cycles 769 0.00 SHM cycles through buffer sm_nreq 18413160 9.92 allocator requests sm_nobj 65318 . outstanding allocations sm_balloc 824430592 . bytes allocated sm_bfree 249311232 . bytes free sma_nreq 0 0.00 SMA allocator requests sma_nobj 0 . SMA outstanding allocations sma_nbytes 0 . SMA outstanding bytes sma_balloc 0 . SMA bytes allocated sma_bfree 0 . SMA bytes free sms_nreq 61 0.00 SMS allocator requests sms_nobj 0 . SMS outstanding allocations sms_nbytes 0 . SMS outstanding bytes sms_balloc 16409 . SMS bytes allocated sms_bfree 16409 . SMS bytes freed backend_req 10367752 5.59 Backend requests made n_vcl 1 0.00 N vcl total n_vcl_avail 1 0.00 N vcl available n_vcl_discard 0 0.00 N vcl discarded n_purge 1 . N total active purges n_purge_add 12314 0.01 N new purges added n_purge_retire 12313 0.01 N old purges deleted n_purge_obj_test 2448163 1.32 N objects tested n_purge_re_test 62064275 33.45 N regexps tested against n_purge_dups 9524 0.01 N duplicate purges removed hcb_nolock 28886624 15.57 HCB Lookups without lock hcb_lock 4243837 2.29 HCB Lookups with lock hcb_insert 4243834 2.29 HCB Inserts esi_parse 0 0.00 Objects ESI parsed (unlock) esi_errors 0 0.00 ESI parse errors (unlock) accept_fail 0 0.00 Accept failures client_drop_late 0 0.00 Connection dropped late uptime 1855569 1.00 Client uptime Is it normal varnish behaviour ? sounds like a bug to me. Am i missing some tuning (lru_interval) to soften the nuking algo ? Do you need more info ? helps appreciated here ;-) Regards, Aurelien Lemaire -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: varnish_expunge-week.png Type: image/png Size: 21317 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: varnish_memory_usage-week.png Type: image/png Size: 27158 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: varnish_objects-week.png Type: image/png Size: 23910 bytes Desc: not available URL: From jnerin+varnish at gmail.com Fri Sep 16 06:55:39 2011 From: jnerin+varnish at gmail.com (=?UTF-8?B?Sm9yZ2UgTmVyw61u?=) Date: Fri, 16 Sep 2011 08:55:39 +0200 Subject: varnish nuking agressively all my objects. In-Reply-To: <4E71C580.9010403@smile.fr> References: <4E71C580.9010403@smile.fr> Message-ID: On Thu, Sep 15, 2011 at 11:29, Aur?lien Lemaire wrote: > ** > Good day folks, > > First of all, varnish is an outstanding piece of software that my company > and i are addicted to. So thanks to all the coders. > > Here is my problem : > I allocated varnish 1G of RAM on a website that can have more than 2 Go of > possible cacheable objects . Not to worry though as any proxy-cache system > should smartly nuke old objects to make place to new one to live peacefully > within its allocated RAM. And that's where Varnish behave unexpectedly : > each time it need to nuke SOME objects : it nukes ALMOST ALL of them (often > ~80% of my 35k objects) which is quite aggressive ; thus i lost almost all > my cache....IRK ! > > 3 Munin graphs attached to see the problem clearly : big drop each time a > nuking happens. > > To make sure my pbr is about varnish nuking system : i increased from 1G to > 3G(more than the max possible 2G cacheable objects) on another varnish of > this platefom (this website is delivered by multiple front/varnish server > all stricly similar and independant) and this issue disappeared (no more > nuking : no lost of ~80%of my objects) > > Here is my env : > Debian 5.0.8 64 bits on 2.6.32-5-openvz-amd64 kernel > Varnish 2.1.3 SVN 5049:5055(debian package 2.1.3-8) > 200 varnish 's worker threads running constantly (no issue on workers) > 30req/s average with 60/s in peak > > Daemon run as such : > /usr/sbin/varnishd -P /var/run/varnishd.pid -a :80 -T localhost:6082 -S > /etc/varnish/secret -f /etc/varnish/serverx.vcl -w 100,1024 -s > file,/var/lib/varnish/serverx/varnish_storage.bin,3G > > Here a quick varnishstat -1 : > > Is it normal varnish behaviour ? sounds like a bug to me. > Am i missing some tuning (lru_interval) to soften the nuking algo ? > Do you need more info ? > helps appreciated here ;-) > > Regards, Aurelien Lemaire > > It could be someone downloading a large file (like a ~700Mb iso file) and varnish nuking objects to make room for this file (even if its configured to not cache it). Try to get a varnishlog trace of the moment the nuking begins. -- Jorge Ner?n -------------- next part -------------- An HTML attachment was scrubbed... URL: From jnerin+varnish at gmail.com Fri Sep 16 07:29:10 2011 From: jnerin+varnish at gmail.com (=?UTF-8?B?Sm9yZ2UgTmVyw61u?=) Date: Fri, 16 Sep 2011 09:29:10 +0200 Subject: varnish nuking agressively all my objects. In-Reply-To: <4E71C580.9010403@smile.fr> References: <4E71C580.9010403@smile.fr> Message-ID: On Thu, Sep 15, 2011 at 11:29, Aur?lien Lemaire wrote: > ** > Good day folks, > > First of all, varnish is an outstanding piece of software that my company > and i are addicted to. So thanks to all the coders. > > Here is my problem : > I allocated varnish 1G of RAM on a website that can have more than 2 Go of > possible cacheable objects . Not to worry though as any proxy-cache system > should smartly nuke old objects to make place to new one to live peacefully > within its allocated RAM. And that's where Varnish behave unexpectedly : > each time it need to nuke SOME objects : it nukes ALMOST ALL of them (often > ~80% of my 35k objects) which is quite aggressive ; thus i lost almost all > my cache....IRK ! > > 3 Munin graphs attached to see the problem clearly : big drop each time a > nuking happens. > > To make sure my pbr is about varnish nuking system : i increased from 1G to > 3G(more than the max possible 2G cacheable objects) on another varnish of > this platefom (this website is delivered by multiple front/varnish server > all stricly similar and independant) and this issue disappeared (no more > nuking : no lost of ~80%of my objects) > > Here is my env : > Debian 5.0.8 64 bits on 2.6.32-5-openvz-amd64 kernel > Varnish 2.1.3 SVN 5049:5055(debian package 2.1.3-8) > 200 varnish 's worker threads running constantly (no issue on workers) > 30req/s average with 60/s in peak > > Daemon run as such : > /usr/sbin/varnishd -P /var/run/varnishd.pid -a :80 -T localhost:6082 -S > /etc/varnish/secret -f /etc/varnish/serverx.vcl -w 100,1024 -s > file,/var/lib/varnish/serverx/varnish_storage.bin,3G > > Here a quick varnishstat -1 : > > client_conn 17177518 9.26 Client connections accepted > client_drop 0 0.00 Connection dropped, no sess/wrk > client_req 31277524 16.86 Client requests received > cache_hit 20909485 11.27 Cache hits > cache_hitpass 2558712 1.38 Cache hits for pass > cache_miss 5422557 2.92 Cache misses > backend_conn 175175 0.09 Backend conn. success > backend_unhealthy 0 0.00 Backend conn. not attempted > backend_busy 0 0.00 Backend conn. too many > backend_fail 0 0.00 Backend conn. failures > backend_reuse 10192538 5.49 Backend conn. reuses > backend_toolate 98629 0.05 Backend conn. was closed > backend_recycle 10291261 5.55 Backend conn. recycles > backend_unused 0 0.00 Backend conn. unused > fetch_head 459553 0.25 Fetch head > fetch_length 9907522 5.34 Fetch with Length > fetch_chunked 33 0.00 Fetch chunked > fetch_eof 0 0.00 Fetch EOF > fetch_bad 0 0.00 Fetch had bad headers > fetch_close 7 0.00 Fetch wanted close > fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed > fetch_zero 638 0.00 Fetch zero len > fetch_failed 0 0.00 Fetch failed > n_sess_mem 1189 . N struct sess_mem > n_sess 890 . N struct sess > n_object 32678 . N struct object > n_vampireobject 0 . N unresurrected objects > n_objectcore 32794 . N struct objectcore > n_objecthead 30698 . N struct objecthead > n_smf 68810 . N struct smf > n_smf_frag 3002 . N small free smf > n_smf_large 490 . N large free smf > n_vbe_conn 8 . N struct vbe_conn > n_wrk 200 . N worker threads > n_wrk_create 200 0.00 N worker threads created > n_wrk_failed 0 0.00 N worker threads not created > n_wrk_max 4133 0.00 N worker threads limited > n_wrk_queue 0 0.00 N queued work requests > n_wrk_overflow 5 0.00 N overflowed work requests > n_wrk_drop 0 0.00 N dropped work requests > n_backend 1 . N backends > n_expired 2291302 . N expired objects > *n_lru_nuked 3098052 . N LRU nuked objects* > n_lru_saved 0 . N LRU saved objects > n_lru_moved 19894617 . N LRU moved objects > n_deathrow 0 . N objects on deathrow > losthdr 0 0.00 HTTP header overflows > n_objsendfile 0 0.00 Objects sent with sendfile > n_objwrite 23874235 12.87 Objects sent with write > n_objoverflow 0 0.00 Objects overflowing workspace > s_sess 17177512 9.26 Total Sessions > s_req 31277524 16.86 Total Requests > s_pipe 0 0.00 Total pipe > s_pass 4945241 2.67 Total pass > s_fetch 10367753 5.59 Total fetch > s_hdrbytes 10732679592 5784.04 Total header bytes > s_bodybytes 316828226025 170744.51 Total body bytes > sess_closed 7071576 3.81 Session Closed > sess_pipeline 13905 0.01 Session Pipeline > sess_readahead 8085 0.00 Session Read Ahead > sess_linger 24485521 13.20 Session Linger > sess_herd 23385907 12.60 Session herd > shm_records 1766166234 951.82 SHM records > shm_writes 128900505 69.47 SHM writes > shm_flushes 13215 0.01 SHM flushes due to overflow > shm_cont 78095 0.04 SHM MTX contention > shm_cycles 769 0.00 SHM cycles through buffer > sm_nreq 18413160 9.92 allocator requests > sm_nobj 65318 . outstanding allocations > sm_balloc 824430592 . bytes allocated > sm_bfree 249311232 . bytes free > sma_nreq 0 0.00 SMA allocator requests > sma_nobj 0 . SMA outstanding allocations > sma_nbytes 0 . SMA outstanding bytes > sma_balloc 0 . SMA bytes allocated > sma_bfree 0 . SMA bytes free > sms_nreq 61 0.00 SMS allocator requests > sms_nobj 0 . SMS outstanding allocations > sms_nbytes 0 . SMS outstanding bytes > sms_balloc 16409 . SMS bytes allocated > sms_bfree 16409 . SMS bytes freed > backend_req 10367752 5.59 Backend requests made > n_vcl 1 0.00 N vcl total > n_vcl_avail 1 0.00 N vcl available > n_vcl_discard 0 0.00 N vcl discarded > n_purge 1 . N total active purges > n_purge_add 12314 0.01 N new purges added > n_purge_retire 12313 0.01 N old purges deleted > n_purge_obj_test 2448163 1.32 N objects tested > n_purge_re_test 62064275 33.45 N regexps tested against > n_purge_dups 9524 0.01 N duplicate purges removed > hcb_nolock 28886624 15.57 HCB Lookups without lock > hcb_lock 4243837 2.29 HCB Lookups with lock > hcb_insert 4243834 2.29 HCB Inserts > esi_parse 0 0.00 Objects ESI parsed (unlock) > esi_errors 0 0.00 ESI parse errors (unlock) > accept_fail 0 0.00 Accept failures > client_drop_late 0 0.00 Connection dropped late > uptime 1855569 1.00 Client uptime > > > Is it normal varnish behaviour ? sounds like a bug to me. > Am i missing some tuning (lru_interval) to soften the nuking algo ? > Do you need more info ? > helps appreciated here ;-) > > Regards, Aurelien Lemaire > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > It could be someone downloading a large file (like a ~700Mb iso file) and varnish nuking objects to make room for this file (even if its configured to not cache it). Try to get a varnishlog trace of the moment the nuking begins. -- Jorge Ner?n -------------- next part -------------- An HTML attachment was scrubbed... URL: From parsimail2001 at yahoo.se Fri Sep 16 08:01:02 2011 From: parsimail2001 at yahoo.se (Your Friend) Date: Fri, 16 Sep 2011 09:01:02 +0100 (BST) Subject: Reducing diskI/O operations Message-ID: <1316160062.90502.YahooMailNeo@web26301.mail.ukl.yahoo.com> Hello, My main question is thatif using malloc instead of file can be usefull to reduce disk I/O operations? Thanks for your answers! -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Fri Sep 16 10:32:11 2011 From: perbu at varnish-software.com (Per Buer) Date: Fri, 16 Sep 2011 12:32:11 +0200 Subject: Reducing diskI/O operations In-Reply-To: <1316160062.90502.YahooMailNeo@web26301.mail.ukl.yahoo.com> References: <1316160062.90502.YahooMailNeo@web26301.mail.ukl.yahoo.com> Message-ID: Hello Friend. On Fri, Sep 16, 2011 at 10:01 AM, Your Friend wrote: > > My main question is thatif using malloc instead of file can be usefull to > reduce disk I/O operations? > Yes. Newer versions of both the Linux and FreeBSD vm's will work quite hard to keep the -sfile file in sync. Malloc will only trigger disk IO when the vm needs to page out parts of varnish to the paging (swap) area. -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From parsimail2001 at yahoo.se Sat Sep 17 09:56:33 2011 From: parsimail2001 at yahoo.se (Your Friend) Date: Sat, 17 Sep 2011 10:56:33 +0100 (BST) Subject: SV: Reducing diskI/O operations In-Reply-To: References: <1316160062.90502.YahooMailNeo@web26301.mail.ukl.yahoo.com> Message-ID: <1316253393.12009.YahooMailNeo@web26306.mail.ukl.yahoo.com> Thanks for your answer and great software! Best regards, Ali ________________________________ Hello Friend. On Fri, Sep 16, 2011 at 10:01 AM, Your Friend wrote: > >My main question is thatif using malloc instead of file can be usefull to reduce disk I/O operations? Yes. Newer versions of both the Linux and FreeBSD vm's will work quite hard to keep the -sfile file in sync. Malloc will only trigger disk IO when the vm needs to page out parts of varnish to the paging (swap) area. -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer Varnish makes websites fly! Whitepapers?| Video?| Twitter? -------------- next part -------------- An HTML attachment was scrubbed... URL: From mailing.lists.wam at gmail.com Fri Sep 16 08:25:47 2011 From: mailing.lists.wam at gmail.com (mailing.lists.wam mailing.lists.wam) Date: Fri, 16 Sep 2011 10:25:47 +0200 Subject: Persistent storage question In-Reply-To: References: Message-ID: I would like to add some details to this case: We encounter various varnish panic (the forked processus crash, won't restart and nothing listen to http/80 port anymore), with persistent storage (tested with 20/35/40/90G) and kernel address randomize On/Off. Same servers with file,malloc parameters instead of persistent are healthy. Feel free to contact me to get the full coredump. All details below :) Varnish Version : 3 - trunk d56069e Sep 06, 2011 d56069e8ef221310d75455feb9b03483c9caf63b Ubuntu 10.04 64Bits Linux 2.6.32-33-generic #72-Ubuntu SMP Fri Jul 29 21:07:13 UTC 2011 x86_64 GNU/Linux 48G RAM / two Intel(R) Xeon(R) CPU L5640 @ 2.27GHz SSD-SATA 90G 2) Startup config : VARNISH_INSTANCE=default START=yes NFILES="131072" MEMLOCK="82000" VARNISH_VCL_CONF=/etc/varnish/default/default.vcl VARNISH_LISTEN_ADDRESS= VARNISH_LISTEN_PORT=80 VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1 VARNISH_ADMIN_LISTEN_PORT=6082 VARNISH_SECRET_FILE=/etc/varnish/default/secret VARNISH_THREAD_POOLS=12 VARNISH_STORAGE_FILE_1=/mnt/ssd/varnish/cachefile1 VARNISH_STORAGE_SIZE=30G VARNISH_STORAGE_1="persistent,${VARNISH_STORAGE_FILE_1},${VARNISH_STORAGE_SIZE}" DAEMON_OPTS=" -n ${VARNISH_INSTANCE} \ -u root \ -a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \ -f ${VARNISH_VCL_CONF} \ -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \ -S ${VARNISH_SECRET_FILE} \ -s ${VARNISH_STORAGE_1} \ -s Transient=malloc,1G\ -p first_byte_timeout=5 \ -p between_bytes_timeout=5 \ -p pipe_timeout=5 \ -p send_timeout=2700 \ -p default_grace=240 \ -p default_ttl=3600 \ -p http_gzip_support=off \ -p http_range_support=on \ -p max_restarts=2 \ -p thread_pool_add_delay=2 \ -p thread_pool_max=4000 \ -p thread_pool_min=80 \ -p thread_pool_timeout=120 \ -p thread_pools=12 \ -p thread_stats_rate=50 #### VCL FILE ##### ### SECDownMod ### https://github.com/footplus/libvmod-secdown import secdown; include "/etc/varnish/backend/director_edge_2xx.vcl"; include "/etc/varnish/acl/purge.vcl"; sub vcl_recv { set req.backend = origin; if (req.request !~ "(GET|HEAD|PURGE)") { error 405 "Not allowed."; } if (req.url ~ "^/files") { set req.url = secdown.check_url(req.url, "MySecretIsNotYourSecret", "/link-expired.html", "/link-error.html"); } # Before anything else we need to fix gzip compression if (req.http.Accept-Encoding) { if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|flv|ts|mp4)$") { # No point in compressing these remove req.http.Accept-Encoding; } else if (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip"; } else if (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate"; } else { # unknown algorithm remove req.http.Accept-Encoding; } } # Allow a PURGE method to clear cache via regular expression. if (req.request == "PURGE") { # If the client has not an authorized IP or # if he comes from the HTTPS proxy on localhost, deny it. if (!client.ip ~ purge || req.http.X-Forwarded-For) { error 405 "Not allowed."; } ban_url(req.url); error 200 "Expression " + req.url + " added to ban.list."; } } sub vcl_pipe { set bereq.http.connection = "close"; } sub vcl_pass { # return (pass); } sub vcl_hash { hash_data(req.url); return (hash); } sub vcl_hit { # return (deliver); } sub vcl_miss { # return (fetch); } sub vcl_fetch { unset beresp.http.expires; set beresp.http.cache-control = "max-age=86400"; set beresp.ttl = 365d; if (beresp.status >= 400) { set beresp.ttl = 1m; } if ((beresp.status == 301) || (beresp.status == 302) || (beresp.status == 401)) { return (hit_for_pass); } } sub vcl_deliver { # Rename Varnish XIDs headers if (resp.http.X-Varnish) { set resp.http.X-Object-ID = resp.http.X-Varnish; unset resp.http.X-Varnish; } remove resp.http.Via; remove resp.http.X-Powered-By; # return (deliver); } sub vcl_error { # Do not reveal what's inside the box :) remove obj.http.Server; set obj.http.Server = "EdgeCache/1.4"; } sub vcl_init { # return (ok); } sub vcl_fini { # return (ok); } 3) Assert Message (from syslog) Sep 15 18:21:02 e101 default[18290]: Child (19438) said Out of space in persistent silo Sep 15 18:21:02 e101 default[18290]: Child (19438) said Committing suicide, restart will make space Sep 15 18:21:02 e101 default[18290]: Child (19438) ended Sep 15 18:21:02 e101 default[18290]: Child cleanup complete Sep 15 18:21:02 e101 default[18290]: child (20924) Started Sep 15 18:21:02 e101 default[18290]: Child (20924) said Child starts Sep 15 18:21:02 e101 default[18290]: Child (20924) said Dropped 11 segments to make free_reserve Sep 15 18:21:02 e101 default[18290]: Child (20924) said Silo completely loaded Sep 15 18:21:27 e101 default[18290]: Child (20924) died signal=6 (core dumped) Sep 15 18:21:27 e101 default[18290]: Child (20924) Panic message: Assert error in smp_oc_getobj(), storage_persistent_silo.c line 401:#012 Condition((o)->mag ic == 0x32851d42) not true.#012thread = (ban-lurker)#012ident = Linux,2.6.32-33-generic,x86_64,-spersistent,-smalloc,-hcritbit,epoll#012Backtrace:#012 0x437e 49: pan_backtrace+19#012 0x43811e: pan_ic+1ad#012 0x45da38: smp_oc_getobj+282#012 0x415407: oc_getobj+14c#012 0x417848: ban_lurker_work+299#012 0x41793d: ban_lurker+5b#012 0x43ad91: wrk_bgthread+184#012 0x7ffff6a9c9ca: _end+7ffff6408692#012 0x7ffff67f970d: _end+7ffff61653d5#012 Sep 15 18:21:27 e101 default[18290]: Child cleanup complete Sep 15 18:21:27 e101 default[18290]: child (21898) Started Sep 15 18:21:27 e101 default[18290]: Pushing vcls failed: CLI communication error (hdr) Sep 15 18:21:27 e101 default[18290]: Stopping Child Sep 15 18:21:27 e101 default[18290]: Child (21898) died signal=6 (core dumped) Sep 15 18:21:27 e101 default[18290]: Child (21898) Panic message: Assert error in smp_open_segs(), storage_persistent.c line 239:#012 Condition(sg1->p.offset != sg->p.offset) not true.#012thread = (cache-main)#012ident = Linux,2.6.32-33-generic,x86_64,-spersistent,-smalloc,-hcritbit,no_waiter#012Backtrace:#012 0x 437e49: pan_backtrace+19#012 0x43811e: pan_ic+1ad#012 0x45a568: smp_open_segs+415#012 0x45ab93: smp_open+236#012 0x456391: STV_open+40#012 0x435fa4: chil d_main+124#012 0x44d3a7: start_child+36a#012 0x44ddce: mgt_sigchld+3e7#012 0x7ffff7bd1fec: _end+7ffff753dcb4#012 0x7ffff7bd2348: _end+7ffff753e010#012 Sep 15 18:21:27 e101 default[18290]: Child (-1) said Child starts Sep 15 18:21:27 e101 default[18290]: Child cleanup complete 4) GDB Core bt (gdb) bt #0 0x00007ffff6746a75 in raise () from /lib/libc.so.6 #1 0x00007ffff674a5c0 in abort () from /lib/libc.so.6 #2 0x00000000004381dd in pan_ic (func=0x482dd5 "smp_open_segs", file=0x4827c4 "storage_persistent.c", line=239, cond=0x48283f "sg1->p.offset != sg->p.offset", err=0, xxx=0) at cache_panic.c:374 #3 0x000000000045a568 in smp_open_segs (sc=0x7ffff6433000, ctx=0x7ffff6433220) at storage_persistent.c:239 #4 0x000000000045ab93 in smp_open (st=0x7ffff64213c0) at storage_persistent.c:331 #5 0x0000000000456391 in STV_open () at stevedore.c:406 #6 0x0000000000435fa4 in child_main () at cache_main.c:128 #7 0x000000000044d3a7 in start_child (cli=0x0) at mgt_child.c:345 #8 0x000000000044ddce in mgt_sigchld (e=0x7ffff64da1d0, what=-1) at mgt_child.c:524 #9 0x00007ffff7bd1fec in vev_sched_signal (evb=0x7ffff6408380) at vev.c:435 #10 0x00007ffff7bd2348 in vev_schedule_one (evb=0x7ffff6408380) at vev.c:478 #11 0x00007ffff7bd1d2a in vev_schedule (evb=0x7ffff6408380) at vev.c:363 #12 0x000000000044e1c9 in MGT_Run () at mgt_child.c:602 #13 0x0000000000461a64 in main (argc=0, argv=0x7fffffffebd0) at varnishd.c:650 5) Last lines of varnishlog 221 SessionOpen c 85.93.199.29 58335 :80 234 SessionOpen c 77.196.147.182 2273 :80 2011/9/15 Aur?lien > Hello, > > I'm currently investigating an issue on some caches we are trying to put in > production, and I think we'll make a separate post about the whole setup, > but i'm currently personnally interested in the following messages: > > default[18290]: Child (19438) said Out of space in persistent silo > default[18290]: Child (19438) said Committing suicide, restart will make > space > > These can be triggered in storage_persistent_silo.c, but I'm not exactly > clear on why varnish commits "suicide", and how this could be a "normal" > condition (exit 0 + auto restart). > > We're using one of the latest trunk versions (d56069e), with various > persistent storage sizes (tried 3*30G, 1*90Gb), on a Linux server with 48Gb > memory. We're caching relatively big files (avg size: ~25 Mb), and they have > a long expiry time (~1year). > > Also, the document I found, > https://www.varnish-cache.org/trac/wiki/ArchitecturePersistentStorage, > does not exactly explain if/how the segments are reused (or I did not > understand it). > > What is the reason and intent behind this restart ? Are the cache contents > lost in this case ? Could this be caused by a certain workflow or > configuration ? > > Thanks, > Best regards, > -- > Aur?lien Guillaume > > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From parsimail2001 at yahoo.se Tue Sep 20 06:57:48 2011 From: parsimail2001 at yahoo.se (Your Friend) Date: Tue, 20 Sep 2011 07:57:48 +0100 (BST) Subject: varnish cach question Message-ID: <1316501868.22837.YahooMailNeo@web26302.mail.ukl.yahoo.com> Hello guys, I'm using malloc for Varnish. I'm using a php application which runs deamons for live streaming (text-streaming) and has a big issue with memory, also memory leak so I desided to use echo 1 > /proc/sys/vm/drop_caches. Here comes the problem! I think this command resets varnish's cache! (not 100% sure.) How can I stop memory leaking without resetting Varnish cache? (my php application is the latest version and I can't replace it with an other application) OS: Ubuntu 9.10 varnish: v. 2.1 memory: 3GB Best regards, Ali -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Tue Sep 20 07:14:51 2011 From: perbu at varnish-software.com (Per Buer) Date: Tue, 20 Sep 2011 09:14:51 +0200 Subject: varnish cach question In-Reply-To: <1316501868.22837.YahooMailNeo@web26302.mail.ukl.yahoo.com> References: <1316501868.22837.YahooMailNeo@web26302.mail.ukl.yahoo.com> Message-ID: Hi Ali, On Tue, Sep 20, 2011 at 8:57 AM, Your Friend wrote: > I'm using malloc for Varnish. I'm using a php application which runs > deamons for live streaming (text-streaming) and has a big issue with memory, > also memory leak so I desided to use echo 1 > /proc/sys/vm/drop_caches. > This has nothing to do with the memory leak in your program. This command flushes out the internal caching in the OS kernel. It will only screw up your IO performance. Here comes the problem! I think this command resets varnish's cache! (not > 100% sure.) > If you are using -s file I think the command might flush out parts of Varnish cache as well. If you are using -s malloc then it won't have any effect. > How can I stop memory leaking without resetting Varnish cache? (my php > application is the latest version and I can't replace it with an other > application) > Restart the PHP application or fix the leak. -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From footplus at gmail.com Tue Sep 20 07:24:48 2011 From: footplus at gmail.com (=?utf-8?Q?Aur=C3=A9llien_Guillaume?=) Date: Tue, 20 Sep 2011 09:24:48 +0200 Subject: varnish cach question In-Reply-To: <1316501868.22837.YahooMailNeo@web26302.mail.ukl.yahoo.com> References: <1316501868.22837.YahooMailNeo@web26302.mail.ukl.yahoo.com> Message-ID: <81C93397-12F2-4B49-983F-18ACFBF3B82E@gmail.com> Le 20 sept. 2011 ? 08:57, Your Friend a ?crit : Hello, > > I'm using malloc for Varnish. I'm using a php application which runs deamons for live streaming (text-streaming) and has a big issue with memory, also memory leak so I desided to use echo 1 > /proc/sys/vm/drop_caches. > I'm not exactly sure of what you mean by leaking memory. On a Linux system, unused memory is considered wasted. The kernel uses this memory to store caches. It will however recycle the pages should an application need some. Here you are dropping the caches, so there is "free" memory, but i dont see the point of doing that. To have a good estimate of how much memory is available to your applications, look on the second line, in the +/- buffers/caches column of the output of the command "free". > Here comes the problem! I think this command resets varnish's cache! (not 100% sure.) Malloc cache should be considered as application data, not as clean pages, and should not be affected by this operation (can someone confirm this ?) > > How can I stop memory leaking without resetting Varnish cache? (my php application is the latest version and I can't replace it with an other application) If your application is relatively standard (not using mmap() or other exotic functions (for a php app), you should try to disable cache drops. Nothing should change, and your apps may even run faster. Best regards, -- Aur?lien GUILLAUME -------------- next part -------------- An HTML attachment was scrubbed... URL: From parsimail2001 at yahoo.se Tue Sep 20 08:09:20 2011 From: parsimail2001 at yahoo.se (Your Friend) Date: Tue, 20 Sep 2011 09:09:20 +0100 (BST) Subject: SV: varnish cach question In-Reply-To: References: <1316501868.22837.YahooMailNeo@web26302.mail.ukl.yahoo.com> Message-ID: <1316506160.41015.YahooMailNeo@web26304.mail.ukl.yahoo.com> Hi Per, " It will only screw up your IO performance " Thanks for mentioning it, I never noticed it because I was more worried about the memory usage and using ? echo 1 > /proc/sys/vm/drop_caches,? echo 1 > /proc/sys/vm/drop_caches ?,?? echo 3 > /proc/sys/vm/drop_caches ?always released 30-40 MB on my server, but it's not worth it if it screws up IO performance. ? I'll try to take a closer look into my PHP application to fix the memory leak. Thank you for taking time to answer to my question. Best regards, Ali ________________________________ Hi Ali, On Tue, Sep 20, 2011 at 8:57 AM, wrote: I'm using malloc for Varnish. I'm using a php application which runs deamons for live streaming (text-streaming) and has a big issue with memory, also memory leak so I desided to use echo 1 > /proc/sys/vm/drop_caches. This has nothing to do with the memory leak in your program. This command flushes out the internal caching in the OS kernel. It will only screw up your IO performance. Here comes the problem! I think this command resets varnish's cache! (not 100% sure.) If you are using -s file I think the command might flush out parts of Varnish cache as well. If you are using -s malloc then it won't have any effect. ? How can I stop memory leaking without resetting Varnish cache? (my php application is the latest version and I can't replace it with an other application) Restart the PHP application or fix the leak. -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer Varnish makes websites fly! Whitepapers?| Video?| Twitter? -------------- next part -------------- An HTML attachment was scrubbed... URL: From rtshilston at gmail.com Tue Sep 20 08:27:04 2011 From: rtshilston at gmail.com (Rob S) Date: Tue, 20 Sep 2011 09:27:04 +0100 Subject: varnish cach question In-Reply-To: <1316506160.41015.YahooMailNeo@web26304.mail.ukl.yahoo.com> References: <1316501868.22837.YahooMailNeo@web26302.mail.ukl.yahoo.com> <1316506160.41015.YahooMailNeo@web26304.mail.ukl.yahoo.com> Message-ID: On Tue, Sep 20, 2011 at 9:09 AM, Your Friend wrote: > I'll try to take a closer look into my PHP application to fix the memory > leak. > One tip for PHP memory leak debugging is to to put: echo memory_get_usage(); at the end of each iteration of your loop. We find that objects sometimes are responsible for an increase in memory usage, and that you need to do $obj = null; unset($obj); at the end of each iteration to ensure they're properly disposed of, and thus ensure memory isn't leaked. Good luck Rob (Apologies for being slightly off topic for Varnish-misc) -------------- next part -------------- An HTML attachment was scrubbed... URL: From parsimail2001 at yahoo.se Tue Sep 20 08:29:34 2011 From: parsimail2001 at yahoo.se (Your Friend) Date: Tue, 20 Sep 2011 09:29:34 +0100 (BST) Subject: SV: varnish cach question In-Reply-To: <81C93397-12F2-4B49-983F-18ACFBF3B82E@gmail.com> References: <1316501868.22837.YahooMailNeo@web26302.mail.ukl.yahoo.com> <81C93397-12F2-4B49-983F-18ACFBF3B82E@gmail.com> Message-ID: <1316507374.63958.YahooMailNeo@web26302.mail.ukl.yahoo.com> Hi, You areright about that, the "drop cache" command doesn't affect Varnish cash if I use malloc and Per (Buer) has confirmed it. I'll disable?cache drops and try to fix the memory leak issue in my PHP application. Thanks for your answer. Best Regards, Ali ________________________________ Le 20 sept. 2011 ? 08:57, ?a ?crit : Hello, > >I'm using malloc for Varnish. I'm using a php application which runs deamons for live streaming (text-streaming) and has a big issue with memory, also memory leak so I desided to use echo 1 > /proc/sys/vm/drop_caches. > > I'm not exactly sure of what you mean by leaking memory. On a Linux system, unused memory is considered wasted. The kernel uses this memory to store caches. It will however recycle the pages should an application need some. Here you are dropping the caches, so there is "free" memory, but i dont see the point of doing that. To have a good estimate of how much memory is available to your applications, look on the second line, in the +/- buffers/caches column of the output of the command "free". Here comes the problem! I think this command resets varnish's cache! (not 100% sure.) Malloc cache should be considered as application data, not as clean pages, and should not be affected by this operation (can someone confirm this ?) > >How can I stop memory leaking without resetting Varnish cache? (my php application is the latest version and I can't replace it with an other application) If your application is relatively standard (not using mmap() or other exotic functions (for a php app), you should try to disable cache drops. Nothing should change, and your apps may even run faster. Best regards, --? Aur?lien GUILLAUME -------------- next part -------------- An HTML attachment was scrubbed... URL: From parsimail2001 at yahoo.se Tue Sep 20 08:37:31 2011 From: parsimail2001 at yahoo.se (Your Friend) Date: Tue, 20 Sep 2011 09:37:31 +0100 (BST) Subject: SV: varnish cach question In-Reply-To: References: <1316501868.22837.YahooMailNeo@web26302.mail.ukl.yahoo.com> <1316506160.41015.YahooMailNeo@web26304.mail.ukl.yahoo.com> Message-ID: <1316507851.89878.YahooMailNeo@web26306.mail.ukl.yahoo.com> Great tips! I can use it in some parts of my PHP application. Thanks for the tip, Best regards, Ali ________________________________ On Tue, Sep 20, 2011 at 9:09 AM, wrote: I'll try to take a closer look into my PHP application to fix the memory leak. One tip for PHP memory leak debugging is to to put: echo memory_get_usage(); at the end of each iteration of your loop. ?We find that objects sometimes are responsible for an increase in memory usage, and that you need to do $obj = null; unset($obj); at the end of each iteration to ensure they're properly disposed of, and thus ensure memory isn't leaked. Good luck Rob (Apologies for being slightly off topic for Varnish-misc) -------------- next part -------------- An HTML attachment was scrubbed... URL: From sime at sime.net.au Wed Sep 21 10:00:45 2011 From: sime at sime.net.au (Simon Males) Date: Wed, 21 Sep 2011 20:00:45 +1000 Subject: Wikimedia releases server configurations (incl. Varnish) Message-ID: Hi, Today I saw via Slashdot that the Wikimedia Foundation is releasing[1] there puppet recipes. Of which includes Varnish[2]. [1] http://blog.wikimedia.org/2011/09/19/ever-wondered-how-the-wikimedia-servers-are-configured/ [2] https://gerrit.wikimedia.org/r/gitweb?p=operations/puppet.git;a=tree;f=templates/varnish;hb=HEAD -- Simon Males From amoiz.shine at gmail.com Wed Sep 21 10:15:44 2011 From: amoiz.shine at gmail.com (Sharl.Jimh.Tsin) Date: Wed, 21 Sep 2011 18:15:44 +0800 Subject: Wikimedia releases server configurations (incl. Varnish) In-Reply-To: References: Message-ID: <1316600144.2036.2.camel@sharl-desktop> ? 2011-09-21?? 20:00 +1000?Simon Males??? > Hi, > > Today I saw via Slashdot that the Wikimedia Foundation is releasing[1] > there puppet recipes. Of which includes Varnish[2]. > > [1] http://blog.wikimedia.org/2011/09/19/ever-wondered-how-the-wikimedia-servers-are-configured/ > [2] https://gerrit.wikimedia.org/r/gitweb?p=operations/puppet.git;a=tree;f=templates/varnish;hb=HEAD > thanks for your valuable URL,it is good to learn. -- Best regards, Sharl.Jimh.Tsin (From China **Obviously Taiwan INCLUDED**) Using Gmail? Please read this important notice: http://www.fsf.org/campaigns/jstrap/gmail?10073. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: This is a digitally signed message part URL: From MSchurenko at airg.com Wed Sep 21 17:15:51 2011 From: MSchurenko at airg.com (Matt Schurenko) Date: Wed, 21 Sep 2011 10:15:51 -0700 Subject: varnish stopped resonding Message-ID: <8B5C6F575422414AA91B46C454126B6C02EB2822FB@EXCHMVS.exchange.airg> Hi, I'm running two varnish servers in production (ver 2.1.5). Both are using the same hardware and have the same amount of RAM (48GB). Last night one of the varnish servers stopped responding on port 80. Since we are using HAproxy in front of both varnish servers for load balancing this did not have much effect on our end users. The symptoms of the problem were either a client ( HAproxy, telnet) could not establish a layer 4 connection to varnish or, if a client could establish a connection and issued an HTTP GET, varnish returned nothing, no HTTP headers, nothing. Running "ps -efL | grep varnish | wc -l" I revelaed that there were ~ 500 varnish threads. I am using the default configuration with regards to threads (max of 500). To me it seemed that when a client tried to connect to varnish there were no thread available to use so the client just hung there until either it or varnish timeout out and disconnected. Unfortunately I didn't have the good sense to capture a "varnishastat -l" after this happened. I was focused on getting the server back to a working state so I ended up restarting varnishd. Here is my varnishd command line followed by a current "varnishstat -l" (I have set the weight for this server to be lower than the other varnish instance so that the cache can "warm up". There is typically 4 x as much traffic): /usr/local/sbin/varnishd -s file,/tmp/varnish-cache,60G -T 127.0.0.1:2000 -a 0.0.0.0:80 -t 604800 -f /usr/local/etc/varnish/default.vcl -p http_headers 384 -p connect_timeout 4.0 client_conn 4985179 120.45 Client connections accepted client_drop 0 0.00 Connection dropped, no sess/wrk client_req 4907077 118.56 Client requests received cache_hit 3356368 81.09 Cache hits cache_hitpass 0 0.00 Cache hits for pass cache_miss 1550606 37.46 Cache misses backend_conn 1530014 36.97 Backend conn. success backend_unhealthy 0 0.00 Backend conn. not attempted backend_busy 0 0.00 Backend conn. too many backend_fail 0 0.00 Backend conn. failures backend_reuse 20690 0.50 Backend conn. reuses backend_toolate 0 0.00 Backend conn. was closed backend_recycle 20691 0.50 Backend conn. recycles backend_unused 0 0.00 Backend conn. unused fetch_head 1 0.00 Fetch head fetch_length 33270 0.80 Fetch with Length fetch_chunked 1517362 36.66 Fetch chunked fetch_eof 0 0.00 Fetch EOF fetch_bad 0 0.00 Fetch had bad headers fetch_close 70 0.00 Fetch wanted close fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed fetch_zero 0 0.00 Fetch zero len fetch_failed 0 0.00 Fetch failed n_sess_mem 262 . N struct sess_mem n_sess 68 . N struct sess n_object 1550439 . N struct object n_vampireobject 0 . N unresurrected objects n_objectcore 1550458 . N struct objectcore n_objecthead 1550412 . N struct objecthead n_smf 3100879 . N struct smf n_smf_frag 0 . N small free smf n_smf_large 1 . N large free smf n_vbe_conn 1 . N struct vbe_conn n_wrk 29 . N worker threads n_wrk_create 870 0.02 N worker threads created n_wrk_failed 0 0.00 N worker threads not created n_wrk_max 3128 0.08 N worker threads limited n_wrk_queue 0 0.00 N queued work requests n_wrk_overflow 4696 0.11 N overflowed work requests n_wrk_drop 0 0.00 N dropped work requests n_backend 2 . N backends n_expired 157 . N expired objects n_lru_nuked 0 . N LRU nuked objects n_lru_saved 0 . N LRU saved objects n_lru_moved 3077705 . N LRU moved objects n_deathrow 0 . N objects on deathrow losthdr 0 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 4817364 116.39 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 4985176 120.45 Total Sessions s_req 4907077 118.56 Total Requests s_pipe 0 0.00 Total pipe s_pass 102 0.00 Total pass s_fetch 1550703 37.47 Total fetch s_hdrbytes 1590643697 38431.56 Total header bytes s_bodybytes 17647134982 426372.59 Total body bytes sess_closed 4522198 109.26 Session Closed sess_pipeline 4 0.00 Session Pipeline sess_readahead 8 0.00 Session Read Ahead sess_linger 469810 11.35 Session Linger sess_herd 476189 11.51 Session herd shm_records 297887487 7197.26 SHM records shm_writes 23469767 567.05 SHM writes shm_flushes 0 0.00 SHM flushes due to overflow shm_cont 51830 1.25 SHM MTX contention shm_cycles 137 0.00 SHM cycles through buffer sm_nreq 3101298 74.93 allocator requests sm_nobj 3100878 . outstanding allocations sm_balloc 13670006784 . bytes allocated sm_bfree 50754502656 . bytes free sma_nreq 0 0.00 SMA allocator requests sma_nobj 0 . SMA outstanding allocations sma_nbytes 0 . SMA outstanding bytes sma_balloc 0 . SMA bytes allocated sma_bfree 0 . SMA bytes free sms_nreq 5 0.00 SMS allocator requests sma_nobj 0 . SMA outstanding allocations sma_nbytes 0 . SMA outstanding bytes sma_balloc 0 . SMA bytes allocated sma_bfree 0 . SMA bytes free sms_nreq 5 0.00 SMS allocator requests sms_nobj 0 . SMS outstanding allocations sms_nbytes 0 . SMS outstanding bytes sms_balloc 2090 . SMS bytes allocated sms_bfree 2090 . SMS bytes freed backend_req 1550708 37.47 Backend requests made n_vcl 1 0.00 N vcl total n_vcl_avail 1 0.00 N vcl available n_vcl_discard 0 0.00 N vcl discarded n_purge 1 . N total active purges n_purge_add 1 0.00 N new purges added n_purge_retire 0 0.00 N old purges deleted n_purge_obj_test 0 0.00 N objects tested n_purge_re_test 0 0.00 N regexps tested against n_purge_dups 0 0.00 N duplicate purges removed hcb_nolock 4906976 118.56 HCB Lookups without lock hcb_lock 1550518 37.46 HCB Lookups with lock hcb_insert 1550517 37.46 HCB Inserts esi_parse 0 0.00 Objects ESI parsed (unlock) esi_errors 0 0.00 ESI parse errors (unlock) accept_fail 0 0.00 Accept failures client_drop_late 0 0.00 Connection dropped late uptime 41389 1.00 Client uptime backend_retry 0 0.00 Backend conn. retry dir_dns_lookups 0 0.00 DNS director lookups dir_dns_failed 0 0.00 DNS director failed lookups dir_dns_hit 0 0.00 DNS director cached lookups hit dir_dns_cache_full 0 0.00 DNS director full dnscache fetch_1xx 0 0.00 Fetch no body (1xx) fetch_204 0 0.00 Fetch no body (204) fetch_304 0 0.00 Fetch no body (304) Could there be something wrong with my configuration that caused this problem? Thanks Matt Schurenko Systems Administrator airG(r) Share Your World Suite 710, 1133 Melville Street Vancouver, BC V6E 4E5 P: +1.604.408.2228 F: +1.866.874.8136 E: MSchurenko at airg.com W: www.airg.com airG is one of BC's Top 55 Employers and Canada's Top Employers for Young People P Please consider the environment before printing this e-mail. The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material communicated under NDA. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From k at kevinkevin.com Wed Sep 21 17:24:55 2011 From: k at kevinkevin.com (Kevin) Date: Wed, 21 Sep 2011 13:24:55 -0400 Subject: Pipe all remaining requests if URI / condition is met Message-ID: <020301cc7883$6288e790$279ab6b0$@com> Hello, I'm curious if its possible to set varnish to pipe all subsequent requests (ignoring any rules that say otherwise) if certain specific conditions are met (i.e. user visits a login page and is returned to the main page). I am having problems with cookies, sessions among many other things and would just like to pipe all logged in traffic after the login page is visited. I realize this isn't efficient, but for troubleshooting purposes and limitations by the CMS I'm using (joomla) , doing this will at least return some function to the site itself. Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From lampe at hauke-lampe.de Wed Sep 21 19:57:16 2011 From: lampe at hauke-lampe.de (Hauke Lampe) Date: Wed, 21 Sep 2011 21:57:16 +0200 Subject: Pipe all remaining requests if URI / condition is met In-Reply-To: <020301cc7883$6288e790$279ab6b0$@com> References: <020301cc7883$6288e790$279ab6b0$@com> Message-ID: <4E7A419C.3020500@hauke-lampe.de> On 21.09.2011 19:24, Kevin wrote: > I'm curious if its possible to set varnish to pipe all subsequent requests > (ignoring any rules that say otherwise) if certain specific conditions are > met (i.e. user visits a login page and is returned to the main page). With the help of a vmod (https://github.com/KristianLyng/libvmod-header), you could add a special cookie to the response and match it early in vcl_recv(). import header; ... if (req.http.Cookie ~ "(^|; )VCLPIPE=1") { return(pipe); } ... if (req.url ~ "^/login") { header.append(beresp.http.Set-Cookie,"VCLPIPE=1; path=/"); } Hauke. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: OpenPGP digital signature URL: From nicole4pt at gmail.com Wed Sep 21 23:35:07 2011 From: nicole4pt at gmail.com (Nicole H.) Date: Wed, 21 Sep 2011 16:35:07 -0700 Subject: A new user with a few questions Message-ID: Hello all I am converting a number of reverse proxy squid servers to Varnish and I have a few issues I haven't been able to solve that I hope you can help me with. I am using Varnish 3.0.1 on FreeBSD 8.X and 3 (smallish) disks for the data 1) Any tunes, recommendations or feelings for/against using ZFS as the file system? I had been using ZFS for squid. I have tried UFS with Soft-Updates off. Seems like UFS would be fine and use less memory but would ZFS provide any real speed difference? I realize more in memory is always better, but I am stuck using older servers with only 4G of memory. 2) When using both ZFS and UFS on FreeBSD, I see a 43G varnish-storage.bin file that has been created on my disk, however df -h only shows 31k or 84K on UFS of disk usage. How can that be? 3) What would be the Varnish equivalent for a squid parameter like this? Basically how to convert/force a full reload request into an If Modified Since? refresh_pattern -i \.xml 0 20% 4320 reload-into-ims 4) When testing, I am seeing these in the headers. I have tried a number of things but I can't seem to remove them. How do I remove these? If not all, especially the Client-Peer IP address. Client-Date: Wed, 21 Sep 2011 01:10:28 GMT Client-Peer: 10.0.2.3:80 Client-Response-Num: 1 Thanks! Nicole -------------- next part -------------- An HTML attachment was scrubbed... URL: From mit at stagename.com Thu Sep 22 05:21:13 2011 From: mit at stagename.com (Mit Rowe) Date: Thu, 22 Sep 2011 01:21:13 -0400 Subject: A new user with a few questions In-Reply-To: References: Message-ID: On Wed, Sep 21, 2011 at 7:35 PM, Nicole H. wrote: > > > > > 4) When testing, I am seeing these in the headers. I have tried a number > of things but I can't seem to remove them. > How do I remove these? If not all, especially the Client-Peer IP address. > > Client-Date: Wed, 21 Sep 2011 01:10:28 GMT > Client-Peer: 10.0.2.3:80 > Client-Response-Num: 1 > > I've seen something like that added by the LWP (libwww-perl) library in the request headers. Are you using that for your tests? I think you can remove them with "remove req.http.Client-Date" in the vcl_recv or vcl_fetch sections -Mit > > > Thanks! > > > Nicole > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- Will 'Mit' Rowe Stagename* *1-866-326-3098 mit at stagename.com www.stagename.com Twitter: @stagename *The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of this information by persons or entities other than the intended recipient is prohibited. If you received this transmission in error, please contact the sender and delete all material contained herein from your computer.* -------------- next part -------------- An HTML attachment was scrubbed... URL: From k at kevinkevin.com Thu Sep 22 05:33:16 2011 From: k at kevinkevin.com (Kevin) Date: Thu, 22 Sep 2011 01:33:16 -0400 Subject: manually refreshing varnish cache In-Reply-To: <18834F5BEC10824891FB8B22AC821A5A019831BE@nucleus-srv01.Nucleus.local> References: <012b01cc5915$73dcfeb0$5b96fc10$@com> <18834F5BEC10824891FB8B22AC821A5A019831BE@nucleus-srv01.Nucleus.local> Message-ID: <02e801cc78e9$21fa8100$65ef8300$@com> Hello Mattias, Doesn't varnishreplay work by reading the live log of transactions and reproducing them as such? How would this work on a standby / "cold" varnish server that is not actively generating said log files? Thanks -----Original Message----- From: Mattias Geniar [mailto:mattias at nucleus.be] Sent: Sunday, August 14, 2011 7:29 AM To: Kevin; varnish-misc at varnish-cache.org Subject: RE: manually refreshing varnish cache > The challenge (not really a hard one mind you) would be to regularly update the > cache in said standby varnish server at scheduled intervals so that the stale cache > is not so far out of date when / if I do need to flip that DNS switch. > If anyone has dealt with this scenario specifically ,or just with crawling varnish in > general, I'd like to hear anyone's thoughts! Hi Kevin, You may want to look into the "varnishreplay" command: https://www.varnish-cache.org/docs/2.1/reference/varnishreplay.html You could restart your Varnish on the standby side, run varnishreplay to cache all the URLs you want with a TTL of > 1 day to keep it in cache. Regards, Mattias From 13505749931 at qq.com Thu Sep 22 05:33:24 2011 From: 13505749931 at qq.com (=?ISO-8859-1?B?Q29uZw==?=) Date: Thu, 22 Sep 2011 13:33:24 +0800 Subject: Error 413 Request Entity Too Large Message-ID: always show this error,and i modify the nginx conf.i use varnish 3.0.1,but the same varnish conf in 2.1.5 all is ok~ Error 413 Request Entity Too Large Request Entity Too Large Guru Meditation: XID: 1978350183 ========================================= my nginx.conf user www www; worker_processes 16; error_log /var/log/nginx/error.log crit; pid /usr/local/nginx/nginx.pid; #Specifies the value for maximum file descriptors that can be opened by this process. worker_rlimit_nofile 65535; events { use epoll; worker_connections 65535; } http { include mime.types; default_type application/octet-stream; #charset gb2312; #server_names_hash_bucket_size 128; #client_header_buffer_size 128k; large_client_header_buffers 4 4k; client_max_body_size 20m; sendfile on; tcp_nopush on; keepalive_timeout 60; tcp_nodelay on; fastcgi_connect_timeout 300; fastcgi_send_timeout 300; fastcgi_read_timeout 300; fastcgi_buffer_size 64k; fastcgi_buffers 4 64k; fastcgi_busy_buffers_size 128k; fastcgi_temp_file_write_size 128k; gzip on; gzip_min_length 1k; gzip_buffers 4 16k; gzip_http_version 1.0; gzip_comp_level 2; gzip_types text/plain application/x-javascript text/css application/xml; gzip_vary on; proxy_ignore_client_abort off; proxy_set_header X-Forwarded-For $remote_addr; #proxy_connect_timeout 90; #proxy_send_timeout 90; #proxy_read_timeout 90; #proxy_buffer_size 4k; #proxy_buffers 4 32k; #proxy_busy_buffers_size 64k; #proxy_temp_file_write_size 64k; #limit_zone crawler $binary_remote_addr 10m; ############################# include website/*.conf; ############################# } -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian.albrecht at easynet.com Thu Sep 22 08:43:49 2011 From: christian.albrecht at easynet.com (Christian Albrecht) Date: Thu, 22 Sep 2011 10:43:49 +0200 Subject: Error 413 Request Entity Too Large In-Reply-To: References: Message-ID: <4E7AF545.1000800@easynet.com> Hi, On 09/22/2011 07:33 AM, Cong wrote: > always show this error,and i modify the nginx conf.i use varnish > 3.0.1,but the same varnish conf in 2.1.5 all is ok~ Error 413 Request > Entity Too Large Request Entity Too Large Guru Meditation: XID: > 1978350183 have a look at https://www.varnish-cache.org/trac/ticket/1016 Brgds, Christian From amoiz.shine at gmail.com Thu Sep 22 09:00:02 2011 From: amoiz.shine at gmail.com (Sharl.Jimh.Tsin) Date: Thu, 22 Sep 2011 17:00:02 +0800 Subject: Error 413 Request Entity Too Large In-Reply-To: References: Message-ID: <1316682002.1963.3.camel@sharl-desktop> ? 2011-09-22?? 13:33 +0800?Cong??? > always show this error,and i modify the nginx conf.i use varnish > 3.0.1,but the same varnish conf in 2.1.5 all is ok~ Error 413 Request > Entity Too Large Request Entity Too Large Guru Meditation: XID: > 1978350183 ========================================= my nginx.conf > user www www; worker_processes 16; error_log /var/log/nginx/error.log > crit; pid /usr/local/nginx/nginx.pid; #Specifies the value for maximum > file descriptors that can be opened by this process. > worker_rlimit_nofile 65535; events { use epoll; worker_connections > 65535; } http { include mime.types; default_type > application/octet-stream; #charset gb2312; > #server_names_hash_bucket_size 128; #client_header_buffer_size 128k; > large_client_header_buffers 4 4k; client_max_body_size 20m; sendfile > on; tcp_nopush on; keepalive_timeout 60; tcp_nodelay on; > fastcgi_connect_timeout 300; fastcgi_send_timeout 300; > fastcgi_read_timeout 300; fastcgi_buffer_size 64k; fastcgi_buffers 4 > 64k; fastcgi_busy_buffers_size 128k; fastcgi_temp_file_write_size > 128k; gzip on; gzip_min_length 1k; gzip_buffers 4 16k; > gzip_http_version 1.0; gzip_comp_level 2; gzip_types text/plain > application/x-javascript text/css application/xml; gzip_vary on; > proxy_ignore_client_abort off; proxy_set_header X-Forwarded-For > $remote_addr; #proxy_connect_timeout 90; #proxy_send_timeout 90; > #proxy_read_timeout 90; #proxy_buffer_size 4k; #proxy_buffers 4 32k; > #proxy_busy_buffers_size 64k; #proxy_temp_file_write_size 64k; > #limit_zone crawler $binary_remote_addr 10m; > ############################# include website/*.conf; > ############################# } > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc check the changeset of version 2.1.5 and version 3.0.1 for some information. BTW,perhaps you should format your message for easily read before send. -- Best regards, Sharl.Jimh.Tsin (From China **Obviously Taiwan INCLUDED**) Using Gmail? Please read this important notice: http://www.fsf.org/campaigns/jstrap/gmail?10073. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: This is a digitally signed message part URL: From MSchurenko at airg.com Thu Sep 22 16:14:38 2011 From: MSchurenko at airg.com (Matt Schurenko) Date: Thu, 22 Sep 2011 09:14:38 -0700 Subject: varnish stops responding and threads do not decrease Message-ID: <8B5C6F575422414AA91B46C454126B6C02EA54EC49@EXCHMVS.exchange.airg> I posted yesterday regarding this issue. It has happened again; however this time I have not restarted the problem varnish node. I am using the default configuration with regards to threads. It seems that once thread_pool_max hits 500 that the server stops responding to any requests. I stopped all connections from the load balancer to the server; however the threads do not decrease. They remain stuck at ~ 509: [root at mvp14 16328]# grep Threads /proc/`pgrep varnishd|tail -n1`/status Threads: 509 The server has been idle for ~ 45 minutes now and there are only a couple of established connections: [root at mvp14 16328]# netstat -ant | grep -w .*:80 | grep EST tcp 254 0 204.92.101.119:80 192.168.105.32:37554 ESTABLISHED tcp 532 0 204.92.101.119:80 192.168.100.153:57722 ESTABLISHED tcp 0 0 192.168.100.56:38818 204.92.101.124:80 ESTABLISHED There are however quite a number of connections in CLOSE_WAIT: [root at mvp14 16328]# netstat -ant | grep -w .*:80 | grep CLOSE_WAIT | wc -l 1118 Here is my varnishd version: [root at mvp14 16328]# varnishd -V varnishd (varnish-2.1.5 SVN ) Copyright (c) 2006-2009 Linpro AS / Verdens Gang ASrsion: Here are the system limits for varnishd: [root at mvp14 16328]# cat /proc/`pgrep varnishd|tail -n1`/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 10485760 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 397312 397312 processes Max open files 131072 131072 files Max locked memory 32768 32768 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 397312 397312 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Here is some memory info on the server: [root at mvp14 16328]# free -m total used free shared buffers cached Mem: 48299 48170 129 0 92 42815 -/+ buffers/cache: 5262 43037 Swap: 15147 0 15147 Here is my varnishd command line: /usr/local/sbin/varnishd -s file,/tmp/varnish-cache,60G -T 127.0.0.1:2000 -a 0.0.0.0:80 -t 604800 -f /usr/local/etc/varnish/default.vcl -p http_headers 384 -p connect_timeout 4.0 Here is the output from 'varnishstat -1': client_conn 32772547 265.09 Client connections accepted client_drop 13103 0.11 Connection dropped, no sess/wrk client_req 32531681 263.14 Client requests received cache_hit 27525134 222.64 Cache hits cache_hitpass 0 0.00 Cache hits for pass cache_miss 5005404 40.49 Cache misses backend_conn 4954451 40.07 Backend conn. success backend_unhealthy 0 0.00 Backend conn. not attempted backend_busy 0 0.00 Backend conn. too many backend_fail 853 0.01 Backend conn. failures backend_reuse 51728 0.42 Backend conn. reuses backend_toolate 13 0.00 Backend conn. was closed backend_recycle 51742 0.42 Backend conn. recycles backend_unused 0 0.00 Backend conn. unused fetch_head 5 0.00 Fetch head fetch_length 81316 0.66 Fetch with Length fetch_chunked 4924086 39.83 Fetch chunked fetch_eof 0 0.00 Fetch EOF fetch_bad 0 0.00 Fetch had bad headers fetch_close 186 0.00 Fetch wanted close fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed fetch_zero 0 0.00 Fetch zero len fetch_failed 0 0.00 Fetch failed n_sess_mem 1268 . N struct sess_mem n_sess 1174 . N struct sess n_object 4922732 . N struct object n_vampireobject 0 . N unresurrected objects n_objectcore 4923144 . N struct objectcore n_objecthead 4642001 . N struct objecthead n_smf 9639146 . N struct smf n_smf_frag 394705 . N small free smf n_smf_large 0 . N large free smf n_vbe_conn 501 . N struct vbe_conn n_wrk 500 . N worker threads n_wrk_create 3622 0.03 N worker threads created n_wrk_failed 0 0.00 N worker threads not created n_wrk_max 4079 0.03 N worker threads limited n_wrk_queue 502 0.00 N queued work requests n_wrk_overflow 65305 0.53 N overflowed work requests n_wrk_drop 13102 0.11 N dropped work requests n_backend 2 . N backends n_expired 1347 . N expired objects n_lru_nuked 381454 . N LRU nuked objects n_lru_saved 0 . N LRU saved objects n_lru_moved 23327252 . N LRU moved objects n_deathrow 0 . N objects on deathrow losthdr 0 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 31912510 258.13 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 32758443 264.97 Total Sessions s_req 32531681 263.14 Total Requests s_pipe 0 0.00 Total pipe s_pass 1134 0.01 Total pass s_fetch 5005593 40.49 Total fetch s_hdrbytes 10659824012 86223.60 Total header bytes s_bodybytes 129812627152 1050009.12 Total body bytes sess_closed 29276120 236.80 Session Closed sess_pipeline 17 0.00 Session Pipeline sess_readahead 32 0.00 Session Read Ahead sess_linger 3510104 28.39 Session Linger sess_herd 3554241 28.75 Session herd shm_records 1725429324 13956.40 SHM records shm_writes 144491896 1168.74 SHM writes shm_flushes 750 0.01 SHM flushes due to overflow shm_cont 494654 4.00 SHM MTX contention shm_cycles 794 0.01 SHM cycles through buffer sm_nreq 10391973 84.06 allocator requests sm_nobj 9244441 . outstanding allocations sm_balloc 41184530432 . bytes allocated sm_bfree 23239979008 . bytes free sma_nreq 0 0.00 SMA allocator requests sma_nobj 0 . SMA outstanding allocations sma_nbytes 0 . SMA outstanding bytes sma_balloc 0 . SMA bytes allocated sma_bfree 0 . SMA bytes free sms_nreq 945 0.01 SMS allocator requests sms_nobj 0 . SMS outstanding allocations sms_nbytes 0 . SMS outstanding bytes sms_balloc 395010 . SMS bytes allocated sms_bfree 395010 . SMS bytes freed backend_req 5006185 40.49 Backend requests made n_vcl 1 0.00 N vcl total n_vcl_avail 1 0.00 N vcl available n_vcl_discard 0 0.00 N vcl discarded n_purge 1 . N total active purges n_purge_add 1 0.00 N new purges added n_purge_retire 0 0.00 N old purges deleted n_purge_obj_test 0 0.00 N objects tested n_purge_re_test 0 0.00 N regexps tested against n_purge_dups 0 0.00 N duplicate purges removed hcb_nolock 32531033 263.13 HCB Lookups without lock hcb_lock 5005369 40.49 HCB Lookups with lock hcb_insert 5005363 40.49 HCB Inserts esi_parse 0 0.00 Objects ESI parsed (unlock) esi_errors 0 0.00 ESI parse errors (unlock) accept_fail 0 0.00 Accept failures client_drop_late 1 0.00 Connection dropped late uptime 123630 1.00 Client uptime backend_retry 0 0.00 Backend conn. retry dir_dns_lookups 0 0.00 DNS director lookups dir_dns_failed 0 0.00 DNS director failed lookups dir_dns_hit 0 0.00 DNS director cached lookups hit dir_dns_cache_full 0 0.00 DNS director full dnscache fetch_1xx 0 0.00 Fetch no body (1xx) fetch_204 0 0.00 Fetch no body (204) fetch_304 0 0.00 Fetch no body (304) Here is some output from varnishlog (Is this normal?): 1442 TTL - 216036249 RFC 604800 1316705246 0 0 0 0 1442 VCL_call - fetch 1442 VCL_return - deliver 1442 ObjProtocol - HTTP/1.1 1442 ObjStatus - 200 1442 ObjResponse - OK 1442 ObjHeader - Date: Thu, 22 Sep 2011 15:27:25 GMT 1442 ObjHeader - Server: Apache/1.3.41 (Unix) mod_perl/1.30 1442 ObjHeader - x-airg-hasbinary: 1 1442 ObjHeader - x-airg-return-contentType: image%2Fjpeg 1442 ObjHeader - x-airg-interfacestatus: 200 1442 ObjHeader - Content-Type: image/jpeg 0 ExpKill - 184966508 LRU 0 ExpKill - 184967370 LRU 0 ExpKill - 184969553 LRU 0 ExpKill - 184970764 LRU 0 ExpKill - 184971732 LRU 0 ExpKill - 184976538 LRU 0 ExpKill - 184977972 LRU 0 ExpKill - 184988825 LRU 0 ExpKill - 184917719 LRU 0 ExpKill - 184997163 LRU 0 ExpKill - 184940621 LRU 0 ExpKill - 185000270 LRU 0 ExpKill - 185001314 LRU 0 ExpKill - 185003793 LRU 0 ExpKill - 185004913 LRU 0 ExpKill - 183651304 LRU 0 ExpKill - 185010145 LRU 0 ExpKill - 185012162 LRU I also noticed in MRTG that when this happens there is a sudden spike in lru nuked activity. It looks like it went from 0 nukes/sec to ~ 200. Do I have something misconfigured? Is varnish running into some kind of resource limitation (memory, file descriptors) which is causing it to hang? Did I set the cache size to large compared to the amount of physcial RAM I have? I am running the same version of varnish on the exact same server and this has not happened. The only difference is that I am using '-s file,/tmp/varnish-cache,48G ' instead of '-s file,/tmp/varnish-cache,60G'. Any help here would be most appreciated. This is in production right now. Matt Schurenko Systems Administrator airG share your world Suite 710, 1133 Melville Street Vancouver, BC V6E 4E5 P: +1.604.408.2228 F: +1.866.874.8136 E: MSchurenko at airg.com W: www.airg.com From apj at mutt.dk Thu Sep 22 16:33:44 2011 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Thu, 22 Sep 2011 18:33:44 +0200 Subject: varnish stopped resonding In-Reply-To: <8B5C6F575422414AA91B46C454126B6C02EB2822FB@EXCHMVS.exchange.airg> References: <8B5C6F575422414AA91B46C454126B6C02EB2822FB@EXCHMVS.exchange.airg> Message-ID: <20110922163344.GA13866@nerd.dk> On Wed, Sep 21, 2011 at 10:15:51AM -0700, Matt Schurenko wrote: > > I'm running two varnish servers in production (ver 2.1.5). Both are using > the same hardware and have the same amount of RAM (48GB). Last night one of > the varnish servers stopped responding on port 80. Since we are using HAproxy > in front of both varnish servers for load balancing this did not have much > effect on our end users. The symptoms of the problem were either a client ( > HAproxy, telnet) could not establish a layer 4 connection to varnish or, if a > client could establish a connection and issued an HTTP GET, varnish returned > nothing, no HTTP headers, nothing. Clients are stuck in the listen queue because you've run out of workers. Are you sure that the workers are not all stuck trying to get content from the backend? You may be experiencing workers piling up on uncacheable urls or urls, that are being refreshed (where the queue will wait for the first request to complete unless you're using grace) You may also want to consider using malloc instead of file storage. -- Andreas From damon at huddler-inc.com Thu Sep 22 16:47:22 2011 From: damon at huddler-inc.com (Damon Snyder) Date: Thu, 22 Sep 2011 09:47:22 -0700 Subject: varnish stops responding and threads do not decrease In-Reply-To: <8B5C6F575422414AA91B46C454126B6C02EA54EC49@EXCHMVS.exchange.airg> References: <8B5C6F575422414AA91B46C454126B6C02EA54EC49@EXCHMVS.exchange.airg> Message-ID: Hi Matt, We had some instability when running varnish using a file backed cache that was larger than RAM. This was a year or more ago, and I don't recall the exact details of the issue. We resolved it by reducing the size of the file cache to be smaller than memory and the problems went away. It looks like you can also increase your thread pool size. We are using something like -w 100,1600,120 at startup and have a similar connection rate. I would also suggest moving to malloc storage type (we are migrating from file to malloc). Kristian is quoted as saying: If you can not contain your cache in memory, first consider if you really > need that big of a cache. Then consider buying more memory. Then sleep on > it. http://kristianlyng.wordpress.com/2010/01/26/varnish-best-practices/ If you do switch to malloc on this box, try something like -smalloc,40G. You don't want to allocate up to the amount of RAM as varnish needs some additional space for other data and you need to make sure your OS and other processes has some margin as well. For my part, I wish there was an interface into the size of the objects being cached so that you can better estimate the size of the cache you need. I understand varnishsizes can do something like but it doesn't tell you what is currently stored in aggregate or allow you to traverse the objects and their sizes. In any case, you can monitor your hit rate and see what impact dropping the extra 12 GB has. I suppose you could argue that your hit rate is moot if you are not responding to connections so just getting rid of the no-response issue should be net positive. Hope this helps, Damon On Thu, Sep 22, 2011 at 9:14 AM, Matt Schurenko wrote: > I posted yesterday regarding this issue. It has happened again; however > this time I have not restarted the problem varnish node. I am using the > default configuration with regards to threads. It seems that once > thread_pool_max hits 500 that the server stops responding to any requests. I > stopped all connections from the load balancer to the server; however the > threads do not decrease. They remain stuck at ~ 509: > > [root at mvp14 16328]# grep Threads /proc/`pgrep varnishd|tail -n1`/status > Threads: 509 > > The server has been idle for ~ 45 minutes now and there are only a couple > of established connections: > [root at mvp14 16328]# netstat -ant | grep -w .*:80 | grep EST > tcp 254 0 204.92.101.119:80 192.168.105.32:37554 > ESTABLISHED > tcp 532 0 204.92.101.119:80 192.168.100.153:57722 > ESTABLISHED > tcp 0 0 192.168.100.56:38818 204.92.101.124:80 > ESTABLISHED > > There are however quite a number of connections in CLOSE_WAIT: > > [root at mvp14 16328]# netstat -ant | grep -w .*:80 | grep CLOSE_WAIT | wc -l > 1118 > > Here is my varnishd version: > > [root at mvp14 16328]# varnishd -V > varnishd (varnish-2.1.5 SVN ) > Copyright (c) 2006-2009 Linpro AS / Verdens Gang ASrsion: > > Here are the system limits for varnishd: > > [root at mvp14 16328]# cat /proc/`pgrep varnishd|tail -n1`/limits > Limit Soft Limit Hard Limit Units > Max cpu time unlimited unlimited seconds > Max file size unlimited unlimited bytes > Max data size unlimited unlimited bytes > Max stack size 10485760 unlimited bytes > Max core file size 0 unlimited bytes > Max resident set unlimited unlimited bytes > Max processes 397312 397312 > processes > Max open files 131072 131072 files > Max locked memory 32768 32768 bytes > Max address space unlimited unlimited bytes > Max file locks unlimited unlimited locks > Max pending signals 397312 397312 signals > Max msgqueue size 819200 819200 bytes > Max nice priority 0 0 > Max realtime priority 0 0 > > Here is some memory info on the server: > > [root at mvp14 16328]# free -m > total used free shared buffers cached > Mem: 48299 48170 129 0 92 42815 > -/+ buffers/cache: 5262 43037 > Swap: 15147 0 15147 > > Here is my varnishd command line: > > /usr/local/sbin/varnishd -s file,/tmp/varnish-cache,60G -T 127.0.0.1:2000-a > 0.0.0.0:80 -t 604800 -f /usr/local/etc/varnish/default.vcl -p http_headers > 384 -p connect_timeout 4.0 > > Here is the output from 'varnishstat -1': > > client_conn 32772547 265.09 Client connections accepted > client_drop 13103 0.11 Connection dropped, no sess/wrk > client_req 32531681 263.14 Client requests received > cache_hit 27525134 222.64 Cache hits > cache_hitpass 0 0.00 Cache hits for pass > cache_miss 5005404 40.49 Cache misses > backend_conn 4954451 40.07 Backend conn. success > backend_unhealthy 0 0.00 Backend conn. not attempted > backend_busy 0 0.00 Backend conn. too many > backend_fail 853 0.01 Backend conn. failures > backend_reuse 51728 0.42 Backend conn. reuses > backend_toolate 13 0.00 Backend conn. was closed > backend_recycle 51742 0.42 Backend conn. recycles > backend_unused 0 0.00 Backend conn. unused > fetch_head 5 0.00 Fetch head > fetch_length 81316 0.66 Fetch with Length > fetch_chunked 4924086 39.83 Fetch chunked > fetch_eof 0 0.00 Fetch EOF > fetch_bad 0 0.00 Fetch had bad headers > fetch_close 186 0.00 Fetch wanted close > fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed > fetch_zero 0 0.00 Fetch zero len > fetch_failed 0 0.00 Fetch failed > n_sess_mem 1268 . N struct sess_mem > n_sess 1174 . N struct sess > n_object 4922732 . N struct object > n_vampireobject 0 . N unresurrected objects > n_objectcore 4923144 . N struct objectcore > n_objecthead 4642001 . N struct objecthead > n_smf 9639146 . N struct smf > n_smf_frag 394705 . N small free smf > n_smf_large 0 . N large free smf > n_vbe_conn 501 . N struct vbe_conn > n_wrk 500 . N worker threads > n_wrk_create 3622 0.03 N worker threads created > n_wrk_failed 0 0.00 N worker threads not created > n_wrk_max 4079 0.03 N worker threads limited > n_wrk_queue 502 0.00 N queued work requests > n_wrk_overflow 65305 0.53 N overflowed work requests > n_wrk_drop 13102 0.11 N dropped work requests > n_backend 2 . N backends > n_expired 1347 . N expired objects > n_lru_nuked 381454 . N LRU nuked objects > n_lru_saved 0 . N LRU saved objects > n_lru_moved 23327252 . N LRU moved objects > n_deathrow 0 . N objects on deathrow > losthdr 0 0.00 HTTP header overflows > n_objsendfile 0 0.00 Objects sent with sendfile > n_objwrite 31912510 258.13 Objects sent with write > n_objoverflow 0 0.00 Objects overflowing workspace > s_sess 32758443 264.97 Total Sessions > s_req 32531681 263.14 Total Requests > s_pipe 0 0.00 Total pipe > s_pass 1134 0.01 Total pass > s_fetch 5005593 40.49 Total fetch > s_hdrbytes 10659824012 86223.60 Total header bytes > s_bodybytes 129812627152 1050009.12 Total body bytes > sess_closed 29276120 236.80 Session Closed > sess_pipeline 17 0.00 Session Pipeline > sess_readahead 32 0.00 Session Read Ahead > sess_linger 3510104 28.39 Session Linger > sess_herd 3554241 28.75 Session herd > shm_records 1725429324 13956.40 SHM records > shm_writes 144491896 1168.74 SHM writes > shm_flushes 750 0.01 SHM flushes due to overflow > shm_cont 494654 4.00 SHM MTX contention > shm_cycles 794 0.01 SHM cycles through buffer > sm_nreq 10391973 84.06 allocator requests > sm_nobj 9244441 . outstanding allocations > sm_balloc 41184530432 . bytes allocated > sm_bfree 23239979008 . bytes free > sma_nreq 0 0.00 SMA allocator requests > sma_nobj 0 . SMA outstanding allocations > sma_nbytes 0 . SMA outstanding bytes > sma_balloc 0 . SMA bytes allocated > sma_bfree 0 . SMA bytes free > sms_nreq 945 0.01 SMS allocator requests > sms_nobj 0 . SMS outstanding allocations > sms_nbytes 0 . SMS outstanding bytes > sms_balloc 395010 . SMS bytes allocated > sms_bfree 395010 . SMS bytes freed > backend_req 5006185 40.49 Backend requests made > n_vcl 1 0.00 N vcl total > n_vcl_avail 1 0.00 N vcl available > n_vcl_discard 0 0.00 N vcl discarded > n_purge 1 . N total active purges > n_purge_add 1 0.00 N new purges added > n_purge_retire 0 0.00 N old purges deleted > n_purge_obj_test 0 0.00 N objects tested > n_purge_re_test 0 0.00 N regexps tested against > n_purge_dups 0 0.00 N duplicate purges removed > hcb_nolock 32531033 263.13 HCB Lookups without lock > hcb_lock 5005369 40.49 HCB Lookups with lock > hcb_insert 5005363 40.49 HCB Inserts > esi_parse 0 0.00 Objects ESI parsed (unlock) > esi_errors 0 0.00 ESI parse errors (unlock) > accept_fail 0 0.00 Accept failures > client_drop_late 1 0.00 Connection dropped late > uptime 123630 1.00 Client uptime > backend_retry 0 0.00 Backend conn. retry > dir_dns_lookups 0 0.00 DNS director lookups > dir_dns_failed 0 0.00 DNS director failed lookups > dir_dns_hit 0 0.00 DNS director cached lookups hit > dir_dns_cache_full 0 0.00 DNS director full dnscache > fetch_1xx 0 0.00 Fetch no body (1xx) > fetch_204 0 0.00 Fetch no body (204) > fetch_304 0 0.00 Fetch no body (304) > > Here is some output from varnishlog (Is this normal?): > 1442 TTL - 216036249 RFC 604800 1316705246 0 0 0 0 > 1442 VCL_call - fetch > 1442 VCL_return - deliver > 1442 ObjProtocol - HTTP/1.1 > 1442 ObjStatus - 200 > 1442 ObjResponse - OK > 1442 ObjHeader - Date: Thu, 22 Sep 2011 15:27:25 GMT > 1442 ObjHeader - Server: Apache/1.3.41 (Unix) mod_perl/1.30 > 1442 ObjHeader - x-airg-hasbinary: 1 > 1442 ObjHeader - x-airg-return-contentType: image%2Fjpeg > 1442 ObjHeader - x-airg-interfacestatus: 200 > 1442 ObjHeader - Content-Type: image/jpeg > 0 ExpKill - 184966508 LRU > 0 ExpKill - 184967370 LRU > 0 ExpKill - 184969553 LRU > 0 ExpKill - 184970764 LRU > 0 ExpKill - 184971732 LRU > 0 ExpKill - 184976538 LRU > 0 ExpKill - 184977972 LRU > 0 ExpKill - 184988825 LRU > 0 ExpKill - 184917719 LRU > 0 ExpKill - 184997163 LRU > 0 ExpKill - 184940621 LRU > 0 ExpKill - 185000270 LRU > 0 ExpKill - 185001314 LRU > 0 ExpKill - 185003793 LRU > 0 ExpKill - 185004913 LRU > 0 ExpKill - 183651304 LRU > 0 ExpKill - 185010145 LRU > 0 ExpKill - 185012162 LRU > > > I also noticed in MRTG that when this happens there is a sudden spike in > lru nuked activity. It looks like it went from 0 nukes/sec to ~ 200. > > Do I have something misconfigured? Is varnish running into some kind of > resource limitation (memory, file descriptors) which is causing it to hang? > Did I set the cache size to large compared to the amount of physcial RAM I > have? I am running the same version of varnish on the exact same server and > this has not happened. The only difference is that I am using '-s > file,/tmp/varnish-cache,48G ' instead of '-s file,/tmp/varnish-cache,60G'. > > Any help here would be most appreciated. This is in production right now. > > > > Matt Schurenko > Systems Administrator > > airG share your world > Suite 710, 1133 Melville Street > Vancouver, BC V6E 4E5 > P: +1.604.408.2228 > F: +1.866.874.8136 > E: MSchurenko at airg.com > W: www.airg.com > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From MSchurenko at airg.com Thu Sep 22 16:48:57 2011 From: MSchurenko at airg.com (Matt Schurenko) Date: Thu, 22 Sep 2011 09:48:57 -0700 Subject: varnish stopped resonding In-Reply-To: <20110922163344.GA13866@nerd.dk> References: <8B5C6F575422414AA91B46C454126B6C02EB2822FB@EXCHMVS.exchange.airg>, <20110922163344.GA13866@nerd.dk> Message-ID: <8B5C6F575422414AA91B46C454126B6C02EA54EC4B@EXCHMVS.exchange.airg> How can I increase the amount of workers, just increase thread_pool_max? It looks like there are a fair amount of sockets between varnish and the backend server that are stuck in CLOSE_WAIT. Could this be what's using all my workers? I would like to use malloc; however our cache size is huge and, from what I read, it is better to use file if your cache will not fit all in RAM. Matt Schurenko Systems Administrator airG share your world Suite 710, 1133 Melville Street Vancouver, BC V6E 4E5 P: +1.604.408.2228 F: +1.866.874.8136 E: MSchurenko at airg.com W: www.airg.com ________________________________________ From: varnish-misc-bounces at varnish-cache.org [varnish-misc-bounces at varnish-cache.org] On Behalf Of Andreas Plesner Jacobsen [apj at mutt.dk] Sent: Thursday, September 22, 2011 9:33 AM To: varnish-misc at varnish-cache.org Subject: Re: varnish stopped resonding On Wed, Sep 21, 2011 at 10:15:51AM -0700, Matt Schurenko wrote: > > I'm running two varnish servers in production (ver 2.1.5). Both are using > the same hardware and have the same amount of RAM (48GB). Last night one of > the varnish servers stopped responding on port 80. Since we are using HAproxy > in front of both varnish servers for load balancing this did not have much > effect on our end users. The symptoms of the problem were either a client ( > HAproxy, telnet) could not establish a layer 4 connection to varnish or, if a > client could establish a connection and issued an HTTP GET, varnish returned > nothing, no HTTP headers, nothing. Clients are stuck in the listen queue because you've run out of workers. Are you sure that the workers are not all stuck trying to get content from the backend? You may be experiencing workers piling up on uncacheable urls or urls, that are being refreshed (where the queue will wait for the first request to complete unless you're using grace) You may also want to consider using malloc instead of file storage. -- Andreas _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From MSchurenko at airg.com Thu Sep 22 17:05:26 2011 From: MSchurenko at airg.com (Matt Schurenko) Date: Thu, 22 Sep 2011 10:05:26 -0700 Subject: varnish stops responding and threads do not decrease In-Reply-To: References: <8B5C6F575422414AA91B46C454126B6C02EA54EC49@EXCHMVS.exchange.airg> Message-ID: <0B7ADE88-CD1D-4A82-8F6E-C7EF5E078914@airg.com> Thanks Damon. I'll put the cache size down. On the other varnish server cache size is equal to physical ram and we haven't had this problem. I have read Kristian's blog and was using malloc before. It was on a server with 32 GB of ram. I set malloc to 28 GB but for some reason it ended up using ~ 15 GB of swap space. From what I understand varnish performs better using mmap then swap so that's why I changed to the file storage type. I'll also try adjusting the thread configuration on all our varnish nodes. Sent from my iPhone On 2011-09-22, at 9:47 AM, "Damon Snyder" > wrote: Hi Matt, We had some instability when running varnish using a file backed cache that was larger than RAM. This was a year or more ago, and I don't recall the exact details of the issue. We resolved it by reducing the size of the file cache to be smaller than memory and the problems went away. It looks like you can also increase your thread pool size. We are using something like -w 100,1600,120 at startup and have a similar connection rate. I would also suggest moving to malloc storage type (we are migrating from file to malloc). Kristian is quoted as saying: If you can not contain your cache in memory, first consider if you really need that big of a cache. Then consider buying more memory. Then sleep on it. http://kristianlyng.wordpress.com/2010/01/26/varnish-best-practices/ If you do switch to malloc on this box, try something like -smalloc,40G. You don't want to allocate up to the amount of RAM as varnish needs some additional space for other data and you need to make sure your OS and other processes has some margin as well. For my part, I wish there was an interface into the size of the objects being cached so that you can better estimate the size of the cache you need. I understand varnishsizes can do something like but it doesn't tell you what is currently stored in aggregate or allow you to traverse the objects and their sizes. In any case, you can monitor your hit rate and see what impact dropping the extra 12 GB has. I suppose you could argue that your hit rate is moot if you are not responding to connections so just getting rid of the no-response issue should be net positive. Hope this helps, Damon On Thu, Sep 22, 2011 at 9:14 AM, Matt Schurenko <MSchurenko at airg.com> wrote: I posted yesterday regarding this issue. It has happened again; however this time I have not restarted the problem varnish node. I am using the default configuration with regards to threads. It seems that once thread_pool_max hits 500 that the server stops responding to any requests. I stopped all connections from the load balancer to the server; however the threads do not decrease. They remain stuck at ~ 509: [root at mvp14 16328]# grep Threads /proc/`pgrep varnishd|tail -n1`/status Threads: 509 The server has been idle for ~ 45 minutes now and there are only a couple of established connections: [root at mvp14 16328]# netstat -ant | grep -w .*:80 | grep EST tcp 254 0 204.92.101.119:80 192.168.105.32:37554 ESTABLISHED tcp 532 0 204.92.101.119:80 192.168.100.153:57722 ESTABLISHED tcp 0 0 192.168.100.56:38818 204.92.101.124:80 ESTABLISHED There are however quite a number of connections in CLOSE_WAIT: [root at mvp14 16328]# netstat -ant | grep -w .*:80 | grep CLOSE_WAIT | wc -l 1118 Here is my varnishd version: [root at mvp14 16328]# varnishd -V varnishd (varnish-2.1.5 SVN ) Copyright (c) 2006-2009 Linpro AS / Verdens Gang ASrsion: Here are the system limits for varnishd: [root at mvp14 16328]# cat /proc/`pgrep varnishd|tail -n1`/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 10485760 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 397312 397312 processes Max open files 131072 131072 files Max locked memory 32768 32768 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 397312 397312 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Here is some memory info on the server: [root at mvp14 16328]# free -m total used free shared buffers cached Mem: 48299 48170 129 0 92 42815 -/+ buffers/cache: 5262 43037 Swap: 15147 0 15147 Here is my varnishd command line: /usr/local/sbin/varnishd -s file,/tmp/varnish-cache,60G -T 127.0.0.1:2000 -a 0.0.0.0:80 -t 604800 -f /usr/local/etc/varnish/default.vcl -p http_headers 384 -p connect_timeout 4.0 Here is the output from 'varnishstat -1': client_conn 32772547 265.09 Client connections accepted client_drop 13103 0.11 Connection dropped, no sess/wrk client_req 32531681 263.14 Client requests received cache_hit 27525134 222.64 Cache hits cache_hitpass 0 0.00 Cache hits for pass cache_miss 5005404 40.49 Cache misses backend_conn 4954451 40.07 Backend conn. success backend_unhealthy 0 0.00 Backend conn. not attempted backend_busy 0 0.00 Backend conn. too many backend_fail 853 0.01 Backend conn. failures backend_reuse 51728 0.42 Backend conn. reuses backend_toolate 13 0.00 Backend conn. was closed backend_recycle 51742 0.42 Backend conn. recycles backend_unused 0 0.00 Backend conn. unused fetch_head 5 0.00 Fetch head fetch_length 81316 0.66 Fetch with Length fetch_chunked 4924086 39.83 Fetch chunked fetch_eof 0 0.00 Fetch EOF fetch_bad 0 0.00 Fetch had bad headers fetch_close 186 0.00 Fetch wanted close fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed fetch_zero 0 0.00 Fetch zero len fetch_failed 0 0.00 Fetch failed n_sess_mem 1268 . N struct sess_mem n_sess 1174 . N struct sess n_object 4922732 . N struct object n_vampireobject 0 . N unresurrected objects n_objectcore 4923144 . N struct objectcore n_objecthead 4642001 . N struct objecthead n_smf 9639146 . N struct smf n_smf_frag 394705 . N small free smf n_smf_large 0 . N large free smf n_vbe_conn 501 . N struct vbe_conn n_wrk 500 . N worker threads n_wrk_create 3622 0.03 N worker threads created n_wrk_failed 0 0.00 N worker threads not created n_wrk_max 4079 0.03 N worker threads limited n_wrk_queue 502 0.00 N queued work requests n_wrk_overflow 65305 0.53 N overflowed work requests n_wrk_drop 13102 0.11 N dropped work requests n_backend 2 . N backends n_expired 1347 . N expired objects n_lru_nuked 381454 . N LRU nuked objects n_lru_saved 0 . N LRU saved objects n_lru_moved 23327252 . N LRU moved objects n_deathrow 0 . N objects on deathrow losthdr 0 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 31912510 258.13 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 32758443 264.97 Total Sessions s_req 32531681 263.14 Total Requests s_pipe 0 0.00 Total pipe s_pass 1134 0.01 Total pass s_fetch 5005593 40.49 Total fetch s_hdrbytes 10659824012 86223.60 Total header bytes s_bodybytes 129812627152 1050009.12 Total body bytes sess_closed 29276120 236.80 Session Closed sess_pipeline 17 0.00 Session Pipeline sess_readahead 32 0.00 Session Read Ahead sess_linger 3510104 28.39 Session Linger sess_herd 3554241 28.75 Session herd shm_records 1725429324 13956.40 SHM records shm_writes 144491896 1168.74 SHM writes shm_flushes 750 0.01 SHM flushes due to overflow shm_cont 494654 4.00 SHM MTX contention shm_cycles 794 0.01 SHM cycles through buffer sm_nreq 10391973 84.06 allocator requests sm_nobj 9244441 . outstanding allocations sm_balloc 41184530432 . bytes allocated sm_bfree 23239979008 . bytes free sma_nreq 0 0.00 SMA allocator requests sma_nobj 0 . SMA outstanding allocations sma_nbytes 0 . SMA outstanding bytes sma_balloc 0 . SMA bytes allocated sma_bfree 0 . SMA bytes free sms_nreq 945 0.01 SMS allocator requests sms_nobj 0 . SMS outstanding allocations sms_nbytes 0 . SMS outstanding bytes sms_balloc 395010 . SMS bytes allocated sms_bfree 395010 . SMS bytes freed backend_req 5006185 40.49 Backend requests made n_vcl 1 0.00 N vcl total n_vcl_avail 1 0.00 N vcl available n_vcl_discard 0 0.00 N vcl discarded n_purge 1 . N total active purges n_purge_add 1 0.00 N new purges added n_purge_retire 0 0.00 N old purges deleted n_purge_obj_test 0 0.00 N objects tested n_purge_re_test 0 0.00 N regexps tested against n_purge_dups 0 0.00 N duplicate purges removed hcb_nolock 32531033 263.13 HCB Lookups without lock hcb_lock 5005369 40.49 HCB Lookups with lock hcb_insert 5005363 40.49 HCB Inserts esi_parse 0 0.00 Objects ESI parsed (unlock) esi_errors 0 0.00 ESI parse errors (unlock) accept_fail 0 0.00 Accept failures client_drop_late 1 0.00 Connection dropped late uptime 123630 1.00 Client uptime backend_retry 0 0.00 Backend conn. retry dir_dns_lookups 0 0.00 DNS director lookups dir_dns_failed 0 0.00 DNS director failed lookups dir_dns_hit 0 0.00 DNS director cached lookups hit dir_dns_cache_full 0 0.00 DNS director full dnscache fetch_1xx 0 0.00 Fetch no body (1xx) fetch_204 0 0.00 Fetch no body (204) fetch_304 0 0.00 Fetch no body (304) Here is some output from varnishlog (Is this normal?): 1442 TTL - 216036249 RFC 604800 1316705246 0 0 0 0 1442 VCL_call - fetch 1442 VCL_return - deliver 1442 ObjProtocol - HTTP/1.1 1442 ObjStatus - 200 1442 ObjResponse - OK 1442 ObjHeader - Date: Thu, 22 Sep 2011 15:27:25 GMT 1442 ObjHeader - Server: Apache/1.3.41 (Unix) mod_perl/1.30 1442 ObjHeader - x-airg-hasbinary: 1 1442 ObjHeader - x-airg-return-contentType: image%2Fjpeg 1442 ObjHeader - x-airg-interfacestatus: 200 1442 ObjHeader - Content-Type: image/jpeg 0 ExpKill - 184966508 LRU 0 ExpKill - 184967370 LRU 0 ExpKill - 184969553 LRU 0 ExpKill - 184970764 LRU 0 ExpKill - 184971732 LRU 0 ExpKill - 184976538 LRU 0 ExpKill - 184977972 LRU 0 ExpKill - 184988825 LRU 0 ExpKill - 184917719 LRU 0 ExpKill - 184997163 LRU 0 ExpKill - 184940621 LRU 0 ExpKill - 185000270 LRU 0 ExpKill - 185001314 LRU 0 ExpKill - 185003793 LRU 0 ExpKill - 185004913 LRU 0 ExpKill - 183651304 LRU 0 ExpKill - 185010145 LRU 0 ExpKill - 185012162 LRU I also noticed in MRTG that when this happens there is a sudden spike in lru nuked activity. It looks like it went from 0 nukes/sec to ~ 200. Do I have something misconfigured? Is varnish running into some kind of resource limitation (memory, file descriptors) which is causing it to hang? Did I set the cache size to large compared to the amount of physcial RAM I have? I am running the same version of varnish on the exact same server and this has not happened. The only difference is that I am using '-s file,/tmp/varnish-cache,48G ' instead of '-s file,/tmp/varnish-cache,60G'. Any help here would be most appreciated. This is in production right now. Matt Schurenko Systems Administrator airG share your world Suite 710, 1133 Melville Street Vancouver, BC V6E 4E5 P: +1.604.408.2228 F: +1.866.874.8136 E: MSchurenko at airg.com W: www.airg.com _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From nicole4pt at gmail.com Thu Sep 22 19:02:46 2011 From: nicole4pt at gmail.com (Nicole H.) Date: Thu, 22 Sep 2011 12:02:46 -0700 Subject: A new user with a few questions In-Reply-To: References: Message-ID: Hi I tried using: remove req.http.Client-Date; remove req.http.Client-Peer; In vcl_recv and in vcl_fetch to no avail. Any other suggestions? Is this normal behavior I am trying to turn off? Thanks Nicole EX: GET -e -d -s http://cachetest-server1.domain.com/image.png 200 OK Cache-Control: public, max-age=259200 Connection: close Date: Thu, 22 Sep 2011 18:57:28 GMT Via: 1.1 varnish Accept-Ranges: bytes Age: 3 Server: lighttpd Content-Length: 19299 Content-Type: image/png Expires: Sun, 25 Sep 2011 18:57:24 GMT Last-Modified: Thu, 18 Mar 2010 19:03:56 GMT Client-Date: Thu, 22 Sep 2011 18:57:28 GMT Client-Peer: 10.0.2.1:80 Client-Response-Num: 1 On Wed, Sep 21, 2011 at 10:21 PM, Mit Rowe wrote: > > > On Wed, Sep 21, 2011 at 7:35 PM, Nicole H. wrote: > >> >> >> >> >> 4) When testing, I am seeing these in the headers. I have tried a number >> of things but I can't seem to remove them. >> How do I remove these? If not all, especially the Client-Peer IP address. >> >> Client-Date: Wed, 21 Sep 2011 01:10:28 GMT >> Client-Peer: 10.0.2.3:80 >> Client-Response-Num: 1 >> >> > > I've seen something like that added by the LWP (libwww-perl) library in the > request headers. Are you using that for your tests? > > I think you can remove them with "remove req.http.Client-Date" in the > vcl_recv or vcl_fetch sections > > -Mit > > > > > > >> >> >> Thanks! >> >> >> Nicole >> >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > > > -- > Will 'Mit' Rowe > Stagename* > *1-866-326-3098 > mit at stagename.com > www.stagename.com > Twitter: @stagename > > *The information transmitted is intended only for the person or entity to > which it is addressed and may contain confidential and/or privileged > material. Any review, retransmission, dissemination or other use of this > information by persons or entities other than the intended recipient is > prohibited. If you received this transmission in error, please contact the > sender and delete all material contained herein from your computer.* > -- --- Nicole Melody Harrington - UNIX Systems Mistress admin1 at picturetrail.com -- http://www.picturetrail.com Obey or the button gets it! -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Thu Sep 22 19:44:24 2011 From: perbu at varnish-software.com (Per Buer) Date: Thu, 22 Sep 2011 21:44:24 +0200 Subject: A new user with a few questions In-Reply-To: References: Message-ID: Hi. If I'm not mistaken those aren't actually headers sent over the the wire, it's just some LWP internal thingy that gets printed out. "varnishlog" will show you what is actually crossing the wire. Per. On Thu, Sep 22, 2011 at 9:02 PM, Nicole H. wrote: > Hi > I tried using: remove req.http.Client-Date; remove req.http.Client-Peer; > In vcl_recv and in vcl_fetch to no avail. > > Any other suggestions? Is this normal behavior I am trying to turn off? > > > Thanks > > > Nicole > > > EX: > > GET -e -d -s http://cachetest-server1.domain.com/image.png > 200 OK > Cache-Control: public, max-age=259200 > Connection: close > Date: Thu, 22 Sep 2011 18:57:28 GMT > Via: 1.1 varnish > Accept-Ranges: bytes > Age: 3 > Server: lighttpd > Content-Length: 19299 > Content-Type: image/png > Expires: Sun, 25 Sep 2011 18:57:24 GMT > Last-Modified: Thu, 18 Mar 2010 19:03:56 GMT > Client-Date: Thu, 22 Sep 2011 18:57:28 GMT > Client-Peer: 10.0.2.1:80 > Client-Response-Num: 1 > > > > > On Wed, Sep 21, 2011 at 10:21 PM, Mit Rowe wrote: > >> >> >> On Wed, Sep 21, 2011 at 7:35 PM, Nicole H. wrote: >> >>> >>> >>> >>> >>> 4) When testing, I am seeing these in the headers. I have tried a number >>> of things but I can't seem to remove them. >>> How do I remove these? If not all, especially the Client-Peer IP >>> address. >>> >>> Client-Date: Wed, 21 Sep 2011 01:10:28 GMT >>> Client-Peer: 10.0.2.3:80 >>> Client-Response-Num: 1 >>> >>> >> >> I've seen something like that added by the LWP (libwww-perl) library in >> the request headers. Are you using that for your tests? >> >> I think you can remove them with "remove req.http.Client-Date" in the >> vcl_recv or vcl_fetch sections >> >> -Mit >> >> >> >> >> >> >>> >>> >>> Thanks! >>> >>> >>> Nicole >>> >>> >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>> >> >> >> >> -- >> Will 'Mit' Rowe >> Stagename* >> *1-866-326-3098 >> mit at stagename.com >> www.stagename.com >> Twitter: @stagename >> >> *The information transmitted is intended only for the person or entity to >> which it is addressed and may contain confidential and/or privileged >> material. Any review, retransmission, dissemination or other use of this >> information by persons or entities other than the intended recipient is >> prohibited. If you received this transmission in error, please contact the >> sender and delete all material contained herein from your computer.* >> > > > > -- > --- > Nicole Melody Harrington - UNIX Systems Mistress > admin1 at picturetrail.com -- http://www.picturetrail.com > Obey or the button gets it! > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon-lists at jfpossibilities.com Thu Sep 22 20:47:21 2011 From: jon-lists at jfpossibilities.com (Jon Foster) Date: Thu, 22 Sep 2011 13:47:21 -0700 Subject: VSL_OpenStats() Message-ID: <4E7B9ED9.1080203@jfpossibilities.com> I see that the VSL_OpenStats() function was removed from the Varnish API in v3. I got lost trying to follow GIT logs so can someone enlighten me as to what its replaced with? THX - Jon -- Jon Foster JF Possibilities, Inc. jon at jfpossibilities.com 541-410-2760 Making computers work for you! From nicole4pt at gmail.com Thu Sep 22 21:03:00 2011 From: nicole4pt at gmail.com (Nicole H.) Date: Thu, 22 Sep 2011 14:03:00 -0700 Subject: A new user with a few questions In-Reply-To: References: Message-ID: DOH! Ok yes, I completely missed that. My bad. Nicole On Thu, Sep 22, 2011 at 12:44 PM, Per Buer wrote: > Hi. > > If I'm not mistaken those aren't actually headers sent over the the wire, > it's just some LWP internal thingy that gets printed out. "varnishlog" will > show you what is actually crossing the wire. > > Per. > > On Thu, Sep 22, 2011 at 9:02 PM, Nicole H. wrote: > >> Hi >> I tried using: remove req.http.Client-Date; remove >> req.http.Client-Peer; In vcl_recv and in vcl_fetch to no avail. >> >> Any other suggestions? Is this normal behavior I am trying to turn off? >> >> >> Thanks >> >> >> Nicole >> >> >> EX: >> >> GET -e -d -s http://cachetest-server1.domain.com/image.png >> 200 OK >> Cache-Control: public, max-age=259200 >> Connection: close >> Date: Thu, 22 Sep 2011 18:57:28 GMT >> Via: 1.1 varnish >> Accept-Ranges: bytes >> Age: 3 >> Server: lighttpd >> Content-Length: 19299 >> Content-Type: image/png >> Expires: Sun, 25 Sep 2011 18:57:24 GMT >> Last-Modified: Thu, 18 Mar 2010 19:03:56 GMT >> Client-Date: Thu, 22 Sep 2011 18:57:28 GMT >> Client-Peer: 10.0.2.1:80 >> Client-Response-Num: 1 >> >> >> >> >> On Wed, Sep 21, 2011 at 10:21 PM, Mit Rowe wrote: >> >>> >>> >>> On Wed, Sep 21, 2011 at 7:35 PM, Nicole H. wrote: >>> >>>> >>>> >>>> >>>> >>>> 4) When testing, I am seeing these in the headers. I have tried a >>>> number of things but I can't seem to remove them. >>>> How do I remove these? If not all, especially the Client-Peer IP >>>> address. >>>> >>>> Client-Date: Wed, 21 Sep 2011 01:10:28 GMT >>>> Client-Peer: 10.0.2.3:80 >>>> Client-Response-Num: 1 >>>> >>>> >>> >>> I've seen something like that added by the LWP (libwww-perl) library in >>> the request headers. Are you using that for your tests? >>> >>> I think you can remove them with "remove req.http.Client-Date" in the >>> vcl_recv or vcl_fetch sections >>> >>> -Mit >>> >>> >>> >>> >>> >>> >>>> >>>> >>>> Thanks! >>>> >>>> >>>> Nicole >>>> >>>> >>>> >>>> _______________________________________________ >>>> varnish-misc mailing list >>>> varnish-misc at varnish-cache.org >>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>> >>> >>> >>> >>> -- >>> Will 'Mit' Rowe >>> Stagename* >>> *1-866-326-3098 >>> mit at stagename.com >>> www.stagename.com >>> Twitter: @stagename >>> >>> *The information transmitted is intended only for the person or entity >>> to which it is addressed and may contain confidential and/or privileged >>> material. Any review, retransmission, dissemination or other use of this >>> information by persons or entities other than the intended recipient is >>> prohibited. If you received this transmission in error, please contact the >>> sender and delete all material contained herein from your computer.* >>> >> >> >> >> -- >> --- >> Nicole Melody Harrington - UNIX Systems Mistress >> admin1 at picturetrail.com -- http://www.picturetrail.com >> Obey or the button gets it! >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > > > -- > Per Buer, CEO > Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer > *Varnish makes websites fly!* > Whitepapers | Video | > Twitter > > > -- --- Nicole Melody Harrington - UNIX Systems Mistress admin1 at picturetrail.com -- http://www.picturetrail.com Obey or the button gets it! -------------- next part -------------- An HTML attachment was scrubbed... URL: From misha at evri.com Sat Sep 24 06:47:30 2011 From: misha at evri.com (Michael Bolotski) Date: Fri, 23 Sep 2011 23:47:30 -0700 Subject: Huge file cache size even though command line specifies 5G? Message-ID: Hello. Any idea why Varnish would create a huge 145GB file when I'm asking it to use only 5GB? The command line is below, and the listing of the file directory is under it. Any thoughts? Any insight is greatly appreciated. /usr/sbin/varnishd -P /var/run/varnish.pid -a :25000 -T :6096 -p thread_pools 2 -p thread_pool_max 4000 -p thread_pool_min 400 -p thread_pool_add_delay 2ms -p session_linger 50 -f /etc/varnish/evri.vcl -u varnish -g varnish -h classic -s file,/var/lib/varnish/datafile.bin,5G ls -l /var/lib/varnish/ total 25284032 drwxr-xr-x 2 root root 4096 Sep 23 23:13 df8-29 -rw------- 1 root root 145840627712 Sep 23 23:45 datafile.bin == Misha Bolotski Evri, Inc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lampe at hauke-lampe.de Sat Sep 24 12:22:02 2011 From: lampe at hauke-lampe.de (Hauke Lampe) Date: Sat, 24 Sep 2011 14:22:02 +0200 Subject: Pipe all remaining requests if URI / condition is met In-Reply-To: <02a401cc78e1$3b6018b0$b2204a10$@com> References: <020301cc7883$6288e790$279ab6b0$@com> <4E7A419C.3020500@hauke-lampe.de> <02a401cc78e1$3b6018b0$b2204a10$@com> Message-ID: <4E7DCB6A.6000605@hauke-lampe.de> Hello Kevin. On 22.09.2011 06:36, Kevin wrote: > I am using varnish 2.1.5 and am getting this make error : vmods are a new feature since varnish 3.0 and not supported in earlier versions. I don't know if there's a way to add multiple headers to the response with varnish 2.x. You'd probably have to encode everything into a single Set-Cookie: header. Hauke. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: OpenPGP digital signature URL: From mailingoijv at gmail.com Mon Sep 26 13:21:24 2011 From: mailingoijv at gmail.com (John S.) Date: Mon, 26 Sep 2011 15:21:24 +0200 Subject: Varnish with many TIME_WAIT sockets and traffic problems Message-ID: Hi, We have an existing infrastructure using Squid for reverse proxy, serving images with nginx as backend on a Debian GNU/Linux 2.6.32-5-amd64 system. We have several hosts behind a pf/relayd load balancer, each machine is a dual-core Xeon with 12GB of memory. We are currently evaluating Varnish (varnish-3.0.1 revision cbf1284) and setup a reverse proxy using it, with the same hardware and software configuration than our Squids. When we put it in our pool of servers, it gets half traffic than our Squids, and we have a lot of sockets stuck in TIME_WAIT state, with few established connexions, cf output of netstat on the varnish host : 6 CLOSING 259 ESTABLISHED 32 FIN_WAIT1 755 FIN_WAIT2 20 LAST_ACK 33 SYN_RECV 2781 TIME_WAIT When we disable keep-alive option the sockets stats return back to normal, but performance is of course very poor and traffic is always half than a squid. On the pf/relayd gateway, we had at the same time ~2110 states related to our Varnish host (Squid hosts were at ~5030), with the following repartition (varnish host only) : 35 CLOSING:ESTABLISHED 3 CLOSING:FIN_WAIT_2 314 ESTABLISHED:ESTABLISHED 5 ESTABLISHED:FIN_WAIT_2 464 FIN_WAIT_2:ESTABLISHED 1100 FIN_WAIT_2:FIN_WAIT_2 11 PROXY:SRC 177 TIME_WAIT:TIME_WAIT We used tcpdump on our varnish host to analyse the traffic, and nothing seems wrong at this level (connections are properly closed). After some research we found this thread https://www.varnish-cache.org/lists/pipermail/varnish-misc/2011-September/006805.html and tried a 10GB malloc cache instead of our previous 60GB disk cache, to no avail. Nothing wrong in syslog. Do we have something misconfigured, or is Varnish running into some kind of resource limitation ? Following some complementary stats. # varnishstat -1 client_conn 964002 77.75 Client connections accepted client_drop 0 0.00 Connection dropped, no sess/wrk client_req 3371261 271.90 Client requests received cache_hit 3225106 260.11 Cache hits cache_hitpass 4 0.00 Cache hits for pass cache_miss 146003 11.78 Cache misses backend_conn 15964 1.29 Backend conn. success backend_unhealthy 0 0.00 Backend conn. not attempted backend_busy 0 0.00 Backend conn. too many backend_fail 0 0.00 Backend conn. failures backend_reuse 130187 10.50 Backend conn. reuses backend_toolate 0 0.00 Backend conn. was closed backend_recycle 130189 10.50 Backend conn. recycles backend_retry 0 0.00 Backend conn. retry fetch_head 0 0.00 Fetch head fetch_length 144362 11.64 Fetch with Length fetch_chunked 1578 0.13 Fetch chunked fetch_eof 0 0.00 Fetch EOF fetch_bad 0 0.00 Fetch had bad headers fetch_close 0 0.00 Fetch wanted close fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed fetch_zero 0 0.00 Fetch zero len fetch_failed 0 0.00 Fetch failed fetch_1xx 0 0.00 Fetch no body (1xx) fetch_204 0 0.00 Fetch no body (204) fetch_304 0 0.00 Fetch no body (304) n_sess_mem 539 . N struct sess_mem n_sess 306 . N struct sess n_object 143971 . N struct object n_vampireobject 0 . N unresurrected objects n_objectcore 144056 . N struct objectcore n_objecthead 144099 . N struct objecthead n_waitinglist 116 . N struct waitinglist n_vbc 2 . N struct vbc n_wrk 100 . N worker threads n_wrk_create 100 0.01 N worker threads created n_wrk_failed 0 0.00 N worker threads not created n_wrk_max 0 0.00 N worker threads limited n_wrk_lqueue 0 0.00 work request queue length n_wrk_queued 6 0.00 N queued work requests n_wrk_drop 0 0.00 N dropped work requests n_backend 1 . N backends n_expired 1827 . N expired objects n_lru_nuked 0 . N LRU nuked objects n_lru_moved 1383093 . N LRU moved objects losthdr 5 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 2652875 213.96 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 963986 77.75 Total Sessions s_req 3371261 271.90 Total Requests s_pipe 5 0.00 Total pipe s_pass 142 0.01 Total pass s_fetch 145940 11.77 Total fetch s_hdrbytes 921297466 74304.18 Total header bytes s_bodybytes 29623671087 2389198.41 Total body bytes sess_closed 127057 10.25 Session Closed sess_pipeline 12585 1.02 Session Pipeline sess_readahead 6810 0.55 Session Read Ahead sess_linger 3316796 267.51 Session Linger sess_herd 2846740 229.59 Session herd shm_records 138220355 11147.70 SHM records shm_writes 9411107 759.02 SHM writes shm_flushes 0 0.00 SHM flushes due to overflow shm_cont 836 0.07 SHM MTX contention shm_cycles 58 0.00 SHM cycles through buffer sms_nreq 210 0.02 SMS allocator requests sms_nobj 0 . SMS outstanding allocations sms_nbytes 0 . SMS outstanding bytes sms_balloc 87855 . SMS bytes allocated sms_bfree 87855 . SMS bytes freed backend_req 146146 11.79 Backend requests made n_vcl 1 0.00 N vcl total n_vcl_avail 1 0.00 N vcl available n_vcl_discard 0 0.00 N vcl discarded n_ban 1 . N total active bans n_ban_add 1 0.00 N new bans added n_ban_retire 0 0.00 N old bans deleted n_ban_obj_test 0 0.00 N objects tested n_ban_re_test 0 0.00 N regexps tested against n_ban_dups 0 0.00 N duplicate bans removed hcb_nolock 3371117 271.89 HCB Lookups without lock hcb_lock 145899 11.77 HCB Lookups with lock hcb_insert 145899 11.77 HCB Inserts esi_errors 0 0.00 ESI parse errors (unlock) esi_warnings 0 0.00 ESI parse warnings (unlock) accept_fail 0 0.00 Accept failures client_drop_late 0 0.00 Connection dropped late uptime 12399 1.00 Client uptime dir_dns_lookups 0 0.00 DNS director lookups dir_dns_failed 0 0.00 DNS director failed lookups dir_dns_hit 0 0.00 DNS director cached lookups hit dir_dns_cache_full 0 0.00 DNS director full dnscache vmods 0 . Loaded VMODs n_gzip 0 0.00 Gzip operations n_gunzip 1741 0.14 Gunzip operations LCK.sms.creat 1 0.00 Created locks LCK.sms.destroy 0 0.00 Destroyed locks LCK.sms.locks 630 0.05 Lock Operations LCK.sms.colls 0 0.00 Collisions LCK.smp.creat 0 0.00 Created locks LCK.smp.destroy 0 0.00 Destroyed locks LCK.smp.locks 0 0.00 Lock Operations LCK.smp.colls 0 0.00 Collisions LCK.sma.creat 2 0.00 Created locks LCK.sma.destroy 0 0.00 Destroyed locks LCK.sma.locks 297818 24.02 Lock Operations LCK.sma.colls 0 0.00 Collisions LCK.smf.creat 0 0.00 Created locks LCK.smf.destroy 0 0.00 Destroyed locks LCK.smf.locks 0 0.00 Lock Operations LCK.smf.colls 0 0.00 Collisions LCK.hsl.creat 0 0.00 Created locks LCK.hsl.destroy 0 0.00 Destroyed locks LCK.hsl.locks 0 0.00 Lock Operations LCK.hsl.colls 0 0.00 Collisions LCK.hcb.creat 1 0.00 Created locks LCK.hcb.destroy 0 0.00 Destroyed locks LCK.hcb.locks 147895 11.93 Lock Operations LCK.hcb.colls 0 0.00 Collisions LCK.hcl.creat 0 0.00 Created locks LCK.hcl.destroy 0 0.00 Destroyed locks LCK.hcl.locks 0 0.00 Lock Operations LCK.hcl.colls 0 0.00 Collisions LCK.vcl.creat 1 0.00 Created locks LCK.vcl.destroy 0 0.00 Destroyed locks LCK.vcl.locks 134 0.01 Lock Operations LCK.vcl.colls 0 0.00 Collisions LCK.stat.creat 1 0.00 Created locks LCK.stat.destroy 0 0.00 Destroyed locks LCK.stat.locks 539 0.04 Lock Operations LCK.stat.colls 0 0.00 Collisions LCK.sessmem.creat 1 0.00 Created locks LCK.sessmem.destroy 0 0.00 Destroyed locks LCK.sessmem.locks 974191 78.57 Lock Operations LCK.sessmem.colls 0 0.00 Collisions LCK.wstat.creat 1 0.00 Created locks LCK.wstat.destroy 0 0.00 Destroyed locks LCK.wstat.locks 2485702 200.48 Lock Operations LCK.wstat.colls 0 0.00 Collisions LCK.herder.creat 1 0.00 Created locks LCK.herder.destroy 0 0.00 Destroyed locks LCK.herder.locks 1 0.00 Lock Operations LCK.herder.colls 0 0.00 Collisions LCK.wq.creat 2 0.00 Created locks LCK.wq.destroy 0 0.00 Destroyed locks LCK.wq.locks 5972534 481.69 Lock Operations LCK.wq.colls 0 0.00 Collisions LCK.objhdr.creat 145983 11.77 Created locks LCK.objhdr.destroy 1884 0.15 Destroyed locks LCK.objhdr.locks 13487826 1087.82 Lock Operations LCK.objhdr.colls 0 0.00 Collisions LCK.exp.creat 1 0.00 Created locks LCK.exp.destroy 0 0.00 Destroyed locks LCK.exp.locks 160025 12.91 Lock Operations LCK.exp.colls 0 0.00 Collisions LCK.lru.creat 2 0.00 Created locks LCK.lru.destroy 0 0.00 Destroyed locks LCK.lru.locks 145799 11.76 Lock Operations LCK.lru.colls 0 0.00 Collisions LCK.cli.creat 1 0.00 Created locks LCK.cli.destroy 0 0.00 Destroyed locks LCK.cli.locks 4145 0.33 Lock Operations LCK.cli.colls 0 0.00 Collisions LCK.ban.creat 1 0.00 Created locks LCK.ban.destroy 0 0.00 Destroyed locks LCK.ban.locks 1382436 111.50 Lock Operations LCK.ban.colls 0 0.00 Collisions LCK.vbp.creat 1 0.00 Created locks LCK.vbp.destroy 0 0.00 Destroyed locks LCK.vbp.locks 0 0.00 Lock Operations LCK.vbp.colls 0 0.00 Collisions LCK.vbe.creat 1 0.00 Created locks LCK.vbe.destroy 0 0.00 Destroyed locks LCK.vbe.locks 31926 2.57 Lock Operations LCK.vbe.colls 0 0.00 Collisions LCK.backend.creat 1 0.00 Created locks LCK.backend.destroy 0 0.00 Destroyed locks LCK.backend.locks 324303 26.16 Lock Operations LCK.backend.colls 0 0.00 Collisions SMA.s0.c_req 291808 23.53 Allocator requests SMA.s0.c_fail 0 0.00 Allocator failures SMA.s0.c_bytes 5460865273 440427.88 Bytes allocated SMA.s0.c_freed 208057151 16780.16 Bytes freed SMA.s0.g_alloc 287944 . Allocations outstanding SMA.s0.g_bytes 5252808122 . Bytes outstanding SMA.s0.g_space 5484610118 . Bytes available SMA.Transient.c_req 284 0.02 Allocator requests SMA.Transient.c_fail 0 0.00 Allocator failures SMA.Transient.c_bytes 9188773 741.09 Bytes allocated SMA.Transient.c_freed 9188773 741.09 Bytes freed SMA.Transient.g_alloc 0 . Allocations outstanding SMA.Transient.g_bytes 0 . Bytes outstanding SMA.Transient.g_space 0 . Bytes available VBE.default(127.0.0.1,,8080).vcls 1 . VCL references VBE.default(127.0.0.1,,8080).happy 0 . Happy health probes # cat /proc/`pgrep varnishd|tail -n1`/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 131072 131072 files Max locked memory 83968000 83968000 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 16382 16382 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Varnish command line : /usr/sbin/varnishd -P /var/run/varnishd.pid -a :80 -f /etc/varnish/conf.vcl -T localhost:6082 -u varnish -g varnish -t 120 -w 50,2000,120 -S /etc/varnish/secret -s malloc,10G -p thread_pool_add_delay 2 -p thread_pools 2 And our conf.vcl : backend default { .host = "127.0.0.1"; .port = "8080"; } sub vcl_recv { if(req.http.host == "www.example.com"){ return(lookup); } } sub vcl_deliver { remove resp.http.Via; remove resp.http.X-Varnish; remove resp.http.Server; remove resp.http.X-Powered-By; } Thank you for your help. From mjy at geizhals.at Mon Sep 26 13:33:57 2011 From: mjy at geizhals.at (Marinos Yannikos) Date: Mon, 26 Sep 2011 15:33:57 +0200 Subject: Varnish with many TIME_WAIT sockets and traffic problems In-Reply-To: References: Message-ID: <4E807F45.9090104@geizhals.at> On 26.09.2011 15:21, John S. wrote: > software configuration than our Squids. When we put it in our pool of > servers, it gets half traffic than our Squids, and we have a lot of > sockets stuck in TIME_WAIT state, with few established connexions, cf > output of netstat on the varnish host : > 6 CLOSING > 259 ESTABLISHED > 32 FIN_WAIT1 > 755 FIN_WAIT2 > 20 LAST_ACK > 33 SYN_RECV > 2781 TIME_WAIT TIME_WAIT looks OK to me, it's not high. Here are some hints for sysctl settings: http://www.mail-archive.com/varnish-misc at projects.linpro.no/msg01571.html (do NOT set tw_recycle=1 though!). Regarding half the traffic, just check whether you have failed requests (e.g. "varnishncsa -m TxStatus:[45]" or similar), but it can be due to the way Varnish handles keepalives differently (what are your Varnish vs. Squid settings for Keepalives?). Regards, Marinos From mailingoijv at gmail.com Mon Sep 26 14:00:22 2011 From: mailingoijv at gmail.com (John S.) Date: Mon, 26 Sep 2011 16:00:22 +0200 Subject: Varnish with many TIME_WAIT sockets and traffic problems In-Reply-To: <4E807F45.9090104@geizhals.at> References: <4E807F45.9090104@geizhals.at> Message-ID: 2011/9/26 Marinos Yannikos > TIME_WAIT looks OK to me, it's not high. Here are some hints for sysctl settings: http://www.mail-archive.com/varnish-misc at projects.linpro.no/msg01571.html (do NOT set tw_recycle=1 though!). It's not very high but twice as much what we have on Squid at the same time. Thank you for your link, we actually use the same sysctls than for Squid - that is, default values for everything except : net.ipv4.ip_local_port_range = 1024 65535 vm.mmap_min_addr=4096 vm.swappiness = 40 > Regarding half the traffic, just check whether you have failed requests (e.g. "varnishncsa -m TxStatus:[45]" or similar), but it can be due to the way Varnish handles keepalives differently (what are your Varnish vs. Squid settings for Keepalives?). We have a lot of 304 (not modified, I assume this is normal) and some 404, not enough to worry about so it seems to work. Considering keep-alives, we have the default for Varnish : default_grace 10.000000 [seconds] default_keep 0.000000 [seconds] default_ttl 120.000000 [seconds] sess_timeout 5 [seconds] sess_workspace 65536 [bytes] session_linger 50 [ms] session_max 100000 [sessions] And for Squid, we have : pconn_timeout 60 seconds request_timeout 5 seconds half_closed_clients off client_persistent_connections on server_persistent_connections on Let me know if you need any other values. From MSchurenko at airg.com Mon Sep 26 19:43:12 2011 From: MSchurenko at airg.com (Matt Schurenko) Date: Mon, 26 Sep 2011 12:43:12 -0700 Subject: varnish backing file gone missing Message-ID: <8B5C6F575422414AA91B46C454126B6C02EB2823B5@EXCHMVS.exchange.airg> I just noticed that the backing file I configured varnish to use is now gone: [root at mvp13 ~]# ps -ef | grep varnishd root 4419 1 0 Sep07 ? 00:00:00 /usr/local/sbin/varnishd -s file,/tmp/varnish-cache,48G -T 127.0.0.1:2000 -a 0.0.0.0:80 -t 604800 -f /usr/local/etc/varnish/default.vcl -p http_headers 384 -p connect_timeout 4.0 [root at mvp13 ~]# ls /tmp/varnish-cache ls: /tmp/varnish-cache: No such file or directory If I do an lsof I see this: varnishd 4420 nobody 4u REG 8,1 51539607552 62160904 /tmp/varnish-cache (deleted) I'm pretty sure I never deleted it. Is this normal behaviour? What are the implications of this file being removed while varnish is running? Varnishd still appears to be working fine. I have another varnish server running but the backing file is still there. BTW I'm running varnish version 2.1.5. Thanks, Matt Schurenko Systems Administrator airG(r) Share Your World Suite 710, 1133 Melville Street Vancouver, BC V6E 4E5 P: +1.604.408.2228 F: +1.866.874.8136 E: MSchurenko at airg.com W: www.airg.com airG is one of BC's Top 55 Employers and Canada's Top Employers for Young People P Please consider the environment before printing this e-mail. The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material communicated under NDA. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From MSchurenko at airg.com Mon Sep 26 19:54:54 2011 From: MSchurenko at airg.com (Matt Schurenko) Date: Mon, 26 Sep 2011 12:54:54 -0700 Subject: varnish backing file gone missing In-Reply-To: <8B5C6F575422414AA91B46C454126B6C02EB2823B5@EXCHMVS.exchange.airg> References: <8B5C6F575422414AA91B46C454126B6C02EB2823B5@EXCHMVS.exchange.airg> Message-ID: <8B5C6F575422414AA91B46C454126B6C02EB2823B6@EXCHMVS.exchange.airg> Update: I noticed that I mistakenly created a 40GB zero-filled file rather than a 48GB one: dd if=/dev/zero of=/tmp/varnish-cache bs=1G count=40 Not sure if that would make any difference. Matt Schurenko Systems Administrator airG(r) Share Your World Suite 710, 1133 Melville Street Vancouver, BC V6E 4E5 P: +1.604.408.2228 F: +1.866.874.8136 E: MSchurenko at airg.com W: www.airg.com From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Matt Schurenko Sent: September-26-11 12:43 PM To: varnish-misc at varnish-cache.org Subject: varnish backing file gone missing I just noticed that the backing file I configured varnish to use is now gone: [root at mvp13 ~]# ps -ef | grep varnishd root 4419 1 0 Sep07 ? 00:00:00 /usr/local/sbin/varnishd -s file,/tmp/varnish-cache,48G -T 127.0.0.1:2000 -a 0.0.0.0:80 -t 604800 -f /usr/local/etc/varnish/default.vcl -p http_headers 384 -p connect_timeout 4.0 [root at mvp13 ~]# ls /tmp/varnish-cache ls: /tmp/varnish-cache: No such file or directory If I do an lsof I see this: varnishd 4420 nobody 4u REG 8,1 51539607552 62160904 /tmp/varnish-cache (deleted) I'm pretty sure I never deleted it. Is this normal behaviour? What are the implications of this file being removed while varnish is running? Varnishd still appears to be working fine. I have another varnish server running but the backing file is still there. BTW I'm running varnish version 2.1.5. Thanks, Matt Schurenko Systems Administrator airG(r) Share Your World Suite 710, 1133 Melville Street Vancouver, BC V6E 4E5 P: +1.604.408.2228 F: +1.866.874.8136 E: MSchurenko at airg.com W: www.airg.com airG is one of BC's Top 55 Employers and Canada's Top Employers for Young People P Please consider the environment before printing this e-mail. The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material communicated under NDA. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From moseleymark at gmail.com Mon Sep 26 20:00:12 2011 From: moseleymark at gmail.com (Mark Moseley) Date: Mon, 26 Sep 2011 13:00:12 -0700 Subject: varnish backing file gone missing In-Reply-To: <8B5C6F575422414AA91B46C454126B6C02EB2823B6@EXCHMVS.exchange.airg> References: <8B5C6F575422414AA91B46C454126B6C02EB2823B5@EXCHMVS.exchange.airg> <8B5C6F575422414AA91B46C454126B6C02EB2823B6@EXCHMVS.exchange.airg> Message-ID: On Mon, Sep 26, 2011 at 12:54 PM, Matt Schurenko wrote: > Update: > > > > I noticed that I mistakenly created a 40GB ?zero-filled file rather than a > 48GB one: > > > > dd if=/dev/zero of=/tmp/varnish-cache bs=1G count=40 > > > > Not sure if that would make any difference. > > > > Matt Schurenko > Systems Administrator > > > airG? Share Your World > Suite 710, 1133 Melville Street > > Vancouver, BC? V6E 4E5 > > P: +1.604.408.2228 > > F: +1.866.874.8136 > > E: MSchurenko at airg.com > > W:?www.airg.com > > From: varnish-misc-bounces at varnish-cache.org > [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Matt Schurenko > Sent: September-26-11 12:43 PM > To: varnish-misc at varnish-cache.org > Subject: varnish backing file gone missing > > > > I just noticed that the backing file I configured varnish to use is now > gone: > > > > [root at mvp13 ~]# ps -ef | grep varnishd > > root????? 4419???? 1? 0 Sep07 ???????? 00:00:00 /usr/local/sbin/varnishd -s > file,/tmp/varnish-cache,48G -T 127.0.0.1:2000 -a 0.0.0.0:80 -t 604800 -f > /usr/local/etc/varnish/default.vcl -p http_headers 384 -p connect_timeout > 4.0 > > > > [root at mvp13 ~]# ls /tmp/varnish-cache > > ls: /tmp/varnish-cache: No such file or directory > > > > If I do an lsof I see this: > > > > varnishd?? 4420 nobody??? 4u????? REG??????????????? 8,1 51539607552 > 62160904 /tmp/varnish-cache (deleted) > > > > I?m pretty sure I never deleted it. Is this normal behaviour? What are the > implications of this file being removed while varnish is running? Varnishd > still appears to be working fine. I have another varnish server running but > the backing file is still there. BTW I?m running varnish version 2.1.5. > > > Perhaps a log rotation script deleted it? If so and varnishd is still running, it should show up in lsof with "(deleted)" next to the pathname. From MSchurenko at airg.com Mon Sep 26 22:37:51 2011 From: MSchurenko at airg.com (Matt Schurenko) Date: Mon, 26 Sep 2011 15:37:51 -0700 Subject: varnish backing file gone missing In-Reply-To: References: <8B5C6F575422414AA91B46C454126B6C02EB2823B5@EXCHMVS.exchange.airg> <8B5C6F575422414AA91B46C454126B6C02EB2823B6@EXCHMVS.exchange.airg> Message-ID: <8B5C6F575422414AA91B46C454126B6C02EB2823C0@EXCHMVS.exchange.airg> Yeah it says "deleted" when I use lsof. Is this going to cause a problem eventually? Since the file is gone is the kernel just using RAM alone now? Swap usage hasn't increased. I'm wondering how it is continuing to function. Matt Schurenko Systems Administrator airG share your world Suite 710, 1133 Melville Street Vancouver, BC V6E 4E5 P: +1.604.408.2228 F: +1.866.874.8136 E: MSchurenko at airg.com W: www.airg.com -----Original Message----- From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Mark Moseley Sent: September-26-11 1:00 PM Cc: varnish-misc at varnish-cache.org Subject: Re: varnish backing file gone missing On Mon, Sep 26, 2011 at 12:54 PM, Matt Schurenko wrote: > Update: > > > > I noticed that I mistakenly created a 40GB ?zero-filled file rather than a > 48GB one: > > > > dd if=/dev/zero of=/tmp/varnish-cache bs=1G count=40 > > > > Not sure if that would make any difference. > > > > Matt Schurenko > Systems Administrator > > > airG? Share Your World > Suite 710, 1133 Melville Street > > Vancouver, BC? V6E 4E5 > > P: +1.604.408.2228 > > F: +1.866.874.8136 > > E: MSchurenko at airg.com > > W:?www.airg.com > > From: varnish-misc-bounces at varnish-cache.org > [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Matt Schurenko > Sent: September-26-11 12:43 PM > To: varnish-misc at varnish-cache.org > Subject: varnish backing file gone missing > > > > I just noticed that the backing file I configured varnish to use is now > gone: > > > > [root at mvp13 ~]# ps -ef | grep varnishd > > root????? 4419???? 1? 0 Sep07 ???????? 00:00:00 /usr/local/sbin/varnishd -s > file,/tmp/varnish-cache,48G -T 127.0.0.1:2000 -a 0.0.0.0:80 -t 604800 -f > /usr/local/etc/varnish/default.vcl -p http_headers 384 -p connect_timeout > 4.0 > > > > [root at mvp13 ~]# ls /tmp/varnish-cache > > ls: /tmp/varnish-cache: No such file or directory > > > > If I do an lsof I see this: > > > > varnishd?? 4420 nobody??? 4u????? REG??????????????? 8,1 51539607552 > 62160904 /tmp/varnish-cache (deleted) > > > > I'm pretty sure I never deleted it. Is this normal behaviour? What are the > implications of this file being removed while varnish is running? Varnishd > still appears to be working fine. I have another varnish server running but > the backing file is still there. BTW I'm running varnish version 2.1.5. > > > Perhaps a log rotation script deleted it? If so and varnishd is still running, it should show up in lsof with "(deleted)" next to the pathname. _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From moseleymark at gmail.com Mon Sep 26 22:51:29 2011 From: moseleymark at gmail.com (Mark Moseley) Date: Mon, 26 Sep 2011 15:51:29 -0700 Subject: varnish backing file gone missing In-Reply-To: <8B5C6F575422414AA91B46C454126B6C02EB2823C0@EXCHMVS.exchange.airg> References: <8B5C6F575422414AA91B46C454126B6C02EB2823B5@EXCHMVS.exchange.airg> <8B5C6F575422414AA91B46C454126B6C02EB2823B6@EXCHMVS.exchange.airg> <8B5C6F575422414AA91B46C454126B6C02EB2823C0@EXCHMVS.exchange.airg> Message-ID: On Mon, Sep 26, 2011 at 3:37 PM, Matt Schurenko wrote: > Yeah it says "deleted" when I use lsof. Is this going to cause a problem eventually? Since the file is gone is the kernel just using RAM alone now? Swap usage hasn't increased. I'm wondering how it is continuing to function. > -----Original Message----- > From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Mark Moseley > Sent: September-26-11 1:00 PM > Cc: varnish-misc at varnish-cache.org > Subject: Re: varnish backing file gone missing > > On Mon, Sep 26, 2011 at 12:54 PM, Matt Schurenko wrote: >> Update: >> >> >> >> I noticed that I mistakenly created a 40GB ?zero-filled file rather than a >> 48GB one: >> >> >> >> dd if=/dev/zero of=/tmp/varnish-cache bs=1G count=40 >> >> >> >> Not sure if that would make any difference. >> >> >> >> Matt Schurenko >> Systems Administrator >> >> >> airG? Share Your World >> Suite 710, 1133 Melville Street >> >> Vancouver, BC? V6E 4E5 >> >> P: +1.604.408.2228 >> >> F: +1.866.874.8136 >> >> E: MSchurenko at airg.com >> >> W:?www.airg.com >> >> From: varnish-misc-bounces at varnish-cache.org >> [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Matt Schurenko >> Sent: September-26-11 12:43 PM >> To: varnish-misc at varnish-cache.org >> Subject: varnish backing file gone missing >> >> >> >> I just noticed that the backing file I configured varnish to use is now >> gone: >> >> >> >> [root at mvp13 ~]# ps -ef | grep varnishd >> >> root????? 4419???? 1? 0 Sep07 ???????? 00:00:00 /usr/local/sbin/varnishd -s >> file,/tmp/varnish-cache,48G -T 127.0.0.1:2000 -a 0.0.0.0:80 -t 604800 -f >> /usr/local/etc/varnish/default.vcl -p http_headers 384 -p connect_timeout >> 4.0 >> >> >> >> [root at mvp13 ~]# ls /tmp/varnish-cache >> >> ls: /tmp/varnish-cache: No such file or directory >> >> >> >> If I do an lsof I see this: >> >> >> >> varnishd?? 4420 nobody??? 4u????? REG??????????????? 8,1 51539607552 >> 62160904 /tmp/varnish-cache (deleted) >> >> >> >> I'm pretty sure I never deleted it. Is this normal behaviour? What are the >> implications of this file being removed while varnish is running? Varnishd >> still appears to be working fine. I have another varnish server running but >> the backing file is still there. BTW I'm running varnish version 2.1.5. >> >> >> > > Perhaps a log rotation script deleted it? If so and varnishd is still > running, it should show up in lsof with "(deleted)" next to the > pathname. It'll work as long as varnishd has it open. It'll only get released when varnished closes it. I'd suggest moving to something like /var/{lib,cache,run}/varnish From bstillwell at photobucket.com Mon Sep 26 22:53:09 2011 From: bstillwell at photobucket.com (Bryan Stillwell) Date: Mon, 26 Sep 2011 16:53:09 -0600 Subject: varnishncsa and negative matches on hostname Message-ID: I'm trying to figure out how I could use varnishncsa with a bunch of vhosts and then have an additional catchall for everything else. For example I have something like: /usr/bin/varnishncsa -a -w /var/log/varnish/foo/access.log -D -P /var/run/varnishncsa-foo.pid -m 'RxHeader:Host: (foo[3-9]).example.com' /usr/bin/varnishncsa -a -w /var/log/varnish/bar/access.log -D -P /var/run/varnishncsa-bar.pid -m 'RxHeader:Host: (bar[2-9]).example.com' Which will put logs destined for foo3-foo9 into /var/log/varnish/foo/access.log, and logs destined for bar2-bar9 into /var/log/varnish/bar/access.log. However, I want everything else (foo1, foo2, bar1, baz, etc.) to go into /var/log/varnish/other/access.log. What I believe I need is some kind of negated match (perl's ~! or grep's -v option). Unfortunately I'm not seeing anything like that in the documentation. Does anyone know how to make something like this work right now? Thanks, Bryan From MSchurenko at airg.com Tue Sep 27 01:32:22 2011 From: MSchurenko at airg.com (Matt Schurenko) Date: Mon, 26 Sep 2011 18:32:22 -0700 Subject: varnish backing file gone missing In-Reply-To: References: <8B5C6F575422414AA91B46C454126B6C02EB2823B5@EXCHMVS.exchange.airg> <8B5C6F575422414AA91B46C454126B6C02EB2823B6@EXCHMVS.exchange.airg> <8B5C6F575422414AA91B46C454126B6C02EB2823C0@EXCHMVS.exchange.airg> Message-ID: <62FD4049-0216-4C3E-B80B-26685180EB55@airg.ca> Thanks for the help. It looks like the previous sys admin had a cron job that deleted everything in /tmp. Sent from my iPhone On 2011-09-26, at 3:52 PM, "Mark Moseley" wrote: > On Mon, Sep 26, 2011 at 3:37 PM, Matt Schurenko wrote: >> Yeah it says "deleted" when I use lsof. Is this going to cause a problem eventually? Since the file is gone is the kernel just using RAM alone now? Swap usage hasn't increased. I'm wondering how it is continuing to function. > >> -----Original Message----- >> From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Mark Moseley >> Sent: September-26-11 1:00 PM >> Cc: varnish-misc at varnish-cache.org >> Subject: Re: varnish backing file gone missing >> >> On Mon, Sep 26, 2011 at 12:54 PM, Matt Schurenko wrote: >>> Update: >>> >>> >>> >>> I noticed that I mistakenly created a 40GB zero-filled file rather than a >>> 48GB one: >>> >>> >>> >>> dd if=/dev/zero of=/tmp/varnish-cache bs=1G count=40 >>> >>> >>> >>> Not sure if that would make any difference. >>> >>> >>> >>> Matt Schurenko >>> Systems Administrator >>> >>> >>> airG? Share Your World >>> Suite 710, 1133 Melville Street >>> >>> Vancouver, BC V6E 4E5 >>> >>> P: +1.604.408.2228 >>> >>> F: +1.866.874.8136 >>> >>> E: MSchurenko at airg.com >>> >>> W: www.airg.com >>> >>> From: varnish-misc-bounces at varnish-cache.org >>> [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Matt Schurenko >>> Sent: September-26-11 12:43 PM >>> To: varnish-misc at varnish-cache.org >>> Subject: varnish backing file gone missing >>> >>> >>> >>> I just noticed that the backing file I configured varnish to use is now >>> gone: >>> >>> >>> >>> [root at mvp13 ~]# ps -ef | grep varnishd >>> >>> root 4419 1 0 Sep07 ? 00:00:00 /usr/local/sbin/varnishd -s >>> file,/tmp/varnish-cache,48G -T 127.0.0.1:2000 -a 0.0.0.0:80 -t 604800 -f >>> /usr/local/etc/varnish/default.vcl -p http_headers 384 -p connect_timeout >>> 4.0 >>> >>> >>> >>> [root at mvp13 ~]# ls /tmp/varnish-cache >>> >>> ls: /tmp/varnish-cache: No such file or directory >>> >>> >>> >>> If I do an lsof I see this: >>> >>> >>> >>> varnishd 4420 nobody 4u REG 8,1 51539607552 >>> 62160904 /tmp/varnish-cache (deleted) >>> >>> >>> >>> I'm pretty sure I never deleted it. Is this normal behaviour? What are the >>> implications of this file being removed while varnish is running? Varnishd >>> still appears to be working fine. I have another varnish server running but >>> the backing file is still there. BTW I'm running varnish version 2.1.5. >>> >>> >>> >> >> Perhaps a log rotation script deleted it? If so and varnishd is still >> running, it should show up in lsof with "(deleted)" next to the >> pathname. > > It'll work as long as varnishd has it open. It'll only get released > when varnished closes it. I'd suggest moving to something like > /var/{lib,cache,run}/varnish > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From mit at stagename.com Tue Sep 27 12:44:52 2011 From: mit at stagename.com (Mit Rowe) Date: Tue, 27 Sep 2011 08:44:52 -0400 Subject: varnish backing file gone missing In-Reply-To: <62FD4049-0216-4C3E-B80B-26685180EB55@airg.ca> References: <8B5C6F575422414AA91B46C454126B6C02EB2823B5@EXCHMVS.exchange.airg> <8B5C6F575422414AA91B46C454126B6C02EB2823B6@EXCHMVS.exchange.airg> <8B5C6F575422414AA91B46C454126B6C02EB2823C0@EXCHMVS.exchange.airg> <62FD4049-0216-4C3E-B80B-26685180EB55@airg.ca> Message-ID: Speaking as a sysadmin: That cron job is a pretty standard thing; /tmp is expected to be volatile and i many operating systems (FreeBSD, Ubuntu, Debian) ship with a default cron job that cleans it up, although at least in the case of FreeBSD it has the ability to add an 'exclusion' list. To avoid issues, I usually use /var/run/ for sockets and such. -Mit On Mon, Sep 26, 2011 at 9:32 PM, Matt Schurenko wrote: > Thanks for the help. It looks like the previous sys admin had a cron job > that deleted everything in /tmp. > > Sent from my iPhone > > On 2011-09-26, at 3:52 PM, "Mark Moseley" wrote: > > > On Mon, Sep 26, 2011 at 3:37 PM, Matt Schurenko > wrote: > >> Yeah it says "deleted" when I use lsof. Is this going to cause a problem > eventually? Since the file is gone is the kernel just using RAM alone now? > Swap usage hasn't increased. I'm wondering how it is continuing to function. > > > >> -----Original Message----- > >> From: varnish-misc-bounces at varnish-cache.org [mailto: > varnish-misc-bounces at varnish-cache.org] On Behalf Of Mark Moseley > >> Sent: September-26-11 1:00 PM > >> Cc: varnish-misc at varnish-cache.org > >> Subject: Re: varnish backing file gone missing > >> > >> On Mon, Sep 26, 2011 at 12:54 PM, Matt Schurenko > wrote: > >>> Update: > >>> > >>> > >>> > >>> I noticed that I mistakenly created a 40GB zero-filled file rather > than a > >>> 48GB one: > >>> > >>> > >>> > >>> dd if=/dev/zero of=/tmp/varnish-cache bs=1G count=40 > >>> > >>> > >>> > >>> Not sure if that would make any difference. > >>> > >>> > >>> > >>> Matt Schurenko > >>> Systems Administrator > >>> > >>> > >>> airG? Share Your World > >>> Suite 710, 1133 Melville Street > >>> > >>> Vancouver, BC V6E 4E5 > >>> > >>> P: +1.604.408.2228 > >>> > >>> F: +1.866.874.8136 > >>> > >>> E: MSchurenko at airg.com > >>> > >>> W: www.airg.com > >>> > >>> From: varnish-misc-bounces at varnish-cache.org > >>> [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Matt > Schurenko > >>> Sent: September-26-11 12:43 PM > >>> To: varnish-misc at varnish-cache.org > >>> Subject: varnish backing file gone missing > >>> > >>> > >>> > >>> I just noticed that the backing file I configured varnish to use is now > >>> gone: > >>> > >>> > >>> > >>> [root at mvp13 ~]# ps -ef | grep varnishd > >>> > >>> root 4419 1 0 Sep07 ? 00:00:00 > /usr/local/sbin/varnishd -s > >>> file,/tmp/varnish-cache,48G -T 127.0.0.1:2000 -a 0.0.0.0:80 -t 604800 > -f > >>> /usr/local/etc/varnish/default.vcl -p http_headers 384 -p > connect_timeout > >>> 4.0 > >>> > >>> > >>> > >>> [root at mvp13 ~]# ls /tmp/varnish-cache > >>> > >>> ls: /tmp/varnish-cache: No such file or directory > >>> > >>> > >>> > >>> If I do an lsof I see this: > >>> > >>> > >>> > >>> varnishd 4420 nobody 4u REG 8,1 51539607552 > >>> 62160904 /tmp/varnish-cache (deleted) > >>> > >>> > >>> > >>> I'm pretty sure I never deleted it. Is this normal behaviour? What are > the > >>> implications of this file being removed while varnish is running? > Varnishd > >>> still appears to be working fine. I have another varnish server running > but > >>> the backing file is still there. BTW I'm running varnish version 2.1.5. > >>> > >>> > >>> > >> > >> Perhaps a log rotation script deleted it? If so and varnishd is still > >> running, it should show up in lsof with "(deleted)" next to the > >> pathname. > > > > It'll work as long as varnishd has it open. It'll only get released > > when varnished closes it. I'd suggest moving to something like > > /var/{lib,cache,run}/varnish > > > > _______________________________________________ > > varnish-misc mailing list > > varnish-misc at varnish-cache.org > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- Will 'Mit' Rowe Stagename* *1-866-326-3098 mit at stagename.com www.stagename.com Twitter: @stagename *The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of this information by persons or entities other than the intended recipient is prohibited. If you received this transmission in error, please contact the sender and delete all material contained herein from your computer.* -------------- next part -------------- An HTML attachment was scrubbed... URL: From mailingoijv at gmail.com Tue Sep 27 12:51:41 2011 From: mailingoijv at gmail.com (John S.) Date: Tue, 27 Sep 2011 14:51:41 +0200 Subject: Varnish with many TIME_WAIT sockets and traffic problems In-Reply-To: <4E807F45.9090104@geizhals.at> References: <4E807F45.9090104@geizhals.at> Message-ID: After more than a day of uptime for our varnish, here are some new stats. Varnish have a full cache as of now (10GB in memory) but it is always half-behind our Squids in name of traffic and requests, as you can see : # squid (via squidclient, all Squids have similar values) Number of HTTP requests received: 419578741 Average HTTP requests per minute since start: 23626.1 Hits as % of all requests: 5min: 98.7%, 60min: 98.8% HTTP/s : 393.77 Hits/s : 389.04 Mb/s : 50.03 # varnish (via varnishstat) client_req 25750851 268.75 Client requests received cache_hit 25175218 262.75 Cache hits uptime 95816 1.00 Client uptime HTTP/s : 268.75 Hits/s : 262.75 Mb/s : 21.67 For netstat, Squid has ~12k sockets ESTABLISHED and ~4k TIME_WAIT while Varnish has ~1k and ~8k respectively. Any clue ? As this is in production right now, I will have to rollback to Squid if I can not figure it out. From mjy at geizhals.at Tue Sep 27 13:04:54 2011 From: mjy at geizhals.at (Marinos Yannikos) Date: Tue, 27 Sep 2011 15:04:54 +0200 Subject: Varnish with many TIME_WAIT sockets and traffic problems In-Reply-To: References: <4E807F45.9090104@geizhals.at> Message-ID: <4E81C9F6.5010104@geizhals.at> On 27.09.2011 14:51, John S. wrote: > Any clue ? As this is in production right now, I will have to rollback > to Squid if I can not figure it out. How are you distributing the traffic between your servers? From the bandwidth vs. hits it seems that connections are handled differently, my guess was keepalives lasting longer with Varnish and therefore if your load balancer uses a "least connections" metric or similar, it will simply send more traffic to servers that close connections earlier. Which means that this says nothing about the actual performance of the servers and whether it would get better or worse if you switched all servers to Varnish. How do the HTTP headers look when coming from Varnish vs. Squid? Perhaps they offer some hints. Regards, Marinos From mailingoijv at gmail.com Tue Sep 27 13:41:20 2011 From: mailingoijv at gmail.com (John S.) Date: Tue, 27 Sep 2011 15:41:20 +0200 Subject: Varnish with many TIME_WAIT sockets and traffic problems In-Reply-To: <4E81C9F6.5010104@geizhals.at> References: <4E807F45.9090104@geizhals.at> <4E81C9F6.5010104@geizhals.at> Message-ID: 2011/9/27 Marinos Yannikos : > How are you distributing the traffic between your servers? From the > bandwidth vs. hits it seems that connections are handled differently, my > guess was keepalives lasting longer with Varnish and therefore if your load > balancer uses a "least connections" metric or similar, it will simply send > more traffic to servers that close connections earlier. Which means that > this says nothing about the actual performance of the servers and whether it > would get better or worse if you switched all servers to Varnish. The loadbalancer is an OpenBSD with a relayd pool in round-robin mode, all hosts having the same weight inside the pool. The timeout for an established session is fixed at 600 seconds, which is more than Varnish's default_ttl (120 seconds). > How do the HTTP headers look when coming from Varnish vs. Squid? Perhaps > they offer some hints. For the same file, we have : Squid : HTTP/1.0 200 OK Server: nginx/1.0.4 Date: Tue, 27 Sep 2011 13:09:21 GMT Content-Type: image/jpeg Content-Length: 95888 Last-Modified: Fri, 23 Sep 2011 08:17:19 GMT Expires: Wed, 28 Sep 2011 13:09:21 GMT Cache-Control: max-age=86400 Accept-Ranges: bytes Age: 795 X-Cache: HIT from www.example.com X-Cache-Lookup: HIT from www.example.com:80 Connection: keep-alive Varnish : HTTP/1.1 200 OK Content-Type: image/jpeg Last-Modified: Fri, 23 Sep 2011 08:17:19 GMT Expires: Wed, 28 Sep 2011 13:11:32 GMT Cache-Control: max-age=86400 Content-Length: 95888 Accept-Ranges: bytes Date: Tue, 27 Sep 2011 13:22:36 GMT Age: 664 Connection: keep-alive The size difference between the two is 108 bytes, not enough to explain the traffic difference. Thank you for your time. From roberto.fernandezcrisial at gmail.com Tue Sep 27 15:38:40 2011 From: roberto.fernandezcrisial at gmail.com (=?ISO-8859-1?Q?Roberto_O=2E_Fern=E1ndez_Crisial?=) Date: Tue, 27 Sep 2011 12:38:40 -0300 Subject: Problem with 3.0.1 Message-ID: Hi guys, I just updated my server from 3.0.0 to 3.0.1 and now my messages log shows that child process exited and started as a new one over and over: Sep 27 12:33:32 calamar14 kernel: pid 99555 (varnishd), uid 1002: exited on signal 11 Sep 27 15:33:32 calamar14 varnishd[99057]: Child (99555) Panic message: Assert error in http_GetHdr(), cache_http.c line 266: Condition(l == strlen(hdr + 1)) not true. thread = (cache-worker) ident = FreeBSD,8.2-RELEASE,amd64,-smalloc,-smalloc,-hcritbit,kqueue Sep 27 15:33:32 calamar14 varnishd[99057]: child (99833) Started Sep 27 15:33:32 calamar14 varnishd[99057]: Child (99833) said Child starts Do you have any idea what would be happen? Any suggestion? Thank you! Roberto O. Fern?ndez Crisial @rofc -------------- next part -------------- An HTML attachment was scrubbed... URL: From viapanda at gmail.com Tue Sep 27 15:45:22 2011 From: viapanda at gmail.com (Olivier G) Date: Tue, 27 Sep 2011 17:45:22 +0200 Subject: Problem with 3.0.1 In-Reply-To: References: Message-ID: <92AD77B9-8AE4-4279-A540-285040266ECD@gmail.com> Might possibly be https://www.varnish-cache.org/trac/ticket/994 Triggers for me under load when a backend sends a "Vary". I tried to apply the patch on the ticket with no luck, and ended-up running the latest development version (which solves this). 2 cents - Olivier Le 27 sept. 2011 ? 17:38, Roberto O. Fern?ndez Crisial a ?crit : > Hi guys, > > I just updated my server from 3.0.0 to 3.0.1 and now my messages log shows that child process exited and started as a new one over and over: > > Sep 27 12:33:32 calamar14 kernel: pid 99555 (varnishd), uid 1002: exited on signal 11 > Sep 27 15:33:32 calamar14 varnishd[99057]: Child (99555) Panic message: Assert error in http_GetHdr(), cache_http.c line 266: Condition(l == strlen(hdr + 1)) not true. thread = (cache-worker) ident = FreeBSD,8.2-RELEASE,amd64,-smalloc,-smalloc,-hcritbit,kqueue > Sep 27 15:33:32 calamar14 varnishd[99057]: child (99833) Started > Sep 27 15:33:32 calamar14 varnishd[99057]: Child (99833) said Child starts > > Do you have any idea what would be happen? Any suggestion? > > Thank you! > > Roberto O. Fern?ndez Crisial > @rofc > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From mailingoijv at gmail.com Tue Sep 27 15:49:14 2011 From: mailingoijv at gmail.com (John S.) Date: Tue, 27 Sep 2011 17:49:14 +0200 Subject: Varnish with many TIME_WAIT sockets and traffic problems In-Reply-To: References: <4E807F45.9090104@geizhals.at> <4E81C9F6.5010104@geizhals.at> Message-ID: For information when we stop Varnish and start Squid on this host, traffic gets back to normal very quickly. We can see on our Munin graphs that both in and out traffic goes down when we use Varnish. From roberto.fernandezcrisial at gmail.com Tue Sep 27 17:34:14 2011 From: roberto.fernandezcrisial at gmail.com (=?ISO-8859-1?Q?Roberto_O=2E_Fern=E1ndez_Crisial?=) Date: Tue, 27 Sep 2011 14:34:14 -0300 Subject: Problem with 3.0.1 In-Reply-To: <92AD77B9-8AE4-4279-A540-285040266ECD@gmail.com> References: <92AD77B9-8AE4-4279-A540-285040266ECD@gmail.com> Message-ID: Olivier, Thank you. I've tried to upgrade from GIT instead apply patch, but it does not compile (at make I found some errors). Do you have the patch url? Regards, Roberto. 2011/9/27 Olivier G > Might possibly be https://www.varnish-cache.org/trac/ticket/994 > Triggers for me under load when a backend sends a "Vary". > > I tried to apply the patch on the ticket with no luck, and ended-up running > the latest development version (which solves this). > > 2 cents > > - Olivier > > > Le 27 sept. 2011 ? 17:38, Roberto O. Fern?ndez Crisial a ?crit : > > Hi guys, > > I just updated my server from 3.0.0 to 3.0.1 and now my messages log shows > that child process exited and started as a new one over and over: > > Sep 27 12:33:32 calamar14 kernel: pid 99555 (varnishd), uid 1002: exited on > signal 11 > Sep 27 15:33:32 calamar14 varnishd[99057]: Child (99555) Panic message: > Assert error in http_GetHdr(), cache_http.c line 266: Condition(l == > strlen(hdr + 1)) not true. thread = (cache-worker) ident = > FreeBSD,8.2-RELEASE,amd64,-smalloc,-smalloc,-hcritbit,kqueue > Sep 27 15:33:32 calamar14 varnishd[99057]: child (99833) Started > Sep 27 15:33:32 calamar14 varnishd[99057]: Child (99833) said Child starts > > Do you have any idea what would be happen? Any suggestion? > > Thank you! > > Roberto O. Fern?ndez Crisial > @rofc > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From MSchurenko at airg.com Tue Sep 27 17:36:01 2011 From: MSchurenko at airg.com (Matt Schurenko) Date: Tue, 27 Sep 2011 10:36:01 -0700 Subject: strace causes varnishd to die Message-ID: <8B5C6F575422414AA91B46C454126B6C02EB282410@EXCHMVS.exchange.airg> I'm trying to use strace to troubleshoot an issue with varnishd. Every time I attach to the child process it gets restarted: [root at mvp14 yum.repos.d]# strace -p 17074 Process 17074 attached - interrupt to quit close(6) = 0 write(1, "Child dies", 10) = 10 write(1, "\n", 1) = 1 exit_group(1) = ? Process 17074 detached Is this indicative of a problem? It doesn't seem normal to me. I'm using varnish 2.1.5 on Centos5.7 64-bit. Thanks Matt Schurenko Systems Administrator airG(r) Share Your World Suite 710, 1133 Melville Street Vancouver, BC V6E 4E5 P: +1.604.408.2228 F: +1.866.874.8136 E: MSchurenko at airg.com W: www.airg.com airG is one of BC's Top 55 Employers and Canada's Top Employers for Young People P Please consider the environment before printing this e-mail. The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material communicated under NDA. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From viapanda at gmail.com Tue Sep 27 17:45:59 2011 From: viapanda at gmail.com (Olivier G) Date: Tue, 27 Sep 2011 19:45:59 +0200 Subject: Problem with 3.0.1 In-Reply-To: References: <92AD77B9-8AE4-4279-A540-285040266ECD@gmail.com> Message-ID: If this is a production setup, maybe you should consider other solutions before patching / upgrading to a potentially unstable version? On top of the head: - downgrade Varnish to 3.0.0 (which seemed to work for you) - or prevent your backends from sending vary headers (assuming you can afford to do that, and assuming this is indeed the trigger in your case) Either way, you should be able to get a patch from the changeset linked in the ticket (and possibly others) from here: https://www.varnish-cache.org/trac/changeset/ec3097a5d70b54fffe594cea7f3267ae7c56063b Best, - Olivier Le 27 sept. 2011 ? 19:34, Roberto O. Fern?ndez Crisial a ?crit : > Olivier, > > Thank you. I've tried to upgrade from GIT instead apply patch, but it does not compile (at make I found some errors). > > Do you have the patch url? > > Regards, > Roberto. > > 2011/9/27 Olivier G > Might possibly be https://www.varnish-cache.org/trac/ticket/994 > Triggers for me under load when a backend sends a "Vary". > > I tried to apply the patch on the ticket with no luck, and ended-up running the latest development version (which solves this). > > 2 cents > > - Olivier > > > Le 27 sept. 2011 ? 17:38, Roberto O. Fern?ndez Crisial a ?crit : > >> Hi guys, >> >> I just updated my server from 3.0.0 to 3.0.1 and now my messages log shows that child process exited and started as a new one over and over: >> >> Sep 27 12:33:32 calamar14 kernel: pid 99555 (varnishd), uid 1002: exited on signal 11 >> Sep 27 15:33:32 calamar14 varnishd[99057]: Child (99555) Panic message: Assert error in http_GetHdr(), cache_http.c line 266: Condition(l == strlen(hdr + 1)) not true. thread = (cache-worker) ident = FreeBSD,8.2-RELEASE,amd64,-smalloc,-smalloc,-hcritbit,kqueue >> Sep 27 15:33:32 calamar14 varnishd[99057]: child (99833) Started >> Sep 27 15:33:32 calamar14 varnishd[99057]: Child (99833) said Child starts >> >> Do you have any idea what would be happen? Any suggestion? >> >> Thank you! >> >> Roberto O. Fern?ndez Crisial >> @rofc >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From damon at huddler-inc.com Tue Sep 27 21:06:01 2011 From: damon at huddler-inc.com (Damon Snyder) Date: Tue, 27 Sep 2011 14:06:01 -0700 Subject: strace causes varnishd to die In-Reply-To: <8B5C6F575422414AA91B46C454126B6C02EB282410@EXCHMVS.exchange.airg> References: <8B5C6F575422414AA91B46C454126B6C02EB282410@EXCHMVS.exchange.airg> Message-ID: Hi Matt, There is a ticket open for this. See https://www.varnish-cache.org/trac/ticket/928. You might also find the debugging instructions helpful particularly the part about setting up the debugging instance. See https://www.varnish-cache.org/trac/wiki/DebuggingVarnish. Damon On Tue, Sep 27, 2011 at 10:36 AM, Matt Schurenko wrote: > I?m trying to use strace to troubleshoot an issue with varnishd. Every time > I attach to the child process it gets restarted:**** > > ** ** > > [root at mvp14 yum.repos.d]# strace -p 17074**** > > Process 17074 attached - interrupt to quit**** > > close(6) = 0**** > > write(1, "Child dies", 10) = 10**** > > write(1, "\n", 1) = 1**** > > exit_group(1) = ?**** > > Process 17074 detached**** > > ** ** > > Is this indicative of a problem? It doesn?t seem normal to me. I?m using > varnish 2.1.5 on Centos5.7 64-bit.**** > > ** ** > > Thanks**** > > ** ** > > * * > > * * > > *Matt Schurenko* > Systems Administrator > **** > > *air**G**?** **Share Your World* > Suite 710, 1133 Melville Street**** > > Vancouver, BC V6E 4E5**** > > P: +1.604.408.2228 **** > > F: +1.866.874.8136**** > > E: MSchurenko at airg.com**** > > W: www.airg.com**** > > ** ** > > *airG is one of BC's Top 55 Employers and** > **Canada's Top Employers for Young People***** > > > P Please consider the environment before printing this e-mail.**** > > The information transmitted is intended only for the person or entity to > which it is addressed and may contain confidential and/or privileged > material communicated under NDA. Any review, retransmission, dissemination > or other use of, or taking of any action in reliance upon, this information > by persons or entities other than the intended recipient is prohibited. If > you received this in error, please contact the sender and delete the > material from any computer.**** > > **** > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfrias at gmail.com Tue Sep 27 23:42:22 2011 From: jfrias at gmail.com (Javier Frias) Date: Tue, 27 Sep 2011 19:42:22 -0400 Subject: Varnish + stud (ssl) + PROXY protocol Message-ID: Yeap, I know the varnish ssl question has been beaten to a pulp. But this one is slightly different.. ( don't we all say that ) Has anyone used stud instead of stunnel with Varnish? And somewhat related, Is there any chance of PROXY support in varnish? https://www.varnish-cache.org/trac/ticket/943 I ideally would like to use https://github.com/bumptech/stud instead of stunnel as my ssl frontend proxy for a new setup i'm working on which will be high TPS all ssl serving. ( Thank to ssl AES-NI and Sandy Bridge awesome performance ) Figured I'd check the list to see if anyone had done it before tackling this. -Javier -------------- next part -------------- An HTML attachment was scrubbed... URL: From aurelien.lemaire at smile.fr Tue Sep 27 10:43:16 2011 From: aurelien.lemaire at smile.fr (=?UTF-8?B?QXVyw6lsaWVuIExlbWFpcmU=?=) Date: Tue, 27 Sep 2011 12:43:16 +0200 Subject: varnish nuking agressively all my objects. In-Reply-To: References: <4E71C580.9010403@smile.fr> Message-ID: <4E81A8C4.7030906@smile.fr> Le 16/09/2011 08:55, Jorge Ner?n a ?crit : > > On Thu, Sep 15, 2011 at 11:29, Aur?lien Lemaire > wrote: > > Good day folks, > > First of all, varnish is an outstanding piece of software that my company > and i are addicted to. So thanks to all the coders. > > Here is my problem : > I allocated varnish 1G of RAM on a website that can have more than 2 Go of > possible cacheable objects . Not to worry though as any proxy-cache system > should smartly nuke old objects to make place to new one to live > peacefully within its allocated RAM. And that's where Varnish behave > unexpectedly : each time it need to nuke SOME objects : it nukes ALMOST > ALL of them (often ~80% of my 35k objects) which is quite aggressive ; > thus i lost almost all my cache....IRK ! > > 3 Munin graphs attached to see the problem clearly : big drop each time a > nuking happens. > > To make sure my pbr is about varnish nuking system : i increased from 1G > to 3G(more than the max possible 2G cacheable objects) on another varnish > of this platefom (this website is delivered by multiple front/varnish > server all stricly similar and independant) and this issue disappeared (no > more nuking : no lost of ~80%of my objects) > > Here is my env : > Debian 5.0.8 64 bits on 2.6.32-5-openvz-amd64 kernel > Varnish 2.1.3 SVN 5049:5055(debian package 2.1.3-8) > 200 varnish 's worker threads running constantly (no issue on workers) > 30req/s average with 60/s in peak > > Daemon run as such : > /usr/sbin/varnishd -P /var/run/varnishd.pid -a :80 -T localhost:6082 -S > /etc/varnish/secret -f /etc/varnish/serverx.vcl -w 100,1024 -s > file,/var/lib/varnish/serverx/varnish_storage.bin,3G > > Here a quick varnishstat -1 : > > Is it normal varnish behaviour ? sounds like a bug to me. > Am i missing some tuning (lru_interval) to soften the nuking algo ? > Do you need more info ? > helps appreciated here ;-) > > Regards, Aurelien Lemaire > > > It could be someone downloading a large file (like a ~700Mb iso file) and > varnish nuking objects to make room for this file (even if its configured to > not cache it). > > Try to get a varnishlog trace of the moment the nuking begins. > > -- > Jorge Ner?n > > This website does not content any files bigger than a couple of Mb. Will see what i can do to get the varnishlog during the nuking. -- aurelien lemaire -------------- next part -------------- An HTML attachment was scrubbed... URL: From aurelien.lemaire at smile.fr Wed Sep 28 13:14:53 2011 From: aurelien.lemaire at smile.fr (=?UTF-8?B?QXVyw6lsaWVuIExlbWFpcmU=?=) Date: Wed, 28 Sep 2011 15:14:53 +0200 Subject: varnish nuking agressively all my objects. In-Reply-To: <4E81A8C4.7030906@smile.fr> References: <4E71C580.9010403@smile.fr> <4E81A8C4.7030906@smile.fr> Message-ID: <4E831DCD.9070000@smile.fr> On 27/09/11 12:43, Aur?lien Lemaire wrote: > > Le 16/09/2011 08:55, Jorge Ner?n a ?crit : >> >> On Thu, Sep 15, 2011 at 11:29, Aur?lien Lemaire > > wrote: >> >> Good day folks, >> >> First of all, varnish is an outstanding piece of software that my company >> and i are addicted to. So thanks to all the coders. >> >> Here is my problem : >> I allocated varnish 1G of RAM on a website that can have more than 2 Go >> of possible cacheable objects . Not to worry though as any proxy-cache >> system should smartly nuke old objects to make place to new one to live >> peacefully within its allocated RAM. And that's where Varnish behave >> unexpectedly : each time it need to nuke SOME objects : it nukes ALMOST >> ALL of them (often ~80% of my 35k objects) which is quite aggressive ; >> thus i lost almost all my cache....IRK ! >> >> 3 Munin graphs attached to see the problem clearly : big drop each time a >> nuking happens. >> >> To make sure my pbr is about varnish nuking system : i increased from 1G >> to 3G(more than the max possible 2G cacheable objects) on another varnish >> of this platefom (this website is delivered by multiple front/varnish >> server all stricly similar and independant) and this issue disappeared >> (no more nuking : no lost of ~80%of my objects) >> >> Here is my env : >> Debian 5.0.8 64 bits on 2.6.32-5-openvz-amd64 kernel >> Varnish 2.1.3 SVN 5049:5055(debian package 2.1.3-8) >> 200 varnish 's worker threads running constantly (no issue on workers) >> 30req/s average with 60/s in peak >> >> Daemon run as such : >> /usr/sbin/varnishd -P /var/run/varnishd.pid -a :80 -T localhost:6082 -S >> /etc/varnish/secret -f /etc/varnish/serverx.vcl -w 100,1024 -s >> file,/var/lib/varnish/serverx/varnish_storage.bin,3G >> >> Here a quick varnishstat -1 : >> >> Is it normal varnish behaviour ? sounds like a bug to me. >> Am i missing some tuning (lru_interval) to soften the nuking algo ? >> Do you need more info ? >> helps appreciated here ;-) >> >> Regards, Aurelien Lemaire >> >> >> It could be someone downloading a large file (like a ~700Mb iso file) and >> varnish nuking objects to make room for this file (even if its configured to >> not cache it). >> >> Try to get a varnishlog trace of the moment the nuking begins. >> >> -- >> Jorge Ner?n >> > > This website does not content any files bigger than a couple of Mb. > > Will see what i can do to get the varnishlog during the nuking. > > -- > aurelien lemaire Providing a reduced-to-the-minimum varnishlog on this stressed prod env won't be easy for me moreover i doubt that nuking activity is reported by varnishlog . So meanwhile : My "agressive nuking" problem seems to be known and often-encountered : - http://www.gossamer-threads.com/lists/varnish/dev/9631 - http://www.mail-archive.com/varnish-misc at projects.linpro.no/msg03684.html - Quite some thread without any outcome found by google search where all the people just fix it by increasign the the "-s file" size which only postpone the problem On this first link Artur ( a varnish dev http://www.gossamer-threads.com/lists/engine?user=137;list=varnish ) shortly admit "/... There are problems with the fragmentation of the store.../" on the 1st of april 2009. On the second link there is all the varnishlog and backtrace infos needed to open the bug report to get it fixed Thus is it possible to open this bug report officially as "to be fixed" ? At the same time i'll try to swith to "-s malloc " as suggested by Artur and keep you all informed. Cheers, -- Aur?lien Lemaire -------------- next part -------------- An HTML attachment was scrubbed... URL: From MSchurenko at airg.com Thu Sep 29 23:34:37 2011 From: MSchurenko at airg.com (Matt Schurenko) Date: Thu, 29 Sep 2011 16:34:37 -0700 Subject: lru nuke causes varnish to stop respnding to client requests Message-ID: <8B5C6F575422414AA91B46C454126B6C02EB2824E7@EXCHMVS.exchange.airg> I've been having this problem for a couple weeks now on one of our varnish servers. I have posted a couple times already. What happens is that the server in questions runs fine until the cache gets full. When it starts to lru nuke the number of worker threads jumps up to thread_pool_max and varnish stops responding to any client requests. I have tried this with Centos5.4, 5.7 and now Slackware (all 64-bit ) and the behaviour is the same. I am using varnish version 2.1.5 on a Dell C6105 with 48GB of RAM. Here is my varnishd command line: /usr/local/sbin/varnishd -s file,/tmp/varnish-cache,48g -T 127.0.0.1:2000 -a 0.0.0.0:80 -t 604800 -f /usr/local/etc/varnish/default.vcl -p http_headers 384 -p connect_timeout 4.0 Here is the output from varnishstat -1: client_conn 38582763 240.38 Client connections accepted client_drop 10950 0.07 Connection dropped, no sess/wrk client_req 38298994 238.61 Client requests received cache_hit 32513762 202.57 Cache hits cache_hitpass 0 0.00 Cache hits for pass cache_miss 5784476 36.04 Cache misses backend_conn 5725540 35.67 Backend conn. success backend_unhealthy 0 0.00 Backend conn. not attempted backend_busy 0 0.00 Backend conn. too many backend_fail 1383 0.01 Backend conn. failures backend_reuse 60837 0.38 Backend conn. reuses backend_toolate 33 0.00 Backend conn. was closed backend_recycle 60870 0.38 Backend conn. recycles backend_unused 0 0.00 Backend conn. unused fetch_head 6 0.00 Fetch head fetch_length 93631 0.58 Fetch with Length fetch_chunked 5689433 35.45 Fetch chunked fetch_eof 0 0.00 Fetch EOF fetch_bad 0 0.00 Fetch had bad headers fetch_close 107 0.00 Fetch wanted close fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed fetch_zero 0 0.00 Fetch zero len fetch_failed 1 0.00 Fetch failed n_sess_mem 7138 . N struct sess_mem n_sess 6970 . N struct sess n_object 5047123 . N struct object n_vampireobject 0 . N unresurrected objects n_objectcore 5048435 . N struct objectcore n_objecthead 4955641 . N struct objecthead n_smf 10139770 . N struct smf n_smf_frag 295671 . N small free smf n_smf_large 0 . N large free smf n_vbe_conn 2997 . N struct vbe_conn n_wrk 3000 . N worker threads n_wrk_create 5739 0.04 N worker threads created n_wrk_failed 0 0.00 N worker threads not created n_wrk_max 4063 0.03 N worker threads limited n_wrk_queue 2861 0.02 N queued work requests n_wrk_overflow 83534 0.52 N overflowed work requests n_wrk_drop 10980 0.07 N dropped work requests n_backend 2 . N backends n_expired 2179 . N expired objects n_lru_nuked 862615 . N LRU nuked objects n_lru_saved 0 . N LRU saved objects n_lru_moved 27156180 . N LRU moved objects n_deathrow 0 . N objects on deathrow losthdr 0 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 37294888 232.35 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 38566049 240.27 Total Sessions s_req 38298994 238.61 Total Requests s_pipe 0 0.00 Total pipe s_pass 266 0.00 Total pass s_fetch 5783176 36.03 Total fetch s_hdrbytes 12570989864 78319.53 Total header bytes s_bodybytes 151327304604 942796.38 Total body bytes sess_closed 34673984 216.03 Session Closed sess_pipeline 187 0.00 Session Pipeline sess_readahead 321 0.00 Session Read Ahead sess_linger 3929378 24.48 Session Linger sess_herd 3929559 24.48 Session herd shm_records 2025645664 12620.14 SHM records shm_writes 169640580 1056.89 SHM writes shm_flushes 41 0.00 SHM flushes due to overflow shm_cont 580515 3.62 SHM MTX contention shm_cycles 933 0.01 SHM cycles through buffer sm_nreq 12431620 77.45 allocator requests sm_nobj 9844099 . outstanding allocations sm_balloc 43855261696 . bytes allocated sm_bfree 7684345856 . bytes free sma_nreq 0 0.00 SMA allocator requests sma_nobj 0 . SMA outstanding allocations sma_nbytes 0 . SMA outstanding bytes sma_balloc 0 . SMA bytes allocated sma_bfree 0 . SMA bytes free sms_nreq 1566 0.01 SMS allocator requests sms_nobj 0 . SMS outstanding allocations sms_nbytes 0 . SMS outstanding bytes sms_balloc 656154 . SMS bytes allocated sms_bfree 656154 . SMS bytes freed backend_req 5786381 36.05 Backend requests made n_vcl 1 0.00 N vcl total n_vcl_avail 1 0.00 N vcl available n_vcl_discard 0 0.00 N vcl discarded n_purge 218 . N total active purges n_purge_add 218 0.00 N new purges added n_purge_retire 0 0.00 N old purges deleted n_purge_obj_test 588742 3.67 N objects tested n_purge_re_test 120444323 750.39 N regexps tested against n_purge_dups 0 0.00 N duplicate purges removed hcb_nolock 38301670 238.63 HCB Lookups without lock hcb_lock 5786309 36.05 HCB Lookups with lock hcb_insert 5786305 36.05 HCB Inserts esi_parse 0 0.00 Objects ESI parsed (unlock) esi_errors 0 0.00 ESI parse errors (unlock) accept_fail 0 0.00 Accept failures client_drop_late 30 0.00 Connection dropped late uptime 160509 1.00 Client uptime backend_retry 25 0.00 Backend conn. retry dir_dns_lookups 0 0.00 DNS director lookups dir_dns_failed 0 0.00 DNS director failed lookups dir_dns_hit 0 0.00 DNS director cached lookups hit dir_dns_cache_full 0 0.00 DNS director full dnscache fetch_1xx 0 0.00 Fetch no body (1xx) fetch_204 0 0.00 Fetch no body (204) fetch_304 0 0.00 Fetch no body (304) Even though I have removed the server from our load balancer there are still a lot of requests going to the backend. Maybe these are all queued up requests that varnish is trying to fulfill? Here is some output from varnishlog -c when I try to connect with curl: root at mvp14:~# varnishlog -c 26 SessionOpen c 192.168.8.41 41942 0.0.0.0:80 26 ReqStart c 192.168.8.41 41942 2108342803 26 RxRequest c GET 26 RxURL c / 26 RxProtocol c HTTP/1.1 26 RxHeader c User-Agent: curl/7.21.4 (x86_64-unknown-linux-gnu) libcurl/7.21.4 OpenSSL/0.9.8n zlib/1.2.5 libidn/1.19 26 RxHeader c Host: mvp14.airg.com 26 RxHeader c Accept: */* 26 VCL_call c recv 26 VCL_return c lookup 26 VCL_call c hash 26 VCL_return c hash The connection just hangs here until it times out. Any help would be appreciated. We are trying to replace our squid caching layer with varnish. However if I can't resolve this issue we will have to go back to squid. Thanks, Matt Schurenko Systems Administrator -------------- next part -------------- An HTML attachment was scrubbed... URL: From damon at huddler-inc.com Fri Sep 30 00:29:53 2011 From: damon at huddler-inc.com (Damon Snyder) Date: Thu, 29 Sep 2011 17:29:53 -0700 Subject: lru nuke causes varnish to stop respnding to client requests In-Reply-To: <8B5C6F575422414AA91B46C454126B6C02EB2824E7@EXCHMVS.exchange.airg> References: <8B5C6F575422414AA91B46C454126B6C02EB2824E7@EXCHMVS.exchange.airg> Message-ID: Hi Matt, It looks like you really need to bump up the number of worker threads. From your stats: n_wrk_queue 2861 0.02 N queued work requests**** n_wrk_overflow 83534 0.52 N overflowed work requests**** n_wrk_drop 10980 0.07 N dropped work requests You have a lot of requests that are on the queue waiting for a worker and you have a lot of requests where varnish has given up trying to fullfil with a worker. You can bump the number of works up using the -w command line option to varnishd. I would suggest something like -w 400,1000,120 to start with (the default is -w2,500,300). This says use 400 at a minimum, 1000 as the maximum, and set the timeout to 120s. According to the stats explanation doc your n_wrk_queue and n_wrk_drop should be 0. If you see these numbers going up again, use -w 500,2000,120 or something like that. Hope this helps, Damon On Thu, Sep 29, 2011 at 4:34 PM, Matt Schurenko wrote: > I?ve been having this problem for a couple weeks now on one of our varnish > servers. I have posted a couple times already. What happens is that the > server in questions runs fine until the cache gets full. When it starts to > lru nuke the number of worker threads jumps up to thread_pool_max and > varnish stops responding to any client requests. I have tried this with > Centos5.4, 5.7 and now Slackware (all 64-bit ) and the behaviour is the > same. **** > > ** ** > > I am using varnish version 2.1.5 on a Dell C6105 with 48GB of RAM.**** > > ** ** > > Here is my varnishd command line:**** > > ** ** > > /usr/local/sbin/varnishd -s file,/tmp/varnish-cache,48g -T 127.0.0.1:2000-a > 0.0.0.0:80 -t 604800 -f /usr/local/etc/varnish/default.vcl -p http_headers > 384 -p connect_timeout 4.0**** > > ** ** > > Here is the output from varnishstat -1:**** > > ** ** > > client_conn 38582763 240.38 Client connections accepted**** > > client_drop 10950 0.07 Connection dropped, no sess/wrk > **** > > client_req 38298994 238.61 Client requests received**** > > cache_hit 32513762 202.57 Cache hits**** > > cache_hitpass 0 0.00 Cache hits for pass**** > > cache_miss 5784476 36.04 Cache misses**** > > backend_conn 5725540 35.67 Backend conn. success**** > > backend_unhealthy 0 0.00 Backend conn. not attempted*** > * > > backend_busy 0 0.00 Backend conn. too many**** > > backend_fail 1383 0.01 Backend conn. failures**** > > backend_reuse 60837 0.38 Backend conn. reuses**** > > backend_toolate 33 0.00 Backend conn. was closed**** > > backend_recycle 60870 0.38 Backend conn. recycles**** > > backend_unused 0 0.00 Backend conn. unused**** > > fetch_head 6 0.00 Fetch head**** > > fetch_length 93631 0.58 Fetch with Length**** > > fetch_chunked 5689433 35.45 Fetch chunked**** > > fetch_eof 0 0.00 Fetch EOF**** > > fetch_bad 0 0.00 Fetch had bad headers**** > > fetch_close 107 0.00 Fetch wanted close**** > > fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed**** > > fetch_zero 0 0.00 Fetch zero len**** > > fetch_failed 1 0.00 Fetch failed**** > > n_sess_mem 7138 . N struct sess_mem**** > > n_sess 6970 . N struct sess**** > > n_object 5047123 . N struct object**** > > n_vampireobject 0 . N unresurrected objects**** > > n_objectcore 5048435 . N struct objectcore**** > > n_objecthead 4955641 . N struct objecthead**** > > n_smf 10139770 . N struct smf**** > > n_smf_frag 295671 . N small free smf**** > > n_smf_large 0 . N large free smf**** > > n_vbe_conn 2997 . N struct vbe_conn**** > > n_wrk 3000 . N worker threads**** > > n_wrk_create 5739 0.04 N worker threads created**** > > n_wrk_failed 0 0.00 N worker threads not created*** > * > > n_wrk_max 4063 0.03 N worker threads limited**** > > n_wrk_queue 2861 0.02 N queued work requests**** > > n_wrk_overflow 83534 0.52 N overflowed work requests**** > > n_wrk_drop 10980 0.07 N dropped work requests**** > > n_backend 2 . N backends**** > > n_expired 2179 . N expired objects**** > > n_lru_nuked 862615 . N LRU nuked objects**** > > n_lru_saved 0 . N LRU saved objects**** > > n_lru_moved 27156180 . N LRU moved objects**** > > n_deathrow 0 . N objects on deathrow**** > > losthdr 0 0.00 HTTP header overflows**** > > n_objsendfile 0 0.00 Objects sent with sendfile**** > > n_objwrite 37294888 232.35 Objects sent with write**** > > n_objoverflow 0 0.00 Objects overflowing workspace** > ** > > s_sess 38566049 240.27 Total Sessions**** > > s_req 38298994 238.61 Total Requests**** > > s_pipe 0 0.00 Total pipe**** > > s_pass 266 0.00 Total pass**** > > s_fetch 5783176 36.03 Total fetch**** > > s_hdrbytes 12570989864 78319.53 Total header bytes**** > > s_bodybytes 151327304604 942796.38 Total body bytes**** > > sess_closed 34673984 216.03 Session Closed**** > > sess_pipeline 187 0.00 Session Pipeline**** > > sess_readahead 321 0.00 Session Read Ahead**** > > sess_linger 3929378 24.48 Session Linger**** > > sess_herd 3929559 24.48 Session herd**** > > shm_records 2025645664 12620.14 SHM records**** > > shm_writes 169640580 1056.89 SHM writes**** > > shm_flushes 41 0.00 SHM flushes due to overflow**** > > shm_cont 580515 3.62 SHM MTX contention**** > > shm_cycles 933 0.01 SHM cycles through buffer**** > > sm_nreq 12431620 77.45 allocator requests**** > > sm_nobj 9844099 . outstanding allocations**** > > sm_balloc 43855261696 . bytes allocated**** > > sm_bfree 7684345856 . bytes free**** > > sma_nreq 0 0.00 SMA allocator requests**** > > sma_nobj 0 . SMA outstanding allocations**** > > sma_nbytes 0 . SMA outstanding bytes**** > > sma_balloc 0 . SMA bytes allocated**** > > sma_bfree 0 . SMA bytes free**** > > sms_nreq 1566 0.01 SMS allocator requests**** > > sms_nobj 0 . SMS outstanding allocations**** > > sms_nbytes 0 . SMS outstanding bytes**** > > sms_balloc 656154 . SMS bytes allocated**** > > sms_bfree 656154 . SMS bytes freed**** > > backend_req 5786381 36.05 Backend requests made**** > > n_vcl 1 0.00 N vcl total**** > > n_vcl_avail 1 0.00 N vcl available**** > > n_vcl_discard 0 0.00 N vcl discarded**** > > n_purge 218 . N total active purges**** > > n_purge_add 218 0.00 N new purges added**** > > n_purge_retire 0 0.00 N old purges deleted**** > > n_purge_obj_test 588742 3.67 N objects tested**** > > n_purge_re_test 120444323 750.39 N regexps tested against**** > > n_purge_dups 0 0.00 N duplicate purges removed**** > > hcb_nolock 38301670 238.63 HCB Lookups without lock**** > > hcb_lock 5786309 36.05 HCB Lookups with lock**** > > hcb_insert 5786305 36.05 HCB Inserts**** > > esi_parse 0 0.00 Objects ESI parsed (unlock)**** > > esi_errors 0 0.00 ESI parse errors (unlock)**** > > accept_fail 0 0.00 Accept failures**** > > client_drop_late 30 0.00 Connection dropped late**** > > uptime 160509 1.00 Client uptime**** > > backend_retry 25 0.00 Backend conn. retry**** > > dir_dns_lookups 0 0.00 DNS director lookups**** > > dir_dns_failed 0 0.00 DNS director failed lookups**** > > dir_dns_hit 0 0.00 DNS director cached lookups hit > **** > > dir_dns_cache_full 0 0.00 DNS director full dnscache*** > * > > fetch_1xx 0 0.00 Fetch no body (1xx)**** > > fetch_204 0 0.00 Fetch no body (204)**** > > fetch_304 0 0.00 Fetch no body (304)**** > > ** ** > > Even though I have removed the server from our load balancer there are > still a lot of requests going to the backend. Maybe these are all queued up > requests that varnish is trying to fulfill? Here is some output from > varnishlog ?c when I try to connect with curl:**** > > ** ** > > root at mvp14:~# varnishlog -c**** > > 26 SessionOpen c 192.168.8.41 41942 0.0.0.0:80**** > > 26 ReqStart c 192.168.8.41 41942 2108342803**** > > 26 RxRequest c GET**** > > 26 RxURL c /**** > > 26 RxProtocol c HTTP/1.1**** > > 26 RxHeader c User-Agent: curl/7.21.4 (x86_64-unknown-linux-gnu) > libcurl/7.21.4 OpenSSL/0.9.8n zlib/1.2.5 libidn/1.19**** > > 26 RxHeader c Host: mvp14.airg.com**** > > 26 RxHeader c Accept: */***** > > 26 VCL_call c recv**** > > 26 VCL_return c lookup**** > > 26 VCL_call c hash**** > > 26 VCL_return c hash**** > > ** ** > > ** ** > > The connection just hangs here until it times out.**** > > ** ** > > Any help would be appreciated. We are trying to replace our squid caching > layer with varnish. However if I can?t resolve this issue we will have to go > back to squid.**** > > ** ** > > Thanks,**** > > ** ** > > ** ** > > ** ** > > *Matt Schurenko* > Systems Administrator **** > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From MSchurenko at airg.com Fri Sep 30 00:42:20 2011 From: MSchurenko at airg.com (Matt Schurenko) Date: Thu, 29 Sep 2011 17:42:20 -0700 Subject: lru nuke causes varnish to stop respnding to client requests In-Reply-To: References: <8B5C6F575422414AA91B46C454126B6C02EB2824E7@EXCHMVS.exchange.airg> Message-ID: <8B5C6F575422414AA91B46C454126B6C02EB2824F2@EXCHMVS.exchange.airg> Sorry. I forgot to mention that I already adjusted thread behaviour via varniahadm while varnish was running. I had it set to use min 50, max 3000 and a thread timeout of 120s. I think the reason why n_wrk_overflow and n_wrk_drop are so high is due to this problem. Before the lru nuke happens the number of worker threads are ~ 100. As soon as it starts to nuke the number of threads jumps to the max. I am monitoring some stats with MRTG. I seem to remember that on the other varnish server it would being to lru nuke long before the cache got full. For this one there is no lru nuke activity until it reaches a certain point and then boom. 3000 threads are used up and no more new clients can connect. Matt Schurenko Systems Administrator airG(r) Share Your World Suite 710, 1133 Melville Street Vancouver, BC V6E 4E5 P: +1.604.408.2228 F: +1.866.874.8136 E: MSchurenko at airg.com W: www.airg.com From: Damon Snyder [mailto:damon at huddler-inc.com] Sent: September-29-11 5:30 PM To: Matt Schurenko Cc: varnish-misc at varnish-cache.org Subject: Re: lru nuke causes varnish to stop respnding to client requests Hi Matt, It looks like you really need to bump up the number of worker threads. From your stats: n_wrk_queue 2861 0.02 N queued work requests n_wrk_overflow 83534 0.52 N overflowed work requests n_wrk_drop 10980 0.07 N dropped work requests You have a lot of requests that are on the queue waiting for a worker and you have a lot of requests where varnish has given up trying to fullfil with a worker. You can bump the number of works up using the -w command line option to varnishd. I would suggest something like -w 400,1000,120 to start with (the default is -w2,500,300). This says use 400 at a minimum, 1000 as the maximum, and set the timeout to 120s. According to the stats explanation doc your n_wrk_queue and n_wrk_drop should be 0. If you see these numbers going up again, use -w 500,2000,120 or something like that. Hope this helps, Damon On Thu, Sep 29, 2011 at 4:34 PM, Matt Schurenko > wrote: I've been having this problem for a couple weeks now on one of our varnish servers. I have posted a couple times already. What happens is that the server in questions runs fine until the cache gets full. When it starts to lru nuke the number of worker threads jumps up to thread_pool_max and varnish stops responding to any client requests. I have tried this with Centos5.4, 5.7 and now Slackware (all 64-bit ) and the behaviour is the same. I am using varnish version 2.1.5 on a Dell C6105 with 48GB of RAM. Here is my varnishd command line: /usr/local/sbin/varnishd -s file,/tmp/varnish-cache,48g -T 127.0.0.1:2000 -a 0.0.0.0:80 -t 604800 -f /usr/local/etc/varnish/default.vcl -p http_headers 384 -p connect_timeout 4.0 Here is the output from varnishstat -1: client_conn 38582763 240.38 Client connections accepted client_drop 10950 0.07 Connection dropped, no sess/wrk client_req 38298994 238.61 Client requests received cache_hit 32513762 202.57 Cache hits cache_hitpass 0 0.00 Cache hits for pass cache_miss 5784476 36.04 Cache misses backend_conn 5725540 35.67 Backend conn. success backend_unhealthy 0 0.00 Backend conn. not attempted backend_busy 0 0.00 Backend conn. too many backend_fail 1383 0.01 Backend conn. failures backend_reuse 60837 0.38 Backend conn. reuses backend_toolate 33 0.00 Backend conn. was closed backend_recycle 60870 0.38 Backend conn. recycles backend_unused 0 0.00 Backend conn. unused fetch_head 6 0.00 Fetch head fetch_length 93631 0.58 Fetch with Length fetch_chunked 5689433 35.45 Fetch chunked fetch_eof 0 0.00 Fetch EOF fetch_bad 0 0.00 Fetch had bad headers fetch_close 107 0.00 Fetch wanted close fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed fetch_zero 0 0.00 Fetch zero len fetch_failed 1 0.00 Fetch failed n_sess_mem 7138 . N struct sess_mem n_sess 6970 . N struct sess n_object 5047123 . N struct object n_vampireobject 0 . N unresurrected objects n_objectcore 5048435 . N struct objectcore n_objecthead 4955641 . N struct objecthead n_smf 10139770 . N struct smf n_smf_frag 295671 . N small free smf n_smf_large 0 . N large free smf n_vbe_conn 2997 . N struct vbe_conn n_wrk 3000 . N worker threads n_wrk_create 5739 0.04 N worker threads created n_wrk_failed 0 0.00 N worker threads not created n_wrk_max 4063 0.03 N worker threads limited n_wrk_queue 2861 0.02 N queued work requests n_wrk_overflow 83534 0.52 N overflowed work requests n_wrk_drop 10980 0.07 N dropped work requests n_backend 2 . N backends n_expired 2179 . N expired objects n_lru_nuked 862615 . N LRU nuked objects n_lru_saved 0 . N LRU saved objects n_lru_moved 27156180 . N LRU moved objects n_deathrow 0 . N objects on deathrow losthdr 0 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 37294888 232.35 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 38566049 240.27 Total Sessions s_req 38298994 238.61 Total Requests s_pipe 0 0.00 Total pipe s_pass 266 0.00 Total pass s_fetch 5783176 36.03 Total fetch s_hdrbytes 12570989864 78319.53 Total header bytes s_bodybytes 151327304604 942796.38 Total body bytes sess_closed 34673984 216.03 Session Closed sess_pipeline 187 0.00 Session Pipeline sess_readahead 321 0.00 Session Read Ahead sess_linger 3929378 24.48 Session Linger sess_herd 3929559 24.48 Session herd shm_records 2025645664 12620.14 SHM records shm_writes 169640580 1056.89 SHM writes shm_flushes 41 0.00 SHM flushes due to overflow shm_cont 580515 3.62 SHM MTX contention shm_cycles 933 0.01 SHM cycles through buffer sm_nreq 12431620 77.45 allocator requests sm_nobj 9844099 . outstanding allocations sm_balloc 43855261696 . bytes allocated sm_bfree 7684345856 . bytes free sma_nreq 0 0.00 SMA allocator requests sma_nobj 0 . SMA outstanding allocations sma_nbytes 0 . SMA outstanding bytes sma_balloc 0 . SMA bytes allocated sma_bfree 0 . SMA bytes free sms_nreq 1566 0.01 SMS allocator requests sms_nobj 0 . SMS outstanding allocations sms_nbytes 0 . SMS outstanding bytes sms_balloc 656154 . SMS bytes allocated sms_bfree 656154 . SMS bytes freed backend_req 5786381 36.05 Backend requests made n_vcl 1 0.00 N vcl total n_vcl_avail 1 0.00 N vcl available n_vcl_discard 0 0.00 N vcl discarded n_purge 218 . N total active purges n_purge_add 218 0.00 N new purges added n_purge_retire 0 0.00 N old purges deleted n_purge_obj_test 588742 3.67 N objects tested n_purge_re_test 120444323 750.39 N regexps tested against n_purge_dups 0 0.00 N duplicate purges removed hcb_nolock 38301670 238.63 HCB Lookups without lock hcb_lock 5786309 36.05 HCB Lookups with lock hcb_insert 5786305 36.05 HCB Inserts esi_parse 0 0.00 Objects ESI parsed (unlock) esi_errors 0 0.00 ESI parse errors (unlock) accept_fail 0 0.00 Accept failures client_drop_late 30 0.00 Connection dropped late uptime 160509 1.00 Client uptime backend_retry 25 0.00 Backend conn. retry dir_dns_lookups 0 0.00 DNS director lookups dir_dns_failed 0 0.00 DNS director failed lookups dir_dns_hit 0 0.00 DNS director cached lookups hit dir_dns_cache_full 0 0.00 DNS director full dnscache fetch_1xx 0 0.00 Fetch no body (1xx) fetch_204 0 0.00 Fetch no body (204) fetch_304 0 0.00 Fetch no body (304) Even though I have removed the server from our load balancer there are still a lot of requests going to the backend. Maybe these are all queued up requests that varnish is trying to fulfill? Here is some output from varnishlog -c when I try to connect with curl: root at mvp14:~# varnishlog -c 26 SessionOpen c 192.168.8.41 41942 0.0.0.0:80 26 ReqStart c 192.168.8.41 41942 2108342803 26 RxRequest c GET 26 RxURL c / 26 RxProtocol c HTTP/1.1 26 RxHeader c User-Agent: curl/7.21.4 (x86_64-unknown-linux-gnu) libcurl/7.21.4 OpenSSL/0.9.8n zlib/1.2.5 libidn/1.19 26 RxHeader c Host: mvp14.airg.com 26 RxHeader c Accept: */* 26 VCL_call c recv 26 VCL_return c lookup 26 VCL_call c hash 26 VCL_return c hash The connection just hangs here until it times out. Any help would be appreciated. We are trying to replace our squid caching layer with varnish. However if I can't resolve this issue we will have to go back to squid. Thanks, Matt Schurenko Systems Administrator _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From damon at huddler-inc.com Fri Sep 30 16:55:45 2011 From: damon at huddler-inc.com (Damon Snyder) Date: Fri, 30 Sep 2011 09:55:45 -0700 Subject: lru nuke causes varnish to stop respnding to client requests In-Reply-To: <8B5C6F575422414AA91B46C454126B6C02EB2824F2@EXCHMVS.exchange.airg> References: <8B5C6F575422414AA91B46C454126B6C02EB2824E7@EXCHMVS.exchange.airg> <8B5C6F575422414AA91B46C454126B6C02EB2824F2@EXCHMVS.exchange.airg> Message-ID: Hi Matt, Oh, sorry, I didn't notice the worker thread count at 3000. I would suggest switching to -smalloc,38G (total mem * 0.8). If you have a lot of small objects or objects of the same size, you could be encountering some excessive nuking. Also, what are your backends doing when this happens? Is the nuking a coincidence or is there an issue further down the stack? Damon On Thu, Sep 29, 2011 at 5:42 PM, Matt Schurenko wrote: > Sorry. I forgot to mention that I already adjusted thread behaviour via > varniahadm while varnish was running. I had it set to use min 50, max 3000 > and a thread timeout of 120s. I think the reason why n_wrk_overflow and > n_wrk_drop are so high is due to this problem. Before the lru nuke happens > the number of worker threads are ~ 100. As soon as it starts to nuke the > number of threads jumps to the max. I am monitoring some stats with MRTG. I > seem to remember that on the other varnish server it would being to lru nuke > long before the cache got full. For this one there is no lru nuke activity > until it reaches a certain point and then boom. 3000 threads are used up and > no more new clients can connect. **** > > ** ** > > > Matt Schurenko > Systems Administrator > **** > > *air**G**?** **Share Your World* > Suite 710, 1133 Melville Street**** > > Vancouver, BC V6E 4E5**** > > P: +1.604.408.2228 **** > > F: +1.866.874.8136**** > > E: MSchurenko at airg.com**** > > W: www.airg.com**** > > *From:* Damon Snyder [mailto:damon at huddler-inc.com] > *Sent:* September-29-11 5:30 PM > *To:* Matt Schurenko > *Cc:* varnish-misc at varnish-cache.org > *Subject:* Re: lru nuke causes varnish to stop respnding to client > requests**** > > ** ** > > Hi Matt,**** > > It looks like you really need to bump up the number of worker threads. From > your stats:**** > > ** ** > > n_wrk_queue 2861 0.02 N queued work requests**** > > n_wrk_overflow 83534 0.52 N overflowed work requests**** > > n_wrk_drop 10980 0.07 N dropped work requests**** > > ** ** > > You have a lot of requests that are on the queue waiting for a worker and > you have a lot of requests where varnish has given up trying to fullfil with > a worker. You can bump the number of works up using the -w command line > option to varnishd. I would suggest something like -w 400,1000,120 to start > with (the default is -w2,500,300). This says use 400 at a minimum, 1000 as > the maximum, and set the timeout to 120s. According to the stats > explanation doc your > n_wrk_queue and n_wrk_drop should be 0. If you see these numbers going up > again, use -w 500,2000,120 or something like that.**** > > ** ** > > Hope this helps,**** > > Damon**** > > ** ** > > On Thu, Sep 29, 2011 at 4:34 PM, Matt Schurenko > wrote:**** > > I?ve been having this problem for a couple weeks now on one of our varnish > servers. I have posted a couple times already. What happens is that the > server in questions runs fine until the cache gets full. When it starts to > lru nuke the number of worker threads jumps up to thread_pool_max and > varnish stops responding to any client requests. I have tried this with > Centos5.4, 5.7 and now Slackware (all 64-bit ) and the behaviour is the > same. **** > > **** > > I am using varnish version 2.1.5 on a Dell C6105 with 48GB of RAM.**** > > **** > > Here is my varnishd command line:**** > > **** > > /usr/local/sbin/varnishd -s file,/tmp/varnish-cache,48g -T 127.0.0.1:2000-a > 0.0.0.0:80 -t 604800 -f /usr/local/etc/varnish/default.vcl -p http_headers > 384 -p connect_timeout 4.0**** > > **** > > Here is the output from varnishstat -1:**** > > **** > > client_conn 38582763 240.38 Client connections accepted**** > > client_drop 10950 0.07 Connection dropped, no sess/wrk > **** > > client_req 38298994 238.61 Client requests received**** > > cache_hit 32513762 202.57 Cache hits**** > > cache_hitpass 0 0.00 Cache hits for pass**** > > cache_miss 5784476 36.04 Cache misses**** > > backend_conn 5725540 35.67 Backend conn. success**** > > backend_unhealthy 0 0.00 Backend conn. not attempted*** > * > > backend_busy 0 0.00 Backend conn. too many**** > > backend_fail 1383 0.01 Backend conn. failures**** > > backend_reuse 60837 0.38 Backend conn. reuses**** > > backend_toolate 33 0.00 Backend conn. was closed**** > > backend_recycle 60870 0.38 Backend conn. recycles**** > > backend_unused 0 0.00 Backend conn. unused**** > > fetch_head 6 0.00 Fetch head**** > > fetch_length 93631 0.58 Fetch with Length**** > > fetch_chunked 5689433 35.45 Fetch chunked**** > > fetch_eof 0 0.00 Fetch EOF**** > > fetch_bad 0 0.00 Fetch had bad headers**** > > fetch_close 107 0.00 Fetch wanted close**** > > fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed**** > > fetch_zero 0 0.00 Fetch zero len**** > > fetch_failed 1 0.00 Fetch failed**** > > n_sess_mem 7138 . N struct sess_mem**** > > n_sess 6970 . N struct sess**** > > n_object 5047123 . N struct object**** > > n_vampireobject 0 . N unresurrected objects**** > > n_objectcore 5048435 . N struct objectcore**** > > n_objecthead 4955641 . N struct objecthead**** > > n_smf 10139770 . N struct smf**** > > n_smf_frag 295671 . N small free smf**** > > n_smf_large 0 . N large free smf**** > > n_vbe_conn 2997 . N struct vbe_conn**** > > n_wrk 3000 . N worker threads**** > > n_wrk_create 5739 0.04 N worker threads created**** > > n_wrk_failed 0 0.00 N worker threads not created*** > * > > n_wrk_max 4063 0.03 N worker threads limited**** > > n_wrk_queue 2861 0.02 N queued work requests**** > > n_wrk_overflow 83534 0.52 N overflowed work requests**** > > n_wrk_drop 10980 0.07 N dropped work requests**** > > n_backend 2 . N backends**** > > n_expired 2179 . N expired objects**** > > n_lru_nuked 862615 . N LRU nuked objects**** > > n_lru_saved 0 . N LRU saved objects**** > > n_lru_moved 27156180 . N LRU moved objects**** > > n_deathrow 0 . N objects on deathrow**** > > losthdr 0 0.00 HTTP header overflows**** > > n_objsendfile 0 0.00 Objects sent with sendfile**** > > n_objwrite 37294888 232.35 Objects sent with write**** > > n_objoverflow 0 0.00 Objects overflowing workspace** > ** > > s_sess 38566049 240.27 Total Sessions**** > > s_req 38298994 238.61 Total Requests**** > > s_pipe 0 0.00 Total pipe**** > > s_pass 266 0.00 Total pass**** > > s_fetch 5783176 36.03 Total fetch**** > > s_hdrbytes 12570989864 78319.53 Total header bytes**** > > s_bodybytes 151327304604 942796.38 Total body bytes**** > > sess_closed 34673984 216.03 Session Closed**** > > sess_pipeline 187 0.00 Session Pipeline**** > > sess_readahead 321 0.00 Session Read Ahead**** > > sess_linger 3929378 24.48 Session Linger**** > > sess_herd 3929559 24.48 Session herd**** > > shm_records 2025645664 12620.14 SHM records**** > > shm_writes 169640580 1056.89 SHM writes**** > > shm_flushes 41 0.00 SHM flushes due to overflow**** > > shm_cont 580515 3.62 SHM MTX contention**** > > shm_cycles 933 0.01 SHM cycles through buffer**** > > sm_nreq 12431620 77.45 allocator requests**** > > sm_nobj 9844099 . outstanding allocations**** > > sm_balloc 43855261696 . bytes allocated**** > > sm_bfree 7684345856 . bytes free**** > > sma_nreq 0 0.00 SMA allocator requests**** > > sma_nobj 0 . SMA outstanding allocations**** > > sma_nbytes 0 . SMA outstanding bytes**** > > sma_balloc 0 . SMA bytes allocated**** > > sma_bfree 0 . SMA bytes free**** > > sms_nreq 1566 0.01 SMS allocator requests**** > > sms_nobj 0 . SMS outstanding allocations**** > > sms_nbytes 0 . SMS outstanding bytes**** > > sms_balloc 656154 . SMS bytes allocated**** > > sms_bfree 656154 . SMS bytes freed**** > > backend_req 5786381 36.05 Backend requests made**** > > n_vcl 1 0.00 N vcl total**** > > n_vcl_avail 1 0.00 N vcl available**** > > n_vcl_discard 0 0.00 N vcl discarded**** > > n_purge 218 . N total active purges**** > > n_purge_add 218 0.00 N new purges added**** > > n_purge_retire 0 0.00 N old purges deleted**** > > n_purge_obj_test 588742 3.67 N objects tested**** > > n_purge_re_test 120444323 750.39 N regexps tested against**** > > n_purge_dups 0 0.00 N duplicate purges removed**** > > hcb_nolock 38301670 238.63 HCB Lookups without lock**** > > hcb_lock 5786309 36.05 HCB Lookups with lock**** > > hcb_insert 5786305 36.05 HCB Inserts**** > > esi_parse 0 0.00 Objects ESI parsed (unlock)**** > > esi_errors 0 0.00 ESI parse errors (unlock)**** > > accept_fail 0 0.00 Accept failures**** > > client_drop_late 30 0.00 Connection dropped late**** > > uptime 160509 1.00 Client uptime**** > > backend_retry 25 0.00 Backend conn. retry**** > > dir_dns_lookups 0 0.00 DNS director lookups**** > > dir_dns_failed 0 0.00 DNS director failed lookups**** > > dir_dns_hit 0 0.00 DNS director cached lookups hit > **** > > dir_dns_cache_full 0 0.00 DNS director full dnscache*** > * > > fetch_1xx 0 0.00 Fetch no body (1xx)**** > > fetch_204 0 0.00 Fetch no body (204)**** > > fetch_304 0 0.00 Fetch no body (304)**** > > **** > > Even though I have removed the server from our load balancer there are > still a lot of requests going to the backend. Maybe these are all queued up > requests that varnish is trying to fulfill? Here is some output from > varnishlog ?c when I try to connect with curl:**** > > **** > > root at mvp14:~# varnishlog -c**** > > 26 SessionOpen c 192.168.8.41 41942 0.0.0.0:80**** > > 26 ReqStart c 192.168.8.41 41942 2108342803**** > > 26 RxRequest c GET**** > > 26 RxURL c /**** > > 26 RxProtocol c HTTP/1.1**** > > 26 RxHeader c User-Agent: curl/7.21.4 (x86_64-unknown-linux-gnu) > libcurl/7.21.4 OpenSSL/0.9.8n zlib/1.2.5 libidn/1.19**** > > 26 RxHeader c Host: mvp14.airg.com**** > > 26 RxHeader c Accept: */***** > > 26 VCL_call c recv**** > > 26 VCL_return c lookup**** > > 26 VCL_call c hash**** > > 26 VCL_return c hash**** > > **** > > **** > > The connection just hangs here until it times out.**** > > **** > > Any help would be appreciated. We are trying to replace our squid caching > layer with varnish. However if I can?t resolve this issue we will have to go > back to squid.**** > > **** > > Thanks,**** > > **** > > **** > > **** > > *Matt Schurenko* > Systems Administrator **** > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc**** > > ** ** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From MSchurenko at airg.com Fri Sep 30 18:06:40 2011 From: MSchurenko at airg.com (Matt Schurenko) Date: Fri, 30 Sep 2011 11:06:40 -0700 Subject: lru nuke causes varnish to stop respnding to client requests In-Reply-To: References: <8B5C6F575422414AA91B46C454126B6C02EB2824E7@EXCHMVS.exchange.airg> <8B5C6F575422414AA91B46C454126B6C02EB2824F2@EXCHMVS.exchange.airg> Message-ID: <8B5C6F575422414AA91B46C454126B6C02EB282506@EXCHMVS.exchange.airg> Hey Damon, Our backends are ok. I have two other varnish servers in front of the same backends, varnish1 and varnish2. varmish1 is a VM with 32 GB of ram that is using a file storage backend of 40GB. varnish2 is a physical machine with 48GB of RAM and a file storage backend of 48GB (this is identical to varnish3, the problem server). varnish1 and varnish2 seem to start to lru nuke much earlier than varnish3. To me it seems like the problem is that varnish3 lets the cache get as full as possible then starts to kind of panic and lru nuke as much as it can which causes the worker threads to skyrocket. That's just my interpretation of it anyways. I could be completely wrong. I also noticed that the n_object counter gets way higher on varnish3. Both varnish1 and varnish2 keep ~ 1.5 million objects in the cache. For varnish3 the number of objects kept climbing past 6 million, at which point it started to lru_nuke and stop responding to clients. I observed this happen three consecutive times on varnsih3. The number of objects keeps climbing. There is no lru activity whatsoever. Then it hits ~ 6 million objects then it starts to nuke. This usually takes about 48 hours to occur. I originally tried using the malloc storage type but kept having problems determining how big I should make the cache. On a server with 32GB I tried setting malloc to 30GB. I noticed that the server used a lot of swap space so I decreased malloc to 28GB. I still saw a lot of swap usage. In fact it used up to 13GB of swap. Since I read that mmap is better performance than swapping I decided to use a storage type of file. Matt Schurenko Systems Administrator airG(r) Share Your World Suite 710, 1133 Melville Street Vancouver, BC V6E 4E5 P: +1.604.408.2228 F: +1.866.874.8136 E: MSchurenko at airg.com W: www.airg.com From: Damon Snyder [mailto:damon at huddler-inc.com] Sent: September-30-11 9:56 AM To: Matt Schurenko Cc: varnish-misc at varnish-cache.org Subject: Re: lru nuke causes varnish to stop respnding to client requests Hi Matt, Oh, sorry, I didn't notice the worker thread count at 3000. I would suggest switching to -smalloc,38G (total mem * 0.8). If you have a lot of small objects or objects of the same size, you could be encountering some excessive nuking. Also, what are your backends doing when this happens? Is the nuking a coincidence or is there an issue further down the stack? Damon On Thu, Sep 29, 2011 at 5:42 PM, Matt Schurenko > wrote: Sorry. I forgot to mention that I already adjusted thread behaviour via varniahadm while varnish was running. I had it set to use min 50, max 3000 and a thread timeout of 120s. I think the reason why n_wrk_overflow and n_wrk_drop are so high is due to this problem. Before the lru nuke happens the number of worker threads are ~ 100. As soon as it starts to nuke the number of threads jumps to the max. I am monitoring some stats with MRTG. I seem to remember that on the other varnish server it would being to lru nuke long before the cache got full. For this one there is no lru nuke activity until it reaches a certain point and then boom. 3000 threads are used up and no more new clients can connect. Matt Schurenko Systems Administrator airG(r) Share Your World Suite 710, 1133 Melville Street Vancouver, BC V6E 4E5 P: +1.604.408.2228 F: +1.866.874.8136 E: MSchurenko at airg.com W: www.airg.com From: Damon Snyder [mailto:damon at huddler-inc.com] Sent: September-29-11 5:30 PM To: Matt Schurenko Cc: varnish-misc at varnish-cache.org Subject: Re: lru nuke causes varnish to stop respnding to client requests Hi Matt, It looks like you really need to bump up the number of worker threads. From your stats: n_wrk_queue 2861 0.02 N queued work requests n_wrk_overflow 83534 0.52 N overflowed work requests n_wrk_drop 10980 0.07 N dropped work requests You have a lot of requests that are on the queue waiting for a worker and you have a lot of requests where varnish has given up trying to fullfil with a worker. You can bump the number of works up using the -w command line option to varnishd. I would suggest something like -w 400,1000,120 to start with (the default is -w2,500,300). This says use 400 at a minimum, 1000 as the maximum, and set the timeout to 120s. According to the stats explanation doc your n_wrk_queue and n_wrk_drop should be 0. If you see these numbers going up again, use -w 500,2000,120 or something like that. Hope this helps, Damon On Thu, Sep 29, 2011 at 4:34 PM, Matt Schurenko > wrote: I've been having this problem for a couple weeks now on one of our varnish servers. I have posted a couple times already. What happens is that the server in questions runs fine until the cache gets full. When it starts to lru nuke the number of worker threads jumps up to thread_pool_max and varnish stops responding to any client requests. I have tried this with Centos5.4, 5.7 and now Slackware (all 64-bit ) and the behaviour is the same. I am using varnish version 2.1.5 on a Dell C6105 with 48GB of RAM. Here is my varnishd command line: /usr/local/sbin/varnishd -s file,/tmp/varnish-cache,48g -T 127.0.0.1:2000 -a 0.0.0.0:80 -t 604800 -f /usr/local/etc/varnish/default.vcl -p http_headers 384 -p connect_timeout 4.0 Here is the output from varnishstat -1: client_conn 38582763 240.38 Client connections accepted client_drop 10950 0.07 Connection dropped, no sess/wrk client_req 38298994 238.61 Client requests received cache_hit 32513762 202.57 Cache hits cache_hitpass 0 0.00 Cache hits for pass cache_miss 5784476 36.04 Cache misses backend_conn 5725540 35.67 Backend conn. success backend_unhealthy 0 0.00 Backend conn. not attempted backend_busy 0 0.00 Backend conn. too many backend_fail 1383 0.01 Backend conn. failures backend_reuse 60837 0.38 Backend conn. reuses backend_toolate 33 0.00 Backend conn. was closed backend_recycle 60870 0.38 Backend conn. recycles backend_unused 0 0.00 Backend conn. unused fetch_head 6 0.00 Fetch head fetch_length 93631 0.58 Fetch with Length fetch_chunked 5689433 35.45 Fetch chunked fetch_eof 0 0.00 Fetch EOF fetch_bad 0 0.00 Fetch had bad headers fetch_close 107 0.00 Fetch wanted close fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed fetch_zero 0 0.00 Fetch zero len fetch_failed 1 0.00 Fetch failed n_sess_mem 7138 . N struct sess_mem n_sess 6970 . N struct sess n_object 5047123 . N struct object n_vampireobject 0 . N unresurrected objects n_objectcore 5048435 . N struct objectcore n_objecthead 4955641 . N struct objecthead n_smf 10139770 . N struct smf n_smf_frag 295671 . N small free smf n_smf_large 0 . N large free smf n_vbe_conn 2997 . N struct vbe_conn n_wrk 3000 . N worker threads n_wrk_create 5739 0.04 N worker threads created n_wrk_failed 0 0.00 N worker threads not created n_wrk_max 4063 0.03 N worker threads limited n_wrk_queue 2861 0.02 N queued work requests n_wrk_overflow 83534 0.52 N overflowed work requests n_wrk_drop 10980 0.07 N dropped work requests n_backend 2 . N backends n_expired 2179 . N expired objects n_lru_nuked 862615 . N LRU nuked objects n_lru_saved 0 . N LRU saved objects n_lru_moved 27156180 . N LRU moved objects n_deathrow 0 . N objects on deathrow losthdr 0 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 37294888 232.35 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 38566049 240.27 Total Sessions s_req 38298994 238.61 Total Requests s_pipe 0 0.00 Total pipe s_pass 266 0.00 Total pass s_fetch 5783176 36.03 Total fetch s_hdrbytes 12570989864 78319.53 Total header bytes s_bodybytes 151327304604 942796.38 Total body bytes sess_closed 34673984 216.03 Session Closed sess_pipeline 187 0.00 Session Pipeline sess_readahead 321 0.00 Session Read Ahead sess_linger 3929378 24.48 Session Linger sess_herd 3929559 24.48 Session herd shm_records 2025645664 12620.14 SHM records shm_writes 169640580 1056.89 SHM writes shm_flushes 41 0.00 SHM flushes due to overflow shm_cont 580515 3.62 SHM MTX contention shm_cycles 933 0.01 SHM cycles through buffer sm_nreq 12431620 77.45 allocator requests sm_nobj 9844099 . outstanding allocations sm_balloc 43855261696 . bytes allocated sm_bfree 7684345856 . bytes free sma_nreq 0 0.00 SMA allocator requests sma_nobj 0 . SMA outstanding allocations sma_nbytes 0 . SMA outstanding bytes sma_balloc 0 . SMA bytes allocated sma_bfree 0 . SMA bytes free sms_nreq 1566 0.01 SMS allocator requests sms_nobj 0 . SMS outstanding allocations sms_nbytes 0 . SMS outstanding bytes sms_balloc 656154 . SMS bytes allocated sms_bfree 656154 . SMS bytes freed backend_req 5786381 36.05 Backend requests made n_vcl 1 0.00 N vcl total n_vcl_avail 1 0.00 N vcl available n_vcl_discard 0 0.00 N vcl discarded n_purge 218 . N total active purges n_purge_add 218 0.00 N new purges added n_purge_retire 0 0.00 N old purges deleted n_purge_obj_test 588742 3.67 N objects tested n_purge_re_test 120444323 750.39 N regexps tested against n_purge_dups 0 0.00 N duplicate purges removed hcb_nolock 38301670 238.63 HCB Lookups without lock hcb_lock 5786309 36.05 HCB Lookups with lock hcb_insert 5786305 36.05 HCB Inserts esi_parse 0 0.00 Objects ESI parsed (unlock) esi_errors 0 0.00 ESI parse errors (unlock) accept_fail 0 0.00 Accept failures client_drop_late 30 0.00 Connection dropped late uptime 160509 1.00 Client uptime backend_retry 25 0.00 Backend conn. retry dir_dns_lookups 0 0.00 DNS director lookups dir_dns_failed 0 0.00 DNS director failed lookups dir_dns_hit 0 0.00 DNS director cached lookups hit dir_dns_cache_full 0 0.00 DNS director full dnscache fetch_1xx 0 0.00 Fetch no body (1xx) fetch_204 0 0.00 Fetch no body (204) fetch_304 0 0.00 Fetch no body (304) Even though I have removed the server from our load balancer there are still a lot of requests going to the backend. Maybe these are all queued up requests that varnish is trying to fulfill? Here is some output from varnishlog -c when I try to connect with curl: root at mvp14:~# varnishlog -c 26 SessionOpen c 192.168.8.41 41942 0.0.0.0:80 26 ReqStart c 192.168.8.41 41942 2108342803 26 RxRequest c GET 26 RxURL c / 26 RxProtocol c HTTP/1.1 26 RxHeader c User-Agent: curl/7.21.4 (x86_64-unknown-linux-gnu) libcurl/7.21.4 OpenSSL/0.9.8n zlib/1.2.5 libidn/1.19 26 RxHeader c Host: mvp14.airg.com 26 RxHeader c Accept: */* 26 VCL_call c recv 26 VCL_return c lookup 26 VCL_call c hash 26 VCL_return c hash The connection just hangs here until it times out. Any help would be appreciated. We are trying to replace our squid caching layer with varnish. However if I can't resolve this issue we will have to go back to squid. Thanks, Matt Schurenko Systems Administrator _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From CoreyH at ganz.com Thu Sep 29 18:52:07 2011 From: CoreyH at ganz.com (Corey Hammerton) Date: Thu, 29 Sep 2011 14:52:07 -0400 Subject: Round-Robin Director Backends Message-ID: <9C8FD2499888F54396801D019DAB849B065993D93A@TORE2K7V01.ganz.ad> Hello, I have an environment with multiple backend servers that require most images (static content) be replicated out to all servers, this is causing unnecessary load on our backend servers. To mitigate this problem we plan to put in a Varnish system to grab the assets from the backend servers and send them to the client VIA a round-robin director. If Varnish cannot find the asset on the current backend server does it automatically go to the next backend in the list? If not what VCL code do we need to insert to accomplish this? Regards, Corey -------------- next part -------------- An HTML attachment was scrubbed... URL: