From phk at phk.freebsd.dk Fri May 1 08:16:27 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 01 May 2009 08:16:27 +0000 Subject: Theoretical connections/second limit using Varnish In-Reply-To: Your message of "Thu, 30 Apr 2009 16:10:15 +0100." <49F9BF57.4020909@loman.net> Message-ID: <48432.1241165787@critter.freebsd.dk> In message <49F9BF57.4020909 at loman.net>, Nick Loman writes: >Precisely, we only have perhaps 50 PHP children serving requests, so if >these are kept open to serve idle keep-alive connections, that severely >limits the numbers of dynamic page requests we can serve. The difference between letting them say open 10msec and 0msec is very significant when it comes to performance. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From nick at loman.net Fri May 1 08:27:59 2009 From: nick at loman.net (Nick Loman) Date: Fri, 01 May 2009 09:27:59 +0100 Subject: Theoretical connections/second limit using Varnish In-Reply-To: <48432.1241165787@critter.freebsd.dk> References: <48432.1241165787@critter.freebsd.dk> Message-ID: <49FAB28F.2040903@loman.net> Poul-Henning Kamp wrote: > In message <49F9BF57.4020909 at loman.net>, Nick Loman writes: > >> Precisely, we only have perhaps 50 PHP children serving requests, so if >> these are kept open to serve idle keep-alive connections, that severely >> limits the numbers of dynamic page requests we can serve. > > The difference between letting them say open 10msec and 0msec is very > significant when it comes to performance. Hi Poul-Henning, Which way round do you mean? Apache specifies Keep-Alive in seconds, and my sites will certainly die if I set it to even 1 second. I've no problem with Varnish performance right now, it works really well for me. And solves the problem nicely of how to give Keep-Alive connections to my users whilst maintaining only a small pool of PHP FastCGI processes. But I would like to eliminate my thousands of TIME_WAIT sockets if at all possible, as this represents a scaleability ceiling right now. Of course I could just throw more webservers at the problem - if necessary. Cheers! Nick. From phk at phk.freebsd.dk Fri May 1 08:35:30 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 01 May 2009 08:35:30 +0000 Subject: Theoretical connections/second limit using Varnish In-Reply-To: Your message of "Fri, 01 May 2009 09:27:59 +0100." <49FAB28F.2040903@loman.net> Message-ID: <72189.1241166930@critter.freebsd.dk> In message <49FAB28F.2040903 at loman.net>, Nick Loman writes: >Poul-Henning Kamp wrote: >Which way round do you mean? > >Apache specifies Keep-Alive in seconds, and my sites will certainly die >if I set it to even 1 second. Tell the apache developers to get their act together then. >But I would like to eliminate my thousands of TIME_WAIT sockets if at >all possible, as this represents a scaleability ceiling right now. Of >course I could just throw more webservers at the problem - if necessary. Couldn't you also just configure multiple IP#'s on your backend ? If you run a tcpdump of the backend traffic, does the connections close properly with FIN or does one of the ends RST them ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From nick at loman.net Fri May 1 10:25:52 2009 From: nick at loman.net (Nick Loman) Date: Fri, 01 May 2009 11:25:52 +0100 Subject: Theoretical connections/second limit using Varnish In-Reply-To: <72189.1241166930@critter.freebsd.dk> References: <72189.1241166930@critter.freebsd.dk> Message-ID: <49FACE30.7010309@loman.net> Poul-Henning Kamp wrote: > In message <49FAB28F.2040903 at loman.net>, Nick Loman writes: >> Poul-Henning Kamp wrote: > >> Which way round do you mean? >> >> Apache specifies Keep-Alive in seconds, and my sites will certainly die >> if I set it to even 1 second. > > Tell the apache developers to get their act together then. Hi Poul-Henning, I'm sure I could find a way of getting Apache to put in a 10msec Keep-Alive timeout if that was necessary. Are you proposing that Varnish would then hold and re-use that backend connection for a waiting request from a different user/TCP connection? >> But I would like to eliminate my thousands of TIME_WAIT sockets if at >> all possible, as this represents a scaleability ceiling right now. Of >> course I could just throw more webservers at the problem - if necessary. > > Couldn't you also just configure multiple IP#'s on your backend ? True, I could set up Apache to listen on 16 different ports each on localhost, then set up Varnish to load-balance to those 16 different backends, giving me much greater pool of unique connection entries to work with. The only thing I don't like about this approach is that the kernel is using a lot of memory tracking localhost connections in TIME_WAIT mode that do not need tracking. > If you run a tcpdump of the backend traffic, does the connections > close properly with FIN or does one of the ends RST them ? A single backend request from Varnish -> Apache yields the following tcpdump. It looks correct in that there is: Varnish -> Apache (FIN) Apache -> Varnish (FIN / ACK) Varnish -> Apache (ACK) 11:16:38.304362 IP (tos 0x0, ttl 64, id 18602, offset 0, flags [DF], proto TCP (6), length 60) 127.0.0.1.52192 > 127.0.0.1.8080: S 1321521977:1321521977(0) win 32792 0x0000: 0000 0000 0000 0000 0000 0000 0800 4500 ..............E. 0x0010: 003c 48aa 4000 4006 f40f 7f00 0001 7f00 . 127.0.0.1.52192: S 1327228413:1327228413(0) ack 1321521978 win 32768 0x0000: 0000 0000 0000 0000 0000 0000 0800 4500 ..............E. 0x0010: 003c 0000 4000 4006 3cba 7f00 0001 7f00 .<.. at .@.<....... 0x0020: 0001 1f90 cbe0 4f1b e5fd 4ec4 d33a a012 ......O...N..:.. 0x0030: 8000 00b5 0000 0204 400c 0402 080a 0a98 ........ at ....... 0x0040: 9b93 0a98 .... 11:16:38.304391 IP (tos 0x0, ttl 64, id 18603, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.52192 > 127.0.0.1.8080: ., cksum 0xe8d8 (correct), ack 1327228414 win 257 0x0000: 0000 0000 0000 0000 0000 0000 0800 4500 ..............E. 0x0010: 0034 48ab 4000 4006 f416 7f00 0001 7f00 .4H. at .@......... 0x0020: 0001 cbe0 1f90 4ec4 d33a 4f1b e5fe 8010 ......N..:O..... 0x0030: 0101 e8d8 0000 0101 080a 0a98 9b93 0a98 ................ 0x0040: 9b93 .. 11:16:38.304552 IP (tos 0x0, ttl 64, id 18604, offset 0, flags [DF], proto TCP (6), length 1190) 127.0.0.1.52192 > 127.0.0.1.8080: P 1321521978:1321523116(1138) ack 1327228414 win 257 0x0000: 0000 0000 0000 0000 0000 0000 0800 4500 ..............E. 0x0010: 04a6 48ac 4000 4006 efa3 7f00 0001 7f00 ..H. at .@......... 0x0020: 0001 cbe0 1f90 4ec4 d33a 4f1b e5fe 8018 ......N..:O..... 0x0030: 0101 029b 0000 0101 080a 0a98 9b93 0a98 ................ 0x0040: 9b93 4745 ..GE 11:16:38.304567 IP (tos 0x0, ttl 64, id 32365, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.8080 > 127.0.0.1.52192: ., cksum 0xe455 (correct), ack 1321523116 win 274 0x0000: 0000 0000 0000 0000 0000 0000 0800 4500 ..............E. 0x0010: 0034 7e6d 4000 4006 be54 7f00 0001 7f00 .4~m at .@..T...... 0x0020: 0001 1f90 cbe0 4f1b e5fe 4ec4 d7ac 8010 ......O...N..... 0x0030: 0112 e455 0000 0101 080a 0a98 9b93 0a98 ...U............ 0x0040: 9b93 .. 11:16:38.410471 IP (tos 0x0, ttl 64, id 32366, offset 0, flags [DF], proto TCP (6), length 13079) 127.0.0.1.8080 > 127.0.0.1.52192: P 1327228414:1327241441(13027) ack 1321523116 win 274 0x0000: 0000 0000 0000 0000 0000 0000 0800 4500 ..............E. 0x0010: 3317 7e6e 4000 4006 8b70 7f00 0001 7f00 3.~n at .@..p...... 0x0020: 0001 1f90 cbe0 4f1b e5fe 4ec4 d7ac 8018 ......O...N..... 0x0030: 0112 310c 0000 0101 080a 0a98 9bfd 0a98 ..1............. 0x0040: 9b93 4854 ..HT 11:16:38.410504 IP (tos 0x0, ttl 64, id 18605, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.52192 > 127.0.0.1.8080: ., cksum 0xb02f (correct), ack 1327241441 win 385 0x0000: 0000 0000 0000 0000 0000 0000 0800 4500 ..............E. 0x0010: 0034 48ad 4000 4006 f414 7f00 0001 7f00 .4H. at .@......... 0x0020: 0001 cbe0 1f90 4ec4 d7ac 4f1c 18e1 8010 ......N...O..... 0x0030: 0181 b02f 0000 0101 080a 0a98 9bfd 0a98 .../............ 0x0040: 9bfd .. 11:16:38.410509 IP (tos 0x0, ttl 64, id 32367, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.8080 > 127.0.0.1.52192: F, cksum 0xb107 (correct), 1327241441:1327241441(0) ack 1321523116 win 274 0x0000: 0000 0000 0000 0000 0000 0000 0800 4500 ..............E. 0x0010: 0034 7e6f 4000 4006 be52 7f00 0001 7f00 .4~o at .@..R...... 0x0020: 0001 1f90 cbe0 4f1c 18e1 4ec4 d7ac 8011 ......O...N..... 0x0030: 0112 b107 0000 0101 080a 0a98 9bfd 0a98 ................ 0x0040: 9b93 .. 11:16:38.410576 IP (tos 0x0, ttl 64, id 18606, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.52192 > 127.0.0.1.8080: F, cksum 0xb02d (correct), 1321523116:1321523116(0) ack 1327241442 win 385 0x0000: 0000 0000 0000 0000 0000 0000 0800 4500 ..............E. 0x0010: 0034 48ae 4000 4006 f413 7f00 0001 7f00 .4H. at .@......... 0x0020: 0001 cbe0 1f90 4ec4 d7ac 4f1c 18e2 8011 ......N...O..... 0x0030: 0181 b02d 0000 0101 080a 0a98 9bfd 0a98 ...-............ 0x0040: 9bfd .. 11:16:38.410587 IP (tos 0x0, ttl 64, id 32368, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.8080 > 127.0.0.1.52192: ., cksum 0xb09c (correct), ack 1321523117 win 274 0x0000: 0000 0000 0000 0000 0000 0000 0800 4500 ..............E. 0x0010: 0034 7e70 4000 4006 be51 7f00 0001 7f00 .4~p at .@..Q...... 0x0020: 0001 1f90 cbe0 4f1c 18e2 4ec4 d7ad 8010 ......O...N..... 0x0030: 0112 b09c 0000 0101 080a 0a98 9bfd 0a98 ................ 0x0040: 9bfd Regards, Nick. From phk at phk.freebsd.dk Fri May 1 10:44:42 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 01 May 2009 10:44:42 +0000 Subject: Theoretical connections/second limit using Varnish In-Reply-To: Your message of "Fri, 01 May 2009 11:25:52 +0100." <49FACE30.7010309@loman.net> Message-ID: <16216.1241174682@critter.freebsd.dk> In message <49FACE30.7010309 at loman.net>, Nick Loman writes: >I'm sure I could find a way of getting Apache to put in a 10msec >Keep-Alive timeout if that was necessary. > >Are you proposing that Varnish would then hold and re-use that backend >connection for a waiting request from a different user/TCP connection? Varnish tries as hard as reasonable to reuse both client and backend connections. If your appache gives varnish just a short chance to do so, I am sure that you can eliminate 40-50% of your connections. >> If you run a tcpdump of the backend traffic, does the connections >> close properly with FIN or does one of the ends RST them ? OK, just making sure. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From jeff at funnyordie.com Fri May 1 15:06:32 2009 From: jeff at funnyordie.com (Jeff Anderson) Date: Fri, 1 May 2009 08:06:32 -0700 Subject: Handling graced objects between varnish servers. In-Reply-To: <22886.1241075795@critter.freebsd.dk> References: <22886.1241075795@critter.freebsd.dk> Message-ID: Is there a way to tell or flag set if the fetched object is a graced object? On Apr 30, 2009, at 12:16 AM, Poul-Henning Kamp wrote: > In message <6D6E4F5A-B9D6-41EB-9B08-27AFEC5D49DB at funnyordie.com>, > Jeff Anderson > writes: > >> How long can I cache objects and keep them >> available as graced objects? > > As long as you want. > >> If a front end varnish cache receives an >> object from a downstream varnish cache that was served as a graced >> object will the front end varnish cache declare it invalid? If so is >> there a way to bypass that invalidation? > > There is no indication passed along that an object is graced, so > the downstream varnish sees it just like any other object and will > interpret its validity based on the headers. > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by > incompetence. --Jeff jeff at funnyordie.com From jeff at funnyordie.com Fri May 1 15:08:47 2009 From: jeff at funnyordie.com (Jeff Anderson) Date: Fri, 1 May 2009 08:08:47 -0700 Subject: Handling graced objects between varnish servers. In-Reply-To: <22886.1241075795@critter.freebsd.dk> References: <22886.1241075795@critter.freebsd.dk> Message-ID: Sorry. You answered that question below. I was hoping to capture graced object statistics and potentially manipulate headers of an object if it is graced. On Apr 30, 2009, at 12:16 AM, Poul-Henning Kamp wrote: > In message <6D6E4F5A-B9D6-41EB-9B08-27AFEC5D49DB at funnyordie.com>, > Jeff Anderson > writes: > >> How long can I cache objects and keep them >> available as graced objects? > > As long as you want. > >> If a front end varnish cache receives an >> object from a downstream varnish cache that was served as a graced >> object will the front end varnish cache declare it invalid? If so is >> there a way to bypass that invalidation? > > There is no indication passed along that an object is graced, so > the downstream varnish sees it just like any other object and will > interpret its validity based on the headers. > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by > incompetence. --Jeff jeff at funnyordie.com From phk at phk.freebsd.dk Sat May 2 16:41:09 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 02 May 2009 16:41:09 +0000 Subject: Handling graced objects between varnish servers. In-Reply-To: Your message of "Fri, 01 May 2009 08:06:32 MST." Message-ID: <7012.1241282469@critter.freebsd.dk> In message , Jeff Anderson writes: >Is there a way to tell or flag set if the fetched object is a graced >object? In VCL you can tell by the negative ttl. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From per.buer at redpill-linpro.com Sun May 3 06:24:40 2009 From: per.buer at redpill-linpro.com (Per Andreas Buer) Date: Sun, 3 May 2009 08:24:40 +0200 (CEST) Subject: Theoretical connections/second limit using Varnish In-Reply-To: <49F9BF57.4020909@loman.net> Message-ID: <292777307.612841241331880828.JavaMail.root@claudius.linpro.no> ----- "Nick Loman" wrote: > Precisely, we only have perhaps 50 PHP children serving requests, so > if these are kept open to serve idle keep-alive connections, that > severely limits the numbers of dynamic page requests we can serve. It sound like you and Michael need a limit on the number of connections for each backend and queuing of requests. I don't think this is possible with Varnish at the moment - but it sounds like a valid feature requests. -- Per Andreas Buer / Business Area Manager Mobile +47 958 39 117 / Phone: +47 21 54 41 21 Redpill Linpro Group - Changing the Game From loco at d0pefish.de Mon May 4 07:12:54 2009 From: loco at d0pefish.de (Peter Hinse) Date: Mon, 04 May 2009 09:12:54 +0200 Subject: Different TTLs dependent upon req.http.host? In-Reply-To: <49F87E5C.5010709@d0pefish.de> References: <49F87E5C.5010709@d0pefish.de> Message-ID: <49FE9576.7080807@d0pefish.de> Peter Hinse schrieb: > Hi all, > > I have a varnish config looking roughly like this: > > backend be1 { > .host = "10.10.10.10"; > .port = "80"; > } > > sub vcl_recv { > elseif (req.http.host ~ "^x.example.com") { > set req.http.Host = "host.example.com"; > set req.backend = be1; > } > elseif (req.http.host ~ "^xi.example.com") { > set req.http.Host = "host.example.com"; > set req.backend = be1; > } > } > > The backend server delivers dynamically rendered images I need to cache. > If the original requests comes in for Host x.example.com, the TTL can be > 12h, if the request is for xi.example.com, the TTL is 15m. > > Since I cannot use "req.http.host" in vcl_fetch and the req.url look the > same, how can I distinguish the two cases and set the obj.ttl differently? No idea? User-defined variables do not work, is there any trick I can use to handle requests to xi.example.com differently? Regards, Peter From phk at phk.freebsd.dk Mon May 4 07:18:35 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 04 May 2009 07:18:35 +0000 Subject: Different TTLs dependent upon req.http.host? In-Reply-To: Your message of "Mon, 04 May 2009 09:12:54 +0200." <49FE9576.7080807@d0pefish.de> Message-ID: <4737.1241421515@critter.freebsd.dk> In message <49FE9576.7080807 at d0pefish.de>, Peter Hinse writes: >No idea? User-defined variables do not work, is there any trick I can >use to handle requests to xi.example.com differently? Which version are you running ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From loco at d0pefish.de Mon May 4 07:35:07 2009 From: loco at d0pefish.de (Peter Hinse) Date: Mon, 04 May 2009 09:35:07 +0200 Subject: Different TTLs dependent upon req.http.host? In-Reply-To: <4737.1241421515@critter.freebsd.dk> References: <4737.1241421515@critter.freebsd.dk> Message-ID: <49FE9AAB.1030204@d0pefish.de> Poul-Henning Kamp schrieb: > In message <49FE9576.7080807 at d0pefish.de>, Peter Hinse writes: > >> No idea? User-defined variables do not work, is there any trick I can >> use to handle requests to xi.example.com differently? > > Which version are you running ? > Hi Poul-Henning, atm, I am running varnish-2.0.2 Regards, Peter From loco at d0pefish.de Mon May 4 12:17:16 2009 From: loco at d0pefish.de (Peter Hinse) Date: Mon, 04 May 2009 14:17:16 +0200 Subject: Different TTLs dependent upon req.http.host? In-Reply-To: <49F87E5C.5010709@d0pefish.de> References: <49F87E5C.5010709@d0pefish.de> Message-ID: <49FEDCCC.6070603@d0pefish.de> Peter Hinse schrieb: > Hi all, > > I have a varnish config looking roughly like this: > > backend be1 { > .host = "10.10.10.10"; > .port = "80"; > } > > sub vcl_recv { > elseif (req.http.host ~ "^x.example.com") { > set req.http.Host = "host.example.com"; > set req.backend = be1; > } > elseif (req.http.host ~ "^xi.example.com") { > set req.http.Host = "host.example.com"; > set req.backend = be1; > } > } > > The backend server delivers dynamically rendered images I need to cache. > If the original requests comes in for Host x.example.com, the TTL can be > 12h, if the request is for xi.example.com, the TTL is 15m. > > Since I cannot use "req.http.host" in vcl_fetch and the req.url look the > same, how can I distinguish the two cases and set the obj.ttl differently? What seems to work (at least the VCL compiles without errors): backend be1 { .host = "10.10.10.10"; .port = "80"; } sub vcl_recv { elseif (req.http.host ~ "^x.example.com") { set req.http.Host = "host.example.com"; set req.http.myhost = "x.example.com"; set req.backend = be1; } elseif (req.http.host ~ "^xi.example.com") { set req.http.Host = "host.example.com"; set req.http.myhost = "xi.example.com"; set req.backend = be1; } } Now I can use the following part in vcl_fetch: if (req.request == "GET" && req.http.myhost == "x.example.com") { set obj.ttl = 12h; deliver; } elseif (req.request == "GET" && req.http.myhost == "xi.example.com") { set obj.ttl = 15m; deliver; } From therziger at idg.de Tue May 12 07:52:29 2009 From: therziger at idg.de (Tobias Herziger) Date: Tue, 12 May 2009 09:52:29 +0200 Subject: ESI problems Message-ID: <1242114749.31972.24.camel@thdesk01> Hi! I'm running varnish 2.0.4 on Ubuntu 9.04 aka Jaunty. I built that package using the Karmic repository with no changes in debian/rules. Apache 2.2 provides the backend on 127.0.0.1:8080. I' using very minimalistic VCL: # cat /etc/varnish/default.vcl backend default { .host = "127.0.0.1"; .port = "8080"; } sub vcl_fetch { if (req.url == "/test.html") { esi; /* Do ESI processing */ set obj.ttl = 30s; } elseif (req.url == "/cgi-bin/date.cgi") { set obj.ttl = 3s; } } Varnish config is pretty simple as well: # sed '/^#/d;/^$/d' /etc/default/varnish NFILES=131072 MEMLOCK=82000 INSTANCE=$(uname -n) DAEMON_OPTS="-a 0.0.0.0:80 \ -f /etc/varnish/default.vcl \ -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G" I extactly follwed the sample that's shown on http://varnish.projects.linpro.no/wiki/ESIfeatures but there's no esi-processing. When I look at the output from varnishlog, there's a line saying ?9 ESI_xmlerror c No ESI processing, first char not '<'? but when i put ? ServerName thdesk01.idgmuc.idg ServerAlias thdesk01 ServerAdmin webmaster at localhost DocumentRoot /var/www ErrorLog /var/log/apache2/error.log CustomLog /var/log/apache2/access.log combined LogLevel warn FileETag none ExpiresActive On ExpiresDefault "access plus 10 seconds" Options FollowSymLinks AllowOverride None Options Indexes FollowSymLinks MultiViews AllowOverride None Order allow,deny allow from all ScriptAlias /cgi-bin /usr/lib/cgi-bin Options +ExecCGI -FollowSymLinks ExpiresDefault "access plus 3 seconds" SetEnv no-gzip and that's the varnishlog: # varnishlog 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1242113246 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1242113249 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1242113252 1.0 9 SessionOpen c 192.168.10.13 36762 0.0.0.0:80 9 ReqStart c 192.168.10.13 36762 58819495 9 RxRequest c GET 9 RxURL c /test.html 9 RxProtocol c HTTP/1.1 9 RxHeader c Host: thdesk01.idgmuc.idg 9 RxHeader c User-Agent: Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.0.10) Gecko/2009042523 Ubuntu/9.04 (jaunty) Firefox/3.0.10 9 RxHeader c Accept: text/html,application/xhtml +xml,application/xml;q=0.9,*/*;q=0.8 9 RxHeader c Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3 9 RxHeader c Accept-Encoding: gzip,deflate 9 RxHeader c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 9 RxHeader c Keep-Alive: 300 9 RxHeader c Connection: keep-alive 9 VCL_call c recv 9 VCL_return c lookup 9 VCL_call c hash 9 VCL_return c hash 9 VCL_call c miss 9 VCL_return c fetch 11 BackendOpen b default 127.0.0.1 48017 127.0.0.1 8080 9 Backend c 11 default default 11 TxRequest b GET 11 TxURL b /test.html 11 TxProtocol b HTTP/1.1 11 TxHeader b Host: thdesk01.idgmuc.idg 11 TxHeader b User-Agent: Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.0.10) Gecko/2009042523 Ubuntu/9.04 (jaunty) Firefox/3.0.10 11 TxHeader b Accept: text/html,application/xhtml +xml,application/xml;q=0.9,*/*;q=0.8 11 TxHeader b Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3 11 TxHeader b Accept-Encoding: gzip,deflate 11 TxHeader b Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 11 TxHeader b X-Varnish: 58819495 11 TxHeader b X-Forwarded-For: 192.168.10.13 11 RxProtocol b HTTP/1.1 11 RxStatus b 200 11 RxResponse b OK 11 RxHeader b Date: Tue, 12 May 2009 07:27:33 GMT 11 RxHeader b Server: Apache 11 RxHeader b Last-Modified: Mon, 11 May 2009 14:13:31 GMT 11 RxHeader b Accept-Ranges: bytes 11 RxHeader b Cache-Control: max-age=10 11 RxHeader b Expires: Tue, 12 May 2009 07:27:43 GMT 11 RxHeader b Vary: Accept-Encoding 11 RxHeader b Content-Encoding: gzip 11 RxHeader b Content-Length: 113 11 RxHeader b Connection: close 11 RxHeader b Content-Type: text/html; charset=UTF-8 9 ObjProtocol c HTTP/1.1 9 ObjStatus c 200 9 ObjResponse c OK 9 ObjHeader c Date: Tue, 12 May 2009 07:27:33 GMT 9 ObjHeader c Server: Apache 9 ObjHeader c Last-Modified: Mon, 11 May 2009 14:13:31 GMT 9 ObjHeader c Cache-Control: max-age=10 9 ObjHeader c Expires: Tue, 12 May 2009 07:27:43 GMT 9 ObjHeader c Vary: Accept-Encoding 9 ObjHeader c Content-Encoding: gzip 9 ObjHeader c Content-Type: text/html; charset=UTF-8 11 BackendClose b default 9 TTL c 58819495 RFC 10 1242113253 0 0 10 0 9 VCL_call c fetch 9 ESI_xmlerror c No ESI processing, first char not '<' 9 TTL c 58819495 VCL 30 1242113253 9 VCL_info c XID 58819495: obj.prefetch (-30) less than ttl (30), ignored. 9 VCL_return c deliver 9 Length c 113 9 VCL_call c deliver 9 VCL_return c deliver 9 TxProtocol c HTTP/1.1 9 TxStatus c 200 9 TxResponse c OK 9 TxHeader c Server: Apache 9 TxHeader c Last-Modified: Mon, 11 May 2009 14:13:31 GMT 9 TxHeader c Cache-Control: max-age=10 9 TxHeader c Expires: Tue, 12 May 2009 07:27:43 GMT 9 TxHeader c Vary: Accept-Encoding 9 TxHeader c Content-Encoding: gzip 9 TxHeader c Content-Type: text/html; charset=UTF-8 9 TxHeader c Content-Length: 113 9 TxHeader c Date: Tue, 12 May 2009 07:27:33 GMT 9 TxHeader c X-Varnish: 58819495 9 TxHeader c Age: 0 9 TxHeader c Via: 1.1 varnish 9 TxHeader c Connection: keep-alive 9 ReqEnd c 58819495 1242113253.012381554 1242113253.013628483 0.000089884 0.001210213 0.000036716 0 StatAddr - 192.168.10.13 0 60153 3 7 0 0 5 2162 1071 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1242113255 1.0 ^C From therziger at idg.de Tue May 12 09:04:49 2009 From: therziger at idg.de (Tobias Herziger) Date: Tue, 12 May 2009 11:04:49 +0200 Subject: ESI problems In-Reply-To: <1242114749.31972.24.camel@thdesk01> References: <1242114749.31972.24.camel@thdesk01> Message-ID: <1242119089.31972.27.camel@thdesk01> Am Dienstag, den 12.05.2009, 09:52 +0200 schrieb Tobias Herziger: > > ScriptAlias /cgi-bin /usr/lib/cgi-bin > > Options +ExecCGI -FollowSymLinks > ExpiresDefault "access plus 3 seconds" > SetEnv no-gzip > > This SetEnv thing here does not do what i want. i unloaded mod_deflate completely and ... ESI works fine. Rgards, Tobi From ingard at startsiden.no Tue May 12 10:18:52 2009 From: ingard at startsiden.no (=?ISO-8859-1?Q?ingard_mev=E5g?=) Date: Tue, 12 May 2009 12:18:52 +0200 Subject: backend probing vs grace on healthy settings Message-ID: <4A094D0C.7010401@startsiden.no> Hi We recently experienced a problem with one of our websites running varnish 2.0.4 with grace enabled. relevant config: backend default { .host = "x.x.x.x"; .port = "80"; .probe = { .timeout = 500 ms; .interval = 5s; .window = 10; .threshold = 1; .request = "GET /index.php HTTP/1.1" "Host: bladiblahdiblah.no" "Connection: close" "Accept-Encoding: text/html" ; } } sub vcl_recv { if (req.backend.healthy) { set req.grace = 30s; } else { set req.grace = 30m; } } The question is: When will the backend be marked sick? Is this after .interval X .window (50 secs) and in that case, what happens after the backend goes down and grace objects are served for 30 secs (as the backend is still marked as healthy). There seems to be 20 secs of "nothing" before the backend is decalred sick. Did I enterpret this correctly? I was told in #varnish that the backend will be marked sick after .threshold of .window probes. This does not seem to be working for me atleast. As excerpt below shows, the backend isnt marked sick until all .window probes has been executed. However, it is marked as healthy after .threshold of .window probes after i start the backend server again. Example: -------------------------------------------------- debug.health 200 624 Backend default is Healthy Current states good: 10 threshold: 1 window: 10 Average responsetime of good probes: 0.001146 Oldest Newest ================================================================ 4444444444444444444444444444444444444444444444444444444444444444 Good IPv4 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Good Xmit SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS Good Shut RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR Good Recv HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH Happy debug.health 200 624 Backend default is Healthy Current states good: 9 threshold: 1 window: 10 Average responsetime of good probes: 0.001146 Oldest Newest ================================================================ 444444444444444444444444444444444444444444444444444444444444444- Good IPv4 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX- Good Xmit SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS- Good Shut RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR- Good Recv HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH- Happy debug.health 200 624 Backend default is Healthy Current states good: 8 threshold: 1 window: 10 Average responsetime of good probes: 0.001146 Oldest Newest ================================================================ 44444444444444444444444444444444444444444444444444444444444444-- Good IPv4 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX-- Good Xmit SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS-- Good Shut RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR-- Good Recv HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH-- Happy debug.health 200 624 Backend default is Healthy Current states good: 7 threshold: 1 window: 10 Average responsetime of good probes: 0.001146 Oldest Newest ================================================================ 4444444444444444444444444444444444444444444444444444444444444--- Good IPv4 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX--- Good Xmit SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS--- Good Shut RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR--- Good Recv HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH--- Happy debug.health 200 624 Backend default is Healthy Current states good: 6 threshold: 1 window: 10 Average responsetime of good probes: 0.001146 Oldest Newest ================================================================ 444444444444444444444444444444444444444444444444444444444444---- Good IPv4 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX---- Good Xmit SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS---- Good Shut RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR---- Good Recv HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH---- Happy debug.health 200 624 Backend default is Healthy Current states good: 5 threshold: 1 window: 10 Average responsetime of good probes: 0.001146 Oldest Newest ================================================================ 44444444444444444444444444444444444444444444444444444444444----- Good IPv4 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX----- Good Xmit SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS----- Good Shut RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR----- Good Recv HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH----- Happy debug.health 200 624 Backend default is Healthy Current states good: 4 threshold: 1 window: 10 Average responsetime of good probes: 0.001146 Oldest Newest ================================================================ 4444444444444444444444444444444444444444444444444444444444------ Good IPv4 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX------ Good Xmit SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS------ Good Shut RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR------ Good Recv HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH------ Happy debug.health 200 624 Backend default is Healthy Current states good: 3 threshold: 1 window: 10 Average responsetime of good probes: 0.001146 Oldest Newest ================================================================ 444444444444444444444444444444444444444444444444444444444------- Good IPv4 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX------- Good Xmit SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS------- Good Shut RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR------- Good Recv HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH------- Happy debug.health 200 624 Backend default is Healthy Current states good: 2 threshold: 1 window: 10 Average responsetime of good probes: 0.001146 Oldest Newest ================================================================ 44444444444444444444444444444444444444444444444444444444-------- Good IPv4 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX-------- Good Xmit SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS-------- Good Shut RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR-------- Good Recv HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH-------- Happy debug.health 200 624 Backend default is Healthy Current states good: 1 threshold: 1 window: 10 Average responsetime of good probes: 0.001146 Oldest Newest ================================================================ 4444444444444444444444444444444444444444444444444444444--------- Good IPv4 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX--------- Good Xmit SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS--------- Good Shut RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR--------- Good Recv HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH--------- Happy debug.health 200 624 Backend default is Healthy Current states good: 1 threshold: 1 window: 10 Average responsetime of good probes: 0.001146 Oldest Newest ================================================================ 4444444444444444444444444444444444444444444444444444444--------- Good IPv4 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX--------- Good Xmit SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS--------- Good Shut RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR--------- Good Recv HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH--------- Happy debug.health 200 621 Backend default is Sick Current states good: 0 threshold: 1 window: 10 Average responsetime of good probes: 0.001146 Oldest Newest ================================================================ 444444444444444444444444444444444444444444444444444444---------- Good IPv4 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX---------- Good Xmit SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS---------- Good Shut RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR---------- Good Recv HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH---------- Happy debug.health 200 624 Backend default is Healthy Current states good: 1 threshold: 1 window: 10 Average responsetime of good probes: 0.001386 Oldest Newest ================================================================ 444444444444444444444444444444444444444444444444444------------4 Good IPv4 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX------------X Good Xmit SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS------------S Good Shut RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR------------R Good Recv HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH------------H Happy From mbaki at aljolit.com Wed May 13 06:05:43 2009 From: mbaki at aljolit.com (Monah Baki) Date: Wed, 13 May 2009 09:05:43 +0300 Subject: Newbie question Message-ID: Hi all, I'm trying to show management as a proof of concept varnish on freebsd 7.2. I know it's supposed to be a reverse proxy and probably locally to your network, but since we have no webserver, I want to set it up and point it to ikea's website (just as a proof of concept). My default.vcl is: backend www { .host = "www.ikea.com.sa"; .port = "80"; } backend www1 { .host = "www.ikeasaudi.com"; .port = "80"; } sub vcl_recv { if (req.http.host ~ "^ikea.com.sa$") { set req.backend = www; } } sub vcl_recv { if (req.http.host ~ "^ikeasaudi.com$") { set req.backend = www1; } } # File type that we will always cache sub vcl_recv { if (req.request == "GET" && req.url ~ "\.(gif|jpg|swf|css|js|png|jpg|jpeg|gif|png|tiff|tif|svg|swf|ico|css|js|vsd|doc|ppt|pps|xls|pdf|mp3|mp4|m4a|ogg|mov|avi|wmv|sxw|zip|gz|bz2|tgz|tar|rar|odc|odb|odf|odg|odi|odp|ods|odt|sxc|sxd|sxi|sxw|dmg|torrent|deb|msi|iso|rpm)$") { lookup; } } Here is part of the logs, how can I tell it's working? 11 SessionOpen c 192.168.1.241 57592 :80 11 ReqStart c 192.168.1.241 57592 419225077 11 RxRequest c GET 11 RxURL c http://www.ikea.com.sa/ 11 RxProtocol c HTTP/1.1 11 RxHeader c Host: www.ikea.com.sa 11 RxHeader c User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.10) Gecko/2009042523 Ubuntu/9.04 (jaunty) Firefox/3.0.10 11 RxHeader c Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 11 RxHeader c Accept-Language: en-us,en;q=0.5 11 RxHeader c Accept-Encoding: gzip,deflate 11 RxHeader c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 11 RxHeader c Keep-Alive: 300 11 RxHeader c Proxy-Connection: keep-alive 11 RxHeader c Cookie: f3571cbb2ae4c36582c82e33e80ba2a8=-; mbfcookie[lang]=en; virtuemart=08490ee0f3529658159d22f4b334078d 11 VCL_call c recv 11 VCL_return c pass 11 VCL_call c pass 11 VCL_return c pass 12 BackendOpen b www 192.168.1.109 59301 70.40.211.108 80 11 Backend c 12 www www 12 TxRequest b GET 12 TxURL b http://www.ikea.com.sa/ 12 TxProtocol b HTTP/1.1 12 TxHeader b Host: www.ikea.com.sa 12 TxHeader b User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.10) Gecko/2009042523 Ubuntu/9.04 (jaunty) Firefox/3.0.10 12 TxHeader b Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 12 TxHeader b Accept-Language: en-us,en;q=0.5 12 TxHeader b Accept-Encoding: gzip,deflate 12 TxHeader b Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 12 TxHeader b Proxy-Connection: keep-alive 12 TxHeader b Cookie: f3571cbb2ae4c36582c82e33e80ba2a8=-; mbfcookie[lang]=en; virtuemart=08490ee0f3529658159d22f4b334078d 12 TxHeader b X-Varnish: 419225077 12 TxHeader b X-Forwarded-For: 192.168.1.241 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1242183717 1.0 12 RxProtocol b HTTP/1.1 12 RxStatus b 200 12 RxResponse b OK 12 RxHeader b Date: Wed, 13 May 2009 06:01:55 GMT 12 RxHeader b Server: Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8i DAV/2 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 12 RxHeader b X-Powered-By: PHP/5.2.9 12 RxHeader b Expires: Mon, 26 Jul 1997 05:00:00 GMT 12 RxHeader b Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 12 RxHeader b Pragma: no-cache 12 RxHeader b Set-Cookie: f3571cbb2ae4c36582c82e33e80ba2a8=45af50eab544822fcdef7e40005cb6d4; path=/ 12 RxHeader b Set-Cookie: lang=deleted; expires=Tue, 13-May-2008 06:01:55 GMT; path=/ 12 RxHeader b Set-Cookie: mbfcookie=deleted; expires=Tue, 13-May-2008 06:01:55 GMT; path=/ 12 RxHeader b Set-Cookie: mbfcookie[lang]=en; expires=Thu, 14-May-2009 06:01:56 GMT; path=/ 12 RxHeader b Last-Modified: Wed, 13 May 2009 06:01:56 GMT 12 RxHeader b Transfer-Encoding: chunked 12 RxHeader b Content-Type: text/html 11 ObjProtocol c HTTP/1.1 11 ObjStatus c 200 11 ObjResponse c OK 11 ObjHeader c Date: Wed, 13 May 2009 06:01:55 GMT 11 ObjHeader c Server: Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8i DAV/2 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 11 ObjHeader c X-Powered-By: PHP/5.2.9 11 ObjHeader c Expires: Mon, 26 Jul 1997 05:00:00 GMT 11 ObjHeader c Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 11 ObjHeader c Pragma: no-cache 11 ObjHeader c Set-Cookie: f3571cbb2ae4c36582c82e33e80ba2a8=45af50eab544822fcdef7e40005cb6d4; path=/ 11 ObjHeader c Set-Cookie: lang=deleted; expires=Tue, 13-May-2008 06:01:55 GMT; path=/ 11 ObjHeader c Set-Cookie: mbfcookie=deleted; expires=Tue, 13-May-2008 06:01:55 GMT; path=/ 11 ObjHeader c Set-Cookie: mbfcookie[lang]=en; expires=Thu, 14-May-2009 06:01:56 GMT; path=/ 11 ObjHeader c Last-Modified: Wed, 13 May 2009 06:01:56 GMT 11 ObjHeader c Content-Type: text/html 12 BackendReuse b www 11 TTL c 419225077 RFC 0 1242183717 1242194515 869893200 0 0 11 VCL_call c fetch 11 VCL_return c pass 11 Length c 41666 11 VCL_call c deliver 11 VCL_return c deliver 11 TxProtocol c HTTP/1.1 11 TxStatus c 200 11 TxResponse c OK 11 TxHeader c Server: Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8i DAV/2 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 11 TxHeader c X-Powered-By: PHP/5.2.9 11 TxHeader c Expires: Mon, 26 Jul 1997 05:00:00 GMT 11 TxHeader c Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 11 TxHeader c Pragma: no-cache 11 TxHeader c Set-Cookie: f3571cbb2ae4c36582c82e33e80ba2a8=45af50eab544822fcdef7e40005cb6d4; path=/ 11 TxHeader c Set-Cookie: lang=deleted; expires=Tue, 13-May-2008 06:01:55 GMT; path=/ 11 TxHeader c Set-Cookie: mbfcookie=deleted; expires=Tue, 13-May-2008 06:01:55 GMT; path=/ 11 TxHeader c Set-Cookie: mbfcookie[lang]=en; expires=Thu, 14-May-2009 06:01:56 GMT; path=/ 11 TxHeader c Last-Modified: Wed, 13 May 2009 06:01:56 GMT 11 TxHeader c Content-Type: text/html 11 TxHeader c Content-Length: 41666 11 TxHeader c Date: Wed, 13 May 2009 03:01:58 GMT 11 TxHeader c X-Varnish: 419225077 11 TxHeader c Age: 1 11 TxHeader c Via: 1.1 varnish 11 TxHeader c Connection: keep-alive 11 ReqEnd c 419225077 1242183716.319736719 1242183718.637232304 0.004083157 2.315966845 0.001528740 0 StatAddr - 192.168.1.241 0 2 1 1 0 1 1 831 41666 11 ReqStart c 192.168.1.241 57592 419225078 11 RxRequest c GET 11 RxURL c http://www.ikeasaudi.com/ikea.com.sa/mambots/system/jceutils/jscripts/jceutils.js 11 RxProtocol c HTTP/1.1 11 RxHeader c Host: www.ikeasaudi.com 11 RxHeader c User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.10) Gecko/2009042523 Ubuntu/9.04 (jaunty) Firefox/3.0.10 11 RxHeader c Accept: */* 11 RxHeader c Accept-Language: en-us,en;q=0.5 11 RxHeader c Accept-Encoding: gzip,deflate 11 RxHeader c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 11 RxHeader c Keep-Alive: 300 11 RxHeader c Proxy-Connection: keep-alive 11 RxHeader c Referer: http://www.ikea.com.sa/ 11 RxHeader c Cookie: f3571cbb2ae4c36582c82e33e80ba2a8=4e2be3c5222672b31f039dd97864ddb5; mbfcookie[lang]=en; virtuemart=e4e5db67f6e31914d714315b3d0048f8 11 RxHeader c If-Modified-Since: Sun, 10 May 2009 21:03:14 GMT 11 RxHeader c If-None-Match: "41e9599-e72-4699530a4e880" 11 VCL_call c recv 11 VCL_return c lookup 11 VCL_call c hash 11 VCL_return c hash 11 VCL_call c miss 11 VCL_return c fetch -------------- next part -------------- An HTML attachment was scrubbed... URL: From v.bilek at 1art.cz Wed May 13 06:36:55 2009 From: v.bilek at 1art.cz (=?UTF-8?B?VsOhY2xhdiBCw61sZWs=?=) Date: Wed, 13 May 2009 08:36:55 +0200 Subject: varnish experience on high request rates Message-ID: <4A0A6A87.3010707@1art.cz> Helo I work on one project, hosted on LVS (linux virtual server) cluster (cca 20 nodes) which makes 25K req/s 800Mbit/s in peaks. I want to use varnish to accelerate that. Question: Is there anyone using varnish on such request rates? If yes, what HW do you use. Thanks for any comment. Vasek Bilek From ingard at startsiden.no Sat May 16 07:43:33 2009 From: ingard at startsiden.no (=?utf-8?Q?Ingard_Mev=C3=A5g?=) Date: Sat, 16 May 2009 09:43:33 +0200 Subject: backend probing vs grace on healthy settings In-Reply-To: <4A094A5A.8000503@startsiden.no> References: <4A094A5A.8000503@startsiden.no> Message-ID: <6B208A84-2E65-4E4D-A912-ADD4145E5205@startsiden.no> Bump On 12. mai. 2009, at 12.07, ingard mev?g wrote: > Hi > > We recently experienced a problem with one of our websites running > varnish 2.0.4 with grace enabled. > > relevant config: > > backend default { > .host = "x.x.x.x"; > .port = "80"; > .probe = { > .timeout = 500 ms; > .interval = 5s; > .window = 10; > .threshold = 1; > .request = > "GET /index.php HTTP/1.1" > "Host: bladiblahdiblah.no" > "Connection: close" > "Accept-Encoding: text/html" ; > } > } > sub vcl_recv { > if (req.backend.healthy) { > set req.grace = 30s; > } else { > set req.grace = 30m; > } > } > The question is: When will the backend be marked sick? Is this > after .interval X .window (50 secs) and in that case, what happens > after the backend goes down and grace objects are served for 30 secs > (as the backend is still marked as healthy). There seems to be 20 > secs of "nothing" before the backend is decalred sick. > > Did I enterpret this correctly? I was told in #varnish that the > backend will be marked sick after .threshold of .window probes. This > does not seem to be working for me atleast. As excerpt below shows, > the backend isnt marked sick until all .window probes has been > executed. However, it is marked as healthy after .threshold > of .window probes after i start the backend server again. > > Example: > -------------------------------------------------- > > debug.health > 200 624 Backend default is Healthy > Current states good: 10 threshold: 1 window: 10 > Average responsetime of good probes: 0.001146 > Oldest Newest > ================================================================ > 4444444444444444444444444444444444444444444444444444444444444444 > Good IPv4 > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > Good Xmit > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS > Good Shut > RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR > Good Recv > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH Happy > > debug.health > 200 624 Backend default is Healthy > Current states good: 9 threshold: 1 window: 10 > Average responsetime of good probes: 0.001146 > Oldest Newest > ================================================================ > 444444444444444444444444444444444444444444444444444444444444444- > Good IPv4 > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX- > Good Xmit > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS- > Good Shut > RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR- > Good Recv > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH- Happy > > debug.health > 200 624 Backend default is Healthy > Current states good: 8 threshold: 1 window: 10 > Average responsetime of good probes: 0.001146 > Oldest Newest > ================================================================ > 44444444444444444444444444444444444444444444444444444444444444-- > Good IPv4 > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX-- > Good Xmit > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS-- > Good Shut > RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR-- > Good Recv > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH-- Happy > > debug.health > 200 624 Backend default is Healthy > Current states good: 7 threshold: 1 window: 10 > Average responsetime of good probes: 0.001146 > Oldest Newest > ================================================================ > 4444444444444444444444444444444444444444444444444444444444444--- > Good IPv4 > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX--- > Good Xmit > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS--- > Good Shut > RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR--- > Good Recv > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH--- Happy > > debug.health > 200 624 Backend default is Healthy > Current states good: 6 threshold: 1 window: 10 > Average responsetime of good probes: 0.001146 > Oldest Newest > ================================================================ > 444444444444444444444444444444444444444444444444444444444444---- > Good IPv4 > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX---- > Good Xmit > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS---- > Good Shut > RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR---- > Good Recv > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH---- Happy > > debug.health > 200 624 Backend default is Healthy > Current states good: 5 threshold: 1 window: 10 > Average responsetime of good probes: 0.001146 > Oldest Newest > ================================================================ > 44444444444444444444444444444444444444444444444444444444444----- > Good IPv4 > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX----- > Good Xmit > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS----- > Good Shut > RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR----- > Good Recv > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH----- Happy > > debug.health > 200 624 Backend default is Healthy > Current states good: 4 threshold: 1 window: 10 > Average responsetime of good probes: 0.001146 > Oldest Newest > ================================================================ > 4444444444444444444444444444444444444444444444444444444444------ > Good IPv4 > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX------ > Good Xmit > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS------ > Good Shut > RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR------ > Good Recv > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH------ Happy > > debug.health > 200 624 Backend default is Healthy > Current states good: 3 threshold: 1 window: 10 > Average responsetime of good probes: 0.001146 > Oldest Newest > ================================================================ > 444444444444444444444444444444444444444444444444444444444------- > Good IPv4 > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX------- > Good Xmit > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS------- > Good Shut > RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR------- > Good Recv > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH------- Happy > > debug.health > 200 624 Backend default is Healthy > Current states good: 2 threshold: 1 window: 10 > Average responsetime of good probes: 0.001146 > Oldest Newest > ================================================================ > 44444444444444444444444444444444444444444444444444444444-------- > Good IPv4 > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX-------- > Good Xmit > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS-------- > Good Shut > RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR-------- > Good Recv > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH-------- Happy > > debug.health > 200 624 Backend default is Healthy > Current states good: 1 threshold: 1 window: 10 > Average responsetime of good probes: 0.001146 > Oldest Newest > ================================================================ > 4444444444444444444444444444444444444444444444444444444--------- > Good IPv4 > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX--------- > Good Xmit > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS--------- > Good Shut > RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR--------- > Good Recv > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH--------- Happy > > debug.health > 200 624 Backend default is Healthy > Current states good: 1 threshold: 1 window: 10 > Average responsetime of good probes: 0.001146 > Oldest Newest > ================================================================ > 4444444444444444444444444444444444444444444444444444444--------- > Good IPv4 > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX--------- > Good Xmit > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS--------- > Good Shut > RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR--------- > Good Recv > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH--------- Happy > > debug.health > 200 621 Backend default is Sick > Current states good: 0 threshold: 1 window: 10 > Average responsetime of good probes: 0.001146 > Oldest Newest > ================================================================ > 444444444444444444444444444444444444444444444444444444---------- > Good IPv4 > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX---------- > Good Xmit > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS---------- > Good Shut > RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR---------- > Good Recv > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH---------- Happy > > debug.health > 200 624 Backend default is Healthy > Current states good: 1 threshold: 1 window: 10 > Average responsetime of good probes: 0.001386 > Oldest Newest > ================================================================ > 444444444444444444444444444444444444444444444444444------------4 > Good IPv4 > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX------------X > Good Xmit > SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS------------S > Good Shut > RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR------------R > Good Recv > HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH------------H Happy > From lukas.loesche at bertelsmann.de Sun May 17 22:53:08 2009 From: lukas.loesche at bertelsmann.de (Loesche, Lukas, scoyo) Date: Mon, 18 May 2009 00:53:08 +0200 Subject: virtual memory/cache hitrate problem In-Reply-To: <4A094D0C.7010401@startsiden.no> Message-ID: Hi everyone, I'm running varnish (2.0.2) on Linux x64 inside Virtuozzo Containers 4.0. The container has got 2GB core memory assigned. I have a backend that holds about 30GB binary objects. Varnish is set to assign a ttl of 30 days to each object it caches. I start varnish with -s file,/var/lib/varnish/varnish_storage.bin,50GB because I assumed it would be wise to have a cache file that is larger than the max. amount of backend objects. However I see a sawtooth behaviour in cache hit rate that corresponds with the server's virtual memory allocation. Munin stats: http://www.flickr.com/photos/25013820 at N02/sets/72157618375799306/ It looks like varnish hits the container's memory limit, purges all memory and all objects from the store and starts from the beginning. If I set -s file to 1500MB virtual memory usage is at a constant 1.5GB as one would expect. However my goal is to have all objects (30GB) on the varnish server's local disk, and the most requested 1.5-2GB in it's core memory. I thought that setting the file store size to 50GB would do the trick and the server than would decide which objects are hold in memory and which are read from local disk, however the 100% MISS ratio whenever the server hit's it's virtual memory allocation limit tells me I've done something wrong. Any ideas as to what that might be? Am I using varnish the wrong way? -- Lukas From tfheen at redpill-linpro.com Mon May 18 09:13:24 2009 From: tfheen at redpill-linpro.com (Tollef Fog Heen) Date: Mon, 18 May 2009 11:13:24 +0200 Subject: Varnish as load balancer In-Reply-To: <69C3BA0F-2591-484A-9B72-DEF696CDB3EC@igloo.cl> (Carlos Oliva G.'s message of "Sun, 19 Apr 2009 23:39:23 -0400") References: <69C3BA0F-2591-484A-9B72-DEF696CDB3EC@igloo.cl> Message-ID: <873ab2ydsb.fsf@qurzaw.linpro.no> ]] "Carlos Oliva G." | In the current setup I'm skipping Lighttpd altogether and began using | Varnish as a load balancer altogether as well, with the 'directors' | feature. However I'm still missing the last bit of the requested | functionality that is to evenly balance all the requests from | 'anonymous' users that aren't registered/logged into the system (which | I am already identifying in the sub_rcvd function of my VCL), but for | registered users to preserve their session by redirecting them always | to the same backend server. I know I can create a 'hash' using some | relatively unique values, as the originating IP address + the User | Agent and a session-dependent cookie value, but what I don't know is | (if possible), how to use this hash to determinate to what backend | server to direct the request to. We currently don't have a hashing load balancer (which is what you seem to want). (Patches accepted, or Redpill Linpro can get you a quote on how much work it is to develop it.) What you could do is have the backend set a cookie with its name when the user logs in, and then choose backend depending on cookie value. Another alternative is to use nginx or similar as Varnish' backend and let that do the hashing and load balancing. I have been told it's quite good, but I don't have experience with it myself. -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 From tfheen at redpill-linpro.com Mon May 18 09:35:16 2009 From: tfheen at redpill-linpro.com (Tollef Fog Heen) Date: Mon, 18 May 2009 11:35:16 +0200 Subject: Newbie question In-Reply-To: (Monah Baki's message of "Wed, 13 May 2009 09:05:43 +0300") References: Message-ID: <87y6suwy7f.fsf@qurzaw.linpro.no> ]] Monah Baki | Here is part of the logs, how can I tell it's working? It doesn't seem to be caching, most likely because the backend sets a cookie (which means we, by default, don't cache). -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 From tfheen at redpill-linpro.com Mon May 18 09:36:28 2009 From: tfheen at redpill-linpro.com (Tollef Fog Heen) Date: Mon, 18 May 2009 11:36:28 +0200 Subject: varnish experience on high request rates In-Reply-To: <4A0A6A87.3010707@1art.cz> (=?utf-8?Q?=22V=C3=A1clav_B=C3=ADl?= =?utf-8?Q?ek=22's?= message of "Wed, 13 May 2009 08:36:55 +0200") References: <4A0A6A87.3010707@1art.cz> Message-ID: <87tz3iwy5f.fsf@qurzaw.linpro.no> ]] V?clav B?lek | Is there anyone using varnish on such request rates? If yes, what HW do | you use. I've had Varnish serve about 20k requests/sec on a single node. I've also had it fill a 1Gbit pipe on a single node. Hardware in both cases was some reasonably new Xeon system with enough memory to fit your working set. -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 From varnish at held-im-ruhestand.de Mon May 18 09:33:31 2009 From: varnish at held-im-ruhestand.de (Christoph Handel) Date: Mon, 18 May 2009 11:33:31 +0200 Subject: Varnish as load balancer In-Reply-To: <873ab2ydsb.fsf@qurzaw.linpro.no> References: <69C3BA0F-2591-484A-9B72-DEF696CDB3EC@igloo.cl> <873ab2ydsb.fsf@qurzaw.linpro.no> Message-ID: <20090518093330.GA10178@cha-desktop> On Mon, May 18, 2009 at 11:13:24AM +0200, Tollef Fog Heen wrote: > ]] "Carlos Oliva G." > > | In the current setup I'm skipping Lighttpd altogether and began using > | Varnish as a load balancer altogether as well, with the 'directors' > | feature. However I'm still missing the last bit of the requested > | functionality that is to evenly balance all the requests from > | 'anonymous' users that aren't registered/logged into the system (which > | I am already identifying in the sub_rcvd function of my VCL), but for > | registered users to preserve their session by redirecting them always > | to the same backend server. I know I can create a 'hash' using some > | relatively unique values, as the originating IP address + the User > | Agent and a session-dependent cookie value, but what I don't know is > | (if possible), how to use this hash to determinate to what backend > | server to direct the request to. > > We currently don't have a hashing load balancer (which is what you seem > to want). (Patches accepted, or Redpill Linpro can get you a quote on > how much work it is to develop it.) > > What you could do is have the backend set a cookie with its name when > the user logs in, and then choose backend depending on cookie value. > > Another alternative is to use nginx or similar as Varnish' backend and > let that do the hashing and load balancing. I have been told it's quite > good, but I don't have experience with it myself. you could take a look at haproxy (http://haproxy.1wt.eu/) copes with pretty high loads and allows you to inject cookies for sticky sessions. Greetings Christoph From kontakt at web30.ch Wed May 20 08:38:14 2009 From: kontakt at web30.ch (Simon Kammerer) Date: Wed, 20 May 2009 10:38:14 +0200 Subject: Cookie validation with varnish? Message-ID: <4A13C176.6020202@web30.ch> hi, has anyone done something like this (high level description...): Web application sets cookie for user authentication, varnish acts as reverse proxy in front of dedicated image servers and checks if the cookie send by the user is a valid cookie set by the web application. Meaning that varnish can validate cookies (or session tokens attached to the GET request) against an external validating service, cache the result for a given TTL and then serve the requested content (or not). Required level of security is low: The idea is to prevent the world from accessing media files on the dedicated image servers without login to the main web application. No superprivate data to protect. If someone could theoretically gain access to a few files due to some TTL race conditions or such, thats no tragedy. No roles, per file permission etc. (for now...). I'm quite sure it's possible by inlining C in VCL. Do you think this could be possible out of the box with some trickery? Like creating an URL from the cookie, check this URL for 200 OK, cache the result, check further requests against the cached results or so? Regards Simon From phk at phk.freebsd.dk Wed May 20 09:28:42 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 20 May 2009 09:28:42 +0000 Subject: Cookie validation with varnish? In-Reply-To: Your message of "Wed, 20 May 2009 10:38:14 +0200." <4A13C176.6020202@web30.ch> Message-ID: <68453.1242811722@critter.freebsd.dk> In message <4A13C176.6020202 at web30.ch>, Simon Kammerer writes: >Web application sets cookie for user authentication, varnish acts as >reverse proxy in front of dedicated image servers and checks if the >cookie send by the user is a valid cookie set by the web application. I would love to extend VCL to allow such stuff to be done, but right now my focus is on persistent storage. Poul-Henning PS: We are happy to hand out "wiki" bits (drop me an email), so people can add ideas to our "shopping list" wiki page: http://varnish.projects.linpro.no/wiki/PostTwoShoppingList -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Wed May 20 13:12:53 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 20 May 2009 13:12:53 +0000 Subject: virtual memory/cache hitrate problem In-Reply-To: Your message of "Mon, 18 May 2009 00:53:08 +0200." Message-ID: <6621.1242825173@critter.freebsd.dk> In message , "Loesche, Lukas, scoy o" writes: >It looks like varnish hits the container's memory limit, purges all memory >and all objects from the store and starts from the beginning. I suspect you run into the LRU purging (and it sounds like it has a bug). You don't say if your 30GB is 30M 1k objects or 3M 10k object, but you need to be aware that there is a per-object overhead (you can trim this somewhat down with the obj_workspace paramter), so 50GB may not be enough to hold 30GB of objects. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From lukas.loesche at bertelsmann.de Wed May 20 14:26:45 2009 From: lukas.loesche at bertelsmann.de (Loesche, Lukas, scoyo) Date: Wed, 20 May 2009 16:26:45 +0200 Subject: virtual memory/cache hitrate problem In-Reply-To: <6621.1242825173@critter.freebsd.dk> Message-ID: > You don't say if your 30GB is 30M 1k objects or 3M 10k object It's mostly >50kb <2MB SWF, PNG, MP3, FLV files. 360232 of them. > trim this somewhat down with the obj_workspace paramter), so 50GB > may not be enough to hold 30GB of objects. Well my problem is that I don't even hit the 30GB limit. The file is 50GB but varnish runs out of memory once it has fetched around 2GB (the 'physical' memory limit of the server) from the backends. My guess is that the problem lies within how Virtuozzo counts memory usage inside it's virtual containers. When I assign 2GB Memory to the virtual Server and some process allocates 1GB virtual memory, while only using 200MB real memory, the Server will have 1 of it's 2 GB memory shown as 'in use'. You can see this in the two munin charts I liked in my original post. The server's memory usage is almost identical to varnish's VM allocation. Basically, what I'd like to know is, am I using varnish correct by assigning 50GB or even more to the cache even if the server only has 2GB physical memory? The way I understood it is that varnish will map the cache file into it's own address space and just let the Kernel decide which parts to hold in memory and which not to. So I would think it's okay to have a cache file that is several times bigger than the physical memory. Thanks, -- Lukas From phk at phk.freebsd.dk Wed May 20 15:04:27 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 20 May 2009 15:04:27 +0000 Subject: virtual memory/cache hitrate problem In-Reply-To: Your message of "Wed, 20 May 2009 16:26:45 +0200." Message-ID: <7079.1242831867@critter.freebsd.dk> In message , "Loesche, Lukas, scoy o" writes: >Basically, what I'd like to know is, am I using varnish correct by assigning >50GB or even more to the cache even if the server only has 2GB physical >memory? Yes, the VM system should take care of the rest. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From jeremy at hinegardner.org Wed May 20 16:35:43 2009 From: jeremy at hinegardner.org (Jeremy Hinegardner) Date: Wed, 20 May 2009 10:35:43 -0600 Subject: dynamic determination of backend? Message-ID: <20090520163543.GK32435@hinegardner.org> Hi all, I'm working on a consistent hashing based system for our asset storage and am considering putting varnish in front of it. We have the library, that given the url can determine the backend server:port, and are wondering the best way to integrate this into VCL. My initial thought is something along the lines of: sub vcl_miss { C{ char *be_name = calloc( 32, sizeof( char )); char done = 0; int n; int i; /* is this the right way to get the url ? */ n = backend_name_for_url( VRT_r_req_url( sp ), be_name ); for(i = 0 ; !done && (i < VCL_conf.ndirector) ; i++ ) { /* what can be compared to be_name to find the right director? */ if ( strncmp( be_name, VCL_conf.director[i]->vcl_name, n ) == 0 ) { VRT_l_req_backend(sp, VCL_conf.director[i]); done = 1; } } free(be_name); }C fetch; } Unfortunately, the compiler doesn't like acessing the director pointer dereference to access 'vcl_name'. I cobbled this together after digging through the source, so I'm sure that there is a better way to do this. I think I'm close with this idea, just having a problem getting the names of the the various backends at runtime. Essentially I'd like to be able to dynamically lookup a backend by name and then directly assign it as that requests backend. Any thoughts? Or am I completely off base here? enjoy, -jeremy -- ======================================================================== Jeremy Hinegardner jeremy at hinegardner.org From alex.mizrahi at gmail.com Sun May 24 17:41:29 2009 From: alex.mizrahi at gmail.com (Alex Mizrahi) Date: Sun, 24 May 2009 20:41:29 +0300 Subject: grace and backend faults Message-ID: <520E90AB304A421AA0C712B9501A0ED8@killer> hi serving stale content during grace period works fine if backend polling can detect that server is down. however in some cases polling thinks that server is fine, but it returns 50x error for some requests (e.g. when database server is overloaded), and it might be better if stale content would be served. (by the way, that's how grace works in nginx web server's cache.) as i understand, varnish does not support this case now. nevertheless, i've tried to hack something with vcl, using example with restarts as inspiration. i've made a nonexistant backend for this purposes, with polling enabled: backend nonexistant { .host = "localhost"; .port = "31337"; .probe = { .interval = 10s; .timeout = 0.3 s; .window = 2; .threshold = 1; } } then it vcl_fetch, when it sees status 500, it calls restart: sub vcl_fetch { if (obj.status == 500) { restart; } } and vcl_recv sets backend to nonexistant in case of restart: sub vcl_recv { if (req.restarts == 0) { set req.grace = 2m; set req.backend = apache; } else { set req.grace = 2m; set req.backend = nonexistant; } } to my surprise, it worked and stale content was served instead of error. but it tries to use backend each time in this case. i've tried to work this out adjusting obj.ttl in vcl_fetch: sub vcl_fetch { if (obj.status == 500) { restart; } if (req.restarts == 1) { set obj.ttl = 120s; } } it did not help, though. is it because ttl can be adjusted only when object is fetched from backend, or something is wrong here? also, if there is no cached version of this page, i'd like it to show error from backend rather than error from varnish, but it is not trivial to do this. i think it can be made via one more restart, but then backend will have two requests instead of one, which is not good. so, this hack in vcl is not very usable, in my situation, at least. are there plans to implement it? i've found this feature requested in ticket #369, but this ticket was last modified 4 monthes ago.. From samcrawford at gmail.com Mon May 25 17:37:48 2009 From: samcrawford at gmail.com (Sam Crawford) Date: Mon, 25 May 2009 18:37:48 +0100 Subject: Cookie validation with varnish? In-Reply-To: <68453.1242811722@critter.freebsd.dk> References: <4A13C176.6020202@web30.ch> <68453.1242811722@critter.freebsd.dk> Message-ID: I too would be very interested in this. I'm not sure if Simon's original suggestion is possible in VCL, but I'm going to have an experiment and see if I can make it work. Thanks, Sam 2009/5/20 Poul-Henning Kamp > In message <4A13C176.6020202 at web30.ch>, Simon Kammerer writes: > > >Web application sets cookie for user authentication, varnish acts as > >reverse proxy in front of dedicated image servers and checks if the > >cookie send by the user is a valid cookie set by the web application. > > I would love to extend VCL to allow such stuff to be done, but right > now my focus is on persistent storage. > > Poul-Henning > > PS: We are happy to hand out "wiki" bits (drop me an email), so > people can add ideas to our "shopping list" wiki page: > > http://varnish.projects.linpro.no/wiki/PostTwoShoppingList > > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.hoopes at gmail.com Mon May 25 17:33:16 2009 From: matthew.hoopes at gmail.com (Matthew Hoopes) Date: Mon, 25 May 2009 13:33:16 -0400 Subject: cache purging questions Message-ID: Hi, I run a site with many subdomains. (Something like username.domain.com). When a user changes something about their data on the backend, I need to purge the varnish caches. So, I'm trying to purge the caches only for that subdomain, and not everything on domain.com . To purge these caches, I (programmatically) ssh to each of my varnish proxy boxes, and run this command: echo url.purge *username.domain.com* | nc -q 1 localhost 6082 Now, this does work. However, I get the debug message: 0 Debug "REGEX: " Is there something I can change with my regex to not have this message? It's strange, because the cache is purged of items requested from that domain. Secondly, when I do this, the first request after that cache purge gives an empty response. hoopes at crotty:/tmp$ curl http://username.domain.com/file.html curl: (52) Empty reply from server This is the only thing that shows up while I am running varnishlog -x CLI 13 SessionOpen c 209.114.135.220 23780 Nothing shows up in varnishncsa, either (leading me to believe varnish doesn't see it as a valid request?). Then, after that, all subsequent requests make it to the backend (and are seen by varnishncsa), and are properly cached again. This happens from any host, not just the one i did the purge from. (After a little testing, if i wait a while (say, enough to read a couple blogs), the request goes through properly...a held connection times out?) If anyone can help me with either of these two problems (or just point me in the correct direction), i'd be much obliged. Thanks, matt hoopes -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.hoopes at gmail.com Mon May 25 17:37:11 2009 From: matthew.hoopes at gmail.com (Matthew Hoopes) Date: Mon, 25 May 2009 13:37:11 -0400 Subject: cache purging questions In-Reply-To: References: Message-ID: On Mon, May 25, 2009 at 1:33 PM, Matthew Hoopes wrote: > Hi, > > I run a site with many subdomains. (Something like username.domain.com). > When a user changes something about their data on the backend, I need to > purge the varnish caches. So, I'm trying to purge the caches only for that > subdomain, and not everything on domain.com . To purge these caches, I > (programmatically) ssh to each of my varnish proxy boxes, and run this > command: > > echo url.purge *username.domain.com* | nc -q 1 localhost 6082 > > Now, this does work. However, I get the debug message: > > 0 Debug "REGEX: " > > > Is there something I can change with my regex to not have this message? > It's strange, because the cache is purged of items requested from that > domain. > > Secondly, when I do this, the first request after that cache purge gives an > empty response. > > hoopes at crotty:/tmp$ curl http://username.domain.com/file.html > curl: (52) Empty reply from server > > This is the only thing that shows up while I am running varnishlog -x CLI > > 13 SessionOpen c 209.114.135.220 23780 > > Nothing shows up in varnishncsa, either (leading me to believe varnish > doesn't see it as a valid request?). Then, after that, all subsequent > requests make it to the backend (and are seen by varnishncsa), and are > properly cached again. This happens from any host, not just the one i did > the purge from. > > (After a little testing, if i wait a while (say, enough to read a couple > blogs), the request goes through properly...a held connection times out?) > > If anyone can help me with either of these two problems (or just point me > in the correct direction), i'd be much obliged. > > Thanks, > matt hoopes Shoot, sorry should have mentioned the version...looks like 1.0.3...should I just upgrade to 2? root at proxy1:/etc/varnish# dpkg-query -s varnish Package: varnish Status: install ok installed Priority: optional Section: web Installed-Size: 736 Maintainer: Ubuntu MOTU Developers Architecture: amd64 Version: 1.0.3-2ubuntu1 Depends: gcc (>= 3.3), libc6 (>= 2.4), libc6-dev, libncurses5 (>= 5.6+20071006-3) Conffiles: /etc/varnish/vcl.conf be65fff525f05bcd5801c8c56d09672f /etc/default/varnish 1eacf59b849abf38adaf756d14978633 /etc/init.d/varnish 674b353ab86aa5d1b37d1140ec9df118 Description: A state-of-the-art, high-performance HTTP accelerator varnish is the server-side alternative to Squid, written primarily with speed in mind, and with a look to implementing full ESI-support in a future release. . The goal of the Varnish project is to develop a state-of-the-art, high-performance HTTP accelerator. . Varnish is targeted primarily at the FreeBSD 6 and Linux 2.6 platforms, and will take full advantage of the advanced I/O features offered by these operating systems. Original-Maintainer: Stig Sandbeck Mathiesen -------------- next part -------------- An HTML attachment was scrubbed... URL: From des at des.no Tue May 26 19:48:36 2009 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Tue, 26 May 2009 21:48:36 +0200 Subject: cache purging questions In-Reply-To: (Matthew Hoopes's message of "Mon, 25 May 2009 13:33:16 -0400") References: Message-ID: <86zlczir1n.fsf@ds4.des.no> Matthew Hoopes writes: > url.purge *username.domain.com* This is not a valid regular expression. DES -- Dag-Erling Sm?rgrav - des at des.no From m.walraven at terantula.com Tue May 26 21:29:08 2009 From: m.walraven at terantula.com (Marco Walraven) Date: Tue, 26 May 2009 23:29:08 +0200 Subject: Varnish restarts when all memory is allocated Message-ID: <20090526212908.GG75996@cotton.terantula.com> Hi, We are testing a Varnish Cache in our production environment with a 500Gb storage file and 32Gb of RAM. Varnish performance is excellent when all of the 32Gb is not allocated yet. The rates I am seeing here are around 40-60Mbit/s, with roughly 2.2M objects in cache and hitting a ratio of ~0.65, even then Varnish can handle it easily. However it is still warming up since we have a lot of objects that need to be cached. The problem I am facing is that as soon as RAM is exhausted Varnish restarts itself. Since this looked like an IO problem, we dropped ext2 in favour of xfs with much better results on writing to disk. However varnishd still stops working after it get to the 32G RAM limit. Note that I don't see any IO until just before it hits the 97% of RAM usage. So we thought to combine the file storage type with malloc and limit the amount of memory Varnish is allowed to allocate, first to 5G and see how that would work out. It turned out that it did not get limited and it seems from reading some posts this is not needed.. I have seen some posts on running large caches with the same kind but not a real approach to a solution. What is the best way to get around this issue ? Below are the init script and output of both varnisstat and top. Hitrate ratio: 3 3 3 Hitrate avg: 0.6008 0.6008 0.6008 10871 1.00 1.17 Client connections accepted 5278218 273.99 566.76 Client requests received 2864011 172.99 307.53 Cache hits 2413896 101.00 259.20 Cache misses 2413920 101.00 259.20 Backend connections success 2391749 99.00 256.82 Backend connections reuses 2391795 99.00 256.82 Backend connections recycles 148 . . N struct sess_mem 29 . . N struct sess 2366595 . . N struct object 2364206 . . N struct objecthead 4733079 . . N struct smf 0 . . N small free smf 1 . . N large free smf 10 . . N struct vbe_conn 96 . . N struct bereq 400 . . N worker threads 400 0.00 0.04 N worker threads created 2 . . N backends 47353 . . N expired objects 2090535 . . N LRU moved objects 5086915 265.99 546.22 Objects sent with write 10867 1.00 1.17 Total Sessions 5278227 273.99 566.76 Total Requests 12 0.00 0.00 Total pipe 13 0.00 0.00 Total pass 2413900 101.00 259.20 Total fetch 1865669893 97172.71 200329.64 Total header bytes 22763257823 1297006.09 2444245.44 Total body bytes 3335 0.00 0.36 Session Closed 5275957 273.99 566.52 Session herd 292178030 14367.51 31373.14 SHM records 7758036 382.99 833.03 SHM writes 6264 2.00 0.67 SHM flushes due to overflow 239 0.00 0.03 SHM MTX contention 125 0.00 0.01 SHM cycles through buffer 4828098 201.99 518.43 allocator requests 4733078 . . outstanding allocations 30790995968 . . bytes allocated 506079916032 . . bytes free 303 0.00 0.03 SMS allocator requests 130986 . . SMS bytes allocated 130986 . . SMS bytes freed 2413909 101.00 259.20 Backend requests made 1 0.00 0.00 N vcl total 1 0.00 0.00 N vcl available 1 . . N total active purges 1 0.00 0.00 N new purges added top - 15:13:40 up 7 days, 33 min, 2 users, load average: 0.14, 0.71, 0.75 Tasks: 116 total, 1 running, 115 sleeping, 0 stopped, 0 zombie Cpu0 : 3.0%us, 1.0%sy, 0.0%ni, 93.0%id, 1.7%wa, 0.0%hi, 1.3%si, 0.0%st Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu2 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu4 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu5 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu6 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu7 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 32942712k total, 32777060k used, 165652k free, 2164k buffers Swap: 506008k total, 25664k used, 480344k free, 29918680k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 26992 nobody 15 0 505g 30g 28g S 4 98.6 17:26.94 varnishd 28537 root 15 0 6628 1208 864 R 0 0.0 0:00.52 top 1 root 15 0 6120 556 500 S 0 0.0 0:08.61 init Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb 2.00 1589.00 14.00 37.00 1960.00 13008.00 293.49 0.16 3.06 0.71 3.60 We are running Varnish 2.0.4 on Linux 2.6. 64bit. Regards, Marco -- Terantula - Industrial Strength Open Source phone:+31 64 3232 400 / www: http://www.terantula.com / pgpkey: E7EE7A46 pgp fingerprint: F2EE 122D 964C DE68 7380 6F95 3710 7719 E7EE 7A46 From gaute at pht.no Wed May 27 08:31:41 2009 From: gaute at pht.no (Gaute Amundsen) Date: Wed, 27 May 2009 10:31:41 +0200 Subject: Disable caching at specific times? ( and a deeper mystery ) Message-ID: <200905271031.42147.gaute@pht.no> Hi We have this most peculiar problem where varnish somehow manages to cache a page with a Content-Type header with charset of utf-8 even though zope correctly answers iso-8859-1 for every combination of request headers I can think of, when accessed directly. The culprit must be some combination factors involving a spider, zope, apache and haproxy, we think, but we have not managed to reproduce the bug yet. While we are digging into this mystery there is one important customer that always seem to run into this problem while publishing articles in the early morning hours. What we need to do is simply return(pass) for this host outside of prime time. We do this manually now, but that is way to easy to forget. We could have zope handle it, but as normal ttl is 2 days, it would be much cleaner if it was possible to do in VCL. My VCLfu, is not quite at that level yet :) Any suggestions? Something involving the Client-Date header perhaps? And any suggestions on the deeper mystery are most welcome as well of course. Right now we are expanding logging, in the hope of catching the bug "in the wild" Regards Gaute Amundsen -- Programmerer - Pixelhospitalet AS Prinsessealleen 50, 0276 Oslo Tlf. 24 12 97 81 - 9074 7344 From kristian at redpill-linpro.com Wed May 27 08:31:30 2009 From: kristian at redpill-linpro.com (Kristian Lyngstol) Date: Wed, 27 May 2009 10:31:30 +0200 Subject: Varnish restarts when all memory is allocated In-Reply-To: <20090526212908.GG75996@cotton.terantula.com> References: <20090526212908.GG75996@cotton.terantula.com> Message-ID: <20090527083130.GJ5857@kjeks.linpro.no> On Tue, May 26, 2009 at 11:29:08PM +0200, Marco Walraven wrote: > We are testing a Varnish Cache in our production environment with a 500Gb storage file and > 32Gb of RAM. Varnish performance is excellent when all of the 32Gb is not allocated yet. > The rates I am seeing here are around 40-60Mbit/s, with roughly 2.2M objects in cache and > hitting a ratio of ~0.65, even then Varnish can handle it easily. However it is still > warming up since we have a lot of objects that need to be cached. > > The problem I am facing is that as soon as RAM is exhausted Varnish restarts itself. > Since this looked like an IO problem, we dropped ext2 in favour of xfs with much > better results on writing to disk. However varnishd still stops working after it get > to the 32G RAM limit. Note that I don't see any IO until just before it hits the 97% of > RAM usage. Can you post the arguments you use to start varnish? -- Kristian Lyngst?l Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From m.walraven at terantula.com Wed May 27 09:32:32 2009 From: m.walraven at terantula.com (Marco Walraven) Date: Wed, 27 May 2009 11:32:32 +0200 Subject: Varnish restarts when all memory is allocated In-Reply-To: <20090527083130.GJ5857@kjeks.linpro.no> References: <20090526212908.GG75996@cotton.terantula.com> <20090527083130.GJ5857@kjeks.linpro.no> Message-ID: <20090527093232.GI75996@cotton.terantula.com> On Wed, May 27, 2009 at 10:31:30AM +0200, Kristian Lyngstol wrote: > Can you post the arguments you use to start varnish? Sure; Varnish runs currently as follows: 5117 ? Ss 0:00 /usr/sbin/varnishd -P /var/run/varnishd.pid -a :80 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 3600 -w 400,4000,60 -s file,/data/varnish/mp-varnish001/varnish_storage.bin,500G -p obj_workspace 4096 -p sess_workspace 262144 -p lru_interval 60 -h classic,4550111 -p listen_depth 8192 -p log_hashstring off -p sess_timeout 10 -p shm_workspace 32768 -p ping_interval 1 -p thread_pools 4 -p thread_pool_min 100 -p thread_pool_max 4000 -p srcaddr_ttl 0 -p esi_syntax 1 5118 ? Sl 0:01 /usr/sbin/varnishd -P /var/run/varnishd.pid -a :80 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 3600 -w 400,4000,60 -s file,/data/varnish/mp-varnish001/varnish_storage.bin,500G -p obj_workspace 4096 -p sess_workspace 262144 -p lru_interval 60 -h classic,4550111 -p listen_depth 8192 -p log_hashstring off -p sess_timeout 10 -p shm_workspace 32768 -p ping_interval 1 -p thread_pools 4 -p thread_pool_min 100 -p thread_pool_max 4000 -p srcaddr_ttl 0 -p esi_syntax 1 I have to note that I have been running with a lru_interval of 3600 which had the same effect e.g. restarting when it hits the memory limit. Marco -- Terantula - Industrial Strength Open Source phone:+31 64 3232 400 / www: http://www.terantula.com / pgpkey: E7EE7A46 pgp fingerprint: F2EE 122D 964C DE68 7380 6F95 3710 7719 E7EE 7A46 From frank at anotheria.net Wed May 27 11:28:54 2009 From: frank at anotheria.net (frank) Date: Wed, 27 May 2009 13:28:54 +0200 Subject: Varnish restarts when all memory is allocated In-Reply-To: <20090526212908.GG75996@cotton.terantula.com> References: <20090526212908.GG75996@cotton.terantula.com> Message-ID: <1243423734.33805.16.camel@nero.internal.friendscout24.de> Hi Guys, On Tue, 2009-05-26 at 23:29 +0200, Marco Walraven wrote: > We are testing a Varnish Cache in our production environment with a 500Gb storage file and > 32Gb of RAM. Varnish performance is excellent when all of the 32Gb is not allocated yet. > The rates I am seeing here are around 40-60Mbit/s, with roughly 2.2M objects in cache and > hitting a ratio of ~0.65, even then Varnish can handle it easily. However it is still > warming up since we have a lot of objects that need to be cached. > > The problem I am facing is that as soon as RAM is exhausted Varnish restarts itself. > Since this looked like an IO problem, we dropped ext2 in favour of xfs with much > better results on writing to disk. However varnishd still stops working after it get > to the 32G RAM limit. Note that I don't see any IO until just before it hits the 97% of > RAM usage. > > So we thought to combine the file storage type with malloc and limit the amount of > memory Varnish is allowed to allocate, first to 5G and see how that would work out. > It turned out that it did not get limited and it seems from reading some posts > this is not needed.. > > I have seen some posts on running large caches with the same kind but not a real > approach to a solution. What is the best way to get around this issue ? I am facing exactly the same problem. Each time Varnish hits the RAM limit, it starts to drop elements from the cache. During this time the system is nearly unusable due to heavy disk io. After that Varnish keeps running but uses just a small part of the available memory. DAEMON_OPTS="-a :3129 \ -T localhost:6082 \ -f /etc/varnish/default.vcl \ -p thread_pool_max=1500 \ -p lru_interval=3600 \ -h classic,500009 \ -p thread_pools=4 \ -p listen_depth=4096 \ -s file,/opt/varnish/cache.bin1,26G" I tried malloc without any results. -s malloc,5G" If i run Varnish with the default setting it does not use the whole memory (8G out of 32G) A second instance which serves other documents runs fine with 1GB cache file and the default settings. Same hardware with debian 4.0 and 2.6.18-6-amd64 Am I missing something? Is there a rule of thumb for calculating the cache file / size? How does Varnish flush objects from the cache? Thanks in advance Frank From havardf at met.no Thu May 28 07:49:46 2009 From: havardf at met.no (=?utf-8?Q?H=C3=A5vard_Futs=C3=A6ter?=) Date: Thu, 28 May 2009 07:49:46 +0000 (UTC) Subject: 'Write error' messages in varnishlog In-Reply-To: <369107460.454911243496962184.JavaMail.root@imap1b> Message-ID: <1208353764.454971243496986821.JavaMail.root@imap1b> Hi. We are seeing quite a few of the following error messages in 'varnishlog -i Debug': 336 Debug c "Write error, len = 121440/185942, errno = Success" 344 Debug c "Write error, len = 75900/184867, errno = Success" 810 Debug - "Write error, len = -1/702, errno = Connection reset by peer" 807 Debug - "Write error, len = -1/702, errno = Connection reset by peer" 216 Debug c "Write error, len = 146280/147661, errno = Success" 843 Debug c "Write error, len = 144900/147661, errno = Success" 203 Debug c "Write error, len = 144900/147661, errno = Success" 836 Debug c "Write error, len = 144900/147661, errno = Success" 30 Debug c "Write error, len = 144900/147661, errno = Success" 324 Debug c "Write error, len = 144900/147661, errno = Success" 297 Debug c "Write error, len = 146280/147661, errno = Resource temporarily unavailable" What is the meaning of these messages? I see that they are generated by WRW_Flush in cache_pool.c. Perhaps they are caused by insufficient memory allocation to the shared memory log? MEMLOCK is set to the default 82000. We have increased shm_workspace quite a bit without any affect. By the way, what is the relation between MEMLOCK and the shm_workspace parameter? Is MEMLOCK the upper limit for shm_workspace? We use varnish 2.0.4 on debian/etch 2.6.18-6-amd64. Here are our current parameter settings: accept_fd_holdoff 50 [ms] acceptor default (epoll, poll) auto_restart on [bool] backend_http11 on [bool] between_bytes_timeout 60.000000 [s] cache_vbe_conns off [bool] cc_command "exec cc -fpic -shared -Wl,-x -o %o %s" cli_buffer 8192 [bytes] cli_timeout 15 [seconds] client_http11 off [bool] clock_skew 10 [s] connect_timeout 0.400000 [s] default_grace 10 default_ttl 5400 [seconds] diag_bitmap 0x0 [bitmap] err_ttl 0 [seconds] esi_syntax 0 [bitmap] fetch_chunksize 128 [kilobytes] first_byte_timeout 60.000000 [s] group nogroup (65534) listen_address 0.0.0.0:80 listen_depth 1024 [connections] log_hashstring off [bool] log_local_address off [bool] lru_interval 120 [seconds] max_esi_includes 5 [includes] max_restarts 4 [restarts] obj_workspace 8192 [bytes] overflow_max 100 [%] ping_interval 6 [seconds] pipe_timeout 60 [seconds] prefer_ipv6 off [bool] purge_dups on [bool] purge_hash off [bool] rush_exponent 3 [requests per request] send_timeout 120 [seconds] sess_timeout 5 [seconds] sess_workspace 16384 [bytes] session_linger 0 [ms] shm_reclen 255 [bytes] shm_workspace 79872 [bytes] srcaddr_hash 6133 [buckets] srcaddr_ttl 30 [seconds] thread_pool_add_delay 20 [milliseconds] thread_pool_add_threshold 2 [requests] thread_pool_fail_delay 200 [milliseconds] thread_pool_max 200 [threads] thread_pool_min 1 [threads] thread_pool_purge_delay 1000 [milliseconds] thread_pool_timeout 120 [seconds] thread_pools 2 [pools] user nobody (65534) vcl_trace off [bool] Best regards, H?vard Futs?ter From phk at phk.freebsd.dk Thu May 28 08:12:17 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 28 May 2009 08:12:17 +0000 Subject: 'Write error' messages in varnishlog In-Reply-To: Your message of "Thu, 28 May 2009 07:49:46 GMT." <1208353764.454971243496986821.JavaMail.root@imap1b> Message-ID: <75146.1243498337@critter.freebsd.dk> > 336 Debug c "Write error, len = 121440/185942, errno = Success" > 344 Debug c "Write error, len = 75900/184867, errno = Success" > 810 Debug - "Write error, len = -1/702, errno = Connection reset by peer" > 807 Debug - "Write error, len = -1/702, errno = Connection reset by peer" > 216 Debug c "Write error, len = 146280/147661, errno = Success" > 843 Debug c "Write error, len = 144900/147661, errno = Success" > 203 Debug c "Write error, len = 144900/147661, errno = Success" > 836 Debug c "Write error, len = 144900/147661, errno = Success" > 30 Debug c "Write error, len = 144900/147661, errno = Success" > 324 Debug c "Write error, len = 144900/147661, errno = Success" > 297 Debug c "Write error, len = 146280/147661, errno = Resource temporarily unavailable" This means that the clients close the TCP connection before receiving the entire contents. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at des.no Thu May 28 18:35:46 2009 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Thu, 28 May 2009 20:35:46 +0200 Subject: cache purging questions In-Reply-To: (Matthew Hoopes's message of "Thu, 28 May 2009 13:46:31 -0400") References: <86zlczir1n.fsf@ds4.des.no> Message-ID: <86d49tqdml.fsf@ds4.des.no> Matthew Hoopes writes: > Is it even possible to clear the cache based on hostname? Yes, but you need to match against the hash string, not the URL. I don't remember how to do that in 1.whateveritwasyouwererunning; you should really upgrade to 2.0.4. DES -- Dag-Erling Sm?rgrav - des at des.no From m.walraven at terantula.com Fri May 29 10:42:40 2009 From: m.walraven at terantula.com (Marco Walraven) Date: Fri, 29 May 2009 12:42:40 +0200 Subject: Varnish restarts when all memory is allocated In-Reply-To: <20090526212908.GG75996@cotton.terantula.com> References: <20090526212908.GG75996@cotton.terantula.com> Message-ID: <20090529104240.GD47535@cotton.terantula.com> On Tue, May 26, 2009 at 11:29:08PM +0200, Marco Walraven wrote: > Hi, > > We are testing a Varnish Cache in our production environment with a 500Gb storage file and > 32Gb of RAM. Varnish performance is excellent when all of the 32Gb is not allocated yet. > The rates I am seeing here are around 40-60Mbit/s, with roughly 2.2M objects in cache and > hitting a ratio of ~0.65, even then Varnish can handle it easily. However it is still > warming up since we have a lot of objects that need to be cached. > > The problem I am facing is that as soon as RAM is exhausted Varnish restarts itself. In the meantime I have been doing some tests tweaking the VM system under Linux, especially vm.min_free_kbytes, leaving some memory for pdflush and kswapd. The results are slightly better, but still varnishd starts to hog the CPU's and restarts. Alternatively we disabled swap, running without it. But also tested with a swap file of 16Gb on a different disk. Again slightly better results but still the same effect in the end. We also ran Varnish without the file storage type having just 8Gb assigned with malloc, this ran longer than the other tests we did. Varnishd did not crash but got extremely high CPU usages 700% and recoverd from that after a minute or 2. The linux system run with the following sysctl config applied: Linux varnish001 2.6.18-6-amd64 #1 SMP Tue May 5 08:01:28 UTC 2009 x86_64 GNU/Linux /etc/systctl.conf net.ipv4.ip_local_port_range = 1024 65536 net.core.rmem_max=16777216 net.core.wmem_max=16777216 net.ipv4.tcp_rmem=4096 87380 16777216 net.ipv4.tcp_wmem=4096 65536 16777216 net.ipv4.tcp_fin_timeout = 3 net.ipv4.tcp_tw_recycle = 1 net.core.netdev_max_backlog = 30000 net.ipv4.tcp_no_metrics_save=1 net.core.somaxconn = 262144 net.ipv4.tcp_syncookies = 0 net.ipv4.tcp_max_orphans = 262144 net.ipv4.tcp_max_syn_backlog = 262144 net.ipv4.tcp_synack_retries = 2 net.ipv4.tcp_syn_retries = 2 vm.swappiness = 0 vm.min_free_kbytes = 4194304 vm.dirty_background_ratio = 25 vm.dirty_expire_centisecs = 1000 vm.dirty_writeback_centisecs = 100 So, yesterday I installed FreeBSD 7.2 STABLE with the lastest CVSup on the second Varnish box and ran Varnish with the exact same config as on the Linux box. Exact same setup, 16 Gb swap file, same arguments for varnishd, same vcl, same amount of traffic, connections etc etc. I did apply perfmance tuning as described on the wiki. Both systems ran Ok until the moment there was little RAM left. Linux showed the exact same behaviour as before, high CPU load, varnishd with high amounts of CPU usages and in the end it varnishd restarted with an ampty cache. FreeBSD kept on going as I expected it to work; however with a higher load but still serving images at 60Mbit/s. I did see that it sometimes needed to recover. Meaning accepting no connections for a few moments and then starting to go on again, but enough to notice. Both systems run varnishd as followed, I changed the amount of buckets to 450011 as opposed to the previous tests I ran. Same for the lru_interval which was 60 and maybe too low. /usr/sbin/varnishd -P /var/run/varnishd.pid -a :80 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 3600 -w 400,4000,60 -s file,500G -p obj_workspace 8192 -p sess_workspace 262144 -p lru_interval 600 -h classic,450011 -p sess_timeout 2 -p listen_depth 8192 -p log_hashstring off -p shm_workspace 32768 -p ping_interval 10 -p srcaddr_ttl 0 -p esi_syntax 1 Below some output of Linux when it started to hog the CPU and output of the FreeBSD system 15 minutes later when it was still going. So is this kind of setup actually possible ? And if so how to get it running smoothly ? So far FreeBSD comes pretty close but not yet there. Thanks for the help, Marco Linux: Hitrate ratio: 10 100 1000 Hitrate avg: 0.6914 0.6843 0.6656 5739 0.00 0.60 Client connections accepted 5197130 0.00 542.72 Client requests received 2929389 0.00 305.91 Cache hits 0 0.00 0.00 Cache hits for pass 2267089 0.00 236.75 Cache misses 2267116 0.00 236.75 Backend connections success 0 0.00 0.00 Backend connections failures 2246281 0.00 234.57 Backend connections reuses 2246300 1.00 234.58 Backend connections recycles 120 . . N struct sess_mem 57 . . N struct sess 2224133 . . N struct object 2222930 . . N struct objecthead 4448208 . . N struct smf 0 . . N small free smf 0 . . N large free smf 33 . . N struct vbe_conn 114 . . N struct bereq 400 . . N worker threads 400 0.00 0.04 N worker threads created 1601 0.00 0.17 N worker threads limited 0 0.00 0.00 N queued work requests 0 0.00 0.00 N overflowed work requests 2 . . N backends 42991 . . N expired objects 1347513 . . N LRU moved objects 5081074 1.00 530.61 Objects sent with write 5725 0.00 0.60 Total Sessions 5197083 1.00 542.72 Total Requests 6 0.00 0.00 Total pipe 21 0.00 0.00 Total pass 2267086 1.00 236.75 Total fetch 1859166711 366.91 194148.57 Total header bytes 21933375988 802.81 2290452.80 Total body bytes 2363 1.00 0.25 Session Closed 5194871 0.00 542.49 Session herd 283299668 45.99 29584.34 SHM records 7496210 3.00 782.81 SHM writes 5065 0.00 0.53 SHM flushes due to overflow 142 0.00 0.01 SHM MTX contention 122 0.00 0.01 SHM cycles through buffer 4534821 0.00 473.56 allocator requests 4448207 . . outstanding allocations 37913182208 . . bytes allocated 498957729792 . . bytes free 605 0.00 0.06 SMS allocator requests 261420 . . SMS bytes allocated 261420 . . SMS bytes freed 2267110 0.00 236.75 Backend requests made 1 0.00 0.00 N vcl total 1 0.00 0.00 N vcl available 1 . . N total active purges 1 0.00 0.00 N new purges added top - 18:56:21 up 1 day, 1:41, 3 users, load average: 10.72, 2.88, 1.34 Tasks: 121 total, 5 running, 116 sleeping, 0 stopped, 0 zombie Cpu0 : 0.7%us, 87.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 12.3%si, 0.0%st Cpu1 : 0.3%us, 99.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st Cpu2 : 0.0%us,100.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu3 : 0.0%us,100.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu4 : 0.3%us, 99.7%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu5 : 0.0%us,100.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu6 : 0.0%us,100.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu7 : 0.0%us,100.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 32942712k total, 30712396k used, 2230316k free, 308k buffers Swap: 16777208k total, 10692k used, 16766516k free, 28230576k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 11740 nobody 18 0 505g 28g 26g S 753 91.8 17:36.41 varnishd 249 root 20 -5 0 0 0 R 46 0.0 13:32.07 kswapd0 12564 root 15 0 6628 1208 864 R 0 0.0 0:04.45 top 1 root 15 0 6120 80 48 S 0 0.0 0:14.35 init 2 root RT 0 0 0 0 S 0 0.0 0:00.01 migration/0 3 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0 FreeBSD: last pid: 12964; load averages: 2.11, 1.63, 0.94 up 0+04:15:40 17:09:45 487 processes: 9 running, 462 sleeping, 16 waiting CPU 0: 2.0% user, 0.0% nice, 25.5% system, 0.7% interrupt, 71.8% idle CPU 1: 0.7% user, 0.0% nice, 27.5% system, 0.0% interrupt, 71.8% idle CPU 2: 1.3% user, 0.0% nice, 65.1% system, 0.0% interrupt, 33.6% idle CPU 3: 1.3% user, 0.0% nice, 36.9% system, 0.0% interrupt, 61.7% idle CPU 4: 0.7% user, 0.0% nice, 12.8% system, 0.7% interrupt, 85.8% idle CPU 5: 0.0% user, 0.0% nice, 0.7% system, 0.0% interrupt, 99.3% idle CPU 6: 0.0% user, 0.0% nice, 2.7% system, 0.0% interrupt, 97.3% idle CPU 7: 0.7% user, 0.0% nice, 0.7% system, 0.0% interrupt, 98.6% idle Mem: 27G Active, 2865M Inact, 727M Wired, 884M Cache, 399M Buf, 120K Free Swap: 16G Total, 1476K Used, 16G Free PID UID THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 12 0 1 171 ki31 0K 16K CPU6 6 252:38 89.99% idle: cpu6 11 0 1 171 ki31 0K 16K CPU7 7 252:34 88.57% idle: cpu7 14 0 1 171 ki31 0K 16K CPU4 4 252:21 88.18% idle: cpu4 13 0 1 171 ki31 0K 16K CPU5 5 252:30 83.06% idle: cpu5 45 0 1 -16 - 0K 16K psleep 1 5:18 81.88% pagedaemon 15 0 1 171 ki31 0K 16K CPU3 3 250:19 75.88% idle: cpu3 18 0 1 171 ki31 0K 16K CPU0 0 242:32 73.00% idle: cpu0 16 0 1 171 ki31 0K 16K RUN 2 245:26 70.90% idle: cpu2 17 0 1 171 ki31 0K 16K CPU1 1 243:08 60.89% idle: cpu1 12033 80 406 45 0 502G 24339M ucond 4 0:08 13.96% varnishd Hitrate ratio: 10 100 1000 Hitrate avg: 0.6868 0.6891 0.6749 15694 2.00 1.51 Client connections accepted 5611010 651.39 540.72 Client requests received 3211214 451.89 309.45 Cache hits 1 0.00 0.00 Cache hits for pass 2399137 198.51 231.20 Cache misses 2399143 198.51 231.20 Backend connections success 2377367 198.51 229.10 Backend connections reuses 2377408 197.51 229.10 Backend connections recycles 0 . . N struct srcaddr 0 . . N active struct srcaddr 316 . . N struct sess_mem 48 . . N struct sess 2349612 . . N struct object 2348460 . . N struct objecthead 4699173 . . N struct smf 0 . . N small free smf 0 . . N large free smf 40 . . N struct vbe_conn 115 . . N struct bereq 400 . . N worker threads 400 0.00 0.04 N worker threads created 0 0.00 0.00 N worker threads limited 0 0.00 0.00 N queued work requests 0 0.00 0.00 N overflowed work requests 2 . . N backends 49564 . . N expired objects 1516192 . . N LRU moved objects 5486698 643.41 528.74 Objects sent with write 15693 1.00 1.51 Total Sessions 5611037 648.40 540.72 Total Requests 8 0.00 0.00 Total pipe 20 0.00 0.00 Total pass 2399127 198.51 231.20 Total fetch 2007812574 234091.74 193486.80 Total header bytes 23594716103 2781420.12 2273751.19 Total body bytes 2576 0.00 0.25 Session Closed 5608555 647.40 540.48 Session herd 304052678 32276.41 29300.63 SHM records 8074243 868.86 778.09 SHM writes 6122 2.99 0.59 SHM flushes due to overflow 11697 5.99 1.13 SHM MTX contention 131 0.00 0.01 SHM cycles through buffer 4798894 396.02 462.45 allocator requests 4699172 . . outstanding allocations 20 0.00 0.00 Total pass 2399127 198.51 231.20 Total fetch 2007812574 234091.74 193486.80 Total header bytes 23594716103 2781420.12 2273751.19 Total body bytes 2576 0.00 0.25 Session Closed 5608555 647.40 540.48 Session herd 304052678 32276.41 29300.63 SHM records 8074243 868.86 778.09 SHM writes 6122 2.99 0.59 SHM flushes due to overflow 11697 5.99 1.13 SHM MTX contention 131 0.00 0.01 SHM cycles through buffer 4798894 396.02 462.45 allocator requests 4699172 . . outstanding allocations 40178368512 . . bytes allocated 496692543488 . . bytes free 575 0.00 0.06 SMS allocator requests 0 . . SMS outstanding allocations 249035 . . SMS bytes allocated 249035 . . SMS bytes freed 2399128 198.51 231.20 Backend requests made 1 0.00 0.00 N vcl total 1 0.00 0.00 N vcl available 1 . . N total active purges 1 0.00 0.00 N new purges added -- Terantula - Industrial Strength Open Source phone:+31 64 3232 400 / www: http://www.terantula.com / pgpkey: E7EE7A46 pgp fingerprint: F2EE 122D 964C DE68 7380 6F95 3710 7719 E7EE 7A46 apt-get install freebsd From michael at dynamine.net Fri May 29 15:17:44 2009 From: michael at dynamine.net (Michael S. Fischer) Date: Fri, 29 May 2009 08:17:44 -0700 Subject: Varnish restarts when all memory is allocated In-Reply-To: <20090529104240.GD47535@cotton.terantula.com> References: <20090526212908.GG75996@cotton.terantula.com> <20090529104240.GD47535@cotton.terantula.com> Message-ID: <1120DC76-6CF3-4ACF-AE0D-BDF68D4408CB@dynamine.net> I think the lesson of these cases is pretty clear: make your cacheable working set fits into the proxy server's available memory -- or, if you want to exceed your available memory, make sure your hit ratio is sufficiently high that the cache server rarely resorts to paging in the data. Otherwise, your cache server will suffer I/O starvation due to excessive paging. This is a general working principle of cache configuration, and not specific to Varnish. After all, the whole point of a cache is to serve cacheable objects quickly. Disk is not fast, and Varnish doesn't make slow disks faster. This goal will become much easier if you can put a "layer 7 switch" in front of a pool of Varnish servers and route HTTP requests to them based on some attribute of the request (e.g., the URI and/or Cookie: header, thereby ensuring efficient use of your cache pool's memory. Best regards, --Michael On May 29, 2009, at 3:42 AM, Marco Walraven wrote: > On Tue, May 26, 2009 at 11:29:08PM +0200, Marco Walraven wrote: >> Hi, >> >> We are testing a Varnish Cache in our production environment with a >> 500Gb storage file and >> 32Gb of RAM. Varnish performance is excellent when all of the 32Gb >> is not allocated yet. >> The rates I am seeing here are around 40-60Mbit/s, with roughly >> 2.2M objects in cache and >> hitting a ratio of ~0.65, even then Varnish can handle it easily. >> However it is still >> warming up since we have a lot of objects that need to be cached. >> >> The problem I am facing is that as soon as RAM is exhausted Varnish >> restarts itself. > > In the meantime I have been doing some tests tweaking the VM system > under Linux, especially > vm.min_free_kbytes, leaving some memory for pdflush and kswapd. The > results are slightly better, > but still varnishd starts to hog the CPU's and restarts. > Alternatively we disabled swap, running > without it. But also tested with a swap file of 16Gb on a different > disk. Again slightly better > results but still the same effect in the end. > > We also ran Varnish without the file storage type having just 8Gb > assigned with malloc, this ran > longer than the other tests we did. Varnishd did not crash but got > extremely high CPU usages 700% > and recoverd from that after a minute or 2. > > The linux system run with the following sysctl config applied: > > Linux varnish001 2.6.18-6-amd64 #1 SMP Tue May 5 08:01:28 UTC 2009 > x86_64 GNU/Linux > > /etc/systctl.conf > > net.ipv4.ip_local_port_range = 1024 65536 > net.core.rmem_max=16777216 > net.core.wmem_max=16777216 > net.ipv4.tcp_rmem=4096 87380 16777216 > net.ipv4.tcp_wmem=4096 65536 16777216 > net.ipv4.tcp_fin_timeout = 3 > net.ipv4.tcp_tw_recycle = 1 > net.core.netdev_max_backlog = 30000 > net.ipv4.tcp_no_metrics_save=1 > net.core.somaxconn = 262144 > net.ipv4.tcp_syncookies = 0 > net.ipv4.tcp_max_orphans = 262144 > net.ipv4.tcp_max_syn_backlog = 262144 > net.ipv4.tcp_synack_retries = 2 > net.ipv4.tcp_syn_retries = 2 > vm.swappiness = 0 > vm.min_free_kbytes = 4194304 > vm.dirty_background_ratio = 25 > vm.dirty_expire_centisecs = 1000 > vm.dirty_writeback_centisecs = 100 > > > So, yesterday I installed FreeBSD 7.2 STABLE with the lastest CVSup > on the second Varnish box and > ran Varnish with the exact same config as on the Linux box. Exact > same setup, 16 Gb swap file, same > arguments for varnishd, same vcl, same amount of traffic, > connections etc etc. I did apply > perfmance tuning as described on the wiki. Both systems ran Ok until > the moment there was little > RAM left. Linux showed the exact same behaviour as before, high CPU > load, varnishd with high amounts > of CPU usages and in the end it varnishd restarted with an ampty > cache. FreeBSD kept on going as I > expected it to work; however with a higher load but still serving > images at 60Mbit/s. I did see that > it sometimes needed to recover. Meaning accepting no connections for > a few moments and then starting > to go on again, but enough to notice. > > Both systems run varnishd as followed, I changed the amount of > buckets to 450011 as opposed to the previous tests I ran. Same for > the lru_interval which was 60 and maybe too low. > > /usr/sbin/varnishd -P /var/run/varnishd.pid -a :80 -f /etc/varnish/ > default.vcl -T 127.0.0.1:6082 -t 3600 -w 400,4000,60 -s file,500G -p > obj_workspace 8192 -p sess_workspace 262144 -p lru_interval 600 -h > classic,450011 -p sess_timeout 2 -p listen_depth 8192 -p > log_hashstring off -p shm_workspace 32768 -p ping_interval 10 -p > srcaddr_ttl 0 -p esi_syntax 1 > > Below some output of Linux when it started to hog the CPU and output > of the FreeBSD system 15 minutes > later when it was still going. > > So is this kind of setup actually possible ? And if so how to get it > running smoothly ? So far FreeBSD > comes pretty close but not yet there. > > Thanks for the help, > > Marco > > > > Linux: > > Hitrate ratio: 10 100 1000 > Hitrate avg: 0.6914 0.6843 0.6656 > > 5739 0.00 0.60 Client connections accepted > 5197130 0.00 542.72 Client requests received > 2929389 0.00 305.91 Cache hits > 0 0.00 0.00 Cache hits for pass > 2267089 0.00 236.75 Cache misses > 2267116 0.00 236.75 Backend connections success > 0 0.00 0.00 Backend connections failures > 2246281 0.00 234.57 Backend connections reuses > 2246300 1.00 234.58 Backend connections recycles > 120 . . N struct sess_mem > 57 . . N struct sess > 2224133 . . N struct object > 2222930 . . N struct objecthead > 4448208 . . N struct smf > 0 . . N small free smf > 0 . . N large free smf > 33 . . N struct vbe_conn > 114 . . N struct bereq > 400 . . N worker threads > 400 0.00 0.04 N worker threads created > 1601 0.00 0.17 N worker threads limited > 0 0.00 0.00 N queued work requests > 0 0.00 0.00 N overflowed work requests > 2 . . N backends > 42991 . . N expired objects > 1347513 . . N LRU moved objects > 5081074 1.00 530.61 Objects sent with write > 5725 0.00 0.60 Total Sessions > 5197083 1.00 542.72 Total Requests > 6 0.00 0.00 Total pipe > 21 0.00 0.00 Total pass > 2267086 1.00 236.75 Total fetch > 1859166711 366.91 194148.57 Total header bytes > 21933375988 802.81 2290452.80 Total body bytes > 2363 1.00 0.25 Session Closed > 5194871 0.00 542.49 Session herd > 283299668 45.99 29584.34 SHM records > 7496210 3.00 782.81 SHM writes > 5065 0.00 0.53 SHM flushes due to overflow > 142 0.00 0.01 SHM MTX contention > 122 0.00 0.01 SHM cycles through buffer > 4534821 0.00 473.56 allocator requests > 4448207 . . outstanding allocations > 37913182208 . . bytes allocated > 498957729792 . . bytes free > 605 0.00 0.06 SMS allocator requests > 261420 . . SMS bytes allocated > 261420 . . SMS bytes freed > 2267110 0.00 236.75 Backend requests made > 1 0.00 0.00 N vcl total > 1 0.00 0.00 N vcl available > 1 . . N total active purges > 1 0.00 0.00 N new purges added > > top - 18:56:21 up 1 day, 1:41, 3 users, load average: 10.72, > 2.88, 1.34 > Tasks: 121 total, 5 running, 116 sleeping, 0 stopped, 0 zombie > Cpu0 : 0.7%us, 87.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, > 12.3%si, 0.0%st > Cpu1 : 0.3%us, 99.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, > 0.3%si, 0.0%st > Cpu2 : 0.0%us,100.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, > 0.0%si, 0.0%st > Cpu3 : 0.0%us,100.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, > 0.0%si, 0.0%st > Cpu4 : 0.3%us, 99.7%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, > 0.0%si, 0.0%st > Cpu5 : 0.0%us,100.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, > 0.0%si, 0.0%st > Cpu6 : 0.0%us,100.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, > 0.0%si, 0.0%st > Cpu7 : 0.0%us,100.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, > 0.0%si, 0.0%st > Mem: 32942712k total, 30712396k used, 2230316k free, 308k > buffers > Swap: 16777208k total, 10692k used, 16766516k free, 28230576k > cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 11740 nobody 18 0 505g 28g 26g S 753 91.8 17:36.41 varnishd > 249 root 20 -5 0 0 0 R 46 0.0 13:32.07 kswapd0 > 12564 root 15 0 6628 1208 864 R 0 0.0 0:04.45 top > 1 root 15 0 6120 80 48 S 0 0.0 0:14.35 init > 2 root RT 0 0 0 0 S 0 0.0 0:00.01 > migration/0 > 3 root 34 19 0 0 0 S 0 0.0 0:00.00 > ksoftirqd/0 > > FreeBSD: > > last pid: 12964; load averages: 2.11, 1.63, > 0.94 up 0+04:15:40 17:09:45 > 487 processes: 9 running, 462 sleeping, 16 waiting > CPU 0: 2.0% user, 0.0% nice, 25.5% system, 0.7% interrupt, 71.8% > idle > CPU 1: 0.7% user, 0.0% nice, 27.5% system, 0.0% interrupt, 71.8% > idle > CPU 2: 1.3% user, 0.0% nice, 65.1% system, 0.0% interrupt, 33.6% > idle > CPU 3: 1.3% user, 0.0% nice, 36.9% system, 0.0% interrupt, 61.7% > idle > CPU 4: 0.7% user, 0.0% nice, 12.8% system, 0.7% interrupt, 85.8% > idle > CPU 5: 0.0% user, 0.0% nice, 0.7% system, 0.0% interrupt, 99.3% > idle > CPU 6: 0.0% user, 0.0% nice, 2.7% system, 0.0% interrupt, 97.3% > idle > CPU 7: 0.7% user, 0.0% nice, 0.7% system, 0.0% interrupt, 98.6% > idle > Mem: 27G Active, 2865M Inact, 727M Wired, 884M Cache, 399M Buf, 120K > Free > Swap: 16G Total, 1476K Used, 16G Free > > PID UID THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > 12 0 1 171 ki31 0K 16K CPU6 6 252:38 89.99% > idle: cpu6 > 11 0 1 171 ki31 0K 16K CPU7 7 252:34 88.57% > idle: cpu7 > 14 0 1 171 ki31 0K 16K CPU4 4 252:21 88.18% > idle: cpu4 > 13 0 1 171 ki31 0K 16K CPU5 5 252:30 83.06% > idle: cpu5 > 45 0 1 -16 - 0K 16K psleep 1 5:18 81.88% > pagedaemon > 15 0 1 171 ki31 0K 16K CPU3 3 250:19 75.88% > idle: cpu3 > 18 0 1 171 ki31 0K 16K CPU0 0 242:32 73.00% > idle: cpu0 > 16 0 1 171 ki31 0K 16K RUN 2 245:26 70.90% > idle: cpu2 > 17 0 1 171 ki31 0K 16K CPU1 1 243:08 60.89% > idle: cpu1 > 12033 80 406 45 0 502G 24339M ucond 4 0:08 13.96% > varnishd > > Hitrate ratio: 10 100 1000 > Hitrate avg: 0.6868 0.6891 0.6749 > > 15694 2.00 1.51 Client connections accepted > 5611010 651.39 540.72 Client requests received > 3211214 451.89 309.45 Cache hits > 1 0.00 0.00 Cache hits for pass > 2399137 198.51 231.20 Cache misses > 2399143 198.51 231.20 Backend connections success > 2377367 198.51 229.10 Backend connections reuses > 2377408 197.51 229.10 Backend connections recycles > 0 . . N struct srcaddr > 0 . . N active struct srcaddr > 316 . . N struct sess_mem > 48 . . N struct sess > 2349612 . . N struct object > 2348460 . . N struct objecthead > 4699173 . . N struct smf > 0 . . N small free smf > 0 . . N large free smf > 40 . . N struct vbe_conn > 115 . . N struct bereq > 400 . . N worker threads > 400 0.00 0.04 N worker threads created > 0 0.00 0.00 N worker threads limited > 0 0.00 0.00 N queued work requests > 0 0.00 0.00 N overflowed work requests > 2 . . N backends > 49564 . . N expired objects > 1516192 . . N LRU moved objects > 5486698 643.41 528.74 Objects sent with write > 15693 1.00 1.51 Total Sessions > 5611037 648.40 540.72 Total Requests > 8 0.00 0.00 Total pipe > 20 0.00 0.00 Total pass > 2399127 198.51 231.20 Total fetch > 2007812574 234091.74 193486.80 Total header bytes > 23594716103 2781420.12 2273751.19 Total body bytes > 2576 0.00 0.25 Session Closed > 5608555 647.40 540.48 Session herd > 304052678 32276.41 29300.63 SHM records > 8074243 868.86 778.09 SHM writes > 6122 2.99 0.59 SHM flushes due to overflow > 11697 5.99 1.13 SHM MTX contention > 131 0.00 0.01 SHM cycles through buffer > 4798894 396.02 462.45 allocator requests > 4699172 . . outstanding allocations > 20 0.00 0.00 Total pass > 2399127 198.51 231.20 Total fetch > 2007812574 234091.74 193486.80 Total header bytes > 23594716103 2781420.12 2273751.19 Total body bytes > 2576 0.00 0.25 Session Closed > 5608555 647.40 540.48 Session herd > 304052678 32276.41 29300.63 SHM records > 8074243 868.86 778.09 SHM writes > 6122 2.99 0.59 SHM flushes due to overflow > 11697 5.99 1.13 SHM MTX contention > 131 0.00 0.01 SHM cycles through buffer > 4798894 396.02 462.45 allocator requests > 4699172 . . outstanding allocations > 40178368512 . . bytes allocated > 496692543488 . . bytes free > 575 0.00 0.06 SMS allocator requests > 0 . . SMS outstanding allocations > 249035 . . SMS bytes allocated > 249035 . . SMS bytes freed > 2399128 198.51 231.20 Backend requests made > 1 0.00 0.00 N vcl total > 1 0.00 0.00 N vcl available > 1 . . N total active purges > 1 0.00 0.00 N new purges added > > -- > Terantula - Industrial Strength Open Source > phone:+31 64 3232 400 / www: http://www.terantula.com / pgpkey: > E7EE7A46 > pgp fingerprint: F2EE 122D 964C DE68 7380 6F95 3710 7719 E7EE 7A46 > apt-get install freebsd > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc From matthew.hoopes at gmail.com Thu May 28 17:46:31 2009 From: matthew.hoopes at gmail.com (Matthew Hoopes) Date: Thu, 28 May 2009 13:46:31 -0400 Subject: cache purging questions In-Reply-To: <86zlczir1n.fsf@ds4.des.no> References: <86zlczir1n.fsf@ds4.des.no> Message-ID: 2009/5/26 Dag-Erling Sm?rgrav > Matthew Hoopes writes: > > url.purge *username.domain.com* > > This is not a valid regular expression. > > DES > -- > Dag-Erling Sm?rgrav - des at des.no Hey, thanks for the response. In the docs, it shows how to purge the index doc and everything in the cache. http://varnish.projects.linpro.no/wiki/FAQ#HowcanIforcearefreshonaobjectcachedbyvarnish I've tried a whole gang of regular expressions involving backslashes and .* everywhere, but I get either "Syntax Error: Illegal backslash sequence" or it just doesn't clear the cache of the objects i'm trying to clear. Is it even possible to clear the cache based on hostname? If someone could show me an example of how to clear the cache of every object from a domain (if possible) i'd be very grateful. thanks again, matt hoopes -------------- next part -------------- An HTML attachment was scrubbed... URL: