From mobiledreamers at gmail.com Mon Mar 26 20:13:18 2007 From: mobiledreamers at gmail.com (mobiledreamers at gmail.com) Date: Mon, 26 Mar 2007 13:13:18 -0700 Subject: Regarding varnish pdf and presentation Message-ID: Read the attached PDF for a presentation and some stats. Im trying to find this pdf The reason I say so is because Varnish uses some "state of the art" features in the OS (Even if they have been around a long time :) Like you say, let the OS do the stuff it knows how to do best.). Right now I use Varnish in a prod. environment with mixed content static/dynamic, and it works amazing. Read the attached PDF for a presentation and some stats. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kristoffer.gleditsch at met.no Tue Mar 27 12:27:19 2007 From: kristoffer.gleditsch at met.no (Kristoffer Gleditsch) Date: Tue, 27 Mar 2007 14:27:19 +0200 Subject: Instability when using management CLI Message-ID: <46090DA7.8070404@met.no> Hi! We're currently testing Varnish in front of a new content generation system. This system doesn't know exactly how long the pages it generates will be valid, but it is important to purge them from the cache once new data is available. In order to do this, we're connecting to the CLI on the administration port, and issuing url.purge commands. Once the purge commands started pouring in, we found some stability issues; Within a few minutes, Varnish would segfault in strange places, or in some cases just keep using 100% CPU without any interaction with anyone. ElectricFence gave me a segfault on this line: http://varnish.projects.linpro.no/browser/tags/varnish-1.0.3/bin/varnishd/mgt_cli.c#L117 (The file doesn't seem to have changed between 1.0.3 and trunk.) I'm not sure I understand every aspect of this function, but I'm unable to see where the space for that extra newline is allocated, and re-compiling version 1.0.3 with the attached patch seems to have stabilized it. It has now been processing some thousands of purge requests for almost 24 hours under ElectricFence without incident. I also tried running it through Valgrind. It complained about lost memory in several places. Would these Valgrind logs be useful to anyone, or should I just try patching Varnish myself instead? :) Regards, Kristoffer. -------------- next part -------------- A non-text attachment was scrubbed... Name: varnish-malloc.patch Type: text/x-patch Size: 407 bytes Desc: not available URL: From des at linpro.no Wed Mar 28 06:44:04 2007 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Wed, 28 Mar 2007 08:44:04 +0200 Subject: Regarding varnish pdf and presentation In-Reply-To: (mobiledreamers@gmail.com's message of "Mon, 26 Mar 2007 13:13:18 -0700") References: Message-ID: mobiledreamers at gmail.com writes: > Read the attached PDF > for a presentation and some stats. > Im trying to find this pdf > > The reason I say so is because Varnish uses some "state > of the art" features in the OS (Even if they have been around a long > time :) Like you say, let the OS do the stuff it knows how to do > best.). Right now I use Varnish in a prod. environment with mixed > content static/dynamic, and it works amazing. Read the attached PDF > for a presentation and some stats. I'm not sure what you are trying to tell us. Are you looking for a PDF about Varnish, or are you trying to send us one? Mailman strips most attachments from incoming mail (with good reason), so if you are trying to send us something, you'd be better off posting it on the web and sending us the URL. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Wed Mar 28 07:35:55 2007 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Wed, 28 Mar 2007 09:35:55 +0200 Subject: Instability when using management CLI In-Reply-To: <46090DA7.8070404@met.no> (Kristoffer Gleditsch's message of "Tue, 27 Mar 2007 14:27:19 +0200") References: <46090DA7.8070404@met.no> Message-ID: Kristoffer Gleditsch writes: > I'm not sure I understand every aspect of this function, but I'm unable > to see where the space for that extra newline is allocated, and > re-compiling version 1.0.3 with the attached patch seems to have > stabilized it. Good catch, but there are a couple of snags. First of all, I think the final newline is meant to replace the trailing space, i.e. ["a" "b" "c"\n] rather than ["a" "b" "c" \n]. However, the for loop escapes double quotes, newlines and backslashes, so the string may grow well beyond the strlen() + 3 set aside for each argument. I'll send you an updated patch later today. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Wed Mar 28 08:00:40 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 28 Mar 2007 08:00:40 +0000 Subject: Instability when using management CLI In-Reply-To: Your message of "Tue, 27 Mar 2007 14:27:19 +0200." <46090DA7.8070404@met.no> Message-ID: <1328.1175068840@critter.freebsd.dk> In message <46090DA7.8070404 at met.no>, Kristoffer Gleditsch writes: >I also tried running it through Valgrind. It complained about lost >memory in several places. Would these Valgrind logs be useful to >anyone, or should I just try patching Varnish myself instead? :) I still havn't gotten Valgrind to work on FreeBSD, so logfiles are most welcome. I'm not sure your patch is enough, I'll look at it in a moment. Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From kristoffer.gleditsch at met.no Thu Mar 29 11:57:08 2007 From: kristoffer.gleditsch at met.no (Kristoffer Gleditsch) Date: Thu, 29 Mar 2007 13:57:08 +0200 Subject: Instability when using management CLI In-Reply-To: <1328.1175068840@critter.freebsd.dk> References: <1328.1175068840@critter.freebsd.dk> Message-ID: <460BA994.1040407@met.no> Poul-Henning Kamp wrote: > I still havn't gotten Valgrind to work on FreeBSD, so logfiles > are most welcome. I've put a tarball of Valgrind logfiles here: http://users.linpro.no/toffer/valgrind-logs.tar.bz2 This is the output after starting valgrind/varnish like this: valgrind -v --leak-check=full --trace-children=yes --num-callers=30 --time-stamp=yes --log-file=/tmp/valgrind-5.varnish /usr/local/varnish/sbin/varnishd -a 0.0.0.0:80 -h classic -f /etc/varnish/vcl.conf -T 0.0.0.0:81 -t 60 -w 1,1,60 -s file,/var/lib/varnish/varnish_storage.bin,10240000 ...and then killing the parent process after 20-ish hours. The Varnish in question was version 1.0.3 from SVN with the patch I got from DES applied. It seems the follow-children followed the compilation of the VCL file as well; I guess valgrind-5.varnish.27584 is the most interesting file. I hope this is useful. I don't have a lot of experience with Valgrind, so if there are any options I should (or shouldn't) have used, please let me know. Regards, Kristoffer. From phk at phk.freebsd.dk Fri Mar 30 19:06:01 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 30 Mar 2007 19:06:01 +0000 Subject: Instability when using management CLI In-Reply-To: Your message of "Thu, 29 Mar 2007 13:57:08 +0200." <460BA994.1040407@met.no> Message-ID: <2520.1175281561@critter.freebsd.dk> In message <460BA994.1040407 at met.no>, Kristoffer Gleditsch writes: >Poul-Henning Kamp wrote: >I've put a tarball of Valgrind logfiles here: >http://users.linpro.no/toffer/valgrind-logs.tar.bz2 >It seems the follow-children followed the compilation of the VCL file as >well; I guess valgrind-5.varnish.27584 is the most interesting file. Thanks a LOT! Yes, the VCL-compiler has some minor leaks I belive there already is a ticket about it. So far the valgrind hasn't found any I didn't know about, but it has found two I forgot I knew about :-) Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From mrz at mozilla.com Fri Mar 30 23:18:04 2007 From: mrz at mozilla.com (matthew zeier) Date: Fri, 30 Mar 2007 16:18:04 -0700 Subject: Project sponsorship ? Message-ID: <460D9AAC.6080409@mozilla.com> Apologies in advance - I couldn't find any other address to send this to. Mozilla's struggling with our current hardware based load balancing solution and Varnish looks like a really great alternative, especially 2.0 (at least from the feature list). Who can I talk to about becoming a sponsor for this project? -- matthew zeier | Network Engineer | Mozilla Corp. | (650)903-0800 x219 From phk at phk.freebsd.dk Sat Mar 31 06:01:45 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 31 Mar 2007 06:01:45 +0000 Subject: Project sponsorship ? In-Reply-To: Your message of "Fri, 30 Mar 2007 16:18:04 MST." <460D9AAC.6080409@mozilla.com> Message-ID: <36818.1175320905@critter.freebsd.dk> In message <460D9AAC.6080409 at mozilla.com>, matthew zeier writes: > >Apologies in advance - I couldn't find any other address to send this to. > >Mozilla's struggling with our current hardware based load balancing solution >and Varnish looks like a really great alternative, especially 2.0 (at least >from the feature list). > >Who can I talk to about becoming a sponsor for this project? You need to get hold of Per Anders Buer and Anders Berg -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence.