From Phil.Cryer at edwardjones.com Thu Nov 1 15:39:58 2007 From: Phil.Cryer at edwardjones.com (Cryer,Phil) Date: Thu, 1 Nov 2007 10:39:58 -0500 Subject: C Compiler requirement Message-ID: <3ECD7F7DDE95BA4FA598E8DDE71F1A5104CACEF8@nwpsrv08.edj.ad.edwardjones.com> So as I understand it, when Varnish runs, it reads in the VCL script (basically the configuration file that I have written) and generates C source from that code, that is then compiled into a shared object to be called by the accelerator. Is there any way Varnish can compile that beforehand, and then have that Shared Object used on the next launch instead of having it rebuilt again? I'm still going around with having a C compiler on Production boxes, and am waiting for a decision from security, but I was wondering about this. Once configured we're not going to change the VCL ever, so if we could have it built once, and then use that, we wouldn't need a C complier on the Prod box. I may be way off, but just an idea I wanted to discuss, as it would make using Varnish easier for enterprises that don't understand Varnish's philosophy. Thanks P If you are not the intended recipient of this message (including attachments), or if you have received this message in error, immediately notify us and delete it and any attachments. If you no longer wish to receive e-mail from Edward Jones, please send this request to messages at edwardjones.com. You must include the e-mail address that you wish not to receive e-mail communications. For important additional information related to this e-mail, visit www.edwardjones.com/US_email_disclosure -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Fri Nov 2 08:58:28 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 02 Nov 2007 08:58:28 +0000 Subject: C Compiler requirement In-Reply-To: Your message of "Thu, 01 Nov 2007 10:39:58 EST." <3ECD7F7DDE95BA4FA598E8DDE71F1A5104CACEF8@nwpsrv08.edj.ad.edwardjones.com> Message-ID: <1498.1193993908@critter.freebsd.dk> In message <3ECD7F7DDE95BA4FA598E8DDE71F1A5104CACEF8 at nwpsrv08.edj.ad.edwardjone s.com>, "Cryer,Phil" writes: >Is there any way Varnish can compile that >beforehand, and then have that Shared Object used on the next launch >instead of having it rebuilt again? It's not inherently impossible, but does raise som concerns about version skew between the compiled VCL and the varnishd binary. It shouldn't be too much work to implement I think. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dwetzel at nerim.net Fri Nov 2 15:22:55 2007 From: dwetzel at nerim.net (Damien Wetzel) Date: Fri, 2 Nov 2007 16:22:55 +0100 Subject: hiding varnish headers from requests Message-ID: <18219.16591.185911.307821@localhost.localdomain> Hello, I wondered if there is a way to prevent Varnish from puting headers like : X-varnish Via: 1.1 varnish in its responses to clients. thanks, -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Damien WETZEL (ATANAR TECHNOLOGIES) ("`-/")_.-'"``-._ http://www.atanar.com . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' Phone:+33 6 62 29 61 77 _.- _..-_/ / ((.' - So much to do, so little time - ((,.-' ((,/ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From dwetzel at nerim.net Mon Nov 5 18:48:32 2007 From: dwetzel at nerim.net (Damien Wetzel) Date: Mon, 5 Nov 2007 19:48:32 +0100 Subject: how to test varnish with port 8080 Message-ID: <18223.25984.330820.354253@localhost.localdomain> Hello, I'm trying varnish on a 64 bit machine on which the port 80 is already active, so i use the port 8080 to talk to varnish. I wondered if someone knew a way to tell firefox to use port 8080 by default to avoid me the pain of adding :8080 on all my requests. Many thanks for any ideas :) Damien, -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Damien WETZEL (ATANAR TECHNOLOGIES) ("`-/")_.-'"``-._ http://www.atanar.com . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' Phone:+33 6 62 29 61 77 _.- _..-_/ / ((.' - So much to do, so little time - ((,.-' ((,/ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From Phil.Cryer at edwardjones.com Mon Nov 5 19:43:47 2007 From: Phil.Cryer at edwardjones.com (Cryer,Phil) Date: Mon, 5 Nov 2007 13:43:47 -0600 Subject: C Compiler requirement In-Reply-To: <1498.1193993908@critter.freebsd.dk> References: Your message of "Thu, 01 Nov 2007 10:39:58 EST." <3ECD7F7DDE95BA4FA598E8DDE71F1A5104CACEF8@nwpsrv08.edj.ad.edwardjones.com> <1498.1193993908@critter.freebsd.dk> Message-ID: <3ECD7F7DDE95BA4FA598E8DDE71F1A5104E14D3C@nwpsrv08.edj.ad.edwardjones.com> >>Is there any way Varnish can compile that beforehand, and then have >>that Shared Object used on the next launch instead of having it rebuilt >>again? > It's not inherently impossible, but does raise som concerns about version skew between > the compiled VCL and the varnishd binary. > It shouldn't be too much work to implement I think. So where is the shared object that Varnish creates on startup? Secondly, is there a DAEMON_OPTS variable I can set to make Varnish use an existing shared object instead of regen'ing a new one on startup? I suspect that last one would be the only functionality missing currently. True, the version issue will be there, but again, I'm thinking of a way to make enterprise implementation easier, and companies are usually pretty strict on new versions. With the proper upgrade path they should know to update the .so and move it in with the install, otherwise a version check on startup could do the trick (make sure 1.1.1 is the version called out in the vcl for example). Thanks Phil If you are not the intended recipient of this message (including attachments), or if you have received this message in error, immediately notify us and delete it and any attachments. If you no longer wish to receive e-mail from Edward Jones, please send this request to messages at edwardjones.com. You must include the e-mail address that you wish not to receive e-mail communications. For important additional information related to this e-mail, visit www.edwardjones.com/US_email_disclosure From Phil.Cryer at edwardjones.com Mon Nov 5 20:07:24 2007 From: Phil.Cryer at edwardjones.com (Cryer,Phil) Date: Mon, 5 Nov 2007 14:07:24 -0600 Subject: C Compiler requirement In-Reply-To: <3ECD7F7DDE95BA4FA598E8DDE71F1A5104E14D3C@nwpsrv08.edj.ad.edwardjones.com> References: Your message of "Thu, 01 Nov 2007 10:39:58 EST."<3ECD7F7DDE95BA4FA598E8DDE71F1A5104CACEF8@nwpsrv08.edj.ad.edwardjones.com><1498.1193993908@critter.freebsd.dk> <3ECD7F7DDE95BA4FA598E8DDE71F1A5104E14D3C@nwpsrv08.edj.ad.edwardjones.com> Message-ID: <3ECD7F7DDE95BA4FA598E8DDE71F1A5104E14E3F@nwpsrv08.edj.ad.edwardjones.com> If you are not the intended recipient of this message (including attachments), or if you have received this message in error, immediately notify us and delete it and any attachments. If you no longer wish to receive e-mail from Edward Jones, please send this request to messages at edwardjones.com. You must include the e-mail address that you wish not to receive e-mail communications. For important additional information related to this e-mail, visit www.edwardjones.com/US_email_disclosure -----Original Message----- From: varnish-misc-bounces at projects.linpro.no [mailto:varnish-misc-bounces at projects.linpro.no] On Behalf Of Cryer,Phil Sent: Monday, November 05, 2007 1:44 PM To: varnish-misc at projects.linpro.no Subject: RE: C Compiler requirement >>Is there any way Varnish can compile that beforehand, and then have >>that Shared Object used on the next launch instead of having it rebuilt >>again? > It's not inherently impossible, but does raise som concerns about version skew between > the compiled VCL and the varnishd binary. > It shouldn't be too much work to implement I think. Replying to myself: || So where is the shared object that Varnish creates on startup? bash-3.00# ls -l /var/lib/varnish/`hostname` total 48 -rwxr-xr-x 1 root root 19800 Nov 5 14:03 bin.XXpiVDV1 -rw------- 1 root root 22836 Nov 5 14:03 _.c -rw-r--r-- 1 root root 8389176 Nov 5 14:03 _.vsl bash-3.00# file * bin.XXpiVDV1: ELF 64-bit LSB shared object, AMD x86-64, version 1 (SYSV), not stripped _.c: ASCII C program text _.vsl: data So I know where the current shared obj is, now I just need to figure out (or have implemented): ||[...] is there a DAEMON_OPTS variable I can set to make Varnish use an existing shared ||object instead of regen'ing a new one on startup? I suspect that last one would be the only ||functionality missing currently. Thanks Phil From ric at digitalmarbles.com Mon Nov 5 22:04:36 2007 From: ric at digitalmarbles.com (Ricardo Newbery) Date: Mon, 5 Nov 2007 14:04:36 -0800 Subject: how to test varnish with port 8080 In-Reply-To: <18223.25984.330820.354253@localhost.localdomain> References: <18223.25984.330820.354253@localhost.localdomain> Message-ID: <4BA17317-15CF-49AA-AF52-77395B03D0B4@digitalmarbles.com> On Nov 5, 2007, at 10:48 AM, Damien Wetzel wrote: > Hello, > I'm trying varnish on a 64 bit machine on which the port 80 is already > active, so i use the port 8080 to talk to varnish. > I wondered if someone knew a way to tell firefox to use port 8080 by > default to avoid me the pain of adding :8080 on all my requests. > Many thanks for any ideas :) > Damien, > -- Use a redirect, proxypass, or rewriterule in the apache config (assuming apache is the one hogging port 80) to send selected requests to port 8080. Ric From duja at torlen.net Tue Nov 6 19:55:32 2007 From: duja at torlen.net (Erik Torlen) Date: Tue, 06 Nov 2007 19:55:32 -0000 Subject: varnishstat vs. telnet stats References: <480b36850709200734m508fdfc0pfbced7537370b95f@mail.gmail.com> Message-ID: <000001c820ad$af475da0$1401010a@Apica.local> Ok. Thanks for all your help mate ;) Erik ----- Original Message ----- From: "Peter Oomen" To: "Erik" Sent: Thursday, September 20, 2007 4:34 PM Subject: Re: varnishstat vs. telnet stats > yes, cache hits mean hits from the cache.. > > > On 9/20/07, Erik wrote: >> Thank you very much. It is up n running ;) >> >> A few question about the cache tho. >> >> The stats are saying that it has fetched like 40 items and have had about >> 650 cache hits. Is this enough to tell if the cache is running, I mean, >> "cache hits"? that must mean that its serving from the cache? >> >> // Erik >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at projects.linpro.no >> http://projects.linpro.no/mailman/listinfo/varnish-misc >> > > > -- > No virus found in this incoming message. > Checked by AVG Free Edition. > Version: 7.5.487 / Virus Database: 269.13.25/1018 - Release Date: > 2007-09-19 15:59 > > From gslin at gslin.org Wed Nov 7 06:16:02 2007 From: gslin at gslin.org (Gea-Suan Lin) Date: Wed, 7 Nov 2007 14:16:02 +0800 Subject: Varnish crash (SIGABRT) about every 10 mins Message-ID: <20071107061602.GA79568@gslin.org> Our server will crash (SIGABRT, signal 6) every 10 mins (approximately). We tried to use "-d" to see error log, but the server hangs and won't listen port 80. It's running FreeBSD 6.2-STABLE (amd64) with ports/www/varnisa. (varnish-1.1.1_2) dmesg and sysctl shows: real memory = 5100273664 (4864 MB) avail memory = 4126154752 (3935 MB) vm.kmem_size_scale: 3 vm.kmem_size_max: 419430400 vm.kmem_size: 419430400 hw.physmem: 4286906368 hw.usermem: 3878264832 hw.realmem: 5100273664 But the real memory is 4096MB not 4864MB, it might be the problem, but I cannot make sure about this. And, we modified some variables in sysctl: kern.ipc.maxsockets: 262144 kern.maxfiles: 1048576 kern.maxfilesperproc: 524288 kern.threads.max_groups_per_proc: 8192 kern.threads.max_threads_per_proc: 8192 The running command is: sudo /usr/bin/env -i /usr/local/sbin/varnishd -a 60.199.247.x:80 -f /usr/local/etc/varnish/image.vcl -h classic,1048583 -P /var/run/varnishd.pid -s file,/home/service/varnish-cache.mmap,32G -T 127.0.0.1:11957 -t 7200 -w 4096 And image.vcl is: # # haproxy - default backend default { set backend.host = "127.0.0.1"; set backend.port = "8080"; } # vip - quota backend vip { set backend.host = "10.1.2.3"; set backend.port = "80"; } sub vcl_recv { if (!req.http.Referer || req.http.Referer ~ "^https?://.*\.pixnet\.net(/|$)") { lookup; } set req.backend = vip; pipe; } -- * Gea-Suan Lin (public key: Using https://keyserver.pgp.com/ to search) * If you cannot convince them, confuse them. -- Harry S Truman From gslin at gslin.org Wed Nov 7 06:30:29 2007 From: gslin at gslin.org (Gea-Suan Lin) Date: Wed, 7 Nov 2007 14:30:29 +0800 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: <20071107061602.GA79568@gslin.org> References: <20071107061602.GA79568@gslin.org> Message-ID: <20071107063029.GA61307@gslin.org> On Wed, Nov 07, 2007 at 02:16:02PM +0800, Gea-Suan Lin wrote: > The running command is: > > sudo /usr/bin/env -i /usr/local/sbin/varnishd -a 60.199.247.x:80 -f /usr/local/etc/varnish/image.vcl -h classic,1048583 -P /var/run/varnishd.pid -s file,/home/service/varnish-cache.mmap,32G -T 127.0.0.1:11957 -t 7200 -w 4096 btw, I've tried another command (just without -t 7200, to use default value 120): sudo /usr/bin/env -i /usr/local/sbin/varnishd -a 60.199.247.x:80 -f /usr/local/etc/varnish/image.vcl -h classic,1048583 -P /var/run/varnishd.pid -s file,/home/service/varnish-cache.mmap,32G -T 127.0.0.1:11957 -w 4096 And this one works fine. (but lower hitrate) -- * Gea-Suan Lin (public key: Using https://keyserver.pgp.com/ to search) * If you cannot convince them, confuse them. -- Harry S Truman From des at linpro.no Wed Nov 7 10:52:32 2007 From: des at linpro.no (=?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?=) Date: Wed, 07 Nov 2007 11:52:32 +0100 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: <20071107061602.GA79568@gslin.org> (Gea-Suan Lin's message of "Wed, 7 Nov 2007 14:16:02 +0800") References: <20071107061602.GA79568@gslin.org> Message-ID: Gea-Suan Lin writes: > Our server will crash (SIGABRT, signal 6) every 10 mins (approximately). > We tried to use "-d" to see error log, but the server hangs and won't > listen port 80. When running in debug mode, you need to type 'start' to start accepting connections. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From Phil.Cryer at edwardjones.com Wed Nov 7 19:07:13 2007 From: Phil.Cryer at edwardjones.com (Cryer,Phil) Date: Wed, 7 Nov 2007 13:07:13 -0600 Subject: Allow Varnish to reuse its shared object Message-ID: <3ECD7F7DDE95BA4FA598E8DDE71F1A5104EDEAEB@nwpsrv08.edj.ad.edwardjones.com> Currently Varnish requires a C compiler to be present on the machine it's running on, since it needs to compile the VCL config file into a shared object each time it starts. During shutdown, Varnish removes this shared object since it will be rebuilt during the next start. This routine repeats regardless of if anything has changed in the VCL config file, and serves as a road bump to getting Varnish into certain production environments since traditionally development applications (such as the C compiler) are not allowed in such instances. For now I am putting aside the arguments as to why it's is acceptable to have development applications in production instances, since that argument's outcome will vary in different situations, and I am aiming for a solution that will cover all instances. My proposal is to allow Varnish to reuse its shared object, instead creating it during on launch, and destroying it on shutdown. The reuse function would be called if a flag was present during startup, either on the command line as -r, or as defined in the DAEMON_OPTS variable. This would cause Varnish to do two things differently than it does now: 1) upon startup it would not create a new shared object, rather it would search for, and use, the first shared object it found in the following location: /var/lib/varnish/`hostname`/bin.* 2) when called to shutdown, Varnish would not remove the existing shared object, thus leaving it to be reused, if called again with the -r flag The shared object would need to be created (and recreated after any changes to the VCL file) on a clone machine that has a C compiler, and then installed to the production machine during a change window, which would be no different from a normal config file change. The upside of this would be that the C compiler requirement would be conditional upon the use of the reuse flag, which would allow smoother integration of Varnish into production environments. Where in the code should look for the startup and shutdown hooks so I can start investigating this further? If you are not the intended recipient of this message (including attachments), or if you have received this message in error, immediately notify us and delete it and any attachments. If you no longer wish to receive e-mail from Edward Jones, please send this request to messages at edwardjones.com. You must include the e-mail address that you wish not to receive e-mail communications. For important additional information related to this e-mail, visit www.edwardjones.com/US_email_disclosure From ask at develooper.com Thu Nov 8 06:51:26 2007 From: ask at develooper.com (=?ISO-8859-1?Q?Ask_Bj=F8rn_Hansen?=) Date: Wed, 7 Nov 2007 22:51:26 -0800 Subject: Varnish - ReProxy In-Reply-To: <11146.1193643751@critter.freebsd.dk> References: <11146.1193643751@critter.freebsd.dk> Message-ID: On Oct 29, 2007, at 0:42, Poul-Henning Kamp wrote: >> Can Varnish ReProxing like perlbal? Or is there any reproxy module >> for >> varnish ? > > It would be trivial to implement in VCL, but I'm not usre I see the > point in bothering an "expensive" backend, just to learn from it that > a cheaper one should be asked instead. I would direct the requests > to the cheap server right away in VCL. The "expensive" backend might be doing access checks for example ("did you really buy this content you are asking for"). - ask -- http://develooper.com/ - http://askask.com/ From gslin at gslin.org Thu Nov 8 11:17:19 2007 From: gslin at gslin.org (Gea-Suan Lin) Date: Thu, 8 Nov 2007 19:17:19 +0800 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: References: <20071107061602.GA79568@gslin.org> Message-ID: <20071108111719.GA63127@gslin.org> Hello, This is "-t 600" case: # sudo /usr/bin/env -i /usr/local/sbin/varnishd -a [our_ip_address]:80 -f /usr/local/etc/varnish/image.vcl -h classic,1048583 -P /var/run/varnishd.pid -s file,/home/service/varnish-cache.mmap,32G -T 127.0.0.1:11957 -t 600 -w 4096 -d -d file /home/service/varnish-cache.mmap size 34359738368 bytes (16777216 fs-blocks, 8388608 pages) Classic hash: 1048583 buckets Using old SHMFILE rolling(1)... rolling(2)... start start child pid 9802 200 0 Child said (2, 9802): <> Child said (2, 9802): <> Child said (2, 9802): <> Child said (2, 9802): <obj) == 0) not true. errno = 0 (Unknown error: 0) >> Cache child died pid=9802 status=0x6 Clean child Child cleaned start child pid 10394 Child said (2, 10394): <> Child said (2, 10394): <> Child said (2, 10394): <obj) == 0) not true. errno = 0 (Unknown error: 0) >> Child not responding to ping Cache child died pid=10394 status=0x6 Clean child Child cleaned start child pid 11575 Child said (2, 11575): <> Child said (2, 11575): <> Child said (2, 11575): <> Child not responding to ping Child not responding to ping Child not responding to ping Cache child died pid=11575 status=0x9 Clean child Child cleaned start child pid 11576 Pushing vcls failed: CLI communication error Child said (1, 11576): <> unlink ./bin.ffNEQwzp gslin at testphp [~] (18:54) On Wed, Nov 07, 2007 at 11:52:32AM +0100, Dag-Erling Sm?grav wrote: > Gea-Suan Lin writes: > > Our server will crash (SIGABRT, signal 6) every 10 mins (approximately). > > We tried to use "-d" to see error log, but the server hangs and won't > > listen port 80. > > When running in debug mode, you need to type 'start' to start > accepting connections. > > DES > -- > Dag-Erling Sm?grav > Senior Software Developer > Linpro AS - www.linpro.no -- * Gea-Suan Lin (public key: Using https://keyserver.pgp.com/ to search) * If you cannot convince them, confuse them. -- Harry S Truman From des at linpro.no Thu Nov 8 11:19:59 2007 From: des at linpro.no (=?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?=) Date: Thu, 08 Nov 2007 12:19:59 +0100 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: <20071108111719.GA63127@gslin.org> (Gea-Suan Lin's message of "Thu, 8 Nov 2007 19:17:19 +0800") References: <20071107061602.GA79568@gslin.org> <20071108111719.GA63127@gslin.org> Message-ID: Gea-Suan Lin writes: > Child said (2, 9802): < Condition((sp->obj) == 0) not true. > errno = 0 (Unknown error: 0) See ticket #162, we're working on it. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From tb-lists at delc.dk Thu Nov 8 11:46:57 2007 From: tb-lists at delc.dk (Thilo Bangert) Date: Thu, 8 Nov 2007 12:46:57 +0100 Subject: Allow Varnish to reuse its shared object In-Reply-To: <3ECD7F7DDE95BA4FA598E8DDE71F1A5104EDEAEB@nwpsrv08.edj.ad.edwardjones.com> References: <3ECD7F7DDE95BA4FA598E8DDE71F1A5104EDEAEB@nwpsrv08.edj.ad.edwardjones.com> Message-ID: <200711081246.57707.tb-lists@delc.dk> Hi Phil, maybe this is the part were you direct some of your enterprise cash in the direction of the developers... it is my understanding, that both PHK and Linpro are for hire with regards to specific feature reguest for varnish. apart from that your proposal sounds reasonable. i would probably grep through the source to find the main() function as a next step. ;) kind regards Thilo From gslin at gslin.org Thu Nov 8 12:55:26 2007 From: gslin at gslin.org (Gea-Suan Lin) Date: Thu, 8 Nov 2007 20:55:26 +0800 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: References: <20071107061602.GA79568@gslin.org> <20071108111719.GA63127@gslin.org> Message-ID: <20071108125526.GA31193@gslin.org> On Thu, Nov 08, 2007 at 12:19:59PM +0100, Dag-Erling Sm?grav wrote: > Gea-Suan Lin writes: > > Child said (2, 9802): < > Condition((sp->obj) == 0) not true. > > errno = 0 (Unknown error: 0) > > See ticket #162, we're working on it. Okay, so I'll upgrade to trunk and try again. -- * Gea-Suan Lin (public key: Using https://keyserver.pgp.com/ to search) * If you cannot convince them, confuse them. -- Harry S Truman From des at linpro.no Thu Nov 8 12:57:36 2007 From: des at linpro.no (=?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?=) Date: Thu, 08 Nov 2007 13:57:36 +0100 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: <20071108125526.GA31193@gslin.org> (Gea-Suan Lin's message of "Thu, 8 Nov 2007 20:55:26 +0800") References: <20071107061602.GA79568@gslin.org> <20071108111719.GA63127@gslin.org> <20071108125526.GA31193@gslin.org> Message-ID: Gea-Suan Lin writes: > On Thu, Nov 08, 2007 at 12:19:59PM +0100, Dag-Erling Sm?rgrav wrote: > > Gea-Suan Lin writes: > > > Child said (2, 9802): < > > Condition((sp->obj) == 0) not true. > > > errno = 0 (Unknown error: 0) > > See ticket #162, we're working on it. > Okay, so I'll upgrade to trunk and try again. We haven't fixed it yet, so upgrading to trunk won't help. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From gslin at gslin.org Thu Nov 8 16:35:03 2007 From: gslin at gslin.org (Gea-Suan Lin) Date: Fri, 9 Nov 2007 00:35:03 +0800 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: References: <20071107061602.GA79568@gslin.org> <20071108111719.GA63127@gslin.org> <20071108125526.GA31193@gslin.org> Message-ID: <20071108163503.GA77445@gslin.org> On Thu, Nov 08, 2007 at 01:57:36PM +0100, Dag-Erling Sm?grav wrote: > Gea-Suan Lin writes: > > On Thu, Nov 08, 2007 at 12:19:59PM +0100, Dag-Erling Sm???rgrav wrote: > > > Gea-Suan Lin writes: > > > > Child said (2, 9802): < > > > Condition((sp->obj) == 0) not true. > > > > errno = 0 (Unknown error: 0) > > > See ticket #162, we're working on it. > > Okay, so I'll upgrade to trunk and try again. > > We haven't fixed it yet, so upgrading to trunk won't help. Is that ok to running 32 varnishd and use HAProxy to redirect to them ? We found they will run out of memory. -- * Gea-Suan Lin (public key: Using https://keyserver.pgp.com/ to search) * If you cannot convince them, confuse them. -- Harry S Truman From des at linpro.no Thu Nov 8 16:39:45 2007 From: des at linpro.no (=?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?=) Date: Thu, 08 Nov 2007 17:39:45 +0100 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: <20071108163503.GA77445@gslin.org> (Gea-Suan Lin's message of "Fri, 9 Nov 2007 00:35:03 +0800") References: <20071107061602.GA79568@gslin.org> <20071108111719.GA63127@gslin.org> <20071108125526.GA31193@gslin.org> <20071108163503.GA77445@gslin.org> Message-ID: Gea-Suan Lin writes: > Is that ok to running 32 varnishd and use HAProxy to redirect to them ? 32 Varnish instances? Sounds like a lot. What kind of site is it? > We found they will run out of memory. Hmm, you must be running 1.1; try switching to branches/1.2. It shouldn't run out of memory. Here's a patch I think should help with the SES_Delete() issue, BTW. I will commit it (or something similar) once I've confirmed that it actually fixes the bug. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: cache_acceptor_kqueue.diff URL: From gslin at gslin.org Thu Nov 8 17:30:46 2007 From: gslin at gslin.org (Gea-Suan Lin) Date: Fri, 9 Nov 2007 01:30:46 +0800 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: References: <20071107061602.GA79568@gslin.org> <20071108111719.GA63127@gslin.org> <20071108125526.GA31193@gslin.org> <20071108163503.GA77445@gslin.org> Message-ID: <20071108173046.GA73399@gslin.org> On Thu, Nov 08, 2007 at 05:39:45PM +0100, Dag-Erling Sm?grav wrote: > Gea-Suan Lin writes: > > Is that ok to running 32 varnishd and use HAProxy to redirect to them ? > > 32 Varnish instances? Sounds like a lot. What kind of site is it? We're trying to serve images since the backend's I/O is not fast enough, so we need cache server to cache them. The original idea is use Varnish with "-s file,filename,100GB" to cache them, but varnish abort in SES_Delete. So the alternative is to run lots of varnishd and use HAProxy to dispatch the request. > > We found they will run out of memory. > > Hmm, you must be running 1.1; try switching to branches/1.2. It > shouldn't run out of memory. > > Here's a patch I think should help with the SES_Delete() issue, BTW. > I will commit it (or something similar) once I've confirmed that it > actually fixes the bug. Yes we run varnish-1.1 (install from ports), I'll try 1.2 with your patch. Thank you in advance. -- * Gea-Suan Lin (public key: Using https://keyserver.pgp.com/ to search) * If you cannot convince them, confuse them. -- Harry S Truman From gslin at gslin.org Thu Nov 8 17:49:58 2007 From: gslin at gslin.org (Gea-Suan Lin) Date: Fri, 9 Nov 2007 01:49:58 +0800 Subject: Varnish 1.1.1 FreeBSD 6.2 In-Reply-To: References: <20070822003343.2a0bc543@mx-1.bksnetworks.com> Message-ID: <20071108174958.GA16750@gslin.org> On Wed, Aug 22, 2007 at 09:32:22AM +0200, Dag-Erling Sm?grav wrote: > jdouglas writes: > > Making all in doc > > --xinclude -o changes-1.0.4.html changes-1.0.4.xml > > --xinclude:No such file or directory > > *** Error code 1 > > You're obviously not building from the release tarball... > > You need xsltproc (from textproc/libxslt) to generate the change logs. Can you update http://varnish.projects.linpro.no/wiki/Installation this page about this ? -- * Gea-Suan Lin (public key: Using https://keyserver.pgp.com/ to search) * If you cannot convince them, confuse them. -- Harry S Truman From gslin at gslin.org Thu Nov 8 18:45:40 2007 From: gslin at gslin.org (Gea-Suan Lin) Date: Fri, 9 Nov 2007 02:45:40 +0800 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: References: <20071107061602.GA79568@gslin.org> <20071108111719.GA63127@gslin.org> <20071108125526.GA31193@gslin.org> <20071108163503.GA77445@gslin.org> Message-ID: <20071108184540.GA71491@gslin.org> On Thu, Nov 08, 2007 at 05:39:45PM +0100, Dag-Erling Sm?grav wrote: > Gea-Suan Lin writes: > > We found they will run out of memory. > > Hmm, you must be running 1.1; try switching to branches/1.2. It > shouldn't run out of memory. > > Here's a patch I think should help with the SES_Delete() issue, BTW. > I will commit it (or something similar) once I've confirmed that it > actually fixes the bug. I checkout the latest version in trunk (r2231), apply the patch, and find no object being cached. (telnet to management port and use "stats" to check) Now I run it without the patch and the server just crash and no information to debug: Child not responding to ping Child not responding to ping Child not responding to ping Cache child died pid=49696 status=0x9 Clean child Child cleaned start child pid 49747 Child said (2, 49747): <> -- * Gea-Suan Lin (public key: Using https://keyserver.pgp.com/ to search) * If you cannot convince them, confuse them. -- Harry S Truman From des at linpro.no Thu Nov 8 19:40:23 2007 From: des at linpro.no (=?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?=) Date: Thu, 08 Nov 2007 20:40:23 +0100 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: <20071108184540.GA71491@gslin.org> (Gea-Suan Lin's message of "Fri, 9 Nov 2007 02:45:40 +0800") References: <20071107061602.GA79568@gslin.org> <20071108111719.GA63127@gslin.org> <20071108125526.GA31193@gslin.org> <20071108163503.GA77445@gslin.org> <20071108184540.GA71491@gslin.org> Message-ID: Gea-Suan Lin writes: > I checkout the latest version in trunk (r2231), apply the patch, and > find no object being cached. (telnet to management port and use "stats" > to check) I'm not psychic, you'll have to actually *show me* the stats. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From gslin at gslin.org Thu Nov 8 19:44:57 2007 From: gslin at gslin.org (Gea-Suan Lin) Date: Fri, 9 Nov 2007 03:44:57 +0800 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: References: <20071107061602.GA79568@gslin.org> <20071108111719.GA63127@gslin.org> <20071108125526.GA31193@gslin.org> <20071108163503.GA77445@gslin.org> <20071108184540.GA71491@gslin.org> Message-ID: <20071108194457.GA78196@gslin.org> On Thu, Nov 08, 2007 at 08:40:23PM +0100, Dag-Erling Sm?grav wrote: > Gea-Suan Lin writes: > > I checkout the latest version in trunk (r2231), apply the patch, and > > find no object being cached. (telnet to management port and use "stats" > > to check) > > I'm not psychic, you'll have to actually *show me* the stats. When I want to try it again, I found it might be my fault :-) http://varnish.projects.linpro.no/attachment/ticket/162/cache_acceptor_kqueue.diff It should be patched to: AZ(kevent(kq, ki, nki, NULL, 0, NULL)); But I patched to: AZ(kevent(kq, ki, nki, NULL, 0, NULL) <= 0); I'll try it again. -- * Gea-Suan Lin (public key: Using https://keyserver.pgp.com/ to search) * If you cannot convince them, confuse them. -- Harry S Truman From des at linpro.no Thu Nov 8 20:15:15 2007 From: des at linpro.no (=?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?=) Date: Thu, 08 Nov 2007 21:15:15 +0100 Subject: Varnish 1.1.1 FreeBSD 6.2 In-Reply-To: <20071108174958.GA16750@gslin.org> (Gea-Suan Lin's message of "Fri, 9 Nov 2007 01:49:58 +0800") References: <20070822003343.2a0bc543@mx-1.bksnetworks.com> <20071108174958.GA16750@gslin.org> Message-ID: Gea-Suan Lin writes: > On Wed, Aug 22, 2007 at 09:32:22AM +0200, Dag-Erling Sm?rgrav wrote: > > You need xsltproc (from textproc/libxslt) to generate the change logs. > Can you update http://varnish.projects.linpro.no/wiki/Installation this > page about this ? Done. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Thu Nov 8 20:16:22 2007 From: des at linpro.no (=?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?=) Date: Thu, 08 Nov 2007 21:16:22 +0100 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: <20071108194457.GA78196@gslin.org> (Gea-Suan Lin's message of "Fri, 9 Nov 2007 03:44:57 +0800") References: <20071107061602.GA79568@gslin.org> <20071108111719.GA63127@gslin.org> <20071108125526.GA31193@gslin.org> <20071108163503.GA77445@gslin.org> <20071108184540.GA71491@gslin.org> <20071108194457.GA78196@gslin.org> Message-ID: Gea-Suan Lin writes: > When I want to try it again, I found it might be my fault :-) > > http://varnish.projects.linpro.no/attachment/ticket/162/cache_acceptor_kqueue.diff > > It should be patched to: > > AZ(kevent(kq, ki, nki, NULL, 0, NULL)); > > But I patched to: > > AZ(kevent(kq, ki, nki, NULL, 0, NULL) <= 0); > > I'll try it again. The correct way to apply a patch is with /usr/bin/patch. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From gslin at gslin.org Thu Nov 8 20:25:19 2007 From: gslin at gslin.org (Gea-Suan Lin) Date: Fri, 9 Nov 2007 04:25:19 +0800 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: <20071108194457.GA78196@gslin.org> References: <20071108111719.GA63127@gslin.org> <20071108125526.GA31193@gslin.org> <20071108163503.GA77445@gslin.org> <20071108184540.GA71491@gslin.org> <20071108194457.GA78196@gslin.org> Message-ID: <20071108202519.GA40203@gslin.org> On Fri, Nov 09, 2007 at 03:44:57AM +0800, Gea-Suan Lin wrote: > On Thu, Nov 08, 2007 at 08:40:23PM +0100, Dag-Erling Sm?grav wrote: > > Gea-Suan Lin writes: > > > I checkout the latest version in trunk (r2231), apply the patch, and > > > find no object being cached. (telnet to management port and use "stats" > > > to check) > > > > I'm not psychic, you'll have to actually *show me* the stats. > > When I want to try it again, I found it might be my fault :-) > > http://varnish.projects.linpro.no/attachment/ticket/162/cache_acceptor_kqueue.diff > > It should be patched to: > > AZ(kevent(kq, ki, nki, NULL, 0, NULL)); > > But I patched to: > > AZ(kevent(kq, ki, nki, NULL, 0, NULL) <= 0); > > I'll try it again. It crash again: gslin at testphp [~] (3:45) sudo /usr/bin/env -i /home/service/varnish/sbin/varnishd -a [our ip address]:80 -f /usr/local/etc/varnish/image.vcl -h classic,1048583 -P /var/run/varnishd.pid -s file,/home/service/varnish-cache.mmap,32G -T 127.0.0.1:11957 -t 604800 -w 32,4096 -d -d storage_file: filename: /home/service/varnish-cache.mmap size 32768 MegaBytes. Classic hash: 1048583 buckets Using old SHMFILE rolling(1)... rolling(2)... start start child pid 57305 200 0 Child said (2, 57305): <> Child not responding to ping Child not responding to ping Child not responding to ping Cache child died pid=57305 status=0x9 Clean child Child cleaned start child pid 58543 Child said (2, 58543): <> Child not responding to ping Child not responding to ping Cache child died pid=58543 status=0x9 Clean child Child cleaned start child pid 58544 Child said (2, 58544): <> -- * Gea-Suan Lin (public key: Using https://keyserver.pgp.com/ to search) * If you cannot convince them, confuse them. -- Harry S Truman From gslin at gslin.org Thu Nov 8 20:27:22 2007 From: gslin at gslin.org (Gea-Suan Lin) Date: Fri, 9 Nov 2007 04:27:22 +0800 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: References: <20071108111719.GA63127@gslin.org> <20071108125526.GA31193@gslin.org> <20071108163503.GA77445@gslin.org> <20071108184540.GA71491@gslin.org> <20071108194457.GA78196@gslin.org> Message-ID: <20071108202722.GA71548@gslin.org> On Thu, Nov 08, 2007 at 09:16:22PM +0100, Dag-Erling Sm?grav wrote: > The correct way to apply a patch is with /usr/bin/patch. Yes I do patch using system's patch this time. -- * Gea-Suan Lin (public key: Using https://keyserver.pgp.com/ to search) * If you cannot convince them, confuse them. -- Harry S Truman From des at linpro.no Thu Nov 8 21:21:22 2007 From: des at linpro.no (=?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?=) Date: Thu, 08 Nov 2007 22:21:22 +0100 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: <20071108202519.GA40203@gslin.org> (Gea-Suan Lin's message of "Fri, 9 Nov 2007 04:25:19 +0800") References: <20071108111719.GA63127@gslin.org> <20071108125526.GA31193@gslin.org> <20071108163503.GA77445@gslin.org> <20071108184540.GA71491@gslin.org> <20071108194457.GA78196@gslin.org> <20071108202519.GA40203@gslin.org> Message-ID: Gea-Suan Lin writes: > It crash again: Actually, it doesn't crash; the child get stuck somewhere and the parent kills it. I'd like to see an 'svn diff' of your tree. Also, if you change SIGKILL in bin/varnishd/mgt_child.c to either SIGABRT or SIGQUIT, you should get a core dump of the child process when the parent kills it. If you don't, 'ulimit -c unlimited' and try again. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From gslin at gslin.org Thu Nov 8 21:31:28 2007 From: gslin at gslin.org (Gea-Suan Lin) Date: Fri, 9 Nov 2007 05:31:28 +0800 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: References: <20071108125526.GA31193@gslin.org> <20071108163503.GA77445@gslin.org> <20071108184540.GA71491@gslin.org> <20071108194457.GA78196@gslin.org> <20071108202519.GA40203@gslin.org> Message-ID: <20071108213128.GA34924@gslin.org> On Thu, Nov 08, 2007 at 10:21:22PM +0100, Dag-Erling Sm?grav wrote: > Gea-Suan Lin writes: > > It crash again: > > Actually, it doesn't crash; the child get stuck somewhere and the > parent kills it. > > I'd like to see an 'svn diff' of your tree. > > Also, if you change SIGKILL in bin/varnishd/mgt_child.c to either > SIGABRT or SIGQUIT, you should get a core dump of the child process > when the parent kills it. If you don't, 'ulimit -c unlimited' and try > again. > > DES gslin at testphp [/tmp] (5:23) svn export ~/svn/varnish/varnish-cache varnish-cache.clean gslin at testphp [/tmp] (5:23) diff -ruN varnish-cache.clean varnish-cache > varnish-cache.diff This file is in http://mail.pixnet.tw/~gslin/varnish-cache.diff.gz gslin at testphp [/tmp] (5:26) cd varnish-cache.clean/ gslin at testphp [/tmp/varnish-cache.clean] (5:26) ./autogen.sh + aclocal + libtoolize --copy --force + autoheader + automake --add-missing --copy --foreign configure.ac: installing `./install-sh' configure.ac: installing `./missing' bin/varnishadm/Makefile.am: installing `./compile' bin/varnishadm/Makefile.am: installing `./depcomp' + autoconf gslin at testphp [/tmp/varnish-cache.clean] (5:26) cd .. gslin at testphp [/tmp] (5:26) diff -ruN varnish-cache.clean varnish-cache > varnish-cache-2.diff This file is in http://mail.pixnet.tw/~gslin/varnish-cache-2.diff.gz gslin at testphp [/tmp/varnish-cache.clean] (5:29) ./configure --prefix=/home/service/varnish (lots of msgs) gslin at testphp [/tmp] (5:30) diff -ruN varnish-cache.clean varnish-cache > varnish-cache-3.diff This file is in http://mail.pixnet.tw/~gslin/varnish-cache-3.diff.gz -- * Gea-Suan Lin (public key: Using https://keyserver.pgp.com/ to search) * If you cannot convince them, confuse them. -- Harry S Truman From des at linpro.no Thu Nov 8 21:37:58 2007 From: des at linpro.no (=?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?=) Date: Thu, 08 Nov 2007 22:37:58 +0100 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: <20071108213128.GA34924@gslin.org> (Gea-Suan Lin's message of "Fri, 9 Nov 2007 05:31:28 +0800") References: <20071108125526.GA31193@gslin.org> <20071108163503.GA77445@gslin.org> <20071108184540.GA71491@gslin.org> <20071108194457.GA78196@gslin.org> <20071108202519.GA40203@gslin.org> <20071108213128.GA34924@gslin.org> Message-ID: Gea-Suan Lin writes: > gslin at testphp [/tmp] (5:23) svn export ~/svn/varnish/varnish-cache varnish-cache.clean > gslin at testphp [/tmp] (5:23) diff -ruN varnish-cache.clean varnish-cache > varnish-cache.diff This is *really* not useful. Please give me a simple 'svn diff', or at least drop the -N. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From gslin at gslin.org Thu Nov 8 21:46:58 2007 From: gslin at gslin.org (Gea-Suan Lin) Date: Fri, 9 Nov 2007 05:46:58 +0800 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: References: <20071108163503.GA77445@gslin.org> <20071108184540.GA71491@gslin.org> <20071108194457.GA78196@gslin.org> <20071108202519.GA40203@gslin.org> <20071108213128.GA34924@gslin.org> Message-ID: <20071108214658.GA71178@gslin.org> On Thu, Nov 08, 2007 at 10:37:58PM +0100, Dag-Erling Sm?grav wrote: > Gea-Suan Lin writes: > > gslin at testphp [/tmp] (5:23) svn export ~/svn/varnish/varnish-cache varnish-cache.clean > > gslin at testphp [/tmp] (5:23) diff -ruN varnish-cache.clean varnish-cache > varnish-cache.diff > > This is *really* not useful. Please give me a simple 'svn diff', or > at least drop the -N. I didn't run autogen.sh & configure & make in working repository, but "svn export varnish-cache /tmp/varnish-cache". This is diff of bin/varnishd/cache_acceptor_kqueue.c: gslin at testphp [/tmp/varnish-cache/bin/varnishd] (5:41) diff -ruN ~/svn/varnish/varnish-cache/bin/varnishd/cache_acceptor_kqueue.c cache_acceptor_kqueue.c --- /net/account/admin/gslin/svn/varnish/varnish-cache/bin/varnishd/cache_acceptor_kqueue.c Fri Nov 9 01:38:12 2007 +++ cache_acceptor_kqueue.c Fri Nov 9 03:43:26 2007 @@ -66,8 +66,8 @@ if (sp->fd < 0) return; EV_SET(&ki[nki], sp->fd, EVFILT_READ, arm, 0, 0, sp); - if (++nki == NKEV) { - assert(kevent(kq, ki, nki, NULL, 0, NULL) <= 0); + if (++nki == NKEV || arm == EV_DELETE) { + AZ(kevent(kq, ki, nki, NULL, 0, NULL)); nki = 0; } } An update of patched varnishd: I use gdb to attach it and it looks fine now, this is varnishstat screenshot, now it can store >5GB. I'll try to run without gdb later. 0+00:27:23 Hitrate ratio: 10 100 1000 Hitrate avg: 0.0933 0.2071 0.2500 30831 13.42 18.77 Client connections accepted 33441 14.38 20.35 Client requests received 5005 0.96 3.05 Cache hits 0 0.00 0.00 Cache hits for pass 28405 14.38 17.29 Cache misses 28436 14.38 17.31 Backend connections success 0 0.00 0.00 Backend connections failures 0 0.00 0.00 Backend connections reuses 0 0.00 0.00 Backend connections recycles 6 -0.96 0.00 Backend connections unused 24 . . N struct srcaddr 4 . . N active struct srcaddr 23 . . N struct sess_mem 5 . . N struct sess 28415 . . N struct object 28415 . . N struct objecthead 56817 . . N struct smf 0 . . N small free smf 1 . . N large free smf 0 . . N struct vbe_conn 17 . . N worker threads 17 0.00 0.01 N worker threads created 0 0.00 0.00 N worker threads not created 0 0.00 0.00 N worker threads limited 0 0.00 0.00 N queued work requests 17 0.00 0.01 N overflowed work requests 0 0.00 0.00 N dropped work requests 0 . . N expired objects 0 . . N LRU nuked objects 0 . . N LRU saved objects 0 . . N objects on deathrow 0 0.00 0.00 HTTP header overflows 0 0.00 0.00 Objects sent with sendfile 31598 14.38 19.23 Objects sent with write 30829 13.42 18.76 Total Sessions 33440 14.38 20.35 Total Requests 31 0.00 0.02 Total pipe 0 0.00 0.00 Total pass 28404 13.42 17.29 Total fetch 7942018 3464.68 4833.85 Total header bytes 5805632005 1854165.04 3533555.69 Total body bytes 30345 13.42 18.47 Session Closed 0 0.00 0.00 Session Pipeline 0 0.00 0.00 Session Read Ahead 3109 0.96 1.89 Session herd 2058773 926.09 1253.06 SHM records 127012 56.56 77.30 SHM writes 13 0.00 0.01 SHM MTX contention 56816 26.84 34.58 allocator requests 56816 . . outstanding allocations 5530710016 . . bytes allocated 28829028352 . . bytes free 28404 14.38 17.29 Backend requests made -- * Gea-Suan Lin (public key: Using https://keyserver.pgp.com/ to search) * If you cannot convince them, confuse them. -- Harry S Truman From gslin at gslin.org Thu Nov 8 21:52:56 2007 From: gslin at gslin.org (Gea-Suan Lin) Date: Fri, 9 Nov 2007 05:52:56 +0800 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: <20071108214658.GA71178@gslin.org> References: <20071108163503.GA77445@gslin.org> <20071108184540.GA71491@gslin.org> <20071108194457.GA78196@gslin.org> <20071108202519.GA40203@gslin.org> <20071108213128.GA34924@gslin.org> <20071108214658.GA71178@gslin.org> Message-ID: <20071108215256.GA52504@gslin.org> Okay it dies, last varnishstat screenshot: 0+00:36:40 Hitrate ratio: 10 100 1000 Hitrate avg: 0.7220 0.2649 0.2517 31612 0.00 14.37 Client connections accepted 34339 0.00 15.61 Client requests received 5134 0.00 2.33 Cache hits 0 0.00 0.00 Cache hits for pass 29165 0.00 13.26 Cache misses 29202 0.00 13.27 Backend connections success 0 0.00 0.00 Backend connections failures 0 0.00 0.00 Backend connections reuses 0 0.00 0.00 Backend connections recycles 6 0.00 0.00 Backend connections unused 25 . . N struct srcaddr 7 . . N active struct srcaddr 23 . . N struct sess_mem 10 . . N struct sess 29176 . . N struct object 29176 . . N struct objecthead 58338 . . N struct smf 0 . . N small free smf 1 . . N large free smf 0 . . N struct vbe_conn 17 . . N worker threads 17 0.00 0.01 N worker threads created 0 0.00 0.00 N worker threads not created 0 0.00 0.00 N worker threads limited 0 0.00 0.00 N queued work requests 17 0.00 0.01 N overflowed work requests 0 0.00 0.00 N dropped work requests 0 . . N expired objects 0 . . N LRU nuked objects 0 . . N LRU saved objects 0 . . N objects on deathrow 0 0.00 0.00 HTTP header overflows 0 0.00 0.00 Objects sent with sendfile 32398 0.00 14.73 Objects sent with write 31606 0.00 14.37 Total Sessions 34335 0.00 15.61 Total Requests 37 0.00 0.02 Total pipe 0 0.00 0.00 Total pass 29164 0.00 13.26 Total fetch 8150499 0.00 3704.77 Total header bytes 5990402809 0.00 2722910.37 Total body bytes 31100 0.00 14.14 Session Closed 0 0.00 0.00 Session Pipeline 0 0.00 0.00 Session Read Ahead 3249 0.00 1.48 Session herd 2113909 0.00 960.87 SHM records 130287 0.00 59.22 SHM writes 13 0.00 0.01 SHM MTX contention 58337 0.00 26.52 allocator requests 58337 . . outstanding allocations 5724512256 . . bytes allocated 28635226112 . . bytes free 29164 0.00 13.26 Backend requests made and gdb console: gslin at testphp [~] (5:15) sudo gdb /home/service/varnish/sbin/varnishd 58794 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Attaching to program: /home/service/varnish/sbin/varnishd, process 58794 Error while mapping shared library sections: ./bin.Z0UDwDnk: No such file or directory. Reading symbols from /home/service/varnish/lib/libvarnish.so.0...done. Loaded symbols for /home/service/varnish/lib/libvarnish.so.0 Reading symbols from /home/service/varnish/lib/libvarnishcompat.so.0...done. Loaded symbols for /home/service/varnish/lib/libvarnishcompat.so.0 Reading symbols from /home/service/varnish/lib/libvcl.so.0...done. Loaded symbols for /home/service/varnish/lib/libvcl.so.0 Reading symbols from /usr/lib/libthr.so.2...done. [New Thread 0xa67d600 (LWP 100358)] [New Thread 0xa67d400 (LWP 100334)] [New Thread 0xa67d200 (LWP 100202)] [New Thread 0xa67d000 (LWP 100186)] [New Thread 0x53de00 (LWP 100184)] [New Thread 0x53da00 (LWP 100092)] [New Thread 0x53d800 (LWP 100067)] [New Thread 0x53d600 (LWP 100056)] [New Thread 0x53d000 (LWP 100235)] Loaded symbols for /usr/lib/libthr.so.2 Reading symbols from /lib/libm.so.4...done. Loaded symbols for /lib/libm.so.4 Reading symbols from /lib/libc.so.6...done. Loaded symbols for /lib/libc.so.6 Error while reading shared library symbols: ./bin.Z0UDwDnk: No such file or directory. Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1 [Switching to Thread 0xa67d600 (LWP 100358)] 0x0000000800c7bf5a in read () from /lib/libc.so.6 (gdb) c Continuing. [New Thread 0xa67d800 (LWP 100407)] [New Thread 0xa67da00 (LWP 100580)] [New Thread 0xa67dc00 (LWP 100660)] [New Thread 0xa67de00 (LWP 100706)] [New Thread 0xad49000 (LWP 100770)] [New Thread 0xad49200 (LWP 100817)] [New Thread 0xad49400 (LWP 100879)] [New Thread 0xad49600 (LWP 100891)] [New Thread 0xad49800 (LWP 101241)] [New Thread 0xad49a00 (LWP 101286)] [New Thread 0xad49c00 (LWP 101292)] [New Thread 0xad49e00 (LWP 101296)] [New Thread 0xbb4e000 (LWP 101298)] [New Thread 0xbb4e200 (LWP 101321)] Program received signal SIGKILL, Killed. [Switching to Thread 0xbb4e200 (LWP 101321)] 0x0000000800c1716f in brk () from /lib/libc.so.6 (gdb) bt #0 0x0000000800c1716f in brk () from /lib/libc.so.6 #1 0x000000000c2f1000 in ?? () #2 0x0000000800c165ba in _UTF8_init () from /lib/libc.so.6 #3 0x0000000800c167e8 in _UTF8_init () from /lib/libc.so.6 #4 0x0000000800c170e6 in _UTF8_init () from /lib/libc.so.6 #5 0x0000000800c7966b in calloc () from /lib/libc.so.6 #6 0x0000000000411262 in HSH_Prealloc (sp=0xa6dd008) at cache_hash.c:80 #7 0x0000000000411875 in HSH_Lookup (sp=0xa6dd008) at cache_hash.c:185 #8 0x000000000040e890 in CNT_Session (sp=0xa6dd008) at cache_center.c:534 #9 0x0000000000416209 in wrk_thread (priv=0x53e5e0) at cache_pool.c:193 #10 0x000000080098729e in pthread_create () from /usr/lib/libthr.so.2 #11 0x0000000000000000 in ?? () Error accessing memory address 0x7ffffd1ea000: Bad address. (gdb) up #1 0x000000000c2f1000 in ?? () (gdb) up #2 0x0000000800c165ba in _UTF8_init () from /lib/libc.so.6 (gdb) up #3 0x0000000800c167e8 in _UTF8_init () from /lib/libc.so.6 (gdb) up #4 0x0000000800c170e6 in _UTF8_init () from /lib/libc.so.6 (gdb) up #5 0x0000000800c7966b in calloc () from /lib/libc.so.6 (gdb) up #6 0x0000000000411262 in HSH_Prealloc (sp=0xa6dd008) at cache_hash.c:80 80 w->nobjhead = calloc(sizeof *w->nobjhead, 1); (gdb) l 75 CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); 76 CHECK_OBJ_NOTNULL(sp->wrk, WORKER_MAGIC); 77 w = sp->wrk; 78 79 if (w->nobjhead == NULL) { 80 w->nobjhead = calloc(sizeof *w->nobjhead, 1); 81 XXXAN(w->nobjhead); 82 w->nobjhead->magic = OBJHEAD_MAGIC; 83 VTAILQ_INIT(&w->nobjhead->objects); 84 MTX_INIT(&w->nobjhead->mtx); (gdb) up #7 0x0000000000411875 in HSH_Lookup (sp=0xa6dd008) at cache_hash.c:185 185 HSH_Prealloc(sp); (gdb) l 180 CHECK_OBJ_NOTNULL(sp->http, HTTP_MAGIC); 181 AN(hash); 182 w = sp->wrk; 183 h = sp->http; 184 185 HSH_Prealloc(sp); 186 if (sp->obj != NULL) { 187 CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC); 188 o = sp->obj; 189 oh = o->objhead; (gdb) up #8 0x000000000040e890 in CNT_Session (sp=0xa6dd008) at cache_center.c:534 534 o = HSH_Lookup(sp); (gdb) l 529 530 VCL_hash_method(sp); 531 /* XXX check error */ 532 } 533 534 o = HSH_Lookup(sp); 535 536 if (o == NULL) { 537 /* 538 * We hit a busy object, disembark worker thread and expect (gdb) up #9 0x0000000000416209 in wrk_thread (priv=0x53e5e0) at cache_pool.c:193 193 CNT_Session(wrq->sess); (gdb) l 188 CHECK_OBJ_NOTNULL(wrq->sess, SESS_MAGIC); 189 wrq->sess->wrk = w; 190 CHECK_OBJ_ORNULL(w->nobj, OBJECT_MAGIC); 191 CHECK_OBJ_ORNULL(w->nobjhead, OBJHEAD_MAGIC); 192 w->used = NAN; 193 CNT_Session(wrq->sess); 194 assert(!isnan(w->used)); 195 CHECK_OBJ_ORNULL(w->nobj, OBJECT_MAGIC); 196 CHECK_OBJ_ORNULL(w->nobjhead, OBJHEAD_MAGIC); 197 w->wrq = NULL; (gdb) up #10 0x000000080098729e in pthread_create () from /usr/lib/libthr.so.2 (gdb) up #11 0x0000000000000000 in ?? () (gdb) up Error accessing memory address 0x7ffffd1ea000: Bad address. (gdb) gslin at testphp [~] (5:14) sudo /usr/bin/env -i /home/service/varnish/sbin/varnishd -a 60.199.247.118:80 -f /usr/local/etc/varn ish/image.vcl -h classic,1048583 -P /var/run/varnishd.pid -s file,/home/service/varnish-cache.mmap,32G -T 127.0.0.1:11957 -t 604800 -w 32,4096 -d -d storage_file: filename: /home/service/varnish-cache.mmap size 32768 MegaBytes. Classic hash: 1048583 buckets Using old SHMFILE rolling(1)... rolling(2)... start start child pid 58794 200 0 Child said (2, 58794): <> 200 0 200 0 Child not responding to ping Child not responding to ping Child not responding to ping Child not responding to ping Child not responding to ping Child not responding to ping Child not responding to ping Child not responding to ping Child not responding to ping Child not responding to ping Child not responding to ping Child not responding to ping (lots of "Child not ...") -- * Gea-Suan Lin (public key: Using https://keyserver.pgp.com/ to search) * If you cannot convince them, confuse them. -- Harry S Truman From des at linpro.no Thu Nov 8 21:57:04 2007 From: des at linpro.no (=?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?=) Date: Thu, 08 Nov 2007 22:57:04 +0100 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: <20071108215256.GA52504@gslin.org> (Gea-Suan Lin's message of "Fri, 9 Nov 2007 05:52:56 +0800") References: <20071108163503.GA77445@gslin.org> <20071108184540.GA71491@gslin.org> <20071108194457.GA78196@gslin.org> <20071108202519.GA40203@gslin.org> <20071108213128.GA34924@gslin.org> <20071108214658.GA71178@gslin.org> <20071108215256.GA52504@gslin.org> Message-ID: Gea-Suan Lin writes: > Program received signal SIGKILL, Killed. Killed by the parent again. > [Switching to Thread 0xbb4e200 (LWP 101321)] > 0x0000000800c1716f in brk () from /lib/libc.so.6 > (gdb) bt > #0 0x0000000800c1716f in brk () from /lib/libc.so.6 > #1 0x000000000c2f1000 in ?? () > #2 0x0000000800c165ba in _UTF8_init () from /lib/libc.so.6 > #3 0x0000000800c167e8 in _UTF8_init () from /lib/libc.so.6 > #4 0x0000000800c170e6 in _UTF8_init () from /lib/libc.so.6 > #5 0x0000000800c7966b in calloc () from /lib/libc.so.6 > #6 0x0000000000411262 in HSH_Prealloc (sp=0xa6dd008) at cache_hash.c:80 > #7 0x0000000000411875 in HSH_Lookup (sp=0xa6dd008) at cache_hash.c:185 > #8 0x000000000040e890 in CNT_Session (sp=0xa6dd008) at cache_center.c:534 > #9 0x0000000000416209 in wrk_thread (priv=0x53e5e0) at cache_pool.c:193 > #10 0x000000080098729e in pthread_create () from /usr/lib/libthr.so.2 > #11 0x0000000000000000 in ?? () > Error accessing memory address 0x7ffffd1ea000: Bad address. This stack frame is of no interest, it is just the thread that happened to be running when the SIGKILL was delivered. What would be helpful at this point is "i thr". As previously mentioned, this will be a lot easier if you modify mgt_child.c to send SIGQUIT instead of SIGKILL so you get a core dump. BTW, I assume this is all on FreeBSD? Which version? DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From gslin at gslin.org Thu Nov 8 22:07:30 2007 From: gslin at gslin.org (Gea-Suan Lin) Date: Fri, 9 Nov 2007 06:07:30 +0800 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: References: <20071108184540.GA71491@gslin.org> <20071108194457.GA78196@gslin.org> <20071108202519.GA40203@gslin.org> <20071108213128.GA34924@gslin.org> <20071108214658.GA71178@gslin.org> <20071108215256.GA52504@gslin.org> Message-ID: <20071108220730.GA66374@gslin.org> On Thu, Nov 08, 2007 at 10:57:04PM +0100, Dag-Erling Sm?grav wrote: > This stack frame is of no interest, it is just the thread that > happened to be running when the SIGKILL was delivered. What would be > helpful at this point is "i thr". > > As previously mentioned, this will be a lot easier if you modify > mgt_child.c to send SIGQUIT instead of SIGKILL so you get a core dump. Okay. I'll run the following version: gslin at testphp [~/svn/varnish] (6:02) svn up At revision 2245. gslin at testphp [~/svn/varnish] (6:02) svn diff Index: varnish-cache/bin/varnishd/mgt_child.c =================================================================== --- varnish-cache/bin/varnishd/mgt_child.c (revision 2245) +++ varnish-cache/bin/varnishd/mgt_child.c (working copy) @@ -114,7 +114,7 @@ return (1); if (child_pid > 0 && mgt_cli_askchild(NULL, NULL, "ping\n")) { fprintf(stderr, "Child not responding to ping\n"); - kill(child_pid, SIGKILL); + kill(child_pid, SIGQUIT); } return (0); } gslin at testphp [~/svn/varnish] (6:02) svn export varnish-cache /tmp/varnish-cache Export complete. gslin at testphp [/tmp/varnish-cache] (6:03) ./autogen.sh + aclocal + libtoolize --copy --force + autoheader + automake --add-missing --copy --foreign configure.ac: installing `./install-sh' configure.ac: installing `./missing' bin/varnishadm/Makefile.am: installing `./compile' bin/varnishadm/Makefile.am: installing `./depcomp' + autoconf gslin at testphp [/tmp/varnish-cache] (6:03) ./configure --prefix=/home/service/varnish (lots of msgs) gslin at testphp [/tmp/varnish-cache] (6:04) make (lots of msgs) gslin at testphp [/tmp/varnish-cache] (6:05) sudo make install clean And, gslin at testphp [/home/service/varnish/sbin] (6:06) ldd varnishd varnishd: libvarnish.so.0 => /home/service/varnish/lib/libvarnish.so.0 (0x800664000) libvarnishcompat.so.0 => /home/service/varnish/lib/libvarnishcompat.so.0 (0x80076b000) libvcl.so.0 => /home/service/varnish/lib/libvcl.so.0 (0x80086c000) libthr.so.2 => /usr/lib/libthr.so.2 (0x80097e000) libm.so.4 => /lib/libm.so.4 (0x800a94000) libc.so.6 => /lib/libc.so.6 (0x800bb0000) > BTW, I assume this is all on FreeBSD? Which version? gslin at testphp [~] (6:05) uname -a FreeBSD testphp.pixnet.tw 6.2-STABLE FreeBSD 6.2-STABLE #2: Mon Mar 26 12:20:42 CST 2007 root at tmp1.pixnet.tw:/usr/obj/usr/src/sys/BLOG amd64 gslin at testphp [~] (6:05) cat /etc/libmap.conf libc_r.so libthr.so libc_r.so.6 libthr.so.2 libpthread.so libthr.so libpthread.so.2 libthr.so.2 -- * Gea-Suan Lin (public key: Using https://keyserver.pgp.com/ to search) * If you cannot convince them, confuse them. -- Harry S Truman From gslin at gslin.org Thu Nov 8 22:32:50 2007 From: gslin at gslin.org (Gea-Suan Lin) Date: Fri, 9 Nov 2007 06:32:50 +0800 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: References: <20071108125526.GA31193@gslin.org> <20071108163503.GA77445@gslin.org> <20071108184540.GA71491@gslin.org> <20071108194457.GA78196@gslin.org> <20071108202519.GA40203@gslin.org> Message-ID: <20071108223250.GA56487@gslin.org> On Thu, Nov 08, 2007 at 10:21:22PM +0100, Dag-Erling Sm?grav wrote: > Gea-Suan Lin writes: > > It crash again: > > Actually, it doesn't crash; the child get stuck somewhere and the > parent kills it. > > I'd like to see an 'svn diff' of your tree. > > Also, if you change SIGKILL in bin/varnishd/mgt_child.c to either > SIGABRT or SIGQUIT, you should get a core dump of the child process > when the parent kills it. If you don't, 'ulimit -c unlimited' and try > again. I can see signal 3 (QUIT) from dmesg: pid 76061 (varnishd), uid 65534: exited on signal 3 pid 76187 (varnishd), uid 65534: exited on signal 3 but I cannot find coredump. I run the following command in /tmp with mode 1777: sudo /usr/bin/env -i /home/service/varnish/sbin/varnishd -a 60.199.247.118:80 -f /usr/local/etc/varnish/image.vcl -h classic,1048583 -P /var/run/varnishd.pid -s file,/home/service/varnish-cache.mmap,32G -T 127.0.0.1:11957 -t 604800 -w 32,4096 And the default coredumpsize is unlimited now. gslin at testphp [~] (6:29) limits Resource limits (current): cputime infinity secs filesize infinity kB datasize 33554432 kB stacksize 524288 kB coredumpsize infinity kB memoryuse infinity kB memorylocked infinity kB maxprocesses 5547 openfiles 524288 sbsize infinity bytes vmemoryuse infinity kB gslin at testphp [~] (6:29) sudo env -i /usr/bin/limits Resource limits (current): cputime infinity secs filesize infinity kB datasize 33554432 kB stacksize 524288 kB coredumpsize infinity kB memoryuse infinity kB memorylocked infinity kB maxprocesses 5547 openfiles 524288 sbsize infinity bytes vmemoryuse infinity kB Suggestion ? -- * Gea-Suan Lin (public key: Using https://keyserver.pgp.com/ to search) * If you cannot convince them, confuse them. -- Harry S Truman From des at linpro.no Thu Nov 8 22:41:36 2007 From: des at linpro.no (=?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?=) Date: Thu, 08 Nov 2007 23:41:36 +0100 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: <20071108223250.GA56487@gslin.org> (Gea-Suan Lin's message of "Fri, 9 Nov 2007 06:32:50 +0800") References: <20071108125526.GA31193@gslin.org> <20071108163503.GA77445@gslin.org> <20071108184540.GA71491@gslin.org> <20071108194457.GA78196@gslin.org> <20071108202519.GA40203@gslin.org> <20071108223250.GA56487@gslin.org> Message-ID: Gea-Suan Lin writes: > I can see signal 3 (QUIT) from dmesg: > > pid 76061 (varnishd), uid 65534: exited on signal 3 > pid 76187 (varnishd), uid 65534: exited on signal 3 > > but I cannot find coredump. I run the following command in /tmp with > mode 1777: Read what I wrote earlier, you need to run 'ulimit -c unlimited' before starting Varnish. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From gslin at gslin.org Thu Nov 8 23:29:59 2007 From: gslin at gslin.org (Gea-Suan Lin) Date: Fri, 9 Nov 2007 07:29:59 +0800 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: References: <20071108163503.GA77445@gslin.org> <20071108184540.GA71491@gslin.org> <20071108194457.GA78196@gslin.org> <20071108202519.GA40203@gslin.org> <20071108223250.GA56487@gslin.org> Message-ID: <20071108232959.GA34048@gslin.org> On Thu, Nov 08, 2007 at 11:41:36PM +0100, Dag-Erling Sm?grav wrote: > Gea-Suan Lin writes: > > I can see signal 3 (QUIT) from dmesg: > > > > pid 76061 (varnishd), uid 65534: exited on signal 3 > > pid 76187 (varnishd), uid 65534: exited on signal 3 > > > > but I cannot find coredump. I run the following command in /tmp with > > mode 1777: > > Read what I wrote earlier, you need to run 'ulimit -c unlimited' > before starting Varnish. I've tried 4 times, but varnishd still not to generate dump file. Here is my step: * /usr/local/sbin/varnishd.sh: #!/bin/sh ulimit -c unlimited /home/service/varnish/sbin/varnishd -a 60.199.247.118:80 -f /usr/local/etc/varnish/image.vcl -h classic,1048583 -P /var/run/varnishd.pid -s file,/home/service/varnish-cache.mmap,32G -T 127.0.0.1:11957 -t 604800 -w 32,4096 * Run it in my home directory with large enough disk and suitable permission directory: gslin at testphp [~/tmp] (7:27) d total 8 drwxrwxrwt 2 gslin admin 4096 Nov 9 07:09 ./ drwxr-xr-x 10 gslin admin 4096 Nov 9 07:19 ../ gslin at testphp [~/tmp] (7:27) df -h Filesystem Size Used Avail Capacity Mounted on /dev/da0s1a 4.8G 89M 4.4G 2% / devfs 1.0K 1.0K 0B 100% /dev /dev/da0s1f 48G 7.5G 36G 17% /home /dev/da0s1d 4.8G 3.2G 1.2G 72% /usr /dev/da0s1e 4.8G 97M 4.4G 2% /var /dev/da1s1d 66G 3.6G 57G 6% /home/logs 10.1.1.100:/vol/home 80G 2.4G 78G 3% /.amd_mnt/10.1.1.100/vol/home gslin at testphp [~/tmp] (7:28) pwd /.amd_mnt/10.1.1.100/vol/home/admin/gslin/tmp dmesg's last messages is: pid 76620 (varnishd), uid 65534: exited on signal 3 pid 76639 (varnishd), uid 65534: exited on signal 6 -- * Gea-Suan Lin (public key: Using https://keyserver.pgp.com/ to search) * If you cannot convince them, confuse them. -- Harry S Truman From des at linpro.no Thu Nov 8 23:32:37 2007 From: des at linpro.no (=?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?=) Date: Fri, 09 Nov 2007 00:32:37 +0100 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: <20071108232959.GA34048@gslin.org> (Gea-Suan Lin's message of "Fri, 9 Nov 2007 07:29:59 +0800") References: <20071108163503.GA77445@gslin.org> <20071108184540.GA71491@gslin.org> <20071108194457.GA78196@gslin.org> <20071108202519.GA40203@gslin.org> <20071108223250.GA56487@gslin.org> <20071108232959.GA34048@gslin.org> Message-ID: Gea-Suan Lin writes: > I've tried 4 times, but varnishd still not to generate dump file. Are you sure? It should be in the run-time state directory, most likely $PREFIX/var/varnish/$HOSTNAME. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From gslin at gslin.org Thu Nov 8 23:50:41 2007 From: gslin at gslin.org (Gea-Suan Lin) Date: Fri, 9 Nov 2007 07:50:41 +0800 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: References: <20071108184540.GA71491@gslin.org> <20071108194457.GA78196@gslin.org> <20071108202519.GA40203@gslin.org> <20071108223250.GA56487@gslin.org> <20071108232959.GA34048@gslin.org> Message-ID: <20071108235041.GA68876@gslin.org> On Fri, Nov 09, 2007 at 12:32:37AM +0100, Dag-Erling Sm?grav wrote: > Gea-Suan Lin writes: > > I've tried 4 times, but varnishd still not to generate dump file. > > Are you sure? It should be in the run-time state directory, > most likely $PREFIX/var/varnish/$HOSTNAME. The core file is not in /home/service/var/varnish/testphp.pixnet.tw/, my guess is it's too small so I retry it: * rm -rf /home/service/varnish/var/varnish/testphp.pixnet.tw * ln -s ~/tmp /home/service/varnish/var/varnish/testphp.pixnet.tw * chmod 1777 ~/tmp So I got: gslin at testphp [~/tmp] (7:47) d total 8244 drwxrwxrwt 2 gslin admin 4096 Nov 9 07:40 ./ drwxr-xr-x 10 gslin admin 4096 Nov 9 07:33 ../ -rw-r--r-- 1 root wheel 8389208 Nov 9 07:43 _.vsl -rwxr-xr-x 1 root wheel 14534 Nov 9 07:40 bin.LxjNthA8* -rwxr-xr-x 1 gslin admin 260 Nov 9 07:33 varnishd.sh* gslin at testphp [~/tmp] (7:47) cat varnishd.sh #!/bin/sh ulimit -c unlimited /home/service/varnish/sbin/varnishd -a 60.199.247.118:80 -f /usr/local/etc/varnish/image.vcl -h classic,1048583 -P /var/run/varnishd.pid -s file,/home/service/varnish-cache.mmap,32G -T 127.0.0.1:11957 -t 604800 -w 32,4096 -d -d gslin at testphp [~/tmp] (7:47) Run it and got SIGQUIT: Nov 9 07:43:26 testphp kernel: pid 76784 (varnishd), uid 65534: exited on signal 3 Nov 9 07:43:51 testphp kernel: pid 76793 (varnishd), uid 65534: exited on signal 3 It's console: gslin at testphp [~/tmp] (7:40) sudo ./varnishd.sh storage_file: filename: /home/service/varnish-cache.mmap size 32768 MegaBytes. Classic hash: 1048583 buckets Using old SHMFILE rolling(1)... rolling(2)... start start child pid 76784 200 0 Child said (2, 76784): <> Child not responding to ping Child not responding to ping Child not responding to ping Child not responding to ping Cache child died pid=76784 status=0x3 Clean child Child cleaned start child pid 76793 Child said (2, 76793): <> Child not responding to ping Child not responding to ping Child not responding to ping Child not responding to ping Child not responding to ping Cache child died pid=76793 status=0x3 Clean child Child cleaned start child pid 76794 Child said (2, 76794): <> -- * Gea-Suan Lin (public key: Using https://keyserver.pgp.com/ to search) * If you cannot convince them, confuse them. -- Harry S Truman From des at linpro.no Thu Nov 8 23:55:57 2007 From: des at linpro.no (=?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?=) Date: Fri, 09 Nov 2007 00:55:57 +0100 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: <20071108235041.GA68876@gslin.org> (Gea-Suan Lin's message of "Fri, 9 Nov 2007 07:50:41 +0800") References: <20071108184540.GA71491@gslin.org> <20071108194457.GA78196@gslin.org> <20071108202519.GA40203@gslin.org> <20071108223250.GA56487@gslin.org> <20071108232959.GA34048@gslin.org> <20071108235041.GA68876@gslin.org> Message-ID: Gea-Suan Lin writes: > gslin at testphp [~/tmp] (7:47) cat varnishd.sh > #!/bin/sh > > ulimit -c unlimited > /home/service/varnish/sbin/varnishd -a 60.199.247.118:80 -f /usr/local/etc/varnish/image.vcl -h classic,1048583 -P /var/run/varnishd.pid -s file,/home/service/varnish-cache.mmap,32G -T 127.0.0.1:11957 -t 604800 -w 32,4096 -d -d > gslin at testphp [~/tmp] (7:47) > > Run it and got SIGQUIT: > > Nov 9 07:43:26 testphp kernel: pid 76784 (varnishd), uid 65534: exited on signal 3 > Nov 9 07:43:51 testphp kernel: pid 76793 (varnishd), uid 65534: exited on signal 3 Still no core file? Try SIGABRT instead. If that doesn't work, I'm out of ideas... though you can still attach directly to the child with gdb. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From gslin at gslin.org Fri Nov 9 00:02:36 2007 From: gslin at gslin.org (Gea-Suan Lin) Date: Fri, 9 Nov 2007 08:02:36 +0800 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: References: <20071108194457.GA78196@gslin.org> <20071108202519.GA40203@gslin.org> <20071108223250.GA56487@gslin.org> <20071108232959.GA34048@gslin.org> <20071108235041.GA68876@gslin.org> Message-ID: <20071109000236.GA6029@gslin.org> On Fri, Nov 09, 2007 at 12:55:57AM +0100, Dag-Erling Sm?grav wrote: > Gea-Suan Lin writes: > > gslin at testphp [~/tmp] (7:47) cat varnishd.sh > > #!/bin/sh > > > > ulimit -c unlimited > > /home/service/varnish/sbin/varnishd -a 60.199.247.118:80 -f /usr/local/etc/varnish/image.vcl -h classic,1048583 -P /var/run/varnishd.pid -s file,/home/service/varnish-cache.mmap,32G -T 127.0.0.1:11957 -t 604800 -w 32,4096 -d -d > > gslin at testphp [~/tmp] (7:47) > > > > Run it and got SIGQUIT: > > > > Nov 9 07:43:26 testphp kernel: pid 76784 (varnishd), uid 65534: exited on signal 3 > > Nov 9 07:43:51 testphp kernel: pid 76793 (varnishd), uid 65534: exited on signal 3 > > Still no core file? Try SIGABRT instead. If that doesn't work, I'm > out of ideas... though you can still attach directly to the child > with gdb. okay I use SIGABRT now. If it still not able to generate, I'll try to use gdb "generate-core-file" command to generate it. -- * Gea-Suan Lin (public key: Using https://keyserver.pgp.com/ to search) * If you cannot convince them, confuse them. -- Harry S Truman From gslin at gslin.org Fri Nov 9 00:38:20 2007 From: gslin at gslin.org (Gea-Suan Lin) Date: Fri, 9 Nov 2007 08:38:20 +0800 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: <20071109000236.GA6029@gslin.org> References: <20071108194457.GA78196@gslin.org> <20071108202519.GA40203@gslin.org> <20071108223250.GA56487@gslin.org> <20071108232959.GA34048@gslin.org> <20071108235041.GA68876@gslin.org> <20071109000236.GA6029@gslin.org> Message-ID: <20071109003820.GA32986@gslin.org> On Fri, Nov 09, 2007 at 08:02:36AM +0800, Gea-Suan Lin wrote: > On Fri, Nov 09, 2007 at 12:55:57AM +0100, Dag-Erling Sm?grav wrote: > > Gea-Suan Lin writes: > > > gslin at testphp [~/tmp] (7:47) cat varnishd.sh > > > #!/bin/sh > > > > > > ulimit -c unlimited > > > /home/service/varnish/sbin/varnishd -a 60.199.247.118:80 -f /usr/local/etc/varnish/image.vcl -h classic,1048583 -P /var/run/varnishd.pid -s file,/home/service/varnish-cache.mmap,32G -T 127.0.0.1:11957 -t 604800 -w 32,4096 -d -d > > > gslin at testphp [~/tmp] (7:47) > > > > > > Run it and got SIGQUIT: > > > > > > Nov 9 07:43:26 testphp kernel: pid 76784 (varnishd), uid 65534: exited on signal 3 > > > Nov 9 07:43:51 testphp kernel: pid 76793 (varnishd), uid 65534: exited on signal 3 > > > > Still no core file? Try SIGABRT instead. If that doesn't work, I'm > > out of ideas... though you can still attach directly to the child > > with gdb. > > okay I use SIGABRT now. If it still not able to generate, I'll try to > use gdb "generate-core-file" command to generate it. I got this: (gdb) generate-core-file Couldn't open /proc/1005/map I'll mount procfs and try again. -- * Gea-Suan Lin (public key: Using https://keyserver.pgp.com/ to search) * If you cannot convince them, confuse them. -- Harry S Truman From des at linpro.no Fri Nov 9 08:23:46 2007 From: des at linpro.no (=?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?=) Date: Fri, 09 Nov 2007 09:23:46 +0100 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: <20071109000236.GA6029@gslin.org> (Gea-Suan Lin's message of "Fri, 9 Nov 2007 08:02:36 +0800") References: <20071108194457.GA78196@gslin.org> <20071108202519.GA40203@gslin.org> <20071108223250.GA56487@gslin.org> <20071108232959.GA34048@gslin.org> <20071108235041.GA68876@gslin.org> <20071109000236.GA6029@gslin.org> Message-ID: Gea-Suan Lin writes: > okay I use SIGABRT now. If it still not able to generate, I'll try to > use gdb "generate-core-file" command to generate it. One last thing to try: 'sysctl kern.coredump=1', though I don't see why it would be 0. There's not much point in using gdb to generate a core dump, though, as the entire point was to avoid having to attach gdb to the child. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From gslin at gslin.org Fri Nov 9 19:07:17 2007 From: gslin at gslin.org (Gea-Suan Lin) Date: Sat, 10 Nov 2007 03:07:17 +0800 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: References: <20071108202519.GA40203@gslin.org> <20071108223250.GA56487@gslin.org> <20071108232959.GA34048@gslin.org> <20071108235041.GA68876@gslin.org> <20071109000236.GA6029@gslin.org> Message-ID: <20071109190717.GA64764@gslin.org> On Fri, Nov 09, 2007 at 09:23:46AM +0100, Dag-Erling Sm?grav wrote: > Gea-Suan Lin writes: > > okay I use SIGABRT now. If it still not able to generate, I'll try to > > use gdb "generate-core-file" command to generate it. > > One last thing to try: 'sysctl kern.coredump=1', though I don't see > why it would be 0. There's not much point in using gdb to generate a > core dump, though, as the entire point was to avoid having to attach > gdb to the child. "kern.coredump" is already 1, but I found "kern.sugid_coredump" in core(5): By default, a process that changes user or group credentials whether real or effective will not create a corefile. This behaviour can be changed to generate a core dump by setting the sysctl(8) variable kern.sugid_coredump to 1. Since varnishd is run by root then setuid to nobody, this rule will apply to. Change kern.sugid_coredump from 0 to 1 then the core file was generated. Core file and execute file is in: http://mail.pixnet.tw/~gslin/tmp/varnishd.core http://mail.pixnet.tw/~gslin/tmp/varnishd -- * Gea-Suan Lin (public key: Using https://keyserver.pgp.com/ to search) * If you cannot convince them, confuse them. -- Harry S Truman From des at linpro.no Fri Nov 9 20:50:31 2007 From: des at linpro.no (=?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?=) Date: Fri, 09 Nov 2007 21:50:31 +0100 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: <20071109190717.GA64764@gslin.org> (Gea-Suan Lin's message of "Sat, 10 Nov 2007 03:07:17 +0800") References: <20071108202519.GA40203@gslin.org> <20071108223250.GA56487@gslin.org> <20071108232959.GA34048@gslin.org> <20071108235041.GA68876@gslin.org> <20071109000236.GA6029@gslin.org> <20071109190717.GA64764@gslin.org> Message-ID: Gea-Suan Lin writes: > Core file and execute file is in: > > http://mail.pixnet.tw/~gslin/tmp/varnishd.core > http://mail.pixnet.tw/~gslin/tmp/varnishd That won't help; at the very least, I'd need all the Varnish libraries and access to a system with the exact same system libraries as yours. What I asked for was the output of "i thr" in gdb. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From gslin at gslin.org Fri Nov 9 21:16:14 2007 From: gslin at gslin.org (Gea-Suan Lin) Date: Sat, 10 Nov 2007 05:16:14 +0800 Subject: Varnish crash (SIGABRT) about every 10 mins In-Reply-To: References: <20071108223250.GA56487@gslin.org> <20071108232959.GA34048@gslin.org> <20071108235041.GA68876@gslin.org> <20071109000236.GA6029@gslin.org> <20071109190717.GA64764@gslin.org> Message-ID: <20071109211614.GA31087@gslin.org> On Fri, Nov 09, 2007 at 09:50:31PM +0100, Dag-Erling Sm?grav wrote: > Gea-Suan Lin writes: > > Core file and execute file is in: > > > > http://mail.pixnet.tw/~gslin/tmp/varnishd.core > > http://mail.pixnet.tw/~gslin/tmp/varnishd > > That won't help; at the very least, I'd need all the Varnish libraries > and access to a system with the exact same system libraries as yours. All files in /home/service/varnish been package in: http://mail.pixnet.tw/~gslin/tmp/varnish-all.tar.gz But you may need *all libraries*, should I create a shell account to let you access ? > What I asked for was the output of "i thr" in gdb. (gdb) i thr 12 Thread 0x53d000 (LWP 100256) 0x0000000800c3356c in poll () from /lib/libc.so.6 11 Thread 0x53d800 (LWP 100159) 0x0000000800c6373c in nanosleep () from /lib/libc.so.6 10 Thread 0x53da00 (LWP 100199) 0x0000000800c6373c in nanosleep () from /lib/libc.so.6 9 Thread 0x53dc00 (LWP 100212) 0x0000000800c6373c in nanosleep () from /lib/libc.so.6 8 Thread 0xa67d000 (LWP 100224) 0x0000000800c432fc in kevent () from /lib/libc.so.6 7 Thread 0xa67d200 (LWP 100225) 0x0000000800c3356c in poll () from /lib/libc.so.6 6 Thread 0xa67d400 (LWP 100226) 0x0000000800bfe07c in _umtx_op () from /lib/libc.so.6 5 Thread 0xa67d600 (LWP 100227) 0x0000000800c7bf5c in read () from /lib/libc.so.6 4 Thread 0xa67d800 (LWP 100228) 0x0000000800c7b926 in memcpy () from /lib/libc.so.6 3 Thread 0xa67da00 (LWP 100229) 0x0000000800c7b8f4 in memset () from /lib/libc.so.6 2 Thread 0xa67dc00 (LWP 100230) 0x0000000800bfe07c in _umtx_op () from /lib/libc.so.6 * 1 Thread 0xa67de00 (LWP 100231) 0x0000000800c7bf5c in read () from /lib/libc.so.6 (gdb) bt #0 0x0000000800c7bf5c in read () from /lib/libc.so.6 #1 0x0000000800984fbb in read () from /usr/lib/libthr.so.2 #2 0x000000000041585e in HTC_Read (htc=0x7ffffe7f2a20, d=0x86dd6c000, len=160763) at cache_httpconn.c:202 #3 0x000000000041095f in Fetch (sp=0xa681008) at cache_fetch.c:72 #4 0x000000000040e42b in CNT_Session (sp=0xa681008) at cache_center.c:323 #5 0x0000000000416209 in wrk_thread (priv=0x53e5e0) at cache_pool.c:193 #6 0x000000080098729e in pthread_create () from /usr/lib/libthr.so.2 #7 0x0000000000000000 in ?? () Cannot access memory at address 0x7ffffe7f5000 (gdb) i fra Stack level 0, frame at 0x7ffffe7f0930: rip = 0x800c7bf5c in read; saved rip 0x800984fbb called by frame at 0x7ffffe7f0960 Arglist at 0x7ffffe7f0920, args: Locals at 0x7ffffe7f0920, Previous frame's sp is 0x7ffffe7f0930 Saved registers: rip at 0x7ffffe7f0928 (gdb) -- * Gea-Suan Lin (public key: Using https://keyserver.pgp.com/ to search) * If you cannot convince them, confuse them. -- Harry S Truman From thetuxkeeper at googlemail.com Sat Nov 10 17:33:49 2007 From: thetuxkeeper at googlemail.com (Daniel) Date: Sat, 10 Nov 2007 18:33:49 +0100 Subject: some strange and blocking problems with varnish In-Reply-To: References: Message-ID: <200711101833.50234.thetuxkeeper@googlemail.com> Hi, first i want to say that varnish is a great program. But i have to say that we have some problems so we have to switch back to squid temporary. We use varnish 1.1 in front of pound (loadbalancer) and behind this the real webservers (apache 2.2 with mod_rewrite, php, ...) First Problem that appeared: We host a vbulletin forum and with switching to varnish the login didn't work correctly anymore. You could login and you were listed as logged in, but the page looks like you're logged out. If you activated the "Remember Me?" checkbox, it worked. We have tested several configurations: - pass/pipe everything of this domain - added the server directly as backend - played with caching or not caching cookie sites Now we set the domain directly to the backend webserver. We haven't found a solution yet. Second problem: varnish produces Internal Server errors though the requested site is available, if i go through pound->backend webserver or directly to the backend servers. One url was unavailable for several hours. One of 100 wget requests ended with a success, but before and after this one only errors. Now this one works again, but we get still nearly randomly internal server errors. No matter how many request have to be handled by varnish. Here my configurations, perhaps there's a fault: vcl: http://pastebin.com/f4aa96552 daemon opts: http://pastebin.com/f5055076c I hope you can help me to solve these things. Atm it's not possible to enable vcl_trace since there are around 200 req/s. I think we'll switch back to squid tomorrow, so i'll be able to trace the problem better. Kind Regards, Daniel From duja at torlen.net Fri Nov 16 10:08:11 2007 From: duja at torlen.net (Erik) Date: Fri, 16 Nov 2007 11:08:11 +0100 Subject: Debug: On Waiting list? Message-ID: <00fkbog5bxjat75.161120071108@server2k7> Hi, Im struggling with a couple of high response times when using varnish on our site. I have configured varnish so it will cache only static content (jpg,gif,png,css,js etc) even if a cookie is set. But all our jpeg's are getting a high response time when Im running some test against the site. One of the jpeg's are pic_mini.jpg, it has a response time of 357ms. I turned on logging and vcl_trace and found the line: 14 Debug c "Hash Match: /Portals/0/Picture/pic_mini.jpg#www3.mydomain.com#" 14 Debug c "on waiting list on obj 1767384108" Does "on waiting list" has something to do with the high response time? The rest fo the log for pic_mini.jpg is: 14 ReqStart c xx.xx.xx.187 41506 1767384112 14 RxRequest c GET 14 RxURL c /Portals/0/Picture/pic_mini.jpg 14 RxProtocol c HTTP/1.1 14 RxHeader c Host: www3.mydomain.com 14 RxHeader c User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; sv-SE; rv:1.8.1.8) Gecko/20071008 Firefox/2.0.0.8 14 RxHeader c Accept: */* 14 RxHeader c Accept-Language: sv,en-us;q=3D0.7,en;q=3D0.3 14 RxHeader c Accept-Encoding: gzip,deflate 14 RxHeader c Accept-Charset:= ISO-8859-1,utf-8;q=3D0.7,*;q=3D0.7 14 RxHeader c Keep-Alive: 300 14 RxHeader c Connection: Keep-Alive 14 RxHeader c Cookie: language=3Den-US; .ASPXANONYMOUS=3DnBeDLiReyAEkAAAAYTIwNmNiYjYtNmRmMS00MmQ4LThlZGItY= mZlNThjYjhjZjky0 14 VCL_call c recv 14 VCL_trace c 1 84.14 14 VCL_trace c 2 87.13 14 VCL_trace c 3 87.51 14 VCL_return c lookup 14 VCL_call c hash 14 VCL_trace c 47 22.14 14 VCL_trace c 48 24.9 14 VCL_trace c 49 24.24 14 VCL_return c hash 14 Debug c "Hash Match: /Portals/0/Picture/pic_mini.jpg#www3.mydomain.com#" 14 Debug c "on waiting list on obj 1767384108" In vcl_recv im fetching static content using: sub vcl_recv { # *.jpg , *.jpeg , *.gif , *.png , *.ico if (req.url ~ ".(jpg|gif|png|jpeg|ico)$") { lookup; } # *.js , *.css , *.swf if (req.url ~ ".(js|css|swf)$") { lookup; } if (req.http.Cookie) { # *.jpg , *.jpeg , *.gif , *.png , *.ico if (req.url ~ ".(jpg|gif|png|jpeg|ico)$") { lookup; } # *.js , *.css , *.swf if (req.url ~ ".(js|css|swf)$") { lookup; } } pass; } #RECV and my vcl_fetch is using: sub vcl_fetch { # *.jpg , *.jpeg , *.gif , *.png , *.ico if (req.url ~ ".(jpg|gif|png|jpeg|ico)$") { insert; } # *.js , *.css , *.swf if (req.url ~ ".(js|css|swf)$") { insert; } if (obj.http.Set-Cookie) { # *.jpg , *.jpeg , *.gif , *.png , *.ico if (req.url ~ ".(jpg|gif|png|jpeg|ico)$") { insert; } # *.js , *.css , *.swf if (req.url ~ ".(js|css|swf)$") { insert; } pass; } # pass; } #FETCH // Erik From phk at phk.freebsd.dk Fri Nov 16 14:12:00 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 16 Nov 2007 14:12:00 +0000 Subject: Debug: On Waiting list? In-Reply-To: Your message of "Fri, 16 Nov 2007 11:08:11 +0100." <00fkbog5bxjat75.161120071108@server2k7> Message-ID: <28866.1195222320@critter.freebsd.dk> In message <00fkbog5bxjat75.161120071108 at server2k7>, Erik writes: >Hi, >Im struggling with a couple of high response times when using > varnish on our site. I have configured varnish so it will > cache only static content (jpg,gif,png,css,js etc) even if a > cookie is set. But all our jpeg's are getting a high response > time when Im running some test against the site. >One of the jpeg's are pic_mini.jpg, it has a response time of 357ms. I turned on logging and vcl_trace and found the line: > > 14 Debug c "Hash Match: > /Portals/0/Picture/pic_mini.jpg#www3.mydomain.com#" > 14 Debug c "on waiting list on obj 1767384108" On waiting list means that somebody else asked for that object from the backend, and that this session now awaits that they get a response before it continues. If the other session gets a cacheable response, this session will return that cached reply, but if the result is not cacheable, this session will then have to go to the backend. The idea here, is that when an object is not in the cache, we should not send hordes of clients to the backend after it, if the first one can do the job alone. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From optilude at gmx.net Sat Nov 17 23:15:15 2007 From: optilude at gmx.net (Martin Aspeli) Date: Sat, 17 Nov 2007 23:15:15 +0000 Subject: Release status Message-ID: Hi, When are we likely to see a new stable Varnish release? Will 1.1.2 ever see the light of day? Unfortunately, Varnish 1.1.1 is unusable for me (stability issues); trunk was fine for a while, but I'm experiencing some more stability issues (and tracking trunk for production systems doesn't seem like a very good idea). I'd be very happy with a stable 1.1.2, or a stable and tested 2.0 if it's not too far away. Cheers, Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book From gslin at gslin.org Mon Nov 19 02:51:12 2007 From: gslin at gslin.org (Gea-Suan Lin) Date: Mon, 19 Nov 2007 10:51:12 +0800 Subject: Create a testbed to make varnishd core dump Message-ID: <20071119025112.GA86365@gslin.org> Hello, I create a testbed to make varnishd core dump under FreeBSD 6.2-STABLE amd64 version. First is install lighttpd, and put the following configuration into /usr/local/etc/lighttpd-testbed.conf: server.modules = ( "mod_rewrite", "mod_accesslog" ) server.document-root = "/tmp/webroot" server.errorlog = "/dev/null" index-file.names = ( "index.html" ) server.event-handler = "freebsd-kqueue" accesslog.filename = "/dev/null" server.port = 54321 server.bind = "127.0.0.1" server.pid-file = "/var/run/lighttpd-testbed.pid" server.username = "www" server.groupname = "www" url.rewrite = ( "^.*$" => "/4MB-zero" ) Then do something to create webroot: # mkdir /tmp/webroot # dd if=/dev/zero of=/tmp/webroot/4MB-zero bs=4096k count=1 You might able to use fetch or wget to test http://localhost:54321/foobar right now. Then install varnish, I choose /home/varnish. So I set backend VCL into /home/varnish/etc/local.vcl: backend default { set backend.host = "127.0.0.1"; set backend.port = "54321"; } And a script to run varnishd in port 12345 with 32GB cache: #!/bin/sh ulimit -c unlimited /home/varnish/sbin/varnishd -a 127.0.0.1:12345 -f /home/varnish/etc/local.vcl -h classic,1048583 -P /var/run/varnishd-testbed.pid -s file,/home/varnish/varnish-cache.mmap,32G -T 127.0.0.1:11957 -t 604800 -w 32,4096 Now you can run top, varnishstat, and everything you want. Then: # jot 4096 0 | sed 's|^|http://localhost:12345/|' | xargs -n 256 wget -O /dev/null I got the crash in every test: pid 1683 (varnishd), uid 65534: exited on signal 6 pid 1762 (varnishd), uid 65534: exited on signal 6 pid 3618 (varnishd), uid 65534: exited on signal 6 (core dumped) pid 35447 (varnishd), uid 0: exited on signal 6 (core dumped) pid 35586 (varnishd), uid 65534: exited on signal 3 pid 75708 (varnishd), uid 65534: exited on signal 3 By the way, I use Amazon EC2 64bits (Linux Xen 2.6.18) to test varnishd, it works perfectly. -- * Gea-Suan Lin (public key: Using https://keyserver.pgp.com/ to search) * If you cannot convince them, confuse them. -- Harry S Truman From jfbustarret at wat.tv Fri Nov 16 11:23:18 2007 From: jfbustarret at wat.tv (BUSTARRET, Jean-francois) Date: Fri, 16 Nov 2007 12:23:18 +0100 Subject: Support Cache-Control no-cache/private as per RFC2616 ? Message-ID: <53C652A09719C54DA24741D0157CB2695FF981@TFPRDEXS1.tf1.groupetf1.fr> "If the no-cache directive does not specify a field-name, then a cache MUST NOT use the response to satisfy a subsequent request without successful revalidation with the origin server. This allows an origin server to prevent caching even by caches that have been configured to return stale responses to client requests." http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9.1 varnish currently only uses max-age to check the cacheability of objects, and not Pragma: no-cache or Cache-Control: no-cache/private. This can be done with some vcl code, but IMHO vcl is not the right place. Why not fully support RFC2616 caching ? Jean-Fran?ois Bustarret WAT - Responsable technique http://www.wat.tv -------------- next part -------------- An HTML attachment was scrubbed... URL: From des at linpro.no Mon Nov 19 15:21:00 2007 From: des at linpro.no (=?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?=) Date: Mon, 19 Nov 2007 16:21:00 +0100 Subject: Support Cache-Control no-cache/private as per RFC2616 ? In-Reply-To: <53C652A09719C54DA24741D0157CB2695FF981@TFPRDEXS1.tf1.groupetf1.fr> (Jean-francois BUSTARRET's message of "Fri, 16 Nov 2007 12:23:18 +0100") References: <53C652A09719C54DA24741D0157CB2695FF981@TFPRDEXS1.tf1.groupetf1.fr> Message-ID: Already answered on varnish-dev. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Mon Nov 19 16:20:14 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 19 Nov 2007 16:20:14 +0000 Subject: Support Cache-Control no-cache/private as per RFC2616 ? In-Reply-To: Your message of "Fri, 16 Nov 2007 12:23:18 +0100." <53C652A09719C54DA24741D0157CB2695FF981@TFPRDEXS1.tf1.groupetf1.fr> Message-ID: <3936.1195489214@critter.freebsd.dk> In message <53C652A09719C54DA24741D0157CB2695FF981 at TFPRDEXS1.tf1.groupetf1.fr>, "BUSTARRET, Jean-francois" writes: >"If the no-cache directive does not specify a field-name, then a cache >MUST NOT [...] Varnish is not a cache in the RFC2616 sense. It more of an extension of the web-server, which is probably best thought of as "a webserver that picks up its contents with HTTP". Therefore, a lot of what RFC2616 has to say about caches do not apply to Varnish. The key distinction is that a RFC2616 cache is not under the content providers control, so it must follow whatever intructions he gives in the HTTP headers, whereas Varnish is under his control and he therefore has other means of instruction (notably VCL). -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Mon Nov 19 17:20:31 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 19 Nov 2007 17:20:31 +0000 Subject: Support Cache-Control no-cache/private as per RFC2616 ? In-Reply-To: Your message of "Mon, 19 Nov 2007 17:50:13 +0100." <53C652A09719C54DA24741D0157CB2695FF997@TFPRDEXS1.tf1.groupetf1.fr> Message-ID: <4333.1195492831@critter.freebsd.dk> In message <53C652A09719C54DA24741D0157CB2695FF997 at TFPRDEXS1.tf1.groupetf1.fr>, "BUSTARRET, Jean-francois" writes: > >Yet http://varnish.projects.linpro.no/wiki/FAQ says "Varnish was written = >from the ground up to be a high performance caching reverse proxy." = >Varnish is a cache, and should follow HTTP/1.1 RFCs. You're welcome to your opinion :-) We belive we have good arguments for the choices we have made, and as development continues, it is not inconceiveable that Varnish might some time in the future grow a "act like an RFC2616 cache", should somebody code the necessary changes or sponsor somebody else to do so. In the mean time, we try to get the maximum bang out of Varnish and think we have made the correct call on this point. One thing we have heard severalt times is that content providers want to be able to use Cache-Control for client instructions and not penalize their Varnish performance with its settings. There is a draft floating around which defines a "Surrogate-Control" along the lines of "Cache-Control" but it seems to have little backing and even less use. Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From jfbustarret at wat.tv Mon Nov 19 16:50:13 2007 From: jfbustarret at wat.tv (BUSTARRET, Jean-francois) Date: Mon, 19 Nov 2007 17:50:13 +0100 Subject: Support Cache-Control no-cache/private as per RFC2616 ? In-Reply-To: <3936.1195489214@critter.freebsd.dk> References: Your message of "Fri, 16 Nov 2007 12:23:18 +0100."<53C652A09719C54DA24741D0157CB2695FF981@TFPRDEXS1.tf1.groupetf1.fr> <3936.1195489214@critter.freebsd.dk> Message-ID: <53C652A09719C54DA24741D0157CB2695FF997@TFPRDEXS1.tf1.groupetf1.fr> Yet http://varnish.projects.linpro.no/wiki/FAQ says "Varnish was written from the ground up to be a high performance caching reverse proxy." Varnish is a cache, and should follow HTTP/1.1 RFCs. http://varnish.projects.linpro.no/wiki/HTTPFeatures also references these features : 355 MUST NOT 14.9.1 cache responses with cache-control: private 356 MUST NOT 14.9.1 use responses with cache-control: no-cache to satisfy other requests without successful revalidation I don't see a problem of having to have the Cache-control logic written in VCL, as long as : - the default configuration uses it (or the vcl code is in the man pages, not in google). - it is as fast as C code (vcl is compiled, so it's ok) - I don't have to use regexps. For now, I have to use two of them : (obj.http.Pragma ~ "no-cache" || obj.http.Cache-Control ~ "no-cache" - I am not using Cache-control: private), but two strpos-like would do the job, with less overhead. Jean-Fran?ois > -----Message d'origine----- > De : varnish-misc-bounces at projects.linpro.no > [mailto:varnish-misc-bounces at projects.linpro.no] De la part > de Poul-Henning Kamp > Envoy? : lundi 19 novembre 2007 17:20 > ? : BUSTARRET, Jean-francois > Cc : varnish-misc at projects.linpro.no > Objet : Re: Support Cache-Control no-cache/private as per RFC2616 ? > > In message > <53C652A09719C54DA24741D0157CB2695FF981 at TFPRDEXS1.tf1.groupetf1.fr>, > "BUSTARRET, Jean-francois" writes: > > >"If the no-cache directive does not specify a field-name, > then a cache > >MUST NOT [...] > > Varnish is not a cache in the RFC2616 sense. It more of an > extension of the web-server, which is probably best thought > of as "a webserver that picks up its contents with HTTP". > > Therefore, a lot of what RFC2616 has to say about caches do > not apply to Varnish. > > The key distinction is that a RFC2616 cache is not under the > content providers control, so it must follow whatever > intructions he gives in the HTTP headers, whereas Varnish is > under his control and he therefore has other means of > instruction (notably VCL). > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by > incompetence. > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc > From des at linpro.no Tue Nov 20 10:23:51 2007 From: des at linpro.no (=?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?=) Date: Tue, 20 Nov 2007 11:23:51 +0100 Subject: Support Cache-Control no-cache/private as per RFC2616 ? In-Reply-To: <4333.1195492831@critter.freebsd.dk> (Poul-Henning Kamp's message of "Mon, 19 Nov 2007 17:20:31 +0000") References: <4333.1195492831@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > There is a draft floating around which defines a "Surrogate-Control" > along the lines of "Cache-Control" but it seems to have little > backing and even less use. It has just as much backing and use as ESI; in fact, it is a prerequisite for ESI. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Tue Nov 20 10:53:18 2007 From: des at linpro.no (=?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?=) Date: Tue, 20 Nov 2007 11:53:18 +0100 Subject: Support Cache-Control no-cache/private as per RFC2616 ? In-Reply-To: <53C652A09719C54DA24741D0157CB2695FF997@TFPRDEXS1.tf1.groupetf1.fr> (Jean-francois BUSTARRET's message of "Mon, 19 Nov 2007 17:50:13 +0100") References: <53C652A09719C54DA24741D0157CB2695FF981@TFPRDEXS1.tf1.groupetf1.fr> <3936.1195489214@critter.freebsd.dk> <53C652A09719C54DA24741D0157CB2695FF997@TFPRDEXS1.tf1.groupetf1.fr> Message-ID: "BUSTARRET, Jean-francois" writes: > Yet http://varnish.projects.linpro.no/wiki/FAQ says "Varnish was > written from the ground up to be a high performance caching reverse > proxy." Varnish is a cache, and should follow HTTP/1.1 RFCs. Excuse me, but who are you to tell us what Varnish is or is not? Do you realize how arrogant that is? That aside, you are trying to fit Varnish into an RFC2616 pigeonhole, but there is no pigeonhole that fits - RFC2616 did not anticipate anything like Varnish. There is a draft W3 specification, the Edge Architecture Specification, which attempts to fill that hole, but it is not widely known, so I'm not sure it would help much to write that Varnish is an HTTP surrogate rather than an HTTP accelerator (I try to avoid the term "reverse proxy"). DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Tue Nov 20 17:19:16 2007 From: des at linpro.no (=?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?=) Date: Tue, 20 Nov 2007 18:19:16 +0100 Subject: Support Cache-Control no-cache/private as per RFC2616 ? In-Reply-To: <53C652A09719C54DA24741D0157CB2695FF99F@TFPRDEXS1.tf1.groupetf1.fr> (Jean-francois BUSTARRET's message of "Tue, 20 Nov 2007 14:49:16 +0100") References: <53C652A09719C54DA24741D0157CB2695FF981@TFPRDEXS1.tf1.groupetf1.fr> <53C652A09719C54DA24741D0157CB2695FF997@TFPRDEXS1.tf1.groupetf1.fr> <200711200630.56746.rudd-o@rudd-o.com> <53C652A09719C54DA24741D0157CB2695FF99F@TFPRDEXS1.tf1.groupetf1.fr> Message-ID: "BUSTARRET, Jean-francois" writes: > IMHO, this is mainly a documentation problem. Why not : > - remove the term "reverse proxy" from the FAQ and replace it by "HTTP Accelerator", > - describe exactly what/when varnish caches by default, > - describe how to build a RFC2616 reverse proxy, and bundle a sample vcl with varnish. > > As Poul-Henning told, someone should contribute/sponsor this. Having > more time than money (and feeling not 100% welcome here), I can > write some documentation if you agree. Documentation is always welcome. The wiki needs a lot of work, and both Poul-Henning and I have far too much to do (both with Varnish and other projects) to be able to spend much time on it. BTW, could I suggest that you subscribe to varnish-misc or varnish-dev (or both) so I don't have to manually approve everything you send to the list? DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From jfbustarret at wat.tv Tue Nov 20 13:49:16 2007 From: jfbustarret at wat.tv (BUSTARRET, Jean-francois) Date: Tue, 20 Nov 2007 14:49:16 +0100 Subject: Support Cache-Control no-cache/private as per RFC2616 ? In-Reply-To: <200711200630.56746.rudd-o@rudd-o.com> References: <53C652A09719C54DA24741D0157CB2695FF981@TFPRDEXS1.tf1.groupetf1.fr><53C652A09719C54DA24741D0157CB2695FF997@TFPRDEXS1.tf1.groupetf1.fr> <200711200630.56746.rudd-o@rudd-o.com> Message-ID: <53C652A09719C54DA24741D0157CB2695FF99F@TFPRDEXS1.tf1.groupetf1.fr> I won't reply to the first part of DES message... I understand the point, and, as I said before, I really like the VCL approach. The real problem is that, for a new varnish user, it is difficult to understand what varnish really is. A new user would read the FAQ and think that varnish is a reverse proxy, but, as said, it isn't (out of the box, I agree that a reverse proxy is only a few lines of VCL code away). IMHO, this is mainly a documentation problem. Why not : - remove the term "reverse proxy" from the FAQ and replace it by "HTTP Accelerator", - describe exactly what/when varnish caches by default, - describe how to build a RFC2616 reverse proxy, and bundle a sample vcl with varnish. As Poul-Henning told, someone should contribute/sponsor this. Having more time than money (and feeling not 100% welcome here), I can write some documentation if you agree. Jean-Fran?ois > -----Message d'origine----- > De : varnish-dev-bounces at projects.linpro.no > [mailto:varnish-dev-bounces at projects.linpro.no] De la part de > Manuel Amador (Rudd-O) > Envoy? : mardi 20 novembre 2007 12:31 > ? : varnish-dev at projects.linpro.no > Objet : Re: Support Cache-Control no-cache/private as per RFC2616 ? > > Do you really need to snap at contributors, Dag-Erling? I > happen to agree with him in the sense that pulling Varnish > (VCL or *not*) in the direction of a complete > standards-compliant configuration-free smart Web accelerator > is a very good idea. > > The grandfather poster may be a bit misguided as to what > standards Varnish would need to comply to, but your comment > would have easily been more constructive if you just had > limited yourself to the Edge mention. > > And, of course, you are already publicly aware of my position > regarding the matter. > > An idea: We need a matrix (OK, a nice table) or a decision > tree of what actions Varnish should take by default, given a > set of requests and content of varied freshness degrees. > Once that work is done, moving VCL (and the default VCL > config) in the direction that will allow us to actually > produce that decision tree, should be much easier and clearer. > > El Mar 20 Nov 2007, Dag-Erling Sm?rgrav escribi?: > > "BUSTARRET, Jean-francois" writes: > > > Yet http://varnish.projects.linpro.no/wiki/FAQ says "Varnish was > > > written from the ground up to be a high performance > caching reverse > > > proxy." Varnish is a cache, and should follow HTTP/1.1 RFCs. > > > > Excuse me, but who are you to tell us what Varnish is or is > not? Do > > you realize how arrogant that is? > > > > That aside, you are trying to fit Varnish into an RFC2616 > pigeonhole, > > but there is no pigeonhole that fits - RFC2616 did not anticipate > > anything like Varnish. There is a draft W3 specification, the Edge > > Architecture Specification, which attempts to fill that > hole, but it > > is not widely known, so I'm not sure it would help much to > write that > > Varnish is an HTTP surrogate rather than an HTTP > accelerator (I try to > > avoid the term "reverse proxy"). > > > > DES > > > > -- > > Manuel Amador (Rudd-O) > Rudd-O.com - http://rudd-o.com/ > GPG key ID 0xC8D28B92 at http://wwwkeys.pgp.net/ > > Now playing, courtesy of Amarok: Voodoo & Serano - Blood is > pumping So this is it. We're going to die. > From jfbustarret at wat.tv Tue Nov 20 17:30:51 2007 From: jfbustarret at wat.tv (BUSTARRET, Jean-francois) Date: Tue, 20 Nov 2007 18:30:51 +0100 Subject: Support Cache-Control no-cache/private as per RFC2616 ? In-Reply-To: References: <53C652A09719C54DA24741D0157CB2695FF981@TFPRDEXS1.tf1.groupetf1.fr><53C652A09719C54DA24741D0157CB2695FF997@TFPRDEXS1.tf1.groupetf1.fr><200711200630.56746.rudd-o@rudd-o.com><53C652A09719C54DA24741D0157CB2695FF99F@TFPRDEXS1.tf1.groupetf1.fr> Message-ID: <53C652A09719C54DA24741D0157CB2695FF9A4@TFPRDEXS1.tf1.groupetf1.fr> > -----Message d'origine----- > De : Dag-Erling Sm?rgrav [mailto:des at linpro.no] > Documentation is always welcome. The wiki needs a lot of > work, and both Poul-Henning and I have far too much to do > (both with Varnish and other projects) to be able to spend > much time on it. I'll send something next week on the list > BTW, could I suggest that you subscribe to varnish-misc or > varnish-dev (or both) so I don't have to manually approve > everything you send to the list? Sorry for that. I'll never get used to what Exchange does to emails... This should be ok. Jean-Fran?ois From des at linpro.no Tue Nov 20 17:42:20 2007 From: des at linpro.no (=?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?=) Date: Tue, 20 Nov 2007 18:42:20 +0100 Subject: Support Cache-Control no-cache/private as per RFC2616 ? In-Reply-To: <53C652A09719C54DA24741D0157CB2695FF9A4@TFPRDEXS1.tf1.groupetf1.fr> (Jean-francois BUSTARRET's message of "Tue, 20 Nov 2007 18:30:51 +0100") References: <53C652A09719C54DA24741D0157CB2695FF981@TFPRDEXS1.tf1.groupetf1.fr> <53C652A09719C54DA24741D0157CB2695FF997@TFPRDEXS1.tf1.groupetf1.fr> <200711200630.56746.rudd-o@rudd-o.com> <53C652A09719C54DA24741D0157CB2695FF99F@TFPRDEXS1.tf1.groupetf1.fr> <53C652A09719C54DA24741D0157CB2695FF9A4@TFPRDEXS1.tf1.groupetf1.fr> Message-ID: "BUSTARRET, Jean-francois" writes: > Dag-Erling Sm?rgrav writes: > > BTW, could I suggest that you subscribe to varnish-misc or > > varnish-dev (or both) so I don't have to manually approve > > everything you send to the list? > Sorry for that. I'll never get used to what Exchange does to > emails... This should be ok. No, you are posting from a different address than that with which you subscribed. The easiest fix is probably to move your subscription over to the address you are posting from. Alternatively, you can subscribe with both addresses and disable one of them so you don't get everything twice; you will then be able to post to the lists using either address. Let me know which you prefer, and I'll take care of it for you. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From duja at torlen.net Wed Nov 21 15:21:50 2007 From: duja at torlen.net (Erik) Date: Wed, 21 Nov 2007 16:21:50 +0100 Subject: Varnish crashes everyday and same time Message-ID: Hi, Varnish crashes everyday, same time. This is what I got from the log files: 12 SessionOpen c xx.xx.xx.xx 31989 0 Debug "Acceptor is epoll" 0 Error CLI read 0 (errno=32) I also found this thread in the mailing archive from July: http://projects.linpro.no/pipermail/varnish-misc/2007-July/000670html That is the last post on that subject, no answer was posted to Anup Shukla. I dont know if that is the same problem but it has the same error in one of the posts. Im running varnish compiled from source on Debian 4.0 Etch. Im gonna start a logjob of the memory to see if thats its the problem. / Erik From chulmin2 at hotmail.com Thu Nov 22 09:19:06 2007 From: chulmin2 at hotmail.com (MontyRee) Date: Thu, 22 Nov 2007 09:19:06 +0000 Subject: varnishd eats CPU more and more.. Message-ID: Hello, all. I have setup varnishd 1.1.1 at several linux servers.(centos 3.x, all versions are same) and works right without any problem except one server. at that server, varnishd process eats the %CPU more and more and grows continuously like below. first %CPU is just 1%, after 10 minutes 10%. after 20 minutes 20%. ...... after one hour 104% like below, load average is over 60 .... USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND nobody 24853 104 2.7 105212236 219976 ? Tl 16:49 81:16 /usr/local/sbin/varnishd -T 127.0.0.1:6082 -f /etc/varnish/vcl.conf -P /var/run/varnish/varn There are just 10Mbps and there is not lots of visitors. How can I solve this problem?? Regards. _________________________________________________________________ MSN ???? ??? ??, Windows Live Messenger! http://windowslive.msn.co.kr/wlm/messenger/ From duja at torlen.net Thu Nov 22 10:18:53 2007 From: duja at torlen.net (Erik) Date: Thu, 22 Nov 2007 11:18:53 +0100 Subject: Varnish crashes everyday and same time Message-ID: <6gdgbgsosm583z3.221120071118@server2k7> Hi again, I made some logging of the memory and it looks fine. I also turned of VCL Trace but that didn't solved it. The crash happened again today but a few hours later then usual. I tried to start varnishd in debug mode but I cant get it to work. When I set it to -d or -d -d it starts but no connection can be made against it. Any ideas? I forgot to mention that Im running varnish on a Virtual Server 2005 with 512 MB RAM (150 MB free) and 10 GB diskspace. / Erik From janis.putrams at delfi.lv Thu Nov 22 11:23:43 2007 From: janis.putrams at delfi.lv (Janis Putrams) Date: Thu, 22 Nov 2007 13:23:43 +0200 Subject: Varnish crashes everyday and same time In-Reply-To: <6gdgbgsosm583z3.221120071118@server2k7> References: <6gdgbgsosm583z3.221120071118@server2k7> Message-ID: <200711221323.43543.janis.putrams@delfi.lv> you need to write "start" after starting varnishd in debug mode. janis a satisfied Varnishd user :) On Thursday 22 November 2007 12:18, Erik wrote: > Hi again, > > I made some logging of the memory and it looks fine. I also turned of VCL > Trace but that didn't solved it. The crash happened again today but a few > hours later then usual. I tried to start varnishd in debug mode but I cant > get it to work. When I set it to -d or -d -d it starts but no connection > can be made against it. Any ideas? > > I forgot to mention that Im running varnish on a Virtual Server 2005 with > 512 MB RAM (150 MB free) and 10 GB diskspace. > > / Erik > > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc From duja at torlen.net Thu Nov 22 13:55:16 2007 From: duja at torlen.net (Erik) Date: Thu, 22 Nov 2007 14:55:16 +0100 Subject: Varnish crashes everyday and same time Message-ID: <6c1ra2fqxe6hdkn.221120071455@server2k7> Hi, This is what I do and what I get: /etc/init.d/varnish start Starting Varnish: varnishUsing old SHMFILE rolling(2)... It seems to me that the varnish is running? But when trying to connect it doesn't work! Althought when I run without -d -d or -d it works! I would really like to commit some logdata from varnishd but since I cant get the debug to work it has to wait :( / Erik Original Message ----------------------- you need to write "start" after starting varnishd in debug mode. janis a satisfied Varnishd user :) On Thursday 22 November 2007 12:18, Erik wrote: > Hi again, > > I made some logging of the memory and it looks fine. I also turned of VCL > Trace but that didn't solved it. The crash happened again today but a few > hours later then usual. I tried to start varnishd in debug mode but I cant > get it to work. When I set it to -d or -d -d it starts but no connection > can be made against it. Any ideas? > > I forgot to mention that Im running varnish on a Virtual Server 2005 with > 512 MB RAM (150 MB free) and 10 GB diskspace. > > / Erik > > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc _______________________________________________ varnish-misc mailing list varnish-misc at projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc From phk at phk.freebsd.dk Fri Nov 23 10:08:34 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 23 Nov 2007 10:08:34 +0000 Subject: varnishd eats CPU more and more.. In-Reply-To: Your message of "Thu, 22 Nov 2007 09:19:06 GMT." Message-ID: <3878.1195812514@critter.freebsd.dk> I'm not very linux-clued, but if you have a facility like FreeBSD's ktrace(1) which logs which system calls a process does, a few hundred lines of output from that would be useful. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From jfbustarret at wat.tv Fri Nov 23 16:07:17 2007 From: jfbustarret at wat.tv (BUSTARRET, Jean-francois) Date: Fri, 23 Nov 2007 17:07:17 +0100 Subject: [varnish] Defining varnish + proposals for the FAQ page Message-ID: <53C652A09719C54DA24741D0157CB2695FF9C0@TFPRDEXS1.tf1.groupetf1.fr> As promised, here is a proposal for a new definition of what varnish is for the FAQ (links in <>, changes in bold, comments in []). I did compile some mails exchanged the last days on varnish-dev, credit for this goes to many people. My mother tongue not being english, there might be quite a few spelling/grammatical errors. What is varnish ? [new] Varnish is a HTTP accelerator : a high performance HTTP server to be put in front of a slow one (like apache) to speed up an existing website. Rather than a true reverse proxy (as in ), varnish is a "HTTP surrogate", as defined in the w3c draft. Varnish can be very easily adapted to any website, and with a few lines of configuration [link to the section in the FAQ, below]. Why bother with Varnish - why not use Squid ? [modified] Varnish was written from the ground up to be a high performance HTTP accelerator. Squid is a forward proxy that can be configured as a reverse proxy, that, as a side effect, can provide HTTP accelerating features. Besides - Squid is rather old, bloated and designed like computer programs where supposed to be designed in 1980. Please see > for details. How do I configure varnish to act like a classical reverse proxy ? [new] Here is a sample VCL to do this : backend default { set backend.host = "backend.example.com"; set backend.port = "http"; } sub vcl_recv { if (req.request != "GET" && req.request != "HEAD") { pipe; } if (req.http.Expect) { pipe; } if (req.http.Authenticate) { pass; } if (req.http.Cache-Control ~ "no-cache") { pass; } lookup; } sub vcl_pipe { pipe; } sub vcl_pass { pass; } sub vcl_hash { set req.hash += req.url; set req.hash += req.http.host; hash; } sub vcl_hit { if (!obj.cacheable) { pass; } deliver; } sub vcl_miss { fetch; } sub vcl_fetch { if (!obj.valid) { error; } if (!obj.cacheable) { pass; } if (obj.http.Set-Cookie) { pass; } if (obj.http.Pragma ~ "no-cache" || obj.http.Cache-Control ~ "(no-cache|private)" ) { pass; } insert; } sub vcl_deliver { deliver; } sub vcl_timeout { discard; } sub vcl_discard { discard; } What do you think of this ? Jean-Fran?ois Bustarret WAT - Responsable technique http://www.wat.tv -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Fri Nov 23 21:09:14 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 23 Nov 2007 21:09:14 +0000 Subject: [varnish] Defining varnish + proposals for the FAQ page In-Reply-To: Your message of "Fri, 23 Nov 2007 17:07:17 +0100." <53C652A09719C54DA24741D0157CB2695FF9C0@TFPRDEXS1.tf1.groupetf1.fr> Message-ID: <7993.1195852154@critter.freebsd.dk> In message <53C652A09719C54DA24741D0157CB2695FF9C0 at TFPRDEXS1.tf1.groupetf1.fr>, "BUSTARRET, Jean-francois" writes: >How do I configure varnish to act like a classical reverse proxy ? [new] > >Here is a sample VCL to do this : There is one aspect of VCL that I need to communicate better: You don't need to copy the default VCL into your vcl files. The way the compiler works is that all definitions of each function are concatenated, and the default function comes last. So if for instance you only write in your VCL code: sub vcl_recv { if (req.url ~ "[.]exe") { error 500 "bugger off"; } } the compiler will append to this, the default vcl code for vcl_recv. All the piping/passing of stuff will happen automatically don't need to put that code in there. The reason I have written the compiler this way, is so that users would only need their own specific code. That way, the default code can change without annoying the users along the way. But if you as a user, include the entire default code in your VCL program, you will have to integrate, by hand, any subsequent changes to the default VCL code yourself. Also, if you want to make varnish behave as much as a RFC2616 proxy as it can, you do not need a VCL program at all, that is, more or less exactly, what the default VCL code is written against: The Principle Of Least Astonishment. Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From jfbustarret at wat.tv Mon Nov 26 08:31:36 2007 From: jfbustarret at wat.tv (BUSTARRET, Jean-francois) Date: Mon, 26 Nov 2007 09:31:36 +0100 Subject: [varnish] Defining varnish + proposals for the FAQ page In-Reply-To: <7993.1195852154@critter.freebsd.dk> References: Your message of "Fri, 23 Nov 2007 17:07:17 +0100." <53C652A09719C54DA24741D0157CB2695FF9C0@TFPRDEXS1.tf1.groupetf1.fr> <7993.1195852154@critter.freebsd.dk> Message-ID: <53C652A09719C54DA24741D0157CB2695FF9C8@TFPRDEXS1.tf1.groupetf1.fr> > -----Message d'origine----- > In message > <53C652A09719C54DA24741D0157CB2695FF9C0 at TFPRDEXS1.tf1.groupetf1.fr>, > "BUSTARRET, Jean-francois" writes: > > >How do I configure varnish to act like a classical reverse proxy ? > >[new] > > > >Here is a sample VCL to do this : [...] > Also, if you want to make varnish behave as much as a RFC2616 > proxy as it can, you do not need a VCL program at all, that > is, more or less exactly, what the default VCL code is > written against: The Principle Of Least Astonishment. > > Poul-Henning Well, that is not the case : the default VCL (and varnishd) does not support "Cache-control: no-cache/private" and "Pragma: no-cache" (which, IMHO, a "classical reverse proxy" should support)... So you need to add some VCL logic. Otherwise, I totally agree with "The Principle Of Least Astonishment". Jean-Fran?ois