From dwetzel at nerim.net Sun Apr 1 06:55:03 2007 From: dwetzel at nerim.net (dwetzel at nerim.net) Date: Sun, 1 Apr 2007 08:55:03 +0200 (CEST) Subject: Delay when fetching a page without Content-Length In-Reply-To: <87648hffku.fsf@topper.koldfront.dk> References: <87abxtfk1x.fsf@topper.koldfront.dk> <81364.1175361430@critter.freebsd.dk> <87648hffku.fsf@topper.koldfront.dk> Message-ID: <3176.82.239.3.208.1175410503.squirrel@webmail.nerim.net> Hello, I had a smilar issue testing varnish last friday, the backend was sending a 302 with a Transfert-Encoding:chunked, Using tcpdump i noticed that varnish was trying to open a new tcp connection with the backend, 3 attempts were made (delay ~15s) before varnish gave up and forwarded back the 302 to the client. there were no authorisation header. Best Regards, > On Sat, 31 Mar 2007 17:17:10 +0000, Poul-Henning wrote: > >> Which version of Varnish are you running ? > > The one from Debian unstable backported to stable, i.e. 1.0.3: > > $ /usr/sbin/varnishd -V > varnishd (varnish-1.0.3) > Copyright (c) 2006 Linpro AS / Verdens Gang AS > $ > >> Can you send me the output from varnishlog for the transaction ? > > Sure: > > $ varnishlog > 0 CLI Rd ping > 0 CLI Wr 0 200 PONG 1175364221 > 13 SessionOpen c 192.168.1.160 52579 > 0 CLI Rd ping > 0 CLI Wr 0 200 PONG 1175364224 > 0 CLI Rd ping > 0 CLI Wr 0 200 PONG 1175364227 > 0 CLI Rd ping > 0 CLI Wr 0 200 PONG 1175364230 > 0 CLI Rd ping > 0 CLI Wr 0 200 PONG 1175364233 > 0 CLI Rd ping > 0 CLI Wr 0 200 PONG 1175364236 > 13 ReqStart c 192.168.1.160 52579 1997137143 > 13 RxRequest c GET > 13 RxURL c /stats/ > 13 RxProtocol c HTTP/1.0 > 13 RxHeader c User-Agent: Wget/1.10.2 > 13 RxHeader c Accept: */* > 13 RxHeader c Host: www.koldfront.local > 13 RxHeader c Connection: Keep-Alive > 13 VCL_call c recv > 13 VCL_return c lookup > 13 VCL_call c miss > 13 VCL_return c fetch > 17 BackendOpen b default 127.0.0.1 1027 127.0.0.1 8080 > 17 BackendXID b 1997137143 > 13 Backend c 17 default > 17 TxRequest b GET > 17 TxURL b /stats/ > 17 TxProtocol b HTTP/1.1 > 17 TxHeader b User-Agent: Wget/1.10.2 > 17 TxHeader b Accept: */* > 17 TxHeader b Host: www.koldfront.local > 17 TxHeader b X-Varnish: 1997137143 > 17 TxHeader b X-Forwarded-for: 192.168.1.160 > 17 RxProtocol b HTTP/1.1 > 17 RxStatus b 401 > 17 RxResponse b Authorization Required > 17 RxHeader b Date: Sat, 31 Mar 2007 18:03:42 GMT > 17 RxHeader b Server: Apache/1.3.33 (Debian GNU/Linux) > mod_fastcgi/2.4.2 mod_perl/1.29 > 17 RxHeader b WWW-Authenticate: Basic realm="Statistics" > 17 RxHeader b Transfer-Encoding: chunked > 17 RxHeader b Content-Type: text/html; charset=iso-8859-1 > 13 TTL c 1997137143 RFC 120 1175364222 1175364222 0 0 0 > 13 VCL_call c fetch > 13 VCL_return c pass > 13 RxProtocol c HTTP/1.1 > 13 RxStatus c 401 > 13 RxResponse c Authorization Required > 13 RxHeader c Date: Sat, 31 Mar 2007 18:03:42 GMT > 13 RxHeader c Server: Apache/1.3.33 (Debian GNU/Linux) > mod_fastcgi/2.4.2 mod_perl/1.29 > 13 RxHeader c WWW-Authenticate: Basic realm="Statistics" > 13 RxHeader c Content-Type: text/html; charset=iso-8859-1 > 13 RxHeader c X-Varnish: 1997137143 > 13 RxHeader c X-Forwarded-for: 192.168.1.160 > 13 RxHeader c Transfer-Encoding: chunked > 17 BackendReuse b default > 13 ReqEnd c 1997137143 1175364222.294967044 1175364238.516689601 > 0.000751112 0.001383653 16.220338904 > 0 StatAddr 192.168.1.160 0 8161 36 37 0 20 11 11486 92507 > 13 HttpError c Received errno 104 > 13 SessionClose c no request > 13 StatSess c 192.168.1.160 52579 16 1 1 0 1 0 322 529 > 0 CLI Rd ping > 0 CLI Wr 0 200 PONG 1175364239 > 0 CLI Rd ping > 0 CLI Wr 0 200 PONG 1175364242 > ^C > $ > > (If I run ngrep I see the request being sent to Apache and the reply > back immediately, and then the pause). > > > Let me know if I should supply more information. > > > Best regards, > > Adam > > -- > "we push onward. to you, it is 2005, to us, it is Adam Sj?gren > 2011. we are always far ahead." asjo at koldfront.dk > > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc > From olivier.dobberkau at dkd.de Sun Apr 1 17:42:09 2007 From: olivier.dobberkau at dkd.de (Olivier Dobberkau) Date: Sun, 01 Apr 2007 19:42:09 +0200 Subject: Questions about Setup In-Reply-To: <19969305.21711175334512466.JavaMail.root@ms1.startsiden.no> Message-ID: am 31.03.2007 11:48 Uhr schrieb Denis Br?khus unter denis at startsiden.no: > Maybe it would be better if you asked questions about the things you do not > understand? Our config is mainly the normal config, add the backend block, and > some specific exceptions for site urls we are sure we do not want cached in > any way. I meant that it would be nice to have some Example Config for us newbies... Olivier -- Olivier Dobberkau . . . . . . . . . . . . . . Je TYPO3, desto d.k.d d.k.d Internet Service GmbH Kaiserstr. 79 60329 Frankfurt/Main Registergericht: Amtsgericht Frankfurt am Main Registernummer: HRB 45590 Gesch?ftsf?hrer: Olivier Dobberkau, G?tz Wegenast. fon: 069 - 43 05 61-70 fax: 069 - 43 05 61-90 mail: olivier.dobberkau at dkd.de home: http://www.dkd.de aktuelle TYPO3-Projekte: www.dosb.de - Relaunch www.gesundheit.de - Relaunch www.wwf.de - barrierefrei, Relaunch From phk at phk.freebsd.dk Sun Apr 1 18:12:51 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sun, 01 Apr 2007 18:12:51 +0000 Subject: Questions about Setup In-Reply-To: Your message of "Sun, 01 Apr 2007 19:42:09 +0200." Message-ID: <72450.1175451171@critter.freebsd.dk> In message , Olivier Dobberkau writes: >am 31.03.2007 11:48 Uhr schrieb Denis Br=E6khus unter denis at startsiden.no: > >> Maybe it would be better if you asked questions about the things you do n= >ot >> understand? Our config is mainly the normal config, add the backend block= >, and >> some specific exceptions for site urls we are sure we do not want cached = >in >> any way. > >I meant that it would be nice to have some Example Config for us newbies... The example you are looking for, is the default VCL code which is used if you don't specify your own VCL code. Right now, that is not obviously available, but look in bin/varnishd/mgt_vcc.c -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Wed Apr 4 09:15:42 2007 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Wed, 04 Apr 2007 11:15:42 +0200 Subject: log files In-Reply-To: <17933.5249.247592.184781@dwetzel@nerim.net> (Damien Wetzel's message of "Fri, 30 Mar 2007 15:45:37 +0200") References: <17931.59232.890894.380967@dwetzel@nerim.net> <17931.64543.652972.864603@dwetzel@nerim.net> <17933.5249.247592.184781@dwetzel@nerim.net> Message-ID: Damien Wetzel writes: > Dag-Erling Sm?rgrav writes: > > Damien Wetzel writes: > > > i think it could be interresting to add a field in the ncsa logs > > > telling if the object was delivered from the cache or had to be > > > fetched. Is there a way to get w3C log ? > > That's what varnishncsa does. > i haven't seen any fields using varnishncsa which refers to cache > hits or miss the man talk about the possibility to get the logs in > combined format but i don't how to do it The output of varnishncsa *is* the so-called combined log format. It contains no information on cache hits / misses. > > > and last is there a ay to remove ping pong lines from the logs ? > > All the information you need is in the man page. > it's my bad the man pages weren't installed after the make install > command. They most likely were. Perhaps your MANPATH is wrong? > PS: I don't think ESI for a v2 are a good addon. ESI is mainly an > akamai tool to catch customers but is not of real use. the standard > hasn't changed since 2002 and doesn't seem to be widely used. You are mistaken. ESI is widely used by large online news outlets and portal sites, including some of Linpro's largest customers. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Wed Apr 4 09:16:48 2007 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Wed, 04 Apr 2007 11:16:48 +0200 Subject: Questions about Setup In-Reply-To: <72450.1175451171@critter.freebsd.dk> (Poul-Henning Kamp's message of "Sun, 01 Apr 2007 18:12:51 +0000") References: <72450.1175451171@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > The example you are looking for, is the default VCL code which is used if > you don't specify your own VCL code. > > Right now, that is not obviously available, but look in > bin/varnishd/mgt_vcc.c Actually, it is reproduced in full in the man page, which also has examples of useful non-default configurations. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Wed Apr 4 09:27:04 2007 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Wed, 04 Apr 2007 11:27:04 +0200 Subject: Virtual hosts and logfiles In-Reply-To: <87mz1tfnip.fsf@topper.koldfront.dk> (Adam =?iso-8859-1?Q?Sj?= =?iso-8859-1?Q?=F8gren's?= message of "Sat, 31 Mar 2007 17:20:14 +0200") References: <87mz1tfnip.fsf@topper.koldfront.dk> Message-ID: asjo at koldfront.dk (Adam Sj?gren) writes: > I wonder if anyone has solved how to make varnishncsa only log > requests to a specific virtualhost? (Then I would just run a > varnishncsa for each virtualhost; problem solved). > > Looking at the man-page, I'm guessing that the '-i' flag perhaps could > be used, but I haven't been able to dig up what "tag" means in this > context. Maybe I'm way off. > > Any pointers where to look? varnishncsa isn't very good at filtering logs, but you can use varnishlog to filter the logs, and run varnishncsa on the output: des at dma ~% varnishlog -w /dev/stdout -c RxHeader '^Host: tinderbox.des.no' | varnishncsa -r /dev/stdin 10.0.11.2 - - [04/Apr/2007:11:24:13 +0200] "GET / HTTP/1.1" 304 - "-" "Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.8.0.5) Gecko/20060727 Firefox/1.5.0.5" 10.0.11.2 - - [04/Apr/2007:11:24:13 +0200] "GET /tb.css HTTP/1.1" 304 - "http://tinderbox.des.no/" "Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.8.0.5) Gecko/20060727 Firefox/1.5.0.5" If you don't need continuously updated NCSA logs, it's probably simpler to just use varnishlog to store raw binary logs to separate files for each virtual hosts, and run varnishncsa on those files later. Ideally, "varnishncsa RxHeader '^Host: tinderbox.des.no'" would do what you want, but we aren't quite there yet. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Wed Apr 4 09:31:14 2007 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Wed, 04 Apr 2007 11:31:14 +0200 Subject: Delay when fetching a page without Content-Length In-Reply-To: <87veghdzw0.fsf@topper.koldfront.dk> (Adam =?iso-8859-1?Q?Sj?= =?iso-8859-1?Q?=F8gren's?= message of "Sat, 31 Mar 2007 20:35:59 +0200") References: <87648hffku.fsf@topper.koldfront.dk> <86144.1175365116@critter.freebsd.dk> <87veghdzw0.fsf@topper.koldfront.dk> Message-ID: asjo at koldfront.dk (Adam Sj?gren) writes: > (Where can I read about meaning of these? I have scanned through > vcl(7), varnishd(1) and the FAQ). The latest version of vcl(7) in Subversion describes this in far better detail than the version in 1.0.3: http://varnish.projects.linpro.no/svn/trunk/varnish-cache/man/vcl.7 DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From zimbatm at oree.ch Wed Apr 4 10:35:40 2007 From: zimbatm at oree.ch (Jonas Pfenniger) Date: Wed, 4 Apr 2007 12:35:40 +0200 Subject: UNIX Sockets support ? Message-ID: Hello, I just discovered Varnish and wanted to know if it supports proxying to UNIX sockets. I've already looked in the man pages so I expect it's not available. The use case I would have for, is use varnish as a frontend on a multiple-users webserver, where some paths are mapped to user-owned file sockets. If classical sockets are used, I cannot guarantee that user A won't take user B's port if user B's webserver crashes. Example : proxy path ~ /~([\w_]) to /home/$1/web.socket -- Cheers, zimbatm From phk at phk.freebsd.dk Wed Apr 4 10:55:39 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 04 Apr 2007 10:55:39 +0000 Subject: UNIX Sockets support ? In-Reply-To: Your message of "Wed, 04 Apr 2007 12:35:40 +0200." Message-ID: <1143.1175684139@critter.freebsd.dk> In message , "Jona s Pfenniger" writes: >Hello, > >I just discovered Varnish and wanted to know if it supports proxying >to UNIX sockets. I've already looked in the man pages so I expect it's >not available. Right now: no. Can we do it ? Sure. Can I get you to open a trac-ticket on our web-page where you describe how you would like to see it work ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dwetzel at nerim.net Wed Apr 4 13:03:55 2007 From: dwetzel at nerim.net (Damien Wetzel) Date: Wed, 4 Apr 2007 15:03:55 +0200 Subject: log files In-Reply-To: References: <17931.59232.890894.380967@dwetzel@nerim.net> <17931.64543.652972.864603@dwetzel@nerim.net> <17933.5249.247592.184781@dwetzel@nerim.net> Message-ID: <17939.41531.500034.358155@dwetzel@nerim.net> Dag-Erling Sm?rgrav writes: > Damien Wetzel writes: > > Dag-Erling Sm?rgrav writes: > > > Damien Wetzel writes: > > > > i think it could be interresting to add a field in the ncsa logs > > > > telling if the object was delivered from the cache or had to be > > > > fetched. Is there a way to get w3C log ? > > > That's what varnishncsa does. > > i haven't seen any fields using varnishncsa which refers to cache > > hits or miss the man talk about the possibility to get the logs in > > combined format but i don't how to do it > > The output of varnishncsa *is* the so-called combined log format. It > contains no information on cache hits / misses. i believe there would be a way to get W3C logs which are widely used in writing a specific program. I still think that information about hits/misses in the logs would be usefull. > > PS: I don't think ESI for a v2 are a good addon. ESI is mainly an > > akamai tool to catch customers but is not of real use. the standard > > hasn't changed since 2002 and doesn't seem to be widely used. > > You are mistaken. ESI is widely used by large online news outlets and > portal sites, including some of Linpro's largest customers. For me ESI are so web 1.0 , now the tendency is to deport the personalisation on the client side. Damien, > > DES > -- > Dag-Erling Sm?rgrav > Senior Software Developer > Linpro AS - www.linpro.no From dwetzel at nerim.net Wed Apr 4 13:51:16 2007 From: dwetzel at nerim.net (Damien Wetzel) Date: Wed, 4 Apr 2007 15:51:16 +0200 Subject: log files In-Reply-To: <28139179.3891175692737360.JavaMail.root@ms1.startsiden.no> References: <6849168.3871175692709549.JavaMail.root@ms1.startsiden.no> <28139179.3891175692737360.JavaMail.root@ms1.startsiden.no> Message-ID: <17939.44372.537796.286223@dwetzel@nerim.net> Denis Br??khus writes: > ----- Damien Wetzel wrote: > And you do have that information, however not in the ncsa logs, due to the fact that the combined logformat does not have a field for that.. --> at the contrary of ncsa logs ,w3c logs are extensible, parametrable and allow the addition of such a field. > Use varnishlog for that information. > > > For me ESI are so web 1.0 , now the tendency is to deport the > > personalisation on the client side. > > Eh.. If you do believe that all web applications from this day on will be javascript apps where all the content handling is on the client side, good for you. Personally I think there will be a large number of sites / applications where that will not apply. If we can have ESI in Varnish that sounds like an excellent feature to me. We might or might not use it, depending on the work to be done. Everyone can have its own opinion, mine is that the server side should do a minimum of work and let the client do it if possible ;) > > Writing it off as "web 1.0" isn't very constructive. > > Regards > -- > Denis Braekhus - Teknisk Ansvarlig ABC Startsiden AS > http://www.startsiden.no > From des at linpro.no Wed Apr 4 15:22:30 2007 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Wed, 04 Apr 2007 17:22:30 +0200 Subject: log files In-Reply-To: <17939.41531.500034.358155@dwetzel@nerim.net> (Damien Wetzel's message of "Wed, 4 Apr 2007 15:03:55 +0200") References: <17931.59232.890894.380967@dwetzel@nerim.net> <17931.64543.652972.864603@dwetzel@nerim.net> <17933.5249.247592.184781@dwetzel@nerim.net> <17939.41531.500034.358155@dwetzel@nerim.net> Message-ID: Damien Wetzel writes: > Dag-Erling Sm?rgrav writes: > > Damien Wetzel writes: > > > PS: I don't think ESI for a v2 are a good addon. ESI is mainly an > > > akamai tool to catch customers but is not of real use. the standard > > > hasn't changed since 2002 and doesn't seem to be widely used. > > You are mistaken. ESI is widely used by large online news outlets > > and portal sites, including some of Linpro's largest customers. > For me ESI are so web 1.0 , now the tendency is to deport the > personalisation on the client side. That may be your opinion, but the reality of the marketplace is that personalization is done on the server side and ESI is alive and well. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From dwetzel at nerim.net Wed Apr 4 15:35:13 2007 From: dwetzel at nerim.net (Damien Wetzel) Date: Wed, 4 Apr 2007 17:35:13 +0200 Subject: log files In-Reply-To: References: <17931.59232.890894.380967@dwetzel@nerim.net> <17931.64543.652972.864603@dwetzel@nerim.net> <17933.5249.247592.184781@dwetzel@nerim.net> <17939.41531.500034.358155@dwetzel@nerim.net> Message-ID: <17939.50609.58939.849295@dwetzel@nerim.net> hello, Do you agree that using ESI means using Akamai ? www.esi.org redirects you through akamai.com What (features) do you thing is usefull in ESI ? Damien, Dag-Erling Sm?rgrav writes: > Damien Wetzel writes: > > Dag-Erling Sm?rgrav writes: > > > Damien Wetzel writes: > > > > PS: I don't think ESI for a v2 are a good addon. ESI is mainly an > > > > akamai tool to catch customers but is not of real use. the standard > > > > hasn't changed since 2002 and doesn't seem to be widely used. > > > You are mistaken. ESI is widely used by large online news outlets > > > and portal sites, including some of Linpro's largest customers. > > For me ESI are so web 1.0 , now the tendency is to deport the > > personalisation on the client side. > > That may be your opinion, but the reality of the marketplace is that > personalization is done on the server side and ESI is alive and well. > > DES > -- > Dag-Erling Sm?rgrav > Senior Software Developer > Linpro AS - www.linpro.no From ask at develooper.com Wed Apr 4 18:00:39 2007 From: ask at develooper.com (=?ISO-8859-1?Q?Ask_Bj=F8rn_Hansen?=) Date: Wed, 4 Apr 2007 11:00:39 -0700 Subject: log files In-Reply-To: <17939.44372.537796.286223@dwetzel@nerim.net> References: <6849168.3871175692709549.JavaMail.root@ms1.startsiden.no> <28139179.3891175692737360.JavaMail.root@ms1.startsiden.no> <17939.44372.537796.286223@dwetzel@nerim.net> Message-ID: On Apr 4, 2007, at 6:51 AM, Damien Wetzel wrote: > Everyone can have its own opinion, mine is that the server side > should do a minimum of > work and let the client do it if possible ;) Hi Damien, If you just need to handle more traffic on your server that might be a good idea. If you want your users to have a good experience... Well, not necessarily such a good idea then. - ask -- http://develooper.com/ - http://askask.com/ From dwetzel at nerim.net Wed Apr 4 22:08:30 2007 From: dwetzel at nerim.net (Damien Wetzel) Date: Thu, 5 Apr 2007 00:08:30 +0200 Subject: log files In-Reply-To: References: <6849168.3871175692709549.JavaMail.root@ms1.startsiden.no> <28139179.3891175692737360.JavaMail.root@ms1.startsiden.no> <17939.44372.537796.286223@dwetzel@nerim.net> Message-ID: <17940.8670.274087.424022@dwetzel@nerim.net> hi ask, Ask Bj?rn Hansen writes: > > On Apr 4, 2007, at 6:51 AM, Damien Wetzel wrote: > > > Everyone can have its own opinion, mine is that the server side > > should do a minimum of > > work and let the client do it if possible ;) > > Hi Damien, > > If you just need to handle more traffic on your server that might be > a good idea. If you want your users to have a good experience... > Well, not necessarily such a good idea then. could detail why leting the client do the work is not a good idea ? Regarding ESI, I'm pretty sure it is a thing of the past, but again it's my opinion. Damien, > > > - ask > > -- > http://develooper.com/ - http://askask.com/ > > From des at linpro.no Wed Apr 4 22:30:47 2007 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Thu, 05 Apr 2007 00:30:47 +0200 Subject: log files In-Reply-To: <17939.50609.58939.849295@dwetzel@nerim.net> (Damien Wetzel's message of "Wed, 4 Apr 2007 17:35:13 +0200") References: <17931.59232.890894.380967@dwetzel@nerim.net> <17931.64543.652972.864603@dwetzel@nerim.net> <17933.5249.247592.184781@dwetzel@nerim.net> <17939.41531.500034.358155@dwetzel@nerim.net> <17939.50609.58939.849295@dwetzel@nerim.net> Message-ID: Damien Wetzel writes: > Do you agree that using ESI means using Akamai ? No. Oracle's CMS (whatever it is called, forget at the moment) uses ESI, and Oracle WebCache supports it. Escenic also supports ESI. > What (features) do you thing is usefull in ESI ? What the name says: edge-side includes. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From ask at develooper.com Wed Apr 4 22:35:19 2007 From: ask at develooper.com (=?ISO-8859-1?Q?Ask_Bj=F8rn_Hansen?=) Date: Wed, 4 Apr 2007 15:35:19 -0700 Subject: log files In-Reply-To: <17940.8670.274087.424022@dwetzel@nerim.net> References: <6849168.3871175692709549.JavaMail.root@ms1.startsiden.no> <28139179.3891175692737360.JavaMail.root@ms1.startsiden.no> <17939.44372.537796.286223@dwetzel@nerim.net> <17940.8670.274087.424022@dwetzel@nerim.net> Message-ID: On Apr 4, 2007, at 15:08, Damien Wetzel wrote: >> If you just need to handle more traffic on your server that might be >> a good idea. If you want your users to have a good experience... >> Well, not necessarily such a good idea then. > could detail why leting the client do the work is not a good idea ? Have you made any busy/complex websites? Getting good client-side performance is usually much harder than scaling and optimizing the server-side. There are things you must do on the client-side for a good ajax-y experience, but you don't want to give the clients extra work "just because you can". - ask -- http://develooper.com/ - http://askask.com/ From des at linpro.no Wed Apr 4 22:37:19 2007 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Thu, 05 Apr 2007 00:37:19 +0200 Subject: log files In-Reply-To: <17940.8670.274087.424022@dwetzel@nerim.net> (Damien Wetzel's message of "Thu, 5 Apr 2007 00:08:30 +0200") References: <6849168.3871175692709549.JavaMail.root@ms1.startsiden.no> <28139179.3891175692737360.JavaMail.root@ms1.startsiden.no> <17939.44372.537796.286223@dwetzel@nerim.net> <17940.8670.274087.424022@dwetzel@nerim.net> Message-ID: Damien Wetzel writes: > Regarding ESI, I'm pretty sure it is a thing of the past, but again > it's my opinion. Yes, it is your opinion. Unfortunately, it is not founded in fact. You are still welcome to express it, but when you persist in doing so despite being repeatedly confronted with conflicting evidence, it becomes tiring. Can we please move on to a more productive topic? DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Wed Apr 11 11:17:58 2007 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Wed, 11 Apr 2007 13:17:58 +0200 Subject: HEADS UP: Varnish on FreeBSD-CURRENT Message-ID: Anyone running Varnish on FreeBSD-CURRENT should set the sendfile_threshold run-time parameter to -1 (disabling the use of sendfile) to work around a bug in the sendfile syscall where the file being transferred will be truncated by an amount equivalent to the size of the HTTP header. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Wed Apr 18 09:10:57 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 18 Apr 2007 09:10:57 +0000 Subject: Operation of varnish In-Reply-To: Your message of "Tue, 17 Apr 2007 10:44:25 +0200." <462488E9.8040402@fer.hr> Message-ID: <16743.1176887457@critter.freebsd.dk> In message <462488E9.8040402 at fer.hr>, Ivan Voras writes: >Does varnish do full buffering from the server-side? I.e. when the >client connects to varnish, and varnish connects to the "real" web >server, does it buffer the web server response fully, and closes the web >server side of the connection (thus freeing it for other uses), then >pipes the data to the client at the rate client's network allows? Depends what varnish (or rather: the VCL program) decides to do. If "pipe" is chosen, Varnish just moves bytes forth and back. In the "trunk" version of varnish, everything else is fully buffered. In the released versions, up to 1.0.3, "pass" mode will not do full buffing, but fetches to cache will. >Can varnish generate HTTP logs in the "combined" format. Pointer to documentation ? >(subquestion: >how about http/1.1 virtual hosts? can each get its own log?) Right now: No. >If I >understand this correctly, varnish would generate its own logs, and then >the apache could generate its own log, but apche's will show every=20 >connection as arriving from 127.0.0.1, right? I'm not sure if Apache logs the tcp source address or the HTTP header client IP# in the logs. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Thu Apr 19 15:34:49 2007 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Thu, 19 Apr 2007 17:34:49 +0200 Subject: Mail server downtime Message-ID: The postgrey (greylisting) server on projects.linpro.no died last Friday for no discernible reason. I was away until today, and nobody else noticed that it wasn't working. This means that from Friday evening until a short while ago, mail *to* the lists was rejected with '450 server configuration error' message. Depending on the configuration of the originating server, some of it may have bounced. Mail *from* the lists was unaffected, so commit messages have gone out as usual (since they originate on projects.linpro.no itself, they are not subject to greylisting). Postgrey is back up, and I've set up a cron job to restart it every night, just in case this happens again. Unfortunately, it left no trace in the logs, and I don't have time to track down why it failed in the first place. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From stonorn at giraffen.dk Thu Apr 19 12:33:14 2007 From: stonorn at giraffen.dk (Anton Stonor) Date: Thu, 19 Apr 2007 14:33:14 +0200 Subject: Checking for purge prevents caching In-Reply-To: <15503.1175205050@critter.freebsd.dk> References: <15503.1175205050@critter.freebsd.dk> Message-ID: <4627618A.8050303@giraffen.dk> Just to share the conclusion on my issue on Varnish restaring unpredictable with an innocent snippet of VCL: sub vcl_hit { if(req.http.Cache-Control == "no-cache") { set obj.ttl = 0s; error 200 "Hey, we got a purge"; } } I have not been able to reproduce it in another environment and with the latest Varnish. So consider issue #92 bogus. /Anton From des at linpro.no Thu Apr 19 15:05:01 2007 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Thu, 19 Apr 2007 17:05:01 +0200 Subject: branches/1.0 updated Message-ID: I've merged trunk into branches/1.0. I actually only intended to merge up to r1288, but SVK ignored the range I specified and merged everything. This does not mean I'm preparing a release (especially now that I can't trust SVK to DTRT), just that I consider some of the changes in trunk to be sufficiently stable and sufficiently useful that it might make sense for people to track branches/1.0 to get something a little newer than the latest release, but not quite bleeding edge. Specifically, I consider trunk up to and including r1288 to be both a significant improvement over 1.0.3 and relatively low-risk. Changes in this range include improved backend handling, a fix for a (non-exploitable) buffer overflow in the management interface, and improved documentation. The corresponding revision on branches/1.0 is r1331, which you can check out with the following Subversion command: % svn co -r1331 http://varnish.projects.linpro.no/svn/branches/1.0 varnish DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From stephan at ymogen.net Thu Apr 19 16:10:31 2007 From: stephan at ymogen.net (Stephan Nedregaard) Date: Thu, 19 Apr 2007 17:10:31 +0100 Subject: Virtual hosts and logfiles -- some problems Message-ID: Hi! I refer to DES' mail to the list 4th April, where he proposed using: $ varnishlog -w /dev/stdout -c RxHeader '^Host: tinderbox.des.no' | varnishncsa -r /dev/stdin We've tried to get NCSA logs out of Varnish based on virtual hosts. The method described doesn't work -- it does not sort by virtual hosts as -o isn't specified. However, -o and -w don't work together, so it's not a matter of simply adding -o. So far, what we've done is to output with -o and the RxHeader rule on the server side and redirected the output to a file. We then want to do the conversion on a separate server. However, varnishncsa doesn't seem to work very welll with stored files on any server. :( varnishncsa -d -r raw.log -- with or without the -d outputs nothing. Some times it segfaults. varnishlog has no problems reading the log when passed the same parameters, although the output is clearly not as intended (see below): 11 RxHeader c Accept-Language: en-gb 11 RxHeader c Accept 25455 (null) ding: gzip, deflate 11 RxHeader c Connection: Keep-Alive 11 VCL_call c recv pass 11 Backend 8291 ObjRequest 13 default 11 RxProtocol c 21584 (null) /1.1 11 RxStatus c 200 11 RxResponse c OK 11 RxHeader c Date: Tue 14112 ObjRequest Apr 2007 11:57:38 GMT 11 RxHeader c Server 28769 ObjRequest che 11 RxHeader c Last-Modified: Wed, 04 Apr 2007 16:20:57 21514 (null) 11 RxHeader c Content-Length: 166 11 RxHeader c Content-Type: i 25903 (null) gif 11 RxHeader c X-Varnish: 1753174786 11 RxHeader c X-Forwarded-for: 194.75.128.200 The version of Varnish used is 1.0.3. This happens both with the standard Gentoo package and a compiled version of the source tarball. If anyone knows any way to reliably get log data per virtual host into NCSA style logfiles, that would be appreciated. Stephan Nedregaard, Ymogen Ltd. From des at linpro.no Thu Apr 19 16:13:29 2007 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Thu, 19 Apr 2007 18:13:29 +0200 Subject: Virtual hosts and logfiles -- some problems In-Reply-To: (Stephan Nedregaard's message of "Thu, 19 Apr 2007 17:10:31 +0100") References: Message-ID: Stephan Nedregaard writes: > So far, what we've done is to output with -o and the RxHeader rule on > the server side and redirected the output to a file. We then want to > do the conversion on a separate server. However, varnishncsa doesn't > seem to work very welll with stored files on any server. :( Of course - varnishncsa can't read the output from varnishlog. It can only read raw logs either from memory or from a file written with varnishlog -w. Basically, we need to fix varnishncsa so you can apply the filter there (i.e. 'varnishncsa RxHeader '^Host: host.example.com') DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From james at nyi.net Thu Apr 19 16:34:15 2007 From: james at nyi.net (James) Date: Thu, 19 Apr 2007 12:34:15 -0400 Subject: Varnishd and Supervisord Message-ID: <46279A07.4080806@nyi.net> Hello all, I'm not sure if this is the right mailing list or not, but I am trying to get Varnish working with Supervisord (http://plope.org/software/supervisor2/). The point of supervisord is to make sure varnish is always running and restart it even in the case of a crash. The problem I am facing is that in order to work with Supervisord, Varnish has to not daemonize. However, with the -d option, the main child thread doesn't start until the start command is given via the telnet interface. Therefore, if varnish crashes and is restarted, the child process will need manual intervention. Is there any way to get the child to start automatically even in debugging mode? -- james From stephan at ymogen.net Tue Apr 17 14:47:12 2007 From: stephan at ymogen.net (Stephan Nedregaard) Date: Tue, 17 Apr 2007 15:47:12 +0100 Subject: Virtual hosts and logfiles -- some problems Message-ID: <4624DDF0.60701@ymogen.net> Hi! I refer to DES` mail to the list 4th April, where he proposed using: des at dma ~% varnishlog -w /dev/stdout -c RxHeader '^Host: tinderbox.des.no' | varnishncsa -r /dev/stdin We've tried to get NCSA logs out of Varnish based on virtual hosts. The method described doesn't work -- it does not sort by virtual hosts as -o isn't specified. However, -o and -w don't work together, so it's not a matter of simply adding -o. So far, what we've done is to output with -o and the RxHeader rule on the server side and redirected the output to a file. We then want to do the conversion on a separate server. However, varnishncsa doesn't seem to work very welll with stored files on any server. :( varnishncsa -d -r raw.log -- with or without the -d outputs nothing. Some times it segfaults. varnishlog has no problems reading the log when passed the same parameters, although the output is clearly not as intended (see below): 11 RxHeader c Accept-Language: en-gb 11 RxHeader c Accept 25455 (null) ding: gzip, deflate 11 RxHeader c Connection: Keep-Alive 11 VCL_call c recv pass 11 Backend 8291 ObjRequest 13 default 11 RxProtocol c 21584 (null) /1.1 11 RxStatus c 200 11 RxResponse c OK 11 RxHeader c Date: Tue 14112 ObjRequest Apr 2007 11:57:38 GMT 11 RxHeader c Server 28769 ObjRequest che 11 RxHeader c Last-Modified: Wed, 04 Apr 2007 16:20:57 21514 (null) 11 RxHeader c Content-Length: 166 11 RxHeader c Content-Type: i 25903 (null) gif 11 RxHeader c X-Varnish: 1753174786 11 RxHeader c X-Forwarded-for: 194.75.128.200 The version of Varnish used is 1.0.3. This happens both with the standard Gentoo package and a compiled version of the source tarball. If anyone knows any way to reliably get log data per virtual host, that would be appreciated. Stephan Nedregaard From stephan at ymogen.net Wed Apr 18 15:58:41 2007 From: stephan at ymogen.net (Stephan Nedregaard) Date: Wed, 18 Apr 2007 16:58:41 +0100 Subject: Virtual hosts and logfiles -- some problems Message-ID: <46264031.5080904@ymogen.net> Hi! I refer to DES` mail to the list 4th April, where he proposed using: des at dma ~% varnishlog -w /dev/stdout -c RxHeader '^Host: tinderbox.des.no' | varnishncsa -r /dev/stdin We've tried to get NCSA logs out of Varnish based on virtual hosts. The method described doesn't work -- it does not sort by virtual hosts as -o isn't specified. However, -o and -w don't work together, so it's not a matter of simply adding -o. So far, what we've done is to output with -o and the RxHeader rule on the server side and redirected the output to a file. We then want to do the conversion on a separate server. However, varnishncsa doesn't seem to work very welll with stored files on any server. :( varnishncsa -d -r raw.log -- with or without the -d outputs nothing. Some times it segfaults. varnishlog has no problems reading the log when passed the same parameters, although the output is clearly not as intended (see below): 11 RxHeader c Accept-Language: en-gb 11 RxHeader c Accept 25455 (null) ding: gzip, deflate 11 RxHeader c Connection: Keep-Alive 11 VCL_call c recv pass 11 Backend 8291 ObjRequest 13 default 11 RxProtocol c 21584 (null) /1.1 11 RxStatus c 200 11 RxResponse c OK 11 RxHeader c Date: Tue 14112 ObjRequest Apr 2007 11:57:38 GMT 11 RxHeader c Server 28769 ObjRequest che 11 RxHeader c Last-Modified: Wed, 04 Apr 2007 16:20:57 21514 (null) 11 RxHeader c Content-Length: 166 11 RxHeader c Content-Type: i 25903 (null) gif 11 RxHeader c X-Varnish: 1753174786 11 RxHeader c X-Forwarded-for: 194.75.128.200 The version of Varnish used is 1.0.3. This happens both with the standard Gentoo package and a compiled version of the source tarball. If anyone knows any way to reliably get log data per virtual host, that would be appreciated. Stephan Nedregaard From ivoras at fer.hr Sun Apr 15 20:46:36 2007 From: ivoras at fer.hr (Ivan Voras) Date: Sun, 15 Apr 2007 22:46:36 +0200 Subject: Operation of varnish Message-ID: <46228F2C.2070105@fer.hr> Does varnish do full buffering from the server-side? I.e. when the client connects to varnish, and varnish connects to the "real" web server, does it buffer the web server response fully, and closes the web server side of the connection (thus freeing it for other uses), then pipes the data to the client at the rate client's network allows? Can varnish generate HTTP logs in the "combined" format. (subquestion: how about http/1.1 virtual hosts? can each get its own log?) If I understand this correctly, varnish would generate its own logs, and then the apache could generate its own log, but it will show every connection as arriving from 127.0.0.1, right? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 250 bytes Desc: OpenPGP digital signature URL: From phk at phk.freebsd.dk Thu Apr 19 18:02:15 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 19 Apr 2007 18:02:15 +0000 Subject: Varnishd and Supervisord In-Reply-To: Your message of "Thu, 19 Apr 2007 12:34:15 -0400." <46279A07.4080806@nyi.net> Message-ID: <46036.1177005735@critter.freebsd.dk> In message <46279A07.4080806 at nyi.net>, James writes: >Hello all, > >I'm not sure if this is the right mailing list or not, but I am trying >to get Varnish working with Supervisord >(http://plope.org/software/supervisor2/). The point of supervisord is >to make sure varnish is always running and restart it even in the case >of a crash. This is sort of pointless, because Varnish already has its own supervisor facility: The management process will restart the worker process if it dies. >The problem I am facing is that in order to work with Supervisord, >Varnish has to not daemonize. However, with the -d option, the main >child thread doesn't start until the start command is given via the >telnet interface. Therefore, if varnish crashes and is restarted, the >child process will need manual intervention. Is there any way to get the >child to start automatically even in debugging mode? I havn't tried this, but you could try to run it as: echo "start" | varnishd -d -d You will want to use the double -d option for something like this. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From lists at dirkgomez.de Thu Apr 19 19:37:19 2007 From: lists at dirkgomez.de (Dirk Gomez) Date: Thu, 19 Apr 2007 21:37:19 +0200 Subject: Varnish logging Message-ID: Hi list, The setting is: a lot of virtual hosts running on top of Zope. We deliver statistics generated with server logfiles. We are perfectly content with post-processing a big logfile with entries for several domains - we had a script like that when we were still using Squid. Comparing the logfiles generated by Squid and Varnish we find that those generated by Varnish lack any Host prefix in the relevant 'GET' field so we are unable to process them as we did with Squid's. As the manpage does not say anything about a configuration option to alter this behaviour, we'd be thankful for hints in this case. Filtering before logging was proposed in an earlier post- in our case this does not seem to be a good solution as there are lots of changing domains involved and so the configuration would be quite tedious- it would be entirely sufficient to have the host logged by default. Appendix: Some lines of the logfiles, for comparison Squid: 89.59.80.86 - - [06/Apr/2007:13:28:05 +0200] "GET http:// op.elseware.de:8080/content/logobao.gif HTTP/1.1" 200 17554 "http:// op.elseware.de/content/e3224/e10/e589/e594/e702/index_ger.html" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727)" TCP_REFRESH_MISS:DIRECT 89.59.80.86 - - [06/Apr/2007:13:28:05 +0200] "GET http:// op.elseware.de:8080/common/masses.jpg HTTP/1.1" 200 2681 "http:// op.elseware.de/content/e3224/e10/e589/e594/e702/index_ger.html" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727)" TCP_MEM_HIT:NONE 89.59.80.86 - - [06/Apr/2007:13:28:05 +0200] "GET http:// op.elseware.de:8080/favicon.ico HTTP/1.1" 200 1407 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727)" TCP_MEM_HIT:NONE 87.160.144.136 - - [06/Apr/2007:13:28:08 +0200] "GET http:// www.csg.elseware.com:8080/content/e1756/index_html? HTTP/1.1" 200 14826 "http://www.csg.elseware.com/content/e1756" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.8.0.11) Gecko/20070312 Firefox/ 1.5.0.11" TCP_MISS:DIRECT Varnish: 87.160.200.220 - - [10/Apr/2007:00:36:05 +0200] "GET /sites/nele- elseware.de/myzms/content/imgb01_ger.jpg HTTP/1.1" 304 - "http:// www.nele-elseware.de/content/e531/index_ger.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; HbTools 4.8.4)" 87.160.200.220 - - [10/Apr/2007:00:36:26 +0200] "GET /sites/nele- elseware.de/myzms/content/e531/e706/milonga_del_serafin_ger.mp3 HTTP/ 1.1" 304 - "http://www.nele-elseware.de/content/e531/index_ger.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; HbTools 4.8.4)" 87.160.200.220 - - [10/Apr/2007:00:36:31 +0200] "GET /sites/nele- elseware.de/myzms/content/e531/e706/milonga_del_serafin_ger.mp3 HTTP/ 1.1" 304 - "-" "NSPlayer/9.0.0.3250 WMFSDK/9.0" 87.160.200.220 - - [10/Apr/2007:00:41:23 +0200] "GET /sites/nele- elseware.de/myzms/content/e531/e686/el_choclo_ger.mp3 HTTP/1.1" 304 - "http://www.nele-elseware.de/content/e531/index_ger.html" "Mozilla/ 4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; HbTools 4.8.4)" 87.160.200.220 - - [10/Apr/2007:00:45:40 +0200] "GET /sites/nele- elseware.de/myzms/content/e531/e543/a_los_paisanos_ger.pdf HTTP/1.1" 200 479110 "http://www.nele-elseware.de/content/e531/index_ger.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; HbTools 4.8.4)" 87.160.200.220 - - [10/Apr/2007:00:46:29 +0200] "GET /common/logo.gif HTTP/1.1" 304 - "http://www.nele-elseware.de/content/e531/ index_ger.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; HbTools 4.8.4)" 87.160.200.220 - - [10/Apr/2007:00:46:29 +0200] "GET /sites/nele- elseware.de/myzms/content/imga01_ger.jpg HTTP/1.1" 304 - "http:// www.nele-elseware.de/content/e531/index_ger.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; HbTools 4.8.4)" 87.160.200.220 - - [10/Apr/2007:00:46:30 +0200] "GET /common/deu.jpg HTTP/1.1" 304 - "http://www.nele-elseware.de/content/e531/ index_ger.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; HbTools 4.8.4)" 87.160.200.220 - - [10/Apr/2007:00:46:30 +0200] "GET /common/eng.jpg HTTP/1.1" 304 - "http://www.nele-elseware.de/content/e531/ index_ger.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; HbTools 4.8.4)" 87.160.200.220 - - [10/Apr/2007:00:46:30 +0200] "GET /common/esp.jpg HTTP/1.1" 304 - "http://www.nele-elseware.de/content/e531/ index_ger.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; HbTools 4.8.4)" 87.160.200.220 - - [10/Apr/2007:00:46:30 +0200] "GET /sites/nele- elseware.de/myzms/content/imgb01_ger.jpg HTTP/1.1" 304 - "http:// www.nele-elseware.de/content/e531/index_ger.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; HbTools 4.8.4)" 87.160.200.220 - - [10/Apr/2007:00:46:30 +0200] "GET /misc_/zms/ mime_type.application_pdf.gif HTTP/1.1" 304 - "http://www.nele- elseware.de/content/e531/index_ger.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; HbTools 4.8.4)" 87.160.200.220 - - [10/Apr/2007:00:46:30 +0200] "GET /misc_/zms/ mime_type.audio_basic.gif HTTP/1.1" 304 - "http://www.nele- elseware.de/content/e531/index_ger.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; HbTools 4.8.4)" 87.160.200.220 - - [10/Apr/2007:00:46:41 +0200] "GET /sites/nele- elseware.de/myzms/content/e531/e684/ahora_no_me_conoces-nele- elseware_ger.mp3 HTTP/1.1" 304 - "http://www.nele-elseware.de/content/ e531/index_ger.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; HbTools 4.8.4)" 87.160.200.220 - - [10/Apr/2007:00:49:59 +0200] "GET /sites/nele- elseware.de/myzms/content/e531/e706/milonga_del_serafin_ger.mp3 HTTP/ 1.1 HTTP/1.1" 304 - "-" "NSPlayer/9.0.0.3250 WMFSDK/9.0" 66.249.65.110 - - [10/Apr/2007:00:53:27 +0200] "GET /e43/e59/ sitemap_ger.html HTTP/1.1 HTTP/1.1" 304 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 88.198.251.202 - - [10/Apr/2007:01:19:14 +0200] "" 304 - "-" "-" 85.25.129.88 - - [10/Apr/2007:01:24:26 +0200] "GET / HTTP/1.0 HTTP/ 1.1" 304 - "-" "Mozilla" 88.198.251.202 - - [10/Apr/2007:03:50:50 +0200] "" 304 - "-" "-" 207.46.98.56 - - [10/Apr/2007:03:55:56 +0200] "GET /robots.txt HTTP/ 1.0 HTTP/1.1" 304 - "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 66.249.65.110 - - [10/Apr/2007:04:08:24 +0200] "GET /e31/e78/ sitemap_ger.html HTTP/1.1 HTTP/1.1" 304 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 65.55.208.24 - - [10/Apr/2007:04:11:10 +0200] "GET /robots.txt HTTP/ 1.0 HTTP/1.1" 304 - "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 65.55.208.24 - - [10/Apr/2007:04:11:13 +0200] "GET /e53/ sitemap_ger.html HTTP/1.0 HTTP/1.1" 304 - "-" "msnbot/1.0 (+http:// search.msn.com/msnbot.htm)" 65.54.188.19 - - [10/Apr/2007:05:00:54 +0200] "GET /robots.txt HTTP/ 1.0 HTTP/1.1" 304 - "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 65.54.188.19 - - [10/Apr/2007:05:00:55 +0200] "GET /content/ index_ger.html HTTP/1.0 HTTP/1.1" 304 - "-" "msnbot/1.0 (+http:// search.msn.com/msnbot.htm)" 207.46.98.56 - - [10/Apr/2007:06:26:38 +0200] "GET /robots.txt HTTP/ 1.0 HTTP/1.1" 304 - "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 207.46.98.56 - - [10/Apr/2007:06:26:39 +0200] "GET /e61/ index_print_ger.html HTTP/1.0 HTTP/1.1" 304 - "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" -- Dirk From phk at phk.freebsd.dk Thu Apr 19 19:40:16 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 19 Apr 2007 19:40:16 +0000 Subject: Varnish logging In-Reply-To: Your message of "Thu, 19 Apr 2007 21:37:19 +0200." Message-ID: <46560.1177011616@critter.freebsd.dk> In message , Dirk Gomez writ es: >Hi list, > >The setting is: a lot of virtual hosts running on top of Zope. We >deliver statistics generated with server logfiles. Basically, you want NCSA format logfiles split by the value of the "Host:" header ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From ingvar at linpro.no Thu Apr 19 19:56:27 2007 From: ingvar at linpro.no (Ingvar Hagelund) Date: Thu, 19 Apr 2007 21:56:27 +0200 Subject: Packaging: A need for a devel package? Message-ID: <1177012587.13533.0.camel@re.e37> Repost, just to make sure it comes through after the greylisting problem. I am putting some effort in the (RedHat) rpm package again. I got a question about a devel package. Usually, on RedHat based systems, one typically puts things like static libraries and header files in a devel package, like "varnish-libs-devel-1.0.3-5.i386.rpm" for instance. Now, I wonder: Would it be appropriate with a devel package at all? Is it thinkable that anyone would use varnish technology to build things outside varnish itself? If so, I could use a list of actual header files and a suggestion on where to put them (/usr/include/varnish?), and maybe some starting point hacking documentation, if that's feasible. If this seems nonsense, please tell, and I'll just skip the devel package. Ingvar -- From phk at phk.freebsd.dk Thu Apr 19 20:20:05 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 19 Apr 2007 20:20:05 +0000 Subject: Packaging: A need for a devel package? In-Reply-To: Your message of "Thu, 19 Apr 2007 21:56:27 +0200." <1177012587.13533.0.camel@re.e37> Message-ID: <46731.1177014005@critter.freebsd.dk> In message <1177012587.13533.0.camel at re.e37>, Ingvar Hagelund writes: >Now, I wonder: Would it be appropriate with a devel package at all? Is >it thinkable that anyone would use varnish technology to build things >outside varnish itself? At this point, I don't think there is any risk of that. Down the road it is anyones guess. I would put a devel package on the "if needs arise" list -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From rmo at sunnmore.net Thu Apr 19 22:21:47 2007 From: rmo at sunnmore.net (Roy-Magne Mo) Date: Fri, 20 Apr 2007 00:21:47 +0200 Subject: Packaging: A need for a devel package? In-Reply-To: <1177012587.13533.0.camel@re.e37> References: <1177012587.13533.0.camel@re.e37> Message-ID: <4627EB7B.9080007@sunnmore.net> Ingvar Hagelund skreiv: > I am putting some effort in the (RedHat) rpm package again. I got a > question about a devel package. Usually, on RedHat based systems, one > typically puts things like static libraries and header files in a devel > package, like "varnish-libs-devel-1.0.3-5.i386.rpm" for instance. > > Now, I wonder: Would it be appropriate with a devel package at all? Is > it thinkable that anyone would use varnish technology to build things > outside varnish itself? If so, I could use a list of actual header files > and a suggestion on where to put them (/usr/include/varnish?), and maybe > some starting point hacking documentation, if that's feasible. It doesn't hurt either, so I think it should stay there. I had a look at the package, and see that you strip the binaries. Stripping is handled by rpm itself, and the debuginfo is automatically put into it's own package. -- Roy-Magne Mo From des at linpro.no Fri Apr 20 03:42:39 2007 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Fri, 20 Apr 2007 05:42:39 +0200 Subject: Packaging: A need for a devel package? In-Reply-To: <1177012587.13533.0.camel@re.e37> (Ingvar Hagelund's message of "Thu, 19 Apr 2007 21:56:27 +0200") References: <1177012587.13533.0.camel@re.e37> Message-ID: Ingvar Hagelund writes: > Now, I wonder: Would it be appropriate with a devel package at all? Is > it thinkable that anyone would use varnish technology to build things > outside varnish itself? Actually, there is an API for reading log data either from shared memory or from a file written by varnishlog, which might be useful if only it were documented... DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From ingvar at linpro.no Wed Apr 18 18:47:08 2007 From: ingvar at linpro.no (Ingvar Hagelund) Date: Wed, 18 Apr 2007 20:47:08 +0200 Subject: Packaging: A need for a devel package? Message-ID: <1176922028.5221.9.camel@re.e37> I am putting some effort in the (RedHat) rpm package again. I got a question about a devel package. Usually, on RedHat based systems, one typically puts things like static libraries and header files in a devel package, like "varnish-libs-devel-1.0.3-5.i386.rpm" for instance. Now, I wonder: Would it be appropriate with a devel package at all? Is it thinkable that anyone would use varnish technology to build things outside varnish itself? If so, I could use a list of actual header files and a suggestion on where to put them (/usr/include/varnish?), and maybe some starting point hacking documentation, if that's feasible. If this seems nonsense, please tell, and I'll just skip the devel package. Ingvar -- From thias at spam.spam.spam.spam.spam.spam.spam.egg.and.spam.freshrpms.net Fri Apr 20 08:08:00 2007 From: thias at spam.spam.spam.spam.spam.spam.spam.egg.and.spam.freshrpms.net (Matthias Saou) Date: Fri, 20 Apr 2007 10:08:00 +0200 Subject: Packaging: A need for a devel package? In-Reply-To: <1177012587.13533.0.camel@re.e37> References: <1177012587.13533.0.camel@re.e37> Message-ID: <20070420100800.51641b68@python3.es.egwn.lan> Ingvar Hagelund wrote : > Repost, just to make sure it comes through after the greylisting problem. > > I am putting some effort in the (RedHat) rpm package again. I got a > question about a devel package. Usually, on RedHat based systems, one > typically puts things like static libraries and header files in a devel > package, like "varnish-libs-devel-1.0.3-5.i386.rpm" for instance. > > Now, I wonder: Would it be appropriate with a devel package at all? Is > it thinkable that anyone would use varnish technology to build things > outside varnish itself? If so, I could use a list of actual header files > and a suggestion on where to put them (/usr/include/varnish?), and maybe > some starting point hacking documentation, if that's feasible. > > If this seems nonsense, please tell, and I'll just skip the devel > package. From what Dag-Erling answered, it seems like the devel package might make sense. If you do decide to have one, your example above isn't good (at least for Red Hat and Fedora, SuSE, Mandriva and others do things differently), as you would need to have for instance : varnish (the main package with the daemon) varnish-libs varnish-devel (and not "varnish-libs-devel") The "libs" only make sense to split out if some programs could require them without requiring the main daemon. Again, from what Dag-Erling wrote, maybe this would make sense if someone writes a varnishlog file parser. Attached are the files I used to build the latest varnish package I used, in case they can be of any help. Matthias -- Clean custom Red Hat Linux rpm packages : http://freshrpms.net/ Fedora Core release 6 (Zod) - Linux kernel 2.6.20-1.2943.fc6 Load : 0.30 0.29 0.27 -------------- next part -------------- A non-text attachment was scrubbed... Name: varnish.conf Type: application/octet-stream Size: 799 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: varnish.init Type: application/octet-stream Size: 1788 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: varnish.spec Type: application/octet-stream Size: 3966 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: vcl.conf Type: application/octet-stream Size: 678 bytes Desc: not available URL: From phk at phk.freebsd.dk Fri Apr 20 08:35:09 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 20 Apr 2007 08:35:09 +0000 Subject: Packaging: A need for a devel package? In-Reply-To: Your message of "Fri, 20 Apr 2007 10:08:00 +0200." <20070420100800.51641b68@python3.es.egwn.lan> Message-ID: <49435.1177058109@critter.freebsd.dk> In message <20070420100800.51641b68 at python3.es.egwn.lan>, Matthias Saou writes: >The "libs" only make sense to split out if some programs could require >them without requiring the main daemon. Again, from what Dag-Erling >wrote, maybe this would make sense if someone writes a varnishlog file >parser. It may make sense down the road, but not yet. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From trondmm-varnish at crusaders.no Fri Apr 20 10:55:17 2007 From: trondmm-varnish at crusaders.no (Trond Michelsen) Date: Fri, 20 Apr 2007 12:55:17 +0200 Subject: Varnishd and Supervisord In-Reply-To: <46036.1177005735@critter.freebsd.dk> References: <46279A07.4080806@nyi.net> <46036.1177005735@critter.freebsd.dk> Message-ID: <20070420105517.GF15752@crusaders.no> On Thu, Apr 19, 2007 at 06:02:15PM +0000, Poul-Henning Kamp wrote: > In message <46279A07.4080806 at nyi.net>, James writes: >> I'm not sure if this is the right mailing list or not, but I am >> trying to get Varnish working with Supervisord >> (http://plope.org/software/supervisor2/). The point of supervisord >> is to make sure varnish is always running and restart it even in >> the case of a crash. > This is sort of pointless, because Varnish already has its own > supervisor facility: The management process will restart the worker > process if it dies. How do you set up Varnish to automatically restart if it dies? We recently had a crash where the varnish process didn't restart by itself, so I'm very interested if there's sonething we can do to make sure it keeps running. Anyway - I found the final entries in the varnishncsa-log to be kind of interesting... --8<-- 195.1.193.62 - - [10/Apr/2007:15:24:39 +0200] "" "-" "-" 195.1.193.62 - - [10/Apr/2007:15:24:39 +0200] "" "-" "-" 195.1.193.62 - - [10/Apr/2007:15:24:39 +0200] "" 200 4378 "-" "-" 195.1.193.62 - - [10/Apr/2007:15:24:39 +0200] "" 200 16500 "-" "-" 195.1.193.62 - - [10/Apr/2007:15:24:39 +0200] "" 200 8077 "-" "-" 195.1.193.62 - - [10/Apr/2007:15:24:39 +0200] "" "-" "-" 195.1.193.62 - - [10/Apr/2007:15:24:39 +0200] "" "-" "-" 195.1.193.62 - - [10/Apr/2007:15:24:39 +0200] "" 200 3989 "-" "-" 195.1.193.62 - - [10/Apr/2007:15:24:39 +0200] "" 200 9087 "-" "-" 195.1.193.62 - - [10/Apr/2007:15:24:39 +0200] "" 200 9709 "-" "-" 195.1.193.62 - - [10/Apr/2007:15:24:39 +0200] "" 200 3278 "-" "-" 195.1.193.62 - - [10/Apr/2007:15:24:39 +0200] "" 200 3894 "-" "-" 195.1.193.62 - - [10/Apr/2007:15:24:39 +0200] "" 200 20217 "-" "-" 195.1.193.62 - - [10/Apr/2007:15:24:39 +0200] "" 200 8145 "-" "-" 195.1.193.62 - - [10/Apr/2007:15:24:39 +0200] "" 200 4638 "-" "-" 195.1.193.62 - - [10/Apr/2007:15:24:39 +0200] "" 200 8611 "-" "-" 195.1.193.62 - - [10/Apr/2007:15:24:39 +0200] "" 200 6313 "-" "-" 195.1.193.62 - - [10/Apr/2007:15:24:39 +0200] "" 200 23572 "-" "-" --8<-- -- Trond Michelsen From phk at phk.freebsd.dk Fri Apr 20 11:11:56 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 20 Apr 2007 11:11:56 +0000 Subject: Varnishd and Supervisord In-Reply-To: Your message of "Fri, 20 Apr 2007 12:55:17 +0200." <20070420105517.GF15752@crusaders.no> Message-ID: <50600.1177067516@critter.freebsd.dk> In message <20070420105517.GF15752 at crusaders.no>, Trond Michelsen writes: >On Thu, Apr 19, 2007 at 06:02:15PM +0000, Poul-Henning Kamp wrote: >> This is sort of pointless, because Varnish already has its own >> supervisor facility: The management process will restart the worker >> process if it dies. > >How do you set up Varnish to automatically restart if it dies? It should be the default, but the parameter auto_restart can be used to disable it. >We recently had a crash where the varnish process didn't restart by >itself, so I'm very interested if there's sonething we can do to make >sure it keeps running. > >Anyway - I found the final entries in the varnishncsa-log to be kind >of interesting... When you have a crash, run "varnishlog -d" to get a snapshot of the "real" log, that would be much more helpful for debugging. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From anders at fupp.net Fri Apr 20 12:50:21 2007 From: anders at fupp.net (Anders Nordby) Date: Fri, 20 Apr 2007 14:50:21 +0200 Subject: Varnishlog -o bug Message-ID: <20070420125021.GA75316@fupp.net> Hi, When using varnishlog -o, log entries for cache misses appear on the same line as the preceding log entry: # varnishlog -o | grep 'VCL_call' | grep miss 12 VCL_call c recv lookup 12 VCL_call c miss fetch 15 VCL_call c recv lookup 15 VCL_call c miss fetch 15 VCL_call c recv lookup 15 VCL_call c miss fetch 12 VCL_call c recv lookup 12 VCL_call c miss fetch 15 VCL_call c recv lookup 15 VCL_call c miss fetch 15 VCL_call c recv lookup 15 VCL_call c miss fetch ^C # varnishlog | grep 'VCL_call' | grep miss 16 VCL_call c miss 17 VCL_call c miss 16 VCL_call c miss 17 VCL_call c miss 21 VCL_call c miss 24 VCL_call c miss ^C I am using Varnish 1.0.3 on FreeBSD/amd64, 6.2-RELEASE. Plz fix k thx! :-) Cheers, -- Anders. From ingvar at linpro.no Fri Apr 20 13:12:17 2007 From: ingvar at linpro.no (Ingvar Hagelund) Date: Fri, 20 Apr 2007 15:12:17 +0200 Subject: Packaging: A need for a devel package? In-Reply-To: <20070420100800.51641b68@python3.es.egwn.lan> References: <1177012587.13533.0.camel@re.e37> <20070420100800.51641b68@python3.es.egwn.lan> Message-ID: <1177074737.6003.26.camel@re.e37> * Matthias Saou > From what Dag-Erling answered, it seems like the devel package might > make sense. Since the documentation mentioned is missing, I'm going to push the devel package for later/request, as proposed by Poul-Henning. > If you do decide to have one, your example above isn't good > (at least for Red Hat and Fedora, SuSE, Mandriva and others do things > differently), as you would need to have for instance : > > varnish (the main package with the daemon) > varnish-libs > varnish-devel (and not "varnish-libs-devel") It's kind of strange, as I get different answers every time I ask anybody about this :-) At the moment, I have a package for review for Fedora. Matthias, could you post comments in RedHat Bugzilla, Bug #230275, please? > The "libs" only make sense to split out if some programs could require > them without requiring the main daemon. Again, from what Dag-Erling > wrote, maybe this would make sense if someone writes a varnishlog file > parser. I guess I will to keep the libs package for future use. It's complete, and thus easier to cope with than a non-existing list of header files and documentation. > Attached are the files I used to build the latest varnish package I > used, in case they can be of any help. Yes, the changes to the initscript and the configuration file are absolutely interesting, though I might insist on putting the config file in /etc/sysconfig. Ingvar -- From ingvar at linpro.no Fri Apr 20 15:13:33 2007 From: ingvar at linpro.no (Ingvar Hagelund) Date: Fri, 20 Apr 2007 17:13:33 +0200 Subject: branches/1.0 updated In-Reply-To: References: Message-ID: <1177082013.6003.58.camel@re.e37> * Dag-Erling Sm?rgrav > I've merged trunk into branches/1.0. > > I actually only intended to merge up to r1288, but SVK ignored the > range I specified and merged everything. > > This does not mean I'm preparing a release (especially now that I > can't trust SVK to DTRT), just that I consider some of the changes in > trunk to be sufficiently stable and sufficiently useful that it might > make sense for people to track branches/1.0 to get something a little > newer than the latest release, but not quite bleeding edge. > > Specifically, I consider trunk up to and including r1288 to be both a > significant improvement over 1.0.3 and relatively low-risk. Changes > in this range include improved backend handling, a fix for a > (non-exploitable) buffer overflow in the management interface, and > improved documentation. If these are not changes enough to roll 1.0.4, when (if ever) can we expect a new 1.0 release? > > The corresponding revision on branches/1.0 is r1331, which you can > check out with the following Subversion command: > > % svn co -r1331 http://varnish.projects.linpro.no/svn/branches/1.0 varnish With these improvements, would you say it's still appropriate to run 1.0.3 in production environments? Ingvar -- From phk at phk.freebsd.dk Fri Apr 20 17:58:03 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 20 Apr 2007 17:58:03 +0000 Subject: branches/1.0 updated In-Reply-To: Your message of "Fri, 20 Apr 2007 17:13:33 +0200." <1177082013.6003.58.camel@re.e37> Message-ID: <52700.1177091883@critter.freebsd.dk> In message <1177082013.6003.58.camel at re.e37>, Ingvar Hagelund writes: >* Dag-Erling Sm?rgrav >> Specifically, I consider trunk up to and including r1288 to be both a >> significant improvement over 1.0.3 and relatively low-risk. Changes >> in this range include improved backend handling, a fix for a >> (non-exploitable) buffer overflow in the management interface, and >> improved documentation. > >If these are not changes enough to roll 1.0.4, when (if ever) can we >expect a new 1.0 release? I can easily be argued that the changes to pass mode alone warrants a 1.0.4 -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From ivoras at fer.hr Thu Apr 19 11:25:16 2007 From: ivoras at fer.hr (Ivan Voras) Date: Thu, 19 Apr 2007 13:25:16 +0200 Subject: Operation of varnish In-Reply-To: <16743.1176887457@critter.freebsd.dk> References: <16743.1176887457@critter.freebsd.dk> Message-ID: <4627519C.1060904@fer.hr> Poul-Henning Kamp wrote: > If "pipe" is chosen, Varnish just moves bytes forth and back. > > In the "trunk" version of varnish, everything else is fully buffered. > In the released versions, up to 1.0.3, "pass" mode will not do full > buffing, but fetches to cache will. > >> Can varnish generate HTTP logs in the "combined" format. > > Pointer to documentation ? From Apache's httpd.conf, it is defined as: LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined Documentation about the substitution codes and meaning is at: http://httpd.apache.org/docs/2.2/logs.html >> If I >> understand this correctly, varnish would generate its own logs, and then >> the apache could generate its own log, but apche's will show every=20 >> connection as arriving from 127.0.0.1, right? > > I'm not sure if Apache logs the tcp source address or the HTTP header > client IP# in the logs. You mean "Forwarded-For:" header? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 249 bytes Desc: OpenPGP digital signature URL: From ivoras at fer.hr Tue Apr 17 08:44:25 2007 From: ivoras at fer.hr (Ivan Voras) Date: Tue, 17 Apr 2007 10:44:25 +0200 Subject: Operation of varnish Message-ID: <462488E9.8040402@fer.hr> Hi! I'm thinking of trying out varnish, but I have several questions about it's operation: Does varnish do full buffering from the server-side? I.e. when the client connects to varnish, and varnish connects to the "real" web server, does it buffer the web server response fully, and closes the web server side of the connection (thus freeing it for other uses), then pipes the data to the client at the rate client's network allows? Can varnish generate HTTP logs in the "combined" format. (subquestion: how about http/1.1 virtual hosts? can each get its own log?) If I understand this correctly, varnish would generate its own logs, and then the apache could generate its own log, but apche's will show every connection as arriving from 127.0.0.1, right? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 249 bytes Desc: OpenPGP digital signature URL: From des at linpro.no Sat Apr 21 17:30:04 2007 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Sat, 21 Apr 2007 19:30:04 +0200 Subject: Operation of varnish In-Reply-To: <4627519C.1060904@fer.hr> (Ivan Voras's message of "Thu, 19 Apr 2007 13:25:16 +0200") References: <16743.1176887457@critter.freebsd.dk> <4627519C.1060904@fer.hr> Message-ID: Ivan Voras writes: > LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined This is exactly what varnishncsa outputs. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Sat Apr 21 17:36:57 2007 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Sat, 21 Apr 2007 19:36:57 +0200 Subject: Varnish logging In-Reply-To: <46560.1177011616@critter.freebsd.dk> (Poul-Henning Kamp's message of "Thu, 19 Apr 2007 19:40:16 +0000") References: <46560.1177011616@critter.freebsd.dk> Message-ID: "Poul-Henning Kamp" writes: > Dirk Gomez writes: > > The setting is: a lot of virtual hosts running on top of Zope. We > > deliver statistics generated with server logfiles. > Basically, you want NCSA format logfiles split by the value of > the "Host:" header ? No, he wants the Host: header prepended to the request URL in NCSA logs. It's easy to do, I'll take care of it. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Sat Apr 21 21:56:28 2007 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Sat, 21 Apr 2007 23:56:28 +0200 Subject: branches/1.0 updated In-Reply-To: <1177082013.6003.58.camel@re.e37> (Ingvar Hagelund's message of "Fri, 20 Apr 2007 17:13:33 +0200") References: <1177082013.6003.58.camel@re.e37> Message-ID: Ingvar Hagelund writes: > Dag-Erling Sm?rgrav writes: > > Specifically, I consider trunk up to and including r1288 to be both a > > significant improvement over 1.0.3 and relatively low-risk. Changes > > in this range include improved backend handling, a fix for a > > (non-exploitable) buffer overflow in the management interface, and > > improved documentation. > If these are not changes enough to roll 1.0.4, when (if ever) can we > expect a new 1.0 release? These changes alone might have warranted a 1.0.4 release on Friday if I had had time to prepare it. As things are, we are most likely looking at 1.1 on May 20th, or possibly a simultaneous release of 1.0.4 (corresponding to r1331) and 1.1 (corresponding to the current state of trunk plus URL rewriting and header manipulation). > > The corresponding revision on branches/1.0 is r1331, which you can > > check out with the following Subversion command: > > > > % svn co -r1331 http://varnish.projects.linpro.no/svn/branches/1.0 varnish > With these improvements, would you say it's still appropriate to run > 1.0.3 in production environments? Yes, definitely. Think of it as 1.0.3 with bug fixes and improved documentation. This is why I set the cutoff at r1288; beyond that point, there are changes in the VCL compiler which will need some time to settle. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Sat Apr 21 21:57:52 2007 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Sat, 21 Apr 2007 23:57:52 +0200 Subject: Varnishlog -o bug In-Reply-To: <20070420125021.GA75316@fupp.net> (Anders Nordby's message of "Fri, 20 Apr 2007 14:50:21 +0200") References: <20070420125021.GA75316@fupp.net> Message-ID: Anders Nordby writes: > When using varnishlog -o, log entries for cache misses appear on the > same line as the preceding log entry: It's not just cache misses - it appears to happen whenever two VCL_call entries are logged in succession. I noticed it on friday, but haven't had time to look into it yet. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Sat Apr 21 22:01:39 2007 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Sun, 22 Apr 2007 00:01:39 +0200 Subject: Varnish logging In-Reply-To: (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav's?= message of "Sat, 21 Apr 2007 19:36:57 +0200") References: <46560.1177011616@critter.freebsd.dk> Message-ID: des at linpro.no (Dag-Erling Sm?rgrav) writes: > "Poul-Henning Kamp" writes: > > Dirk Gomez writes: > > > The setting is: a lot of virtual hosts running on top of Zope. We > > > deliver statistics generated with server logfiles. > > Basically, you want NCSA format logfiles split by the value of > > the "Host:" header ? > No, he wants the Host: header prepended to the request URL in NCSA > logs. It's easy to do, I'll take care of it. Done in r1362. Dirk, if you are comfortable building Varnish from sources, try grabbing the following file: http://varnish.projects.linpro.no/svn/trunk/varnish-cache/bin/varnishncsa/varnishncsa.c Stick it in the obvious place in your source tree, rebuild and reinstall. It will work fine in a 1.0 tree. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From lists at dirkgomez.de Sun Apr 22 09:34:00 2007 From: lists at dirkgomez.de (Dirk Gomez) Date: Sun, 22 Apr 2007 11:34:00 +0200 Subject: Varnish logging In-Reply-To: References: <46560.1177011616@critter.freebsd.dk> Message-ID: >>> The setting is: a lot of virtual hosts running on top of Zope. We >>> deliver statistics generated with server logfiles. >> Basically, you want NCSA format logfiles split by the value of >> the "Host:" header ? > > No, he wants the Host: header prepended to the request URL in NCSA > logs. It's easy to do, I'll take care of it. Exactly. We have statistics processing on one logfile that has worked for years and we would like to continue using it. From des at linpro.no Sun Apr 22 12:45:07 2007 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Sun, 22 Apr 2007 14:45:07 +0200 Subject: New mailing list: varnish-dist Message-ID: We have a new mailing list, varnish-dist, which will be used to plan releases and coordinate the efforts of packagers. Anyone interested in these topics is welcome to subscribe at the following URL: http://projects.linpro.no/mailman/listinfo/varnish-dist/ DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From james at nyi.net Mon Apr 23 03:09:55 2007 From: james at nyi.net (James) Date: Sun, 22 Apr 2007 23:09:55 -0400 Subject: Varnishd and Supervisord In-Reply-To: <46036.1177005735@critter.freebsd.dk> References: <46036.1177005735@critter.freebsd.dk> Message-ID: <462C2383.9030203@nyi.net> Poul-Henning Kamp wrote: > This is sort of pointless, because Varnish already has its own > supervisor facility: The management process will restart > the worker process if it dies. Well, I was more worried about the management process dying for no good reason. I would imagine that it would be stable, but It would be nice to get supervisord running with varnish for some other reasons as well. > I havn't tried this, but you could try to run it as: > > echo "start" | varnishd -d -d > > You will want to use the double -d option for something like > this Sadly, that didn't work. -- james From anders at fupp.net Mon Apr 23 14:20:51 2007 From: anders at fupp.net (Anders Nordby) Date: Mon, 23 Apr 2007 16:20:51 +0200 Subject: Varnishlog -o bug In-Reply-To: References: <20070420125021.GA75316@fupp.net> Message-ID: <20070423142051.GA16422@fupp.net> Hi, On Sat, Apr 21, 2007 at 11:57:52PM +0200, Dag-Erling Sm?rgrav wrote: >> When using varnishlog -o, log entries for cache misses appear on the >> same line as the preceding log entry: > It's not just cache misses - it appears to happen whenever two > VCL_call entries are logged in succession. I noticed it on friday, > but haven't had time to look into it yet. One more thing: # varnishstat -1 Cannot open /tmp/_.vsl: No such file or directory Segmentation fault: 11 (core dumped) This is if varnish is not running. Cheers, -- Anders. From feedreader at yahoo.com Mon Apr 23 15:17:33 2007 From: feedreader at yahoo.com (Ken Ara) Date: Mon, 23 Apr 2007 08:17:33 -0700 (PDT) Subject: cachefile Message-ID: <261901.43163.qm@web53302.mail.re2.yahoo.com> >Every time I restart varnishd it needs to grab every object again from >the backend. Is this intentional because varnishd does not like to >save the objects to the disk? > >I know that varnish should not be restarted, but it happens and then >a nice built up cache is empty and needs to fill up again. Oh, what a shame... I had such high hopes for Varnish. I am setting up a shiny new server (multi-cpu, FreeBSD 6.2) to run a busy Zope site that has so far cranked out 300k+ pages currently residing in Squid's cache... Yes, we give our Zope a workout and our quite expensive pages basically don't expire. Squid has performed very well, but Varnish promises so much more! For what it's worth, I cast my vote for persistent storage in a future version! Ken __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From phk at phk.freebsd.dk Mon Apr 23 15:26:09 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 23 Apr 2007 15:26:09 +0000 Subject: cachefile In-Reply-To: Your message of "Mon, 23 Apr 2007 08:17:33 MST." <261901.43163.qm@web53302.mail.re2.yahoo.com> Message-ID: <2602.1177341969@critter.freebsd.dk> In message <261901.43163.qm at web53302.mail.re2.yahoo.com>, Ken Ara writes: >Oh, what a shame... I had such high hopes for Varnish. >I am setting up a shiny new server (multi-cpu, FreeBSD >6.2) to run a busy Zope site that has so far cranked >out 300k+ pages currently residing in Squid's cache... > >Yes, we give our Zope a workout and our quite >expensive pages basically don't expire. Squid has >performed very well, but Varnish promises so much >more! I think you'll still find Varnish delivering much more :-) >For what it's worth, I cast my vote for persistent >storage in a future version! Noted. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dirk at dirkgomez.de Sun Apr 22 20:35:19 2007 From: dirk at dirkgomez.de (Dirk Gomez) Date: Sun, 22 Apr 2007 22:35:19 +0200 Subject: Varnish logging In-Reply-To: References: <46560.1177011616@critter.freebsd.dk> Message-ID: <39DE4D78-2B6A-4A1E-89AB-7A7431133AD8@dirkgomez.de> > Done in r1362. > > Dirk, if you are comfortable building Varnish from sources, try > grabbing the following file: > > http://varnish.projects.linpro.no/svn/trunk/varnish-cache/bin/ > varnishncsa/varnishncsa.c > > Stick it in the obvious place in your source tree, rebuild and > reinstall. It will work fine in a 1.0 tree. DES, I checked out varnishncsa from trunk head, compiled it and replaced the binary that came with Debian (new)stable with the newly built one - works fine. Thanks a lot! -- Dirk From des at linpro.no Tue Apr 24 06:36:21 2007 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Tue, 24 Apr 2007 08:36:21 +0200 Subject: Varnishd and Supervisord In-Reply-To: <462C2383.9030203@nyi.net> (james@nyi.net's message of "Sun, 22 Apr 2007 23:09:55 -0400") References: <46036.1177005735@critter.freebsd.dk> <462C2383.9030203@nyi.net> Message-ID: James writes: > Well, I was more worried about the management process dying for no > good reason. I would imagine that it would be stable, but It would be > nice to get supervisord running with varnish for some other reasons as > well. Please open a ticket requesting supervisord interoperability, and we will consider it for 1.1 / 2.0. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Tue Apr 24 06:40:03 2007 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Tue, 24 Apr 2007 08:40:03 +0200 Subject: cachefile In-Reply-To: <261901.43163.qm@web53302.mail.re2.yahoo.com> (Ken Ara's message of "Mon, 23 Apr 2007 08:17:33 -0700 (PDT)") References: <261901.43163.qm@web53302.mail.re2.yahoo.com> Message-ID: Ken Ara writes: > Denis Ahrens writes: > > Every time I restart varnishd it needs to grab every object again from > > the backend. Is this intentional because varnishd does not like to > > save the objects to the disk? > Oh, what a shame... I had such high hopes for Varnish. Give it a spin. I think you'll be surprised at how well it performs, even without persistent caching. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From anders at fupp.net Tue Apr 24 08:31:46 2007 From: anders at fupp.net (Anders Nordby) Date: Tue, 24 Apr 2007 10:31:46 +0200 Subject: Delivering expired documents when backend is down Message-ID: <20070424083146.GA67181@fupp.net> Hi! One of the design goals of Varnish was to be able to deliver content when your backend is down, as far as I can remember. Now, I may be stupid, but in mgt_vcc.c, there is this default VCL which is confusing: sub vcl_timeout{ discard; } Is the object really discarded? What sort of trouble would I be getting into if I remove discard from vcl_timeout? Basically, I want to know, how do I deliver expired content if backend is down. There's no toggle to switch on and off such a behaviour? Is the answer to use a sufficiently high ttl? Bye, -- Anders. From dwetzel at nerim.net Tue Apr 24 08:49:57 2007 From: dwetzel at nerim.net (Damien Wetzel) Date: Tue, 24 Apr 2007 10:49:57 +0200 Subject: varnish anti DOS feature Message-ID: <17965.50357.545551.224112@dwetzel@nerim.net> Hello all, Coming from the CDN space, one of the main reasons that makes people giving up extraordinary amount of money to CDNs is to prevent against DOS. I wondered if you have thought about protecting varnish against DOS when designing it or if you will ? Best regards, -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Damien WETZEL ("`-/")_.-'"``-._ ATANAR TECHNOLOGIES . . `; -._ )-;-,_`) (v_,)' _ )`-.\ ``-' Phone:+33 1 45 43 02 90 _.- _..-_/ / ((.' - So much to do, so little time - ((,.-' ((,/ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From phk at phk.freebsd.dk Tue Apr 24 09:54:17 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 24 Apr 2007 09:54:17 +0000 Subject: varnish anti DOS feature In-Reply-To: Your message of "Tue, 24 Apr 2007 10:49:57 +0200." <17965.50357.545551.224112@dwetzel@nerim.net> Message-ID: <1339.1177408457@critter.freebsd.dk> In message <17965.50357.545551.224112 at dwetzel@nerim.net>, Damien Wetzel writes: >Hello all, >Coming from the CDN space, one of the main reasons that >makes people giving up extraordinary amount of money to CDNs is >to prevent against DOS. >I wondered if you have thought about protecting varnish against DOS >when designing it or if you will ? We did think about it a bit, and it is more or less the only reason we keep per-source-ip statistics. You will be able to do something like if (client.bandwidth > 1 mbit/s) { sleep 1 s; } and similar once I get to those pieces. As always: Ideas are most welcome -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Tue Apr 24 08:35:06 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 24 Apr 2007 08:35:06 +0000 Subject: Delivering expired documents when backend is down In-Reply-To: Your message of "Tue, 24 Apr 2007 10:31:46 +0200." <20070424083146.GA67181@fupp.net> Message-ID: <1051.1177403706@critter.freebsd.dk> In message <20070424083146.GA67181 at fupp.net>, Anders Nordby writes: >Hi! > >One of the design goals of Varnish was to be able to deliver content >when your backend is down, as far as I can remember. Now, I may be >stupid, but in mgt_vcc.c, there is this default VCL which is confusing: > >sub vcl_timeout{ > discard; >} > >Is the object really discarded? What sort of trouble would I be >getting into if I remove discard from vcl_timeout? Right now: quite a lot :-) vcl_timeout is the background job that is run when the object is 30 seconds from expiring. the 30 seconds will become configurable and vcl_timeout will be where gzip compression is initiated and where prefetching can be started. >Basically, I want to know, how do I deliver expired content if backend >is down. There's no toggle to switch on and off such a behaviour? > >Is the answer to use a sufficiently high ttl? That's certainly part of it. Right now we don't have a good concept of "backend is down" in Varnish so implementing a ttl overriding check is pretty hard as it is. There's a reason why we look for sponsors :-) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From thomas.westlund at aftenposten.no Wed Apr 25 12:10:16 2007 From: thomas.westlund at aftenposten.no (Thomas Westlund) Date: Wed, 25 Apr 2007 14:10:16 +0200 Subject: Problems with purging /. Message-ID: <20070425121016.GA17668@aftenposten.no> Hi, I have a problem purging the / pages on a varnish installation. I have configured the following in varnish.vcl sub vcl_hit { if (req.request == "PURGE") { set obj.ttl = 0s; error 200 "Purged."; } else { deliver; } } and sub vcl_timeout{ discard; } When I try to purge an url via the http-call it works nicely for everyting execpt the / (for example the front page). The call I send to varnish is constructed like this: # telnet varnish 80 PURGE / HTTP/1.0 Host: www.somedomain.no And the response it gives is this: 200 OK

Error 200 OK

Purged.

Guru Meditation:

XID: 326200233

Varnish I get the same result if i try to purge another url like this: PURGE /somefolder/somearticle.ece HTTP/1.0 Host: www.somedomain.no The domainnames and url's have been changed to hide the sites identity -- Thomas Westlund Aftenposten AS, UNIX/nettverksavd. Postboks 1, 0051 Oslo Tlf: +47 98 20 30 33 Fax: +47 22 86 40 74 From phk at phk.freebsd.dk Wed Apr 25 12:15:17 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 25 Apr 2007 12:15:17 +0000 Subject: Problems with purging /. In-Reply-To: Your message of "Wed, 25 Apr 2007 14:10:16 +0200." <20070425121016.GA17668@aftenposten.no> Message-ID: <8324.1177503317@critter.freebsd.dk> In message <20070425121016.GA17668 at aftenposten.no>, Thomas Westlund writes: >Hi, > >I have a problem purging the / pages on a varnish installation. > >I have configured the following in varnish.vcl Uhm, the PURGE request gets the "200" answer you quote from the: error 200 "Purged."; What makes you think the object is not purged ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Wed Apr 25 12:20:13 2007 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Wed, 25 Apr 2007 14:20:13 +0200 Subject: Problems with purging /. In-Reply-To: <20070425121016.GA17668@aftenposten.no> (Thomas Westlund's message of "Wed, 25 Apr 2007 14:10:16 +0200") References: <20070425121016.GA17668@aftenposten.no> Message-ID: Thomas Westlund writes: > > > > 200 OK > > >

Error 200 OK

>

Purged.

>

Guru Meditation:

>

XID: 326200233

> Varnish > > We need to see the Varnish log for this transaction. Run 'varnishlog -w /tmp/varnish.log' on the server. Send a request that triggers the bug, then wait a few seconds and stop varnishlog. You can then extract the relevant entry like this: # varnishlog -r /tmp/varnish.log -c -o ReqStart XID where XID is the number shown in the error message (e.g. 326200233). DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From thomas.westlund at aftenposten.no Wed Apr 25 13:11:51 2007 From: thomas.westlund at aftenposten.no (Thomas Westlund) Date: Wed, 25 Apr 2007 15:11:51 +0200 Subject: Problems with purging /. In-Reply-To: <8324.1177503317@critter.freebsd.dk> References: <20070425121016.GA17668@aftenposten.no> <8324.1177503317@critter.freebsd.dk> Message-ID: <20070425131151.GA19155@aftenposten.no> Hi, On Wed, Apr 25, 2007 at 12:15:17PM +0000, Poul-Henning Kamp wrote: > Uhm, the PURGE request gets the "200" answer you quote from > the: > error 200 "Purged."; > > What makes you think the object is not purged ? If I update for example the front page on the backend and run the purge-routine for / as I discribed earlier,the document is not updated when I test it via the cache, but it is when I access it directly on the backend. -- Thomas Westlund Aftenposten AS, UNIX/nettverksavd. Postboks 1, 0051 Oslo Tlf: +47 98 20 30 33 Fax: +47 22 86 40 74 From thomas.westlund at aftenposten.no Wed Apr 25 13:18:50 2007 From: thomas.westlund at aftenposten.no (Thomas Westlund) Date: Wed, 25 Apr 2007 15:18:50 +0200 Subject: Problems with purging /. In-Reply-To: References: <20070425121016.GA17668@aftenposten.no> Message-ID: <20070425131850.GB19155@aftenposten.no> Hi, On Wed, Apr 25, 2007 at 02:20:13PM +0200, Dag-Erling Sm?rgrav wrote: > Thomas Westlund writes: > > > > > > > > 200 OK > > > > > >

Error 200 OK

> >

Purged.

> >

Guru Meditation:

> >

XID: 326200233

> > Varnish > > > > > > We need to see the Varnish log for this transaction. This is the output: 329 SessionOpen c **.**.**.** 63375 329 VCL_acl c MATCH purge "**.**.**.**"/32 329 SessionClose c Purged. 329 ReqStart c **.**.**.** 63375 326308940 329 RxRequest c PURGE 329 RxURL c / 329 RxProtocol c HTTP/1.0 329 RxHeader c Host: www.**.no 329 VCL_call c recv lookup 329 Hit c 326307698 329 VCL_call c hit 329 TTL c 326307698 VCL 0 1177506682 329 VCL_return c error 329 TxStatus c 200 329 TxProtocol c HTTP/1.1 329 TxResponse c OK 329 Req IPs and hostnames are changed to preserver privacy. Iv'e already checked the logs to see what happens, as far as I'm able to se it works as it should, but the / pages are stil not updated when I try to acesss them via the cache, but they are on the backend-server. -- Thomas Westlund Aftenposten AS, UNIX/nettverksavd. Postboks 1, 0051 Oslo Tlf: +47 98 20 30 33 Fax: +47 22 86 40 74 From phk at phk.freebsd.dk Wed Apr 25 13:20:06 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 25 Apr 2007 13:20:06 +0000 Subject: Problems with purging /. In-Reply-To: Your message of "Wed, 25 Apr 2007 15:18:50 +0200." <20070425131850.GB19155@aftenposten.no> Message-ID: <8612.1177507206@critter.freebsd.dk> In message <20070425131850.GB19155 at aftenposten.no>, Thomas Westlund writes: >This is the output: > > 329 SessionOpen c **.**.**.** 63375 > 329 VCL_acl c MATCH purge "**.**.**.**"/32 Just an idea: we need an "anonymous" option to varnishlog, where all sensitive data gets replaced by random but consistent garbage. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Wed Apr 25 13:21:18 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 25 Apr 2007 13:21:18 +0000 Subject: Problems with purging /. In-Reply-To: Your message of "Wed, 25 Apr 2007 15:18:50 +0200." <20070425131850.GB19155@aftenposten.no> Message-ID: <8622.1177507278@critter.freebsd.dk> In message <20070425131850.GB19155 at aftenposten.no>, Thomas Westlund writes: >> We need to see the Varnish log for this transaction. >This is the output: > > [...] > >IPs and hostnames are changed to preserver privacy. > >Iv'e already checked the logs to see what happens, as far as I'm able to se it works as it should, >but the / pages are stil not updated when I try to acesss them via the cache, but they are on the backend-server. What we really need to see is the log for the next GET of / -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Wed Apr 25 13:37:49 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 25 Apr 2007 13:37:49 +0000 Subject: Problems with purging /. In-Reply-To: Your message of "Wed, 25 Apr 2007 15:11:51 +0200." <20070425131151.GA19155@aftenposten.no> Message-ID: <9195.1177508269@critter.freebsd.dk> In message <20070425131151.GA19155 at aftenposten.no>, Thomas Westlund writes: >Hi, > >On Wed, Apr 25, 2007 at 12:15:17PM +0000, Poul-Henning Kamp wrote: >> Uhm, the PURGE request gets the "200" answer you quote from >> the: >> error 200 "Purged."; >> >> What makes you think the object is not purged ? > >If I update for example the front page on the backend and run the purge-routine for / as I discribed earlier,the document is not updated when I test it via the cache, >but it is when I access it directly on the backend. And / is not by any chance hit by a 3xx redirect ? If it is, then you need to purge the destination instead. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Wed Apr 25 14:08:50 2007 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Wed, 25 Apr 2007 16:08:50 +0200 Subject: Problems with purging /. In-Reply-To: <20070425131850.GB19155@aftenposten.no> (Thomas Westlund's message of "Wed, 25 Apr 2007 15:18:50 +0200") References: <20070425121016.GA17668@aftenposten.no> <20070425131850.GB19155@aftenposten.no> Message-ID: Thomas Westlund writes: > This is the output: > > 329 SessionOpen c **.**.**.** 63375 > 329 VCL_acl c MATCH purge "**.**.**.**"/32 > 329 SessionClose c Purged. > 329 ReqStart c **.**.**.** 63375 326308940 > 329 RxRequest c PURGE > 329 RxURL c / > 329 RxProtocol c HTTP/1.0 > 329 RxHeader c Host: www.**.no > 329 VCL_call c recv lookup > 329 Hit c 326307698 > 329 VCL_call c hit > 329 TTL c 326307698 VCL 0 1177506682 > 329 VCL_return c error > 329 TxStatus c 200 > 329 TxProtocol c HTTP/1.1 > 329 TxResponse c OK > 329 Req > > IPs and hostnames are changed to preserver privacy. Everything here is fine. Varnish did precisely what you asked it to do. The error is either in your backend or your expectations... DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From thomas.westlund at aftenposten.no Wed Apr 25 14:17:57 2007 From: thomas.westlund at aftenposten.no (Thomas Westlund) Date: Wed, 25 Apr 2007 16:17:57 +0200 Subject: Problems with purging /. In-Reply-To: References: <20070425121016.GA17668@aftenposten.no> <20070425131850.GB19155@aftenposten.no> Message-ID: <20070425141757.GA20988@aftenposten.no> Hi, On Wed, Apr 25, 2007 at 04:08:50PM +0200, Dag-Erling Sm?rgrav wrote: > Thomas Westlund writes: > > This is the output: > > > > 329 SessionOpen c **.**.**.** 63375 > > 329 VCL_acl c MATCH purge "**.**.**.**"/32 > > 329 SessionClose c Purged. > > 329 ReqStart c **.**.**.** 63375 326308940 > > 329 RxRequest c PURGE > > 329 RxURL c / > > 329 RxProtocol c HTTP/1.0 > > 329 RxHeader c Host: www.**.no > > 329 VCL_call c recv lookup > > 329 Hit c 326307698 > > 329 VCL_call c hit > > 329 TTL c 326307698 VCL 0 1177506682 > > 329 VCL_return c error > > 329 TxStatus c 200 > > 329 TxProtocol c HTTP/1.1 > > 329 TxResponse c OK > > 329 Req > > > > IPs and hostnames are changed to preserver privacy. > > Everything here is fine. Varnish did precisely what you asked it to > do. The error is either in your backend or your expectations... > > DES Thanks for the help so far, I've managed to find logs that confirms that varnish does in fact fetch the page anew after a purge. I'll have the customer look at their pages so confirmed that we are not fooled by linked pages somehow. Sorry for the inconvenience... -- Thomas Westlund Aftenposten AS, UNIX/nettverksavd. Postboks 1, 0051 Oslo Tlf: +47 98 20 30 33 Fax: +47 22 86 40 74 From denis at startsiden.no Wed Apr 25 18:33:25 2007 From: denis at startsiden.no (=?utf-8?Q?Denis_Br=C3=A6khus?=) Date: Wed, 25 Apr 2007 20:33:25 +0200 (CEST) Subject: Gzip issues with Varnish Message-ID: <13408569.151821177526005830.JavaMail.root@ms1.startsiden.no> Hi, One of my developer colleagues has experienced an issue at Aftenposten with regards to gzip and varnish. I think you are experiencing the same issue we had, that is Varnish doesn't really handle gzipped data from the backend intelligently yet. So you could have the following scenario: 0. Varnish startup 1. Client A (with a gzip capable client) requests object 1 2. Varnish will fetch the object from the backend and pass it on to the client, and store the gzipped object in the cache 3. Client B (with a non gzip capable client) requests object 1 4. Varnish will pass on the gzipped object to the client 5. The client will not be able to interpret the data received, and you get various errors.. We "solved" the issue by disabling gzip on the backend until Varnish will handle this properly. For us the sacrifice in bandwidth savings was more than negligible when we take into consideration application server performance. Regards -- Denis Braekhus - Teknisk Ansvarlig ABC Startsiden AS http://www.startsiden.no From dirk at dirkgomez.de Fri Apr 27 12:22:50 2007 From: dirk at dirkgomez.de (Dirk Gomez) Date: Fri, 27 Apr 2007 14:22:50 +0200 Subject: Varnish logging In-Reply-To: References: <46560.1177011616@critter.freebsd.dk> Message-ID: <3E18A25B-EE46-49CB-BB3D-28BB9BA2B5ED@dirkgomez.de> > Dirk, if you are comfortable building Varnish from sources, try > grabbing the following file: > > http://varnish.projects.linpro.no/svn/trunk/varnish-cache/bin/ > varnishncsa/varnishncsa.c > > Stick it in the obvious place in your source tree, rebuild and > reinstall. It will work fine in a 1.0 tree. DES, here's what I did: * Checked out the 1.0.3 tag. * Checkout above file from trunk. * Stuck it into the obvious place * Compiled, created and installed a Debian varnish-103 package. varnishncsa doesn't write to a log file anymore though. execve("/usr/bin/varnishncsa", ["/usr/bin/varnishncsa", "-w", "foo"], [/* 19 vars */]) = 0 uname({sys="Linux", node="test01.dirkgomez.de", ...}) = 0 brk(0) = 0x804b000 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40017000 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0644, st_size=19929, ...}) = 0 mmap2(NULL, 19929, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40019000 close(3) = 0 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) open("/usr/lib/libvarnish.so.0", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\20\21\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0644, st_size=16468, ...}) = 0 mmap2(NULL, 19480, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x4001e000 mmap2(0x40022000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED| MAP_DENYWRITE, 3, 0x3) = 0x40022000 close(3) = 0 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) open("/usr/lib/libvarnishapi.so.0", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\360\f\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0644, st_size=11352, ...}) = 0 mmap2(NULL, 14600, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x40023000 mmap2(0x40026000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED| MAP_DENYWRITE, 3, 0x2) = 0x40026000 close(3) = 0 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) open("/lib/tls/libpthread.so.0", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\360G\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=85010, ...}) = 0 mmap2(NULL, 70104, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x40166000 mmap2(0x40174000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED| MAP_DENYWRITE, 3, 0xd) = 0x40174000 mmap2(0x40176000, 4568, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED| MAP_ANONYMOUS, -1, 0) = 0x40176000 close(3) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40178000 mprotect(0x4015b000, 20480, PROT_READ) = 0 set_thread_area({entry_number:-1 -> 6, base_addr:0x401786c0, limit: 1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0 munmap(0x40019000, 19929) = 0 set_tid_address(0x40178708) = 8910 rt_sigaction(SIGRTMIN, {0x4016a450, [], SA_SIGINFO}, NULL, 8) = 0 rt_sigaction(SIGRT_1, {0x4016a3c0, [], SA_RESTART|SA_SIGINFO}, NULL, 8) = 0 rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0 getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0 uname({sys="Linux", node="test01.dirkgomez.de", ...}) = 0 brk(0) = 0x804b000 brk(0x807c000) = 0x807c000 open("/tmp/_.vsl", O_RDONLY) = 3 read(3, "2\332y\371\214\1\0\0=\3501F\344\1\0\0\0\0\200\0+\362\1"..., 396) = 396 mmap2(NULL, 8389004, PROT_READ, MAP_SHARED, 3, 0) = 0x40179000 open("foo", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 4 rt_sigaction(SIGHUP, {0x8048b90, [HUP], SA_RESTART}, {SIG_DFL}, 8) = 0 nanosleep({0, 50000000}, NULL) = 0 nanosleep({0, 50000000}, NULL) = 0 nanosleep({0, 50000000}, NULL) = 0 ... and then it continues to nanosleep. varnishlog though does show activity on the server. -- Dirk From lists at dirkgomez.de Mon Apr 30 10:58:03 2007 From: lists at dirkgomez.de (Dirk Gomez) Date: Mon, 30 Apr 2007 12:58:03 +0200 Subject: Varnish logging In-Reply-To: References: <46560.1177011616@critter.freebsd.dk> <3E18A25B-EE46-49CB-BB3D-28BB9BA2B5ED@dirkgomez.de> Message-ID: > You mean -w is broken, or that it stops? Does it work if you do not > specify -w? What exact command line are you using? No output is written with this command: varnishncsa -adw foo (Just tried: it now segfaults) From des at linpro.no Mon Apr 30 11:15:44 2007 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Mon, 30 Apr 2007 13:15:44 +0200 Subject: Varnish logging In-Reply-To: (Dirk Gomez's message of "Mon, 30 Apr 2007 12:58:03 +0200") References: <46560.1177011616@critter.freebsd.dk> <3E18A25B-EE46-49CB-BB3D-28BB9BA2B5ED@dirkgomez.de> Message-ID: Dirk Gomez writes: > No output is written with this command: > > varnishncsa -adw foo > > (Just tried: it now segfaults) I'm unable to reproduce this. Are you sure varnishd and varnishncsa are using the same library versions? And do you have a backtrace for the segfault? DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Mon Apr 30 10:18:38 2007 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Mon, 30 Apr 2007 12:18:38 +0200 Subject: Varnish logging In-Reply-To: <3E18A25B-EE46-49CB-BB3D-28BB9BA2B5ED@dirkgomez.de> (Dirk Gomez's message of "Fri, 27 Apr 2007 14:22:50 +0200") References: <46560.1177011616@critter.freebsd.dk> <3E18A25B-EE46-49CB-BB3D-28BB9BA2B5ED@dirkgomez.de> Message-ID: Dirk Gomez writes: > DES, here's what I did: > > * Checked out the 1.0.3 tag. > * Checkout above file from trunk. > * Stuck it into the obvious place > * Compiled, created and installed a Debian varnish-103 package. > > varnishncsa doesn't write to a log file anymore though. You mean -w is broken, or that it stops? Does it work if you do not specify -w? What exact command line are you using? DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Mon Apr 30 15:40:36 2007 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Mon, 30 Apr 2007 17:40:36 +0200 Subject: Gzip issues with Varnish In-Reply-To: <13408569.151821177526005830.JavaMail.root@ms1.startsiden.no> (Denis =?iso-8859-1?Q?Br=E6khus's?= message of "Wed, 25 Apr 2007 20:33:25 +0200 (CEST)") References: <13408569.151821177526005830.JavaMail.root@ms1.startsiden.no> Message-ID: Denis Br?khus writes: > One of my developer colleagues has experienced an issue at > Aftenposten with regards to gzip and varnish. > > I think you are experiencing the same issue we had, that is Varnish > doesn't really handle gzipped data from the backend intelligently > yet. So you could have the following scenario: > > 0. Varnish startup > 1. Client A (with a gzip capable client) requests object 1 > 2. Varnish will fetch the object from the backend and pass it on to > the client, and store the gzipped object in the cache > 3. Client B (with a non gzip capable client) requests object 1 > 4. Varnish will pass on the gzipped object to the client > 5. The client will not be able to interpret the data received, and > you get various errors.. Yes, this is exactly what will happen, because Varnish does not yet understand the Vary: header which the server uses to indicate that the document it returns can vary depending on request headers. > We "solved" the issue by disabling gzip on the backend until Varnish > will handle this properly. For us the sacrifice in bandwidth savings > was more than negligible when we take into consideration application > server performance. Unfortunately, for some applications, the bandwidth savings are considerable... which is why Vary: handling is very high on our priority list. We expect to have it working in trunk within a couple of months. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Mon Apr 30 15:36:11 2007 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Mon, 30 Apr 2007 17:36:11 +0200 Subject: Varnish logging In-Reply-To: <21F37A05-AC36-4A02-93A7-3F6EBB6C8A72@dirkgomez.de> (Dirk Gomez's message of "Mon, 30 Apr 2007 17:32:56 +0200") References: <46560.1177011616@critter.freebsd.dk> <3E18A25B-EE46-49CB-BB3D-28BB9BA2B5ED@dirkgomez.de> <2BFC60A3-E620-46AC-9174-AC41766563BC@dirkgomez.de> <21F37A05-AC36-4A02-93A7-3F6EBB6C8A72@dirkgomez.de> Message-ID: Dirk Gomez writes: > varnishncsa -d outputs something now - just for one of our "virtual > hosts", none other makes it to varnishncsa's output. varnishlog works > fine tho: it outputs the other vhosts as well. OK, I would like you to do the following: - capture a few minutes of traffic to a file using 'varnishlog -w file' - verify that 'varnishncsa -r file' exhibits the bug you experience when running it "live" - send me that file (directly, not to the list) Hopefully, I will be able to find and correct the error quickly. I'm very busy this week, though, so please be patient. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Mon Apr 30 12:18:43 2007 From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) Date: Mon, 30 Apr 2007 14:18:43 +0200 Subject: Varnish logging In-Reply-To: <2BFC60A3-E620-46AC-9174-AC41766563BC@dirkgomez.de> (Dirk Gomez's message of "Mon, 30 Apr 2007 13:41:49 +0200") References: <46560.1177011616@critter.freebsd.dk> <3E18A25B-EE46-49CB-BB3D-28BB9BA2B5ED@dirkgomez.de> <2BFC60A3-E620-46AC-9174-AC41766563BC@dirkgomez.de> Message-ID: Dirk Gomez writes: > (Debian Etch, self-built varnish-103 package with your "HOST" patch > from trunk for varnishncsa) I assume you rebuilt and reinstalled the entire package? > varnishncsa returns to its nanosleep routine. Does varnishlog behave normally? > Should I recompile with --enable-developer-warnings and --enable- > debugging-symbols set to yes and retry strace? Those flags won't make any difference for strace, but they will make it easier to get a backtrace if it segfaults again. Please remember to 'make clean' before rebuilding with these flags, otherwise they won't be applied. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From denis at startsiden.no Mon Apr 30 15:38:11 2007 From: denis at startsiden.no (=?utf-8?Q?Denis_Br=C3=A6khus?=) Date: Mon, 30 Apr 2007 17:38:11 +0200 (CEST) Subject: Gzip issues with Varnish In-Reply-To: Message-ID: <9609640.180151177947491492.JavaMail.root@ms1.startsiden.no> ----- Dag-Erling Sm?rgrav wrote: > Denis Br?khus writes: > Yes, this is exactly what will happen, because Varnish does not yet > understand the Vary: header which the server uses to indicate that > the document it returns can vary depending on request headers. I forgot to mention that it was of course DES who solved the issue for us. > > We "solved" the issue by disabling gzip on the backend until > > Varnish will handle this properly. For us the sacrifice in bandwidth > > savings was more than negligible when we take into consideration > > application server performance. > Unfortunately, for some applications, the bandwidth savings are > considerable... which is why Vary: handling is very high on our > priority list. We expect to have it working in trunk within a couple > of months. Yes I know, that's why i said "for us" :) And I would of course love to have both a speedy appserver and gzipped transfers. Meanwhile I think this is an issue sites using Varnish should be aware of. And Aftenposten should probably fix their setup.. Regards -- Denis Braekhus - Teknisk Ansvarlig ABC Startsiden AS http://www.startsiden.no From jean-marc.pouchoulon at ac-montpellier.fr Mon Apr 30 12:14:42 2007 From: jean-marc.pouchoulon at ac-montpellier.fr (jean-marc pouchoulon) Date: Mon, 30 Apr 2007 14:14:42 +0200 Subject: multiple backend + apache bottleneck Message-ID: <4635DDB2.1050400@ac-montpellier.fr> Hello all , We are using zope/nuxeo cps with 8 zeo clients and I test varnish 1.3 in place of squid 2.6. squid uses 8 zeo peers and if I've understood well varnish can have "only one" host per backend. is there is a way to load balance upon multiples hosts with varnish ? Varnish is so fast that the bottleneck is now apache ( we are using apache with MPM workers + mod_rewrite in front of our site ). Performances are equals in that design with squid / varnish ( near 1000 r/s on a quadriprocessor) Not really a varnish question but maybe you can give me an advice to have a full profit of varnish. jean-marc. From lists at dirkgomez.de Mon Apr 30 15:32:56 2007 From: lists at dirkgomez.de (Dirk Gomez) Date: Mon, 30 Apr 2007 17:32:56 +0200 Subject: Varnish logging In-Reply-To: References: <46560.1177011616@critter.freebsd.dk> <3E18A25B-EE46-49CB-BB3D-28BB9BA2B5ED@dirkgomez.de> <2BFC60A3-E620-46AC-9174-AC41766563BC@dirkgomez.de> Message-ID: <21F37A05-AC36-4A02-93A7-3F6EBB6C8A72@dirkgomez.de> >> (Debian Etch, self-built varnish-103 package with your "HOST" patch >> from trunk for varnishncsa) > > I assume you rebuilt and reinstalled the entire package? Yes (so that everything fits together). > >> varnishncsa returns to its nanosleep routine. > > Does varnishlog behave normally? Yes. >> Should I recompile with --enable-developer-warnings and --enable- >> debugging-symbols set to yes and retry strace? > > Those flags won't make any difference for strace, but they will make > it easier to get a backtrace if it segfaults again. Please remember > to 'make clean' before rebuilding with these flags, otherwise they > won't be applied. No more segfaults. varnishncsa -d outputs something now - just for one of our "virtual hosts", none other makes it to varnishncsa's output. varnishlog works fine tho: it outputs the other vhosts as well. From lists at dirkgomez.de Mon Apr 30 11:41:49 2007 From: lists at dirkgomez.de (Dirk Gomez) Date: Mon, 30 Apr 2007 13:41:49 +0200 Subject: Varnish logging In-Reply-To: References: <46560.1177011616@critter.freebsd.dk> <3E18A25B-EE46-49CB-BB3D-28BB9BA2B5ED@dirkgomez.de> Message-ID: <2BFC60A3-E620-46AC-9174-AC41766563BC@dirkgomez.de> >> No output is written with this command: >> >> varnishncsa -adw foo >> >> (Just tried: it now segfaults) > > I'm unable to reproduce this. Are you sure varnishd and varnishncsa > are using the same library versions? And do you have a backtrace for > the segfault? Hmm, not able to reproduce twenty minutes later. Here are the library versions: anadyr:~# ldd /usr/bin/varnishncsa linux-gate.so.1 => (0xffffe000) libvarnish.so.0 => /usr/lib/libvarnish.so.0 (0x4001e000) libvarnishapi.so.0 => /usr/lib/libvarnishapi.so.0 (0x40023000) libdl.so.2 => /lib/tls/libdl.so.2 (0x40027000) librt.so.1 => /lib/tls/librt.so.1 (0x4002b000) libc.so.6 => /lib/tls/libc.so.6 (0x40033000) /lib/ld-linux.so.2 (0x40000000) libpthread.so.0 => /lib/tls/libpthread.so.0 (0x40166000) anadyr:~# ldd /usr/sbin/varnishd linux-gate.so.1 => (0xffffe000) libvarnish.so.0 => /usr/lib/libvarnish.so.0 (0x4001e000) libvcl.so.0 => /usr/lib/libvcl.so.0 (0x40023000) libdl.so.2 => /lib/tls/libdl.so.2 (0x40030000) librt.so.1 => /lib/tls/librt.so.1 (0x40034000) libc.so.6 => /lib/tls/libc.so.6 (0x4003c000) libpthread.so.0 => /lib/tls/libpthread.so.0 (0x4016f000) /lib/ld-linux.so.2 (0x40000000) (Debian Etch, self-built varnish-103 package with your "HOST" patch from trunk for varnishncsa) varnishncsa returns to its nanosleep routine. Should I recompile with --enable-developer-warnings and --enable- debugging-symbols set to yes and retry strace?