From tfheen at linpro.no Mon Dec 1 10:11:42 2008 From: tfheen at linpro.no (Tollef Fog Heen) Date: Mon, 01 Dec 2008 11:11:42 +0100 Subject: varnish2.0.2 on Suse 10.3 In-Reply-To: <75cf5800811262158n7819ac40m83f4ddea2d71093e@mail.gmail.com> (Paras Fadte's message of "Thu, 27 Nov 2008 11:28:32 +0530") References: <75cf5800811200134y7cbcc0bew2dd2f113ce710309@mail.gmail.com> <86db848d0811200242j5543663yc9fc09857aae2a81@mail.gmail.com> <75cf5800811200746x74e6ba8fyf9711e1680e28d1f@mail.gmail.com> <878wre9uf1.fsf@qurzaw.linpro.no> <75cf5800811232042j77251155s452df10e499f8dc8@mail.gmail.com> <75cf5800811262158n7819ac40m83f4ddea2d71093e@mail.gmail.com> Message-ID: <87wsek5hnl.fsf@qurzaw.linpro.no> ]] "Paras Fadte" | child (28673) Started | Pushing vcls failed: Internal error: No VCL_conf symbol It seems to fail to load the configuration, probably because the .so with the configuration ends up stripped. Why that happens, I have no idea. -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 From perbu at linpro.no Mon Dec 1 12:33:57 2008 From: perbu at linpro.no (Per Buer) Date: Mon, 01 Dec 2008 13:33:57 +0100 Subject: varnish2.0.2 on Suse 10.3 In-Reply-To: <87wsek5hnl.fsf@qurzaw.linpro.no> References: <75cf5800811200134y7cbcc0bew2dd2f113ce710309@mail.gmail.com> <86db848d0811200242j5543663yc9fc09857aae2a81@mail.gmail.com> <75cf5800811200746x74e6ba8fyf9711e1680e28d1f@mail.gmail.com> <878wre9uf1.fsf@qurzaw.linpro.no> <75cf5800811232042j77251155s452df10e499f8dc8@mail.gmail.com> <75cf5800811262158n7819ac40m83f4ddea2d71093e@mail.gmail.com> <87wsek5hnl.fsf@qurzaw.linpro.no> Message-ID: <4933D9B5.6010709@linpro.no> Tollef Fog Heen skrev: > ]] "Paras Fadte" > > | child (28673) Started > | Pushing vcls failed: Internal error: No VCL_conf symbol > > It seems to fail to load the configuration, probably because the .so > with the configuration ends up stripped. Why that happens, I have no > idea. It seems to be a recurring issue on OpenSUSE. There have been earlier reports on the same issue. -- Per Buer - Leder Infrastruktur og Drift - Redpill Linpro Telefon: 21 54 41 21 - Mobil: 958 39 117 http://linpro.no/ | http://redpill.se/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 252 bytes Desc: OpenPGP digital signature URL: From jeff at funnyordie.com Mon Dec 1 23:43:53 2008 From: jeff at funnyordie.com (Jeff Anderson) Date: Mon, 1 Dec 2008 15:43:53 -0800 Subject: Vary Header Strange Behavior Message-ID: The latest version of Varnish seems to behave unexpectedly with regard to the Vary on Accept-Encoding headers. Our production servers report back different headers than our development and test varnish servers even though they are identical VCL. In production it serves up uncompressed versions no matter what version is requested. The vary header doesn't even show up like it does on the development severs. $ curl http://prodweb9 --head --compressed HTTP/1.1 200 OK Content-Type: text/html; charset=utf-8 X-Runtime: 0.58210 ETag: "359535e914cddcc7ce0c879f1b3d4668" Cache-Control: no-cache, public, max-age=300 Server: LiteSpeed Content-Length: 60711 Date: Mon, 01 Dec 2008 23:10:47 GMT X-Varnish: 1013927736 1013926899 Age: 59 Via: 1.1 varnish Served-by: prodweb9/prodapp7 Connection: close $ curl http://prodweb9 --head HTTP/1.1 200 OK Content-Type: text/html; charset=utf-8 X-Runtime: 0.58210 ETag: "359535e914cddcc7ce0c879f1b3d4668" Cache-Control: no-cache, public, max-age=300 Server: LiteSpeed Content-Length: 60711 Date: Mon, 01 Dec 2008 23:10:56 GMT X-Varnish: 1013927873 1013926899 Age: 69 Via: 1.1 varnish Served-by: prodweb9/prodapp7 Connection: close However on our development servers: $ curl http://devweb1 --head HTTP/1.1 200 OK Content-Type: text/html; charset=utf-8 ETag: "4ebc8c72bde3a68dc965f52d392ac630" X-Runtime: 0.12171 Cache-Control: no-cache, public, max-age=300 Vary: Accept-Encoding X-Varnish: 191681248 Age: 0 Via: 1.1 varnish Served-by: devweb1/devapp1 Date: Mon, 01 Dec 2008 23:04:15 GMT Server: LiteSpeed Connection: Keep-Alive Keep-Alive: timeout=5, max=100 WWW-Authenticate: Basic realm="Protected Area" $ curl http://devweb1 --head --compressed HTTP/1.1 200 OK Content-Type: text/html; charset=utf-8 ETag: "4ebc8c72bde3a68dc965f52d392ac630" X-Runtime: 0.12171 Cache-Control: no-cache, public, max-age=300 Content-Encoding: gzip Vary: Accept-Encoding X-Varnish: 191681255 191681248 Age: 30 Via: 1.1 varnish Served-by: devweb1/devapp1 Date: Mon, 01 Dec 2008 23:04:45 GMT Server: LiteSpeed Connection: Keep-Alive Keep-Alive: timeout=5, max=100 WWW-Authenticate: Basic realm="Protected Area" The lines below were added to the VCL to resolve the problem. And while they did resolve the issue it seems to have made varnish double the look up of incoming requests according to the varnishncsa log. sub vcl_hash { if (req.http.Accept-Encoding ~ "gzip" || req.http.Accept-Encoding ~ "deflate") { set req.hash += req.http.Accept-Encoding; } Is this normal or is something overriding the proper functioning of varnish? Thanks, --Jeff From plfgoa at gmail.com Tue Dec 2 06:23:27 2008 From: plfgoa at gmail.com (Paras Fadte) Date: Tue, 2 Dec 2008 11:53:27 +0530 Subject: varnish2.0.2 on Suse 10.3 In-Reply-To: <4933D9B5.6010709@linpro.no> References: <75cf5800811200134y7cbcc0bew2dd2f113ce710309@mail.gmail.com> <86db848d0811200242j5543663yc9fc09857aae2a81@mail.gmail.com> <75cf5800811200746x74e6ba8fyf9711e1680e28d1f@mail.gmail.com> <878wre9uf1.fsf@qurzaw.linpro.no> <75cf5800811232042j77251155s452df10e499f8dc8@mail.gmail.com> <75cf5800811262158n7819ac40m83f4ddea2d71093e@mail.gmail.com> <87wsek5hnl.fsf@qurzaw.linpro.no> <4933D9B5.6010709@linpro.no> Message-ID: <75cf5800812012223t2334829cm8371b77e05c9e70a@mail.gmail.com> Hi Per, But it runs fine on Suse 10.1 when built from source . Any major changes from suse 10.1 to sue 10.3 that could have caused the issue ? Would be happy if Poul-Henning Kamp can put some light on this issue . Thank you. -Paras On Mon, Dec 1, 2008 at 6:03 PM, Per Buer wrote: > Tollef Fog Heen skrev: >> ]] "Paras Fadte" >> >> | child (28673) Started >> | Pushing vcls failed: Internal error: No VCL_conf symbol >> >> It seems to fail to load the configuration, probably because the .so >> with the configuration ends up stripped. Why that happens, I have no >> idea. > > It seems to be a recurring issue on OpenSUSE. There have been earlier > reports on the same issue. > > -- > Per Buer - Leder Infrastruktur og Drift - Redpill Linpro > Telefon: 21 54 41 21 - Mobil: 958 39 117 > http://linpro.no/ | http://redpill.se/ > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc > > From plfgoa at gmail.com Tue Dec 2 06:24:20 2008 From: plfgoa at gmail.com (Paras Fadte) Date: Tue, 2 Dec 2008 11:54:20 +0530 Subject: varnish2.0.2 on Suse 10.3 In-Reply-To: <87wsek5hnl.fsf@qurzaw.linpro.no> References: <75cf5800811200134y7cbcc0bew2dd2f113ce710309@mail.gmail.com> <86db848d0811200242j5543663yc9fc09857aae2a81@mail.gmail.com> <75cf5800811200746x74e6ba8fyf9711e1680e28d1f@mail.gmail.com> <878wre9uf1.fsf@qurzaw.linpro.no> <75cf5800811232042j77251155s452df10e499f8dc8@mail.gmail.com> <75cf5800811262158n7819ac40m83f4ddea2d71093e@mail.gmail.com> <87wsek5hnl.fsf@qurzaw.linpro.no> Message-ID: <75cf5800812012224r13d009b0s6ed434c66fb6e107@mail.gmail.com> Would be happy if Poul-Henning Kamp can put some light on this issue . On Mon, Dec 1, 2008 at 3:41 PM, Tollef Fog Heen wrote: > ]] "Paras Fadte" > > | child (28673) Started > | Pushing vcls failed: Internal error: No VCL_conf symbol > > It seems to fail to load the configuration, probably because the .so > with the configuration ends up stripped. Why that happens, I have no > idea. > > -- > Tollef Fog Heen > Redpill Linpro -- Changing the game! > t: +47 21 54 41 73 > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc > From plfgoa at gmail.com Tue Dec 2 11:21:28 2008 From: plfgoa at gmail.com (Paras Fadte) Date: Tue, 2 Dec 2008 16:51:28 +0530 Subject: sendfile() and Varnish2.0.2 Message-ID: <75cf5800812020321t1cbf0686t5f170d301c42d678@mail.gmail.com> Hi , Are there issues associated with using sendfile() in varnish2.0.2 ? Does varnish 2.0.2 use sendfile() by default ? I have read that currently it supports sendfile() only on FreeBSD or does it support OS like SUSE LINUX 10.1 (X86-64) and Kernel 2.6.16.13-4-smp ? I also tend to see increase in "overflowed work requests" over a period of time . How can they be controlled ? Thank you. -Paras From plfgoa at gmail.com Tue Dec 2 11:28:00 2008 From: plfgoa at gmail.com (Paras Fadte) Date: Tue, 2 Dec 2008 16:58:00 +0530 Subject: sendfile() and Varnish2.0.2 Message-ID: <75cf5800812020328u6b238d4dg55b55ba7d1222383@mail.gmail.com> Hi , Are there issues associated with using sendfile() in varnish2.0.2 ? Does varnish 2.0.2 use sendfile() by default ? I have read that currently it supports sendfile() only on FreeBSD or does it support OS like SUSE LINUX 10.1 (X86-64) and Kernel 2.6.16.13-4-smp ? I also tend to see increase in "overflowed work requests" over a period of time . How can they be controlled ? Thank you. -Paras From perbu at linpro.no Tue Dec 2 14:32:44 2008 From: perbu at linpro.no (Per Buer) Date: Tue, 02 Dec 2008 15:32:44 +0100 Subject: sendfile() and Varnish2.0.2 In-Reply-To: <75cf5800812020321t1cbf0686t5f170d301c42d678@mail.gmail.com> References: <75cf5800812020321t1cbf0686t5f170d301c42d678@mail.gmail.com> Message-ID: <4935470C.30200@linpro.no> Paras Fadte skrev: > Hi , > > Are there issues associated with using sendfile() in varnish2.0.2 ? Yes. sendfile is turned off on Linux and FreeBSD. The performance impact is quite minimal. > Does varnish 2.0.2 use sendfile() by default ? Only on Solaris. -- Per Buer - Leder Infrastruktur og Drift - Redpill Linpro Telefon: 21 54 41 21 - Mobil: 958 39 117 http://linpro.no/ | http://redpill.se/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 252 bytes Desc: OpenPGP digital signature URL: From phk at phk.freebsd.dk Tue Dec 2 14:39:13 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 02 Dec 2008 14:39:13 +0000 Subject: sendfile() and Varnish2.0.2 In-Reply-To: Your message of "Tue, 02 Dec 2008 15:32:44 +0100." <4935470C.30200@linpro.no> Message-ID: <9488.1228228753@critter.freebsd.dk> In message <4935470C.30200 at linpro.no>, Per Buer writes: >> Are there issues associated with using sendfile() in varnish2.0.2 ? > >Yes. sendfile is turned off on Linux and FreeBSD. The performance impact >is quite minimal. It works on FreeBSD 8.* version, but you have to enable it by setting a threshold above which it will be used. It is not obvious that it is a benefit to use sendfile on any platform. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at des.no Tue Dec 2 15:23:15 2008 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Tue, 02 Dec 2008 16:23:15 +0100 Subject: sendfile() and Varnish2.0.2 In-Reply-To: <4935470C.30200@linpro.no> (Per Buer's message of "Tue, 02 Dec 2008 15:32:44 +0100") References: <75cf5800812020321t1cbf0686t5f170d301c42d678@mail.gmail.com> <4935470C.30200@linpro.no> Message-ID: <8663m24n4s.fsf@ds4.des.no> Per Buer writes: > Paras Fadte writes: > > Are there issues associated with using sendfile() in varnish2.0.2 ? > Yes. sendfile is turned off on Linux and FreeBSD. The performance impact > is quite minimal. > > > Does varnish 2.0.2 use sendfile() by default ? > Only on Solaris. No. Neither Linux nor Solaris have a version of sendfile that Varnish can use. FreeBSD 8 does (and so would FreeBSD 7 if phk had remembered to MFC it). Enabling sendfile on anything other than FreeBSD 8 is a no-op. In any case, sendfile only works with file storage, not malloc, and due to the overhead of setting up the VM mapping for the transfer, the break-even point is very high - you're not likely to see any measurable performance improvement unless you're serving very large objects. DES -- Dag-Erling Sm?rgrav - des at des.no From phk at phk.freebsd.dk Tue Dec 2 15:38:20 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 02 Dec 2008 15:38:20 +0000 Subject: sendfile() and Varnish2.0.2 In-Reply-To: Your message of "Tue, 02 Dec 2008 16:23:15 +0100." <8663m24n4s.fsf@ds4.des.no> Message-ID: <9734.1228232300@critter.freebsd.dk> In message <8663m24n4s.fsf at ds4.des.no>, =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= wr ites: > No. Neither Linux nor Solaris have a version of sendfile that Varnish > can use. FreeBSD 8 does (and so would FreeBSD 7 if phk had remembered > to MFC it). MFC'ing it changes an obscure kernel-API slightly, so absent some really good evidence that it does wonders, I didn't feel it was worth the MFC. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From nork at ninth-nine.com Tue Dec 2 17:48:12 2008 From: nork at ninth-nine.com (Norikatsu Shigemura) Date: Wed, 3 Dec 2008 02:48:12 +0900 Subject: sendfile() and Varnish2.0.2 In-Reply-To: <9734.1228232300@critter.freebsd.dk> References: <8663m24n4s.fsf@ds4.des.no> <9734.1228232300@critter.freebsd.dk> Message-ID: <20081203024812.6f00266d.nork@ninth-nine.com> Hi phk. On Tue, 02 Dec 2008 15:38:20 +0000 "Poul-Henning Kamp" wrote: > > No. Neither Linux nor Solaris have a version of sendfile that Varnish > > can use. FreeBSD 8 does (and so would FreeBSD 7 if phk had remembered > > to MFC it). > MFC'ing it changes an obscure kernel-API slightly, so absent some really > good evidence that it does wonders, I didn't feel it was worth the MFC. What is the difference between FreeBSD 8 and 7? From phk at phk.freebsd.dk Tue Dec 2 18:09:20 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 02 Dec 2008 18:09:20 +0000 Subject: sendfile() and Varnish2.0.2 In-Reply-To: Your message of "Wed, 03 Dec 2008 02:48:12 +0900." <20081203024812.6f00266d.nork@ninth-nine.com> Message-ID: <12832.1228241360@critter.freebsd.dk> In message <20081203024812.6f00266d.nork at ninth-nine.com>, Norikatsu Shigemura w rites: > What is the difference between FreeBSD 8 and 7? Presently 8 is "-current" which means it is the development branch of FreeBSD, so it is pretty much a moving target. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From jeff at funnyordie.com Wed Dec 3 02:54:39 2008 From: jeff at funnyordie.com (Jeff Anderson) Date: Tue, 2 Dec 2008 18:54:39 -0800 Subject: Varnish Serves only uncompressed objects if they are requested first Message-ID: <4DB04145-6ECB-4887-AEAC-BD7AF508E28F@funnyordie.com> It looks like if the first requested page is for an uncompressed page varnish will only deliver the uncompressed page from cache even if a compressed page is requested. --After a fresh bounce of varnish--- **** Requesting uncompressed page first Curl request: $ curl http://prodweb9/ --head HTTP/1.1 200 OK Content-Type: text/html; charset=utf-8 X-Runtime: 2.06915 ETag: "c10816a9ec10679b3ac800a4b2e3e70b" Cache-Control: no-cache, public, max-age=300 Server: LiteSpeed Content-Length: 60261 Date: Wed, 03 Dec 2008 02:19:15 GMT X-Varnish: 353939895 Age: 0 Via: 1.1 varnish Served-by: prodweb9/prodapp5 Connection: close Varnishlog output: 8 SessionOpen c 172.16.200.145 56473 :80 8 ReqStart c 172.16.200.145 56473 353939895 8 RxRequest c HEAD 8 RxURL c / 8 RxProtocol c HTTP/1.1 8 RxHeader c User-Agent: curl/7.16.3 (powerpc-apple- darwin9.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3 8 RxHeader c Host: prodweb9 8 RxHeader c Accept: */* 8 VCL_call c recv 8 VCL_return c lookup 8 VCL_call c hash 8 VCL_return c hash 8 VCL_call c miss 8 VCL_return c fetch 9 BackendClose b default 9 BackendOpen b default 172.16.160.85 23565 172.16.160.174 80 8 Backend c 9 default default 9 TxRequest b GET 9 TxURL b / 9 TxProtocol b HTTP/1.1 9 TxHeader b User-Agent: curl/7.16.3 (powerpc-apple- darwin9.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3 9 TxHeader b Host: prodweb9 9 TxHeader b Accept: */* 9 TxHeader b X-Varnish: 353939895 9 TxHeader b X-Forwarded-For: 172.16.200.145 9 RxProtocol b HTTP/1.1 9 RxStatus b 200 9 RxResponse b OK 9 RxHeader b Content-Type: text/html; charset=utf-8 9 RxHeader b Set-Cookie: _fod_session =BAh7BiIKZmxhc2hJQzonQWN0aW9uQ29udHJvbGxlcjo6Rmxhc2g6OkZsYXNo %0ASGFzaHsGOg5yZXR1cm5fdG8iBi8GOgpAdXNlZHsGOwZG --5181ccb5b65c2f75cafa4a6a89f0932ab07d1391; path=/ 9 RxHeader b X-Runtime: 2.06915 9 RxHeader b ETag: "c10816a9ec10679b3ac800a4b2e3e70b" 9 RxHeader b Served-by: prodapp5 9 RxHeader b Cache-Control: no-cache, public, max-age=300 9 RxHeader b Content-Length: 60261 9 RxHeader b Date: Wed, 03 Dec 2008 02:19:15 GMT 9 RxHeader b Server: LiteSpeed 9 RxHeader b Connection: Keep-Alive 9 RxHeader b Keep-Alive: timeout=5, max=100 8 ObjProtocol c HTTP/1.1 8 ObjStatus c 200 8 ObjResponse c OK 8 ObjHeader c Content-Type: text/html; charset=utf-8 8 ObjHeader c Set-Cookie: _fod_session =BAh7BiIKZmxhc2hJQzonQWN0aW9uQ29udHJvbGxlcjo6Rmxhc2g6OkZsYXNo %0ASGFzaHsGOg5yZXR1cm5fdG8iBi8GOgpAdXNlZHsGOwZG --5181ccb5b65c2f75cafa4a6a89f0932ab07d1391; path=/ 8 ObjHeader c X-Runtime: 2.06915 8 ObjHeader c ETag: "c10816a9ec10679b3ac800a4b2e3e70b" 8 ObjHeader c Served-by: prodapp5 8 ObjHeader c Cache-Control: no-cache, public, max-age=300 8 ObjHeader c Date: Wed, 03 Dec 2008 02:19:15 GMT 8 ObjHeader c Server: LiteSpeed 8 ObjHeader c Keep-Alive: timeout=5, max=100 9 BackendReuse b default 8 TTL c 353939895 RFC 300 1228270755 0 0 300 0 8 VCL_call c fetch 8 VCL_return c deliver 8 Length c 60261 8 VCL_call c deliver 8 VCL_return c deliver 8 TxProtocol c HTTP/1.1 8 TxStatus c 200 8 TxResponse c OK 8 TxHeader c Content-Type: text/html; charset=utf-8 8 TxHeader c X-Runtime: 2.06915 8 TxHeader c ETag: "c10816a9ec10679b3ac800a4b2e3e70b" 8 TxHeader c Cache-Control: no-cache, public, max-age=300 8 TxHeader c Server: LiteSpeed 8 TxHeader c Content-Length: 60261 8 TxHeader c Date: Wed, 03 Dec 2008 02:19:15 GMT 8 TxHeader c X-Varnish: 353939895 8 TxHeader c Age: 0 8 TxHeader c Via: 1.1 varnish 8 TxHeader c Served-by: prodweb9/prodapp5 8 TxHeader c Connection: close 8 ReqEnd c 353939895 1228270752.682591915 1228270755.543001890 0.000190020 2.860359192 0.000050783 0 StatAddr - 172.16.200.145 0 60 2 2 0 0 2 761 0 8 SessionClose c no request 8 StatSess c 172.16.200.145 56473 3 1 1 0 0 1 357 0 **** Requesting same url with compression Curl Command: $ curl http://prodweb9/ --head --compressed HTTP/1.1 200 OK Content-Type: text/html; charset=utf-8 X-Runtime: 2.06915 ETag: "c10816a9ec10679b3ac800a4b2e3e70b" Cache-Control: no-cache, public, max-age=300 Server: LiteSpeed Content-Length: 60261 Date: Wed, 03 Dec 2008 02:22:20 GMT X-Varnish: 353939896 353939895 Age: 185 Via: 1.1 varnish Served-by: prodweb9/prodapp5 Connection: close Varnishlog output: 8 SessionOpen c 172.16.200.145 56529 :80 8 ReqStart c 172.16.200.145 56529 353939896 8 RxRequest c HEAD 8 RxURL c / 8 RxProtocol c HTTP/1.1 8 RxHeader c User-Agent: curl/7.16.3 (powerpc-apple- darwin9.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3 8 RxHeader c Host: prodweb9 8 RxHeader c Accept: */* 8 RxHeader c Accept-Encoding: deflate, gzip 8 VCL_call c recv 8 VCL_return c lookup 8 VCL_call c hash 8 VCL_return c hash 8 Hit c 353939895 8 VCL_call c hit 8 VCL_return c deliver 8 Length c 60261 8 VCL_call c deliver 8 VCL_return c deliver 8 TxProtocol c HTTP/1.1 8 TxStatus c 200 8 TxResponse c OK 8 TxHeader c Content-Type: text/html; charset=utf-8 8 TxHeader c X-Runtime: 2.06915 8 TxHeader c ETag: "c10816a9ec10679b3ac800a4b2e3e70b" 8 TxHeader c Cache-Control: no-cache, public, max-age=300 8 TxHeader c Server: LiteSpeed 8 TxHeader c Content-Length: 60261 8 TxHeader c Date: Wed, 03 Dec 2008 02:22:20 GMT 8 TxHeader c X-Varnish: 353939896 353939895 8 TxHeader c Age: 185 8 TxHeader c Via: 1.1 varnish 8 TxHeader c Served-by: prodweb9/prodapp5 8 TxHeader c Connection: close 8 ReqEnd c 353939896 1228270940.695084095 1228270940.695233107 0.009155035 0.000080824 0.000068188 0 StatAddr - 172.16.200.145 0 245 3 3 0 0 2 1130 0 8 SessionClose c no request 8 StatSess c 172.16.200.145 56529 0 1 1 0 0 0 369 0 ---- Bounce Varnish again ---- Now request compressed page first: $ curl http://prodweb9/ --head --compressed HTTP/1.1 200 OK Content-Type: text/html; charset=utf-8 X-Runtime: 1.96633 ETag: "d0c769c5047af7cef664e2f941011667" Cache-Control: no-cache, public, max-age=300 Content-Encoding: gzip Vary: Accept-Encoding Server: LiteSpeed Content-Length: 11576 Date: Wed, 03 Dec 2008 02:39:14 GMT X-Varnish: 1723012051 Age: 0 Via: 1.1 varnish Served-by: prodweb9/prodapp12 Connection: close Varnishlog output: 8 SessionOpen c 172.16.200.145 56707 :80 8 ReqStart c 172.16.200.145 56707 1723012051 8 RxRequest c HEAD 8 RxURL c / 8 RxProtocol c HTTP/1.1 8 RxHeader c User-Agent: curl/7.16.3 (powerpc-apple- darwin9.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3 8 RxHeader c Host: prodweb9 8 RxHeader c Accept: */* 8 RxHeader c Accept-Encoding: deflate, gzip 8 VCL_call c recv 8 VCL_return c lookup 8 VCL_call c hash 8 VCL_return c hash 8 VCL_call c miss 8 VCL_return c fetch 9 BackendOpen b default 172.16.160.85 22005 172.16.160.174 80 8 Backend c 9 default default 9 TxRequest b GET 9 TxURL b / 9 TxProtocol b HTTP/1.1 9 TxHeader b User-Agent: curl/7.16.3 (powerpc-apple- darwin9.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3 9 TxHeader b Host: prodweb9 9 TxHeader b Accept: */* 9 TxHeader b Accept-Encoding: gzip 9 TxHeader b X-Varnish: 1723012051 9 TxHeader b X-Forwarded-For: 172.16.200.145 9 RxProtocol b HTTP/1.1 9 RxStatus b 200 9 RxResponse b OK 9 RxHeader b Content-Type: text/html; charset=utf-8 9 RxHeader b Set-Cookie: _fod_session =BAh7BiIKZmxhc2hJQzonQWN0aW9uQ29udHJvbGxlcjo6Rmxhc2g6OkZsYXNo %0ASGFzaHsGOg5yZXR1cm5fdG8iBi8GOgpAdXNlZHsGOwZG --5181ccb5b65c2f75cafa4a6a89f0932ab07d1391; path=/ 9 RxHeader b X-Runtime: 1.96633 9 RxHeader b ETag: "d0c769c5047af7cef664e2f941011667" 9 RxHeader b Served-by: prodapp12 9 RxHeader b Cache-Control: no-cache, public, max-age=300 9 RxHeader b Content-Encoding: gzip 9 RxHeader b Vary: Accept-Encoding 9 RxHeader b Transfer-Encoding: chunked 9 RxHeader b Date: Wed, 03 Dec 2008 02:39:14 GMT 9 RxHeader b Server: LiteSpeed 9 RxHeader b Connection: Keep-Alive 9 RxHeader b Keep-Alive: timeout=5, max=100 8 ObjProtocol c HTTP/1.1 8 ObjStatus c 200 8 ObjResponse c OK 8 ObjHeader c Content-Type: text/html; charset=utf-8 8 ObjHeader c Set-Cookie: _fod_session =BAh7BiIKZmxhc2hJQzonQWN0aW9uQ29udHJvbGxlcjo6Rmxhc2g6OkZsYXNo %0ASGFzaHsGOg5yZXR1cm5fdG8iBi8GOgpAdXNlZHsGOwZG --5181ccb5b65c2f75cafa4a6a89f0932ab07d1391; path=/ 8 ObjHeader c X-Runtime: 1.96633 8 ObjHeader c ETag: "d0c769c5047af7cef664e2f941011667" 8 ObjHeader c Served-by: prodapp12 8 ObjHeader c Cache-Control: no-cache, public, max-age=300 8 ObjHeader c Content-Encoding: gzip 8 ObjHeader c Vary: Accept-Encoding 8 ObjHeader c Date: Wed, 03 Dec 2008 02:39:14 GMT 8 ObjHeader c Server: LiteSpeed 8 ObjHeader c Keep-Alive: timeout=5, max=100 9 BackendReuse b default 8 TTL c 1723012051 RFC 300 1228271954 0 0 300 0 8 VCL_call c fetch 8 VCL_return c deliver 8 Length c 11576 8 VCL_call c deliver 8 VCL_return c deliver 8 TxProtocol c HTTP/1.1 8 TxStatus c 200 8 TxResponse c OK 8 TxHeader c Content-Type: text/html; charset=utf-8 8 TxHeader c X-Runtime: 1.96633 8 TxHeader c ETag: "d0c769c5047af7cef664e2f941011667" 8 TxHeader c Cache-Control: no-cache, public, max-age=300 8 TxHeader c Content-Encoding: gzip 8 TxHeader c Vary: Accept-Encoding 8 TxHeader c Server: LiteSpeed 8 TxHeader c Content-Length: 11576 8 TxHeader c Date: Wed, 03 Dec 2008 02:39:14 GMT 8 TxHeader c X-Varnish: 1723012051 8 TxHeader c Age: 0 8 TxHeader c Via: 1.1 varnish 8 TxHeader c Served-by: prodweb9/prodapp12 8 TxHeader c Connection: close 8 ReqEnd c 1723012051 1228271952.806010008 1228271954.783368111 0.000576019 1.977247953 0.000110149 0 StatAddr - 172.16.200.145 0 2 1 1 0 0 1 406 0 8 SessionClose c no request 8 StatSess c 172.16.200.145 56707 2 1 1 0 0 1 406 0 Now request uncompressed page... $ curl http://prodweb9/ --head HTTP/1.1 200 OK Content-Type: text/html; charset=utf-8 X-Runtime: 0.54823 ETag: "309fb08bc00884c3b2d52449906da9b4" Cache-Control: no-cache, public, max-age=300 Server: LiteSpeed Content-Length: 60171 Date: Wed, 03 Dec 2008 02:39:36 GMT X-Varnish: 1723012052 Age: 0 Via: 1.1 varnish Served-by: prodweb9/prodapp5 Connection: close Varnish log output: 8 SessionOpen c 172.16.200.145 56710 :80 8 ReqStart c 172.16.200.145 56710 1723012052 8 RxRequest c HEAD 8 RxURL c / 8 RxProtocol c HTTP/1.1 8 RxHeader c User-Agent: curl/7.16.3 (powerpc-apple- darwin9.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3 8 RxHeader c Host: prodweb9 8 RxHeader c Accept: */* 8 VCL_call c recv 8 VCL_return c lookup 8 VCL_call c hash 8 VCL_return c hash 8 VCL_call c miss 8 VCL_return c fetch 9 BackendClose b default 9 BackendOpen b default 172.16.160.85 22006 172.16.160.174 80 8 Backend c 9 default default 9 TxRequest b GET 9 TxURL b / 9 TxProtocol b HTTP/1.1 9 TxHeader b User-Agent: curl/7.16.3 (powerpc-apple- darwin9.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3 9 TxHeader b Host: prodweb9 9 TxHeader b Accept: */* 9 TxHeader b X-Varnish: 1723012052 9 TxHeader b X-Forwarded-For: 172.16.200.145 9 RxProtocol b HTTP/1.1 9 RxStatus b 200 9 RxResponse b OK 9 RxHeader b Content-Type: text/html; charset=utf-8 9 RxHeader b Set-Cookie: _fod_session =BAh7BiIKZmxhc2hJQzonQWN0aW9uQ29udHJvbGxlcjo6Rmxhc2g6OkZsYXNo %0ASGFzaHsGOg5yZXR1cm5fdG8iBi8GOgpAdXNlZHsGOwZG --5181ccb5b65c2f75cafa4a6a89f0932ab07d1391; path=/ 9 RxHeader b X-Runtime: 0.54823 9 RxHeader b ETag: "309fb08bc00884c3b2d52449906da9b4" 9 RxHeader b Served-by: prodapp5 9 RxHeader b Cache-Control: no-cache, public, max-age=300 9 RxHeader b Content-Length: 60171 9 RxHeader b Date: Wed, 03 Dec 2008 02:39:36 GMT 9 RxHeader b Server: LiteSpeed 9 RxHeader b Connection: Keep-Alive 9 RxHeader b Keep-Alive: timeout=5, max=100 8 ObjProtocol c HTTP/1.1 8 ObjStatus c 200 8 ObjResponse c OK 8 ObjHeader c Content-Type: text/html; charset=utf-8 8 ObjHeader c Set-Cookie: _fod_session =BAh7BiIKZmxhc2hJQzonQWN0aW9uQ29udHJvbGxlcjo6Rmxhc2g6OkZsYXNo %0ASGFzaHsGOg5yZXR1cm5fdG8iBi8GOgpAdXNlZHsGOwZG --5181ccb5b65c2f75cafa4a6a89f0932ab07d1391; path=/ 8 ObjHeader c X-Runtime: 0.54823 8 ObjHeader c ETag: "309fb08bc00884c3b2d52449906da9b4" 8 ObjHeader c Served-by: prodapp5 8 ObjHeader c Cache-Control: no-cache, public, max-age=300 8 ObjHeader c Date: Wed, 03 Dec 2008 02:39:36 GMT 8 ObjHeader c Server: LiteSpeed 8 ObjHeader c Keep-Alive: timeout=5, max=100 9 BackendReuse b default 8 TTL c 1723012052 RFC 300 1228271976 0 0 300 0 8 VCL_call c fetch 8 VCL_return c deliver 8 Length c 60171 8 VCL_call c deliver 8 VCL_return c deliver 8 TxProtocol c HTTP/1.1 8 TxStatus c 200 8 TxResponse c OK 8 TxHeader c Content-Type: text/html; charset=utf-8 8 TxHeader c X-Runtime: 0.54823 8 TxHeader c ETag: "309fb08bc00884c3b2d52449906da9b4" 8 TxHeader c Cache-Control: no-cache, public, max-age=300 8 TxHeader c Server: LiteSpeed 8 TxHeader c Content-Length: 60171 8 TxHeader c Date: Wed, 03 Dec 2008 02:39:36 GMT 8 TxHeader c X-Varnish: 1723012052 8 TxHeader c Age: 0 8 TxHeader c Via: 1.1 varnish 8 TxHeader c Served-by: prodweb9/prodapp5 8 TxHeader c Connection: close 8 ReqEnd c 1723012052 1228271975.969137907 1228271976.524655104 0.000959873 0.555464983 0.000052214 0 StatAddr - 172.16.200.145 0 24 2 2 0 0 2 764 0 8 SessionClose c no request 8 StatSess c 172.16.200.145 56710 1 1 1 0 0 1 358 0 What could be causing this? The only way to fix appears to be to add the lines below: sub vcl_hash { if (req.http.Accept-Encoding ~ "gzip" || req.http.Accept-Encoding ~ "deflate") { set req.hash += req.http.Accept-Encoding; } -------------- next part -------------- An HTML attachment was scrubbed... URL: From plfgoa at gmail.com Wed Dec 3 04:12:43 2008 From: plfgoa at gmail.com (Paras Fadte) Date: Wed, 3 Dec 2008 09:42:43 +0530 Subject: sendfile() and Varnish2.0.2 In-Reply-To: <12832.1228241360@critter.freebsd.dk> References: <20081203024812.6f00266d.nork@ninth-nine.com> <12832.1228241360@critter.freebsd.dk> Message-ID: <75cf5800812022012i7fbd1174l53e5435063fb514c@mail.gmail.com> Thanks for the response , I would like to know how can one minimize the "overflowed work requests" ? On Tue, Dec 2, 2008 at 11:39 PM, Poul-Henning Kamp wrote: > In message <20081203024812.6f00266d.nork at ninth-nine.com>, Norikatsu Shigemura w > rites: > >> What is the difference between FreeBSD 8 and 7? > > Presently 8 is "-current" which means it is the development branch of > FreeBSD, so it is pretty much a moving target. > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc > From perbu at linpro.no Wed Dec 3 06:36:46 2008 From: perbu at linpro.no (Per Buer) Date: Wed, 03 Dec 2008 07:36:46 +0100 Subject: Varnish Serves only uncompressed objects if they are requested first In-Reply-To: <4DB04145-6ECB-4887-AEAC-BD7AF508E28F@funnyordie.com> References: <4DB04145-6ECB-4887-AEAC-BD7AF508E28F@funnyordie.com> Message-ID: <493628FE.9000207@linpro.no> Jeff Anderson skrev: > It looks like if the first requested page is for an uncompressed page > varnish will only deliver the uncompressed page from cache even if a > compressed page is requested. As long as you don't Vary: on the Accept-Encoding I guess that is expected. Varnish doesn't not understand the Accept-Encoding header. > (..) > What could be causing this? The only way to fix appears to be to add > the lines below: > > sub vcl_hash { > if (req.http.Accept-Encoding ~ "gzip" || req.http.Accept-Encoding ~ > "deflate") { > set req.hash += req.http.Accept-Encoding; > } Either use the fix you suggested or add a Vary: Accept-Encoding on the backend. -- Per Buer - Leder Infrastruktur og Drift - Redpill Linpro Telefon: 21 54 41 21 - Mobil: 958 39 117 http://linpro.no/ | http://redpill.se/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 252 bytes Desc: OpenPGP digital signature URL: From plfgoa at gmail.com Wed Dec 3 06:48:41 2008 From: plfgoa at gmail.com (Paras Fadte) Date: Wed, 3 Dec 2008 12:18:41 +0530 Subject: Varnish Performance Issue Message-ID: <75cf5800812022248n6704cef5k44ea4ba44876a31c@mail.gmail.com> Hi, I am faced with following performace issue with respect to Varnish. Load on Varnish increases suddenly at times. When this happens , "varnishhist" shows a high number of misses and the cache hit ratio decreases and the response time increases. Also there is a gradual increase in the number of "overflowed work requests" over a period of time. I have attached mrtg of CPU utilisation for perusal. It shows a very spiky graph. What could be the issue ? What parameters do I need to adjust to overcome this load issue ? The settings used are : lru_interval = 30 thread_pool_max = 750 (initially I had set this to 300 but the load issue was the same) overflow_max = 100000 sess_timeout = 2 listen_depth = 4096 Cache size is 3GB ( ramdisk) Varnish Version:2.0.2 (built from source) OS:SUSE LINUX 10.1 (X86-64) RAM:20GB CPU: Intel(R) Xeon(R) CPU E5430 @ 2.66GHz Thanks in Advance. -Paras -------------- next part -------------- A non-text attachment was scrubbed... Name: CPU_usage.jpg Type: image/jpeg Size: 26400 bytes Desc: not available URL: From phk at phk.freebsd.dk Wed Dec 3 08:04:40 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 03 Dec 2008 08:04:40 +0000 Subject: sendfile() and Varnish2.0.2 In-Reply-To: Your message of "Wed, 03 Dec 2008 09:42:43 +0530." <75cf5800812022012i7fbd1174l53e5435063fb514c@mail.gmail.com> Message-ID: <79953.1228291480@critter.freebsd.dk> In message <75cf5800812022012i7fbd1174l53e5435063fb514c at mail.gmail.com>, "Paras Fadte" writes: >Thanks for the response , I would like to know how can one minimize >the "overflowed work requests" ? overflowed work requests are not bad per se, they are the signal we need to increase the worker thread pool. it's the dropped requests you want to keep at zero. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From plfgoa at gmail.com Wed Dec 3 09:53:17 2008 From: plfgoa at gmail.com (Paras Fadte) Date: Wed, 3 Dec 2008 15:23:17 +0530 Subject: sendfile() and Varnish2.0.2 In-Reply-To: <79953.1228291480@critter.freebsd.dk> References: <75cf5800812022012i7fbd1174l53e5435063fb514c@mail.gmail.com> <79953.1228291480@critter.freebsd.dk> Message-ID: <75cf5800812030153h762711f3q1ff2b31819970d70@mail.gmail.com> Hi Poul, Thanks for the response. So "thread_pools" parameter should be increased or the "thread_pool_max" or both should be increased ? Currently I have "thread_pools" as 4 and "thread_pool_max" as 750 Thanks. -Paras On Wed, Dec 3, 2008 at 1:34 PM, Poul-Henning Kamp wrote: > In message <75cf5800812022012i7fbd1174l53e5435063fb514c at mail.gmail.com>, "Paras > Fadte" writes: > >>Thanks for the response , I would like to know how can one minimize >>the "overflowed work requests" ? > > overflowed work requests are not bad per se, they are the signal > we need to increase the worker thread pool. > > it's the dropped requests you want to keep at zero. > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > From phk at phk.freebsd.dk Wed Dec 3 09:58:07 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 03 Dec 2008 09:58:07 +0000 Subject: sendfile() and Varnish2.0.2 In-Reply-To: Your message of "Wed, 03 Dec 2008 15:23:17 +0530." <75cf5800812030153h762711f3q1ff2b31819970d70@mail.gmail.com> Message-ID: <15664.1228298287@critter.freebsd.dk> In message <75cf5800812030153h762711f3q1ff2b31819970d70 at mail.gmail.com>, "Paras Fadte" writes: >Hi Poul, > >Thanks for the response. So "thread_pools" parameter should be >increased or the "thread_pool_max" or both should be increased ? > >Currently I have "thread_pools" as 4 and "thread_pool_max" as 750 You should only increase thread_pools if you have performance problems due to lock-contention. You should only increas thread_pool_max if you regularly have bursts of dropped sessions. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From plfgoa at gmail.com Wed Dec 3 10:05:20 2008 From: plfgoa at gmail.com (Paras Fadte) Date: Wed, 3 Dec 2008 15:35:20 +0530 Subject: sendfile() and Varnish2.0.2 In-Reply-To: <15664.1228298287@critter.freebsd.dk> References: <75cf5800812030153h762711f3q1ff2b31819970d70@mail.gmail.com> <15664.1228298287@critter.freebsd.dk> Message-ID: <75cf5800812030205i3cc29eedjda904410f0b7891a@mail.gmail.com> Thanks Poul. On Wed, Dec 3, 2008 at 3:28 PM, Poul-Henning Kamp wrote: > In message <75cf5800812030153h762711f3q1ff2b31819970d70 at mail.gmail.com>, "Paras > Fadte" writes: >>Hi Poul, >> >>Thanks for the response. So "thread_pools" parameter should be >>increased or the "thread_pool_max" or both should be increased ? >> >>Currently I have "thread_pools" as 4 and "thread_pool_max" as 750 > > You should only increase thread_pools if you have performance > problems due to lock-contention. > > You should only increas thread_pool_max if you regularly have > bursts of dropped sessions. > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > From plfgoa at gmail.com Wed Dec 3 10:10:07 2008 From: plfgoa at gmail.com (Paras Fadte) Date: Wed, 3 Dec 2008 15:40:07 +0530 Subject: sendfile() and Varnish2.0.2 In-Reply-To: <15664.1228298287@critter.freebsd.dk> References: <75cf5800812030153h762711f3q1ff2b31819970d70@mail.gmail.com> <15664.1228298287@critter.freebsd.dk> Message-ID: <75cf5800812030210x2e1e9743h287056990571b6c3@mail.gmail.com> Hi Poul, Can you please shed some light on the mail that I had sent regarding varnish performance ? In the same mail I have attached a mrtg graph of CPU utilisation which is very spiky. Thank you. -Paras On Wed, Dec 3, 2008 at 3:28 PM, Poul-Henning Kamp wrote: > In message <75cf5800812030153h762711f3q1ff2b31819970d70 at mail.gmail.com>, "Paras > Fadte" writes: >>Hi Poul, >> >>Thanks for the response. So "thread_pools" parameter should be >>increased or the "thread_pool_max" or both should be increased ? >> >>Currently I have "thread_pools" as 4 and "thread_pool_max" as 750 > > You should only increase thread_pools if you have performance > problems due to lock-contention. > > You should only increas thread_pool_max if you regularly have > bursts of dropped sessions. > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > From phk at phk.freebsd.dk Wed Dec 3 10:13:07 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 03 Dec 2008 10:13:07 +0000 Subject: Varnish Performance Issue In-Reply-To: Your message of "Wed, 03 Dec 2008 12:18:41 +0530." <75cf5800812022248n6704cef5k44ea4ba44876a31c@mail.gmail.com> Message-ID: <15757.1228299187@critter.freebsd.dk> In message <75cf5800812022248n6704cef5k44ea4ba44876a31c at mail.gmail.com>, "Paras Fadte" writes: >I am faced with following performace issue with respect to Varnish. > >Load on Varnish increases suddenly at times. When this happens , >"varnishhist" shows a high number of misses and the cache hit ratio >decreases and the response time increases. I think this sounds pretty normal, try examining the traffic pattern and see if coincides with the arrival of robots/spiders at your site. If you want to, you can return a temporary error in vcl_miss if the user-agent indicates a robot or spider, that way they don't pull your entire site content into the cache. Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From plfgoa at gmail.com Wed Dec 3 10:30:29 2008 From: plfgoa at gmail.com (Paras Fadte) Date: Wed, 3 Dec 2008 16:00:29 +0530 Subject: Varnish Performance Issue In-Reply-To: <15757.1228299187@critter.freebsd.dk> References: <75cf5800812022248n6704cef5k44ea4ba44876a31c@mail.gmail.com> <15757.1228299187@critter.freebsd.dk> Message-ID: <75cf5800812030230v62dccd54m6e1538ad28795916@mail.gmail.com> Hi Poul, The traffic is steady and the server is not directly externally available , It acts as a caching server behind a load balancer for the origin servers. I have attached bandwidth graph for our perusal. What confuses is the sudden Spike in CPU usage . Thank you. -Paras On Wed, Dec 3, 2008 at 3:43 PM, Poul-Henning Kamp wrote: > In message <75cf5800812022248n6704cef5k44ea4ba44876a31c at mail.gmail.com>, "Paras > Fadte" writes: > >>I am faced with following performace issue with respect to Varnish. >> >>Load on Varnish increases suddenly at times. When this happens , >>"varnishhist" shows a high number of misses and the cache hit ratio >>decreases and the response time increases. > > I think this sounds pretty normal, try examining the traffic pattern > and see if coincides with the arrival of robots/spiders at your > site. > > If you want to, you can return a temporary error in vcl_miss if the > user-agent indicates a robot or spider, that way they don't pull > your entire site content into the cache. > > Poul-Henning > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > -------------- next part -------------- A non-text attachment was scrubbed... Name: traffic.JPG Type: image/jpeg Size: 27623 bytes Desc: not available URL: From phk at phk.freebsd.dk Wed Dec 3 10:37:54 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 03 Dec 2008 10:37:54 +0000 Subject: Varnish Performance Issue In-Reply-To: Your message of "Wed, 03 Dec 2008 16:00:29 +0530." <75cf5800812030230v62dccd54m6e1538ad28795916@mail.gmail.com> Message-ID: <18661.1228300674@critter.freebsd.dk> In message <75cf5800812030230v62dccd54m6e1538ad28795916 at mail.gmail.com>, "Paras Fadte" writes: >The traffic is steady and the server is not directly externally >available , It acts as a caching server behind a load balancer for the >origin servers. I have attached bandwidth graph for our perusal. What >confuses is the sudden Spike in CPU usage . As I said, I don't see anything abnormal, you said yourself that the spikes match times with lower hit rates. Picking up things from the backend takes significantly more effort than just serving them out of the cache. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From plfgoa at gmail.com Wed Dec 3 10:47:54 2008 From: plfgoa at gmail.com (Paras Fadte) Date: Wed, 3 Dec 2008 16:17:54 +0530 Subject: Varnish Performance Issue In-Reply-To: <18661.1228300674@critter.freebsd.dk> References: <75cf5800812030230v62dccd54m6e1538ad28795916@mail.gmail.com> <18661.1228300674@critter.freebsd.dk> Message-ID: <75cf5800812030247l1f09f967v839e8dd58d3b88d4@mail.gmail.com> Hi Poul, What I wanted to know was whether the sudden spikes would be caused due to cache management by varnish which it could be doing periodically? Thank you. -Paras On Wed, Dec 3, 2008 at 4:07 PM, Poul-Henning Kamp wrote: > In message <75cf5800812030230v62dccd54m6e1538ad28795916 at mail.gmail.com>, "Paras > Fadte" writes: > >>The traffic is steady and the server is not directly externally >>available , It acts as a caching server behind a load balancer for the >>origin servers. I have attached bandwidth graph for our perusal. What >>confuses is the sudden Spike in CPU usage . > > As I said, I don't see anything abnormal, you said yourself that > the spikes match times with lower hit rates. Picking up things > from the backend takes significantly more effort than just > serving them out of the cache. > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > From phk at phk.freebsd.dk Wed Dec 3 10:50:50 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 03 Dec 2008 10:50:50 +0000 Subject: Varnish Performance Issue In-Reply-To: Your message of "Wed, 03 Dec 2008 16:17:54 +0530." <75cf5800812030247l1f09f967v839e8dd58d3b88d4@mail.gmail.com> Message-ID: <32554.1228301450@critter.freebsd.dk> In message <75cf5800812030247l1f09f967v839e8dd58d3b88d4 at mail.gmail.com>, "Paras Fadte" writes: >Hi Poul, > >What I wanted to know was whether the sudden spikes would be caused >due to cache management by varnish which it could be doing >periodically? Examine your varnishlog output -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From plfgoa at gmail.com Wed Dec 3 11:10:59 2008 From: plfgoa at gmail.com (Paras Fadte) Date: Wed, 3 Dec 2008 16:40:59 +0530 Subject: Varnish Performance Issue In-Reply-To: <32554.1228301450@critter.freebsd.dk> References: <75cf5800812030247l1f09f967v839e8dd58d3b88d4@mail.gmail.com> <32554.1228301450@critter.freebsd.dk> Message-ID: <75cf5800812030310g5d4266f8m661b668308027d04@mail.gmail.com> Hi Poul, What I also saw during high CPU usage was substantial increase in "small free smf" shown by "varnishstat" Thank you. -Paras On Wed, Dec 3, 2008 at 4:20 PM, Poul-Henning Kamp wrote: > In message <75cf5800812030247l1f09f967v839e8dd58d3b88d4 at mail.gmail.com>, "Paras > Fadte" writes: >>Hi Poul, >> >>What I wanted to know was whether the sudden spikes would be caused >>due to cache management by varnish which it could be doing >>periodically? > > Examine your varnishlog output > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > From johan at allerinternett.no Wed Dec 3 13:08:09 2008 From: johan at allerinternett.no (Grasmo, Johan) Date: Wed, 3 Dec 2008 14:08:09 +0100 Subject: Strange varnish behavior - not answering requests Message-ID: Hi, We're tired of squid and it's quirks and want to advance into the information age with varnish. I've been testing varnish the last couple of days and it has been behaving perfectly until last night. When stresstesting the varnishserver it suddenly stopped handling requests. Varnish returns to normal operation after a restart. Symptom: Accessing a page on the varnishsserver results in a timeout in the browser I am able to telnet to the varnishserver on port 80 but a GET doesn't result in anything ("hangs"). I am able to telnet to port 6082 and run commands: [snip] status 200 22 Child in state running [/snip] Varnishlog returns: [snip] 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1228307253 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1228307256 1.0 0 CLI - Rd ping [/snip] Varnishncsa doesn't return anything. I've verified with tcpdump that varnish ack's requests. So "everything seems normal" even though clients get timeout. FF returned "Connection Interrupted The document contains no data". If you need any logs/configs please let me know :) I haven't exhausted the search function for the mailinglist so sorry if this topic has been an issue before. I'm running varnish 2.0 on an Ubuntu 8.04 distro. Cheers and thanks for replies, Johan Grasmo From espen at linpro.no Wed Dec 3 14:37:06 2008 From: espen at linpro.no (Espen Braastad) Date: Wed, 03 Dec 2008 15:37:06 +0100 Subject: How override/remove cache headers from backend? Message-ID: <49369992.7070709@linpro.no> Hi, I'm having a bit of a problem with a not-so-trustworthy backend which is sending the following headers: [...] Cache-Control: no-cache, no-store, max-age=0, must-revalidate Pragma: no-cache Vary: Accept-Encoding,Cookie [...] The goal is to get varnish to ignore/remove the headers and still put the element into cache. I've tried the following minimal configuration unsuccessfully: [...] sub vcl_recv { remove req.http.Cookie; } sub vcl_fetch { remove obj.http.Cache-control; remove obj.http.Expires; remove obj.http.Pragma; remove obj.http.Vary; deliver; } [...] This configuration strips the headers *after* varnish determines if the element should be put in cache or not; something like: 1. Fetching element from backend. 2. Determine if element should be put in cache => in this case don't. 3. Remove the headers. 4. Deliver element to client (without the cache headers). Is there a way to manipulate reply headers from the backend before varnish decides whether or not to put the element into cache? The complete varnishlog from one request (with the previous mentioned configuration): 9 SessionOpen c 10.0.0.2 59453 0.0.0.0:80 9 ReqStart c 10.0.0.2 59453 683810183 9 RxRequest c GET 9 RxURL c /page/Main_Page 9 RxProtocol c HTTP/1.1 9 RxHeader c Connection: close 9 RxHeader c Host: www.example.com 9 RxHeader c User-Agent: lwp-request/2.08 9 VCL_call c recv 9 VCL_return c lookup 9 VCL_call c hash 9 VCL_return c hash 9 VCL_call c miss 9 VCL_return c fetch 10 BackendClose - default 10 BackendOpen b default 127.0.0.1 47939 127.0.0.1 8080 9 Backend c 10 default default 10 TxRequest b GET 10 TxURL b /page/Main_Page 10 TxProtocol b HTTP/1.1 10 TxHeader b Host: www.example.com 10 TxHeader b User-Agent: lwp-request/2.08 10 TxHeader b X-Varnish: 683810183 10 TxHeader b X-Forwarded-For: 10.0.0.2 10 RxProtocol b HTTP/1.1 10 RxStatus b 200 10 RxResponse b OK 10 RxHeader b Date: Wed, 03 Dec 2008 14:28:14 GMT 10 RxHeader b Server: Apache 10 RxHeader b Content-language: en 10 RxHeader b Vary: Accept-Encoding,Cookie 10 RxHeader b Expires: Thu, 01 Jan 1970 00:00:00 GMT 10 RxHeader b Cache-Control: no-cache, no-store, max-age=0, must-revalidate 10 RxHeader b Pragma: no-cache 10 RxHeader b Transfer-Encoding: chunked 10 RxHeader b Content-Type: text/html; charset=utf-8 9 ObjProtocol c HTTP/1.1 9 ObjStatus c 200 9 ObjResponse c OK 9 ObjHeader c Date: Wed, 03 Dec 2008 14:28:14 GMT 9 ObjHeader c Server: Apache 9 ObjHeader c Content-language: en 9 ObjHeader c Vary: Accept-Encoding,Cookie 9 ObjHeader c Expires: Thu, 01 Jan 1970 00:00:00 GMT 9 ObjHeader c Cache-Control: no-cache, no-store, max-age=0, must-revalidate 9 ObjHeader c Pragma: no-cache 9 ObjHeader c Content-Type: text/html; charset=utf-8 10 BackendReuse b default 9 TTL c 683810183 RFC 0 1228314496 0 0 0 0 9 VCL_call c fetch 9 VCL_return c deliver 9 Length c 18524 9 VCL_call c deliver 9 VCL_return c deliver 9 TxProtocol c HTTP/1.1 9 TxStatus c 200 9 TxResponse c OK 9 TxHeader c Server: Apache 9 TxHeader c Content-language: en 9 TxHeader c Content-Type: text/html; charset=utf-8 9 TxHeader c Content-Length: 18524 9 TxHeader c Date: Wed, 03 Dec 2008 14:28:16 GMT 9 TxHeader c X-Varnish: 683810183 9 TxHeader c Age: 0 9 TxHeader c Via: 1.1 varnish 9 TxHeader c Connection: close 9 ReqEnd c 683810183 1228314494.178344011 1228314496.532316923 0.001534939 2.351898909 0.002074003 9 SessionClose c Connection: close 9 StatSess c 10.0.0.2 59453 2 1 1 0 0 1 224 18524 Thanks, -- Espen Braastad Redpill Linpro AB - Changing the game Kontor: +47 21 54 41 37 From espen at linpro.no Wed Dec 3 15:30:27 2008 From: espen at linpro.no (Espen Braastad) Date: Wed, 03 Dec 2008 16:30:27 +0100 Subject: How override/remove cache headers from backend? In-Reply-To: <49369992.7070709@linpro.no> References: <49369992.7070709@linpro.no> Message-ID: <4936A613.2040801@linpro.no> Espen Braastad wrote: > Hi, > > I'm having a bit of a problem with a not-so-trustworthy backend which is > sending the following headers: > > [...] > Cache-Control: no-cache, no-store, max-age=0, must-revalidate > Pragma: no-cache > Vary: Accept-Encoding,Cookie > [...] > > The goal is to get varnish to ignore/remove the headers and still put > the element into cache. > The correct method (thanks to Lars Erler) is: sub vcl_fetch { if(obj.ttl < 120s){ set obj.ttl = 120s; } } -- Espen Braastad Redpill Linpro AB - Changing the game Kontor: +47 21 54 41 37 From eden at mojiti.com Wed Dec 3 17:48:59 2008 From: eden at mojiti.com (Eden Li) Date: Wed, 3 Dec 2008 09:48:59 -0800 Subject: ESI objects hang for HTTP/1.0 clients Message-ID: <180e6a10812030948j4763275elb682e639ed992ba3@mail.gmail.com> I'm testing out Varnish ESI, and I notice that HTTP/1.0 requests to ESI-enabled cache objects hang for about 5 seconds before returning while HTTP/1.1 requests return immediately. $ time echo -e "GET /esi HTTP/1.0\r\n\r\n" | nc host 80 > /dev/null real 0m5.005s user 0m0.000s sys 0m0.002s $ time echo -e "GET /esi HTTP/1.1\r\n\r\n" | nc host 80 > /dev/null real 0m0.003s user 0m0.000s sys 0m0.001s It appears the entire response is generated immediately but Varnish keeps the connection open for about 5 seconds afterwards. Is there something wrong with my test setup? I'm running varnish 2.0.2 in front of a very small cherrypy script, both the script and VCL are at http://pastie.org/329955 From phk at phk.freebsd.dk Wed Dec 3 17:54:24 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 03 Dec 2008 17:54:24 +0000 Subject: ESI objects hang for HTTP/1.0 clients In-Reply-To: Your message of "Wed, 03 Dec 2008 09:48:59 PST." <180e6a10812030948j4763275elb682e639ed992ba3@mail.gmail.com> Message-ID: <44421.1228326864@critter.freebsd.dk> In message <180e6a10812030948j4763275elb682e639ed992ba3 at mail.gmail.com>, "Eden Li" writes: >I'm testing out Varnish ESI, [...] Which Varnish version ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phil at reflected.net Wed Dec 3 18:11:53 2008 From: phil at reflected.net (Phil Doroff) Date: Wed, 03 Dec 2008 12:11:53 -0600 Subject: Strange varnish behavior - not answering requests In-Reply-To: References: Message-ID: <4936CBE9.3030807@reflected.net> Please excuse the top posting. We also are seeing either this exact issue, or one very like it. Unfortunately, we only see this problem in production and are unable to reproduce it in a test environment like you are. I've been meaning to post, but have yet to get more detailed varnishlog information like you have been able to. Due to these problems, we are about to migrate off Varnish very reluctantly, if we cannot resolve them soon. Varnish works great, until 2, 3, 6, 12 - seems random, hours of production use for our application. When the problem starts, all requests appear to "hang". You can either restart varnish manually at that point, to get back to normal operation - or after some time (roughly 5-15 minutes, it varies) varnish will actually restart the child thread and the services come back to normal operation automatically. We see this behavior both in 2.0.1 and 2.0.2 Sorry I don't have more information to add, but I figured I would drop a "me too" reply since we understand how frustrating this problem can be. I am also very willing to hire a varnish developer to diagnose and fix this problem, giving them full access to the machines in question. I was actually going to make a post mentioning this, but was going to wait until I could collect proper troubleshooting information. Other than this (admittedly show-stopper) issue, varnish is absolutely wonderful. Total time to learn about it, download, compile, and create our custom VCL was maybe 30 minutes. Regards, -Phil Grasmo, Johan wrote: > Hi, > > We're tired of squid and it's quirks and want to advance into the information age with varnish. > > I've been testing varnish the last couple of days and it has been behaving perfectly until last night. When stresstesting the varnishserver it suddenly stopped handling requests. Varnish returns to normal operation after a restart. > > Symptom: > Accessing a page on the varnishsserver results in a timeout in the browser > > I am able to telnet to the varnishserver on port 80 but a GET doesn't result in anything ("hangs"). > I am able to telnet to port 6082 and run commands: > [snip] > status > 200 22 > Child in state running > [/snip] > > Varnishlog returns: > [snip] > 0 CLI - Rd ping > 0 CLI - Wr 0 200 PONG 1228307253 1.0 > 0 CLI - Rd ping > 0 CLI - Wr 0 200 PONG 1228307256 1.0 > 0 CLI - Rd ping > [/snip] > > Varnishncsa doesn't return anything. > > I've verified with tcpdump that varnish ack's requests. > > So "everything seems normal" even though clients get timeout. FF returned "Connection Interrupted The document contains no data". If you need any logs/configs please let me know :) > > I haven't exhausted the search function for the mailinglist so sorry if this topic has been an issue before. > > I'm running varnish 2.0 on an Ubuntu 8.04 distro. > > > Cheers and thanks for replies, > > Johan Grasmo > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc > From eden at mojiti.com Wed Dec 3 18:27:37 2008 From: eden at mojiti.com (Eden Li) Date: Wed, 3 Dec 2008 10:27:37 -0800 Subject: ESI objects hang for HTTP/1.0 clients In-Reply-To: <44421.1228326864@critter.freebsd.dk> References: <180e6a10812030948j4763275elb682e639ed992ba3@mail.gmail.com> <44421.1228326864@critter.freebsd.dk> Message-ID: <180e6a10812031027x1b366462tdf3b2b30fc7e1547@mail.gmail.com> Varnish 2.0.2 using the options: -a 0.0.0.0:80 -T 127.0.0.1:9999 -f /usr/local/etc/vcl.conf -s file,/var/varnish/cache The VCL is listed in http://pastie.org/329955 On Wed, Dec 3, 2008 at 9:54 AM, Poul-Henning Kamp wrote: > In message <180e6a10812030948j4763275elb682e639ed992ba3 at mail.gmail.com>, "Eden > Li" writes: > >>I'm testing out Varnish ESI, [...] > > Which Varnish version ? > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > From phk at phk.freebsd.dk Wed Dec 3 18:46:08 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 03 Dec 2008 18:46:08 +0000 Subject: ESI objects hang for HTTP/1.0 clients In-Reply-To: Your message of "Wed, 03 Dec 2008 10:27:37 PST." <180e6a10812031027x1b366462tdf3b2b30fc7e1547@mail.gmail.com> Message-ID: <44609.1228329968@critter.freebsd.dk> In message <180e6a10812031027x1b366462tdf3b2b30fc7e1547 at mail.gmail.com>, "Eden Li" writes: Check that the parameter backend_http11 is set to "on" Poul-Henning >Varnish 2.0.2 using the options: > >-a 0.0.0.0:80 -T 127.0.0.1:9999 -f /usr/local/etc/vcl.conf -s >file,/var/varnish/cache > >The VCL is listed in http://pastie.org/329955 > >On Wed, Dec 3, 2008 at 9:54 AM, Poul-Henning Kamp wrote: >> In message <180e6a10812030948j4763275elb682e639ed992ba3 at mail.gmail.com>, "Eden >> Li" writes: >> >>>I'm testing out Varnish ESI, [...] >> >> Which Varnish version ? >> >> -- >> Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 >> phk at FreeBSD.ORG | TCP/IP since RFC 956 >> FreeBSD committer | BSD since 4.3-tahoe >> Never attribute to malice what can adequately be explained by incompetence. >> > -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From chris2 at rixon.net Wed Dec 3 18:40:59 2008 From: chris2 at rixon.net (Chris) Date: Wed, 3 Dec 2008 18:40:59 +0000 Subject: obj.ttl not working ? Message-ID: <77de6c9a0812031040s3f7b4886ie3ac4470c4d4fd23@mail.gmail.com> We are trying to get varnish to remove from its store all objects older than 1 day. Here's the vcl: sub vcl_fetch { set obj.grace = 2m; if( obj.ttl > 1d) { set obj.ttl = 1d; } if (obj.http.Pragma ~ "no-cache" || obj.http.Cache-Control ~ "no-cache" || obj.http.Cache-Control ~ "private") { pass; } } It doesn't seem to work the objects just go up and up :-( Varnish 2.0.1 From phk at phk.freebsd.dk Wed Dec 3 19:58:17 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 03 Dec 2008 19:58:17 +0000 Subject: ESI objects hang for HTTP/1.0 clients In-Reply-To: Your message of "Wed, 03 Dec 2008 11:54:05 PST." <180e6a10812031154wcfb577cxe7af56c852de244@mail.gmail.com> Message-ID: <44934.1228334297@critter.freebsd.dk> In message <180e6a10812031154wcfb577cxe7af56c852de244 at mail.gmail.com>, "Eden Li " writes: >On Wed, Dec 3, 2008 at 10:46 AM, Poul-Henning Kamp wrote: >> Check that the parameter backend_http11 is set to "on" > >It wasn't set, but the results were the same after I set it to "on". >I also tried setting client_http11 to "on", but still saw the same >results. > >If I set the backend to return non-XML formatted data or if I set >esi_syntax to "2", it seems to fix this (e.g. there's no delay). > >Is this setting required if we expect to deliver XML data to HTTP/1.0 clients? backend_http11 should probably be on always these days. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From eden at mojiti.com Wed Dec 3 20:09:11 2008 From: eden at mojiti.com (Eden Li) Date: Wed, 3 Dec 2008 12:09:11 -0800 Subject: ESI objects hang for HTTP/1.0 clients In-Reply-To: <44934.1228334297@critter.freebsd.dk> References: <180e6a10812031154wcfb577cxe7af56c852de244@mail.gmail.com> <44934.1228334297@critter.freebsd.dk> Message-ID: <180e6a10812031209y3a94f376j55f3b57396f593dc@mail.gmail.com> On Wed, Dec 3, 2008 at 11:58 AM, Poul-Henning Kamp wrote: > In message <180e6a10812031154wcfb577cxe7af56c852de244 at mail.gmail.com>, "Eden Li > " writes: >>On Wed, Dec 3, 2008 at 10:46 AM, Poul-Henning Kamp wrote: >>> Check that the parameter backend_http11 is set to "on" >> >>It wasn't set, but the results were the same after I set it to "on". >>I also tried setting client_http11 to "on", but still saw the same >>results. >> >>If I set the backend to return non-XML formatted data or if I set >>esi_syntax to "2", it seems to fix this (e.g. there's no delay). >> >>Is this setting required if we expect to deliver XML data to HTTP/1.0 clients? > > backend_http11 should probably be on always these days. OK. Should esi_syntax always be set to 2 as well? From phk at phk.freebsd.dk Wed Dec 3 20:25:30 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 03 Dec 2008 20:25:30 +0000 Subject: ESI objects hang for HTTP/1.0 clients In-Reply-To: Your message of "Wed, 03 Dec 2008 12:09:11 PST." <180e6a10812031209y3a94f376j55f3b57396f593dc@mail.gmail.com> Message-ID: <45101.1228335930@critter.freebsd.dk> In message <180e6a10812031209y3a94f376j55f3b57396f593dc at mail.gmail.com>, "Eden Li" writes: >OK. Should esi_syntax always be set to 2 as well? That is more of a site-specific question depending on what kind of objects you handle with ESI... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From tim at metaweb.com Wed Dec 3 23:51:23 2008 From: tim at metaweb.com (Tim Kientzle) Date: Wed, 3 Dec 2008 15:51:23 -0800 Subject: default.vcl insufficient Message-ID: <266C5169-9A5D-4385-B846-9A5302E089BB@metaweb.com> According to the "Getting Started" guide on the website, I should be able to just run varnish with the default VCL, but that doesn't actually work. I get this error: "No backends or directors found in VCL program, at least one is necessary. VCL compilation failed" I recommend that the "backend default" definition in default.vcl be uncommented in order to make it easier for people to get started. Tim From tim at metaweb.com Thu Dec 4 00:08:47 2008 From: tim at metaweb.com (Tim Kientzle) Date: Wed, 3 Dec 2008 16:08:47 -0800 Subject: default.vcl insufficient In-Reply-To: <266C5169-9A5D-4385-B846-9A5302E089BB@metaweb.com> References: <266C5169-9A5D-4385-B846-9A5302E089BB@metaweb.com> Message-ID: <197DF6EF-2FEE-401B-A833-F336625D412D@metaweb.com> Varnish's startup error messaging seems slightly broken. It took me a while to track down why varnish wasn't starting for me, since it just silently exited with nothing to the console and nothing to the varnishlog. By trial and error, I figured out that I needed an explicit -a option (why can't this be set in the VCL?). I then found this in the source code: In mgt_child.c, around line 245: if (open_sockets() != 0) { .... ... REPORT0(LOG_ERR, "Child start failed: could not open sockets"); return; } And REPORT0 calls both fprintf(stderr) and syslog(). So now I know to look in syslog. But I'm not sure why the fprintf(stderr) went nowhere. This seems to be before the child is forked and stdio redirected. Tim On Dec 3, 2008, at 3:51 PM, Tim Kientzle wrote: > According to the "Getting Started" guide on the website, I should be > able to just run varnish with the default VCL, but that doesn't > actually work. I get this error: > > "No backends or directors found in VCL program, at least one is > necessary. > VCL compilation failed" > > I recommend that the "backend default" definition in default.vcl be > uncommented in order to make it easier for people to get started. > > Tim > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc From plfgoa at gmail.com Thu Dec 4 05:33:21 2008 From: plfgoa at gmail.com (Paras Fadte) Date: Thu, 4 Dec 2008 11:03:21 +0530 Subject: small free smf Message-ID: <75cf5800812032133y7311f356n49abd97dadf86bb7@mail.gmail.com> Hi, What does "small free smf" shown by "varnishstat" represent ? I see substantial increase at times and the CPU usage increases. Thank you. -Paras From plfgoa at gmail.com Thu Dec 4 07:13:02 2008 From: plfgoa at gmail.com (Paras Fadte) Date: Thu, 4 Dec 2008 12:43:02 +0530 Subject: 5xx errors and Varnish Message-ID: <75cf5800812032313t477d26depf8ec5ccc5e3c3225@mail.gmail.com> Hi, What setting can get rid of 5xx errors for a valid request in varnish acting as a caching server for number of backend origin servers? Thank you. -Paras From phk at phk.freebsd.dk Thu Dec 4 10:29:46 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 04 Dec 2008 10:29:46 +0000 Subject: VCL program naming Message-ID: <1287.1228386586@critter.freebsd.dk> I have never been quite happy with the current namespace for vcl programs and the recent work on management tools have shown that it is indeed not very practical. Having pondered the question, I have come up with the following idea instead: we add version numbers to the names of loaded VCLs, and allow CLI commands to optionally be specify them. Here is a mockup to show how it works: $ varnishd -f myvcl.vcl -d -d ... [...] vcl.list 200 # active 21 boot:1 2008-12-04 10:03:06 UTC -f myvcl.vcl # The -b or -f argument, always turns into a VCL named "boot" # Notice the ":1" version number. vcl.load test myvcl2.vcl 200 # vcl.list 200 # active 34 boot:1 2008-12-04 10:03:06 UTC -f myvcl.vcl - 0 test:1 2008-12-04 10:07:16 UTC myvcl2.vcl by stdin # we have loaded the test vcl, it gets version 1 as well. # VCLs do not get activated automatically. vcl.use test 200 # vcl.list 200 # - 34 boot:1 2008-12-04 10:03:06 UTC -f myvcl.vcl active 3 test:1 2008-12-04 10:07:16 UTC myvcl2.vcl by stdin # Now we are running the test vcl vcl.load boot myvcl.vcl 200 # vcl.list 200 # - 14 boot.1 2008-12-04 10:03:06 UTC -f myvcl.vcl - 0 boot:2 2008-12-04 10:08:34 UTC myvcl.vcl by stdin active 13 test:1 2008-12-04 10:07:16 UTC myvcl2.vcl by stdin # we have loaded another "boot" vcl and it gets version # number two. vcl.use boot 200 # vcl.list 200 # - 08 boot.1 2008-12-04 10:03:06 UTC -f myvcl.vcl active 12 boot:2 2008-12-04 10:08:34 UTC myvcl.vcl by stdin - 12 test:1 2008-12-04 10:07:16 UTC myvcl2.vcl by stdin # If no version is specified, the highst version number # is assumed. vcl.discard test 200 # vcl.list 200 # active 22 boot.1 2008-12-04 10:03:06 UTC -f myvcl.vcl - 6 boot:2 2008-12-04 10:08:34 UTC myvcl.vcl by stdin # Discard without version number removes all non-active VCLs # of that # name. vcl.load boot myvcl.vcl 200 # vcl.list 200 # active 22 boot.1 2008-12-04 10:03:06 UTC -f myvcl.vcl - 6 boot:2 2008-12-04 10:08:34 UTC myvcl.vcl by stdin - 0 boot:3 2008-12-04 10:10:51 UTC myvcl.vcl by stdin # Another version of the boot vcl vcl.use boot:2 200 # vcl.list 200 # - 18 boot.1 2008-12-04 10:03:06 UTC -f myvcl.vcl active 15 boot:2 2008-12-04 10:08:34 UTC myvcl.vcl by stdin - 0 boot:3 2008-12-04 10:10:51 UTC myvcl.vcl by stdin # We can say which exact version we want vcl.discard boot 200 # vcl.list 200 # active 15 boot:2 2008-12-04 10:08:34 UTC myvcl.vcl by stdin # And discard without version purges all non-active versions, # even if they are higher version numbers than the active. vcl.list -d 200 # discard 7 boot.1 2008-12-04 10:03:06 UTC -f myvcl.vcl active 23 boot:2 2008-12-04 10:08:34 UTC myvcl.vcl by stdin discard 2 test:1 2008-12-04 10:07:16 UTC myvcl2.vcl by stdin # the -d option also shows discarded VCLs that are still not # cleaned up, because sessions still reference them. Input & comments welcome! Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From jeff at funnyordie.com Thu Dec 4 19:59:37 2008 From: jeff at funnyordie.com (Jeff Anderson) Date: Thu, 4 Dec 2008 11:59:37 -0800 Subject: Varnish Serves only uncompressed objects if they are requested first In-Reply-To: <493628FE.9000207@linpro.no> References: <4DB04145-6ECB-4887-AEAC-BD7AF508E28F@funnyordie.com> <493628FE.9000207@linpro.no> Message-ID: <95F4A52A-9C64-452E-A59E-636A85CA7048@funnyordie.com> Our app servers are sending the Vary on the Accept-Encoding when compression is requested. If compression is not requested they do not perform the Vary. Does that mean we should find a way to send a Vary: Accept-Encoding: null,gzip,deflate or something? Is there a 'no compression' accept-encoding header? On Dec 2, 2008, at 10:36 PM, Per Buer wrote: > Jeff Anderson skrev: >> It looks like if the first requested page is for an uncompressed page >> varnish will only deliver the uncompressed page from cache even if a >> compressed page is requested. > > As long as you don't Vary: on the Accept-Encoding I guess that is > expected. Varnish doesn't not understand the Accept-Encoding header. > >> (..) >> What could be causing this? The only way to fix appears to be to add >> the lines below: >> >> sub vcl_hash { >> if (req.http.Accept-Encoding ~ "gzip" || req.http.Accept-Encoding ~ >> "deflate") { >> set req.hash += req.http.Accept-Encoding; >> } > > Either use the fix you suggested or add a Vary: Accept-Encoding on the > backend. > > -- > Per Buer - Leder Infrastruktur og Drift - Redpill Linpro > Telefon: 21 54 41 21 - Mobil: 958 39 117 > http://linpro.no/ | http://redpill.se/ > > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc From ric at digitalmarbles.com Thu Dec 4 22:49:44 2008 From: ric at digitalmarbles.com (Ricardo Newbery) Date: Thu, 4 Dec 2008 14:49:44 -0800 Subject: [varnish] Re: Varnish Serves only uncompressed objects if they are requested first In-Reply-To: <95F4A52A-9C64-452E-A59E-636A85CA7048@funnyordie.com> References: <4DB04145-6ECB-4887-AEAC-BD7AF508E28F@funnyordie.com> <493628FE.9000207@linpro.no> <95F4A52A-9C64-452E-A59E-636A85CA7048@funnyordie.com> Message-ID: <3C79B184-790A-4878-9BDD-63873DDF8EFD@digitalmarbles.com> Your app server should set the Vary on *all* responses if *any* response can vary. Ric On Dec 4, 2008, at 11:59 AM, Jeff Anderson wrote: > Our app servers are sending the Vary on the Accept-Encoding when > compression is requested. If compression is not requested they do not > perform the Vary. Does that mean we should find a way to send a Vary: > Accept-Encoding: null,gzip,deflate or something? Is there a 'no > compression' accept-encoding header? > > > On Dec 2, 2008, at 10:36 PM, Per Buer wrote: > >> Jeff Anderson skrev: >>> It looks like if the first requested page is for an uncompressed >>> page >>> varnish will only deliver the uncompressed page from cache even if a >>> compressed page is requested. >> >> As long as you don't Vary: on the Accept-Encoding I guess that is >> expected. Varnish doesn't not understand the Accept-Encoding header. >> >>> (..) >>> What could be causing this? The only way to fix appears to be to >>> add >>> the lines below: >>> >>> sub vcl_hash { >>> if (req.http.Accept-Encoding ~ "gzip" || req.http.Accept-Encoding ~ >>> "deflate") { >>> set req.hash += req.http.Accept-Encoding; >>> } >> >> Either use the fix you suggested or add a Vary: Accept-Encoding on >> the >> backend. >> >> -- >> Per Buer - Leder Infrastruktur og Drift - Redpill Linpro >> Telefon: 21 54 41 21 - Mobil: 958 39 117 >> http://linpro.no/ | http://redpill.se/ >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at projects.linpro.no >> http://projects.linpro.no/mailman/listinfo/varnish-misc > > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc From jeff at funnyordie.com Thu Dec 4 22:54:06 2008 From: jeff at funnyordie.com (Jeff Anderson) Date: Thu, 4 Dec 2008 14:54:06 -0800 Subject: [varnish] Re: Varnish Serves only uncompressed objects if they are requested first In-Reply-To: <3C79B184-790A-4878-9BDD-63873DDF8EFD@digitalmarbles.com> References: <4DB04145-6ECB-4887-AEAC-BD7AF508E28F@funnyordie.com> <493628FE.9000207@linpro.no> <95F4A52A-9C64-452E-A59E-636A85CA7048@funnyordie.com> <3C79B184-790A-4878-9BDD-63873DDF8EFD@digitalmarbles.com> Message-ID: <20BCFC2D-DDDE-487D-A010-55B6AC1DC3C0@funnyordie.com> Thank you. On Dec 4, 2008, at 2:49 PM, Ricardo Newbery wrote: > > > Your app server should set the Vary on *all* responses if *any* > response can vary. > > Ric > > > > On Dec 4, 2008, at 11:59 AM, Jeff Anderson wrote: > >> Our app servers are sending the Vary on the Accept-Encoding when >> compression is requested. If compression is not requested they do >> not >> perform the Vary. Does that mean we should find a way to send a >> Vary: >> Accept-Encoding: null,gzip,deflate or something? Is there a 'no >> compression' accept-encoding header? >> >> >> On Dec 2, 2008, at 10:36 PM, Per Buer wrote: >> >>> Jeff Anderson skrev: >>>> It looks like if the first requested page is for an uncompressed >>>> page >>>> varnish will only deliver the uncompressed page from cache even >>>> if a >>>> compressed page is requested. >>> >>> As long as you don't Vary: on the Accept-Encoding I guess that is >>> expected. Varnish doesn't not understand the Accept-Encoding header. >>> >>>> (..) >>>> What could be causing this? The only way to fix appears to be to >>>> add >>>> the lines below: >>>> >>>> sub vcl_hash { >>>> if (req.http.Accept-Encoding ~ "gzip" || req.http.Accept-Encoding ~ >>>> "deflate") { >>>> set req.hash += req.http.Accept-Encoding; >>>> } >>> >>> Either use the fix you suggested or add a Vary: Accept-Encoding on >>> the >>> backend. >>> >>> -- >>> Per Buer - Leder Infrastruktur og Drift - Redpill Linpro >>> Telefon: 21 54 41 21 - Mobil: 958 39 117 >>> http://linpro.no/ | http://redpill.se/ >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at projects.linpro.no >>> http://projects.linpro.no/mailman/listinfo/varnish-misc >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at projects.linpro.no >> http://projects.linpro.no/mailman/listinfo/varnish-misc > From chris2 at rixon.net Fri Dec 5 11:05:31 2008 From: chris2 at rixon.net (Chris) Date: Fri, 5 Dec 2008 11:05:31 +0000 Subject: obj.ttl not working ? In-Reply-To: <77de6c9a0812031040s3f7b4886ie3ac4470c4d4fd23@mail.gmail.com> References: <77de6c9a0812031040s3f7b4886ie3ac4470c4d4fd23@mail.gmail.com> Message-ID: <77de6c9a0812050305n721aa6c4t14951d7cbbbe431d@mail.gmail.com> Here is a picture of what we are seeing http://i38.tinypic.com/2zs8iu0.png From des at des.no Fri Dec 5 11:45:47 2008 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Fri, 05 Dec 2008 12:45:47 +0100 Subject: default.vcl insufficient In-Reply-To: <266C5169-9A5D-4385-B846-9A5302E089BB@metaweb.com> (Tim Kientzle's message of "Wed, 3 Dec 2008 15:51:23 -0800") References: <266C5169-9A5D-4385-B846-9A5302E089BB@metaweb.com> Message-ID: <86k5aehml0.fsf@ds4.des.no> Tim Kientzle writes: > According to the "Getting Started" guide on the website, I should be > able to just run varnish with the default VCL, but that doesn't > actually work. Yes, it does. Varnish requires that you specify *either* a VCL script *or* a backend on the command line. What you did wrong was that you specified an empty VCL script. The default VCL script is a copy of the build-in default script that Varnish uses if you don't specify one, and is provided for documentation purposes only; you should never use it. (and yes, the varnishd rc script in the FreeBSD port is incorrect in this respect; patches are welcome) DES -- Dag-Erling Sm?rgrav - des at des.no From tim at metaweb.com Fri Dec 5 18:48:42 2008 From: tim at metaweb.com (Tim Kientzle) Date: Fri, 5 Dec 2008 10:48:42 -0800 Subject: default.vcl insufficient In-Reply-To: <86k5aehml0.fsf@ds4.des.no> References: <266C5169-9A5D-4385-B846-9A5302E089BB@metaweb.com> <86k5aehml0.fsf@ds4.des.no> Message-ID: On Dec 5, 2008, at 3:45 AM, Dag-Erling Sm?rgrav wrote: > Tim Kientzle writes: >> According to the "Getting Started" guide on the website, I should be >> able to just run varnish with the default VCL, but that doesn't >> actually work. > > Yes, it does. Varnish requires that you specify *either* a VCL script > *or* a backend on the command line. What you did wrong was that you > specified an empty VCL script. The default VCL script is a copy of > the > build-in default script that Varnish uses if you don't specify one, > and > is provided for documentation purposes only; you should never use it. I think you just managed to say both "Yes, it does work to just run varnish with the default VCL" and "you should never use the default VCL script." ;-) In either case, the online documentation could use an update. http://varnish.projects.linpro.no/static/getting-started.html Under "Installing Varnish from source", it claims you should configure, make, make install, then: "# Start Varnish by typing: varnishd -f /etc/varnish/default.vcl" Which clearly doesn't actually work. It's good to have an easy "make sure it works" step here. I would suggest providing a separate "varnish.vcl" file specifically for people to customize that by default is just enough to get varnish running, installing this along side default.vcl, and editing the documentation here to refer to "varnish.vcl". Your comment above should also be added to the varnishd.1 man page for the -b option to make clear that it's required if there is no backend defined in the specified VCL file. Tim From phk at phk.freebsd.dk Fri Dec 5 19:01:58 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 05 Dec 2008 19:01:58 +0000 Subject: default.vcl insufficient In-Reply-To: Your message of "Fri, 05 Dec 2008 10:48:42 PST." Message-ID: <23468.1228503718@critter.freebsd.dk> In message , Tim Kientzle wri tes: > "# Start Varnish by typing: varnishd -f /etc/varnish/default.vcl" > >Which clearly doesn't actually work. This is wrong and should be fixed. The default VCL does not contain any backend specifications, as we have no idea where to find the users backend. THe default.vcl is actually compiled in, and need not be provided, a simple backend specification is all that is necessary. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From lists at rebert.name Mon Dec 8 10:19:58 2008 From: lists at rebert.name (Rebert Luc) Date: Mon, 8 Dec 2008 11:19:58 +0100 Subject: Varnish performance tests Message-ID: <48295.1228731598@rebert.name> Hello, In our studies we have a project which consists in testing the performance of Varnish in order to make a comparative with and without the proxy cache. Does anyone know which utilities to employ ? (knowing that the aim is to justify our tests) Thanks in advance. REBERT Luc & MORTREAU Alexandre -------------- next part -------------- An HTML attachment was scrubbed... URL: From johan.grasmo at allerinternett.no Wed Dec 3 12:32:43 2008 From: johan.grasmo at allerinternett.no (Grasmo, Johan) Date: Wed, 3 Dec 2008 13:32:43 +0100 Subject: Strange varnish behavior - not answering requests Message-ID: Hi, We're tired of squid and it's quirks and want to advance into the information age with varnish. I've been testing varnish the last couple of days and it has been behaving perfectly until last night. When stresstesting the varnishserver it suddenly stopped handling requests. Varnish returns to normal operation after a restart. Symptom: Accessing a page on the varnishsserver results in a timeout in the browser I am able to telnet to the varnishserver on port 80 but a GET doesn't result in anything ("hangs"). I am able to telnet to port 6082 and run commands: [snip] status 200 22 Child in state running [/snip] Varnishlog returns: [snip] 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1228307253 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1228307256 1.0 0 CLI - Rd ping [/snip] Varnishncsa doesn't return anything. I've verified with tcpdump that varnish ack's requests. So "everything seems normal" even though clients get timeout. FF returned "Connection Interrupted The document contains no data". If you need any logs/configs please let me know :) I haven't exhausted the search function for the mailinglist so sorry if this topic has been an issue before. I'm running varnish on an Ubuntu 8.04 distro. Cheers and thanks for replies, Johan Grasmo From tfheen at linpro.no Mon Dec 8 14:39:05 2008 From: tfheen at linpro.no (Tollef Fog Heen) Date: Mon, 08 Dec 2008 15:39:05 +0100 Subject: How much of the ESI spec is implemented? In-Reply-To: <114A665E8FCADD4199D425621C5FF9C6789F3662@34093-MBX-C02.mex07a.mlsrvr.com> (Kurt Mackey's message of "Thu, 9 Oct 2008 07:38:33 -0500") References: <114A665E8FCADD4199D425621C5FF9C6789F3662@34093-MBX-C02.mex07a.mlsrvr.com> Message-ID: <87oczmenp2.fsf@qurzaw.linpro.no> ]] Kurt Mackey First, sorry for taking ages to get back to you. | So ESI looks interesting, but I can't quite figure out the extent to | which Varnish supports it. We just support the simple esi:include, ESI comments and esi:remove, not anything more at the moment. | For instance, the ESI expressions look ridiculously useful in | conjunction with the include tag. Can you do things like this? | | Yes, it'd be quite useful. | I'm not good enough to completely understand the source, but it looks like that example may not be possible. | | It would be awesome to see expression support and support for the | choose/when/otherwise tags as well. Is that the type of thing that | someone could sponsor for development? I don't believe we will want to do choose/when/otherwise, but sponsoring of includes based on cookies is something that would be interesting to us. Feel free to get in touch with varnish at linpro about it. -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 From perbu at linpro.no Mon Dec 8 17:03:43 2008 From: perbu at linpro.no (Per Buer) Date: Mon, 08 Dec 2008 18:03:43 +0100 Subject: Varnish performance tests In-Reply-To: <48295.1228731598@rebert.name> References: <48295.1228731598@rebert.name> Message-ID: <493D536F.3030103@linpro.no> Rebert Luc wrote: > Hello, > > In our studies we have a project which consists in testing the > performance of Varnish in order to make a comparative with and without > the proxy cache. > Does anyone know which utilities to employ ? (knowing that the aim is to > justify our tests) You could have a look at varnishreplay to "replay" earlier recorded varnishlogs. If you also want to synthesize traffic you could also check out siege and curl-loader. If you have a high hitrate in your cache you'll most likely end up benchmarking the tools rather then varnish. I think I measured siege to be ~3x slower then varnish. -- Per Buer - Leder Infrastruktur og Drift - Redpill Linpro Telefon: 21 54 41 21 - Mobil: 958 39 117 http://linpro.no/ | http://redpill.se/ From michael at dynamine.net Mon Dec 8 17:05:38 2008 From: michael at dynamine.net (Michael S. Fischer) Date: Mon, 8 Dec 2008 09:05:38 -0800 Subject: Varnish performance tests In-Reply-To: <493D536F.3030103@linpro.no> References: <48295.1228731598@rebert.name> <493D536F.3030103@linpro.no> Message-ID: On Dec 8, 2008, at 9:03 AM, Per Buer wrote: > Rebert Luc wrote: >> Hello, >> >> In our studies we have a project which consists in testing the >> performance of Varnish in order to make a comparative with and >> without >> the proxy cache. >> Does anyone know which utilities to employ ? (knowing that the aim >> is to >> justify our tests) > > You could have a look at varnishreplay to "replay" earlier recorded > varnishlogs. If you also want to synthesize traffic you could also > check > out siege and curl-loader. > > If you have a high hitrate in your cache you'll most likely end up > benchmarking the tools rather then varnish. I think I measured siege > to > be ~3x slower then varnish. I used httperf with great success. However, it's only single- threaded, so you may want to put a wrapper around it to test request concurrencies > 1. --Michael From tfheen at linpro.no Tue Dec 9 00:15:27 2008 From: tfheen at linpro.no (Tollef Fog Heen) Date: Tue, 09 Dec 2008 01:15:27 +0100 Subject: Varnish performance tests In-Reply-To: <48295.1228731598@rebert.name> (Rebert Luc's message of "Mon, 8 Dec 2008 11:19:58 +0100") References: <48295.1228731598@rebert.name> Message-ID: <874p1ew6e8.fsf@qurzaw.linpro.no> ]] "Rebert Luc" | In our studies we have a project which consists in testing the | performance of Varnish in order to make a comparative with and without | the proxy cache. | Does anyone know which utilities to employ ? (knowing that the aim | is to justify our tests) http://www.web-cache.com/benchmarking.html has some pointers. I've had interesting numbers come out of polygraph. -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 From tfheen at linpro.no Tue Dec 9 09:35:23 2008 From: tfheen at linpro.no (Tollef Fog Heen) Date: Tue, 09 Dec 2008 10:35:23 +0100 Subject: 5xx errors and Varnish In-Reply-To: <75cf5800812032313t477d26depf8ec5ccc5e3c3225@mail.gmail.com> (Paras Fadte's message of "Thu, 4 Dec 2008 12:43:02 +0530") References: <75cf5800812032313t477d26depf8ec5ccc5e3c3225@mail.gmail.com> Message-ID: <87r64h3d44.fsf@qurzaw.linpro.no> ]] "Paras Fadte" | What setting can get rid of 5xx errors for a valid request in varnish | acting as a caching server for number of backend origin servers? I think you need to describe your problem in a bit more concrete terms. What are you trying to do, what does your setup look like and what happens when you do what? -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 From tfheen at linpro.no Tue Dec 9 09:40:43 2008 From: tfheen at linpro.no (Tollef Fog Heen) Date: Tue, 09 Dec 2008 10:40:43 +0100 Subject: small free smf In-Reply-To: <75cf5800812032133y7311f356n49abd97dadf86bb7@mail.gmail.com> (Paras Fadte's message of "Thu, 4 Dec 2008 11:03:21 +0530") References: <75cf5800812032133y7311f356n49abd97dadf86bb7@mail.gmail.com> Message-ID: <87myf53cv8.fsf@qurzaw.linpro.no> ]] "Paras Fadte" | What does "small free smf" shown by "varnishstat" represent ? I see | substantial increase at times and the CPU usage increases. It's just showing you that it is removing objects and putting memory pages back on the ?free? list. -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 From tfheen at linpro.no Tue Dec 9 09:45:49 2008 From: tfheen at linpro.no (Tollef Fog Heen) Date: Tue, 09 Dec 2008 10:45:49 +0100 Subject: Varnishreplay question In-Reply-To: <478514.24557.qm@web26501.mail.ukl.yahoo.com> (Serge Helly's message of "Sun, 12 Oct 2008 12:07:42 +0000 (GMT)") References: <478514.24557.qm@web26501.mail.ukl.yahoo.com> Message-ID: <87iqpt3cmq.fsf@qurzaw.linpro.no> ]] Serge Helly | I've been playing with Varnish for a couple of months now: impressive, really ! Thanks! | I would like to replay Apache logs. | Is there a way to run "varnishreplay -r ncsa-file" ? Currently, there's no way to do it, at least. | If no, could that be consider as a future enhancement ? I doubt it's something we'll want to put significant effort into, but if somebody shows up with a patch, I'll be happy to have it applied. -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 From plzeton at gmail.com Tue Dec 9 21:52:10 2008 From: plzeton at gmail.com (=?ISO-8859-2?Q?Tomasz_Leszczy=F1ski?=) Date: Tue, 09 Dec 2008 22:52:10 +0100 Subject: Varnish ALWAYS getting index from backend Message-ID: <493EE88A.4040308@gmail.com> Headers from backend (connect directly to backend): HTTP/1.x 200 OK Date: Tue, 09 Dec 2008 21:42:54 GMT Server: Apache/1.3.33 (Debian GNU/Linux) PHP/4.3.10-18 X-Powered-By: PHP/4.3.10-18 Set-Cookie: CMS_Session=95775f09228158a4a0e16a617d8c2372; path=/ Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Keep-Alive: timeout=15, max=100 Connection: Keep-Alive Transfer-Encoding: chunked Content-Type: text/html; charset=iso-8859-2 I try remove cookies... not work. Removing Cache-Control, pragma etc. in vcl_fetch subroutine - not effect. Line from varnishncsa -b: 192.168.3.34 - - [09/Dec/2008:21:49:00 +0000] "GET http://192.168.3.34/ HTTP/1.1" 200 - "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; pl; rv:1.9.0.4) Gecko/2008102920 Firefox/3.0.4" (rows are broken) Backend is SLOW. Very slow. From malevo at gmail.com Tue Dec 9 22:38:29 2008 From: malevo at gmail.com (=?ISO-8859-1?Q?Pablo_Garc=EDa?=) Date: Tue, 9 Dec 2008 20:38:29 -0200 Subject: Varnish ALWAYS getting index from backend In-Reply-To: <493EE88A.4040308@gmail.com> References: <493EE88A.4040308@gmail.com> Message-ID: <4ec3c3f70812091438m70db8775i68ecc78de09b6421@mail.gmail.com> Have you removed expires header ? Also, you removed those headers, but do you specify any expiration ? Regards, Pablo 2008/12/9, Tomasz Leszczy?ski : > Headers from backend (connect directly to backend): > > HTTP/1.x 200 OK > Date: Tue, 09 Dec 2008 21:42:54 GMT > Server: Apache/1.3.33 (Debian GNU/Linux) PHP/4.3.10-18 > X-Powered-By: PHP/4.3.10-18 > Set-Cookie: CMS_Session=95775f09228158a4a0e16a617d8c2372; path=/ > Expires: Thu, 19 Nov 1981 08:52:00 GMT > Cache-Control: no-store, no-cache, must-revalidate, post-check=0, > pre-check=0 > Pragma: no-cache > Keep-Alive: timeout=15, max=100 > Connection: Keep-Alive > Transfer-Encoding: chunked > Content-Type: text/html; charset=iso-8859-2 > > > > I try remove cookies... not work. Removing Cache-Control, pragma etc. in > vcl_fetch subroutine - not effect. > > > Line from varnishncsa -b: > > 192.168.3.34 - - [09/Dec/2008:21:49:00 +0000] "GET http://192.168.3.34/ > HTTP/1.1" 200 - "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; pl; > rv:1.9.0.4) Gecko/2008102920 Firefox/3.0.4" > > (rows are broken) > > Backend is SLOW. Very slow. > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc > From plzeton at gmail.com Tue Dec 9 22:48:16 2008 From: plzeton at gmail.com (=?ISO-8859-2?Q?Tomasz_Leszczy=F1ski?=) Date: Tue, 09 Dec 2008 23:48:16 +0100 Subject: Varnish ALWAYS getting index from backend In-Reply-To: <4ec3c3f70812091438m70db8775i68ecc78de09b6421@mail.gmail.com> References: <493EE88A.4040308@gmail.com> <4ec3c3f70812091438m70db8775i68ecc78de09b6421@mail.gmail.com> Message-ID: <493EF5B0.6050503@gmail.com> Pablo Garc?a pisze: > Have you removed expires header ? Also, you removed those headers, but > do you specify any expiration ? Something like that: sub vcl_fetch { remove obj.http.Cache-Control; remove obj.http.Pragma; set obj.http.Expires = "Thu, 19 Nov 2010 08:52:00 GMT"; } or like that: sub vcl_fetch { remove obj.http.Cache-Control; remove obj.http.Pragma; remove obj.http.Expires; set obj.http.Expires = "Thu, 19 Nov 2010 08:52:00 GMT"; } Not work :| From Neil.Scholten at scoyo.com Wed Dec 10 08:30:01 2008 From: Neil.Scholten at scoyo.com (Neil.Scholten at scoyo.com) Date: Wed, 10 Dec 2008 09:30:01 +0100 Subject: AW: Varnish ALWAYS getting index from backend In-Reply-To: <493EF5B0.6050503@gmail.com> References: <493EE88A.4040308@gmail.com><4ec3c3f70812091438m70db8775i68ecc78de09b6421@mail.gmail.com> <493EF5B0.6050503@gmail.com> Message-ID: Hi, How about the example from the varnish-site? http://varnish.projects.linpro.no/wiki/VCLExampleLongerCaching Here's the example: sub vcl_fetch { if (obj.cacheable) { /* Remove Expires from backend, it's not long enough */ unset obj.http.expires; /* Set the clients TTL on this object */ set obj.http.cache-control = "max-age = 900"; /* Set how long Varnish will keep it */ set obj.ttl = 1w; /* marker for vcl_deliver to reset Age: */ set obj.http.magicmarker = 1; } } sub vcl_deliver { if (resp.http.magicmarker) { /* Remove the magic marker */ unset resp.http.magicmarker; /* By definition we have a fresh object */ set resp.http.age = "0"; } } Worked for us so far. --neil -----Urspr?ngliche Nachricht----- Von: varnish-misc-bounces at projects.linpro.no [mailto:varnish-misc-bounces at projects.linpro.no] Im Auftrag von Tomasz Leszczynski Gesendet: Dienstag, 9. Dezember 2008 23:48 An: Pablo Garc?a Cc: varnish-misc at projects.linpro.no Betreff: Re: Varnish ALWAYS getting index from backend Pablo Garc?a pisze: > Have you removed expires header ? Also, you removed those headers, but > do you specify any expiration ? Something like that: sub vcl_fetch { remove obj.http.Cache-Control; remove obj.http.Pragma; set obj.http.Expires = "Thu, 19 Nov 2010 08:52:00 GMT"; } or like that: sub vcl_fetch { remove obj.http.Cache-Control; remove obj.http.Pragma; remove obj.http.Expires; set obj.http.Expires = "Thu, 19 Nov 2010 08:52:00 GMT"; } Not work :| _______________________________________________ varnish-misc mailing list varnish-misc at projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc From plzeton at gmail.com Wed Dec 10 08:42:33 2008 From: plzeton at gmail.com (=?windows-1252?Q?Tomasz_Leszczyn=27ski?=) Date: Wed, 10 Dec 2008 09:42:33 +0100 Subject: AW: Varnish ALWAYS getting index from backend In-Reply-To: References: <493EE88A.4040308@gmail.com><4ec3c3f70812091438m70db8775i68ecc78de09b6421@mail.gmail.com> <493EF5B0.6050503@gmail.com> Message-ID: <493F80F9.8090103@gmail.com> Neil.Scholten at scoyo.com pisze: > Hi, > > How about the example from the varnish-site? > > http://varnish.projects.linpro.no/wiki/VCLExampleLongerCaching > > Here's the example: > > sub vcl_fetch { > > if (obj.cacheable) { > /* Remove Expires from backend, it's not long enough */ > unset obj.http.expires; > > /* Set the clients TTL on this object */ > set obj.http.cache-control = "max-age = 900"; > > /* Set how long Varnish will keep it */ > set obj.ttl = 1w; > > /* marker for vcl_deliver to reset Age: */ > set obj.http.magicmarker = 1; > } > } > > sub vcl_deliver { > if (resp.http.magicmarker) { > /* Remove the magic marker */ > unset resp.http.magicmarker; > > /* By definition we have a fresh object */ > set resp.http.age = "0"; > } > } > > Worked for us so far. Expected string variable or constant (/etc/varnish.conf Line 19 Pos 44) set obj.http.magicmarker = 1; -------------------------------------------#- VCL compilation failed Version: piorin:~# varnishd -V varnishd (varnish-2.0.2) Copyright (c) 2006-2008 Linpro AS / Verdens Gang AS From Neil.Scholten at scoyo.com Wed Dec 10 09:51:46 2008 From: Neil.Scholten at scoyo.com (Neil.Scholten at scoyo.com) Date: Wed, 10 Dec 2008 10:51:46 +0100 Subject: AW: AW: Varnish ALWAYS getting index from backend In-Reply-To: <493F80F9.8090103@gmail.com> References: <493EE88A.4040308@gmail.com><4ec3c3f70812091438m70db8775i68ecc78de09b6421@mail.gmail.com> <493EF5B0.6050503@gmail.com> <493F80F9.8090103@gmail.com> Message-ID: Hi, Sorry, just Cut&Pasted the example. I had a look in our configuration: /* marker for vcl_deliver to reset Age: */ set obj.http.magicmarker = "1"; Just add the "" and it should work. The error you had pointed the direction: "Expected string variable or constant" --neil -----Urspr?ngliche Nachricht----- Von: varnish-misc-bounces at projects.linpro.no [mailto:varnish-misc-bounces at projects.linpro.no] Im Auftrag von Tomasz Leszczyn'ski Gesendet: Mittwoch, 10. Dezember 2008 09:43 An: Scholten, Neil, scoyo Cc: varnish-misc at projects.linpro.no Betreff: Re: AW: Varnish ALWAYS getting index from backend Neil.Scholten at scoyo.com pisze: > Hi, > > How about the example from the varnish-site? > > http://varnish.projects.linpro.no/wiki/VCLExampleLongerCaching > > Here's the example: > > sub vcl_fetch { > > if (obj.cacheable) { > /* Remove Expires from backend, it's not long enough */ > unset obj.http.expires; > > /* Set the clients TTL on this object */ > set obj.http.cache-control = "max-age = 900"; > > /* Set how long Varnish will keep it */ > set obj.ttl = 1w; > > /* marker for vcl_deliver to reset Age: */ > set obj.http.magicmarker = 1; > } > } > > sub vcl_deliver { > if (resp.http.magicmarker) { > /* Remove the magic marker */ > unset resp.http.magicmarker; > > /* By definition we have a fresh object */ > set resp.http.age = "0"; > } > } > > Worked for us so far. Expected string variable or constant (/etc/varnish.conf Line 19 Pos 44) set obj.http.magicmarker = 1; -------------------------------------------#- VCL compilation failed Version: piorin:~# varnishd -V varnishd (varnish-2.0.2) Copyright (c) 2006-2008 Linpro AS / Verdens Gang AS _______________________________________________ varnish-misc mailing list varnish-misc at projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc From plzeton at gmail.com Wed Dec 10 10:50:08 2008 From: plzeton at gmail.com (=?windows-1252?Q?Tomasz_Leszczyn=27ski?=) Date: Wed, 10 Dec 2008 11:50:08 +0100 Subject: AW: AW: Varnish ALWAYS getting index from backend In-Reply-To: References: <493EE88A.4040308@gmail.com><4ec3c3f70812091438m70db8775i68ecc78de09b6421@mail.gmail.com> <493EF5B0.6050503@gmail.com> <493F80F9.8090103@gmail.com> Message-ID: <493F9EE0.8010709@gmail.com> Neil.Scholten at scoyo.com pisze: > Hi, > > Sorry, just Cut&Pasted the example. > I had a look in our configuration: > > /* marker for vcl_deliver to reset Age: */ > set obj.http.magicmarker = "1"; > > Just add the "" and it should work. > > The error you had pointed the direction: > > "Expected string variable or constant" > > --neil My fault :) Anyway... don't work on my test.php :) Varnish always getting this from backend :| Once more: test.php: Refreshing webpage on client (addres: http://192.168.3.34/test.php) display current time. Commenting line number 2 helps with this vcl: backend default { .host = "192.168.3.5"; // use your own backend ip address .port = "80"; // use your own backend port } Simple, isn't :D Headers Cache-Control and Pragma not affect but for Expires header commands: remove, unset or set not helps (in vcl_fetch sub). Unfortunately my backend application is IonCubed (encrypted) and is from third party :D From des at des.no Wed Dec 10 13:32:49 2008 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Wed, 10 Dec 2008 14:32:49 +0100 Subject: Varnishreplay question In-Reply-To: <87iqpt3cmq.fsf@qurzaw.linpro.no> (Tollef Fog Heen's message of "Tue, 09 Dec 2008 10:45:49 +0100") References: <478514.24557.qm@web26501.mail.ukl.yahoo.com> <87iqpt3cmq.fsf@qurzaw.linpro.no> Message-ID: <86vdtsqhoe.fsf@ds4.des.no> Tollef Fog Heen writes: > Serge Helly writes: > > I would like to replay Apache logs. > > Is there a way to run "varnishreplay -r ncsa-file" ? > > If no, could that be consider as a future enhancement ? > I doubt it's something we'll want to put significant effort into, but if > somebody shows up with a patch, I'll be happy to have it applied. There isn't enough information in NCSA "combined" logs to replay the traffic that generated them, but you can set up Varnish in front of Apache to log HTTP traffic (just "pass" at the top of vcl_recv). DES -- Dag-Erling Sm?rgrav - des at des.no From des at des.no Wed Dec 10 13:36:04 2008 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Wed, 10 Dec 2008 14:36:04 +0100 Subject: Varnish ALWAYS getting index from backend In-Reply-To: <493EF5B0.6050503@gmail.com> ("Tomasz =?utf-8?Q?Leszczy=C5=84?= =?utf-8?Q?ski=22's?= message of "Tue, 09 Dec 2008 23:48:16 +0100") References: <493EE88A.4040308@gmail.com> <4ec3c3f70812091438m70db8775i68ecc78de09b6421@mail.gmail.com> <493EF5B0.6050503@gmail.com> Message-ID: <86r64gqhiz.fsf@ds4.des.no> Tomasz Leszczy?ski writes: > sub vcl_fetch { > remove obj.http.Cache-Control; > remove obj.http.Pragma; > set obj.http.Expires = "Thu, 19 Nov 2010 08:52:00 GMT"; > } That will change the stored header, but not Varnish's idea of the object's TTL (which is computed before vcl_fetch is called). The best solution is to set the correct time on the backend; second-best is to set obj.ttl to a non-zero value in vcl_fetch. DES -- Dag-Erling Sm?rgrav - des at des.no From plzeton at gmail.com Wed Dec 10 18:05:23 2008 From: plzeton at gmail.com (=?UTF-8?B?VG9tYXN6IExlc3pjennFhHNraQ==?=) Date: Wed, 10 Dec 2008 19:05:23 +0100 Subject: Varnish ALWAYS getting index from backend In-Reply-To: <86r64gqhiz.fsf@ds4.des.no> References: <493EE88A.4040308@gmail.com> <4ec3c3f70812091438m70db8775i68ecc78de09b6421@mail.gmail.com> <493EF5B0.6050503@gmail.com> <86r64gqhiz.fsf@ds4.des.no> Message-ID: <494004E3.1060207@gmail.com> Dag-Erling Sm?rgrav pisze: > The best > solution is to set the correct time on the backend; Mission impossible :| There are third party application on backend and encrypted. Third party is in reorganisation and we are unvisible for him now. XSS hole is signalized on april :| Without any effect. Budda not response. This application (CMS) is horrible slow. VARNISH are slavation :D > second-best is to > set obj.ttl to a non-zero value in vcl_fetch. Thats it. I know. But I completely don't understand why this not work. Beacuse not work. From alecshenry at gmail.com Wed Dec 10 19:22:33 2008 From: alecshenry at gmail.com (Alecs Henry) Date: Wed, 10 Dec 2008 17:22:33 -0200 Subject: Variable for backend host Message-ID: <3c54843f0812101122j7b1ba7end9930e2eea0a93ee@mail.gmail.com> Hi All! Is there any way to access the backend host name in VCL? What about obj.ttl? What I'd like to do is place that information in a header, and maybe log it. While we're at it, is there a listing of the available variables to be used in VCL? Thanks! Alecs -------------- next part -------------- An HTML attachment was scrubbed... URL: From plfgoa at gmail.com Thu Dec 11 12:43:57 2008 From: plfgoa at gmail.com (Paras Fadte) Date: Thu, 11 Dec 2008 18:13:57 +0530 Subject: 5xx errors and Varnish In-Reply-To: <87r64h3d44.fsf@qurzaw.linpro.no> References: <75cf5800812032313t477d26depf8ec5ccc5e3c3225@mail.gmail.com> <87r64h3d44.fsf@qurzaw.linpro.no> Message-ID: <75cf5800812110443h27be60f8lffb2bf1f45356dd4@mail.gmail.com> Hi Tollef , Set up is client requests --> Loadbalancer --> caching servers --> origin servers If I tend to get more 5xx errors when requests come to caching servers from load balancer than when the requests go directly to the origin servers from load balancer could it be then something related to with number of probes etc specified in varnish ? Can specifying a lower number of probes reduce the 5xx errors ? What do you suggest ? Thank you. -Paras On Tue, Dec 9, 2008 at 3:05 PM, Tollef Fog Heen wrote: > ]] "Paras Fadte" > > | What setting can get rid of 5xx errors for a valid request in varnish > | acting as a caching server for number of backend origin servers? > > I think you need to describe your problem in a bit more concrete terms. > What are you trying to do, what does your setup look like and what > happens when you do what? > > -- > Tollef Fog Heen > Redpill Linpro -- Changing the game! > t: +47 21 54 41 73 > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc > From plfgoa at gmail.com Thu Dec 11 12:28:09 2008 From: plfgoa at gmail.com (Paras Fadte) Date: Thu, 11 Dec 2008 17:58:09 +0530 Subject: small free smf In-Reply-To: <87myf53cv8.fsf@qurzaw.linpro.no> References: <75cf5800812032133y7311f356n49abd97dadf86bb7@mail.gmail.com> <87myf53cv8.fsf@qurzaw.linpro.no> Message-ID: <75cf5800812110428n1172c39bg220d6dfe082c5cb0@mail.gmail.com> Hi Tollef, Thanks for the reply I would also like to know is how to interpret "Backend connections failures" ? if I specify a file in probe and if it is moved temporarily , varnish would term it is "sick" till it finds the file ? is that considered as one of the "Backend connections failures" ? Example: backend o1 { .host = "10.1.0.100"; .probe = { .url = "/check.gif"; .timeout = 100 ms; .interval = 1s; .window = 10; .threshold = 6; } .port = "80"; } In the above if check.gif on 10.1.0.100 is not found , will each attempt of probing 10.1.0.100l be considered as a Backend connection failure ? Thank you. -Paras On Tue, Dec 9, 2008 at 3:10 PM, Tollef Fog Heen wrote: > ]] "Paras Fadte" > > | What does "small free smf" shown by "varnishstat" represent ? I see > | substantial increase at times and the CPU usage increases. > > It's just showing you that it is removing objects and putting memory > pages back on the ?free? list. > > -- > Tollef Fog Heen > Redpill Linpro -- Changing the game! > t: +47 21 54 41 73 > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc > From alex at davz.net Thu Dec 11 17:21:40 2008 From: alex at davz.net (Alex Davies) Date: Thu, 11 Dec 2008 17:21:40 +0000 Subject: Example VCL for backend check Message-ID: <5fb622120812110921x2833df5u38a5c89105fb79b9@mail.gmail.com> Hi, I'm trying to setup varnish in a fairly simple situation - I have a box with 15G of RAM and 15G of disk space. I want Varnish to use the lot. I've got 4 backend webservers, and I want varnish to test them all and ensure that they return a 200 code. However, i'm really struggling with the VCL examples - is there a reference that a non-programmer such as I can use? I'm figuring that I need something like # web1 backend default { .host = "xxx.xxx"; .port = "80"; .probe = { .url = "/"; .timeout = 34 ms; .interval = 1s; .window = 10; .threshold = 8; } } # web2 backend default { .host = "xxx.xxx"; .port = "80"; .probe = { .url = "/"; .timeout = 34 ms; .interval = 1s; .window = 10; .threshold = 8; } } And then something to glue all the backends together, but I see many different options in the manuals! I'd appreciate a pointer in the right direction if anyone could spare one. The vcl man page isnt entirely obvious to me! Many thanks, Alex -- Alex Davies This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately by e-mail and delete this e-mail permanently. From perbu at linpro.no Thu Dec 11 18:03:09 2008 From: perbu at linpro.no (Per Buer) Date: Thu, 11 Dec 2008 19:03:09 +0100 Subject: Example VCL for backend check In-Reply-To: <5fb622120812110921x2833df5u38a5c89105fb79b9@mail.gmail.com> References: <5fb622120812110921x2833df5u38a5c89105fb79b9@mail.gmail.com> Message-ID: <494155DD.3000702@linpro.no> Hi Alex, Alex Davies wrote: > I'm figuring that I need something like > > # web1 > backend default { backends must have uniqe names so you might use: backend web1 { > .host = "xxx.xxx"; > .port = "80"; > .probe = { > .url = "/"; > .timeout = 34 ms; > .interval = 1s; > .window = 10; > .threshold = 8; > } > } > > # web2 > backend default { backend web2{ > .host = "xxx.xxx"; > .port = "80"; > .probe = { > .url = "/"; > .timeout = 34 ms; > .interval = 1s; > .window = 10; > .threshold = 8; > } > } > > And then something to glue all the backends together, but I see many > different options in the manuals! I'd appreciate a pointer in the > right direction if anyone could spare one. The vcl man page isnt > entirely obvious to me! Use a director, like this: director www-director round-robin { { .backend = web1; .weight = 100; } { .backend = web2; .weight = 200; } } -- Per Buer - Leder Infrastruktur og Drift - Redpill Linpro Telefon: 21 54 41 21 - Mobil: 958 39 117 http://linpro.no/ | http://redpill.se/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 260 bytes Desc: OpenPGP digital signature URL: From alecshenry at gmail.com Thu Dec 11 21:29:13 2008 From: alecshenry at gmail.com (Alecs Henry) Date: Thu, 11 Dec 2008 19:29:13 -0200 Subject: Varnishncsa Message-ID: <3c54843f0812111329r79c0c267n94b051ebf1ac9c10@mail.gmail.com> Hi there! Is there any way to tweak the varnishncsa log? For instance, if I wanted to log to a file a specific header set by varnish itself. Or a hedaer sent by the backend. Thanks! Alecs -------------- next part -------------- An HTML attachment was scrubbed... URL: From des at des.no Thu Dec 11 22:34:08 2008 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Thu, 11 Dec 2008 23:34:08 +0100 Subject: Varnishncsa In-Reply-To: <3c54843f0812111329r79c0c267n94b051ebf1ac9c10@mail.gmail.com> (Alecs Henry's message of "Thu, 11 Dec 2008 19:29:13 -0200") References: <3c54843f0812111329r79c0c267n94b051ebf1ac9c10@mail.gmail.com> Message-ID: <863agus5nj.fsf@ds4.des.no> "Alecs Henry" writes: > Is there any way to tweak the varnishncsa log? Not currently. Patches are welcome. DES -- Dag-Erling Sm?rgrav - des at des.no From petter at linpro.no Fri Dec 12 12:42:30 2008 From: petter at linpro.no (Petter Knudsen) Date: Fri, 12 Dec 2008 13:42:30 +0100 Subject: Web GUI for varnish Message-ID: <49425C36.2020404@linpro.no> As an early christmas gift, here is a snapshot of the web GUI being developed for the 2.1 release: http://varnish.projects.linpro.no/wiki/WebGui It is not 100% finished (saving state and security being the major issues and the code needs some cleaning), but I hope people can test and play with it and give some input to the final version due in february. You MUST read the README file before using it, as it will describe it all. WARNING: This is, as mentioned, not finished, so don't use it on anything critical, as the VCL and parameters might be overwritten without warning. Petter, ready for the holidays From alecshenry at gmail.com Fri Dec 12 15:53:21 2008 From: alecshenry at gmail.com (Alecs Henry) Date: Fri, 12 Dec 2008 13:53:21 -0200 Subject: Varnishncsa In-Reply-To: <863agus5nj.fsf@ds4.des.no> References: <3c54843f0812111329r79c0c267n94b051ebf1ac9c10@mail.gmail.com> <863agus5nj.fsf@ds4.des.no> Message-ID: <3c54843f0812120753t5f61e31eh8caedceb3b4cb7db@mail.gmail.com> I wish I was a developer.. That would make my life so much easier! What about cookies? Can varnish log cookie values? Or does it fit in the same problem as the first message? Thanks! Alecs On Thu, Dec 11, 2008 at 8:34 PM, Dag-Erling Sm?rgrav wrote: > "Alecs Henry" writes: > > Is there any way to tweak the varnishncsa log? > > Not currently. Patches are welcome. > > DES > -- > Dag-Erling Sm?rgrav - des at des.no > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alecshenry at gmail.com Fri Dec 12 20:53:52 2008 From: alecshenry at gmail.com (Alecs Henry) Date: Fri, 12 Dec 2008 18:53:52 -0200 Subject: What to do when the backend sends no caching information? Message-ID: <3c54843f0812121253o4a795932k2b150e3f4b8a5509@mail.gmail.com> Hi All, In testing varnish in different situations, I came accross a backend that sends no caching information in its responses, no Expires, cache-control or anything. Ok, I thought, I'll just set the TTL on that object (namely an html page) in VCL. Well, turns out varnish may be setting the ttl all right (I do it in vcl_fetch, but frankly there really is no way to see if it is set correctly or not! or is there?) but the page always gets a PASS, never a HIT. Age: is set to 0 by varnish. Thanks for all the help! Alecs -------------- next part -------------- An HTML attachment was scrubbed... URL: From alecshenry at gmail.com Fri Dec 12 21:05:36 2008 From: alecshenry at gmail.com (Alecs Henry) Date: Fri, 12 Dec 2008 19:05:36 -0200 Subject: Backend response Message-ID: <3c54843f0812121305v4bb629a1i52773549530d9ec4@mail.gmail.com> Hi! Is there a place where I can alter the backend response before it is inserted into the cache? Thanks! Alecs -------------- next part -------------- An HTML attachment was scrubbed... URL: From des at des.no Fri Dec 12 21:08:27 2008 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Fri, 12 Dec 2008 22:08:27 +0100 Subject: Varnishncsa In-Reply-To: <3c54843f0812120753t5f61e31eh8caedceb3b4cb7db@mail.gmail.com> (Alecs Henry's message of "Fri, 12 Dec 2008 13:53:21 -0200") References: <3c54843f0812111329r79c0c267n94b051ebf1ac9c10@mail.gmail.com> <863agus5nj.fsf@ds4.des.no> <3c54843f0812120753t5f61e31eh8caedceb3b4cb7db@mail.gmail.com> Message-ID: <86y6ylqeyc.fsf@ds4.des.no> "Alecs Henry" writes: > What about cookies? Can varnish log cookie values? Or does it fit in > the same problem as the first message? Varnish can and does log cookies. See the varnishlog(1) man page. DES -- Dag-Erling Sm?rgrav - des at des.no From perbu at linpro.no Fri Dec 12 21:20:36 2008 From: perbu at linpro.no (Per Buer) Date: Fri, 12 Dec 2008 22:20:36 +0100 Subject: Backend response In-Reply-To: <3c54843f0812121305v4bb629a1i52773549530d9ec4@mail.gmail.com> References: <3c54843f0812121305v4bb629a1i52773549530d9ec4@mail.gmail.com> Message-ID: <4942D5A4.5000608@linpro.no> Alecs Henry wrote: > Hi! > > Is there a place where I can alter the backend response before it is > inserted into the cache? You can remove, add and alter headers - but there isn't much you can do to the content. Except to ESI, process it, I guess. Per. -- http://linpro.no/ | http://redpill.se/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 260 bytes Desc: OpenPGP digital signature URL: From perbu at linpro.no Fri Dec 12 21:21:42 2008 From: perbu at linpro.no (Per Buer) Date: Fri, 12 Dec 2008 22:21:42 +0100 Subject: What to do when the backend sends no caching information? In-Reply-To: <3c54843f0812121253o4a795932k2b150e3f4b8a5509@mail.gmail.com> References: <3c54843f0812121253o4a795932k2b150e3f4b8a5509@mail.gmail.com> Message-ID: <4942D5E6.1060507@linpro.no> Hi, Alecs Henry wrote: > > Well, turns out varnish may be setting the ttl all right (I do it in > vcl_fetch, but frankly there really is no way to see if it is set > correctly or not! or is there?) but the page always gets a PASS, never a > HIT. Age: is set to 0 by varnish. The client probably sends a cookie. When varnish sees a cookie it does a pass. Please see the FAQ on notes on how to deal with this. Per. -- http://linpro.no/ | http://redpill.se/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 260 bytes Desc: OpenPGP digital signature URL: From alecshenry at gmail.com Fri Dec 12 21:28:42 2008 From: alecshenry at gmail.com (Alecs Henry) Date: Fri, 12 Dec 2008 19:28:42 -0200 Subject: Varnishncsa In-Reply-To: <86y6ylqeyc.fsf@ds4.des.no> References: <3c54843f0812111329r79c0c267n94b051ebf1ac9c10@mail.gmail.com> <863agus5nj.fsf@ds4.des.no> <3c54843f0812120753t5f61e31eh8caedceb3b4cb7db@mail.gmail.com> <86y6ylqeyc.fsf@ds4.des.no> Message-ID: <3c54843f0812121328u717d5b8fuf2f3a76698f02211@mail.gmail.com> Thanks Dag, I must have missed it! Alecs On Fri, Dec 12, 2008 at 7:08 PM, Dag-Erling Sm?rgrav wrote: > "Alecs Henry" writes: > > What about cookies? Can varnish log cookie values? Or does it fit in > > the same problem as the first message? > > Varnish can and does log cookies. See the varnishlog(1) man page. > > DES > -- > Dag-Erling Sm?rgrav - des at des.no > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alecshenry at gmail.com Fri Dec 12 21:30:33 2008 From: alecshenry at gmail.com (Alecs Henry) Date: Fri, 12 Dec 2008 19:30:33 -0200 Subject: Backend response In-Reply-To: <4942D5A4.5000608@linpro.no> References: <3c54843f0812121305v4bb629a1i52773549530d9ec4@mail.gmail.com> <4942D5A4.5000608@linpro.no> Message-ID: <3c54843f0812121330u8351b1dn226d53dbeecc7616@mail.gmail.com> Hi Per, I'm looking into changing the headers all right! Where do I do that in VCL? I wanted to try to set some cache control headers before the object was put into varnish cache. Does that even make sense? Thanks! Alecs On Fri, Dec 12, 2008 at 7:20 PM, Per Buer wrote: > Alecs Henry wrote: > > Hi! > > > > Is there a place where I can alter the backend response before it is > > inserted into the cache? > > You can remove, add and alter headers - but there isn't much you can do > to the content. Except to ESI, process it, I guess. > > Per. > > -- > http://linpro.no/ | http://redpill.se/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alecshenry at gmail.com Fri Dec 12 21:41:36 2008 From: alecshenry at gmail.com (Alecs Henry) Date: Fri, 12 Dec 2008 19:41:36 -0200 Subject: What to do when the backend sends no caching information? In-Reply-To: <4942D5E6.1060507@linpro.no> References: <3c54843f0812121253o4a795932k2b150e3f4b8a5509@mail.gmail.com> <4942D5E6.1060507@linpro.no> Message-ID: <3c54843f0812121341h61998679r6fc5f9564c8f4672@mail.gmail.com> Hi Per, I'm actually using the cookies in the cache (using the information provided on the website -- set req.hash += req.http.cookie;) and it works just fine! One thing that the backend does is vary on user-agent, not only content-encoding (I've removed the UA vary on the response, though). So looking into varnishlog I can see the pass, even after the hash is shown correctly. Any pointers? THanks! Alecs On Fri, Dec 12, 2008 at 7:21 PM, Per Buer wrote: > Hi, > > Alecs Henry wrote: > > > > Well, turns out varnish may be setting the ttl all right (I do it in > > vcl_fetch, but frankly there really is no way to see if it is set > > correctly or not! or is there?) but the page always gets a PASS, never a > > HIT. Age: is set to 0 by varnish. > > The client probably sends a cookie. When varnish sees a cookie it does a > pass. Please see the FAQ on notes on how to deal with this. > > Per. > > -- > http://linpro.no/ | http://redpill.se/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at linpro.no Fri Dec 12 21:41:55 2008 From: perbu at linpro.no (Per Buer) Date: Fri, 12 Dec 2008 22:41:55 +0100 Subject: Backend response In-Reply-To: <3c54843f0812121330u8351b1dn226d53dbeecc7616@mail.gmail.com> References: <3c54843f0812121305v4bb629a1i52773549530d9ec4@mail.gmail.com> <4942D5A4.5000608@linpro.no> <3c54843f0812121330u8351b1dn226d53dbeecc7616@mail.gmail.com> Message-ID: <4942DAA3.3010707@linpro.no> Hi Alecs, You can modify the object in vcl_fetch. See http://varnish.projects.linpro.no/wiki/VCLExampleDefault for a flow chart of VCL. Also read through "man vcl". You can alter the headers in vcl_fetch before doing "deliver" on them which will store the object and insert it into the cache. Yes, it makes sense. Per. Alecs Henry wrote: > Hi Per, > > I'm looking into changing the headers all right! Where do I do that in VCL? > I wanted to try to set some cache control headers before the object was > put into varnish cache. > > Does that even make sense? > > Thanks! > > Alecs > > On Fri, Dec 12, 2008 at 7:20 PM, Per Buer > wrote: > > Alecs Henry wrote: > > Hi! > > > > Is there a place where I can alter the backend response before it is > > inserted into the cache? > > You can remove, add and alter headers - but there isn't much you can do > to the content. Except to ESI, process it, I guess. > > Per. > > -- > http://linpro.no/ | http://redpill.se/ > > -- Per Buer - Leder Infrastruktur og Drift - Redpill Linpro Telefon: 21 54 41 21 - Mobil: 958 39 117 http://linpro.no/ | http://redpill.se/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 260 bytes Desc: OpenPGP digital signature URL: From perbu at linpro.no Fri Dec 12 21:52:11 2008 From: perbu at linpro.no (Per Buer) Date: Fri, 12 Dec 2008 22:52:11 +0100 Subject: What to do when the backend sends no caching information? In-Reply-To: <3c54843f0812121341h61998679r6fc5f9564c8f4672@mail.gmail.com> References: <3c54843f0812121253o4a795932k2b150e3f4b8a5509@mail.gmail.com> <4942D5E6.1060507@linpro.no> <3c54843f0812121341h61998679r6fc5f9564c8f4672@mail.gmail.com> Message-ID: <4942DD0B.7040908@linpro.no> Hi, If you have a look at the default VCL you'll see that varnish does a pass when the object is not cacheable (see man vcl again) or if it Sets a cookie. I think you have to override this manually, both in vcl_fetch and in vcl_recv. Per. Alecs Henry wrote: > Hi Per, > > I'm actually using the cookies in the cache (using the information > provided on the website -- set req.hash += req.http.cookie;) and it > works just fine! > One thing that the backend does is vary on user-agent, not only > content-encoding (I've removed the UA vary on the response, though). > > So looking into varnishlog I can see the pass, even after the hash is > shown correctly. > > Any pointers? > > THanks! > > Alecs > > On Fri, Dec 12, 2008 at 7:21 PM, Per Buer > wrote: > > Hi, > > Alecs Henry wrote: > > > > Well, turns out varnish may be setting the ttl all right (I do it in > > vcl_fetch, but frankly there really is no way to see if it is set > > correctly or not! or is there?) but the page always gets a PASS, > never a > > HIT. Age: is set to 0 by varnish. > > The client probably sends a cookie. When varnish sees a cookie it does a > pass. Please see the FAQ on notes on how to deal with this. > > Per. > > -- > http://linpro.no/ | http://redpill.se/ > > -- Per Buer - Leder Infrastruktur og Drift - Redpill Linpro Telefon: 21 54 41 21 - Mobil: 958 39 117 http://linpro.no/ | http://redpill.se/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 260 bytes Desc: OpenPGP digital signature URL: From karl at vomba.com Sun Dec 14 19:35:35 2008 From: karl at vomba.com (Karl Bernard) Date: Sun, 14 Dec 2008 14:35:35 -0500 Subject: Unexplainable 503 error Message-ID: <49456007.2060707@vomba.com> Hello All, I have been trying to upgrade my production instance of Varnish 1.1.2. to 2.01, but I have a problem that I can't figure out in the new version. When loading a web page with a lot of images, some of the images do not get displayed and instead I get a 503 Service Unavailable. When looking at my lighty logs, I don't see those requests at all. When looking at varnishlog, I can see "vcl_call error deliver". I'm not sure what to do to identify why those 503 are happening, they seem to be random, but only when I have a miss. I am not having this problem with 1.1.2. Does anyone have an idea what could be happening? Here's an excerpt from varnishlog: 9 ReqStart c 70.83.84.29 4747 406814454 9 RxRequest c GET 9 RxURL c /starpreviews/118x118/0/0/974.jpg 9 RxProtocol c HTTP/1.1 9 RxHeader c Host: img1.sxx.com 9 RxHeader c User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.4) Gecko/2008102920 Firefox/3.0.4 9 RxHeader c Accept: image/png,image/*;q=0.8,*/*;q=0.5 9 RxHeader c Accept-Language: en-us,en;q=0.5 9 RxHeader c Accept-Encoding: gzip,deflate 9 RxHeader c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 9 RxHeader c Keep-Alive: 300 9 RxHeader c Connection: keep-alive 9 RxHeader c Referer: http://www.sxx.com/stars/index/sex/female/list/E.html 9 RxHeader c Cookie: __utma=57286504.1072416550.1198780078.1229275966.1229278040.29; __utmz=57286504.1221787148.9.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmb=57286504.5.10.1229278040; __utmc=57286504 9 VCL_call c recv lookup 9 VCL_call c hash hash 9 VCL_call c miss fetch 9 Backend c 32 default default * 9 VCL_call c error deliver* 9 Length c 452 9 VCL_call c deliver deliver 9 TxProtocol c HTTP/1.1 9 TxStatus c 503 9 TxResponse c Service Unavailable 9 TxHeader c Server: Varnish 9 TxHeader c Retry-After: 0 9 TxHeader c Content-Type: text/html; charset=utf-8 9 TxHeader c Content-Length: 452 9 TxHeader c Date: Sun, 14 Dec 2008 18:15:22 GMT 9 TxHeader c X-Varnish: 406814454 9 TxHeader c Age: 2 9 TxHeader c Via: 1.1 varnish 9 TxHeader c Connection: close 9 ReqEnd c 406814454 1229278520.274427891 1229278522.459613085 0.000026941 2.185151100 0.000034094 9 SessionClose c error 9 StatSess c 70.83.84.29 4747 9 1 4 0 0 3 1251 15914 My configuration: backend default { .host = "66.152.93.60"; .port = "8082"; } sub vcl_recv { #We change the url: if (req.http.host ~ "^(img0|img1|img2|img3|img).") { set req.http.host = "imgtest.sxx.com"; } # remove useless headers unset req.http.Cookie; unset req.http.Authenticate; unset req.http.Accept; unset req.http.Accept-Language; unset req.http.Accept-Encoding; unset req.http.Accept-Charset; unset req.http.Referer; } My varnishstat: http://pastebin.com/f6e5d3b97 Karl -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex at davz.net Mon Dec 15 20:21:47 2008 From: alex at davz.net (Alex Davies) Date: Mon, 15 Dec 2008 20:21:47 +0000 Subject: Example VCL for backend check In-Reply-To: <494155DD.3000702@linpro.no> References: <5fb622120812110921x2833df5u38a5c89105fb79b9@mail.gmail.com> <494155DD.3000702@linpro.no> Message-ID: <5fb622120812151221jc8949e0l8bae4be022f7e326@mail.gmail.com> Hi Per, Thank-you for your help? With a minor change, your configuration worked extremely well - is there a wiki for Varnish that I can post it? I've written out a full, working, configuration for a two-host director below which might be of use in the VCL examples page. Another question - is there an easy way to monitor the average pageload time that varnish experiences when contacting the backends? Many thanks for your help, Alex -- backend wwxx { .host = "xx.xx.xx.xx"; .port = "80"; .probe = { .url = "/"; .timeout = 34 ms; .interval = 1s; .window = 10; .threshold = 8; } } backend wwxx { .host = "xx.xx.xx.xx"; .port = "80"; .probe = { .url = "/"; .timeout = 34 ms; .interval = 1s; .window = 10; .threshold = 8; } } director director random { { .backend = wwxx; .weight = 100; } { .backend = wwxx; .weight = 100; } } sub vcl_recv { if (req.http.host ~ "^(www.)?mysite.com$") { set req.backend = director; } } On Thu, Dec 11, 2008 at 6:03 PM, Per Buer wrote: > Hi Alex, > > Alex Davies wrote: > >> I'm figuring that I need something like >> >> # web1 >> backend default { > > backends must have uniqe names so you might use: > > backend web1 { >> .host = "xxx.xxx"; >> .port = "80"; >> .probe = { >> .url = "/"; >> .timeout = 34 ms; >> .interval = 1s; >> .window = 10; >> .threshold = 8; >> } >> } >> >> # web2 >> backend default { > > backend web2{ > >> .host = "xxx.xxx"; >> .port = "80"; >> .probe = { >> .url = "/"; >> .timeout = 34 ms; >> .interval = 1s; >> .window = 10; >> .threshold = 8; >> } >> } >> >> And then something to glue all the backends together, but I see many >> different options in the manuals! I'd appreciate a pointer in the >> right direction if anyone could spare one. The vcl man page isnt >> entirely obvious to me! > > Use a director, like this: > > director www-director round-robin { > { > .backend = web1; > .weight = 100; > } > { > .backend = web2; > .weight = 200; > } > } > > > -- > Per Buer - Leder Infrastruktur og Drift - Redpill Linpro > Telefon: 21 54 41 21 - Mobil: 958 39 117 > http://linpro.no/ | http://redpill.se/ > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc > > -- Alex Davies This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately by e-mail and delete this e-mail permanently. -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at linpro.no Mon Dec 15 20:42:37 2008 From: perbu at linpro.no (Per Buer) Date: Mon, 15 Dec 2008 21:42:37 +0100 Subject: Example VCL for backend check In-Reply-To: <5fb622120812151221jc8949e0l8bae4be022f7e326@mail.gmail.com> References: <5fb622120812110921x2833df5u38a5c89105fb79b9@mail.gmail.com> <494155DD.3000702@linpro.no> <5fb622120812151221jc8949e0l8bae4be022f7e326@mail.gmail.com> Message-ID: <4946C13D.7030802@linpro.no> Hi Alex. Good to hear you got it working.:-) I written a page about load balancing a few days ago and I've added your vcl_recv there. It's located at http://varnish.projects.linpro.no/wiki/LoadBalancing - linked from http://varnish.projects.linpro.no/wiki/VarnishFeatures - it might need some proofreading if anyone got some spare time. I'm not aware of any way to monitor pageload time from varnish. I guess the information is available in varnish so it might not be to hard to expose this in VCL. At the moment its not on the list, however. Per. Alex Davies wrote: > Hi Per, > > Thank-you for your help? With a minor change, your configuration worked > extremely well - is there a wiki for Varnish that I can post it? I've > written out a full, working, configuration for a two-host director below > which might be of use in the VCL examples page. > > Another question - is there an easy way to monitor the average pageload > time that varnish experiences when contacting the backends? > > Many thanks for your help, > > Alex > > -- > > backend wwxx { > .host = "xx.xx.xx.xx"; > .port = "80"; > .probe = { > .url = "/"; > .timeout = 34 ms; > .interval = 1s; > .window = 10; > .threshold = 8; > } > } > > backend wwxx { > .host = "xx.xx.xx.xx"; > .port = "80"; > .probe = { > .url = "/"; > .timeout = 34 ms; > .interval = 1s; > .window = 10; > .threshold = 8; > } > } > > director director random { > { > .backend = wwxx; > .weight = 100; > } > { > .backend = wwxx; > .weight = 100; > } > } > > sub vcl_recv { > if (req.http.host ~ "^(www.)?mysite.com $") { > set req.backend = director; > } > } > > > > On Thu, Dec 11, 2008 at 6:03 PM, Per Buer > wrote: >> Hi Alex, >> >> Alex Davies wrote: >> >>> I'm figuring that I need something like >>> >>> # web1 >>> backend default { >> >> backends must have uniqe names so you might use: >> >> backend web1 { >>> .host = "xxx.xxx"; >>> .port = "80"; >>> .probe = { >>> .url = "/"; >>> .timeout = 34 ms; >>> .interval = 1s; >>> .window = 10; >>> .threshold = 8; >>> } >>> } >>> >>> # web2 >>> backend default { >> >> backend web2{ >> >>> .host = "xxx.xxx"; >>> .port = "80"; >>> .probe = { >>> .url = "/"; >>> .timeout = 34 ms; >>> .interval = 1s; >>> .window = 10; >>> .threshold = 8; >>> } >>> } >>> >>> And then something to glue all the backends together, but I see many >>> different options in the manuals! I'd appreciate a pointer in the >>> right direction if anyone could spare one. The vcl man page isnt >>> entirely obvious to me! >> >> Use a director, like this: >> >> director www-director round-robin { >> { >> .backend = web1; >> .weight = 100; >> } >> { >> .backend = web2; >> .weight = 200; >> } >> } >> >> >> -- >> Per Buer - Leder Infrastruktur og Drift - Redpill Linpro >> Telefon: 21 54 41 21 - Mobil: 958 39 117 >> http://linpro.no/ | http://redpill.se/ >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at projects.linpro.no >> http://projects.linpro.no/mailman/listinfo/varnish-misc >> >> > > > > -- > Alex Davies > > This email and any files transmitted with it are confidential and > intended solely for the use of the individual or entity to whom they > are addressed. If you have received this email in error please notify > the sender immediately by e-mail and delete this e-mail permanently. > > > ------------------------------------------------------------------------ > > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc -- Per Buer - Leder Infrastruktur og Drift - Redpill Linpro Telefon: 21 54 41 21 - Mobil: 958 39 117 http://linpro.no/ | http://redpill.se/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 260 bytes Desc: OpenPGP digital signature URL: From lukas.loesche at scoyo.com Mon Dec 15 15:53:12 2008 From: lukas.loesche at scoyo.com (Loesche, Lukas, scoyo) Date: Mon, 15 Dec 2008 16:53:12 +0100 Subject: caching everything incl. 403? In-Reply-To: <49456007.2060707@vomba.com> Message-ID: Hi, our goal is to cache 'everything' for a while. Status 200 responses should be kept for a week, everything else for 5 minutes. However I had some problems figuring out what makes an object "cacheable" and what doesn't. Our vcl: ... ... sub vcl_fetch { /* Set NON-200 caching */ if (obj.status != 200) { set obj.ttl = 300s; } elseif (obj.cacheable) { /* Remove Expires from backend */ unset obj.http.expires; /* Set the clients TTL on this object */ set obj.http.cache-control = "max-age = 21600"; /* Set how long Varnish will keep it */ set obj.ttl = 604800s; /* marker for vcl_deliver to set Age: */ set obj.http.magicmarker = "1"; } } ... ... Now the obj.ttl=300s Part works for Status "404 not found" responses, however it doesn't for instance for "403 forbidden". I was wondering, why. Why does a 403 always hit the backend with aboves vcl? Also, what criteria has to be meet for if(obj.cacheable) to be true? Best regards, -- Lukas From tfheen at linpro.no Tue Dec 16 14:55:24 2008 From: tfheen at linpro.no (Tollef Fog Heen) Date: Tue, 16 Dec 2008 15:55:24 +0100 Subject: Unexplainable 503 error In-Reply-To: <49456007.2060707@vomba.com> (Karl Bernard's message of "Sun, 14 Dec 2008 14:35:35 -0500") References: <49456007.2060707@vomba.com> Message-ID: <87myew890j.fsf@qurzaw.linpro.no> ]] Karl Bernard | I have been trying to upgrade my production instance of Varnish | 1.1.2. to 2.01, but I have a problem that I can't figure out in the | new version. I recommend going to 2.0.2, not 2.0.1 | 9 VCL_call c miss fetch | 9 Backend c 32 default default | * 9 VCL_call c error deliver* This means it fails when fetching the object. I suspect this is a limitation in the number of connections to the backend or something like that. Maybe wireshark or tcpdump will be able to shed more light on what's going on here. -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 From tfheen at linpro.no Tue Dec 16 15:42:59 2008 From: tfheen at linpro.no (Tollef Fog Heen) Date: Tue, 16 Dec 2008 16:42:59 +0100 Subject: caching everything incl. 403? In-Reply-To: (Lukas Loesche's message of "Mon, 15 Dec 2008 16:53:12 +0100") References: Message-ID: <87iqpk86t8.fsf@qurzaw.linpro.no> ]] "Loesche, Lukas, scoyo" | Also, what criteria has to be meet for if(obj.cacheable) to be true? from bin/varnishd/rfc2616.c: switch (sp->obj->response) { case 200: /* OK */ case 203: /* Non-Authoritative Information */ case 300: /* Multiple Choices */ case 301: /* Moved Permanently */ case 302: /* Moved Temporarily */ case 410: /* Gone */ case 404: /* Not Found */ sp->obj->cacheable = 1; [...] default: sp->obj->cacheable = 0; In other words, by default it'll be marked as non-cacheable. You should be able to do something like: if (obj.status == 403) { set obj.cacheable = 1; } in your VCL. -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 From erik at okcupid.com Tue Dec 16 16:44:44 2008 From: erik at okcupid.com (Erik Steigler) Date: Tue, 16 Dec 2008 11:44:44 -0500 Subject: Persistent Storage Message-ID: <4947DAFC.5010104@okcupid.com> I was wondering if there's a time line for when persistent storage will be available. Right now I'm considering running squid with a disk based cache which will query the origin servers and having varnish in front of that since squid is unable to handle our load it seems. (And Varnish makes a whole lot more sense to me) Also on some test instances I get error messages such as these: Dec 2 08:37:00 ip-2 varnishd[76185]: Child (138) not responding to ping, killing it. Dec 2 08:37:00 ip-2 varnishd[76185]: child (8721) Started Dec 2 08:37:01 ip-2 varnishd[76185]: Child (8721) said Closed fds: 4 5 6 7 11 12 14 15 Dec 2 08:37:01 ip-2 varnishd[76185]: Child (8721) said Child starts Dec 2 08:37:01 ip-2 varnishd[76185]: Child (8721) said managed to mmap 10737418240 bytes of 10737418240 Dec 2 08:37:01 ip-2 varnishd[76185]: Child (8721) said Ready Dec 2 14:25:15 ip-2 varnishd[76185]: Child (8721) not responding to ping, killing it. Dec 2 09:25:17 ip-2 kernel: pid 8721 (varnishd), uid 65534: exited on signal 3 Dec 2 14:25:17 ip-2 varnishd[76185]: Child (8721) not responding to ping, killing it. Dec 2 14:25:17 ip-2 varnishd[76185]: child (26870) Started Dec 2 14:25:18 ip-2 varnishd[76185]: Child (26870) said Closed fds: 4 5 6 7 11 12 14 15 Dec 2 14:25:18 ip-2 varnishd[76185]: Child (26870) said Child starts Dec 2 14:25:18 ip-2 varnishd[76185]: Child (26870) said managed to mmap 10737418240 bytes of 10737418240 Dec 2 14:25:18 ip-2 varnishd[76185]: Child (26870) said Ready Is that indicative of a problem or should I just up the timeout value for pings? This is using either revision 3459 or 2.0.2 out of ports on FreeBSD 6.3. I'm going to be trying another test on FreeBSD 7.0 with either 2.0.2 or the latest checkout. Thanks From jfbustarret at wat.tv Tue Dec 16 16:54:20 2008 From: jfbustarret at wat.tv (BUSTARRET, Jean-francois) Date: Tue, 16 Dec 2008 17:54:20 +0100 Subject: Sharing VCL code ? Message-ID: <53C652A09719C54DA24741D0157CB26901124100@TFPRDEXS1.tf1.groupetf1.fr> We all have some nice VCL snippets. It would be nice to have some way of sharing VCL code (some kind of "Varnish exchange") ! Jean-Fran?ois Bustarret WAT - Responsable technique http://www.wat.tv -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex at davz.net Tue Dec 16 17:27:38 2008 From: alex at davz.net (Alex Davies) Date: Tue, 16 Dec 2008 17:27:38 +0000 Subject: varnishstat build error on RHEL5 Message-ID: <5fb622120812160927h2466ef12lf85974cc99bc2532@mail.gmail.com> Hi, I'm trying to build varnishstat on RHEL5. It was not installed by my standard ./configure;make;make install but I notice that there is a folder in "bin" called varnishstat. However, when I attempt to run "make", I get an error: [root at frontend varnishstat]# make clean rm -f varnishstat varnishstat rm -rf .libs _libs rm -f *.o rm -f *.lo [root at frontend varnishstat]# make gcc -DHAVE_CONFIG_H -I. -I../.. -I../../include -g -O2 -MT varnishstat.o -MD -MP -MF .deps/varnishstat.Tpo -c -o varnishstat.o varnishstat.c mv -f .deps/varnishstat.Tpo .deps/varnishstat.Po /bin/sh ../../libtool --tag=CC --mode=link gcc -g -O2 -o varnishstat varnishstat.o ../../lib/libvarnish/libvarnish.la ../../lib/libvarnishcompat/libvarnishcompat.la ../../lib/libvarnishapi/libvarnishapi.la -lrt -lpthread mkdir .libs gcc -g -O2 -o .libs/varnishstat varnishstat.o ../../lib/libvarnish/.libs/libvarnish.so ../../lib/libvarnishcompat/.libs/libvarnishcompat.so ../../lib/libvarnishapi/.libs/libvarnishapi.so -lrt -lpthread -Wl,--rpath -Wl,/usr/local/lib varnishstat.o: In function `do_curses': /usr/src/varnish-2.0.2/bin/varnishstat/varnishstat.c:111: undefined reference to `initscr' /usr/src/varnish-2.0.2/bin/varnishstat/varnishstat.c:112: undefined reference to `raw' /usr/src/varnish-2.0.2/bin/varnishstat/varnishstat.c:113: undefined reference to `noecho' /usr/src/varnish-2.0.2/bin/varnishstat/varnishstat.c:114: undefined reference to `nonl' /usr/src/varnish-2.0.2/bin/varnishstat/varnishstat.c:115: undefined reference to `stdscr' /usr/src/varnish-2.0.2/bin/varnishstat/varnishstat.c:115: undefined reference to `intrflush' /usr/src/varnish-2.0.2/bin/varnishstat/varnishstat.c:116: undefined reference to `curs_set' /usr/src/varnish-2.0.2/bin/varnishstat/varnishstat.c:117: undefined reference to `stdscr' /usr/src/varnish-2.0.2/bin/varnishstat/varnishstat.c:117: undefined reference to `werase' /usr/src/varnish-2.0.2/bin/varnishstat/varnishstat.c:128: undefined reference to `COLS' /usr/src/varnish-2.0.2/bin/varnishstat/varnishstat.c:128: undefined reference to `mvprintw' /usr/src/varnish-2.0.2/bin/varnishstat/varnishstat.c:129: undefined reference to `mvprintw' /usr/src/varnish-2.0.2/bin/varnishstat/varnishstat.c:142: undefined reference to `mvprintw' /usr/src/varnish-2.0.2/bin/varnishstat/varnishstat.c:143: undefined reference to `mvprintw' varnishstat.o: In function `do_curses': /usr/src/varnish-2.0.2/bin/varnishstat/../../include/stat_field.h:32: undefined reference to `LINES' /usr/src/varnish-2.0.2/bin/varnishstat/../../include/stat_field.h:33: undefined reference to `LINES' /usr/src/varnish-2.0.2/bin/varnishstat/../../include/stat_field.h:35: undefined reference to `LINES' /usr/src/varnish-2.0.2/bin/varnishstat/../../include/stat_field.h:36: undefined reference to `LINES' /usr/src/varnish-2.0.2/bin/varnishstat/../../include/stat_field.h:37: undefined reference to `LINES' varnishstat.o:/usr/src/varnish-2.0.2/bin/varnishstat/../../include/stat_field.h:39: more undefined references to `LINES' follow varnishstat.o: In function `do_curses': /usr/src/varnish-2.0.2/bin/varnishstat/varnishstat.c:161: undefined reference to `stdscr' /usr/src/varnish-2.0.2/bin/varnishstat/varnishstat.c:161: undefined reference to `wrefresh' /usr/src/varnish-2.0.2/bin/varnishstat/varnishstat.c:162: undefined reference to `stdscr' /usr/src/varnish-2.0.2/bin/varnishstat/varnishstat.c:162: undefined reference to `wtimeout' /usr/src/varnish-2.0.2/bin/varnishstat/varnishstat.c:163: undefined reference to `stdscr' /usr/src/varnish-2.0.2/bin/varnishstat/varnishstat.c:163: undefined reference to `wgetch' /usr/src/varnish-2.0.2/bin/varnishstat/varnishstat.c:200: undefined reference to `beep' varnishstat.o: In function `do_curses': /usr/src/varnish-2.0.2/bin/varnishstat/../../include/stat_field.h:32: undefined reference to `mvprintw' varnishstat.o: In function `do_curses': /usr/src/varnish-2.0.2/bin/varnishstat/varnishstat.c:185: undefined reference to `endwin' varnishstat.o: In function `do_curses': /usr/src/varnish-2.0.2/bin/varnishstat/../../include/stat_field.h:33: undefined reference to `mvprintw' /usr/src/varnish-2.0.2/bin/varnishstat/../../include/stat_field.h:35: undefined reference to `mvprintw' /usr/src/varnish-2.0.2/bin/varnishstat/../../include/stat_field.h:36: undefined reference to `mvprintw' /usr/src/varnish-2.0.2/bin/varnishstat/../../include/stat_field.h:101: undefined reference to `mvprintw' /usr/src/varnish-2.0.2/bin/varnishstat/../../include/stat_field.h:102: undefined reference to `mvprintw' varnishstat.o:/usr/src/varnish-2.0.2/bin/varnishstat/../../include/stat_field.h:104: more undefined references to `mvprintw' follow varnishstat.o: In function `do_curses': /usr/src/varnish-2.0.2/bin/varnishstat/varnishstat.c:173: undefined reference to `stdscr' /usr/src/varnish-2.0.2/bin/varnishstat/varnishstat.c:173: undefined reference to `wredrawln' /usr/src/varnish-2.0.2/bin/varnishstat/varnishstat.c:174: undefined reference to `stdscr' /usr/src/varnish-2.0.2/bin/varnishstat/varnishstat.c:174: undefined reference to `wrefresh' /usr/src/varnish-2.0.2/bin/varnishstat/varnishstat.c:168: undefined reference to `stdscr' /usr/src/varnish-2.0.2/bin/varnishstat/varnishstat.c:168: undefined reference to `werase' collect2: ld returned 1 exit status make: *** [varnishstat] Error 1 Could anyone be so kind as to advise me in the correct way to get varnishstat working? Many thanks, Alex -- Alex Davies This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately by e-mail and delete this e-mail permanently. From des at des.no Tue Dec 16 18:10:18 2008 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Tue, 16 Dec 2008 19:10:18 +0100 Subject: varnishstat build error on RHEL5 In-Reply-To: <5fb622120812160927h2466ef12lf85974cc99bc2532@mail.gmail.com> (Alex Davies's message of "Tue, 16 Dec 2008 17:27:38 +0000") References: <5fb622120812160927h2466ef12lf85974cc99bc2532@mail.gmail.com> Message-ID: <86prjs3sad.fsf@ds4.des.no> "Alex Davies" writes: > I'm trying to build varnishstat on RHEL5. It was not installed by my > standard ./configure;make;make install but I notice that there is a > folder in "bin" called varnishstat. You need to install ncurses-devel first. DES -- Dag-Erling Sm?rgrav - des at des.no From perbu at linpro.no Tue Dec 16 19:21:30 2008 From: perbu at linpro.no (Per Buer) Date: Tue, 16 Dec 2008 20:21:30 +0100 Subject: Sharing VCL code ? In-Reply-To: <53C652A09719C54DA24741D0157CB26901124100@TFPRDEXS1.tf1.groupetf1.fr> References: <53C652A09719C54DA24741D0157CB26901124100@TFPRDEXS1.tf1.groupetf1.fr> Message-ID: <4947FFBA.3020800@linpro.no> BUSTARRET, Jean-francois wrote: > We all have some nice VCL snippets. > > It would be nice to have some way of sharing VCL code (some kind of > "Varnish exchange") ! That might be a really good idea. I guess the Wiki would be a natural place to start this off - the page http://varnish.projects.linpro.no/wiki/VCLExamples does already contain some usefull examples. Could you add your snippets to that page? We'll get the page organized in a more sensible way as it grows. I've added the magic bit to your wiki user - so you're free to edit it. -- Per Buer - Leder Infrastruktur og Drift - Redpill Linpro Telefon: 21 54 41 21 - Mobil: 958 39 117 http://linpro.no/ | http://redpill.se/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 260 bytes Desc: OpenPGP digital signature URL: From alex at davz.net Tue Dec 16 22:30:50 2008 From: alex at davz.net (Alex Davies) Date: Tue, 16 Dec 2008 22:30:50 +0000 Subject: varnishstat build error on RHEL5 In-Reply-To: <86prjs3sad.fsf@ds4.des.no> References: <5fb622120812160927h2466ef12lf85974cc99bc2532@mail.gmail.com> <86prjs3sad.fsf@ds4.des.no> Message-ID: <5fb622120812161430j2bfb344ase209e33abf8abdc4@mail.gmail.com> Hi Des, Thank-you for your suggestion. Unfortunately Package ncurses-devel - 5.5-24.20060715.i386 is already installed. Package ncurses-devel - 5.5-24.20060715.x86_64 is already installed. ? Many thanks, Alex On Tue, Dec 16, 2008 at 6:10 PM, Dag-Erling Sm?rgrav wrote: > "Alex Davies" writes: > > I'm trying to build varnishstat on RHEL5. It was not installed by my > > standard ./configure;make;make install but I notice that there is a > > folder in "bin" called varnishstat. > > You need to install ncurses-devel first. > > DES > -- > Dag-Erling Sm?rgrav - des at des.no > -- Alex Davies This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately by e-mail and delete this e-mail permanently. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex at davz.net Tue Dec 16 23:07:04 2008 From: alex at davz.net (Alex Davies) Date: Tue, 16 Dec 2008 23:07:04 +0000 Subject: Locating source of errors to clients Message-ID: <5fb622120812161507k7c4ee1acg35770392bd6f4ced@mail.gmail.com> Hi All, I've setup a custom "sub vcl_error {" in my vcl file, and 1-2% of all requests are getting this error page for reasons that I dont entirely understand. Running varnishlog -i Error and varnishlog -i HttpError returns nothing even when visitors are getting this error. Am I using the wrong tag? Unfortunately, this is a busy site (thousands of req a second) so varnishlog on its own isnt much use to me! What should I put inside sub vcl_error to print out a more interesting error detail that might help me find the problem? Is there any way to use the request ID to look at logs back in time, without running varnishlog and piping the log output to disk constantly? Many thanks, Alex -- Alex Davies This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately by e-mail and delete this e-mail permanently. -------------- next part -------------- An HTML attachment was scrubbed... URL: From des at des.no Wed Dec 17 10:21:36 2008 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Wed, 17 Dec 2008 11:21:36 +0100 Subject: varnishstat build error on RHEL5 In-Reply-To: <5fb622120812161430j2bfb344ase209e33abf8abdc4@mail.gmail.com> (Alex Davies's message of "Tue, 16 Dec 2008 22:30:50 +0000") References: <5fb622120812160927h2466ef12lf85974cc99bc2532@mail.gmail.com> <86prjs3sad.fsf@ds4.des.no> <5fb622120812161430j2bfb344ase209e33abf8abdc4@mail.gmail.com> Message-ID: <86iqpjgkzz.fsf@ds4.des.no> "Alex Davies" writes: > Dag-Erling Sm?rgrav writes: > > You need to install ncurses-devel first. > Package ncurses-devel - 5.5-24.20060715.i386 is already installed. > Package ncurses-devel - 5.5-24.20060715.x86_64 is already installed. Please don't top-post. Check config.log to see why configure decided that you didn't have curses. DES -- Dag-Erling Sm?rgrav - des at des.no From jayk at ion0.com Wed Dec 17 17:24:21 2008 From: jayk at ion0.com (Jason Kuri) Date: Wed, 17 Dec 2008 10:24:21 -0700 Subject: Articles regarding Varnish, VCL and ESI Message-ID: <02C72C68-941A-4834-9A8C-519FE1A3D9DA@ion0.com> Hi All, Been lurking on the list for awhile, I've been playing with Varnish for some time also. I'm very involved with the Catalyst community and recently wrote a pair of articles that might be of interest to the folks on the varnish list so I wanted to send them on. Both are oriented towards Catalyst developers, but they also contain a lot of info that is not specific to Catalyst. They are: "Making Catalyst Sites Shine with Varnish" - includes an in-depth exploration of VCL http://www.catalystframework.org/calendar/2008/14 and "Site Building Hotness with Edge Side Includes" - which covers the benefits of using ESI and includes my JS based ESI parser that is useful for developing ESI sites when you aren't behind Varnish. http://www.catalystframework.org/calendar/2008/17 Hope they are of use, Jay From darryl.dixon at winterhouseconsulting.com Wed Dec 17 21:09:34 2008 From: darryl.dixon at winterhouseconsulting.com (Darryl Dixon - Winterhouse Consulting) Date: Thu, 18 Dec 2008 10:09:34 +1300 (NZDT) Subject: Articles regarding Varnish, VCL and ESI In-Reply-To: <02C72C68-941A-4834-9A8C-519FE1A3D9DA@ion0.com> References: <02C72C68-941A-4834-9A8C-519FE1A3D9DA@ion0.com> Message-ID: <58432.58.28.153.120.1229548174.squirrel@services.directender.co.nz> Hi Jason, [...snip...] > "Making Catalyst Sites Shine with Varnish" - includes an in-depth > exploration of VCL > > http://www.catalystframework.org/calendar/2008/14 This is a great introduction to VCL - I'm quite certain I'll be directing newbies to it in the future for a gentle introduction. > "Site Building Hotness with Edge Side Includes" - which covers the > benefits of using ESI and includes my JS based ESI parser that is > useful for developing ESI sites when you aren't behind Varnish. > > http://www.catalystframework.org/calendar/2008/17 I am especially excited about the JS ESI parser - the concept is an excellent one and it directly addresses an otherwise difficult problem - kudos to you, I'll be giving it a spin shortly. This whole second article in particular is excellent and very timely for me, as I have been thinking about using Varnish's ESI support for a new project I am working on. Currently my strategy for low-overhead dynamic pages revolves around just-in-time ajax. I am definitely inspired now to give it a crack in ESI instead. Thank you very much for this indeed. regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com From jayk at ion0.com Thu Dec 18 01:41:56 2008 From: jayk at ion0.com (Jason Kuri) Date: Wed, 17 Dec 2008 18:41:56 -0700 Subject: Articles regarding Varnish, VCL and ESI In-Reply-To: <58432.58.28.153.120.1229548174.squirrel@services.directender.co.nz> References: <02C72C68-941A-4834-9A8C-519FE1A3D9DA@ion0.com> <58432.58.28.153.120.1229548174.squirrel@services.directender.co.nz> Message-ID: <6926ED5C-D438-42BD-A21F-78A9C21555F5@ion0.com> >> "Making Catalyst Sites Shine with Varnish" - includes an in-depth >> exploration of VCL >> http://www.catalystframework.org/calendar/2008/14 > > This is a great introduction to VCL - I'm quite certain I'll be > directing > newbies to it in the future for a gentle introduction. Thanks. Glad to hear it will be useful. >> "Site Building Hotness with Edge Side Includes" - which covers the >> benefits of using ESI and includes my JS based ESI parser that is >> useful for developing ESI sites when you aren't behind Varnish. >> http://www.catalystframework.org/calendar/2008/17 > > I am especially excited about the JS ESI parser - the concept is an > excellent one and it directly addresses an otherwise difficult > problem - > kudos to you, I'll be giving it a spin shortly. Thanks again. :-) it works very well for me. > This whole second article in particular is excellent and very timely > for > me, as I have been thinking about using Varnish's ESI support for a > new > project I am working on. Currently my strategy for low-overhead > dynamic > pages revolves around just-in-time ajax. I am definitely inspired > now to > give it a crack in ESI instead. *nod* One of the side advantages of using ESI is that your content blocks are already factored out and accessible directly, which makes it easy to load in ESI, and then reload dynamically using ajax later. > Thank you very much for this indeed. No worries. :-) Jay From Neil.Scholten at scoyo.com Sun Dec 21 13:18:56 2008 From: Neil.Scholten at scoyo.com (Neil.Scholten at scoyo.com) Date: Sun, 21 Dec 2008 14:18:56 +0100 Subject: AW: caching everything incl. 403? In-Reply-To: <87iqpk86t8.fsf@qurzaw.linpro.no> References: <87iqpk86t8.fsf@qurzaw.linpro.no> Message-ID: Hi, It doesn't work :/ sub vcl_fetch { /* Set NON-200 caching */ if (obj.status != 200) { set obj.cacheable = "1"; set obj.ttl = 300s; } [...] Error-Message: Expected ID got '"1"' (program line 290), at (/etc/varnish/default.vcl Line 20 Pos 37) set obj.cacheable = "1"; ------------------------------------###- VCL compilation failed Any idea? --neil -----Urspr?ngliche Nachricht----- Von: varnish-misc-bounces at projects.linpro.no [mailto:varnish-misc-bounces at projects.linpro.no] Im Auftrag von Tollef Fog Heen Gesendet: Dienstag, 16. Dezember 2008 16:43 An: varnish-misc at projects.linpro.no Betreff: Re: caching everything incl. 403? ]] "Loesche, Lukas, scoyo" | Also, what criteria has to be meet for if(obj.cacheable) to be true? from bin/varnishd/rfc2616.c: switch (sp->obj->response) { case 200: /* OK */ case 203: /* Non-Authoritative Information */ case 300: /* Multiple Choices */ case 301: /* Moved Permanently */ case 302: /* Moved Temporarily */ case 410: /* Gone */ case 404: /* Not Found */ sp->obj->cacheable = 1; [...] default: sp->obj->cacheable = 0; In other words, by default it'll be marked as non-cacheable. You should be able to do something like: if (obj.status == 403) { set obj.cacheable = 1; } in your VCL. -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 _______________________________________________ varnish-misc mailing list varnish-misc at projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc From ottolski at web.de Sun Dec 21 13:55:11 2008 From: ottolski at web.de (Sascha Ottolski) Date: Sun, 21 Dec 2008 14:55:11 +0100 Subject: next release? Message-ID: <200812211455.11650.ottolski@web.de> Hi folks, I'm curious when the next stable (minor) release is planned. I'm especially interested in "Date: 2008-11-14 01:19:33 +0100 (Fri, 14 Nov 2008) New Revision: 3390 Modified: trunk/varnish-cache/lib/libvarnish/binary_heap.c Log: Rework the binary heap, we use for expiry processing, to deal more gracefully with large number of objects." which seem to be in the trunk, but didn't made it into 2.0.2. If no release is planned, which trunk revision could be recommended for productive environment? Thanks, Sascha From ottolski at web.de Sun Dec 21 13:41:28 2008 From: ottolski at web.de (Sascha Ottolski) Date: Sun, 21 Dec 2008 14:41:28 +0100 Subject: AW: caching everything incl. 403? In-Reply-To: References: <87iqpk86t8.fsf@qurzaw.linpro.no> Message-ID: <200812211441.28344.ottolski@web.de> Am Sonntag 21 Dezember 2008 14:18:56 schrieb Neil.Scholten at scoyo.com: > Hi, > > It doesn't work :/ > > sub vcl_fetch { > ???????????????? > ????????/* Set NON-200 caching */ > ????????if (obj.status != 200) > ????????{ > ????????????????set obj.cacheable = "1"; > ????????????????set obj.ttl = 300s; > ????????} > ????????[...] > > Error-Message: > > Expected ID got '"1"' > (program line 290), at > (/etc/varnish/default.vcl Line 20 Pos 37) > ? ? ? ? ? ? ? ? set obj.cacheable = "1"; > ------------------------------------###- > > VCL compilation failed > > Any idea? you should probably omit the quotes? set obj.cacheable = 1; Cheers, Sascha From phk at phk.freebsd.dk Sun Dec 21 14:23:15 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sun, 21 Dec 2008 14:23:15 +0000 Subject: next release? In-Reply-To: Your message of "Sun, 21 Dec 2008 14:55:11 +0100." <200812211455.11650.ottolski@web.de> Message-ID: <9661.1229869395@critter.freebsd.dk> In message <200812211455.11650.ottolski at web.de>, Sascha Ottolski writes: >I'm curious when the next stable (minor) release is planned. I'm >especially interested in I don't know if we have a plan as such, but I expect a minor release before I start integrating the persistent storage stuff. -trunk right now should be safe. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Sun Dec 21 14:29:38 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sun, 21 Dec 2008 14:29:38 +0000 Subject: AW: caching everything incl. 403? In-Reply-To: Your message of "Sun, 21 Dec 2008 14:41:28 +0100." <200812211441.28344.ottolski@web.de> Message-ID: <9729.1229869778@critter.freebsd.dk> In message <200812211441.28344.ottolski at web.de>, Sascha Ottolski writes: >Am Sonntag 21 Dezember 2008 14:18:56 schrieb Neil.Scholten at scoyo.com: >you should probably omit the quotes? > > set obj.cacheable = 1; set obj.cacheable = true; It's a boolean. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From plfgoa at gmail.com Tue Dec 23 11:48:27 2008 From: plfgoa at gmail.com (Paras Fadte) Date: Tue, 23 Dec 2008 17:18:27 +0530 Subject: Origin server aborts Message-ID: <75cf5800812230348p34e1c787qb2a8de51e4c39fec@mail.gmail.com> Hi, How do we minimize origin server aborts ? if a request is legitimate and if varnish refers the request back to a backend and if the backend or the origin server doesn't reply would that result in origin abort ? What could be the reason for such origin abort ? thank you. -Paras From plfgoa at gmail.com Wed Dec 24 08:22:19 2008 From: plfgoa at gmail.com (Paras Fadte) Date: Wed, 24 Dec 2008 13:52:19 +0530 Subject: Strange varnish behavior - not answering requests In-Reply-To: References: Message-ID: <75cf5800812240022jc8a0918ydcd3ba968db79558@mail.gmail.com> Hi all, I have also encountered similar issue 3-4 times but don't know the exact reason. Could this be useful http://varnish.projects.linpro.no/ticket/55 ? Thank you. -Paras On Wed, Dec 3, 2008 at 6:02 PM, Grasmo, Johan wrote: > Hi, > > We're tired of squid and it's quirks and want to advance into the information age with varnish. > > I've been testing varnish the last couple of days and it has been behaving perfectly until last night. When stresstesting the varnishserver it suddenly stopped handling requests. Varnish returns to normal operation after a restart. > > Symptom: > Accessing a page on the varnishsserver results in a timeout in the browser > > I am able to telnet to the varnishserver on port 80 but a GET doesn't result in anything ("hangs"). > I am able to telnet to port 6082 and run commands: > [snip] > status > 200 22 > Child in state running > [/snip] > > Varnishlog returns: > [snip] > 0 CLI - Rd ping > 0 CLI - Wr 0 200 PONG 1228307253 1.0 > 0 CLI - Rd ping > 0 CLI - Wr 0 200 PONG 1228307256 1.0 > 0 CLI - Rd ping > [/snip] > > Varnishncsa doesn't return anything. > > I've verified with tcpdump that varnish ack's requests. > > So "everything seems normal" even though clients get timeout. FF returned "Connection Interrupted The document contains no data". If you need any logs/configs please let me know :) > > I haven't exhausted the search function for the mailinglist so sorry if this topic has been an issue before. > > I'm running varnish on an Ubuntu 8.04 distro. > > > Cheers and thanks for replies, > > Johan Grasmo > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc > From plfgoa at gmail.com Wed Dec 24 09:22:29 2008 From: plfgoa at gmail.com (Paras Fadte) Date: Wed, 24 Dec 2008 14:52:29 +0530 Subject: varnish2.0.2 on Suse 10.3 In-Reply-To: <4933D9B5.6010709@linpro.no> References: <75cf5800811200134y7cbcc0bew2dd2f113ce710309@mail.gmail.com> <86db848d0811200242j5543663yc9fc09857aae2a81@mail.gmail.com> <75cf5800811200746x74e6ba8fyf9711e1680e28d1f@mail.gmail.com> <878wre9uf1.fsf@qurzaw.linpro.no> <75cf5800811232042j77251155s452df10e499f8dc8@mail.gmail.com> <75cf5800811262158n7819ac40m83f4ddea2d71093e@mail.gmail.com> <87wsek5hnl.fsf@qurzaw.linpro.no> <4933D9B5.6010709@linpro.no> Message-ID: <75cf5800812240122q879e20tcd331c6a790d0336@mail.gmail.com> Downloaded it from trunk and it seems to start without any issues now. On Mon, Dec 1, 2008 at 6:03 PM, Per Buer wrote: > Tollef Fog Heen skrev: >> ]] "Paras Fadte" >> >> | child (28673) Started >> | Pushing vcls failed: Internal error: No VCL_conf symbol >> >> It seems to fail to load the configuration, probably because the .so >> with the configuration ends up stripped. Why that happens, I have no >> idea. > > It seems to be a recurring issue on OpenSUSE. There have been earlier > reports on the same issue. > > -- > Per Buer - Leder Infrastruktur og Drift - Redpill Linpro > Telefon: 21 54 41 21 - Mobil: 958 39 117 > http://linpro.no/ | http://redpill.se/ > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc > > From andrew at scoop.co.nz Mon Dec 29 12:33:54 2008 From: andrew at scoop.co.nz (Andrew McNaughton) Date: Mon, 29 Dec 2008 23:33:54 +1100 Subject: Surrogate-Capability Header Message-ID: <4958C3B2.8090208@scoop.co.nz> I'm currently writing an extension to perl's HTML::Mason code to support ESI. I'd like to allow page construction to happen at either the origin server or the Surrogate without change to the code on the origin. In the Edge Architecture Spec (EAS), there's mention of a Surrogate-Capability Header which "allows surrogates to advertise their capabilities with capability tokens". I'd like to use this, or something like this, to trigger the code on the origin server to decide whether to issue esi:include tags and have the pages constructed at the edge, or to construct the pages fully at the origin. vcl can do what I need, but I wonder what people think would be the best way to advertise the ESI capabilty, and whether there's any interest in doing this in varnish itself. Even just a Varnish version number in the request headers would be useful and would allow my code to require that little bit less external configuration for it to work. Advertising "ESI/1.0" in the Surrogate-Capability Header is problematic in that only part of that spec is supported. Perhaps Varnish should use it's own token here, or perhaps it should just issue a separate header saying that varnish is running and what version. Any thoughts? Andrew McNaughton // From plfgoa at gmail.com Tue Dec 30 05:52:36 2008 From: plfgoa at gmail.com (Paras Fadte) Date: Tue, 30 Dec 2008 11:22:36 +0530 Subject: Varnish hangs . Message-ID: <75cf5800812292152x794e8d34w2350ba545ad3a1fa@mail.gmail.com> Hi, I have encountered a strange problem with varnish wherein it just hangs and stops responding to http requests even though it is in running state . The backends run without issue , one can telnet to varnish console also . What could be the issue ? I had downloaded varnish from trunk also to check if this issue goes but it tends to reoccur . I get following messages in varnishlog . 0 VCL_return - discard 0 ExpKill - 958922603 -30 0 ExpPick - 958922604 ttl 0 VCL_call - timeout 0 VCL_return - discard 0 ExpKill - 958922604 -30 0 ExpPick - 958922605 ttl 0 VCL_call - timeout 0 VCL_return - discard 0 ExpKill - 958922605 -30 0 ExpPick - 958930071 prefetch 0 VCL_call - prefetch 0 VCL_return - fetch 0 Debug - "Attempt Prefetch 958930071" 0 ExpPick - 958922606 ttl 0 VCL_call - timeout 0 VCL_return - discard 0 ExpKill - 958922606 -30 0 ExpPick - 958930072 prefetch 0 VCL_call - prefetch 0 VCL_return - fetch 0 Debug - "Attempt Prefetch 958930072" 0 ExpPick - 959145014 ttl 0 VCL_call - timeout 0 VCL_return - discard Thanks in advance. -Paras From balghi at gmail.com Tue Dec 30 09:40:05 2008 From: balghi at gmail.com (Babblu Ghimire) Date: Tue, 30 Dec 2008 01:40:05 -0800 Subject: Any one can help me to run varnish on Linux (Redhat Enterprise)? Message-ID: <91720cac0812300140u30d09aeal7eb2ede784e5c0fe@mail.gmail.com> I had installed Varnish in My Redhat Enterprise Workstation, it's working But can't operate as proxy server. My network is 192.168.0.0/24 I issued such command from terminal: #varnishd -a "192.168.0.1"/8000 (avail varnish to my network) #varnishd -b "192.168.0.1"/3128 (pointed to my squid server) What mistake I make ? What change I have to made ? How can I define and load my own vcl ? Regards, Balram From des at des.no Tue Dec 30 11:29:19 2008 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Tue, 30 Dec 2008 12:29:19 +0100 Subject: Any one can help me to run varnish on Linux (Redhat Enterprise)? In-Reply-To: <91720cac0812300140u30d09aeal7eb2ede784e5c0fe@mail.gmail.com> (Babblu Ghimire's message of "Tue, 30 Dec 2008 01:40:05 -0800") References: <91720cac0812300140u30d09aeal7eb2ede784e5c0fe@mail.gmail.com> Message-ID: <86myed52b4.fsf@ds4.des.no> "Babblu Ghimire" writes: > I issued such command from terminal: > #varnishd -a "192.168.0.1"/8000 (avail varnish to my network) > #varnishd -b "192.168.0.1"/3128 (pointed to my squid server) Neither of these will work, because a) the address syntax is incorrect and b) you need to specify both a listen address and a backend address. The details are in the man page. DES -- Dag-Erling Sm?rgrav - des at des.no From svein-listmail at stillbilde.net Wed Dec 31 11:48:45 2008 From: svein-listmail at stillbilde.net (Svein Skogen (List Mail Account)) Date: Wed, 31 Dec 2008 12:48:45 +0100 Subject: Configuration problem, possibly a PEBCAK. Message-ID: <495B5C1D.5010302@stillbilde.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'm in the process of setting up a solution with Varnish running in the front-end VM of a VMWare ESX solution. The idea being to run a setup similar to this: HTTP--Pound\ ,---apache22 >-Varnish--< HTTPS-Pound/ `---IIS However, when requesting large files (50MB +) from the backends, the pound<-->varnish connection times out, since varnish insists on fetching the entire file before starting to send it to the pound frontend. Is there a way to configure varnish to start sending what it receives immediately, while still retaining a copy in the cache, or do I have to make a choice between caching, or forwarding? I have a similar problem with the gallery2 backend, which uses a continuous updating http stream for "progress-bars" when performing long operations (like rebuilding thumnails and resizes for 19K pictures!). However I suspect that this problem is better solved by NOT caching those pages (if I can find a pattern to exclude them, that is). Any help would be appreciated. And, btw, a wish for a happy new year to all list subscribers. //Svein -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAklbXB0ACgkQtVbTV+BEzaOL3ACfQ3FZh+Q4iwzdXnZMBBOiD6Kn kLQAoM9CLNcWsHK18VVr0KkUnS5lLuHH =nuMy -----END PGP SIGNATURE----- From ottolski at web.de Wed Dec 31 13:58:42 2008 From: ottolski at web.de (Sascha Ottolski) Date: Wed, 31 Dec 2008 14:58:42 +0100 Subject: "uptime" not available to nagios plugin? Message-ID: <200812311458.42349.ottolski@web.de> Hi, bug or feature? The nagios plugin seems not to be able to extract the uptime: img-proxy1:~# /usr/local/libexec/check_varnish -p uptime Unknown parameter 'uptime' VARNISH UNKNOWN: (null) (3)|uptime=3 but it probably should, shouldn't it? img-proxy1:~# varnishstat -l 2>&1|grep uptime uptime Child uptime Happens both with trunk and 2.02. Thanks and a good party tonight to everyone, Sascha