From ccripy at gmail.com Fri Jun 1 17:53:38 2007 From: ccripy at gmail.com (cripy) Date: Fri, 1 Jun 2007 13:53:38 -0400 Subject: Varnish Stats Message-ID: Hello all, I'm curious what kind of varnish stats everyone here has recorded and what the "maximum" uptime you have with varnish not restarting itself. I'm trying to track down a nasty memory leak I believe exists in varnish which you can see for yourself in "top". It appears varnishs memory just increases infinitely until it crashes and restarts itself. Under a good amount of traffic (20 Mbit/s out) I can't seem to get varnish to stay up more than 5 hours without the memory leak kicking its ass-- however, here are the stats from a varnish server that's only pushing out about 5-9Mbit/s. Please share your stats and information that relates to this possible leak that we have going so we can track this sucker down! 1d+13:18:50 Hitrate ratio: 10 14 14 Hitrate avg: 0.9915 0.9907 0.9907 190399 2.99 1.42 Client connections accepted 747515 5.99 5.56 Client requests received 4659 0.00 0.03 Cache hits 543 0.00 0.00 Cache hits for pass 692 0.00 0.01 Cache misses 1487211 10.98 11.07 Backend connections success 0 0.00 0.00 Backend connections failures 608752 1.00 4.53 Backend connections reuses 642171 2.00 4.78 Backend connections recycles 26 -1.00 0.00 Backend connections unused 1075 0.00 0.01 N struct srcaddr 17 1.00 0.00 N active struct srcaddr 59 0.00 0.00 N struct sess_mem 21 0.00 0.00 N struct sess 809 1.00 0.01 N struct object 10122 0.00 0.08 N struct objecthead 684 0.00 0.01 N struct smf 4 0.00 0.00 N small free smf 89 0.00 0.00 N large free smf 93 0.00 0.00 N struct vbe_conn 17 0.00 0.00 N worker threads 9483 0.00 0.07 N worker threads created 0 0.00 0.00 N worker threads not created 0 0.00 0.00 N worker threads limited 0 0.00 0.00 N queued work requests 9483 0.00 0.07 N overflowed work requests 0 0.00 0.00 N dropped work requests 0 0.00 0.00 N expired objects 0 0.00 0.00 N objects on deathrow 0 0.00 0.00 HTTP header overflows 0 0.00 0.00 Objects sent with sendfile 574340 2.00 4.28 Objects sent with write 190390 2.00 1.42 Total Sessions 747503 4.99 5.56 Total Requests 0 0.00 0.00 Total pipe 0 0.00 0.00 Total pass 743562 4.99 5.54 Total fetch 156546112 1072.94 1165.38 Total header bytes 244767426611 7896.83 1822135.24 Total body bytes 36295 0.00 0.27 Session Closed 4190 0.00 0.03 Session Pipeline 1322 0.00 0.01 Session Read Ahead 707433 4.99 5.27 Session herd 54319017 373.28 404.37 SHM records 1516417 16.97 11.29 SHM writes 7 0.00 0.00 SHM MTX contention -------------- next part -------------- An HTML attachment was scrubbed... URL: From nwmcsween at gmail.com Tue Jun 5 00:39:30 2007 From: nwmcsween at gmail.com (Nathan McSween) Date: Mon, 4 Jun 2007 17:39:30 -0700 Subject: Varnish the poor mans reverse proxy? Message-ID: <000301c7a709$fbaafda0$6600a8c0@octane> Is it possible to use varnish as a fake layer 7 load balancer by redirecting static content (e.g. .png, .jpg, etc) to a fast static server and dynamic to another? -------------- next part -------------- An HTML attachment was scrubbed... URL: From des at linpro.no Tue Jun 5 07:57:36 2007 From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Tue, 05 Jun 2007 09:57:36 +0200 Subject: Varnish the poor mans reverse proxy? In-Reply-To: <000301c7a709$fbaafda0$6600a8c0@octane> (Nathan McSween's message of "Mon\, 4 Jun 2007 17\:39\:30 -0700") References: <000301c7a709$fbaafda0$6600a8c0@octane> Message-ID: <87k5uiomtr.fsf@des.linpro.no> Nathan McSween writes: > Is it possible to use varnish as a fake layer 7 load balancer by redirecting > static content (e.g. .png, .jpg, etc) to a fast static server and dynamic to > another? Yes. I hope you meant to write "the poor man's load balancer", because there's nothing second-rate about Varnish's abilities as a reverse proxy... DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Tue Jun 5 07:58:10 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 05 Jun 2007 07:58:10 +0000 Subject: Varnish the poor mans reverse proxy? In-Reply-To: Your message of "Mon, 04 Jun 2007 17:39:30 MST." <000301c7a709$fbaafda0$6600a8c0@octane> Message-ID: <65685.1181030290@critter.freebsd.dk> In message <000301c7a709$fbaafda0$6600a8c0 at octane>, Nathan McSween writes: >Is it possible to use varnish as a fake layer 7 load balancer by redirecting >static content (e.g. .png, .jpg, etc) to a fast static server and dynamic to >another? Yes. Declare two backends in your VCL program and select the appropriate one in vcl_miss{} -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dikshie at gmail.com Tue Jun 5 10:12:46 2007 From: dikshie at gmail.com (dikshie) Date: Tue, 5 Jun 2007 19:12:46 +0900 Subject: multiple backend Message-ID: <910e60e80706050312u726f9a72i6f19e4b2fa05884a@mail.gmail.com> Hi, this is newbie question: in my university there are a lot of web server (especially department's and faculty's web server) say for example www.math.itb.ac.jp, www.physics.itb.ac.jp, etc. how to define backend servers for *.itb.ac.jp ? I'm thinking to use if statement url_request for every web server but it seems too long :-) thanks ! -- -dikshie- From phk at phk.freebsd.dk Tue Jun 5 10:25:56 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 05 Jun 2007 10:25:56 +0000 Subject: multiple backend In-Reply-To: Your message of "Tue, 05 Jun 2007 19:12:46 +0900." <910e60e80706050312u726f9a72i6f19e4b2fa05884a@mail.gmail.com> Message-ID: <66274.1181039156@critter.freebsd.dk> In message <910e60e80706050312u726f9a72i6f19e4b2fa05884a at mail.gmail.com>, diksh ie writes: >Hi, >this is newbie question: >in my university there are a lot of web server (especially >department's and faculty's web server) say for example >www.math.itb.ac.jp, www.physics.itb.ac.jp, etc. > >how to define backend servers for *.itb.ac.jp ? > >I'm thinking to use if statement url_request for every web server >but it seems too long :-) That is unfortunately the way you have to do it. I have been thinking of a way to allow dynamic backends, but it is not on the current plan for version2. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Tue Jun 5 10:26:59 2007 From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Tue, 05 Jun 2007 12:26:59 +0200 Subject: multiple backend In-Reply-To: <910e60e80706050312u726f9a72i6f19e4b2fa05884a@mail.gmail.com> (dikshie@gmail.com's message of "Tue\, 5 Jun 2007 19\:12\:46 +0900") References: <910e60e80706050312u726f9a72i6f19e4b2fa05884a@mail.gmail.com> Message-ID: <87fy56n1cc.fsf@des.linpro.no> dikshie writes: > how to define backend servers for *.itb.ac.jp ? > > I'm thinking to use if statement url_request for every web server > but it seems too long :-) There is currently no other way to do it. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From nwmcsween at gmail.com Wed Jun 6 05:19:54 2007 From: nwmcsween at gmail.com (Nathan McSween) Date: Tue, 5 Jun 2007 22:19:54 -0700 Subject: Varnish multiple backend configuration? Message-ID: <001201c7a7fa$51ca1810$6600a8c0@octane> Would this work as a simple load balancer configuration? backend apache { set backend.host = ""; set backend.port = "80"; } backend nginx { set backend.host = ""; set backend.port = "80"; } sub vcl_recv { if (req.request != "GET" && req.request != "HEAD") { pipe; } if (req.http.Expect) { pipe; } if (req.http.Authenticate || req.http.Cookie) { pass; } lookup; } sub vcl_pipe { pipe; } sub vcl_pipe { pipe; } sub vcl_pass { pass; } sub vcl_hash { hash; } sub vcl_hit { if (!obj.cacheable) { pass; } deliver; } sub vcl_miss { if (req.url ~ "/.(php|py)$") { set req.backend = apache; fetch; } else (req.url ~ "/.(*)$") { set req.backend = nginx; fetch; } } sub vcl_fetch { if (!obj.valid) { error; } if (!obj.cacheable) { pass; } if (resp.http.Set-Cookie) { pass; } insert; } sub vcl_timeout { discard; } From phk at phk.freebsd.dk Wed Jun 6 09:25:40 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 06 Jun 2007 09:25:40 +0000 Subject: Varnish multiple backend configuration? In-Reply-To: Your message of "Tue, 05 Jun 2007 22:19:54 MST." <001201c7a7fa$51ca1810$6600a8c0@octane> Message-ID: <72010.1181121940@critter.freebsd.dk> In message <001201c7a7fa$51ca1810$6600a8c0 at octane>, Nathan McSween writes: >Would this work as a simple load balancer configuration? I don't think so, you will risk multiple concurrent transactions getting mixed up about which backend they connect to or using cached backend connections going to different places. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Wed Jun 6 09:27:05 2007 From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Wed, 06 Jun 2007 11:27:05 +0200 Subject: Varnish multiple backend configuration? In-Reply-To: <001201c7a7fa$51ca1810$6600a8c0@octane> (Nathan McSween's message of "Tue\, 5 Jun 2007 22\:19\:54 -0700") References: <001201c7a7fa$51ca1810$6600a8c0@octane> Message-ID: <877iqhl9g6.fsf@des.linpro.no> Nathan McSween writes: > Would this work as a simple load balancer configuration? Yes, except > backend apache { > set backend.host = ""; set backend.port = "80"; > } > > backend nginx { > set backend.host = ""; set backend.port = "80"; > } I assume you removed the host names intentionally before posting > sub vcl_miss { > if (req.url ~ "/.(php|py)$") { > set req.backend = apache; > fetch; > } else (req.url ~ "/.(*)$") { > set req.backend = nginx; > fetch; > } > } "else" does not take a condition, did you mean "else if"? In any case, the second condition is not necessary, and neither are the two "fetch" statements. The following should do: sub vcl_miss { if (req.url ~ "/.(php|py)$") { set req.backend = apache; } else { set req.backend = nginx; } } DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From admin at adofms.com.au Fri Jun 8 03:00:44 2007 From: admin at adofms.com.au (ADOFMS Admin, SteveOC) Date: Fri, 08 Jun 2007 12:30:44 +0930 Subject: Passing through client IP address to server Message-ID: <4668C65C.1080400@adofms.com.au> Im running both Varnish and my LAMP application on the same machine, so all requests hitting the apache server now have an incoming IP address of 127.0.0.1. Is there something that I can do in VCL to set the client IP address that apache sees as being the real client IP address ? My PHP app has a requirement to record both client IP address, timestamp, and browser version for all accesses to the system, so I eventually need to make this happen. Thnks SteveOC Apologies if this topic has been covered before - Ive done a good search using google, and couldnt find an answer to this one .. thanks. From Kenneth.Rorvik at hio.no Fri Jun 8 05:38:01 2007 From: Kenneth.Rorvik at hio.no (=?ISO-8859-1?Q?Kenneth_R=F8rvik?=) Date: Fri, 08 Jun 2007 07:38:01 +0200 Subject: Passing through client IP address to server In-Reply-To: <4668C65C.1080400@adofms.com.au> References: <4668C65C.1080400@adofms.com.au> Message-ID: <4668EB39.2050606@hio.no> ADOFMS Admin, SteveOC wrote: > Im running both Varnish and my LAMP application on the same machine, so > all requests hitting the apache server now have an incoming IP address > of 127.0.0.1. > > Is there something that I can do in VCL to set the client IP address > that apache sees as being the real client IP address ? > > My PHP app has a requirement to record both client IP address, > timestamp, and browser version for all accesses to the system, so I > eventually need to make this happen. Take a look at the X-Forwarded-for header. -- Kenneth R?rvik, IT HiO Tlf 22 45 20 83 Kenneth.Rorvik at hio.no From admin at adofms.com.au Fri Jun 8 06:01:31 2007 From: admin at adofms.com.au (ADOFMS Admin, SteveOC) Date: Fri, 08 Jun 2007 15:31:31 +0930 Subject: Passing through client IP address to server In-Reply-To: <4668EB39.2050606@hio.no> References: <4668C65C.1080400@adofms.com.au> <4668EB39.2050606@hio.no> Message-ID: <4668F0BB.8090509@adofms.com.au> Kenneth R?rvik wrote: > ADOFMS Admin, SteveOC wrote: >> Im running both Varnish and my LAMP application on the same machine, so >> all requests hitting the apache server now have an incoming IP address >> of 127.0.0.1. >> >> Is there something that I can do in VCL to set the client IP address >> that apache sees as being the real client IP address ? >> My PHP app has a requirement to record both client IP address, >> timestamp, and browser version for all accesses to the system, so I >> eventually need to make this happen. > > Take a look at the X-Forwarded-for header. > Brilliant !! Got it now, didnt notice that being set before. but that will do the trick. There is also a HTTP_X_VIA being set sometimes, which reports the hostname of the client machine. This variable isnt set on every request though. There is also a HTTP_X_VARNISH variable set as well, which is just a number. Any idea what this one does ? From denis at startsiden.no Fri Jun 8 05:57:37 2007 From: denis at startsiden.no (=?utf-8?Q?Denis_Br=C3=A6khus?=) Date: Fri, 8 Jun 2007 07:57:37 +0200 (CEST) Subject: Passing through client IP address to server In-Reply-To: <24857753.159081181282229200.JavaMail.root@ms1.startsiden.no> Message-ID: <3232401.159101181282257770.JavaMail.root@ms1.startsiden.no> ----- ADOFMS Admin, SteveOC wrote: > There is also a HTTP_X_VARNISH variable set as well, which is just a > number. Any idea what this one does ? This number is a request number you can use for debugging. varnishlog outputs these identifiers and you can cross check with the number from the header. Regards -- Denis Braekhus - Teknisk Ansvarlig ABC Startsiden AS http://www.startsiden.no From gaute at pht.no Fri Jun 8 09:22:40 2007 From: gaute at pht.no (Gaute Amundsen) Date: Fri, 8 Jun 2007 11:22:40 +0200 Subject: varnishlog filtering on host Message-ID: <200706081122.40789.gaute@pht.no> Hi. Just a quick question before I dig into it myself. How would you call varnishlog to verify hits and misses when adding a new host to a varnish that allready has some traffic? I need to work a bit to clean up old sinns with unnessesary cookies, forms with GET methods, and the like... I am looking at piping varnishlog -o thru a script to filter on hostname, and then pick out the relevant fields. Or is this doable with the right regexp perhaps? Regards Gaute Amundsen From des at linpro.no Fri Jun 8 14:15:41 2007 From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Fri, 08 Jun 2007 16:15:41 +0200 Subject: varnishlog filtering on host In-Reply-To: <200706081122.40789.gaute@pht.no> (Gaute Amundsen's message of "Fri\, 8 Jun 2007 11\:22\:40 +0200") References: <200706081122.40789.gaute@pht.no> Message-ID: <871wgmjzw2.fsf@des.linpro.no> Gaute Amundsen writes: > How would you call varnishlog to verify hits and misses when adding a > new host to a varnish that allready has some traffic? Try 'varnishlog -c -o VCL_call hit' and 'varnishlog -c VCL_call miss'. Also, varnishhist will show you the relative proportions of hits and misses in real time (so will varnishstat, for that matter, but not as prettily) DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Fri Jun 8 14:11:40 2007 From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Fri, 08 Jun 2007 16:11:40 +0200 Subject: Passing through client IP address to server In-Reply-To: <4668F0BB.8090509@adofms.com.au> (ADOFMS Admin's message of "Fri\, 08 Jun 2007 15\:31\:31 +0930") References: <4668C65C.1080400@adofms.com.au> <4668EB39.2050606@hio.no> <4668F0BB.8090509@adofms.com.au> Message-ID: <87645yk02r.fsf@des.linpro.no> "ADOFMS Admin, SteveOC" writes: > There is also a HTTP_X_VIA being set sometimes, which reports the > hostname of the client machine. This variable isnt set on every request > though. X-Via most likely comes from a downstream cache, possibly a client-side Squid cache or something similar. > There is also a HTTP_X_VARNISH variable set as well, which is just a > number. Any idea what this one does ? It is a reference to the internal request ID, which is recorded in the log file. For a cache hit, X-Varnish will contain both the ID of the current request and the ID of the request that populated the cache. It makes debugging Varnish a *lot* easier. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Sun Jun 10 08:55:42 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sun, 10 Jun 2007 08:55:42 +0000 Subject: please test Varnish "Vary:" matching Message-ID: <14630.1181465742@critter.freebsd.dk> I have implemented a very efficient and simple "Vary:" processing in varnish(-trunk) now. The simplicity is based on a number of assumptions which I have very little supporting evidence for, apart from my informed guesses. Please read: http://varnish.projects.linpro.no/wiki/ArchitectureVary And report back whether it works for you or not... Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Sun Jun 10 12:07:30 2007 From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Sun, 10 Jun 2007 14:07:30 +0200 Subject: please test Varnish "Vary:" matching In-Reply-To: <14630.1181465742@critter.freebsd.dk> (Poul-Henning Kamp's message of "Sun\, 10 Jun 2007 08\:55\:42 +0000") References: <14630.1181465742@critter.freebsd.dk> Message-ID: <87tztg8131.fsf@des.linpro.no> Poul-Henning Kamp writes: > I have implemented a very efficient and simple "Vary:" processing > in varnish(-trunk) now. > [...] > And report back whether it works for you or not... Works for me for deflate. Very useful for tinderbox.des.no, as the typical compression ratio for tinderbox logs is 15:1. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Sun Jun 10 13:37:48 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sun, 10 Jun 2007 13:37:48 +0000 Subject: please test Varnish "Vary:" matching In-Reply-To: Your message of "Sun, 10 Jun 2007 14:07:30 +0200." <87tztg8131.fsf@des.linpro.no> Message-ID: <1293.1181482668@critter.freebsd.dk> In message <87tztg8131.fsf at des.linpro.no>, =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= writes: >Poul-Henning Kamp writes: >> I have implemented a very efficient and simple "Vary:" processing >> in varnish(-trunk) now. >> [...] >> And report back whether it works for you or not... > >Works for me for deflate. Very useful for tinderbox.des.no, as the >typical compression ratio for tinderbox logs is 15:1. Please pay attention to the increase in cache size, in particular if you Vary: on User-Agent: -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Sun Jun 10 13:46:54 2007 From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Sun, 10 Jun 2007 15:46:54 +0200 Subject: please test Varnish "Vary:" matching In-Reply-To: <1293.1181482668@critter.freebsd.dk> (Poul-Henning Kamp's message of "Sun\, 10 Jun 2007 13\:37\:48 +0000") References: <1293.1181482668@critter.freebsd.dk> Message-ID: <87k5ub9b1t.fsf@des.linpro.no> "Poul-Henning Kamp" writes: > Dag-Erling Sm?rgrav writes: > > Works for me for deflate. Very useful for tinderbox.des.no, as the > > typical compression ratio for tinderbox logs is 15:1. > Please pay attention to the increase in cache size, in particular > if you Vary: on User-Agent: It's a low-traffic site, so it's hard to measure... and I only Vary: on Accept-Encoding. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From phk at phk.freebsd.dk Sun Jun 10 13:51:15 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sun, 10 Jun 2007 13:51:15 +0000 Subject: please test Varnish "Vary:" matching In-Reply-To: Your message of "Sun, 10 Jun 2007 15:46:54 +0200." <87k5ub9b1t.fsf@des.linpro.no> Message-ID: <1591.1181483475@critter.freebsd.dk> In message <87k5ub9b1t.fsf at des.linpro.no>, =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= writes: >"Poul-Henning Kamp" writes: >> Dag-Erling Sm=C3=B8rgrav writes: >> > Works for me for deflate. Very useful for tinderbox.des.no, as the >> > typical compression ratio for tinderbox logs is 15:1. >> Please pay attention to the increase in cache size, in particular >> if you Vary: on User-Agent: > >It's a low-traffic site, so it's hard to measure... and I only Vary: on >Accept-Encoding. I guess we can pull some stats out of the shmlog by looking at XID numbers in hit or something... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From anup at iamcool.net Mon Jun 11 08:07:28 2007 From: anup at iamcool.net (Anup Shukla) Date: Mon, 11 Jun 2007 16:07:28 +0800 Subject: Newbie : Static content caching Message-ID: <466D02C0.8010701@iamcool.net> Hello all, I have been using Varnish with its "default configuration" for quite sometime now. Absolutely nothing has changed except for the backed servers port number (set to 8000). Needless to say that it has been performing brilliantly and has reduced the load on our webservers considerably. Just noticed something strange though. Before i explain further, please let me clarify that i am not an expert in caching by any standard, so this happens to be more of a newbie question. I am running varnish and the backend apache on the same server. The backend listens on the loopback address while varnish listens on the interface facing the load balancer. If i request, say for example, a static jpeg, i assume that the first request will be served by the webserver via varnish, and all later requests would be served by varnish, without asking for it from the webserver, that is till the object exceeds its lifetime in the cache. What i observe is this: the request for the jpeg is sent to the webserver everytime, the webserver send the entire content to varnish, and varnish sends a 304 Not Modified to the client browser. Is this the expected behaviour? Do i need to do some VCL'ing to get it work the way i expect? Any help or pointers will be very much appreciated. Thank you. From anup at iamcool.net Mon Jun 11 09:19:54 2007 From: anup at iamcool.net (Anup Shukla) Date: Mon, 11 Jun 2007 17:19:54 +0800 Subject: Newbie : Static content caching In-Reply-To: <466D02C0.8010701@iamcool.net> References: <466D02C0.8010701@iamcool.net> Message-ID: <466D13BA.8050106@iamcool.net> Hello all, Please accept my apologies for posting this without "complete" understanding about varnish. The reason for the static content not being cached was "cookies". Everything is well and good after a slight tinkering of the configuration file. I, again, apologies for not being responsible enough before posting. Just that, in this case, the light flickered, almost immediately after i hit the send button. Thank you. Anup Shukla wrote: > Hello all, > > I have been using Varnish with its "default configuration" for quite > sometime now. > Absolutely nothing has changed except for the backed servers port number > (set to 8000). > > Needless to say that it has been performing brilliantly and has reduced > the load on our webservers considerably. > > Just noticed something strange though. > Before i explain further, please let me clarify that i am not an expert > in caching by any standard, so this happens to be more of a newbie question. > > I am running varnish and the backend apache on the same server. > The backend listens on the loopback address while varnish listens on the > interface facing the load balancer. > > If i request, say for example, a static jpeg, i assume that the first > request will be served by the webserver via varnish, and all later > requests would be served by varnish, without asking for it from the > webserver, that is till the object exceeds its lifetime in the cache. > > What i observe is this: > the request for the jpeg is sent to the webserver everytime, the > webserver send the entire content to varnish, and varnish sends a 304 > Not Modified to the client browser. > > Is this the expected behaviour? > Do i need to do some VCL'ing to get it work the way i expect? > > Any help or pointers will be very much appreciated. > > Thank you. > > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc > From anup at iamcool.net Mon Jun 11 09:35:34 2007 From: anup at iamcool.net (Anup Shukla) Date: Mon, 11 Jun 2007 17:35:34 +0800 Subject: Newbie : Static content caching In-Reply-To: <874plederj.fsf@des.linpro.no> References: <466D02C0.8010701@iamcool.net> <874plederj.fsf@des.linpro.no> Message-ID: <466D1766.3020004@iamcool.net> Hi Dag-Erling Sm?rgrav, Thank you for your response. It was the cookies. Forcing varnish to cache requests even if cookies are present does the trick. Thank you again. Dag-Erling Sm?rgrav wrote: > Anup Shukla writes: >> If i request, say for example, a static jpeg, i assume that the first >> request will be served by the webserver via varnish, and all later >> requests would be served by varnish, without asking for it from the >> webserver, that is till the object exceeds its lifetime in the cache. >> >> What i observe is this: the request for the jpeg is sent to the >> webserver everytime, the webserver send the entire content to varnish, >> and varnish sends a 304 Not Modified to the client browser. > > Most likely, the backend does not set an expiry time on the requested > image, so Varnish uses the default TTL (normally 120s). > > Another possibility is that your site uses cookies; by default, Varnish > will not serve requests that come with a cookie from its cache. > > I can provide a more detailed (and correct) explanation if you send me a > raw varnish log off-list. > > DES From des at linpro.no Mon Jun 11 09:25:20 2007 From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Mon, 11 Jun 2007 11:25:20 +0200 Subject: Newbie : Static content caching In-Reply-To: <466D02C0.8010701@iamcool.net> (Anup Shukla's message of "Mon\, 11 Jun 2007 16\:07\:28 +0800") References: <466D02C0.8010701@iamcool.net> Message-ID: <874plederj.fsf@des.linpro.no> Anup Shukla writes: > If i request, say for example, a static jpeg, i assume that the first > request will be served by the webserver via varnish, and all later > requests would be served by varnish, without asking for it from the > webserver, that is till the object exceeds its lifetime in the cache. > > What i observe is this: the request for the jpeg is sent to the > webserver everytime, the webserver send the entire content to varnish, > and varnish sends a 304 Not Modified to the client browser. Most likely, the backend does not set an expiry time on the requested image, so Varnish uses the default TTL (normally 120s). Another possibility is that your site uses cookies; by default, Varnish will not serve requests that come with a cookie from its cache. I can provide a more detailed (and correct) explanation if you send me a raw varnish log off-list. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From andrearo at pvv.ntnu.no Mon Jun 11 21:14:53 2007 From: andrearo at pvv.ntnu.no (=?ISO-8859-1?Q?Andreas_R=F8sdal?=) Date: Mon, 11 Jun 2007 23:14:53 +0200 (CEST) Subject: Varnishstat: bandwith usage Message-ID: <20070611225521.C34016@verden.pvv.ntnu.no> Hi! I have a question about measuring bandwith usage with varnishstat. I would expect 'Total header bytes' and 'Total header bytes' to be the total number of bytes sent to clients since varnishd was started? Measuring these numbers at different points in time could therefore be used to calculate the bandwith usage. So here are my measurements: * Initial values: 556874988 Total header bytes 20238688347 Total body bytes * 75 minutes later: 848969031 Total header bytes 30800272652 Total body bytes The problem is that the correct values are 40 times larger than what I get from varnishstat. So is this a bug in varnishstat? I notice that the total body bytes is larger than 2^32, perhaps there is an integer overflow somewhere? Or am I interpreting these values wrong? - Andreas From phk at phk.freebsd.dk Mon Jun 11 21:22:28 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 11 Jun 2007 21:22:28 +0000 Subject: Varnishstat: bandwith usage In-Reply-To: Your message of "Mon, 11 Jun 2007 23:14:53 +0200." <20070611225521.C34016@verden.pvv.ntnu.no> Message-ID: <37806.1181596948@critter.freebsd.dk> In message <20070611225521.C34016 at verden.pvv.ntnu.no>, =?ISO-8859-1?Q?Andreas_R =F8sdal?= writes: >* Initial values: >556874988 Total header bytes >20238688347 Total body bytes > >* 75 minutes later: >848969031 Total header bytes >30800272652 Total body bytes > >The problem is that the correct values are 40 times larger >than what I get from varnishstat. So is this a bug in varnishstat? You're probably running out of the 32bit counter, we need to expand some of those to 64 bits... Can I get you to open a ticket in the varnish trac system ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Mon Jun 11 21:35:15 2007 From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Mon, 11 Jun 2007 23:35:15 +0200 Subject: Varnishstat: bandwith usage In-Reply-To: <20070611225521.C34016@verden.pvv.ntnu.no> ("Andreas =?utf-8?Q?R=C3=B8sdal=22's?= message of "Mon\, 11 Jun 2007 23\:14\:53 +0200 \(CEST\)") References: <20070611225521.C34016@verden.pvv.ntnu.no> Message-ID: <87bqfm41kc.fsf@des.linpro.no> Andreas R?sdal writes: > I would expect 'Total header bytes' and 'Total header bytes' to be > the total number of bytes sent to clients since varnishd was started? Yes. > Measuring these numbers at different points in time could therefore be > used to calculate the bandwith usage. So here are my measurements: > > * Initial values: > 556874988 Total header bytes > 20238688347 Total body bytes > > * 75 minutes later: > 848969031 Total header bytes > 30800272652 Total body bytes A bit of simple arithmetic gives 10853678348 bytes (10 GB) served in 75 minutes, or 2411928 Bps, or 19295428 bps (19.3 Mbps) which is a fair amount of traffic. > The problem is that the correct values are 40 times larger than what I > get from varnishstat. How do you derive "the correct values"? 40 times what you have here is nearly 800 Mbps, or twice as much as the daily average of Norway's most visited web site (vg.no). > I notice that the total body bytes is larger than 2^32, perhaps there > is an integer overflow somewhere? No, these are 64-bit counters. Perhaps there is an integer overflow in your calculator? ;) DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Mon Jun 11 21:37:37 2007 From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Mon, 11 Jun 2007 23:37:37 +0200 Subject: Varnishstat: bandwith usage In-Reply-To: <37806.1181596948@critter.freebsd.dk> (Poul-Henning Kamp's message of "Mon\, 11 Jun 2007 21\:22\:28 +0000") References: <37806.1181596948@critter.freebsd.dk> Message-ID: <877iqa41ge.fsf@des.linpro.no> "Poul-Henning Kamp" writes: > You're probably running out of the 32bit counter, we need to expand > some of those to 64 bits... It *is* a 64-bit counter. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From florian.schulze at gmx.net Mon Jun 11 21:26:04 2007 From: florian.schulze at gmx.net (Florian Schulze) Date: Mon, 11 Jun 2007 23:26:04 +0200 Subject: Mac OS X patch Message-ID: Hi! I attached a patch with the changes necessary to get the 1.0 branch to compile *and* run on Mac OS X. I don't know autoconf well enough to integrate it properly, so it's just the changes, not a full mergable patch. The changes are very simple and should apply to trunk as well. On trunk I had some more problems which I didn't investigate further yet. Regards, Florian Schulze -------------- next part -------------- A non-text attachment was scrubbed... Name: macosx.patch Type: application/octet-stream Size: 1169 bytes Desc: not available URL: From swann at cybercartes.com Tue Jun 12 10:23:23 2007 From: swann at cybercartes.com (swann) Date: Tue, 12 Jun 2007 12:23:23 +0200 Subject: where are logs Message-ID: <466E741B.2010200@cybercartes.com> Hi, I've problem with logs when i stress varnish with Apache Bench. I use varnishlog to store logs in logfile, and i (try to) get ncsa format log with varnishncsa to generate statistics with awstats It works fine with some requests but when I attac varnish by 100000 requests with 500 in concurrency (this is my requirements ) I loose info in logs ! Some output of varnisncsa looks like that : - - - [12/Jun/2007:11:32:06 +0200] "(null) (null) (null)" (null) (null) "-" "-" and the number of requests from varnishncsa is not coherent with the number of launched requests with AB: 25000 for 100000 original requests. I try to change run time parameters like 'thread_pools' to 500 but it is the same result.. Another parameters can fixe the problem ? I can't determine where is the problem, varnishlog , varnishncsa, varnishd, any idea? thank you. my config: FreeBSD 6.2 i386 /usr/local/sbin/varnishd -f /usr/local/etc/varnish.vcl -a 0.0.0.0:80 -P /var/run/varnish.pid -s file,/var/tmp/varnish/cache ,300m -T 127.0.0.1:9999 /usr/local/bin/varnishlog -D -a -P /var/run/varnishlog.pid -w /var/log/varnish.log From phk at phk.freebsd.dk Tue Jun 12 10:51:25 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 12 Jun 2007 10:51:25 +0000 Subject: where are logs In-Reply-To: Your message of "Tue, 12 Jun 2007 12:23:23 +0200." <466E741B.2010200@cybercartes.com> Message-ID: <41162.1181645485@critter.freebsd.dk> In message <466E741B.2010200 at cybercartes.com>, swann writes: >Hi, > >I've problem with logs when i stress varnish with Apache Bench. >I use varnishlog to store logs in logfile, and i (try to) get ncsa >format log with varnishncsa to generate statistics with awstats > >It works fine with some requests but when I attac varnish by 100000 >requests with 500 in concurrency (this is my requirements ) I loose info >in logs ! There are some parameters with respect to the shared memory log we have not tuned yet, the most important one being the size of the shared memory we use. If you look at the end of varnishd.c you will find a line like this: VSL_MgtInit(SHMLOG_FILENAME, 8*1024*1024); That sets the size to 8MB. Apart from some RAM/disk space, there is no penalty for increasing that size, so give it a fact 10 and see if that helps. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From nwmcsween at gmail.com Tue Jun 12 11:08:14 2007 From: nwmcsween at gmail.com (Nathan McSween) Date: Tue, 12 Jun 2007 04:08:14 -0700 Subject: Apache and lighttpd backend Message-ID: <000301c7ace1$fa5e94b0$6600a8c0@octane> Does anyone know of a way to make apache the default handler for php,pl,py,rb AND ?/*? while lighttpd handles everything else isn?t dynamic? I cant regex */* to apache for just that reason and the only other way I can see doing this is setting every single static extension type to lighttpd which would be impossible since a huge amount of extension types are undocumented. From des at linpro.no Tue Jun 12 13:19:28 2007 From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Tue, 12 Jun 2007 15:19:28 +0200 Subject: Apache and lighttpd backend In-Reply-To: <000301c7ace1$fa5e94b0$6600a8c0@octane> (Nathan McSween's message of "Tue\, 12 Jun 2007 04\:08\:14 -0700") References: <000301c7ace1$fa5e94b0$6600a8c0@octane> Message-ID: <87myz5l38f.fsf@des.linpro.no> Nathan McSween writes: > Does anyone know of a way to make apache the default handler for > php,pl,py,rb AND "/*" while lighttpd handles everything else > isn't dynamic? I cant regex */* to apache for just that reason and > the only other way I can see doing this is setting every single static > extension type to lighttpd which would be impossible since a huge > amount of extension types are undocumented. Sure, if you can express one of these sets as a regular expression. But there really isn't much point in splitting your backend in two when you have Varnish in front; just have Apache serve everything. Cache misses will be slightly slower, but cache hits will be just as fast. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From gaute at pht.no Tue Jun 19 11:41:38 2007 From: gaute at pht.no (Gaute Amundsen) Date: Tue, 19 Jun 2007 13:41:38 +0200 Subject: log files In-Reply-To: References: <17931.59232.890894.380967@dwetzel@nerim.net> Message-ID: <200706191341.38606.gaute@pht.no> On Thursday 29 March 2007 18:56, Dag-Erling Sm?rgrav wrote: > No, you have to run varnishlog and / or varnishncsa yourself to > capture the logs to a file. For the next release, I will work with > the packagers to make sure binary packages ship with an init script > which starts varnishlog. > > DES Did you ever get around to writing that init script? If not, how would you recommend going about it? Adding it to /etc/init.d/varnish, or perhaps a separate initscript? This would be the bare necessities I guess? /usr/bin/nohup \ /usr/bin/varnishncsa -c -a -w /var/log/httpd/varnish_combined-access_log & Regards Gaute Amundsen From ssm at linpro.no Tue Jun 19 11:45:07 2007 From: ssm at linpro.no (Stig Sandbeck Mathisen) Date: Tue, 19 Jun 2007 13:45:07 +0200 Subject: log files In-Reply-To: <200706191341.38606.gaute@pht.no> (Gaute Amundsen's message of "Tue, 19 Jun 2007 13:41:38 +0200") References: <17931.59232.890894.380967@dwetzel@nerim.net> <200706191341.38606.gaute@pht.no> Message-ID: <7xhcp42mnw.fsf@iostat.e.linpro.no> Gaute Amundsen writes: > Did you ever get around to writing that init script? If not, how > would you recommend going about it? > > Adding it to /etc/init.d/varnish, or perhaps a separate initscript? > > This would be the bare necessities I guess? > > /usr/bin/nohup \ > /usr/bin/varnishncsa -c -a -w /var/log/httpd/varnish_combined-access_log & One separate init script (for Debian) is available at http://varnish.projects.linpro.no/browser/trunk/varnish-cache/debian/varnish.varnishlog.init -- Stig Sandbeck Mathisen, Linpro From ingvar at linpro.no Tue Jun 19 11:58:24 2007 From: ingvar at linpro.no (Ingvar Hagelund) Date: Tue, 19 Jun 2007 13:58:24 +0200 Subject: log files In-Reply-To: <7xhcp42mnw.fsf@iostat.e.linpro.no> References: <17931.59232.890894.380967@dwetzel@nerim.net> <200706191341.38606.gaute@pht.no> <7xhcp42mnw.fsf@iostat.e.linpro.no> Message-ID: <4677C4E0.5000709@linpro.no> * Gaute Amundsen >> Did you ever get around to writing that init script? If not, how >> would you recommend going about it? >> >> Adding it to /etc/init.d/varnish, or perhaps a separate initscript? >> >> This would be the bare necessities I guess? >> >> /usr/bin/nohup \ >> /usr/bin/varnishncsa -c -a -w /var/log/httpd/varnish_combined-access_log & * Stig Sandbeck Mathisen > One separate init script (for Debian) is available at > http://varnish.projects.linpro.no/browser/trunk/varnish-cache/debian/varnish.varnishlog.init Just for the record, the RedHat subdirectory has a similar script. It is also included in the RHEL packages at sourceforge and in the Fedora package. Ingvar -- At barnebokfigurer er snille er bare ljug Mummipappa dater Barbamamma i smug From gaute at pht.no Tue Jun 19 13:56:33 2007 From: gaute at pht.no (Gaute Amundsen) Date: Tue, 19 Jun 2007 15:56:33 +0200 Subject: log files In-Reply-To: <4677C4E0.5000709@linpro.no> References: <17931.59232.890894.380967@dwetzel@nerim.net> <7xhcp42mnw.fsf@iostat.e.linpro.no> <4677C4E0.5000709@linpro.no> Message-ID: <200706191556.33623.gaute@pht.no> > * Stig Sandbeck Mathisen > > > One separate init script (for Debian) is available at > > http://varnish.projects.linpro.no/browser/trunk/varnish-cache/debian/varn > >ish.varnishlog.init > > Just for the record, the RedHat subdirectory has a similar script. It is > also included in the RHEL packages at sourceforge and in the Fedora > package. > > Ingvar Sorry folks, it seems I'm totaly blind. I've had /etc/init.d/varnishlog all the time. (since I upgraded to 1.0.4 I guess) So then you would pipe this log into "varnishncsa -r /dev/stdin"? This seems to work for me in AWstats :-) LogFile="cat /var/log/varnish/varnish.log | varnishncsa -c -r /dev/stdin | " or perhaps better: LogFile="/usr/bin/varnishncsa -c -r /var/log/varnish/varnish.log | " Seems there are ways to set up awstats to process this separately for the different vhosts too :-) G. From des at linpro.no Tue Jun 19 22:45:27 2007 From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Wed, 20 Jun 2007 00:45:27 +0200 Subject: log files In-Reply-To: <200706191341.38606.gaute@pht.no> (Gaute Amundsen's message of "Tue\, 19 Jun 2007 13\:41\:38 +0200") References: <17931.59232.890894.380967@dwetzel@nerim.net> <200706191341.38606.gaute@pht.no> Message-ID: <87odjblg1k.fsf@des.linpro.no> Gaute Amundsen writes: > Did you ever get around to writing that init script? Yes, 1.0.4 ships with init scripts for both varnishd and varnishlog. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From kimmo at glaborg.com Wed Jun 20 13:11:10 2007 From: kimmo at glaborg.com (=?ISO-8859-1?Q?Kimmo_Gl=E4borg?=) Date: Wed, 20 Jun 2007 15:11:10 +0200 Subject: Varnish as web page cache for dynamic content Message-ID: Hello, I'm totally new to Varnish and working for the largest IT news portal in Scandinavia. We're looking to replacing our current server side page cache environment with a hardware solution (such as Netapp NetCache for example). Now would this be possible to do maybe with Varnish instead? So I don't want to use Varnish for client proxy anyhow, just website content cache/acceleration and if possible also distribution for streaming media (webtv).. // kimmo From phk at phk.freebsd.dk Wed Jun 20 13:34:07 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 20 Jun 2007 13:34:07 +0000 Subject: Varnish as web page cache for dynamic content In-Reply-To: Your message of "Wed, 20 Jun 2007 15:11:10 +0200." Message-ID: <5536.1182346447@critter.freebsd.dk> In message , "=?ISO- 8859-1?Q?Kimmo_Gl=E4borg?=" writes: >Hello, > >I'm totally new to Varnish and working for the largest IT news portal in >Scandinavia. We're looking to replacing our current server side page >cache environment with a hardware solution (such as Netapp NetCache >for example). Now would this be possible to do maybe with Varnish instead? > >So I don't want to use Varnish for client proxy anyhow, just website >content cache/acceleration and if possible also distribution for >streaming media (webtv).. That's exactly what Varnish is built for :-) Dag-Erling and I talked briefly about real-time streaming apps yesterday and I think I can say definitively, that it is at earliest a v3 feature, if we do it at all. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From kimmo at glaborg.com Wed Jun 20 13:48:24 2007 From: kimmo at glaborg.com (=?ISO-8859-1?Q?Kimmo_Gl=E4borg?=) Date: Wed, 20 Jun 2007 15:48:24 +0200 Subject: Varnish as web page cache for dynamic content In-Reply-To: <5536.1182346447@critter.freebsd.dk> References: <5536.1182346447@critter.freebsd.dk> Message-ID: Can varnish run on separate machines? We wan't to place the future cache machine's between load balancer and web front ends. We have two systems we would like to use same caching boxes for; one unix based CMS and also one iis environment. Is this doable? // kimmo On 6/20/07, Poul-Henning Kamp wrote: > In message , "=?ISO- > 8859-1?Q?Kimmo_Gl=E4borg?=" writes: > >Hello, > > > >I'm totally new to Varnish and working for the largest IT news portal in > >Scandinavia. We're looking to replacing our current server side page > >cache environment with a hardware solution (such as Netapp NetCache > >for example). Now would this be possible to do maybe with Varnish instead? > > > >So I don't want to use Varnish for client proxy anyhow, just website > >content cache/acceleration and if possible also distribution for > >streaming media (webtv).. > > That's exactly what Varnish is built for :-) > > Dag-Erling and I talked briefly about real-time streaming apps yesterday > and I think I can say definitively, that it is at earliest a v3 feature, > if we do it at all. > > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > From phk at phk.freebsd.dk Wed Jun 20 13:50:12 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 20 Jun 2007 13:50:12 +0000 Subject: Varnish as web page cache for dynamic content In-Reply-To: Your message of "Wed, 20 Jun 2007 15:48:24 +0200." Message-ID: <5629.1182347412@critter.freebsd.dk> In message , "=?IS O-8859-1?Q?Kimmo_Gl=E4borg?=" writes: >Can varnish run on separate machines? Sure. >We wan't to place the future cache machine's between load balancer >and web front ends. We have two systems we would like to use same >caching boxes for; one unix based CMS and also one iis environment. >Is this doable? Absolutely. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From kimmo at glaborg.com Wed Jun 20 13:56:26 2007 From: kimmo at glaborg.com (=?ISO-8859-1?Q?Kimmo_Gl=E4borg?=) Date: Wed, 20 Jun 2007 15:56:26 +0200 Subject: Varnish as web page cache for dynamic content In-Reply-To: <5629.1182347412@critter.freebsd.dk> References: <5629.1182347412@critter.freebsd.dk> Message-ID: Sounds wonderful. Sorry my incompetence but, just so I get this right, in our IIS environment we use an old style "burning" batch jobs that burns out static html pages from an .asp site. Could external Varnish boxes replace & include a similar functionality? // kimmo On 6/20/07, Poul-Henning Kamp wrote: > In message , "=?IS > O-8859-1?Q?Kimmo_Gl=E4borg?=" writes: > > >Can varnish run on separate machines? > > Sure. > > >We wan't to place the future cache machine's between load balancer > >and web front ends. We have two systems we would like to use same > >caching boxes for; one unix based CMS and also one iis environment. > >Is this doable? > > Absolutely. > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > From phk at phk.freebsd.dk Wed Jun 20 14:09:33 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 20 Jun 2007 14:09:33 +0000 Subject: Varnish as web page cache for dynamic content In-Reply-To: Your message of "Wed, 20 Jun 2007 15:56:26 +0200." Message-ID: <5713.1182348573@critter.freebsd.dk> In message , "=?IS O-8859-1?Q?Kimmo_Gl=E4borg?=" writes: >Sounds wonderful. Sorry my incompetence but, just so I get this right, >in our IIS environment we use an old style "burning" batch jobs that >burns out static html pages from an .asp site. Could external Varnish >boxes replace & include a similar functionality? Yes, that's what it is designed for. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From kimmo at glaborg.com Wed Jun 20 14:21:26 2007 From: kimmo at glaborg.com (=?ISO-8859-1?Q?Kimmo_Gl=E4borg?=) Date: Wed, 20 Jun 2007 16:21:26 +0200 Subject: Varnish as web page cache for dynamic content In-Reply-To: <5713.1182348573@critter.freebsd.dk> References: <5713.1182348573@critter.freebsd.dk> Message-ID: That just did my day // kimmo On 6/20/07, Poul-Henning Kamp wrote: > In message , "=?IS > O-8859-1?Q?Kimmo_Gl=E4borg?=" writes: > > >Sounds wonderful. Sorry my incompetence but, just so I get this right, > >in our IIS environment we use an old style "burning" batch jobs that > >burns out static html pages from an .asp site. Could external Varnish > >boxes replace & include a similar functionality? > > Yes, that's what it is designed for. > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > From kimmo at glaborg.com Wed Jun 20 14:30:56 2007 From: kimmo at glaborg.com (=?ISO-8859-1?Q?Kimmo_Gl=E4borg?=) Date: Wed, 20 Jun 2007 16:30:56 +0200 Subject: Varnish as web page cache for dynamic content In-Reply-To: <5713.1182348573@critter.freebsd.dk> References: <5713.1182348573@critter.freebsd.dk> Message-ID: Oh, one more thing. One of our requirements it the possibility to choose which content to cache and not. All our pages are individual with ad strings etc. How do an external varnish box act, as a router or a switch between networks? Can it do loadbalancing? // kimmo On 6/20/07, Poul-Henning Kamp wrote: > In message , "=?IS > O-8859-1?Q?Kimmo_Gl=E4borg?=" writes: > > >Sounds wonderful. Sorry my incompetence but, just so I get this right, > >in our IIS environment we use an old style "burning" batch jobs that > >burns out static html pages from an .asp site. Could external Varnish > >boxes replace & include a similar functionality? > > Yes, that's what it is designed for. > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > From phk at phk.freebsd.dk Wed Jun 20 14:53:11 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 20 Jun 2007 14:53:11 +0000 Subject: Varnish as web page cache for dynamic content In-Reply-To: Your message of "Wed, 20 Jun 2007 16:30:56 +0200." Message-ID: <5890.1182351191@critter.freebsd.dk> In message , "=?IS O-8859-1?Q?Kimmo_Gl=E4borg?=" writes: >Oh, one more thing. One of our requirements it the possibility to >choose which content to cache and not. Varnish will inspect the HTTP headers from your backend, and will do as told. If you want to do something different, you can override the caching decision and lifetime in the VCL configuration language. >How do an external varnish box act, as a router >or a switch between networks? Varnishd doesn't care, it receives requests and answers them, using HTTP to contact the backend server when necessary. You can put it on the same machine, on a different machine with any number of network interfaces you like. >Can it do loadbalancing? Not currently. We're working on that. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From denis at zeno.org Fri Jun 22 14:07:17 2007 From: denis at zeno.org (Denis Ahrens) Date: Fri, 22 Jun 2007 16:07:17 +0200 Subject: r1533 does not run at all Message-ID: Hi I have several problems with revision 1533: 1. When starting (with empty tmp) it simply does not start at all. 2. When using varnishd with -n parameter a second time it says: Parameter errors: could not create /tmp/zenotest: Directory not empty and stops. 3. when starting with parameter -d and then manually 'start' it runs but throws errors and does not answer any request. Every child dies immediately. when I use an older version like r1532 varnishd works but the childs are crashing when there is an http-requestheader like "Cache-Control max- age=0". >> Child said (2, 46606): <> Cache child died pid=46606 status=0xb Clean child Child cleaned start child pid 46607 Child said (2, 46607): < References: <3fc325330706212118w740d5659x99e30828d1f8d82c@mail.gmail.com> Message-ID: <3fc325330706250718y7d41a0bev91ae23f946b5b7dc@mail.gmail.com> Hi, I'd like to use Varnish as nothing more than a basic Internet security appliance, that is set it up with a whitelist of sites and direct all other domains to a local page which will ask for authentication, then forward along if passed. I'm trying to just get google.com and wikipedia.orgworking as pass-through with the following VCL... and I'm having trouble. Eventually I'd like to implement caching, but I'd like to just get the simple stuff going for now. Thanks for any insight you can provide. ---- # This is a basic vcl.conf file for varnish. # Modifying this file should be where you store your modification to # varnish. Settnigs here will override defaults. backend default { set backend.host = "127.0.0.1"; set backend.port = "80"; } backend google { set backend.host = " www.google.com"; set backend.port = "80"; } backend wikipedia { set backend.host = "www.wikipedia.org "; set backend.port = "80"; } sub vcl_recv { if (req.request == "POST") { pipe; } if ( req.http.host ~ "^(www\.)?google\.com$") { set req.backend = google; pass; } elseif (req.http.host ~ "^(www\.)?wikipedia\.org$") { set req.backend = wikipedia; pass; } elseif (req.http.host ~ "^(en\.)?wikipedia\.org$") { set req.backend = wikipedia; pass; } # force lookup even when cookies are present if (req.request == "GET" && req.http.cookie) { lookup; } } sub vcl_fetch { # force minimum ttl of 180 seconds if (obj.ttl < 180s) { set obj.ttl = 180s; } } -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Mon Jun 25 14:20:39 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 25 Jun 2007 14:20:39 +0000 Subject: Fwd: Newbie varnish configuration In-Reply-To: Your message of "Mon, 25 Jun 2007 09:18:50 EST." <3fc325330706250718y7d41a0bev91ae23f946b5b7dc@mail.gmail.com> Message-ID: <35350.1182781239@critter.freebsd.dk> In message <3fc325330706250718y7d41a0bev91ae23f946b5b7dc at mail.gmail.com>, "Andr ew Baudouin" writes: >I'm having trouble. Me too. For one thing your report contains absolutely no indication of waht kind of trouble you might be having. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From des at linpro.no Mon Jun 25 14:27:35 2007 From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Mon, 25 Jun 2007 16:27:35 +0200 Subject: Fwd: Newbie varnish configuration In-Reply-To: <3fc325330706250718y7d41a0bev91ae23f946b5b7dc@mail.gmail.com> (Andrew Baudouin's message of "Mon\, 25 Jun 2007 09\:18\:50 -0500") References: <3fc325330706212118w740d5659x99e30828d1f8d82c@mail.gmail.com> <3fc325330706250718y7d41a0bev91ae23f946b5b7dc@mail.gmail.com> Message-ID: <871wg01548.fsf@des.linpro.no> "Andrew Baudouin" writes: > I'd like to use Varnish as nothing more than a basic Internet security > appliance, that is set it up with a whitelist of sites and direct all > other domains to a local page which will ask for authentication, then > forward along if passed. I'm trying to just get google.com and > wikipedia.orgworking as pass-through with the following VCL... and I'm > having trouble. Eventually I'd like to implement caching, but I'd > like to just get the simple stuff going for now. Varnish is not a client-side cache, and can't be used for what you're trying to do. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From anup at iamcool.net Tue Jun 26 07:40:19 2007 From: anup at iamcool.net (Anup Shukla) Date: Tue, 26 Jun 2007 13:10:19 +0530 Subject: Best Practices - Suggestions Request Message-ID: <4680C2E3.2010108@iamcool.net> Hi All, We have been running varnish for over a month on our production servers now. We are using the default configuration with a small change, that is to cache request with Cookies. We have a number of web servers behind a load balancer, each web server running varnish + apache. I understand that running both services on the same server is probably a bad idea, but due to time constraints i had no option but to go ahead with the current setup. The problem is, the varnish process dies every 7 - 10 days, and i have to manually restart it. One solution would be to write a script which monitors varnishd and starts it if not running. I, however, would like to understand "Why it dies"... I have been monitoring the servers for long times at a stretch and it seems that a flood of requests for big files (100+ MB) seems to get varnishd down. I cannot be very sure about this though. Is it really a bad idea to run varnishd on the same server with apache? Another thing is, is it possible to configure varnish such that it adds a custom header to the response (for example, the ip of the server which processed the request or any custom value). This would greatly assist in knowing which server served the request. Is it possible to make varnish refuse connections from the clients it it detects that the backend has stopped responding and start servicing again the moment it detects that the backend is up? Sorry, if i have asked too many questions at one go. Any help/pointers would be greatly appreciated. Regards A.S. From phk at phk.freebsd.dk Tue Jun 26 09:17:37 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 26 Jun 2007 09:17:37 +0000 Subject: Best Practices - Suggestions Request In-Reply-To: Your message of "Tue, 26 Jun 2007 13:10:19 +0530." <4680C2E3.2010108@iamcool.net> Message-ID: <62510.1182849457@critter.freebsd.dk> In message <4680C2E3.2010108 at iamcool.net>, Anup Shukla writes: >I understand that running both services on the same server is probably a >bad idea, but due to time constraints i had no option but to go ahead >with the current setup. It is not an obviously bad idea, and in many cases it is likely to work quite well. If it works for you, be happy about the saved hardware :-) >The problem is, the varnish process dies every 7 - 10 days, and i have >to manually restart it. >One solution would be to write a script which monitors varnishd and >starts it if not running. > >I, however, would like to understand "Why it dies"... I have been >monitoring the servers for long times at a stretch and it seems that a >flood of requests for big files (100+ MB) seems to get varnishd down. I >cannot be very sure about this though. It's not much to go from, but here are some ideas: 1. Do you have enough storage space ? By default Varnish takes a fixed fraction of the free space in /tmp, that may not be enough. Use the "-s" argument to specify a different directory and/or how much space you want to use. >Another thing is, is it possible to configure varnish such that it adds >a custom header to the response (for example, the ip of the server which >processed the request or any custom value). This would greatly assist in >knowing which server served the request. I'm working on VCL support for such headermodification right now. >Is it possible to make varnish refuse connections from the clients it it >detects that the backend has stopped responding and start servicing >again the moment it detects that the backend is up? At some point we will be able to, but not right now. >Sorry, if i have asked too many questions at one go. You're welcome :-) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From anup at iamcool.net Tue Jun 26 10:23:46 2007 From: anup at iamcool.net (Anup Shukla) Date: Tue, 26 Jun 2007 15:53:46 +0530 Subject: Best Practices - Suggestions Request In-Reply-To: <62510.1182849457@critter.freebsd.dk> References: <62510.1182849457@critter.freebsd.dk> Message-ID: <4680E932.5040906@iamcool.net> Poul-Henning Kamp wrote: > In message <4680C2E3.2010108 at iamcool.net>, Anup Shukla writes: > > >> I understand that running both services on the same server is probably a >> bad idea, but due to time constraints i had no option but to go ahead >> with the current setup. >> > > It is not an obviously bad idea, and in many cases it is likely to > work quite well. > > If it works for you, be happy about the saved hardware :-) > That makes me feel a lot better now. Thank you. :) > It's not much to go from, but here are some ideas: > > 1. Do you have enough storage space ? By default Varnish takes a > fixed fraction of the free space in /tmp, that may not be enough. > Use the "-s" argument to specify a different directory and/or how > much space you want to use. > I have put the cache file (or whatever its called as) under /cache/varnish the parameters are .................... -s file,/cache/varnish/varnish_storage.bin,2G Disk space should not be a problem as there is plenty available. Initially i had kept the size to 8G, but changed it 2G later. I doubt if that has any affect apart from being able to store not more than 2G of cached content. The only reason i changed it to 2G was that, i noticed the size of varnishd process kept on increasing. Now, i am not a Linux expert so did not have the knowledge about the whole idea of using disk space as RAM. However a bit of Google search cleared up that my fears were baseless. Or is it that there is a relation between the process size and the on-disk cache file? ( I hope this is not out-of-place) Did not change it back to 8G though, it works fine for me with the way its configured. ;) I will keep an eye on the process to see if i can find anything specific to relate to the problem of "Varnish dying in 7-10 days". Do i need to check for something specific? > >> Sorry, if i have asked too many questions at one go. >> > > You're welcome :-) > > Much appreciated. Thanks again. :) Regards A.S From phk at phk.freebsd.dk Tue Jun 26 10:42:48 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 26 Jun 2007 10:42:48 +0000 Subject: Best Practices - Suggestions Request In-Reply-To: Your message of "Tue, 26 Jun 2007 15:53:46 +0530." <4680E932.5040906@iamcool.net> Message-ID: <94058.1182854568@critter.freebsd.dk> In message <4680E932.5040906 at iamcool.net>, Anup Shukla writes: >Poul-Henning Kamp wrote: >I have put the cache file (or whatever its called as) under /cache/varnish >the parameters are .................... -s >file,/cache/varnish/varnish_storage.bin,2G > >Disk space should not be a problem as there is plenty available. >Initially i had kept the size to 8G, but changed it 2G later. >I doubt if that has any affect apart from being able to store not more >than 2G of cached content. Well, that could be your problem, right now Varnish doesn't deal well with running out of storage, (DES is working on that right now) >The only reason i changed it to 2G was that, i noticed the size of >varnishd process kept on increasing. That's normal, expected and shouldn't worry you unless Varnish forceses more important processes into the defensive. Varnish is designed around virtual memory and very few people have learned yet, that allocating a lot of virtual memory is not by definition a bad thing, it all depends how you use it. I would try to give it a bit more space if the crashes persist. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From anup at iamcool.net Tue Jun 26 11:09:42 2007 From: anup at iamcool.net (Anup Shukla) Date: Tue, 26 Jun 2007 16:39:42 +0530 Subject: Best Practices - Suggestions Request In-Reply-To: <94058.1182854568@critter.freebsd.dk> References: <94058.1182854568@critter.freebsd.dk> Message-ID: <4680F3F6.4080909@iamcool.net> Poul-Henning Kamp wrote: > In message <4680E932.5040906 at iamcool.net>, Anup Shukla writes: > >> Poul-Henning Kamp wrote: >> > > >> I have put the cache file (or whatever its called as) under /cache/varnish >> the parameters are .................... -s >> file,/cache/varnish/varnish_storage.bin,2G >> >> Disk space should not be a problem as there is plenty available. >> Initially i had kept the size to 8G, but changed it 2G later. >> I doubt if that has any affect apart from being able to store not more >> than 2G of cached content. >> > > Well, that could be your problem, right now Varnish doesn't deal > well with running out of storage, (DES is working on that right now) > > Okay. Then i just increase the storage size to a large but acceptable number. Thank you for the help. Will surely update you if i still have the crashes going on. Thank you again. Regards A.S From gaute at pht.no Tue Jun 26 13:52:03 2007 From: gaute at pht.no (Gaute Amundsen) Date: Tue, 26 Jun 2007 15:52:03 +0200 Subject: More logging questions - hostname again Message-ID: <200706261552.05388.gaute@pht.no> Hi again. Still fumbling around with the logs.. I have a feeling this has been mentioned before, but I can't find it right now. The problem is that AWstats really need a separate field for server hostname (apache: %v) to filter separate vhosts on. I'ts not good to use a simple regexp filter since referer could trigger a false match. I've tried various variants of: varnishncsa -c -I 'hostname' -r /var/log/varnish/varnish.log but without luck. The only option I can think of right now is to make my own log preprocessor to split the hostname out of the request. (quite unusual place to have it, is it not?) Any sed wizards out there? sed -e 's#\("\([A-Z]\+\) http://\([^/]*\)\)#\3 \1#g' Or any better sugestions? Regards Gaute Amundsen From des at linpro.no Wed Jun 27 20:05:50 2007 From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Wed, 27 Jun 2007 22:05:50 +0200 Subject: Best Practices - Suggestions Request In-Reply-To: <94058.1182854568@critter.freebsd.dk> (Poul-Henning Kamp's message of "Tue\, 26 Jun 2007 10\:42\:48 +0000") References: <94058.1182854568@critter.freebsd.dk> Message-ID: <87ved9f9i9.fsf@des.linpro.no> "Poul-Henning Kamp" writes: > Anup Shukla writes: > > Disk space should not be a problem as there is plenty available. > > Initially i had kept the size to 8G, but changed it 2G later. I > > doubt if that has any affect apart from being able to store not more > > than 2G of cached content. > Well, that could be your problem, right now Varnish doesn't deal > well with running out of storage, (DES is working on that right now) Still, only the child should die, the parent should automatically restart it. He says he has to restart it manually, which worries me... Anup, what version are you running? DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Wed Jun 27 20:10:42 2007 From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Wed, 27 Jun 2007 22:10:42 +0200 Subject: More logging questions - hostname again In-Reply-To: <200706261552.05388.gaute@pht.no> (Gaute Amundsen's message of "Tue\, 26 Jun 2007 15\:52\:03 +0200") References: <200706261552.05388.gaute@pht.no> Message-ID: <87r6nxf9a5.fsf@des.linpro.no> Gaute Amundsen writes: > The problem is that AWstats really need a separate field for server > hostname (apache: %v) to filter separate vhosts on. [...] The only > option I can think of right now is to make my own log preprocessor to > split the hostname out of the request. [...] Or any better > sugestions? Take the varnishncsa source code and modify it to produce what you need? It's really not that hard, just modify the bottom half of h_ncsa(). I intend to eventually modify varnishncsa to understand Apache CustomLog format strings, but I have more pressing issues to handle first. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Wed Jun 27 20:13:37 2007 From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Wed, 27 Jun 2007 22:13:37 +0200 Subject: Varnish Stats In-Reply-To: (ccripy@gmail.com's message of "Fri\, 1 Jun 2007 13\:53\:38 -0400") References: Message-ID: <87myylf95a.fsf@des.linpro.no> cripy writes: > I'm curious what kind of varnish stats everyone here has recorded and > what the "maximum" uptime you have with varnish not restarting itself. > I'm trying to track down a nasty memory leak I believe exists in > varnish which you can see for yourself in "top". It appears varnishs > memory just increases infinitely until it crashes and restarts itself. What you are seeing is Varnish using more and more of its cache (hence increasing its RSS aka VSIZE) until it runs out of cache space and restarts. With the LRU code I committed today, once the cache fills up it will start discarding the least recently used objects instead of restarting. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Wed Jun 27 20:25:32 2007 From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Wed, 27 Jun 2007 22:25:32 +0200 Subject: r1533 does not run at all In-Reply-To: (Denis Ahrens's message of "Fri\, 22 Jun 2007 16\:07\:17 +0200") References: Message-ID: <87ir99f8lf.fsf@des.linpro.no> Denis Ahrens writes: > I have several problems with revision 1533: There were issues with the instance naming code which have now been resolved. In more general terms, it might be safe to run trunk at a particular point in time, but it's not necessarily safe to *track* trunk, especially now that we are actively working on 1.1 / 2.0. I will probably branch 1.1 some time next week, after which it should be safe to track branches/1.1 but almost certainly unsafe to track trunk, which will continue to move steadily towards 2.0. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From james at nyi.net Wed Jun 27 21:16:34 2007 From: james at nyi.net (James Quacinella) Date: Wed, 27 Jun 2007 17:16:34 -0400 Subject: Latest trunk build will not start (rev1586) Message-ID: <4682D3B2.7030200@nyi.net> Hello everyone. I know tracking trunk is typically a bad idea, but I just wanted to post this up. It seems the latest revision is giving me errors on startup ("Cannot create working directory NONE/var/$hostname") but I haven't changed the startup command line parameters, which are as such: varnishd -a 10.10.10.5:80 -T 10.10.10.5:8080 -b 10.11.11.200:80 -s file, /var/cache/varnish/,10G -p thread_pool_max=1500 -p thread_pools=5 -p listen_depth=512 -p client_http11=on -p backend_http11=on -p default_ttl=36000 -P /var/run/varnishd.pid The error is from varnishd.c, line 549 - 553 if (mkdir(dirname, 0755) < 0 && errno != EEXIST) { fprintf(stderr, "Cannot create working directory '%s': %s\n", dirname, strerror(errno)); exit(1); } I'm not 100% sure where dirname gets set though. Don't know if this is a bug or I am missing some parameter, thanks for any and all info :-) -- james From anup at iamcool.net Thu Jun 28 03:14:37 2007 From: anup at iamcool.net (Anup Shukla) Date: Thu, 28 Jun 2007 08:44:37 +0530 Subject: Best Practices - Suggestions Request In-Reply-To: <87ved9f9i9.fsf@des.linpro.no> References: <94058.1182854568@critter.freebsd.dk> <87ved9f9i9.fsf@des.linpro.no> Message-ID: <4683279D.7020509@iamcool.net> Dag-Erling Sm?rgrav wrote: > Still, only the child should die, the parent should automatically > restart it. He says he has to restart it manually, which worries me... > > Anup, what version are you running? > > DES > I am running version 1.0.4 Is that too old to use? If so, i will update it immediately. Thank you Regards, A.S From anup at iamcool.net Thu Jun 28 03:23:21 2007 From: anup at iamcool.net (Anup Shukla) Date: Thu, 28 Jun 2007 08:53:21 +0530 Subject: Best Practices - Suggestions Request In-Reply-To: <4683279D.7020509@iamcool.net> References: <94058.1182854568@critter.freebsd.dk> <87ved9f9i9.fsf@des.linpro.no> <4683279D.7020509@iamcool.net> Message-ID: <468329A9.5080903@iamcool.net> Anup Shukla wrote: > Dag-Erling Sm?rgrav wrote: > >> Still, only the child should die, the parent should automatically >> restart it. He says he has to restart it manually, which worries me... >> >> Anup, what version are you running? >> >> DES >> >> > > > I am running version 1.0.4 > Is that too old to use? > If so, i will update it immediately. > Sorry for missing out on the release date. It is version 1.0.4 released on May 20, 2007. From admin at adofms.com.au Thu Jun 28 04:37:09 2007 From: admin at adofms.com.au (ADOFMS Admin, SteveOC) Date: Thu, 28 Jun 2007 14:07:09 +0930 Subject: Best Practices - Suggestions Request In-Reply-To: <468329A9.5080903@iamcool.net> References: <94058.1182854568@critter.freebsd.dk> <87ved9f9i9.fsf@des.linpro.no> <4683279D.7020509@iamcool.net> <468329A9.5080903@iamcool.net> Message-ID: <46833AF5.6030501@adofms.com.au> >> I am running version 1.0.4 >> Is that too old to use? >> If so, i will update it immediately. >> >> > > Sorry for missing out on the release date. It is version 1.0.4 released > on May 20, 2007. > > hmm ... I am running 1.0.4 as well, but I am building mine from the SVN repository rather than from the 1.0.4 tarball that was available at the time through the gentoo ebuilds. I originally just did an emerge to get varnish running, but found some issues with VCL, and it was suggested that I get the SVN release and try some VCL extensions. My SVN update was from the 29th May 2007, and there was sufficient functionality difference between the SVN source compared to the origiinal 1.0.4 release. Not a whole lot of code, but it added a few VCL commands that got the job done for me, which were not working in the original 20-May 1.0.4 release. In my case, the following VCL code requires the SVN 1.0.4 release : sub vcl_hash { set req.hash += req.url; set req.hash += req.http.host; set req.hash += req.http.cookie; hash; } Not suggesting that the SVN code would cure your crashes - but it definitely gives you a better VCL environment than the stock release. I havnt seen ANY stability or performance, or memory leak issues with Varnish. My system gets hit hard enough, but in my case, varnish is only serving smallish chunks, and has no constraints on disk space or RAM. My main machine runs apache + php + mysql + varnish + a number of cronjobs written in C, and even during very busy periods, it never goes to swap. It has 'only' 4GB of physical RAM, and varnish is compiled in 64bit mode. Not a problem on my setup, but every case is different. When it comes to your situation with varnish dying - that really sounds like a resource problem from the outside here. Hard to tell without having your system right in front of me and watching what it is doing. ie - I would be very keen to run one of your varnish instances under gdb and try to simulate the point where it is dying, and see if you can find a pattern there ??. Or at least run it from a console under gdb and let it go for several days and hope that it crashes whilst in the debugger. Whilst commercial reality dictates that we need to find workarounds to make things work from one day to the next (and that applies to EVERYTHING) - Id encourage you to keep looking and find out exactly why the process is stopping. Even if a new release of code fixes the problem, its good for you to know why the old setup didnt work, and the new one does. Keep looking :) Sounds like Dag's latest code (which drops items from cache on a LRU scheme as memory fills up) is more likely to solve your problems longer term. I assume that comes out first in SVN, so thats another good reason to try the SVN release. From des at linpro.no Thu Jun 28 06:54:24 2007 From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Thu, 28 Jun 2007 08:54:24 +0200 Subject: Best Practices - Suggestions Request In-Reply-To: <4683279D.7020509@iamcool.net> (Anup Shukla's message of "Thu\, 28 Jun 2007 08\:44\:37 +0530") References: <94058.1182854568@critter.freebsd.dk> <87ved9f9i9.fsf@des.linpro.no> <4683279D.7020509@iamcool.net> Message-ID: <87ejjwfu1r.fsf@des.linpro.no> Anup Shukla writes: > Dag-Erling Sm?rgrav writes: > > Anup, what version are you running? > I am running version 1.0.4 Is that too old to use? If so, i will > update it immediately. No, 1.0.4 should be fine, but I'm not aware of any bugs in it that might cause the behaviour you're seeing. What hardware and OS are you running it on? DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From ssm at linpro.no Thu Jun 28 06:54:45 2007 From: ssm at linpro.no (Stig Sandbeck Mathisen) Date: Thu, 28 Jun 2007 08:54:45 +0200 Subject: More logging questions - hostname again In-Reply-To: <87r6nxf9a5.fsf@des.linpro.no> (Dag-Erling =?utf-8?Q?Sm=C3=B8?= =?utf-8?Q?rgrav's?= message of "Wed, 27 Jun 2007 22:10:42 +0200") References: <200706261552.05388.gaute@pht.no> <87r6nxf9a5.fsf@des.linpro.no> Message-ID: <7xlke4lgay.fsf@iostat.e.linpro.no> Dag-Erling Sm?rgrav writes: > I intend to eventually modify varnishncsa to understand Apache > CustomLog format strings Now, that's really high on the nice-to-have list. :D -- Stig Sandbeck Mathisen, Linpro From des at linpro.no Thu Jun 28 06:56:07 2007 From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Thu, 28 Jun 2007 08:56:07 +0200 Subject: Latest trunk build will not start (rev1586) In-Reply-To: <4682D3B2.7030200@nyi.net> (James Quacinella's message of "Wed\, 27 Jun 2007 17\:16\:34 -0400") References: <4682D3B2.7030200@nyi.net> Message-ID: <87abukftyw.fsf@des.linpro.no> James Quacinella writes: > I know tracking trunk is typically a bad idea, but I just wanted to > post this up. It seems the latest revision is giving me errors on > startup ("Cannot create working directory NONE/var/$hostname") but I > haven't changed the startup command line parameters, which are as > such: Did you re-run autogen.sh after updating? In the top-level source directory, what does 'grep STATE config.h' say? DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Thu Jun 28 07:01:57 2007 From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Thu, 28 Jun 2007 09:01:57 +0200 Subject: Best Practices - Suggestions Request In-Reply-To: <46833AF5.6030501@adofms.com.au> (ADOFMS Admin's message of "Thu\, 28 Jun 2007 14\:07\:09 +0930") References: <94058.1182854568@critter.freebsd.dk> <87ved9f9i9.fsf@des.linpro.no> <4683279D.7020509@iamcool.net> <468329A9.5080903@iamcool.net> <46833AF5.6030501@adofms.com.au> Message-ID: <876458ftp6.fsf@des.linpro.no> "ADOFMS Admin, SteveOC" writes: > Sounds like Dag's latest code (which drops items from cache on a LRU > scheme as memory fills up) is more likely to solve your problems longer > term. I assume that comes out first in SVN, so thats another good reason > to try the SVN release. I would like to stress once more that it is not always a good idea to track trunk, especially now that we are in an active development phase. For instance, we now know that the workspace management code that was committed on June 4th had a small but significant (performance- and correctness-affecting) bug that was fixed only yesterday, and the instance-naming code we committed last week had serious issues that were fixed on Tuesday. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From denis at directmedia.de Thu Jun 28 10:23:03 2007 From: denis at directmedia.de (Denis Ahrens) Date: Thu, 28 Jun 2007 12:23:03 +0200 Subject: Latest trunk build will not start (rev1586) In-Reply-To: <4682D3B2.7030200@nyi.net> References: <4682D3B2.7030200@nyi.net> Message-ID: On 27.06.2007, at 23:16, James Quacinella wrote: > Hello everyone. > > I know tracking trunk is typically a bad idea, but I just wanted to > post > this up. It seems the latest revision is giving me errors on startup > ("Cannot create working directory NONE/var/$hostname") but I haven't > changed the startup command line parameters, which are as such: > > varnishd -a 10.10.10.5:80 -T 10.10.10.5:8080 -b 10.11.11.200:80 -s > file, /var/cache/varnish/,10G -p thread_pool_max=1500 -p > thread_pools=5 > -p listen_depth=512 -p client_http11=on -p backend_http11=on -p > default_ttl=36000 -P /var/run/varnishd.pid > > The error is from varnishd.c, line 549 - 553 > > if (mkdir(dirname, 0755) < 0 && errno != EEXIST) { > fprintf(stderr, "Cannot create working directory '%s': %s\n", > dirname, strerror(errno)); > exit(1); > } > > I'm not 100% sure where dirname gets set though. Don't know if this > is a > bug or I am missing some parameter, thanks for any and all info :-) > > -- james You need to start configure with option --prefix=/usr/local Denis From des at linpro.no Thu Jun 28 10:34:24 2007 From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Thu, 28 Jun 2007 12:34:24 +0200 Subject: Latest trunk build will not start (rev1586) In-Reply-To: (Denis Ahrens's message of "Thu\, 28 Jun 2007 12\:23\:03 +0200") References: <4682D3B2.7030200@nyi.net> Message-ID: <87odj0e5an.fsf@des.linpro.no> Denis Ahrens writes: > James Quacinella writes: > > I know tracking trunk is typically a bad idea, but I just wanted to > > post this up. It seems the latest revision is giving me errors on > > startup ("Cannot create working directory NONE/var/$hostname") but I > > haven't changed the startup command line parameters, which are as > > such: > You need to start configure with option --prefix=/usr/local Not necessarily. Regardless of the selected prefix, ${localstatedir}/varnish (normally ${prefix}/var/varnish) should be created during installation. In any case, this particular failure is most likely a result of not re-running autogen.sh and configure after updating the source tree. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From gaute at pht.no Thu Jun 28 12:13:40 2007 From: gaute at pht.no (Gaute Amundsen) Date: Thu, 28 Jun 2007 14:13:40 +0200 Subject: More logging questions - hostname again In-Reply-To: <87r6nxf9a5.fsf@des.linpro.no> References: <200706261552.05388.gaute@pht.no> <87r6nxf9a5.fsf@des.linpro.no> Message-ID: <200706281413.41112.gaute@pht.no> On Wednesday 27 June 2007 22:10, Dag-Erling Sm?rgrav wrote: > Gaute Amundsen writes: > > The problem is that AWstats really need a separate field for server > > hostname (apache: %v) to filter separate vhosts on. [...] The only > > option I can think of right now is to make my own log preprocessor to > > split the hostname out of the request. [...] Or any better > > sugestions? > > Take the varnishncsa source code and modify it to produce what you need? > It's really not that hard, just modify the bottom half of h_ncsa(). Thanks for the pointer. I guess I'm sometimes more reluctant to touch C that is strictly waranted :-/ > I intend to eventually modify varnishncsa to understand Apache CustomLog > format strings, but I have more pressing issues to handle first. > > DES That would be just the thing :) But for now it seems that sed pipe seems to be pretty stable. I guess I ought to add it to the wiki or make a tiny howto or something...? Gaute From trondmm-varnish at crusaders.no Thu Jun 28 12:28:01 2007 From: trondmm-varnish at crusaders.no (Trond Michelsen) Date: Thu, 28 Jun 2007 14:28:01 +0200 Subject: Best Practices - Suggestions Request In-Reply-To: <87ejjwfu1r.fsf@des.linpro.no> References: <94058.1182854568@critter.freebsd.dk> <87ved9f9i9.fsf@des.linpro.no> <4683279D.7020509@iamcool.net> <87ejjwfu1r.fsf@des.linpro.no> Message-ID: <20070628122801.GK25095@crusaders.no> On Thu, Jun 28, 2007 at 08:54:24AM +0200, Dag-Erling Sm?rgrav wrote: > Anup Shukla writes: >> Dag-Erling Sm??rgrav writes: >>> Anup, what version are you running? >> I am running version 1.0.4 Is that too old to use? If so, i will >> update it immediately. > No, 1.0.4 should be fine, but I'm not aware of any bugs in it that might > cause the behaviour you're seeing. What hardware and OS are you running > it on? We're running Varnish 1.0.3, so this might not be related, but we've also run into a bug that causes varnish to die. In our setup, varnish is prone to crash if /tmp is cleaned up (we had a process that removed all files that hadn't been accessed in 7 days from /tmp). What happened in our case was that something (we couldn't figure out what) would make varnish reload the vcl-files. I'm not sure why it would read these files from /tmp, but I'm guessing this is where varnish places the compiled versions. Anyway - if these files were missing, all varnish processes would simply die. -- Trond Michelsen From des at linpro.no Thu Jun 28 12:51:24 2007 From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Thu, 28 Jun 2007 14:51:24 +0200 Subject: Best Practices - Suggestions Request In-Reply-To: <20070628122801.GK25095@crusaders.no> (Trond Michelsen's message of "Thu\, 28 Jun 2007 14\:28\:01 +0200") References: <94058.1182854568@critter.freebsd.dk> <87ved9f9i9.fsf@des.linpro.no> <4683279D.7020509@iamcool.net> <87ejjwfu1r.fsf@des.linpro.no> <20070628122801.GK25095@crusaders.no> Message-ID: <87hcos6y43.fsf@des.linpro.no> Trond Michelsen writes: > We're running Varnish 1.0.3, so this might not be related, but we've > also run into a bug that causes varnish to die. 1.0.3 has numerous known bugs... > In our setup, varnish is prone to crash if /tmp is cleaned up (we had > a process that removed all files that hadn't been accessed in 7 days > from /tmp). What happened in our case was that something (we couldn't > figure out what) would make varnish reload the vcl-files. I'm not sure > why it would read these files from /tmp, but I'm guessing this is > where varnish places the compiled versions. Anyway - if these files > were missing, all varnish processes would simply die. If the child process dies (due to a bug or running out of space), it will not be able to restart without the compiled VCL file, and the parent will probably bail out... DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From james at nyi.net Thu Jun 28 15:48:55 2007 From: james at nyi.net (James Quacinella) Date: Thu, 28 Jun 2007 11:48:55 -0400 Subject: Latest trunk build will not start (rev1586) In-Reply-To: <87odj0e5an.fsf@des.linpro.no> References: <4682D3B2.7030200@nyi.net> <87odj0e5an.fsf@des.linpro.no> Message-ID: <4683D867.3020503@nyi.net> Dag-Erling Sm?rgrav wrote: > In any case, this particular failure is most likely a result of not > re-running autogen.sh and configure after updating the source tree. > Doh! You're right. I usually do, I'm not sure how I overlooked that. Oh well, sorry for the mailing list noise. Thanks! -- james From jean-marc.pouchoulon at ac-montpellier.fr Fri Jun 29 17:39:45 2007 From: jean-marc.pouchoulon at ac-montpellier.fr (jean-marc pouchoulon) Date: Fri, 29 Jun 2007 19:39:45 +0200 Subject: setting varnish storage size Message-ID: <468543E1.705@ac-montpellier.fr> Hi , I'm a bit confused about /var/lib/varnish/varnish_storage.bin I use varnish 1.0.0.4 under linux fedora 6 ( using a vserver ). I set in /etc/sysconfig varnish VARNISH_STORAGE_SIZE=2G and the size of the file still remains to one GIGA -rw------- 1 root root 1,0G jun 29 19:27 /var/lib/varnish/varnish_storage.bin setting VARNISH_STORAGE_SIZE=2000G /var/lib/varnish/varnish_storage.bin setting 3000G I get -rw------- 1 root root 1,5G jun 29 19:29 /var/lib/varnish/varnish_storage.bin mm what I don't understand ? ( the % parameter ?) thanks From anup at iamcool.net Sat Jun 30 03:50:59 2007 From: anup at iamcool.net (Anup Shukla) Date: Sat, 30 Jun 2007 09:20:59 +0530 Subject: Best Practices - Suggestions Request Message-ID: <4685D323.1080307@iamcool.net> Hi All. Apologies for not being able to respond to my own problems ;) I would personally like to give the latest SVN version a try. I had encountered a situation where the /tmp was mounted noexec and varnish failed to load. This made me aware that Varnish uses /tmp to keep the compiled VCL. I have removed the noexec option and its fine now.. would love to have a configuration option to set the directory though :) . As for the crashes, its still a mystery. The system does not swap and everything seems to be quite fine. Must be something else which is affecting Varnish. There is one possibility though. There is a lot of old unorganized php code on the servers that i have never had time to inspect. Its working so i did not need to tinker with it. I assume that some one has used the /tmp to cache query responses and might be cleaning up the entire /tmp to flush the caches. Sounds a bit stupid, but a very possible scenario in my case. Will need some time to find if that is really the case. Thanks to DES for pointing this /tmp scenario (in his reply to Tron Michelson) Regards A.S From anup at iamcool.net Sat Jun 30 03:51:24 2007 From: anup at iamcool.net (Anup Shukla) Date: Sat, 30 Jun 2007 09:21:24 +0530 Subject: setting varnish storage size Message-ID: <4685D33C.4060103@iamcool.net> jean-marc pouchoulon wrote: > Hi , > I'm a bit confused about /var/lib/varnish/varnish_storage.bin > I use varnish 1.0.0.4 under linux fedora 6 ( using a vserver ). > > I set in /etc/sysconfig varnish VARNISH_STORAGE_SIZE=2G > and the size of the file still remains to one GIGA > > -rw------- 1 root root 1,0G jun 29 19:27 > /var/lib/varnish/varnish_storage.bin > > setting VARNISH_STORAGE_SIZE=2000G /var/lib/varnish/varnish_storage.bin > > > setting 3000G I get > -rw------- 1 root root 1,5G jun 29 19:29 > /var/lib/varnish/varnish_storage.bin > > mm what I don't understand ? ( the % parameter ?) > > thanks > Are you using 32-bit hardware? If yes, there lies the problem. I had similar issues when testing on my local 32-bit PC. However once i installed on our actual servers (64-bit), the 1G limit went away. Have never tried setting the storage file to sizes in thousands of G though. Also, as far as i know, Linux will not support files bigger than 4G unless the support is added to the kernel. Not sure if Fedora Core 6 has the patch applied. I cannot be very sure of this though. Regards A.S From denis at zeno.org Sat Jun 30 18:29:51 2007 From: denis at zeno.org (Denis Ahrens) Date: Sat, 30 Jun 2007 20:29:51 +0200 Subject: missing Content-Lenght header Message-ID: Hi I have varnish head running and everything now looks good. But after some while varnishd stops to send out Content-Length header. A simple restart and it works again for some hours then it happens again. The logoutput with the missing Content-Length header ends with "SessionClose c timeout", when it works it shows "SessionClose c EOF" I have also seen "ObjLostHeader" in the logfile. But I don't know if that was related. I will report if I see it again. Denis From phk at phk.freebsd.dk Sat Jun 30 18:57:02 2007 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 30 Jun 2007 18:57:02 +0000 Subject: missing Content-Lenght header In-Reply-To: Your message of "Sat, 30 Jun 2007 20:29:51 +0200." Message-ID: <13906.1183229822@critter.freebsd.dk> In message , Denis Ahrens writes : >I have varnish head running and everything now looks good. How recent is that copy ? I (belive I) fixed that bug this week. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From denis at zeno.org Sat Jun 30 19:33:42 2007 From: denis at zeno.org (Denis Ahrens) Date: Sat, 30 Jun 2007 21:33:42 +0200 Subject: missing Content-Lenght header In-Reply-To: <13906.1183229822@critter.freebsd.dk> References: <13906.1183229822@critter.freebsd.dk> Message-ID: On 30.06.2007, at 20:57, Poul-Henning Kamp wrote: > In message , Denis > Ahrens writes > : > >> I have varnish head running and everything now looks good. > > How recent is that copy ? I (belive I) fixed that bug > this week. svnversion says 1589M. you made no changes after that. Denis From des at linpro.no Sat Jun 30 21:36:10 2007 From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Sat, 30 Jun 2007 23:36:10 +0200 Subject: utma utmz In-Reply-To: <1183147489.32019.23.camel@gabriela> (Manuel Amador's message of "Fri\, 29 Jun 2007 15\:04\:49 -0500") References: <1183147489.32019.23.camel@gabriela> Message-ID: <87ir9515x1.fsf@des.linpro.no> [moved from -dev to -misc] "Manuel Amador (Rudd-O)" writes: > My site always feeds the utma and utmz (google analytics) cookies to > requests. How can I specify in the varnish configuration file > default.vcl that if *only* those two cookies are present, it should > supply files from cache, but if any other cookies are present, it should > contact the backend server (or use the appropriate caching policy)? There's no easy way to do that with the current code base, but we expect to have the required functionality in place within a couple of weeks. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Sat Jun 30 21:41:58 2007 From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Sat, 30 Jun 2007 23:41:58 +0200 Subject: Problem with varnish and caching In-Reply-To: <1183146070.32019.20.camel@gabriela> (Manuel Amador's message of "Fri\, 29 Jun 2007 14\:41\:10 -0500") References: <1183146070.32019.20.camel@gabriela> Message-ID: <87ejjt15nd.fsf@des.linpro.no> [moved from -dev to -misc] "Manuel Amador (Rudd-O)" writes: > As you might already have noted, I reported a bug on varnish caching > files indiscriminately. This is not a bug, it is a misunderstanding. It appears you expect Varnish to act like an RFC 2616 "shared cache" whereas it is in fact a "surrogate" (see the "Edge Architecture Specification" by Oracle and Akamai, although Varnish does not yet fully implement that specification either) > My page sets a few cookies. That'd be okay and it should produce > dynamic pages, which Varnish is furnishing through my backend. The > thing is, these cookies are sent along requests for CSS and PNG and JPG > and JS files, which causes varnish to contact the backend. I don't want > that to happen (I'm happy with them being cached by Varnish 120 > seconds). > > How can I tell Varnish that requests with a response that includes ETag > (a discriminant for static files) should be forcibly cached? This is basically the same issue as in your previous email, and the answer is the same. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From des at linpro.no Sat Jun 30 21:45:45 2007 From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Sat, 30 Jun 2007 23:45:45 +0200 Subject: missing Content-Lenght header In-Reply-To: (Denis Ahrens's message of "Sat\, 30 Jun 2007 21\:33\:42 +0200") References: <13906.1183229822@critter.freebsd.dk> Message-ID: <87abuh15h2.fsf@des.linpro.no> Denis Ahrens writes: > svnversion says 1589M. you made no changes after that. Are you sure your tree is up-to-date and you rebuilt everything? r1584 should have fixed that bug. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no From denis at zeno.org Sat Jun 30 22:06:33 2007 From: denis at zeno.org (Denis Ahrens) Date: Sun, 1 Jul 2007 00:06:33 +0200 Subject: missing Content-Lenght header In-Reply-To: <87abuh15h2.fsf@des.linpro.no> References: <13906.1183229822@critter.freebsd.dk> <87abuh15h2.fsf@des.linpro.no> Message-ID: <4212A4E9-F93C-4A07-9654-16C3D917A8FB@zeno.org> On 30.06.2007, at 23:45, Dag-Erling Sm?rgrav wrote: > Denis Ahrens writes: >> svnversion says 1589M. you made no changes after that. > > Are you sure your tree is up-to-date and you rebuilt everything? > r1584 > should have fixed that bug. I made the binary one day later, but maybe I forgot to checkout before. Anyway, I made a new one from HEAD and will look at it and report back when it happens again. Denis PS: I still need to add --prefix=/usr/local to configure. Do I need to remove some files which autogen.sh builds new then? From des at linpro.no Sat Jun 30 22:37:27 2007 From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Sun, 01 Jul 2007 00:37:27 +0200 Subject: missing Content-Lenght header In-Reply-To: <4212A4E9-F93C-4A07-9654-16C3D917A8FB@zeno.org> (Denis Ahrens's message of "Sun\, 1 Jul 2007 00\:06\:33 +0200") References: <13906.1183229822@critter.freebsd.dk> <87abuh15h2.fsf@des.linpro.no> <4212A4E9-F93C-4A07-9654-16C3D917A8FB@zeno.org> Message-ID: <876455132w.fsf@des.linpro.no> Denis Ahrens writes: > PS: I still need to add --prefix=/usr/local to configure. Do I need to > remove some files which autogen.sh builds new then? I'm not sure I understand what you mean - normally, you run autogen.sh first, then ./configure with whatever options you want. However, if you're running on FreeBSD, /usr/local is the default prefix; there is no need to specify it explicitly. You can also use autogen.des, which runs autogen.sh *and* configure with options useful for development and debugging; it sets the prefix to /opt/varnish. DES -- Dag-Erling Sm?rgrav Senior Software Developer Linpro AS - www.linpro.no