From ottolski at web.de Sun Feb 1 19:31:59 2009 From: ottolski at web.de (Sascha Ottolski) Date: Sun, 1 Feb 2009 20:31:59 +0100 Subject: Panic message: Assert error in exp_timer(), cache_expire.c line 303 In-Reply-To: <30248.1233136367@critter.freebsd.dk> References: <30248.1233136367@critter.freebsd.dk> Message-ID: <200902012032.00280.ottolski@web.de> Am Mittwoch 28 Januar 2009 10:52:47 schrieb Poul-Henning Kamp: > In message <200901260917.45223.ottolski at web.de>, Sascha Ottolski writes: > >Assert error in exp_timer(), cache_expire.c line 303: > >Condition(oe2->timer_when >= oe->timer_when) not true. thread = > >(cache-timeout) > > > >[...] > > > >It all happens with trunk, r3497. > > Fixed in r3547 thanks very much. I upgraded to r3563, and now had another crash (after approx. 54 hours runtime): Feb 1 19:54:21 localhost varnishd[3146]: Child (3147) not responding to ping, killing it. Feb 1 19:54:22 localhost varnishd[3146]: Child (3147) died signal=6 Feb 1 19:54:22 localhost varnishd[3146]: Child (3147) Panic message: Assert error in EXP_Rearm(), cache_expire.c line 255: Condition(oe->timer_idx != BINHEAP_NOIDX) not true. thread = (cache-worker)sp = 0x2b1b405ac008 { fd = 3797, id = 3797, xid = 2021543483, client = 217.234.68.98:1869, step = STP_HIT, handling = error, err_code = 200, err_reason = (null), ws = 0x2b1b405ac078 { id = "sess", {s,f,r,e} = {0x2b1b405ac808,,+938,(nil),+16384}, }, worker = 0x59831c80 { }, vcl = { srcname = { "input", "Default", }, }, obj = 0x2ab1ab8b8000 { refcnt = 2, xid = 1987221953, ws = 0x2ab1ab8b8028 { id = "obj", {s,f,r,e} = {0x2ab1ab8b8358,, +241,(nil),+3240}, }, http = { ws = 0x2ab1ab8b8028 { id = "obj", {s,f,r,e} = {0x2ab1ab8b8358,,+241,(nil), +3240}, }, hd = { "Date: Fri, 30 Jan 2009 14:22:59 GMT", "Server: Apache", "X-SF-Stats: stf-img7", "Last-Modified: Thu, Feb 1 19:54:22 localhost varnishd[3146]: Child cleanup complete Please let me know if you nedd additional informations. Cheers, Sascha From matko.bregovic at travellab.com Mon Feb 2 13:27:55 2009 From: matko.bregovic at travellab.com (Matko Bregovic) Date: Mon, 02 Feb 2009 14:27:55 +0100 Subject: Cache dynamic pages based on some (and not all) query parameters? Message-ID: <1233581275.11992.75.camel@rant-e-minor> Hi, I wonder if it's possible to cache pages based on some query parameters. All (GET) query parameters for search.jsp, but 'search.x=', 'search.y=' and 'random=' should be part of the cache key uniqueness. An example URL: search.jsp?homeLeaveMonth=200902&numberOfRooms=1&destLeaveDay=12&isReturn=true&paxAdults=1&homeCity=SAO&search.y=0&search.x=0&view=air&paxKids=0&destLeaveMonth=200902&extPartner=&destCity=PAR&paxTeens=0&specificAirline=all&formActive=true&paxInfants=0&homeLeaveDay=5&random=1906 Another question is if pages can be completely ignored if there's a cookie in place with format: "loggedIn=true". // Matko From phk at phk.freebsd.dk Mon Feb 2 14:20:58 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 02 Feb 2009 14:20:58 +0000 Subject: Cache dynamic pages based on some (and not all) query parameters? In-Reply-To: Your message of "Mon, 02 Feb 2009 14:27:55 +0100." <1233581275.11992.75.camel@rant-e-minor> Message-ID: <1596.1233584458@critter.freebsd.dk> In message <1233581275.11992.75.camel at rant-e-minor>, Matko Bregovic writes: >Hi, > >I wonder if it's possible to cache pages based on some query parameters. > >All (GET) query parameters for search.jsp, but 'search.x=', 'search.y=' >and 'random=' should be part of the cache key uniqueness. You decide your own lookup string in vcl_hash, so just don't use the entire req.url, but only the bits you want. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Mon Feb 2 16:48:56 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 02 Feb 2009 16:48:56 +0000 Subject: Panic message: Assert error in exp_timer(), cache_expire.c line 303 In-Reply-To: Your message of "Sun, 01 Feb 2009 20:31:59 +0100." <200902012032.00280.ottolski@web.de> Message-ID: <5233.1233593336@critter.freebsd.dk> I think I fixed this in r3565. Poul-Henning In message <200902012032.00280.ottolski at web.de>, Sascha Ottolski writes: >Am Mittwoch 28 Januar 2009 10:52:47 schrieb Poul-Henning Kamp: >> In message <200901260917.45223.ottolski at web.de>, Sascha Ottolski >writes: >> >Assert error in exp_timer(), cache_expire.c line 303: >> >Condition(oe2->timer_when >= oe->timer_when) not true. thread = >> >(cache-timeout) >> > >> >[...] >> > >> >It all happens with trunk, r3497. >> >> Fixed in r3547 > >thanks very much. I upgraded to r3563, and now had another crash (after >approx. 54 hours runtime): > > >Feb 1 19:54:21 localhost varnishd[3146]: Child (3147) not responding to >ping, killing it. >Feb 1 19:54:22 localhost varnishd[3146]: Child (3147) died signal=6 >Feb 1 19:54:22 localhost varnishd[3146]: Child (3147) Panic message: >Assert error in EXP_Rearm(), cache_expire.c line 255: >Condition(oe->timer_idx != BINHEAP_NOIDX) not true. thread = >(cache-worker)sp = 0x2b1b405ac008 { fd = 3797, id = 3797, xid = >2021543483, client = 217.234.68.98:1869, step = STP_HIT, handling >= error, err_code = 200, err_reason = (null), ws = 0x2b1b405ac078 { >id = "sess", {s,f,r,e} = {0x2b1b405ac808,,+938,(nil),+16384}, }, >worker = 0x59831c80 { }, vcl = { srcname = >{ "input", "Default", }, }, obj = >0x2ab1ab8b8000 { refcnt = 2, xid = 1987221953, ws = >0x2ab1ab8b8028 { id = "obj", {s,f,r,e} = {0x2ab1ab8b8358,, >+241,(nil),+3240}, }, http = { ws = 0x2ab1ab8b8028 { >id = "obj", {s,f,r,e} = {0x2ab1ab8b8358,,+241,(nil), >+3240}, }, hd = { "Date: Fri, 30 Jan 2009 14:22:59 >GMT", "Server: Apache", "X-SF-Stats: >stf-img7", "Last-Modified: Thu, >Feb 1 19:54:22 localhost varnishd[3146]: Child cleanup complete > > >Please let me know if you nedd additional informations. > > >Cheers, > >Sascha >_______________________________________________ >varnish-misc mailing list >varnish-misc at projects.linpro.no >http://projects.linpro.no/mailman/listinfo/varnish-misc > -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From steven.toensing at artundweise.de Tue Feb 3 11:10:06 2009 From: steven.toensing at artundweise.de (=?ISO-8859-1?Q?Steven_T=F6nsing?=) Date: Tue, 3 Feb 2009 12:10:06 +0100 Subject: Cookie Expiring Date Message-ID: <2B1AD134-AE8B-425A-951A-CB1CAC2A3180@artundweise.de> Hi everyone, I'm just trying to get the vanish 2.0.2 running in front of a cms. The problem is, the user need to be passed to the backend if the "admin" cookie is set and valid. Therefore i'm wondering if there is a way to find out if the expiring date of a cookie is in the future or in the past. My vcl is something like this. if ( (req.http.cookie ~ "MYCOOKIE") && (isExpiringDateOfCookieInFuture) ){ pass; } Thanks Steven From tfheen at redpill-linpro.com Tue Feb 3 14:25:18 2009 From: tfheen at redpill-linpro.com (Tollef Fog Heen) Date: Tue, 03 Feb 2009 15:25:18 +0100 Subject: Cookie Expiring Date In-Reply-To: <2B1AD134-AE8B-425A-951A-CB1CAC2A3180@artundweise.de> ("Steven =?utf-8?Q?T=C3=B6nsing=22's?= message of "Tue, 3 Feb 2009 12:10:06 +0100") References: <2B1AD134-AE8B-425A-951A-CB1CAC2A3180@artundweise.de> Message-ID: <87prhzha29.fsf@qurzaw.linpro.no> ]] Steven T?nsing | Hi everyone, | I'm just trying to get the vanish 2.0.2 running in front of a cms. The | problem is, the user need to be passed to the backend if the "admin" | cookie is set and valid. | Therefore i'm wondering if there is a way to find out if the expiring | date of a cookie is in the future or in the past. If it has expired, the client just won't send it, so just check req.http.cookie for the relevant cookie and you'll be fine. -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 From michael at dynamine.net Tue Feb 3 16:59:07 2009 From: michael at dynamine.net (Michael S. Fischer) Date: Tue, 3 Feb 2009 08:59:07 -0800 Subject: Cookie Expiring Date In-Reply-To: <87prhzha29.fsf@qurzaw.linpro.no> References: <2B1AD134-AE8B-425A-951A-CB1CAC2A3180@artundweise.de> <87prhzha29.fsf@qurzaw.linpro.no> Message-ID: <140D65AC-A883-414D-BE89-5FAF59C71E28@dynamine.net> On Feb 3, 2009, at 6:25 AM, Tollef Fog Heen wrote: > If it has expired, the client just won't send it, so just check > req.http.cookie for the relevant cookie and you'll be fine. I strongly advise against this, as it could subject you to replay attacks. That said, the client does not include an expiration date with the Cookie: header in an HTTP request. You'll have to check the validity of the header on the backend, or modify Varnish to do it for you. --Michael From justin at redwiredesign.com Wed Feb 4 12:06:46 2009 From: justin at redwiredesign.com (Justin Finkelstein) Date: Wed, 04 Feb 2009 12:06:46 +0000 Subject: Disconnection time Message-ID: <1233749206.15900.47.camel@localhost.localdomain> Hi guys I've been using varnish for a little while now and have noticed behaviour that I'd like to alter. When I telnet into varnish to do a manual GET and I hesitate for a couple of seconds, it closes the connection. Is there a way that this delay can be configured or altered? Thanks, Justin -- Redwire Design Limited 54 Maltings Place 169 Tower Bridge Road London SE1 3LJ www.redwiredesign.com [ 020 7403 1444 ] - voice [ 020 7378 8711 ] - fax -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfheen at redpill-linpro.com Wed Feb 4 13:47:16 2009 From: tfheen at redpill-linpro.com (Tollef Fog Heen) Date: Wed, 04 Feb 2009 14:47:16 +0100 Subject: Disconnection time In-Reply-To: <1233749206.15900.47.camel@localhost.localdomain> (Justin Finkelstein's message of "Wed, 04 Feb 2009 12:06:46 +0000") References: <1233749206.15900.47.camel@localhost.localdomain> Message-ID: <87zlh25n6j.fsf@qurzaw.linpro.no> ]] Justin Finkelstein | I've been using varnish for a little while now and have noticed | behaviour that I'd like to alter. | When I telnet into varnish to do a manual GET and I hesitate for a | couple of seconds, it closes the connection. | | Is there a way that this delay can be configured or altered? Set sess_timeout to something greater than 5 (which is the default). -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 From tfheen at redpill-linpro.com Wed Feb 4 14:33:29 2009 From: tfheen at redpill-linpro.com (Tollef Fog Heen) Date: Wed, 04 Feb 2009 15:33:29 +0100 Subject: Disconnection time In-Reply-To: <1233756208.569.4.camel@localhost.localdomain> (Justin Finkelstein's message of "Wed, 04 Feb 2009 14:03:28 +0000") References: <1233749206.15900.47.camel@localhost.localdomain> <87zlh25n6j.fsf@qurzaw.linpro.no> <1233756208.569.4.camel@localhost.localdomain> Message-ID: <87myd25l1i.fsf@qurzaw.linpro.no> ]] Justin Finkelstein (please keep questions and discussions on-list so others can read them too.) | Where do I set this, please? Just add -p sess_timeout=30 in the varnish startup script. (Or telnet to the management port and do param.set sess_timeout 30 -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 From justin at redwiredesign.com Wed Feb 4 14:46:02 2009 From: justin at redwiredesign.com (Justin Finkelstein) Date: Wed, 04 Feb 2009 14:46:02 +0000 Subject: Disconnection time In-Reply-To: <87myd25l1i.fsf@qurzaw.linpro.no> References: <1233749206.15900.47.camel@localhost.localdomain> <87zlh25n6j.fsf@qurzaw.linpro.no> <1233756208.569.4.camel@localhost.localdomain> <87myd25l1i.fsf@qurzaw.linpro.no> Message-ID: <1233758762.569.7.camel@localhost.localdomain> Thanks; can you tell me where I'd find a list of all of these parameters? There doesn't seem to be anything on the wiki about this. On Wed, 2009-02-04 at 15:33 +0100, Tollef Fog Heen wrote: > ]] Justin Finkelstein > > (please keep questions and discussions on-list so others can read them > too.) > > | Where do I set this, please? > > Just add -p sess_timeout=30 in the varnish startup script. (Or telnet > to the management port and do param.set sess_timeout 30 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at automattic.com Wed Feb 4 15:44:29 2009 From: barry at automattic.com (Barry Abrahamson) Date: Wed, 4 Feb 2009 10:44:29 -0500 Subject: is 2.0.2 not as efficient as 1.1.2 was? In-Reply-To: <492C7E1A.1060304@apokalyptik.com> References: <492C7E1A.1060304@apokalyptik.com> Message-ID: <37EADDE4-A23A-4204-B04A-46D47D348D7A@automattic.com> On Nov 25, 2008, at 5:37 PM, Demitrious Kelly wrote: > Hello, > > We run Gravatar.com and use varnish to cache avatar responses. There > are a ton of very small objects and lots of requests per second. Last > week we were using 1.1.2 compiled against tcmalloc (-t 600 -w 1,4000,5 > -h classic,500009 -p thread_pools 10 -p listen_depth 4096 -s > malloc,16G). This used an nginx load balancer on a separate host as > its > back end which distributed varnish's requests to our pool of webs. > All > was well. > > This week we upgraded to 2.0.2 and are using varnish's back end & > director configuration for the same work. What we are seeing is that > 2.0.2 holds about 60% of the objects in the same amount of cache space > as 1.1.2 did (we tried tcmalloc, jemalloc, and mmap.) This caused us > quite a few problems after the upgrade as varnish would start spiking > the load on the boxes into the hundreds. We attempted tuning the > lru_interval (up) and obj_workspace (down) but we couldn't get varnish > to hold the same data that it used to on the same machines. > > Right now we've reduced the time that we keep cached objects > drastically, bringing our cache hit rate down to 92% from 96% which > roughly doubled the requests (and load) on the web servers. It is, > however, stable at this point. Obviously the idea of not keeping up > with the latest versions of varnish is not what we want to do, however > effectively doubling requirements for scaling the service is just as > unappealing. > > So, what we're asking is... how do we get varnish 2 to be as efficient > as varnish 1 was? We're glad to try things... It takes a while to > fill > up the cache to the point that it can cause problems so testing and > reporting back will take some time, but we'd like this fixed and will > put in some work. We're currently running the following cli options: > > -a 0.0.0.0:80 -f ... -P ... -T 10.1.94.43:6969 -t 600 -w 1,4000,5 -h > classic,500009 -p thread_pools 10 -p listen_depth 4096 -s malloc,16G > > And our VCL looks like this (with most of the webs taken out for > brevity > since they're repeated verbatim with only numbers changed) > > backend web11 { .host = "xxx"; .port = "8088"; .probe = > { .url = "xxx"; .timeout = 50 ms; .interval = 5s; > .window = 2; .threshold = 1; } > } > backend web12 { .host = "xxx"; .port = "8088"; .probe = > { .url = "xxx"; .timeout = 50 ms; .interval = 5s; > .window = 2; .threshold = 1; } > } > > director default random { > .retries = 3; > { .backend = web11; .weight = 1; } > { .backend = web12; .weight = 1; } > } > > sub vcl_recv { > set req.backend = default; > set req.grace = 30s; > if ( req.url ~ "^/(avatar|userimage)" && req.http.cookie ) { > lookup; > } > } > > sub vcl_fetch { > if (obj.ttl < 600s) { > set obj.ttl = 600s; > } > if (obj.status == 404) { > set obj.ttl = 30s; > } > if (obj.status == 500 || obj.status == 503 ) { > pass; > } > set obj.grace = 30s; > deliver; > } > > sub vcl_deliver { > remove resp.http.Expires; > remove resp.http.Cache-Control; > set resp.http.Cache-Control = "public, max-age=600, proxy- > revalidate"; > deliver; > } Bump :) Is anyone else seeing the same thing? I think it may be a result of the fact that a lot of the cached responses are just headers (302 redirects) and don't have any actual content. That is the only thing I can think of why we would be seeing this issue and others wouldn't. I suspect most people using varnish dont have stats that look like this: 10094887744 960644.65 847668.80 Total header bytes 22230934332 2174908.58 1866733.93 Total body bytes I don't really want to revert to 1.1.2 because I like the general stability and features of 2.x, but I don't have any real ideas on how to troubleshoot why this would be happening. Any ideas would be appreciated. -- Barry Abrahamson | Systems Wrangler | Automattic Blog: http://barry.wordpress.com From phk at phk.freebsd.dk Wed Feb 4 18:53:43 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 04 Feb 2009 18:53:43 +0000 Subject: is 2.0.2 not as efficient as 1.1.2 was? In-Reply-To: Your message of "Wed, 04 Feb 2009 10:44:29 EST." <37EADDE4-A23A-4204-B04A-46D47D348D7A@automattic.com> Message-ID: <64131.1233773623@critter.freebsd.dk> In message <37EADDE4-A23A-4204-B04A-46D47D348D7A at automattic.com>, Barry Abraham son writes: >> This week we upgraded to 2.0.2 and are using varnish's back end & >> director configuration for the same work. What we are seeing is that >> 2.0.2 holds about 60% of the objects in the same amount of cache space >> as 1.1.2 did (we tried tcmalloc, jemalloc, and mmap.) Your description does not make it obvious to me what is causing this but one candidate could be the stored hash-string, in particular if your URLS are long. The new purge code (likely included in 2.0.3, but already available in -trunk) dispenses with the need to store the hash-string so theory could be tested. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From per.buer at redpill-linpro.com Thu Feb 5 06:05:25 2009 From: per.buer at redpill-linpro.com (Per Andreas Buer) Date: Thu, 5 Feb 2009 07:05:25 +0100 (CET) Subject: Disconnection time In-Reply-To: <1233758762.569.7.camel@localhost.localdomain> Message-ID: <25499071.561233813919764.JavaMail.perbu@thimk> ----- "Justin Finkelstein" skrev: > Thanks; can you tell me where I'd find a list of all of these > parameters? "man vcl". You can also list them in the CLI. -- Per Andreas Buer / Business Area Manager Redpill Linpro - Changing the Game From justin at redwiredesign.com Thu Feb 5 10:31:51 2009 From: justin at redwiredesign.com (Justin Finkelstein) Date: Thu, 05 Feb 2009 10:31:51 +0000 Subject: Disconnection time In-Reply-To: <25499071.561233813919764.JavaMail.perbu@thimk> References: <25499071.561233813919764.JavaMail.perbu@thimk> Message-ID: <1233829911.569.42.camel@localhost.localdomain> Aha! Always read the man pages :o) Thanks! On Thu, 2009-02-05 at 07:05 +0100, Per Andreas Buer wrote: > ----- "Justin Finkelstein" skrev: > > > Thanks; can you tell me where I'd find a list of all of these > > parameters? > > "man vcl". You can also list them in the CLI. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfbustarret at wat.tv Thu Feb 5 11:55:37 2009 From: jfbustarret at wat.tv (BUSTARRET, Jean-francois) Date: Thu, 5 Feb 2009 12:55:37 +0100 Subject: [PATCH] More varnish statistics Message-ID: <53C652A09719C54DA24741D0157CB26902BD0A85@TFPRDEXS1.tf1.groupetf1.fr> Varnish has already many useful statistics, but it lacks a few to be perfect. Here is a patch to add : - n_lru_inserted : number of objets inserted in the LRU (coupled with n_lru_nuked, it can be a nice way of monitoring cache storage usage) - esi_included : number of ESI includes found It also includes esi_parsed, backported from trunk. The patch is against 2.0.2. Hope you will find it useful. Jean-Fran?ois Bustarret WAT - Responsable technique http://www.wat.tv -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: varnish-stats.patch Type: application/octet-stream Size: 2270 bytes Desc: varnish-stats.patch URL: From jfbustarret at wat.tv Thu Feb 5 14:16:41 2009 From: jfbustarret at wat.tv (BUSTARRET, Jean-francois) Date: Thu, 5 Feb 2009 15:16:41 +0100 Subject: Understanding nukes & the expiration thread Message-ID: <53C652A09719C54DA24741D0157CB26902BD0B09@TFPRDEXS1.tf1.groupetf1.fr> If I understand correctly how varnish works : - the expiration thread (exp_timer) only checks the root of the binheap (= the oldest object currently in cache) - a worker needing to free some space in the cache will nuke (EXP_NukeOne) the least recently used object (not taking into account lru_timeout) which means : - exp_timer can be "forever" checking a valid/very high ttl content - EXP_Nuke can often nuke valid (medium traffic/high ttl) content, leaving expired (high traffic/low ttl) untouched Am I right ? I yes, shouldn't be better to have exp_timer look deeper ? Jean-Fran?ois Bustarret WAT - Responsable technique http://www.wat.tv -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeff at funnyordie.com Thu Feb 5 16:35:56 2009 From: jeff at funnyordie.com (Jeff Anderson) Date: Thu, 5 Feb 2009 08:35:56 -0800 Subject: Caching details of objects in varnish... Message-ID: Is there a way in VCL to cache just the base html of a page without its parameters? For example: /advertproviderformat.html?provider=2342342&foo=3434&bar=34213142 Is the entire url cached or just the advertprovider.html portion? It seems the whole url is cached because I observe so many misses for these calls. The parameters are nearly always random so caching the entire url is very inefficient. I've tried in vcl_fetch: if obj.url ~ 'advertproviderformat.html { set obj.ttl = 24h; } But I think it is just caching the entire url which extremely inefficient. How can I just instruct varnish to serve the advertproviderformat.html when it receives one of the full url requests? How can I handle this in VCL? Thanks, --Jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: From des at des.no Thu Feb 5 23:10:03 2009 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Fri, 06 Feb 2009 00:10:03 +0100 Subject: Disconnection time In-Reply-To: <25499071.561233813919764.JavaMail.perbu@thimk> (Per Andreas Buer's message of "Thu, 5 Feb 2009 07:05:25 +0100 (CET)") References: <25499071.561233813919764.JavaMail.perbu@thimk> Message-ID: <86d4dwcwfo.fsf@ds4.des.no> Per Andreas Buer writes: > "Justin Finkelstein" writes: > > Thanks; can you tell me where I'd find a list of all of these > > parameters? > "man vcl". You can also list them in the CLI. No, "man varnishd" DES -- Dag-Erling Sm?rgrav - des at des.no From des at des.no Thu Feb 5 23:11:36 2009 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Fri, 06 Feb 2009 00:11:36 +0100 Subject: Caching details of objects in varnish... In-Reply-To: (Jeff Anderson's message of "Thu, 5 Feb 2009 08:35:56 -0800") References: Message-ID: <868wokcwd3.fsf@ds4.des.no> Jeff Anderson writes: > Is there a way in VCL to cache just the base html of a page without > its parameters? For example: > /advertproviderformat.html?provider=2342342&foo=3434&bar=34213142 Yes, hook into vcl_hash and modify the hash string as appropriate. DES -- Dag-Erling Sm?rgrav - des at des.no From justin at redwiredesign.com Fri Feb 6 10:13:16 2009 From: justin at redwiredesign.com (Justin Finkelstein) Date: Fri, 06 Feb 2009 10:13:16 +0000 Subject: Disconnection time In-Reply-To: <86d4dwcwfo.fsf@ds4.des.no> References: <25499071.561233813919764.JavaMail.perbu@thimk> <86d4dwcwfo.fsf@ds4.des.no> Message-ID: <1233915196.20123.27.camel@localhost.localdomain> That's the one; I found it in the end. It's just a shame that there aren't HTML versions of the man pages in the wiki. If I get some time I think I might write an introductory post about to help clear some of these things up for those of us less sysadmin-y types! On Fri, 2009-02-06 at 00:10 +0100, Dag-Erling Sm?rgrav wrote: > Per Andreas Buer writes: > > "Justin Finkelstein" writes: > > > Thanks; can you tell me where I'd find a list of all of these > > > parameters? > > "man vcl". You can also list them in the CLI. > > No, "man varnishd" > > DES -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeff at funnyordie.com Fri Feb 6 20:01:45 2009 From: jeff at funnyordie.com (Jeff Anderson) Date: Fri, 6 Feb 2009 12:01:45 -0800 Subject: Caching details of objects in varnish... In-Reply-To: <868wokcwd3.fsf@ds4.des.no> References: <868wokcwd3.fsf@ds4.des.no> Message-ID: <4F5A178A-70A0-4447-A81F-0C8AA98A1440@funnyordie.com> Can you please give me a brief code example regarding how to do this in vcl? Thanks, On Feb 5, 2009, at 3:11 PM, Dag-Erling Sm?rgrav wrote: > Jeff Anderson writes: >> Is there a way in VCL to cache just the base html of a page without >> its parameters? For example: >> /advertproviderformat.html?provider=2342342&foo=3434&bar=34213142 > > Yes, hook into vcl_hash and modify the hash string as appropriate. > > DES > -- > Dag-Erling Sm?rgrav - des at des.no --Jeff jeff at funnyordie.com From des at des.no Fri Feb 6 21:22:58 2009 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Fri, 06 Feb 2009 22:22:58 +0100 Subject: Caching details of objects in varnish... In-Reply-To: <4F5A178A-70A0-4447-A81F-0C8AA98A1440@funnyordie.com> (Jeff Anderson's message of "Fri, 6 Feb 2009 12:01:45 -0800") References: <868wokcwd3.fsf@ds4.des.no> <4F5A178A-70A0-4447-A81F-0C8AA98A1440@funnyordie.com> Message-ID: <86y6wjb6q5.fsf@ds4.des.no> Jeff Anderson writes: > Dag-Erling Sm?rgrav writes: > > Jeff Anderson writes: > > > Is there a way in VCL to cache just the base html of a page > > > without its parameters? For example: > > > /advertproviderformat.html?provider=2342342&foo=3434&bar=34213142 > > Yes, hook into vcl_hash and modify the hash string as appropriate. > Can you please give me a brief code example regarding how to do this > in vcl? Read vcl(7); basically, you want to replace the stock vcl_hash (a copy of which is included in the man page) with a version that uses regsub() to extract the relevant portion of the URL. DES -- Dag-Erling Sm?rgrav - des at des.no From jeff at funnyordie.com Fri Feb 6 23:18:14 2009 From: jeff at funnyordie.com (Jeff Anderson) Date: Fri, 6 Feb 2009 15:18:14 -0800 Subject: Caching details of objects in varnish... In-Reply-To: <86y6wjb6q5.fsf@ds4.des.no> References: <868wokcwd3.fsf@ds4.des.no> <4F5A178A-70A0-4447-A81F-0C8AA98A1440@funnyordie.com> <86y6wjb6q5.fsf@ds4.des.no> Message-ID: <258AAA37-B9A9-4C18-8959-E1ED8E9B0D7F@funnyordie.com> Thanks. On Feb 6, 2009, at 1:22 PM, Dag-Erling Sm?rgrav wrote: > Jeff Anderson writes: >> Dag-Erling Sm?rgrav writes: >>> Jeff Anderson writes: >>>> Is there a way in VCL to cache just the base html of a page >>>> without its parameters? For example: >>>> /advertproviderformat.html?provider=2342342&foo=3434&bar=34213142 >>> Yes, hook into vcl_hash and modify the hash string as appropriate. >> Can you please give me a brief code example regarding how to do this >> in vcl? > > Read vcl(7); basically, you want to replace the stock vcl_hash (a copy > of which is included in the man page) with a version that uses > regsub() > to extract the relevant portion of the URL. > > DES > -- > Dag-Erling Sm?rgrav - des at des.no --Jeff jeff at funnyordie.com From phk at phk.freebsd.dk Fri Feb 6 23:32:55 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 06 Feb 2009 23:32:55 +0000 Subject: Understanding nukes & the expiration thread In-Reply-To: Your message of "Thu, 05 Feb 2009 15:16:41 +0100." <53C652A09719C54DA24741D0157CB26902BD0B09@TFPRDEXS1.tf1.groupetf1.fr> Message-ID: <24370.1233963175@critter.freebsd.dk> In message <53C652A09719C54DA24741D0157CB26902BD0B09 at TFPRDEXS1.tf1.groupetf1.fr >, "BUSTARRET, Jean-francois" writes: > >If I understand correctly how varnish works : >- the expiration thread (exp_timer) only checks the root of the binheap >(= the oldest object currently in cache) Not the oldest object, the object closest to expiring. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From per.buer at redpill-linpro.com Mon Feb 9 08:48:36 2009 From: per.buer at redpill-linpro.com (Per Andreas Buer) Date: Mon, 09 Feb 2009 09:48:36 +0100 Subject: Should we keep the man pages? Message-ID: <498FEDE4.7060108@redpill-linpro.com> Hi, There is a lot of confusion regarding the documentation. Some of it is in the wiki and some of it is in the manual. Some of it is missing - but as always, we're working on that. :-) Should we drop the man pages? Should we drop all of them or just "man vcl"? Dropping a man page would men we would incorporate the content into the wiki - maybe adding a few extra ACL's on the pages to restrict editing. From phk at phk.freebsd.dk Mon Feb 9 08:50:49 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 09 Feb 2009 08:50:49 +0000 Subject: Should we keep the man pages? In-Reply-To: Your message of "Mon, 09 Feb 2009 09:48:36 +0100." <498FEDE4.7060108@redpill-linpro.com> Message-ID: <37883.1234169449@critter.freebsd.dk> In message <498FEDE4.7060108 at redpill-linpro.com>, Per Andreas Buer writes: >Hi, > >There is a lot of confusion regarding the documentation. Some of it is >in the wiki and some of it is in the manual. Some of it is missing - but >as always, we're working on that. :-) > >Should we drop the man pages? Should we drop all of them or just "man >vcl"? Dropping a man page would men we would incorporate the content >into the wiki - maybe adding a few extra ACL's on the pages to restrict >editing. I think there should be a manpage for the executables, varnishd, varnishlog, varnishstat etc. But for documenting the overall workings and VCL in particular, manpages suffer badly from the inability to include sensible illustrations. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From justin at redwiredesign.com Mon Feb 9 13:26:04 2009 From: justin at redwiredesign.com (Justin Finkelstein) Date: Mon, 09 Feb 2009 13:26:04 +0000 Subject: Should we keep the man pages? In-Reply-To: <37883.1234169449@critter.freebsd.dk> References: <37883.1234169449@critter.freebsd.dk> Message-ID: <1234185964.20123.63.camel@localhost.localdomain> I agree; man pages are normally used (in my experience) to describe how to access program control instructions, such as parameters, description and exit codes. I think the best approach would be this split, with a static html (or other format) of the wiki documentation included in the distributable archive. On Mon, 2009-02-09 at 08:50 +0000, Poul-Henning Kamp wrote: > In message <498FEDE4.7060108 at redpill-linpro.com>, Per Andreas Buer writes: > >Hi, > > > >There is a lot of confusion regarding the documentation. Some of it is > >in the wiki and some of it is in the manual. Some of it is missing - but > >as always, we're working on that. :-) > > > >Should we drop the man pages? Should we drop all of them or just "man > >vcl"? Dropping a man page would men we would incorporate the content > >into the wiki - maybe adding a few extra ACL's on the pages to restrict > >editing. > > I think there should be a manpage for the executables, varnishd, varnishlog, > varnishstat etc. > > But for documenting the overall workings and VCL in particular, manpages > suffer badly from the inability to include sensible illustrations. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From list at zaide.org Mon Feb 9 17:51:45 2009 From: list at zaide.org (Damien Desmarets) Date: Mon, 09 Feb 2009 18:51:45 +0100 Subject: Configuration question about a domain list for a http.req.host comparaison Message-ID: <49906D31.70302@zaide.org> Hello, I have 1 question for you. I haven't found the response in the documentation/wiki. I think you have a big job to wrtie a good documentation ... the current is really poor ! So the question. I have a list of differents domains: azeazeaze.com eruigeiogr.com zezeirjiorf.com ... My configuration file is actually like this: if (req.http.host ~ "^(azeazeaze.com|eruigeiogr.com|zezeirjiorf.com|...)$") { set req.backend = principal_delivery; } elsif (req.http.host ~ "^(aazeazeazeaze.com|azeiazjeaioze.com|azejoazjeaiozejoaz.com|...)$") { set req.backend = pictures_delivery; } else { error 403 "Not supported" } The regex comparison is very big with maybe 100 domains. It's not clear inside my file, and difficult to made change. Is it possible to include those domains in a list and match this list to the req.http.host. Like this maybe: #define DOMAINS_PRINCIPAL_LIST "domains_principal_list.txt" #define DOMAINS_PICTURES_LIST "domains_pictures_list.txt" if (req.http.host ~ DOMAINS_PINCIPAL_LIST) { set req.backend = principal_delivery; } elsif (req.http.host ~ DOMAINS_PICTURES_LIST) { set req.backend = pictures_delivery; } else { error 403 "Not supported" } I know the constant don't contain regex valid content but it's a pseudo-code to show you the idea. And my file "domains_principal_list.txt" contains for exemple: azeazeaze.com eruigeiogr.com zezeirjiorf.com Is it possible to do this in VCL origin configuration? Thank for your help. Regards, Damien Desmarets From phk at phk.freebsd.dk Mon Feb 9 17:59:14 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 09 Feb 2009 17:59:14 +0000 Subject: Configuration question about a domain list for a http.req.host comparaison In-Reply-To: Your message of "Mon, 09 Feb 2009 18:51:45 +0100." <49906D31.70302@zaide.org> Message-ID: <39771.1234202354@critter.freebsd.dk> In message <49906D31.70302 at zaide.org>, Damien Desmarets writes: >The regex comparison is very big with maybe 100 domains. Presently we have no better mechanism. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From alecshenry at gmail.com Mon Feb 9 19:54:58 2009 From: alecshenry at gmail.com (Alecs Henry) Date: Mon, 9 Feb 2009 17:54:58 -0200 Subject: Weird log entries Message-ID: <3c54843f0902091154i56e4007fy520c50d9fffd92d2@mail.gmail.com> Hi guys, I was wondering if any of you has seen the following log entries when using varnishncsa: 127.0.0.1 - - [00/Jan/1900:00:00:00 +0000] "(null) (null) (null)" (null) - "-" "-" Those are coupled with: 127.0.0.1 - - [09/Feb/2009:19:39:46 +0000] "(null) (null) (null)" 200 39678 "-" "-" I can see an object in the page that has that size (image) -- through firebug, but the object didn't load into the browser until I hit reload. And there are also many entries like the following: 127.0.0.1 - - [09/Feb/2009:19:39:52 +0000] "GET http://www.example.com/img/mod.gif HTTP/1.1" 200 951 " http://www.example.com/style.css" "Mozilla/5.0 (X11; U; Linux x86_64; en; rv:1.9.0.5) Gecko/2008121623 Ubuntu/8.10 (intrepid) Firefox/3.0.5" These entries are from communication with the backend server I expected it to show the backend name instead of 127.0.0.1. Is there an explanation? Thanks! Alecs -------------- next part -------------- An HTML attachment was scrubbed... URL: From nick at loman.net Mon Feb 9 20:20:05 2009 From: nick at loman.net (Nick Loman) Date: Mon, 09 Feb 2009 20:20:05 +0000 Subject: Weird log entries In-Reply-To: <3c54843f0902091154i56e4007fy520c50d9fffd92d2@mail.gmail.com> References: <3c54843f0902091154i56e4007fy520c50d9fffd92d2@mail.gmail.com> Message-ID: <49908FF5.4080501@loman.net> Alecs Henry wrote: > Those are coupled with: > 127.0.0.1 - - [09/Feb/2009:19:39:46 +0000] "(null) (null) (null)" 200 > 39678 "-" "-" > I can see an object in the page that has that size (image) -- through > firebug, but the object didn't load into the browser until I hit reload. I've seen log entries like this for HTTP 1.0 requests (using Varnish 1.2), as that version of Varnish did not have support for parsing HTTP 1.0 headers. Cheers, Nick. From alecshenry at gmail.com Mon Feb 9 20:29:04 2009 From: alecshenry at gmail.com (Alecs Henry) Date: Mon, 9 Feb 2009 18:29:04 -0200 Subject: Weird log entries In-Reply-To: <49908FF5.4080501@loman.net> References: <3c54843f0902091154i56e4007fy520c50d9fffd92d2@mail.gmail.com> <49908FF5.4080501@loman.net> Message-ID: <3c54843f0902091229w2914b616sac6806f9c326167c@mail.gmail.com> Hi Nick, I forgot to say... It's varnish-trunk. But I saw it on both varnish 2.0.1 and 2.0.2. Alecs On Mon, Feb 9, 2009 at 6:20 PM, Nick Loman wrote: > Alecs Henry wrote: > > Those are coupled with: >> 127.0.0.1 - - [09/Feb/2009:19:39:46 +0000] "(null) (null) (null)" 200 >> 39678 "-" "-" >> I can see an object in the page that has that size (image) -- through >> firebug, but the object didn't load into the browser until I hit reload. >> > > I've seen log entries like this for HTTP 1.0 requests (using Varnish 1.2), > as that version of Varnish did not have support for parsing HTTP 1.0 > headers. > > Cheers, > > Nick. > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From per.buer at redpill-linpro.com Mon Feb 9 20:33:40 2009 From: per.buer at redpill-linpro.com (Per Andreas Buer) Date: Mon, 9 Feb 2009 21:33:40 +0100 (CET) Subject: Should we keep the man pages? In-Reply-To: <1827724874.143381234211528019.JavaMail.root@claudius.linpro.no> Message-ID: <790730168.143401234211620890.JavaMail.root@claudius.linpro.no> ----- "Poul-Henning Kamp" wrote: > In message <498FEDE4.7060108 at redpill-linpro.com>, Per Andreas Buer > I think there should be a manpage for the executables, varnishd, > varnishlog, varnishstat etc. > > But for documenting the overall workings and VCL in particular, > manpages suffer badly from the inability to include sensible illustrations. Ok. The first step is done. The content of "man vcl" now resides in the wiki. http://varnish.projects.linpro.no/wiki/VCL - feedback is welcome. As of now, there are no special restrictions on editing this page. -- Redpill Linpro - Changing the Game From plfgoa at gmail.com Tue Feb 10 09:39:48 2009 From: plfgoa at gmail.com (Paras Fadte) Date: Tue, 10 Feb 2009 15:09:48 +0530 Subject: Caching details of objects in varnish... In-Reply-To: References: Message-ID: <75cf5800902100139o3fc56c11gcf4e05b8550bc13f@mail.gmail.com> query stripping ? On Thu, Feb 5, 2009 at 10:05 PM, Jeff Anderson wrote: > Is there a way in VCL to cache just the base html of a page without > its parameters? For example: > /advertproviderformat.html?provider=2342342&foo=3434&bar=34213142 > Is the entire url cached or just the advertprovider.html portion? It seems > the whole url is cached because I observe so many misses for these calls. > The parameters are nearly always random so caching the entire url is > very inefficient. I've tried in vcl_fetch: > if obj.url ~ 'advertproviderformat.html { > set obj.ttl = 24h; > } > But I think it is just caching the entire url which extremely inefficient. > How can I just instruct varnish to serve the advertproviderformat.html when > it receives one of the full url requests? How can I handle this in VCL? > > Thanks, > --Jeff > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc > > From quasirob at googlemail.com Tue Feb 10 12:48:02 2009 From: quasirob at googlemail.com (Rob Ayres) Date: Tue, 10 Feb 2009 12:48:02 +0000 Subject: running varnish-2.0.2 on Sun Message-ID: Hi, hope this is a good place to post this. I have been trying to build Varnish on a Sun (SunOS 5.10 Generic_127111-03 sun4u sparc SUNW,Sun-Fire-V490, gcc version 4.2.1 ) and are experiencing many problems, if anyone has any hints please help. Have built it on freebsd and linux with no problems, unfortunately we need it on the Sun machines! We have built varnish, but upon running it get the following messages : sbin/varnishd -a :8080 -b localhost:8081 C-compiler said: "./vcl.ORk8t3RP.c", line 38: warning: const is a keyword in ANSI C C-compiler said: "./vcl.ORk8t3RP.c", line 38: syntax error before or at: const C-compiler said: "./vcl.ORk8t3RP.c", line 39: syntax error before or at: const C-compiler said: "./vcl.ORk8t3RP.c", line 109: syntax error before or at: const C-compiler said: "./vcl.ORk8t3RP.c", line 110: syntax error before or at: const C-compiler said: "./vcl.ORk8t3RP.c", line 121: syntax error before or at: const C-compiler said: "./vcl.ORk8t3RP.c", line 122: syntax error before or at: const C-compiler said: "./vcl.ORk8t3RP.c", line 124: syntax error before or at: const C-compiler said: "./vcl.ORk8t3RP.c", line 126: syntax error before or at: const C-compiler said: "./vcl.ORk8t3RP.c", line 127: syntax error before or at: const C-compiler said: "./vcl.ORk8t3RP.c", line 139: syntax error before or at: const C-compiler said: "./vcl.ORk8t3RP.c", line 140: syntax error before or at: const C-compiler said: "./vcl.ORk8t3RP.c", line 148: syntax error before or at: const C-compiler said: "./vcl.ORk8t3RP.c", line 153: syntax error before or at: const C-compiler said: "./vcl.ORk8t3RP.c", line 156: syntax error before or at: const C-compiler said: "./vcl.ORk8t3RP.c", line 164: syntax error before or at: const C-compiler said: "./vcl.ORk8t3RP.c", line 168: syntax error before or at: const C-compiler said: "./vcl.ORk8t3RP.c", line 170: syntax error before or at: const C-compiler said: "./vcl.ORk8t3RP.c", line 185: syntax error before or at: const C-compiler said: "./vcl.ORk8t3RP.c", line 191: syntax error before or at: struct C-compiler said: "./vcl.ORk8t3RP.c", line 191: warning: undefined or missing type for: struct C-compiler said: "./vcl.ORk8t3RP.c", line 191: parameter redeclared: const C-compiler said: "./vcl.ORk8t3RP.c", line 191: warning: undefined or missing type for: char C-compiler said: "./vcl.ORk8t3RP.c", line 194: syntax error before or at: const C-compiler said: "./vcl.ORk8t3RP.c", line 194: warning: undefined or missing type for: const C-compiler said: "./vcl.ORk8t3RP.c", line 194: warning: undefined or missing type for: int C-compiler said: "./vcl.ORk8t3RP.c", line 196: syntax error before or at: char C-compiler said: "./vcl.ORk8t3RP.c", line 196: warning: undefined or missing type for: char C-compiler said: "./vcl.ORk8t3RP.c", line 196: warning: undefined or missing type for: void ....much more of this..... C-compiler said: "./vcl.ORk8t3RP.c", line 880: warning: improper pointer/integer combination: op "=" C-compiler said: "./vcl.ORk8t3RP.c", line 886: too many struct/union initializers C-compiler said: cc: acomp failed for ./vcl.ORk8t3RP.c mgt_run_cc(): Compiler failed, exit 2 VCL compilation failed Does anyone have an idea what has caused this? The VCC_CC in config.h is set to: #define VCC_CC "cc -Kpic -G -o %o %s" $ cc -V cc: Sun C 5.9 SunOS_sparc Patch 124867-01 2007/07/12 We also got NaN errors during "make" which were fixed by inserting the following into config.h #ifndef NAN #define NAN 0.0/0.0 #endif Thanks for any help! -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at automattic.com Tue Feb 10 17:22:04 2009 From: barry at automattic.com (Barry Abrahamson) Date: Tue, 10 Feb 2009 12:22:04 -0500 Subject: is 2.0.2 not as efficient as 1.1.2 was? In-Reply-To: <64131.1233773623@critter.freebsd.dk> References: <64131.1233773623@critter.freebsd.dk> Message-ID: <4D484E2D-0553-43D8-84D3-1020E0784E7D@automattic.com> On Feb 4, 2009, at 1:53 PM, Poul-Henning Kamp wrote: > In message <37EADDE4-A23A-4204-B04A-46D47D348D7A at automattic.com>, > Barry Abraham > son writes: > >>> This week we upgraded to 2.0.2 and are using varnish's back end & >>> director configuration for the same work. What we are seeing is >>> that >>> 2.0.2 holds about 60% of the objects in the same amount of cache >>> space >>> as 1.1.2 did (we tried tcmalloc, jemalloc, and mmap.) > > Your description does not make it obvious to me what is causing this > but one candidate could be the stored hash-string, in particular if > your URLS are long. > > The new purge code (likely included in 2.0.3, but already available > in -trunk) dispenses with the need to store the hash-string so theory > could be tested. Upgraded to trunk, didn't help. -- Barry Abrahamson | Systems Wrangler | Automattic Blog: http://barry.wordpress.com From jeff at funnyordie.com Tue Feb 10 17:52:31 2009 From: jeff at funnyordie.com (Jeff Anderson) Date: Tue, 10 Feb 2009 09:52:31 -0800 Subject: Caching details of objects in varnish... In-Reply-To: <75cf5800902100139o3fc56c11gcf4e05b8550bc13f@mail.gmail.com> References: <75cf5800902100139o3fc56c11gcf4e05b8550bc13f@mail.gmail.com> Message-ID: <87235176-3A32-4F1B-88D7-5FCBE9322D49@funnyordie.com> Actually yes query stripping is what I want to do on the inbound request not on the fetch. My mistake for not being clear. On Feb 10, 2009, at 1:39 AM, Paras Fadte wrote: > query stripping ? > > > On Thu, Feb 5, 2009 at 10:05 PM, Jeff Anderson > wrote: >> Is there a way in VCL to cache just the base html of a page without >> its parameters? For example: >> /advertproviderformat.html?provider=2342342&foo=3434&bar=34213142 >> Is the entire url cached or just the advertprovider.html portion? >> It seems >> the whole url is cached because I observe so many misses for these >> calls. >> The parameters are nearly always random so caching the entire url is >> very inefficient. I've tried in vcl_fetch: >> if obj.url ~ 'advertproviderformat.html { >> set obj.ttl = 24h; >> } >> But I think it is just caching the entire url which extremely >> inefficient. >> How can I just instruct varnish to serve the >> advertproviderformat.html when >> it receives one of the full url requests? How can I handle this in >> VCL? >> >> Thanks, >> --Jeff >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at projects.linpro.no >> http://projects.linpro.no/mailman/listinfo/varnish-misc >> >> --Jeff jeff at funnyordie.com From tfheen at redpill-linpro.com Wed Feb 11 07:53:14 2009 From: tfheen at redpill-linpro.com (Tollef Fog Heen) Date: Wed, 11 Feb 2009 08:53:14 +0100 Subject: Configuration question about a domain list for a http.req.host comparaison In-Reply-To: <49906D31.70302@zaide.org> (Damien Desmarets's message of "Mon, 09 Feb 2009 18:51:45 +0100") References: <49906D31.70302@zaide.org> Message-ID: <877i3xjtp1.fsf@qurzaw.linpro.no> ]] Damien Desmarets | I have 1 question for you. I haven't found the response in the | documentation/wiki. I think you have a big job to wrtie a good | documentation ... the current is really poor ! If you want to help out with writing docs or suggest weak points, you are more than welcome. | Is it possible to do this in VCL origin configuration? Not at the moment, no. What I would suggest is you write a small script in your favourite language that generates the VCL for you. -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 From olau at iola.dk Wed Feb 11 12:00:42 2009 From: olau at iola.dk (Ole Laursen) Date: Wed, 11 Feb 2009 12:00:42 +0000 (UTC) Subject: Default behaviour with regards to Cache-Control Message-ID: Hi! Why doesn't Varnish respect Cache-Control: private and Cache-Control: no-cache out of the box? My backend is a Django backend which is mostly sensible about what it sends out. So it's a bit strange that Varnish doesn't just do the right thing unless you configure it to do so. I have a URL which redirects based on the clients IP-address (to auto-select the country/currency). I tried sending out "Vary: *", but that doesn't seem to work. So I ended up with Cache-Control: private just to find out that it didn't work either (this is Varnish 2.0.2). I've put in sub vcl_fetch { if(obj.http.Pragma ~ "no-cache" || obj.http.Cache-Control ~ "no-cache" || obj.http.Cache-Control ~ "private") { pass; } } as hinted in the introduction on the Wiki. Now it works, but I'm curious why this isn't the default? Ole From varnish-lists at andersground.net Wed Feb 11 13:14:22 2009 From: varnish-lists at andersground.net (Florian Gilcher) Date: Wed, 11 Feb 2009 14:14:22 +0100 Subject: ESI include: src, alt, onerror weirdness Message-ID: <354F062B-1013-4CA6-BC1F-2440979593ED@andersground.net> Hi, being rather new to using varnish, I don't want to file this as a bug right away. First of all, my .vcl in use: === backend www { .host = "127.0.0.1"; .port = "4567"; } sub vcl_fetch { esi; } === I have two problems. I have an include-element of the following form: === === First, I (accidentally) forged "/include" to return 404 and no body, which leads to the following output: === === So, the ESI processing stops in the moment that the include is empty string. The scenario second problem is as follows: 1. "/include" returns 404 and the string "some text" as body. 2. "/alt" returns 500 and the string "some other text" as body. Reading the (rather unspecific) ESI Language Specification, I expect that the include-element is deleted silently, because: 1. "src" "is not found". 2. "ESI can fetch neither the src nor the alt" So, I would expect: === === But, much to my surprise, the return is: === some text === It also happens if "/include" returns 500. I checked the headers, they are okay. It happens both with 2.0.2 and trunk, OS is Gentoo Linux and OS X. So, I am beginning to wonder on how esi:include is implemented in varnish or what I am doing wrong. Because - granted - the ESI specification could be interpreted to include the element without giving much thought on what the returned entity actually represents. Then this would be the correct behaviour, as the backend is obviously responding. To potentially save you some work, i'll include my demo script (in ruby/sinatra) and the vcl below. Thank you for your time, Florian Gilcher -- esi.rb: === #!/usr/bin/env ruby # to use this, you need ruby, configured with zlib and rubygems installed # to install sinatra, run "gem install sinatra" require 'rubygems' require 'sinatra' get '/include' do response.status = 500 "some text" end get '/alt' do response.status = 500 "some other text" end get '/' do ' Page Title ' end === esi.vcl === backend www { .host = "127.0.0.1"; .port = "4567"; } sub vcl_fetch { esi; } === From afassl at progis.de Wed Feb 11 12:47:16 2009 From: afassl at progis.de (Andreas Fassl) Date: Wed, 11 Feb 2009 13:47:16 +0100 Subject: Varnish and mp3 streaming on demand Message-ID: <4992C8D4.10705@progis.de> Hi, after reading the docs it looks like I need an apache server to serve the cached mp3 content for streaming on demand. Any experience in configuration of this setup? Any pointer/hint is greatly appreciated. Best Regards Andreas From plfgoa at gmail.com Wed Feb 11 16:02:19 2009 From: plfgoa at gmail.com (Paras Fadte) Date: Wed, 11 Feb 2009 21:32:19 +0530 Subject: Caching details of objects in varnish... In-Reply-To: <87235176-3A32-4F1B-88D7-5FCBE9322D49@funnyordie.com> References: <75cf5800902100139o3fc56c11gcf4e05b8550bc13f@mail.gmail.com> <87235176-3A32-4F1B-88D7-5FCBE9322D49@funnyordie.com> Message-ID: <75cf5800902110802k77b4ff2ct39a752f6bad5fca8@mail.gmail.com> http://varnish.projects.linpro.no/wiki/FAQ On Tue, Feb 10, 2009 at 11:22 PM, Jeff Anderson wrote: > Actually yes query stripping is what I want to do on the inbound request not > on the fetch. My mistake for not being clear. > > > On Feb 10, 2009, at 1:39 AM, Paras Fadte wrote: > >> query stripping ? >> >> >> On Thu, Feb 5, 2009 at 10:05 PM, Jeff Anderson >> wrote: >>> >>> Is there a way in VCL to cache just the base html of a page without >>> its parameters? For example: >>> /advertproviderformat.html?provider=2342342&foo=3434&bar=34213142 >>> Is the entire url cached or just the advertprovider.html portion? It >>> seems >>> the whole url is cached because I observe so many misses for these calls. >>> The parameters are nearly always random so caching the entire url is >>> very inefficient. I've tried in vcl_fetch: >>> if obj.url ~ 'advertproviderformat.html { >>> set obj.ttl = 24h; >>> } >>> But I think it is just caching the entire url which extremely >>> inefficient. >>> How can I just instruct varnish to serve the advertproviderformat.html >>> when >>> it receives one of the full url requests? How can I handle this in VCL? >>> >>> Thanks, >>> --Jeff >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at projects.linpro.no >>> http://projects.linpro.no/mailman/listinfo/varnish-misc >>> >>> > > --Jeff > jeff at funnyordie.com > > > > From phk at phk.freebsd.dk Thu Feb 12 09:04:21 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 12 Feb 2009 09:04:21 +0000 Subject: Default behaviour with regards to Cache-Control In-Reply-To: Your message of "Wed, 11 Feb 2009 12:00:42 GMT." Message-ID: <36046.1234429461@critter.freebsd.dk> In message , Ole Laursen writes: >Why doesn't Varnish respect Cache-Control: private and Cache-Control: no-cache >out of the box? Because we see those as headers you want non-friendly caches to act on, whereas we consider Varnish a friendly cache, under your control. Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From olau at iola.dk Thu Feb 12 09:57:02 2009 From: olau at iola.dk (Ole Laursen) Date: Thu, 12 Feb 2009 09:57:02 +0000 (UTC) Subject: Default behaviour with regards to Cache-Control References: <36046.1234429461@critter.freebsd.dk> Message-ID: Poul-Henning Kamp writes: > In message , Ole Laursen writes: > > >Why doesn't Varnish respect Cache-Control: private and Cache-Control: no-cache > >out of the box? > > Because we see those as headers you want non-friendly caches to act on, > whereas we consider Varnish a friendly cache, under your control. OK, I didn't expect that reply. :-) Is it because some backends send out no-cache and private even when they don't mean it? Otherwise I don't get it. If they do mean it, returning the same object is not safe, e.g. in my case it meant that if someone from Germany had visited the URL, all Danes would be redirected to the euro-as-currency page until Varnish let the object go. I looked up private here http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html and it says Indicates that all or part of the response message is intended for a single user and MUST NOT be cached by a shared cache. This allows an origin server to state that the specified parts of the response are intended for only one user and are not a valid response for requests by other users I'm not saying that Varnish should follow this just because it's in the standard, but I think it should follow it (as default) unless we know that most backends use private the wrong way. Ole From phk at phk.freebsd.dk Thu Feb 12 10:06:35 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 12 Feb 2009 10:06:35 +0000 Subject: Default behaviour with regards to Cache-Control In-Reply-To: Your message of "Thu, 12 Feb 2009 09:57:02 GMT." Message-ID: <1903.1234433195@critter.freebsd.dk> In message , Ole Laursen writes: >Poul-Henning Kamp writes: >I looked up private here > > http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html > >and it says > > Indicates that all or part of the response message is intended > for a single user and MUST NOT be cached by a shared cache. We don't consider varnish a "shared cache" in the RFC2616 sense of the concept, because the varnish instance is fully under the control of the servers administrator, and should therefore be considered part of the server. The fine point here is that you *know* why your headers are the way the are, and can implement the policy you desire in your VCL, whereas a "hostile" shared cache can only look at the headers and do as told, not knowing the underlying reason. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From olau at iola.dk Thu Feb 12 11:20:39 2009 From: olau at iola.dk (Ole Laursen) Date: Thu, 12 Feb 2009 11:20:39 +0000 (UTC) Subject: Default behaviour with regards to Cache-Control References: <1903.1234433195@critter.freebsd.dk> Message-ID: Poul-Henning Kamp writes: > We don't consider varnish a "shared cache" in the RFC2616 sense of > the concept, because the varnish instance is fully under the control > of the servers administrator, and should therefore be considered > part of the server. As I read that part of the RFC, shared simply means shared between more than one user, the key part here is that it's used to signal that it's "intended for only one user and not a valid response for requests by other users". > The fine point here is that you *know* why your headers are the way > the are, and can implement the policy you desire in your VCL, > whereas a "hostile" shared cache can only look at the headers > and do as told, not knowing the underlying reason. Yes, all well, but that's an argument for being able to configure it (which is cool), not really an argument for or against what the default behaviour is. When it comes to cookies, Varnish is as default verging on the side of safety in by-passing the cache. Why wouldn't you do the same to private? Principle of least surprise? Ole From phk at phk.freebsd.dk Thu Feb 12 11:34:59 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 12 Feb 2009 11:34:59 +0000 Subject: Default behaviour with regards to Cache-Control In-Reply-To: Your message of "Thu, 12 Feb 2009 11:20:39 GMT." Message-ID: <1528.1234438499@critter.freebsd.dk> In message , Ole Laursen writes: >Poul-Henning Kamp writes: > >> We don't consider varnish a "shared cache" in the RFC2616 sense of >> the concept, because the varnish instance is fully under the control >> of the servers administrator, and should therefore be considered >> part of the server. > >As I read that part of the RFC, shared simply means shared between more than one >user, the key part here is that it's used to signal that it's "intended for only >one user and not a valid response for requests by other users". If you look *really* carefully through the RFC2616, you will find one reference to server side caches -- which they forgot to remove. In the end, the standards group realized, that a cache under the control of the web-server owner, is just a web-server that picks up its contents with HTTP. The reasons why client-side caches have to be so paranoid and fail back to "don't cache if confused", is that they have no knowledge or ownership of the content. A webserver owner, even if he uses Varnish as his webserver, has knowledge and ownership, and configuring things correctly is his job. >When it comes to cookies, Varnish is as default verging on the side of safety in >by-passing the cache. > >Why wouldn't you do the same to private? Principle of least surprise? Well, if people in general think our defaults should be that way, we can change them, our defaults are whatever the consensus can agree on. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From olau at iola.dk Thu Feb 12 14:05:58 2009 From: olau at iola.dk (Ole Laursen) Date: Thu, 12 Feb 2009 14:05:58 +0000 (UTC) Subject: Default behaviour with regards to Cache-Control References: <1528.1234438499@critter.freebsd.dk> Message-ID: Poul-Henning Kamp writes: > If you look *really* carefully through the RFC2616, you will find one > reference to server side caches -- which they forgot to remove. I get your point (the RFC doesn't apply to Varnish). It wasn't my intention to slam Varnish for standards violation, though, sorry if it came across like that. > Well, if people in general think our defaults should be that way, we > can change them, our defaults are whatever the consensus can agree on. Cool! That'd make one particular webserver owner's job a bit easier, and I suspect many others too. :) Pros: - safer - principle of least surprise - easier management (configure this caching aspect once: in the web app) Cons - ? I guess I should open an issue about it? Ole From tfheen at redpill-linpro.com Thu Feb 12 15:26:43 2009 From: tfheen at redpill-linpro.com (Tollef Fog Heen) Date: Thu, 12 Feb 2009 16:26:43 +0100 Subject: ESI include: src, alt, onerror weirdness In-Reply-To: <354F062B-1013-4CA6-BC1F-2440979593ED@andersground.net> (Florian Gilcher's message of "Wed, 11 Feb 2009 14:14:22 +0100") References: <354F062B-1013-4CA6-BC1F-2440979593ED@andersground.net> Message-ID: <87k57vislo.fsf@qurzaw.linpro.no> ]] Florian Gilcher | So, I am beginning to wonder on how esi:include is implemented in | varnish or what I am doing wrong. Because - granted - the ESI | specification could be interpreted to include the element without | giving much thought on what the returned entity actually represents. | Then this would be the correct behaviour, as the backend is obviously | responding. We don't support onerror, nor alt at the moment. -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 From tfheen at redpill-linpro.com Thu Feb 12 15:27:53 2009 From: tfheen at redpill-linpro.com (Tollef Fog Heen) Date: Thu, 12 Feb 2009 16:27:53 +0100 Subject: running varnish-2.0.2 on Sun In-Reply-To: (Rob Ayres's message of "Tue, 10 Feb 2009 12:48:02 +0000") References: Message-ID: <87fxijisjq.fsf@qurzaw.linpro.no> ]] Rob Ayres | Does anyone have an idea what has caused this? Not really, no. I'm going to get a buildbot slave going on Solaris so we'll hopefully be able to avoid such bugs in the future. If you have found a solution, patches are more than welcome. -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 From olau at iola.dk Thu Feb 12 15:49:23 2009 From: olau at iola.dk (Ole Laursen) Date: Thu, 12 Feb 2009 15:49:23 +0000 (UTC) Subject: Varnish and mp3 streaming on demand References: <4992C8D4.10705@progis.de> Message-ID: Andreas Fassl writes: > after reading the docs it looks like I need an apache server to serve > the cached mp3 content for streaming on demand. > Any experience in configuration of this setup? No, but I have set up a site with videos streamed with a Flash widget. We let the video files pass through the cache unaltered with something like if (req.url ~ "^/media/content") { pass; } and send the request to a lighttpd backend. Lighttpd is easy to setup to serve static files, modest in resource usage for this task and scales very well. The disk cache in the operating system makes sure the files are in memory - it's just a matter of having enough RAM to keep the requested data in memory. For this reason, I don't think there's any point in using Varnish's cache in this kind of setup. Ole From olau at iola.dk Thu Feb 12 15:55:58 2009 From: olau at iola.dk (Ole Laursen) Date: Thu, 12 Feb 2009 15:55:58 +0000 (UTC) Subject: Varnish and mp3 streaming on demand References: <4992C8D4.10705@progis.de> Message-ID: Ole Laursen writes: > just a matter of having enough RAM to keep the requested data in memory. For > this reason, I don't think there's any point in using Varnish's cache in this > kind of setup. In other words, unless you need some other feature of Varnish, you might as well route the requests directly to lighttpd. In my case, we had one server only so Varnish was convenient for multiplexing port 80 in addition to caching the dynamic HTML parts from an Apache web app. Ole From michael at dynamine.net Thu Feb 12 16:49:29 2009 From: michael at dynamine.net (Michael S. Fischer) Date: Thu, 12 Feb 2009 08:49:29 -0800 Subject: Default behaviour with regards to Cache-Control In-Reply-To: <1528.1234438499@critter.freebsd.dk> References: <1528.1234438499@critter.freebsd.dk> Message-ID: <3615ACD2-AA69-452C-A9B5-08510490E895@dynamine.net> On Feb 12, 2009, at 3:34 AM, Poul-Henning Kamp wrote: > Well, if people in general think our defaults should be that way, we > can change them, our defaults are whatever the consensus can agree on. I'm with the OP. Regardless of the finer details of the RFC, if I'm a web developer and I set the "Cache-Control:" header to "private" or "no-cache," I would expect it not to be cached by any midstream proxy, regardless of who controls it. This would be especially true if I worked for a larger organization, where some folks in another country, despite receiving a paycheck signed by the same person as me, may control a proxy layer I'm not even aware of. --Michael From afassl at progis.de Thu Feb 12 17:24:33 2009 From: afassl at progis.de (Andreas Fassl) Date: Thu, 12 Feb 2009 18:24:33 +0100 Subject: Varnish and mp3 streaming on demand In-Reply-To: References: <4992C8D4.10705@progis.de> Message-ID: <49945B51.90407@progis.de> Hi, especially the caching is very important for us, because we want to keep traffic away from the mp3 repository server. So you recommend: Client requests streaming on demand mp3 - lighthttpd does streaming and requests from - varnish as reverse proxy/cache from - mp3 repository Best regards Andreas Ole Laursen schrieb: > Andreas Fassl writes: > >> after reading the docs it looks like I need an apache server to serve >> the cached mp3 content for streaming on demand. >> Any experience in configuration of this setup? >> > > No, but I have set up a site with videos streamed with a Flash widget. We let > the video files pass through the cache unaltered with something like > > if (req.url ~ "^/media/content") { > pass; > } > > and send the request to a lighttpd backend. Lighttpd is easy to setup to serve > static files, modest in resource usage for this task and scales very well. The > disk cache in the operating system makes sure the files are in memory - it's > just a matter of having enough RAM to keep the requested data in memory. For > this reason, I don't think there's any point in using Varnish's cache in this > kind of setup. > > > Ole > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Thu Feb 12 18:18:32 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 12 Feb 2009 18:18:32 +0000 Subject: Varnish and mp3 streaming on demand In-Reply-To: Your message of "Thu, 12 Feb 2009 18:24:33 +0100." <49945B51.90407@progis.de> Message-ID: <1318.1234462712@critter.freebsd.dk> In message <49945B51.90407 at progis.de>, Andreas Fassl writes: >Hi, especially the caching is very important for us, because we want to >keep traffic away from the mp3 repository server. >So you recommend: >Client requests streaming on demand mp3 >- lighthttpd does streaming and requests from >- varnish as reverse proxy/cache from >- mp3 repository For your information: One of the changes I'm working on right now is to move vcl_fetch{} up to after receiving the headers from the backend, but before dealing with the backend. This is necessary to be able to decide, per object, if it should be stored in temporary (malloc) or persistent (disk) storage. With some extra work, this will allow pass to become streaming. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From varnish-lists at andersground.net Fri Feb 13 08:57:17 2009 From: varnish-lists at andersground.net (Florian Gilcher) Date: Fri, 13 Feb 2009 09:57:17 +0100 Subject: ESI include: src, alt, onerror weirdness In-Reply-To: <87zlgqd9mr.fsf@qurzaw.linpro.no> References: <354F062B-1013-4CA6-BC1F-2440979593ED@andersground.net> <87k57vislo.fsf@qurzaw.linpro.no> <5AF5271B-39F9-4FC8-8928-014C29A126D0@andersground.net> <87zlgqd9mr.fsf@qurzaw.linpro.no> Message-ID: <7151C7A0-A3F7-4A56-8D9A-2F4913E3B2FC@andersground.net> On Feb 13, 2009, at 9:27 AM, Tollef Fog Heen wrote: > ]] Florian Gilcher > > | > | How about the unexpected break when retrieving an empty document? > > That's weird. Can you try again using 2.0.3? Same issue. I'll report it. > | [1]: And undocumented. The documentation sadly leads to believe that > | include is implemented (as in: full, not half). Do you prefer if I > | file a bug or apply for the magic wiki bit and do it on my own? > > Just tell me your wiki user name and I'll add the bit. Skade Sorry for replying off-list by accident. Regards, Florian Gilcher From per.buer at redpill-linpro.com Fri Feb 13 09:02:21 2009 From: per.buer at redpill-linpro.com (Per Andreas Buer) Date: Fri, 13 Feb 2009 10:02:21 +0100 (CET) Subject: ESI include: src, alt, onerror weirdness In-Reply-To: <7151C7A0-A3F7-4A56-8D9A-2F4913E3B2FC@andersground.net> Message-ID: <1023923573.72191234515741019.JavaMail.root@claudius.linpro.no> ----- "Florian Gilcher" wrote: > Skade Added to the "wiki" group. -- Redpill Linpro - Changing the Game From justin at redwiredesign.com Fri Feb 13 10:07:05 2009 From: justin at redwiredesign.com (Justin Finkelstein) Date: Fri, 13 Feb 2009 10:07:05 +0000 Subject: Varnish and virtual hosting Message-ID: <1234519625.24088.12.camel@localhost.localdomain> Hi guys I'm fairly new to Varnish and it's goodness and not that brilliant a sysadmin (I'm a developer, really) and I have some questions about how to go about setting up Varnish in a virtual hosting environment. I've had a lot of trouble combining the technologies I use for non-proxied hosting with Varnish and what I'd like to do is start a thread to get other people's experiences then combined all this into a bit of documentation for the wiki. So first, when I host a website in Apache, I use it's log piping capability to parse the output of its access logs for each virtual host through Cronolog, so that we can have one log file per day, organised in folders of hostname/year/month. We then chuck all this data at awstats to generate traffic logs based on this data. When I was trying to set this up to work with varnish, I ran into a couple of problems: 1. There didn't seem to be a way to seperate out log data for each virtual host using varnishncsa. 2. There doesn't seem to be a way to get log data out of varnish in a 'live' fashion or in a historical way; meaning (from my view) that if the server crashes of varnish fails, I have to do some work to get the log data out. 3. There doesn't seem to be a way to process live varnishncsa data via a pipe. If this were possible, then I think I would find this whole process a lot easier. Instead, it seems there are two options: to historically dump log data via an option, or run as a daemon and send output to a file (which cannot be via a pipe). I've read the respective man pages for all of the various applications several times and couldn't find a clear solution, so if anyone on this list has been through something similar, or has any ideas on the subject, please contribute them and I'll write the whole thing up as a kind of how-to. Thanks, Justin -- Redwire Design Limited 54 Maltings Place 169 Tower Bridge Road London SE1 3LJ www.redwiredesign.com [ 020 7403 1444 ] - voice [ 020 7378 8711 ] - fax -------------- next part -------------- An HTML attachment was scrubbed... URL: From olau at iola.dk Fri Feb 13 10:08:38 2009 From: olau at iola.dk (Ole Laursen) Date: Fri, 13 Feb 2009 10:08:38 +0000 (UTC) Subject: Varnish and mp3 streaming on demand References: <4992C8D4.10705@progis.de> <49945B51.90407@progis.de> Message-ID: Andreas Fassl writes: > Hi, especially the caching is very important for us, because we want to > keep traffic away from the mp3 repository server. > So you recommend: > Client requests streaming on demand mp3 > - lighthttpd does streaming and requests from > - varnish as reverse proxy/cache from > - mp3 repository Well, I'd setup lighttpd on the MP3 repository server to serve the MP3 files. So client -> varnish cache server -> lighttpd on mp3 repository server However, why do you want to keep traffic away from the MP3 repository? If the server can handle it, letting clients talk directly to it is clearly the most efficient way of serving the files. If you stream MP3, it's probably < 256 kbit. If you can get 1 Gbit out of your repository server, we're talking maybe 3000-4000 simultaneous listeners at 256 kbit. That sounds like a lot. I guess it makes sense if you're a really large radio station or music site and you need to scale up beyond that single server. You could then setup several Varnish servers and have them cache the files automatically instead of distributing the files manually over the nodes. Hmmm. Ole From quasirob at googlemail.com Fri Feb 13 10:17:10 2009 From: quasirob at googlemail.com (Rob Ayres) Date: Fri, 13 Feb 2009 10:17:10 +0000 Subject: running varnish-2.0.2 on Sun In-Reply-To: <87fxijisjq.fsf@qurzaw.linpro.no> References: <87fxijisjq.fsf@qurzaw.linpro.no> Message-ID: 2009/2/12 Tollef Fog Heen > ]] Rob Ayres > > | Does anyone have an idea what has caused this? > > Not really, no. I'm going to get a buildbot slave going on Solaris so > we'll hopefully be able to avoid such bugs in the future. If you have > found a solution, patches are more than welcome. > > We set VCC_CC in config.h to be gcc rather than cc and now it compiles the VCL file with no problem. -------------- next part -------------- An HTML attachment was scrubbed... URL: From olau at iola.dk Fri Feb 13 10:19:52 2009 From: olau at iola.dk (Ole Laursen) Date: Fri, 13 Feb 2009 10:19:52 +0000 (UTC) Subject: Varnish and mp3 streaming on demand References: <49945B51.90407@progis.de> <1318.1234462712@critter.freebsd.dk> Message-ID: Poul-Henning Kamp writes: > This is necessary to be able to decide, per object, if it should be > stored in temporary (malloc) or persistent (disk) storage. > > With some extra work, this will allow pass to become streaming. Right now pass in vcl_recv is streaming, right? You're only talking about if the object is entered into the cache? Or both cases? Ole From phk at phk.freebsd.dk Fri Feb 13 10:30:52 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 13 Feb 2009 10:30:52 +0000 Subject: Varnish and mp3 streaming on demand In-Reply-To: Your message of "Fri, 13 Feb 2009 10:19:52 GMT." Message-ID: <49407.1234521052@critter.freebsd.dk> In message , Ole Laursen writes: >Poul-Henning Kamp writes: > >> This is necessary to be able to decide, per object, if it should be >> stored in temporary (malloc) or persistent (disk) storage. >> >> With some extra work, this will allow pass to become streaming. > >Right now pass in vcl_recv is streaming, right? >You're only talking about if the >object is entered into the cache? Or both cases? No, we never stream pass, the current design is aimed at freeing up the backend as fast as possible, and we only start to transmit the object to the client once we have all of it. This is the same for both caching and passing. pipe is, by nature of how it works, streaming. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From olau at iola.dk Fri Feb 13 10:37:22 2009 From: olau at iola.dk (Ole Laursen) Date: Fri, 13 Feb 2009 10:37:22 +0000 (UTC) Subject: Varnish and mp3 streaming on demand References: <49407.1234521052@critter.freebsd.dk> Message-ID: Poul-Henning Kamp writes: > >Right now pass in vcl_recv is streaming, right? > >You're only talking about if the > >object is entered into the cache? Or both cases? > > No, we never stream pass, the current design is aimed at freeing > up the backend as fast as possible, and we only start to transmit > the object to the client once we have all of it. OK, thanks! That's good to know. Also that you're fixing it. :) Ole From des at des.no Fri Feb 13 12:54:14 2009 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Fri, 13 Feb 2009 13:54:14 +0100 Subject: Default behaviour with regards to Cache-Control In-Reply-To: (Ole Laursen's message of "Thu, 12 Feb 2009 09:57:02 +0000 (UTC)") References: <36046.1234429461@critter.freebsd.dk> Message-ID: <86r622zedl.fsf@ds4.des.no> Ole Laursen writes: > I looked up private here > > http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html > > and it says > > Indicates that all or part of the response message is intended > for a single user and MUST NOT be cached by a shared cache. This > allows an origin server to state that the specified parts of the > response are intended for only one user and are not a valid > response for requests by other users Varnish is not a shared cache, it's a surrogate (not covered by RFC2616) DES -- Dag-Erling Sm?rgrav - des at des.no From des at des.no Fri Feb 13 13:00:47 2009 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Fri, 13 Feb 2009 14:00:47 +0100 Subject: Varnish and mp3 streaming on demand In-Reply-To: <49407.1234521052@critter.freebsd.dk> (Poul-Henning Kamp's message of "Fri, 13 Feb 2009 10:30:52 +0000") References: <49407.1234521052@critter.freebsd.dk> Message-ID: <86mycqze2o.fsf@ds4.des.no> "Poul-Henning Kamp" writes: > No, we never stream pass, the current design is aimed at freeing > up the backend as fast as possible, and we only start to transmit > the object to the client once we have all of it. Perhaps it is time to consider adding a third option: "stream", which streams the current request, then reverts to normal operation. DES -- Dag-Erling Sm?rgrav - des at des.no From des at des.no Fri Feb 13 13:05:43 2009 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Fri, 13 Feb 2009 14:05:43 +0100 Subject: Varnish and virtual hosting In-Reply-To: <1234519625.24088.12.camel@localhost.localdomain> (Justin Finkelstein's message of "Fri, 13 Feb 2009 10:07:05 +0000") References: <1234519625.24088.12.camel@localhost.localdomain> Message-ID: <86iqnezdug.fsf@ds4.des.no> Justin Finkelstein writes: > 1. There didn't seem to be a way to seperate out log data for each > virtual host using varnishncsa. Correct. This has been discussed in the past, search the list archives. > 2. There doesn't seem to be a way to get log data out of varnish in > a 'live' fashion or in a historical way; meaning (from my view) that if > the server crashes of varnish fails, I have to do some work to get the > log data out. Wrong. Read the varnishlog and varnishncsa man pages. > 3. There doesn't seem to be a way to process live varnishncsa data > via a pipe. If this were possible, then I think I would find this whole > process a lot easier. Instead, it seems there are two options: to > historically dump log data via an option, or run as a daemon and send > output to a file (which cannot be via a pipe). Wrong. Both varnishlog and varnishncsa will, by default, process live data and print it to stdout, where you can pipe it to whatever you like. DES -- Dag-Erling Sm?rgrav - des at des.no From olau at iola.dk Fri Feb 13 13:36:00 2009 From: olau at iola.dk (Ole Laursen) Date: Fri, 13 Feb 2009 13:36:00 +0000 (UTC) Subject: Default behaviour with regards to Cache-Control References: <36046.1234429461@critter.freebsd.dk> <86r622zedl.fsf@ds4.des.no> Message-ID: Dag-Erling Sm?rgrav writes: > Varnish is not a shared cache, it's a surrogate (not covered by RFC2616) The only reason I quoted the RFC was to explain why I was surprised that my web app broke: Cache-Control: private is a strong hint that the object cannot be shared between users. Ole From ric at digitalmarbles.com Fri Feb 13 13:45:49 2009 From: ric at digitalmarbles.com (Ricardo Newbery) Date: Fri, 13 Feb 2009 05:45:49 -0800 Subject: [varnish] Re: Default behaviour with regards to Cache-Control In-Reply-To: <86r622zedl.fsf@ds4.des.no> References: <36046.1234429461@critter.freebsd.dk> <86r622zedl.fsf@ds4.des.no> Message-ID: On Feb 13, 2009, at 4:54 AM, Dag-Erling Sm?rgrav wrote: > Ole Laursen writes: >> I looked up private here >> >> http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html >> >> and it says >> >> Indicates that all or part of the response message is intended >> for a single user and MUST NOT be cached by a shared cache. This >> allows an origin server to state that the specified parts of the >> response are intended for only one user and are not a valid >> response for requests by other users > > Varnish is not a shared cache, it's a surrogate (not covered by > RFC2616) > > DES Speaking of which... It would be super handy if Varnish supported Surrogate-Control. I sympathize with the Varnish developers who feel that everything can be done in vcl but some of us really do need the backend to control the surrogate cache behavior on a more granular level. Ric From afassl at progis.de Fri Feb 13 14:37:13 2009 From: afassl at progis.de (Andreas Fassl (proGIS)) Date: Fri, 13 Feb 2009 15:37:13 +0100 Subject: AW: Varnish and mp3 streaming on demand Message-ID: <1713457020.3791234535833476.JavaMail.root@postoffice.progis.de> If I use varnish as cache between mp3 repository and lighttpd, lighttpd will stream the cached contents, that is sufficient for us. If varnish could stream, that would be fine, but I don't think that it is part of the original concept, or? ----- Originalnachricht ----- Von: Dag-Erling Sm?rgrav Gesendet: Fre, 13.2.2009 14:00 An: Poul-Henning Kamp Cc: Ole Laursen ; varnish-misc at projects.linpro.no Betreff: Re: Varnish and mp3 streaming on demand "Poul-Henning Kamp" writes: > No, we never stream pass, the current design is aimed at freeing > up the backend as fast as possible, and we only start to transmit > the object to the client once we have all of it. Perhaps it is time to consider adding a third option: "stream", which streams the current request, then reverts to normal operation. DES -- Dag-Erling Sm?rgrav - des at des.no _______________________________________________ varnish-misc mailing list varnish-misc at projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc From kitai at ya.com Fri Feb 13 13:49:21 2009 From: kitai at ya.com (Kitai) Date: Fri, 13 Feb 2009 14:49:21 +0100 Subject: Varnish and virtual hosting In-Reply-To: <86iqnezdug.fsf@ds4.des.no> References: <1234519625.24088.12.camel@localhost.localdomain> <86iqnezdug.fsf@ds4.des.no> Message-ID: <48d8853d0902130549x532d6000y52e7b4d254ce7d26@mail.gmail.com> > >> 3. There doesn't seem to be a way to process live varnishncsa data >> via a pipe. If this were possible, then I think I would find this whole >> process a lot easier. Instead, it seems there are two options: to >> historically dump log data via an option, or run as a daemon and send >> output to a file (which cannot be via a pipe). > > Wrong. Both varnishlog and varnishncsa will, by default, process live > data and print it to stdout, where you can pipe it to whatever you like. > Agree. You can post-process varnishncsa data to any post-processing program you want. Or, for example, using Perl ..you may launch the program and take the output directly to the files you want. #!/usr/bin/perl open (COMMAND, "varnishncsa | ") or die "Could not open varnishncsa...\n"; while () { # bla bla bla print $_; # Print actual line } close COMMAND; Every request is logged with full DN in the URL. 62.151.2.42 - - [29/Jan/2009:16:56:16 +0100] "GET http://www.FULLDOMAIN.info/static HTTP/1.0" 503 466 "referer" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)" So it's not difficult to make a program to separate logs depending on your needs. Btw, a program in C made by varnish programmers to accomplish this may be greatly appreciated by the community ;-) Kitai From list at zaide.org Fri Feb 13 16:34:40 2009 From: list at zaide.org (Damien Desmarets) Date: Fri, 13 Feb 2009 17:34:40 +0100 Subject: Configuration question about a domain list for a http.req.host comparaison In-Reply-To: <877i3xjtp1.fsf@qurzaw.linpro.no> References: <49906D31.70302@zaide.org> <877i3xjtp1.fsf@qurzaw.linpro.no> Message-ID: <4995A120.4030803@zaide.org> Tollef Fog Heen a ?crit : > ]] Damien Desmarets > > | I have 1 question for you. I haven't found the response in the > | documentation/wiki. I think you have a big job to wrtie a good > | documentation ... the current is really poor ! > > If you want to help out with writing docs or suggest weak points, you > are more than welcome. > Yes of course, but to begin I need a good documentation to have "solids foundations"! And after this, be sure i will post modification/add in your wiki documentation. > | Is it possible to do this in VCL origin configuration? > > Not at the moment, no. What I would suggest is you write a small script > in your favourite language that generates the VCL for you. > > Yes sure, and modify the init.d script ... or I can try to add this particularity inside libvcl directly. I'm going to see the difficulty to implement this. Thank for the support, Damien Desmarets From armando.singer at gmail.com Sat Feb 14 00:35:06 2009 From: armando.singer at gmail.com (Armando Singer) Date: Fri, 13 Feb 2009 16:35:06 -0800 Subject: running varnish-2.0.2 on Sun In-Reply-To: References: <87fxijisjq.fsf@qurzaw.linpro.no> Message-ID: <6f201f470902131635w6b62fa72n90d4dcc167578af1@mail.gmail.com> >> | Does anyone have an idea what has caused this? >> >> Not really, no. I'm going to get a buildbot slave going on Solaris so >> we'll hopefully be able to avoid such bugs in the future. If you have >> found a solution, patches are more than welcome. >> > We set VCC_CC in config.h to be gcc rather than cc and now it compiles the > VCL file with no problem. Or, if using Sun Studio Compiler (I'm using Sun Studio Express 11/08) Change the following: #define VCC_CC "cc -Kpic -G -o %o %s" to: #define VCC_CC "cc -m64 -mt -Kpic -G -o %o %s" For 64 bit architectures -m64 must be set. This should probably be default. Not sure if compiled VCL is multithreaded, but i put in -mt. You might want to put -fast in there as well. Cheers, Armando From olau at iola.dk Mon Feb 16 13:21:38 2009 From: olau at iola.dk (Ole Laursen) Date: Mon, 16 Feb 2009 13:21:38 +0000 (UTC) Subject: Varnish and mp3 streaming on demand References: <4992C8D4.10705@progis.de> <49945B51.90407@progis.de> Message-ID: Ole Laursen writes: > Andreas Fassl writes: > > Hi, especially the caching is very important for us, because we want to > > keep traffic away from the mp3 repository server. > > So you recommend: > > Client requests streaming on demand mp3 > > - lighthttpd does streaming and requests from > > - varnish as reverse proxy/cache from > > - mp3 repository I reread your original post, and I can see know that you're probably referring to "Range header support" on http://varnish.projects.linpro.no/wiki/PostTwoShoppingList which is presumably missing. So forget the last thing I said. Ole From jyri at virkki.com Wed Feb 18 07:33:42 2009 From: jyri at virkki.com (Jyri J. Virkki) Date: Tue, 17 Feb 2009 23:33:42 -0800 Subject: running varnish-2.0.2 on Sun In-Reply-To: References: Message-ID: <20090218073342.GA11446@localhost.localdomain> Once upon a time Rob Ayres wrote: > > Hi, hope this is a good place to post this. I have been trying to build > Varnish on a Sun (SunOS 5.10 Generic_127111-03 sun4u sparc > SUNW,Sun-Fire-V490, gcc version 4.2.1 ) and are experiencing many problems, > if anyone has any hints please help. Have built it on freebsd and linux with > no problems, unfortunately we need it on the Sun machines! FYI I've packaged 2.0.2 for OpenSolaris and hosting it experimentally at a repository at http://pkg.opensolaris.org/webstack/en/catalog.shtml in case it helps. I hadn't tried on Solaris 10 for a while but just attempted a build from the same sources on S10 and saw no issues (but haven't tested the bits). I build with Sun Studio not gcc. -- Jyri J. Virkki - Santa Cruz, CA -- From liukaixuan at gmail.com Tue Feb 24 02:54:55 2009 From: liukaixuan at gmail.com (=?GB2312?B?wfW/qtD+?=) Date: Tue, 24 Feb 2009 10:54:55 +0800 Subject: How to rotate varnish log files with cronolog? Message-ID: <73bbf2560902231854o17e05913x773ad5eaa0eb8e6b@mail.gmail.com> Does Varnish support cronolog? I have searched over Google and the archived maillist for a whole day, but find nothing useful. My Varnish is the lastest 2.0.3. I have tried this: /opt/varnish203/bin/varnishncsa -a -w |/usr/bin/cronolog /logs/mysite_access_%Y%m%d.log /opt/varnish203/bin/varnishncsa: option requires an argument -- w usage: varnishncsa [-bCcd] [-i tag] [-I regexp] [-k keep] [-r file] [-s skip] [-X regexp] [-x tag] [-aDV] [-n varnish_name] [-P file] [-w file] Thanks in advance for any help, Louis From tfheen at redpill-linpro.com Tue Feb 24 11:25:58 2009 From: tfheen at redpill-linpro.com (Tollef Fog Heen) Date: Tue, 24 Feb 2009 12:25:58 +0100 Subject: How to rotate varnish log files with cronolog? In-Reply-To: <73bbf2560902231854o17e05913x773ad5eaa0eb8e6b@mail.gmail.com> (=?utf-8?B?IuWImAnlvIDnjoQiJ3M=?= message of "Tue, 24 Feb 2009 10:54:55 +0800") References: <73bbf2560902231854o17e05913x773ad5eaa0eb8e6b@mail.gmail.com> Message-ID: <87d4d8yt2x.fsf@qurzaw.linpro.no> ]] ??? | Does Varnish support cronolog? I have searched over Google and the | archived maillist for a whole day, but find nothing useful. My Varnish | is the lastest 2.0.3. I'm not sure how cronolog works, but it seems like it takes log files from stdin and handles the rotation automatically. Just use ?/opt/varnish203/bin/varnishncsa |/usr/bin/cronolog /logs/mysite_access_%Y%m%d.log? and you should be fine. -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 From gaute at pht.no Wed Feb 25 08:02:24 2009 From: gaute at pht.no (Gaute Amundsen) Date: Wed, 25 Feb 2009 09:02:24 +0100 Subject: varnishadm purge.list broken? Message-ID: <200902250902.24352.gaute@pht.no> Red Hat Enterprise Linux ES release 4 2.6.9-67.ELsmp varnish-2.0.3 # varnishadm -T localhost:90 purge.list An error occured in receiving answer. Doing a few purges both by hash and url to fill the list makes no difference. Is this supposed to work? Anything I should try? We are upgrading servers soon, will a newer distribution make a difference? ( to ubuntu server jeos 8.04 ) Regards Gaute Amundsen From des at des.no Wed Feb 25 16:30:59 2009 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Wed, 25 Feb 2009 17:30:59 +0100 Subject: varnishadm purge.list broken? In-Reply-To: <200902250902.24352.gaute@pht.no> (Gaute Amundsen's message of "Wed, 25 Feb 2009 09:02:24 +0100") References: <200902250902.24352.gaute@pht.no> Message-ID: <86ljrueaws.fsf@ds4.des.no> Gaute Amundsen writes: > # varnishadm -T localhost:90 purge.list > An error occured in receiving answer. There is no "purge.list" command. DES -- Dag-Erling Sm?rgrav - des at des.no From gaute at pht.no Wed Feb 25 17:24:18 2009 From: gaute at pht.no (Gaute Amundsen) Date: Wed, 25 Feb 2009 18:24:18 +0100 Subject: varnishadm purge.list broken? In-Reply-To: <86ljrueaws.fsf@ds4.des.no> References: <200902250902.24352.gaute@pht.no> <86ljrueaws.fsf@ds4.des.no> Message-ID: <200902251824.18943.gaute@pht.no> On Wednesday 25 February 2009 17:30:59 Dag-Erling Sm?rgrav wrote: > Gaute Amundsen writes: > > # varnishadm -T localhost:90 purge.list > > An error occured in receiving answer. > > There is no "purge.list" command. > > DES # varnishadm -T localhost:90 help help [command] purge.url purge.hash purge [&& ]... purge.list # varnishadm -T localhost:90 help purge.list purge.list List the active purges. # grep -ir 'purge.list' varnish-2.0.3/* | wc -l 26 No? G. From des at des.no Wed Feb 25 18:55:54 2009 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Wed, 25 Feb 2009 19:55:54 +0100 Subject: varnishadm purge.list broken? In-Reply-To: <200902251824.18943.gaute@pht.no> (Gaute Amundsen's message of "Wed, 25 Feb 2009 18:24:18 +0100") References: <200902250902.24352.gaute@pht.no> <86ljrueaws.fsf@ds4.des.no> <200902251824.18943.gaute@pht.no> Message-ID: <86wsbecpmt.fsf@ds4.des.no> Gaute Amundsen writes: > Dag-Erling Sm?rgrav writes: > > There is no "purge.list" command. > No? Sorry, I was looking at the parent commands; purge.list is a child command. You didn't mention which version you're running, but it looks like a varnishadm or cli bug. You shouldn't get "an error occured in receiving answer" when the list is empty - you should get an empty result. Have you tried connecting directly with telnet? DES -- Dag-Erling Sm?rgrav - des at des.no From gaute at pht.no Wed Feb 25 21:18:49 2009 From: gaute at pht.no (Gaute Amundsen) Date: Wed, 25 Feb 2009 22:18:49 +0100 Subject: varnishadm purge.list broken? In-Reply-To: <86wsbecpmt.fsf@ds4.des.no> References: <200902250902.24352.gaute@pht.no> <200902251824.18943.gaute@pht.no> <86wsbecpmt.fsf@ds4.des.no> Message-ID: <200902252218.50112.gaute@pht.no> On Wednesday 25 February 2009 19:55:54 Dag-Erling Sm?rgrav wrote: > Gaute Amundsen writes: > > Dag-Erling Sm?rgrav writes: > > > There is no "purge.list" command. > > > > No? > > Sorry, I was looking at the parent commands; purge.list is a child > command. > > You didn't mention which version you're running, but it looks like a > varnishadm or cli bug. You shouldn't get "an error occured in receiving > answer" when the list is empty - you should get an empty result. Have > you tried connecting directly with telnet? > > DES I mentioned varnish-2.0.3, i don't think varnishadm has it's ow version number? Telnet works! I thought I had tried it.. Returns a _massive_ list. Any ref. on how to interpret what I see? Hey! Wait! This time of night I can restart varnish and guess what? varnishadm purge.list started working! Must have been something in that list. We pass 90% of traffic here, so far. Perhaps related? another matter: purge [&& ]... I can't seem to find any info on how to use this. Gaute -- Programmerer - Pixelhospitalet AS Prinsessealleen 50, 0276 Oslo Tlf. 24 12 97 81 - 9074 7344 From liukaixuan at gmail.com Thu Feb 26 02:24:25 2009 From: liukaixuan at gmail.com (=?GB2312?B?wfW/qtD+?=) Date: Thu, 26 Feb 2009 10:24:25 +0800 Subject: How to rotate varnish log files with cronolog? In-Reply-To: <87d4d8yt2x.fsf@qurzaw.linpro.no> References: <73bbf2560902231854o17e05913x773ad5eaa0eb8e6b@mail.gmail.com> <87d4d8yt2x.fsf@qurzaw.linpro.no> Message-ID: <73bbf2560902251824ref3a9c4jd8eecb1f46fd280b@mail.gmail.com> It works. thank you, it really help me out! On Tue, Feb 24, 2009 at 7:25 PM, Tollef Fog Heen wrote: > > ]] ??? > > | Does Varnish support cronolog? I have searched over Google and the > | archived maillist for a whole day, but find nothing useful. My Varnish > | is the lastest 2.0.3. > > I'm not sure how cronolog works, but it seems like it takes log files > from stdin and handles the rotation automatically. > > Just use <> > and you should be fine. > > -- > Tollef Fog Heen > Redpill Linpro -- Changing the game! > t: +47 21 54 41 73 From list at zaide.org Thu Feb 26 15:57:39 2009 From: list at zaide.org (Damien Desmarets) Date: Thu, 26 Feb 2009 16:57:39 +0100 Subject: How to enable HTCP/ICP in varnish? Message-ID: <49A6BBF3.3070103@zaide.org> Hello, I'm searching how to activate an inter-cache coordination between 2 varnish servers. But i can't found any documentation, and no trace of htcp/icp in the source code (grep -REi "htcp|icp" varnish-2.0.3/). Could you help me to activate this functionality? I seen that varnish support this (yes ok it's a draft): http://varnish.projects.linpro.no/wiki/VarnishSpecDraft 3.2.2 ICP ? Varnish could support ICP for inter-cache coordination. ICP support may be a compile-time option. 3.2.3 HTCP ? Varnish could support HTCP for inter-cache coordination. HTCP support may be a compile-time option. Thank Regards, Damien Desmarets From ljorg6 at gmail.com Thu Feb 26 16:20:47 2009 From: ljorg6 at gmail.com (=?ISO-8859-1?Q?Lars_J=F8rgensen?=) Date: Thu, 26 Feb 2009 17:20:47 +0100 Subject: Varnish 2.0.3 hanging after a period of activity Message-ID: <9ED87A80-4D08-402C-BCF3-FF297139F1E5@gmail.com> Hi, We're currently employing Varnish to cache a largish web site but are experiencing difficulties. The site went live today and the increased load seemed to kill off Varnish every so often. Version 2.0.2 would hang with timeouts after about 5-7 minutes of run time. The backends are responding fine if we hit them directly. Restarting Varnish immediately makes it work fine again for another 5 minutes or so. I upgraded to version 2.0.3 which was running fine for about an hour but then the same thing happened. I don't see any particular errors in the log, except: 0 StatAddr - 91.194.202.5 0 552 1 163 0 0 0 28688 0 0 ExpPick - 843611540 ttl 0 VCL_call - timeout 0 VCL_return - discard 0 ExpKill - 843611540 -30 0 ExpPick - 843611541 ttl 0 VCL_call - timeout 0 VCL_return - discard 0 ExpKill - 843611541 -30 0 ExpPick - 843611555 ttl 0 VCL_call - timeout 0 VCL_return - discard 0 ExpKill - 843611555 -30 0 ExpPick - 843611556 ttl 0 VCL_call - timeout 0 VCL_return - discard 0 ExpKill - 843611556 -30 0 ExpPick - 843611557 ttl 0 VCL_call - timeout 0 VCL_return - discard 0 ExpKill - 843611557 -30 0 ExpPick - 843611561 ttl 0 VCL_call - timeout 0 VCL_return - discard 0 ExpKill - 843611561 -30 0 ExpPick - 843611562 ttl 0 VCL_call - timeout 0 VCL_return - discard 0 ExpKill - 843611562 -30 Right now I have a cron job restarting Varnish every 30 minutes but that seems less than optimal. I'm very new to Varnish and I have Googled as much as I could. Is there anything I should be aware of? Any kind of tuning of Varnish or the OS I should do? I've disabled KeepAlives on the backend Apache-servers. That seemed to help a bit with 2.0.2 but I don't know if that is still an issue on 2.0.3. Varnish is running on Debian Etch on a virtual machine with 4GB of memory. The cache file is 512MB. I start it with: /usr/local/sbin/varnishd -a 123.123.123.123:80 -T localhost:5000 -P / var/run/varnishd.pid -f /etc/varnish/website.vcl -s file,/var/varnish/ varnish_storage.bin,512M I could post the VCL if you need it, but I don't want to spam your mailboxes unnecessarily. -- Lars From phk at phk.freebsd.dk Thu Feb 26 16:24:32 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 26 Feb 2009 16:24:32 +0000 Subject: How to enable HTCP/ICP in varnish? In-Reply-To: Your message of "Thu, 26 Feb 2009 16:57:39 +0100." <49A6BBF3.3070103@zaide.org> Message-ID: <53466.1235665472@critter.freebsd.dk> In message <49A6BBF3.3070103 at zaide.org>, Damien Desmarets writes: >Hello, >I'm searching how to activate an inter-cache coordination between 2 >varnish servers. >But i can't found any documentation, and no trace of htcp/icp in the >source code (grep -REi "htcp|icp" varnish-2.0.3/). We do not support it. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From plfgoa at gmail.com Fri Feb 27 10:59:55 2009 From: plfgoa at gmail.com (Paras Fadte) Date: Fri, 27 Feb 2009 16:29:55 +0530 Subject: Unknown variable 'obj.prefetch' Message-ID: <75cf5800902270259l121bcf5as5d3cd474e748f990@mail.gmail.com> Hi, I downloaded varnish from trunk(Revision3832) and after installation when trying to start it , it gives error saying "Unknown variable 'obj.prefetch' " which is defined in varnish.vcl file . It worked fine for 2.0.2 Has obj.prefetch been removed ? Thank you. -Paras From tfheen at redpill-linpro.com Fri Feb 27 11:14:11 2009 From: tfheen at redpill-linpro.com (Tollef Fog Heen) Date: Fri, 27 Feb 2009 12:14:11 +0100 Subject: Unknown variable 'obj.prefetch' In-Reply-To: <75cf5800902270259l121bcf5as5d3cd474e748f990@mail.gmail.com> (Paras Fadte's message of "Fri, 27 Feb 2009 16:29:55 +0530") References: <75cf5800902270259l121bcf5as5d3cd474e748f990@mail.gmail.com> Message-ID: <87k57cjfng.fsf@qurzaw.linpro.no> ]] Paras Fadte | I downloaded varnish from trunk(Revision3832) and after installation | when trying to start it , it gives error saying "Unknown variable | 'obj.prefetch' " which is defined in varnish.vcl file . It worked fine | for 2.0.2 Has obj.prefetch been removed ? Yes. It was never used for anything. -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 From plfgoa at gmail.com Fri Feb 27 11:16:11 2009 From: plfgoa at gmail.com (Paras Fadte) Date: Fri, 27 Feb 2009 16:46:11 +0530 Subject: Unknown variable 'obj.prefetch' In-Reply-To: <87k57cjfng.fsf@qurzaw.linpro.no> References: <75cf5800902270259l121bcf5as5d3cd474e748f990@mail.gmail.com> <87k57cjfng.fsf@qurzaw.linpro.no> Message-ID: <75cf5800902270316q5786dd22hca405a081fd52c79@mail.gmail.com> So its no longer supported but it never did any function also ? -Paras On Fri, Feb 27, 2009 at 4:44 PM, Tollef Fog Heen wrote: > ]] Paras Fadte > > | I downloaded varnish from trunk(Revision3832) and after installation > | when trying to start it , it gives error ?saying "Unknown variable > | 'obj.prefetch' " which is defined in varnish.vcl file . It worked fine > | for 2.0.2 ?Has obj.prefetch been removed ? > > Yes. ?It was never used for anything. > > -- > Tollef Fog Heen > Redpill Linpro -- Changing the game! > t: +47 21 54 41 73 > From plfgoa at gmail.com Fri Feb 27 11:25:34 2009 From: plfgoa at gmail.com (Paras Fadte) Date: Fri, 27 Feb 2009 16:55:34 +0530 Subject: Unknown variable 'obj.prefetch' In-Reply-To: <75cf5800902270316q5786dd22hca405a081fd52c79@mail.gmail.com> References: <75cf5800902270259l121bcf5as5d3cd474e748f990@mail.gmail.com> <87k57cjfng.fsf@qurzaw.linpro.no> <75cf5800902270316q5786dd22hca405a081fd52c79@mail.gmail.com> Message-ID: <75cf5800902270325v189803cbg4ee3e05edfcfc773@mail.gmail.com> Whats functionality was it supposed to provide ? On Fri, Feb 27, 2009 at 4:46 PM, Paras Fadte wrote: > So its no longer supported but it never did any function also ? > > -Paras > > On Fri, Feb 27, 2009 at 4:44 PM, Tollef Fog Heen > wrote: >> ]] Paras Fadte >> >> | I downloaded varnish from trunk(Revision3832) and after installation >> | when trying to start it , it gives error ?saying "Unknown variable >> | 'obj.prefetch' " which is defined in varnish.vcl file . It worked fine >> | for 2.0.2 ?Has obj.prefetch been removed ? >> >> Yes. ?It was never used for anything. >> >> -- >> Tollef Fog Heen >> Redpill Linpro -- Changing the game! >> t: +47 21 54 41 73 >> > From phk at phk.freebsd.dk Sat Feb 28 22:34:57 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 28 Feb 2009 22:34:57 +0000 Subject: Man at Work Message-ID: <54223.1235860497@critter.freebsd.dk> Those of you who get the commit mails will have noticed that I have started pushings bits around to enable persistent storage. Most of the code for persistance will be isolated in the storage module, but for reasons of I/O layout optimization and general stateiness[1], I need to rearrange the order of some events and add some callbacks and API functions to the core code. The biggest item on that list was postponing object allocation until after we have decided in vcl_fetch{} if we even want to keep the object in the cache. That change is hopefully in place now. As always I would love if some of you have time to run -trunk a bit every so often to help weed out the bugs. Poul-Henning [1] Yes, I watch too much Colbert Report. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence.