From martin at varnish-software.com Tue Aug 2 14:06:02 2011 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 2 Aug 2011 16:06:02 +0200 Subject: Patch review for limiting backend retry on post Message-ID: Please find attached patch file. Any comments appreciated. -Martin -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: limit-retry-on-post.patch Type: application/octet-stream Size: 2461 bytes Desc: not available URL: From martin at varnish-software.com Tue Aug 2 21:16:22 2011 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 2 Aug 2011 23:16:22 +0200 Subject: Patches to fix leaking busyobj Message-ID: Find attached two patches to deal with leaking busyobj structures. First one frees any busyobj structure attached to the worker on thread exit. Second one deals with early deref's of objects (before HSH_Unbusy, e.g. due to failures), returning the busyobj to the worker. Any comments appreciated. Regards, Martin Blix Grydeland -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: free-busyobj.patch Type: application/octet-stream Size: 551 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: hsh_deref-return-busyobj.patch Type: application/octet-stream Size: 985 bytes Desc: not available URL: From phk at phk.freebsd.dk Wed Aug 3 08:40:38 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 03 Aug 2011 08:40:38 +0000 Subject: Patches to fix leaking busyobj In-Reply-To: Your message of "Tue, 02 Aug 2011 23:16:22 +0200." Message-ID: <2357.1312360838@critter.freebsd.dk> Looks good. Nice catch. >Find attached two patches to deal with leaking busyobj structures. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Wed Aug 3 08:50:49 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 03 Aug 2011 08:50:49 +0000 Subject: Patch review for limiting backend retry on post In-Reply-To: Your message of "Tue, 02 Aug 2011 16:06:02 +0200." Message-ID: <2424.1312361449@critter.freebsd.dk> In message , Martin Blix Grydeland writes: > Do not retry a backend connection if we have ever read any of the > request body bytes. I'm not keen in the pointer retry-arg, it should just be another return {enum} value, for sanity something like: -2 body write failed -1 body read failed 0 no body 1 body successfully transfered. In reality, the retry decision should not be based on the body-xfer but on the HTTP request: Only GET and HEAD are really safe to retry. This can be extended so that if we know that the server never had a chance to act, we can also retry POST/PUT/... but that is a somewhat slippery criteria to test, for a relatively small gain. At some point we probably should bite the bullet and buffer the request body so we can do this sensibly. That would also allow people to do things with it from VCL/VMOD. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From martin at varnish-software.com Wed Aug 3 12:07:50 2011 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Wed, 3 Aug 2011 14:07:50 +0200 Subject: Patch review for limiting backend retry on post In-Reply-To: <2424.1312361449@critter.freebsd.dk> References: <2424.1312361449@critter.freebsd.dk> Message-ID: Thanks for the feedback Poul-Henning. I've implemented what you suggested in the attached two patches. First changes FetchReqBody to report what it's done, and then only allow a retry if it reported no body. Second sets the retry variable only if we are doing a GET or HEAD. Another point for making Varnish buffer request bodies before doing any backend work at all is dealing with resource hungry backends and slow sending clients. We would then be able to buffer the data before tying up a backend resource and send it all in one fast swoop. -Martin -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Make-FetchReqBody-report-what-was-done-and-don-t-all.patch Type: application/octet-stream Size: 2325 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-Only-retry-a-recycled-backend-connection-on-GET-HEAD.patch Type: application/octet-stream Size: 821 bytes Desc: not available URL: From martin at varnish-software.com Thu Aug 4 19:42:15 2011 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Thu, 4 Aug 2011 21:42:15 +0200 Subject: Proposed patch to fix #972 Message-ID: Find attached patch to fix bug #972. This disables 304's when we are doing streaming, as we are already commited to stream the object when we get there. I guess this is an interim solution, as we would be able to do real 304s also when streaming when the final streaming bits are in place. Any comments appreciated. -Martin -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: no-cond-when-streaming.patch Type: application/octet-stream Size: 1374 bytes Desc: not available URL: From apj at mutt.dk Thu Aug 4 21:17:36 2011 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Thu, 4 Aug 2011 23:17:36 +0200 Subject: Doc: Upgrading to 3.0 Message-ID: <20110804211736.GK4654@nerd.dk> Based on questions in the IRC channel, I've compiled a small list of changes to make it easier to upgrade from 2.1 to 3.0 with the hope that they can be included in the docs. rst is attached. I'm sure more gotchas can be added as more people start using 3.0 -- Andreas -------------- next part -------------- %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Upgrading from Varnish 2.1 to 3.0 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% This is a compilation of items you need to pay attention to when upgrading from Varnish 2.1 to 3.0 Changes to VCL ============== In most cases you need to update your VCL since there has been some changes to the syntax. String concatenation did not have an operator previously, but this has now been changed to ``+``. ``log`` has moved to the std vmod: log "log something"; becomes import std; std.log "log something"; You only need to import std once. ``purge()`` and ``purge_url()`` are now respectively ``ban()`` and ``ban_url()``, so you should replace all occurences: purge("req.url = " req.url); becomes ban("req.url = " + req.url); ``purge`` does not take any arguments anymore, but can be used in vcl_hit or vcl_miss to purge the item from the cache, where you would reduce ttl to 0 in Varnish 2.1. sub vcl_hit { if (req.request == "PURGE") { set obj.ttl = 0s; error 200 "Purged."; } } becomes sub vcl_hit { if (req.request == "PURGE") { purge; error 200 "Purged."; } } ``beresp.cacheable`` is gone, and can be replaced with ``beresp.ttl > 0`` ``pass``, ``pipe``, ``lookup``, ``deliver``, ``fetch``, ``hash``, ``pipe`` and ``restart`` are no longer keywords, but arguments to ``return()``, so sub vcl_pass { pass; } becomes sub vcl_pass { return(pass); } You no longer append to the hash with +=, so set req.hash += req.url; becomes hash_data(req.url); Changes to behaviour ==================== Varnish will return an error when headers are too large instead of just ignoring them. If the limits are too low, Varnish will return HTTP 413. You can change the limits by increasing http_req_hdr_len and http_req_size. From martin at varnish-software.com Fri Aug 5 08:39:26 2011 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 5 Aug 2011 10:39:26 +0200 Subject: Proposed patch to fix #972 In-Reply-To: References: Message-ID: After discussions with DocWilco and Tollef, we came to the conclusion that disabling streaming when meeting the conditions for 304 is a better solution. Attached patch refactors the conditionals handling into a check and do function, and performs an extra check and disables if met when do_stream is true. -Martin -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: no-streaming-when-cond.patch Type: application/octet-stream Size: 3174 bytes Desc: not available URL: From tfheen at varnish-software.com Fri Aug 5 08:46:41 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Fri, 05 Aug 2011 10:46:41 +0200 Subject: Proposed patch to fix #972 In-Reply-To: (Martin Blix Grydeland's message of "Fri, 5 Aug 2011 10:39:26 +0200") References: Message-ID: <87y5z8gs9a.fsf@qurzaw.varnish-software.com> ]] Martin Blix Grydeland | After discussions with DocWilco and Tollef, we came to the conclusion that | disabling streaming when meeting the conditions for 304 is a better | solution. | | Attached patch refactors the conditionals handling into a check and do | function, and performs an extra check and disables if met when do_stream is | true. Maybe check sp->http->conds either in RES_Check_Conds or in cnt_fetchbody too? Apart from that, it looks reasonable to me. -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From tfheen at varnish-software.com Tue Aug 9 09:10:29 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 09 Aug 2011 11:10:29 +0200 Subject: Doc: Upgrading to 3.0 In-Reply-To: <20110804211736.GK4654@nerd.dk> (Andreas Plesner Jacobsen's message of "Thu, 4 Aug 2011 23:17:36 +0200") References: <20110804211736.GK4654@nerd.dk> Message-ID: <87r54vc5mi.fsf@qurzaw.varnish-software.com> ]] Andreas Plesner Jacobsen | Based on questions in the IRC channel, I've compiled a small list of changes to | make it easier to upgrade from 2.1 to 3.0 with the hope that they can be | included in the docs. rst is attached. Thanks a lot, I've glued this into the docs now after adding hit_for_pass and fixing up the markup slightly. https://www.varnish-cache.org/docs/trunk/installation/upgrade.html for the online version; I'll merge it to 3.0 later. Best regards, -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From kristian at varnish-software.com Mon Aug 15 14:15:04 2011 From: kristian at varnish-software.com (Kristian Lyngstol) Date: Mon, 15 Aug 2011 16:15:04 +0200 Subject: [PATCH] Re-arrange the random dir for proper retries Message-ID: <20110815141504.GB3995@freud.kly.no> The rearrangement consists of the following parts: 1. Creating a vdi_random_seed_init() function for appearances. vdi_random_seed_init() returns the initial random value to be used to pick the first backend, regardless of which of the random-based directors we use. 2. Split most of the SHA256-related code out into vdi_random_sha(). This is to allow simple reuse during retries. 3. Remove the hash/client-specific attempt that was made before the for() loop with retries. Hash/client now always uses the same algorithm as random. 4. Move the re-calculation of the random value below the code that picks a backend. This is possible because vdi_random_seed_init() has already given us the first value, and makes it easy to re-compute the SHA256 for client/hash only when a retry is about to happen. As a side-effect, this means that .retries param now works correctly too. The .retries-param was only considered for the for()-loop that the random director used, meaning that .retries = 1 for random meant "only try once" while it meant "try once, then only retry once after that" for hash/client. The test-case (which is not moved into tests/* here) demonstrates the issue. See #977 for the origin of this. --- bin/varnishd/cache_dir_random.c | 126 +++++++++++++++++------------ bin/varnishtest/tests.disabled/r00977.vtc | 28 +++++- 2 files changed, 97 insertions(+), 57 deletions(-) diff --git a/bin/varnishd/cache_dir_random.c b/bin/varnishd/cache_dir_random.c index f85d905..2d13cd6 100644 --- a/bin/varnishd/cache_dir_random.c +++ b/bin/varnishd/cache_dir_random.c @@ -77,22 +77,36 @@ struct vdi_random { unsigned nhosts; }; -static struct vbc * -vdi_random_getfd(const struct director *d, struct sess *sp) +/* + * Applies sha256 using the given context and input/length, returning a + * usable double. + */ +static double +vdi_random_sha(const struct sess *sp, struct SHA256Context *ctx, + uint8_t *sign, char *input, ssize_t len) { - int i, k; - struct vdi_random *vs; - double r, s1; - unsigned u = 0; - struct vbc *vbe; - struct director *d2; - struct SHA256Context ctx; - uint8_t sign[SHA256_LEN], *hp = NULL; - - CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); - CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC); - CAST_OBJ_NOTNULL(vs, d->priv, VDI_RANDOM_MAGIC); + double u,r; + uint8_t *hp = NULL; + SHA256_Init(ctx); + AN(input); + SHA256_Update(ctx, input, len); + SHA256_Final(sign, ctx); + hp = sign; + AN(hp); + u = vle32dec(hp); + r = u / 4294967296.0; + return r; +} +/* + * Sets up the initial seed for picking a backend. + * + * First tries the canonical backend for that hash/url. + */ +static double +vdi_random_init_seed(struct vdi_random *vs, struct sess *sp, + struct SHA256Context *ctx, uint8_t *sign) +{ if (vs->criteria == c_client) { /* * Hash the client IP# ascii representation, rather than @@ -101,50 +115,54 @@ vdi_random_getfd(const struct director *d, struct sess *sp) * We do not hash the port number, to make everybody behind * a given NAT gateway fetch from the same backend. */ - SHA256_Init(&ctx); - AN(sp->addr); + if (sp->client_identity != NULL) - SHA256_Update(&ctx, sp->client_identity, - strlen(sp->client_identity)); + return vdi_random_sha(sp, ctx, sign, + sp->client_identity, + strlen(sp->client_identity)); else - SHA256_Update(&ctx, sp->addr, strlen(sp->addr)); - SHA256_Final(sign, &ctx); - hp = sign; + return vdi_random_sha(sp, ctx, sign, sp->addr, + strlen(sp->addr)); } + if (vs->criteria == c_hash) { /* * Reuse the hash-string, the objective here is to fetch the * same object on the same backend all the time */ - hp = sp->digest; + AN(sp->digest); + return vle32dec(sp->digest) / 4294967296.0; } /* - * If we are hashing, first try to hit our "canonical backend" - * If that fails, we fall through, and select a weighted backend - * amongst the healthy set. + * Only c_random left */ - if (vs->criteria != c_random) { - AN(hp); - u = vle32dec(hp); - r = u / 4294967296.0; - assert(r >= 0.0 && r < 1.0); - r *= vs->tot_weight; - s1 = 0.0; - for (i = 0; i < vs->nhosts; i++) { - s1 += vs->hosts[i].weight; - if (r >= s1) - continue; - d2 = vs->hosts[i].backend; - if (!VDI_Healthy(d2, sp)) - break; - vbe = VDI_GetFd(d2, sp); - if (vbe != NULL) - return (vbe); - break; - } - } + return random() / 2147483648.0; /* 2^31 */ +} + +static struct vbc * +vdi_random_getfd(const struct director *d, struct sess *sp) +{ + int i, k; + struct vdi_random *vs; + double r, s1; + unsigned u = 0; + struct vbc *vbe; + struct director *d2; + struct SHA256Context ctx; + uint8_t sign[SHA256_LEN]; + CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); + CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC); + CAST_OBJ_NOTNULL(vs, d->priv, VDI_RANDOM_MAGIC); + + r = vdi_random_init_seed(vs, sp, &ctx, sign); + + /* + * Cycle through N retries. For each retry, re-calculate r, either + * randomly or (with hash/client) using the old value ran through + * sha256 again. + */ for (k = 0; k < vs->retries; ) { /* Sum up the weights of healty backends */ s1 = 0.0; @@ -158,12 +176,6 @@ vdi_random_getfd(const struct director *d, struct sess *sp) if (s1 == 0.0) return (NULL); - if (vs->criteria != c_random) { - r = u / 4294967296.0; - } else { - /* Pick a random threshold in that interval */ - r = random() / 2147483648.0; /* 2^31 */ - } assert(r >= 0.0 && r < 1.0); r *= s1; @@ -180,6 +192,16 @@ vdi_random_getfd(const struct director *d, struct sess *sp) return (vbe); break; } + + /* + * Recompute the hash for the next retry. + */ + if (vs->criteria != c_random) { + r = vdi_random_sha(sp, &ctx, sign, (char *)&r, + sizeof(r)); + } else { + r = random() / 2147483648.0; /* 2^31 */ + } k++; } return (NULL); diff --git a/bin/varnishtest/tests.disabled/r00977.vtc b/bin/varnishtest/tests.disabled/r00977.vtc index e6c61da..d15b55d 100644 --- a/bin/varnishtest/tests.disabled/r00977.vtc +++ b/bin/varnishtest/tests.disabled/r00977.vtc @@ -1,7 +1,10 @@ varnishtest "Test proper fallbacks of client director" -server s1 -repeat 1 { +server s1 { + rxreq + txresp -status 200 + accept rxreq txresp -status 200 } -start @@ -12,17 +15,32 @@ varnish v1 -vcl+backend { { .backend = { .host = "${bad_ip}"; .port = "9090"; } .weight = 1; } { .backend = s1; .weight = 1;} } + director bar client{ + .retries = 1; + { .backend = { .host = "${bad_ip}"; .port = "9090"; } .weight = 1; } + { .backend = s1; .weight = 1;} + } sub vcl_recv { - set req.backend = foo; - set client.identity = "44.452"; + if (req.url ~ "/one") { + set req.backend = foo; + } else { + set req.backend = bar; + } + # Carefully chosen seed that'll give us bad backend on + # first try and good on second. + set client.identity = "1.4"; return (pass); } } -start client c1 { - txreq + txreq -url "/one" rxresp expect resp.status == 200 + + txreq -url "/two" + rxresp + expect resp.status == 503 } -run -varnish v1 -expect backend_fail == 1 +varnish v1 -expect backend_fail == 2 -- 1.7.4.1 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: Digital signature URL: From apj at mutt.dk Tue Aug 16 14:24:25 2011 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Tue, 16 Aug 2011 16:24:25 +0200 Subject: Document one more pitfall when upgrading Message-ID: <20110816142425.GB1944@nerd.dk> This could bite some users when upgrading large installations. -- Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-Add-note-about-thread_pool_max.patch Type: text/x-diff Size: 957 bytes Desc: not available URL: From martin at varnish-software.com Tue Aug 16 20:42:34 2011 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 16 Aug 2011 22:42:34 +0200 Subject: Proposed patch to fix #978 Message-ID: Find attached proposed patch to fix #978 Regards, Martin -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: fix-for-978.patch Type: text/x-patch Size: 1508 bytes Desc: not available URL: From martin at varnish-software.com Tue Aug 16 21:57:35 2011 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 16 Aug 2011 23:57:35 +0200 Subject: Proposed fix for #979 Message-ID: Find attached proposed fix and test case for #979 Regards, Martin -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: fix-for-979.patch Type: text/x-patch Size: 1326 bytes Desc: not available URL: From martin at varnish-software.com Tue Aug 16 23:20:06 2011 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Wed, 17 Aug 2011 01:20:06 +0200 Subject: Proposed fix for #980 Message-ID: Find attached proposed fix for #980. Regards, Martin -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: fix-for-980.patch Type: text/x-patch Size: 1635 bytes Desc: not available URL: From ask at develooper.com Wed Aug 17 06:48:05 2011 From: ask at develooper.com (=?iso-8859-1?Q?Ask_Bj=F8rn_Hansen?=) Date: Tue, 16 Aug 2011 23:48:05 -0700 Subject: Documentation bugs Message-ID: <765BF75C-8791-4BA8-8733-3D47B65A9AA9@develooper.com> I tried logging in to my trac account; but it was too hard. On https://www.varnish-cache.org/docs/3.0/reference/vcl.html the vcl_fetch documentation is listing 'pass' as a valid return. Looks like it's missing being updated to pass_for_hit. On https://www.varnish-cache.org/docs/trunk/installation/upgrade.html the std.log examples are missing ()'s. On the same page it'd be nice if the "beresp.cacheable is gone" section had an appropriate example of how to test for !beresp.cacheable. beresp.ttl < 1s ? beresp.ttl == 0s? Or? In the varnishlog man page the documentation both says "-o is ignored" and then later has an explanation about how -m is ignored when -o is used. - ask -- http://develooper.com/ - http://askask.com/ -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4798 bytes Desc: not available URL: From phk at phk.freebsd.dk Wed Aug 17 07:10:58 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 17 Aug 2011 07:10:58 +0000 Subject: Proposed fix for #980 In-Reply-To: Your message of "Wed, 17 Aug 2011 01:20:06 +0200." Message-ID: <51696.1313565058@critter.freebsd.dk> In message , Martin Blix Grydeland writes: >Find attached proposed fix for #980. Almost correct: The C-L and do_stream tests belong together. Committed in fixed form. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From apj at mutt.dk Wed Aug 17 07:25:42 2011 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Wed, 17 Aug 2011 09:25:42 +0200 Subject: Documentation bugs In-Reply-To: <765BF75C-8791-4BA8-8733-3D47B65A9AA9@develooper.com> References: <765BF75C-8791-4BA8-8733-3D47B65A9AA9@develooper.com> Message-ID: <20110817072542.GC1944@nerd.dk> On Tue, Aug 16, 2011 at 11:48:05PM -0700, Ask Bj?rn Hansen wrote: > > On https://www.varnish-cache.org/docs/trunk/installation/upgrade.html the > std.log examples are missing ()'s. This is also the case for man vcl. > In the varnishlog man page the documentation both says "-o is ignored" and > then later has an explanation about how -m is ignored when -o is used. Where are you seeing this? Are you checking the latest version? The only mentions of -o in mine are in the synopsis and: -o Ignored for compatibility with earlier versions. -- Andreas From perbu at varnish-software.com Wed Aug 17 08:08:55 2011 From: perbu at varnish-software.com (Per Buer) Date: Wed, 17 Aug 2011 10:08:55 +0200 Subject: Documentation bugs In-Reply-To: <765BF75C-8791-4BA8-8733-3D47B65A9AA9@develooper.com> References: <765BF75C-8791-4BA8-8733-3D47B65A9AA9@develooper.com> Message-ID: Hi Ask, On Wed, Aug 17, 2011 at 8:48 AM, Ask Bj?rn Hansen wrote: > I tried logging in to my trac account; but it was too hard. > > On https://www.varnish-cache.org/docs/3.0/reference/vcl.html the vcl_fetch > documentation is listing 'pass' as a valid return. Looks like it's missing > being updated to pass_for_hit. > Fixed. On https://www.varnish-cache.org/docs/trunk/installation/upgrade.html the > std.log examples are missing ()'s. > Fixed. > On the same page it'd be nice if the "beresp.cacheable is gone" section had > an appropriate example of how to test for !beresp.cacheable. beresp.ttl < > 1s ? beresp.ttl == 0s? Or? > If anyone has a confirmed example I'll be happy to push it in. In the varnishlog man page the documentation both says "-o is ignored" and > then later has an explanation about how -m is ignored when -o is used. > Fixed. Thanks, Ask. -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Wed Aug 17 08:19:07 2011 From: perbu at varnish-software.com (Per Buer) Date: Wed, 17 Aug 2011 10:19:07 +0200 Subject: Documentation bugs In-Reply-To: <20110817072542.GC1944@nerd.dk> References: <765BF75C-8791-4BA8-8733-3D47B65A9AA9@develooper.com> <20110817072542.GC1944@nerd.dk> Message-ID: Hi Andreas On Wed, Aug 17, 2011 at 9:25 AM, Andreas Plesner Jacobsen wrote: > On Tue, Aug 16, 2011 at 11:48:05PM -0700, Ask Bj?rn Hansen wrote: > > > > On https://www.varnish-cache.org/docs/trunk/installation/upgrade.htmlthe > > std.log examples are missing ()'s. > > This is also the case for man vcl. > man vcl and the above URL are generated from the same source. -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From apj at mutt.dk Wed Aug 17 08:39:35 2011 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Wed, 17 Aug 2011 10:39:35 +0200 Subject: Documentation bugs In-Reply-To: References: <765BF75C-8791-4BA8-8733-3D47B65A9AA9@develooper.com> <20110817072542.GC1944@nerd.dk> Message-ID: <20110817083935.GD1944@nerd.dk> On Wed, Aug 17, 2011 at 10:19:07AM +0200, Per Buer wrote: > > > > > > On https://www.varnish-cache.org/docs/trunk/installation/upgrade.htmlthe > > > std.log examples are missing ()'s. > > > > This is also the case for man vcl. > > man vcl and the above URL are generated from the same source. No. Check EXAMPLES section in the man page. -- Andreas From perbu at varnish-software.com Wed Aug 17 09:07:18 2011 From: perbu at varnish-software.com (Per Buer) Date: Wed, 17 Aug 2011 11:07:18 +0200 Subject: Documentation bugs In-Reply-To: <20110817083935.GD1944@nerd.dk> References: <765BF75C-8791-4BA8-8733-3D47B65A9AA9@develooper.com> <20110817072542.GC1944@nerd.dk> <20110817083935.GD1944@nerd.dk> Message-ID: On Wed, Aug 17, 2011 at 10:39 AM, Andreas Plesner Jacobsen wrote: > On Wed, Aug 17, 2011 at 10:19:07AM +0200, Per Buer wrote: > > > > > > > > On > https://www.varnish-cache.org/docs/trunk/installation/upgrade.htmlthe > > > > std.log examples are missing ()'s. > > > > > > This is also the case for man vcl. > > > > man vcl and the above URL are generated from the same source. > > No. Check EXAMPLES section in the man page. > Right. More coffe, please. -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfheen at varnish-software.com Thu Aug 18 12:39:07 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Thu, 18 Aug 2011 14:39:07 +0200 Subject: vmod ABI checking Message-ID: <87ty9ec2s4.fsf@qurzaw.varnish-software.com> Hi, I've written a patch to do ABI checking for vmods, it currently uses the version from configure.ac and the git commit ID. I tried to make it use all the .h files too, but that turned out to not be stable enough, so I dropped it. We can always add it later if we see people managing to load vmods against the wrong ABI. Do you have any thoughts about this? Patch is attached. -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 -------------- next part -------------- A non-text attachment was scrubbed... Name: varnish_vmod_abi_check.diff Type: text/x-diff Size: 4900 bytes Desc: not available URL: From slink at schokola.de Thu Aug 18 14:05:32 2011 From: slink at schokola.de (Nils Goroll) Date: Thu, 18 Aug 2011 16:05:32 +0200 Subject: [PATCH] vmod_digest In-Reply-To: References: <4D42F382.5050402@schokola.de> Message-ID: <4E4D1C2C.3000005@schokola.de> Hi Laurence and all, sorry for the _very_ late response, I've been highly customer-demand driven and (and chased by deadlines) during the last months, so much work I started on Varnish stayed unfinished. The good news is that customers now demand more Varnish work again, so I hope to return to the more active circle again this year. >> phk had spotted a race condition in base64_init which I had overlooked (and even >> commented on sarcastically), so I've added code to pre-generate the base64 >> lookup tables. >> >> All other changes are due to the changes to varnish code since then. > > This looks really interesting. Do you have an up to date version of > this patch? It no longer applies and I was unable to find the base git > revision in the repository (maybe my lack of git-fu.) phk wasn't happy with the patch at the time for two reasons, IIUC: - he didn't like the generator for the base64_tables.h to be done in python (which I didn't quite understand at the time because he uses python scripts to generate such things all over the place) - he had asked me to clean up the digest functions code to use algorithm-specific function pointers rather than the case statements Because, at the time, I had no resources to finish these two things, the module did not make it any official repo at the time. I am using the code in production, but not on current versions of varnish. I will definitely need a Varnish 3-compatible digest module by the end of 2011, so I will come back to this. If anyone else wants to finish this, feel free. Nils From tfheen at varnish-software.com Fri Aug 19 13:17:58 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Fri, 19 Aug 2011 15:17:58 +0200 Subject: [PATCH] Lock panic buffer before mangling it Message-ID: <1313759878-7836-1-git-send-email-tfheen@varnish-software.com> If we were panic-ing in multiple threads at the same time, the panic buffer would be partially overwritten. Prevent this with a mutex --- bin/varnishd/cache_panic.c | 5 +++++ bin/varnishd/cache_shmlog.c | 2 +- 2 files changed, 6 insertions(+), 1 deletions(-) diff --git a/bin/varnishd/cache_panic.c b/bin/varnishd/cache_panic.c index c443b67..9279029 100644 --- a/bin/varnishd/cache_panic.c +++ b/bin/varnishd/cache_panic.c @@ -34,6 +34,7 @@ #include #include #include +#include #ifndef HAVE_EXECINFO_H #include "compat/execinfo.h" @@ -56,6 +57,7 @@ */ static struct vsb vsps, *vsp; +pthread_mutex_t panicstr_mtx = PTHREAD_MUTEX_INITIALIZER; /*--------------------------------------------------------------------*/ @@ -294,6 +296,9 @@ pan_ic(const char *func, const char *file, int line, const char *cond, const char *q; const struct sess *sp; + AZ(pthread_mutex_lock(&panicstr_mtx)); /* Won't be released, + we're going to die + anyway */ switch(xxx) { case 3: VSB_printf(vsp, diff --git a/bin/varnishd/cache_shmlog.c b/bin/varnishd/cache_shmlog.c index e0b332d..023c20a 100644 --- a/bin/varnishd/cache_shmlog.c +++ b/bin/varnishd/cache_shmlog.c @@ -288,7 +288,7 @@ VSL_Init(void) vsl_wrap(); VSM_head->starttime = (intmax_t)TIM_real(); - VSM_head->panicstr[0] = '\0'; + memset(VSM_head->panicstr, '\0', sizeof *VSM_head->panicstr); memset(VSC_C_main, 0, sizeof *VSC_C_main); VSM_head->child_pid = getpid(); } -- 1.7.5.4 From phk at phk.freebsd.dk Fri Aug 19 16:46:47 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 19 Aug 2011 16:46:47 +0000 Subject: [PATCH] Lock panic buffer before mangling it In-Reply-To: Your message of "Fri, 19 Aug 2011 15:17:58 +0200." <1313759878-7836-1-git-send-email-tfheen@varnish-software.com> Message-ID: <3201.1313772407@critter.freebsd.dk> In message <1313759878-7836-1-git-send-email-tfheen at varnish-software.com>, Toll ef Fog Heen writes: Go for it... >If we were panic-ing in multiple threads at the same time, the panic >buffer would be partially overwritten. Prevent this with a mutex >--- > bin/varnishd/cache_panic.c | 5 +++++ > bin/varnishd/cache_shmlog.c | 2 +- > 2 files changed, 6 insertions(+), 1 deletions(-) > >diff --git a/bin/varnishd/cache_panic.c b/bin/varnishd/cache_panic.c >index c443b67..9279029 100644 >--- a/bin/varnishd/cache_panic.c >+++ b/bin/varnishd/cache_panic.c >@@ -34,6 +34,7 @@ > #include > #include > #include >+#include > > #ifndef HAVE_EXECINFO_H > #include "compat/execinfo.h" >@@ -56,6 +57,7 @@ > */ > > static struct vsb vsps, *vsp; >+pthread_mutex_t panicstr_mtx = PTHREAD_MUTEX_INITIALIZER; > > /*--------------------------------------------------------------------*/ > >@@ -294,6 +296,9 @@ pan_ic(const char *func, const char *file, int line, const char *cond, > const char *q; > const struct sess *sp; > >+ AZ(pthread_mutex_lock(&panicstr_mtx)); /* Won't be released, >+ we're going to die >+ anyway */ > switch(xxx) { > case 3: > VSB_printf(vsp, >diff --git a/bin/varnishd/cache_shmlog.c b/bin/varnishd/cache_shmlog.c >index e0b332d..023c20a 100644 >--- a/bin/varnishd/cache_shmlog.c >+++ b/bin/varnishd/cache_shmlog.c >@@ -288,7 +288,7 @@ VSL_Init(void) > > vsl_wrap(); > VSM_head->starttime = (intmax_t)TIM_real(); >- VSM_head->panicstr[0] = '\0'; >+ memset(VSM_head->panicstr, '\0', sizeof *VSM_head->panicstr); > memset(VSC_C_main, 0, sizeof *VSC_C_main); > VSM_head->child_pid = getpid(); > } >-- >1.7.5.4 > > >_______________________________________________ >varnish-dev mailing list >varnish-dev at varnish-cache.org >https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From varnish at bsdchicks.com Mon Aug 22 22:43:06 2011 From: varnish at bsdchicks.com (Rogier R. Mulhuijzen) Date: Tue, 23 Aug 2011 00:43:06 +0200 (CEST) Subject: [PATCH] Fallback director (update) Message-ID: <20110822232823.I37583@ishtar.drwilco.net> By Tollef's request, here's the fallback director patch that applies cleanly to current trunk. I managed to get something else into the previous patch I linked from github. Cheers, Rogier/DocWilco -------------- next part -------------- diff --git a/bin/varnishd/cache_backend.h b/bin/varnishd/cache_backend.h index d2de06e..21a7061 100644 --- a/bin/varnishd/cache_backend.h +++ b/bin/varnishd/cache_backend.h @@ -156,4 +156,5 @@ dir_init_f VRT_init_dir_dns; dir_init_f VRT_init_dir_hash; dir_init_f VRT_init_dir_random; dir_init_f VRT_init_dir_round_robin; +dir_init_f VRT_init_dir_fallback; dir_init_f VRT_init_dir_client; diff --git a/bin/varnishd/cache_backend_cfg.c b/bin/varnishd/cache_backend_cfg.c index e6c935c..0582f72 100644 --- a/bin/varnishd/cache_backend_cfg.c +++ b/bin/varnishd/cache_backend_cfg.c @@ -256,6 +256,8 @@ VRT_init_dir(struct cli *cli, struct director **dir, const char *name, VRT_init_dir_dns(cli, dir, idx, priv); else if (!strcmp(name, "round-robin")) VRT_init_dir_round_robin(cli, dir, idx, priv); + else if (!strcmp(name, "fallback")) + VRT_init_dir_fallback(cli, dir, idx, priv); else if (!strcmp(name, "client")) VRT_init_dir_client(cli, dir, idx, priv); else diff --git a/bin/varnishd/cache_dir_round_robin.c b/bin/varnishd/cache_dir_round_robin.c index 2a6009a..61d80fc 100644 --- a/bin/varnishd/cache_dir_round_robin.c +++ b/bin/varnishd/cache_dir_round_robin.c @@ -47,10 +47,13 @@ struct vdi_round_robin_host { struct director *backend; }; +enum mode_e { m_round_robin, m_fallback }; + struct vdi_round_robin { unsigned magic; #define VDI_ROUND_ROBIN_MAGIC 0x2114a178 struct director dir; + enum mode_e mode; struct vdi_round_robin_host *hosts; unsigned nhosts; unsigned next_host; @@ -68,9 +71,17 @@ vdi_round_robin_getfd(const struct director *d, struct sess *sp) CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC); CAST_OBJ_NOTNULL(vs, d->priv, VDI_ROUND_ROBIN_MAGIC); + /* + * In fallback mode we ignore the next_host and always grab the + * first healthy backend we can find. + */ for (i = 0; i < vs->nhosts; i++) { - backend = vs->hosts[vs->next_host].backend; - vs->next_host = (vs->next_host + 1) % vs->nhosts; + if (vs->mode == m_round_robin) { + backend = vs->hosts[vs->next_host].backend; + vs->next_host = (vs->next_host + 1) % vs->nhosts; + } else /* m_fallback */ { + backend = vs->hosts[i].backend; + } if (!VDI_Healthy(backend, sp)) continue; vbe = VDI_GetFd(backend, sp); @@ -114,9 +125,9 @@ vdi_round_robin_fini(const struct director *d) FREE_OBJ(vs); } -void -VRT_init_dir_round_robin(struct cli *cli, struct director **bp, int idx, - const void *priv) +static void +vrt_init_dir(struct cli *cli, struct director **bp, int idx, + const void *priv, enum mode_e mode) { const struct vrt_dir_round_robin *t; struct vdi_round_robin *vs; @@ -141,6 +152,7 @@ VRT_init_dir_round_robin(struct cli *cli, struct director **bp, int idx, vs->dir.fini = vdi_round_robin_fini; vs->dir.healthy = vdi_round_robin_healthy; + vs->mode = mode; vh = vs->hosts; te = t->members; for (i = 0; i < t->nmember; i++, vh++, te++) { @@ -152,3 +164,18 @@ VRT_init_dir_round_robin(struct cli *cli, struct director **bp, int idx, bp[idx] = &vs->dir; } + +void +VRT_init_dir_round_robin(struct cli *cli, struct director **bp, int idx, + const void *priv) +{ + vrt_init_dir(cli, bp, idx, priv, m_round_robin); +} + +void +VRT_init_dir_fallback(struct cli *cli, struct director **bp, int idx, + const void *priv) +{ + vrt_init_dir(cli, bp, idx, priv, m_fallback); +} + diff --git a/bin/varnishtest/tests/v00036.vtc b/bin/varnishtest/tests/v00036.vtc new file mode 100644 index 0000000..c039650 --- /dev/null +++ b/bin/varnishtest/tests/v00036.vtc @@ -0,0 +1,132 @@ +varnishtest "Test fallback director" + +server s1 { + rxreq + expect req.url == "/" + txresp -body "slash" +} -start + +server s2 { + rxreq + expect req.url == "/" + txresp -body "slash" +} -start + +server s3 { + rxreq + expect req.url == "/foo" + txresp -hdr "Foo: 3" -body "foobar" +} -start + +varnish v1 -vcl { + + probe p1 { + .url = "/"; + .timeout = 1s; + .interval = 1s; + .window = 4; + .threshold = 3; + .initial = 0; + } + probe p2 { + .url = "/"; + .timeout = 1s; + .interval = 1s; + .window = 3; + .threshold = 2; + .initial = 0; + } + + backend b1 { + .host = "${s1_addr}"; + .port = "${s1_port}"; + .max_connections = 1; + .probe = p1; + } + backend b2 { + .host = "${s2_addr}"; + .port = "${s2_port}"; + .max_connections = 1; + .probe = p2; + } + backend b3 { + .host = "${s3_addr}"; + .port = "${s3_port}"; + } + director f1 fallback { + { .backend = b1; } + { .backend = b2; } + { .backend = b3; } + } + + sub vcl_recv { + set req.backend = f1; + return(pass); + } +} -start + +# s1 & s2 have both had 1 probe, so both are unhealthy + +client c1 { + txreq -url "/foo" + rxresp + expect resp.http.foo == "3" +} -run + +# setup for probe #2 + +server s1 { + rxreq + expect req.url == "/" + txresp -body "slash" +} -start + +server s2 { + rxreq + expect req.url == "/" + txresp -body "slash" +} -start + +# if we muck with a running server, the test will wait until it's done, +# which will be after probe #2 completes. b2 will then be healthy. + +server s2 { + rxreq + expect req.url == "/foo" + txresp -hdr "Foo: 2" -body "foobar" +} -start + +client c1 { + txreq -url "/foo" + rxresp + expect resp.http.foo == "2" +} -run + +# setup for probe #3 + +server s1 { + rxreq + expect req.url == "/" + txresp -body "slash" +} -start + +server s2 { + rxreq + expect req.url == "/" + txresp -body "slash" +} -start + +# after probe #3 b1 should be healthy. + +server s1 { + rxreq + expect req.url == "/foo" + txresp -hdr "Foo: 1" -body "foobar" +} -start + +client c1 { + txreq -url "/foo" + rxresp + expect resp.http.foo == "1" +} -run + diff --git a/doc/sphinx/reference/vcl.rst b/doc/sphinx/reference/vcl.rst index 281b653..4645cd1 100644 --- a/doc/sphinx/reference/vcl.rst +++ b/doc/sphinx/reference/vcl.rst @@ -221,6 +221,22 @@ The above example will append "internal.example.net" to the incoming Host header supplied by the client, before looking it up. All settings are optional. +The fallback director +~~~~~~~~~~~~~~~~~~~~~ + +The fallback director will pick the first backend that is healthy. It +considers them in the order in which they are listed in its definition. + +The fallback director does not take any options. + +An example of a fallback director:: + + director b3 fallback { + { .backend = www1; } + { .backend = www2; } // will only be used if www1 is unhealthy. + { .backend = www3; } // will only be used if both www1 and www2 + // are unhealthy. + } Backend probes -------------- diff --git a/lib/libvcl/vcc_backend.c b/lib/libvcl/vcc_backend.c index babfe01..b896a7d 100644 --- a/lib/libvcl/vcc_backend.c +++ b/lib/libvcl/vcc_backend.c @@ -695,6 +695,7 @@ static const struct dirlist { { "random", vcc_ParseRandomDirector }, { "client", vcc_ParseRandomDirector }, { "round-robin", vcc_ParseRoundRobinDirector }, + { "fallback", vcc_ParseRoundRobinDirector }, { "dns", vcc_ParseDnsDirector }, { NULL, NULL } }; From apj at mutt.dk Tue Aug 23 07:14:05 2011 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Tue, 23 Aug 2011 09:14:05 +0200 Subject: default hit_for_pass usage Message-ID: <20110823071405.GE1944@nerd.dk> I was just going through the docs, updating examples to 3.0 syntax. I fell over something that bothered me: https://www.varnish-cache.org/docs/trunk/faq/general.html#troubleshooting Has a section on increasing hit_for_pass ttl. I discussed rewriting this section with DocWilco on irc, and we came to the conclusion that the current default vcl may not be completely sane. For example, it looks like a 500 will cause varnish to hit_for_pass for the next two minutes, even though the next request could result in a nice cacheable 200. Any ideas? Was this already considered when writing the default? -- Andreas From varnish at bsdchicks.com Tue Aug 23 12:50:51 2011 From: varnish at bsdchicks.com (Rogier R. Mulhuijzen) Date: Tue, 23 Aug 2011 14:50:51 +0200 (CEST) Subject: default hit_for_pass usage In-Reply-To: <20110823071405.GE1944@nerd.dk> References: <20110823071405.GE1944@nerd.dk> Message-ID: <20110823142710.M37583@ishtar.drwilco.net> What we object to specifically was that hit_for_pass is returned when (beresp.ttl <= 0). Which is true not just for things like 401, but also 500 and 503. My primary objection is the 503. Like Andreas says, it possible to get a 500 or 503 on a page either due to load or some temporary error, where the next request could result in a perfectly cacheable page. But with the hit_for_pass in place, all workers blocking on that URL before then will just go through to the backend en-masse for the next two minutes. Especially with a 503, which most probably means the backend isn't coping very well due to load, and I'd hate to see an error like that cause even more load on a backend. I think the default.vcl should be more conservative, and looking at all the status codes other than the ones deemed cacheable by default, I can only think of 401 and 403 as needing a hit_for_pass. Cheers, Rogier/DocWilco On Tue, 23 Aug 2011, Andreas Plesner Jacobsen wrote: > I was just going through the docs, updating examples to 3.0 syntax. > > I fell over something that bothered me: > > https://www.varnish-cache.org/docs/trunk/faq/general.html#troubleshooting > > Has a section on increasing hit_for_pass ttl. I discussed rewriting this > section with DocWilco on irc, and we came to the conclusion that the current > default vcl may not be completely sane. For example, it looks like a 500 will > cause varnish to hit_for_pass for the next two minutes, even though the next > request could result in a nice cacheable 200. > > Any ideas? Was this already considered when writing the default? > > -- > Andreas > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > From apj at mutt.dk Wed Aug 24 07:28:37 2011 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Wed, 24 Aug 2011 09:28:37 +0200 Subject: [PATCH] Documentation fixes for 3.0-changes Message-ID: <20110824072836.GF1944@nerd.dk> A few fixes for docs --- doc/sphinx/faq/general.rst | 18 +++++++++--------- doc/sphinx/faq/http.rst | 6 +++--- doc/sphinx/tutorial/vcl.rst | 18 +++++++++--------- 3 files changed, 21 insertions(+), 21 deletions(-) diff --git a/doc/sphinx/faq/general.rst b/doc/sphinx/faq/general.rst index 5a1fbac..d40ef9f 100644 --- a/doc/sphinx/faq/general.rst +++ b/doc/sphinx/faq/general.rst @@ -115,7 +115,7 @@ You can use the ``bereq`` object for altering requests going to the backend, but sub vcl_miss { set bereq.url = regsub(req.url,"stream/","/"); - fetch; + return(fetch); } **How do I force the backend to send Vary headers?** @@ -148,18 +148,18 @@ A custom error page can be generated by adding a ``vcl_error`` to your configura "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> - "} obj.status " " obj.response {" + "} + obj.status + " " + obj.response + {" -

Error "} obj.status " " obj.response {"

-

"} obj.response {"

+

Error "} + obj.status + " " + obj.response + {"

+

"} + obj.response + {"

Guru Meditation:

-

XID: "} req.xid {"

+

XID: "} + req.xid + {"

Varnish
"}; - deliver; + return(deliver); } **How do I instruct varnish to ignore the query parameters and only cache one instance of an object?** @@ -235,8 +235,8 @@ HTTPS proxy such as nginx or pound. Yes, you need VCL code like this:: director foobar round-robin { - { .backend = { .host = "www1.example.com; .port = "http"; } } - { .backend = { .host = "www2.example.com; .port = "http"; } } + { .backend = { .host = "www1.example.com"; .port = "http"; } } + { .backend = { .host = "www2.example.com"; .port = "http"; } } } sub vcl_recv { @@ -337,7 +337,7 @@ Varnish has a feature called **hit for pass**, which is used when Varnish gets a * Client 2..N are now given the **hit for pass** object instructing them to go to the backend The **hit for pass** object will stay cached for the duration of its ttl. This means that subsequent clients requesting /foo will be sent straight to the backend as long as the **hit for pass** object exists. -The :command:`varnishstat` can tell you how many **hit for pass** objects varnish has served. You can lower the ttl for such an object if you are sure this is needed, using the following logic:: +The :command:`varnishstat` can tell you how many **hit for pass** objects varnish has served. The default vcl will set ttl for a hit_for_pass object to 120s. But you can override this, using the following logic: sub vcl_fetch { if (!obj.cacheable) { diff --git a/doc/sphinx/faq/http.rst b/doc/sphinx/faq/http.rst index 3d8f64a..1e276ba 100644 --- a/doc/sphinx/faq/http.rst +++ b/doc/sphinx/faq/http.rst @@ -16,10 +16,10 @@ To add a HTTP header, unless you want to add something about the client/request, sub vcl_fetch { # Add a unique header containing the cache servers IP address: - remove obj.http.X-Varnish-IP; - set obj.http.X-Varnish-IP = server.ip; + remove beresp.http.X-Varnish-IP; + set beresp.http.X-Varnish-IP = server.ip; # Another header: - set obj.http.Foo = "bar"; + set beresp.http.Foo = "bar"; } **How can I log the client IP address on the backend?** diff --git a/doc/sphinx/tutorial/vcl.rst b/doc/sphinx/tutorial/vcl.rst index 39b1e92..923c023 100644 --- a/doc/sphinx/tutorial/vcl.rst +++ b/doc/sphinx/tutorial/vcl.rst @@ -53,29 +53,29 @@ headers from the backend. actions ~~~~~~~ -The most common actions to call are these: +The most common results to return are these: *pass* - When you call pass the request and subsequent response will be passed - to and from the backend server. It won't be cached. pass can be called - in both vcl_recv and vcl_fetch. + When you return pass the request and subsequent response will be passed to + and from the backend server. It won't be cached. pass can be returned from + both vcl_recv and vcl_fetch. *lookup* - When you call lookup from vcl_recv you tell Varnish to deliver content + When you return lookup from vcl_recv you tell Varnish to deliver content from cache even if the request othervise indicates that the request - should be passed. You can't call lookup from vcl_fetch. + should be passed. You can't return lookup from vcl_fetch. *pipe* - Pipe can be called from vcl_recv as well. Pipe short circuits the + Pipe can be returned from vcl_recv as well. Pipe short circuits the client and the backend connections and Varnish will just sit there and shuffle bytes back and forth. Varnish will not look at the data being send back and forth - so your logs will be incomplete. Beware that with HTTP 1.1 a client can send several requests on the same connection and so you should instruct Varnish to add a "Connection: close" - header before actually calling pipe. + header before actually returning pipe. *deliver* - Deliver the cached object to the client. Usually called in vcl_fetch. + Deliver the cached object to the client. Usually returned from vcl_fetch. Requests, responses and objects ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- 1.7.2.3 From apj at mutt.dk Wed Aug 24 07:36:58 2011 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Wed, 24 Aug 2011 09:36:58 +0200 Subject: [PATCH] Documentation fixes for 3.0-changes In-Reply-To: <20110824072836.GF1944@nerd.dk> References: <20110824072836.GF1944@nerd.dk> Message-ID: <20110824073658.GG1944@nerd.dk> On Wed, Aug 24, 2011 at 09:28:37AM +0200, Andreas Plesner Jacobsen wrote: > A few fixes for docs I now realize that I screwed up the nomenclature in doc/sphinx/tutorial/vcl.rst Try this instead --- doc/sphinx/faq/general.rst | 18 +++++++++--------- doc/sphinx/faq/http.rst | 6 +++--- doc/sphinx/tutorial/vcl.rst | 18 +++++++++--------- 3 files changed, 21 insertions(+), 21 deletions(-) diff --git a/doc/sphinx/faq/general.rst b/doc/sphinx/faq/general.rst index 5a1fbac..d40ef9f 100644 --- a/doc/sphinx/faq/general.rst +++ b/doc/sphinx/faq/general.rst @@ -115,7 +115,7 @@ You can use the ``bereq`` object for altering requests going to the backend, but sub vcl_miss { set bereq.url = regsub(req.url,"stream/","/"); - fetch; + return(fetch); } **How do I force the backend to send Vary headers?** @@ -148,18 +148,18 @@ A custom error page can be generated by adding a ``vcl_error`` to your configura "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> - "} obj.status " " obj.response {" + "} + obj.status + " " + obj.response + {" -

Error "} obj.status " " obj.response {"

-

"} obj.response {"

+

Error "} + obj.status + " " + obj.response + {"

+

"} + obj.response + {"

Guru Meditation:

-

XID: "} req.xid {"

+

XID: "} + req.xid + {"

Varnish
"}; - deliver; + return(deliver); } **How do I instruct varnish to ignore the query parameters and only cache one instance of an object?** @@ -235,8 +235,8 @@ HTTPS proxy such as nginx or pound. Yes, you need VCL code like this:: director foobar round-robin { - { .backend = { .host = "www1.example.com; .port = "http"; } } - { .backend = { .host = "www2.example.com; .port = "http"; } } + { .backend = { .host = "www1.example.com"; .port = "http"; } } + { .backend = { .host = "www2.example.com"; .port = "http"; } } } sub vcl_recv { @@ -337,7 +337,7 @@ Varnish has a feature called **hit for pass**, which is used when Varnish gets a * Client 2..N are now given the **hit for pass** object instructing them to go to the backend The **hit for pass** object will stay cached for the duration of its ttl. This means that subsequent clients requesting /foo will be sent straight to the backend as long as the **hit for pass** object exists. -The :command:`varnishstat` can tell you how many **hit for pass** objects varnish has served. You can lower the ttl for such an object if you are sure this is needed, using the following logic:: +The :command:`varnishstat` can tell you how many **hit for pass** objects varnish has served. The default vcl will set ttl for a hit_for_pass object to 120s. But you can override this, using the following logic: sub vcl_fetch { if (!obj.cacheable) { diff --git a/doc/sphinx/faq/http.rst b/doc/sphinx/faq/http.rst index 3d8f64a..1e276ba 100644 --- a/doc/sphinx/faq/http.rst +++ b/doc/sphinx/faq/http.rst @@ -16,10 +16,10 @@ To add a HTTP header, unless you want to add something about the client/request, sub vcl_fetch { # Add a unique header containing the cache servers IP address: - remove obj.http.X-Varnish-IP; - set obj.http.X-Varnish-IP = server.ip; + remove beresp.http.X-Varnish-IP; + set beresp.http.X-Varnish-IP = server.ip; # Another header: - set obj.http.Foo = "bar"; + set beresp.http.Foo = "bar"; } **How can I log the client IP address on the backend?** diff --git a/doc/sphinx/tutorial/vcl.rst b/doc/sphinx/tutorial/vcl.rst index 39b1e92..923c023 100644 --- a/doc/sphinx/tutorial/vcl.rst +++ b/doc/sphinx/tutorial/vcl.rst @@ -53,29 +53,29 @@ headers from the backend. actions ~~~~~~~ -The most common actions to call are these: +The most common actions to return are these: *pass* - When you call pass the request and subsequent response will be passed - to and from the backend server. It won't be cached. pass can be called - in both vcl_recv and vcl_fetch. + When you return pass the request and subsequent response will be passed to + and from the backend server. It won't be cached. pass can be returned from + both vcl_recv and vcl_fetch. *lookup* - When you call lookup from vcl_recv you tell Varnish to deliver content + When you return lookup from vcl_recv you tell Varnish to deliver content from cache even if the request othervise indicates that the request - should be passed. You can't call lookup from vcl_fetch. + should be passed. You can't return lookup from vcl_fetch. *pipe* - Pipe can be called from vcl_recv as well. Pipe short circuits the + Pipe can be returned from vcl_recv as well. Pipe short circuits the client and the backend connections and Varnish will just sit there and shuffle bytes back and forth. Varnish will not look at the data being send back and forth - so your logs will be incomplete. Beware that with HTTP 1.1 a client can send several requests on the same connection and so you should instruct Varnish to add a "Connection: close" - header before actually calling pipe. + header before actually returning pipe. *deliver* - Deliver the cached object to the client. Usually called in vcl_fetch. + Deliver the cached object to the client. Usually returned from vcl_fetch. Requests, responses and objects ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- 1.7.2.3 From phk at phk.freebsd.dk Wed Aug 24 10:30:50 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 24 Aug 2011 10:30:50 +0000 Subject: default hit_for_pass usage In-Reply-To: Your message of "Tue, 23 Aug 2011 09:14:05 +0200." <20110823071405.GE1944@nerd.dk> Message-ID: <49934.1314181850@critter.freebsd.dk> In message <20110823071405.GE1944 at nerd.dk>, Andreas Plesner Jacobsen writes: >I was just going through the docs, updating examples to 3.0 syntax. > >I fell over something that bothered me: > >https://www.varnish-cache.org/docs/trunk/faq/general.html#troubleshooting > >Has a section on increasing hit_for_pass ttl. I discussed rewriting this >section with DocWilco on irc, and we came to the conclusion that the current >default vcl may not be completely sane. For example, it looks like a 500 will >cause varnish to hit_for_pass for the next two minutes, even though the next >request could result in a nice cacheable 200. > >Any ideas? Was this already considered when writing the default? No it was not. The basic premise was that we don't want to pile up the waiting list too often, and in fact much longer TTLs were proposed. I have no opinion on this matter. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Thu Aug 25 12:15:09 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 25 Aug 2011 12:15:09 +0000 Subject: [PATCH] Fallback director (update) In-Reply-To: Your message of "Tue, 23 Aug 2011 00:43:06 +0200." <20110822232823.I37583@ishtar.drwilco.net> Message-ID: <92332.1314274509@critter.freebsd.dk> In message <20110822232823.I37583 at ishtar.drwilco.net>, "Rogier R. Mulhuijzen" writ es: Committed, thanks! -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From tfheen at varnish-software.com Thu Aug 25 12:27:30 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Thu, 25 Aug 2011 14:27:30 +0200 Subject: Document one more pitfall when upgrading In-Reply-To: <20110816142425.GB1944@nerd.dk> (Andreas Plesner Jacobsen's message of "Tue, 16 Aug 2011 16:24:25 +0200") References: <20110816142425.GB1944@nerd.dk> Message-ID: <87sjopsmkt.fsf@qurzaw.varnish-software.com> ]] Andreas Plesner Jacobsen | This could bite some users when upgrading large installations. Cheers, applied -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From tfheen at varnish-software.com Thu Aug 25 12:28:57 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Thu, 25 Aug 2011 14:28:57 +0200 Subject: [PATCH] Documentation fixes for 3.0-changes In-Reply-To: <20110824073658.GG1944@nerd.dk> (Andreas Plesner Jacobsen's message of "Wed, 24 Aug 2011 09:36:58 +0200") References: <20110824072836.GF1944@nerd.dk> <20110824073658.GG1944@nerd.dk> Message-ID: <87liuhsmie.fsf@qurzaw.varnish-software.com> ]] Andreas Plesner Jacobsen | On Wed, Aug 24, 2011 at 09:28:37AM +0200, Andreas Plesner Jacobsen wrote: | | > A few fixes for docs | | I now realize that I screwed up the nomenclature in doc/sphinx/tutorial/vcl.rst And thanks for this too; applied. :-) -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From jbq at caraldi.com Thu Aug 25 15:10:13 2011 From: jbq at caraldi.com (Jean-Baptiste Quenot) Date: Thu, 25 Aug 2011 17:10:13 +0200 Subject: varnishncsa 3.0.0 crash In-Reply-To: References: Message-ID: Do you believe this is fixed by https://www.varnish-cache.org/trac/ticket/944 ? Cheers, -- Jean-Baptiste Quenot From tfheen at varnish-software.com Fri Aug 26 07:09:41 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Fri, 26 Aug 2011 09:09:41 +0200 Subject: varnishncsa 3.0.0 crash In-Reply-To: (Jean-Baptiste Quenot's message of "Thu, 25 Aug 2011 17:10:13 +0200") References: Message-ID: <87d3fssl6y.fsf@qurzaw.varnish-software.com> ]] Jean-Baptiste Quenot | Do you believe this is fixed by https://www.varnish-cache.org/trac/ticket/944 ? Maybe, try 3.0.1 rc 1 and give feedback. -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From jbq at caraldi.com Mon Aug 29 13:03:18 2011 From: jbq at caraldi.com (Jean-Baptiste Quenot) Date: Mon, 29 Aug 2011 15:03:18 +0200 Subject: [PATCH] Fix time_firstbyte bug in varnishncsa Message-ID: Hi there, Here is a patch against 3.0.1 rc1 to fix handling of the %{Varnish:time_firstbyte}x format flag in varnishncsa: http://dl.dropbox.com/u/5836407/Patches/Varnish_3.0.1-rc1-Fix-time_firstbyte-bug-in-varnishncsa.patch Otherwise, I tested the feature to log any response header, and it works fine for logging the object age: %{Host}i %h %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i" %{Varnish:time_firstbyte}x %{Age}o Cheers, -- Jean-Baptiste Quenot -------------- next part -------------- An HTML attachment was scrubbed... URL: From varnish at bsdchicks.com Mon Aug 29 23:34:10 2011 From: varnish at bsdchicks.com (Rogier R. Mulhuijzen) Date: Tue, 30 Aug 2011 01:34:10 +0200 (CEST) Subject: [PATCH] varnishlog functionality in varnishtest Message-ID: <20110830012018.M37583@ishtar.drwilco.net> A while in the making, but 2 full rewrites later, here it is. Please be sure to apply https://www.varnish-cache.org/trac/attachment/ticket/995/995.patch as well, or there's a good chance you'll get errors during "make check". What this adds: - When given a second -v, varnishlog output is added to the overall test log. - The varnish component gets a -logexpecttag action. It takes 2 arguments, a number (0+) and a VSL tag. It will see how many of said tag have passed by since the child was started, and match it with the number. - And it gets a -logexpectregex action. Again 2 arguments, a number (0+) and a regex. The regex is matched with the contents of the loglines, just like varnishlog et al do. Do give the varnishd a smidgen of time to write to the VSL before doing your expects. How much is a bit of an inexact science, but 0.2 has been working for me. (See a00012.vtc) Any comments are welcome, as are suggestions for better alternatives to -logexpect(tag|regex). They're a bit horrible in length. Also, I'm tempted to scrap the second verbosity level and always include VSL in the test log. Opinions on that? Cheers, Doc -------------- next part -------------- diff --git a/bin/varnishtest/tests/a00012.vtc b/bin/varnishtest/tests/a00012.vtc new file mode 100644 index 0000000..215024c --- /dev/null +++ b/bin/varnishtest/tests/a00012.vtc @@ -0,0 +1,94 @@ +varnishtest "Test varnishlog function" + +server s1 { + rxreq + txresp -hdr "X-Foo: Bar" -hdr "Cache-Control: max-age=0" +} -start +server s2 { + rxreq + txresp -hdr "X-Foo: Bar" -hdr "Cache-Control: max-age=0" +} -start + +varnish v1 -vcl+backend { +import std from "${topbuild}/lib/libvmod_std/.libs/libvmod_std.so"; + sub vcl_recv { + set req.backend = s2; + set req.backend = s1; + std.log("this is v1"); + } +} -start + +varnish v2 -vcl+backend { +import std from "${topbuild}/lib/libvmod_std/.libs/libvmod_std.so"; + sub vcl_recv { + set req.backend = s1; + set req.backend = s2; + std.log("this is v2"); + std.log("no really!"); + } +} -start + +client c1 { + txreq + rxresp + expect resp.http.X-Foo == "Bar" +} -run + +client c2 -connect ${v2_sock} { + txreq + rxresp + expect resp.http.X-Foo == "Bar" +} -run + +# give the log a moment to catch up +delay 0.2 + +# there should be no crosstalk +varnish v1 -logexpecttag 1 VCL_Log +varnish v2 -logexpecttag 2 VCL_Log + +# shouldn't be case sensitive +varnish v1 -logexpecttag 1 vcl_log +varnish v1 -logexpecttag 1 vcl_lOG + +varnish v1 -logexpecttag 1 RxURL + +server s1 -wait + +varnish v1 -stop -start + +server s1 { + rxreq + txresp -hdr "X-Foo: Bar" +} -start + +varnish v1 -logexpecttag 0 RxURL + +client c1 { + txreq + rxresp + expect resp.http.X-Foo == "Bar" +} -run + +delay 0.2 + +varnish v1 -logexpectregex 3 "X-Foo: Bar" +varnish v1 -logexpectregex 3 "X-Foo: B.." + +# check proper multiline handling +varnish v1 -logexpectregex 3 "^X-Foo: Bar" +varnish v1 -logexpectregex 3 "X-Foo: Bar$" +varnish v1 -logexpectregex 0 "^Bar" +varnish v1 -logexpectregex 0 "X-Foo$" + +# there should be no crosstalk +varnish v1 -logexpectregex 1 "^this is v1$" +varnish v1 -logexpectregex 0 "^this is v2$" +varnish v2 -logexpectregex 0 "^this is v1$" +varnish v2 -logexpectregex 1 "^this is v2$" + +# and finally, nothing should be found after a restart +varnish v1 -stop -start +delay 0.2 +varnish v1 -logexpectregex 0 "X-Foo: Bar" +varnish v1 -logexpecttag 0 VCL_Log diff --git a/bin/varnishtest/vtc.h b/bin/varnishtest/vtc.h index 16d577f..6a309d7 100644 --- a/bin/varnishtest/vtc.h +++ b/bin/varnishtest/vtc.h @@ -61,6 +61,7 @@ cmd_f cmd_sema; extern volatile sig_atomic_t vtc_error; /* Error, bail out */ extern int vtc_stop; /* Abandon current test, no error */ extern pthread_t vtc_thread; +extern int vtc_verbosity; void init_sema(void); diff --git a/bin/varnishtest/vtc_main.c b/bin/varnishtest/vtc_main.c index b280022..131a662 100644 --- a/bin/varnishtest/vtc_main.c +++ b/bin/varnishtest/vtc_main.c @@ -83,11 +83,12 @@ static int njob = 0; static int npar = 1; /* Number of parallel tests */ static unsigned vtc_maxdur = 60; /* Max duration of any test */ static int vtc_continue; /* Continue on error */ -static int vtc_verbosity = 1; /* Verbosity Level */ static int vtc_good; static int vtc_fail; static int leave_temp; +int vtc_verbosity = 1; /* Verbosity Level */ + /********************************************************************** * Parse a -D option argument into a name/val pair, and insert * into extmacro list @@ -370,7 +371,7 @@ main(int argc, char * const *argv) vtc_maxdur = strtoul(optarg, NULL, 0); break; case 'v': - if (vtc_verbosity < 2) + if (vtc_verbosity < 3) vtc_verbosity++; break; default: diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c index 2b6e3d4..a6e8e7c 100644 --- a/bin/varnishtest/vtc_varnish.c +++ b/bin/varnishtest/vtc_varnish.c @@ -37,6 +37,7 @@ #include #include #include +#include #include #include @@ -50,6 +51,7 @@ #include "cli_common.h" #include "vss.h" #include "vsb.h" +#include "vsl.h" #include "vtc.h" @@ -57,8 +59,10 @@ struct varnish { unsigned magic; #define VARNISH_MAGIC 0x208cd8e3 char *name; + /* one log per thread */ struct vtclog *vl; struct vtclog *vl1; + struct vtclog *vl2; VTAILQ_ENTRY(varnish) list; struct vsb *storage; @@ -68,16 +72,28 @@ struct varnish { pid_t pid; pthread_t tp; + pthread_t ltp; + pthread_mutex_t lsmtx; + pthread_cond_t lscnd; int cli_fd; int vcl_nbr; char *workdir; struct VSM_data *vd; + + /* XXX merge lvd with vd once VSM_Setup allows it */ + struct VSM_data *lvd; + int lr; + int ts[256]; + struct VSM_data *revd; + int reh; }; #define NONSENSE "%XJEIFLH|)Xspa8P" +#define SLEEP_USEC (50*1000) + static VTAILQ_HEAD(, varnish) varnishes = VTAILQ_HEAD_INITIALIZER(varnishes); @@ -116,6 +132,51 @@ varnish_ask_cli(const struct varnish *v, const char *cmd, char **repl) } /********************************************************************** + * Varnishlog gatherer + thread + */ + +static int +h_addlog(void *priv, enum VSL_tag_e tag, unsigned fd, unsigned len, + unsigned spec, const char *ptr, uint64_t bitmap) +{ + struct varnish *v; + int type; + + (void) bitmap; + + type = (spec & VSL_S_CLIENT) ? 'c' : + (spec & VSL_S_BACKEND) ? 'b' : '-'; + CAST_OBJ_NOTNULL(v, priv, VARNISH_MAGIC); + + v->ts[tag]++; + + if (vtc_verbosity > 2) + vtc_log(v->vl2, 4, "varnishlog| %5u %-12s %c %.*s", fd, + VSL_tags[tag], type, len, ptr); + return (0); +} + +static void * +varnishlog_thread(void *priv) +{ + struct varnish *v; + + CAST_OBJ_NOTNULL(v, priv, VARNISH_MAGIC); + while (v->lr && VSL_Open(v->lvd, 0) != 0) { + assert(usleep(SLEEP_USEC) == 0 || errno == EINTR); + } + while (v->lr) { + (void) VSL_Dispatch(v->lvd, h_addlog, v); + /* tell main thread we're at the end of the log */ + pthread_mutex_lock(&v->lsmtx); + pthread_cond_signal(&v->lscnd); + pthread_mutex_unlock(&v->lsmtx); + assert(usleep(SLEEP_USEC) == 0 || errno == EINTR); + } + return (NULL); +} + +/********************************************************************** * Allocate and initialize a varnish */ @@ -157,6 +218,19 @@ varnish_new(const char *name) VSB_printf(v->storage, "-sfile,%s,10M", v->workdir); AZ(VSB_finish(v->storage)); + v->vl2 = vtc_logopen(name); + AN(v->vl2); + + v->lvd = VSM_New(); + VSL_Setup(v->lvd); + + (void)VSL_Arg(v->lvd, 'n', v->workdir); + (void)VSL_Arg(v->lvd, 'd', NULL); + v->lr = 1; + AZ(pthread_mutex_init(&v->lsmtx, NULL)); + AZ(pthread_cond_init(&v->lscnd, NULL)); + AZ(pthread_create(&v->ltp, NULL, varnishlog_thread, v)); + v->cli_fd = -1; VTAILQ_INSERT_TAIL(&varnishes, v, list); @@ -171,13 +245,23 @@ varnish_new(const char *name) static void varnish_delete(struct varnish *v) { + void *p; CHECK_OBJ_NOTNULL(v, VARNISH_MAGIC); + v->lr = 0; + AZ(pthread_join(v->ltp, &p)); + AZ(pthread_mutex_destroy(&v->lsmtx)); + AZ(pthread_cond_destroy(&v->lscnd)); vtc_logclose(v->vl); + vtc_logclose(v->vl1); + vtc_logclose(v->vl2); free(v->name); free(v->workdir); + VSB_delete(v->storage); if (v->vd != NULL) VSM_Delete(v->vd); + if (v->lvd != NULL) + VSM_Delete(v->lvd); /* * We do not delete the workdir, it may contain stuff people @@ -353,7 +437,7 @@ varnish_launch(struct varnish *v) vtc_log(v->vl, 0, "CLI auth command failed: %u %s", u, r); free(r); - (void)VSL_Arg(v->vd, 'n', v->workdir); + (void)VSC_Arg(v->vd, 'n', v->workdir); AZ(VSC_Open(v->vd, 1)); } @@ -371,6 +455,7 @@ varnish_start(struct varnish *v) varnish_launch(v); if (vtc_error) return; + memset(&v->ts, 0, sizeof v->ts); vtc_log(v->vl, 2, "Start"); u = varnish_ask_cli(v, "start", &resp); if (vtc_error) @@ -663,6 +748,103 @@ varnish_expect(const struct varnish *v, char * const *av) { } /********************************************************************** + * Check varnishlog for VSL tags + */ + +static void +varnish_logexpecttag(struct varnish *v, const int exp, const char *tag) +{ + int i, j, tl, ul; + const char *t, *u; + + /* Make sure log is caught up */ + pthread_mutex_lock(&v->lsmtx); + pthread_cond_wait(&v->lscnd, &v->lsmtx); + pthread_mutex_unlock(&v->lsmtx); + + tl = strlen(tag); + for (i = 0; i < 256; i++) { + if (VSL_tags[i] == NULL) + continue; + u = VSL_tags[i]; + ul = strlen(u); + if (ul != tl) + continue; + t = tag; + for (j = 0; j < tl; j++) + if (tolower(*t++) != tolower(*u++)) + break; + if (j == tl) { + j = v->ts[i]; /* single read, so no mutex */ + if (j == exp) + vtc_log(v->vl, 2, "as expected, found %s %d " + "times", VSL_tags[i], exp); + else + vtc_log(v->vl, 0, "Found %s %d times, expected" + " %d times", VSL_tags[i], j, exp); + return; + } + } + vtc_log(v->vl, 0, "Unknown tag: %s", tag); +} + +/********************************************************************** + * Check varnishlog for regexes + */ + +static int +h_count(void *priv, enum VSL_tag_e tag, unsigned fd, unsigned len, + unsigned spec, const char *ptr, uint64_t bitmap) +{ + struct varnish *v; + + (void) tag; + (void) fd; + (void) len; + (void) spec; + (void) ptr; + (void) bitmap; + CAST_OBJ_NOTNULL(v, priv, VARNISH_MAGIC); + v->reh++; + return (0); +} + +static void +varnish_logexpectregex(struct varnish *v, const int exp, const char *pat) +{ + + v->revd = VSM_New(); + VSL_Setup(v->revd); + + (void)VSL_Arg(v->revd, 'n', v->workdir); + (void)VSL_Arg(v->revd, 'd', NULL); + if (!VSL_Arg(v->revd, 'I', pat)) { + vtc_log(v->vl, 0, "Regex error: %s", pat); + return; + } + + v->reh = 0; + + if (VSL_Open(v->revd, 0)) { + vtc_log(v->vl, 0, "VSL_Open failed"); + return; + } + if (VSL_Dispatch(v->revd, h_count, v) != -1) { + vtc_log(v->vl, 0, "VSL_Dispatch failed"); + return; + } + + VSM_Delete(v->revd); + + if (exp == v->reh) + vtc_log(v->vl, 2, "as expected, matched \"%s\" %d times", pat, + exp); + else + vtc_log(v->vl, 0, "Matched \"%s\" %d times, expected %d", pat, + v->reh, exp); + +} +/********************************************************************** * Varnish server cmd dispatch */ @@ -764,11 +946,28 @@ cmd_varnish(CMD_ARGS) continue; } if (!strcmp(*av, "-expect")) { + AN(av[1]); + AN(av[2]); + AN(av[3]); av++; varnish_expect(v, av); av += 2; continue; } + if (!strcmp(*av, "-logexpecttag")) { + AN(av[1]); + AN(av[2]); + varnish_logexpecttag(v, atoi(av[1]), av[2]); + av += 2; + continue; + } + if (!strcmp(*av, "-logexpectregex")) { + AN(av[1]); + AN(av[2]); + varnish_logexpectregex(v, atoi(av[1]), av[2]); + av += 2; + continue; + } vtc_log(v->vl, 0, "Unknown varnish argument: %s", *av); } } From tfheen at varnish-software.com Tue Aug 30 07:08:07 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 30 Aug 2011 09:08:07 +0200 Subject: [PATCH] Fix time_firstbyte bug in varnishncsa In-Reply-To: (Jean-Baptiste Quenot's message of "Mon, 29 Aug 2011 15:03:18 +0200") References: Message-ID: <87ippfs7fs.fsf@qurzaw.varnish-software.com> ]] Jean-Baptiste Quenot | Here is a patch against 3.0.1 rc1 to fix handling of the | %{Varnish:time_firstbyte}x format flag in varnishncsa: | | http://dl.dropbox.com/u/5836407/Patches/Varnish_3.0.1-rc1-Fix-time_firstbyte-bug-in-varnishncsa.patch Ooops, thanks, applied. | Otherwise, I tested the feature to log any response header, and it works | fine for logging the object age: Good to hear. :-) -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From jbq at caraldi.com Tue Aug 30 21:40:42 2011 From: jbq at caraldi.com (Jean-Baptiste Quenot) Date: Tue, 30 Aug 2011 23:40:42 +0200 Subject: Varnish 3.0.0 crash In-Reply-To: References: <7840.1310504660@critter.freebsd.dk> Message-ID: Got the same crash with 3.0.1 rc1. I added CFLAGS+=-O0 to make your life easier. https://gist.github.com/1182132 -- Jean-Baptiste Quenot From phk at phk.freebsd.dk Wed Aug 31 15:31:16 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 31 Aug 2011 15:31:16 +0000 Subject: Varnish 3.0.0 crash In-Reply-To: Your message of "Tue, 30 Aug 2011 23:40:42 +0200." Message-ID: <92453.1314804676@critter.freebsd.dk> In message , Jean-Baptiste Quenot writes: Sorry for not getting back to you earlier, I lost you in the mailbox. Try trunk as of [master] c0e2acc When headers are surplus to limits, That should fix the panic. You underlying problem is probably that your have (set-)cookie headers which run afoul of the new http_resp_hdr_len/http_req_hdr_len parameters. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence.