From jocelyn.delarosa at smartjog.com Mon Feb 6 15:53:22 2012 From: jocelyn.delarosa at smartjog.com (Jocelyn De La Rosa) Date: Mon, 06 Feb 2012 16:53:22 +0100 Subject: [PATCH] Fixes for range support with streaming - varnish 3.0S Message-ID: <4F2FF772.1020105@smartjog.com> Hi, Some time ago Thomas Souvignet sent a patch on the varnish-dev mailing list that implements range support with streaming (for varnish 3.0S). There were some bugs with his implementation so here are three patches that correct them. Regards, -- Jocelyn De La Rosa - Developer, Research& Engineering SmartJog | www.smartjog.com 27 Blvd Hippolyte Marqu?s, 94200 Ivry-sur-Seine, France -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Fix-http_range_support-flag-off-with-streaming-mode.patch Type: text/x-patch Size: 862 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-Fix-concecutive-range-requests-whith-streaming-mode.patch Type: text/x-patch Size: 1332 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0003-Add-object-response-check-to-range-support.patch Type: text/x-patch Size: 918 bytes Desc: not available URL: From phk at phk.freebsd.dk Thu Feb 9 07:55:01 2012 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 09 Feb 2012 07:55:01 +0000 Subject: No more worker-thread workspace Message-ID: <45985.1328774101@critter.freebsd.dk> I have completed the much mumbled about move of the thread workspace to busyobj. This should save a fair amount of memory and thread-stack. A quick experiment showed that I could run the test-cases with only 32k of thread-stack. See commit messages for more details. Feedback welcome. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From mad.krinke at googlemail.com Fri Feb 10 13:49:32 2012 From: mad.krinke at googlemail.com (Moritz Krinke) Date: Fri, 10 Feb 2012 14:49:32 +0100 Subject: Possible race condition between ban and fetch within esi processing Message-ID: Hello, i'm adding bans (lurker friendly) in vcl using headers set by the backend. I'm also using ESI includes. Usually the bans match a specific esi include. I'm banning before the call to process the request for esi includes. Usage example: "You have 2 unread notifications" -> click -> backend marks notifications as read, backends add header for varnish -> varnish adds ban, requests new partial -> "you have 0 unread notifications" This works quite fine. But sometimes it doesnt. Varnish seems to deliver the old cached partial rather than fetching a new one from the backend. This does not happen a lot and i havent found a way yet to reproduce it. One could write a small programm to do stresstest and prove this problem, but before i do that i would like to know if you can confirm that there might occur such a race condition (maybe from own experience, or you know the code) vcl example: sub vcl_fetch { ban("obj.http.x-url ~ /some/uri"); set beresp.do_esi = true; } im using varnish 3.0.0 (havent upgraded yet because of the bug about the ban lurker) and my ban list is usually below 200. Thanks a lot, Moritz -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfheen at varnish-software.com Mon Feb 13 09:34:35 2012 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Mon, 13 Feb 2012 10:34:35 +0100 Subject: [PATCH 1/3] Fix up some sess_workspace leftovers In-Reply-To: <1326880276-18354-1-git-send-email-github@bsdchicks.com> (Rogier Mulhuijzen's message of "Wed, 18 Jan 2012 10:51:14 +0100") References: <1326880276-18354-1-git-send-email-github@bsdchicks.com> Message-ID: <87k43rcc0k.fsf@qurzaw.varnish-software.com> Thanks, the whole series has been applied. (The first patch needed some hand-holding due to a bug in git-send-email, I think (it failed to apply the second chunk due to the function prototype changing and then got thoroughly confused. It also needed a small adjustment for some changes in git master, but no worries about that.) Thanks and sorry for the delay in getting this applied! -- Tollef Fog Heen Technical lead, Varnish Software t: +47 21 98 92 64 From tfheen at varnish-software.com Mon Feb 13 09:36:25 2012 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Mon, 13 Feb 2012 10:36:25 +0100 Subject: deactivate obj updates In-Reply-To: <18AD381627308B47A6AED6D2E69D552F18E68857F0@IE2RD2XVS201.red002.local> (Uxio Faria Giraldez's message of "Tue, 10 Jan 2012 07:52:11 -0800") References: <18AD381627308B47A6AED6D2E69D552F18E68857F0@IE2RD2XVS201.red002.local> Message-ID: <87fwefcbxi.fsf@qurzaw.varnish-software.com> ]] Uxio Faria Giraldez > Hello list. Last day i was wondering why varnish writes so much info to disc even if hot data > keeps in memory and having high hitrate. Assuming you're using -s file, that's under control of the VM, not in Varnish's hands. > I supposse is due to 'obj.' variables that needs to be updated: > obj.hits > obj.lastuse > I think no more variables are updated on object HITS. This shouldn't cause any significant churn, I believe. -- Tollef Fog Heen Technical lead, Varnish Software t: +47 21 98 92 64 From martin at varnish-software.com Tue Feb 14 14:04:52 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 14 Feb 2012 15:04:52 +0100 Subject: [PATCH] Honor remove-flag also when processing comments in ESI parsing. Message-ID: <1329228292-16580-1-git-send-email-martin@varnish-software.com> Fixes: #1092 --- bin/varnishd/cache/cache_esi_parse.c | 1 - bin/varnishtest/tests/r01092.vtc | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 34 insertions(+), 1 deletions(-) create mode 100644 bin/varnishtest/tests/r01092.vtc diff --git a/bin/varnishd/cache/cache_esi_parse.c b/bin/varnishd/cache/cache_esi_parse.c index 1d705b1..dcd2428 100644 --- a/bin/varnishd/cache/cache_esi_parse.c +++ b/bin/varnishd/cache/cache_esi_parse.c @@ -706,7 +706,6 @@ VEP_Parse(const struct worker *wrk, const char *p, size_t l) vep->until_p = vep->until = "-->"; vep->until_s = VEP_NEXTTAG; vep->state = VEP_UNTIL; - vep_mark_verbatim(vep, p); break; } p++; diff --git a/bin/varnishtest/tests/r01092.vtc b/bin/varnishtest/tests/r01092.vtc new file mode 100644 index 0000000..26b0b9c --- /dev/null +++ b/bin/varnishtest/tests/r01092.vtc @@ -0,0 +1,34 @@ +varnishtest "Test case for #1092 - esi:remove and comments" + +server s1 { + rxreq + txresp -body { + + Keep-1 + + + + Remove-1 + + + Remove-4444 + + Keep-4444 + + } +} -start + +varnish v1 -vcl+backend { + sub vcl_fetch { + set beresp.do_esi = true; + } +} -start -cliok "param.set esi_syntax 4" + +client c1 { + txreq + rxresp + expect resp.bodylen == 80 +} + +client c1 -run +varnish v1 -expect esi_errors == 1 -- 1.7.4.1 From fgsch at lodoss.net Wed Feb 15 01:27:54 2012 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Wed, 15 Feb 2012 01:27:54 +0000 Subject: [PATCH] add a least-connection director In-Reply-To: <20120121101427.1e4d788b.fgsch@lodoss.net> References: <20120121101427.1e4d788b.fgsch@lodoss.net> Message-ID: <20120215012754.6c676cc2.fgsch@lodoss.net> On Sat, 21 Jan 2012 10:14:27 +0000 "Federico G. Schwindt" wrote: > Hi, > > The diff below adds a least-connection director based on the random director. > Documentation and tests included. > Comments? OKs? Updated diff against trunk. Anyone? bin/varnishd/cache/cache_backend.h | 1 + bin/varnishd/cache/cache_backend_cfg.c | 2 + bin/varnishd/cache/cache_dir_random.c | 52 ++++++++++++++++++-- bin/varnishtest/tests/v00038.vtc | 86 ++++++++++++++++++++++++++++++++ doc/sphinx/reference/vcl.rst | 11 ++++ lib/libvcl/vcc_backend.c | 1 + 6 files changed, 149 insertions(+), 4 deletions(-) diff --git a/bin/varnishd/cache/cache_backend.h b/bin/varnishd/cache/cache_backend.h index b1688f9..82f32c2 100644 --- a/bin/varnishd/cache/cache_backend.h +++ b/bin/varnishd/cache/cache_backend.h @@ -192,3 +192,4 @@ dir_init_f VRT_init_dir_random; dir_init_f VRT_init_dir_round_robin; dir_init_f VRT_init_dir_fallback; dir_init_f VRT_init_dir_client; +dir_init_f VRT_init_dir_least_connection; diff --git a/bin/varnishd/cache/cache_backend_cfg.c b/bin/varnishd/cache/cache_backend_cfg.c index c2d022d..9bed6ed 100644 --- a/bin/varnishd/cache/cache_backend_cfg.c +++ b/bin/varnishd/cache/cache_backend_cfg.c @@ -263,6 +263,8 @@ VRT_init_dir(struct cli *cli, struct director **dir, const char *name, VRT_init_dir_fallback(cli, dir, idx, priv); else if (!strcmp(name, "client")) VRT_init_dir_client(cli, dir, idx, priv); + else if (!strcmp(name, "least-connection")) + VRT_init_dir_least_connection(cli, dir, idx, priv); else INCOMPL(); } diff --git a/bin/varnishd/cache/cache_dir_random.c b/bin/varnishd/cache/cache_dir_random.c index b323343..7cd0708 100644 --- a/bin/varnishd/cache/cache_dir_random.c +++ b/bin/varnishd/cache/cache_dir_random.c @@ -26,8 +26,9 @@ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * - * This code is shared between the random, client and hash directors, because - * they share the same properties and most of the same selection logic. + * This code is shared between the random, client, hash and least-connection + * directors, because they share the same properties and most of the same + * selection logic. * * The random director picks a backend on random. * @@ -35,6 +36,9 @@ * * The client director picks based on client identity or IP-address * + * The least-connection director picks a backend with the least number of + * established connections. + * * In all cases, the choice is by weight of the healthy subset of * configured backends. * @@ -63,7 +67,7 @@ struct vdi_random_host { double weight; }; -enum crit_e {c_random, c_hash, c_client}; +enum crit_e {c_random, c_hash, c_client, c_least_connection}; struct vdi_random { unsigned magic; @@ -185,6 +189,36 @@ vdi_random_getfd(const struct director *d, struct sess *sp) return (NULL); } +static struct vbc * +vdi_least_connection_getfd(const struct director *d, struct sess *sp) +{ + int i, m; + struct vdi_random *vs; + struct backend *bi, *bm; + + CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); + CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC); + CAST_OBJ_NOTNULL(vs, d->priv, VDI_RANDOM_MAGIC); + + for (m = 0; m < vs->nhosts; m++) { + if (!VDI_Healthy(vs->hosts[m].backend, sp)) + continue; + bm = vdi_get_backend_if_simple(vs->hosts[m].backend); + for (i = m + 1; i < vs->nhosts; i++) { + if (!VDI_Healthy(vs->hosts[i].backend, sp)) + continue; + bi = vdi_get_backend_if_simple(vs->hosts[i].backend); + if (bm->refcount * vs->hosts[i].weight > + bi->refcount * vs->hosts[m].weight) { + bm = bi; + m = i; + } + } + return (VDI_GetFd(vs->hosts[m].backend, sp)); + } + return (NULL); +} + /* * Healthy if just a single backend is... */ @@ -241,7 +275,10 @@ vrt_init(struct cli *cli, struct director **bp, int idx, vs->dir.priv = vs; vs->dir.name = "random"; REPLACE(vs->dir.vcl_name, t->name); - vs->dir.getfd = vdi_random_getfd; + if (criteria == c_least_connection) + vs->dir.getfd = vdi_least_connection_getfd; + else + vs->dir.getfd = vdi_random_getfd; vs->dir.fini = vdi_random_fini; vs->dir.healthy = vdi_random_healthy; @@ -283,3 +320,10 @@ VRT_init_dir_client(struct cli *cli, struct director **bp, int idx, { vrt_init(cli, bp, idx, priv, c_client); } + +void +VRT_init_dir_least_connection(struct cli *cli, struct director **bp, int idx, + const void *priv) +{ + vrt_init(cli, bp, idx, priv, c_least_connection); +} diff --git a/bin/varnishtest/tests/v00038.vtc b/bin/varnishtest/tests/v00038.vtc new file mode 100644 index 0000000..f392f6f --- /dev/null +++ b/bin/varnishtest/tests/v00038.vtc @@ -0,0 +1,86 @@ +varnishtest "Test least-connection director" + +server s1 { + rxreq + sema r1 sync 3 + sema r1 sync 2 + txresp -hdr "be: s1" + close + + accept + rxreq + txresp -hdr "be: s1" +} -start + +server s2 { + rxreq + sema r1 sync 3 + txresp -hdr "be: s2" + close + + accept + rxreq + txresp -hdr "be: s2" +} -start + +server s3 { + rxreq + txresp -hdr "be: s3" + close + + accept + rxreq + txresp -hdr "be: s3" +} -start + +varnish v1 -vcl+backend { + director lc least-connection { + { .backend = s1; .weight = 1; } + { .backend = s2; .weight = 1; } + { .backend = s3; .weight = 1; } + } + sub vcl_recv { + set req.backend = lc; + return (pass); + } +} -start + +client c1 { + txreq + rxresp + expect resp.status == 200 +} -start + +client c2 { + txreq + rxresp + expect resp.status == 200 +} -start + +delay .5 + +client c3 { + txreq + rxresp + expect resp.status == 200 + + txreq + rxresp + expect resp.http.be == "s3" + + sema r1 sync 3 + + txreq + rxresp + expect resp.http.be == "s2" + + sema r1 sync 2 + + txreq + rxresp + expect resp.http.be == "s1" +} -start + +client c1 -wait +client c2 -wait +client c3 -wait diff --git a/doc/sphinx/reference/vcl.rst b/doc/sphinx/reference/vcl.rst index 1ef14d6..373e6f8 100644 --- a/doc/sphinx/reference/vcl.rst +++ b/doc/sphinx/reference/vcl.rst @@ -275,6 +275,17 @@ An example of a fallback director:: // are unhealthy. } +The least-connection director +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The least-connection director will pick a backend with the least number of +established connections. + +Each backend requires a .weight option which sets the amount of traffic +each backend will get compared to the others. Equal weight means equal +traffic. A backend with lower weight than an other will get proportionally +less traffic. + Backend probes -------------- diff --git a/lib/libvcl/vcc_backend.c b/lib/libvcl/vcc_backend.c index fb160e7..d1a8a31 100644 --- a/lib/libvcl/vcc_backend.c +++ b/lib/libvcl/vcc_backend.c @@ -688,6 +688,7 @@ static const struct dirlist { { "hash", vcc_ParseRandomDirector }, { "random", vcc_ParseRandomDirector }, { "client", vcc_ParseRandomDirector }, + { "least-connection", vcc_ParseRandomDirector }, { "round-robin", vcc_ParseRoundRobinDirector }, { "fallback", vcc_ParseRoundRobinDirector }, { "dns", vcc_ParseDnsDirector }, From varnish at bsdchicks.com Wed Feb 15 22:58:45 2012 From: varnish at bsdchicks.com (Rogier R. Mulhuijzen) Date: Wed, 15 Feb 2012 23:58:45 +0100 (CET) Subject: libvmod_urlcode Message-ID: <20120215235520.G37583@ishtar.drwilco.net> L.S., I made a vmod! Process was slightly painful, even starting from vmod_example. Turns out vmod_example doesn't like master, for instance. Anyhoo, for people needing URL(percent) encoding in their VCL, there now is: https://github.com/fastly/libvmod-urlcode Courtesy of Fastly Inc. Cheers, Doc From perbu at varnish-software.com Thu Feb 16 14:45:27 2012 From: perbu at varnish-software.com (Per Buer) Date: Thu, 16 Feb 2012 15:45:27 +0100 Subject: VMOD listing Message-ID: We finally got the vmod listing up and running. We had some weird problems with drupal and a server crash which didn't really help and it all took a tad bit too long. Anyway, the idea is to have a listing of the all the VMODs that are in active use. If you want to register your VMOD just sign up for an account at https://www.varnish-cache.org/user/register and then go to https://www.varnish-cache.org/node/add/vmod (or click add content / VMOD) and fill out the form. Once we verify there is not goatse references in the listing it will be published on https://www.varnish-cache.org/vmods We're having some weird integration issues with Drupal 7.12 and Varnish (we blame drupal, of course :-) so there is some latency before stuff shows up (last when we tried enabling Varnish integration the whole session handling went down the drain) but we'll probably figure it out. Temporarily there is a picture of me on all the vmods I submitted but we'll clear that soon. In the mean time you can all enjoy my pretty face. Cheers, Per. -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From ruben at varnish-software.com Thu Feb 16 18:29:51 2012 From: ruben at varnish-software.com (=?UTF-8?Q?Rub=C3=A9n_Romero?=) Date: Thu, 16 Feb 2012 19:29:51 +0100 Subject: VMOD listing In-Reply-To: References: Message-ID: On Thu, Feb 16, 2012 at 3:45 PM, Per Buer wrote: > We finally got the vmod listing up and running. We had some weird problems > with drupal and a server crash which didn't really help and it all took a > tad bit too long. > > So, I had my own document on the currently available VMODs. We have now 16 in the overview and I know of at least 3 additional VMODs that will be added in the future (2 of them doing GeoIP). Before I can do that I need the MIT License to be added to the license choices. Could you please make it happen or shall I just Other? Thanks to you (and Tollef) for getting this up and running! :-) Best, -- Rub?n Romero Phone: +47 21 98 92 62 / Mobile: +47 959 64 088 / Skype: ruben_varnish / GTalk: *Varnish makes websites fly!* Whitepapers | Video | Twitter | LinkedIn -------------- next part -------------- An HTML attachment was scrubbed... URL: From falcao at intelie.com.br Fri Feb 17 14:55:08 2012 From: falcao at intelie.com.br (=?ISO-8859-1?Q?Jorge_Falc=E3o?=) Date: Fri, 17 Feb 2012 12:55:08 -0200 Subject: FastCGI backend Message-ID: Is somebody working at some "fastcgi-backend" implementation? -- Jorge Falc?o, Intelie - Intelig?ncia em opera??o http://www.intelie.com.br/ 55 21 2240-1193 / 9373-8671 From Gauthier.Delacroix at coreye.fr Fri Feb 17 15:49:32 2012 From: Gauthier.Delacroix at coreye.fr (Delacroix, Gauthier) Date: Fri, 17 Feb 2012 16:49:32 +0100 Subject: Size issues in ncsa log with gzip Message-ID: <8349678DF4CCC3428201CED9DF7B4D9C1187445F@richter.lild01.pictime.fr> Hi, I'm currently migrating from Varnish 2.0.6 to 3.0.2 (yup, big jump!) and played a little with gzip feature but I'm a little disappointed about its consequences on ncsa logs (I need to parse this log to build per host statistics). 1. It sound logical but NCSA size of a gzipped request is the compressed size, and I see no way to get uncompressed size for my statistics 2. When responding to a client not supporting compression (no "Accept-Encoding" header), Varnish send a chunked response. Of course there is no "Content-Length", but varnishncsa also shows zero as a size... Is there a way to ask varnishncsa to log the "real size" of an object instead of its "Content-Length" value ? It seems obvious that varnish do not know the size of the gzipped object, but could there be a way to "store" the real size with the object itself ? Thanks ! Gauthier From jspam at skopis.com Fri Feb 17 19:52:30 2012 From: jspam at skopis.com (John Skopis) Date: Fri, 17 Feb 2012 13:52:30 -0600 Subject: feasability of adding field to sess struct Message-ID: Hi, I posted to varnish-misc recently asking how I might be able to propgate the http status code from an esi subrequest to the parent request, ie esi beresp.status is 5xx, set resp.status to 5xx in vcl_deliver. At this point I am pretty sure its not possible. I am wondering if the best way to make it possible is to add a member to struct sess { int esi_status; }? What I was thinking was that somewhere there is a function that sets up a session where wsi_status would be initialized to 200. Somewhere there is a function that does the fetch, in which I would check the http status and if it's not 200 updating esi_status on the session. Additionally I would expose esi_status for reading and writing in vcl. For me at least the canonical way to use esi_status in vcl would be to check if the http response is in an esi context (using a header set by the backend) and possibly override the esi_status, and ultimately set the resp.status to req.esi_status. I am not too familiar with the varnish source and before I jump into the magic that is the varnish FSM I am wondering Would you do this? Is there a better way? How might I go about doing this? I am not sure if esi requests are synchronous or not. Is a mutex required to update the esi_status on the session? Anything else? Thanks, John From phk at phk.freebsd.dk Fri Feb 17 20:12:19 2012 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 17 Feb 2012 20:12:19 +0000 Subject: feasability of adding field to sess struct In-Reply-To: Your message of "Fri, 17 Feb 2012 13:52:30 CST." Message-ID: <6634.1329509539@critter.freebsd.dk> In message , John Skopis writes: >I posted to varnish-misc recently asking how I might be able to >propgate the http status code from an esi subrequest to the parent >request, ie esi beresp.status is 5xx, set resp.status to 5xx in >vcl_deliver. That is not currently possible, because we deliver the ESI response as we build it, so by the time the sub request is handled, the status code is long since sent. Somewhere on the todo list is "buffered ESI" which would assemble the entire response in memory, before sending it, and then we can start to look at such features. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From jammy.linux at gmail.com Mon Feb 20 03:05:22 2012 From: jammy.linux at gmail.com (Jammy) Date: Mon, 20 Feb 2012 11:05:22 +0800 Subject: chunked response within first session Message-ID: Hi~@list I noticed that in the fetch session the chunked responses would be aggregated and returned. Is there any concern about this behavior? Can we return chunked response even if in the fetch session? Any feedback will be appreciated. Thanks. ---------------------------------- Best wishes, Jammy -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfheen at varnish-software.com Tue Feb 21 11:01:14 2012 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 21 Feb 2012 12:01:14 +0100 Subject: chunked response within first session In-Reply-To: References: Message-ID: <20120221110114.GA18101@err.no> ]] Jammy > I noticed that in the fetch session the chunked responses would be aggregated and returned. Yes this is normal. > Is there any concern about this behavior? Can we return chunked response even if in the fetch session? Why would you want to do that? -- Tollef Fog Heen Technical lead, Varnish Software t: +47 21 98 92 64 From tfheen at varnish-software.com Tue Feb 21 11:03:29 2012 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 21 Feb 2012 12:03:29 +0100 Subject: Size issues in ncsa log with gzip In-Reply-To: <8349678DF4CCC3428201CED9DF7B4D9C1187445F@richter.lild01.pictime.fr> References: <8349678DF4CCC3428201CED9DF7B4D9C1187445F@richter.lild01.pictime.fr> Message-ID: <20120221110329.GB18101@err.no> ]] Delacroix, Gauthier > 1. It sound logical but NCSA size of a gzipped request is the compressed > size, and I see no way to get uncompressed size for my statistics 2. > When responding to a client not supporting compression (no > "Accept-Encoding" header), Varnish send a chunked response. Of course > there is no "Content-Length", but varnishncsa also shows zero as a > size... The missing size for chunked requests sounds like a bug. Please file one. > Is there a way to ask varnishncsa to log the "real size" of an object > instead of its "Content-Length" value ? > It seems obvious that varnish do not know the size of the gzipped > object, but could there be a way to "store" the real size with the > object itself ? Assuming your backend sends content-length you can store it as X-Content-Length and then grab that header instead for the length field. -- Tollef Fog Heen Technical lead, Varnish Software t: +47 21 98 92 64 From jammy.linux at gmail.com Wed Feb 22 01:59:55 2012 From: jammy.linux at gmail.com (Jammy) Date: Wed, 22 Feb 2012 09:59:55 +0800 Subject: chunked response within first session In-Reply-To: <20120221110114.GA18101@err.no> References: <20120221110114.GA18101@err.no> Message-ID: <62680C39-4D32-41A7-AF99-F1BE20B55941@gmail.com> On 21 Feb, 2012, at 7:01 PM, Tollef Fog Heen wrote: > ]] Jammy > >> I noticed that in the fetch session the chunked responses would be aggregated and returned. > > Yes this is normal. > >> Is there any concern about this behavior? Can we return chunked response even if in the fetch session? > > Why would you want to do that? I'd like to have varnish stand in front of the web-server, even for HTML assets. This normal behavior may ruin the Time-To-FirstByte metrics;) > > -- > Tollef Fog Heen > Technical lead, Varnish Software > t: +47 21 98 92 64 ---------------------------------- Best wishes, Jammy -------------- next part -------------- An HTML attachment was scrubbed... URL: From l at lrowe.co.uk Wed Feb 22 18:31:08 2012 From: l at lrowe.co.uk (Laurence Rowe) Date: Wed, 22 Feb 2012 18:31:08 +0000 Subject: chunked response within first session In-Reply-To: <62680C39-4D32-41A7-AF99-F1BE20B55941@gmail.com> References: <20120221110114.GA18101@err.no> <62680C39-4D32-41A7-AF99-F1BE20B55941@gmail.com> Message-ID: On 22 February 2012 01:59, Jammy wrote: > > > On 21 Feb, 2012, at 7:01 PM, Tollef Fog Heen wrote: > > ]] Jammy > > I noticed that in the fetch session the chunked responses would be > aggregated and returned. > > > Yes this is normal. > > Is there any concern about this behavior? Can we return chunked response > even if in the fetch session? > > > Why would you want to do that? > > > I'd like to have varnish stand in front of the web-server, even for HTML > assets. This normal behavior may ruin the Time-To-FirstByte metrics;) Perhaps you should investigate the streaming support in Varnish three: https://www.varnish-software.com/blog/http-streaming-varnish and currently being worked on: https://www.varnish-software.com/blog/http-streaming-varnish Laurence From artur at fastly.com Tue Feb 21 21:10:17 2012 From: artur at fastly.com (Artur Bergman) Date: Tue, 21 Feb 2012 13:10:17 -0800 Subject: Fix for saintmode Message-ID: <57B3EDB1-4BDB-4656-B1F8-50366D970610@fastly.com> Saintmode sadly locks on every request to vbe_healthy, this means if you do req.backend.healty you acquire the lock. It does this even if the list is empty. Following patch fix this, it introduces a tiny race which I am happy to live with. https://raw.github.com/gist/1878965/b1b31dd188d524e8f56b4700da34f2233b125578/saintmode-sane --- a/varnish-cache/bin/varnishd/cache_backend.c +++ b/varnish-cache/bin/varnishd/cache_backend.c @@ -266,7 +266,7 @@ vbe_Healthy(double now, uintptr_t target, struct backend *backend) threshold = backend->saintmode_threshold; /* Saintmode is disabled */ - if (threshold == 0) + if (threshold == 0 || VTAILQ_EMPTY(&backend->troublelist)) return (1); /* No need to test if we don't have an object head to test against. From artur at fastly.com Tue Feb 21 21:10:17 2012 From: artur at fastly.com (Artur Bergman) Date: Tue, 21 Feb 2012 13:10:17 -0800 Subject: Fix for saintmode Message-ID: <57B3EDB1-4BDB-4656-B1F8-50366D970610@fastly.com> Saintmode sadly locks on every request to vbe_healthy, this means if you do req.backend.healty you acquire the lock. It does this even if the list is empty. Following patch fix this, it introduces a tiny race which I am happy to live with. https://raw.github.com/gist/1878965/b1b31dd188d524e8f56b4700da34f2233b125578/saintmode-sane --- a/varnish-cache/bin/varnishd/cache_backend.c +++ b/varnish-cache/bin/varnishd/cache_backend.c @@ -266,7 +266,7 @@ vbe_Healthy(double now, uintptr_t target, struct backend *backend) threshold = backend->saintmode_threshold; /* Saintmode is disabled */ - if (threshold == 0) + if (threshold == 0 || VTAILQ_EMPTY(&backend->troublelist)) return (1); /* No need to test if we don't have an object head to test against. From github at bsdchicks.com Wed Feb 22 05:42:33 2012 From: github at bsdchicks.com (Rogier 'DocWilco' Mulhuijzen) Date: Tue, 21 Feb 2012 21:42:33 -0800 Subject: [PATCH] Don't keep object contents around for hit-for-pass after delivering Message-ID: <1329889353-16850-1-git-send-email-github@bsdchicks.com> So, while going over various things, we discovered that hit-for-pass objects actually keep the content around as well. We found that to be an enormous waste of space, and decided to do something about it. Namely: --- bin/varnishd/cache/cache_center.c | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-) diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c index b6ee5a5..cfe2619 100644 --- a/bin/varnishd/cache/cache_center.c +++ b/bin/varnishd/cache/cache_center.c @@ -333,6 +333,13 @@ cnt_deliver(struct sess *sp, struct worker *wrk, struct req *req) req->restarts = 0; RES_WriteObj(sp); + + CHECK_OBJ_NOTNULL(req->obj, OBJECT_MAGIC); + if (req->obj->objcore != NULL) { + CHECK_OBJ_NOTNULL(req->obj->objcore, OBJCORE_MAGIC); + if (req->obj->objcore->flags & OC_F_PASS) + STV_Freestore(req->obj); + } assert(WRW_IsReleased(wrk)); (void)HSH_Deref(&wrk->stats, NULL, &req->obj); -- 1.7.5.4 From jammy.linux at gmail.com Thu Feb 23 06:00:56 2012 From: jammy.linux at gmail.com (Jammy) Date: Thu, 23 Feb 2012 14:00:56 +0800 Subject: chunked response within first session In-Reply-To: References: <20120221110114.GA18101@err.no> <62680C39-4D32-41A7-AF99-F1BE20B55941@gmail.com> Message-ID: <3833A438-2FEC-402E-A1D4-8157243ADBA4@gmail.com> Thank you for your pointer. On 23 Feb, 2012, at 2:31 AM, Laurence Rowe wrote: > On 22 February 2012 01:59, Jammy wrote: >> >> >> On 21 Feb, 2012, at 7:01 PM, Tollef Fog Heen wrote: >> >> ]] Jammy >> >> I noticed that in the fetch session the chunked responses would be >> aggregated and returned. >> >> >> Yes this is normal. >> >> Is there any concern about this behavior? Can we return chunked response >> even if in the fetch session? >> >> >> Why would you want to do that? >> >> >> I'd like to have varnish stand in front of the web-server, even for HTML >> assets. This normal behavior may ruin the Time-To-FirstByte metrics;) > > Perhaps you should investigate the streaming support in Varnish three: > > https://www.varnish-software.com/blog/http-streaming-varnish > > and currently being worked on: > > https://www.varnish-software.com/blog/http-streaming-varnish > > Laurence ---------------------------------- Best wishes, Jammy -------------- next part -------------- An HTML attachment was scrubbed... URL: From fgsch at lodoss.net Thu Feb 23 15:39:18 2012 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Thu, 23 Feb 2012 15:39:18 +0000 Subject: [PATCH] some documentation love Message-ID: <20120223153918.ff3e891a.fgsch@lodoss.net> Minor touches: - mention vmod_std(7) - HEADER is released and used by vmod_std(7) so put it in its own line. The explanation is still needed though. - for consistency replace leading tabs with spaces - *fallback* are not strings so avoid mentioning the type - use ascii characters in vmod_std(7) OK? diff --git a/doc/sphinx/reference/vcl.rst b/doc/sphinx/reference/vcl.rst index 9e71fbd..43197f6 100644 --- a/doc/sphinx/reference/vcl.rst +++ b/doc/sphinx/reference/vcl.rst @@ -275,6 +275,17 @@ An example of a fallback director:: // are unhealthy. } +The least-connection director +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The least-connection director will pick a backend with the least number of +established connections. + +Each backend requires a .weight option which sets the amount of traffic +each backend will get compared to the others. Equal weight means equal +traffic. A backend with lower weight than an other will get proportionally +less traffic. + Backend probes -------------- diff --git a/doc/sphinx/reference/vmod.rst b/doc/sphinx/reference/vmod.rst index 57163c5..a76dee1 100644 --- a/doc/sphinx/reference/vmod.rst +++ b/doc/sphinx/reference/vmod.rst @@ -22,7 +22,7 @@ For instance:: The "std" vmod is one you get with Varnish, it will always be there and we will put "boutique" functions in it, such as the "toupper" function shown above. The full contents of the "std" module is -documented in XXX:TBW. +documented in vmod_std(7). This part of the manual is about how you go about writing your own VMOD, how the language interface between C and VCC works etc. This @@ -185,7 +185,12 @@ VOID Can only be used for return-value, which makes the function a VCL procedure. -IP, BOOL, HEADER +HEADER + C-type: ``enum gethdr_e, const char *`` + + XXX: explain me + +IP, BOOL XXX: these types are not released for use in vmods yet. diff --git a/doc/sphinx/reference/vmod_std.rst b/doc/sphinx/reference/vmod_std.rst index 1ea8193..f1a8ba2 100644 --- a/doc/sphinx/reference/vmod_std.rst +++ b/doc/sphinx/reference/vmod_std.rst @@ -33,7 +33,7 @@ toupper Prototype toupper(STRING S) Return value - String + String Description Converts the STRING S to upper case. Example @@ -44,18 +44,18 @@ tolower Prototype tolower(STRING S) Return value - String + String Description Converts the STRING to lower case. Example - set beresp.http.x-nice = std.tolower("VerY"); + set beresp.http.x-nice = std.tolower("VerY"); set_up_tos ---------- Prototype set_ip_tos(INT I) Return value - Void + Void Description Sets the Type-of-Service flag for the current session. Please note that the TOS flag is not removed by the end of the @@ -71,7 +71,7 @@ random Prototype random(REAL a, REAL b) Return value - Real + Real Description Returns a random REAL number between *a* and *b*. Example @@ -82,7 +82,7 @@ log Prototype log(STRING string) Return value - Void + Void Description Logs string to the shared memory log. Example @@ -93,7 +93,7 @@ syslog Prototype syslog(INT priority, STRING string) Return value - Void + Void Description Logs *string* to syslog marked with *priority*. Example @@ -104,7 +104,7 @@ fileread Prototype fileread(STRING filename) Return value - String + String Description Reads a file and returns a string with the content. Please note that it is not recommended to send variables to this @@ -118,12 +118,11 @@ duration Prototype duration(STRING s, DURATION fallback) Return value - Duration + Duration Description Converts the string s to seconds. s can be quantified with the usual s (seconds), m (minutes), h (hours), d (days) and w - (weeks) units. If it fails to parse the string *fallback* - will be used + (weeks) units. If it fails to parse *fallback* will be used Example set beresp.ttl = std.duration("1w", 3600); @@ -132,19 +131,19 @@ integer Prototype integer(STRING s, INT fallback) Return value - Int + Int Description - Converts the string s to an integer. If it fails to parse the - string *fallback* will be used + Converts the string s to an integer. If it fails to parse + *fallback* will be used Example - if (std.integer(beresp.http.x-foo, 0) > 5) { ? } + if (std.integer(beresp.http.x-foo, 0) > 5) { ... } collect ------- Prototype collect(HEADER header) Return value - Void + Void Description Collapses the header, joining the headers into one. Example From fgsch at lodoss.net Thu Feb 23 16:22:08 2012 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Thu, 23 Feb 2012 16:22:08 +0000 Subject: [PATCH] some documentation love In-Reply-To: <20120223153918.ff3e891a.fgsch@lodoss.net> References: <20120223153918.ff3e891a.fgsch@lodoss.net> Message-ID: <20120223162208.908f34f8.fgsch@lodoss.net> Ops, you can ignore the vcl.rst change (first chunk). That's part of another diff I sent previously. On Thu, 23 Feb 2012 15:39:18 +0000 "Federico G. Schwindt" wrote: > Minor touches: > > - mention vmod_std(7) > - HEADER is released and used by vmod_std(7) so put it in its own line. > The explanation is still needed though. > - for consistency replace leading tabs with spaces > - *fallback* are not strings so avoid mentioning the type > - use ascii characters in vmod_std(7) > > OK? > > diff --git a/doc/sphinx/reference/vcl.rst b/doc/sphinx/reference/vcl.rst > index 9e71fbd..43197f6 100644 > --- a/doc/sphinx/reference/vcl.rst > +++ b/doc/sphinx/reference/vcl.rst > @@ -275,6 +275,17 @@ An example of a fallback director:: > // are unhealthy. > } > > +The least-connection director > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +The least-connection director will pick a backend with the least number of > +established connections. > + > +Each backend requires a .weight option which sets the amount of traffic > +each backend will get compared to the others. Equal weight means equal > +traffic. A backend with lower weight than an other will get proportionally > +less traffic. > + > Backend probes > -------------- > > diff --git a/doc/sphinx/reference/vmod.rst b/doc/sphinx/reference/vmod.rst > index 57163c5..a76dee1 100644 > --- a/doc/sphinx/reference/vmod.rst > +++ b/doc/sphinx/reference/vmod.rst > @@ -22,7 +22,7 @@ For instance:: > The "std" vmod is one you get with Varnish, it will always be there > and we will put "boutique" functions in it, such as the "toupper" > function shown above. The full contents of the "std" module is > -documented in XXX:TBW. > +documented in vmod_std(7). > > This part of the manual is about how you go about writing your own > VMOD, how the language interface between C and VCC works etc. This > @@ -185,7 +185,12 @@ VOID > Can only be used for return-value, which makes the function a VCL > procedure. > > -IP, BOOL, HEADER > +HEADER > + C-type: ``enum gethdr_e, const char *`` > + > + XXX: explain me > + > +IP, BOOL > XXX: these types are not released for use in vmods yet. > > > diff --git a/doc/sphinx/reference/vmod_std.rst b/doc/sphinx/reference/vmod_std.rst > index 1ea8193..f1a8ba2 100644 > --- a/doc/sphinx/reference/vmod_std.rst > +++ b/doc/sphinx/reference/vmod_std.rst > @@ -33,7 +33,7 @@ toupper > Prototype > toupper(STRING S) > Return value > - String > + String > Description > Converts the STRING S to upper case. > Example > @@ -44,18 +44,18 @@ tolower > Prototype > tolower(STRING S) > Return value > - String > + String > Description > Converts the STRING to lower case. > Example > - set beresp.http.x-nice = std.tolower("VerY"); > + set beresp.http.x-nice = std.tolower("VerY"); > > set_up_tos > ---------- > Prototype > set_ip_tos(INT I) > Return value > - Void > + Void > Description > Sets the Type-of-Service flag for the current session. Please > note that the TOS flag is not removed by the end of the > @@ -71,7 +71,7 @@ random > Prototype > random(REAL a, REAL b) > Return value > - Real > + Real > Description > Returns a random REAL number between *a* and *b*. > Example > @@ -82,7 +82,7 @@ log > Prototype > log(STRING string) > Return value > - Void > + Void > Description > Logs string to the shared memory log. > Example > @@ -93,7 +93,7 @@ syslog > Prototype > syslog(INT priority, STRING string) > Return value > - Void > + Void > Description > Logs *string* to syslog marked with *priority*. > Example > @@ -104,7 +104,7 @@ fileread > Prototype > fileread(STRING filename) > Return value > - String > + String > Description > Reads a file and returns a string with the content. Please > note that it is not recommended to send variables to this > @@ -118,12 +118,11 @@ duration > Prototype > duration(STRING s, DURATION fallback) > Return value > - Duration > + Duration > Description > Converts the string s to seconds. s can be quantified with the > usual s (seconds), m (minutes), h (hours), d (days) and w > - (weeks) units. If it fails to parse the string *fallback* > - will be used > + (weeks) units. If it fails to parse *fallback* will be used > Example > set beresp.ttl = std.duration("1w", 3600); > > @@ -132,19 +131,19 @@ integer > Prototype > integer(STRING s, INT fallback) > Return value > - Int > + Int > Description > - Converts the string s to an integer. If it fails to parse the > - string *fallback* will be used > + Converts the string s to an integer. If it fails to parse > + *fallback* will be used > Example > - if (std.integer(beresp.http.x-foo, 0) > 5) { ? } > + if (std.integer(beresp.http.x-foo, 0) > 5) { ... } > > collect > ------- > Prototype > collect(HEADER header) > Return value > - Void > + Void > Description > Collapses the header, joining the headers into one. > Example From apj at mutt.dk Thu Feb 23 17:00:47 2012 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Thu, 23 Feb 2012 18:00:47 +0100 Subject: [PATCH] some documentation love In-Reply-To: <20120223153918.ff3e891a.fgsch@lodoss.net> References: <20120223153918.ff3e891a.fgsch@lodoss.net> Message-ID: <20120223170047.GR12685@nerd.dk> On Thu, Feb 23, 2012 at 03:39:18PM +0000, Federico G. Schwindt wrote: > Minor touches: > > - mention vmod_std(7) > - HEADER is released and used by vmod_std(7) so put it in its own line. > The explanation is still needed though. > - for consistency replace leading tabs with spaces > - *fallback* are not strings so avoid mentioning the type > - use ascii characters in vmod_std(7) Thanks, applied (with some few extra changes) -- Andreas