From slink at schokola.de Wed Mar 6 14:41:49 2013 From: slink at schokola.de (Nils Goroll) Date: Wed, 06 Mar 2013 15:41:49 +0100 Subject: full disclosure reports Message-ID: <513755AD.2010808@schokola.de> FYI: * http://www.gossamer-threads.com/lists/fulldisc/full-disclosure/89110 -> looks like https://www.varnish-cache.org/trac/ticket/927 at first sight * http://www.gossamer-threads.com/lists/fulldisc/full-disclosure/89115 -> another one with ridiculously high Content-Length these ones are also reported for 3.0.3 and look like genuine issues to me: * http://www.gossamer-threads.com/lists/fulldisc/full-disclosure/89113 -> new report? (does not look like a new issue to me regarding GetHdr, but in the context of Vary parsing) * http://www.gossamer-threads.com/lists/fulldisc/full-disclosure/89107 -> Vary parsing IIUC to exploit any of these one would need access to a backend or at least some way to make a backend produce certain response headers. Nils From daghf at varnish-software.com Wed Mar 6 15:07:37 2013 From: daghf at varnish-software.com (Dag Haavi Finstad) Date: Wed, 6 Mar 2013 16:07:37 +0100 Subject: full disclosure reports In-Reply-To: <513755AD.2010808@schokola.de> References: <513755AD.2010808@schokola.de> Message-ID: For the record, I have added #1274 and #1275 in trac for the last two: https://www.varnish-cache.org/trac/ticket/1274 https://www.varnish-cache.org/trac/ticket/1275 On Wed, Mar 6, 2013 at 3:41 PM, Nils Goroll wrote: > FYI: > > * http://www.gossamer-threads.**com/lists/fulldisc/full-**disclosure/89110 > -> looks like https://www.varnish-cache.org/**trac/ticket/927at first sight > > * http://www.gossamer-threads.**com/lists/fulldisc/full-**disclosure/89115 > -> another one with ridiculously high Content-Length > > these ones are also reported for 3.0.3 and look like genuine issues to me: > > * http://www.gossamer-threads.**com/lists/fulldisc/full-**disclosure/89113 > -> new report? (does not look like a new issue to me regarding GetHdr, > but in the context of Vary parsing) > > * http://www.gossamer-threads.**com/lists/fulldisc/full-**disclosure/89107 > -> Vary parsing > > IIUC to exploit any of these one would need access to a backend or at > least some way to make a backend produce certain response headers. > > Nils > > ______________________________**_________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/**lists/mailman/listinfo/**varnish-dev > -- *Dag Haavi Finstad* Developer | Varnish Software AS Phone: +47 21 98 92 60 We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Wed Mar 6 15:38:47 2013 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 06 Mar 2013 15:38:47 +0000 Subject: full disclosure reports In-Reply-To: <513755AD.2010808@schokola.de> References: <513755AD.2010808@schokola.de> Message-ID: <10521.1362584327@critter.freebsd.dk> In message <513755AD.2010808 at schokola.de>, Nils Goroll writes: >IIUC to exploit any of these one would need access to a backend or at least some >way to make a backend produce certain response headers. They contacted me up front, I told them we don't consider it a security problem, because Varnish has to trust the backend being sensible. We'd be just as hosed if the backend started sending only 1TB objects. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From sky at crucially.net Wed Mar 6 15:45:29 2013 From: sky at crucially.net (Artur Bergman) Date: Wed, 6 Mar 2013 07:45:29 -0800 Subject: full disclosure reports In-Reply-To: <10521.1362584327@critter.freebsd.dk> References: <513755AD.2010808@schokola.de> <10521.1362584327@critter.freebsd.dk> Message-ID: On Mar 6, 2013, at 7:38 AM, "Poul-Henning Kamp" wrote: > In message <513755AD.2010808 at schokola.de>, Nils Goroll writes: > >> IIUC to exploit any of these one would need access to a backend or at least some >> way to make a backend produce certain response headers. > > They contacted me up front, I told them we don't consider it a security > problem, because Varnish has to trust the backend being sensible. > > We'd be just as hosed if the backend started sending only 1TB objects. > > -- Thank you for that very pragmatic and mature view of the world. Cheers Artur For the sarcasm deficient people out there, this email contains 1000% sarcasm. From phk at phk.freebsd.dk Wed Mar 6 16:23:34 2013 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 06 Mar 2013 16:23:34 +0000 Subject: full disclosure reports In-Reply-To: References: <513755AD.2010808@schokola.de> <10521.1362584327@critter.freebsd.dk> Message-ID: <10681.1362587014@critter.freebsd.dk> In message , Artur Bergman writes: >> They contacted me up front, I told them we don't consider it a security >> problem, because Varnish has to trust the backend being sensible. >> >> We'd be just as hosed if the backend started sending only 1TB objects. > >Thank you for that very pragmatic and mature view of the world. I didn't say I don't consider those issues problems, I do, I just don't consider them security problems. I do realize that Fastly's usage of Varnish is different from pretty much everybody else in the world, but that is a risk Fastly has chosen to take and the consequences are theirs to bear. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From sky at crucially.net Wed Mar 6 16:31:18 2013 From: sky at crucially.net (sky at crucially.net) Date: Wed, 6 Mar 2013 16:31:18 +0000 Subject: full disclosure reports Message-ID: <1759739879-1362587494-cardhu_decombobulator_blackberry.rim.net-68758731-@b15.c3.bise6.blackberry> I am not just speaking for us. (we didn't report some of the issues because of the seemingly lack of interest). I also don't think it is just about us, I know a lot of people front web (word press, drupal, etc) farms with varnish. Where someone could change a header but not generate 1TB objects. And previously, you have not considered these a problem at all. If you are fronting a website that you completely own, and you have 300 developers working on it. It isn't entirely unlikely one of them will forget a , in a vary. And boom your varnish is gone. Or god forbid some funny person sets a response code over 999. (oh is that new disclosure?) I've argued many times: Asserting on user input from the network should not be done, it is bad form and sloppy. EOM ------Original Message------ From: Poul-Henning Kamp To: Artur Bergman Cc: Nils Goroll Cc: varnish-dev at varnish-cache.org Subject: Re: full disclosure reports Sent: Mar 6, 2013 08:23 In message , Artur Bergman writes: >> They contacted me up front, I told them we don't consider it a security >> problem, because Varnish has to trust the backend being sensible. >> >> We'd be just as hosed if the backend started sending only 1TB objects. > >Thank you for that very pragmatic and mature view of the world. I didn't say I don't consider those issues problems, I do, I just don't consider them security problems. I do realize that Fastly's usage of Varnish is different from pretty much everybody else in the world, but that is a risk Fastly has chosen to take and the consequences are theirs to bear. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. Sent via BlackBerry by AT&T From phk at phk.freebsd.dk Wed Mar 6 17:02:33 2013 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 06 Mar 2013 17:02:33 +0000 Subject: full disclosure reports In-Reply-To: <1759739879-1362587494-cardhu_decombobulator_blackberry.rim.net-68758731-@b15.c3.bise6.blackberry> References: <1759739879-1362587494-cardhu_decombobulator_blackberry.rim.net-68758731-@b15.c3.bise6.blackberry> Message-ID: <10823.1362589353@critter.freebsd.dk> > And previously, you have not considered these a problem at all. I do and have always consider them problems, but I have a very long list of problems to deal with and limited amount of time, so things gets sorted by some kind of priority. I'm fully aware that you might prefer a different prioritization, but under present circumstances, your input in this respect is just one of many, I get to try to balance them all, and in this particular case you loose: These are not security problems. > I've argued many times: And I've said many times: Please send patches. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From ruben at varnish-software.com Wed Mar 6 21:17:29 2013 From: ruben at varnish-software.com (=?UTF-8?Q?Rub=C3=A9n_Romero?=) Date: Wed, 6 Mar 2013 22:17:29 +0100 Subject: VUG6 report and Development news Message-ID: Hello, Finally I managed to get the VUG6 report out and, in case you missed it, the agenda, slides, notes and videos from both the user and developer days are available online: https://www.varnish-cache.org/vug6 https://www.varnish-cache.org/vug6-report https://docs.google.com/a/varnish-software.com/spreadsheet/ccc?key=0Apb8MyYPKVqZdDlNTHl2dExMRllrSV9HUnBLdkxwb3c#gid=0 (VUG6 Dev Day discussion) I know it has been ages since the meeting, but the report and notes gives some nice insights into the user community and what people think about Varnish. As for development news since then, Lasse has been blogging about what is going on: http://lassekarstensen.wordpress.com/2013/01/24/varnish-cache-development-news/ http://lassekarstensen.wordpress.com/2013/02/27/varnish-development-news-feb-2013/ Enjoy! Cheers, -- *Rub?n Romero* Global Sales Executive & Community Cheerleader | Varnish Software AS Cell: +47 95964088 / Office: +47 21989260 Skype & Twitter: ruben_varnish We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From theixos at gmail.com Thu Mar 7 15:53:50 2013 From: theixos at gmail.com (The Ixos) Date: Thu, 7 Mar 2013 16:53:50 +0100 Subject: Adding statistics to varnish Message-ID: Hi, I would like to add to Varnish some statistics (which are not available in the VCL). They will be logged by calling syslog in cnt_done after the end of each request. I am just beginning the adventure of varnish and therefore I have a few questions: 1) where to put the statistics variable (int, char *) so that they won't be overwritten by another thread - the struct sess, sessmem or somewhere else? 2) where to put initialization of variables that they will be initialized before each request - VCA_Prep? When I view logs I see that Content-Lengt << clinet_out, I mean Content-Length is couple times smaller then client_out. It looks like clinet_out isn't cleared at the beggining of request or another thread add something to it between sending data and saving logs. While the same time backend_in, backend_out and client_in data seems to be correct. This error occurs in about 5-10% of requests and I'm not able to reproduce it on a machine. Here's how I'm doing it now: /* cache.h */ struct conn_stat{ unsigned magic; #define CONN_STAT_MAGIC 0xf6d1bf char *host; char *addr; unsigned long long client_in; unsigned long long client_out; unsigned long long backend_in; unsigned long long backend_out; }; struct sess { /* ... */ struct conn_stat conn_stat; unsigned long long *wbytes; /* Where to put number of bytes written */ unsigned long long *rbytes; /* Where to put number of bytes read */ /* ... */ }; /* cache_session.c: */ struct sessmem { /* ... */ struct conn_stat *conn_stat; /* ... */ }; //ses_sm_alloc() sm = (void*)p; p += sizeof *sm; sm->magic = SESSMEM_MAGIC; sm->workspace = nws; sm->conn_stat = p; p += cl; sm->http[0] = HTTP_create(p, nhttp); p += hl; sm->http[1] = HTTP_create(p, nhttp); p += hl; sm->wsp = p; p += nws; /* cache_center.c */ static int cnt_streamdeliver(struct sess *sp) { struct busyobj *bo; bo = sp->stream_busyobj; CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC); sp->wbytes = &sp->conn_stat.client_out; *sp->wbytes = 0; RES_StreamStart(sp); sp->wrk->h_content_length = NULL; if (sp->wantbody) RES_StreamBody(sp); RES_StreamEnd(sp); sp->wbytes = NULL; /* ... */ static int cnt_done(struct sess *sp) { double dh, dp, da; int i; CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); CHECK_OBJ_ORNULL(sp->vcl, VCL_CONF_MAGIC); if(sp->vcl != NULL) { syslog(LOG_INFO, "%llu", sp->conn_stat->client_out); } -------------- next part -------------- An HTML attachment was scrubbed... URL: From fgsch at lodoss.net Sat Mar 9 07:09:52 2013 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Sat, 9 Mar 2013 07:09:52 +0000 Subject: [PATCH] Fix compilation under OpenBSD Message-ID: <20130309070952.a89aaf397a54067f3e4bd880@lodoss.net> OpenBSD is not defined (unless sys/param.h is included) so simply use the gcc builtin define for it. diff --git a/bin/varnishd/mgt/mgt_shmem.c b/bin/varnishd/mgt/mgt_shmem.c index 7707309..3974b43 100644 --- a/bin/varnishd/mgt/mgt_shmem.c +++ b/bin/varnishd/mgt/mgt_shmem.c @@ -247,7 +247,7 @@ mgt_SHM_Create(void) exit (-1); } -#ifdef OpenBSD +#if defined(__OpenBSD__) /* Commit changes, for OS's without coherent VM/buf */ AZ(msync(p, getpagesize(), MS_SYNC)); #endif From bkerensa at ubuntu.com Mon Mar 11 02:28:58 2013 From: bkerensa at ubuntu.com (Benjamin Kerensa) Date: Sun, 10 Mar 2013 19:28:58 -0700 Subject: [PATCH] Fix for the typos Message-ID: >From ff54983150df5c9c5a33a6f749cc52ffc17724ac Mon Sep 17 00:00:00 2001 From: Benjamin Kerensa Date: Sun, 10 Mar 2013 19:23:52 -0700 Subject: [PATCH] Fix Typos --- bin/varnishd/mgt/mgt_main.c | 2 +- bin/varnishd/mgt/mgt_param.c | 2 +- doc/sphinx/reference/params.rst | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c index b41133c..a111481 100644 --- a/bin/varnishd/mgt/mgt_main.c +++ b/bin/varnishd/mgt/mgt_main.c @@ -180,7 +180,7 @@ usage(void) fprintf(stderr, FMT, "-T address:port", "Telnet listen address and port"); fprintf(stderr, FMT, "-V", "version"); - fprintf(stderr, FMT, "-u user", "Priviledge separation user id"); + fprintf(stderr, FMT, "-u user", "Privilege separation user id"); #undef FMT exit(1); } diff --git a/bin/varnishd/mgt/mgt_param.c b/bin/varnishd/mgt/mgt_param.c index ebb784d..f83fb84 100644 --- a/bin/varnishd/mgt/mgt_param.c +++ b/bin/varnishd/mgt/mgt_param.c @@ -725,7 +725,7 @@ mcf_param_show(struct cli *cli, const char * const *av, void *priv) } /*-------------------------------------------------------------------- - * Mark paramters as protected + * Mark parameters as protected */ void diff --git a/doc/sphinx/reference/params.rst b/doc/sphinx/reference/params.rst index 1f9a486..1abc0fe 100644 --- a/doc/sphinx/reference/params.rst +++ b/doc/sphinx/reference/params.rst @@ -230,7 +230,7 @@ http_max_hdr - Default: 64 Maximum number of HTTP headers we will deal with in client request or backend reponses. Note that the first line occupies five header fields. - This paramter does not influence storage consumption, objects allocate exact space for the headers they store. + This parameter does not influence storage consumption, objects allocate exact space for the headers they store. http_range_support - Units: bool @@ -533,7 +533,7 @@ thread_stats_rate - Flags: experimental Worker threads accumulate statistics, and dump these into the global stats counters if the lock is free when they finish a request. - This parameters defines the maximum number of requests a worker thread may handle, before it is forced to dump its accumulated stats into the global counters. + This parameter defines the maximum number of requests a worker thread may handle, before it is forced to dump its accumulated stats into the global counters. user - Default: magic -- 1.8.1.2 From tfheen at varnish-software.com Mon Mar 11 08:00:57 2013 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Mon, 11 Mar 2013 09:00:57 +0100 Subject: [PATCH] Fix for the typos In-Reply-To: References: Message-ID: <20130311080057.GC25825@err.no> ]] Benjamin Kerensa > From ff54983150df5c9c5a33a6f749cc52ffc17724ac Mon Sep 17 00:00:00 2001 > From: Benjamin Kerensa > Date: Sun, 10 Mar 2013 19:23:52 -0700 > Subject: [PATCH] Fix Typos Thanks, I've applied the bits of this that still made sense. Part of the docs are auto-generated, so that bit went away automatically. -- Tollef Fog Heen Technical lead | Varnish Software AS t: +47 21 98 92 64 We Make Websites Fly! From tfheen at varnish-software.com Mon Mar 11 09:19:16 2013 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Mon, 11 Mar 2013 10:19:16 +0100 Subject: full disclosure reports In-Reply-To: <1759739879-1362587494-cardhu_decombobulator_blackberry.rim.net-68758731-@b15.c3.bise6.blackberry> References: <1759739879-1362587494-cardhu_decombobulator_blackberry.rim.net-68758731-@b15.c3.bise6.blackberry> Message-ID: <20130311091916.GD25825@err.no> > Or god forbid some funny person sets a response code over 999. (oh is > that new disclosure?) Are you referring to #506, fixed in 2009? -- Tollef Fog Heen Technical lead | Varnish Software AS t: +47 21 98 92 64 We Make Websites Fly! From martin at varnish-software.com Mon Mar 18 16:57:22 2013 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 18 Mar 2013 17:57:22 +0100 Subject: [PATCH 1/8] Resource cleanup in cnt_error() when Transient is full. Message-ID: <1363625849-2790-1-git-send-email-martin@varnish-software.com> Fixes: #1283 --- bin/varnishd/cache/cache_hash.c | 3 +- bin/varnishd/cache/cache_req_fsm.c | 5 ++++ bin/varnishtest/tests/r01283.vtc | 57 ++++++++++++++++++++++++++++++++++++ 3 files changed, 64 insertions(+), 1 deletion(-) create mode 100644 bin/varnishtest/tests/r01283.vtc diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c index db906ec..b680605 100644 --- a/bin/varnishd/cache/cache_hash.c +++ b/bin/varnishd/cache/cache_hash.c @@ -682,7 +682,8 @@ HSH_Deref(struct dstat *ds, struct objcore *oc, struct object **oo) BAN_DestroyObj(oc); AZ(oc->ban); - } + } else + AZ(oc->refcnt); if (oc->methods != NULL) { oc_freeobj(oc); diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index b2bdd50..a7f8778 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -280,11 +280,16 @@ cnt_error(struct worker *wrk, struct req *req) if (req->obj == NULL) { req->doclose = SC_OVERLOAD; req->director = NULL; + AZ(HSH_Deref(&wrk->stats, req->objcore, NULL)); + req->objcore = NULL; http_Teardown(bo->beresp); http_Teardown(bo->bereq); + VBO_DerefBusyObj(wrk, &req->busyobj); + AZ(req->busyobj); return (REQ_FSM_DONE); } CHECK_OBJ_NOTNULL(req->obj, OBJECT_MAGIC); + AZ(req->objcore); req->obj->vxid = bo->vsl->wid; req->obj->exp.entered = req->t_req; diff --git a/bin/varnishtest/tests/r01283.vtc b/bin/varnishtest/tests/r01283.vtc new file mode 100644 index 0000000..668deda --- /dev/null +++ b/bin/varnishtest/tests/r01283.vtc @@ -0,0 +1,57 @@ +varnishtest "#1283 - Test failure to allocate object for error (transient full)" + +server s1 { + rxreq + expect req.url == "/obj1" + txresp -bodylen 1048000 + rxreq + expect req.url == "/obj2" + txresp -bodylen 1048000 +} -start + +varnish v1 -arg "-p nuke_limit=0" -storage "-smalloc,1m -sTransient=malloc,1m" -vcl+backend { + sub vcl_recv { + if (req.http.x-do-error) { + error 500; + } + } +} -start + +varnish v1 -expect n_objcore == 0 + +client c1 { + # Fill s0 + txreq -url "/obj1" + rxresp + expect resp.status == 200 +} -run + +varnish v1 -expect SMA.s0.g_bytes > 1048000 +varnish v1 -expect SMA.s0.g_space < 200 + +client c1 { + # Fill transient + txreq -url "/obj2" + rxresp + expect resp.status == 200 +} -run + +varnish v1 -expect SMA.Transient.g_bytes > 1048000 +varnish v1 -expect SMA.Transient.g_space < 200 +varnish v1 -expect SMA.Transient.c_fail == 0 + +client c1 { + # Trigger vcl_error. Don't wait for reply as Varnish will not send one + # due to Transient full + txreq -hdr "X-Do-Error: true" + delay 1 +} -run + +varnish v1 -expect SMA.Transient.c_fail == 1 + +client c1 { + # Check that Varnish is still alive + txreq -url "/obj1" + rxresp + expect resp.status == 200 +} -run -- 1.7.10.4 From martin at varnish-software.com Mon Mar 18 16:57:23 2013 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 18 Mar 2013 17:57:23 +0100 Subject: [PATCH 2/8] Properly clean up resources after STV_NewObject fails before jumping to error. In-Reply-To: <1363625849-2790-1-git-send-email-martin@varnish-software.com> References: <1363625849-2790-1-git-send-email-martin@varnish-software.com> Message-ID: <1363625849-2790-2-git-send-email-martin@varnish-software.com> Fixes: #1284 --- bin/varnishd/cache/cache_req_fsm.c | 4 ++- bin/varnishtest/tests/r01284.vtc | 53 ++++++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+), 1 deletion(-) create mode 100644 bin/varnishtest/tests/r01284.vtc diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index a7f8778..49a6732 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -487,6 +487,7 @@ cnt_fetchbody(struct worker *wrk, struct req *req) CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); CHECK_OBJ_NOTNULL(req, REQ_MAGIC); + CHECK_OBJ_NOTNULL(req->objcore, OBJCORE_MAGIC); bo = req->busyobj; CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC); @@ -571,7 +572,6 @@ cnt_fetchbody(struct worker *wrk, struct req *req) /* Create Vary instructions */ if (req->objcore->objhead != NULL) { - CHECK_OBJ_NOTNULL(req->objcore, OBJCORE_MAGIC); vary = VRY_Create(req, bo->beresp); if (vary != NULL) { varyl = VSB_len(vary); @@ -610,6 +610,8 @@ cnt_fetchbody(struct worker *wrk, struct req *req) if (req->obj == NULL) { req->err_code = 503; req->req_step = R_STP_ERROR; + AZ(HSH_Deref(&wrk->stats, req->objcore, NULL)); + req->objcore = NULL; VDI_CloseFd(&bo->vbc); VBO_DerefBusyObj(wrk, &req->busyobj); return (REQ_FSM_MORE); diff --git a/bin/varnishtest/tests/r01284.vtc b/bin/varnishtest/tests/r01284.vtc new file mode 100644 index 0000000..1289bd5 --- /dev/null +++ b/bin/varnishtest/tests/r01284.vtc @@ -0,0 +1,53 @@ +varnishtest "#1284 - Test resource cleanup after STV_NewObject fail in fetch" + +server s1 { + rxreq + expect req.url == "/obj1" + txresp -bodylen 1048000 + rxreq + expect req.url == "/obj2" + txresp -bodylen 1048000 + rxreq + expect req.url == "/obj3" + txresp -hdr "Long: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" -hdr "Long2: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" +} -start + +varnish v1 -arg "-p nuke_limit=0" -storage "-smalloc,1m -sTransient=malloc,1m" -vcl+backend { +} -start + +client c1 { + # Fill s0 + txreq -url "/obj1" + rxresp + expect resp.status == 200 +} -run + +varnish v1 -expect SMA.s0.g_bytes > 1048000 +varnish v1 -expect SMA.s0.g_space < 200 + +client c1 { + # Fill transient + txreq -url "/obj2" + rxresp + expect resp.status == 200 +} -run + +varnish v1 -expect SMA.Transient.g_bytes > 1048000 +varnish v1 -expect SMA.Transient.g_space < 200 + +client c1 { + # No space for this object (more than 256 bytes in headers). Don't wait + # for reply as Varnish will not send one due to Transient full. + txreq -url "/obj3" + delay 1 +} -run + +# Two failures, one for obj3 and one for the attempt at sending error +varnish v1 -expect SMA.Transient.c_fail == 2 + +client c1 { + # Check that Varnish is still alive + txreq -url "/obj1" + rxresp + expect resp.status == 200 +} -run -- 1.7.10.4 From martin at varnish-software.com Mon Mar 18 16:57:24 2013 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 18 Mar 2013 17:57:24 +0100 Subject: [PATCH 3/8] Fix check on object pass in fetchbody to check on req->objcore->objhead instead of req->objcore. This to fit the new world order. (This check might be OBE?) In-Reply-To: <1363625849-2790-1-git-send-email-martin@varnish-software.com> References: <1363625849-2790-1-git-send-email-martin@varnish-software.com> Message-ID: <1363625849-2790-3-git-send-email-martin@varnish-software.com> --- bin/varnishd/cache/cache_req_fsm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index 49a6732..af2fa21 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -587,7 +587,7 @@ cnt_fetchbody(struct worker *wrk, struct req *req) l += strlen("Content-Length: XxxXxxXxxXxxXxxXxx") + sizeof(void *); if (bo->exp.ttl < cache_param->shortlived || - req->objcore == NULL) + req->objcore->objhead == NULL) req->storage_hint = TRANSIENT_STORAGE; AZ(bo->stats); -- 1.7.10.4 From martin at varnish-software.com Mon Mar 18 16:57:25 2013 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 18 Mar 2013 17:57:25 +0100 Subject: [PATCH 4/8] Be able to detect parse error during processing of backend Vary headers, and do 503 on parse error. In-Reply-To: <1363625849-2790-1-git-send-email-martin@varnish-software.com> References: <1363625849-2790-1-git-send-email-martin@varnish-software.com> Message-ID: <1363625849-2790-4-git-send-email-martin@varnish-software.com> --- bin/varnishd/cache/cache.h | 2 +- bin/varnishd/cache/cache_req_fsm.c | 22 ++++++++++++++---- bin/varnishd/cache/cache_vary.c | 45 ++++++++++++++++++++++++++---------- 3 files changed, 51 insertions(+), 18 deletions(-) diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 6be2c73..a63c36e 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -1003,7 +1003,7 @@ void RES_BuildHttp(struct req *); void RES_WriteObj(struct req *); /* cache_vary.c */ -struct vsb *VRY_Create(struct req *sp, const struct http *hp); +int VRY_Create(struct req *req, const struct http *hp, struct vsb **psb); int VRY_Match(struct req *, const uint8_t *vary); void VRY_Validate(const uint8_t *vary); void VRY_Prep(struct req *); diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index af2fa21..19c9eaa 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -572,12 +572,24 @@ cnt_fetchbody(struct worker *wrk, struct req *req) /* Create Vary instructions */ if (req->objcore->objhead != NULL) { - vary = VRY_Create(req, bo->beresp); - if (vary != NULL) { - varyl = VSB_len(vary); - assert(varyl > 0); + varyl = VRY_Create(req, bo->beresp, &vary); + if (varyl > 0) { + AN(vary); + assert(varyl == VSB_len(vary)); l += varyl; - } + } else if (varyl < 0) { + /* Vary parse error */ + AZ(vary); + req->err_code = 503; + req->req_step = R_STP_ERROR; + AZ(HSH_Deref(&wrk->stats, req->objcore, NULL)); + req->objcore = NULL; + VDI_CloseFd(&bo->vbc); + VBO_DerefBusyObj(wrk, &req->busyobj); + return (REQ_FSM_MORE); + } else + /* No vary */ + AZ(vary); } /* diff --git a/bin/varnishd/cache/cache_vary.c b/bin/varnishd/cache/cache_vary.c index d4319b2..34f1891 100644 --- a/bin/varnishd/cache/cache_vary.c +++ b/bin/varnishd/cache/cache_vary.c @@ -61,20 +61,33 @@ #include "vct.h" #include "vend.h" -struct vsb * -VRY_Create(struct req *req, const struct http *hp) +/********************************************************************** + * Create a Vary matching string from a Vary header + * + * Return value: + * <0: Parse error + * 0: No Vary header on object + * >0: Length of Vary matching string in *psb + */ + +int +VRY_Create(struct req *req, const struct http *hp, struct vsb **psb) { char *v, *p, *q, *h, *e; - struct vsb *sb, *sbh; + struct vsb *sbh; unsigned l; + int error = 0; + + AN(psb); + AZ(*psb); /* No Vary: header, no worries */ if (!http_GetHdr(hp, H_Vary, &v)) - return (NULL); + return (0); /* For vary matching string */ - sb = VSB_new_auto(); - AN(sb); + *psb = VSB_new_auto(); + AN(*psb); /* For header matching strings */ sbh = VSB_new_auto(); @@ -112,11 +125,11 @@ VRY_Create(struct req *req, const struct http *hp) e = h; l = 0xffff; } - VSB_printf(sb, "%c%c", (int)(l >> 8), (int)(l & 0xff)); + VSB_printf(*psb, "%c%c", (int)(l >> 8), (int)(l & 0xff)); /* Append to vary matching string */ - VSB_bcat(sb, VSB_data(sbh), VSB_len(sbh)); + VSB_bcat(*psb, VSB_data(sbh), VSB_len(sbh)); if (e != h) - VSB_bcat(sb, h, e - h); + VSB_bcat(*psb, h, e - h); while (vct_issp(*q)) q++; @@ -125,12 +138,20 @@ VRY_Create(struct req *req, const struct http *hp) xxxassert(*q == ','); p = q; } + + if (error) { + VSB_delete(sbh); + VSB_delete(*psb); + *psb = NULL; + return (-1); + } + /* Terminate vary matching string */ - VSB_printf(sb, "%c%c%c", 0xff, 0xff, 0); + VSB_printf(*psb, "%c%c%c", 0xff, 0xff, 0); VSB_delete(sbh); - AZ(VSB_finish(sb)); - return(sb); + AZ(VSB_finish(*psb)); + return (VSB_len(*psb)); } /* -- 1.7.10.4 From martin at varnish-software.com Mon Mar 18 16:57:26 2013 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 18 Mar 2013 17:57:26 +0100 Subject: [PATCH 5/8] Return 503 when Vary-headers references header names more than 127 (out limit) characters long. In-Reply-To: <1363625849-2790-1-git-send-email-martin@varnish-software.com> References: <1363625849-2790-1-git-send-email-martin@varnish-software.com> Message-ID: <1363625849-2790-5-git-send-email-martin@varnish-software.com> Fixes: #1274 Test case by: Dag Haavi Finstad --- bin/varnishd/cache/cache_vary.c | 7 +++++++ bin/varnishtest/tests/r01274.vtc | 15 +++++++++++++++ 2 files changed, 22 insertions(+) create mode 100644 bin/varnishtest/tests/r01274.vtc diff --git a/bin/varnishd/cache/cache_vary.c b/bin/varnishd/cache/cache_vary.c index 34f1891..2a1cb6d 100644 --- a/bin/varnishd/cache/cache_vary.c +++ b/bin/varnishd/cache/cache_vary.c @@ -106,6 +106,13 @@ VRY_Create(struct req *req, const struct http *hp, struct vsb **psb) for (q = p; *q && !vct_issp(*q) && *q != ','; q++) continue; + if (q - p > INT8_MAX) { + VSLb(req->vsl, SLT_Error, + "Vary header name length exceeded"); + error = 1; + break; + } + /* Build a header-matching string out of it */ VSB_clear(sbh); VSB_printf(sbh, "%c%.*s:%c", diff --git a/bin/varnishtest/tests/r01274.vtc b/bin/varnishtest/tests/r01274.vtc new file mode 100644 index 0000000..fe427cc --- /dev/null +++ b/bin/varnishtest/tests/r01274.vtc @@ -0,0 +1,15 @@ +varnishtest "#1274 - panic when Vary field-name is too large to fit in a signed char" + +server s1 { + rxreq + # Vary header more than 127 characters long + txresp -hdr "Vary: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" +} -start + +varnish v1 -vcl+backend { } -start + +client c1 { + txreq + rxresp + expect resp.status == 503 +} -run -- 1.7.10.4 From martin at varnish-software.com Mon Mar 18 16:57:27 2013 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 18 Mar 2013 17:57:27 +0100 Subject: [PATCH 6/8] Don't panic on malformed Vary headers. In-Reply-To: <1363625849-2790-1-git-send-email-martin@varnish-software.com> References: <1363625849-2790-1-git-send-email-martin@varnish-software.com> Message-ID: <1363625849-2790-6-git-send-email-martin@varnish-software.com> Fixes: #1275 Test case by: Dag Haavi Finstad --- bin/varnishd/cache/cache_vary.c | 6 +++++- bin/varnishtest/tests/r01275.vtc | 14 ++++++++++++++ 2 files changed, 19 insertions(+), 1 deletion(-) create mode 100644 bin/varnishtest/tests/r01275.vtc diff --git a/bin/varnishd/cache/cache_vary.c b/bin/varnishd/cache/cache_vary.c index 2a1cb6d..30d77ed 100644 --- a/bin/varnishd/cache/cache_vary.c +++ b/bin/varnishd/cache/cache_vary.c @@ -142,7 +142,11 @@ VRY_Create(struct req *req, const struct http *hp, struct vsb **psb) q++; if (*q == '\0') break; - xxxassert(*q == ','); + if (*q != ',') { + VSLb(req->vsl, SLT_Error, "Malformed Vary header"); + error = 1; + break; + } p = q; } diff --git a/bin/varnishtest/tests/r01275.vtc b/bin/varnishtest/tests/r01275.vtc new file mode 100644 index 0000000..72c7184 --- /dev/null +++ b/bin/varnishtest/tests/r01275.vtc @@ -0,0 +1,14 @@ +varnishtest "#1275 - panic with malformed Vary header" + +server s1 { + rxreq + txresp -hdr "Vary: foo bar" +} -start + +varnish v1 -vcl+backend { } -start + +client c1 { + txreq + rxresp + expect resp.status == 503 +} -run -- 1.7.10.4 From martin at varnish-software.com Mon Mar 18 17:07:27 2013 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 18 Mar 2013 18:07:27 +0100 Subject: [PATCH 5/8] Return 503 when Vary-headers references header names more than 127 (out limit) characters long. In-Reply-To: References: <1363625849-2790-1-git-send-email-martin@varnish-software.com> <1363625849-2790-5-git-send-email-martin@varnish-software.com> Message-ID: True, but that isn't the only place where that is used as an unsigned char (e.g. http_findhdr). So Varnish header names are limited to signed char I believe. -Martin On 18 March 2013 18:02, Rogier 'DocWilco' Mulhuijzen wrote: > If you cast the 1st char to unsigned when assigning to len in VRY_Match, > you can get up to 255. > >> >> _______________________________________________ >> varnish-dev mailing list >> varnish-dev at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev >> >> -- *Martin Blix Grydeland* Senior Developer | Varnish Software AS Cell: +47 21 98 92 60 We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at varnish-software.com Mon Mar 18 16:57:29 2013 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 18 Mar 2013 17:57:29 +0100 Subject: [PATCH 8/8] Throw 503 on exceeding max 65k Vary header constituent limit. In-Reply-To: <1363625849-2790-1-git-send-email-martin@varnish-software.com> References: <1363625849-2790-1-git-send-email-martin@varnish-software.com> Message-ID: <1363625849-2790-8-git-send-email-martin@varnish-software.com> --- bin/varnishd/cache/cache_vary.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/bin/varnishd/cache/cache_vary.c b/bin/varnishd/cache/cache_vary.c index f66044a..3aab8a3 100644 --- a/bin/varnishd/cache/cache_vary.c +++ b/bin/varnishd/cache/cache_vary.c @@ -122,7 +122,12 @@ VRY_Create(struct req *req, const struct http *hp, struct vsb **psb) e--; /* Encode two byte length and contents */ l = e - h; - assert(!(l & ~0xffff)); + if (l > 0xffff - 1) { + VSLb(req->vsl, SLT_Error, + "Vary header maximum length exceeded"); + error = 1; + break; + } } else { e = h; l = 0xffff; -- 1.7.10.4 From martin at varnish-software.com Mon Mar 18 16:57:28 2013 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 18 Mar 2013 17:57:28 +0100 Subject: [PATCH 7/8] Remove dodgy workaround introduced for #763. In-Reply-To: <1363625849-2790-1-git-send-email-martin@varnish-software.com> References: <1363625849-2790-1-git-send-email-martin@varnish-software.com> Message-ID: <1363625849-2790-7-git-send-email-martin@varnish-software.com> With fix for #1275 we can now do a proper 503 when malformed Vary-headers are received from backend. Error out instead of accepting bogus ':'. --- bin/varnishd/cache/cache_vary.c | 5 ----- bin/varnishtest/tests/r00763.vtc | 2 +- 2 files changed, 1 insertion(+), 6 deletions(-) diff --git a/bin/varnishd/cache/cache_vary.c b/bin/varnishd/cache/cache_vary.c index 30d77ed..f66044a 100644 --- a/bin/varnishd/cache/cache_vary.c +++ b/bin/varnishd/cache/cache_vary.c @@ -93,11 +93,6 @@ VRY_Create(struct req *req, const struct http *hp, struct vsb **psb) sbh = VSB_new_auto(); AN(sbh); - if (*v == ':') { - VSLb(req->vsl, SLT_Error, - "Vary header had extra ':', fix backend"); - v++; - } for (p = v; *p; p++) { /* Find next header-name */ diff --git a/bin/varnishtest/tests/r00763.vtc b/bin/varnishtest/tests/r00763.vtc index 24a4d6f..72fced4 100644 --- a/bin/varnishtest/tests/r00763.vtc +++ b/bin/varnishtest/tests/r00763.vtc @@ -10,5 +10,5 @@ varnish v1 -vcl+backend {} -start client c1 { txreq rxresp - expect resp.http.vary == ": foo" + expect resp.status == 503 } -run -- 1.7.10.4 From phk at phk.freebsd.dk Tue Mar 19 12:50:46 2013 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 19 Mar 2013 12:50:46 +0000 Subject: [PATCH 5/8] Return 503 when Vary-headers references header names more than 127 (out limit) characters long. In-Reply-To: References: <1363625849-2790-1-git-send-email-martin@varnish-software.com> <1363625849-2790-5-git-send-email-martin@varnish-software.com> Message-ID: <40262.1363697446@critter.freebsd.dk> In message , Martin Blix Grydeland wr ites: >True, but that isn't the only place where that is used as an unsigned char >(e.g. http_findhdr). So Varnish header names are limited to signed char I >believe. Are there real world examples of non-insane uses of headers longer than 127 characters ? Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Wed Mar 20 12:05:43 2013 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 20 Mar 2013 12:05:43 +0000 Subject: V4 VCL roadmappery... Message-ID: <8648.1363781143@critter.freebsd.dk> "I have a plan" (Insert obligatory Olsen Banden reply-quote here: _____) The VCLv4 will have two distinct parts: client and backend which will mostly run in separate threads. After looking at various approaches, it seems to me that it will be the easiest rebuilding, if we start from the backend side and end up with the client side of VCLv4, so I've made the following plan: vcl_fetch{} becomes vcl_reponse{} This is a straight renaming. vcl_response{} is called when we have the header back from the backend and its main job is to validate the response, and pick optional procesing (gzip, gunzip, esi, non-streaming) A new vcl_fetch{} This will (a little later) replace vcl_miss{} and vcl_pass{} Pipe (probably) doesn't go here, it's too magical. vcl_fetch{} is called to pick backend and polish the bereq for backend, including disabling conditional fetches. (req.backend becomes bereq.backend) Make it possible to run vcl_fetch{} and vcl_response{} without req.* This is to enable them to run (to completion) in a parallel thread, even if the instigating client have moved on, but they must still be able to run on req::thread for pass. The operation starts in a few minutes... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From magnus at hagander.net Wed Mar 20 12:34:27 2013 From: magnus at hagander.net (Magnus Hagander) Date: Wed, 20 Mar 2013 12:34:27 +0000 Subject: V4 VCL roadmappery... In-Reply-To: <8648.1363781143@critter.freebsd.dk> References: <8648.1363781143@critter.freebsd.dk> Message-ID: On Wed, Mar 20, 2013 at 12:05 PM, Poul-Henning Kamp wrote: > > "I have a plan" > > (Insert obligatory Olsen Banden reply-quote here: _____) > > The VCLv4 will have two distinct parts: client and backend which > will mostly run in separate threads. > > After looking at various approaches, it seems to me that it will > be the easiest rebuilding, if we start from the backend side and > end up with the client side of VCLv4, so I've made the following > plan: > > vcl_fetch{} becomes vcl_reponse{} > This is a straight renaming. > > vcl_response{} is called when we have the header back from > the backend and its main job is to validate the response, > and pick optional procesing (gzip, gunzip, esi, non-streaming) > > A new vcl_fetch{} > This will (a little later) replace vcl_miss{} and vcl_pass{} > > Pipe (probably) doesn't go here, it's too magical. > > vcl_fetch{} is called to pick backend and polish the bereq > for backend, including disabling conditional fetches. > > (req.backend becomes bereq.backend) Maybe I'm coming in late to this game, but is it really necessary to reuse the old name? I'm seeing a lot of confusion amongst people for the simple reuse of "purge" in version 3 to mean something completely different to what it used to mean. This sounds like it could be even worse... Probably a lot less confusing if a completely new name can be invented, if it's a new function that does something else... -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ From lkarsten at varnish-software.com Wed Mar 20 12:42:28 2013 From: lkarsten at varnish-software.com (Lasse Karstensen) Date: Wed, 20 Mar 2013 13:42:28 +0100 Subject: V4 VCL roadmappery... In-Reply-To: References: <8648.1363781143@critter.freebsd.dk> Message-ID: <20130320124228.GB16203@immer.varnish-software.com> On Wed, Mar 20, 2013 at 12:34:27PM +0000, Magnus Hagander wrote: > On Wed, Mar 20, 2013 at 12:05 PM, Poul-Henning Kamp wrote: > > A new vcl_fetch{} > > This will (a little later) replace vcl_miss{} and vcl_pass{} > > Pipe (probably) doesn't go here, it's too magical. > > vcl_fetch{} is called to pick backend and polish the bereq > > for backend, including disabling conditional fetches. > > (req.backend becomes bereq.backend) > Maybe I'm coming in late to this game, but is it really necessary to > reuse the old name? I'm seeing a lot of confusion amongst people for > the simple reuse of "purge" in version 3 to mean something completely > different to what it used to mean. This sounds like it could be even > worse... Probably a lot less confusing if a completely new name can be > invented, if it's a new function that does something else... +1. -- With regards, Lasse Karstensen Varnish Software AS From phk at phk.freebsd.dk Wed Mar 20 12:56:30 2013 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 20 Mar 2013 12:56:30 +0000 Subject: V4 VCL roadmappery... In-Reply-To: References: <8648.1363781143@critter.freebsd.dk> Message-ID: <61572.1363784190@critter.freebsd.dk> In message , Magnus Hagander writes: >> vcl_fetch{} becomes vcl_reponse{} >> >> A new vcl_fetch{} >> >Maybe I'm coming in late to this game, but is it really necessary to >reuse the old name? I'm seeing a lot of confusion amongst people for >the simple reuse of "purge" in version 3 to mean something completely >different to what it used to mean. This sounds like it could be even >worse... Probably a lot less confusing if a completely new name can be >invented, if it's a new function that does something else... Suggest a better name ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi.boukelmoune at zenika.com Wed Mar 20 13:07:53 2013 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Wed, 20 Mar 2013 14:07:53 +0100 Subject: V4 VCL roadmappery... In-Reply-To: <61572.1363784190@critter.freebsd.dk> References: <8648.1363781143@critter.freebsd.dk> <61572.1363784190@critter.freebsd.dk> Message-ID: vcl_request vcl_bereq vcl_forward vcl_transmit I agree this is not easy to find a meaningful name, those might be confusing :( Best Regards, Dridi On Wed, Mar 20, 2013 at 1:56 PM, Poul-Henning Kamp wrote: > In message < > CABUevEw2P64dcayQ2_rH_ZR-_sX7YcG95qb7q2wQJ4o_hNHyCg at mail.gmail.com>, > Magnus Hagander writes: > > >> vcl_fetch{} becomes vcl_reponse{} > >> > >> A new vcl_fetch{} > >> > > >Maybe I'm coming in late to this game, but is it really necessary to > >reuse the old name? I'm seeing a lot of confusion amongst people for > >the simple reuse of "purge" in version 3 to mean something completely > >different to what it used to mean. This sounds like it could be even > >worse... Probably a lot less confusing if a completely new name can be > >invented, if it's a new function that does something else... > > Suggest a better name ? > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at uplex.de Wed Mar 20 13:13:24 2013 From: geoff at uplex.de (Geoff Simmons) Date: Wed, 20 Mar 2013 14:13:24 +0100 Subject: V4 VCL roadmappery... In-Reply-To: References: <8648.1363781143@critter.freebsd.dk> <61572.1363784190@critter.freebsd.dk> Message-ID: <5149B5F4.2050300@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 03/20/2013 02:07 PM, Dridi Boukelmoune wrote: > vcl_request > vcl_bereq > vcl_forward > vcl_transmit Maybe vcl_prefetch(), since this will be just before a fetch. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJRSbX0AAoJEOUwvh9pJNURFUwP/11y/fE9ZCmj9Xknqm3P+bHJ tFjJnhqnX1TeppVqXG8XhVaQH6wwy0qJLFRiFUQInpUDubzhd/Gk5vLEMXuSNKwb ybHro7UB+75tmZK3uJehgptmrUKZRyuk0YDDy5QrIwOTCwGEWsopyqg5BQBsWzfP eO/r9M5QxyGd/jytJEogLycFg8CivtATgE4ZSyYhc959/ilHEjWNh+f9i2cZO6K5 WuAaPHCa6v3choXgRw+jvrB4QOas/EagFLC6Gc7hb7eKILgKgqnkutJHfFITCquq GT/6Gen3czoAHWL59bqZn2K61l4Dd9tevYkksVBprI5lPL7P+K8h1YibMCnXlmkN 4ycEIvYgoMsI6KyDvKvPO0nDjIZf3UrkWips4hpCZB/WodVOVxcPl2Y839conKxh ODSAam+ibL7DWOkY1f7C2C2TUnQiqAAsJG3D0kHQUb7wPwKx3v22X1pQL+1npuIx 6fUKFoSbJh0SKEJQcu2ebsAXBnIzp4KBuWmgPs3yNwww0R96Cs0xyqB8rafPuaWU AW0sOt/yBtL/l449uvFTGZK4QPZ1aMXQ5qnjjjQnrDIgvaOF9GD/AyqBTuFHl9f8 9nZ/I7ffoGtU6n4c/6TzTm0n1P2WaJ3cwk8GZMKFfxL1c+lhtk+QRWTaCmC73m3U N9cXuToESiCTEFTLWRqX =HPpT -----END PGP SIGNATURE----- From magnus at hagander.net Wed Mar 20 13:15:22 2013 From: magnus at hagander.net (Magnus Hagander) Date: Wed, 20 Mar 2013 13:15:22 +0000 Subject: V4 VCL roadmappery... In-Reply-To: <5149B5F4.2050300@uplex.de> References: <8648.1363781143@critter.freebsd.dk> <61572.1363784190@critter.freebsd.dk> <5149B5F4.2050300@uplex.de> Message-ID: On Wed, Mar 20, 2013 at 1:13 PM, Geoff Simmons wrote: > On 03/20/2013 02:07 PM, Dridi Boukelmoune wrote: >> vcl_request >> vcl_bereq >> vcl_forward >> vcl_transmit > > Maybe vcl_prefetch(), since this will be just before a fetch. +1, I like that one. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ From dridi.boukelmoune at zenika.com Wed Mar 20 13:17:55 2013 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Wed, 20 Mar 2013 14:17:55 +0100 Subject: V4 VCL roadmappery... In-Reply-To: References: <8648.1363781143@critter.freebsd.dk> <61572.1363784190@critter.freebsd.dk> <5149B5F4.2050300@uplex.de> Message-ID: +1 ! On Wed, Mar 20, 2013 at 2:15 PM, Magnus Hagander wrote: > On Wed, Mar 20, 2013 at 1:13 PM, Geoff Simmons wrote: > > On 03/20/2013 02:07 PM, Dridi Boukelmoune wrote: > >> vcl_request > >> vcl_bereq > >> vcl_forward > >> vcl_transmit > > > > Maybe vcl_prefetch(), since this will be just before a fetch. > > +1, I like that one. > > -- > Magnus Hagander > Me: http://www.hagander.net/ > Work: http://www.redpill-linpro.com/ > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ksorensen at nordija.com Wed Mar 20 13:20:52 2013 From: ksorensen at nordija.com (Kristian =?ISO-8859-1?Q?Gr=F8nfeldt_S=F8rensen?=) Date: Wed, 20 Mar 2013 14:20:52 +0100 Subject: V4 VCL roadmappery... In-Reply-To: References: <8648.1363781143@critter.freebsd.dk> <61572.1363784190@critter.freebsd.dk> <5149B5F4.2050300@uplex.de> Message-ID: <1363785652.11607.68.camel@kriller.nordija.dk> On Wed, 2013-03-20 at 13:15 +0000, Magnus Hagander wrote: > On Wed, Mar 20, 2013 at 1:13 PM, Geoff Simmons wrote: > > On 03/20/2013 02:07 PM, Dridi Boukelmoune wrote: > >> vcl_request > >> vcl_bereq > >> vcl_forward > >> vcl_transmit > > > > Maybe vcl_prefetch(), since this will be just before a fetch. > > +1, I like that one. +1 /Kristian From phk at phk.freebsd.dk Wed Mar 20 13:25:00 2013 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 20 Mar 2013 13:25:00 +0000 Subject: V4 VCL roadmappery... In-Reply-To: References: <8648.1363781143@critter.freebsd.dk> <61572.1363784190@critter.freebsd.dk> Message-ID: <93158.1363785900@critter.freebsd.dk> In message , Dridi Boukelmoune writes : >--20cf3071c6b05bedd204d85aeb63 >Content-Type: text/plain; charset=UTF-8 > >vcl_request >vcl_bereq >vcl_forward >vcl_transmit I had considered vcl_bereq{} + vcl_beresp{}, but decided against them because they simple don't work for me. However: I'm open to suggestions and consensus -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From sky at crucially.net Wed Mar 20 13:26:30 2013 From: sky at crucially.net (sky at crucially.net) Date: Wed, 20 Mar 2013 13:26:30 +0000 Subject: V4 VCL roadmappery... In-Reply-To: <93158.1363785900@critter.freebsd.dk> References: <8648.1363781143@critter.freebsd.dk> <61572.1363784190@critter.freebsd.dk> <93158.1363785900@critter.freebsd.dk> Message-ID: <243279930-1363785991-cardhu_decombobulator_blackberry.rim.net-616716734-@b15.c3.bise6.blackberry> vcl_prepare_request ? It is not like we type it very often. prefetch can mean two thins. Artur Sent via BlackBerry by AT&T -----Original Message----- From: "Poul-Henning Kamp" Sender: varnish-dev-bounces at varnish-cache.org Date: Wed, 20 Mar 2013 13:25:00 To: Dridi Boukelmoune Cc: Subject: Re: V4 VCL roadmappery... In message , Dridi Boukelmoune writes : >--20cf3071c6b05bedd204d85aeb63 >Content-Type: text/plain; charset=UTF-8 > >vcl_request >vcl_bereq >vcl_forward >vcl_transmit I had considered vcl_bereq{} + vcl_beresp{}, but decided against them because they simple don't work for me. However: I'm open to suggestions and consensus -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. _______________________________________________ varnish-dev mailing list varnish-dev at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev From deltawasp at gmail.com Thu Mar 21 10:26:01 2013 From: deltawasp at gmail.com (Delta Wasp) Date: Thu, 21 Mar 2013 10:26:01 +0000 Subject: Request duration Message-ID: <66F1427B-0777-4E02-A967-B8EC99171CE1@gmail.com> Hi I'm trying to find out how to log request duration using varnishncsa. The nearest thing I've seen is Klaus's posting earlier this year: https://www.varnish-cache.org/lists/pipermail/varnish-dev/2013-January/007461.html But this code doesn't appear to have made into any release. Can anybody tell me how to go about getting this code tested and live? I'm new to this, so apologies if I'm not going about the right way. Regards Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: From jos at dwim.org Thu Mar 21 10:49:53 2013 From: jos at dwim.org (Jos Boumans) Date: Thu, 21 Mar 2013 11:49:53 +0100 Subject: Request duration In-Reply-To: <66F1427B-0777-4E02-A967-B8EC99171CE1@gmail.com> References: <66F1427B-0777-4E02-A967-B8EC99171CE1@gmail.com> Message-ID: <59612C4F-03E0-43B6-B6E3-C0525A13256A@dwim.org> Hi Jon, we tried to do exactly the same, and I ended up writing this vmod to accomplish the task: https://www.varnish-cache.org/vmod/varnish-timers-get-timingduration-directly-varnish This let's you capture the timers (and store them in a header if you like for logging) without having to patch varnishncsa itself. We use that together with another vmod I wrote to send stats directly to statsd. Here's a blogpost that explains the process, together with a VCL you can include to make it work for you: http://jiboumans.wordpress.com/2013/02/27/realtime-stats-from-varnish/ Cheers, -Jos On 21 Mar 2013, at 11:26, Delta Wasp wrote: > Hi > > I'm trying to find out how to log request duration using varnishncsa. The nearest thing I've seen is Klaus's posting earlier this year: > > https://www.varnish-cache.org/lists/pipermail/varnish-dev/2013-January/007461.html > > But this code doesn't appear to have made into any release. Can anybody tell me how to go about getting this code tested and live? > > I'm new to this, so apologies if I'm not going about the right way. > > Regards > > Jon > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev From jos at dwim.org Wed Mar 27 00:00:32 2013 From: jos at dwim.org (Jos Boumans) Date: Wed, 27 Mar 2013 00:00:32 +0000 Subject: Is request time available in VCL? In-Reply-To: References: Message-ID: <191C3888-1EE4-4A02-873C-40797636A33C@dwim.org> Hi Jean-Baptiste, it's not available in VCL, but you can get at it with this Vmod I wrote for exactly that purpose: https://www.varnish-cache.org/vmod/varnish-timers-get-timingduration-directly-varnish Cheers, -Jos On 25 Jan 2011, at 11:40, Jean-Baptiste Quenot wrote: > Hi, > > I'm using a varnishncsa patch that logs how long the server took to reply: > > case SLT_ReqEnd: > if (sscanf(ptr, "%*u %*u.%*u %ld.%*u %lf %lf %lf", &l, > &idle_t, &proc_t, &xmit_t) != 4) { > lp->bogus = 1; > } else { > t = l; > lp->df_D= (proc_t + xmit_t) * 1000000; > localtime_r(&t, &lp->df_t); > } > /* got it all */ > return (0); > > I'd like to add a X-Varnish-Time header in the response to indicate > the same number, for service health checks. Is this time information > available in VCL? > > Thanks in advance, > -- > Jean-Baptiste Quenot > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > http://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev