From phk at phk.freebsd.dk Mon Jan 2 12:11:22 2012 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 02 Jan 2012 12:11:22 +0000 Subject: [PATCH] Enable kqueue() support on OpenBSD. In-Reply-To: Your message of "Fri, 23 Dec 2011 09:49:12 EST." <20111223144912.GA16802@rox.home.comstyle.com> Message-ID: <35102.1325506282@critter.freebsd.dk> Fixed. In message <20111223144912.GA16802 at rox.home.comstyle.com>, Brad writes: >The following diff fixes the autoconf script to properly detect >the presence of kqueue() on OpenBSD. > > >diff --git a/configure.ac b/configure.ac >index 8404de4..425d925 100644 >--- a/configure.ac >+++ b/configure.ac >@@ -335,7 +335,7 @@ AC_ARG_ENABLE(kqueue, > > if test "$enable_kqueue" = yes; then > case $target in >- *-*-freebsd* | *-*-darwin9* | *-*-darwin11* | *-*-netbsd* ) >+ *-*-freebsd* | *-*-darwin9* | *-*-darwin11* | *-*-netbsd* | *-*-openbsd*) > AC_CHECK_FUNCS([kqueue]) > ;; > *-*-bsd*) > >-- >This message has been scanned for viruses and >dangerous content by MailScanner, and is >believed to be clean. > > >_______________________________________________ >varnish-dev mailing list >varnish-dev at varnish-cache.org >https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From brad at comstyle.com Mon Jan 2 17:03:59 2012 From: brad at comstyle.com (Brad) Date: Mon, 02 Jan 2012 12:03:59 -0500 Subject: [PATCH] Enable kqueue() support on OpenBSD. In-Reply-To: <20111223144912.GA16802@rox.home.comstyle.com> References: <20111223144912.GA16802@rox.home.comstyle.com> Message-ID: <4F01E37F.80405@comstyle.com> On 23/12/11 9:49 AM, Brad wrote: > The following diff fixes the autoconf script to properly detect > the presence of kqueue() on OpenBSD. Sorry phk I had deleted your response before noticing what you had actually commited. Just wanted to point out this comment above the kqueue() section could be removed now.. # Check which mechanism to use for the acceptor. We look for kqueue # only on platforms on which we know that it works, because there are # platforms where a simple AC_CHECK_FUNCS([kqueue]) would succeed but # the build would fail. We also allow the user to disable mechanisms # he doesn't want to use. > diff --git a/configure.ac b/configure.ac > index 8404de4..425d925 100644 > --- a/configure.ac > +++ b/configure.ac > @@ -335,7 +335,7 @@ AC_ARG_ENABLE(kqueue, > > if test "$enable_kqueue" = yes; then > case $target in > - *-*-freebsd* | *-*-darwin9* | *-*-darwin11* | *-*-netbsd* ) > + *-*-freebsd* | *-*-darwin9* | *-*-darwin11* | *-*-netbsd* | *-*-openbsd*) > AC_CHECK_FUNCS([kqueue]) > ;; > *-*-bsd*) -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From brad at comstyle.com Mon Jan 2 19:03:46 2012 From: brad at comstyle.com (Brad) Date: Mon, 2 Jan 2012 14:03:46 -0500 Subject: [PATCH] Fix libedit detection on *BSD OS's Message-ID: <20120102190345.GA24654@rox.home.comstyle.com> The following patch allows the autoconf script to detect the presence of libedit when there isn't a pkg-config file present and thus allowing Varnish to detect libedit on OpenBSD/FreeBSD/NetBSD/DragonFly. diff --git a/bin/varnishadm/varnishadm.c b/bin/varnishadm/varnishadm.c index bb5cc8e..90532a8 100644 --- a/bin/varnishadm/varnishadm.c +++ b/bin/varnishadm/varnishadm.c @@ -33,8 +33,12 @@ #ifdef HAVE_LIBEDIT #include +#ifdef HAVE_READLINE_READLINE_H +#include +#else #include #endif +#endif #include #include diff --git a/configure.ac b/configure.ac index 01eac28..7d33c32 100644 --- a/configure.ac +++ b/configure.ac @@ -142,9 +142,16 @@ fi AC_SUBST(PCRE_CFLAGS) AC_SUBST(PCRE_LIBS) -PKG_CHECK_MODULES([LIBEDIT], [libedit], +PKG_CHECK_MODULES([LIBEDIT], [libedit], [AC_DEFINE([HAVE_LIBEDIT], [1], [Define we have libedit])], - [AC_MSG_WARN([libedit not found, disabling libedit support])]) + [AC_CHECK_HEADERS([readline/readline.h]) + AC_CHECK_LIB(edit, el_init, + [ AC_DEFINE([HAVE_LIBEDIT], [1], [Define we have libedit]) + LIBEDIT_CFLAGS="" + LIBEDIT_LIBS="-ledit ${CURSES_LIBS}" + ], + [AC_MSG_WARN([libedit not found, disabling libedit support])], + [${CURSES_LIBS}])]) # Checks for header files. AC_HEADER_STDC -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From apj at mutt.dk Tue Jan 3 14:43:10 2012 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Tue, 3 Jan 2012 15:43:10 +0100 Subject: Upgrading from 2.1.5 to 3 In-Reply-To: References: Message-ID: <20120103144309.GS3214@nerd.dk> On Wed, Dec 28, 2011 at 09:40:59PM +0530, paramvir kaler wrote: > > I am facing a lot of problems with .vcl file after upgrading. Can anyone > make the changes in .vcl file for me and make it compatible with varnish 3 ? https://www.varnish-cache.org/docs/trunk/installation/upgrade.html -- Andreas From ruben at varnish-software.com Thu Jan 5 17:45:04 2012 From: ruben at varnish-software.com (=?UTF-8?Q?Rub=C3=A9n_Romero?=) Date: Thu, 5 Jan 2012 18:45:04 +0100 Subject: Welcome to VUG5 in Paris, France - March 22-23, 2012 Message-ID: Hello to you all, The year has started and we need to plan ahead. So it is my pleasure to announce that our Fifth Varnish User Group meeting, VUG5, will happen in Paris, France. The dates are March 22nd (User Day) and 23rd (Dev Day), 2012. For all the gory details and registration visit the following page on the Varnish Cache community website: As we have a rather limited number of seats we need to know the number of assistants in advance, so register even if you are not 100% sure about coming to Paris. We will send you a confirmation reminder later. We need YOU! -> Do not forget to add your favorite discussion item (or your own Varnish use case) to the agenda. This goes both for the User and Dev days. We want to know how you use the software, which problems you solve with it and what challenges you have so we can together make an even better Varnish Cache! If you have *any* questions related to this meeting, do not hesitate in contacting me directly and I will get back to you. Last but not least: Hope to see you all in Paris! Best wishes, -- Rub?n Romero, Self-appointed <3 Non-Official VUG Coordinator & Cheerleader Phone: +47 21 98 92 62 / Mobile: +47 959 64 088 / Skype: ruben_varnish / GTalk: *Varnish makes websites fly!* Whitepapers | Video | Twitter | LinkedIn -------------- next part -------------- An HTML attachment was scrubbed... URL: From github at bsdchicks.com Fri Jan 6 03:22:02 2012 From: github at bsdchicks.com (Rogier 'DocWilco' Mulhuijzen) Date: Fri, 6 Jan 2012 04:22:02 +0100 Subject: [PATCH] Drop old grace objects when we have a new object that matches the Vary Message-ID: <1325820122-7794-1-git-send-email-github@bsdchicks.com> Also add EXP_Remove, to allow removing objects with touching them. This patch makes it possible to have a very short TTL, but a long grace. Normally, if you were to set a 1s TTL and a 86400s grace, you would end up with 86400 objects on a single objecthead, not even taking into account Vary. With this patch, any object found in grace is kept with the request, and upon succesful retrieval of a new object from the backend, the grace object is removed. EXP_Remove was added so that the TTL and grace on the object aren't touched. This, in combination with some other work we've done, prevents pages from getting dirtied and thus having to be written out to disk in a environment where the cache cannot be purely kept in memory. (Side note, the testcase needs a proper filename, but waiting on review before picking it.) --- bin/varnishd/cache/cache.h | 2 + bin/varnishd/cache/cache_center.c | 4 +- bin/varnishd/cache/cache_expire.c | 47 +++++++++++++ bin/varnishd/cache/cache_hash.c | 48 ++++++++++---- bin/varnishd/hash/hash_slinger.h | 2 +- bin/varnishtest/tests/grace-drop.vtc | 120 ++++++++++++++++++++++++++++++++++ include/tbl/vsc_f_main.h | 5 ++ 7 files changed, 213 insertions(+), 15 deletions(-) create mode 100644 bin/varnishtest/tests/grace-drop.vtc diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 6b0d0a9..cb8fe4e 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -584,6 +584,7 @@ struct req { const char *doclose; struct exp exp; + struct objcore *grace_oc; unsigned cur_method; unsigned handling; unsigned char sendbody; @@ -753,6 +754,7 @@ void EXP_Init(void); void EXP_Rearm(const struct object *o); int EXP_Touch(struct objcore *oc); int EXP_NukeOne(struct worker *w, struct lru *lru); +void EXP_Remove(struct worker *w, struct objcore *oc); /* cache_fetch.c */ struct storage *FetchStorage(struct worker *w, ssize_t sz); diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c index 4967c82..df9b9cd 100644 --- a/bin/varnishd/cache/cache_center.c +++ b/bin/varnishd/cache/cache_center.c @@ -928,7 +928,7 @@ cnt_fetchbody(struct sess *sp, struct worker *wrk, struct req *req) EXP_Insert(req->obj); AN(req->obj->objcore); AN(req->obj->objcore->ban); - HSH_Unbusy(wrk); + HSH_Unbusy(wrk, 1); } VBO_DerefBusyObj(wrk, &wrk->busyobj); wrk->acct_tmp.fetch++; @@ -987,7 +987,7 @@ cnt_streambody(struct sess *sp, struct worker *wrk, struct req *req) EXP_Insert(req->obj); AN(req->obj->objcore); AN(req->obj->objcore->ban); - HSH_Unbusy(wrk); + HSH_Unbusy(wrk, 1); } else { req->doclose = "Stream error"; } diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c index e93a1aa..1f1749b 100644 --- a/bin/varnishd/cache/cache_expire.c +++ b/bin/varnishd/cache/cache_expire.c @@ -451,6 +451,53 @@ EXP_NukeOne(struct worker *wrk, struct lru *lru) } /*-------------------------------------------------------------------- + * Remove from expiry binheap. Caller should have a ref if it wants + * to do anything with it afterwards. + */ + +void +EXP_Remove(struct worker *w, struct objcore *oc) +{ + struct object *o; + struct lru *l; + + CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); + o = oc_getobj(w, oc); + CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC); + l = oc_getlru(oc); + CHECK_OBJ_NOTNULL(l, LRU_MAGIC); + + Lck_Lock(&l->mtx); + Lck_Lock(&exp_mtx); + + if (oc->timer_idx == BINHEAP_NOIDX) { /* exp_timer has it */ + Lck_Unlock(&exp_mtx); + Lck_Unlock(&l->mtx); + return; + } + + /* remove from binheap */ + binheap_delete(exp_heap, oc->timer_idx); + assert(oc->timer_idx == BINHEAP_NOIDX); + + /* And from LRU */ + CHECK_OBJ_NOTNULL(l, LRU_MAGIC); +#ifdef WIKIA_STABLE + if (l->memmark == oc) + l->memmark = VLIST_NEXT(oc, lru_list); +#endif + VTAILQ_REMOVE(&l->lru_head, oc, lru_list); + + Lck_Unlock(&exp_mtx); + Lck_Unlock(&l->mtx); + + w->stats.c_removed++; + + WSL(w, SLT_ExpKill, 0, "%u Removed", o->xid); + HSH_Deref(w, NULL, &o); +} + +/*-------------------------------------------------------------------- * BinHeap helper functions for objcore. */ diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c index d967291..a3eb77c 100644 --- a/bin/varnishd/cache/cache_hash.c +++ b/bin/varnishd/cache/cache_hash.c @@ -296,7 +296,7 @@ HSH_Lookup(struct sess *sp, struct objhead **poh) struct worker *wrk; struct objhead *oh; struct objcore *oc; - struct objcore *busy_oc, *grace_oc; + struct objcore *busy_oc; struct object *o; double grace_ttl; @@ -332,7 +332,7 @@ HSH_Lookup(struct sess *sp, struct objhead **poh) Lck_Lock(&oh->mtx); assert(oh->refcnt > 0); busy_oc = NULL; - grace_oc = NULL; + sp->req->grace_oc = NULL; grace_ttl = NAN; VTAILQ_FOREACH(oc, &oh->objcs, list) { /* Must be at least our own ref + the objcore we examine */ @@ -372,9 +372,9 @@ HSH_Lookup(struct sess *sp, struct objhead **poh) * and if there are several, use the least expired one. */ if (EXP_Grace(sp, o) >= sp->t_req) { - if (grace_oc == NULL || + if (sp->req->grace_oc == NULL || grace_ttl < o->exp.entered + o->exp.ttl) { - grace_oc = oc; + sp->req->grace_oc = oc; grace_ttl = o->exp.entered + o->exp.ttl; } } @@ -391,15 +391,16 @@ HSH_Lookup(struct sess *sp, struct objhead **poh) */ AZ(sp->req->objcore); - sp->req->objcore = grace_oc; /* XXX: Hack-ish */ + sp->req->objcore = sp->req->grace_oc; /* XXX: Hack-ish */ if (oc == NULL /* We found no live object */ - && grace_oc != NULL /* There is a grace candidate */ + && sp->req->grace_oc != NULL /* There is a grace candidate */ && (busy_oc != NULL /* Somebody else is already busy */ || !VDI_Healthy(sp->req->director, sp))) { /* Or it is impossible to fetch */ - o = oc_getobj(sp->wrk, grace_oc); + o = oc_getobj(sp->wrk, sp->req->grace_oc); CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC); - oc = grace_oc; + oc = sp->req->grace_oc; + sp->req->grace_oc = NULL; } sp->req->objcore = NULL; @@ -415,6 +416,7 @@ HSH_Lookup(struct sess *sp, struct objhead **poh) assert(oh->refcnt > 1); Lck_Unlock(&oh->mtx); assert(hash->deref(oh)); + sp->req->grace_oc = NULL; *poh = oh; return (oc); } @@ -441,6 +443,7 @@ HSH_Lookup(struct sess *sp, struct objhead **poh) */ sp->req->hash_objhead = oh; sp->wrk = NULL; + sp->req->grace_oc = NULL; Lck_Unlock(&oh->mtx); return (NULL); } @@ -450,6 +453,15 @@ HSH_Lookup(struct sess *sp, struct objhead **poh) wrk->nobjcore = NULL; AN(oc->flags & OC_F_BUSY); oc->refcnt = 1; + /* + * make sure that we don't expire the graced object while we're fetching + * so that we don't panic/segfault after fetch + */ + if (sp->req->grace_oc) { + CHECK_OBJ_NOTNULL(sp->req->grace_oc, OBJCORE_MAGIC); + sp->req->grace_oc->refcnt++; + assert(sp->req->grace_oc->refcnt > 0); + } AZ(wrk->busyobj); wrk->busyobj = VBO_GetBusyObj(wrk); @@ -586,16 +598,16 @@ HSH_Drop(struct worker *wrk) AssertObjCorePassOrBusy(o->objcore); o->exp.ttl = -1.; if (o->objcore != NULL) /* Pass has no objcore */ - HSH_Unbusy(wrk); + HSH_Unbusy(wrk, 0); (void)HSH_Deref(wrk, NULL, &wrk->sp->req->obj); } void -HSH_Unbusy(struct worker *wrk) +HSH_Unbusy(struct worker *wrk, int dropgrace) { - struct object *o; + struct object *o, *go; struct objhead *oh; - struct objcore *oc; + struct objcore *oc, *grace_oc; CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); o = wrk->sp->req->obj; @@ -627,6 +639,18 @@ HSH_Unbusy(struct worker *wrk) hsh_rush(oh); AN(oc->ban); Lck_Unlock(&oh->mtx); + if (wrk->sp->req->grace_oc) { + grace_oc = wrk->sp->req->grace_oc; + CHECK_OBJ_NOTNULL(grace_oc, OBJCORE_MAGIC); + go = oc_getobj(wrk, grace_oc); + CHECK_OBJ_NOTNULL(go, OBJECT_MAGIC); + if (dropgrace) { + wrk->stats.c_grace_obj_dropped++; + EXP_Remove(wrk, grace_oc); + } + assert(grace_oc->refcnt > 0); + HSH_Deref(wrk, NULL, &go); + } assert(oc_getobj(wrk, oc) == o); } diff --git a/bin/varnishd/hash/hash_slinger.h b/bin/varnishd/hash/hash_slinger.h index b45e604..c0cd54c 100644 --- a/bin/varnishd/hash/hash_slinger.h +++ b/bin/varnishd/hash/hash_slinger.h @@ -54,7 +54,7 @@ struct hash_slinger { void HSH_Prealloc(const struct sess *sp); void HSH_Cleanup(struct worker *w); struct objcore *HSH_Lookup(struct sess *sp, struct objhead **poh); -void HSH_Unbusy(struct worker *wrk); +void HSH_Unbusy(struct worker *wrk, int dropgrace); void HSH_Ref(struct objcore *o); void HSH_Drop(struct worker *wrk); void HSH_Init(const struct hash_slinger *slinger); diff --git a/bin/varnishtest/tests/grace-drop.vtc b/bin/varnishtest/tests/grace-drop.vtc new file mode 100644 index 0000000..8f51d60 --- /dev/null +++ b/bin/varnishtest/tests/grace-drop.vtc @@ -0,0 +1,120 @@ +varnishtest "Drop graced object when new object arrives" + +server s1 { + rxreq + txresp -hdr "Vary: X-Foo" -hdr "Cache-Control: max-age=1" -body "1" + rxreq + txresp -hdr "Vary: X-Foo" -hdr "Cache-Control: max-age=1" -body "12" + rxreq + txresp -hdr "Vary: X-Foo" -hdr "Cache-Control: max-age=1" -body "123" +} -start + +varnish v1 -vcl+backend { + sub vcl_recv { + set req.grace = 0s; + } + sub vcl_fetch { + set beresp.grace = 3600s; + } +} -start + +varnish v1 -expect n_object == 0 +varnish v1 -expect sess_conn == 0 +varnish v1 -expect client_req == 0 +varnish v1 -expect cache_miss == 0 +varnish v1 -expect cache_hit == 0 +varnish v1 -expect s_sess == 0 +varnish v1 -expect s_req == 0 + +client c1 { + txreq -hdr "X-Foo: bar" + rxresp + expect resp.status == 200 + expect resp.bodylen == 1 + txreq -hdr "X-Foo: blargh" + rxresp + expect resp.status == 200 + expect resp.bodylen == 2 +} -run + +delay 1.1 + +varnish v1 -expect n_object == 2 +varnish v1 -expect sess_conn == 1 +varnish v1 -expect client_req == 2 +varnish v1 -expect cache_miss == 2 +varnish v1 -expect cache_hit == 0 +varnish v1 -expect s_sess == 1 +varnish v1 -expect s_req == 2 + +client c1 { + txreq -hdr "X-Foo: bar" + rxresp + expect resp.status == 200 + expect resp.bodylen == 3 +} -run + +varnish v1 -expect n_object == 2 +varnish v1 -expect sess_conn == 2 +varnish v1 -expect client_req == 3 +varnish v1 -expect cache_miss == 3 +varnish v1 -expect cache_hit == 0 +varnish v1 -expect s_sess == 2 +varnish v1 -expect s_req == 3 + +server s1 -wait + +delay 1.1 + +client c1 { + txreq -hdr "X-Foo: bar" + rxresp + expect resp.status == 503 +} -run + +varnish v1 -expect n_object == 2 +varnish v1 -expect sess_conn == 3 +varnish v1 -expect client_req == 4 +varnish v1 -expect cache_miss == 4 +varnish v1 -expect cache_hit == 0 +varnish v1 -expect s_sess == 3 +varnish v1 -expect s_req == 4 + +varnish v1 -vcl { + backend default { + .host = "${s1_addr}"; + .port = "${s1_port}"; + .max_connections = 1; + .probe = { + .url = "/"; + .timeout = 1s; + .interval = 1000s; + .window = 1; + .threshold = 1; + .initial = 0; + } + } + + sub vcl_recv { + set req.grace = 3600s; + } +} + +client c1 { + txreq -hdr "X-Foo: bar" + rxresp + expect resp.status == 200 + expect resp.bodylen == 3 + txreq -hdr "X-Foo: blargh" + rxresp + expect resp.status == 200 + expect resp.bodylen == 2 +} -run + +varnish v1 -expect n_object == 2 +varnish v1 -expect sess_conn == 4 +varnish v1 -expect client_req == 6 +varnish v1 -expect cache_miss == 4 +varnish v1 -expect cache_hit == 2 +varnish v1 -expect s_sess == 4 +varnish v1 -expect s_req == 6 diff --git a/include/tbl/vsc_f_main.h b/include/tbl/vsc_f_main.h index df7984a..3b89faa 100644 --- a/include/tbl/vsc_f_main.h +++ b/include/tbl/vsc_f_main.h @@ -232,7 +232,12 @@ VSC_F(n_backend, uint64_t, 0, 'i', "N backends", "") VSC_F(n_expired, uint64_t, 0, 'i', "N expired objects", "") VSC_F(n_lru_nuked, uint64_t, 0, 'i', "N LRU nuked objects", "") +VSC_F(c_removed, uint64_t, 1, 'a', "N removed objects", "") VSC_F(n_lru_moved, uint64_t, 0, 'i', "N LRU moved objects", "") +VSC_F(c_grace_obj_dropped, uint64_t, 1, 'i', + "Objects in grace dropped due to new object from backend", "") +VSC_F(c_objcore_zero_ref, uint64_t, 1, 'i', + "objcore refcnt down to zero, shouldn't happen", "") VSC_F(losthdr, uint64_t, 0, 'a', "HTTP header overflows", "") -- 1.7.5.4 From varnish at bsdchicks.com Fri Jan 6 03:28:30 2012 From: varnish at bsdchicks.com (Rogier R. Mulhuijzen) Date: Fri, 6 Jan 2012 04:28:30 +0100 (CET) Subject: [PATCH] Drop old grace objects when we have a new object that matches the Vary In-Reply-To: <1325820122-7794-1-git-send-email-github@bsdchicks.com> References: <1325820122-7794-1-git-send-email-github@bsdchicks.com> Message-ID: <20120106042754.C37583@ishtar.drwilco.net> And of course, I forgot to take out a debugging stat counter. Please ignore its existence. Cheers, Doc From perbu at varnish-software.com Fri Jan 6 09:47:57 2012 From: perbu at varnish-software.com (Per Buer) Date: Fri, 6 Jan 2012 10:47:57 +0100 Subject: [PATCH] Drop old grace objects when we have a new object that matches the Vary In-Reply-To: <1325820122-7794-1-git-send-email-github@bsdchicks.com> References: <1325820122-7794-1-git-send-email-github@bsdchicks.com> Message-ID: On Fri, Jan 6, 2012 at 4:22 AM, Rogier 'DocWilco' Mulhuijzen < github at bsdchicks.com> wrote: > Also add EXP_Remove, to allow removing objects with touching them. > > This patch makes it possible to have a very short TTL, but a long grace. > Normally, if you were to set a 1s TTL and a 86400s grace, you would end > up with 86400 objects on a single objecthead, not even taking into > account Vary. With this patch, any object found in grace is kept with > the request, and upon succesful retrieval of a new object from the > backend, the grace object is removed. > Yes! Excellent work. -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From github at bsdchicks.com Fri Jan 6 15:54:46 2012 From: github at bsdchicks.com (Rogier 'DocWilco' Mulhuijzen) Date: Fri, 6 Jan 2012 16:54:46 +0100 Subject: [PATCH] Drop old grace objects when we have a new object that matches the Vary In-Reply-To: <1325820122-7794-1-git-send-email-github@bsdchicks.com> References: <1325820122-7794-1-git-send-email-github@bsdchicks.com> Message-ID: <1325865286-13513-1-git-send-email-github@bsdchicks.com> (Lessen learned: do not email patches at 6am. I've done a few small fixups, so here it is again.) Also add EXP_Remove, to allow removing objects with touching them. This patch makes it possible to have a very short TTL, but a long grace. Normally, if you were to set a 1s TTL and a 86400s grace, you would end up with 86400 objects on a single objecthead, not even taking into account Vary. With this patch, any object found in grace is kept with the request, and upon succesful retrieval of a new object from the backend, the grace object is removed. EXP_Remove was added so that the TTL and grace on the object aren't touched. This, in combination with some other work we've done, prevents pages from getting dirtied and thus having to be written out to disk in a environment where the cache cannot be purely kept in memory. --- bin/varnishd/cache/cache.h | 2 + bin/varnishd/cache/cache_center.c | 4 +- bin/varnishd/cache/cache_expire.c | 43 ++++++++++++ bin/varnishd/cache/cache_hash.c | 48 ++++++++++---- bin/varnishd/hash/hash_slinger.h | 2 +- bin/varnishtest/tests/grace-drop.vtc | 122 ++++++++++++++++++++++++++++++++++ include/tbl/vsc_f_main.h | 3 + 7 files changed, 209 insertions(+), 15 deletions(-) create mode 100644 bin/varnishtest/tests/grace-drop.vtc diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 6b0d0a9..cb8fe4e 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -584,6 +584,7 @@ struct req { const char *doclose; struct exp exp; + struct objcore *grace_oc; unsigned cur_method; unsigned handling; unsigned char sendbody; @@ -753,6 +754,7 @@ void EXP_Init(void); void EXP_Rearm(const struct object *o); int EXP_Touch(struct objcore *oc); int EXP_NukeOne(struct worker *w, struct lru *lru); +void EXP_Remove(struct worker *w, struct objcore *oc); /* cache_fetch.c */ struct storage *FetchStorage(struct worker *w, ssize_t sz); diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c index 4967c82..df9b9cd 100644 --- a/bin/varnishd/cache/cache_center.c +++ b/bin/varnishd/cache/cache_center.c @@ -928,7 +928,7 @@ cnt_fetchbody(struct sess *sp, struct worker *wrk, struct req *req) EXP_Insert(req->obj); AN(req->obj->objcore); AN(req->obj->objcore->ban); - HSH_Unbusy(wrk); + HSH_Unbusy(wrk, 1); } VBO_DerefBusyObj(wrk, &wrk->busyobj); wrk->acct_tmp.fetch++; @@ -987,7 +987,7 @@ cnt_streambody(struct sess *sp, struct worker *wrk, struct req *req) EXP_Insert(req->obj); AN(req->obj->objcore); AN(req->obj->objcore->ban); - HSH_Unbusy(wrk); + HSH_Unbusy(wrk, 1); } else { req->doclose = "Stream error"; } diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c index e93a1aa..a8c4254 100644 --- a/bin/varnishd/cache/cache_expire.c +++ b/bin/varnishd/cache/cache_expire.c @@ -451,6 +451,49 @@ EXP_NukeOne(struct worker *wrk, struct lru *lru) } /*-------------------------------------------------------------------- + * Remove from expiry binheap. Caller must have a ref on the oc. + */ + +void +EXP_Remove(struct worker *w, struct objcore *oc) +{ + struct object *o; + struct lru *l; + + CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); + o = oc_getobj(w, oc); + CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC); + l = oc_getlru(oc); + CHECK_OBJ_NOTNULL(l, LRU_MAGIC); + + Lck_Lock(&l->mtx); + Lck_Lock(&exp_mtx); + + if (oc->timer_idx == BINHEAP_NOIDX) { /* exp_timer has it */ + Lck_Unlock(&exp_mtx); + Lck_Unlock(&l->mtx); + return; + } + + /* remove from binheap */ + binheap_delete(exp_heap, oc->timer_idx); + assert(oc->timer_idx == BINHEAP_NOIDX); + + /* And from LRU */ + CHECK_OBJ_NOTNULL(l, LRU_MAGIC); + VTAILQ_REMOVE(&l->lru_head, oc, lru_list); + + Lck_Unlock(&exp_mtx); + Lck_Unlock(&l->mtx); + + w->stats.c_removed++; + + WSL(w, SLT_ExpKill, 0, "%u Removed", o->xid); + assert(oc->refcnt >= 2); /* exp ref and caller ref */ + HSH_Deref(w, NULL, &o); +} + +/*-------------------------------------------------------------------- * BinHeap helper functions for objcore. */ diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c index d967291..a3eb77c 100644 --- a/bin/varnishd/cache/cache_hash.c +++ b/bin/varnishd/cache/cache_hash.c @@ -296,7 +296,7 @@ HSH_Lookup(struct sess *sp, struct objhead **poh) struct worker *wrk; struct objhead *oh; struct objcore *oc; - struct objcore *busy_oc, *grace_oc; + struct objcore *busy_oc; struct object *o; double grace_ttl; @@ -332,7 +332,7 @@ HSH_Lookup(struct sess *sp, struct objhead **poh) Lck_Lock(&oh->mtx); assert(oh->refcnt > 0); busy_oc = NULL; - grace_oc = NULL; + sp->req->grace_oc = NULL; grace_ttl = NAN; VTAILQ_FOREACH(oc, &oh->objcs, list) { /* Must be at least our own ref + the objcore we examine */ @@ -372,9 +372,9 @@ HSH_Lookup(struct sess *sp, struct objhead **poh) * and if there are several, use the least expired one. */ if (EXP_Grace(sp, o) >= sp->t_req) { - if (grace_oc == NULL || + if (sp->req->grace_oc == NULL || grace_ttl < o->exp.entered + o->exp.ttl) { - grace_oc = oc; + sp->req->grace_oc = oc; grace_ttl = o->exp.entered + o->exp.ttl; } } @@ -391,15 +391,16 @@ HSH_Lookup(struct sess *sp, struct objhead **poh) */ AZ(sp->req->objcore); - sp->req->objcore = grace_oc; /* XXX: Hack-ish */ + sp->req->objcore = sp->req->grace_oc; /* XXX: Hack-ish */ if (oc == NULL /* We found no live object */ - && grace_oc != NULL /* There is a grace candidate */ + && sp->req->grace_oc != NULL /* There is a grace candidate */ && (busy_oc != NULL /* Somebody else is already busy */ || !VDI_Healthy(sp->req->director, sp))) { /* Or it is impossible to fetch */ - o = oc_getobj(sp->wrk, grace_oc); + o = oc_getobj(sp->wrk, sp->req->grace_oc); CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC); - oc = grace_oc; + oc = sp->req->grace_oc; + sp->req->grace_oc = NULL; } sp->req->objcore = NULL; @@ -415,6 +416,7 @@ HSH_Lookup(struct sess *sp, struct objhead **poh) assert(oh->refcnt > 1); Lck_Unlock(&oh->mtx); assert(hash->deref(oh)); + sp->req->grace_oc = NULL; *poh = oh; return (oc); } @@ -441,6 +443,7 @@ HSH_Lookup(struct sess *sp, struct objhead **poh) */ sp->req->hash_objhead = oh; sp->wrk = NULL; + sp->req->grace_oc = NULL; Lck_Unlock(&oh->mtx); return (NULL); } @@ -450,6 +453,15 @@ HSH_Lookup(struct sess *sp, struct objhead **poh) wrk->nobjcore = NULL; AN(oc->flags & OC_F_BUSY); oc->refcnt = 1; + /* + * make sure that we don't expire the graced object while we're fetching + * so that we don't panic/segfault after fetch + */ + if (sp->req->grace_oc) { + CHECK_OBJ_NOTNULL(sp->req->grace_oc, OBJCORE_MAGIC); + sp->req->grace_oc->refcnt++; + assert(sp->req->grace_oc->refcnt > 0); + } AZ(wrk->busyobj); wrk->busyobj = VBO_GetBusyObj(wrk); @@ -586,16 +598,16 @@ HSH_Drop(struct worker *wrk) AssertObjCorePassOrBusy(o->objcore); o->exp.ttl = -1.; if (o->objcore != NULL) /* Pass has no objcore */ - HSH_Unbusy(wrk); + HSH_Unbusy(wrk, 0); (void)HSH_Deref(wrk, NULL, &wrk->sp->req->obj); } void -HSH_Unbusy(struct worker *wrk) +HSH_Unbusy(struct worker *wrk, int dropgrace) { - struct object *o; + struct object *o, *go; struct objhead *oh; - struct objcore *oc; + struct objcore *oc, *grace_oc; CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); o = wrk->sp->req->obj; @@ -627,6 +639,18 @@ HSH_Unbusy(struct worker *wrk) hsh_rush(oh); AN(oc->ban); Lck_Unlock(&oh->mtx); + if (wrk->sp->req->grace_oc) { + grace_oc = wrk->sp->req->grace_oc; + CHECK_OBJ_NOTNULL(grace_oc, OBJCORE_MAGIC); + go = oc_getobj(wrk, grace_oc); + CHECK_OBJ_NOTNULL(go, OBJECT_MAGIC); + if (dropgrace) { + wrk->stats.c_grace_obj_dropped++; + EXP_Remove(wrk, grace_oc); + } + assert(grace_oc->refcnt > 0); + HSH_Deref(wrk, NULL, &go); + } assert(oc_getobj(wrk, oc) == o); } diff --git a/bin/varnishd/hash/hash_slinger.h b/bin/varnishd/hash/hash_slinger.h index b45e604..c0cd54c 100644 --- a/bin/varnishd/hash/hash_slinger.h +++ b/bin/varnishd/hash/hash_slinger.h @@ -54,7 +54,7 @@ struct hash_slinger { void HSH_Prealloc(const struct sess *sp); void HSH_Cleanup(struct worker *w); struct objcore *HSH_Lookup(struct sess *sp, struct objhead **poh); -void HSH_Unbusy(struct worker *wrk); +void HSH_Unbusy(struct worker *wrk, int dropgrace); void HSH_Ref(struct objcore *o); void HSH_Drop(struct worker *wrk); void HSH_Init(const struct hash_slinger *slinger); diff --git a/bin/varnishtest/tests/grace-drop.vtc b/bin/varnishtest/tests/grace-drop.vtc new file mode 100644 index 0000000..05cdbac --- /dev/null +++ b/bin/varnishtest/tests/grace-drop.vtc @@ -0,0 +1,122 @@ +varnishtest "Drop graced object when new object arrives" + +server s1 { + rxreq + txresp -hdr "Vary: X-Foo" -hdr "Cache-Control: max-age=1" -body "1" + rxreq + txresp -hdr "Vary: X-Foo" -hdr "Cache-Control: max-age=1" -body "12" + rxreq + txresp -hdr "Vary: X-Foo" -hdr "Cache-Control: max-age=1" -body "123" +} -start + +varnish v1 -vcl+backend { + sub vcl_recv { + set req.grace = 0s; + } + sub vcl_fetch { + set beresp.grace = 3600s; + } +} -start + +varnish v1 -expect n_object == 0 +varnish v1 -expect sess_conn == 0 +varnish v1 -expect client_req == 0 +varnish v1 -expect cache_miss == 0 +varnish v1 -expect cache_hit == 0 +varnish v1 -expect s_sess == 0 +varnish v1 -expect s_req == 0 + +client c1 { + txreq -hdr "X-Foo: bar" + rxresp + expect resp.status == 200 + expect resp.bodylen == 1 + txreq -hdr "X-Foo: blargh" + rxresp + expect resp.status == 200 + expect resp.bodylen == 2 +} -run + +delay 1.1 + +varnish v1 -expect n_object == 2 +varnish v1 -expect sess_conn == 1 +varnish v1 -expect client_req == 2 +varnish v1 -expect cache_miss == 2 +varnish v1 -expect cache_hit == 0 +varnish v1 -expect s_sess == 1 +varnish v1 -expect s_req == 2 + +client c1 { + txreq -hdr "X-Foo: bar" + rxresp + expect resp.status == 200 + expect resp.bodylen == 3 +} -run + +varnish v1 -expect n_object == 2 +varnish v1 -expect sess_conn == 2 +varnish v1 -expect client_req == 3 +varnish v1 -expect cache_miss == 3 +varnish v1 -expect cache_hit == 0 +varnish v1 -expect s_sess == 2 +varnish v1 -expect s_req == 3 + +server s1 -wait + +delay 1.1 + +client c1 { + txreq -hdr "X-Foo: bar" + rxresp + expect resp.status == 503 +} -run + +varnish v1 -expect n_object == 2 +varnish v1 -expect sess_conn == 3 +varnish v1 -expect client_req == 4 +varnish v1 -expect cache_miss == 4 +varnish v1 -expect cache_hit == 0 +varnish v1 -expect s_sess == 3 +varnish v1 -expect s_req == 4 + +varnish v1 -vcl { + backend default { + .host = "${s1_addr}"; + .port = "${s1_port}"; + .max_connections = 1; + .probe = { + .url = "/"; + .timeout = 1s; + .interval = 1000s; + .window = 1; + .threshold = 1; + .initial = 0; + } + } + + sub vcl_recv { + set req.grace = 3600s; + } +} + +delay 0.1 + +client c1 { + txreq -hdr "X-Foo: bar" + rxresp + expect resp.status == 200 + expect resp.bodylen == 3 + txreq -hdr "X-Foo: blargh" + rxresp + expect resp.status == 200 + expect resp.bodylen == 2 +} -run + +varnish v1 -expect n_object == 2 +varnish v1 -expect sess_conn == 4 +varnish v1 -expect client_req == 6 +varnish v1 -expect cache_miss == 4 +varnish v1 -expect cache_hit == 2 +varnish v1 -expect s_sess == 4 +varnish v1 -expect s_req == 6 diff --git a/include/tbl/vsc_f_main.h b/include/tbl/vsc_f_main.h index df7984a..60401a6 100644 --- a/include/tbl/vsc_f_main.h +++ b/include/tbl/vsc_f_main.h @@ -232,7 +232,10 @@ VSC_F(n_backend, uint64_t, 0, 'i', "N backends", "") VSC_F(n_expired, uint64_t, 0, 'i', "N expired objects", "") VSC_F(n_lru_nuked, uint64_t, 0, 'i', "N LRU nuked objects", "") +VSC_F(c_removed, uint64_t, 1, 'a', "N removed objects", "") VSC_F(n_lru_moved, uint64_t, 0, 'i', "N LRU moved objects", "") +VSC_F(c_grace_obj_dropped, uint64_t, 1, 'i', + "Objects in grace dropped due to new object from backend", "") VSC_F(losthdr, uint64_t, 0, 'a', "HTTP header overflows", "") -- 1.7.5.4 From sky at crucially.net Fri Jan 6 17:47:16 2012 From: sky at crucially.net (Artur Bergman) Date: Fri, 6 Jan 2012 09:47:16 -0800 Subject: [PATCH] Drop old grace objects when we have a new object that matches the Vary In-Reply-To: References: <1325820122-7794-1-git-send-email-github@bsdchicks.com> Message-ID: On Jan 6, 2012, at 1:47 AM, Per Buer wrote: > On Fri, Jan 6, 2012 at 4:22 AM, Rogier 'DocWilco' Mulhuijzen wrote: > Also add EXP_Remove, to allow removing objects with touching them. > > This patch makes it possible to have a very short TTL, but a long grace. > Normally, if you were to set a 1s TTL and a 86400s grace, you would end > up with 86400 objects on a single objecthead, not even taking into > account Vary. With this patch, any object found in grace is kept with > the request, and upon succesful retrieval of a new object from the > backend, the grace object is removed. > > Yes! Excellent work. > It should be able to be modified for 304 if-modified too. Artur -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at varnish-software.com Mon Jan 9 13:08:04 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 9 Jan 2012 14:08:04 +0100 Subject: Streaming patches Message-ID: Hi PHK, These are the streaming patches as they look now. I would very much like to have a discussion round this structure, the use of different modes when we don't have a thread available etc. See specifically the areas marked "MBGXXX" -Martin -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Functionality-for-a-worker-to-grab-another-worker-to.patch Type: text/x-patch Size: 8575 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-Make-FetchBody-take-a-busyobj-struct-as-parameter-in.patch Type: text/x-patch Size: 2944 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0003-Add-a-configurable-through-parameter-stream_maxchunk.patch Type: text/x-patch Size: 2154 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0004-Don-t-free-the-object-store-when-fetch-fails-and-str.patch Type: text/x-patch Size: 1239 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0005-Add-a-condvar-to-struct-vbo.patch Type: text/x-patch Size: 1089 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0006-Make-VBO_DerefBusyObj-return-refcount-after-decremen.patch Type: text/x-patch Size: 1494 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0007-Add-stream-data-synchronization-functions-to-cache_b.patch Type: text/x-patch Size: 3065 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0008-Rework-RES_StreamPoll-to-use-the-VBO_StreamData-and-.patch Type: text/x-patch Size: 5536 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0009-Add-VBO_StreamStopped-and-VBO_StreamWait-thread-sync.patch Type: text/x-patch Size: 2947 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0010-Add-general-lock-functions-for-busyobjs.patch Type: text/x-patch Size: 1616 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0011-Lock-busyobj-when-doing-late-object-header-changes-e.patch Type: text/x-patch Size: 1897 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0012-Use-background-thread-fetching-when-streaming.patch Type: text/x-patch Size: 13161 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0013-Add-a-couple-of-streaming-test-cases.patch Type: text/x-patch Size: 2271 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0014-Don-t-double-allocate-busyobj-on-pipes.patch Type: text/x-patch Size: 812 bytes Desc: not available URL: From jocelyn.delarosa at smartjog.com Tue Jan 10 15:41:48 2012 From: jocelyn.delarosa at smartjog.com (Jocelyn De La Rosa) Date: Tue, 10 Jan 2012 16:41:48 +0100 Subject: [PATCH][experimental-ims] Fix conditional backend request in streaming mode Message-ID: <4F0C5C3C.2040903@smartjog.com> Hello, First of all thank you Geoff for updating the experimental-ims branch yesterday! After some tests, I found out that there was a problem using streaming (do_stream = true) with conditional backend request. Here's a small patch that fixes the bug :-) Best, -- Jocelyn De La Rosa Smartjog - Developer - Research& Engineering -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Fix-conditional-backend-request-in-streaming-mode.patch Type: text/x-patch Size: 1029 bytes Desc: not available URL: From ufaria at prisadigital.com Tue Jan 10 15:52:11 2012 From: ufaria at prisadigital.com (Uxio Faria Giraldez) Date: Tue, 10 Jan 2012 07:52:11 -0800 Subject: deactivate obj updates Message-ID: <18AD381627308B47A6AED6D2E69D552F18E68857F0@IE2RD2XVS201.red002.local> Hello list. Last day i was wondering why varnish writes so much info to disc even if hot data keeps in memory and having high hitrate. I supposse is due to 'obj.' variables that needs to be updated: obj.hits obj.lastuse I think no more variables are updated on object HITS. If we do not need that info, is any way to deactivate that object updates to minify disc access and lower disc IO requeriments? thanks a lot! -- uxio faria -- -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at uplex.de Tue Jan 10 19:57:22 2012 From: geoff at uplex.de (Geoff Simmons) Date: Tue, 10 Jan 2012 20:57:22 +0100 Subject: [PATCH][experimental-ims] Fix conditional backend request in streaming mode In-Reply-To: <4F0C5C3C.2040903@smartjog.com> References: <4F0C5C3C.2040903@smartjog.com> Message-ID: <4F0C9822.1010108@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 01/10/12 04:41 PM, Jocelyn De La Rosa wrote: > > First of all thank you Geoff for updating the experimental-ims branch > yesterday! You're welcome, but I hope the update didn't create a lot conflicts for you. I didn't want to announce anything until after I'd done more stress testing, but if I'm not the only person working on the branch I'll speak up sooner. So, the belated info is: the experimental-ims branch is now merged with master as of yesterday, and passes all of the test cases. But I don't think it's ready to pass all of the stress tests yet. > After some tests, I found out that there was a problem using streaming > (do_stream = true) with conditional backend request. > > Here's a small patch that fixes the bug :-) Thanks, I had noticed the bug but hadn't been able to investigate it. And didn't expect it to be so simple. %^) I'll try it out. This has raised some questions in my mind about if and how IMS should interact with streaming, something we haven't considered at all so far. That may need some discussion, but first I'll try the patch. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Schwanenwik 24 22087 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (SunOS) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJPDJghAAoJEOUwvh9pJNURBTYQAJnjVH5Od4XOwbKOY2o6L8vq +2dK9B54a9ltHAZ+SxVvw70cho4E3MPxO7TjKrl1ab2PBMfJ1cL6NVCJyTKh6yzJ FFOcT2VUrk5q08nsqLnp1zF9BVpKuLF/7PKMjEq5un8Ge5YuVtAta2AwMnkzAGXG Kq5WazDMlc0G9qgEnq/uytGBChRu5x1ZMotZ3PE2UL+rpaxgDlllJGj3405/QfDc 91EcPbG5k0e+aGq6hrsLjDgOK82uMZ/Dc6bUjpnUhXpaO8UoiGbnVX7FWZp0/nj7 45Ii2Ekib7ZQEOGsFbGu7jGhMwmAS/uulRYmuseL/KdCzL/raedVYcOmtuNetU6n mx6NQ1ymM971KhZl6WWny5Px2x50ZICJl6l/506va3grbau9PieG6im3eZyrEV1t TZb7Ubg/ugIhHz+uFSyEj6XZG5fNSGPcTAhYM34kaBefy/NrKdXrk0cSWAGtZe8j 3nvZljL7T6oMcPs0D/p3QGS2wNGueCszmZuM3TBdOAYIBfTYHhX53ZrqLuY/W9Yw EVR1Jl6ZbV/Q/uUr80FBS2TTok5Kam31apK7nYlOg+G+O8hpG9e8voHTXQ/aTJVk AYEddlQT33FE4WzYMRTpGuC7rTgSI0vBL0VvySOsmqVpf1Viiz9i5fMX76yE5NmA XejXRjVipnG3VAR2WnKL =wRdD -----END PGP SIGNATURE----- From geoff at uplex.de Thu Jan 12 11:32:02 2012 From: geoff at uplex.de (Geoff Simmons) Date: Thu, 12 Jan 2012 12:32:02 +0100 Subject: Streaming and backend conditional requests Message-ID: <4F0EC4B2.7090102@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hello all, Yesterday there was a discussion about this subject on the #varnish-hacking IRC channel, and we decided that a summary should be sent up to varnish-dev, to ask for feedback. @phk, I hope we can get your opinions. For background on backend conditional requests (usually If-Modified-Since), currently implemented in the branch experimental-ims of the source repository, look here: https://www.varnish-cache.org/trac/wiki/BackendConditionalRequests This implementation never chooses a busy object during hash lookup as the stale object (the candidate object from cache to be validated with the conditional request). The reason for this was mainly just to not change too much about the way Varnish works. If a matching busy object is found in HSH_Lookup(), then it returns just as it does in the current master; a stale object is taken into consideration only if that doesn't happen. In 3.0 streaming, a streaming object always has the busy flag set; so a streaming object can never become the stale object in experimental-ims. As I've understood Martin, this will change in new streaming -- IIUC a streaming object won't have the busy flag set, but will have a non-NULL busyobj. So something will have to change for IMS to work with new streaming. The question, however, is whether passing over all currently streaming objects as potential stale_objs for IMS is a good idea. Suppose a very large object is streaming, which takes so long that its TTL elapses before streaming is finished. One might say that this is a poorly configured TTL, but it can nevertheless happen. When a new client request for this object now arrives, it could conceivably be validated against the backend as "Not Modified", so that it can be streamed from Varnish to the second client as well, without having the backend send it again. Reducing the bandwidth from backends for large objects is, after all, a main goal of having conditional requests. experimental-ims as currently implemented couldn't do that. If the stale_obj is validated by the backend, then: - - HTTP headers that the stale_obj has but the beresp object created for the response doesn't have are filtered into the beresp object. - - Storage for the stale_obj is duped into the storage for the beresp object. "Dupe" is currently implemented just by copying the storage, although the interfaces have been set up so that in the future, something more efficient could be implemented -- what we've had in mind is that storage can be shared with pointers by more than one object, and storage objects have refcounts. Last year, phk advised us to just implement the storage dupe with copying for now, and implement something like refcounting in a future version. As I understood it, this was so that we don't try to change too many things at once; and I think it wasn't entirely clear how to implement something like shared storage and refcounting for the persistent stevedore (I've had a look and it hasn't been clear to me). So as things are, experimental-ims couldn't try to validate an object that is currently streaming, with or without the busy flag, because it has to be able to copy its storage. The questions are: - - Should backend validation stay this way, or should we try to change it as suggested above (validate a currently streaming object)? (Opinions in IRC were that this would definitely be worthwhile.) - - If so, how should we manage the storage for the stale_obj and the beresp object? Dupe by copying, as now implemented, would have to be changed. An idea on IRC was: rather than introducing refcounting into storage and sharing it among objects, have the objects point to each other, and make use of the refcounting for objects that already exists. So the beresp object, with the updated headers, would have no storage of its own, but would point to the stale_obj, indicating that its storage is to be found there. I haven't fully seen through all of this (and may have restated it incorrectly); but the idea of using a refcounting solution that has existed and been tested for a long time seems promising. Opinions, comments? What do you think? Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Schwanenwik 24 22087 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (SunOS) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJPDsSyAAoJEOUwvh9pJNUR3ZcP/1kRr4zttzTWoCiOyMShjfs4 IKG42fFhet0rHbQqR8osSkhZndofoaAe/x5rK5ZD7+eACKL0DC+5CcOxbHHFjd0J 6YKoAQQ0p6KNont7LFrKovq4pUUDXIFQ6CovWju/VO6EeRbN2xvjephpS7It9Vhe 5W/ooJSwcP9az4V5kx6nXBq+KW1nOgVS4koW5AtQhuDmKptOjHTX9ZajQN+rGXjv 2Yyt5c/DbRgJLUcWs3q2RVUWB/o2toeyl6K7H34tfEz40tB12+7MGdWzKHQoqUcI 7Y53Gd8/wjQ3BTFcIG59qQT6dcQTxNHXuQXM4vvqOA5+FFsKVqw1kpFAMhdVcTNA 6YOvFeN51y5lUzhxD4Jltt+2Xar1MIaCqsiV48JWerLFxE3QlX53IWdifvIq0Hqi DU8+QXkaGhg2n4lc+AalXNXbe0iMO5qij935W1sCPGivxXjdsbUaYhooh868jhRS kVAZ5M1FcW24myo1RkuyQIjhdb1t8+/Dyv4l2fsgo3LMXjvZusA1rnXd14rB1aho SK1BSpzBQ9ZePNPknj+HSpTTYeffQLknt9ijx7osqwcs+zYLjZJVRPbInVKrB/yN QsU+cW7Toprfdu4AwsK0MsKqsDwTDvh0Q5wp6cGkGe6me92o2EDd8jEMVp4toDRu V5KmQhFb2jm37H/kLG3v =bp9k -----END PGP SIGNATURE----- From tfheen at varnish-software.com Thu Jan 12 11:58:52 2012 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Thu, 12 Jan 2012 12:58:52 +0100 Subject: vcl_error with streaming Message-ID: <20120112115852.GA12620@err.no> We have talked a bit about how vcl_error interacts with streaming, and the idea so far has been to remove vcl_error and add a synthetic capability to vcl_deliver. Last night, phk had an idea and he and I had a chat about how we could go about not getting rid of vcl_error and its implications. Further, we had a bit of discussion in the office this morning about this and some sort of consensus emerged. - vcl_error stays, but delivers directly and does not pass vcl_deliver. vcl_error is an error handler and should not be used for redirects and similar - vcl_synth is added, for any kind of synthetic content, typically redirects. The default vcl_synth can include the necessary couple of lines to handle redirects directly, so you can then just do return(synth(301, "http://varnish-cache.org/")); wherever you want to redirect. Both vcl_error and vcl_synth terminate and does not go through vcl_deliver, vcl_fetch or any other place. Does this sound roughly sane (and could Kristian and Martin please correct any inaccuracies)? -- Tollef Fog Heen Technical lead, Varnish Software t: +47 21 98 92 64 From martin at varnish-software.com Thu Jan 12 14:30:06 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Thu, 12 Jan 2012 15:30:06 +0100 Subject: Streaming documentation Message-ID: Hi PHK, Attached is a patch with proposed documentation changes for streaming-ng. (Not all of the functionality referenced there is implemented yet). Example VCL usage follows: sub vcl_fetch { set beresp.do_stream = true; if (req.http.host ~ "my.dvdimage.host") { /* These hosts serve static dvdimage's from nginx, don't try to cache and don't waste cache space on them */ set beresp.stream_pass_bufsize = 10M; return (hit-for-pass); } else if (req.http.host ~ "custom.image.host") { /* Serves custum tailored images, non-cacheable. Forking webserver, keep backend connection time to a minimum */ set beresp.stream_pass_bufsize = 0; return (hit-for-pass); } } Martin -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: streaming-documentation.patch Type: text/x-patch Size: 3610 bytes Desc: not available URL: From sky at crucially.net Thu Jan 12 17:29:23 2012 From: sky at crucially.net (Artur Bergman) Date: Thu, 12 Jan 2012 09:29:23 -0800 Subject: Streaming and backend conditional requests In-Reply-To: <4F0EC4B2.7090102@uplex.de> References: <4F0EC4B2.7090102@uplex.de> Message-ID: I have no idea why all this complexity exist, and I have no idea why it is exposed to VCL. It seems to me that an IMS patch can be fairly transparent, and very very simple. Artur On Jan 12, 2012, at 3:32 AM, Geoff Simmons wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Hello all, > > Yesterday there was a discussion about this subject on the > #varnish-hacking IRC channel, and we decided that a summary should be > sent up to varnish-dev, to ask for feedback. @phk, I hope we can get > your opinions. > > For background on backend conditional requests (usually > If-Modified-Since), currently implemented in the branch experimental-ims > of the source repository, look here: > > https://www.varnish-cache.org/trac/wiki/BackendConditionalRequests > > This implementation never chooses a busy object during hash lookup as > the stale object (the candidate object from cache to be validated with > the conditional request). The reason for this was mainly just to not > change too much about the way Varnish works. If a matching busy object > is found in HSH_Lookup(), then it returns just as it does in the current > master; a stale object is taken into consideration only if that doesn't > happen. > > In 3.0 streaming, a streaming object always has the busy flag set; so a > streaming object can never become the stale object in experimental-ims. > As I've understood Martin, this will change in new streaming -- IIUC a > streaming object won't have the busy flag set, but will have a non-NULL > busyobj. > > So something will have to change for IMS to work with new streaming. The > question, however, is whether passing over all currently streaming > objects as potential stale_objs for IMS is a good idea. > > Suppose a very large object is streaming, which takes so long that its > TTL elapses before streaming is finished. One might say that this is a > poorly configured TTL, but it can nevertheless happen. When a new client > request for this object now arrives, it could conceivably be validated > against the backend as "Not Modified", so that it can be streamed from > Varnish to the second client as well, without having the backend send it > again. Reducing the bandwidth from backends for large objects is, after > all, a main goal of having conditional requests. > > experimental-ims as currently implemented couldn't do that. If the > stale_obj is validated by the backend, then: > > - - HTTP headers that the stale_obj has but the beresp object created for > the response doesn't have are filtered into the beresp object. > > - - Storage for the stale_obj is duped into the storage for the beresp > object. > > "Dupe" is currently implemented just by copying the storage, although > the interfaces have been set up so that in the future, something more > efficient could be implemented -- what we've had in mind is that storage > can be shared with pointers by more than one object, and storage objects > have refcounts. > > Last year, phk advised us to just implement the storage dupe with > copying for now, and implement something like refcounting in a future > version. As I understood it, this was so that we don't try to change too > many things at once; and I think it wasn't entirely clear how to > implement something like shared storage and refcounting for the > persistent stevedore (I've had a look and it hasn't been clear to me). > > So as things are, experimental-ims couldn't try to validate an object > that is currently streaming, with or without the busy flag, because it > has to be able to copy its storage. The questions are: > > - - Should backend validation stay this way, or should we try to change it > as suggested above (validate a currently streaming object)? (Opinions in > IRC were that this would definitely be worthwhile.) > > - - If so, how should we manage the storage for the stale_obj and the > beresp object? Dupe by copying, as now implemented, would have to be > changed. > > An idea on IRC was: rather than introducing refcounting into storage and > sharing it among objects, have the objects point to each other, and make > use of the refcounting for objects that already exists. So the beresp > object, with the updated headers, would have no storage of its own, but > would point to the stale_obj, indicating that its storage is to be found > there. I haven't fully seen through all of this (and may have restated > it incorrectly); but the idea of using a refcounting solution that has > existed and been tested for a long time seems promising. > > Opinions, comments? What do you think? > > > Best, > Geoff > - -- > ** * * UPLEX - Nils Goroll Systemoptimierung > > Schwanenwik 24 > 22087 Hamburg > > Tel +49 40 2880 5731 > Mob +49 176 636 90917 > Fax +49 40 42949753 > > http://uplex.de > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.10 (SunOS) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iQIcBAEBCAAGBQJPDsSyAAoJEOUwvh9pJNUR3ZcP/1kRr4zttzTWoCiOyMShjfs4 > IKG42fFhet0rHbQqR8osSkhZndofoaAe/x5rK5ZD7+eACKL0DC+5CcOxbHHFjd0J > 6YKoAQQ0p6KNont7LFrKovq4pUUDXIFQ6CovWju/VO6EeRbN2xvjephpS7It9Vhe > 5W/ooJSwcP9az4V5kx6nXBq+KW1nOgVS4koW5AtQhuDmKptOjHTX9ZajQN+rGXjv > 2Yyt5c/DbRgJLUcWs3q2RVUWB/o2toeyl6K7H34tfEz40tB12+7MGdWzKHQoqUcI > 7Y53Gd8/wjQ3BTFcIG59qQT6dcQTxNHXuQXM4vvqOA5+FFsKVqw1kpFAMhdVcTNA > 6YOvFeN51y5lUzhxD4Jltt+2Xar1MIaCqsiV48JWerLFxE3QlX53IWdifvIq0Hqi > DU8+QXkaGhg2n4lc+AalXNXbe0iMO5qij935W1sCPGivxXjdsbUaYhooh868jhRS > kVAZ5M1FcW24myo1RkuyQIjhdb1t8+/Dyv4l2fsgo3LMXjvZusA1rnXd14rB1aho > SK1BSpzBQ9ZePNPknj+HSpTTYeffQLknt9ijx7osqwcs+zYLjZJVRPbInVKrB/yN > QsU+cW7Toprfdu4AwsK0MsKqsDwTDvh0Q5wp6cGkGe6me92o2EDd8jEMVp4toDRu > V5KmQhFb2jm37H/kLG3v > =bp9k > -----END PGP SIGNATURE----- > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev From sky at crucially.net Thu Jan 12 17:34:18 2012 From: sky at crucially.net (Artur Bergman) Date: Thu, 12 Jan 2012 09:34:18 -0800 Subject: vcl_error with streaming In-Reply-To: <20120112115852.GA12620@err.no> References: <20120112115852.GA12620@err.no> Message-ID: <0F410A27-62F6-4B34-A700-24A5CB47021B@crucially.net> Why can't vcl_error/vcl_synth go through deliver? We have a fair amount of stuff in deliver that would have to be replicated. Artur On Jan 12, 2012, at 3:58 AM, Tollef Fog Heen wrote: > We have talked a bit about how vcl_error interacts with streaming, and > the idea so far has been to remove vcl_error and add a synthetic > capability to vcl_deliver. Last night, phk had an idea and he and I had > a chat about how we could go about not getting rid of vcl_error and its > implications. Further, we had a bit of discussion in the office this > morning about this and some sort of consensus emerged. > > - vcl_error stays, but delivers directly and does not pass vcl_deliver. > vcl_error is an error handler and should not be used for redirects and > similar > > - vcl_synth is added, for any kind of synthetic content, typically > redirects. The default vcl_synth can include the necessary couple of > lines to handle redirects directly, so you can then just do > return(synth(301, "http://varnish-cache.org/")); wherever you want to > redirect. > > Both vcl_error and vcl_synth terminate and does not go through > vcl_deliver, vcl_fetch or any other place. > > Does this sound roughly sane (and could Kristian and Martin please > correct any inaccuracies)? > > -- > Tollef Fog Heen > Technical lead, Varnish Software > t: +47 21 98 92 64 > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev From geoff at uplex.de Thu Jan 12 18:34:10 2012 From: geoff at uplex.de (Geoff Simmons) Date: Thu, 12 Jan 2012 19:34:10 +0100 Subject: Streaming and backend conditional requests In-Reply-To: References: <4F0EC4B2.7090102@uplex.de> Message-ID: <4F0F27A2.5020306@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 01/12/12 06:29 PM, Artur Bergman wrote: > I have no idea why all this complexity exist, and I have no idea why it is exposed to VCL. > > It seems to me that an IMS patch can be fairly transparent, and very very simple. Fair enough. But actually the considerations here don't involve VCL at all. One way or another, streaming and IMS will want to co-exist in Varnish. We could avoid the complexity by leaving it as it is now -- then IMS will never attempt to validate an object while it's still streaming. But it doesn't seem unreasonable to want to take advantage of IMS to avoid fetching a large object that Varnish already has. Especially when new streaming arrives, which IIUC aims to make a streaming object available to more than one client at a time. If we want to have that, then the simple solution in place now won't work. We'll have to work out how IMS interacts with streaming and storage. Any working solution that keeps things simple would certainly be preferable. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Schwanenwik 24 22087 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (SunOS) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJPDyeiAAoJEOUwvh9pJNURtq4P/1soeHZb7ROz6rkmrZ8wigvE ti4KMybz0rz7gYfDMtj57Bra3+Ti2Pig5mrN9eHNMsPwOFprrXy8kI6f/akRHKLh JbaPuEH3QMqPC68AdNyExyt6nkbFy4LkzcZ+qmq3r5LN3GSM7jF2gXXiuvQmdAce NWwqg58HD9BqyOBjQ/hC4Wp0SfGUaCQPslSBebk2W6m6r4/6tE/+tPMjtj2ROHRj yaM/7CzBVunqzCmZs+JkjNGNE2kF11/LIBAQbeeap4R70iXmgmiPcZnlsDAxBIbx 17iMFBPoFNVXRhaEAv5GGbob5AoQzcHUtKY1oyYrjlN2dFzcy7bzYxoenBMwSTOf 8SuJDw88PkSl58dq65GJj0WHX68eeW95lOa4ZN/tIFOIvrnHN8GvQYCoe5Gjskrf UdOqdlrrai8rBXtIJS/JaWcIGhxeowdoFJAYPmoJr2dlAih8hRrLgOKepB1QDeMm ijWhCH72Oapb0rW7I1mHSPRllQyou9IqbjS4G+SNr1UW78d8y+X+ZkHmlzI+DE44 9C6mQXKj3zsscejl1u8j1rc04t/kwvAZDsb8+g0rXtw7Qch/A+bL0Tv+Fp/pAV8M YLK9aVG5nXgB4Ht5s2o279pUMJlOU98ROD+IfrYfnnL1WzpkgqchcDCCXtEeuFgx q0jPAdHpekVNDO5Je1zF =1rZP -----END PGP SIGNATURE----- From phk at phk.freebsd.dk Fri Jan 13 13:18:16 2012 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 13 Jan 2012 13:18:16 +0000 Subject: Streaming and backend conditional requests In-Reply-To: Your message of "Thu, 12 Jan 2012 12:32:02 +0100." <4F0EC4B2.7090102@uplex.de> Message-ID: <5156.1326460696@critter.freebsd.dk> Sorry for being a bit late to this discussion, I'm still trying to get through my contractually obligated New Year ritual. I think the discussion is somewhat muddled by looking too much at how to move forward, I would far rather discuss where we want to go in the first place. My goal, is to have backend-IMS and streaming fetch both be default-on features. Usually the way we've done things is one release with $newfeature being default-off, then make it default-on in the next release, but I am seriously considering skipping that intermediate step, because it invariably means VCL changes which become pointless a release later. So, going directly after the IMS+Streaming situation, there's how I see the VCL: Pipe is pipe and nothing changes there. Once you issue a fetch, (from vcl_pass/vcl_miss) you _always_ end up in vcl_fetch{}, but you may have a synthetic 503 object if we failed to get a backend connection. This allows the 503 to be cached (ie: single-backend, no-probe) By default all backend fetches which can find a candidate for it, will try to IMS against the backend. If you do not want that, you remove the bereq.http.IMS header in vcl_pass{}/vcl_miss{} In vcl_fetch{} we add a "BOOL fetch_body()" function, so that you can insist on pulling the entire object from the backend, before accepting it into the cache, before deciding to deliver it to the client, or for doing (vmod-) modifications to it before delivery. If you do not call fetch_body(), only the headers are available in vcl_fetch{} and the body will be fetched asynchronously and delivered to any client hitting this object in parallel. vcl_error{} becomes vcl_synth{}, and you call it like: return(synth(202)); return(synth(578, "You got to be kidding")); And the arguments go into resp.status and resp.response vcl_synth{} will only have access to req.* and resp.*, no obj.* or bere*.* Two issues need resolved: Can you cache synth objects ? Do they get delivered through vcl_deliver ? My initial idea was "no & no", but I am willing to be persuaded otherwise, and Artur may be halfway there already. The argument for no caching is simply that Varnish is not a webserver and I'd hate to see people try to turn it into one. The second is more involved: Synth replies will not have an obj.* and therefore delivery is mechanically quite different from an obj.* based delivery. Also, we won't be doing gunzip or ESI on synth replies. Now, there is at least one inconsistency in the above, and I'm open for input on how to best solve it: You cannot change the body of the 503 you get in vcl_fetch{}, unless you call vcl_synth{}, but then you can't cache that 503... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From geoff at uplex.de Fri Jan 13 15:14:41 2012 From: geoff at uplex.de (Geoff Simmons) Date: Fri, 13 Jan 2012 16:14:41 +0100 Subject: [PATCH] need to base port_getn timeout on sp->t_idle Message-ID: <4F104A61.2080704@uplex.de> The attached patch (by slink) is a one-liner for waiter/cache_waiter_ports.c, the code was referencing sp->t_open where it should have been sp->t_idle, which was causing assertion failures on idle session timeouts. -- ** * * UPLEX - Nils Goroll Systemoptimierung Schwanenwik 24 22087 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: 0001-need-to-base-port_getn-timeout-on-sp-t_idle.patch URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 896 bytes Desc: OpenPGP digital signature URL: From martin at varnish-software.com Fri Jan 13 18:00:00 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 13 Jan 2012 19:00:00 +0100 Subject: Streaming patches Message-ID: Hi PHK, Sending you the latest streaming patches so you have the up-to-date picture when you take a look at them. New from latest submission is: - Multiple streaming clients is supported - Flip-flop fallback is implemented - Bufsize throttling on pass is implemented - More test-cases Martin -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: streaming-part1-take3.tar.gz Type: application/x-gzip Size: 23161 bytes Desc: not available URL: From geoff at uplex.de Sat Jan 14 13:56:32 2012 From: geoff at uplex.de (Geoff Simmons) Date: Sat, 14 Jan 2012 14:56:32 +0100 Subject: Streaming and backend conditional requests In-Reply-To: <5156.1326460696@critter.freebsd.dk> References: <5156.1326460696@critter.freebsd.dk> Message-ID: <4F118990.7020702@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 1/13/12 2:18 PM, Poul-Henning Kamp wrote: > > By default all backend fetches which can find a candidate for it, > will try to IMS against the backend. If you do not want that, you > remove the bereq.http.IMS header in vcl_pass{}/vcl_miss{} What got the conversation going about this is that in the current implementation, a busy object can never be the candidate for a validation. So in 3.0 streaming, it can never be an object that is still streaming. If that's the right solution now and in the future, then everything's fine. When new streaming arrives, if I've understood correctly, a second request for a streaming object can be served from the same object in the cache. If this happens after TTL has already elapsed, then a validation against the backend could avoid another fetch for a large object that hasn't changed. If we want that, the innards of IMS implementation would have to change, because now it can only work with an object stored in entirety in the cache. If not, IMS would at least have to recognize that an object is currently streaming (to be able to exclude it), since Martin has advised that busy flag won't be set. Thanks for the update. Geoff - -- UPLEX Systemoptimierung Schwanenwik 24 22087 Hamburg http://uplex.de/ Mob: +49-176-63690917 -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJPEYmPAAoJEOUwvh9pJNURAIoP/A2eNUq32SnQztU/X/zqx6xZ +2XUVQKdsOnU5TbG5sV53cFvO1UybRGI5VNGs++KhxkEv1IHAwIEq2Kr19QVrKHC 6oIb+dhb/820xGhpC0P0PJBVCkCuiAevuFy7NwIWl9eZDrkxZJqgXiCmQ6ip/rvW mBw4lAmnIgaAIVidmFRGeO5R5Xk/JrIf6e00c7sGP95WZMd01E7d9pfegb6Hav8w tm52SZ4bam6D9FI9jqgagGfZJp/bWWXhyc9+ZN1NQeO8/jh/iHqih6rG3dv/XcPH YO6rSVuGefdeNqcjrnAAObOsmu37nnmWINydgA3k2cdROQJ5ZMGuToNtAc4wUcgk uy/9wz7fVBxFdX0pWZz1umhkgCJpsYBlDHie+lmhuog5EyCFFnJCgItaUnbDg71V W5RDp5ZU4LpvJ2p876fT8Hgsb5cJe6svnLO838hm4APqY00WzpzkDpm7B6zoojeK 2AvGXMrEZ4Kogrc0rIMoXMWA0KSwc3tMZda58SKW1bBakqxUNJzs6BZ3m75hgwXQ 6En3f40CbSsC7p/TFhYtJphO/28Ky8es3xlVCFFXPzEOKDq7rMmlqRALUF1sTwR7 AMjoQSUgsxxVmkzZlyn3LJgcsSbcIvHgPHJUqNSE8JjiXo3r4SL5PVRThqfT+Usu +lOhja0Fl/P+W1BkTL4V =lG6D -----END PGP SIGNATURE----- From github at bsdchicks.com Sat Jan 14 17:49:55 2012 From: github at bsdchicks.com (Rogier 'DocWilco' Mulhuijzen) Date: Sat, 14 Jan 2012 18:49:55 +0100 Subject: [PATCH] Add __printflike to all printflike functions and fix all that uncovers Message-ID: <1326563395-8543-1-git-send-email-github@bsdchicks.com> --- bin/varnishd/cache/cache.h | 12 +++++--- bin/varnishd/cache/cache_center.c | 6 ++-- bin/varnishd/cache/cache_shmlog.c | 4 +++ bin/varnishd/cache/cache_vrt_vmod.c | 2 +- bin/varnishd/mgt/mgt.h | 4 ++- bin/varnishd/mgt/mgt_param.c | 2 +- bin/varnishreplay/varnishreplay.c | 5 +++ bin/varnishtest/vtc.c | 8 +++--- bin/varnishtest/vtc.h | 9 ++++-- bin/varnishtest/vtc_http.c | 6 ++-- bin/varnishtest/vtc_sema.c | 2 +- include/printflike.h | 51 +++++++++++++++++++++++++++++++++++ include/vcli_priv.h | 5 +++- include/vsb.h | 6 +--- lib/libvarnishapi/vsm_api.h | 5 +++- lib/libvcl/vcc_acl.c | 9 +++--- lib/libvcl/vcc_backend.c | 2 +- lib/libvcl/vcc_compile.c | 2 +- lib/libvcl/vcc_compile.h | 15 +++++++--- lib/libvcl/vcc_dir_dns.c | 4 +- lib/libvcl/vcc_expr.c | 6 +++- lib/libvcl/vcc_parse.c | 4 +- 22 files changed, 125 insertions(+), 44 deletions(-) create mode 100644 include/printflike.h diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 6b0d0a9..356d138 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -809,7 +809,8 @@ void http_PutStatus(struct http *to, uint16_t status); void http_PutResponse(struct worker *w, unsigned vsl_id, const struct http *to, const char *response); void http_PrintfHeader(struct worker *w, unsigned vsl_id, struct http *to, - const char *fmt, ...); + const char *fmt, ...) + __printflike(4, 5); void http_SetHeader(struct worker *w, unsigned vsl_id, struct http *to, const char *hdr); void http_SetH(const struct http *to, unsigned n, const char *fm); @@ -928,10 +929,13 @@ void *VSM_Alloc(unsigned size, const char *class, const char *type, const char *ident); void VSM_Free(void *ptr); #ifdef VSL_ENDMARKER -void VSL(enum VSL_tag_e tag, int id, const char *fmt, ...); +void VSL(enum VSL_tag_e tag, int id, const char *fmt, ...) + __printflike(3, 4); void WSLR(struct worker *w, enum VSL_tag_e tag, int id, txt t); -void WSL(struct worker *w, enum VSL_tag_e tag, int id, const char *fmt, ...); -void WSLB(struct worker *w, enum VSL_tag_e tag, const char *fmt, ...); +void WSL(struct worker *w, enum VSL_tag_e tag, int id, const char *fmt, ...) + __printflike(4, 5); +void WSLB(struct worker *w, enum VSL_tag_e tag, const char *fmt, ...) + __printflike(3, 4); void WSL_Flush(struct worker *w, int overflow); diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c index 4967c82..1b62937 100644 --- a/bin/varnishd/cache/cache_center.c +++ b/bin/varnishd/cache/cache_center.c @@ -1655,12 +1655,12 @@ cnt_diag(struct sess *sp, const char *state) } if (sp->wrk != NULL) { - WSP(sp, SLT_Debug, "thr %p STP_%s sp %p obj %p vcl %p", + WSP(sp, SLT_Debug, "thr %lx STP_%s sp %p obj %p vcl %p", pthread_self(), state, sp, obj, vcl); WSL_Flush(sp->wrk, 0); } else { VSL(SLT_Debug, sp->vsl_id, - "thr %p STP_%s sp %p obj %p vcl %p", + "thr %lx STP_%s sp %p obj %p vcl %p", pthread_self(), state, sp, obj, vcl); } } @@ -1776,7 +1776,7 @@ cli_debug_srandom(struct cli *cli, const char * const *av, void *priv) seed = strtoul(av[2], NULL, 0); srandom(seed); srand48(random()); - VCLI_Out(cli, "Random(3) seeded with %lu", seed); + VCLI_Out(cli, "Random(3) seeded with %u", seed); } static struct cli_proto debug_cmds[] = { diff --git a/bin/varnishd/cache/cache_shmlog.c b/bin/varnishd/cache/cache_shmlog.c index c29f5a1..a839f5a 100644 --- a/bin/varnishd/cache/cache_shmlog.c +++ b/bin/varnishd/cache/cache_shmlog.c @@ -236,6 +236,10 @@ WSLR(struct worker *wrk, enum VSL_tag_e tag, int id, txt t) static void wsl(struct worker *wrk, enum VSL_tag_e tag, int id, const char *fmt, va_list ap) + __printflike(4, 0); + +static void +wsl(struct worker *wrk, enum VSL_tag_e tag, int id, const char *fmt, va_list ap) { char *p; unsigned n, mlen; diff --git a/bin/varnishd/cache/cache_vrt_vmod.c b/bin/varnishd/cache/cache_vrt_vmod.c index 6b3b846..c67e85d 100644 --- a/bin/varnishd/cache/cache_vrt_vmod.c +++ b/bin/varnishd/cache/cache_vrt_vmod.c @@ -104,7 +104,7 @@ VRT_Vmod_Init(void **hdl, void *ptr, int len, const char *nm, AN(w); if (strcmp(x, nm)) { VCLI_Out(cli, "Loading VMOD %s from %s:\n", nm, path); - VCLI_Out(cli, "File contain wrong VMOD (\"%s\")\n", x); + VCLI_Out(cli, "File contain wrong VMOD (\"%s\")\n", (char *) x); VCLI_Out(cli, "Check relative pathnames ?.\n"); (void)dlclose(v->hdl); FREE_OBJ(v); diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h index 0e3eb9e..15666ca 100644 --- a/bin/varnishd/mgt/mgt.h +++ b/bin/varnishd/mgt/mgt.h @@ -31,6 +31,7 @@ #include #include "common/common.h" +#include "printflike.h" struct cli; @@ -50,7 +51,8 @@ void MGT_Child_Cli_Fail(void); typedef void mgt_cli_close_f(void *priv); void mgt_cli_setup(int fdi, int fdo, int verbose, const char *ident, mgt_cli_close_f *close_func, void *priv); -int mgt_cli_askchild(unsigned *status, char **resp, const char *fmt, ...); +int mgt_cli_askchild(unsigned *status, char **resp, const char *fmt, ...) + __printflike(3, 4); void mgt_cli_start_child(int fdi, int fdo); void mgt_cli_stop_child(void); void mgt_cli_telnet(const char *T_arg); diff --git a/bin/varnishd/mgt/mgt_param.c b/bin/varnishd/mgt/mgt_param.c index c488f23..8123bfc 100644 --- a/bin/varnishd/mgt/mgt_param.c +++ b/bin/varnishd/mgt/mgt_param.c @@ -261,7 +261,7 @@ tweak_generic_uint(struct cli *cli, volatile unsigned *dest, const char *arg, } *dest = u; } else if (*dest == UINT_MAX) { - VCLI_Out(cli, "unlimited", *dest); + VCLI_Out(cli, "unlimited"); } else { VCLI_Out(cli, "%u", *dest); } diff --git a/bin/varnishreplay/varnishreplay.c b/bin/varnishreplay/varnishreplay.c index 8e694da..cdc9cf7 100644 --- a/bin/varnishreplay/varnishreplay.c +++ b/bin/varnishreplay/varnishreplay.c @@ -44,6 +44,7 @@ #include #include +#include "printflike.h" #include "vapi/vsl.h" #include "vapi/vsm.h" #include "vas.h" @@ -171,6 +172,10 @@ static pthread_mutex_t log_mutex = PTHREAD_MUTEX_INITIALIZER; static void thread_log(int lvl, int errcode, const char *fmt, ...) + __printflike(3, 4); + +static void +thread_log(int lvl, int errcode, const char *fmt, ...) { va_list ap; diff --git a/bin/varnishtest/vtc.c b/bin/varnishtest/vtc.c index 9581b3c..b8556ce 100644 --- a/bin/varnishtest/vtc.c +++ b/bin/varnishtest/vtc.c @@ -449,7 +449,7 @@ cmd_random(CMD_ARGS) l = random(); if (l == random_expect[i]) continue; - vtc_log(vl, 4, "random[%d] = 0x%x (expect 0x%x)", + vtc_log(vl, 4, "random[%d] = 0x%x (expect 0x%lx)", i, l, random_expect[i]); vtc_log(vl, 1, "SKIPPING test: unknown srandom(1) sequence."); vtc_stop = 1; @@ -538,17 +538,17 @@ exec_file(const char *fn, const char *script, const char *tmpdir, /* Apply extmacro definitions */ VTAILQ_FOREACH(m, &extmacro_list, list) - macro_def(vltop, NULL, m->name, m->val); + macro_def(vltop, NULL, m->name, "%s", m->val); /* Other macro definitions */ cwd = getcwd(NULL, PATH_MAX); - macro_def(vltop, NULL, "pwd", cwd); + macro_def(vltop, NULL, "pwd", "%s", cwd); macro_def(vltop, NULL, "topbuild", "%s/%s", cwd, TOP_BUILDDIR); macro_def(vltop, NULL, "bad_ip", "10.255.255.255"); /* Move into our tmpdir */ AZ(chdir(tmpdir)); - macro_def(vltop, NULL, "tmpdir", tmpdir); + macro_def(vltop, NULL, "tmpdir", "%s", tmpdir); /* Drop file to tell what was going on here */ f = fopen("INFO", "w"); diff --git a/bin/varnishtest/vtc.h b/bin/varnishtest/vtc.h index ee5308a..be20cb9 100644 --- a/bin/varnishtest/vtc.h +++ b/bin/varnishtest/vtc.h @@ -77,7 +77,8 @@ void cmd_server_genvcl(struct vsb *vsb); void vtc_loginit(char *buf, unsigned buflen); struct vtclog *vtc_logopen(const char *id); void vtc_logclose(struct vtclog *vl); -void vtc_log(struct vtclog *vl, int lvl, const char *fmt, ...); +void vtc_log(struct vtclog *vl, int lvl, const char *fmt, ...) + __printflike(3, 4); void vtc_dump(struct vtclog *vl, int lvl, const char *pfx, const char *str, int len); void vtc_hexdump(struct vtclog *vl, int lvl, const char *pfx, @@ -87,7 +88,9 @@ int exec_file(const char *fn, const char *script, const char *tmpdir, char *logbuf, unsigned loglen); void macro_def(struct vtclog *vl, const char *instance, const char *name, - const char *fmt, ...); + const char *fmt, ...) + __printflike(4, 5); struct vsb *macro_expand(struct vtclog *vl, const char *text); -void extmacro_def(const char *name, const char *fmt, ...); +void extmacro_def(const char *name, const char *fmt, ...) + __printflike(2, 3); diff --git a/bin/varnishtest/vtc_http.c b/bin/varnishtest/vtc_http.c index d3753a5..00be0be 100644 --- a/bin/varnishtest/vtc_http.c +++ b/bin/varnishtest/vtc_http.c @@ -133,13 +133,13 @@ synth_body(const char *len, int rnd) static void http_write(const struct http *hp, int lvl, const char *pfx) { - int l; + ssize_t l; AZ(VSB_finish(hp->vsb)); vtc_dump(hp->vl, lvl, pfx, VSB_data(hp->vsb), VSB_len(hp->vsb)); l = write(hp->fd, VSB_data(hp->vsb), VSB_len(hp->vsb)); if (l != VSB_len(hp->vsb)) - vtc_log(hp->vl, hp->fatal, "Write failed: (%d vs %d) %s", + vtc_log(hp->vl, hp->fatal, "Write failed: (%zd vs %zd) %s", l, VSB_len(hp->vsb), strerror(errno)); } @@ -387,7 +387,7 @@ http_rxchunk(struct http *hp) bprintf(hp->chunklen, "%d", i); if ((q == hp->rxbuf + l) || (*q != '\0' && !vct_islws(*q))) { - vtc_log(hp->vl, hp->fatal, "chunked fail %02x @ %d", + vtc_log(hp->vl, hp->fatal, "chunked fail %02x @ %td", *q, q - (hp->rxbuf + l)); } assert(q != hp->rxbuf + l); diff --git a/bin/varnishtest/vtc_sema.c b/bin/varnishtest/vtc_sema.c index 48d4c68..fd4c561 100644 --- a/bin/varnishtest/vtc_sema.c +++ b/bin/varnishtest/vtc_sema.c @@ -63,7 +63,7 @@ sema_new(char *name, struct vtclog *vl) AN(r); r->name = name; if (*name != 'r') - vtc_log(vl, 0, "Sema name must start with 'r' (%s)", *name); + vtc_log(vl, 0, "Sema name must start with 'r' (%s)", name); AZ(pthread_mutex_init(&r->mtx, NULL)); AZ(pthread_cond_init(&r->cond, NULL)); diff --git a/include/printflike.h b/include/printflike.h new file mode 100644 index 0000000..ab76204 --- /dev/null +++ b/include/printflike.h @@ -0,0 +1,51 @@ +/*- + * Copyright (c) 2012 Fastly Inc + * All rights reserved. + * + * Author: Rogier 'DocWilco' Mulhuijzen + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer + * in this position and unchanged. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + */ + +#ifndef PRINTFLIKE_H_INCLUDED +#define PRINTFLIKE_H_INCLUDED + +#ifndef __GNUC_PREREQ +# if defined __GNUC__ && defined __GNUC_MINOR__ +# define __GNUC_PREREQ(maj, min) \ + (__GNUC__ > (maj) || (__GNUC__ == (maj) && __GNUC_MINOR__ >= (min))) +# else +# define __GNUC_PREREQ(maj, min) 0 +# endif +#endif + +#ifndef __printflike +# if __GNUC_PREREQ(2, 95) || defined(__INTEL_COMPILER) +# define __printflike(f,a) __attribute__((format(printf, f, a))) +# else +# define __printflike(f,a) +# endif +#endif + +#endif /* PRINTFLIKE_H_INCLUDED */ diff --git a/include/vcli_priv.h b/include/vcli_priv.h index a265d2e..ad4d343 100644 --- a/include/vcli_priv.h +++ b/include/vcli_priv.h @@ -31,6 +31,8 @@ * XXX: at a latter date we may want to move some to cli.h/libvarnishapi */ +#include "printflike.h" + #define CLI_PRIV_H struct cli; /* NB: struct cli is opaque at this level. */ @@ -53,6 +55,7 @@ struct cli_proto { /* The implementation must provide these functions */ int VCLI_Overflow(struct cli *cli); -void VCLI_Out(struct cli *cli, const char *fmt, ...); +void VCLI_Out(struct cli *cli, const char *fmt, ...) + __printflike(2, 3); void VCLI_Quote(struct cli *cli, const char *str); void VCLI_SetResult(struct cli *cli, unsigned r); diff --git a/include/vsb.h b/include/vsb.h index 0a8e2bf..af70def 100644 --- a/include/vsb.h +++ b/include/vsb.h @@ -31,6 +31,8 @@ #ifndef VSB_H_INCLUDED #define VSB_H_INCLUDED +#include "printflike.h" + /* * Structure definition */ @@ -49,10 +51,6 @@ struct vsb { int s_flags; /* flags */ }; -#ifndef __printflike -#define __printflike(a,b) -#endif - #ifdef __cplusplus extern "C" { #endif diff --git a/lib/libvarnishapi/vsm_api.h b/lib/libvarnishapi/vsm_api.h index 79be982..5698311 100644 --- a/lib/libvarnishapi/vsm_api.h +++ b/lib/libvarnishapi/vsm_api.h @@ -28,6 +28,8 @@ * */ +#include "printflike.h" + struct vsc; struct vsb; @@ -51,6 +53,7 @@ struct VSM_data { struct vsl *vsl; }; -int vsm_diag(struct VSM_data *vd, const char *fmt, ...); +int vsm_diag(struct VSM_data *vd, const char *fmt, ...) + __printflike(2, 3); void VSC_Delete(struct VSM_data *vd); void VSL_Delete(struct VSM_data *vd); diff --git a/lib/libvcl/vcc_acl.c b/lib/libvcl/vcc_acl.c index c962044..e6a1065 100644 --- a/lib/libvcl/vcc_acl.c +++ b/lib/libvcl/vcc_acl.c @@ -362,12 +362,12 @@ vcc_acl_emit(const struct vcc *tl, const char *acln, int anon) Fh(tl, 0, "\n"); Fh(tl, 0, "\ta = p;\n"); - Fh(tl, 0, "\tVRT_memmove(&fam, a + %d, sizeof fam);\n", + Fh(tl, 0, "\tVRT_memmove(&fam, a + %zd, sizeof fam);\n", offsetof(struct sockaddr, sa_family)); Fh(tl, 0, "\tif (fam == %d)\n", PF_INET); - Fh(tl, 0, "\t\ta += %d;\n", offsetof(struct sockaddr_in, sin_addr)); + Fh(tl, 0, "\t\ta += %zd;\n", offsetof(struct sockaddr_in, sin_addr)); Fh(tl, 0, "\telse if (fam == %d)\n", PF_INET6); - Fh(tl, 0, "\t\ta += %d;\n", offsetof(struct sockaddr_in6, sin6_addr)); + Fh(tl, 0, "\t\ta += %zd;\n", offsetof(struct sockaddr_in6, sin6_addr)); Fh(tl, 0, "\telse {\n"); Fh(tl, 0, "\t\tVRT_acl_log(sp, \"NO_FAM %s\");\n", acln); Fh(tl, 0, "\t\treturn(0);\n"); @@ -422,8 +422,7 @@ vcc_acl_emit(const struct vcc *tl, const char *acln, int anon) if (!anon) { Fh(tl, 0, "\t%*sVRT_acl_log(sp, \"%sMATCH %s \" ", - -i, "", ae->not ? "NEG_" : "", acln, - PF(ae->t_addr)); + -i, "", ae->not ? "NEG_" : "", acln); EncToken(tl->fh, ae->t_addr); if (ae->t_mask != NULL) Fh(tl, 0, " \"/%.*s\" ", PF(ae->t_mask)); diff --git a/lib/libvcl/vcc_backend.c b/lib/libvcl/vcc_backend.c index 03482c4..fb160e7 100644 --- a/lib/libvcl/vcc_backend.c +++ b/lib/libvcl/vcc_backend.c @@ -353,7 +353,7 @@ vcc_ParseProbeSpec(struct vcc *tl) if (t_initial != NULL) Fh(tl, 0, "\t.initial = %u,\n", initial); else - Fh(tl, 0, "\t.initial = ~0U,\n", initial); + Fh(tl, 0, "\t.initial = ~0U,\n"); if (status > 0) Fh(tl, 0, "\t.exp_status = %u,\n", status); Fh(tl, 0, "};\n"); diff --git a/lib/libvcl/vcc_compile.c b/lib/libvcl/vcc_compile.c index 27447fe..66d89f5 100644 --- a/lib/libvcl/vcc_compile.c +++ b/lib/libvcl/vcc_compile.c @@ -286,7 +286,7 @@ LocTable(const struct vcc *tl) pos++; } - Fc(tl, 0, " [%3u] = { %d, %8u, %4u, %3u, 0, ", + Fc(tl, 0, " [%3u] = { %d, %8tu, %4u, %3u, 0, ", t->cnt, sp->idx, t->b - sp->b, lin, pos + 1); if (t->tok == CSRC) Fc(tl, 0, " \"C{\"},\n"); diff --git a/lib/libvcl/vcc_compile.h b/lib/libvcl/vcc_compile.h index caacd73..b64564e 100644 --- a/lib/libvcl/vcc_compile.h +++ b/lib/libvcl/vcc_compile.h @@ -247,11 +247,16 @@ extern struct method method_tab[]; * I -> Initializer function * F -> Finish function */ -void Fh(const struct vcc *tl, int indent, const char *fmt, ...); -void Fc(const struct vcc *tl, int indent, const char *fmt, ...); -void Fb(const struct vcc *tl, int indent, const char *fmt, ...); -void Fi(const struct vcc *tl, int indent, const char *fmt, ...); -void Ff(const struct vcc *tl, int indent, const char *fmt, ...); +void Fh(const struct vcc *tl, int indent, const char *fmt, ...) + __printflike(3, 4); +void Fc(const struct vcc *tl, int indent, const char *fmt, ...) + __printflike(3, 4); +void Fb(const struct vcc *tl, int indent, const char *fmt, ...) + __printflike(3, 4); +void Fi(const struct vcc *tl, int indent, const char *fmt, ...) + __printflike(3, 4); +void Ff(const struct vcc *tl, int indent, const char *fmt, ...) + __printflike(3, 4); void EncToken(struct vsb *sb, const struct token *t); int IsMethod(const struct token *t); void *TlAlloc(struct vcc *tl, unsigned len); diff --git a/lib/libvcl/vcc_dir_dns.c b/lib/libvcl/vcc_dir_dns.c index 174d1ab..55ed921 100644 --- a/lib/libvcl/vcc_dir_dns.c +++ b/lib/libvcl/vcc_dir_dns.c @@ -92,9 +92,9 @@ print_backend(struct vcc *tl, Fb(tl, 0, "\t.hosthdr = \""); if (b_defaults.hostheader != NULL) - Fb(tl,0, b_defaults.hostheader); + Fb(tl, 0, "%s", b_defaults.hostheader); else - Fb(tl,0, strip); + Fb(tl, 0, "%s", strip); Fb(tl, 0, "\",\n"); Fb(tl, 0, "\t.saintmode_threshold = %d,\n",b_defaults.saint); diff --git a/lib/libvcl/vcc_expr.c b/lib/libvcl/vcc_expr.c index 3dfeeaa..82e53ef 100644 --- a/lib/libvcl/vcc_expr.c +++ b/lib/libvcl/vcc_expr.c @@ -253,6 +253,10 @@ vcc_new_expr(void) static struct expr * vcc_mk_expr(enum var_type fmt, const char *str, ...) + __printflike(2, 3); + +static struct expr * +vcc_mk_expr(enum var_type fmt, const char *str, ...) { va_list ap; struct expr *e; @@ -537,7 +541,7 @@ vcc_Eval_Func(struct vcc *tl, struct expr **e, const struct symbol *sym) r = strchr(sym->name, '.'); AN(r); e1 = vcc_mk_expr(VOID, "&vmod_priv_%.*s", - r - sym->name, sym->name); + (int) (r - sym->name), sym->name); p += strlen(p) + 1; } else if (fmt == VOID && !strcmp(p, "PRIV_CALL")) { bprintf(buf, "vmod_priv_%u", tl->nvmodpriv++); diff --git a/lib/libvcl/vcc_parse.c b/lib/libvcl/vcc_parse.c index 4023287..d8e74a4 100644 --- a/lib/libvcl/vcc_parse.c +++ b/lib/libvcl/vcc_parse.c @@ -154,7 +154,7 @@ vcc_Compound(struct vcc *tl) return; case CSRC: Fb(tl, 1, "%.*s\n", - tl->t->e - (tl->t->b + 2), + (int) (tl->t->e - (tl->t->b + 2)), tl->t->b + 1); vcc_NextToken(tl); break; @@ -274,7 +274,7 @@ vcc_Parse(struct vcc *tl) switch (tl->t->tok) { case CSRC: Fc(tl, 0, "%.*s\n", - tl->t->e - (tl->t->b + 4), tl->t->b + 2); + (int) (tl->t->e - (tl->t->b + 4)), tl->t->b + 2); vcc_NextToken(tl); break; case EOI: -- 1.7.5.4 From phk at phk.freebsd.dk Sat Jan 14 18:48:18 2012 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 14 Jan 2012 18:48:18 +0000 Subject: [PATCH] Add __printflike to all printflike functions and fix all that uncovers In-Reply-To: Your message of "Sat, 14 Jan 2012 18:49:55 +0100." <1326563395-8543-1-git-send-email-github@bsdchicks.com> Message-ID: <6846.1326566898@critter.freebsd.dk> In message <1326563395-8543-1-git-send-email-github at bsdchicks.com>, Rogier 'Doc Wilco' Mulhuijzen writes: >diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c >index 4967c82..1b62937 100644 >--- a/bin/varnishd/cache/cache_center.c >+++ b/bin/varnishd/cache/cache_center.c >@@ -1655,12 +1655,12 @@ cnt_diag(struct sess *sp, const char *state) > } > > if (sp->wrk != NULL) { >- WSP(sp, SLT_Debug, "thr %p STP_%s sp %p obj %p vcl %p", >+ WSP(sp, SLT_Debug, "thr %lx STP_%s sp %p obj %p vcl %p", > pthread_self(), state, sp, obj, vcl); Neither %p nor %lx are correct, in fact there is by standard no correct way to print a pthread_t (thanks POSIX, next time think, OK ?) We probably need to tag all our threads some way to solve this, or simply remove printing the pthread_self(). Since we have the sp->vsl_id I think that's the best compromise. >@@ -1776,7 +1776,7 @@ cli_debug_srandom(struct cli *cli, const char * const *av, void *priv) > seed = strtoul(av[2], NULL, 0); > srandom(seed); > srand48(random()); >- VCLI_Out(cli, "Random(3) seeded with %lu", seed); >+ VCLI_Out(cli, "Random(3) seeded with %u", seed); This is the wrong fix, both srandom and srand48 takes a (unsigned) long arg. > if (strcmp(x, nm)) { > VCLI_Out(cli, "Loading VMOD %s from %s:\n", nm, path); >- VCLI_Out(cli, "File contain wrong VMOD (\"%s\")\n", x); >+ VCLI_Out(cli, "File contain wrong VMOD (\"%s\")\n", (char *) x); Please split elongated lines lines properly at 80char boundary. >+#include "printflike.h" I've been thinking of introducing a "vdefs.h" for stuff like this, to contain compiler and C-dialect dependencies to a single file, we should use this chance to found that file. The #include hierarchy would be something like: include/vdefs.h included from: varnishd/common/common.h varnishstat/varnishstat.h varnishtest/vtc.h etc. So as to minimize the number of #include lines, given that we want total exposure for this file. We may want to move AZ(), AN() etc. over there subsequently. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From sky at crucially.net Sat Jan 14 18:54:08 2012 From: sky at crucially.net (Artur Bergman) Date: Sat, 14 Jan 2012 10:54:08 -0800 Subject: Streaming and backend conditional requests In-Reply-To: <5156.1326460696@critter.freebsd.dk> References: <5156.1326460696@critter.freebsd.dk> Message-ID: <5EBB67DC-B916-4BC2-9C9A-10205EDF713C@crucially.net> On Jan 13, 2012, at 5:18 AM, Poul-Henning Kamp wrote: > In vcl_fetch{} we add a "BOOL fetch_body()" function, so that you > can insist on pulling the entire object from the backend, before > accepting it into the cache, before deciding to deliver it to > the client, or for doing (vmod-) modifications to it before delivery. > > If you do not call fetch_body(), only the headers are available in > vcl_fetch{} and the body will be fetched asynchronously and delivered > to any client hitting this object in parallel. > > vcl_error{} becomes vcl_synth{}, and you call it like: > > return(synth(202)); > return(synth(578, "You got to be kidding")); > And the arguments go into resp.status and resp.response > > vcl_synth{} will only have access to req.* and resp.*, no obj.* or bere*.* > > Two issues need resolved: > Can you cache synth objects ? > Do they get delivered through vcl_deliver ? > > My initial idea was "no & no", but I am willing to be persuaded otherwise, > and Artur may be halfway there already. > > The argument for no caching is simply that Varnish is not a webserver > and I'd hate to see people try to turn it into one. > > The second is more involved: Synth replies will not have an obj.* > and therefore delivery is mechanically quite different from an obj.* > based delivery. Also, we won't be doing gunzip or ESI on synth > replies. Couple of points. I agree with all, except I think they should be delivered through vcl_deliver, or you have to duplicate header logic in both of them. Regarding ESI, as long as an ESI can have an fragment be a synth reply, I am happy. Artur From phk at phk.freebsd.dk Sat Jan 14 19:07:54 2012 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 14 Jan 2012 19:07:54 +0000 Subject: Streaming and backend conditional requests In-Reply-To: Your message of "Sat, 14 Jan 2012 10:54:08 PST." <5EBB67DC-B916-4BC2-9C9A-10205EDF713C@crucially.net> Message-ID: <7000.1326568074@critter.freebsd.dk> In message <5EBB67DC-B916-4BC2-9C9A-10205EDF713C at crucially.net>, Artur Bergman writes: >Couple of points. > >I agree with all, except I think they should be delivered through = >vcl_deliver, or you have to duplicate header logic in both of them. Presently we allow limited obj.* access in vcl_deliver{}, but that's actually limited to the obj.hits and obj.lastuse variables which are under dispute for causing write-backs to object storage. So if we eliminate those two, using vcl_deliver{} should be feasible. >Regarding ESI, as long as an ESI can have an fragment be a synth reply, = >I am happy. It can, but the synth reply cannot be ESI parsed. I did consider the alternative, where you call synth() from vcl_recv{} and then go from there to vcl_fetch{}, in essense making synth{} a virtual backend. Would that be better ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Sat Jan 14 19:10:41 2012 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 14 Jan 2012 19:10:41 +0000 Subject: Streaming and backend conditional requests In-Reply-To: Your message of "Sat, 14 Jan 2012 14:56:32 +0100." <4F118990.7020702@uplex.de> Message-ID: <7013.1326568241@critter.freebsd.dk> In message <4F118990.7020702 at uplex.de>, Geoff Simmons writes: >On 1/13/12 2:18 PM, Poul-Henning Kamp wrote: >> >> By default all backend fetches which can find a candidate for it, >> will try to IMS against the backend. If you do not want that, you >> remove the bereq.http.IMS header in vcl_pass{}/vcl_miss{} > >What got the conversation going about this is that in the current >implementation, a busy object can never be the candidate for a >validation. So in 3.0 streaming, it can never be an object that is >still streaming. If that's the right solution now and in the future, >then everything's fine. Actually once Martins streaming is in, we will have three object states: busy Not available for anybody but the fetching thread. incomplete We don't have the body yet, but can stream complete "unbusy" today. I will argue that only "complete" objects are IMS candidates, because they are the only ones we can serve, should we get an affirmative answer from the backend. I guess we could IMS "incomplete" also, but that sounds more complicated and I would need to study it more. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From sky at crucially.net Sat Jan 14 19:17:42 2012 From: sky at crucially.net (Artur Bergman) Date: Sat, 14 Jan 2012 11:17:42 -0800 Subject: Streaming and backend conditional requests In-Reply-To: <7000.1326568074@critter.freebsd.dk> References: <7000.1326568074@critter.freebsd.dk> Message-ID: <674C6610-199B-43B7-874E-7113361B4CA9@crucially.net> On Jan 14, 2012, at 11:07 AM, Poul-Henning Kamp wrote: >> Couple of points. >> >> I agree with all, except I think they should be delivered through = >> vcl_deliver, or you have to duplicate header logic in both of them. > > Presently we allow limited obj.* access in vcl_deliver{}, but that's > actually limited to the obj.hits and obj.lastuse variables which are > under dispute for causing write-backs to object storage. > I would love to keep the obj.hits and obj.lastuse in the struct obscure. > So if we eliminate those two, using vcl_deliver{} should be feasible. > >> Regarding ESI, as long as an ESI can have an fragment be a synth reply, = >> I am happy. > > It can, but the synth reply cannot be ESI parsed. > > I did consider the alternative, where you call synth() from vcl_recv{} > and then go from there to vcl_fetch{}, in essense making synth{} a > virtual backend. > > Would that be better ? What happens in an error and you want to go vcl_synth? Artur From phk at phk.freebsd.dk Sat Jan 14 19:27:21 2012 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 14 Jan 2012 19:27:21 +0000 Subject: Streaming and backend conditional requests In-Reply-To: Your message of "Sat, 14 Jan 2012 11:17:42 PST." <674C6610-199B-43B7-874E-7113361B4CA9@crucially.net> Message-ID: <7201.1326569241@critter.freebsd.dk> In message <674C6610-199B-43B7-874E-7113361B4CA9 at crucially.net>, Artur Bergman writes: >> I did consider the alternative, where you call synth() from vcl_recv{} >> and then go from there to vcl_fetch{}, in essense making synth{} a >> virtual backend. >> >> Would that be better ? > >What happens in an error and you want to go vcl_synth? You mean a 503 ? Same is in my proposal from yesterday: You get into vcl_fetch{} with a synthetic 503 response In some ways this way of doing things is cleaner, but I fear also more confusing. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From sky at crucially.net Sat Jan 14 19:50:53 2012 From: sky at crucially.net (Artur Bergman) Date: Sat, 14 Jan 2012 11:50:53 -0800 Subject: Streaming and backend conditional requests In-Reply-To: <7201.1326569241@critter.freebsd.dk> References: <7201.1326569241@critter.freebsd.dk> Message-ID: On Jan 14, 2012, at 11:27 AM, Poul-Henning Kamp wrote: > You mean a 503 ? Same is in my proposal from yesterday: You get > into vcl_fetch{} with a synthetic 503 response > > In some ways this way of doing things is cleaner, but I fear also > more confusing. No, I mean vcl_fetch returns something that you don't like and you want to error out. Artur From phk at phk.freebsd.dk Sat Jan 14 19:55:45 2012 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 14 Jan 2012 19:55:45 +0000 Subject: Streaming and backend conditional requests In-Reply-To: Your message of "Sat, 14 Jan 2012 11:50:53 PST." Message-ID: <7315.1326570945@critter.freebsd.dk> In message , Artur Bergman writes: >> You mean a 503 ? Same is in my proposal from yesterday: You get >> into vcl_fetch{} with a synthetic 503 response >> >> In some ways this way of doing things is cleaner, but I fear also >> more confusing. > >No, I mean vcl_fetch returns something that you don't like and you want >to error out. Yes, that's one of the reasons I found it confusing: You end up in vcl_fetch{} again, this time with the synth response. To avoid the obvious loop, we'd have to count that as a restart or something similar. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From cyberroadie at gmail.com Sat Jan 14 22:25:11 2012 From: cyberroadie at gmail.com (Olivier Van Acker) Date: Sat, 14 Jan 2012 22:25:11 +0000 Subject: new module In-Reply-To: References: Message-ID: Hi, I created a new varnish module, it orders the parameters in an url alphabetically Code can be found here: https://github.com/cyberroadie/varnish-urlsort And a blogpost describing it here: http://cyberroadie.wordpress.com/2012/01/05/varnish-reordering-query-string/ Is it possible to get it listed on the https://www.varnish-cache.org/project/modules page? Olivier -------------- next part -------------- An HTML attachment was scrubbed... URL: From ask at develooper.com Sat Jan 14 23:37:36 2012 From: ask at develooper.com (=?iso-8859-1?Q?Ask_Bj=F8rn_Hansen?=) Date: Sat, 14 Jan 2012 15:37:36 -0800 Subject: name changes for the upcoming 4.0 In-Reply-To: References: Message-ID: <4BD31D35-EA22-4858-9F81-B357646BBD11@develooper.com> On Dec 5, 2011, at 2:34, Per Buer wrote: > One forth and not related issue is "ban". We've already renamed this > once. Most people seem to get anchored to some perception that this > has to do with banning someone from getting content. My best > suggestion for a new name would be "purge-filter". Tollefs suggestion > is "invalidate" which might cover it better. Since you were bike-shedding, I'd add my 2 cents that I'd find "invalidate" much easier to understand than "ban". Invalidate clearly acts on "stuff from the past", where "ban" seems like it'd operate on data in the future. (Even if that's how it works, the effect of a ban is to invalidate old data). Ask From github at bsdchicks.com Sun Jan 15 16:39:54 2012 From: github at bsdchicks.com (Rogier 'DocWilco' Mulhuijzen) Date: Sun, 15 Jan 2012 17:39:54 +0100 Subject: [PATCH 1/2] Use sp->vsl_id as thread identifier In-Reply-To: <6846.1326566898@critter.freebsd.dk> References: <6846.1326566898@critter.freebsd.dk> Message-ID: <1326645595-5585-1-git-send-email-github@bsdchicks.com> POSIX threads don't have a thread identifier that can be printed in a portable manner. --- bin/varnishd/cache/cache_center.c | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c index 4967c82..a9d791e 100644 --- a/bin/varnishd/cache/cache_center.c +++ b/bin/varnishd/cache/cache_center.c @@ -1655,13 +1655,13 @@ cnt_diag(struct sess *sp, const char *state) } if (sp->wrk != NULL) { - WSP(sp, SLT_Debug, "thr %p STP_%s sp %p obj %p vcl %p", - pthread_self(), state, sp, obj, vcl); + WSP(sp, SLT_Debug, "thr %x STP_%s sp %p obj %p vcl %p", + sp->vsl_id, state, sp, obj, vcl); WSL_Flush(sp->wrk, 0); } else { VSL(SLT_Debug, sp->vsl_id, - "thr %p STP_%s sp %p obj %p vcl %p", - pthread_self(), state, sp, obj, vcl); + "thr %x STP_%s sp %p obj %p vcl %p", + sp->vsl_id, state, sp, obj, vcl); } } -- 1.7.5.4 From github at bsdchicks.com Sun Jan 15 16:39:55 2012 From: github at bsdchicks.com (Rogier 'DocWilco' Mulhuijzen) Date: Sun, 15 Jan 2012 17:39:55 +0100 Subject: [PATCH 2/2] Add __printflike and fix everything that brings to light In-Reply-To: <1326645595-5585-1-git-send-email-github@bsdchicks.com> References: <6846.1326566898@critter.freebsd.dk> <1326645595-5585-1-git-send-email-github@bsdchicks.com> Message-ID: <1326645595-5585-2-git-send-email-github@bsdchicks.com> --- bin/varnishd/cache/cache.h | 12 ++++++++---- bin/varnishd/cache/cache_center.c | 2 +- bin/varnishd/cache/cache_shmlog.c | 4 ++++ bin/varnishd/cache/cache_vrt_vmod.c | 3 ++- bin/varnishd/mgt/mgt.h | 3 ++- bin/varnishd/mgt/mgt_param.c | 2 +- bin/varnishreplay/varnishreplay.c | 5 +++++ bin/varnishtest/vtc.c | 8 ++++---- bin/varnishtest/vtc.h | 9 ++++++--- bin/varnishtest/vtc_http.c | 6 +++--- bin/varnishtest/vtc_sema.c | 2 +- include/vcli_common.h | 2 ++ include/vcli_priv.h | 3 ++- include/vdef.h | 28 +++++++++++++++++++++++++++- include/vsb.h | 6 ++---- lib/libvarnishapi/vsc.c | 1 + lib/libvarnishapi/vsl.c | 1 + lib/libvarnishapi/vsl_arg.c | 1 + lib/libvarnishapi/vsm.c | 1 + lib/libvarnishapi/vsm_api.h | 3 ++- lib/libvcl/vcc_acl.c | 9 ++++----- lib/libvcl/vcc_backend.c | 2 +- lib/libvcl/vcc_compile.c | 2 +- lib/libvcl/vcc_compile.h | 15 ++++++++++----- lib/libvcl/vcc_dir_dns.c | 4 ++-- lib/libvcl/vcc_expr.c | 6 +++++- lib/libvcl/vcc_parse.c | 4 ++-- 27 files changed, 101 insertions(+), 43 deletions(-) diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 6b0d0a9..356d138 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -809,7 +809,8 @@ void http_PutStatus(struct http *to, uint16_t status); void http_PutResponse(struct worker *w, unsigned vsl_id, const struct http *to, const char *response); void http_PrintfHeader(struct worker *w, unsigned vsl_id, struct http *to, - const char *fmt, ...); + const char *fmt, ...) + __printflike(4, 5); void http_SetHeader(struct worker *w, unsigned vsl_id, struct http *to, const char *hdr); void http_SetH(const struct http *to, unsigned n, const char *fm); @@ -928,10 +929,13 @@ void *VSM_Alloc(unsigned size, const char *class, const char *type, const char *ident); void VSM_Free(void *ptr); #ifdef VSL_ENDMARKER -void VSL(enum VSL_tag_e tag, int id, const char *fmt, ...); +void VSL(enum VSL_tag_e tag, int id, const char *fmt, ...) + __printflike(3, 4); void WSLR(struct worker *w, enum VSL_tag_e tag, int id, txt t); -void WSL(struct worker *w, enum VSL_tag_e tag, int id, const char *fmt, ...); -void WSLB(struct worker *w, enum VSL_tag_e tag, const char *fmt, ...); +void WSL(struct worker *w, enum VSL_tag_e tag, int id, const char *fmt, ...) + __printflike(4, 5); +void WSLB(struct worker *w, enum VSL_tag_e tag, const char *fmt, ...) + __printflike(3, 4); void WSL_Flush(struct worker *w, int overflow); diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c index a9d791e..7c8cea4 100644 --- a/bin/varnishd/cache/cache_center.c +++ b/bin/varnishd/cache/cache_center.c @@ -1776,7 +1776,7 @@ cli_debug_srandom(struct cli *cli, const char * const *av, void *priv) seed = strtoul(av[2], NULL, 0); srandom(seed); srand48(random()); - VCLI_Out(cli, "Random(3) seeded with %lu", seed); + VCLI_Out(cli, "Random(3) seeded with %u", seed); } static struct cli_proto debug_cmds[] = { diff --git a/bin/varnishd/cache/cache_shmlog.c b/bin/varnishd/cache/cache_shmlog.c index c29f5a1..a839f5a 100644 --- a/bin/varnishd/cache/cache_shmlog.c +++ b/bin/varnishd/cache/cache_shmlog.c @@ -236,6 +236,10 @@ WSLR(struct worker *wrk, enum VSL_tag_e tag, int id, txt t) static void wsl(struct worker *wrk, enum VSL_tag_e tag, int id, const char *fmt, va_list ap) + __printflike(4, 0); + +static void +wsl(struct worker *wrk, enum VSL_tag_e tag, int id, const char *fmt, va_list ap) { char *p; unsigned n, mlen; diff --git a/bin/varnishd/cache/cache_vrt_vmod.c b/bin/varnishd/cache/cache_vrt_vmod.c index 6b3b846..93d82f9 100644 --- a/bin/varnishd/cache/cache_vrt_vmod.c +++ b/bin/varnishd/cache/cache_vrt_vmod.c @@ -104,7 +104,8 @@ VRT_Vmod_Init(void **hdl, void *ptr, int len, const char *nm, AN(w); if (strcmp(x, nm)) { VCLI_Out(cli, "Loading VMOD %s from %s:\n", nm, path); - VCLI_Out(cli, "File contain wrong VMOD (\"%s\")\n", x); + VCLI_Out(cli, "File contain wrong VMOD (\"%s\")\n", + (char *) x); VCLI_Out(cli, "Check relative pathnames ?.\n"); (void)dlclose(v->hdl); FREE_OBJ(v); diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h index 0e3eb9e..ffaef56 100644 --- a/bin/varnishd/mgt/mgt.h +++ b/bin/varnishd/mgt/mgt.h @@ -50,7 +50,8 @@ void MGT_Child_Cli_Fail(void); typedef void mgt_cli_close_f(void *priv); void mgt_cli_setup(int fdi, int fdo, int verbose, const char *ident, mgt_cli_close_f *close_func, void *priv); -int mgt_cli_askchild(unsigned *status, char **resp, const char *fmt, ...); +int mgt_cli_askchild(unsigned *status, char **resp, const char *fmt, ...) + __printflike(3, 4); void mgt_cli_start_child(int fdi, int fdo); void mgt_cli_stop_child(void); void mgt_cli_telnet(const char *T_arg); diff --git a/bin/varnishd/mgt/mgt_param.c b/bin/varnishd/mgt/mgt_param.c index c488f23..8123bfc 100644 --- a/bin/varnishd/mgt/mgt_param.c +++ b/bin/varnishd/mgt/mgt_param.c @@ -261,7 +261,7 @@ tweak_generic_uint(struct cli *cli, volatile unsigned *dest, const char *arg, } *dest = u; } else if (*dest == UINT_MAX) { - VCLI_Out(cli, "unlimited", *dest); + VCLI_Out(cli, "unlimited"); } else { VCLI_Out(cli, "%u", *dest); } diff --git a/bin/varnishreplay/varnishreplay.c b/bin/varnishreplay/varnishreplay.c index 8e694da..02ad83e 100644 --- a/bin/varnishreplay/varnishreplay.c +++ b/bin/varnishreplay/varnishreplay.c @@ -44,6 +44,7 @@ #include #include +#include "vdef.h" #include "vapi/vsl.h" #include "vapi/vsm.h" #include "vas.h" @@ -171,6 +172,10 @@ static pthread_mutex_t log_mutex = PTHREAD_MUTEX_INITIALIZER; static void thread_log(int lvl, int errcode, const char *fmt, ...) + __printflike(3, 4); + +static void +thread_log(int lvl, int errcode, const char *fmt, ...) { va_list ap; diff --git a/bin/varnishtest/vtc.c b/bin/varnishtest/vtc.c index 9581b3c..b8556ce 100644 --- a/bin/varnishtest/vtc.c +++ b/bin/varnishtest/vtc.c @@ -449,7 +449,7 @@ cmd_random(CMD_ARGS) l = random(); if (l == random_expect[i]) continue; - vtc_log(vl, 4, "random[%d] = 0x%x (expect 0x%x)", + vtc_log(vl, 4, "random[%d] = 0x%x (expect 0x%lx)", i, l, random_expect[i]); vtc_log(vl, 1, "SKIPPING test: unknown srandom(1) sequence."); vtc_stop = 1; @@ -538,17 +538,17 @@ exec_file(const char *fn, const char *script, const char *tmpdir, /* Apply extmacro definitions */ VTAILQ_FOREACH(m, &extmacro_list, list) - macro_def(vltop, NULL, m->name, m->val); + macro_def(vltop, NULL, m->name, "%s", m->val); /* Other macro definitions */ cwd = getcwd(NULL, PATH_MAX); - macro_def(vltop, NULL, "pwd", cwd); + macro_def(vltop, NULL, "pwd", "%s", cwd); macro_def(vltop, NULL, "topbuild", "%s/%s", cwd, TOP_BUILDDIR); macro_def(vltop, NULL, "bad_ip", "10.255.255.255"); /* Move into our tmpdir */ AZ(chdir(tmpdir)); - macro_def(vltop, NULL, "tmpdir", tmpdir); + macro_def(vltop, NULL, "tmpdir", "%s", tmpdir); /* Drop file to tell what was going on here */ f = fopen("INFO", "w"); diff --git a/bin/varnishtest/vtc.h b/bin/varnishtest/vtc.h index ee5308a..be20cb9 100644 --- a/bin/varnishtest/vtc.h +++ b/bin/varnishtest/vtc.h @@ -77,7 +77,8 @@ void cmd_server_genvcl(struct vsb *vsb); void vtc_loginit(char *buf, unsigned buflen); struct vtclog *vtc_logopen(const char *id); void vtc_logclose(struct vtclog *vl); -void vtc_log(struct vtclog *vl, int lvl, const char *fmt, ...); +void vtc_log(struct vtclog *vl, int lvl, const char *fmt, ...) + __printflike(3, 4); void vtc_dump(struct vtclog *vl, int lvl, const char *pfx, const char *str, int len); void vtc_hexdump(struct vtclog *vl, int lvl, const char *pfx, @@ -87,7 +88,9 @@ int exec_file(const char *fn, const char *script, const char *tmpdir, char *logbuf, unsigned loglen); void macro_def(struct vtclog *vl, const char *instance, const char *name, - const char *fmt, ...); + const char *fmt, ...) + __printflike(4, 5); struct vsb *macro_expand(struct vtclog *vl, const char *text); -void extmacro_def(const char *name, const char *fmt, ...); +void extmacro_def(const char *name, const char *fmt, ...) + __printflike(2, 3); diff --git a/bin/varnishtest/vtc_http.c b/bin/varnishtest/vtc_http.c index d3753a5..00be0be 100644 --- a/bin/varnishtest/vtc_http.c +++ b/bin/varnishtest/vtc_http.c @@ -133,13 +133,13 @@ synth_body(const char *len, int rnd) static void http_write(const struct http *hp, int lvl, const char *pfx) { - int l; + ssize_t l; AZ(VSB_finish(hp->vsb)); vtc_dump(hp->vl, lvl, pfx, VSB_data(hp->vsb), VSB_len(hp->vsb)); l = write(hp->fd, VSB_data(hp->vsb), VSB_len(hp->vsb)); if (l != VSB_len(hp->vsb)) - vtc_log(hp->vl, hp->fatal, "Write failed: (%d vs %d) %s", + vtc_log(hp->vl, hp->fatal, "Write failed: (%zd vs %zd) %s", l, VSB_len(hp->vsb), strerror(errno)); } @@ -387,7 +387,7 @@ http_rxchunk(struct http *hp) bprintf(hp->chunklen, "%d", i); if ((q == hp->rxbuf + l) || (*q != '\0' && !vct_islws(*q))) { - vtc_log(hp->vl, hp->fatal, "chunked fail %02x @ %d", + vtc_log(hp->vl, hp->fatal, "chunked fail %02x @ %td", *q, q - (hp->rxbuf + l)); } assert(q != hp->rxbuf + l); diff --git a/bin/varnishtest/vtc_sema.c b/bin/varnishtest/vtc_sema.c index 48d4c68..fd4c561 100644 --- a/bin/varnishtest/vtc_sema.c +++ b/bin/varnishtest/vtc_sema.c @@ -63,7 +63,7 @@ sema_new(char *name, struct vtclog *vl) AN(r); r->name = name; if (*name != 'r') - vtc_log(vl, 0, "Sema name must start with 'r' (%s)", *name); + vtc_log(vl, 0, "Sema name must start with 'r' (%s)", name); AZ(pthread_mutex_init(&r->mtx, NULL)); AZ(pthread_cond_init(&r->cond, NULL)); diff --git a/include/vcli_common.h b/include/vcli_common.h index 8d0a623..fbb4e9f 100644 --- a/include/vcli_common.h +++ b/include/vcli_common.h @@ -28,6 +28,8 @@ * */ +#include "vdef.h" + struct vlu; struct VCLS; diff --git a/include/vcli_priv.h b/include/vcli_priv.h index a265d2e..0129b83 100644 --- a/include/vcli_priv.h +++ b/include/vcli_priv.h @@ -53,6 +53,7 @@ struct cli_proto { /* The implementation must provide these functions */ int VCLI_Overflow(struct cli *cli); -void VCLI_Out(struct cli *cli, const char *fmt, ...); +void VCLI_Out(struct cli *cli, const char *fmt, ...) + __printflike(2, 3); void VCLI_Quote(struct cli *cli, const char *str); void VCLI_SetResult(struct cli *cli, unsigned r); diff --git a/include/vdef.h b/include/vdef.h index c9b24c5..63a2a6b 100644 --- a/include/vdef.h +++ b/include/vdef.h @@ -1,10 +1,14 @@ /*- * Copyright (c) 2006 Verdens Gang AS - * Copyright (c) 2006-2011 Varnish Software AS + * Copyright (c) 2006-2012 Varnish Software AS + * Copyright (c) 2012 Fastly Inc * All rights reserved. * * Author: Poul-Henning Kamp + * Author: Rogier 'DocWilco' Mulhuijzen * + * Inspired by FreeBSD's + * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: @@ -28,6 +32,9 @@ * */ +#ifndef VDEF_H_INCLUDED +#define VDEF_H_INCLUDED + /* Safe printf into a fixed-size buffer */ #define bprintf(buf, fmt, ...) \ do { \ @@ -42,3 +49,22 @@ < sizeof buf); \ } while (0) + +#ifndef __GNUC_PREREQ +# if defined __GNUC__ && defined __GNUC_MINOR__ +# define __GNUC_PREREQ(maj, min) \ + (__GNUC__ > (maj) || (__GNUC__ == (maj) && __GNUC_MINOR__ >= (min))) +# else +# define __GNUC_PREREQ(maj, min) 0 +# endif +#endif + +#ifndef __printflike +# if __GNUC_PREREQ(2, 95) || defined(__INTEL_COMPILER) +# define __printflike(f,a) __attribute__((format(printf, f, a))) +# else +# define __printflike(f,a) +# endif +#endif + +#endif /* VDEF_H_INCLUDED */ diff --git a/include/vsb.h b/include/vsb.h index 0a8e2bf..e17fe5b 100644 --- a/include/vsb.h +++ b/include/vsb.h @@ -31,6 +31,8 @@ #ifndef VSB_H_INCLUDED #define VSB_H_INCLUDED +#include "vdef.h" + /* * Structure definition */ @@ -49,10 +51,6 @@ struct vsb { int s_flags; /* flags */ }; -#ifndef __printflike -#define __printflike(a,b) -#endif - #ifdef __cplusplus extern "C" { #endif diff --git a/lib/libvarnishapi/vsc.c b/lib/libvarnishapi/vsc.c index accef82..5f992cf 100644 --- a/lib/libvarnishapi/vsc.c +++ b/lib/libvarnishapi/vsc.c @@ -39,6 +39,7 @@ #include "miniobj.h" #include "vas.h" +#include "vdef.h" #include "vapi/vsc.h" #include "vapi/vsm.h" diff --git a/lib/libvarnishapi/vsl.c b/lib/libvarnishapi/vsl.c index 6950c1b..8e7001a 100644 --- a/lib/libvarnishapi/vsl.c +++ b/lib/libvarnishapi/vsl.c @@ -41,6 +41,7 @@ #include "miniobj.h" #include "vas.h" +#include "vdef.h" #include "vapi/vsl.h" #include "vapi/vsm.h" diff --git a/lib/libvarnishapi/vsl_arg.c b/lib/libvarnishapi/vsl_arg.c index 068d539..8587fb1 100644 --- a/lib/libvarnishapi/vsl_arg.c +++ b/lib/libvarnishapi/vsl_arg.c @@ -43,6 +43,7 @@ #include "miniobj.h" #include "vas.h" +#include "vdef.h" #include "vapi/vsl.h" #include "vapi/vsm.h" diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c index e11fe26..c175f10 100644 --- a/lib/libvarnishapi/vsm.c +++ b/lib/libvarnishapi/vsm.c @@ -44,6 +44,7 @@ #include "miniobj.h" #include "vas.h" +#include "vdef.h" #include "vapi/vsm.h" #include "vapi/vsm_int.h" diff --git a/lib/libvarnishapi/vsm_api.h b/lib/libvarnishapi/vsm_api.h index 79be982..af99798 100644 --- a/lib/libvarnishapi/vsm_api.h +++ b/lib/libvarnishapi/vsm_api.h @@ -51,6 +51,7 @@ struct VSM_data { struct vsl *vsl; }; -int vsm_diag(struct VSM_data *vd, const char *fmt, ...); +int vsm_diag(struct VSM_data *vd, const char *fmt, ...) + __printflike(2, 3); void VSC_Delete(struct VSM_data *vd); void VSL_Delete(struct VSM_data *vd); diff --git a/lib/libvcl/vcc_acl.c b/lib/libvcl/vcc_acl.c index c962044..e6a1065 100644 --- a/lib/libvcl/vcc_acl.c +++ b/lib/libvcl/vcc_acl.c @@ -362,12 +362,12 @@ vcc_acl_emit(const struct vcc *tl, const char *acln, int anon) Fh(tl, 0, "\n"); Fh(tl, 0, "\ta = p;\n"); - Fh(tl, 0, "\tVRT_memmove(&fam, a + %d, sizeof fam);\n", + Fh(tl, 0, "\tVRT_memmove(&fam, a + %zd, sizeof fam);\n", offsetof(struct sockaddr, sa_family)); Fh(tl, 0, "\tif (fam == %d)\n", PF_INET); - Fh(tl, 0, "\t\ta += %d;\n", offsetof(struct sockaddr_in, sin_addr)); + Fh(tl, 0, "\t\ta += %zd;\n", offsetof(struct sockaddr_in, sin_addr)); Fh(tl, 0, "\telse if (fam == %d)\n", PF_INET6); - Fh(tl, 0, "\t\ta += %d;\n", offsetof(struct sockaddr_in6, sin6_addr)); + Fh(tl, 0, "\t\ta += %zd;\n", offsetof(struct sockaddr_in6, sin6_addr)); Fh(tl, 0, "\telse {\n"); Fh(tl, 0, "\t\tVRT_acl_log(sp, \"NO_FAM %s\");\n", acln); Fh(tl, 0, "\t\treturn(0);\n"); @@ -422,8 +422,7 @@ vcc_acl_emit(const struct vcc *tl, const char *acln, int anon) if (!anon) { Fh(tl, 0, "\t%*sVRT_acl_log(sp, \"%sMATCH %s \" ", - -i, "", ae->not ? "NEG_" : "", acln, - PF(ae->t_addr)); + -i, "", ae->not ? "NEG_" : "", acln); EncToken(tl->fh, ae->t_addr); if (ae->t_mask != NULL) Fh(tl, 0, " \"/%.*s\" ", PF(ae->t_mask)); diff --git a/lib/libvcl/vcc_backend.c b/lib/libvcl/vcc_backend.c index 03482c4..fb160e7 100644 --- a/lib/libvcl/vcc_backend.c +++ b/lib/libvcl/vcc_backend.c @@ -353,7 +353,7 @@ vcc_ParseProbeSpec(struct vcc *tl) if (t_initial != NULL) Fh(tl, 0, "\t.initial = %u,\n", initial); else - Fh(tl, 0, "\t.initial = ~0U,\n", initial); + Fh(tl, 0, "\t.initial = ~0U,\n"); if (status > 0) Fh(tl, 0, "\t.exp_status = %u,\n", status); Fh(tl, 0, "};\n"); diff --git a/lib/libvcl/vcc_compile.c b/lib/libvcl/vcc_compile.c index 27447fe..66d89f5 100644 --- a/lib/libvcl/vcc_compile.c +++ b/lib/libvcl/vcc_compile.c @@ -286,7 +286,7 @@ LocTable(const struct vcc *tl) pos++; } - Fc(tl, 0, " [%3u] = { %d, %8u, %4u, %3u, 0, ", + Fc(tl, 0, " [%3u] = { %d, %8tu, %4u, %3u, 0, ", t->cnt, sp->idx, t->b - sp->b, lin, pos + 1); if (t->tok == CSRC) Fc(tl, 0, " \"C{\"},\n"); diff --git a/lib/libvcl/vcc_compile.h b/lib/libvcl/vcc_compile.h index caacd73..b64564e 100644 --- a/lib/libvcl/vcc_compile.h +++ b/lib/libvcl/vcc_compile.h @@ -247,11 +247,16 @@ extern struct method method_tab[]; * I -> Initializer function * F -> Finish function */ -void Fh(const struct vcc *tl, int indent, const char *fmt, ...); -void Fc(const struct vcc *tl, int indent, const char *fmt, ...); -void Fb(const struct vcc *tl, int indent, const char *fmt, ...); -void Fi(const struct vcc *tl, int indent, const char *fmt, ...); -void Ff(const struct vcc *tl, int indent, const char *fmt, ...); +void Fh(const struct vcc *tl, int indent, const char *fmt, ...) + __printflike(3, 4); +void Fc(const struct vcc *tl, int indent, const char *fmt, ...) + __printflike(3, 4); +void Fb(const struct vcc *tl, int indent, const char *fmt, ...) + __printflike(3, 4); +void Fi(const struct vcc *tl, int indent, const char *fmt, ...) + __printflike(3, 4); +void Ff(const struct vcc *tl, int indent, const char *fmt, ...) + __printflike(3, 4); void EncToken(struct vsb *sb, const struct token *t); int IsMethod(const struct token *t); void *TlAlloc(struct vcc *tl, unsigned len); diff --git a/lib/libvcl/vcc_dir_dns.c b/lib/libvcl/vcc_dir_dns.c index 174d1ab..55ed921 100644 --- a/lib/libvcl/vcc_dir_dns.c +++ b/lib/libvcl/vcc_dir_dns.c @@ -92,9 +92,9 @@ print_backend(struct vcc *tl, Fb(tl, 0, "\t.hosthdr = \""); if (b_defaults.hostheader != NULL) - Fb(tl,0, b_defaults.hostheader); + Fb(tl, 0, "%s", b_defaults.hostheader); else - Fb(tl,0, strip); + Fb(tl, 0, "%s", strip); Fb(tl, 0, "\",\n"); Fb(tl, 0, "\t.saintmode_threshold = %d,\n",b_defaults.saint); diff --git a/lib/libvcl/vcc_expr.c b/lib/libvcl/vcc_expr.c index 3dfeeaa..82e53ef 100644 --- a/lib/libvcl/vcc_expr.c +++ b/lib/libvcl/vcc_expr.c @@ -253,6 +253,10 @@ vcc_new_expr(void) static struct expr * vcc_mk_expr(enum var_type fmt, const char *str, ...) + __printflike(2, 3); + +static struct expr * +vcc_mk_expr(enum var_type fmt, const char *str, ...) { va_list ap; struct expr *e; @@ -537,7 +541,7 @@ vcc_Eval_Func(struct vcc *tl, struct expr **e, const struct symbol *sym) r = strchr(sym->name, '.'); AN(r); e1 = vcc_mk_expr(VOID, "&vmod_priv_%.*s", - r - sym->name, sym->name); + (int) (r - sym->name), sym->name); p += strlen(p) + 1; } else if (fmt == VOID && !strcmp(p, "PRIV_CALL")) { bprintf(buf, "vmod_priv_%u", tl->nvmodpriv++); diff --git a/lib/libvcl/vcc_parse.c b/lib/libvcl/vcc_parse.c index 4023287..d8e74a4 100644 --- a/lib/libvcl/vcc_parse.c +++ b/lib/libvcl/vcc_parse.c @@ -154,7 +154,7 @@ vcc_Compound(struct vcc *tl) return; case CSRC: Fb(tl, 1, "%.*s\n", - tl->t->e - (tl->t->b + 2), + (int) (tl->t->e - (tl->t->b + 2)), tl->t->b + 1); vcc_NextToken(tl); break; @@ -274,7 +274,7 @@ vcc_Parse(struct vcc *tl) switch (tl->t->tok) { case CSRC: Fc(tl, 0, "%.*s\n", - tl->t->e - (tl->t->b + 4), tl->t->b + 2); + (int) (tl->t->e - (tl->t->b + 4)), tl->t->b + 2); vcc_NextToken(tl); break; case EOI: -- 1.7.5.4 From phk at phk.freebsd.dk Sun Jan 15 17:15:16 2012 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sun, 15 Jan 2012 17:15:16 +0000 Subject: name changes for the upcoming 4.0 In-Reply-To: Your message of "Sat, 14 Jan 2012 15:37:36 PST." <4BD31D35-EA22-4858-9F81-B357646BBD11@develooper.com> Message-ID: <43361.1326647716@critter.freebsd.dk> In message <4BD31D35-EA22-4858-9F81-B357646BBD11 at develooper.com>, =?iso-8859-1? Q?Ask_Bj=F8rn_Hansen?= writes: >Since you were bike-shedding, I'd add my 2 cents that I'd find >"invalidate" much easier to understand than "ban". Invalidate >clearly acts on "stuff from the past", where "ban" seems like it'd >operate on data in the future. (Even if that's how it works, the >effect of a ban is to invalidate old data). The problem is that a "ban" does operate in the future: It bans (matching) cached objects from being served. "invalidate" implies immediate action, and the entire point is to avoid just that. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From brad at comstyle.com Sun Jan 15 22:11:02 2012 From: brad at comstyle.com (Brad) Date: Sun, 15 Jan 2012 17:11:02 -0500 Subject: [PATCH] Fix libedit detection on *BSD OS's In-Reply-To: <20120102190345.GA24654@rox.home.comstyle.com> References: <20120102190345.GA24654@rox.home.comstyle.com> Message-ID: <4F134EF6.50104@comstyle.com> On 02/01/12 2:03 PM, Brad wrote: > The following patch allows the autoconf script to detect the presence of > libedit when there isn't a pkg-config file present and thus allowing > Varnish to detect libedit on OpenBSD/FreeBSD/NetBSD/DragonFly. ping. -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From github at bsdchicks.com Sun Jan 15 22:39:15 2012 From: github at bsdchicks.com (Rogier 'DocWilco' Mulhuijzen) Date: Sun, 15 Jan 2012 23:39:15 +0100 Subject: [PATCH] Simplify vry_cmp's signature Message-ID: <1326667155-9543-1-git-send-email-github@bsdchicks.com> --- bin/varnishd/cache/cache_vary.c | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-) diff --git a/bin/varnishd/cache/cache_vary.c b/bin/varnishd/cache/cache_vary.c index 453b52d..9ad41d0 100644 --- a/bin/varnishd/cache/cache_vary.c +++ b/bin/varnishd/cache/cache_vary.c @@ -145,18 +145,18 @@ vry_len(const uint8_t *p) * Compare two vary entries */ static int -vry_cmp(const uint8_t * const *v1, uint8_t * const *v2) +vry_cmp(const uint8_t *v1, const uint8_t *v2) { unsigned retval = 0; - if (!memcmp(*v1, *v2, vry_len(*v1))) { + if (!memcmp(v1, v2, vry_len(v1))) { /* Same same */ retval = 0; - } else if (memcmp((*v1) + 2, (*v2) + 2, (*v1)[2] + 2)) { + } else if (memcmp(v1 + 2, v2 + 2, v1[2] + 2)) { /* Different header */ retval = 1; } else if (cache_param->http_gzip_support && - !strcasecmp(H_Accept_Encoding, (const char*)((*v1)+2))) { + !strcasecmp(H_Accept_Encoding, (const char*) v1 + 2)) { /* * If we do gzip processing, we do not vary on Accept-Encoding, * because we want everybody to get the gzip'ed object, and @@ -183,7 +183,7 @@ VRY_Match(const struct sess *sp, const uint8_t *vary) AN(vsp); while (vary[2]) { - i = vry_cmp(&vary, &vsp); + i = vry_cmp(vary, vsp); if (i == 1) { /* Build a new entry */ @@ -222,7 +222,7 @@ VRY_Match(const struct sess *sp, const uint8_t *vary) } else vsp[2 + vary[2] + 2 + 2] = '\0'; - i = vry_cmp(&vary, &vsp); + i = vry_cmp(vary, vsp); assert(i != 1); /* hdr must be the same now */ } if (i != 0) -- 1.7.5.4 From github at bsdchicks.com Sun Jan 15 23:08:24 2012 From: github at bsdchicks.com (Rogier 'DocWilco' Mulhuijzen) Date: Mon, 16 Jan 2012 00:08:24 +0100 Subject: [PATCH] Drop old grace objects when we have a new object that matches the Vary In-Reply-To: <1325865286-13513-1-git-send-email-github@bsdchicks.com> References: <1325865286-13513-1-git-send-email-github@bsdchicks.com> Message-ID: <1326668904-24963-1-git-send-email-github@bsdchicks.com> (Now with yummy VRY_Compare!) Also add EXP_Remove, to allow removing objects with touching them. This patch makes it possible to have a very short TTL, but a long grace. Normally, if you were to set a 1s TTL and a 86400s grace, you would end up with 86400 objects on a single objecthead, not even taking into account Vary. With this patch, any object found in grace is kept with the request, and upon succesful retrieval of a new object from the backend, the grace object is removed. EXP_Remove was added so that the TTL and grace on the object aren't touched. This, in combination with some other work we've done, prevents pages from getting dirtied and thus having to be written out to disk in a environment where the cache cannot be purely kept in memory. VRY_Compare compares 2 vary strings for equality (taking gzip into account) and was added so that the grace object is only dropped if its vary matches to the vary on the new object. --- bin/varnishd/cache/cache.h | 3 + bin/varnishd/cache/cache_center.c | 4 +- bin/varnishd/cache/cache_expire.c | 43 ++++++++++++ bin/varnishd/cache/cache_hash.c | 52 +++++++++++---- bin/varnishd/cache/cache_vary.c | 23 +++++++ bin/varnishd/hash/hash_slinger.h | 2 +- bin/varnishtest/tests/grace-drop.vtc | 122 ++++++++++++++++++++++++++++++++++ include/tbl/vsc_f_main.h | 3 + 8 files changed, 235 insertions(+), 17 deletions(-) create mode 100644 bin/varnishtest/tests/grace-drop.vtc diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 356d138..d60ad8b 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -584,6 +584,7 @@ struct req { const char *doclose; struct exp exp; + struct objcore *grace_oc; /* == stale_oc */ unsigned cur_method; unsigned handling; unsigned char sendbody; @@ -753,6 +754,7 @@ void EXP_Init(void); void EXP_Rearm(const struct object *o); int EXP_Touch(struct objcore *oc); int EXP_NukeOne(struct worker *w, struct lru *lru); +void EXP_Remove(struct worker *w, struct objcore *oc); /* cache_fetch.c */ struct storage *FetchStorage(struct worker *w, ssize_t sz); @@ -969,6 +971,7 @@ void RES_StreamPoll(struct worker *); /* cache_vary.c */ struct vsb *VRY_Create(const struct sess *sp, const struct http *hp); +int VRY_Compare(const uint8_t *vary1, const uint8_t *vary2); int VRY_Match(const struct sess *sp, const uint8_t *vary); void VRY_Validate(const uint8_t *vary); diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c index d80842a..a4aef8b 100644 --- a/bin/varnishd/cache/cache_center.c +++ b/bin/varnishd/cache/cache_center.c @@ -928,7 +928,7 @@ cnt_fetchbody(struct sess *sp, struct worker *wrk, struct req *req) EXP_Insert(req->obj); AN(req->obj->objcore); AN(req->obj->objcore->ban); - HSH_Unbusy(wrk); + HSH_Unbusy(wrk, 1); } VBO_DerefBusyObj(wrk, &wrk->busyobj); wrk->acct_tmp.fetch++; @@ -987,7 +987,7 @@ cnt_streambody(struct sess *sp, struct worker *wrk, struct req *req) EXP_Insert(req->obj); AN(req->obj->objcore); AN(req->obj->objcore->ban); - HSH_Unbusy(wrk); + HSH_Unbusy(wrk, 1); } else { req->doclose = "Stream error"; } diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c index e93a1aa..a8c4254 100644 --- a/bin/varnishd/cache/cache_expire.c +++ b/bin/varnishd/cache/cache_expire.c @@ -451,6 +451,49 @@ EXP_NukeOne(struct worker *wrk, struct lru *lru) } /*-------------------------------------------------------------------- + * Remove from expiry binheap. Caller must have a ref on the oc. + */ + +void +EXP_Remove(struct worker *w, struct objcore *oc) +{ + struct object *o; + struct lru *l; + + CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); + o = oc_getobj(w, oc); + CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC); + l = oc_getlru(oc); + CHECK_OBJ_NOTNULL(l, LRU_MAGIC); + + Lck_Lock(&l->mtx); + Lck_Lock(&exp_mtx); + + if (oc->timer_idx == BINHEAP_NOIDX) { /* exp_timer has it */ + Lck_Unlock(&exp_mtx); + Lck_Unlock(&l->mtx); + return; + } + + /* remove from binheap */ + binheap_delete(exp_heap, oc->timer_idx); + assert(oc->timer_idx == BINHEAP_NOIDX); + + /* And from LRU */ + CHECK_OBJ_NOTNULL(l, LRU_MAGIC); + VTAILQ_REMOVE(&l->lru_head, oc, lru_list); + + Lck_Unlock(&exp_mtx); + Lck_Unlock(&l->mtx); + + w->stats.c_removed++; + + WSL(w, SLT_ExpKill, 0, "%u Removed", o->xid); + assert(oc->refcnt >= 2); /* exp ref and caller ref */ + HSH_Deref(w, NULL, &o); +} + +/*-------------------------------------------------------------------- * BinHeap helper functions for objcore. */ diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c index d967291..22ad508 100644 --- a/bin/varnishd/cache/cache_hash.c +++ b/bin/varnishd/cache/cache_hash.c @@ -296,7 +296,7 @@ HSH_Lookup(struct sess *sp, struct objhead **poh) struct worker *wrk; struct objhead *oh; struct objcore *oc; - struct objcore *busy_oc, *grace_oc; + struct objcore *busy_oc; struct object *o; double grace_ttl; @@ -332,7 +332,7 @@ HSH_Lookup(struct sess *sp, struct objhead **poh) Lck_Lock(&oh->mtx); assert(oh->refcnt > 0); busy_oc = NULL; - grace_oc = NULL; + sp->req->grace_oc = NULL; grace_ttl = NAN; VTAILQ_FOREACH(oc, &oh->objcs, list) { /* Must be at least our own ref + the objcore we examine */ @@ -372,9 +372,9 @@ HSH_Lookup(struct sess *sp, struct objhead **poh) * and if there are several, use the least expired one. */ if (EXP_Grace(sp, o) >= sp->t_req) { - if (grace_oc == NULL || + if (sp->req->grace_oc == NULL || grace_ttl < o->exp.entered + o->exp.ttl) { - grace_oc = oc; + sp->req->grace_oc = oc; grace_ttl = o->exp.entered + o->exp.ttl; } } @@ -391,15 +391,16 @@ HSH_Lookup(struct sess *sp, struct objhead **poh) */ AZ(sp->req->objcore); - sp->req->objcore = grace_oc; /* XXX: Hack-ish */ - if (oc == NULL /* We found no live object */ - && grace_oc != NULL /* There is a grace candidate */ - && (busy_oc != NULL /* Somebody else is already busy */ + sp->req->objcore = sp->req->grace_oc; /* XXX: Hack-ish */ + if (oc == NULL /* We found no live object */ + && sp->req->grace_oc != NULL /* There is a grace candidate */ + && (busy_oc != NULL /* Somebody else is already busy */ || !VDI_Healthy(sp->req->director, sp))) { /* Or it is impossible to fetch */ - o = oc_getobj(sp->wrk, grace_oc); + o = oc_getobj(sp->wrk, sp->req->grace_oc); CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC); - oc = grace_oc; + oc = sp->req->grace_oc; + sp->req->grace_oc = NULL; } sp->req->objcore = NULL; @@ -415,6 +416,7 @@ HSH_Lookup(struct sess *sp, struct objhead **poh) assert(oh->refcnt > 1); Lck_Unlock(&oh->mtx); assert(hash->deref(oh)); + sp->req->grace_oc = NULL; *poh = oh; return (oc); } @@ -441,6 +443,7 @@ HSH_Lookup(struct sess *sp, struct objhead **poh) */ sp->req->hash_objhead = oh; sp->wrk = NULL; + sp->req->grace_oc = NULL; Lck_Unlock(&oh->mtx); return (NULL); } @@ -450,6 +453,15 @@ HSH_Lookup(struct sess *sp, struct objhead **poh) wrk->nobjcore = NULL; AN(oc->flags & OC_F_BUSY); oc->refcnt = 1; + /* + * make sure that we don't expire the graced object while we're fetching + * so that we don't panic/segfault after fetch + */ + if (sp->req->grace_oc) { + CHECK_OBJ_NOTNULL(sp->req->grace_oc, OBJCORE_MAGIC); + sp->req->grace_oc->refcnt++; + assert(sp->req->grace_oc->refcnt > 0); + } AZ(wrk->busyobj); wrk->busyobj = VBO_GetBusyObj(wrk); @@ -586,16 +598,16 @@ HSH_Drop(struct worker *wrk) AssertObjCorePassOrBusy(o->objcore); o->exp.ttl = -1.; if (o->objcore != NULL) /* Pass has no objcore */ - HSH_Unbusy(wrk); + HSH_Unbusy(wrk, 0); (void)HSH_Deref(wrk, NULL, &wrk->sp->req->obj); } void -HSH_Unbusy(struct worker *wrk) +HSH_Unbusy(struct worker *wrk, int dropgrace) { - struct object *o; + struct object *o, *go; struct objhead *oh; - struct objcore *oc; + struct objcore *oc, *grace_oc; CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); o = wrk->sp->req->obj; @@ -627,6 +639,18 @@ HSH_Unbusy(struct worker *wrk) hsh_rush(oh); AN(oc->ban); Lck_Unlock(&oh->mtx); + if (wrk->sp->req->grace_oc) { + grace_oc = wrk->sp->req->grace_oc; + CHECK_OBJ_NOTNULL(grace_oc, OBJCORE_MAGIC); + go = oc_getobj(wrk, grace_oc); + CHECK_OBJ_NOTNULL(go, OBJECT_MAGIC); + if (dropgrace && VRY_Compare(o->vary, go->vary)) { + wrk->stats.c_grace_obj_dropped++; + EXP_Remove(wrk, grace_oc); + } + assert(grace_oc->refcnt > 0); + HSH_Deref(wrk, NULL, &go); + } assert(oc_getobj(wrk, oc) == o); } diff --git a/bin/varnishd/cache/cache_vary.c b/bin/varnishd/cache/cache_vary.c index 9ad41d0..dea9d95 100644 --- a/bin/varnishd/cache/cache_vary.c +++ b/bin/varnishd/cache/cache_vary.c @@ -174,6 +174,29 @@ vry_cmp(const uint8_t *v1, const uint8_t *v2) } int +VRY_Compare(const uint8_t *vary1, const uint8_t *vary2) +{ + + if (vary1 == NULL && vary2 == NULL) + /* both are NULL */ + return 1; + if (vary1 == NULL || vary2 == NULL) + /* only one of them is NULL */ + return 0; + while (vary1[2] && vary2[2]) { + if (vry_cmp(vary1, vary2)) + /* header doesn't match */ + return 0; + vary1 += vry_len(vary1); + vary2 += vry_len(vary2); + } + if (vary1[2] || vary2[2]) + /* number of headers doesn't match */ + return 0; + return 1; +} + +int VRY_Match(const struct sess *sp, const uint8_t *vary) { uint8_t *vsp = sp->req->vary_b; diff --git a/bin/varnishd/hash/hash_slinger.h b/bin/varnishd/hash/hash_slinger.h index b45e604..c0cd54c 100644 --- a/bin/varnishd/hash/hash_slinger.h +++ b/bin/varnishd/hash/hash_slinger.h @@ -54,7 +54,7 @@ struct hash_slinger { void HSH_Prealloc(const struct sess *sp); void HSH_Cleanup(struct worker *w); struct objcore *HSH_Lookup(struct sess *sp, struct objhead **poh); -void HSH_Unbusy(struct worker *wrk); +void HSH_Unbusy(struct worker *wrk, int dropgrace); void HSH_Ref(struct objcore *o); void HSH_Drop(struct worker *wrk); void HSH_Init(const struct hash_slinger *slinger); diff --git a/bin/varnishtest/tests/grace-drop.vtc b/bin/varnishtest/tests/grace-drop.vtc new file mode 100644 index 0000000..05cdbac --- /dev/null +++ b/bin/varnishtest/tests/grace-drop.vtc @@ -0,0 +1,122 @@ +varnishtest "Drop graced object when new object arrives" + +server s1 { + rxreq + txresp -hdr "Vary: X-Foo" -hdr "Cache-Control: max-age=1" -body "1" + rxreq + txresp -hdr "Vary: X-Foo" -hdr "Cache-Control: max-age=1" -body "12" + rxreq + txresp -hdr "Vary: X-Foo" -hdr "Cache-Control: max-age=1" -body "123" +} -start + +varnish v1 -vcl+backend { + sub vcl_recv { + set req.grace = 0s; + } + sub vcl_fetch { + set beresp.grace = 3600s; + } +} -start + +varnish v1 -expect n_object == 0 +varnish v1 -expect sess_conn == 0 +varnish v1 -expect client_req == 0 +varnish v1 -expect cache_miss == 0 +varnish v1 -expect cache_hit == 0 +varnish v1 -expect s_sess == 0 +varnish v1 -expect s_req == 0 + +client c1 { + txreq -hdr "X-Foo: bar" + rxresp + expect resp.status == 200 + expect resp.bodylen == 1 + txreq -hdr "X-Foo: blargh" + rxresp + expect resp.status == 200 + expect resp.bodylen == 2 +} -run + +delay 1.1 + +varnish v1 -expect n_object == 2 +varnish v1 -expect sess_conn == 1 +varnish v1 -expect client_req == 2 +varnish v1 -expect cache_miss == 2 +varnish v1 -expect cache_hit == 0 +varnish v1 -expect s_sess == 1 +varnish v1 -expect s_req == 2 + +client c1 { + txreq -hdr "X-Foo: bar" + rxresp + expect resp.status == 200 + expect resp.bodylen == 3 +} -run + +varnish v1 -expect n_object == 2 +varnish v1 -expect sess_conn == 2 +varnish v1 -expect client_req == 3 +varnish v1 -expect cache_miss == 3 +varnish v1 -expect cache_hit == 0 +varnish v1 -expect s_sess == 2 +varnish v1 -expect s_req == 3 + +server s1 -wait + +delay 1.1 + +client c1 { + txreq -hdr "X-Foo: bar" + rxresp + expect resp.status == 503 +} -run + +varnish v1 -expect n_object == 2 +varnish v1 -expect sess_conn == 3 +varnish v1 -expect client_req == 4 +varnish v1 -expect cache_miss == 4 +varnish v1 -expect cache_hit == 0 +varnish v1 -expect s_sess == 3 +varnish v1 -expect s_req == 4 + +varnish v1 -vcl { + backend default { + .host = "${s1_addr}"; + .port = "${s1_port}"; + .max_connections = 1; + .probe = { + .url = "/"; + .timeout = 1s; + .interval = 1000s; + .window = 1; + .threshold = 1; + .initial = 0; + } + } + + sub vcl_recv { + set req.grace = 3600s; + } +} + +delay 0.1 + +client c1 { + txreq -hdr "X-Foo: bar" + rxresp + expect resp.status == 200 + expect resp.bodylen == 3 + txreq -hdr "X-Foo: blargh" + rxresp + expect resp.status == 200 + expect resp.bodylen == 2 +} -run + +varnish v1 -expect n_object == 2 +varnish v1 -expect sess_conn == 4 +varnish v1 -expect client_req == 6 +varnish v1 -expect cache_miss == 4 +varnish v1 -expect cache_hit == 2 +varnish v1 -expect s_sess == 4 +varnish v1 -expect s_req == 6 diff --git a/include/tbl/vsc_f_main.h b/include/tbl/vsc_f_main.h index df7984a..60401a6 100644 --- a/include/tbl/vsc_f_main.h +++ b/include/tbl/vsc_f_main.h @@ -232,7 +232,10 @@ VSC_F(n_backend, uint64_t, 0, 'i', "N backends", "") VSC_F(n_expired, uint64_t, 0, 'i', "N expired objects", "") VSC_F(n_lru_nuked, uint64_t, 0, 'i', "N LRU nuked objects", "") +VSC_F(c_removed, uint64_t, 1, 'a', "N removed objects", "") VSC_F(n_lru_moved, uint64_t, 0, 'i', "N LRU moved objects", "") +VSC_F(c_grace_obj_dropped, uint64_t, 1, 'i', + "Objects in grace dropped due to new object from backend", "") VSC_F(losthdr, uint64_t, 0, 'a', "HTTP header overflows", "") -- 1.7.5.4 From perbu at varnish-software.com Mon Jan 16 08:24:28 2012 From: perbu at varnish-software.com (Per Buer) Date: Mon, 16 Jan 2012 09:24:28 +0100 Subject: new module In-Reply-To: References: Message-ID: On Sat, Jan 14, 2012 at 11:25 PM, Olivier Van Acker wrote: > Hi, > > I created a new varnish module, it orders the parameters in an url > alphabetically > Code can be found here: > https://github.com/cyberroadie/varnish-urlsort > > And a blogpost describing it here: > > http://cyberroadie.wordpress.com/2012/01/05/varnish-reordering-query-string/ > > Is it possible to get it listed on the > https://www.varnish-cache.org/project/modules page? > Yes. That would be great. We're in the middle of changing the management system for the modules so you can more or less self-manage your VMODs. I'll get back to you in a week or so and pester you for some information. Prod me if you haven't heard anything by the end of next week. -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From cyberroadie at gmail.com Mon Jan 16 09:33:56 2012 From: cyberroadie at gmail.com (Olivier Van Acker) Date: Mon, 16 Jan 2012 09:33:56 +0000 Subject: new module In-Reply-To: References: Message-ID: Great, thanks :-) Olivier On 16 January 2012 08:24, Per Buer wrote: > On Sat, Jan 14, 2012 at 11:25 PM, Olivier Van Acker > wrote: > >> Hi, >> >> I created a new varnish module, it orders the parameters in an url >> alphabetically >> Code can be found here: >> https://github.com/cyberroadie/varnish-urlsort >> >> And a blogpost describing it here: >> >> http://cyberroadie.wordpress.com/2012/01/05/varnish-reordering-query-string/ >> >> Is it possible to get it listed on the >> https://www.varnish-cache.org/project/modules page? >> > > Yes. That would be great. We're in the middle of changing the management > system for the modules so you can more or less self-manage your VMODs. I'll > get back to you in a week or so and pester you for some information. Prod > me if you haven't heard anything by the end of next week. > > > -- > Per Buer, CEO > Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer > *Varnish makes websites fly!* > Whitepapers | Video | > Twitter > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Tue Jan 17 08:21:50 2012 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 17 Jan 2012 08:21:50 +0000 Subject: [PATCH] Drop old grace objects when we have a new object that matches the Vary In-Reply-To: Your message of "Mon, 16 Jan 2012 00:08:24 +0100." <1326668904-24963-1-git-send-email-github@bsdchicks.com> Message-ID: <17514.1326788510@critter.freebsd.dk> In message <1326668904-24963-1-git-send-email-github at bsdchicks.com>, Rogier 'Do cWilco' Mulhuijzen writes: A bunch of comments of mostly stylistic nature >+ struct objcore *grace_oc; /* == stale_oc */ Which is it ? grace_oc or stale_oc ? choose one. >+void >+EXP_Remove(struct worker *w, struct objcore *oc) >+{ >[...] >+ Lck_Unlock(&exp_mtx); >+ Lck_Unlock(&l->mtx); >+ >+ w->stats.c_removed++; Shouldn't that stats counter be inside the lock for consistency ? > void >-HSH_Unbusy(struct worker *wrk) >+HSH_Unbusy(struct worker *wrk, int dropgrace) > { >- struct object *o; >+ struct object *o, *go; > struct objhead *oh; >- struct objcore *oc; >+ struct objcore *oc, *grace_oc; > > CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); > o = wrk->sp->req->obj; >@@ -627,6 +639,18 @@ HSH_Unbusy(struct worker *wrk) > hsh_rush(oh); > AN(oc->ban); > Lck_Unlock(&oh->mtx); >+ if (wrk->sp->req->grace_oc) { >+ grace_oc = wrk->sp->req->grace_oc; Please NULL sp->req->grace_oc when you take over ownership of the oc pointed to. >+ CHECK_OBJ_NOTNULL(grace_oc, OBJCORE_MAGIC); >+ go = oc_getobj(wrk, grace_oc); >+ CHECK_OBJ_NOTNULL(go, OBJECT_MAGIC); >+ if (dropgrace && VRY_Compare(o->vary, go->vary)) { >+ wrk->stats.c_grace_obj_dropped++; >+ EXP_Remove(wrk, grace_oc); >+ } >+ assert(grace_oc->refcnt > 0); >+ HSH_Deref(wrk, NULL, &go); >+ } > assert(oc_getobj(wrk, oc) == o); > } Looking at this, I fail to see why it needs to be stuffed into HSH_Unbusy() adding a "what is it that flag means" argument, instead of being its own function with a sensible name ? I can see the "but it automatically catches all places we might need to get rid of the grace_oc" argument, but I don't find it particularly convincing. > int >+VRY_Compare(const uint8_t *vary1, const uint8_t *vary2) >+{ >+ >+ if (vary1 == NULL && vary2 == NULL) >+ /* both are NULL */ >+ return 1; Please use (...) around return values for consistency. >+ if (vary1[2] || vary2[2]) >+ /* number of headers doesn't match */ >+ return 0; We should always put {...} around if-bodies, if they are more than one line, even if the second line is a comment. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From github at bsdchicks.com Wed Jan 18 09:51:15 2012 From: github at bsdchicks.com (Rogier 'DocWilco' Mulhuijzen) Date: Wed, 18 Jan 2012 10:51:15 +0100 Subject: [PATCH 2/3] Fixup printf format on 32bit platforms In-Reply-To: <1326880276-18354-1-git-send-email-github@bsdchicks.com> References: <1326880276-18354-1-git-send-email-github@bsdchicks.com> Message-ID: <1326880276-18354-2-git-send-email-github@bsdchicks.com> --- bin/varnishd/cache/cache_wrw.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/bin/varnishd/cache/cache_wrw.c b/bin/varnishd/cache/cache_wrw.c index c344f9c..e33be6c 100644 --- a/bin/varnishd/cache/cache_wrw.c +++ b/bin/varnishd/cache/cache_wrw.c @@ -163,14 +163,14 @@ WRW_Flush(struct worker *wrk) cache_param->send_timeout) { WSL(wrk, SLT_Debug, *wrw->wfd, "Hit total send timeout, " - "wrote = %ld/%ld; not retrying", + "wrote = %zd/%zd; not retrying", i, wrw->liov); i = -1; break; } WSL(wrk, SLT_Debug, *wrw->wfd, - "Hit send timeout, wrote = %ld/%ld; retrying", + "Hit send timeout, wrote = %zd/%zd; retrying", i, wrw->liov); wrw_prune(wrw, i); -- 1.7.5.4 From github at bsdchicks.com Wed Jan 18 09:51:14 2012 From: github at bsdchicks.com (Rogier 'DocWilco' Mulhuijzen) Date: Wed, 18 Jan 2012 10:51:14 +0100 Subject: [PATCH 1/3] Fix up some sess_workspace leftovers Message-ID: <1326880276-18354-1-git-send-email-github@bsdchicks.com> Found by: Joakim Berg --- bin/varnishd/cache/cache_wrk.c | 8 ++++---- bin/varnishd/mgt/mgt_main.c | 2 +- bin/varnishd/mgt/mgt_param.c | 8 ++++---- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/bin/varnishd/cache/cache_wrk.c b/bin/varnishd/cache/cache_wrk.c index 6087991..2b6cc23 100644 --- a/bin/varnishd/cache/cache_wrk.c +++ b/bin/varnishd/cache/cache_wrk.c @@ -131,13 +131,13 @@ WRK_BgThread(pthread_t *thr, const char *name, bgthread_t *func, void *priv) /*--------------------------------------------------------------------*/ static void * -wrk_thread_real(void *priv, unsigned shm_workspace, unsigned sess_workspace, - unsigned siov) +wrk_thread_real(void *priv, unsigned shm_workspace, + unsigned wthread_workspace, unsigned siov) { struct worker *w, ww; uint32_t wlog[shm_workspace / 4]; /* XXX: can we trust these to be properly aligned ? */ - unsigned char ws[sess_workspace]; + unsigned char ws[wthread_workspace]; struct iovec iov[siov]; THR_SetName("cache-worker"); @@ -152,7 +152,7 @@ wrk_thread_real(void *priv, unsigned shm_workspace, unsigned sess_workspace, w->wrw.ciov = siov; AZ(pthread_cond_init(&w->cond, NULL)); - WS_Init(w->ws, "wrk", ws, sess_workspace); + WS_Init(w->ws, "wrk", ws, wthread_workspace); VSL(SLT_WorkThread, 0, "%p start", w); diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c index cc918c1..6405cdd 100644 --- a/bin/varnishd/mgt/mgt_main.c +++ b/bin/varnishd/mgt/mgt_main.c @@ -406,7 +406,7 @@ main(int argc, char * const *argv) * Adjust default parameters for 32 bit systems to conserve * VM space. */ - MCF_ParamSet(cli, "sess_workspace", "16384"); + MCF_ParamSet(cli, "workspace_client", "16384"); cli_check(cli); MCF_ParamSet(cli, "thread_pool_workspace", "16384"); diff --git a/bin/varnishd/mgt/mgt_param.c b/bin/varnishd/mgt/mgt_param.c index 8123bfc..e3dfd12 100644 --- a/bin/varnishd/mgt/mgt_param.c +++ b/bin/varnishd/mgt/mgt_param.c @@ -705,8 +705,8 @@ static const struct parspec input_parspec[] = { "Maximum number of bytes of HTTP client request we will deal " "with. This is a limit on all bytes up to the double blank " "line which ends the HTTP request.\n" - "The memory for the request is allocated from the session " - "workspace (param: sess_workspace) and this parameter limits " + "The memory for the request is allocated from the client " + "workspace (param: workspace_client) and this parameter limits " "how much of that the request is allowed to take up.", 0, "32k", "bytes" }, @@ -724,8 +724,8 @@ static const struct parspec input_parspec[] = { "with. This is a limit on all bytes up to the double blank " "line which ends the HTTP request.\n" "The memory for the request is allocated from the worker " - "workspace (param: sess_workspace) and this parameter limits " - "how much of that the request is allowed to take up.", + "workspace (param: thread_pool_workspace) and this parameter " + "limits how much of that the request is allowed to take up.", 0, "32k", "bytes" }, { "http_max_hdr", tweak_uint, &mgt_param.http_max_hdr, 32, 65535, -- 1.7.5.4 From github at bsdchicks.com Wed Jan 18 09:51:16 2012 From: github at bsdchicks.com (Rogier 'DocWilco' Mulhuijzen) Date: Wed, 18 Jan 2012 10:51:16 +0100 Subject: [PATCH 3/3] Wait a bit before trying to start/use varnishd again In-Reply-To: <1326880276-18354-1-git-send-email-github@bsdchicks.com> References: <1326880276-18354-1-git-send-email-github@bsdchicks.com> Message-ID: <1326880276-18354-3-git-send-email-github@bsdchicks.com> --- bin/varnishtest/tests/p00007.vtc | 2 ++ bin/varnishtest/tests/v00010.vtc | 2 ++ 2 files changed, 4 insertions(+), 0 deletions(-) diff --git a/bin/varnishtest/tests/p00007.vtc b/bin/varnishtest/tests/p00007.vtc index 78aa8fb..7dc2e68 100644 --- a/bin/varnishtest/tests/p00007.vtc +++ b/bin/varnishtest/tests/p00007.vtc @@ -53,6 +53,8 @@ varnish v1 -cliok "debug.persistent s0 dump" # Panic worker so second segment does not get closed varnish v1 -clierr 400 "debug.panic.worker" +delay 0.5 + # start again varnish v1 -start diff --git a/bin/varnishtest/tests/v00010.vtc b/bin/varnishtest/tests/v00010.vtc index 45dbc79..b44fc13 100644 --- a/bin/varnishtest/tests/v00010.vtc +++ b/bin/varnishtest/tests/v00010.vtc @@ -42,6 +42,8 @@ varnish v1 -cliok "start" varnish v1 -wait-running sema r1 sync 2 +delay 0.5 + client c1 { txreq -url "/" rxresp -- 1.7.5.4 From github at bsdchicks.com Wed Jan 18 18:03:42 2012 From: github at bsdchicks.com (Rogier 'DocWilco' Mulhuijzen) Date: Wed, 18 Jan 2012 19:03:42 +0100 Subject: [PATCH] Don't forget to do the stats when a request is aborted Message-ID: <1326909822-25213-1-git-send-email-github@bsdchicks.com> Without doing SES_Charge, the worker will panic at the end of CNT_Session() when a request is aborted before it is complete. Found by: Federico G. Schwindt --- bin/varnishd/cache/cache_center.c | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c index a9c233b..b03ba4b 100644 --- a/bin/varnishd/cache/cache_center.c +++ b/bin/varnishd/cache/cache_center.c @@ -139,10 +139,12 @@ cnt_wait(struct sess *sp, struct worker *wrk, struct req *req) break; } if (i == -1) { + SES_Charge(sp); SES_Delete(sp, "EOF", now); return (1); } if (i == -2) { + SES_Charge(sp); SES_Delete(sp, "overflow", now); return (1); } @@ -150,6 +152,7 @@ cnt_wait(struct sess *sp, struct worker *wrk, struct req *req) /* Nothing but whitespace */ when = sp->t_idle + cache_param->timeout_idle; if (when < now) { + SES_Charge(sp); SES_Delete(sp, "timeout", now); return (1); } @@ -168,6 +171,7 @@ cnt_wait(struct sess *sp, struct worker *wrk, struct req *req) when = sp->t_req + cache_param->timeout_req; tmo = (int)(1e3 * (when - now)); if (when < now || tmo == 0) { + SES_Charge(sp); SES_Delete(sp, "req timeout", now); return (1); } -- 1.7.5.4 From phk at phk.freebsd.dk Thu Jan 19 07:55:05 2012 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 19 Jan 2012 07:55:05 +0000 Subject: [PATCH] Don't forget to do the stats when a request is aborted In-Reply-To: Your message of "Wed, 18 Jan 2012 19:03:42 +0100." <1326909822-25213-1-git-send-email-github@bsdchicks.com> Message-ID: <20121.1326959705@critter.freebsd.dk> In message <1326909822-25213-1-git-send-email-github at bsdchicks.com>, Rogier 'Do cWilco' Mulhuijzen writes: Looks good, commit it. >Without doing SES_Charge, the worker will panic at the end of >CNT_Session() when a request is aborted before it is complete. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From kristian at varnish-software.com Thu Jan 19 12:27:28 2012 From: kristian at varnish-software.com (Kristian Lyngstol) Date: Thu, 19 Jan 2012 13:27:28 +0100 Subject: FYI: Fryer hacking ensues Message-ID: <20120119122728.GJ18108@freud.kly.no> I'm hacking on Fryer these days, so expect some strange reports to varnish-test and possibly delayed executions. If you have any wish list items, now is the time to make them. So far, the plan is: - Finding out why siege fails so badly on certain tests - Dealing with empty stat counters (e.g: counters that change name or are removed) more gracefully. - Adding tests that use randomized mixed sizes for objects. - Printing the aggregated expectations whenever a test fails. - Persistent storage checks - Random stuff I encounter along the way. - Kristian From martin at varnish-software.com Thu Jan 19 15:35:58 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Thu, 19 Jan 2012 16:35:58 +0100 Subject: [PATCH] Stop accept() timeouts from being counted as sess_fail in varnishstat Message-ID: <1326987358-26699-1-git-send-email-martin@varnish-software.com> Varnish sets SO_RCVTIMEO to cache_param->timeout_idle on the listen sockets, so that the value is inherited to the accept()ed connections. This has the side effect of making accept() returning with EAGAIN after cache_param->timeout_idle seconds when there are no connections coming in, and each of these timeouts is counted as a sess_failed. Also the pacing sleep time is increased for each of these timeouts, causing Varnish to react slowly at first when connections start coming in. Patch changes to loop around the call to accept() while EAGAIN --- bin/varnishd/cache/cache_acceptor.c | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-) diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c index d55545b..7918a6a 100644 --- a/bin/varnishd/cache/cache_acceptor.c +++ b/bin/varnishd/cache/cache_acceptor.c @@ -186,7 +186,10 @@ VCA_Accept(struct listen_sock *ls, struct wrk_accept *wa) (void)usleep(100*1000); wa->acceptaddrlen = sizeof wa->acceptaddr; - i = accept(ls->sock, (void*)&wa->acceptaddr, &wa->acceptaddrlen); + do { + i = accept(ls->sock, (void*)&wa->acceptaddr, + &wa->acceptaddrlen); + } while (i < 0 && errno == EAGAIN); if (i < 0) { switch (errno) { -- 1.7.4.1 From phk at phk.freebsd.dk Thu Jan 19 16:05:11 2012 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 19 Jan 2012 16:05:11 +0000 Subject: [PATCH] Stop accept() timeouts from being counted as sess_fail in varnishstat In-Reply-To: Your message of "Thu, 19 Jan 2012 16:35:58 +0100." <1326987358-26699-1-git-send-email-martin@varnish-software.com> Message-ID: <29391.1326989111@critter.freebsd.dk> In message <1326987358-26699-1-git-send-email-martin at varnish-software.com>, Mar tin Blix Grydeland writes: Commit if tested. Poul-Henning >Varnish sets SO_RCVTIMEO to cache_param->timeout_idle on the listen >sockets, so that the value is inherited to the accept()ed >connections. This has the side effect of making accept() returning >with EAGAIN after cache_param->timeout_idle seconds when there are no >connections coming in, and each of these timeouts is counted as a >sess_failed. Also the pacing sleep time is increased for each of these >timeouts, causing Varnish to react slowly at first when connections >start coming in. > >Patch changes to loop around the call to accept() while EAGAIN >--- > bin/varnishd/cache/cache_acceptor.c | 5 ++++- > 1 files changed, 4 insertions(+), 1 deletions(-) > >diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c >index d55545b..7918a6a 100644 >--- a/bin/varnishd/cache/cache_acceptor.c >+++ b/bin/varnishd/cache/cache_acceptor.c >@@ -186,7 +186,10 @@ VCA_Accept(struct listen_sock *ls, struct wrk_accept *wa) > (void)usleep(100*1000); > > wa->acceptaddrlen = sizeof wa->acceptaddr; >- i = accept(ls->sock, (void*)&wa->acceptaddr, &wa->acceptaddrlen); >+ do { >+ i = accept(ls->sock, (void*)&wa->acceptaddr, >+ &wa->acceptaddrlen); >+ } while (i < 0 && errno == EAGAIN); > > if (i < 0) { > switch (errno) { >-- >1.7.4.1 > > >_______________________________________________ >varnish-dev mailing list >varnish-dev at varnish-cache.org >https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From apj at mutt.dk Thu Jan 19 17:19:42 2012 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Thu, 19 Jan 2012 18:19:42 +0100 Subject: [PATCH] Cleanup references to sess_workspace in docs help Message-ID: <1326993583-7127-1-git-send-email-apj@mutt.dk> This should only touch doc-type stuff From apj at mutt.dk Thu Jan 19 17:19:43 2012 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Thu, 19 Jan 2012 18:19:43 +0100 Subject: [PATCH] Cleanup references to sess_workspace in docs help In-Reply-To: <1326993583-7127-1-git-send-email-apj@mutt.dk> References: <1326993583-7127-1-git-send-email-apj@mutt.dk> Message-ID: <1326993583-7127-2-git-send-email-apj@mutt.dk> Found by: Federico G. Schwindt --- bin/varnishd/mgt/mgt_param.c | 12 ++++++------ doc/sphinx/reference/params.rst | 12 ++---------- doc/sphinx/reference/varnishd.rst | 6 ++++-- doc/sphinx/reference/vmod.rst | 2 +- 4 files changed, 13 insertions(+), 19 deletions(-) diff --git a/bin/varnishd/mgt/mgt_param.c b/bin/varnishd/mgt/mgt_param.c index c984aae..c8f652b 100644 --- a/bin/varnishd/mgt/mgt_param.c +++ b/bin/varnishd/mgt/mgt_param.c @@ -705,9 +705,9 @@ static const struct parspec input_parspec[] = { "Maximum number of bytes of HTTP client request we will deal " "with. This is a limit on all bytes up to the double blank " "line which ends the HTTP request.\n" - "The memory for the request is allocated from the session " - "workspace (param: sess_workspace) and this parameter limits " - "how much of that the request is allowed to take up.", + "The memory for the request is allocated from the client " + "workspace (param: workspace_client) and this parameter " + "limits how much of that the request is allowed to take up.", 0, "32k", "bytes" }, { "http_resp_hdr_len", @@ -723,9 +723,9 @@ static const struct parspec input_parspec[] = { "Maximum number of bytes of HTTP backend resonse we will deal " "with. This is a limit on all bytes up to the double blank " "line which ends the HTTP request.\n" - "The memory for the request is allocated from the worker " - "workspace (param: sess_workspace) and this parameter limits " - "how much of that the request is allowed to take up.", + "The memory for the request is allocated from the thread pool " + "workspace (param: thread_pool_workspace) and this parameter " + "limits how much of that the request is allowed to take up.", 0, "32k", "bytes" }, { "http_max_hdr", tweak_uint, &mgt_param.http_max_hdr, 32, 65535, diff --git a/doc/sphinx/reference/params.rst b/doc/sphinx/reference/params.rst index f161596..1f9a486 100644 --- a/doc/sphinx/reference/params.rst +++ b/doc/sphinx/reference/params.rst @@ -250,7 +250,7 @@ http_req_size - Default: 32768 Maximum number of bytes of HTTP client request we will deal with. This is a limit on all bytes up to the double blank line which ends the HTTP request. - The memory for the request is allocated from the session workspace (param: sess_workspace) and this parameter limits how much of that the request is allowed to take up. + The memory for the request is allocated from the client workspace (param: workspace_client) and this parameter limits how much of that the request is allowed to take up. http_resp_hdr_len - Units: bytes @@ -263,7 +263,7 @@ http_resp_size - Default: 32768 Maximum number of bytes of HTTP backend resonse we will deal with. This is a limit on all bytes up to the double blank line which ends the HTTP request. - The memory for the request is allocated from the worker workspace (param: sess_workspace) and this parameter limits how much of that the request is allowed to take up. + The memory for the request is allocated from the thread pool workspace (param: thread_pool_workspace) and this parameter limits how much of that the request is allowed to take up. listen_address - Default: :80 @@ -378,14 +378,6 @@ sess_timeout Idle timeout for persistent sessions. If a HTTP request has not been received in this many seconds, the session is closed. -sess_workspace - - Units: bytes - - Default: 65536 - - Flags: delayed - - Bytes of HTTP protocol workspace allocated for sessions. This space must be big enough for the entire HTTP protocol header and any edits done to it in the VCL code. - Minimum is 1024 bytes. - session_linger - Units: ms - Default: 50 diff --git a/doc/sphinx/reference/varnishd.rst b/doc/sphinx/reference/varnishd.rst index 4e2b3f2..351c7b3 100644 --- a/doc/sphinx/reference/varnishd.rst +++ b/doc/sphinx/reference/varnishd.rst @@ -274,8 +274,10 @@ Here is a list of all parameters, current as of last time we remembered to updat produced from the same text you will find in the CLI if you use the param.show command, so should there be a new parameter which is not listed here, you can find the description using the CLI commands. -Be aware that on 32 bit systems, certain default values, such as sess_workspace (=16k) and thread_pool_stack -(=64k) are reduced relative to the values listed here, in order to conserve VM space. +Be aware that on 32 bit systems, certain default values, such as +workspace_client (=16k), thread_pool_workspace (=16k), http_resp_size (=8k), +http_req_size (=12k), gzip_stack_buffer (=4k) and thread_pool_stack (=64k) are +reduced relative to the values listed here, in order to conserve VM space. .. include:: params.rst diff --git a/doc/sphinx/reference/vmod.rst b/doc/sphinx/reference/vmod.rst index 2b515c1..57163c5 100644 --- a/doc/sphinx/reference/vmod.rst +++ b/doc/sphinx/reference/vmod.rst @@ -171,7 +171,7 @@ STRING_LIST a function, we may relax that at a latter time. If you don't want to bother with STRING_LIST, just use STRING - and make sure your sess_workspace param is big enough. + and make sure your thread_pool_workspace param is big enough. PRIV_VCL See below -- 1.7.8.3 From apj at mutt.dk Thu Jan 19 18:06:09 2012 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Thu, 19 Jan 2012 19:06:09 +0100 Subject: [PATCH] Cleanup references to sess_workspace in docs help In-Reply-To: <1326993583-7127-1-git-send-email-apj@mutt.dk> References: <1326993583-7127-1-git-send-email-apj@mutt.dk> Message-ID: <20120119180609.GX3214@nerd.dk> On Thu, Jan 19, 2012 at 06:19:42PM +0100, Andreas Plesner Jacobsen wrote: > This should only touch doc-type stuff DocWilco already had the changes to bin/, so all of mine is doc changes, which I'll just commit. -- Andreas From fgsch at lodoss.net Sat Jan 21 10:14:27 2012 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Sat, 21 Jan 2012 10:14:27 +0000 Subject: [PATCH] add a least-connection director Message-ID: <20120121101427.1e4d788b.fgsch@lodoss.net> Hi, The diff below adds a least-connection director based on the random director. Documentation and tests included. Comments? OKs? f.- bin/varnishd/cache/cache_backend.h | 1 + bin/varnishd/cache/cache_backend_cfg.c | 2 + bin/varnishd/cache/cache_dir_random.c | 52 ++++++++++++++++++-- bin/varnishtest/tests/v00037.vtc | 86 ++++++++++++++++++++++++++++++++ doc/sphinx/reference/vcl.rst | 11 ++++ lib/libvcl/vcc_backend.c | 1 + 7 files changed, 150 insertions(+), 5 deletions(-) diff --git a/bin/varnishd/cache/cache_backend.h b/bin/varnishd/cache/cache_backend.h index ad71169..6105ed3 100644 --- a/bin/varnishd/cache/cache_backend.h +++ b/bin/varnishd/cache/cache_backend.h @@ -190,3 +190,4 @@ dir_init_f VRT_init_dir_random; dir_init_f VRT_init_dir_round_robin; dir_init_f VRT_init_dir_fallback; dir_init_f VRT_init_dir_client; +dir_init_f VRT_init_dir_least_connection; diff --git a/bin/varnishd/cache/cache_backend_cfg.c b/bin/varnishd/cache/cache_backend_cfg.c index c2d022d..9bed6ed 100644 --- a/bin/varnishd/cache/cache_backend_cfg.c +++ b/bin/varnishd/cache/cache_backend_cfg.c @@ -263,6 +263,8 @@ VRT_init_dir(struct cli *cli, struct director **dir, const char *name, VRT_init_dir_fallback(cli, dir, idx, priv); else if (!strcmp(name, "client")) VRT_init_dir_client(cli, dir, idx, priv); + else if (!strcmp(name, "least-connection")) + VRT_init_dir_least_connection(cli, dir, idx, priv); else INCOMPL(); } diff --git a/bin/varnishd/cache/cache_dir_random.c b/bin/varnishd/cache/cache_dir_random.c index b323343..84a1224 100644 --- a/bin/varnishd/cache/cache_dir_random.c +++ b/bin/varnishd/cache/cache_dir_random.c @@ -26,8 +26,9 @@ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * - * This code is shared between the random, client and hash directors, because - * they share the same properties and most of the same selection logic. + * This code is shared between the random, client, hash and least-connection + * directors, because they share the same properties and most of the same + * selection logic. * * The random director picks a backend on random. * @@ -35,6 +36,9 @@ * * The client director picks based on client identity or IP-address * + * The least-connection director picks a backend with the least number of + * established connections. + * * In all cases, the choice is by weight of the healthy subset of * configured backends. * @@ -63,7 +67,7 @@ struct vdi_random_host { double weight; }; -enum crit_e {c_random, c_hash, c_client}; +enum crit_e {c_random, c_hash, c_client, c_least_connection}; struct vdi_random { unsigned magic; @@ -185,6 +189,36 @@ vdi_random_getfd(const struct director *d, struct sess *sp) return (NULL); } +static struct vbc * +vdi_least_connection_getfd(const struct director *d, struct sess *sp) +{ + int i, m; + struct vdi_random *vs; + struct backend *b1, *b2; + + CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); + CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC); + CAST_OBJ_NOTNULL(vs, d->priv, VDI_RANDOM_MAGIC); + + for (m = 0; m < vs->nhosts; m++) { + if (!VDI_Healthy(vs->hosts[m].backend, sp)) + continue; + b1 = vdi_get_backend_if_simple(vs->hosts[m].backend); + for (i = m + 1; i < vs->nhosts; i++) { + if (!VDI_Healthy(vs->hosts[i].backend, sp)) + continue; + b2 = vdi_get_backend_if_simple(vs->hosts[i].backend); + if (b1->refcount * vs->hosts[i].weight > + b2->refcount * vs->hosts[m].weight) { + b1 = b2; + m = i; + } + } + return (VDI_GetFd(vs->hosts[m].backend, sp)); + } + return (NULL); +} + /* * Healthy if just a single backend is... */ @@ -241,7 +275,10 @@ vrt_init(struct cli *cli, struct director **bp, int idx, vs->dir.priv = vs; vs->dir.name = "random"; REPLACE(vs->dir.vcl_name, t->name); - vs->dir.getfd = vdi_random_getfd; + if (criteria == c_least_connection) + vs->dir.getfd = vdi_least_connection_getfd; + else + vs->dir.getfd = vdi_random_getfd; vs->dir.fini = vdi_random_fini; vs->dir.healthy = vdi_random_healthy; @@ -283,3 +320,10 @@ VRT_init_dir_client(struct cli *cli, struct director **bp, int idx, { vrt_init(cli, bp, idx, priv, c_client); } + +void +VRT_init_dir_least_connection(struct cli *cli, struct director **bp, int idx, + const void *priv) +{ + vrt_init(cli, bp, idx, priv, c_least_connection); +} diff --git a/bin/varnishtest/tests/v00037.vtc b/bin/varnishtest/tests/v00037.vtc new file mode 100644 index 0000000..f392f6f --- /dev/null +++ b/bin/varnishtest/tests/v00037.vtc @@ -0,0 +1,86 @@ +varnishtest "Test least-connection director" + +server s1 { + rxreq + sema r1 sync 3 + sema r1 sync 2 + txresp -hdr "be: s1" + close + + accept + rxreq + txresp -hdr "be: s1" +} -start + +server s2 { + rxreq + sema r1 sync 3 + txresp -hdr "be: s2" + close + + accept + rxreq + txresp -hdr "be: s2" +} -start + +server s3 { + rxreq + txresp -hdr "be: s3" + close + + accept + rxreq + txresp -hdr "be: s3" +} -start + +varnish v1 -vcl+backend { + director lc least-connection { + { .backend = s1; .weight = 1; } + { .backend = s2; .weight = 1; } + { .backend = s3; .weight = 1; } + } + sub vcl_recv { + set req.backend = lc; + return (pass); + } +} -start + +client c1 { + txreq + rxresp + expect resp.status == 200 +} -start + +client c2 { + txreq + rxresp + expect resp.status == 200 +} -start + +delay .5 + +client c3 { + txreq + rxresp + expect resp.status == 200 + + txreq + rxresp + expect resp.http.be == "s3" + + sema r1 sync 3 + + txreq + rxresp + expect resp.http.be == "s2" + + sema r1 sync 2 + + txreq + rxresp + expect resp.http.be == "s1" +} -start + +client c1 -wait +client c2 -wait +client c3 -wait diff --git a/doc/sphinx/reference/vcl.rst b/doc/sphinx/reference/vcl.rst index 2c185ed..68be210 100644 --- a/doc/sphinx/reference/vcl.rst +++ b/doc/sphinx/reference/vcl.rst @@ -275,6 +275,17 @@ An example of a fallback director:: // are unhealthy. } +The least-connection director +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The least-connection director will pick a backend with the least number of +established connections. + +Each backend requires a .weight option which sets the amount of traffic +each backend will get compared to the others. Equal weight means equal +traffic. A backend with lower weight than an other will get proportionally +less traffic. + Backend probes -------------- diff --git a/lib/libvcl/vcc_backend.c b/lib/libvcl/vcc_backend.c index fb160e7..d1a8a31 100644 --- a/lib/libvcl/vcc_backend.c +++ b/lib/libvcl/vcc_backend.c @@ -688,6 +688,7 @@ static const struct dirlist { { "hash", vcc_ParseRandomDirector }, { "random", vcc_ParseRandomDirector }, { "client", vcc_ParseRandomDirector }, + { "least-connection", vcc_ParseRandomDirector }, { "round-robin", vcc_ParseRoundRobinDirector }, { "fallback", vcc_ParseRoundRobinDirector }, { "dns", vcc_ParseDnsDirector }, From martin at varnish-software.com Sun Jan 22 16:59:28 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Sun, 22 Jan 2012 17:59:28 +0100 Subject: [PATCH] Fix for #1086 - break out of loop in VGZ_WrwGunzip when end of gzip data condition is met Message-ID: <1327251568-21117-1-git-send-email-martin@varnish-software.com> Fix breaks out of the input data loop when end of gzip data is found, causing all subsequent input data to be ignored. --- bin/varnishd/cache/cache_gzip.c | 2 + bin/varnishtest/tests/r01086.vtc | 41 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 43 insertions(+), 0 deletions(-) create mode 100644 bin/varnishtest/tests/r01086.vtc diff --git a/bin/varnishd/cache/cache_gzip.c b/bin/varnishd/cache/cache_gzip.c index 1b732ca..61eb42c 100644 --- a/bin/varnishd/cache/cache_gzip.c +++ b/bin/varnishd/cache/cache_gzip.c @@ -380,6 +380,8 @@ VGZ_WrwGunzip(struct worker *wrk, struct vgz *vg, const void *ibuf, *obufp = 0; VGZ_Obuf(vg, obuf + *obufp, obufl - *obufp); } + if (i == VGZ_END) + break; } while (!VGZ_IbufEmpty(vg)); if (i == VGZ_STUCK) i = VGZ_OK; diff --git a/bin/varnishtest/tests/r01086.vtc b/bin/varnishtest/tests/r01086.vtc new file mode 100644 index 0000000..0fabfde --- /dev/null +++ b/bin/varnishtest/tests/r01086.vtc @@ -0,0 +1,41 @@ +varnishtest "#1086 junk after gzip from backend and streaming enabled" + +server s1 { + rxreq + txresp -nolen \ + -hdr "Content-Encoding: gzip" \ + -hdr "Transfer-Encoding: Chunked" + send "14\r\n" + # An empty gzip file: + sendhex "1f8b" + sendhex "08" + sendhex "00" + sendhex "00000000" + sendhex "00" + sendhex "03" + sendhex "0300" + sendhex "00000000" + sendhex "00000000" + send "\r\n" + + chunked "FOOBAR" + + chunkedlen 0 + +} -start + +varnish v1 -arg {-p diag_bitmap=0x00010000} -vcl+backend { + sub vcl_fetch { + set beresp.do_stream = true; + } +} + +varnish v1 -start + +client c1 { + txreq + rxresp + expect resp.status == 200 + expect resp.bodylen == 0 +} -run + -- 1.7.4.1 From martin at varnish-software.com Sun Jan 22 17:52:23 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Sun, 22 Jan 2012 18:52:23 +0100 Subject: Request for code review of streaming patches and merging into master Message-ID: The streaming implementation has reached a state now where I believe it should be considered for merging into master. I will post all of the patches to varnish-dev. There are some outstanding points left, outlined below. And also there will be some cache_center.c restructuring coming up, but it should be possible to do that without needing major rewriting of the streaming stuff, and having the code in master when starting on that should be an advantage. Any comments and inputs much appreciated. Outstanding streaming work items: * Streaming and client 304 If we can deliver a 304, we should for the pass-case just drop the backend connection (alternately do the bg fetch and throw away the result?). For the non-pass-case we should perform the bg fetch while still delivering a 304 to the client. The decision making will have to be influenced by wether we manage to grab a thread or not, and what to do when we don't have a 2nd thread has to be decided. This is best implemented after the vcl changes and cache_center.c changes coming from that. * Streaming and range Currently ranged delivery isn't performed when doing streaming, full object is always delivered when do_stream is true. This should be fixed. Some patches from the 3.0 streaming by Thomas Souvignet has been mailed to varnish-dev. Should be possible to use this as input when adding it to the new streaming implementation. * Break backend connection when streaming, pass-mode and client goes away If we loose the client while streaming in pass-mode we should probably just drop the backend connection as well. * Adaptive streaming I don't think it makes sense to do streaming for very small objects. For things less than a kilobyte, by the time we have the 2nd thread ready the full object is probably already on the wire and it would be faster to just read this in full before delivering the object. This is easy if there is a content-length header, we could just test this against a configurable minimum size. This decision policy probably makes the much sense in VCL? For chunked/eof transfer from backend, we can't know the size at decision making time. We could perhaps implement that streaming is first started after having received a minimum number of bytes? -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at varnish-software.com Sun Jan 22 17:53:07 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Sun, 22 Jan 2012 18:53:07 +0100 Subject: [PATCH 01/25] Functionality for a worker to grab another worker to perform a background task. Will be used for streamed fetching. Message-ID: <1327254811-29629-1-git-send-email-martin@varnish-software.com> Thread grabbing statistics --- bin/varnishd/cache/cache.h | 16 ++++++ bin/varnishd/cache/cache_pool.c | 106 +++++++++++++++++++++++++++++++++++- bin/varnishd/cache/cache_session.c | 21 +++++++ bin/varnishd/cache/cache_wrk.c | 18 ++++++ include/tbl/vsc_f_main.h | 15 +++++ 5 files changed, 174 insertions(+), 2 deletions(-) diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 817b7c7..11f0f6a 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -294,8 +294,11 @@ enum e_do_what { pool_do_accept, pool_do_nothing, pool_do_die, + pool_do_task, }; +typedef void taskfunc(struct worker *, void *priv); + struct worker { unsigned magic; #define WORKER_MAGIC 0x6391adcf @@ -331,6 +334,15 @@ struct worker { /* Stream state */ struct stream_ctx *sctx; + /* Task */ + taskfunc *taskfunc; + void *taskpriv; + + /* ESI delivery stuff */ + int gzip_resp; + ssize_t l_crc; + uint32_t crc; + /* Timeouts */ double connect_timeout; double first_byte_timeout; @@ -894,6 +906,8 @@ void PipeSession(struct sess *sp); void Pool_Init(void); void Pool_Work_Thread(void *priv, struct worker *w); int Pool_Schedule(struct pool *pp, struct sess *sp); +struct worker *Pool_GrabWorker(struct pool *pp, struct worker *wrk, + unsigned timeout); #define WRW_IsReleased(w) ((w)->wrw.wfd == NULL) int WRW_Error(const struct worker *w); @@ -920,6 +934,7 @@ int SES_Schedule(struct sess *sp); void SES_Handle(struct sess *sp, double now); void SES_GetReq(struct sess *sp); void SES_ReleaseReq(struct sess *sp); +struct worker *SES_GrabWorker(struct sess *sp, unsigned timeout); /* cache_shmlog.c */ extern struct VSC_C_main *VSC_C_main; @@ -1007,6 +1022,7 @@ void *WRK_thread(void *priv); typedef void *bgthread_t(struct sess *, void *priv); void WRK_BgThread(pthread_t *thr, const char *name, bgthread_t *func, void *priv); +void WRK_DoTask(struct worker *wrk, taskfunc *func, void *priv); /* cache_ws.c */ diff --git a/bin/varnishd/cache/cache_pool.c b/bin/varnishd/cache/cache_pool.c index 84a6cfd..89c96ec 100644 --- a/bin/varnishd/cache/cache_pool.c +++ b/bin/varnishd/cache/cache_pool.c @@ -89,6 +89,14 @@ struct poolsock { struct listen_sock *lsock; }; +struct grabber { + unsigned magic; +#define GRABBER_MAGIC 0x1f6f53f5 + VTAILQ_ENTRY(grabber) list; + struct worker *wrk_waiting; + struct worker *wrk_grabbed; +}; + /* Number of work requests queued in excess of worker threads available */ struct pool { @@ -104,6 +112,7 @@ struct pool { struct workerhead idle; VTAILQ_HEAD(, sess) queue; VTAILQ_HEAD(, poolsock) socks; + VTAILQ_HEAD(, grabber) grabqueue; unsigned nthr; unsigned lqueue; unsigned last_lqueue; @@ -183,6 +192,7 @@ Pool_Work_Thread(void *priv, struct worker *wrk) struct pool *pp; int stats_clean, i; struct poolsock *ps; + struct grabber *grabber; CAST_OBJ_NOTNULL(pp, priv, POOL_MAGIC); wrk->pool = pp; @@ -197,8 +207,22 @@ Pool_Work_Thread(void *priv, struct worker *wrk) WS_Reset(wrk->ws, NULL); - wrk->sp = VTAILQ_FIRST(&pp->queue); - if (wrk->sp != NULL) { + grabber = VTAILQ_FIRST(&pp->grabqueue); + if (grabber == NULL) + wrk->sp = VTAILQ_FIRST(&pp->queue); + if (grabber != NULL) { + /* We've been grabbed */ + assert(pp->lqueue > 0); + VTAILQ_REMOVE(&pp->grabqueue, grabber, list); + pp->lqueue--; + CHECK_OBJ_NOTNULL(grabber, GRABBER_MAGIC); + CHECK_OBJ_NOTNULL(grabber->wrk_waiting, WORKER_MAGIC); + AZ(grabber->wrk_grabbed); + grabber->wrk_grabbed = wrk; + AZ(pthread_cond_signal(&grabber->wrk_waiting->cond)); + /* Wait for work to be assigned to us */ + (void)Lck_CondWait(&wrk->cond, &pp->mtx, NULL); + } else if (wrk->sp != NULL) { /* Process queued requests, if any */ assert(pp->lqueue > 0); VTAILQ_REMOVE(&pp->queue, wrk->sp, list); @@ -271,6 +295,18 @@ Pool_Work_Thread(void *priv, struct worker *wrk) if (wrk->vcl != NULL) VCL_Rel(&wrk->vcl); } + } else if (wrk->do_what == pool_do_task) { + stats_clean = 0; + wrk->lastused = NAN; + + AZ(wrk->sp); + AN(wrk->taskfunc); + wrk->taskfunc(wrk, wrk->taskpriv); + wrk->taskfunc = NULL; + wrk->taskpriv = NULL; + + WS_Assert(wrk->ws); + AZ(wrk->busyobj); } else if (wrk->do_what == pool_do_nothing) { /* we already did */ } else { @@ -334,6 +370,71 @@ Pool_Schedule(struct pool *pp, struct sess *sp) } /*-------------------------------------------------------------------- + * Grab an idle worker to do a task, waiting up to timeout + * milliseconds for one to become idle. + * Returns: + * pointer to an idle worker + * NULL on failure to acquire idle worker within timeout + */ + +struct worker * +Pool_GrabWorker(struct pool *pp, struct worker *wrk, unsigned timeout) +{ + struct grabber grabber; + struct timespec ts; + + CHECK_OBJ_NOTNULL(pp, POOL_MAGIC); + CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); + + memset(&grabber, 0, sizeof grabber); + grabber.magic = GRABBER_MAGIC; + + Lck_Lock(&pp->mtx); + + grabber.wrk_grabbed = VTAILQ_FIRST(&pp->idle); + if (grabber.wrk_grabbed != NULL) { + VTAILQ_REMOVE(&pp->idle, grabber.wrk_grabbed, list); + VSC_C_main->threadgrab_idle++; + } + + if (grabber.wrk_grabbed == NULL && timeout > 0 && + pp->lqueue < (cache_param->queue_max * pp->nthr) / 100) { + /* Wait up to timeout milliseconds for a worker to + * become idle */ + AZ(clock_gettime(CLOCK_REALTIME, &ts)); + ts.tv_sec += timeout / 1000; + ts.tv_nsec += (timeout % 1000) * 1000000; + if (ts.tv_nsec >= 1000000000) { + ts.tv_sec++; + ts.tv_nsec -= 1000000000; + } + grabber.wrk_waiting = wrk; + VTAILQ_INSERT_TAIL(&pp->grabqueue, &grabber, list); + pp->lqueue++; + AZ(pthread_cond_signal(&pp->herder_cond)); + (void)Lck_CondWait(&wrk->cond, &pp->mtx, &ts); + if (grabber.wrk_grabbed != NULL) { + VSC_C_main->threadgrab_waiting++; + } else { + VTAILQ_REMOVE(&pp->grabqueue, &grabber, list); + pp->lqueue--; + } + } + + if (grabber.wrk_grabbed != NULL) { + CHECK_OBJ_NOTNULL(grabber.wrk_grabbed, WORKER_MAGIC); + AZ(grabber.wrk_grabbed->taskfunc); + AZ(grabber.wrk_grabbed->taskpriv); + } else { + VSC_C_main->threadgrab_failed++; + } + + Lck_Unlock(&pp->mtx); + + return (grabber.wrk_grabbed); +} + +/*-------------------------------------------------------------------- * Create another thread, if necessary & possible */ @@ -480,6 +581,7 @@ pool_mkpool(unsigned pool_no) XXXAN(pp); Lck_New(&pp->mtx, lck_wq); + VTAILQ_INIT(&pp->grabqueue); VTAILQ_INIT(&pp->queue); VTAILQ_INIT(&pp->idle); VTAILQ_INIT(&pp->socks); diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c index d1dbaba..ca93bb5 100644 --- a/bin/varnishd/cache/cache_session.c +++ b/bin/varnishd/cache/cache_session.c @@ -159,6 +159,27 @@ SES_Schedule(struct sess *sp) } /*-------------------------------------------------------------------- + * Grab an idle worker to do a task, waiting up to timeout + * milliseconds for one to become idle. + * Returns: + * pointer to an idle woker + * NULL on failure to acquire idle worker within timeout + */ + +struct worker * +SES_GrabWorker(struct sess *sp, unsigned timeout) +{ + struct sesspool *pp; + + CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); + pp = sp->sesspool; + CHECK_OBJ_NOTNULL(pp, SESSPOOL_MAGIC); + AN(pp->pool); + + return (Pool_GrabWorker(pp->pool, sp->wrk, timeout)); +} + +/*-------------------------------------------------------------------- * Handle a session (from waiter) */ diff --git a/bin/varnishd/cache/cache_wrk.c b/bin/varnishd/cache/cache_wrk.c index 6087991..e866aa1 100644 --- a/bin/varnishd/cache/cache_wrk.c +++ b/bin/varnishd/cache/cache_wrk.c @@ -128,6 +128,24 @@ WRK_BgThread(pthread_t *thr, const char *name, bgthread_t *func, void *priv) AZ(pthread_create(thr, NULL, wrk_bgthread, bt)); } +/*-------------------------------------------------------------------- + * Make the referenced worker execute the given task + */ + +void +WRK_DoTask(struct worker *wrk, taskfunc *func, void *priv) +{ + CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); + AN(func); + + AZ(wrk->taskfunc); + AZ(wrk->taskpriv); + wrk->taskfunc = func; + wrk->taskpriv = priv; + wrk->do_what = pool_do_task; + AZ(pthread_cond_signal(&wrk->cond)); +} + /*--------------------------------------------------------------------*/ static void * diff --git a/include/tbl/vsc_f_main.h b/include/tbl/vsc_f_main.h index df7984a..b6b8dc9 100644 --- a/include/tbl/vsc_f_main.h +++ b/include/tbl/vsc_f_main.h @@ -193,6 +193,21 @@ VSC_F(thread_queue_len, uint64_t, 0, 'g', " See also param queue_max." ) +VSC_F(threadgrab_idle, uint64_t, 0, 'c', + "Threads grabbed while idle", + "Number of times an idle thread was grabbed." +) + +VSC_F(threadgrab_waiting, uint64_t, 0, 'c', + "Threads grabbed while waiting", + "Number of times an idle thread was grabbed while waiting." +) + +VSC_F(threadgrab_failed, uint64_t, 0, 'c', + "Thread grab failed", + "Number of times failed to grab a thread within the timeout." +) + VSC_F(sess_queued, uint64_t, 0, 'c', "Sessions queued for thread", "Number of times session was queued waiting for a thread." -- 1.7.4.1 From martin at varnish-software.com Sun Jan 22 17:53:09 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Sun, 22 Jan 2012 18:53:09 +0100 Subject: [PATCH 03/25] BusyObj handling improvements In-Reply-To: <1327254811-29629-1-git-send-email-martin@varnish-software.com> References: <1327254811-29629-1-git-send-email-martin@varnish-software.com> Message-ID: <1327254811-29629-3-git-send-email-martin@varnish-software.com> Fix double allocations of busyobj in some cases Make VBO_DerefBusyObj return refcount after decrementation Make VBO_RefBusyObj return a pointer to the ref'ed busyobj Make objcore's have a refcount on the busyobj Drop const from VBO_RefBusyObj arg BusyObj lock-less mode --- bin/varnishd/cache/cache.h | 5 +++-- bin/varnishd/cache/cache_busyobj.c | 22 +++++++++++++++------- bin/varnishd/cache/cache_center.c | 36 +++++++++++++++++++----------------- bin/varnishd/cache/cache_hash.c | 10 ++++++++-- 4 files changed, 45 insertions(+), 28 deletions(-) diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 7fcd41e..4f260e9 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -501,6 +501,7 @@ struct busyobj { unsigned magic; #define BUSYOBJ_MAGIC 0x23b95567 struct vbo *vbo; + unsigned use_locks; uint8_t *vary; unsigned is_gzip; @@ -733,8 +734,8 @@ double BAN_Time(const struct ban *ban); /* cache_busyobj.c */ void VBO_Init(void); struct busyobj *VBO_GetBusyObj(struct worker *wrk); -void VBO_RefBusyObj(const struct busyobj *busyobj); -void VBO_DerefBusyObj(struct worker *wrk, struct busyobj **busyobj); +struct busyobj *VBO_RefBusyObj(struct busyobj *busyobj); +unsigned VBO_DerefBusyObj(struct worker *wrk, struct busyobj **busyobj); void VBO_Free(struct vbo **vbo); /* cache_center.c [CNT] */ diff --git a/bin/varnishd/cache/cache_busyobj.c b/bin/varnishd/cache/cache_busyobj.c index ab4db76..b8b6fd7 100644 --- a/bin/varnishd/cache/cache_busyobj.c +++ b/bin/varnishd/cache/cache_busyobj.c @@ -144,21 +144,25 @@ VBO_GetBusyObj(struct worker *wrk) return (&vbo->bo); } -void -VBO_RefBusyObj(const struct busyobj *busyobj) +struct busyobj * +VBO_RefBusyObj(struct busyobj *busyobj) { struct vbo *vbo; CHECK_OBJ_NOTNULL(busyobj, BUSYOBJ_MAGIC); vbo = busyobj->vbo; CHECK_OBJ_NOTNULL(vbo, VBO_MAGIC); - Lck_Lock(&vbo->mtx); + if (busyobj->use_locks) + Lck_Lock(&vbo->mtx); assert(vbo->refcount > 0); vbo->refcount++; - Lck_Unlock(&vbo->mtx); + if (busyobj->use_locks) + Lck_Unlock(&vbo->mtx); + + return (busyobj); } -void +unsigned VBO_DerefBusyObj(struct worker *wrk, struct busyobj **pbo) { struct busyobj *bo; @@ -172,10 +176,12 @@ VBO_DerefBusyObj(struct worker *wrk, struct busyobj **pbo) CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC); vbo = bo->vbo; CHECK_OBJ_NOTNULL(vbo, VBO_MAGIC); - Lck_Lock(&vbo->mtx); + if (bo->use_locks) + Lck_Lock(&vbo->mtx); assert(vbo->refcount > 0); r = --vbo->refcount; - Lck_Unlock(&vbo->mtx); + if (bo->use_locks) + Lck_Unlock(&vbo->mtx); if (r == 0) { /* XXX: Sanity checks & cleanup */ @@ -196,4 +202,6 @@ VBO_DerefBusyObj(struct worker *wrk, struct busyobj **pbo) VBO_Free(&vbo); } } + + return (r); } diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c index f9908b3..2909212 100644 --- a/bin/varnishd/cache/cache_center.c +++ b/bin/varnishd/cache/cache_center.c @@ -119,6 +119,7 @@ cnt_wait(struct sess *sp, struct worker *wrk, struct req *req) AZ(req->esi_level); assert(req->xid == 0); req->t_resp = NAN; + AZ(wrk->busyobj); assert(!isnan(sp->t_req)); tmo = (int)(1e3 * cache_param->timeout_linger); @@ -381,12 +382,11 @@ cnt_done(struct sess *sp, struct worker *wrk, struct req *req) CHECK_OBJ_ORNULL(req->vcl, VCL_CONF_MAGIC); AZ(req->obj); + AZ(req->objcore); AZ(wrk->busyobj); req->director = NULL; req->restarts = 0; - wrk->busyobj = NULL; - SES_Charge(sp); /* If we did an ESI include, don't mess up our state */ @@ -535,7 +535,7 @@ cnt_error(struct sess *sp, struct worker *wrk, struct req *req) if (req->handling == VCL_RET_RESTART && req->restarts < cache_param->max_restarts) { HSH_Drop(wrk); - VBO_DerefBusyObj(wrk, &wrk->busyobj); + (void)VBO_DerefBusyObj(wrk, &wrk->busyobj); req->director = NULL; req->restarts++; sp->step = STP_RECV; @@ -552,7 +552,7 @@ cnt_error(struct sess *sp, struct worker *wrk, struct req *req) req->err_code = 0; req->err_reason = NULL; http_Setup(wrk->busyobj->bereq, NULL); - VBO_DerefBusyObj(wrk, &wrk->busyobj); + (void)VBO_DerefBusyObj(wrk, &wrk->busyobj); sp->step = STP_PREPRESP; return (0); } @@ -680,7 +680,7 @@ cnt_fetch(struct sess *sp, struct worker *wrk, struct req *req) AZ(HSH_Deref(wrk, req->objcore, NULL)); req->objcore = NULL; } - VBO_DerefBusyObj(wrk, &wrk->busyobj); + (void)VBO_DerefBusyObj(wrk, &wrk->busyobj); req->director = NULL; req->storage_hint = NULL; @@ -852,7 +852,7 @@ cnt_fetchbody(struct sess *sp, struct worker *wrk, struct req *req) req->err_code = 503; sp->step = STP_ERROR; VDI_CloseFd(wrk, &wrk->busyobj->vbc); - VBO_DerefBusyObj(wrk, &wrk->busyobj); + (void)VBO_DerefBusyObj(wrk, &wrk->busyobj); return (0); } CHECK_OBJ_NOTNULL(req->obj, OBJECT_MAGIC); @@ -922,7 +922,7 @@ cnt_fetchbody(struct sess *sp, struct worker *wrk, struct req *req) if (i) { HSH_Drop(wrk); - VBO_DerefBusyObj(wrk, &wrk->busyobj); + (void)VBO_DerefBusyObj(wrk, &wrk->busyobj); AZ(req->obj); req->err_code = 503; sp->step = STP_ERROR; @@ -935,7 +935,7 @@ cnt_fetchbody(struct sess *sp, struct worker *wrk, struct req *req) AN(req->obj->objcore->ban); HSH_Unbusy(wrk); } - VBO_DerefBusyObj(wrk, &wrk->busyobj); + (void)VBO_DerefBusyObj(wrk, &wrk->busyobj); wrk->acct_tmp.fetch++; sp->step = STP_PREPRESP; return (0); @@ -1011,7 +1011,7 @@ cnt_streambody(struct sess *sp, struct worker *wrk, struct req *req) assert(WRW_IsReleased(wrk)); assert(wrk->wrw.ciov == wrk->wrw.siov); (void)HSH_Deref(wrk, NULL, &req->obj); - VBO_DerefBusyObj(wrk, &wrk->busyobj); + (void)VBO_DerefBusyObj(wrk, &wrk->busyobj); http_Setup(req->resp, NULL); sp->step = STP_DONE; return (0); @@ -1103,6 +1103,10 @@ cnt_hit(struct sess *sp, struct worker *wrk, struct req *req) /* Drop our object, we won't need it */ (void)HSH_Deref(wrk, NULL, &req->obj); req->objcore = NULL; + if (wrk->busyobj != NULL) { + CHECK_OBJ_NOTNULL(wrk->busyobj, BUSYOBJ_MAGIC); + (void)VBO_DerefBusyObj(wrk, &wrk->busyobj); + } switch(req->handling) { case VCL_RET_PASS: @@ -1274,7 +1278,6 @@ cnt_miss(struct sess *sp, struct worker *wrk, struct req *req) AN(req->objcore); CHECK_OBJ_NOTNULL(wrk->busyobj, BUSYOBJ_MAGIC); WS_Reset(wrk->ws, NULL); - wrk->busyobj = VBO_GetBusyObj(wrk); http_Setup(wrk->busyobj->bereq, wrk->ws); http_FilterReq(sp, HTTPH_R_FETCH); http_ForceGet(wrk->busyobj->bereq); @@ -1299,13 +1302,13 @@ cnt_miss(struct sess *sp, struct worker *wrk, struct req *req) AZ(HSH_Deref(wrk, req->objcore, NULL)); req->objcore = NULL; http_Setup(wrk->busyobj->bereq, NULL); - VBO_DerefBusyObj(wrk, &wrk->busyobj); + (void)VBO_DerefBusyObj(wrk, &wrk->busyobj); sp->step = STP_ERROR; return (0); case VCL_RET_PASS: AZ(HSH_Deref(wrk, req->objcore, NULL)); req->objcore = NULL; - VBO_DerefBusyObj(wrk, &wrk->busyobj); + (void)VBO_DerefBusyObj(wrk, &wrk->busyobj); sp->step = STP_PASS; return (0); case VCL_RET_FETCH: @@ -1315,7 +1318,7 @@ cnt_miss(struct sess *sp, struct worker *wrk, struct req *req) case VCL_RET_RESTART: AZ(HSH_Deref(wrk, req->objcore, NULL)); req->objcore = NULL; - VBO_DerefBusyObj(wrk, &wrk->busyobj); + (void)VBO_DerefBusyObj(wrk, &wrk->busyobj); INCOMPL(); default: WRONG("Illegal action in vcl_miss{}"); @@ -1364,7 +1367,6 @@ cnt_pass(struct sess *sp, struct worker *wrk, struct req *req) AZ(req->obj); AZ(wrk->busyobj); - wrk->busyobj = VBO_GetBusyObj(wrk); WS_Reset(wrk->ws, NULL); wrk->busyobj = VBO_GetBusyObj(wrk); http_Setup(wrk->busyobj->bereq, wrk->ws); @@ -1376,7 +1378,7 @@ cnt_pass(struct sess *sp, struct worker *wrk, struct req *req) VCL_pass_method(sp); if (req->handling == VCL_RET_ERROR) { http_Setup(wrk->busyobj->bereq, NULL); - VBO_DerefBusyObj(wrk, &wrk->busyobj); + (void)VBO_DerefBusyObj(wrk, &wrk->busyobj); sp->step = STP_ERROR; return (0); } @@ -1423,7 +1425,6 @@ cnt_pipe(struct sess *sp, struct worker *wrk, const struct req *req) AZ(wrk->busyobj); wrk->acct_tmp.pipe++; - wrk->busyobj = VBO_GetBusyObj(wrk); WS_Reset(wrk->ws, NULL); wrk->busyobj = VBO_GetBusyObj(wrk); http_Setup(wrk->busyobj->bereq, wrk->ws); @@ -1438,7 +1439,7 @@ cnt_pipe(struct sess *sp, struct worker *wrk, const struct req *req) PipeSession(sp); assert(WRW_IsReleased(wrk)); http_Setup(wrk->busyobj->bereq, NULL); - VBO_DerefBusyObj(wrk, &wrk->busyobj); + (void)VBO_DerefBusyObj(wrk, &wrk->busyobj); sp->step = STP_DONE; return (0); } @@ -1577,6 +1578,7 @@ cnt_start(struct sess *sp, struct worker *wrk, struct req *req) AZ(req->vcl); EXP_Clr(&req->exp); AZ(req->esi_level); + AZ(wrk->busyobj); /* Update stats of various sorts */ wrk->stats.client_req++; diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c index d967291..e78eb60 100644 --- a/bin/varnishd/cache/cache_hash.c +++ b/bin/varnishd/cache/cache_hash.c @@ -306,6 +306,7 @@ HSH_Lookup(struct sess *sp, struct objhead **poh) AN(sp->req->director); AN(hash); wrk = sp->wrk; + AZ(wrk->busyobj); HSH_Prealloc(sp); memcpy(sp->wrk->nobjhead->digest, sp->req->digest, @@ -459,7 +460,7 @@ HSH_Lookup(struct sess *sp, struct objhead **poh) wrk->busyobj->vary = sp->req->vary_b; else wrk->busyobj->vary = NULL; - oc->busyobj = wrk->busyobj; + oc->busyobj = VBO_RefBusyObj(wrk->busyobj); /* * Busy objects go on the tail, so they will not trip up searches. @@ -622,7 +623,8 @@ HSH_Unbusy(struct worker *wrk) VTAILQ_REMOVE(&oh->objcs, oc, list); VTAILQ_INSERT_HEAD(&oh->objcs, oc, list); oc->flags &= ~OC_F_BUSY; - oc->busyobj = NULL; + if (oc->busyobj != NULL) + (void)VBO_DerefBusyObj(wrk, &oc->busyobj); if (oh->waitinglist != NULL) hsh_rush(oh); AN(oc->ban); @@ -710,6 +712,10 @@ HSH_Deref(struct worker *wrk, struct objcore *oc, struct object **oo) BAN_DestroyObj(oc); AZ(oc->ban); + if (oc->busyobj != NULL) + (void)VBO_DerefBusyObj(wrk, &oc->busyobj); + AZ(oc->busyobj); + if (oc->methods != NULL) { oc_freeobj(oc); wrk->stats.n_object--; -- 1.7.4.1 From martin at varnish-software.com Sun Jan 22 17:53:08 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Sun, 22 Jan 2012 18:53:08 +0100 Subject: [PATCH 02/25] Make FetchBody take a busyobj struct as parameter instead of obj, and prepare the busyobj->fetch_obj before calling FetchBody. In-Reply-To: <1327254811-29629-1-git-send-email-martin@varnish-software.com> References: <1327254811-29629-1-git-send-email-martin@varnish-software.com> Message-ID: <1327254811-29629-2-git-send-email-martin@varnish-software.com> --- bin/varnishd/cache/cache.h | 2 +- bin/varnishd/cache/cache_center.c | 10 ++++++++-- bin/varnishd/cache/cache_fetch.c | 7 +++---- 3 files changed, 12 insertions(+), 7 deletions(-) diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 11f0f6a..7fcd41e 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -771,7 +771,7 @@ struct storage *FetchStorage(struct worker *w, ssize_t sz); int FetchError(struct worker *w, const char *error); int FetchError2(struct worker *w, const char *error, const char *more); int FetchHdr(struct sess *sp, int need_host_hdr); -int FetchBody(struct worker *w, struct object *obj); +int FetchBody(struct worker *w, struct busyobj *bo); int FetchReqBody(const struct sess *sp); void Fetch_Init(void); diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c index b03ba4b..f9908b3 100644 --- a/bin/varnishd/cache/cache_center.c +++ b/bin/varnishd/cache/cache_center.c @@ -908,7 +908,10 @@ cnt_fetchbody(struct sess *sp, struct worker *wrk, struct req *req) } /* Use unmodified headers*/ - i = FetchBody(wrk, req->obj); + AZ(wrk->busyobj->fetch_obj); + wrk->busyobj->fetch_obj = req->obj; + i = FetchBody(wrk, wrk->busyobj); + AZ(wrk->busyobj->fetch_obj); http_Setup(wrk->busyobj->bereq, NULL); http_Setup(wrk->busyobj->beresp, NULL); @@ -977,7 +980,10 @@ cnt_streambody(struct sess *sp, struct worker *wrk, struct req *req) AssertObjCorePassOrBusy(req->obj->objcore); - i = FetchBody(wrk, req->obj); + AZ(wrk->busyobj->fetch_obj); + wrk->busyobj->fetch_obj = req->obj; + i = FetchBody(wrk, wrk->busyobj); + AZ(wrk->busyobj->fetch_obj); http_Setup(wrk->busyobj->bereq, NULL); http_Setup(wrk->busyobj->beresp, NULL); diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index 7070612..06d23e6 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -484,20 +484,19 @@ FetchHdr(struct sess *sp, int need_host_hdr) /*--------------------------------------------------------------------*/ int -FetchBody(struct worker *wrk, struct object *obj) +FetchBody(struct worker *wrk, struct busyobj *bo) { int cls; struct storage *st; int mklen; ssize_t cl; struct http_conn *htc; - struct busyobj *bo; + struct object *obj; CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); - bo = wrk->busyobj; CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC); - AZ(bo->fetch_obj); CHECK_OBJ_NOTNULL(bo->vbc, VBC_MAGIC); + obj = bo->fetch_obj; CHECK_OBJ_NOTNULL(obj, OBJECT_MAGIC); CHECK_OBJ_NOTNULL(obj->http, HTTP_MAGIC); -- 1.7.4.1 From martin at varnish-software.com Sun Jan 22 17:53:10 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Sun, 22 Jan 2012 18:53:10 +0100 Subject: [PATCH 04/25] Add a (configurable through parameter stream_maxchunksize) chunk size limit when streaming. In-Reply-To: <1327254811-29629-1-git-send-email-martin@varnish-software.com> References: <1327254811-29629-1-git-send-email-martin@varnish-software.com> Message-ID: <1327254811-29629-4-git-send-email-martin@varnish-software.com> --- bin/varnishd/cache/cache_fetch.c | 2 ++ bin/varnishd/common/params.h | 1 + bin/varnishd/mgt/mgt_param.c | 9 +++++++++ 3 files changed, 12 insertions(+), 0 deletions(-) diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index 06d23e6..8cf9350 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -189,6 +189,8 @@ FetchStorage(struct worker *wrk, ssize_t sz) l = sz; if (l == 0) l = cache_param->fetch_chunksize; + if (wrk->busyobj->do_stream && l > cache_param->stream_maxchunksize) + l = cache_param->stream_maxchunksize; st = STV_alloc(wrk, l); if (st == NULL) { (void)FetchError(wrk, "Could not get storage"); diff --git a/bin/varnishd/common/params.h b/bin/varnishd/common/params.h index 6c1899a..f0445ae 100644 --- a/bin/varnishd/common/params.h +++ b/bin/varnishd/common/params.h @@ -98,6 +98,7 @@ struct params { /* Fetcher hints */ ssize_t fetch_chunksize; ssize_t fetch_maxchunksize; + ssize_t stream_maxchunksize; unsigned nuke_limit; #ifdef SENDFILE_WORKS diff --git a/bin/varnishd/mgt/mgt_param.c b/bin/varnishd/mgt/mgt_param.c index c984aae..227ea3a 100644 --- a/bin/varnishd/mgt/mgt_param.c +++ b/bin/varnishd/mgt/mgt_param.c @@ -837,6 +837,15 @@ static const struct parspec input_parspec[] = { "fragmentation.\n", EXPERIMENTAL, "256m", "bytes" }, + { "stream_maxchunksize", + tweak_bytes_u, + &mgt_param.stream_maxchunksize, 4 * 1024, UINT_MAX, + "The maximum chunksize we attempt to allocate from storage " + "when streaming. This also defines the intervals at which " + "the streaming clients receive notifications about new " + "data available.\n", + EXPERIMENTAL, + "256k", "bytes" }, #ifdef SENDFILE_WORKS { "sendfile_threshold", tweak_bytes, &mgt_param.sendfile_threshold, 0, HUGE_VAL, -- 1.7.4.1 From martin at varnish-software.com Sun Jan 22 17:53:12 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Sun, 22 Jan 2012 18:53:12 +0100 Subject: [PATCH 06/25] Don't try to stream when there is no body. In-Reply-To: <1327254811-29629-1-git-send-email-martin@varnish-software.com> References: <1327254811-29629-1-git-send-email-martin@varnish-software.com> Message-ID: <1327254811-29629-6-git-send-email-martin@varnish-software.com> --- bin/varnishd/cache/cache_center.c | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c index f074c4b..55e06c6 100644 --- a/bin/varnishd/cache/cache_center.c +++ b/bin/varnishd/cache/cache_center.c @@ -811,6 +811,10 @@ cnt_fetchbody(struct sess *sp, struct worker *wrk, struct req *req) if (!req->wantbody) wrk->busyobj->do_stream = 0; + /* No reason to try streaming a non-existing body */ + if (wrk->busyobj->body_status == BS_NONE) + wrk->busyobj->do_stream = 0; + l = http_EstimateWS(wrk->busyobj->beresp, pass ? HTTPH_R_PASS : HTTPH_A_INS, &nhttp); -- 1.7.4.1 From martin at varnish-software.com Sun Jan 22 17:53:13 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Sun, 22 Jan 2012 18:53:13 +0100 Subject: [PATCH 07/25] Don't free the object store when fetch fails and streaming is in effect, as there might be threads reading this data. In-Reply-To: <1327254811-29629-1-git-send-email-martin@varnish-software.com> References: <1327254811-29629-1-git-send-email-martin@varnish-software.com> Message-ID: <1327254811-29629-7-git-send-email-martin@varnish-software.com> --- bin/varnishd/cache/cache_fetch.c | 13 ++++++++----- 1 files changed, 8 insertions(+), 5 deletions(-) diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index b9e7ef0..f69d80f 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -580,11 +580,14 @@ FetchBody(struct worker *wrk, struct busyobj *bo) if (cls < 0) { wrk->stats.fetch_failed++; - /* XXX: Wouldn't this store automatically be released ? */ - while (!VTAILQ_EMPTY(&obj->store)) { - st = VTAILQ_FIRST(&obj->store); - VTAILQ_REMOVE(&obj->store, st, list); - STV_free(st); + if (bo->do_stream == 0) { + /* XXX: Wouldn't this store automatically be + * released ? */ + while (!VTAILQ_EMPTY(&obj->store)) { + st = VTAILQ_FIRST(&obj->store); + VTAILQ_REMOVE(&obj->store, st, list); + STV_free(st); + } } VDI_CloseFd(wrk, &bo->vbc); obj->len = 0; -- 1.7.4.1 From martin at varnish-software.com Sun Jan 22 17:53:14 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Sun, 22 Jan 2012 18:53:14 +0100 Subject: [PATCH 08/25] Add a condvar to struct vbo In-Reply-To: <1327254811-29629-1-git-send-email-martin@varnish-software.com> References: <1327254811-29629-1-git-send-email-martin@varnish-software.com> Message-ID: <1327254811-29629-8-git-send-email-martin@varnish-software.com> --- bin/varnishd/cache/cache_busyobj.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/bin/varnishd/cache/cache_busyobj.c b/bin/varnishd/cache/cache_busyobj.c index b8b6fd7..423df93 100644 --- a/bin/varnishd/cache/cache_busyobj.c +++ b/bin/varnishd/cache/cache_busyobj.c @@ -42,6 +42,7 @@ struct vbo { unsigned magic; #define VBO_MAGIC 0xde3d8223 struct lock mtx; + pthread_cond_t cond; unsigned refcount; uint16_t nhttp; struct busyobj bo; @@ -80,6 +81,7 @@ vbo_New(void) vbo->magic = VBO_MAGIC; vbo->nhttp = nhttp; Lck_New(&vbo->mtx, lck_busyobj); + AZ(pthread_cond_init(&vbo->cond, NULL)); return (vbo); } @@ -94,6 +96,7 @@ VBO_Free(struct vbo **vbop) CHECK_OBJ_NOTNULL(vbo, VBO_MAGIC); AZ(vbo->refcount); Lck_Delete(&vbo->mtx); + AZ(pthread_cond_destroy(&vbo->cond)); FREE_OBJ(vbo); } -- 1.7.4.1 From martin at varnish-software.com Sun Jan 22 17:53:16 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Sun, 22 Jan 2012 18:53:16 +0100 Subject: [PATCH 10/25] Rework RES_StreamPoll to use the VBO_StreamData and VBO_StreamSync functions, and make the object data access thread safe In-Reply-To: <1327254811-29629-1-git-send-email-martin@varnish-software.com> References: <1327254811-29629-1-git-send-email-martin@varnish-software.com> Message-ID: <1327254811-29629-10-git-send-email-martin@varnish-software.com> --- bin/varnishd/cache/cache.h | 4 +- bin/varnishd/cache/cache_response.c | 67 ++++++++++++++++++++++------------ 2 files changed, 45 insertions(+), 26 deletions(-) diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 0dd8ef4..b59a217 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -269,7 +269,7 @@ struct stream_ctx { /* Next byte we will take from storage */ ssize_t stream_next; - /* First byte of storage if we free it as we go (pass) */ + /* Point in storage chunk chain we have reached */ ssize_t stream_front; struct storage *stream_frontchunk; @@ -998,7 +998,7 @@ void RES_BuildHttp(const struct sess *sp); void RES_WriteObj(struct sess *sp); void RES_StreamStart(struct sess *sp); void RES_StreamEnd(struct sess *sp); -void RES_StreamPoll(struct worker *); +void RES_StreamPoll(struct worker *wrk); /* cache_vary.c */ struct vsb *VRY_Create(const struct sess *sp, const struct http *hp); diff --git a/bin/varnishd/cache/cache_response.c b/bin/varnishd/cache/cache_response.c index 86f4517..cb6ddd6 100644 --- a/bin/varnishd/cache/cache_response.c +++ b/bin/varnishd/cache/cache_response.c @@ -370,23 +370,39 @@ RES_StreamPoll(struct worker *wrk) { struct stream_ctx *sctx; struct storage *st; - ssize_t l, l2; + struct object *fetch_obj; + ssize_t l, l2, stlen; void *ptr; CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); - CHECK_OBJ_NOTNULL(wrk->busyobj->fetch_obj, OBJECT_MAGIC); + CHECK_OBJ_NOTNULL(wrk->sp->req->obj, OBJECT_MAGIC); sctx = wrk->sctx; CHECK_OBJ_NOTNULL(sctx, STREAM_CTX_MAGIC); - if (wrk->busyobj->fetch_obj->len == sctx->stream_next) + + VBO_StreamData(wrk->busyobj); + VBO_StreamSync(wrk); + + if (sctx->stream_max == sctx->stream_next) return; - assert(wrk->busyobj->fetch_obj->len > sctx->stream_next); + assert(sctx->stream_max > sctx->stream_next); + l = sctx->stream_front; - VTAILQ_FOREACH(st, &wrk->busyobj->fetch_obj->store, list) { - if (st->len + l <= sctx->stream_next) { - l += st->len; + st = sctx->stream_frontchunk; + if (st == NULL) + st = VTAILQ_FIRST(&wrk->sp->req->obj->store); + for (; st != NULL; st = VTAILQ_NEXT(st, list)) { + CHECK_OBJ_NOTNULL(st, STORAGE_MAGIC); + sctx->stream_front = l; + sctx->stream_frontchunk = st; + stlen = st->len; + if (l + stlen <= sctx->stream_next) { + l += stlen; continue; } - l2 = st->len + l - sctx->stream_next; + assert(l + stlen > sctx->stream_next); + l2 = l + stlen - sctx->stream_next; + if (sctx->stream_next + l2 > sctx->stream_max) + l2 = sctx->stream_max - sctx->stream_next; ptr = st->ptr + (sctx->stream_next - l); if (wrk->res_mode & RES_GUNZIP) { (void)VGZ_WrwGunzip(wrk, sctx->vgz, ptr, l2, @@ -394,27 +410,30 @@ RES_StreamPoll(struct worker *wrk) } else { (void)WRW_Write(wrk, ptr, l2); } - l += st->len; sctx->stream_next += l2; + if (sctx->stream_next == sctx->stream_max) + break; + AN(VTAILQ_NEXT(st, list)); + l += st->len; } if (!(wrk->res_mode & RES_GUNZIP)) (void)WRW_Flush(wrk); - if (wrk->busyobj->fetch_obj->objcore == NULL || - (wrk->busyobj->fetch_obj->objcore->flags & OC_F_PASS)) { - /* - * This is a pass object, release storage as soon as we - * have delivered it. - */ - while (1) { - st = VTAILQ_FIRST(&wrk->busyobj->fetch_obj->store); - if (st == NULL || - sctx->stream_front + st->len > sctx->stream_next) - break; - VTAILQ_REMOVE(&wrk->busyobj->fetch_obj->store, st, list); - sctx->stream_front += st->len; - STV_free(st); - } + if (wrk->busyobj->stream_frontchunk == NULL) + return; + + /* It's a pass - remove chunks already delivered */ + fetch_obj = wrk->busyobj->fetch_obj; + CHECK_OBJ_NOTNULL(fetch_obj, OBJECT_MAGIC); + assert(fetch_obj->objcore == NULL || + (fetch_obj->objcore->flags & OC_F_PASS)); + while (1) { + st = VTAILQ_FIRST(&fetch_obj->store); + if (st == NULL || st == wrk->busyobj->stream_frontchunk) + break; + CHECK_OBJ_NOTNULL(st, STORAGE_MAGIC); + VTAILQ_REMOVE(&fetch_obj->store, st, list); + STV_free(st); } } -- 1.7.4.1 From martin at varnish-software.com Sun Jan 22 17:53:25 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Sun, 22 Jan 2012 18:53:25 +0100 Subject: [PATCH 19/25] Flip-flop test case In-Reply-To: <1327254811-29629-1-git-send-email-martin@varnish-software.com> References: <1327254811-29629-1-git-send-email-martin@varnish-software.com> Message-ID: <1327254811-29629-19-git-send-email-martin@varnish-software.com> --- bin/varnishtest/tests/t00008.vtc | 45 ++++++++++++++++++++++++++++++++++++++ 1 files changed, 45 insertions(+), 0 deletions(-) create mode 100644 bin/varnishtest/tests/t00008.vtc diff --git a/bin/varnishtest/tests/t00008.vtc b/bin/varnishtest/tests/t00008.vtc new file mode 100644 index 0000000..076da71 --- /dev/null +++ b/bin/varnishtest/tests/t00008.vtc @@ -0,0 +1,45 @@ +varnishtest "Test flipflop streaming" + +server s1 { + rxreq + txresp -nolen -hdr "Transfer-encoding: chunked" + chunked "<1>------------------------<1>\n" + sema r1 sync 2 + chunked "<2>------------------------<2>\n" + sema r2 sync 2 + chunkedlen 0 +} -start + +varnish v1 -arg "-p thread_pools=1 -p thread_pool_min=2 -p thread_pool_max=2 -p stream_grab_timeout=0 -p thread_pool_add_threshold=1" -vcl+backend { + sub vcl_recv { + return (pass); + } + + sub vcl_fetch { + set beresp.do_stream = true; + } +} -start + +client c1 { + txreq -hdr "foo: /foo" + rxresp -no_obj + + rxchunk + expect resp.chunklen == 31 + sema r1 sync 2 + + rxchunk + expect resp.chunklen == 31 + sema r2 sync 2 + + rxchunk + expect resp.chunklen == 0 + expect resp.bodylen == 62 +} -run + +varnish v1 -expect threadgrab_idle == 0 +varnish v1 -expect threadgrab_waiting == 0 +varnish v1 -expect threadgrab_failed == 1 +varnish v1 -expect fetch_threaded == 0 +varnish v1 -expect fetch_flipflop == 1 +varnish v1 -expect s_streamed == 1 -- 1.7.4.1 From martin at varnish-software.com Sun Jan 22 17:53:27 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Sun, 22 Jan 2012 18:53:27 +0100 Subject: [PATCH 21/25] Add run-time parameter stream_pass_bufsize and VCL variable beresp.stream_pass_bufsize with test case In-Reply-To: <1327254811-29629-1-git-send-email-martin@varnish-software.com> References: <1327254811-29629-1-git-send-email-martin@varnish-software.com> Message-ID: <1327254811-29629-21-git-send-email-martin@varnish-software.com> --- bin/varnishd/cache/cache.h | 1 + bin/varnishd/cache/cache_busyobj.c | 2 ++ bin/varnishd/cache/cache_vrt_var.c | 19 +++++++++++++++++++ bin/varnishd/common/params.h | 2 ++ bin/varnishd/mgt/mgt_param.c | 6 ++++++ bin/varnishtest/tests/t00009.vtc | 33 +++++++++++++++++++++++++++++++++ lib/libvcl/generate.py | 6 ++++++ 7 files changed, 69 insertions(+), 0 deletions(-) create mode 100644 bin/varnishtest/tests/t00009.vtc diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 880f5e3..fe65dbd 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -543,6 +543,7 @@ struct busyobj { ssize_t stream_max; struct storage *stream_frontchunk; unsigned stream_stopped; + ssize_t stream_pass_bufsize; }; /* Object structure --------------------------------------------------*/ diff --git a/bin/varnishd/cache/cache_busyobj.c b/bin/varnishd/cache/cache_busyobj.c index 6eec464..b39c170 100644 --- a/bin/varnishd/cache/cache_busyobj.c +++ b/bin/varnishd/cache/cache_busyobj.c @@ -144,6 +144,8 @@ VBO_GetBusyObj(struct worker *wrk) p += HTTP_estimate(vbo->nhttp); vbo->bo.beresp = HTTP_create(p, vbo->nhttp); + vbo->bo.stream_pass_bufsize = cache_param->stream_pass_bufsize; + return (&vbo->bo); } diff --git a/bin/varnishd/cache/cache_vrt_var.c b/bin/varnishd/cache/cache_vrt_var.c index e051dbb..e655abf 100644 --- a/bin/varnishd/cache/cache_vrt_var.c +++ b/bin/varnishd/cache/cache_vrt_var.c @@ -301,6 +301,25 @@ VRT_l_beresp_storage(struct sess *sp, const char *str, ...) sp->req->storage_hint = b; } +double +VRT_r_beresp_stream_pass_bufsize(const struct sess *sp) +{ + CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); + CHECK_OBJ_NOTNULL(sp->wrk->busyobj, BUSYOBJ_MAGIC); + return (sp->wrk->busyobj->stream_pass_bufsize); +} + +void +VRT_l_beresp_stream_pass_bufsize(const struct sess *sp, double val) +{ + CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); + CHECK_OBJ_NOTNULL(sp->wrk->busyobj, BUSYOBJ_MAGIC); + if (val >= 0.) + sp->wrk->busyobj->stream_pass_bufsize = val; + else + sp->wrk->busyobj->stream_pass_bufsize = 0; +} + /*--------------------------------------------------------------------*/ void diff --git a/bin/varnishd/common/params.h b/bin/varnishd/common/params.h index f726ce8..20b346a 100644 --- a/bin/varnishd/common/params.h +++ b/bin/varnishd/common/params.h @@ -100,6 +100,8 @@ struct params { ssize_t fetch_maxchunksize; ssize_t stream_maxchunksize; unsigned stream_grab_timeout; + ssize_t stream_pass_bufsize; + unsigned nuke_limit; #ifdef SENDFILE_WORKS diff --git a/bin/varnishd/mgt/mgt_param.c b/bin/varnishd/mgt/mgt_param.c index 9b1467f..3a64b96 100644 --- a/bin/varnishd/mgt/mgt_param.c +++ b/bin/varnishd/mgt/mgt_param.c @@ -852,6 +852,12 @@ static const struct parspec input_parspec[] = { "grabbing a thread for background fetch.\n", EXPERIMENTAL, "100", "milliseconds" }, + { "stream_pass_bufsize", + tweak_bytes_u, &mgt_param.stream_pass_bufsize, 0, UINT_MAX, + "Default max amount of data to buffer when streaming passes. " + "Zero means unlimited.\n", + EXPERIMENTAL, + "10mb", "bytes" }, #ifdef SENDFILE_WORKS { "sendfile_threshold", tweak_bytes, &mgt_param.sendfile_threshold, 0, HUGE_VAL, diff --git a/bin/varnishtest/tests/t00009.vtc b/bin/varnishtest/tests/t00009.vtc new file mode 100644 index 0000000..834ba80 --- /dev/null +++ b/bin/varnishtest/tests/t00009.vtc @@ -0,0 +1,33 @@ +varnishtest "Test streaming vcl syntax" + +server s1 { + rxreq + txresp +} -start + +varnish v1 -arg "-p stream_pass_bufsize=1M" -vcl+backend { + sub vcl_fetch { + set beresp.do_stream = true; + set beresp.http.spb-orig = beresp.stream_pass_bufsize; + set beresp.stream_pass_bufsize = 1024B; + set beresp.stream_pass_bufsize = 1KB; + set beresp.stream_pass_bufsize = 10MB; + set beresp.stream_pass_bufsize = 1GB; + set beresp.stream_pass_bufsize = 0B - 1B; + set beresp.http.spb = beresp.stream_pass_bufsize; + } +} -start + +client c1 { + txreq + rxresp + expect resp.http.spb-orig == "1048576.000" + expect resp.http.spb == "0.000" +} -run + +varnish v1 -badvcl { + backend default { .host = "127.0.0.1"; } + sub vcl_fetch { + set beresp.stream_pass_bufsize = 0; + } +} diff --git a/lib/libvcl/generate.py b/lib/libvcl/generate.py index b91f675..51b2294 100755 --- a/lib/libvcl/generate.py +++ b/lib/libvcl/generate.py @@ -331,6 +331,12 @@ sp_variables = ( ( 'fetch',), 'const struct sess *' ), + ('beresp.stream_pass_bufsize', + 'BYTES', + ( 'fetch',), + ( 'fetch',), + 'const struct sess *' + ), ('beresp.ttl', 'DURATION', ( 'fetch',), -- 1.7.4.1 From martin at varnish-software.com Sun Jan 22 17:53:29 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Sun, 22 Jan 2012 18:53:29 +0100 Subject: [PATCH 23/25] Add test case for pass bufsize throttling In-Reply-To: <1327254811-29629-1-git-send-email-martin@varnish-software.com> References: <1327254811-29629-1-git-send-email-martin@varnish-software.com> Message-ID: <1327254811-29629-23-git-send-email-martin@varnish-software.com> --- bin/varnishtest/tests/t00010.vtc | 68 ++++++++++++++++++++++++++++++++++++++ 1 files changed, 68 insertions(+), 0 deletions(-) create mode 100644 bin/varnishtest/tests/t00010.vtc diff --git a/bin/varnishtest/tests/t00010.vtc b/bin/varnishtest/tests/t00010.vtc new file mode 100644 index 0000000..eec3a5d --- /dev/null +++ b/bin/varnishtest/tests/t00010.vtc @@ -0,0 +1,68 @@ +varnishtest "Test streaming recepient in pass mode with low bufsize" + +server s1 { + rxreq + txresp -nolen -hdr "Transfer-encoding: chunked" + chunkedlen 1024 + sema r1 sync 2 + chunkedlen 1024 + sema r2 sync 2 + chunkedlen 1024 + sema r1 sync 2 + chunkedlen 1024 + sema r2 sync 2 + chunkedlen 0 +} -repeat 2 -start + +# First test with flip-flop +varnish v1 -arg "-p thread_pools=1 -p thread_pool_min=2 -p thread_pool_max=2 -p stream_grab_timeout=0 -p thread_pool_add_threshold=1" -vcl+backend { + sub vcl_recv { + return (pass); + } + + sub vcl_fetch { + set beresp.do_stream = true; + set beresp.stream_pass_bufsize = 8B; + } +} -start + +client c1 { + txreq -hdr "foo: /foo" + rxresp -no_obj + + rxchunk + expect resp.chunklen == 1024 + sema r1 sync 2 + + rxchunk + expect resp.chunklen == 1024 + sema r2 sync 2 + + rxchunk + expect resp.chunklen == 1024 + sema r1 sync 2 + + rxchunk + expect resp.chunklen == 1024 + sema r2 sync 2 + + rxchunk + expect resp.chunklen == 0 + expect resp.bodylen == 4096 +} -run + +# Allow thread creation so next test will be with fetch thread +varnish v1 -cliok "param.set thread_pool_max 4" +varnish v1 -cliok "param.set thread_pool_add_threshold 0" +varnish v1 -cliok "param.set stream_grab_timeout 100" + +client c1 -run + +# Allow statistics to get ajour before testing +delay 0.1 + +varnish v1 -expect threadgrab_failed == 1 +varnish v1 -expect threadgrab_waiting == 1 +varnish v1 -expect fetch_threaded == 1 +varnish v1 -expect fetch_flipflop == 1 +varnish v1 -expect s_streamed == 2 -- 1.7.4.1 From martin at varnish-software.com Sun Jan 22 17:53:11 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Sun, 22 Jan 2012 18:53:11 +0100 Subject: [PATCH 05/25] Remove busyobj->h_content_length pointer into workspace In-Reply-To: <1327254811-29629-1-git-send-email-martin@varnish-software.com> References: <1327254811-29629-1-git-send-email-martin@varnish-software.com> Message-ID: <1327254811-29629-5-git-send-email-martin@varnish-software.com> The busyobj->h_content_length is a pointer to the header on the workspace of the fetch initiating worker. This is not safe when doing streaming. This patch removes busyobj->h_content_length and replaces it with a flag and an attribute instead. --- bin/varnishd/cache/cache.h | 4 +++- bin/varnishd/cache/cache_center.c | 3 +-- bin/varnishd/cache/cache_fetch.c | 9 +++++---- bin/varnishd/cache/cache_response.c | 5 +++-- bin/varnishd/cache/cache_rfc2616.c | 5 +++-- 5 files changed, 15 insertions(+), 11 deletions(-) diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 4f260e9..0bc3f59 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -523,7 +523,8 @@ struct busyobj { struct vef_priv *vef_priv; unsigned should_close; - char *h_content_length; + unsigned has_content_length; + ssize_t content_length; unsigned do_esi; unsigned do_gzip; @@ -768,6 +769,7 @@ int EXP_Touch(struct objcore *oc); int EXP_NukeOne(struct worker *w, struct lru *lru); /* cache_fetch.c */ +ssize_t FetchNumber(const char *nbr, int radix); struct storage *FetchStorage(struct worker *w, ssize_t sz); int FetchError(struct worker *w, const char *error); int FetchError2(struct worker *w, const char *error, const char *more); diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c index 2909212..f074c4b 100644 --- a/bin/varnishd/cache/cache_center.c +++ b/bin/varnishd/cache/cache_center.c @@ -233,8 +233,7 @@ cnt_prepresp(struct sess *sp, struct worker *wrk, struct req *req) wrk->res_mode |= RES_LEN; if (wrk->busyobj != NULL && - (wrk->busyobj->h_content_length != NULL || - !wrk->busyobj->do_stream) && + (wrk->busyobj->has_content_length || !wrk->busyobj->do_stream) && !wrk->busyobj->do_gzip && !wrk->busyobj->do_gunzip) wrk->res_mode |= RES_LEN; diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index 8cf9350..b9e7ef0 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -205,8 +205,8 @@ FetchStorage(struct worker *wrk, ssize_t sz) * Convert a string to a size_t safely */ -static ssize_t -fetch_number(const char *nbr, int radix) +ssize_t +FetchNumber(const char *nbr, int radix) { uintmax_t cll; ssize_t cl; @@ -291,7 +291,7 @@ fetch_chunked(struct worker *wrk, struct http_conn *htc) return (FetchError(wrk,"chunked header no NL")); buf[u] = '\0'; - cl = fetch_number(buf, 16); + cl = FetchNumber(buf, 16); if (cl < 0) return (FetchError(wrk,"chunked header number syntax")); @@ -527,7 +527,8 @@ FetchBody(struct worker *wrk, struct busyobj *bo) mklen = 1; break; case BS_LENGTH: - cl = fetch_number(bo->h_content_length, 10); + AN(bo->has_content_length); + cl = bo->content_length; bo->vfp->begin(wrk, cl > 0 ? cl : 0); cls = fetch_straight(wrk, htc, cl); mklen = 1; diff --git a/bin/varnishd/cache/cache_response.c b/bin/varnishd/cache/cache_response.c index 5724f07..86f4517 100644 --- a/bin/varnishd/cache/cache_response.c +++ b/bin/varnishd/cache/cache_response.c @@ -353,9 +353,10 @@ RES_StreamStart(struct sess *sp) http_Unset(sp->req->resp, H_Content_Encoding); if (!(sp->wrk->res_mode & RES_CHUNKED) && - sp->wrk->busyobj->h_content_length != NULL) + sp->wrk->busyobj->has_content_length && + !http_GetHdr(sp->req->resp, H_Content_Length, NULL)) http_PrintfHeader(sp->wrk, sp->vsl_id, sp->req->resp, - "Content-Length: %s", sp->wrk->busyobj->h_content_length); + "Content-Length: %zd", sp->wrk->busyobj->content_length); sp->wrk->acct_tmp.hdrbytes += http_Write(sp->wrk, sp->vsl_id, sp->req->resp, 1); diff --git a/bin/varnishd/cache/cache_rfc2616.c b/bin/varnishd/cache/cache_rfc2616.c index 1b67d81..59d4e2c 100644 --- a/bin/varnishd/cache/cache_rfc2616.c +++ b/bin/varnishd/cache/cache_rfc2616.c @@ -239,8 +239,9 @@ RFC2616_Body(const struct sess *sp) return (BS_ERROR); } - if (http_GetHdr(hp, H_Content_Length, - &sp->wrk->busyobj->h_content_length)) { + if (http_GetHdr(hp, H_Content_Length, &b)) { + sp->wrk->busyobj->has_content_length = 1; + sp->wrk->busyobj->content_length = FetchNumber(b, 10); sp->wrk->stats.fetch_length++; return (BS_LENGTH); } -- 1.7.4.1 From martin at varnish-software.com Sun Jan 22 17:53:20 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Sun, 22 Jan 2012 18:53:20 +0100 Subject: [PATCH 14/25] Add a couple of streaming test cases In-Reply-To: <1327254811-29629-1-git-send-email-martin@varnish-software.com> References: <1327254811-29629-1-git-send-email-martin@varnish-software.com> Message-ID: <1327254811-29629-14-git-send-email-martin@varnish-software.com> --- bin/varnishtest/tests/t00002.vtc | 42 ++++++++++++++++++++++++++++++++++++++ bin/varnishtest/tests/t00003.vtc | 37 +++++++++++++++++++++++++++++++++ 2 files changed, 79 insertions(+), 0 deletions(-) create mode 100644 bin/varnishtest/tests/t00002.vtc create mode 100644 bin/varnishtest/tests/t00003.vtc diff --git a/bin/varnishtest/tests/t00002.vtc b/bin/varnishtest/tests/t00002.vtc new file mode 100644 index 0000000..fa0e8de --- /dev/null +++ b/bin/varnishtest/tests/t00002.vtc @@ -0,0 +1,42 @@ +varnishtest "Test streaming recepient in pass mode" + +server s1 { + rxreq + txresp -nolen -hdr "Transfer-encoding: chunked" + chunked "<1>------------------------<1>\n" + sema r1 sync 2 + chunked "<2>------------------------<2>\n" + sema r2 sync 2 + chunkedlen 0 +} -start + +varnish v1 -vcl+backend { + sub vcl_recv { + return (pass); + } + + sub vcl_fetch { + set beresp.do_stream = true; + } +} -start + +client c1 { + txreq -hdr "foo: /foo" + rxresp -no_obj + + rxchunk + expect resp.chunklen == 31 + sema r1 sync 2 + + rxchunk + expect resp.chunklen == 31 + sema r2 sync 2 + + rxchunk + expect resp.chunklen == 0 + expect resp.bodylen == 62 +} -run + +varnish v1 -expect threadgrab_idle == 1 +varnish v1 -expect fetch_threaded == 1 +varnish v1 -expect s_streamed == 1 diff --git a/bin/varnishtest/tests/t00003.vtc b/bin/varnishtest/tests/t00003.vtc new file mode 100644 index 0000000..76deb2b --- /dev/null +++ b/bin/varnishtest/tests/t00003.vtc @@ -0,0 +1,37 @@ +varnishtest "Test failed streamed delivery not entering cache" + +server s1 { + rxreq + txresp -nolen -hdr "Transfer-encoding: chunked" + chunked "<1>------------------------<1>\n" + chunked "<2>------------------------<2>\n" + accept + rxreq + txresp -nolen -hdr "Transfer-encoding: chunked" + chunked "<1>------------------------<1>\n" + chunked "<2>------------------------<2>\n" + chunked "<3>------------------------<3>\n" + chunkedlen 0 +} -start + +varnish v1 -vcl+backend { + sub vcl_fetch { + set beresp.do_stream = true; + } +} -start + +client c1 { + txreq + rxresp + delay 1 +} -run + +client c2 { + txreq + rxresp + expect resp.bodylen == 93 +} -run + +varnish v1 -expect threadgrab_idle == 2 +varnish v1 -expect fetch_threaded == 2 +varnish v1 -expect s_streamed == 2 -- 1.7.4.1 From martin at varnish-software.com Sun Jan 22 17:53:31 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Sun, 22 Jan 2012 18:53:31 +0100 Subject: [PATCH 25/25] Test case for thundering horde algorithm In-Reply-To: <1327254811-29629-1-git-send-email-martin@varnish-software.com> References: <1327254811-29629-1-git-send-email-martin@varnish-software.com> Message-ID: <1327254811-29629-25-git-send-email-martin@varnish-software.com> --- bin/varnishtest/tests/t00011.vtc | 130 ++++++++++++++++++++++++++++++++++++++ 1 files changed, 130 insertions(+), 0 deletions(-) create mode 100644 bin/varnishtest/tests/t00011.vtc diff --git a/bin/varnishtest/tests/t00011.vtc b/bin/varnishtest/tests/t00011.vtc new file mode 100644 index 0000000..29cb889 --- /dev/null +++ b/bin/varnishtest/tests/t00011.vtc @@ -0,0 +1,130 @@ +varnishtest "Test multiple streaming recepients and low token count" + +server s1 { + rxreq + txresp -nolen -hdr "Transfer-encoding: chunked" + chunked "<1>------------------------<1>\n" + sema r1 sync 5 + chunked "<2>------------------------<2>\n" + sema r2 sync 5 + chunked "<3>------------------------<3>\n" + sema r1 sync 5 + chunked "<4>------------------------<4>\n" + sema r2 sync 5 + chunkedlen 0 +} -start + +varnish v1 -vcl+backend { + sub vcl_fetch { + set beresp.do_stream = true; + set beresp.stream_tokens = 1; + } +} -start + +client c1 { + txreq -hdr "foo: /foo" + rxresp -no_obj + + rxchunk + expect resp.chunklen == 31 + sema r1 sync 5 + + rxchunk + expect resp.chunklen == 31 + sema r2 sync 5 + + rxchunk + expect resp.chunklen == 31 + sema r1 sync 5 + + rxchunk + expect resp.chunklen == 31 + sema r2 sync 5 + + rxchunk + expect resp.chunklen == 0 + expect resp.bodylen == 124 +} -start + +client c2 { + txreq -hdr "foo: /foo" + rxresp -no_obj + + rxchunk + expect resp.chunklen == 31 + sema r1 sync 5 + + rxchunk + expect resp.chunklen == 31 + sema r2 sync 5 + + rxchunk + expect resp.chunklen == 31 + sema r1 sync 5 + + rxchunk + expect resp.chunklen == 31 + sema r2 sync 5 + + rxchunk + expect resp.chunklen == 0 + expect resp.bodylen == 124 +} -start + +client c3 { + txreq -hdr "foo: /foo" + rxresp -no_obj + + rxchunk + expect resp.chunklen == 31 + sema r1 sync 5 + + rxchunk + expect resp.chunklen == 31 + sema r2 sync 5 + + rxchunk + expect resp.chunklen == 31 + sema r1 sync 5 + + rxchunk + expect resp.chunklen == 31 + sema r2 sync 5 + + rxchunk + expect resp.chunklen == 0 + expect resp.bodylen == 124 +} -start + +client c4 { + txreq -hdr "foo: /foo" + rxresp -no_obj + + rxchunk + expect resp.chunklen == 31 + sema r1 sync 5 + + rxchunk + expect resp.chunklen == 31 + sema r2 sync 5 + + rxchunk + expect resp.chunklen == 31 + sema r1 sync 5 + + rxchunk + expect resp.chunklen == 31 + sema r2 sync 5 + + rxchunk + expect resp.chunklen == 0 + expect resp.bodylen == 124 +} -start + +client c1 -wait +client c2 -wait +client c3 -wait +client c4 -wait + +varnish v1 -expect fetch_threaded == 1 +varnish v1 -expect s_streamed == 4 -- 1.7.4.1 From martin at varnish-software.com Sun Jan 22 17:53:15 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Sun, 22 Jan 2012 18:53:15 +0100 Subject: [PATCH 09/25] Add stream data synchronization functions to cache_busyobj.c in preparation of threaded streaming. In-Reply-To: <1327254811-29629-1-git-send-email-martin@varnish-software.com> References: <1327254811-29629-1-git-send-email-martin@varnish-software.com> Message-ID: <1327254811-29629-9-git-send-email-martin@varnish-software.com> VBO_StreamData is called by the fetch to update the busyobj with how much data is available. VBO_StreamSync is called by the dilvery to update it's local stream_ctx struct with the new pointers. VBO_StreamStopped signals object fetch has finished. VBO_StreamWait waits for the streaming thread to finish looking at other thread's workspace. --- bin/varnishd/cache/cache.h | 16 ++++++ bin/varnishd/cache/cache_busyobj.c | 91 ++++++++++++++++++++++++++++++++++++ 2 files changed, 107 insertions(+), 0 deletions(-) diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 0bc3f59..0dd8ef4 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -271,6 +271,13 @@ struct stream_ctx { /* First byte of storage if we free it as we go (pass) */ ssize_t stream_front; + struct storage *stream_frontchunk; + + /* Max byte we can stream */ + ssize_t stream_max; + + /* Backend fetch has finished */ + unsigned stream_stopped; }; /*--------------------------------------------------------------------*/ @@ -530,6 +537,11 @@ struct busyobj { unsigned do_gzip; unsigned do_gunzip; unsigned do_stream; + + /* Stream stuff */ + ssize_t stream_max; + struct storage *stream_frontchunk; + unsigned stream_stopped; }; /* Object structure --------------------------------------------------*/ @@ -738,6 +750,10 @@ struct busyobj *VBO_GetBusyObj(struct worker *wrk); struct busyobj *VBO_RefBusyObj(struct busyobj *busyobj); unsigned VBO_DerefBusyObj(struct worker *wrk, struct busyobj **busyobj); void VBO_Free(struct vbo **vbo); +void VBO_StreamStopped(struct busyobj *busyobj); +void VBO_StreamWait(struct busyobj *busyobj); +void VBO_StreamData(struct busyobj *busyobj); +void VBO_StreamSync(struct worker *wrk); /* cache_center.c [CNT] */ void CNT_Session(struct sess *sp); diff --git a/bin/varnishd/cache/cache_busyobj.c b/bin/varnishd/cache/cache_busyobj.c index 423df93..7651999 100644 --- a/bin/varnishd/cache/cache_busyobj.c +++ b/bin/varnishd/cache/cache_busyobj.c @@ -208,3 +208,94 @@ VBO_DerefBusyObj(struct worker *wrk, struct busyobj **pbo) return (r); } + +/* Signal that the fetch thread has stopped */ +void +VBO_StreamStopped(struct busyobj *busyobj) +{ + if (!busyobj->use_locks) { + busyobj->stream_stopped = 1; + return; + } + Lck_Lock(&busyobj->vbo->mtx); + busyobj->stream_stopped = 1; + AZ(pthread_cond_broadcast(&busyobj->vbo->cond)); + Lck_Unlock(&busyobj->vbo->mtx); +} + +/* Wait for the fetch thread to finish reading the pipeline buffer */ +void +VBO_StreamWait(struct busyobj *busyobj) +{ + if (!busyobj->use_locks) + return; + Lck_Lock(&busyobj->vbo->mtx); + while (busyobj->htc.pipeline.b != NULL && busyobj->stream_stopped == 0) + Lck_CondWait(&busyobj->vbo->cond, &busyobj->vbo->mtx, NULL); + Lck_Unlock(&busyobj->vbo->mtx); +} + +/* Signal additional data available */ +void +VBO_StreamData(struct busyobj *busyobj) +{ + CHECK_OBJ_NOTNULL(busyobj, BUSYOBJ_MAGIC); + CHECK_OBJ_NOTNULL(busyobj->fetch_obj, OBJECT_MAGIC); + + if (busyobj->use_locks) + Lck_Lock(&busyobj->vbo->mtx); + assert(busyobj->fetch_obj->len >= busyobj->stream_max); + if (busyobj->fetch_obj->len > busyobj->stream_max) { + busyobj->stream_max = busyobj->fetch_obj->len; + if (busyobj->use_locks) + AZ(pthread_cond_broadcast(&busyobj->vbo->cond)); + } + if (busyobj->use_locks) + Lck_Unlock(&busyobj->vbo->mtx); +} + +/* Sync the client's stream_ctx with the busyobj, and block on no more + * data available */ +void +VBO_StreamSync(struct worker *wrk) +{ + struct busyobj *busyobj; + struct stream_ctx *sctx; + + CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); + CHECK_OBJ_NOTNULL(wrk->busyobj, BUSYOBJ_MAGIC); + busyobj = wrk->busyobj; + CHECK_OBJ_NOTNULL(wrk->sp, SESS_MAGIC); + CHECK_OBJ_NOTNULL(wrk->sp->req, REQ_MAGIC); + CHECK_OBJ_NOTNULL(wrk->sp->req->obj, OBJECT_MAGIC); + CHECK_OBJ_NOTNULL(wrk->sctx, STREAM_CTX_MAGIC); + sctx = wrk->sctx; + + if (busyobj->use_locks) + Lck_Lock(&busyobj->vbo->mtx); + assert(sctx->stream_max <= busyobj->stream_max); + + if (wrk->sp->req->obj->objcore == NULL || + (wrk->sp->req->obj->objcore->flags & OC_F_PASS)) { + /* Give notice to backend fetch that we are finished + * with all chunks before this one */ + busyobj->stream_frontchunk = sctx->stream_frontchunk; + } + + sctx->stream_stopped = busyobj->stream_stopped; + sctx->stream_max = busyobj->stream_max; + + if (busyobj->use_locks && !sctx->stream_stopped && + sctx->stream_next == sctx->stream_max) { + while (!busyobj->stream_stopped && + sctx->stream_max == busyobj->stream_max) { + Lck_CondWait(&busyobj->vbo->cond, &busyobj->vbo->mtx, + NULL); + } + sctx->stream_stopped = busyobj->stream_stopped; + sctx->stream_max = busyobj->stream_max; + } + + if (busyobj->use_locks) + Lck_Unlock(&busyobj->vbo->mtx); +} -- 1.7.4.1 From martin at varnish-software.com Sun Jan 22 17:53:19 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Sun, 22 Jan 2012 18:53:19 +0100 Subject: [PATCH 13/25] Streaming statistics In-Reply-To: <1327254811-29629-1-git-send-email-martin@varnish-software.com> References: <1327254811-29629-1-git-send-email-martin@varnish-software.com> Message-ID: <1327254811-29629-13-git-send-email-martin@varnish-software.com> --- bin/varnishd/cache/cache_center.c | 7 ++++++- include/tbl/acct_fields.h | 1 + include/tbl/vsc_f_main.h | 3 +++ 3 files changed, 10 insertions(+), 1 deletions(-) diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c index 1f1a239..7a9134e 100644 --- a/bin/varnishd/cache/cache_center.c +++ b/bin/varnishd/cache/cache_center.c @@ -983,6 +983,9 @@ cnt_streambody_task(struct worker *wrk, void *priv) objcore = obj->objcore; wrk->busyobj->fetch_failed = FetchBody(wrk, wrk->busyobj); + + wrk->stats.fetch_threaded++; + AZ(wrk->busyobj->fetch_obj); AZ(wrk->busyobj->vbc); wrk->busyobj->vfp = NULL; @@ -1028,6 +1031,7 @@ cnt_streambody(struct sess *sp, struct worker *wrk, struct req *req) * fetch at all. This code now assumes all passes through here * needs to do the fetch as well. (Multiple streaming clients * not implemented yet) */ + wrk->acct_tmp.fetch++; AZ(wrk->busyobj->fetch_obj); wrk->busyobj->fetch_obj = req->obj; http_Setup(wrk->busyobj->bereq, NULL); @@ -1051,6 +1055,7 @@ cnt_streambody(struct sess *sp, struct worker *wrk, struct req *req) * streaming. (MBGXXX: Flipflop not finished * yet) */ wrk->busyobj->do_stream_flipflop = 1; + wrk->stats.fetch_flipflop++; } wrk->busyobj->fetch_failed = FetchBody(sp->wrk, wrk->busyobj); VBO_StreamStopped(wrk->busyobj); @@ -1072,11 +1077,11 @@ cnt_streambody(struct sess *sp, struct worker *wrk, struct req *req) AN(req->obj->objcore->ban); HSH_Unbusy(wrk); } - wrk->acct_tmp.fetch++; req->director = NULL; req->restarts = 0; RES_StreamEnd(sp); + wrk->acct_tmp.streamed++; if (wrk->res_mode & RES_GUNZIP) (void)VGZ_Destroy(&sctx.vgz, sp->vsl_id); diff --git a/include/tbl/acct_fields.h b/include/tbl/acct_fields.h index 154f106..c58bcb1 100644 --- a/include/tbl/acct_fields.h +++ b/include/tbl/acct_fields.h @@ -38,3 +38,4 @@ ACCT(pass) ACCT(fetch) ACCT(hdrbytes) ACCT(bodybytes) +ACCT(streamed) diff --git a/include/tbl/vsc_f_main.h b/include/tbl/vsc_f_main.h index b6b8dc9..5761c51 100644 --- a/include/tbl/vsc_f_main.h +++ b/include/tbl/vsc_f_main.h @@ -143,6 +143,8 @@ VSC_F(fetch_failed, uint64_t, 1, 'a', "Fetch failed", "") VSC_F(fetch_1xx, uint64_t, 1, 'a', "Fetch no body (1xx)", "") VSC_F(fetch_204, uint64_t, 1, 'a', "Fetch no body (204)", "") VSC_F(fetch_304, uint64_t, 1, 'a', "Fetch no body (304)", "") +VSC_F(fetch_threaded, uint64_t, 1, 'a', "Fetch threaded", "") +VSC_F(fetch_flipflop, uint64_t, 1, 'a', "Fetch flipflop", "") /*--------------------------------------------------------------------- * Pools, threads, and sessions @@ -266,6 +268,7 @@ VSC_F(s_req, uint64_t, 1, 'a', "Total Requests", "") VSC_F(s_pipe, uint64_t, 1, 'a', "Total pipe", "") VSC_F(s_pass, uint64_t, 1, 'a', "Total pass", "") VSC_F(s_fetch, uint64_t, 1, 'a', "Total fetch", "") +VSC_F(s_streamed, uint64_t, 1, 'a', "Total streamed delivery", "") VSC_F(s_hdrbytes, uint64_t, 1, 'a', "Total header bytes", "") VSC_F(s_bodybytes, uint64_t, 1, 'a', "Total body bytes", "") -- 1.7.4.1 From martin at varnish-software.com Sun Jan 22 17:53:24 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Sun, 22 Jan 2012 18:53:24 +0100 Subject: [PATCH 18/25] Allow flip-flop streaming on pass In-Reply-To: <1327254811-29629-1-git-send-email-martin@varnish-software.com> References: <1327254811-29629-1-git-send-email-martin@varnish-software.com> Message-ID: <1327254811-29629-18-git-send-email-martin@varnish-software.com> --- bin/varnishd/cache/cache_center.c | 4 ---- bin/varnishd/cache/cache_response.c | 1 - 2 files changed, 0 insertions(+), 5 deletions(-) diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c index aaab4fd..96e23fc 100644 --- a/bin/varnishd/cache/cache_center.c +++ b/bin/varnishd/cache/cache_center.c @@ -1061,9 +1061,6 @@ cnt_streambody(struct sess *sp, struct worker *wrk, struct req *req) /* We have no fetch worker */ if (req->obj->objcore == NULL || req->obj->objcore->flags & OC_F_PASS) { - /* It's a pass, prefer flipflop - * streaming. (MBGXXX: Flipflop not - * finished yet) */ wrk->busyobj->do_stream_flipflop = 1; wrk->stats.fetch_flipflop++; } @@ -1090,7 +1087,6 @@ cnt_streambody(struct sess *sp, struct worker *wrk, struct req *req) if (wrk->busyobj->htc.ws == wrk->ws) { /* Busyobj's htc has buffer on our workspace, wait for it to be released */ - AZ(wrk->busyobj->do_stream_flipflop); VBO_StreamWait(wrk->busyobj); } diff --git a/bin/varnishd/cache/cache_response.c b/bin/varnishd/cache/cache_response.c index a49acbe..6bd4da7 100644 --- a/bin/varnishd/cache/cache_response.c +++ b/bin/varnishd/cache/cache_response.c @@ -465,7 +465,6 @@ RES_StreamPoll(struct worker *wrk) VBO_StreamData(wrk->busyobj); if (wrk->busyobj->do_stream_flipflop == 1) { AN(wrk->sctx); - /* MBGXXX: Do flip-flop streaming */ /* MBGXXX: Loop around waiting for the lag behind to * be less than some configurable size, to keep the * cache memory usage low (this for streaming -- 1.7.4.1 From martin at varnish-software.com Sun Jan 22 17:53:26 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Sun, 22 Jan 2012 18:53:26 +0100 Subject: [PATCH 20/25] Streaming documentation In-Reply-To: <1327254811-29629-1-git-send-email-martin@varnish-software.com> References: <1327254811-29629-1-git-send-email-martin@varnish-software.com> Message-ID: <1327254811-29629-20-git-send-email-martin@varnish-software.com> --- doc/sphinx/reference/vcl.rst | 49 +++++++++++++++++++++++++++++++++++++++-- 1 files changed, 46 insertions(+), 3 deletions(-) diff --git a/doc/sphinx/reference/vcl.rst b/doc/sphinx/reference/vcl.rst index 2c185ed..5341239 100644 --- a/doc/sphinx/reference/vcl.rst +++ b/doc/sphinx/reference/vcl.rst @@ -791,11 +791,14 @@ The following variables are available after the requested object has been retrieved from the backend, before it is entered into the cache. In other words, they are available in vcl_fetch: -beresp.do_stream +beresp.do_stream Deliver the object to the client directly without fetching the whole object into varnish. If this request is pass'ed it will not be - stored in memory. As of Varnish Cache 3.0 the object will marked as busy - as it is delivered so only client can access the object. + stored in memory (up to *stream_pass_bufsize*). + +beresp.stream_pass_bufsize + When passing and streaming, do not use more than this amount of + cache space. beresp.do_esi Boolean. ESI-process the object after fetching it. Defaults to @@ -937,6 +940,46 @@ If there are no more backends or if you hit *max_restarts* and we have an object that is younger than what you set beresp.saintmode to be Varnish will serve the object, even if it is stale. +Streaming +--------- + +When setting the beresp.do_stream flag to true in vcl_fetch, you +instruct Varnish to stream the object to the client. This means that +(after receiving the backend headers), the object body will be +delivered to the client while it is received from the backend. For +non-passes, the object is not busy while receiving the data from the +backend, and any subsequent requests for the object will be served the +content as it is received from the backend. + +For streaming passes, Varnish has a configurable buffer size for the +data that is read from the backend connection and will be passed on to +the client. This is useful for limiting the cache space usage when +passing large static objects, which would otherwise purge most of the +cache content to make room for the temporary object. Varnish will then +read only up to this buffer size from the backend, and then stall the +connection until the the data has been delivered to the client. The +pass buffer size defaults to *stream_pass_bufsize*, but can also be +set per request through VCL in vcl_fetch by setting +beresp.stream_pass_bufsize. If the resource cost on the backend of +keeping the connection open is high, you might want to keep the time +this connection is kept open down to a minimum also for passes. This +can be achieved by setting *stream_pass_bufsize* to zero, which means +unlimited. + +Varnish will use an additional worker thread for the backend body +fetch when streaming. This thread is taken from the idle worker list, +with a configurable wait timeout. If an idle thread is not found +within *stream_grab_timeout* milliseconds, Varnish will fetch the body +content from the requesting thread instead. For passes, the fall-back +mechanism will be a flip-flop style streaming, where the thread +alternates between reading data from the backend connection and +writing to the client connection. For non-passes the entire body will +be fetched in one go from the client thread, before writing the data +to the client (much the same as a non-streamed case). The object will +not be busy while reading the body though, and subsequent requesting +clients for the same object will deliver the streamed body content as +it is read from the backend. + EXAMPLES ======== -- 1.7.4.1 From martin at varnish-software.com Sun Jan 22 17:53:28 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Sun, 22 Jan 2012 18:53:28 +0100 Subject: [PATCH 22/25] Implement stream_pass_bufsize throttling of streaming in pass mode In-Reply-To: <1327254811-29629-1-git-send-email-martin@varnish-software.com> References: <1327254811-29629-1-git-send-email-martin@varnish-software.com> Message-ID: <1327254811-29629-22-git-send-email-martin@varnish-software.com> --- bin/varnishd/cache/cache.h | 3 +- bin/varnishd/cache/cache_busyobj.c | 5 ++- bin/varnishd/cache/cache_response.c | 67 +++++++++++++++++++++------------- 3 files changed, 46 insertions(+), 29 deletions(-) diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index fe65dbd..a64a0ae 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -541,7 +541,8 @@ struct busyobj { /* Stream stuff */ ssize_t stream_max; - struct storage *stream_frontchunk; + volatile ssize_t stream_next; + volatile struct storage *stream_frontchunk; unsigned stream_stopped; ssize_t stream_pass_bufsize; }; diff --git a/bin/varnishd/cache/cache_busyobj.c b/bin/varnishd/cache/cache_busyobj.c index b39c170..8b97d4a 100644 --- a/bin/varnishd/cache/cache_busyobj.c +++ b/bin/varnishd/cache/cache_busyobj.c @@ -295,14 +295,15 @@ VBO_StreamSync(struct worker *wrk) (wrk->sp->req->obj->objcore->flags & OC_F_PASS)) { /* Give notice to backend fetch that we are finished * with all chunks before this one */ + assert(busyobj->stream_next <= sctx->stream_next); + busyobj->stream_next = sctx->stream_next; busyobj->stream_frontchunk = sctx->stream_frontchunk; } sctx->stream_stopped = busyobj->stream_stopped; sctx->stream_max = busyobj->stream_max; - if (busyobj->use_locks && !sctx->stream_stopped && - sctx->stream_next == sctx->stream_max) { + if (busyobj->use_locks && sctx->stream_next == sctx->stream_max) { while (!busyobj->stream_stopped && sctx->stream_max == busyobj->stream_max) { Lck_CondWait(&busyobj->vbo->cond, &busyobj->vbo->mtx, diff --git a/bin/varnishd/cache/cache_response.c b/bin/varnishd/cache/cache_response.c index 6bd4da7..bc7f122 100644 --- a/bin/varnishd/cache/cache_response.c +++ b/bin/varnishd/cache/cache_response.c @@ -459,36 +459,51 @@ RES_StreamPoll(struct worker *wrk) { struct object *fetch_obj; struct storage *st; + struct busyobj *bo; + int pass = 0; - CHECK_OBJ_NOTNULL(wrk->busyobj, BUSYOBJ_MAGIC); - - VBO_StreamData(wrk->busyobj); - if (wrk->busyobj->do_stream_flipflop == 1) { - AN(wrk->sctx); - /* MBGXXX: Loop around waiting for the lag behind to - * be less than some configurable size, to keep the - * cache memory usage low (this for streaming - * extremely large objects with pass) */ - VBO_StreamSync(wrk); - RES_StreamWrite(wrk->sp); - } - - if (wrk->busyobj->stream_frontchunk == NULL) - return; - - /* It's a pass - remove chunks already delivered. Should be OK - * to do lock-free, as we are not fiddling pointers of any - * storage chunk passed busyobj->stream_frontchunk */ + bo = wrk->busyobj; + CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC); fetch_obj = wrk->busyobj->fetch_obj; CHECK_OBJ_NOTNULL(fetch_obj, OBJECT_MAGIC); - assert(fetch_obj->objcore == NULL || - (fetch_obj->objcore->flags & OC_F_PASS)); + + if (fetch_obj->objcore == NULL || fetch_obj->objcore->flags & OC_F_PASS) + pass = 1; + + VBO_StreamData(bo); while (1) { - st = VTAILQ_FIRST(&fetch_obj->store); - if (st == NULL || st == wrk->busyobj->stream_frontchunk) + if (bo->do_stream_flipflop == 1) { + CHECK_OBJ_NOTNULL(wrk->sctx, STREAM_CTX_MAGIC); + VBO_StreamSync(wrk); + RES_StreamWrite(wrk->sp); + } + + if (bo->stream_frontchunk != NULL) { + /* It's a pass - remove chunks already + * delivered. Should be OK to do lock-free, as + * we are not fiddling pointers of any storage + * chunk beyond busyobj->stream_frontchunk */ + AN(pass); + while (1) { + st = VTAILQ_FIRST(&fetch_obj->store); + if (st == NULL || + st == bo->stream_frontchunk) + break; + CHECK_OBJ_NOTNULL(st, STORAGE_MAGIC); + VTAILQ_REMOVE(&fetch_obj->store, st, list); + STV_free(st); + } + } + + if (pass == 0 || bo->stream_pass_bufsize == 0 || + bo->stream_max - bo->stream_next < bo->stream_pass_bufsize) break; - CHECK_OBJ_NOTNULL(st, STORAGE_MAGIC); - VTAILQ_REMOVE(&fetch_obj->store, st, list); - STV_free(st); + + /* Loop around waiting for the lag behind to be less + * than some configurable size, to keep the cache + * memory usage low (this for streaming large objects + * with pass) */ + if (bo->do_stream_flipflop == 0) + VTIM_sleep(0.1); } } -- 1.7.4.1 From martin at varnish-software.com Sun Jan 22 17:53:23 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Sun, 22 Jan 2012 18:53:23 +0100 Subject: [PATCH 17/25] Add a run-time parameter for stream fetch thread grab timeout. In-Reply-To: <1327254811-29629-1-git-send-email-martin@varnish-software.com> References: <1327254811-29629-1-git-send-email-martin@varnish-software.com> Message-ID: <1327254811-29629-17-git-send-email-martin@varnish-software.com> --- bin/varnishd/cache/cache_center.c | 4 +--- bin/varnishd/common/params.h | 1 + bin/varnishd/mgt/mgt_param.c | 6 ++++++ 3 files changed, 8 insertions(+), 3 deletions(-) diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c index 23b5f71..aaab4fd 100644 --- a/bin/varnishd/cache/cache_center.c +++ b/bin/varnishd/cache/cache_center.c @@ -1041,9 +1041,7 @@ cnt_streambody(struct sess *sp, struct worker *wrk, struct req *req) wrk->busyobj->fetch_obj = req->obj; http_Setup(wrk->busyobj->bereq, NULL); http_Setup(wrk->busyobj->beresp, NULL); - wrk_ex = SES_GrabWorker(sp, 100); /* MBGXXX: Configurable - * thread grabbing - * timeout */ + wrk_ex = SES_GrabWorker(sp, cache_param->stream_grab_timeout); if (wrk_ex != NULL) wrk->busyobj->use_locks = 1; if (req->obj->objcore != NULL) { diff --git a/bin/varnishd/common/params.h b/bin/varnishd/common/params.h index f0445ae..f726ce8 100644 --- a/bin/varnishd/common/params.h +++ b/bin/varnishd/common/params.h @@ -99,6 +99,7 @@ struct params { ssize_t fetch_chunksize; ssize_t fetch_maxchunksize; ssize_t stream_maxchunksize; + unsigned stream_grab_timeout; unsigned nuke_limit; #ifdef SENDFILE_WORKS diff --git a/bin/varnishd/mgt/mgt_param.c b/bin/varnishd/mgt/mgt_param.c index 227ea3a..9b1467f 100644 --- a/bin/varnishd/mgt/mgt_param.c +++ b/bin/varnishd/mgt/mgt_param.c @@ -846,6 +846,12 @@ static const struct parspec input_parspec[] = { "data available.\n", EXPERIMENTAL, "256k", "bytes" }, + { "stream_grab_timeout", + tweak_uint, &mgt_param.stream_grab_timeout, 0, UINT_MAX, + "Wait at most this long for an idle worker to appear when " + "grabbing a thread for background fetch.\n", + EXPERIMENTAL, + "100", "milliseconds" }, #ifdef SENDFILE_WORKS { "sendfile_threshold", tweak_bytes, &mgt_param.sendfile_threshold, 0, HUGE_VAL, -- 1.7.4.1 From martin at varnish-software.com Sun Jan 22 17:53:21 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Sun, 22 Jan 2012 18:53:21 +0100 Subject: [PATCH 15/25] Make streaming work with multiple streaming clients In-Reply-To: <1327254811-29629-1-git-send-email-martin@varnish-software.com> References: <1327254811-29629-1-git-send-email-martin@varnish-software.com> Message-ID: <1327254811-29629-15-git-send-email-martin@varnish-software.com> --- bin/varnishd/cache/cache_center.c | 117 ++++++++++++++++++++++--------------- bin/varnishd/cache/cache_expire.c | 1 - bin/varnishd/cache/cache_fetch.c | 2 - bin/varnishd/cache/cache_hash.c | 33 ++++++++++- bin/varnishd/hash/hash_slinger.h | 1 + 5 files changed, 103 insertions(+), 51 deletions(-) diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c index 7a9134e..23b5f71 100644 --- a/bin/varnishd/cache/cache_center.c +++ b/bin/varnishd/cache/cache_center.c @@ -224,7 +224,6 @@ cnt_prepresp(struct sess *sp, struct worker *wrk, struct req *req) if (wrk->busyobj != NULL) { CHECK_OBJ_NOTNULL(wrk->busyobj, BUSYOBJ_MAGIC); AN(wrk->busyobj->do_stream); - AssertObjCorePassOrBusy(req->obj->objcore); } wrk->res_mode = 0; @@ -301,14 +300,17 @@ cnt_prepresp(struct sess *sp, struct worker *wrk, struct req *req) case VCL_RET_RESTART: if (req->restarts >= cache_param->max_restarts) break; - if (wrk->busyobj != NULL) { + if (wrk->busyobj != NULL && + (req->obj->objcore == NULL || + req->obj->objcore->flags & OC_F_BUSY)) { AN(wrk->busyobj->do_stream); VDI_CloseFd(wrk, &wrk->busyobj->vbc); HSH_Drop(wrk); - VBO_DerefBusyObj(wrk, &wrk->busyobj); } else { (void)HSH_Deref(wrk, NULL, &req->obj); } + if (wrk->busyobj != NULL) + (void)VBO_DerefBusyObj(wrk, &wrk->busyobj); AZ(req->obj); req->restarts++; req->director = NULL; @@ -318,8 +320,8 @@ cnt_prepresp(struct sess *sp, struct worker *wrk, struct req *req) default: WRONG("Illegal action in vcl_deliver{}"); } - if (wrk->busyobj != NULL && wrk->busyobj->do_stream) { - AssertObjCorePassOrBusy(req->obj->objcore); + if (wrk->busyobj != NULL) { + AN(wrk->busyobj->do_stream); sp->step = STP_STREAMBODY; } else { sp->step = STP_DELIVER; @@ -983,13 +985,18 @@ cnt_streambody_task(struct worker *wrk, void *priv) objcore = obj->objcore; wrk->busyobj->fetch_failed = FetchBody(wrk, wrk->busyobj); + VBO_StreamStopped(wrk->busyobj); wrk->stats.fetch_threaded++; + if (obj->objcore != NULL) { + HSH_RemoveBusyObj(wrk, obj->objcore); + if (wrk->busyobj->fetch_failed == 0) + EXP_Insert(obj); + } AZ(wrk->busyobj->fetch_obj); AZ(wrk->busyobj->vbc); wrk->busyobj->vfp = NULL; - VBO_StreamStopped(wrk->busyobj); u = VBO_DerefBusyObj(wrk, &wrk->busyobj); if (objcore != NULL || u == 0) { @@ -1013,6 +1020,7 @@ cnt_streambody(struct sess *sp, struct worker *wrk, struct req *req) CHECK_OBJ_NOTNULL(req, REQ_MAGIC); CHECK_OBJ_NOTNULL(wrk->busyobj, BUSYOBJ_MAGIC); + AN(wrk->busyobj->do_stream); memset(&sctx, 0, sizeof sctx); sctx.magic = STREAM_CTX_MAGIC; AZ(wrk->sctx); @@ -1024,58 +1032,72 @@ cnt_streambody(struct sess *sp, struct worker *wrk, struct req *req) sctx.obuf_len = sizeof (obuf); } - AssertObjCorePassOrBusy(req->obj->objcore); RES_StreamStart(sp); - /* MBGXXX: Test on OC_F_BUSY to see if we should initiate - * fetch at all. This code now assumes all passes through here - * needs to do the fetch as well. (Multiple streaming clients - * not implemented yet) */ - wrk->acct_tmp.fetch++; - AZ(wrk->busyobj->fetch_obj); - wrk->busyobj->fetch_obj = req->obj; - http_Setup(wrk->busyobj->bereq, NULL); - http_Setup(wrk->busyobj->beresp, NULL); - wrk_ex = SES_GrabWorker(sp, 100); /* MBGXXX: Configurable - * thread grabbing - * timeout */ - if (wrk_ex != NULL) { - /* Set up separate thread fetch */ - wrk->busyobj->use_locks = 1; - if (req->obj->objcore != NULL) - /* Grab a ref on the objcore for the other thread */ - HSH_Ref(req->obj->objcore); - VBO_RefBusyObj(wrk->busyobj); /* Ref for the other thread */ - WRK_DoTask(wrk_ex, cnt_streambody_task, wrk->busyobj); - } else { - /* We have no worker */ - if (wrk->busyobj->fetch_obj->objcore == NULL || - wrk->busyobj->fetch_obj->objcore->flags & OC_F_PASS) { - /* It's a pass, prefer flipflop - * streaming. (MBGXXX: Flipflop not finished - * yet) */ - wrk->busyobj->do_stream_flipflop = 1; - wrk->stats.fetch_flipflop++; + if (req->obj->objcore == NULL || req->obj->objcore->flags & OC_F_BUSY) { + /* Initiate a fetch of the body */ + wrk->acct_tmp.fetch++; + AZ(wrk->busyobj->fetch_obj); + wrk->busyobj->fetch_obj = req->obj; + http_Setup(wrk->busyobj->bereq, NULL); + http_Setup(wrk->busyobj->beresp, NULL); + wrk_ex = SES_GrabWorker(sp, 100); /* MBGXXX: Configurable + * thread grabbing + * timeout */ + if (wrk_ex != NULL) + wrk->busyobj->use_locks = 1; + if (req->obj->objcore != NULL) { + AN(req->obj->objcore->ban); + HSH_Unbusy(wrk); + AN(req->obj->objcore->busyobj); + } + if (wrk_ex != NULL) { + /* Set up separate thread fetch */ + if (req->obj->objcore != NULL) + /* Grab a ref on the objcore for the + * other thread */ + HSH_Ref(req->obj->objcore); + WRK_DoTask(wrk_ex, cnt_streambody_task, + VBO_RefBusyObj(wrk->busyobj)); + } else { + /* We have no fetch worker */ + if (req->obj->objcore == NULL || + req->obj->objcore->flags & OC_F_PASS) { + /* It's a pass, prefer flipflop + * streaming. (MBGXXX: Flipflop not + * finished yet) */ + wrk->busyobj->do_stream_flipflop = 1; + wrk->stats.fetch_flipflop++; + } + wrk->busyobj->fetch_failed = + FetchBody(wrk, wrk->busyobj); + VBO_StreamStopped(wrk->busyobj); + if (req->obj->objcore != NULL) { + HSH_RemoveBusyObj(wrk, req->obj->objcore); + if (wrk->busyobj->fetch_failed == 0) + EXP_Insert(req->obj); + } + AZ(wrk->busyobj->fetch_obj); + AZ(wrk->busyobj->vbc); + wrk->busyobj->vfp = NULL; + VBO_StreamSync(wrk); } - wrk->busyobj->fetch_failed = FetchBody(sp->wrk, wrk->busyobj); - VBO_StreamStopped(wrk->busyobj); } - RES_StreamBody(sp); + if (wrk->busyobj->do_stream_flipflop == 0) + RES_StreamBody(sp); + else + AN(wrk->sctx->stream_stopped); - if (wrk->busyobj->htc.ws == wrk->ws) + if (wrk->busyobj->htc.ws == wrk->ws) { /* Busyobj's htc has buffer on our workspace, wait for it to be released */ + AZ(wrk->busyobj->do_stream_flipflop); VBO_StreamWait(wrk->busyobj); + } if (wrk->busyobj->fetch_failed) { req->doclose = "Stream error"; - } else if (req->obj->objcore != NULL) { - /* MBGXXX: This should be done on the bg task */ - EXP_Insert(req->obj); - AN(req->obj->objcore); - AN(req->obj->objcore->ban); - HSH_Unbusy(wrk); } req->director = NULL; req->restarts = 0; @@ -1169,7 +1191,6 @@ cnt_hit(struct sess *sp, struct worker *wrk, struct req *req) CHECK_OBJ_NOTNULL(req->obj, OBJECT_MAGIC); CHECK_OBJ_NOTNULL(req->vcl, VCL_CONF_MAGIC); - AZ(wrk->busyobj); assert(!(req->obj->objcore->flags & OC_F_PASS)); @@ -1315,6 +1336,8 @@ cnt_lookup(struct sess *sp, struct worker *wrk, struct req *req) wrk->stats.cache_hitpass++; WSP(sp, SLT_HitPass, "%u", req->obj->xid); (void)HSH_Deref(wrk, NULL, &req->obj); + if (wrk->busyobj != NULL) + (void)VBO_DerefBusyObj(wrk, &wrk->busyobj); req->objcore = NULL; sp->step = STP_PASS; return (0); diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c index e93a1aa..212adf3 100644 --- a/bin/varnishd/cache/cache_expire.c +++ b/bin/varnishd/cache/cache_expire.c @@ -224,7 +224,6 @@ EXP_Insert(struct object *o) CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC); oc = o->objcore; CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); - AssertObjBusy(o); HSH_Ref(oc); assert(o->exp.entered != 0 && !isnan(o->exp.entered)); diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index d5ae4d5..0643389 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -507,8 +507,6 @@ FetchBody(struct worker *wrk, struct busyobj *bo) if (bo->vfp == NULL) bo->vfp = &vfp_nop; - AssertObjCorePassOrBusy(obj->objcore); - AZ(bo->vgz_rx); AZ(VTAILQ_FIRST(&obj->store)); diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c index e78eb60..a12d43a 100644 --- a/bin/varnishd/cache/cache_hash.c +++ b/bin/varnishd/cache/cache_hash.c @@ -63,6 +63,8 @@ static const struct hash_slinger *hash; +static void hsh_rush(struct objhead *oh); + /*---------------------------------------------------------------------*/ /* Precreate an objhead and object for later use */ void @@ -414,6 +416,13 @@ HSH_Lookup(struct sess *sp, struct objhead **poh) if (o->hits < INT_MAX) o->hits++; assert(oh->refcnt > 1); + if (oc->busyobj != NULL) { + /* It's streamable */ + CHECK_OBJ_NOTNULL(oc->busyobj, BUSYOBJ_MAGIC); + wrk->busyobj = VBO_RefBusyObj(oc->busyobj); + if (oh->waitinglist != NULL) + hsh_rush(oh); + } Lck_Unlock(&oh->mtx); assert(hash->deref(oh)); *poh = oh; @@ -623,7 +632,7 @@ HSH_Unbusy(struct worker *wrk) VTAILQ_REMOVE(&oh->objcs, oc, list); VTAILQ_INSERT_HEAD(&oh->objcs, oc, list); oc->flags &= ~OC_F_BUSY; - if (oc->busyobj != NULL) + if (oc->busyobj->do_stream == 0) (void)VBO_DerefBusyObj(wrk, &oc->busyobj); if (oh->waitinglist != NULL) hsh_rush(oh); @@ -632,6 +641,28 @@ HSH_Unbusy(struct worker *wrk) assert(oc_getobj(wrk, oc) == o); } +/*--------------------------------------------------------------------- + * Drop the objcore's ref on the busyobj while holding the objhead mutex + */ + +void +HSH_RemoveBusyObj(struct worker *wrk, struct objcore *oc) +{ + struct objhead *oh; + + CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); + CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); + oh = oc->objhead; + CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC); + + AZ(oc->flags & OC_F_BUSY); + AN(oc->busyobj); + assert(oc->busyobj->do_stream == 0 || oc->busyobj->stream_stopped == 1); + Lck_Lock(&oh->mtx); + (void)VBO_DerefBusyObj(wrk, &oc->busyobj); + Lck_Unlock(&oh->mtx); +} + void HSH_Ref(struct objcore *oc) { diff --git a/bin/varnishd/hash/hash_slinger.h b/bin/varnishd/hash/hash_slinger.h index b45e604..8affa42 100644 --- a/bin/varnishd/hash/hash_slinger.h +++ b/bin/varnishd/hash/hash_slinger.h @@ -55,6 +55,7 @@ void HSH_Prealloc(const struct sess *sp); void HSH_Cleanup(struct worker *w); struct objcore *HSH_Lookup(struct sess *sp, struct objhead **poh); void HSH_Unbusy(struct worker *wrk); +void HSH_RemoveBusyObj(struct worker *wrk, struct objcore *oc); void HSH_Ref(struct objcore *o); void HSH_Drop(struct worker *wrk); void HSH_Init(const struct hash_slinger *slinger); -- 1.7.4.1 From martin at varnish-software.com Sun Jan 22 17:53:30 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Sun, 22 Jan 2012 18:53:30 +0100 Subject: [PATCH 24/25] Thundering horde elimination In-Reply-To: <1327254811-29629-1-git-send-email-martin@varnish-software.com> References: <1327254811-29629-1-git-send-email-martin@varnish-software.com> Message-ID: <1327254811-29629-24-git-send-email-martin@varnish-software.com> Deal with thundering horde problem by limiting the number of clients allowed to wait for additional data on the stream to stream_tokens. Only threads that acquire a token will be allowed to wait on the condvar. Others will have to go on a queue. Tokens are passed to queued clients when a token holding thread has sent some data to the client. The maximum wait time to get a token is controlled through the stream_token_timeout run-time parameter. This to prevent a token holding client that suddenly blocks from starving another client waiting for a token. --- bin/varnishd/cache/cache.h | 5 ++ bin/varnishd/cache/cache_busyobj.c | 96 +++++++++++++++++++++++++++++++++++- bin/varnishd/cache/cache_center.c | 4 ++ bin/varnishd/cache/cache_vrt_var.c | 20 ++++++++ bin/varnishd/common/params.h | 2 + bin/varnishd/mgt/mgt_param.c | 12 +++++ include/tbl/vsc_f_main.h | 3 + lib/libvcl/generate.py | 6 ++ 8 files changed, 146 insertions(+), 2 deletions(-) diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index a64a0ae..b02bea6 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -278,6 +278,9 @@ struct stream_ctx { /* Backend fetch has finished */ unsigned stream_stopped; + + /* Are we currently holding a token from the busyobj */ + unsigned has_token; }; /*--------------------------------------------------------------------*/ @@ -545,6 +548,7 @@ struct busyobj { volatile struct storage *stream_frontchunk; unsigned stream_stopped; ssize_t stream_pass_bufsize; + unsigned stream_tokens; }; /* Object structure --------------------------------------------------*/ @@ -759,6 +763,7 @@ void VBO_StreamStopped(struct busyobj *busyobj); void VBO_StreamWait(struct busyobj *busyobj); void VBO_StreamData(struct busyobj *busyobj); void VBO_StreamSync(struct worker *wrk); +void VBO_ReleaseToken(struct worker *wrk, struct busyobj *busyobj); /* cache_center.c [CNT] */ void CNT_Session(struct sess *sp); diff --git a/bin/varnishd/cache/cache_busyobj.c b/bin/varnishd/cache/cache_busyobj.c index 8b97d4a..2b6ebc0 100644 --- a/bin/varnishd/cache/cache_busyobj.c +++ b/bin/varnishd/cache/cache_busyobj.c @@ -43,6 +43,7 @@ struct vbo { #define VBO_MAGIC 0xde3d8223 struct lock mtx; pthread_cond_t cond; + VTAILQ_HEAD(, worker) token_wait_queue; unsigned refcount; uint16_t nhttp; struct busyobj bo; @@ -51,6 +52,9 @@ struct vbo { static struct lock vbo_mtx; static struct vbo *nvbo; +static void vbo_acquire_token(struct worker *wrk, struct busyobj *busyobj); +static void vbo_release_token(struct worker *wrk, struct busyobj *busyobj); + void VBO_Init(void) { @@ -82,6 +86,7 @@ vbo_New(void) vbo->nhttp = nhttp; Lck_New(&vbo->mtx, lck_busyobj); AZ(pthread_cond_init(&vbo->cond, NULL)); + VTAILQ_INIT(&vbo->token_wait_queue); return (vbo); } @@ -145,6 +150,7 @@ VBO_GetBusyObj(struct worker *wrk) vbo->bo.beresp = HTTP_create(p, vbo->nhttp); vbo->bo.stream_pass_bufsize = cache_param->stream_pass_bufsize; + vbo->bo.stream_tokens = cache_param->stream_tokens; return (&vbo->bo); } @@ -291,6 +297,11 @@ VBO_StreamSync(struct worker *wrk) Lck_Lock(&busyobj->vbo->mtx); assert(sctx->stream_max <= busyobj->stream_max); + if (sctx->has_token) + /* Release token to give other clients that has + * reached the end of data a chance */ + vbo_release_token(wrk, busyobj); + if (wrk->sp->req->obj->objcore == NULL || (wrk->sp->req->obj->objcore->flags & OC_F_PASS)) { /* Give notice to backend fetch that we are finished @@ -303,8 +314,11 @@ VBO_StreamSync(struct worker *wrk) sctx->stream_stopped = busyobj->stream_stopped; sctx->stream_max = busyobj->stream_max; - if (busyobj->use_locks && sctx->stream_next == sctx->stream_max) { - while (!busyobj->stream_stopped && + if (busyobj->use_locks && !sctx->stream_stopped && + sctx->stream_next == sctx->stream_max) { + /* We've exhausted available data, wait for more */ + vbo_acquire_token(wrk, busyobj); + while (sctx->has_token && !busyobj->stream_stopped && sctx->stream_max == busyobj->stream_max) { Lck_CondWait(&busyobj->vbo->cond, &busyobj->vbo->mtx, NULL); @@ -316,3 +330,81 @@ VBO_StreamSync(struct worker *wrk) if (busyobj->use_locks) Lck_Unlock(&busyobj->vbo->mtx); } + +/* Acquire a token for the worker from the busyobj. If none available, + wait for stream_token_timeout ms on the queue. + wrk->sctx->has_token will be true if succesful */ +static void +vbo_acquire_token(struct worker *wrk, struct busyobj *busyobj) +{ + struct timespec ts; + + CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); + CHECK_OBJ_NOTNULL(wrk->sctx, STREAM_CTX_MAGIC); + AZ(wrk->sctx->has_token); + CHECK_OBJ_NOTNULL(busyobj, BUSYOBJ_MAGIC); + AN(busyobj->use_locks); + + if (busyobj->stream_tokens > 0) { + busyobj->stream_tokens--; + wrk->sctx->has_token = 1; + return; + } + + AZ(clock_gettime(CLOCK_REALTIME, &ts)); + ts.tv_sec += cache_param->stream_token_timeout / 1000; + ts.tv_nsec += (cache_param->stream_token_timeout % 1000) * 1000000; + if (ts.tv_nsec >= 1000000000) { + ts.tv_sec++; + ts.tv_nsec -= 1000000000; + } + VTAILQ_INSERT_TAIL(&busyobj->vbo->token_wait_queue, wrk, list); + Lck_CondWait(&wrk->cond, &busyobj->vbo->mtx, &ts); + if (wrk->sctx->has_token == 0) { + VTAILQ_REMOVE(&busyobj->vbo->token_wait_queue, wrk, list); + wrk->stats.n_tokentimeout++; + } +} + +/* Release our currently held token, passing it on to the first on the + queue if the queue is non-empty. Resets wrk->sctx->has_token */ +static void +vbo_release_token(struct worker *wrk, struct busyobj *busyobj) +{ + struct worker *wrk_waiting; + + CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); + CHECK_OBJ_NOTNULL(wrk->sctx, STREAM_CTX_MAGIC); + AN(wrk->sctx->has_token); + CHECK_OBJ_NOTNULL(busyobj, BUSYOBJ_MAGIC); + AN(busyobj->use_locks); + + wrk_waiting = VTAILQ_FIRST(&busyobj->vbo->token_wait_queue); + if (wrk_waiting != NULL) { + /* Transfer our token to the first on the waiting + list, and wake it */ + CHECK_OBJ_NOTNULL(wrk_waiting, WORKER_MAGIC); + CHECK_OBJ_NOTNULL(wrk_waiting->sctx, STREAM_CTX_MAGIC); + VTAILQ_REMOVE(&busyobj->vbo->token_wait_queue, wrk_waiting, + list); + AZ(wrk_waiting->sctx->has_token); + wrk_waiting->sctx->has_token = 1; + AZ(pthread_cond_signal(&wrk_waiting->cond)); + } else { + busyobj->stream_tokens++; + } + + wrk->sctx->has_token = 0; +} + +void +VBO_ReleaseToken(struct worker *wrk, struct busyobj *busyobj) +{ + CHECK_OBJ_NOTNULL(busyobj, BUSYOBJ_MAGIC); + + if (busyobj->use_locks == 0) + return; + Lck_Lock(&busyobj->vbo->mtx); + vbo_release_token(wrk, busyobj); + Lck_Unlock(&busyobj->vbo->mtx); +} diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c index 96e23fc..5daa949 100644 --- a/bin/varnishd/cache/cache_center.c +++ b/bin/varnishd/cache/cache_center.c @@ -1084,6 +1084,10 @@ cnt_streambody(struct sess *sp, struct worker *wrk, struct req *req) else AN(wrk->sctx->stream_stopped); + if (wrk->sctx->has_token) + VBO_ReleaseToken(wrk, wrk->busyobj); + AZ(wrk->sctx->has_token); + if (wrk->busyobj->htc.ws == wrk->ws) { /* Busyobj's htc has buffer on our workspace, wait for it to be released */ diff --git a/bin/varnishd/cache/cache_vrt_var.c b/bin/varnishd/cache/cache_vrt_var.c index e655abf..82d77c6 100644 --- a/bin/varnishd/cache/cache_vrt_var.c +++ b/bin/varnishd/cache/cache_vrt_var.c @@ -320,6 +320,26 @@ VRT_l_beresp_stream_pass_bufsize(const struct sess *sp, double val) sp->wrk->busyobj->stream_pass_bufsize = 0; } +int +VRT_r_beresp_stream_tokens(const struct sess *sp) +{ + CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); + CHECK_OBJ_NOTNULL(sp->wrk->busyobj, BUSYOBJ_MAGIC); + return (sp->wrk->busyobj->stream_tokens); +} + +void +VRT_l_beresp_stream_tokens(const struct sess *sp, int val) +{ + CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); + CHECK_OBJ_NOTNULL(sp->wrk->busyobj, BUSYOBJ_MAGIC); + if (val >= 1) + sp->wrk->busyobj->stream_tokens = val; + else + sp->wrk->busyobj->stream_tokens = 1; +} + + /*--------------------------------------------------------------------*/ void diff --git a/bin/varnishd/common/params.h b/bin/varnishd/common/params.h index 20b346a..a68d608 100644 --- a/bin/varnishd/common/params.h +++ b/bin/varnishd/common/params.h @@ -101,6 +101,8 @@ struct params { ssize_t stream_maxchunksize; unsigned stream_grab_timeout; ssize_t stream_pass_bufsize; + unsigned stream_tokens; + unsigned stream_token_timeout; unsigned nuke_limit; diff --git a/bin/varnishd/mgt/mgt_param.c b/bin/varnishd/mgt/mgt_param.c index 3a64b96..e5d6888 100644 --- a/bin/varnishd/mgt/mgt_param.c +++ b/bin/varnishd/mgt/mgt_param.c @@ -858,6 +858,18 @@ static const struct parspec input_parspec[] = { "Zero means unlimited.\n", EXPERIMENTAL, "10mb", "bytes" }, + { "stream_tokens", + tweak_uint, &mgt_param.stream_tokens, 1, UINT_MAX, + "Default number of tokens available for racing streaming " + "clients.\n", + EXPERIMENTAL, + "10", "tokens" }, + { "stream_token_timeout", + tweak_uint, &mgt_param.stream_token_timeout, 1, UINT_MAX, + "Timeout for acquiring token during streaming and waiting " + "for more data.\n", + EXPERIMENTAL, + "100", "ms" }, #ifdef SENDFILE_WORKS { "sendfile_threshold", tweak_bytes, &mgt_param.sendfile_threshold, 0, HUGE_VAL, diff --git a/include/tbl/vsc_f_main.h b/include/tbl/vsc_f_main.h index 5761c51..415b07d 100644 --- a/include/tbl/vsc_f_main.h +++ b/include/tbl/vsc_f_main.h @@ -262,6 +262,9 @@ VSC_F(n_objwrite, uint64_t, 0, 'a', "Objects sent with write", "or if the sendfile call has been disabled") VSC_F(n_objoverflow, uint64_t, 1, 'a', "Objects overflowing workspace", "") +VSC_F(n_tokentimeout, uint64_t, 1, 'a', "Token wait timeouts", + "The number of times a fast streaming client has timed out waiting for " + "token to be notified about new incoming data.") VSC_F(s_sess, uint64_t, 1, 'a', "Total Sessions", "") VSC_F(s_req, uint64_t, 1, 'a', "Total Requests", "") diff --git a/lib/libvcl/generate.py b/lib/libvcl/generate.py index 51b2294..1dab49d 100755 --- a/lib/libvcl/generate.py +++ b/lib/libvcl/generate.py @@ -337,6 +337,12 @@ sp_variables = ( ( 'fetch',), 'const struct sess *' ), + ('beresp.stream_tokens', + 'INT', + ( 'fetch',), + ( 'fetch',), + 'const struct sess *' + ), ('beresp.ttl', 'DURATION', ( 'fetch',), -- 1.7.4.1 From martin at varnish-software.com Sun Jan 22 17:53:22 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Sun, 22 Jan 2012 18:53:22 +0100 Subject: [PATCH 16/25] Multiple streaming test cases In-Reply-To: <1327254811-29629-1-git-send-email-martin@varnish-software.com> References: <1327254811-29629-1-git-send-email-martin@varnish-software.com> Message-ID: <1327254811-29629-16-git-send-email-martin@varnish-software.com> --- bin/varnishtest/tests/t00004.vtc | 76 +++++++++++++++++++++++++++++++++++++ bin/varnishtest/tests/t00005.vtc | 77 ++++++++++++++++++++++++++++++++++++++ bin/varnishtest/tests/t00006.vtc | 57 ++++++++++++++++++++++++++++ bin/varnishtest/tests/t00007.vtc | 75 +++++++++++++++++++++++++++++++++++++ 4 files changed, 285 insertions(+), 0 deletions(-) create mode 100644 bin/varnishtest/tests/t00004.vtc create mode 100644 bin/varnishtest/tests/t00005.vtc create mode 100644 bin/varnishtest/tests/t00006.vtc create mode 100644 bin/varnishtest/tests/t00007.vtc diff --git a/bin/varnishtest/tests/t00004.vtc b/bin/varnishtest/tests/t00004.vtc new file mode 100644 index 0000000..e7667e9 --- /dev/null +++ b/bin/varnishtest/tests/t00004.vtc @@ -0,0 +1,76 @@ +varnishtest "Test multiple streaming recepients in pass mode" + +server s1 { + rxreq + txresp -nolen -hdr "Transfer-encoding: chunked" + chunked "<1>------------------------<1>\n" + sema r1 sync 2 + chunked "<2>------------------------<2>\n" + sema r2 sync 2 + chunkedlen 0 +} -start + +server s2 { + rxreq + txresp -nolen -hdr "Transfer-encoding: chunked" + chunked "<1>------------------------<1>\n" + sema r3 sync 2 + chunked "<2>------------------------<2>\n" + sema r4 sync 2 + chunkedlen 0 +} -start + +varnish v1 -vcl+backend { + sub vcl_recv { + if (req.http.client == "c1") { + set req.backend = s1; + } else { + set req.backend = s2; + } + return (pass); + } + + sub vcl_fetch { + set beresp.do_stream = true; + } +} -start + +client c1 { + txreq -hdr "client: c1" -hdr "foo: /foo" + rxresp -no_obj + + rxchunk + expect resp.chunklen == 31 + sema r1 sync 2 + + rxchunk + expect resp.chunklen == 31 + sema r2 sync 2 + + rxchunk + expect resp.chunklen == 0 + expect resp.bodylen == 62 +} -start + +client c2 { + txreq -hdr "client: c2" -hdr "foo: /foo" + rxresp -no_obj + + rxchunk + expect resp.chunklen == 31 + sema r3 sync 2 + + rxchunk + expect resp.chunklen == 31 + sema r4 sync 2 + + rxchunk + expect resp.chunklen == 0 + expect resp.bodylen == 62 +} -start + +client c1 -wait +client c2 -wait + +varnish v1 -expect fetch_threaded == 2 +varnish v1 -expect s_streamed == 2 diff --git a/bin/varnishtest/tests/t00005.vtc b/bin/varnishtest/tests/t00005.vtc new file mode 100644 index 0000000..d6fc47a --- /dev/null +++ b/bin/varnishtest/tests/t00005.vtc @@ -0,0 +1,77 @@ +varnishtest "Test multiple streaming recepients in hit-for-pass mode" + +server s1 { + rxreq + txresp -nolen -hdr "Transfer-encoding: chunked" + chunked "<1>------------------------<1>\n" + sema r1 sync 2 + chunked "<2>------------------------<2>\n" + sema r2 sync 2 + chunkedlen 0 +} -start + +server s2 { + rxreq + txresp -nolen -hdr "Transfer-encoding: chunked" + chunked "<1>------------------------<1>\n" + sema r3 sync 2 + chunked "<2>------------------------<2>\n" + sema r4 sync 2 + chunkedlen 0 +} -start + +varnish v1 -vcl+backend { + sub vcl_recv { + if (req.http.client == "c1") { + set req.backend = s1; + } else { + set req.backend = s2; + } + return (pass); + } + + sub vcl_fetch { + set beresp.do_stream = true; + return (hit_for_pass); + } +} -start + +client c1 { + txreq -hdr "client: c1" -hdr "foo: /foo" + rxresp -no_obj + + rxchunk + expect resp.chunklen == 31 + sema r1 sync 2 + + rxchunk + expect resp.chunklen == 31 + sema r2 sync 2 + + rxchunk + expect resp.chunklen == 0 + expect resp.bodylen == 62 +} -start + +client c2 { + txreq -hdr "client: c2" -hdr "foo: /foo" + rxresp -no_obj + + rxchunk + expect resp.chunklen == 31 + sema r3 sync 2 + + rxchunk + expect resp.chunklen == 31 + sema r4 sync 2 + + rxchunk + expect resp.chunklen == 0 + expect resp.bodylen == 62 +} -start + +client c1 -wait +client c2 -wait + +varnish v1 -expect fetch_threaded == 2 +varnish v1 -expect s_streamed == 2 diff --git a/bin/varnishtest/tests/t00006.vtc b/bin/varnishtest/tests/t00006.vtc new file mode 100644 index 0000000..76fd8ca --- /dev/null +++ b/bin/varnishtest/tests/t00006.vtc @@ -0,0 +1,57 @@ +varnishtest "Test multiple streaming recepients" + +server s1 { + rxreq + txresp -nolen -hdr "Transfer-encoding: chunked" + chunked "<1>------------------------<1>\n" + sema r1 sync 3 + chunked "<2>------------------------<2>\n" + sema r2 sync 3 + chunkedlen 0 +} -start + +varnish v1 -vcl+backend { + sub vcl_fetch { + set beresp.do_stream = true; + } +} -start + +client c1 { + txreq -hdr "foo: /foo" + rxresp -no_obj + + rxchunk + expect resp.chunklen == 31 + sema r1 sync 3 + + rxchunk + expect resp.chunklen == 31 + sema r2 sync 3 + + rxchunk + expect resp.chunklen == 0 + expect resp.bodylen == 62 +} -start + +client c2 { + txreq -hdr "foo: /foo" + rxresp -no_obj + + rxchunk + expect resp.chunklen == 31 + sema r1 sync 3 + + rxchunk + expect resp.chunklen == 31 + sema r2 sync 3 + + rxchunk + expect resp.chunklen == 0 + expect resp.bodylen == 62 +} -start + +client c1 -wait +client c2 -wait + +varnish v1 -expect fetch_threaded == 1 +varnish v1 -expect s_streamed == 2 diff --git a/bin/varnishtest/tests/t00007.vtc b/bin/varnishtest/tests/t00007.vtc new file mode 100644 index 0000000..bd3f0ed --- /dev/null +++ b/bin/varnishtest/tests/t00007.vtc @@ -0,0 +1,75 @@ +varnishtest "Test multiple rushing of waiting list when streaming" + +server s1 { + rxreq + txresp -nolen -hdr "Transfer-encoding: chunked" + chunked "<1>------------------------<1>\n" + sema r1 sync 5 + chunkedlen 0 +} -start + +varnish v1 -arg "-p rush_exponent=2" -vcl+backend { + sub vcl_fetch { + set beresp.do_stream = true; + } +} -start + +client c1 { + txreq -hdr "foo: /foo" + rxresp -no_obj + + rxchunk + expect resp.chunklen == 31 + sema r1 sync 5 + + rxchunk + expect resp.chunklen == 0 + expect resp.bodylen == 31 +} -start + +client c2 { + txreq -hdr "foo: /foo" + rxresp -no_obj + + rxchunk + expect resp.chunklen == 31 + sema r1 sync 5 + + rxchunk + expect resp.chunklen == 0 + expect resp.bodylen == 31 +} -start + +client c3 { + txreq -hdr "foo: /foo" + rxresp -no_obj + + rxchunk + expect resp.chunklen == 31 + sema r1 sync 5 + + rxchunk + expect resp.chunklen == 0 + expect resp.bodylen == 31 +} -start + +client c4 { + txreq -hdr "foo: /foo" + rxresp -no_obj + + rxchunk + expect resp.chunklen == 31 + sema r1 sync 5 + + rxchunk + expect resp.chunklen == 0 + expect resp.bodylen == 31 +} -start + +client c1 -wait +client c2 -wait +client c3 -wait +client c4 -wait + +varnish v1 -expect fetch_threaded == 1 +varnish v1 -expect s_streamed == 4 -- 1.7.4.1 From martin at varnish-software.com Sun Jan 22 17:53:17 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Sun, 22 Jan 2012 18:53:17 +0100 Subject: [PATCH 11/25] Lock busyobj when doing late object header changes (end of FetchBody creating a Content-Length header), and when copying the object headers in RES_BuildHTTP. In-Reply-To: <1327254811-29629-1-git-send-email-martin@varnish-software.com> References: <1327254811-29629-1-git-send-email-martin@varnish-software.com> Message-ID: <1327254811-29629-11-git-send-email-martin@varnish-software.com> Add general lock functions for busyobjs --- bin/varnishd/cache/cache.h | 2 ++ bin/varnishd/cache/cache_busyobj.c | 14 ++++++++++++++ bin/varnishd/cache/cache_center.c | 10 +++++++++- bin/varnishd/cache/cache_fetch.c | 4 ++++ 4 files changed, 29 insertions(+), 1 deletions(-) diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index b59a217..da06965 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -750,6 +750,8 @@ struct busyobj *VBO_GetBusyObj(struct worker *wrk); struct busyobj *VBO_RefBusyObj(struct busyobj *busyobj); unsigned VBO_DerefBusyObj(struct worker *wrk, struct busyobj **busyobj); void VBO_Free(struct vbo **vbo); +void VBO_LockBusyObj(struct busyobj *busyobj); +void VBO_UnlockBusyObj(struct busyobj *busyobj); void VBO_StreamStopped(struct busyobj *busyobj); void VBO_StreamWait(struct busyobj *busyobj); void VBO_StreamData(struct busyobj *busyobj); diff --git a/bin/varnishd/cache/cache_busyobj.c b/bin/varnishd/cache/cache_busyobj.c index 7651999..6eec464 100644 --- a/bin/varnishd/cache/cache_busyobj.c +++ b/bin/varnishd/cache/cache_busyobj.c @@ -209,6 +209,20 @@ VBO_DerefBusyObj(struct worker *wrk, struct busyobj **pbo) return (r); } +void +VBO_LockBusyObj(struct busyobj *busyobj) +{ + if (busyobj->use_locks) + Lck_Lock(&busyobj->vbo->mtx); +} + +void +VBO_UnlockBusyObj(struct busyobj *busyobj) +{ + if (busyobj->use_locks) + Lck_Unlock(&busyobj->vbo->mtx); +} + /* Signal that the fetch thread has stopped */ void VBO_StreamStopped(struct busyobj *busyobj) diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c index 55e06c6..386012c 100644 --- a/bin/varnishd/cache/cache_center.c +++ b/bin/varnishd/cache/cache_center.c @@ -285,7 +285,15 @@ cnt_prepresp(struct sess *sp, struct worker *wrk, struct req *req) req->obj->last_use = req->t_resp; /* XXX: locking ? */ } http_Setup(req->resp, req->ws); - RES_BuildHttp(sp); + if (wrk->busyobj != NULL) { + /* We are streaming, read the object headers while + * holding the lock as the fetching thread might + * update these headers (set Content-Length) */ + VBO_LockBusyObj(wrk->busyobj); + RES_BuildHttp(sp); + VBO_UnlockBusyObj(wrk->busyobj); + } else + RES_BuildHttp(sp); VCL_deliver_method(sp); switch (req->handling) { case VCL_RET_DELIVER: diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index f69d80f..d5ae4d5 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -616,9 +616,13 @@ FetchBody(struct worker *wrk, struct busyobj *bo) } if (mklen > 0) { + /* Lock the busyobj during late header changes, as a + * streaming client may be reading these */ + VBO_LockBusyObj(bo); http_Unset(obj->http, H_Content_Length); http_PrintfHeader(wrk, bo->vbc->vsl_id, obj->http, "Content-Length: %zd", obj->len); + VBO_UnlockBusyObj(bo); } if (cls) -- 1.7.4.1 From martin at varnish-software.com Sun Jan 22 17:53:18 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Sun, 22 Jan 2012 18:53:18 +0100 Subject: [PATCH 12/25] Use background thread fetching when streaming In-Reply-To: <1327254811-29629-1-git-send-email-martin@varnish-software.com> References: <1327254811-29629-1-git-send-email-martin@varnish-software.com> Message-ID: <1327254811-29629-12-git-send-email-martin@varnish-software.com> --- bin/varnishd/cache/cache.h | 5 ++ bin/varnishd/cache/cache_center.c | 96 +++++++++++++++++++++++++++++----- bin/varnishd/cache/cache_response.c | 84 ++++++++++++++++++++++-------- 3 files changed, 149 insertions(+), 36 deletions(-) diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index da06965..880f5e3 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -537,6 +537,7 @@ struct busyobj { unsigned do_gzip; unsigned do_gunzip; unsigned do_stream; + unsigned do_stream_flipflop; /* Stream stuff */ ssize_t stream_max; @@ -793,6 +794,8 @@ int FetchError(struct worker *w, const char *error); int FetchError2(struct worker *w, const char *error, const char *more); int FetchHdr(struct sess *sp, int need_host_hdr); int FetchBody(struct worker *w, struct busyobj *bo); +void FetchBodyBackground(struct sess *sp, struct busyobj *bo); +void FetchBodyWait(struct busyobj *bo); int FetchReqBody(const struct sess *sp); void Fetch_Init(void); @@ -999,6 +1002,8 @@ void WSL_Flush(struct worker *w, int overflow); void RES_BuildHttp(const struct sess *sp); void RES_WriteObj(struct sess *sp); void RES_StreamStart(struct sess *sp); +void RES_StreamBody(struct sess *sp); +void RES_StreamWrite(struct sess *sp); void RES_StreamEnd(struct sess *sp); void RES_StreamPoll(struct worker *wrk); diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c index 386012c..1f1a239 100644 --- a/bin/varnishd/cache/cache_center.c +++ b/bin/varnishd/cache/cache_center.c @@ -963,13 +963,47 @@ DOT } DOT streambody -> DONE [style=bold,color=cyan] */ +/* Background fetch task. Should be called with ref on busyobj, and + the objcore if present */ + +static void +cnt_streambody_task(struct worker *wrk, void *priv) +{ + struct object *obj; + struct objcore *objcore; + unsigned u; + + AZ(wrk->busyobj); + CAST_OBJ_NOTNULL(wrk->busyobj, priv, BUSYOBJ_MAGIC); + AN(wrk->busyobj->use_locks); + + CHECK_OBJ_NOTNULL(wrk->busyobj->fetch_obj, OBJECT_MAGIC); + AN(wrk->busyobj->vbc); + obj = wrk->busyobj->fetch_obj; + objcore = obj->objcore; + + wrk->busyobj->fetch_failed = FetchBody(wrk, wrk->busyobj); + AZ(wrk->busyobj->fetch_obj); + AZ(wrk->busyobj->vbc); + wrk->busyobj->vfp = NULL; + VBO_StreamStopped(wrk->busyobj); + + u = VBO_DerefBusyObj(wrk, &wrk->busyobj); + if (objcore != NULL || u == 0) { + /* Only deref object if it has it's own refcnt, or we + * were the last to deref the busyobj */ + (void)HSH_Deref(wrk, NULL, &obj); + } +} + static int cnt_streambody(struct sess *sp, struct worker *wrk, struct req *req) { - int i; struct stream_ctx sctx; uint8_t obuf[sp->wrk->res_mode & RES_GUNZIP ? cache_param->gzip_stack_buffer : 1]; + struct worker *wrk_ex; + unsigned u; CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); @@ -987,28 +1021,56 @@ cnt_streambody(struct sess *sp, struct worker *wrk, struct req *req) sctx.obuf_len = sizeof (obuf); } - RES_StreamStart(sp); - AssertObjCorePassOrBusy(req->obj->objcore); + RES_StreamStart(sp); + /* MBGXXX: Test on OC_F_BUSY to see if we should initiate + * fetch at all. This code now assumes all passes through here + * needs to do the fetch as well. (Multiple streaming clients + * not implemented yet) */ AZ(wrk->busyobj->fetch_obj); wrk->busyobj->fetch_obj = req->obj; - i = FetchBody(wrk, wrk->busyobj); - AZ(wrk->busyobj->fetch_obj); - http_Setup(wrk->busyobj->bereq, NULL); http_Setup(wrk->busyobj->beresp, NULL); - wrk->busyobj->vfp = NULL; - AZ(wrk->busyobj->vbc); - AN(req->director); + wrk_ex = SES_GrabWorker(sp, 100); /* MBGXXX: Configurable + * thread grabbing + * timeout */ + if (wrk_ex != NULL) { + /* Set up separate thread fetch */ + wrk->busyobj->use_locks = 1; + if (req->obj->objcore != NULL) + /* Grab a ref on the objcore for the other thread */ + HSH_Ref(req->obj->objcore); + VBO_RefBusyObj(wrk->busyobj); /* Ref for the other thread */ + WRK_DoTask(wrk_ex, cnt_streambody_task, wrk->busyobj); + } else { + /* We have no worker */ + if (wrk->busyobj->fetch_obj->objcore == NULL || + wrk->busyobj->fetch_obj->objcore->flags & OC_F_PASS) { + /* It's a pass, prefer flipflop + * streaming. (MBGXXX: Flipflop not finished + * yet) */ + wrk->busyobj->do_stream_flipflop = 1; + } + wrk->busyobj->fetch_failed = FetchBody(sp->wrk, wrk->busyobj); + VBO_StreamStopped(wrk->busyobj); + } - if (!i && req->obj->objcore != NULL) { + RES_StreamBody(sp); + + if (wrk->busyobj->htc.ws == wrk->ws) + /* Busyobj's htc has buffer on our workspace, + wait for it to be released */ + VBO_StreamWait(wrk->busyobj); + + if (wrk->busyobj->fetch_failed) { + req->doclose = "Stream error"; + } else if (req->obj->objcore != NULL) { + /* MBGXXX: This should be done on the bg task */ EXP_Insert(req->obj); AN(req->obj->objcore); AN(req->obj->objcore->ban); HSH_Unbusy(wrk); - } else { - req->doclose = "Stream error"; } wrk->acct_tmp.fetch++; req->director = NULL; @@ -1021,8 +1083,14 @@ cnt_streambody(struct sess *sp, struct worker *wrk, struct req *req) wrk->sctx = NULL; assert(WRW_IsReleased(wrk)); assert(wrk->wrw.ciov == wrk->wrw.siov); - (void)HSH_Deref(wrk, NULL, &req->obj); - (void)VBO_DerefBusyObj(wrk, &wrk->busyobj); + u = VBO_DerefBusyObj(wrk, &wrk->busyobj); + if (req->obj->objcore != NULL || u == 0) { + /* Only deref object if it has it's own refcnt, or we + * were the last to deref the busyobj */ + (void)HSH_Deref(wrk, NULL, &req->obj); + } else + /* Object will be deref'ed by fetch thread */ + req->obj = NULL; http_Setup(req->resp, NULL); sp->step = STP_DONE; return (0); diff --git a/bin/varnishd/cache/cache_response.c b/bin/varnishd/cache/cache_response.c index cb6ddd6..a49acbe 100644 --- a/bin/varnishd/cache/cache_response.c +++ b/bin/varnishd/cache/cache_response.c @@ -366,22 +366,39 @@ RES_StreamStart(struct sess *sp) } void -RES_StreamPoll(struct worker *wrk) +RES_StreamBody(struct sess *sp) { struct stream_ctx *sctx; + struct busyobj *bo; + + sctx = sp->wrk->sctx; + CHECK_OBJ_NOTNULL(sctx, STREAM_CTX_MAGIC); + bo = sp->wrk->busyobj; + CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC); + AN(sp->req->wantbody); + + while (!sctx->stream_stopped || sctx->stream_next < sctx->stream_max) { + VBO_StreamSync(sp->wrk); + RES_StreamWrite(sp); + } +} + +void +RES_StreamWrite(struct sess *sp) +{ + struct worker *wrk; + struct stream_ctx *sctx; struct storage *st; - struct object *fetch_obj; ssize_t l, l2, stlen; void *ptr; + CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); + wrk = sp->wrk; CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); CHECK_OBJ_NOTNULL(wrk->sp->req->obj, OBJECT_MAGIC); sctx = wrk->sctx; CHECK_OBJ_NOTNULL(sctx, STREAM_CTX_MAGIC); - VBO_StreamData(wrk->busyobj); - VBO_StreamSync(wrk); - if (sctx->stream_max == sctx->stream_next) return; assert(sctx->stream_max > sctx->stream_next); @@ -418,23 +435,6 @@ RES_StreamPoll(struct worker *wrk) } if (!(wrk->res_mode & RES_GUNZIP)) (void)WRW_Flush(wrk); - - if (wrk->busyobj->stream_frontchunk == NULL) - return; - - /* It's a pass - remove chunks already delivered */ - fetch_obj = wrk->busyobj->fetch_obj; - CHECK_OBJ_NOTNULL(fetch_obj, OBJECT_MAGIC); - assert(fetch_obj->objcore == NULL || - (fetch_obj->objcore->flags & OC_F_PASS)); - while (1) { - st = VTAILQ_FIRST(&fetch_obj->store); - if (st == NULL || st == wrk->busyobj->stream_frontchunk) - break; - CHECK_OBJ_NOTNULL(st, STORAGE_MAGIC); - VTAILQ_REMOVE(&fetch_obj->store, st, list); - STV_free(st); - } } void @@ -453,3 +453,43 @@ RES_StreamEnd(struct sess *sp) if (WRW_FlushRelease(sp->wrk)) SES_Close(sp, "remote closed"); } + +void +RES_StreamPoll(struct worker *wrk) +{ + struct object *fetch_obj; + struct storage *st; + + CHECK_OBJ_NOTNULL(wrk->busyobj, BUSYOBJ_MAGIC); + + VBO_StreamData(wrk->busyobj); + if (wrk->busyobj->do_stream_flipflop == 1) { + AN(wrk->sctx); + /* MBGXXX: Do flip-flop streaming */ + /* MBGXXX: Loop around waiting for the lag behind to + * be less than some configurable size, to keep the + * cache memory usage low (this for streaming + * extremely large objects with pass) */ + VBO_StreamSync(wrk); + RES_StreamWrite(wrk->sp); + } + + if (wrk->busyobj->stream_frontchunk == NULL) + return; + + /* It's a pass - remove chunks already delivered. Should be OK + * to do lock-free, as we are not fiddling pointers of any + * storage chunk passed busyobj->stream_frontchunk */ + fetch_obj = wrk->busyobj->fetch_obj; + CHECK_OBJ_NOTNULL(fetch_obj, OBJECT_MAGIC); + assert(fetch_obj->objcore == NULL || + (fetch_obj->objcore->flags & OC_F_PASS)); + while (1) { + st = VTAILQ_FIRST(&fetch_obj->store); + if (st == NULL || st == wrk->busyobj->stream_frontchunk) + break; + CHECK_OBJ_NOTNULL(st, STORAGE_MAGIC); + VTAILQ_REMOVE(&fetch_obj->store, st, list); + STV_free(st); + } +} -- 1.7.4.1 From bedis9 at gmail.com Thu Jan 5 20:55:16 2012 From: bedis9 at gmail.com (Baptiste) Date: Thu, 05 Jan 2012 20:55:16 -0000 Subject: Welcome to VUG5 in Paris, France - March 22-23, 2012 In-Reply-To: References: Message-ID: Hey Ruben, Just registred! I'm looking forward to meet you again. cheers 2012/1/5 Rub?n Romero > Hello to you all, > > The year has started and we need to plan ahead. So it is my pleasure to > announce that our Fifth Varnish User Group meeting, VUG5, will happen in > Paris, France. The dates are March 22nd (User Day) and 23rd (Dev Day), > 2012. For all the gory details and registration visit the following page on > the Varnish Cache community website: > > > > As we have a rather limited number of seats we need to know the number of > assistants in advance, so register even if you are not 100% sure about > coming to Paris. We will send you a confirmation reminder later. > > We need YOU! -> Do not forget to add your favorite discussion item (or > your own Varnish use case) to the agenda. This goes both for the User and > Dev days. We want to know how you use the software, which problems you > solve with it and what challenges you have so we can together make an even > better Varnish Cache! > > If you have *any* questions related to this meeting, do not hesitate in > contacting me directly and I will get back to you. > > Last but not least: Hope to see you all in Paris! > > > Best wishes, > -- > Rub?n Romero, Self-appointed <3 Non-Official VUG Coordinator & Cheerleader > Phone: +47 21 98 92 62 / Mobile: +47 959 64 088 / Skype: ruben_varnish / > GTalk: > *Varnish makes websites fly!* > Whitepapers | Video > | Twitter | LinkedIn > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From drwilco at drwilco.net Fri Jan 6 03:26:50 2012 From: drwilco at drwilco.net (Rogier R. Mulhuijzen) Date: Fri, 06 Jan 2012 03:26:50 -0000 Subject: [PATCH] Drop old grace objects when we have a new object that matches the Vary In-Reply-To: <1325820122-7794-1-git-send-email-github@bsdchicks.com> References: <1325820122-7794-1-git-send-email-github@bsdchicks.com> Message-ID: <20120106042609.L37583@ishtar.drwilco.net> And of course, I forgot to take out a debugging stat counter. Please ignore its existence. From cyberroadie at gmail.com Fri Jan 6 17:21:06 2012 From: cyberroadie at gmail.com (Olivier Van Acker) Date: Fri, 06 Jan 2012 17:21:06 -0000 Subject: new module Message-ID: Hi, I created a new varnish module, it orders the parameters in an url alphabetically Code can be found here: https://github.com/cyberroadie/varnish-urlsort And a blogpost describing it here: http://cyberroadie.wordpress.com/2012/01/05/varnish-reordering-query-string/ Is it possible to get it listed on the https://www.varnish-cache.org/project/modules page? Olivier -------------- next part -------------- An HTML attachment was scrubbed... URL: