From phk at phk.freebsd.dk Mon May 5 07:29:24 2014 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 05 May 2014 07:29:24 +0000 Subject: [PATCH] Remember to signal the thread to exit when decimating the flock In-Reply-To: <1398858672-8534-1-git-send-email-martin@varnish-software.com> References: <1398858672-8534-1-git-send-email-martin@varnish-software.com> Message-ID: <74886.1399274964@critter.freebsd.dk> In message <1398858672-8534-1-git-send-email-martin at varnish-software.com>, Mart in Blix Grydeland writes: Good catch. > if (wrk != NULL) { >+ /* Update counters */ Is this really an XXX comment ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From slink at schokola.de Mon May 5 11:40:21 2014 From: slink at schokola.de (Nils Goroll) Date: Mon, 05 May 2014 13:40:21 +0200 Subject: [PATCH] [EXPERIMENTAL] autocrap autohardening In-Reply-To: <5329F3C8.2040707@schokola.de> References: <52CEDCFE.2090508@schokola.de> <20140130074848.GB10577@err.no> <5329F3C8.2040707@schokola.de> Message-ID: <536778A5.8030707@schokola.de> brief summary of a #varnish-hacking discussion from my understanding (please correct or comment if you disagree): * we do want hardening options by default * The patch needs to be tested against gcc and llvm/clang * phk would like the compiler flag tests be implemented using the methodology from Makefile.phk / config.phk in favor of wrap-compiler-for-flag-check * look after vcc cflags related change: * we want three basic compiler-settings - developer - production - performance in the order of decreasing safety settings. People should be discouraged from using "performance" unless absolutely necessary From martin at varnish-software.com Mon May 5 12:24:48 2014 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 5 May 2014 14:24:48 +0200 Subject: [PATCH] Release backend connection before setting BOS_FINISHED Message-ID: <1399292688-22386-1-git-send-email-martin@varnish-software.com> This to give predictable backend reuse behavior for varnishtest, making the test cases non-racy. Fixes: 1491 --- bin/varnishd/cache/cache_fetch.c | 35 ++++++++++++++++++++++++++++------- 1 file changed, 28 insertions(+), 7 deletions(-) diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index dded999..32790cf 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -40,6 +40,23 @@ #include "vtim.h" /*-------------------------------------------------------------------- + * Release the VBC of the busyobj if still open + */ +static void +vbf_release_vbc(struct busyobj *bo) +{ + + CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC); + if (bo->vbc != NULL) { + if (bo->should_close) + VDI_CloseFd(&bo->vbc, &bo->acct); + else + VDI_RecycleFd(&bo->vbc, &bo->acct); + AZ(bo->vbc); + } +} + +/*-------------------------------------------------------------------- * Allocate an object, with fall-back to Transient. * XXX: This somewhat overlaps the stuff in stevedore.c * XXX: Should this be merged over there ? @@ -548,6 +565,11 @@ vbf_stp_fetch(struct worker *wrk, struct busyobj *bo) HSH_Unbusy(&wrk->stats, obj->objcore); } VSLb_ts_busyobj(bo, "BerespBody", W_TIM_real(wrk)); + + /* Release the backend connection before setting BOS_FINISHED to + give predictable backend reuse behavior for varnishtest */ + vbf_release_vbc(bo); + VBO_setstate(bo, BOS_FINISHED); return (F_STP_DONE); } @@ -639,6 +661,11 @@ vbf_stp_condfetch(struct worker *wrk, struct busyobj *bo) assert(al == bo->ims_obj->len); assert(obj->len == al); EXP_Rearm(bo->ims_obj, bo->ims_obj->exp.t_origin, 0, 0, 0); + + /* Release the backend connection before setting BOS_FINISHED to + give predictable backend reuse behavior for varnishtest */ + vbf_release_vbc(bo); + VBO_setstate(bo, BOS_FINISHED); VSLb_ts_busyobj(bo, "BerespBody", W_TIM_real(wrk)); return (F_STP_DONE); @@ -789,13 +816,7 @@ vbf_fetch_thread(struct worker *wrk, void *priv) bo->stats = NULL; - if (bo->vbc != NULL) { - if (bo->should_close) - VDI_CloseFd(&bo->vbc, &bo->acct); - else - VDI_RecycleFd(&bo->vbc, &bo->acct); - AZ(bo->vbc); - } + vbf_release_vbc(bo); http_Teardown(bo->bereq); http_Teardown(bo->beresp); -- 1.9.2 From martin at varnish-software.com Mon May 5 14:25:24 2014 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 5 May 2014 16:25:24 +0200 Subject: [PATCH] Recycle backend connection before setting BOS_FINISHED (take two) Message-ID: <1399299924-28085-1-git-send-email-martin@varnish-software.com> This to give predictable backend reuse behavior for varnishtest, making the test cases non-racy. 2nd version of this patch. This version will only deal with recyclable backend connections before BOS_FINISHED, leaving the slower close path to the clean up code that is run after the BOS_FINISHED state. Fixes: #1491 --- bin/varnishd/cache/cache_fetch.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index dded999..ac4c478 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -548,6 +548,14 @@ vbf_stp_fetch(struct worker *wrk, struct busyobj *bo) HSH_Unbusy(&wrk->stats, obj->objcore); } VSLb_ts_busyobj(bo, "BerespBody", W_TIM_real(wrk)); + + /* Recycle the backend connection before setting BOS_FINISHED to + give predictable backend reuse behavior for varnishtest */ + if (bo->vbc != NULL && !(bo->should_close)) { + VDI_RecycleFd(&bo->vbc, &bo->acct); + AZ(bo->vbc); + } + VBO_setstate(bo, BOS_FINISHED); return (F_STP_DONE); } @@ -639,6 +647,14 @@ vbf_stp_condfetch(struct worker *wrk, struct busyobj *bo) assert(al == bo->ims_obj->len); assert(obj->len == al); EXP_Rearm(bo->ims_obj, bo->ims_obj->exp.t_origin, 0, 0, 0); + + /* Recycle the backend connection before setting BOS_FINISHED to + give predictable backend reuse behavior for varnishtest */ + if (bo->vbc != NULL && !(bo->should_close)) { + VDI_RecycleFd(&bo->vbc, &bo->acct); + AZ(bo->vbc); + } + VBO_setstate(bo, BOS_FINISHED); VSLb_ts_busyobj(bo, "BerespBody", W_TIM_real(wrk)); return (F_STP_DONE); -- 1.9.2 From slink at schokola.de Mon May 5 16:38:17 2014 From: slink at schokola.de (Nils Goroll) Date: Mon, 05 May 2014 18:38:17 +0200 Subject: auto-generated rst docs: top_srcdir vs top_builddir Message-ID: <5367BE79.5050603@schokola.de> Working on https://www.varnish-cache.org/trac/changeset/5321145882f6edd65bbeae24a7ff23cc6985f57e/ I noticed that the same inconsistencies exist for the docs - authored vs. generated: When srcdir != builddir, paths to generated rsts are wrong, so building in a separate builddir does not work, e.g.: # pristine source tree slink at haggis:/tmp/varnish-cache$ mkdir build slink at haggis:/tmp/varnish-cache$ cd build slink at haggis:/tmp/varnish-cache/build$ ../configure ... slink at haggis:/tmp/varnish-cache/build$ ../make ... make[2]: Entering directory `/tmp/varnish-cache/build/man' CC vsc2rst.o CCLD vsc2rst rst2man ../../doc/sphinx/reference/varnish-cli.rst varnish-cli.7 ./vsc2rst | rst2man - varnish-counters.7 rst2man ../../doc/sphinx/reference/vcl.rst vcl.7 ../../doc/sphinx/reference/vcl.rst:347: (SEVERE/4) Problems with "include" directive path: InputError: [Errno 2] No such file or directory: '../../doc/sphinx/include/vcl_var.rst'. Exiting due to level-4 (SEVERE) system message. make[2]: *** [vcl.7] Error 1 make[2]: Leaving directory `/tmp/varnish-cache/build/man' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/tmp/varnish-cache/build' make: *** [all] Error 2 I'd suggest the attached patch, but it is pretty useless as long as we can't use dynamic paths in .rst like this (does not work! exemplary only!): --- a/doc/sphinx/reference/vcl.rst +++ b/doc/sphinx/reference/vcl.rst @@ -344,7 +344,7 @@ In VCL you have access to certain variable objects. These contain requests and responses currently being worked on. What variables are available depends on context. -.. include:: ../include/vcl_var.rst +.. include:: $(top_builddir)/doc/sphinx/include/vcl_var.rst Functions Does anyone have a good solution here (except for "separate build directory not supported")? Nils -------------- next part -------------- A non-text attachment was scrubbed... Name: docs_srcdir_vs_builddir.patch Type: text/x-patch Size: 2855 bytes Desc: not available URL: From slink at schokola.de Mon May 5 16:55:08 2014 From: slink at schokola.de (Nils Goroll) Date: Mon, 05 May 2014 18:55:08 +0200 Subject: automake 1.14 and subdir-objects Message-ID: <5367C26C.4050008@schokola.de> The automake authors are planning to activate subdir-objects unconditionally: http://lists.gnu.org/archive/html/automake/2013-06/msg00040.html - The next major Automake version (2.0) will unconditionally activate the 'subdir-objects' option. In order to smooth out the transition, we now give a warning (in the category 'unsupported') whenever a source file is present in a subdirectory but the 'subdir-object' is not enabled. automake 1.14 complains when building Varnish: bin/varnishadm/Makefile.am:9: warning: source file '$(top_srcdir)/lib/libvarnish/vsa.c' is in a subdirectory, bin/varnishadm/Makefile.am:9: but option 'subdir-objects' is disabled bin/varnishadm/Makefile.am:9: warning: source file '$(top_srcdir)/lib/libvarnish/vtcp.c' is in a subdirectory, bin/varnishadm/Makefile.am:9: but option 'subdir-objects' is disabled (yaddayadda) I have checked if we could enable subdir-objects now, but we basically run into the dilemma summarized here: http://curl.haxx.se/mail/lib-2013-07/0172.html IIUC, the punchline is that we may choose from these options: - hope for automake maintainers fix subdir-objects and ignore the warnings for the time being - use subdir-objects, but disable dependency tracking - hack our Makefiles to work around (don't) Nils From slink at schokola.de Mon May 5 17:20:56 2014 From: slink at schokola.de (Nils Goroll) Date: Mon, 05 May 2014 19:20:56 +0200 Subject: [PATCH] SO_REUSEPORT ? Message-ID: <5367C878.3010406@schokola.de> On Solaris 11, I see intermittent varnishtest failures on arbitrary tests due to bind failing with EADDRINUSE : # previous test finishing up ... 22591/1: nanosleep(0xFFFF80FFBFFFE010, 0x00000000) = 0 22591/1: tmout: 0.100000000 sec 22591/1: lwp_wait(2, 0xFFFF80FFBFFFDF9C) = 0 22591/1: lwpid: 2 22591/1: close(3) = 0 22591/1: so_socket(PF_INET, SOCK_STREAM, IPPROTO_TCP, 0, SOV_DEFAULT) = 3 22591/1: setsockopt(3, SOL_SOCKET, SO_REUSEADDR, 0xFFFF80FFBFFFDFA8, 4, SOV_DEFAULT) = 0 22591/1: bind(3, 0x004B8660, 16, SOV_SOCKBSD) Err#125 EADDRINUSE 22591/1: AF_INET name = 127.0.0.1 port = 45199 22591/1: write(2, " b i n d ( )", 6) = 6 22591/1: write(2, " : ", 2) = 2 22578: pollsys(0x00431F80, 1, 0xFFFF80FFBFFFF7E0, 0x00000000) = 1 22578: fd=3 ev=POLLIN|POLLERR|POLLHUP rev=POLLIN 22578: timeout: 59.999000000 sec 22591/1: write(2, 0xFFFF80FFBF48A4CE, 22) = 22 22591/1: A d d r e s s a l r e a d y i n u s e 22591/1: write(2, "\n", 1) = 1 ... The problem can be avoided by not only setting SO_REUSEADDR, but also SO_REUSEPORT in VCC_bind as shown in the attached patch. According to http://lwn.net/Articles/542629/, this could also help performance on huge multiple-acceptor-thread-setups: the SO_REUSEPORT implementation distributes connections evenly across all of the threads (or processes) that are blocked in accept() on the same port. Nils -------------- next part -------------- A non-text attachment was scrubbed... Name: 0004-use-SO_REUSEPORT-where-availab-e.patch Type: text/x-patch Size: 1434 bytes Desc: not available URL: From slink at schokola.de Mon May 5 18:31:06 2014 From: slink at schokola.de (Nils Goroll) Date: Mon, 05 May 2014 20:31:06 +0200 Subject: [PATCH] SO_REUSEPORT ? In-Reply-To: <5367C878.3010406@schokola.de> References: <5367C878.3010406@schokola.de> Message-ID: <5367D8EA.3050300@schokola.de> A note on an irc chat with phk: The issue can be avoided by not re-opening the socket in vtc_server (that often). At this point it remains unclear to me why Solaris returns EADDRINUSE sometimes. The suggestion to use SO_REUSEPORT was of more general nature, as I stumbled over it. Another side note: With many repetitions of c56.vtc with SO_REUSEPORT enabled, I got a listen(3, 10, SOV_DEFAULT) Err#122 EOPNOTSUPP From fgsch at lodoss.net Tue May 6 01:43:11 2014 From: fgsch at lodoss.net (Federico Schwindt) Date: Tue, 6 May 2014 02:43:11 +0100 Subject: Thoughs on VMODs and VCL Message-ID: Hi, As I mentioned a while ago on #varnish-hacking I've been thinking it'd be kinda nice to have a delete keyword. One of the reasons was to use malloc'd memory. The other was to express transformations like: mod.func1(mod.func2(mod.func3(foo))) In what I think a more friendly way: bar.func1(); bar.func2(); bar.func3(); Or perhaps even: bar.func1().func2().func3(); This weekend, while I was talking to Dag, I thought of another reason I'd like to see this in: to have per-request states. Currently this is done manually using ctx->req->sp->fd as an index to a malloc'd list. The other option is to use thread-specific data but I was explained this won't work over waitinglist'd requests. Both of these approaches only work for single states btw. A PRIV_REQ type will solve single state cases and has the extra benefit of somewhat automatic clean up when the request is done. Thoughts? f.- -------------- next part -------------- An HTML attachment was scrubbed... URL: From slink at schokola.de Tue May 6 06:39:16 2014 From: slink at schokola.de (Nils Goroll) Date: Tue, 06 May 2014 08:39:16 +0200 Subject: Thoughs on VMODs and VCL In-Reply-To: References: Message-ID: <53688394.3030301@schokola.de> Hi Federico, I must say I don't understand the first part of your email about the delete keyword and what you intend to express about calling vmod methods. On 06/05/14 03:43, Federico Schwindt wrote: > Currently this is done manually using ctx->req->sp->fd as an index to a malloc'd > list. The other option is to use thread-specific data but I was explained this > won't work over waitinglist'd requests. Both of these approaches only work for > single states btw. > > A PRIV_REQ type will solve single state cases and has the extra benefit of > somewhat automatic clean up when the request is done. > > Thoughts? This has got a long-standing vote from me too, the request lives here: https://www.varnish-cache.org/trac/wiki/Future_VCL#VMODs : * VMOD {{{PRIV_REQUEST}}} {{{PRIV_SESSION}}} state * array indexed by vmod id (for those using the state) hanging off session, request and busy obj * VMOD needs to define free callback - same as now * free will be called at the end of req/session * where to allocate memory from? * session: malloc or own pool * request: get from {{{req->ws}}}, no free required then From geoff at uplex.de Tue May 6 08:17:54 2014 From: geoff at uplex.de (Geoff Simmons) Date: Tue, 06 May 2014 10:17:54 +0200 Subject: Thoughs on VMODs and VCL In-Reply-To: <53688394.3030301@schokola.de> References: <53688394.3030301@schokola.de> Message-ID: <53689AB2.5070907@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 05/06/2014 08:39 AM, Nils Goroll wrote: > Hi Federico, > > I must say I don't understand the first part of your email about > the delete keyword and what you intend to express about calling > vmod methods. Frederico had written: > One of the reasons was to use malloc'd memory. That sounds to me as if VCL authors become responsible for memory management, in that they will have to free malloc'd memory, and could create memory leaks if they forget to delete. But maybe I'm misunderstanding, because it sounds very dangerous. +1 to not understanding the bit about the VMOD methods. >> A PRIV_REQ type will solve single state cases and has the extra >> benefit of somewhat automatic clean up when the request is done. >> >> Thoughts? > > This has got a long-standing vote from me too, the request lives > here: https://www.varnish-cache.org/trac/wiki/Future_VCL#VMODs : > > * VMOD {{{PRIV_REQUEST}}} {{{PRIV_SESSION}}} state +1 for PRIV_REQUEST and PRIV_SESSION, +2 or more if I could. I actually thought they would make it into Varnish 4.0. Another reason: Using the file descriptors as indexes into per-session state tables means that you have to allocate a table that is much larger than will usually be necessary. To be safe, it has to be as large as ulimit -n, and then you worry about whether the Varnish process owner can increase it ulimits. That doesn't have to be so bad, if it's just a table of pointers, but it's unnecessarily wasteful. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Icedove - http://www.enigmail.net/ iQIbBAEBCAAGBQJTaJqlAAoJEOUwvh9pJNURCzYP92KXuL0WgUGh4ngskE8ilvwW yfsJFd8cOvipEZUcnhS8gVPGNpnoHToVQ1fe4+zWWsD+fZx2osneyN6Q9pv7tlRo J3ei/moCnqQ1IwDyOdEVAWriV1x4jsxGR1uReFDqZCKmHzcOfzSkN8Osbel4gpXz PDkRR/pkMM14iziXyFx2Bumb28fYsva+sKpK4237+x6WmQigVSOHpvCblb9jx3SK Lz4E8BR63vGsB5e1dTofpyA4QKFIk8PdlSSm6rFYh+HELQiWabOmCJSskiwRycmi ujnmGhFEBLEzAfjXLL3jx7rUqxb7d7ie/4DByTD6CEXm6qeVmOkboYGHfudu05EZ HiTKNRItu9Q6+NZihbb4BgF0mC4SXgje/MlbsL9ldCBMnMScXxqulPGUCv+PoMLL 0I7zz6OC0HFqQxRuLy2I2M3+EMsjMCH/uXilLOhQJZOHOE9i68B8w01mx2rrkePJ nXmSzpVC8hg2lAEOCliP5irtYFkNDgxZ/gDCVwcz8BZ6J0CDo6RNlUUnqTtsagBn Bp88rBCKpeJzwCL5xprpHEB9kQtmsegDNrmhSaMK/KfHojdlflUK5hlcSxBrpGO9 Z7ZFJDGx4KWwzn/6ljBe86fbfjlTG0rgp8b5NzJiXdm37Squ+dsESIGPpaPdLMMS xVbeYUnIbPy52o9c0UU= =B6d0 -----END PGP SIGNATURE----- From slink at schokola.de Tue May 6 13:16:12 2014 From: slink at schokola.de (Nils Goroll) Date: Tue, 06 May 2014 15:16:12 +0200 Subject: nit in hcb_deref? Message-ID: <5368E09C.2050507@schokola.de> Trying to understand if I race a saw in 2.0.x still exists in master, I reviewed some of the cash_hash.c and hash_*.c code. The issue I feared could exist seems not to, but I believe I spottet a little nit: * HSH_DeleteObjHead is only called if hash_deref(oh) returned 0 - hcb_deref always returns 1, so for hcb we never call Lck_Delete(&oh->mtx); ds->n_objecthead--; FREE_OBJ(oh); looks like a resource leak to me simple fix IMHO: diff --git a/bin/varnishd/hash/hash_critbit.c b/bin/varnishd/hash/hash_critbit.c index 4e14bdd..d27e3ce 100644 --- a/bin/varnishd/hash/hash_critbit.c +++ b/bin/varnishd/hash/hash_critbit.c @@ -408,6 +408,7 @@ hcb_deref(struct objhead *oh) Lck_Unlock(&hcb_mtx); assert(VTAILQ_EMPTY(&oh->objcs)); AZ(oh->waitinglist); + r = 0; } Lck_Unlock(&oh->mtx); #ifdef PHK - hsl_deref / hcl_deref look ok to me Nils From slink at schokola.de Tue May 6 14:25:58 2014 From: slink at schokola.de (Nils Goroll) Date: Tue, 06 May 2014 16:25:58 +0200 Subject: nit in hcb_deref? In-Reply-To: <5368E09C.2050507@schokola.de> References: <5368E09C.2050507@schokola.de> Message-ID: <5368F0F6.3090002@schokola.de> On 06/05/14 15:16, Nils Goroll wrote: > * HSH_DeleteObjHead is only called if hash_deref(oh) returned 0 > > - hcb_deref always returns 1, so for hcb we never call my fault: hcb_cleaner calls HSH_DeleteObjHead, so this is correct. Thanks to Martin for the explanation on irc. Nils From phk at phk.freebsd.dk Tue May 6 19:57:06 2014 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 06 May 2014 19:57:06 +0000 Subject: Thoughs on VMODs and VCL In-Reply-To: <53689AB2.5070907@uplex.de> References: <53688394.3030301@schokola.de> <53689AB2.5070907@uplex.de> Message-ID: <32275.1399406226@critter.freebsd.dk> In message <53689AB2.5070907 at uplex.de>, Geoff Simmons writes: >> * VMOD {{{PRIV_REQUEST}}} {{{PRIV_SESSION}}} state > >+1 for PRIV_REQUEST and PRIV_SESSION, +2 or more if I could. I >actually thought they would make it into Varnish 4.0. Now that we have a per-session workspace to put them in, it's just a matter of code. Not trivial code, but code. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Tue May 6 21:28:30 2014 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 06 May 2014 21:28:30 +0000 Subject: [PATCH] [EXPERIMENTAL] autocrap autohardening In-Reply-To: <536778A5.8030707@schokola.de> References: <52CEDCFE.2090508@schokola.de> <20140130074848.GB10577@err.no> <5329F3C8.2040707@schokola.de> <536778A5.8030707@schokola.de> Message-ID: <44096.1399411710@critter.freebsd.dk> In message <536778A5.8030707 at schokola.de>, Nils Goroll writes: >brief summary of a #varnish-hacking discussion from my understanding (please >correct or comment if you disagree): > >* we do want hardening options by default In general Varnish is a paranoid source code, so anything the compiler can do to help we welcome. Obviously there are things too intrusive/pointless/useless for production, and that's why I'm aiming for three "levels": > - developer > - production > - performance The last one is intented only for people who want to shave the last epsilon performance out of the bottom of the barrel and it should be the same as -production, less only those compiler options proven to cause a performance hit. >* The patch needs to be tested against gcc and llvm/clang Well, it needs to not make jenkins cry... >* phk would like the compiler flag tests be implemented using the methodology > from Makefile.phk / config.phk in favor of wrap-compiler-for-flag-check It's not so much a "like" as a "I don't think the autocrap* way scales" GCC and LLVM is locked in mortal combat for being "the best" compiler right now, and new (and useful!) -W and -f options sprout like weeds. LLVM has sort of taken the consequences and introduced a "ignore -W and -f options you don't know" but I don't belive GCC has done that, which means that selecting the right CFLAGS is a rather nightmarish (and slow!) process, unless we do something smarter. The idea I played with in the "Makefile.phk" prototype is to run a file through the compiler: #if WARNS >= 1 "-Wall" "-Werror" #endif #if WARNS >= 2 "-W" "-fstack-protector" "-Wno-format-y2k" "-Wstrict-prototypes" [...] #endif And then use #ifdefs against the compilers predefines to sort compiler/version/OS specific options out: #if defined(__SVR4) && defined(sun) "-Wspecial_option_forSolaris" #endif #if defined(LLVM) && LLVM_Version > 234345 "-Wfind_bugs_like_heartbleed" #endif and so on. Once the file is run through "${CC} -E" you have only have to polish off the #line and quotes, and your ${CFLAGS} is all cooked. The big benefit is that it gives us very finegrained control and it runs very very fast and reliably, compared to having autocrap check for each and every option. Also, I find it a LOT more readable. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Tue May 6 21:45:37 2014 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 06 May 2014 21:45:37 +0000 Subject: Thinking about 4.1 softban/purges (yes, already...) Message-ID: <45710.1399412737@critter.freebsd.dk> I'm starting to think about 4.1 and one of the first things that pop up is the "softbans" and the purge ticket in #1139 I also noticed that we forgot to move the "purge" direct action (ie: from vcl_hit{}) in 4.0 to VMOD.std. Today a ban basically expires anything which it hits, but in general that doesn't have to be the case, the ban mechanism could be used to do any useful thing to the set of matching objects, for instance extend the grace period by 10 minutes. That's basically what "softbans" covers, the abilit to do something like: ban -grace +10m req.url ~ [.]jpeg$ There are outstanding design issues with respect to specifying the arguments, because of the "delayed action", those need to be sorted out. I want to do something similar for purge. If you return(purge) from vcl_recv{}, the stuff's going to be gone. But I want to add a std.purge(ttl, grace, keep) where you can modify all three as you like. Both of these pressume that you can modify ttl/grace/keep on an object and let it survive, and that's somewhat contrary to our pre-4 decision to make objects read-only. That decision in turn changes the previous calculus on where the ttl|grace|keep timers live and I think they should move from obj to objcore in 4.1. That increases the size of objcore by 24 bytes or so, and since it is magically 128 bytes right now I'm not happy about that, but I think it will be worth the cost. It also means adding a oc->method() so we can tell -spersistent that we muked about with exp.*. Comments ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From martin at varnish-software.com Wed May 7 14:53:34 2014 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Wed, 7 May 2014 16:53:34 +0200 Subject: [PATCH] Move struct exp from object to objcore Message-ID: <1399474414-21108-1-git-send-email-martin@varnish-software.com> --- bin/varnishd/cache/cache.h | 17 +++++---- bin/varnishd/cache/cache_ban.c | 4 +-- bin/varnishd/cache/cache_expire.c | 49 +++++++++++--------------- bin/varnishd/cache/cache_fetch.c | 34 +++++++++--------- bin/varnishd/cache/cache_hash.c | 28 +++++++-------- bin/varnishd/cache/cache_req_fsm.c | 2 +- bin/varnishd/cache/cache_rfc2616.c | 2 +- bin/varnishd/cache/cache_vrt_var.c | 22 ++++++------ bin/varnishd/storage/stevedore.c | 7 ++-- bin/varnishd/storage/storage_persistent.c | 9 +++-- bin/varnishd/storage/storage_persistent_silo.c | 36 +++++++++++-------- include/persistent.h | 5 +-- 12 files changed, 105 insertions(+), 110 deletions(-) diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index b0740c1..c15473d 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -404,7 +404,7 @@ struct storage { typedef struct object *getobj_f(struct dstat *ds, struct objcore *oc); typedef unsigned getxid_f(struct dstat *ds, struct objcore *oc); typedef void updatemeta_f(struct objcore *oc); -typedef void freeobj_f(struct objcore *oc); +typedef void freeobj_f(struct dstat *ds, struct objcore *oc); typedef struct lru *getlru_f(const struct objcore *oc); struct objcore_methods { @@ -446,6 +446,8 @@ struct objcore { double last_lru; VTAILQ_ENTRY(objcore) ban_list; struct ban *ban; + + struct exp exp; }; static inline unsigned @@ -480,13 +482,13 @@ oc_updatemeta(struct objcore *oc) } static inline void -oc_freeobj(struct objcore *oc) +oc_freeobj(struct dstat *ds, struct objcore *oc) { CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); AN(oc->methods); AN(oc->methods->freeobj); - oc->methods->freeobj(oc); + oc->methods->freeobj(ds, oc); } static inline struct lru * @@ -553,7 +555,6 @@ struct busyobj { struct objcore *fetch_objcore; struct object *fetch_obj; - struct exp exp; struct http_conn htc; struct pool_task fetch_task; @@ -612,8 +613,6 @@ struct object { ssize_t len; - struct exp exp; - /* VCL only variables */ double last_modified; @@ -917,11 +916,11 @@ extern pthread_t cli_thread; /* cache_expiry.c */ void EXP_Clr(struct exp *e); -double EXP_Ttl(const struct req *, const struct object*); +double EXP_Ttl(const struct req *, const struct objcore *); void EXP_Insert(struct objcore *oc); -void EXP_Inject(struct objcore *oc, struct lru *lru, double when); +void EXP_Inject(struct objcore *oc, struct lru *lru, const struct exp *exp); void EXP_Init(void); -void EXP_Rearm(struct object *o, double now, double ttl, double grace, +void EXP_Rearm(struct objcore *oc, double now, double ttl, double grace, double keep); void EXP_Touch(struct objcore *oc, double now); int EXP_NukeOne(struct busyobj *, struct lru *lru); diff --git a/bin/varnishd/cache/cache_ban.c b/bin/varnishd/cache/cache_ban.c index 99f2914..b60ed9e 100644 --- a/bin/varnishd/cache/cache_ban.c +++ b/bin/varnishd/cache/cache_ban.c @@ -964,7 +964,7 @@ ban_check_object(struct object *o, struct vsl_log *vsl, oc->ban = NULL; VSLb(vsl, SLT_ExpBan, "%u banned lookup", o->vxid); VSC_C_main->bans_obj_killed++; - EXP_Rearm(o, o->exp.t_origin, 0, 0, 0); // XXX fake now + EXP_Rearm(oc, oc->exp.t_origin, 0, 0, 0); // XXX fake now return (1); } } @@ -1105,7 +1105,7 @@ ban_lurker_test_ban(struct worker *wrk, struct vsl_log *vsl, struct ban *bt, } if (i) { VSLb(vsl, SLT_ExpBan, "%u banned by lurker", o->vxid); - EXP_Rearm(o, o->exp.t_origin, 0, 0, 0); // XXX fake now + EXP_Rearm(oc, oc->exp.t_origin, 0, 0, 0);// XXX fake now VSC_C_main->bans_lurker_obj_killed++; } (void)HSH_DerefObjCore(&wrk->stats, &oc); diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c index b5ecfbe..864d679 100644 --- a/bin/varnishd/cache/cache_expire.c +++ b/bin/varnishd/cache/cache_expire.c @@ -76,14 +76,14 @@ EXP_Clr(struct exp *e) */ double -EXP_Ttl(const struct req *req, const struct object *o) +EXP_Ttl(const struct req *req, const struct objcore *oc) { double r; - r = o->exp.ttl; + r = oc->exp.ttl; if (req != NULL && req->d_ttl > 0. && req->d_ttl < r) r = req->d_ttl; - return (o->exp.t_origin + r); + return (oc->exp.t_origin + r); } /*-------------------------------------------------------------------- @@ -91,13 +91,11 @@ EXP_Ttl(const struct req *req, const struct object *o) */ static double -exp_when(const struct object *o) +exp_when(const struct exp *exp) { double when; - CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC); - - when = o->exp.t_origin + o->exp.ttl + o->exp.grace + o->exp.keep; + when = exp->t_origin + exp->ttl + exp->grace + exp->keep; AZ(isnan(when)); return (when); } @@ -130,7 +128,7 @@ exp_mail_it(struct objcore *oc) */ void -EXP_Inject(struct objcore *oc, struct lru *lru, double when) +EXP_Inject(struct objcore *oc, struct lru *lru, const struct exp *exp) { CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); @@ -146,10 +144,12 @@ EXP_Inject(struct objcore *oc, struct lru *lru, double when) Lck_Lock(&lru->mtx); lru->n_objcore++; oc->exp_flags |= OC_EF_OFFLRU | OC_EF_INSERT | OC_EF_EXP; - if (when < 0) + if (exp == NULL) oc->exp_flags |= OC_EF_MOVE; - else - oc->timer_when = when; + else { + oc->exp = *exp; + oc->timer_when = exp_when(exp); + } Lck_Unlock(&lru->mtx); exp_mail_it(oc); @@ -168,7 +168,7 @@ EXP_Insert(struct objcore *oc) CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); HSH_Ref(oc); - EXP_Inject(oc, NULL, -1); + EXP_Inject(oc, NULL, NULL); } /*-------------------------------------------------------------------- @@ -223,32 +223,29 @@ EXP_Touch(struct objcore *oc, double now) */ void -EXP_Rearm(struct object *o, double now, double ttl, double grace, double keep) +EXP_Rearm(struct objcore *oc, double now, double ttl, double grace, double keep) { - struct objcore *oc; struct lru *lru; double when; - CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC); - oc = o->objcore; CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); assert(oc->refcnt > 0); AN(oc->exp_flags & OC_EF_EXP); if (!isnan(ttl)) - o->exp.ttl = now + ttl - o->exp.t_origin; + oc->exp.ttl = now + ttl - oc->exp.t_origin; if (!isnan(grace)) - o->exp.grace = grace; + oc->exp.grace = grace; if (!isnan(keep)) - o->exp.keep = keep; + oc->exp.keep = keep; - when = exp_when(o); + when = exp_when(&oc->exp); VSL(SLT_ExpKill, 0, "EXP_Rearm p=%p E=%.9f e=%.9f f=0x%x", oc, oc->timer_when, when, oc->flags); - if (when > o->exp.t_origin && when > oc->timer_when) + if (when > oc->exp.t_origin && when > oc->timer_when) return; lru = oc_getlru(oc); @@ -419,7 +416,6 @@ exp_inbox(struct exp_priv *ep, struct objcore *oc, double now) { unsigned flags; struct lru *lru; - struct object *o; CHECK_OBJ_NOTNULL(ep, EXP_PRIV_MAGIC); CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); @@ -457,9 +453,7 @@ exp_inbox(struct exp_priv *ep, struct objcore *oc, double now) } if (flags & OC_EF_MOVE) { - o = oc_getobj(&ep->wrk->stats, oc); - CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC); - oc->timer_when = exp_when(o); + oc->timer_when = exp_when(&oc->exp); oc_updatemeta(oc); } @@ -494,7 +488,6 @@ exp_expire(struct exp_priv *ep, double now) { struct lru *lru; struct objcore *oc; - struct object *o; CHECK_OBJ_NOTNULL(ep, EXP_PRIV_MAGIC); @@ -532,11 +525,9 @@ exp_expire(struct exp_priv *ep, double now) assert(oc->timer_idx == BINHEAP_NOIDX); CHECK_OBJ_NOTNULL(oc->objhead, OBJHEAD_MAGIC); - o = oc_getobj(&ep->wrk->stats, oc); - CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC); VSLb(&ep->vsl, SLT_ExpKill, "EXP_Expired x=%u t=%.0f", oc_getxid(&ep->wrk->stats, oc) & VSL_IDENTMASK, - EXP_Ttl(NULL, o) - now); + EXP_Ttl(NULL, oc) - now); (void)HSH_DerefObjCore(&ep->wrk->stats, &oc); return (0); } diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index f8e2848..0176309 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -55,7 +55,8 @@ vbf_allocobj(struct busyobj *bo, unsigned l, uint16_t nhttp) CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC); CHECK_OBJ_NOTNULL(bo->fetch_objcore, OBJCORE_MAGIC); - lifetime = bo->exp.ttl + bo->exp.grace + bo->exp.keep; + lifetime = bo->fetch_objcore->exp.ttl + bo->fetch_objcore->exp.grace + + bo->fetch_objcore->exp.keep; if (bo->uncacheable || lifetime < cache_param->shortlived) storage_hint = TRANSIENT_STORAGE; @@ -77,10 +78,10 @@ vbf_allocobj(struct busyobj *bo, unsigned l, uint16_t nhttp) * on Transient storage. */ - if (bo->exp.ttl > cache_param->shortlived) - bo->exp.ttl = cache_param->shortlived; - bo->exp.grace = 0.0; - bo->exp.keep = 0.0; + if (bo->fetch_objcore->exp.ttl > cache_param->shortlived) + bo->fetch_objcore->exp.ttl = cache_param->shortlived; + bo->fetch_objcore->exp.grace = 0.0; + bo->fetch_objcore->exp.keep = 0.0; obj = STV_NewObject(bo, TRANSIENT_STORAGE, l, nhttp); return (obj); } @@ -162,7 +163,7 @@ vbf_beresp2obj(struct busyobj *bo) if (http_GetHdr(hp, H_Last_Modified, &b)) obj->last_modified = VTIM_parse(b); else - obj->last_modified = floor(bo->exp.t_origin); + obj->last_modified = floor(bo->fetch_objcore->exp.t_origin); return (0); } @@ -346,12 +347,12 @@ vbf_stp_startfetch(struct worker *wrk, struct busyobj *bo) /* * What does RFC2616 think about TTL ? */ - EXP_Clr(&bo->exp); + EXP_Clr(&bo->fetch_objcore->exp); RFC2616_Ttl(bo, now); /* private objects have negative TTL */ if (bo->fetch_objcore->flags & OC_F_PRIVATE) - bo->exp.ttl = -1.; + bo->fetch_objcore->exp.ttl = -1.; AZ(bo->do_esi); @@ -531,9 +532,8 @@ vbf_stp_fetch(struct worker *wrk, struct busyobj *bo) if (bo->failed && !bo->do_stream) { assert(bo->state < BOS_STREAM); if (bo->fetch_obj != NULL) { - oc_freeobj(bo->fetch_objcore); + oc_freeobj(bo->stats, bo->fetch_objcore); bo->fetch_obj = NULL; - bo->stats->n_object--; } return (F_STP_ERROR); } @@ -646,7 +646,8 @@ vbf_stp_condfetch(struct worker *wrk, struct busyobj *bo) assert(al == bo->ims_obj->len); assert(obj->len == al); - EXP_Rearm(bo->ims_obj, bo->ims_obj->exp.t_origin, 0, 0, 0); + EXP_Rearm(bo->ims_obj->objcore, bo->ims_obj->objcore->exp.t_origin, + 0, 0, 0); /* Recycle the backend connection before setting BOS_FINISHED to give predictable backend reuse behavior for varnishtest */ @@ -692,10 +693,10 @@ vbf_stp_error(struct worker *wrk, struct busyobj *bo) http_PrintfHeader(bo->beresp, "Date: %s", time_str); http_SetHeader(bo->beresp, "Server: Varnish"); - bo->exp.t_origin = bo->t_prev; - bo->exp.ttl = 0; - bo->exp.grace = 0; - bo->exp.keep = 0; + bo->fetch_objcore->exp.t_origin = bo->t_prev; + bo->fetch_objcore->exp.ttl = 0; + bo->fetch_objcore->exp.grace = 0; + bo->fetch_objcore->exp.keep = 0; VCL_backend_error_method(bo->vcl, wrk, NULL, bo, bo->bereq->ws); @@ -751,7 +752,8 @@ vbf_stp_fail(struct worker *wrk, struct busyobj *bo) if (bo->fetch_objcore->exp_flags & OC_EF_EXP) { /* Already unbusied - expire it */ AN(bo->fetch_obj); - EXP_Rearm(bo->fetch_obj, bo->fetch_obj->exp.t_origin, 0, 0, 0); + EXP_Rearm(bo->fetch_obj->objcore, + bo->fetch_obj->objcore->exp.t_origin, 0, 0, 0); } wrk->stats.fetch_failed++; VBO_setstate(bo, BOS_FAILED); diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c index f4bbae0..3234571 100644 --- a/bin/varnishd/cache/cache_hash.c +++ b/bin/varnishd/cache/cache_hash.c @@ -420,17 +420,18 @@ HSH_Lookup(struct req *req, struct objcore **ocp, struct objcore **bocp, continue; } + if (oc->exp.ttl <= 0.) + continue; + o = oc_getobj(&wrk->stats, oc); CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC); - if (o->exp.ttl <= 0.) - continue; if (BAN_CheckObject(o, req)) continue; if (o->vary != NULL && !VRY_Match(req, o->vary)) continue; - if (EXP_Ttl(req, o) >= req->t_req) { + if (EXP_Ttl(req, oc) >= req->t_req) { /* If still valid, use it */ assert(oh->refcnt > 1); assert(oc->objhead == oh); @@ -442,12 +443,12 @@ HSH_Lookup(struct req *req, struct objcore **ocp, struct objcore **bocp, *ocp = oc; return (HSH_HIT); } - if (o->exp.t_origin > exp_t_origin && + if (oc->exp.t_origin > exp_t_origin && !(oc->flags & OC_F_PASS)) { /* record the newest object */ exp_oc = oc; exp_o = o; - exp_t_origin = o->exp.t_origin; + exp_t_origin = oc->exp.t_origin; } } @@ -559,7 +560,6 @@ HSH_Purge(struct worker *wrk, struct objhead *oh, double ttl, double grace) { struct objcore *oc, **ocp; unsigned spc, nobj, n; - struct object *o; double now; CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); @@ -594,12 +594,10 @@ HSH_Purge(struct worker *wrk, struct objhead *oh, double ttl, double grace) for (n = 0; n < nobj; n++) { oc = ocp[n]; CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); - o = oc_getobj(&wrk->stats, oc); - if (o == NULL) + if (!(oc->exp_flags & OC_EF_EXP)) continue; - CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC); - EXP_Rearm(o, now, ttl, grace, NAN); // XXX: Keep ? - (void)HSH_DerefObj(&wrk->stats, &o); + EXP_Rearm(oc, now, ttl, grace, NAN); // XXX: Keep ? + (void)HSH_DerefObjCore(&wrk->stats, &oc); } WS_Release(wrk->aws, 0); } @@ -616,7 +614,7 @@ HSH_Drop(struct worker *wrk, struct object **oo) CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); AN(oo); CHECK_OBJ_NOTNULL(*oo, OBJECT_MAGIC); - (*oo)->exp.ttl = -1.; + (*oo)->objcore->exp.ttl = -1.; AZ(HSH_DerefObj(&wrk->stats, oo)); } @@ -802,10 +800,8 @@ HSH_DerefObjCore(struct dstat *ds, struct objcore **ocp) BAN_DestroyObj(oc); AZ(oc->ban); - if (oc->methods != NULL) { - oc_freeobj(oc); - ds->n_object--; - } + if (oc->methods != NULL) + oc_freeobj(ds, oc); FREE_OBJ(oc); ds->n_objectcore--; diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index d465ba4..2201049 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -122,7 +122,7 @@ cnt_deliver(struct worker *wrk, struct req *req) age. Truncate to zero in that case). */ http_PrintfHeader(req->resp, "Age: %.0f", - fmax(0., req->t_prev - req->obj->exp.t_origin)); + fmax(0., req->t_prev - req->obj->objcore->exp.t_origin)); http_SetHeader(req->resp, "Via: 1.1 varnish (v4)"); diff --git a/bin/varnishd/cache/cache_rfc2616.c b/bin/varnishd/cache/cache_rfc2616.c index bb0a85e..8ada968 100644 --- a/bin/varnishd/cache/cache_rfc2616.c +++ b/bin/varnishd/cache/cache_rfc2616.c @@ -72,7 +72,7 @@ RFC2616_Ttl(struct busyobj *bo, double now) struct exp *expp; CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC); - expp = &bo->exp; + expp = &bo->fetch_objcore->exp; hp = bo->beresp; diff --git a/bin/varnishd/cache/cache_vrt_var.c b/bin/varnishd/cache/cache_vrt_var.c index b68031c..ca84bb0 100644 --- a/bin/varnishd/cache/cache_vrt_var.c +++ b/bin/varnishd/cache/cache_vrt_var.c @@ -461,17 +461,17 @@ VRT_r_##which##_##fld(const struct vrt_ctx *ctx) \ return(0.0); \ } -VRT_DO_EXP_R(obj, ctx->req->obj->exp, ttl, - (ctx->req->t_req - ctx->req->obj->exp.t_origin)) -VRT_DO_EXP_R(obj, ctx->req->obj->exp, grace, 0) -VRT_DO_EXP_R(obj, ctx->req->obj->exp, keep, 0) - -VRT_DO_EXP_L(beresp, ctx->bo->exp, ttl) -VRT_DO_EXP_R(beresp, ctx->bo->exp, ttl, 0) -VRT_DO_EXP_L(beresp, ctx->bo->exp, grace) -VRT_DO_EXP_R(beresp, ctx->bo->exp, grace, 0) -VRT_DO_EXP_L(beresp, ctx->bo->exp, keep) -VRT_DO_EXP_R(beresp, ctx->bo->exp, keep, 0) +VRT_DO_EXP_R(obj, ctx->req->obj->objcore->exp, ttl, + (ctx->req->t_req - ctx->req->obj->objcore->exp.t_origin)) +VRT_DO_EXP_R(obj, ctx->req->obj->objcore->exp, grace, 0) +VRT_DO_EXP_R(obj, ctx->req->obj->objcore->exp, keep, 0) + +VRT_DO_EXP_L(beresp, ctx->bo->fetch_objcore->exp, ttl) +VRT_DO_EXP_R(beresp, ctx->bo->fetch_objcore->exp, ttl, 0) +VRT_DO_EXP_L(beresp, ctx->bo->fetch_objcore->exp, grace) +VRT_DO_EXP_R(beresp, ctx->bo->fetch_objcore->exp, grace, 0) +VRT_DO_EXP_L(beresp, ctx->bo->fetch_objcore->exp, keep) +VRT_DO_EXP_R(beresp, ctx->bo->fetch_objcore->exp, keep, 0) /*-------------------------------------------------------------------- * [be]req.xid diff --git a/bin/varnishd/storage/stevedore.c b/bin/varnishd/storage/stevedore.c index 5e54ed6..6fc3ce2 100644 --- a/bin/varnishd/storage/stevedore.c +++ b/bin/varnishd/storage/stevedore.c @@ -70,7 +70,7 @@ default_oc_getobj(struct dstat *ds, struct objcore *oc) } static void -default_oc_freeobj(struct objcore *oc) +default_oc_freeobj(struct dstat *ds, struct objcore *oc) { struct object *o; @@ -81,6 +81,8 @@ default_oc_freeobj(struct objcore *oc) STV_Freestore(o); STV_free(o->objstore); + + ds->n_object--; } static struct lru * @@ -283,9 +285,7 @@ STV_MkObject(struct stevedore *stv, struct busyobj *bo, HTTP_Setup(o->http, bo->ws_o, bo->vsl, SLT_ObjMethod); o->http->magic = HTTP_MAGIC; - o->exp = bo->exp; VTAILQ_INIT(&o->store); - bo->stats->n_object++; o->objcore = bo->fetch_objcore; @@ -321,6 +321,7 @@ stv_default_allocobj(struct stevedore *stv, struct busyobj *bo, o = STV_MkObject(stv, bo, st->ptr, ltot, soc); CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC); o->objstore = st; + bo->stats->n_object++; return (o); } diff --git a/bin/varnishd/storage/storage_persistent.c b/bin/varnishd/storage/storage_persistent.c index d7506f0..a931f81 100644 --- a/bin/varnishd/storage/storage_persistent.c +++ b/bin/varnishd/storage/storage_persistent.c @@ -474,7 +474,7 @@ smp_allocx(struct stevedore *st, size_t min_size, size_t max_size, sc->next_top -= sizeof(**so); *so = (void*)(sc->base + sc->next_top); /* Render this smp_object mostly harmless */ - (*so)->ttl = 0.; + EXP_Clr(&(*so)->exp); (*so)->ban = 0.; (*so)->ptr = 0; sg->objs = *so; @@ -521,7 +521,9 @@ smp_allocobj(struct stevedore *stv, struct busyobj *bo, if (bo->fetch_objcore == NULL) return (NULL); /* from cnt_error */ CAST_OBJ_NOTNULL(sc, stv->priv, SMP_SC_MAGIC); - AN((bo->exp.ttl + bo->exp.grace + bo->exp.keep) > 0.); + assert((bo->fetch_objcore->exp.ttl + + bo->fetch_objcore->exp.grace + + bo->fetch_objcore->exp.keep) > 0.); ltot = IRNUP(sc, ltot); @@ -535,6 +537,7 @@ smp_allocobj(struct stevedore *stv, struct busyobj *bo, o = STV_MkObject(stv, bo, st->ptr, ltot, soc); CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC); o->objstore = st; + bo->stats->n_object++; oc = o->objcore; CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); @@ -546,7 +549,7 @@ smp_allocobj(struct stevedore *stv, struct busyobj *bo, /* We have to do this somewhere, might as well be here... */ assert(sizeof so->hash == DIGEST_LEN); memcpy(so->hash, oc->objhead->digest, DIGEST_LEN); - so->ttl = oc->timer_when; + so->exp = oc->exp; so->ptr = (uint8_t*)o - sc->base; so->ban = BAN_Time(oc->ban); diff --git a/bin/varnishd/storage/storage_persistent_silo.c b/bin/varnishd/storage/storage_persistent_silo.c index f27ccd9..f561159 100644 --- a/bin/varnishd/storage/storage_persistent_silo.c +++ b/bin/varnishd/storage/storage_persistent_silo.c @@ -127,6 +127,7 @@ smp_load_seg(struct worker *wrk, const struct smp_sc *sc, uint32_t no; double t_now = VTIM_real(); struct smp_signctx ctx[1]; + const struct exp *exp; ASSERT_SILO_THREAD(sc); CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); @@ -149,7 +150,9 @@ smp_load_seg(struct worker *wrk, const struct smp_sc *sc, /* Clear the bogus "hold" count */ sg->nobj = 0; for (;no > 0; so++,no--) { - if (so->ttl == 0 || so->ttl < t_now) + exp = &so->exp; + if (exp->ttl < 0. || + (exp->t_origin + exp->ttl + exp->grace + exp->keep < t_now)) continue; ALLOC_OBJ(oc, OBJCORE_MAGIC); AN(oc); @@ -158,7 +161,7 @@ smp_load_seg(struct worker *wrk, const struct smp_sc *sc, smp_init_oc(oc, sg, no); oc->ban = BAN_RefBan(oc, so->ban, sc->tailban); HSH_Insert(wrk, so->hash, oc); - EXP_Inject(oc, sg->lru, so->ttl); + EXP_Inject(oc, sg->lru, &so->exp); sg->nobj++; } WRK_SumStat(wrk); @@ -415,7 +418,7 @@ smp_oc_getobj(struct dstat *ds, struct objcore *oc) struct smp_object *so; struct storage *st; uint64_t l; - int bad; + int bad = 0; /* Some calls are direct, but they should match anyway */ assert(oc->methods->getobj == smp_oc_getobj); @@ -451,7 +454,6 @@ smp_oc_getobj(struct dstat *ds, struct objcore *oc) /* We trust caller to have a refcnt for us */ o->objcore = oc; - bad = 0; l = 0; VTAILQ_FOREACH(st, &o->store, list) { bad |= smp_loaded_st(sg->sc, sg, st); @@ -462,10 +464,8 @@ smp_oc_getobj(struct dstat *ds, struct objcore *oc) if (l != o->len) bad |= 0x100; - if(bad) { - o->exp.ttl = -1; - so->ttl = 0; - } + if(bad) + EXP_Clr(&oc->exp); sg->nfixed++; ds->n_object++; @@ -473,7 +473,8 @@ smp_oc_getobj(struct dstat *ds, struct objcore *oc) oc->flags &= ~OC_F_NEEDFIXUP; } Lck_Unlock(&sg->sc->mtx); - EXP_Rearm(o, NAN, NAN, NAN, NAN); // XXX: Shouldn't be needed + if (bad) + EXP_Rearm(oc, NAN, NAN, NAN, NAN); return (o); } @@ -496,16 +497,16 @@ smp_oc_updatemeta(struct objcore *oc) /* Lock necessary, we might race close_seg */ Lck_Lock(&sg->sc->mtx); so->ban = BAN_Time(oc->ban); - so->ttl = oc->timer_when; + so->exp = oc->exp; Lck_Unlock(&sg->sc->mtx); } else { so->ban = BAN_Time(oc->ban); - so->ttl = oc->timer_when; + so->exp = oc->exp; } } static void __match_proto__() -smp_oc_freeobj(struct objcore *oc) +smp_oc_freeobj(struct dstat *ds, struct objcore *oc) { struct smp_seg *sg; struct smp_object *so; @@ -516,13 +517,18 @@ smp_oc_freeobj(struct objcore *oc) so = smp_find_so(sg, oc->priv2); Lck_Lock(&sg->sc->mtx); - so->ttl = 0; + EXP_Clr(&so->exp); so->ptr = 0; assert(sg->nobj > 0); - assert(sg->nfixed > 0); sg->nobj--; - sg->nfixed--; + if (oc->flags & OC_F_NEEDFIXUP) + ds->n_vampireobject--; + else { + assert(sg->nfixed > 0); + sg->nfixed--; + ds->n_object--; + } Lck_Unlock(&sg->sc->mtx); } diff --git a/include/persistent.h b/include/persistent.h index f0cb7f5..0c48154 100644 --- a/include/persistent.h +++ b/include/persistent.h @@ -131,14 +131,11 @@ struct smp_segptr { /* * An object descriptor - * - * A positive ttl is obj.ttl with obj.grace being NAN - * A negative ttl is - (obj.ttl + obj.grace) */ struct smp_object { uint8_t hash[32]; /* really: DIGEST_LEN */ - double ttl; + struct exp exp; double ban; uint64_t ptr; /* rel to silo */ }; -- 2.0.0.rc0 From martin at varnish-software.com Wed May 7 14:56:41 2014 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Wed, 7 May 2014 16:56:41 +0200 Subject: Thinking about 4.1 softban/purges (yes, already...) In-Reply-To: <45710.1399412737@critter.freebsd.dk> References: <45710.1399412737@critter.freebsd.dk> Message-ID: I think this sounds great!. As this change became a blocker for my current project, I ended up implementing the change. Patch sent to the -dev list. In my patch I didn't implement any additional oc->method() for the exp changes, AFAICS the existing updatemeta() function suffices. Martin On 6 May 2014 23:45, Poul-Henning Kamp wrote: > > I'm starting to think about 4.1 and one of the first things that > pop up is the "softbans" and the purge ticket in #1139 > > I also noticed that we forgot to move the "purge" direct > action (ie: from vcl_hit{}) in 4.0 to VMOD.std. > > Today a ban basically expires anything which it hits, but in > general that doesn't have to be the case, the ban mechanism > could be used to do any useful thing to the set of matching > objects, for instance extend the grace period by 10 minutes. > > That's basically what "softbans" covers, the abilit to do something > like: > ban -grace +10m req.url ~ [.]jpeg$ > > There are outstanding design issues with respect to specifying > the arguments, because of the "delayed action", those need to > be sorted out. > > I want to do something similar for purge. > > If you return(purge) from vcl_recv{}, the stuff's going to be > gone. > > But I want to add a std.purge(ttl, grace, keep) where you can > modify all three as you like. > > Both of these pressume that you can modify ttl/grace/keep on > an object and let it survive, and that's somewhat contrary > to our pre-4 decision to make objects read-only. > > That decision in turn changes the previous calculus on where the > ttl|grace|keep timers live and I think they should move from obj > to objcore in 4.1. > > That increases the size of objcore by 24 bytes or so, and since > it is magically 128 bytes right now I'm not happy about that, > but I think it will be worth the cost. > > It also means adding a oc->method() so we can tell -spersistent > that we muked about with exp.*. > > Comments ? > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -- *Martin Blix Grydeland* Senior Developer | Varnish Software AS Cell: +47 21 98 92 60 We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Wed May 7 14:58:24 2014 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 07 May 2014 14:58:24 +0000 Subject: Thinking about 4.1 softban/purges (yes, already...) In-Reply-To: References: <45710.1399412737@critter.freebsd.dk> Message-ID: <49538.1399474704@critter.freebsd.dk> In message , Martin Blix Grydeland writes: >I think this sounds great!. As this change became a blocker for my current >project, I ended up implementing the change. Patch sent to the -dev list. Isn't it a bit brutal to slam a change like this into 4.0.x ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From martin at varnish-software.com Wed May 7 18:38:53 2014 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Wed, 7 May 2014 20:38:53 +0200 Subject: Thinking about 4.1 softban/purges (yes, already...) In-Reply-To: <49538.1399474704@critter.freebsd.dk> References: <45710.1399412737@critter.freebsd.dk> <49538.1399474704@critter.freebsd.dk> Message-ID: I don't think it's too much to add to a 4.0.x release, specially when we're at 4.0.0 now. The changes are for the most part limited to pointer changes. Only place with minor logic changes is for persistent, but that's pretty experimental anyways. But if it isn't for 4.0.x, then maybe it can be applied to 4.1 (in a dev branch or when that time comes). I will use it for my project, and I'd thought I'd share it now that it's done. Any feed back would be most welcome, so I don't stray too far away from what will become 4.1. Martin On 7 May 2014 16:58, Poul-Henning Kamp wrote: > In message vMf7rw at mail.gmail.com> > , Martin Blix Grydeland writes: > > >I think this sounds great!. As this change became a blocker for my current > >project, I ended up implementing the change. Patch sent to the -dev list. > > Isn't it a bit brutal to slam a change like this into 4.0.x ? > > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > -- *Martin Blix Grydeland* Senior Developer | Varnish Software AS Cell: +47 21 98 92 60 We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at varnish-software.com Thu May 8 15:49:04 2014 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Thu, 8 May 2014 17:49:04 +0200 Subject: [PATCH] Deref the old object after finishing with it in cnt_miss() Message-ID: <1399564144-27225-1-git-send-email-martin@varnish-software.com> Fixes: #1499 --- bin/varnishd/cache/cache_req_fsm.c | 2 ++ bin/varnishtest/tests/r01499.vtc | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 34 insertions(+) create mode 100644 bin/varnishtest/tests/r01499.vtc diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index d465ba4..40c0912 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -498,6 +498,8 @@ cnt_miss(struct worker *wrk, struct req *req) wrk->stats.cache_miss++; VBF_Fetch(wrk, req, req->objcore, o, VBF_NORMAL); req->req_step = R_STP_FETCH; + if (o != NULL) + (void)HSH_DerefObj(&wrk->stats, &o); return (REQ_FSM_MORE); case VCL_RET_SYNTH: req->req_step = R_STP_SYNTH; diff --git a/bin/varnishtest/tests/r01499.vtc b/bin/varnishtest/tests/r01499.vtc new file mode 100644 index 0000000..7b438b2 --- /dev/null +++ b/bin/varnishtest/tests/r01499.vtc @@ -0,0 +1,32 @@ +varnishtest "#1499 - objcore ref leak on IMS update" + +server s1 { + rxreq + txresp -hdr "Etag: foo" + rxreq + expect req.http.if-none-match == "foo" + txresp -hdr "X-Resp: 2" +} -start + +varnish v1 -vcl+backend { + sub vcl_backend_response { + set beresp.ttl = 0.0001s; + set beresp.grace = 0s; + set beresp.keep = 2s; + } +} -start + +client c1 { + txreq + rxresp + + delay 0.5 + + txreq + rxresp + expect resp.http.x-resp == "2" +} -run + +delay 3 + +varnish v1 -expect n_object == 0 -- 2.0.0.rc0 From davidr at crane7.co.uk Wed May 21 18:14:04 2014 From: davidr at crane7.co.uk (David Robertson) Date: Wed, 21 May 2014 19:14:04 +0100 Subject: [PATCH] Added %I and %O formatters to varnishncsa (byte counts) Message-ID: This adds %I and %O formatters to varnishncsa, which are total bytes received and total bytes sent from/to the client respectively. These are similar to the same formatters in apache (with mod_logio). Useful for bandwidth accounting. Thanks, David. --- bin/varnishncsa/varnishncsa.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c index 2162118..5aa0361 100644 --- a/bin/varnishncsa/varnishncsa.c +++ b/bin/varnishncsa/varnishncsa.c @@ -45,6 +45,8 @@ * %b Length of reply body, or "-" * %{Referer}i Contents of "Referer" request header * %{User-agent}i Contents of "User-agent" request header + * %I Total bytes recieved from client + * %O Total bytes sent to client * * Actually, we cheat a little and replace "%r" with something close to * "%m http://%{Host}i%U%q %H", where the additional fields are: @@ -95,6 +97,8 @@ enum e_frag { F_h, /* %h Host name / IP Address */ F_m, /* %m Method */ F_s, /* %s Status */ + F_I, /* %I Bytes recieved */ + F_O, /* %O Bytes sent */ F_tstart, /* Time start */ F_tend, /* Time end */ F_ttfb, /* %{Varnish:time_firstbyte}x */ @@ -574,6 +578,12 @@ parse_format(const char *format) case 'U': /* URL */ addf_fragment(&CTX.frag[F_U], "-"); break; + case 'I': /* Bytes recieved */ + addf_fragment(&CTX.frag[F_I], "-"); + break; + case 'O': /* Bytes sent */ + addf_fragment(&CTX.frag[F_O], "-"); + break; case '{': p++; q = p; @@ -783,6 +793,8 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[], break; case SLT_ReqAcct: frag_fields(b, e, 5, &CTX.frag[F_b], 0, NULL); + frag_fields(b, e, 3, &CTX.frag[F_I], 0, NULL); + frag_fields(b, e, 6, &CTX.frag[F_O], 0, NULL); break; case SLT_Timestamp: if (isprefix(b, "Start:", e, &p)) { -- 1.9.1 From davidr at crane7.co.uk Wed May 21 18:40:13 2014 From: davidr at crane7.co.uk (David Robertson) Date: Wed, 21 May 2014 19:40:13 +0100 Subject: [PATCH v2] Added %I and %O formatters to varnishncsa (byte counts) Message-ID: <1400697613-18239-1-git-send-email-david@davidr.me> (My email client messed with the whitespace in the patch) This adds %I and %O formatters to varnishncsa, which are total bytes received and total bytes sent. These are similar to the same formatters in apache (with mod_logio). Useful for bandwidth accounting. Thanks, David. --- bin/varnishncsa/varnishncsa.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c index 2162118..5aa0361 100644 --- a/bin/varnishncsa/varnishncsa.c +++ b/bin/varnishncsa/varnishncsa.c @@ -45,6 +45,8 @@ * %b Length of reply body, or "-" * %{Referer}i Contents of "Referer" request header * %{User-agent}i Contents of "User-agent" request header + * %I Total bytes recieved from client + * %O Total bytes sent to client * * Actually, we cheat a little and replace "%r" with something close to * "%m http://%{Host}i%U%q %H", where the additional fields are: @@ -95,6 +97,8 @@ enum e_frag { F_h, /* %h Host name / IP Address */ F_m, /* %m Method */ F_s, /* %s Status */ + F_I, /* %I Bytes recieved */ + F_O, /* %O Bytes sent */ F_tstart, /* Time start */ F_tend, /* Time end */ F_ttfb, /* %{Varnish:time_firstbyte}x */ @@ -574,6 +578,12 @@ parse_format(const char *format) case 'U': /* URL */ addf_fragment(&CTX.frag[F_U], "-"); break; + case 'I': /* Bytes recieved */ + addf_fragment(&CTX.frag[F_I], "-"); + break; + case 'O': /* Bytes sent */ + addf_fragment(&CTX.frag[F_O], "-"); + break; case '{': p++; q = p; @@ -783,6 +793,8 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[], break; case SLT_ReqAcct: frag_fields(b, e, 5, &CTX.frag[F_b], 0, NULL); + frag_fields(b, e, 3, &CTX.frag[F_I], 0, NULL); + frag_fields(b, e, 6, &CTX.frag[F_O], 0, NULL); break; case SLT_Timestamp: if (isprefix(b, "Start:", e, &p)) { -- 1.9.1 From daghf at varnish-software.com Thu May 22 14:36:31 2014 From: daghf at varnish-software.com (Dag Haavi Finstad) Date: Thu, 22 May 2014 16:36:31 +0200 Subject: Roadblock for Varnish 4 vmod_saintmode Message-ID: Hi all, I was about to do saintmode in a VMOD for 4.0. My idea was to implement saintmode as a director, that lets you decorate a backend (or director) with saintmode capabilities. The vdi_healthy function for the saintmode director will then evaluate whether the backend is healthy based on the current object in question, just like saintmode in 3.0. In VCL, a typical usage example sub vcl_init { # first param is a backend, second param is saintmode_threshold new sm0 = saintmode.saintmode(b0, 10); new sm1 = saintmode.saintmode(b1, 10); new rr = directors.round_robin(); rr.add_backend(sm0.backend()); rr.add_backend(sm1.backend()); } sub vcl_backend_fetch { set bereq.backend = rr.backend(); } sub vcl_backend_response { if (beresp.status == 500) { saintmode.blacklist(20s); return (retry); } } In doing this, I ran into some problems when implementing the vdi_healthy function. This function is of course invoked outside of the typical VCL/vmod scope, thus we don't have the vrt_ctx available. This makes it hard to do a saintmode vmod, when there is no reliable way to get access to the objhead digest - which is very much needed to see if this object is present on the trouble list. In 3.0 this was solved by vdi_healthy getting a struct sess passed in, typedef unsigned vdi_healthy(const struct director *, const struct sess *sp); compared with what we have currently, typedef unsigned vdi_healthy(const struct director *, double *changed); I would like to discuss the possibility of adding a vrt_ctx parameter (or simply the digest) to the vdi_healthy function in Varnish 4.0. Other suggestions would also be welcome. :) I can probably suggest a patch for re-introducing a digest parameter, if wanted. Regards, Dag -- Dag Haavi Finstad Software Developer | Varnish Software AS Mobile: +47 476 64 134 We Make Websites Fly! From phk at phk.freebsd.dk Fri May 23 07:32:14 2014 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 23 May 2014 07:32:14 +0000 Subject: Roadblock for Varnish 4 vmod_saintmode In-Reply-To: References: Message-ID: <6671.1400830334@critter.freebsd.dk> In message , Dag Haavi Finstad wri tes: >In 3.0 this was solved by vdi_healthy getting a struct sess passed >in, >I would like to discuss the possibility of adding a vrt_ctx parameter >(or simply the digest) to the vdi_healthy function in Varnish >4.0. Other suggestions would also be welcome. :) This runs into a very big question: Are backends (and should they be ?) visible on the request side at all ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Fri May 23 08:44:22 2014 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 23 May 2014 08:44:22 +0000 Subject: Roadblock for Varnish 4 vmod_saintmode In-Reply-To: <6671.1400830334@critter.freebsd.dk> References: <6671.1400830334@critter.freebsd.dk> Message-ID: <7291.1400834662@critter.freebsd.dk> In message <6671.1400830334 at critter.freebsd.dk>, "Poul-Henning Kamp" writes: >In message , Dag Haavi Finstad wri >tes: > >>In 3.0 this was solved by vdi_healthy getting a struct sess passed >>in, > >>I would like to discuss the possibility of adding a vrt_ctx parameter >>(or simply the digest) to the vdi_healthy function in Varnish >>4.0. Other suggestions would also be welcome. :) > >This runs into a very big question: > >Are backends (and should they be ?) visible on the request side at all ? Sorry, that was probably too terse... The question is probably better expressed this way ? Do the saint::health need access to req.* ? or bereq.* ? What will it do ? Make grace decisions on the client side or fetch decisions on the backend side ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From lkarsten at varnish-software.com Fri May 23 10:00:17 2014 From: lkarsten at varnish-software.com (Lasse Karstensen) Date: Fri, 23 May 2014 12:00:17 +0200 Subject: Bugwash on hangout Monday May 26th 12:00 CEST Message-ID: <20140523100016.GA17382@immer.varnish-software.com> Hi all. As we discussed on IRC for post-4.0 process improvements, I'd like to try doing the bugwash meetings using videoconferencing once a month. We'll do bugwash on Google Hangout on Monday May 26th at 12:00 CEST. Agenda is: 1. Quick introduction (if we have new people joining.) 2. Walkthrough/triage of new bugs from [1]. 3. In-depth discussion of important bugs/issues underway. 4. Other issues. I'll make a list to run through as these creep up during the meeting. If anyone have something significant for #3, please reply with ticket numbers in advance so we get time to read through them and prepare ourselves. We shouldn't spend more than an hour on this. Things that need serious thought can be discussed further on IRC afterwards. I'll set up a Google Hangout session and distribute the invitations on Monday. Send me an email with your g+-enabled account so I know who to invite. On the technical side; please get hold of a microphone headset, so we avoid unecessary feedback and forced muting rounds. (and please test it beforehand :)) 1: https://www.varnish-cache.org/trac/report/13 -- Lasse Karstensen With release manager hat. From daghf at varnish-software.com Fri May 23 10:34:22 2014 From: daghf at varnish-software.com (Dag Haavi Finstad) Date: Fri, 23 May 2014 12:34:22 +0200 Subject: Roadblock for Varnish 4 vmod_saintmode In-Reply-To: <7291.1400834662@critter.freebsd.dk> References: <6671.1400830334@critter.freebsd.dk> <7291.1400834662@critter.freebsd.dk> Message-ID: On Fri, May 23, 2014 at 10:44 AM, Poul-Henning Kamp wrote: > In message <6671.1400830334 at critter.freebsd.dk>, "Poul-Henning Kamp" writes: >>In message , Dag Haavi Finstad wri >>tes: >> >>>In 3.0 this was solved by vdi_healthy getting a struct sess passed >>>in, >> >>>I would like to discuss the possibility of adding a vrt_ctx parameter >>>(or simply the digest) to the vdi_healthy function in Varnish >>>4.0. Other suggestions would also be welcome. :) >> >>This runs into a very big question: >> >>Are backends (and should they be ?) visible on the request side at all ? > > > Sorry, that was probably too terse... > > > The question is probably better expressed this way ? > > Do the saint::health need access to req.* ? or bereq.* ? > > What will it do ? Make grace decisions on the client side or fetch > decisions on the backend side ? The way I envisioned it is purely on the fetch side. So 'saintmode' in this case will be a tool that helps us avoid picking a backend for certain URLs. Unlike 3.0, it's not meant to be used for deciding whether or not to serve a graced object, and I think that's fine since 4.0 does "stale-while-revalidate" and serves grace candidates by default anyway. It would of course be nice to be able to avoid kicking off a fetch thread if we had that information at hand, but I don't see how we could do that without severely breaking the rules of client/fetch side separation. -- Dag Haavi Finstad Software Developer | Varnish Software AS Mobile: +47 476 64 134 We Make Websites Fly! From kenshaw at gmail.com Sun May 25 04:58:54 2014 From: kenshaw at gmail.com (Kenneth Shaw) Date: Sun, 25 May 2014 11:58:54 +0700 Subject: vmod 4 / libvmod-example (and derivatives) / aclocal Message-ID: Hi, With the new 4.0 varnish path discovery for vmods (through the AC tools/macros), if you install varnish in a non-standard directory (ie, with the default prefix of /usr/local), then libvmod-example (and its derivatives -- for instance, libvmod-dns that I originally wrote), will not compile out of the box using the autogen.sh script. The reason for this is in varnish's autogen.sh, and the call to aclocal: aclocal -I m4 Specifically, because (as per http://lassekarstensen.wordpress.com/2013/12/19/converting-a-varnish-3-0-vmod-to-4-0/) the AC tools need to have access to the varnish.m4 which would cause the varnish.m4 file to be installed in the /usr/local/share/aclocal directory. As per http://www.gnu.org/software/automake/manual/html_node/Macro-Search-Path.html#ACLOCAL_005fPATHone can fix this causing aclocal to search on a different path, and can be accomplished by putting a 'dirlist' file in /usr/share/aclocal or by running autogen.sh for a vmod with the ACLOCAL_PATH set: ACLOCAL_PATH=/usr/local/share/aclocal ./autogen.sh While this is an OK workaround, for compiling the vmods, that to me seems to defeats the purpose of path autodiscovery for the vmods. I believe that the right solution to this would be to change varnish's autogen.sh file, so that the call to aclocal would instead look like the following (ie, add the --install parameter): aclocal -I m4 --install This causes aclocal to install varnish.m4 into the /usr/share/aclocal path instead of /usr/local/share/aclocal Anyway, I'm not sure if this is 100% the "right" way to fix this issue, but without changing varnish's autogen.sh file, then I would suggest at minimum the ACLOCAL_PATH should be documented in libvmod-example. Input would be greatly appreciated. Thanks in advance! -Ken -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Mon May 26 09:00:46 2014 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 26 May 2014 09:00:46 +0000 Subject: Bugwash on hangout Monday May 26th 12:00 CEST In-Reply-To: <20140523100016.GA17382@immer.varnish-software.com> References: <20140523100016.GA17382@immer.varnish-software.com> Message-ID: <58099.1401094846@critter.freebsd.dk> In message <20140523100016.GA17382 at immer.varnish-software.com>, Lasse Karstense n writes: >As we discussed on IRC for post-4.0 process improvements, I'd like to >try doing the bugwash meetings using videoconferencing once a month. > >We'll do bugwash on Google Hangout on Monday May 26th at 12:00 CEST. > >Agenda is: > > 1. Quick introduction (if we have new people joining.) > 2. Walkthrough/triage of new bugs from [1]. Actually, we will be doing "reverse" order, starting from oldest bugs to newest. I would also like if people would do a quick scan of the patchworks list to eliminate patches overtaken by events, so that we can go through that. https://www.varnish-cache.org/patchwork/project/varnish-cache/list/ -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Mon May 26 09:04:49 2014 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 26 May 2014 09:04:49 +0000 Subject: Roadblock for Varnish 4 vmod_saintmode In-Reply-To: References: <6671.1400830334@critter.freebsd.dk> <7291.1400834662@critter.freebsd.dk> Message-ID: <59017.1401095089@critter.freebsd.dk> In message , Dag Haavi Finstad writes: >The way I envisioned it is purely on the fetch side. So 'saintmode' in >this case will be a tool that helps us avoid picking a backend for >certain URLs. So how exactly does that work ? Once you havn't picked the backend, what backend do you then use ? Does the saint director pick another one ? (by what algorithm ?) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From martin at varnish-software.com Mon May 26 13:05:07 2014 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 26 May 2014 15:05:07 +0200 Subject: [PATCH] Expire callback implementation Message-ID: <1401109507-20292-1-git-send-email-martin@varnish-software.com> --- bin/varnishd/cache/cache.h | 19 ++++++- bin/varnishd/cache/cache_expire.c | 76 +++++++++++++++++++++++++- bin/varnishd/cache/cache_hash.c | 2 +- bin/varnishd/storage/storage_persistent_silo.c | 2 +- 4 files changed, 92 insertions(+), 7 deletions(-) diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index c8d8d89..4c0196a 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -291,6 +291,18 @@ struct exp { /*--------------------------------------------------------------------*/ +typedef void exp_callback_f(struct dstat *ds, struct objcore *oc, void *priv); +struct exp_callback { + unsigned magic; +#define EXP_CALLBACK_MAGIC 0xAB956EB1 + exp_callback_f *cb_insert; + exp_callback_f *cb_remove; + void *priv; + VTAILQ_ENTRY(exp_callback) list; +}; + +/*--------------------------------------------------------------------*/ + struct vsl_log { uint32_t *wlb, *wlp, *wle; unsigned wlr; @@ -919,14 +931,17 @@ extern pthread_t cli_thread; void EXP_Clr(struct exp *e); double EXP_Ttl(const struct req *, const struct object*); -void EXP_Insert(struct objcore *oc); -void EXP_Inject(struct objcore *oc, struct lru *lru, double when); +void EXP_Insert(struct dstat *ds, struct objcore *oc); +void EXP_Inject(struct dstat *ds, struct objcore *oc, struct lru *lru, + double when); void EXP_Init(void); void EXP_Rearm(struct object *o, double now, double ttl, double grace, double keep); void EXP_Touch(struct objcore *oc, double now); int EXP_NukeOne(struct busyobj *, struct lru *lru); void EXP_NukeLRU(struct worker *wrk, struct vsl_log *vsl, struct lru *lru); +void EXP_Reg_Callback(struct exp_callback *cb); +void EXP_Dereg_Callback(struct exp_callback *cb); /* cache_fetch.c */ enum vbf_fetch_mode_e { diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c index 8bd0707..e076fec 100644 --- a/bin/varnishd/cache/cache_expire.c +++ b/bin/varnishd/cache/cache_expire.c @@ -52,10 +52,47 @@ struct exp_priv { VTAILQ_HEAD(,objcore) inbox; struct binheap *heap; pthread_cond_t condvar; + + VTAILQ_HEAD(,exp_callback) cb_list; + pthread_rwlock_t cb_mtx; }; static struct exp_priv *exphdl; +static void +exp_insert_cb(struct dstat *ds, struct objcore *oc) +{ + struct exp_callback *cb; + + if (VTAILQ_EMPTY(&exphdl->cb_list)) + return; + + AZ(pthread_rwlock_rdlock(&exphdl->cb_mtx)); + VTAILQ_FOREACH(cb, &exphdl->cb_list, list) { + CHECK_OBJ_NOTNULL(cb, EXP_CALLBACK_MAGIC); + if (cb->cb_insert) + cb->cb_insert(ds, oc, cb->priv); + } + AZ(pthread_rwlock_unlock(&exphdl->cb_mtx)); +} + +static void +exp_remove_cb(struct dstat *ds, struct objcore *oc) +{ + struct exp_callback *cb; + + if (VTAILQ_EMPTY(&exphdl->cb_list)) + return; + + AZ(pthread_rwlock_rdlock(&exphdl->cb_mtx)); + VTAILQ_FOREACH(cb, &exphdl->cb_list, list) { + CHECK_OBJ_NOTNULL(cb, EXP_CALLBACK_MAGIC); + if (cb->cb_remove) + cb->cb_remove(ds, oc, cb->priv); + } + AZ(pthread_rwlock_unlock(&exphdl->cb_mtx)); +} + /*-------------------------------------------------------------------- * struct exp manipulations */ @@ -130,7 +167,7 @@ exp_mail_it(struct objcore *oc) */ void -EXP_Inject(struct objcore *oc, struct lru *lru, double when) +EXP_Inject(struct dstat *ds, struct objcore *oc, struct lru *lru, double when) { CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); @@ -152,6 +189,8 @@ EXP_Inject(struct objcore *oc, struct lru *lru, double when) oc->timer_when = when; Lck_Unlock(&lru->mtx); + exp_insert_cb(ds, oc); + exp_mail_it(oc); } @@ -163,12 +202,12 @@ EXP_Inject(struct objcore *oc, struct lru *lru, double when) */ void -EXP_Insert(struct objcore *oc) +EXP_Insert(struct dstat *ds, struct objcore *oc) { CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); HSH_Ref(oc); - EXP_Inject(oc, NULL, -1); + EXP_Inject(ds, oc, NULL, -1); } /*-------------------------------------------------------------------- @@ -338,6 +377,7 @@ EXP_NukeOne(struct busyobj *bo, struct lru *lru) oc_getxid(bo->stats, oc) & VSL_IDENTMASK); AN(bo->stats); AN(oc); + exp_remove_cb(bo->stats, oc); (void)HSH_DerefObjCore(bo->stats, &oc); return (1); } @@ -397,6 +437,7 @@ EXP_NukeLRU(struct worker *wrk, struct vsl_log *vsl, struct lru *lru) oc_getxid(&wrk->stats, oc) & VSL_IDENTMASK, EXP_Ttl(NULL, o) - t); o->exp.ttl = 0.0; + exp_remove_cb(&wrk->stats, oc); (void)HSH_DerefObjCore(&wrk->stats, &oc); } @@ -409,6 +450,31 @@ EXP_NukeLRU(struct worker *wrk, struct vsl_log *vsl, struct lru *lru) #endif +void +EXP_Reg_Callback(struct exp_callback *cb) +{ + CHECK_OBJ_NOTNULL(cb, EXP_CALLBACK_MAGIC); + AZ(pthread_rwlock_wrlock(&exphdl->cb_mtx)); + VTAILQ_INSERT_TAIL(&exphdl->cb_list, cb, list); + AZ(pthread_rwlock_unlock(&exphdl->cb_mtx)); +} + +void +EXP_Dereg_Callback(struct exp_callback *cb) +{ + struct exp_callback *cb2; + + CHECK_OBJ_NOTNULL(cb, EXP_CALLBACK_MAGIC); + AZ(pthread_rwlock_wrlock(&exphdl->cb_mtx)); + VTAILQ_FOREACH(cb2, &exphdl->cb_list, list) { + CHECK_OBJ_NOTNULL(cb2, EXP_CALLBACK_MAGIC); + if (cb2 == cb) + break; + } + AN(cb2); + VTAILQ_REMOVE(&exphdl->cb_list, cb2, list); + AZ(pthread_rwlock_unlock(&exphdl->cb_mtx)); +} /*-------------------------------------------------------------------- * Handle stuff in the inbox @@ -452,6 +518,7 @@ exp_inbox(struct exp_priv *ep, struct objcore *oc, double now) binheap_delete(ep->heap, oc->timer_idx); } assert(oc->timer_idx == BINHEAP_NOIDX); + exp_remove_cb(&ep->wrk->stats, oc); (void)HSH_DerefObjCore(&ep->wrk->stats, &oc); return; } @@ -537,6 +604,7 @@ exp_expire(struct exp_priv *ep, double now) VSLb(&ep->vsl, SLT_ExpKill, "EXP_Expired x=%u t=%.0f", oc_getxid(&ep->wrk->stats, oc) & VSL_IDENTMASK, EXP_Ttl(NULL, o) - now); + exp_remove_cb(&ep->wrk->stats, oc); (void)HSH_DerefObjCore(&ep->wrk->stats, &oc); return (0); } @@ -618,6 +686,8 @@ EXP_Init(void) Lck_New(&ep->mtx, lck_exp); AZ(pthread_cond_init(&ep->condvar, NULL)); VTAILQ_INIT(&ep->inbox); + AZ(pthread_rwlock_init(&ep->cb_mtx, NULL)); + VTAILQ_INIT(&ep->cb_list); exphdl = ep; WRK_BgThread(&pt, "cache-timeout", exp_thread, ep); } diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c index f4bbae0..ed269c6 100644 --- a/bin/varnishd/cache/cache_hash.c +++ b/bin/varnishd/cache/cache_hash.c @@ -683,7 +683,7 @@ HSH_Unbusy(struct dstat *ds, struct objcore *oc) if (!(oc->flags & OC_F_PRIVATE)) { BAN_NewObjCore(oc); - EXP_Insert(oc); + EXP_Insert(ds, oc); AN(oc->exp_flags & OC_EF_EXP); AN(oc->ban); } diff --git a/bin/varnishd/storage/storage_persistent_silo.c b/bin/varnishd/storage/storage_persistent_silo.c index f27ccd9..dd93276 100644 --- a/bin/varnishd/storage/storage_persistent_silo.c +++ b/bin/varnishd/storage/storage_persistent_silo.c @@ -158,7 +158,7 @@ smp_load_seg(struct worker *wrk, const struct smp_sc *sc, smp_init_oc(oc, sg, no); oc->ban = BAN_RefBan(oc, so->ban, sc->tailban); HSH_Insert(wrk, so->hash, oc); - EXP_Inject(oc, sg->lru, so->ttl); + EXP_Inject(&wrk->stats, oc, sg->lru, so->ttl); sg->nobj++; } WRK_SumStat(wrk); -- 2.0.0.rc2 From daghf at varnish-software.com Mon May 26 15:04:08 2014 From: daghf at varnish-software.com (Dag Haavi Finstad) Date: Mon, 26 May 2014 17:04:08 +0200 Subject: Roadblock for Varnish 4 vmod_saintmode In-Reply-To: <59017.1401095089@critter.freebsd.dk> References: <6671.1400830334@critter.freebsd.dk> <7291.1400834662@critter.freebsd.dk> <59017.1401095089@critter.freebsd.dk> Message-ID: On Mon, May 26, 2014 at 11:04 AM, Poul-Henning Kamp wrote: > In message > , Dag Haavi Finstad writes: > >>The way I envisioned it is purely on the fetch side. So 'saintmode' in >>this case will be a tool that helps us avoid picking a backend for >>certain URLs. > > So how exactly does that work ? > > Once you havn't picked the backend, what backend do you then use ? > > Does the saint director pick another one ? (by what algorithm ?) The saintmode director is only a simple decorator around a single backend. It will not pick a different backend by itself, it will merely report as sick. If an object is found to be on its trouble list in be->healthy(), it will be reported as sick. Likewise, be->getfd() will return NULL and it will be a 503 if it ends up getting selected anyway. The policy of how a different backend is picked is left up to whichever other director is in use, or by a vcl decision. I'll revisit the previous example, with a few more comments added: sub vcl_init { # sm0 and sm1 represent separate backends, with each their # separate trouble lists. new sm0 = saintmode.saintmode(b0, 10); new sm1 = saintmode.saintmode(b1, 10); new rr = directors.round_robin(); rr.add_backend(sm0.backend()); rr.add_backend(sm1.backend()); # .backend() here returns a pointer to a 'struct director' # representing that specific saintmode director } sub vcl_backend_fetch { set bereq.backend = rr.backend(); # the round-robin director will evaluate be->healthy() upon # picking a backend. The healthy() function in the selected saint # mode dir will then report as sick in the case that the specific # object is on its trouble list. The choice is then left to the # r-r director to pick the next healthy backend. } sub vcl_backend_response { if (beresp.status == 500) { saintmode.blacklist(20s); # equivalent of 3.0's set beresp.saintmode = 20s return (retry); # upon retry, the offending backend (saintmode dir) will # report as sick. # .blacklist() should be a function and not an object method, # since when saintmode objects are used in a nested fashion like # here, we wouldn't be able to tell which 'sm' object to invoke # the .blacklist() method on. The .blacklist() function will be able to # figure out which backend and hence saintmode object to use. } } Regards, -- Dag Haavi Finstad Software Developer | Varnish Software AS Mobile: +47 476 64 134 We Make Websites Fly! From fgsch at lodoss.net Mon May 26 23:41:49 2014 From: fgsch at lodoss.net (Federico Schwindt) Date: Tue, 27 May 2014 00:41:49 +0100 Subject: vmod 4 / libvmod-example (and derivatives) / aclocal In-Reply-To: References: Message-ID: Hi, Personally I think if you install Varnish in a non-standard location you know what you are doing so you should use ACLOCAL_PATH or go for the dirlist option. As Lasse mentioned in hist post: "If you installed Varnish in the standard path/prefix, that should work out of the box. Otherwise, you might to add some symlinks for pkg-config and automake to find the source." AFAICT --install copies the varnish.m4 file into the -I directory instead of stuffing everything into aclocal.m4 so this won't help. IIUC your suggestion could be done modifying Varnish's Makefile.am to use /usr/share insted of $(datadir)/aclocal but installing varnish.m4 under /usr and the rest of Varnish under /usr/local doesn't make much sense to me and will break systems that install aclocal under /usr/local. I'm not sure how many people installs Varnish in a prefix different to where aclocal lives TBH but I'd say documenting it is the way to go. On Sun, May 25, 2014 at 5:58 AM, Kenneth Shaw wrote: > Hi, > > With the new 4.0 varnish path discovery for vmods (through the AC > tools/macros), if you install varnish in a non-standard directory (ie, with > the default prefix of /usr/local), then libvmod-example (and its > derivatives -- for instance, libvmod-dns that I originally wrote), will not > compile out of the box using the autogen.sh script. > > The reason for this is in varnish's autogen.sh, and the call to aclocal: > > aclocal -I m4 > > Specifically, because (as per > http://lassekarstensen.wordpress.com/2013/12/19/converting-a-varnish-3-0-vmod-to-4-0/) the AC tools need to have access to the varnish.m4 which would cause the > varnish.m4 file to be installed in the /usr/local/share/aclocal directory. > > As per > http://www.gnu.org/software/automake/manual/html_node/Macro-Search-Path.html#ACLOCAL_005fPATHone can fix this causing aclocal to search on a different path, and can be > accomplished by putting a 'dirlist' file in /usr/share/aclocal or by > running autogen.sh for a vmod with the ACLOCAL_PATH set: > > ACLOCAL_PATH=/usr/local/share/aclocal ./autogen.sh > > While this is an OK workaround, for compiling the vmods, that to me seems > to defeats the purpose of path autodiscovery for the vmods. > > I believe that the right solution to this would be to change varnish's > autogen.sh file, so that the call to aclocal would instead look like the > following (ie, add the --install parameter): > > aclocal -I m4 --install > > This causes aclocal to install varnish.m4 into the /usr/share/aclocal path > instead of /usr/local/share/aclocal > > Anyway, I'm not sure if this is 100% the "right" way to fix this issue, > but without changing varnish's autogen.sh file, then I would suggest at > minimum the ACLOCAL_PATH should be documented in libvmod-example. > > Input would be greatly appreciated. Thanks in advance! > > -Ken > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdzstz at gmail.com Wed May 28 23:01:21 2014 From: jdzstz at gmail.com (jdzstz) Date: Thu, 29 May 2014 01:01:21 +0200 Subject: Updated varnish cygwin patches for 4.0.x varnish version - merge? In-Reply-To: References: Message-ID: <53866AC1.1000706@gmail.com> Hi all, I have prepared a version of cygwin patches for 4.0.x varnish versions, It would be nice to merge it with varnish master and 4.x code. I tested them against last snapshots, my last tests was against 25/May/2014 snapshot ( http://repo.varnish-cache.org/snapshots/varnish-4.0.0+2014-05-25.tar.gz ) There are three patch files, I attach them at email: 1. Varnish minor bugs, detected during Cygwin tests: *varnish-4.0.0+2014-05-25-1.bugs.patch* 2. Varnish code modification to allow windows paths at Cygwin: *varnish-4.0.0+2014-05-25-1.cygwin-src-winpaths.patch* 3. Varnish makefile modifications to allow build application at Cygwin: *varnish-4.0.0+2014-05-25-1.cygwin-make.patch* The changes has been tested at Cygwin and Linux, so my changes should not break anything at Linux build. It will save to me a lot of time to me during Cygwin package compilation/creation if you add the code to master, because will avoid me to compare/merge old and new codes when ".patch" file is not applied fine due line changes at master. _ _ _More detail about my changes:_ *1) **varnish-4.0.0+2014-05-25-1.bugs.patch* - minor bugs - At _bin/varnishd/cache/cache_acceptor.c_ A << #if defined(SO_SNDTIMEO_WORKS) || defined(SO_RCVTIMEO_WORKS) >> is need to avoid warning with "timeval tv;" when SO_SNDTIMEO or SO_RCVTIMEO are not available - At _doc/sphinx/Makefile.am and man/Makefile.am_ $(top_srcdir) must be replaced with $(top_builddir) to avoid errors when $(top_srcdir) != $(top_builddir) because generated RST files are not created at source directories A mkdir -p $(shell dirname $@) must be also added at RST creation. At Cygwin, by default, the cygport program always separe src and build dir during package build. At Linux it can be reproduced if you execute "./configure" and "make" outside src directory, in order to separate built files: 1. mkdir ~/builddir 2. cd ~/builddir 3. ~/src/varnish4/configure --srcdir=~/src/varnish4 (inside builddir) 4. make (inside builddir) - At _lib/libvarnishapi/Makefile.am:_ vsl2rst must be replaced with vsl2rst$(EXEEXT) because at cygwin, the executable is appended a ".exe" (at linux it does nothing) - At _lib/libvarnish/vfil.c_: The code: bprintf(fnb, "/%s/%s", pfx, fn); Add a initial "/" but pfx already starts with "/", duplicating the char: //usr/local/... This double // seems that does not cause trouble at Linux but at Cygwin causes issues because //computer is used to access to other computers. Solution: replace bprintf(fnb, "/%s/%s", pfx, fn); with bprintf(fnb, "%s/%s", pfx, fn); *2) varnish-4.0.0+2014-05-25-1.cygwin-src-winpaths.patch* - allow windows paths Minor changes at *bin/varnishd/mgt/mgt_vcc.c* and***lib/libvarnish/vin.c* to allow windows paths and "cmd.exe" shell when we are compiling varnish at cygwin *3) **varnish-4.0.0+2014-05-25-1.cygwin-make.patch * - cygwin compilation* * Makefile and configure changes, to allow Cygwin compilation: - lib vmod are compiled after "bin" compilation to avoid some errors: at windows those libs are linked with varnishd.exe file, created at "bin" compilation, and standard compilation order causes trouble with vmod dependences - VCC_CC variable (and others) definition at "configure" - Conditional definition and usage of lots of variables : AC_SUBST(CYGWIN_varnishd_LDFLAGS) , AC_SUBST(CYGWIN_libvarnishd_OBJECTS) , AC_SUBST(CYGWIN_libvcc_LIBADD) , AC_SUBST(CYGWIN_libvmod_std_la_LIBADD) , AC_SUBST(CYGWIN_lib_la_LDFLAGS_no_undefined) only at Cygwin compilations - etc... Tell me if you want more information about any cygwin change. Regards, Jorge D?az --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- --- origsrc/varnish-4.0.0+2014-05-25/bin/varnishd/mgt/mgt_vcc.c 2014-05-25 02:15:09.000000000 +0200 +++ src/varnish-4.0.0+2014-05-25/bin/varnishd/mgt/mgt_vcc.c 2014-05-26 01:02:04.942359200 +0200 @@ -176,8 +176,25 @@ run_vcc(void *priv) static void run_cc(void *priv) { + int i; mgt_sandbox(SANDBOX_CC); - (void)execl("/bin/sh", "/bin/sh", "-c", priv, (char*)0); +#if defined(__CYGWIN__) + if( access( "/bin/sh", X_OK ) != -1 ) + { + i=execl("/bin/sh", "/bin/sh", "-c", priv, (char*)0); + } + else + { + i=execl(getenv("COMSPEC"), getenv("COMSPEC"), "/c", priv, (char*)0); + } +#else + i=execl("/bin/sh", "/bin/sh", "-c", priv, (char*)0); +#endif + if(i) + { + printf("Error %i in run_cc: %s\n",errno,strerror(errno)); + printf("run_cc command: %s\n", (char*)priv); + } } /*-------------------------------------------------------------------- --- origsrc/varnish-4.0.0+2014-05-25/lib/libvarnish/vin.c 2014-05-25 02:15:09.000000000 +0200 +++ src/varnish-4.0.0+2014-05-25/lib/libvarnish/vin.c 2014-05-26 01:03:39.232885000 +0200 @@ -36,6 +36,9 @@ #include #include #include +#if defined(__CYGWIN__) +#include +#endif #include "vapi/vsm_int.h" #include "vas.h" @@ -66,6 +69,13 @@ VIN_N_Arg(const char *n_arg, char **name if (*nm == '/') strcpy(dn, nm); +#if defined(__CYGWIN__) +/* In Windows we also tolerate other absolute paths: \\hostname\path, c:/path and c:\path */ + else if (strlen(nm)>2 && nm[0] == '\\' && nm[1] == '\\') + strcpy(dn, nm); + else if (strlen(nm)>3 && isalpha((unsigned char)nm[0]) && nm[1] == ':' && (nm[2] == '/'||nm[2] == '\\')) + strcpy(dn, nm); +#endif else if (strlen(VARNISH_STATE_DIR) + 1 + strlen(nm) >= sizeof dn){ /* preliminary length check to avoid overflowing dm */ errno = ENAMETOOLONG; -------------- next part -------------- --- origsrc/varnish-4.0.0+2014-05-25/bin/varnishd/cache/cache_acceptor.c 2014-05-25 02:15:09.000000000 +0200 +++ src/varnish-4.0.0+2014-05-25/bin/varnishd/cache/cache_acceptor.c 2014-05-26 01:00:35.831931000 +0200 @@ -128,7 +128,9 @@ vca_tcp_opt_init(void) int one = 1; // int zero = 0; struct tcp_opt *to; +#if defined(SO_SNDTIMEO_WORKS) || defined(SO_RCVTIMEO_WORKS) struct timeval tv; +#endif int chg = 0; #ifdef HAVE_TCP_KEEP int x; --- origsrc/varnish-4.0.0+2014-05-25/doc/sphinx/Makefile.am 2014-05-25 02:15:09.000000000 +0200 +++ src/varnish-4.0.0+2014-05-25/doc/sphinx/Makefile.am 2014-05-26 02:11:26.850983700 +0200 @@ -144,11 +144,13 @@ include/varnishhist_synopsis.rst: $(top_ BUILT_SOURCES += include/varnishhist_options.rst \ include/varnishhist_synopsis.rst -reference/vmod_std.generated.rst: $(top_srcdir)/lib/libvmod_std/vmod_std.rst +reference/vmod_std.generated.rst: $(top_builddir)/lib/libvmod_std/vmod_std.rst + mkdir -p $(shell dirname $@) cp $? $@ BUILT_SOURCES += reference/vmod_std.generated.rst -reference/vmod_directors.generated.rst: $(top_srcdir)/lib/libvmod_directors/vmod_directors.rst +reference/vmod_directors.generated.rst: $(top_builddir)/lib/libvmod_directors/vmod_directors.rst + mkdir -p $(shell dirname $@) cp $? $@ BUILT_SOURCES += reference/vmod_directors.generated.rst --- origsrc/varnish-4.0.0+2014-05-25/lib/libvarnish/vfil.c 2014-05-25 02:15:09.000000000 +0200 +++ src/varnish-4.0.0+2014-05-25/lib/libvarnish/vfil.c 2014-05-26 01:03:29.609480000 +0200 @@ -110,7 +110,7 @@ VFIL_readfile(const char *pfx, const cha if (fn[0] == '/') fd = open(fn, O_RDONLY); else if (pfx != NULL) { - bprintf(fnb, "/%s/%s", pfx, fn); + bprintf(fnb, "%s/%s", pfx, fn); /* XXX: graceful length check */ fd = open(fnb, O_RDONLY); } else --- origsrc/varnish-4.0.0+2014-05-25/lib/libvarnishapi/Makefile.am 2014-05-25 02:15:09.000000000 +0200 +++ src/varnish-4.0.0+2014-05-25/lib/libvarnishapi/Makefile.am 2014-05-26 01:03:56.186163300 +0200 @@ -69,14 +69,14 @@ CLEANFILES = \ MAINTAINERCLEANFILES = \ vsl-tags.rst -noinst_PROGRAMS = vsl2rst +noinst_PROGRAMS = vsl2rst$(EXEEXT) vsl2rst_SOURCES = \ vsl2rst.c \ $(top_srcdir)/include/tbl/vsl_tags.h \ $(top_srcdir)/include/tbl/vsl_tags_http.h -vsl-tags.rst: vsl2rst +vsl-tags.rst: vsl2rst$(EXEEXT) ./vsl2rst > $@ vxp_fixed_token.c vxp_tokens.h: \ --- origsrc/varnish-4.0.0+2014-05-25/man/Makefile.am 2014-05-25 02:15:09.000000000 +0200 +++ src/varnish-4.0.0+2014-05-25/man/Makefile.am 2014-05-26 01:07:31.665109700 +0200 @@ -86,8 +86,8 @@ varnishhist.1: \ $(top_srcdir)/doc/sphinx/include/varnishhist_synopsis.rst ${RST2MAN} $(RST2ANY_FLAGS) $(top_srcdir)/doc/sphinx/reference/varnishhist.rst $@ -vmod_std.3: $(top_srcdir)/lib/libvmod_std/vmod_std.man.rst +vmod_std.3: $(top_builddir)/lib/libvmod_std/vmod_std.man.rst ${RST2MAN} $(RST2ANY_FLAGS) $? $@ -vmod_directors.3: $(top_srcdir)/lib/libvmod_directors/vmod_directors.man.rst +vmod_directors.3: $(top_builddir)/lib/libvmod_directors/vmod_directors.man.rst ${RST2MAN} $(RST2ANY_FLAGS) $? $@ -------------- next part -------------- --- origsrc/varnish-4.0.0+2014-05-25/Makefile.am 2014-05-25 02:15:09.000000000 +0200 +++ src/varnish-4.0.0+2014-05-25/Makefile.am 2014-05-26 00:59:53.380056800 +0200 @@ -1,6 +1,6 @@ ACLOCAL_AMFLAGS = -I m4 -SUBDIRS = include lib bin etc doc man redhat +SUBDIRS = include lib bin lib/libvmod_debug lib/libvmod_std lib/libvmod_directors etc doc man redhat pkgconfigdir = $(libdir)/pkgconfig pkgconfig_DATA = varnishapi.pc --- origsrc/varnish-4.0.0+2014-05-25/bin/varnishd/Makefile.am 2014-05-25 02:15:09.000000000 +0200 +++ src/varnish-4.0.0+2014-05-25/bin/varnishd/Makefile.am 2014-05-26 01:01:31.507921400 +0200 @@ -122,9 +122,15 @@ varnishd_LDADD = \ $(top_builddir)/lib/libvgz/libvgz.la \ @JEMALLOC_LDADD@ \ @PCRE_LIBS@ \ - ${DL_LIBS} ${PTHREAD_LIBS} ${NET_LIBS} ${LIBM} ${LIBUMEM} + ${DL_LIBS} ${PTHREAD_LIBS} ${NET_LIBS} ${LIBM} ${LIBUMEM} ${CYGWIN_varnishd_LDFLAGS} EXTRA_DIST = builtin.vcl + +EXTRA_SCRIPTS = + +sbin_SCRIPTS = ${CYGWIN_libvarnishd_OBJECTS} + + DISTCLEANFILES = builtin_vcl.h # --- origsrc/varnish-4.0.0+2014-05-25/configure.ac 2014-05-25 02:15:09.000000000 +0200 +++ src/varnish-4.0.0+2014-05-25/configure.ac 2014-05-26 00:59:16.626614500 +0200 @@ -459,6 +459,12 @@ gl_LD_VERSION_SCRIPT # failures. CFLAGS="${CFLAGS} -Wall -Werror" OCFLAGS="${OCFLAGS} -Wall -Werror" +case $target in +*-*-cygwin*) + CFLAGS="${CFLAGS} -Wno-char-subscripts" + OCFLAGS="${OCFLAGS} -Wno-char-subscripts" + ;; +esac AX_CHECK_COMPILE_FLAG([-Werror=unused-result], [CFLAGS="${CFLAGS} -Wno-error=unused-result" OCFLAGS="${OCFLAGS} -Wno-error=unused-result"], @@ -569,6 +575,19 @@ else *-*-darwin*) VCC_CC="exec cc $OCFLAGS -dynamiclib -Wl,-undefined,dynamic_lookup -o %o %s" ;; + *-*-cygwin*) + #calculate ${exec_prefix}/lib, because it is asigned later, and we need now + prefix_aux=`eval echo ${prefix}` + test "x$prefix_aux" = xNONE && prefix_aux=`eval echo ${ac_default_prefix}` + # Let make expand exec_prefix. + exec_prefix_aux=`eval echo ${exec_prefix}` + test "x$exec_prefix_aux" = xNONE && exec_prefix_aux=`eval echo ${prefix_aux}` + # Remove "/" if we are in Root + test "x$exec_prefix_aux" = "x/" && exec_prefix_aux="" + #calculate compilation path + comp_prefix=`pwd` + VCC_CC="$PTHREAD_CC $OCFLAGS $PTHREAD_CFLAGS -shared %s -L${comp_prefix}/lib/libvarnish/.libs -L${comp_prefix}/bin/varnishd -L${exec_prefix_aux}/lib -L${exec_prefix_aux}/sbin -L${exec_prefix_aux}/bin -lvarnish -lvarnishd -o %o" + ;; *) VCC_CC="exec $PTHREAD_CC $OCFLAGS $PTHREAD_CFLAGS -fpic -shared -Wl,-x -o %o %s" ;; @@ -595,6 +614,31 @@ AC_SUBST(VTC_TESTS) # Make sure this include dir exists AC_CONFIG_COMMANDS([mkdir], [$MKDIR_P doc/sphinx/include]) +# Add library dependencies with libvarnish (linker needs it in CYGWIN during compilation time) +# Generate fake library libvarnishd.dll.a in order to successful linking DLL when resolving varnishd.exe simbols +# Add -no-undefined to LDFLAGS +case $target in +*-*-cygwin*) + CYGWIN_varnishd_LDFLAGS=' -Wl,-out-implib,libvarnishd.dll.a' + CYGWIN_libvarnishd_OBJECTS='libvarnishd.dll.a' + CYGWIN_libvcc_LIBADD='$(top_builddir)/lib/libvarnish/libvarnish.la' + CYGWIN_libvmod_std_la_LIBADD='$(top_builddir)/lib/libvarnish/libvarnish.la -lvarnishd -L$(top_builddir)/bin/varnishd' + CYGWIN_lib_la_LDFLAGS_no_undefined='-no-undefined' + AC_SUBST(CYGWIN_varnishd_LDFLAGS) + AC_SUBST(CYGWIN_libvarnishd_OBJECTS) + AC_SUBST(CYGWIN_libvcc_LIBADD) + AC_SUBST(CYGWIN_libvmod_std_la_LIBADD) + AC_SUBST(CYGWIN_lib_la_LDFLAGS_no_undefined) + cygwin=true + ;; +*) + cygwin=false + ;; +esac + +AM_CONDITIONAL([CYGWIN_COMPILATION], test x$cygwin = xtrue) + + # Generate output AC_CONFIG_FILES([ Makefile --- origsrc/varnish-4.0.0+2014-05-25/lib/Makefile.am 2014-05-25 02:15:09.000000000 +0200 +++ src/varnish-4.0.0+2014-05-25/lib/Makefile.am 2014-05-26 01:06:46.265206900 +0200 @@ -6,10 +6,7 @@ SUBDIRS = \ libvarnishapi \ libvarnishtools \ libvcc \ - libvgz \ - libvmod_debug \ - libvmod_std \ - libvmod_directors + libvgz DIST_SUBDIRS = \ libvarnishcompat \ --- origsrc/varnish-4.0.0+2014-05-25/lib/libvarnish/Makefile.am 2014-05-25 02:15:09.000000000 +0200 +++ src/varnish-4.0.0+2014-05-25/lib/libvarnish/Makefile.am 2014-05-26 01:03:18.837312800 +0200 @@ -7,7 +7,7 @@ AM_LDFLAGS = $(AM_LT_LDFLAGS) pkglib_LTLIBRARIES = libvarnish.la -libvarnish_la_LDFLAGS = $(AM_LDFLAGS) -avoid-version +libvarnish_la_LDFLAGS = $(AM_LDFLAGS) -avoid-version ${CYGWIN_lib_la_LDFLAGS_no_undefined} libvarnish_la_SOURCES = \ vav.c \ --- origsrc/varnish-4.0.0+2014-05-25/lib/libvarnishapi/Makefile.am 2014-05-25 02:15:09.000000000 +0200 +++ src/varnish-4.0.0+2014-05-25/lib/libvarnishapi/Makefile.am 2014-05-26 01:03:56.186163300 +0200 @@ -9,7 +9,7 @@ AM_CPPFLAGS = \ lib_LTLIBRARIES = libvarnishapi.la -libvarnishapi_la_LDFLAGS = $(AM_LDFLAGS) -version-info 1:0:0 +libvarnishapi_la_LDFLAGS = $(AM_LDFLAGS) -version-info 1:0:0 ${CYGWIN_lib_la_LDFLAGS_no_undefined} libvarnishapi_la_SOURCES = \ vsm_api.h \ --- origsrc/varnish-4.0.0+2014-05-25/lib/libvarnishcompat/Makefile.am 2014-05-25 02:15:09.000000000 +0200 +++ src/varnish-4.0.0+2014-05-25/lib/libvarnishcompat/Makefile.am 2014-05-26 01:04:07.376608800 +0200 @@ -8,7 +8,7 @@ AM_LDFLAGS = $(AM_LT_LDFLAGS) pkglib_LTLIBRARIES = libvarnishcompat.la -libvarnishcompat_la_LDFLAGS = $(AM_LDFLAGS) -avoid-version +libvarnishcompat_la_LDFLAGS = $(AM_LDFLAGS) -avoid-version ${CYGWIN_lib_la_LDFLAGS_no_undefined} libvarnishcompat_la_SOURCES = \ daemon.c \ --- origsrc/varnish-4.0.0+2014-05-25/lib/libvcc/Makefile.am 2014-05-25 02:15:09.000000000 +0200 +++ src/varnish-4.0.0+2014-05-25/lib/libvcc/Makefile.am 2014-05-26 01:04:17.984667700 +0200 @@ -8,7 +8,7 @@ AM_CPPFLAGS = \ pkglib_LTLIBRARIES = libvcc.la -libvcc_la_LDFLAGS = $(AM_LDFLAGS) -avoid-version +libvcc_la_LDFLAGS = $(AM_LDFLAGS) -avoid-version ${CYGWIN_lib_la_LDFLAGS_no_undefined} libvcc_la_SOURCES = \ vcc_compile.h \ @@ -30,6 +30,8 @@ libvcc_la_SOURCES = \ vcc_var.c \ vcc_vmod.c \ vcc_xref.c + +libvcc_la_LIBADD = ${CYGWIN_libvcc_LIBADD} EXTRA_DIST = \ generate.py --- origsrc/varnish-4.0.0+2014-05-25/lib/libvgz/Makefile.am 2014-05-25 02:15:09.000000000 +0200 +++ src/varnish-4.0.0+2014-05-25/lib/libvgz/Makefile.am 2014-05-26 01:04:37.848008800 +0200 @@ -3,7 +3,7 @@ AM_LDFLAGS = $(AM_LT_LDFLAGS) pkglib_LTLIBRARIES = libvgz.la -libvgz_la_LDFLAGS = $(AM_LDFLAGS) -avoid-version +libvgz_la_LDFLAGS = $(AM_LDFLAGS) -avoid-version ${CYGWIN_lib_la_LDFLAGS_no_undefined} libvgz_la_CFLAGS = -D_LARGEFILE64_SOURCE=1 -DZLIB_CONST $(libvgz_extra_cflags) libvgz_la_SOURCES = \ --- origsrc/varnish-4.0.0+2014-05-25/lib/libvmod_debug/Makefile.am 2014-05-25 02:15:09.000000000 +0200 +++ src/varnish-4.0.0+2014-05-25/lib/libvmod_debug/Makefile.am 2014-05-26 01:05:38.039203500 +0200 @@ -13,12 +13,14 @@ vmodtoolargs = --strict noinst_LTLIBRARIES = libvmod_debug.la -libvmod_debug_la_LDFLAGS = $(AM_LDFLAGS) -module -export-dynamic -avoid-version -shared -rpath /nowhere +libvmod_debug_la_LDFLAGS = $(AM_LDFLAGS) -module -export-dynamic -avoid-version -shared -rpath /nowhere ${CYGWIN_lib_la_LDFLAGS_no_undefined} libvmod_debug_la_SOURCES = \ vmod_debug.c \ vmod_debug_obj.c +libvmod_debug_la_LIBADD = ${CYGWIN_libvmod_std_la_LIBADD} + nodist_libvmod_debug_la_SOURCES = \ vcc_if.c \ vcc_if.h --- origsrc/varnish-4.0.0+2014-05-25/lib/libvmod_directors/Makefile.am 2014-05-25 02:15:09.000000000 +0200 +++ src/varnish-4.0.0+2014-05-25/lib/libvmod_directors/Makefile.am 2014-05-26 01:06:00.593210200 +0200 @@ -12,7 +12,7 @@ vmodtool = $(top_srcdir)/lib/libvcc/vmod vmodtoolargs = --strict vmod_LTLIBRARIES = libvmod_directors.la -libvmod_directors_la_LDFLAGS = $(AM_LDFLAGS) -module -export-dynamic -avoid-version -shared +libvmod_directors_la_LDFLAGS = $(AM_LDFLAGS) -module -export-dynamic -avoid-version -shared ${CYGWIN_lib_la_LDFLAGS_no_undefined} libvmod_directors_la_SOURCES = \ vdir.c \ @@ -22,6 +22,8 @@ libvmod_directors_la_SOURCES = \ random.c \ round_robin.c +libvmod_directors_la_LIBADD = ${CYGWIN_libvmod_std_la_LIBADD} + nodist_libvmod_directors_la_SOURCES = \ vcc_if.c \ vcc_if.h --- origsrc/varnish-4.0.0+2014-05-25/lib/libvmod_std/Makefile.am 2014-05-25 02:15:09.000000000 +0200 +++ src/varnish-4.0.0+2014-05-25/lib/libvmod_std/Makefile.am 2014-05-26 01:06:33.500105900 +0200 @@ -13,13 +13,15 @@ vmodtool = $(top_srcdir)/lib/libvcc/vmod vmodtoolargs = --strict vmod_LTLIBRARIES = libvmod_std.la -libvmod_std_la_LDFLAGS = $(AM_LDFLAGS) -module -export-dynamic -avoid-version -shared +libvmod_std_la_LDFLAGS = $(AM_LDFLAGS) -module -export-dynamic -avoid-version -shared ${CYGWIN_lib_la_LDFLAGS_no_undefined} libvmod_std_la_SOURCES = \ vmod_std.c \ vmod_std_fileread.c \ vmod_std_conversions.c +libvmod_std_la_LIBADD = ${CYGWIN_libvmod_std_la_LIBADD} + nodist_libvmod_std_la_SOURCES = \ vcc_if.c \ vcc_if.h