From fgsch at lodoss.net Sun Oct 2 16:31:05 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Sun, 02 Oct 2016 18:31:05 +0200 Subject: [master] f43e9ae Don't check twice Message-ID: commit f43e9ae45ddb7b6bf666ea1b3d035f03697b17be Author: Federico G. Schwindt Date: Sun Oct 2 16:21:39 2016 +0100 Don't check twice diff --git a/bin/varnishd/cache/cache_ws.c b/bin/varnishd/cache/cache_ws.c index 57f334f..2eb443f 100644 --- a/bin/varnishd/cache/cache_ws.c +++ b/bin/varnishd/cache/cache_ws.c @@ -176,8 +176,6 @@ WS_Printf(struct ws *ws, const char *fmt, ...) va_list ap; char *p; - WS_Assert(ws); - assert(ws->r == NULL); u = WS_Reserve(ws, 0); p = ws->f; va_start(ap, fmt); From fgsch at lodoss.net Sun Oct 2 16:31:05 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Sun, 02 Oct 2016 18:31:05 +0200 Subject: [master] dfcf44c Flexelint cleanup Message-ID: commit dfcf44c67263e91b266d2c8b86cbe0cd39b523b7 Author: Federico G. Schwindt Date: Sun Oct 2 16:22:26 2016 +0100 Flexelint cleanup diff --git a/bin/varnishd/flint.lnt b/bin/varnishd/flint.lnt index 360706e..80d5c78 100644 --- a/bin/varnishd/flint.lnt +++ b/bin/varnishd/flint.lnt @@ -31,14 +31,6 @@ // XXX: I think this is a flexelint bug: -esym(522, vbit_clr) -// Stuff used outside varnishd --esym(759, BackSlash) --esym(765, BackSlash) --esym(759, BackSlashDecode) --esym(765, BackSlashDecode) --esym(757, base64*) - - // Stuff used by compiled VCL -esym(757, VRT_*) -esym(759, VRT_*) @@ -52,7 +44,6 @@ // FLINT Bug20090910_838 -efunc(838, VRT_purge) --efunc(838, VRT_purge_string) // Stuff in VMODs which is used through dl*(3) functions -esym(754, Vmod_*_Func::*) @@ -89,7 +80,6 @@ -sem(exp_timer, thread_mono) -sem(wrk_herdtimer_thread, thread_mono) -sem(wrk_herder_thread, thread_mono) --esym(458, VSL_stats) -esym(458, heritage) -esym(458, name_key) @@ -144,7 +134,6 @@ // cache_vcl.c -esym(528, vcl_call_method) // Not referenced --esym(528, vcl_handlingname) // Not referenced -e441 // for clause irregularity: loop variable '___' not found in 2nd for expression diff --git a/bin/varnishncsa/flint.lnt b/bin/varnishncsa/flint.lnt index fd2c864..5faddb8 100644 --- a/bin/varnishncsa/flint.lnt +++ b/bin/varnishncsa/flint.lnt @@ -8,14 +8,7 @@ -e785 // Too few initializers for aggregate -e786 // String concatenation within initializer -// cache_center.c --efunc(525, CNT_Session) // Negative indentation from line --efunc(525, http_FilterHeader) // Negative indentation from line --efunc(539, http_FilterHeader) // Positive indentation from line - // cache_vcl.c --efunc(525, vcl_handlingname) // Negative indentation from line --esym(528, vcl_handlingname) // Not referenced -e641 // Converting enum 'cli_status_e' to int // Review all below this line /////////////////////////////////////////////// diff --git a/bin/varnishtest/flint.lnt b/bin/varnishtest/flint.lnt index a527ec3..3c31604 100644 --- a/bin/varnishtest/flint.lnt +++ b/bin/varnishtest/flint.lnt @@ -3,7 +3,6 @@ -function(__assert(1), vtc_log(2)) -esym(850, av) --esym(528, iter_call) // Flexelint bug -esym(534, snprintf) // Only for varnishtest, and not really nice diff --git a/lib/libvarnish/flint.lnt b/lib/libvarnish/flint.lnt index c98d791..881e8f8 100644 --- a/lib/libvarnish/flint.lnt +++ b/lib/libvarnish/flint.lnt @@ -54,14 +54,7 @@ // cache.h -emacro(506, INCOMPL) // Constant value Boolean -// cache_center.c --efunc(525, CNT_Session) // Negative indentation from line --efunc(525, http_FilterFields) // Negative indentation from line --efunc(539, http_FilterFields) // Positive indentation from line - // cache_vcl.c --efunc(525, vcl_handlingname) // Negative indentation from line --esym(528, vcl_handlingname) // Not referenced -e641 // Converting enum 'cli_status_e' to int // Review all below this line /////////////////////////////////////////////// From nils.goroll at uplex.de Tue Oct 4 15:19:04 2016 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 04 Oct 2016 17:19:04 +0200 Subject: [master] 2bbcde1 document bereq preparation in core Message-ID: commit 2bbcde1d33dd6583405af876a5064fda6a0fe513 Author: Nils Goroll Date: Tue Oct 4 17:14:40 2016 +0200 document bereq preparation in core doc fixes #2107 diff --git a/doc/sphinx/users-guide/vcl-built-in-subs.rst b/doc/sphinx/users-guide/vcl-built-in-subs.rst index 21e5873..d5be45d 100644 --- a/doc/sphinx/users-guide/vcl-built-in-subs.rst +++ b/doc/sphinx/users-guide/vcl-built-in-subs.rst @@ -271,6 +271,32 @@ The `vcl_backend_fetch` subroutine may terminate with calling background fetch, control is passed to :ref:`vcl_synth` on the client side with ``resp.status`` preset to 503. +Before calling `vcl_backend_fetch`, varnish core prepares the `bereq` +backend request as follows: + +* ``set bereq.method = "GET"`` and ``set bereq.proto = "HTTP/1.1"``, + unless the request is a `pass` + +* ``set bereq.http.Accept_Encoding = "gzip"`` if + :ref:`ref_param_http_gzip_support` is enabled + +* If there is an existing cache object to be revalidated, set + ``bereq.http.If-Modified-Since`` from its ``Last-Modified`` header + and/or set ``bereq.http.If-None-Match`` from its ``Etag`` header + +* Set ``bereq.http.X-Varnish`` to the current transaction id (`vxid`) + +These changes can be undone or modified in `vcl_backend_fetch` before +the backend request is issued. + +In particular, to cache non-GET requests, ``req.method`` needs to be +saved to a header or variable in :ref:`vcl_recv` and restored to +``bereq.method``. Notice that caching non-GET requests typically also +requires changing the cache key in :ref:`vcl_hash` e.g. by also +hashing the request method and/or request body. + +HEAD request can be satisfied from cached GET responses. + .. _vcl_backend_response: vcl_backend_response From nils.goroll at uplex.de Tue Oct 4 18:06:05 2016 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 04 Oct 2016 20:06:05 +0200 Subject: [master] f134293 fgs thinks docs should not use (pseudo) VCL to describe what is happening in core Message-ID: commit f13429359ab2821ec458afc2722fffc00de7ca8c Author: Nils Goroll Date: Tue Oct 4 20:04:52 2016 +0200 fgs thinks docs should not use (pseudo) VCL to describe what is happening in core diff --git a/doc/sphinx/users-guide/vcl-built-in-subs.rst b/doc/sphinx/users-guide/vcl-built-in-subs.rst index d5be45d..e69a441 100644 --- a/doc/sphinx/users-guide/vcl-built-in-subs.rst +++ b/doc/sphinx/users-guide/vcl-built-in-subs.rst @@ -274,10 +274,10 @@ The `vcl_backend_fetch` subroutine may terminate with calling Before calling `vcl_backend_fetch`, varnish core prepares the `bereq` backend request as follows: -* ``set bereq.method = "GET"`` and ``set bereq.proto = "HTTP/1.1"``, +* Set ``bereq.method`` to ``GET`` and ``bereq.proto`` to ``HTTP/1.1``, unless the request is a `pass` -* ``set bereq.http.Accept_Encoding = "gzip"`` if +* Set ``bereq.http.Accept_Encoding`` to ``gzip`` if :ref:`ref_param_http_gzip_support` is enabled * If there is an existing cache object to be revalidated, set From nils.goroll at uplex.de Tue Oct 4 18:10:05 2016 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 04 Oct 2016 20:10:05 +0200 Subject: [master] b698aec Also bereq.http.Accept_Encoding only gets set for passes Message-ID: commit b698aec2f65dacbe7a069ffd3313aabc4ba3563d Author: Nils Goroll Date: Tue Oct 4 20:08:27 2016 +0200 Also bereq.http.Accept_Encoding only gets set for passes diff --git a/doc/sphinx/users-guide/vcl-built-in-subs.rst b/doc/sphinx/users-guide/vcl-built-in-subs.rst index e69a441..0128233 100644 --- a/doc/sphinx/users-guide/vcl-built-in-subs.rst +++ b/doc/sphinx/users-guide/vcl-built-in-subs.rst @@ -274,11 +274,13 @@ The `vcl_backend_fetch` subroutine may terminate with calling Before calling `vcl_backend_fetch`, varnish core prepares the `bereq` backend request as follows: -* Set ``bereq.method`` to ``GET`` and ``bereq.proto`` to ``HTTP/1.1``, - unless the request is a `pass` +* Unless the request is a `pass`, -* Set ``bereq.http.Accept_Encoding`` to ``gzip`` if - :ref:`ref_param_http_gzip_support` is enabled + * set ``bereq.method`` to ``GET`` and ``bereq.proto`` to + ``HTTP/1.1`` and + + * set ``bereq.http.Accept_Encoding`` to ``gzip`` if + :ref:`ref_param_http_gzip_support` is enabled. * If there is an existing cache object to be revalidated, set ``bereq.http.If-Modified-Since`` from its ``Last-Modified`` header From dridi.boukelmoune at gmail.com Wed Oct 5 15:26:05 2016 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Wed, 05 Oct 2016 17:26:05 +0200 Subject: [master] 6c49da8 Bust DIST_SUBDIRS since there is no conditionals Message-ID: commit 6c49da843c009a98208820a4c8bec49f70ed5b8d Author: Dridi Boukelmoune Date: Wed Oct 5 17:21:25 2016 +0200 Bust DIST_SUBDIRS since there is no conditionals diff --git a/lib/Makefile.am b/lib/Makefile.am index 2fd8999..579cebd 100644 --- a/lib/Makefile.am +++ b/lib/Makefile.am @@ -9,13 +9,3 @@ SUBDIRS = \ libvmod_debug \ libvmod_std \ libvmod_directors - -DIST_SUBDIRS = \ - libvarnishcompat \ - libvarnish \ - libvarnishapi \ - libvcc \ - libvgz \ - libvmod_debug \ - libvmod_std \ - libvmod_directors From fgsch at lodoss.net Thu Oct 6 21:13:05 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Thu, 06 Oct 2016 23:13:05 +0200 Subject: [master] 119d128 Log the correct attribute name on error Message-ID: commit 119d12890f8b72c3ff3fbcd226ec33f5477ab267 Author: Federico G. Schwindt Date: Thu Oct 6 22:11:46 2016 +0100 Log the correct attribute name on error diff --git a/bin/varnishd/cache/cache_vrt_var.c b/bin/varnishd/cache/cache_vrt_var.c index 0667bde..d7e3be4 100644 --- a/bin/varnishd/cache/cache_vrt_var.c +++ b/bin/varnishd/cache/cache_vrt_var.c @@ -342,7 +342,7 @@ VRT_l_beresp_storage_hint(VRT_CTX, const char *str, ...) b = VRT_String(ctx->bo->ws, NULL, str, ap); // XXX: ctx->ws ? va_end(ap); if (b == NULL) { - VSLb(ctx->vsl, SLT_LostHeader, "storage.hint"); + VSLb(ctx->vsl, SLT_LostHeader, "storage_hint"); WS_MarkOverflow(ctx->bo->beresp->ws); return; } From phk at FreeBSD.org Mon Oct 10 10:11:04 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 10 Oct 2016 12:11:04 +0200 Subject: [master] 2b1b002 This test is quite sensitive to DNS trouble, so extend client impatience to make -t argument controlling Message-ID: commit 2b1b0022b529920393d61b65855302a9984c0814 Author: Poul-Henning Kamp Date: Mon Oct 10 10:10:12 2016 +0000 This test is quite sensitive to DNS trouble, so extend client impatience to make -t argument controlling diff --git a/bin/varnishtest/tests/m00011.vtc b/bin/varnishtest/tests/m00011.vtc index 1c51a07..acf3912 100644 --- a/bin/varnishtest/tests/m00011.vtc +++ b/bin/varnishtest/tests/m00011.vtc @@ -19,6 +19,8 @@ varnish v1 -vcl+backend { } -start client c1 { + # Allow for DNS trouble with bogos domains + timeout 60 txreq rxresp expect resp.http.foo0 == "${localhost}" From phk at FreeBSD.org Mon Oct 10 13:57:04 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 10 Oct 2016 15:57:04 +0200 Subject: [master] efa265b Make this test less sensitive to varnishd running in the background Message-ID: commit efa265b06600c8badbd62e3926248716c4ef0816 Author: Poul-Henning Kamp Date: Mon Oct 10 13:55:51 2016 +0000 Make this test less sensitive to varnishd running in the background diff --git a/bin/varnishtest/tests/u00002.vtc b/bin/varnishtest/tests/u00002.vtc index 9f083ec..662c124 100644 --- a/bin/varnishtest/tests/u00002.vtc +++ b/bin/varnishtest/tests/u00002.vtc @@ -10,14 +10,14 @@ varnish v1 -vcl+backend {} -start delay 1 process p1 {varnishstat -1 -n ${v1_name} -f ^LCK.vbe.destroy \ - -f LCK.vbe.* -f LCK.mempool.*} -run + -f ^*.locks -f LCK.vbe.* -f LCK.mempool.*} -run shell "grep -q vbe ${tmpdir}/p1/stdout" shell "grep -q mempool ${tmpdir}/p1/stdout" err_shell "" "grep -q LCK.vbe.destroy ${tmpdir}/p1/stdout" process p2 {varnishstat -1 -n ${v1_name} -f ^*vbe.destroy \ - -f *vbe* -f *mempool*} -run + -f ^*.locks -f *vbe* -f *mempool*} -run shell "grep -q vbe ${tmpdir}/p2/stdout" shell "grep -q mempool ${tmpdir}/p2/stdout" From phk at FreeBSD.org Mon Oct 10 16:21:04 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 10 Oct 2016 18:21:04 +0200 Subject: [master] 96bb4f7 Move clearing of cli->vsb up to also cover centralized error checks Message-ID: commit 96bb4f7f604a141a6812e1d856f4ee05c7b78fee Author: Poul-Henning Kamp Date: Mon Oct 10 16:19:48 2016 +0000 Move clearing of cli->vsb up to also cover centralized error checks diff --git a/lib/libvarnish/vcli_serve.c b/lib/libvarnish/vcli_serve.c index 80d09de..119b5f1 100644 --- a/lib/libvarnish/vcli_serve.c +++ b/lib/libvarnish/vcli_serve.c @@ -194,6 +194,8 @@ cls_dispatch(struct cli *cli, const struct cli_proto *cp, AN(av); + VSB_clear(cli->sb); + if (ac > 1 && !strcmp(av[2], "-j")) json = 1; @@ -221,7 +223,6 @@ cls_dispatch(struct cli *cli, const struct cli_proto *cp, } cli->result = CLIS_OK; - VSB_clear(cli->sb); if (json) cp->jsonfunc(cli, (const char * const *)av, cp->priv); else From phk at FreeBSD.org Mon Oct 10 16:41:04 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 10 Oct 2016 18:41:04 +0200 Subject: [master] 0124731 Dont eat newlines in VSB_QUOTE_CSTR mode Message-ID: commit 01247317a3aa09967d243d47ef3a7a4511ec4669 Author: Poul-Henning Kamp Date: Mon Oct 10 16:39:43 2016 +0000 Dont eat newlines in VSB_QUOTE_CSTR mode Spotted by: gquintar diff --git a/lib/libvarnish/vsb.c b/lib/libvarnish/vsb.c index 6a6f169..23fa142 100644 --- a/lib/libvarnish/vsb.c +++ b/lib/libvarnish/vsb.c @@ -555,7 +555,7 @@ VSB_quote(struct vsb *s, const void *v, int len, int how) break; case '\n': if (how & VSB_QUOTE_CSTR) - (void)VSB_cat(s, "\"\n\t\""); + (void)VSB_cat(s, "\\n\"\n\t\""); else if (how & VSB_QUOTE_NONL) (void)VSB_cat(s, "\n"); else From phk at FreeBSD.org Tue Oct 11 09:14:05 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 11 Oct 2016 11:14:05 +0200 Subject: [master] f52d5a2 Refactor return(bla) code slightly Message-ID: commit f52d5a2214c91443dbe6a6ad3e087f12a95dc477 Author: Poul-Henning Kamp Date: Tue Oct 11 08:31:52 2016 +0000 Refactor return(bla) code slightly diff --git a/bin/varnishtest/tests/v00018.vtc b/bin/varnishtest/tests/v00018.vtc index 7fe29e3..53d96c9 100644 --- a/bin/varnishtest/tests/v00018.vtc +++ b/bin/varnishtest/tests/v00018.vtc @@ -103,7 +103,17 @@ varnish v1 -errvcl {Unknown token '<<' when looking for STRING_LIST} { sub vcl_synth { synthetic( << "foo"; } } -varnish v1 -errvcl {Syntax has changed, use:} { +varnish v1 -errvcl {Missing argument.} { backend b { .host = "127.0.0.1"; } sub vcl_recv { return(synth); } } + +varnish v1 -errvcl {Arguments not allowed.} { + backend b { .host = "127.0.0.1"; } + sub vcl_recv { return(pipe(XXX); } +} + +varnish v1 -errvcl {Expected return action name.} { + backend b { .host = "127.0.0.1"; } + sub vcl_recv { return(foobar); } +} diff --git a/lib/libvcc/vcc_action.c b/lib/libvcc/vcc_action.c index 200ba97..ced924c 100644 --- a/lib/libvcc/vcc_action.c +++ b/lib/libvcc/vcc_action.c @@ -195,17 +195,6 @@ static void parse_return_synth(struct vcc *tl) { - AN(vcc_IdIs(tl->t, "synth")); - vcc_NextToken(tl); - if (tl->t->tok == ')') { - VSB_printf(tl->sb, - "Syntax has changed, use:\n" - "\treturn(synth(999));\n" - "or\n" - "\treturn(synth(999, \"Response text\"));\n"); - vcc_ErrWhere(tl, tl->t); - return; - } ExpectErr(tl, '('); vcc_NextToken(tl); Fb(tl, 1, "VRT_synth(ctx,\n"); @@ -224,11 +213,6 @@ parse_return_synth(struct vcc *tl) ExpectErr(tl, ')'); vcc_NextToken(tl); Fb(tl, 1, ");\n"); - Fb(tl, 1, "VRT_handling(ctx, VCL_RET_SYNTH);\n"); - Fb(tl, 1, "return (1);\n"); - vcc_ProcAction(tl->curproc, VCL_RET_SYNTH, tl->t); - ExpectErr(tl, ')'); - vcc_NextToken(tl); } /*--------------------------------------------------------------------*/ @@ -240,8 +224,6 @@ parse_return_vcl(struct vcc *tl) struct inifin *p; char buf[1024]; - AN(vcc_IdIs(tl->t, "vcl")); - vcc_NextToken(tl); ExpectErr(tl, '('); vcc_NextToken(tl); ExpectErr(tl, ID); @@ -270,10 +252,6 @@ parse_return_vcl(struct vcc *tl) } Fb(tl, 1, "VRT_vcl_select(ctx, %s);\t/* %.*s */\n", (const char*)sym->eval_priv, PF(tl->t)); - Fb(tl, 1, "VRT_handling(ctx, VCL_RET_VCL);\n"); - Fb(tl, 1, "return (1);\n"); - vcc_NextToken(tl); - ExpectErr(tl, ')'); vcc_NextToken(tl); ExpectErr(tl, ')'); vcc_NextToken(tl); @@ -284,39 +262,50 @@ parse_return_vcl(struct vcc *tl) static void parse_return(struct vcc *tl) { - int retval = 0; + unsigned hand; + const char *h; vcc_NextToken(tl); ExpectErr(tl, '('); vcc_NextToken(tl); ExpectErr(tl, ID); - if (vcc_IdIs(tl->t, "synth")) { - parse_return_synth(tl); - return; - } - if (vcc_IdIs(tl->t, "vcl")) { - parse_return_vcl(tl); - return; - } - -#define VCL_RET_MAC(l, U, B) \ - do { \ - if (vcc_IdIs(tl->t, #l)) { \ - Fb(tl, 1, "VRT_handling(ctx, VCL_RET_" #U ");\n"); \ - Fb(tl, 1, "return (1);\n"); \ - vcc_ProcAction(tl->curproc, VCL_RET_##U, tl->t);\ - retval = 1; \ - } \ - } while (0); + hand = VCL_RET_MAX; + h = NULL; +#define VCL_RET_MAC(l, U, B) \ + if (vcc_IdIs(tl->t, #l)) { \ + hand = VCL_RET_ ## U; \ + h = #U; \ + } #include "tbl/vcl_returns.h" #undef VCL_RET_MAC - if (!retval) { + if (h == NULL) { VSB_printf(tl->sb, "Expected return action name.\n"); vcc_ErrWhere(tl, tl->t); ERRCHK(tl); } + assert(hand < VCL_RET_MAX); + + vcc_ProcAction(tl->curproc, hand, tl->t); vcc_NextToken(tl); + if (tl->t->tok == '(') { + if (hand == VCL_RET_SYNTH) + parse_return_synth(tl); + else if (hand == VCL_RET_VCL) + parse_return_vcl(tl); + else { + VSB_printf(tl->sb, "Arguments not allowed.\n"); + vcc_ErrWhere(tl, tl->t); + } + } else { + if (hand == VCL_RET_SYNTH || hand == VCL_RET_VCL) { + VSB_printf(tl->sb, "Missing argument.\n"); + vcc_ErrWhere(tl, tl->t); + } + } + ERRCHK(tl); + Fb(tl, 1, "VRT_handling(ctx, VCL_RET_%s);\n", h); + Fb(tl, 1, "return (1);\n"); ExpectErr(tl, ')'); vcc_NextToken(tl); } From phk at FreeBSD.org Tue Oct 11 09:14:05 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 11 Oct 2016 11:14:05 +0200 Subject: [master] 4fdcdca Another attempt to make this test more reliable: zero all numbers Message-ID: commit 4fdcdcaf295dce1e9e08b2094fc381793e28d5ff Author: Poul-Henning Kamp Date: Tue Oct 11 08:43:16 2016 +0000 Another attempt to make this test more reliable: zero all numbers diff --git a/bin/varnishtest/tests/u00002.vtc b/bin/varnishtest/tests/u00002.vtc index 662c124..d6aa4ef 100644 --- a/bin/varnishtest/tests/u00002.vtc +++ b/bin/varnishtest/tests/u00002.vtc @@ -10,14 +10,14 @@ varnish v1 -vcl+backend {} -start delay 1 process p1 {varnishstat -1 -n ${v1_name} -f ^LCK.vbe.destroy \ - -f ^*.locks -f LCK.vbe.* -f LCK.mempool.*} -run + -f LCK.vbe.* -f LCK.mempool.* | tr '[1-9]' '0'} -run shell "grep -q vbe ${tmpdir}/p1/stdout" shell "grep -q mempool ${tmpdir}/p1/stdout" err_shell "" "grep -q LCK.vbe.destroy ${tmpdir}/p1/stdout" process p2 {varnishstat -1 -n ${v1_name} -f ^*vbe.destroy \ - -f ^*.locks -f *vbe* -f *mempool*} -run + -f *vbe* -f *mempool* | tr '[1-9]' '0'} -run shell "grep -q vbe ${tmpdir}/p2/stdout" shell "grep -q mempool ${tmpdir}/p2/stdout" From hermunn at varnish-software.com Tue Oct 11 15:15:05 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Tue, 11 Oct 2016 17:15:05 +0200 Subject: [4.1] d3df37f Document %r behaviour Message-ID: commit d3df37f2c00a0f3f1bc645edee4af99286999730 Author: Federico G. Schwindt Date: Tue Sep 27 09:44:56 2016 +0100 Document %r behaviour Fixes #2017. diff --git a/doc/sphinx/reference/varnishncsa.rst b/doc/sphinx/reference/varnishncsa.rst index 94730a8..55c38a8 100644 --- a/doc/sphinx/reference/varnishncsa.rst +++ b/doc/sphinx/reference/varnishncsa.rst @@ -118,7 +118,7 @@ Supported formatters are: %r The first line of the request. Synthesized from other fields, so it - may not be the request verbatim. + may not be the request verbatim. See the NOTES section. %s Status sent to the client. In backend mode, status received from @@ -187,6 +187,12 @@ SIGHUP SIGUSR1 Flush any outstanding transactions +NOTES +===== + +The %r formatter is equivalent to "%m http://%{Host}i%U%q %H". This +differs from apache's %r behavior. For the latter use "%m %U%q %H". + EXAMPLE ======= From hermunn at varnish-software.com Tue Oct 11 15:17:05 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Tue, 11 Oct 2016 17:17:05 +0200 Subject: [4.1] fd95845 Fix test for duplicate links in vsl_dispatch Message-ID: commit fd958451b81a8edc092e538ca947b1fea04cc81a Author: Martin Blix Grydeland Date: Fri Sep 30 15:21:13 2016 +0200 Fix test for duplicate links in vsl_dispatch This test wasn't working as it was supposed to, and would cause the forward links graph building of log transactions to fail. The fail was soft, as the reverse mapping still succeeded creating the right mapping. Depending on when the local thread log buffer was flushed, the problem would show as spurious "duplicate link" log records to be inserted. Fixes: #1830 diff --git a/lib/libvarnishapi/vsl_dispatch.c b/lib/libvarnishapi/vsl_dispatch.c index bbcd7cc..7b733ad 100644 --- a/lib/libvarnishapi/vsl_dispatch.c +++ b/lib/libvarnishapi/vsl_dispatch.c @@ -834,7 +834,7 @@ vtx_scan_link(struct VSLQ *vslq, struct vtx *vtx, const uint32_t *ptr) if (c_vtx->parent == vtx) /* Link already exists */ return (0); - if (c_vtx->parent != vtx) + if (c_vtx->parent != NULL && c_vtx->parent != vtx) return (vtx_diag_tag(vtx, ptr, "duplicate link")); if (c_vtx->flags & VTX_F_COMPLETE) return (vtx_diag_tag(vtx, ptr, "link too late")); From phk at FreeBSD.org Tue Oct 11 20:31:05 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 11 Oct 2016 22:31:05 +0200 Subject: [master] 5486482 Polish Message-ID: commit 54864828605461dd3c2a153af0830daba669e4e7 Author: Poul-Henning Kamp Date: Tue Oct 11 20:28:44 2016 +0000 Polish diff --git a/lib/libvcc/vcc_compile.c b/lib/libvcc/vcc_compile.c index 32f245a..b2b6cab 100644 --- a/lib/libvcc/vcc_compile.c +++ b/lib/libvcc/vcc_compile.c @@ -371,8 +371,10 @@ EmitInitFini(const struct vcc *tl) VTAILQ_FOREACH_REVERSE(p, &tl->inifin, inifinhead, list) { if (VSB_len(p->event)) { Fc(tl, 0, "\t/* %u */\n", p->n); - Fc(tl, 0, "\tif (vgc_warmupstep >= %u &&\n", p->n); - Fc(tl, 0, "\t %s != 0)\n", VSB_data(p->event)); + Fc(tl, 0, + "\tif (vgc_warmupstep >= %u &&\n", p->n); + Fc(tl, 0, + "\t %s != 0)\n", VSB_data(p->event)); Fc(tl, 0, "\t\tretval = 1;\n\n"); } VSB_destroy(&p->event); diff --git a/lib/libvcc/vcc_parse.c b/lib/libvcc/vcc_parse.c index 64ac0d9..53bc29e 100644 --- a/lib/libvcc/vcc_parse.c +++ b/lib/libvcc/vcc_parse.c @@ -331,6 +331,7 @@ vcc_Parse(struct vcc *tl) struct toplev *tp; struct token *tok; + AZ(tl->indent); if (tl->t->tok != ID || !vcc_IdIs(tl->t, "vcl")) { VSB_printf(tl->sb, "VCL version declaration missing\n" @@ -395,4 +396,5 @@ vcc_Parse(struct vcc *tl) return; } } + AZ(tl->indent); } From phk at FreeBSD.org Tue Oct 11 20:31:05 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 11 Oct 2016 22:31:05 +0200 Subject: [master] 97a2a9e Lay down some work for HFP mkII Message-ID: commit 97a2a9efef26a9006751ad12e33c9b54057d1165 Author: Poul-Henning Kamp Date: Tue Oct 11 20:30:03 2016 +0000 Lay down some work for HFP mkII diff --git a/include/vrt.h b/include/vrt.h index 901e13f..8aaa551 100644 --- a/include/vrt.h +++ b/include/vrt.h @@ -287,6 +287,7 @@ void VRT_purge(VRT_CTX, double ttl, double grace, double keep); void VRT_count(VRT_CTX, unsigned); void VRT_synth(VRT_CTX, unsigned, const char *); +void VRT_hit_for_pass(VRT_CTX, VCL_DURATION); struct http *VRT_selecthttp(VRT_CTX, enum gethdr_e); const char *VRT_GetHdr(VRT_CTX, const struct gethdr_s *); diff --git a/lib/libvcc/vcc_action.c b/lib/libvcc/vcc_action.c index ced924c..60525a7 100644 --- a/lib/libvcc/vcc_action.c +++ b/lib/libvcc/vcc_action.c @@ -192,6 +192,23 @@ parse_hash_data(struct vcc *tl) /*--------------------------------------------------------------------*/ static void +parse_return_pass(struct vcc *tl) +{ + + ExpectErr(tl, '('); + vcc_NextToken(tl); + Fb(tl, 1, "VRT_hit_for_pass(ctx,\n"); + tl->indent += INDENT; + vcc_Expr(tl, DURATION); + ERRCHK(tl); + ExpectErr(tl, ')'); + vcc_NextToken(tl); + Fb(tl, 1, ");\n"); + tl->indent -= INDENT; +} +/*--------------------------------------------------------------------*/ + +static void parse_return_synth(struct vcc *tl) { @@ -272,7 +289,7 @@ parse_return(struct vcc *tl) hand = VCL_RET_MAX; h = NULL; -#define VCL_RET_MAC(l, U, B) \ +#define VCL_RET_MAC(l, U, B) \ if (vcc_IdIs(tl->t, #l)) { \ hand = VCL_RET_ ## U; \ h = #U; \ @@ -285,7 +302,7 @@ parse_return(struct vcc *tl) ERRCHK(tl); } assert(hand < VCL_RET_MAX); - + vcc_ProcAction(tl->curproc, hand, tl->t); vcc_NextToken(tl); if (tl->t->tok == '(') { @@ -293,6 +310,8 @@ parse_return(struct vcc *tl) parse_return_synth(tl); else if (hand == VCL_RET_VCL) parse_return_vcl(tl); + else if (hand == VCL_RET_PASS) + parse_return_pass(tl); else { VSB_printf(tl->sb, "Arguments not allowed.\n"); vcc_ErrWhere(tl, tl->t); From hermunn at varnish-software.com Wed Oct 12 08:37:04 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Wed, 12 Oct 2016 10:37:04 +0200 Subject: [master] 0f291ee Add error checking to confirm bug Message-ID: commit 0f291ee97dd6afe4e9c5500d4db15cc39c86c7c9 Author: P?l Hermunn Johansen Date: Wed Sep 28 14:54:29 2016 +0200 Add error checking to confirm bug There seems to be an error in the varnish jail design, which makes the mgt process uable to kill the child process. To confirm this, add some error checking to the relevant code. Related to: #2010 diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c index 58588db..e486d55 100644 --- a/bin/varnishd/mgt/mgt_child.c +++ b/bin/varnishd/mgt/mgt_child.c @@ -34,6 +34,7 @@ #include #include +#include #include #include #include @@ -583,17 +584,22 @@ mgt_reap_child(void) void MGT_Child_Cli_Fail(void) { + int i; if (child_state != CH_RUNNING) return; if (child_pid < 0) return; - MGT_complain(C_ERR, "Child (%jd) not responding to CLI, killing it.", - (intmax_t)child_pid); if (MGT_FEATURE(FEATURE_NO_COREDUMP)) - (void)kill(child_pid, SIGKILL); + i = kill(child_pid, SIGKILL); + else + i = kill(child_pid, SIGQUIT); + if (i == 0) + MGT_complain(C_ERR, "Child (%jd) not responding to CLI," + " killing it.", (intmax_t)child_pid); else - (void)kill(child_pid, SIGQUIT); + MGT_complain(C_ERR, "Failed to kill child with PID %jd: %s", + (intmax_t)child_pid, strerror(errno)); } /*===================================================================== From hermunn at varnish-software.com Wed Oct 12 08:37:04 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Wed, 12 Oct 2016 10:37:04 +0200 Subject: [master] c189a22 Move killing the child process into separate function Message-ID: commit c189a22703fc61d1cd56dba2ee745dc5aed1bb52 Author: P?l Hermunn Johansen Date: Tue Oct 4 11:01:14 2016 +0200 Move killing the child process into separate function A new function, kill_child() is introduced, so that kill() is only called from this function. This is a preparation of fixing the jail implementation. diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c index e486d55..a2c7fb1 100644 --- a/bin/varnishd/mgt/mgt_child.c +++ b/bin/varnishd/mgt/mgt_child.c @@ -453,6 +453,14 @@ mgt_launch_child(struct cli *cli) * Cleanup when child dies. */ +static int +kill_child(void) { + if (MGT_FEATURE(FEATURE_NO_COREDUMP)) + return (kill(child_pid, SIGKILL)); + else + return (kill(child_pid, SIGQUIT)); +} + static void mgt_reap_child(void) { @@ -502,10 +510,7 @@ mgt_reap_child(void) VSB_printf(vsb, "Child (%jd) not dying, killing", (intmax_t)r); /* Kick it Jim... */ - if (MGT_FEATURE(FEATURE_NO_COREDUMP)) - (void)kill(child_pid, SIGKILL); - else - (void)kill(child_pid, SIGQUIT); + (void)kill_child(); r = waitpid(child_pid, &status, 0); } if (r != child_pid) @@ -584,17 +589,12 @@ mgt_reap_child(void) void MGT_Child_Cli_Fail(void) { - int i; if (child_state != CH_RUNNING) return; if (child_pid < 0) return; - if (MGT_FEATURE(FEATURE_NO_COREDUMP)) - i = kill(child_pid, SIGKILL); - else - i = kill(child_pid, SIGQUIT); - if (i == 0) + if (kill_child() == 0) MGT_complain(C_ERR, "Child (%jd) not responding to CLI," " killing it.", (intmax_t)child_pid); else From hermunn at varnish-software.com Wed Oct 12 08:37:04 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Wed, 12 Oct 2016 10:37:04 +0200 Subject: [master] c41c30a New jail level - kill - used when killing is happening Message-ID: commit c41c30aa7f7dfba228af12d99987dac479494910 Author: P?l Hermunn Johansen Date: Tue Oct 11 13:42:55 2016 +0200 New jail level - kill - used when killing is happening A new jail level, JAIL_MASTER_KILL, is introduced. The mgt process takes this level before killing the child process. Fixes: #2010 diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h index 590372d..a4d835a 100644 --- a/bin/varnishd/mgt/mgt.h +++ b/bin/varnishd/mgt/mgt.h @@ -88,6 +88,7 @@ enum jail_master_e { JAIL_MASTER_FILE, JAIL_MASTER_STORAGE, JAIL_MASTER_PRIVPORT, + JAIL_MASTER_KILL, }; typedef int jail_init_f(char **); diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c index a2c7fb1..b8c75e7 100644 --- a/bin/varnishd/mgt/mgt_child.c +++ b/bin/varnishd/mgt/mgt_child.c @@ -455,10 +455,17 @@ mgt_launch_child(struct cli *cli) static int kill_child(void) { + int i, error; + + VJ_master(JAIL_MASTER_KILL); if (MGT_FEATURE(FEATURE_NO_COREDUMP)) - return (kill(child_pid, SIGKILL)); + i = kill(child_pid, SIGKILL); else - return (kill(child_pid, SIGQUIT)); + i = kill(child_pid, SIGQUIT); + error = errno; + VJ_master(JAIL_MASTER_LOW); + errno = error; + return (i); } static void From hermunn at varnish-software.com Wed Oct 12 08:37:04 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Wed, 12 Oct 2016 10:37:04 +0200 Subject: [master] df4ec11 Updated error message in MGT_Child_Cli_Fail Message-ID: commit df4ec11dc46ad4d0c3c147b3504e9980b2f4f2bf Author: P?l Hermunn Johansen Date: Tue Oct 11 15:37:38 2016 +0200 Updated error message in MGT_Child_Cli_Fail Updated error message in MGT_Child_Cli_Fail to indicate that we actually have sent the kill signal to the child process. diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c index b8c75e7..9264f65 100644 --- a/bin/varnishd/mgt/mgt_child.c +++ b/bin/varnishd/mgt/mgt_child.c @@ -603,7 +603,7 @@ MGT_Child_Cli_Fail(void) return; if (kill_child() == 0) MGT_complain(C_ERR, "Child (%jd) not responding to CLI," - " killing it.", (intmax_t)child_pid); + " killed it.", (intmax_t)child_pid); else MGT_complain(C_ERR, "Failed to kill child with PID %jd: %s", (intmax_t)child_pid, strerror(errno)); From fgsch at lodoss.net Wed Oct 12 13:57:04 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Wed, 12 Oct 2016 15:57:04 +0200 Subject: [master] 0bb94ca Some style(9) Message-ID: commit 0bb94ca79e347cbd4ef577984edc97581a33576c Author: Federico G. Schwindt Date: Wed Oct 12 09:07:36 2016 +0100 Some style(9) diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c index 9264f65..8bba5b7 100644 --- a/bin/varnishd/mgt/mgt_child.c +++ b/bin/varnishd/mgt/mgt_child.c @@ -454,7 +454,8 @@ mgt_launch_child(struct cli *cli) */ static int -kill_child(void) { +kill_child(void) +{ int i, error; VJ_master(JAIL_MASTER_KILL); diff --git a/bin/varnishd/mgt/mgt_cli.c b/bin/varnishd/mgt/mgt_cli.c index b16597d..84a853a 100644 --- a/bin/varnishd/mgt/mgt_cli.c +++ b/bin/varnishd/mgt/mgt_cli.c @@ -174,7 +174,8 @@ static struct cli_proto cli_askchild[] = { */ int -mgt_cli_askchild(unsigned *status, char **resp, const char *fmt, ...) { +mgt_cli_askchild(unsigned *status, char **resp, const char *fmt, ...) +{ int i, j; va_list ap; unsigned u; diff --git a/bin/varnishd/mgt/mgt_jail_solaris.c b/bin/varnishd/mgt/mgt_jail_solaris.c index 1723e3e..a9a624e 100644 --- a/bin/varnishd/mgt/mgt_jail_solaris.c +++ b/bin/varnishd/mgt/mgt_jail_solaris.c @@ -262,7 +262,8 @@ vjs_init(char **args) /* for priv_delset() and priv_addset() */ static inline int -priv_setop_check(int a) { +priv_setop_check(int a) +{ if (a == 0) return (1); if (errno == EINVAL) @@ -280,7 +281,8 @@ priv_setop_check(int a) { /* for setppriv */ static inline int -setppriv_check(int a) { +setppriv_check(int a) +{ if (a == 0) return (1); if (errno == EPERM) diff --git a/bin/varnishd/waiter/cache_waiter_ports.c b/bin/varnishd/waiter/cache_waiter_ports.c index 3b4f59c..f208f8a 100644 --- a/bin/varnishd/waiter/cache_waiter_ports.c +++ b/bin/varnishd/waiter/cache_waiter_ports.c @@ -107,7 +107,8 @@ vws_del(struct vws *vws, int fd) } static inline void -vws_port_ev(struct vws *vws, struct waiter *w, port_event_t *ev, double now) { +vws_port_ev(struct vws *vws, struct waiter *w, port_event_t *ev, double now) +{ struct waited *wp; if(ev->portev_source == PORT_SOURCE_USER) { CAST_OBJ_NOTNULL(wp, ev->portev_user, WAITED_MAGIC); diff --git a/bin/varnishtest/vtc_http2.c b/bin/varnishtest/vtc_http2.c index b4cb96f..f1b5360 100644 --- a/bin/varnishtest/vtc_http2.c +++ b/bin/varnishtest/vtc_http2.c @@ -559,7 +559,8 @@ parse_prio(struct stream *s, struct frame *f) } static void -parse_rst(const struct stream *s, struct frame *f) { +parse_rst(const struct stream *s, struct frame *f) +{ struct http *hp; uint32_t err; const char *buf; @@ -2059,7 +2060,8 @@ cmd_txwinup(CMD_ARGS) } static struct frame * -rxstuff(struct stream *s) { +rxstuff(struct stream *s) +{ struct frame *f; CHECK_OBJ_NOTNULL(s, STREAM_MAGIC); @@ -2284,7 +2286,8 @@ cmd_rxreqsp(CMD_ARGS) * */ static void -cmd_rxpush(CMD_ARGS) { +cmd_rxpush(CMD_ARGS) +{ struct stream *s; struct frame *f = NULL; char *p; @@ -2374,7 +2377,8 @@ RXFUNC(winup, WINDOW_UPDATE) * Receive a frame, any frame. */ static void -cmd_rxframe(CMD_ARGS) { +cmd_rxframe(CMD_ARGS) +{ struct stream *s; (void)cmd; (void)vl; diff --git a/include/vbm.h b/include/vbm.h index 94497ab..f56447d 100644 --- a/include/vbm.h +++ b/include/vbm.h @@ -44,7 +44,8 @@ #define VBITMAP_BIT(n) (1U << ((n) % VBITMAP_WORD)) static inline unsigned -vbit_rndup(unsigned bit, unsigned to) { +vbit_rndup(unsigned bit, unsigned to) +{ bit += to - 1; bit -= (bit % to); diff --git a/lib/libvmod_directors/vmod_shard.c b/lib/libvmod_directors/vmod_shard.c index 52c9710..031e364 100644 --- a/lib/libvmod_directors/vmod_shard.c +++ b/lib/libvmod_directors/vmod_shard.c @@ -104,7 +104,8 @@ VCL_INT __match_proto__(td_directors_shard_key) VCL_VOID __match_proto__(td_directors_set_warmup) vmod_shard_set_warmup(VRT_CTX, struct vmod_directors_shard *vshard, - VCL_REAL probability) { + VCL_REAL probability) +{ CHECK_OBJ_NOTNULL(vshard, VMOD_SHARD_SHARD_MAGIC); if (probability < 0 || probability >= 1) { shard_err(ctx, vshard->shardd, From hermunn at varnish-software.com Wed Oct 12 14:12:05 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Wed, 12 Oct 2016 16:12:05 +0200 Subject: [4.1] c6efe04 Add error checking to confirm bug Message-ID: commit c6efe045ff12c1f192d15caf5683f65b0aca34e2 Author: P?l Hermunn Johansen Date: Wed Sep 28 14:54:29 2016 +0200 Add error checking to confirm bug There seems to be an error in the varnish jail design, which makes the mgt process uable to kill the child process. To confirm this, add some error checking to the relevant code. Related to: #2010 diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c index 2fa0795..f89f6d4 100644 --- a/bin/varnishd/mgt/mgt_child.c +++ b/bin/varnishd/mgt/mgt_child.c @@ -34,6 +34,7 @@ #include #include +#include #include #include #include @@ -587,17 +588,22 @@ mgt_reap_child(void) void MGT_Child_Cli_Fail(void) { + int i; if (child_state != CH_RUNNING) return; if (child_pid < 0) return; - MGT_complain(C_ERR, "Child (%jd) not responding to CLI, killing it.", - (intmax_t)child_pid); if (MGT_FEATURE(FEATURE_NO_COREDUMP)) - (void)kill(child_pid, SIGKILL); + i = kill(child_pid, SIGKILL); + else + i = kill(child_pid, SIGQUIT); + if (i == 0) + MGT_complain(C_ERR, "Child (%jd) not responding to CLI," + " killing it.", (intmax_t)child_pid); else - (void)kill(child_pid, SIGQUIT); + MGT_complain(C_ERR, "Failed to kill child with PID %jd: %s", + (intmax_t)child_pid, strerror(errno)); } /*===================================================================== From hermunn at varnish-software.com Wed Oct 12 14:12:05 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Wed, 12 Oct 2016 16:12:05 +0200 Subject: [4.1] f9526a5 Move killing the child process into separate function Message-ID: commit f9526a52c5cbbd3bdd3e524d2f93949da4c411b4 Author: P?l Hermunn Johansen Date: Tue Oct 4 11:01:14 2016 +0200 Move killing the child process into separate function A new function, kill_child() is introduced, so that kill() is only called from this function. This is a preparation of fixing the jail implementation. diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c index f89f6d4..8acde5e 100644 --- a/bin/varnishd/mgt/mgt_child.c +++ b/bin/varnishd/mgt/mgt_child.c @@ -457,6 +457,14 @@ mgt_launch_child(struct cli *cli) * Cleanup when child dies. */ +static int +kill_child(void) { + if (MGT_FEATURE(FEATURE_NO_COREDUMP)) + return (kill(child_pid, SIGKILL)); + else + return (kill(child_pid, SIGQUIT)); +} + static void mgt_reap_child(void) { @@ -506,10 +514,7 @@ mgt_reap_child(void) VSB_printf(vsb, "Child (%jd) not dying, killing", (intmax_t)r); /* Kick it Jim... */ - if (MGT_FEATURE(FEATURE_NO_COREDUMP)) - (void)kill(child_pid, SIGKILL); - else - (void)kill(child_pid, SIGQUIT); + (void)kill_child(); r = waitpid(child_pid, &status, 0); } if (r != child_pid) @@ -588,17 +593,12 @@ mgt_reap_child(void) void MGT_Child_Cli_Fail(void) { - int i; if (child_state != CH_RUNNING) return; if (child_pid < 0) return; - if (MGT_FEATURE(FEATURE_NO_COREDUMP)) - i = kill(child_pid, SIGKILL); - else - i = kill(child_pid, SIGQUIT); - if (i == 0) + if (kill_child() == 0) MGT_complain(C_ERR, "Child (%jd) not responding to CLI," " killing it.", (intmax_t)child_pid); else From hermunn at varnish-software.com Wed Oct 12 14:12:05 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Wed, 12 Oct 2016 16:12:05 +0200 Subject: [4.1] 61cdc9d New jail level - kill - used when killing is happening Message-ID: commit 61cdc9d8416a27eb33948e9b838e25a8eb07fa5f Author: P?l Hermunn Johansen Date: Tue Oct 11 13:42:55 2016 +0200 New jail level - kill - used when killing is happening A new jail level, JAIL_MASTER_KILL, is introduced. The mgt process takes this level before killing the child process. Fixes: #2010 diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h index e59ec7f..a850a08 100644 --- a/bin/varnishd/mgt/mgt.h +++ b/bin/varnishd/mgt/mgt.h @@ -82,6 +82,7 @@ enum jail_master_e { JAIL_MASTER_FILE, JAIL_MASTER_STORAGE, JAIL_MASTER_PRIVPORT, + JAIL_MASTER_KILL, }; typedef int jail_init_f(char **); diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c index 8acde5e..d8ab431 100644 --- a/bin/varnishd/mgt/mgt_child.c +++ b/bin/varnishd/mgt/mgt_child.c @@ -459,10 +459,17 @@ mgt_launch_child(struct cli *cli) static int kill_child(void) { + int i, error; + + VJ_master(JAIL_MASTER_KILL); if (MGT_FEATURE(FEATURE_NO_COREDUMP)) - return (kill(child_pid, SIGKILL)); + i = kill(child_pid, SIGKILL); else - return (kill(child_pid, SIGQUIT)); + i = kill(child_pid, SIGQUIT); + error = errno; + VJ_master(JAIL_MASTER_LOW); + errno = error; + return (i); } static void From hermunn at varnish-software.com Wed Oct 12 14:12:05 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Wed, 12 Oct 2016 16:12:05 +0200 Subject: [4.1] 2c4ceea Updated error message in MGT_Child_Cli_Fail Message-ID: commit 2c4ceead04fb14c6bec0d57b6d0aaef876d5ddff Author: P?l Hermunn Johansen Date: Tue Oct 11 15:37:38 2016 +0200 Updated error message in MGT_Child_Cli_Fail Updated error message in MGT_Child_Cli_Fail to indicate that we actually have sent the kill signal to the child process. diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c index d8ab431..1154de5 100644 --- a/bin/varnishd/mgt/mgt_child.c +++ b/bin/varnishd/mgt/mgt_child.c @@ -607,7 +607,7 @@ MGT_Child_Cli_Fail(void) return; if (kill_child() == 0) MGT_complain(C_ERR, "Child (%jd) not responding to CLI," - " killing it.", (intmax_t)child_pid); + " killed it.", (intmax_t)child_pid); else MGT_complain(C_ERR, "Failed to kill child with PID %jd: %s", (intmax_t)child_pid, strerror(errno)); From nils.goroll at uplex.de Thu Oct 13 06:24:05 2016 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 13 Oct 2016 08:24:05 +0200 Subject: [master] 4b0f93f Also use the C locale for re-checking our list of tests Message-ID: commit 4b0f93f09ea32248605357acf48297fb349e95e9 Author: Nils Goroll Date: Thu Oct 13 07:58:16 2016 +0200 Also use the C locale for re-checking our list of tests otherwise, when the tests directory contains vtc files whose names get sorted differently in the current environment's locale than in C, config.status will get re-run for every make check invocation, which is particularly annoying with parallel makes diff --git a/bin/varnishtest/Makefile.am b/bin/varnishtest/Makefile.am index 1c519e1..26563e2 100644 --- a/bin/varnishtest/Makefile.am +++ b/bin/varnishtest/Makefile.am @@ -9,6 +9,7 @@ check: check-am check-local check-am: check-local # See if list of checks have changed, recheck check-local: + LC_ALL=C; \ if [ "$$(cd $(srcdir) && echo tests/*.vtc)" != "@VTC_TESTS@" ]; then \ cd $(top_builddir) && ./config.status --recheck ; \ fi From nils.goroll at uplex.de Thu Oct 13 06:24:05 2016 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 13 Oct 2016 08:24:05 +0200 Subject: [master] 9c89055 Remove duplicate copyright section, remove history Message-ID: commit 9c890553ac7ff1cc7462362687e9a00e286ed012 Author: Nils Goroll Date: Thu Oct 13 08:17:36 2016 +0200 Remove duplicate copyright section, remove history vmodtool already includes the copyright from the boilerplate Authors and contributors are numerous, so we'd either need to name them all or none. diff --git a/lib/libvmod_std/vmod.vcc b/lib/libvmod_std/vmod.vcc index 7808e14..dfedbd7 100644 --- a/lib/libvmod_std/vmod.vcc +++ b/lib/libvmod_std/vmod.vcc @@ -280,18 +280,3 @@ SEE ALSO * :ref:`varnishd(1)` * :ref:`vsl(7)` - -HISTORY -======= - -The Varnish standard module was released along with Varnish Cache 3.0. -This manual page was written by Per Buer with help from Martin Blix -Grydeland. - -COPYRIGHT -========= - -This document is licensed under the same licence as Varnish -itself. See LICENCE for details. - -* Copyright (c) 2010-2015 Varnish Software AS From fgsch at lodoss.net Thu Oct 13 07:34:04 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Thu, 13 Oct 2016 09:34:04 +0200 Subject: [master] 361e393 Rename for clarity Message-ID: commit 361e393076e22e4681aaa02b30300dc9b256a2c9 Author: Federico G. Schwindt Date: Wed Oct 12 21:50:36 2016 +0100 Rename for clarity diff --git a/bin/varnishd/storage/storage.h b/bin/varnishd/storage/storage.h index 0f85255..db8856b 100644 --- a/bin/varnishd/storage/storage.h +++ b/bin/varnishd/storage/storage.h @@ -65,7 +65,7 @@ struct storage { typedef void storage_init_f(struct stevedore *, int ac, char * const *av); typedef void storage_open_f(struct stevedore *); typedef int storage_allocobj_f(struct worker *, const struct stevedore *, - struct objcore *, unsigned ltot, int really); + struct objcore *, unsigned, int); typedef void storage_close_f(const struct stevedore *, int pass); typedef int storage_baninfo_f(const struct stevedore *, enum baninfo event, const uint8_t *ban, unsigned len); diff --git a/bin/varnishd/storage/storage_persistent.c b/bin/varnishd/storage/storage_persistent.c index 6cc8542..2b7bf72 100644 --- a/bin/varnishd/storage/storage_persistent.c +++ b/bin/varnishd/storage/storage_persistent.c @@ -505,7 +505,7 @@ smp_allocx(const struct stevedore *st, size_t min_size, size_t max_size, static int __match_proto__(storage_allocobj_f) smp_allocobj(struct worker *wrk, const struct stevedore *stv, - struct objcore *oc, unsigned wsl, int really) + struct objcore *oc, unsigned wsl, int nuke_limit) { struct object *o; struct storage *st; @@ -518,7 +518,7 @@ smp_allocobj(struct worker *wrk, const struct stevedore *stv, CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); CAST_OBJ_NOTNULL(sc, stv->priv, SMP_SC_MAGIC); - assert(really >= 0); + assert(nuke_limit >= 0); /* Don't entertain already dead objects */ if (oc->flags & OC_F_DYING) @@ -532,10 +532,10 @@ smp_allocobj(struct worker *wrk, const struct stevedore *stv, ltot = IRNUP(sc, ltot); while (1) { - if (really > 0) { + if (nuke_limit > 0) { if (!LRU_NukeOne(wrk, stv->lru)) return (0); - really--; + nuke_limit--; } st = smp_allocx(stv, ltot, ltot, &so, &objidx, &sg); if (st != NULL && st->space < ltot) { @@ -544,7 +544,7 @@ smp_allocobj(struct worker *wrk, const struct stevedore *stv, } if (st != NULL) break; - if (!really) + if (!nuke_limit) return (0); } diff --git a/bin/varnishd/storage/storage_simple.c b/bin/varnishd/storage/storage_simple.c index ef427cc..1385927 100644 --- a/bin/varnishd/storage/storage_simple.c +++ b/bin/varnishd/storage/storage_simple.c @@ -131,7 +131,7 @@ SML_MkObject(const struct stevedore *stv, struct objcore *oc, void *ptr) int __match_proto__(storage_allocobj_f) SML_allocobj(struct worker *wrk, const struct stevedore *stv, - struct objcore *oc, unsigned wsl, int really) + struct objcore *oc, unsigned wsl, int nuke_limit) { struct object *o; struct storage *st; @@ -139,13 +139,13 @@ SML_allocobj(struct worker *wrk, const struct stevedore *stv, CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); - assert(really >= 0); + assert(nuke_limit >= 0); ltot = sizeof(struct object) + PRNDUP(wsl); while (1) { - if (really > 0) { + if (nuke_limit > 0) { if (!LRU_NukeOne(wrk, stv->lru)) return (0); - really--; + nuke_limit--; } AN(stv->sml_alloc); st = stv->sml_alloc(stv, ltot); @@ -155,7 +155,7 @@ SML_allocobj(struct worker *wrk, const struct stevedore *stv, } if (st != NULL) break; - if (!really) + if (!nuke_limit) return (0); } AN(st); From hermunn at varnish-software.com Thu Oct 13 08:56:04 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Thu, 13 Oct 2016 10:56:04 +0200 Subject: [4.1] c9598de document bereq preparation in core Message-ID: commit c9598debac3407add0ab084313f2f63a7fc09c9d Author: Nils Goroll Date: Tue Oct 4 17:14:40 2016 +0200 document bereq preparation in core (Backport of 2bbcde1d with the fix b698aec2 squashed in) doc fixes #2107 diff --git a/doc/sphinx/users-guide/vcl-built-in-subs.rst b/doc/sphinx/users-guide/vcl-built-in-subs.rst index 21e5873..0128233 100644 --- a/doc/sphinx/users-guide/vcl-built-in-subs.rst +++ b/doc/sphinx/users-guide/vcl-built-in-subs.rst @@ -271,6 +271,34 @@ The `vcl_backend_fetch` subroutine may terminate with calling background fetch, control is passed to :ref:`vcl_synth` on the client side with ``resp.status`` preset to 503. +Before calling `vcl_backend_fetch`, varnish core prepares the `bereq` +backend request as follows: + +* Unless the request is a `pass`, + + * set ``bereq.method`` to ``GET`` and ``bereq.proto`` to + ``HTTP/1.1`` and + + * set ``bereq.http.Accept_Encoding`` to ``gzip`` if + :ref:`ref_param_http_gzip_support` is enabled. + +* If there is an existing cache object to be revalidated, set + ``bereq.http.If-Modified-Since`` from its ``Last-Modified`` header + and/or set ``bereq.http.If-None-Match`` from its ``Etag`` header + +* Set ``bereq.http.X-Varnish`` to the current transaction id (`vxid`) + +These changes can be undone or modified in `vcl_backend_fetch` before +the backend request is issued. + +In particular, to cache non-GET requests, ``req.method`` needs to be +saved to a header or variable in :ref:`vcl_recv` and restored to +``bereq.method``. Notice that caching non-GET requests typically also +requires changing the cache key in :ref:`vcl_hash` e.g. by also +hashing the request method and/or request body. + +HEAD request can be satisfied from cached GET responses. + .. _vcl_backend_response: vcl_backend_response From hermunn at varnish-software.com Thu Oct 13 13:30:06 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Thu, 13 Oct 2016 15:30:06 +0200 Subject: [4.1] 9ec3d3d Changelog update for 4.1.4-beta2 Message-ID: commit 9ec3d3d687425735445eef8e81c1701804415791 Author: P?l Hermunn Johansen Date: Thu Oct 13 15:28:27 2016 +0200 Changelog update for 4.1.4-beta2 diff --git a/doc/changes.rst b/doc/changes.rst index 66ab184..6ca2418 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -1,4 +1,22 @@ ====================================== +Varnish Cache 4.1.4-beta2 (2016-10-13) +====================================== + +Bugs fixed +---------- + +* 1830_ - VSL API: "duplicate link" errors in request grouping when + vsl_buffer is increased +* 2010_ - varnishadm CLI behaving weirdly +* 2017_ - varnishncsa docfix: "%r" field is wrong +* 2107_ - (docfix) HEAD requestes changed to GET + +.. _1830: https://github.com/varnishcache/varnish-cache/issues/1830 +.. _2010: https://github.com/varnishcache/varnish-cache/issues/2010 +.. _2017: https://github.com/varnishcache/varnish-cache/issues/2017 +.. _2107: https://github.com/varnishcache/varnish-cache/issues/2107 + +====================================== Varnish Cache 4.1.4-beta1 (2016-09-14) ====================================== From hermunn at varnish-software.com Thu Oct 13 13:32:04 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Thu, 13 Oct 2016 15:32:04 +0200 Subject: [4.1] e77e315 Prepare for 4.1.4-beta2 release Message-ID: commit e77e315c32be15c22c3e82226d8f06828aee59ce Author: P?l Hermunn Johansen Date: Thu Oct 13 15:30:54 2016 +0200 Prepare for 4.1.4-beta2 release diff --git a/configure.ac b/configure.ac index 02603be..1653b25 100644 --- a/configure.ac +++ b/configure.ac @@ -2,7 +2,7 @@ AC_PREREQ(2.59) AC_COPYRIGHT([Copyright (c) 2006 Verdens Gang AS Copyright (c) 2006-2016 Varnish Software]) AC_REVISION([$Id$]) -AC_INIT([Varnish], [4.1.4-beta1], [varnish-dev at varnish-cache.org]) +AC_INIT([Varnish], [4.1.4-beta2], [varnish-dev at varnish-cache.org]) AC_CONFIG_SRCDIR(include/miniobj.h) AC_CONFIG_HEADERS([config.h]) AC_CONFIG_MACRO_DIR([m4]) From dridi.boukelmoune at gmail.com Thu Oct 13 13:43:05 2016 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 13 Oct 2016 15:43:05 +0200 Subject: [master] ce2cfa7 Make -msg an alias to -reason Message-ID: commit ce2cfa7f3b1d1d553e47c87ad8b1e9ebc309134d Author: Dridi Boukelmoune Date: Wed Sep 21 14:54:16 2016 +0200 Make -msg an alias to -reason When we renamed resp.msg to resp.reason to match RFC and VCL termonology we forgot to rename its counterpart. Instead of renaming -msg to -reason the former will instead become an alias of the latter. This way it can be applied to older branches (modulus inline documentation). diff --git a/bin/varnishtest/tests/a00002.vtc b/bin/varnishtest/tests/a00002.vtc index 79c5517..2aa6f64 100644 --- a/bin/varnishtest/tests/a00002.vtc +++ b/bin/varnishtest/tests/a00002.vtc @@ -5,7 +5,7 @@ server s1 { expect req.method == PUT expect req.proto == HTTP/1.0 expect req.url == "/foo" - txresp -proto HTTP/1.2 -status 201 -msg Foo + txresp -proto HTTP/1.2 -status 201 -reason Foo } server s1 -start diff --git a/bin/varnishtest/tests/a00003.vtc b/bin/varnishtest/tests/a00003.vtc index b924c76..8855dba 100644 --- a/bin/varnishtest/tests/a00003.vtc +++ b/bin/varnishtest/tests/a00003.vtc @@ -5,7 +5,7 @@ server s1 { expect req.method == PUT expect req.proto == HTTP/1.0 expect req.url == "/foo" - txresp -proto HTTP/1.2 -status 201 -msg Foo + txresp -proto HTTP/1.2 -status 201 -reason Foo } server s2 { diff --git a/bin/varnishtest/tests/a00004.vtc b/bin/varnishtest/tests/a00004.vtc index ce3a3b5..be185a6 100644 --- a/bin/varnishtest/tests/a00004.vtc +++ b/bin/varnishtest/tests/a00004.vtc @@ -5,7 +5,7 @@ server s1 -repeat 2 { expect req.method == PUT expect req.proto == HTTP/1.0 expect req.url == "/foo" - txresp -proto HTTP/1.2 -status 201 -msg Foo + txresp -proto HTTP/1.2 -status 201 -reason Foo } server s1 -start diff --git a/bin/varnishtest/tests/a00005.vtc b/bin/varnishtest/tests/a00005.vtc index 3d8aa49..7de7675 100644 --- a/bin/varnishtest/tests/a00005.vtc +++ b/bin/varnishtest/tests/a00005.vtc @@ -5,7 +5,7 @@ server s1 { expect req.method == PUT expect req.proto == HTTP/1.0 expect req.url == "/foo" - txresp -proto HTTP/1.2 -status 201 -msg Foo + txresp -proto HTTP/1.2 -status 201 -reason Foo } server s2 { diff --git a/bin/varnishtest/tests/a00006.vtc b/bin/varnishtest/tests/a00006.vtc index 160c883..c01522d 100644 --- a/bin/varnishtest/tests/a00006.vtc +++ b/bin/varnishtest/tests/a00006.vtc @@ -5,7 +5,7 @@ server s1 { expect req.method == PUT expect req.proto == HTTP/1.0 expect req.url == "/foo" - txresp -proto HTTP/1.2 -status 201 -msg Foo \ + txresp -proto HTTP/1.2 -status 201 -reason Foo \ -body "987654321\n" } diff --git a/bin/varnishtest/tests/r00962.vtc b/bin/varnishtest/tests/r00962.vtc index c66dc84..2b6d03e 100644 --- a/bin/varnishtest/tests/r00962.vtc +++ b/bin/varnishtest/tests/r00962.vtc @@ -38,7 +38,7 @@ varnish v1 -stop server s1 { rxreq - txresp -status 400 -msg "Persistent Object Not Found" + txresp -status 400 -reason "Persistent Object Not Found" } -start diff --git a/bin/varnishtest/tests/r01485.vtc b/bin/varnishtest/tests/r01485.vtc index 302f177..1163ff4 100644 --- a/bin/varnishtest/tests/r01485.vtc +++ b/bin/varnishtest/tests/r01485.vtc @@ -6,7 +6,7 @@ server s1 { rxreq expect req.http.If-None-Match == {"foo"} - txresp -status 304 -msg "Not Modified" + txresp -status 304 -reason "Not Modified" } -start varnish v1 -vcl+backend { diff --git a/bin/varnishtest/vtc_http.c b/bin/varnishtest/vtc_http.c index 13671cd..0894f81 100644 --- a/bin/varnishtest/vtc_http.c +++ b/bin/varnishtest/vtc_http.c @@ -945,9 +945,12 @@ http_tx_parse_args(char * const *av, struct vtclog *vl, struct http *hp, * \-status NUMBER (txresp only) * What status code to return (default 200). * - * \-msg STRING (txresp only) + * \-reason STRING (txresp only) * What message to put in the status line (default: "OK"). * + * \-msg STRING (txresp only) + * An alias to -reason. + * * These three switches can appear in any order but must come before the * following ones. * @@ -991,7 +994,7 @@ cmd_http_txresp(CMD_ARGS) struct http *hp; const char *proto = "HTTP/1.1"; const char *status = "200"; - const char *msg = "OK"; + const char *reason = "OK"; char* body = NULL; (void)cmd; @@ -1010,15 +1013,15 @@ cmd_http_txresp(CMD_ARGS) } else if (!strcmp(*av, "-status")) { status = av[1]; av++; - } else if (!strcmp(*av, "-msg")) { - msg = av[1]; + } else if (!strcmp(*av, "-reason") || !strcmp(*av, "-msg")) { + reason = av[1]; av++; continue; } else break; } - VSB_printf(hp->vsb, "%s %s %s%s", proto, status, msg, nl); + VSB_printf(hp->vsb, "%s %s %s%s", proto, status, reason, nl); /* send a "Content-Length: 0" header unless something else happens */ REPLACE(body, ""); From dridi.boukelmoune at gmail.com Thu Oct 13 13:43:05 2016 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 13 Oct 2016 15:43:05 +0200 Subject: [master] 62c3d56 Drop support for -msg Message-ID: commit 62c3d56214e1f3fee8bbf81ae8180ddb6b61e1a5 Author: Dridi Boukelmoune Date: Wed Sep 21 14:59:59 2016 +0200 Drop support for -msg diff --git a/bin/varnishtest/vtc_http.c b/bin/varnishtest/vtc_http.c index 0894f81..e691198 100644 --- a/bin/varnishtest/vtc_http.c +++ b/bin/varnishtest/vtc_http.c @@ -948,9 +948,6 @@ http_tx_parse_args(char * const *av, struct vtclog *vl, struct http *hp, * \-reason STRING (txresp only) * What message to put in the status line (default: "OK"). * - * \-msg STRING (txresp only) - * An alias to -reason. - * * These three switches can appear in any order but must come before the * following ones. * @@ -1013,7 +1010,7 @@ cmd_http_txresp(CMD_ARGS) } else if (!strcmp(*av, "-status")) { status = av[1]; av++; - } else if (!strcmp(*av, "-reason") || !strcmp(*av, "-msg")) { + } else if (!strcmp(*av, "-reason")) { reason = av[1]; av++; continue; From dridi.boukelmoune at gmail.com Thu Oct 13 13:51:05 2016 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 13 Oct 2016 15:51:05 +0200 Subject: [4.1] 33666d6 Make -msg an alias to -reason Message-ID: commit 33666d69faa26824600fa8e9525a8bc995bb028d Author: Dridi Boukelmoune Date: Wed Sep 21 14:54:16 2016 +0200 Make -msg an alias to -reason When we renamed resp.msg to resp.reason to match RFC and VCL terminology we forgot to rename its counterpart. Instead of renaming -msg to -reason the former will instead become an alias of the latter. This way it can be applied to older branches (modulus inline documentation). diff --git a/bin/varnishtest/tests/a00002.vtc b/bin/varnishtest/tests/a00002.vtc index 39b8334..0429309 100644 --- a/bin/varnishtest/tests/a00002.vtc +++ b/bin/varnishtest/tests/a00002.vtc @@ -5,7 +5,7 @@ server s1 { expect req.method == PUT expect req.proto == HTTP/1.0 expect req.url == "/foo" - txresp -proto HTTP/1.2 -status 201 -msg Foo + txresp -proto HTTP/1.2 -status 201 -reason Foo } server s1 -start diff --git a/bin/varnishtest/tests/a00003.vtc b/bin/varnishtest/tests/a00003.vtc index 810da48..b6453a8 100644 --- a/bin/varnishtest/tests/a00003.vtc +++ b/bin/varnishtest/tests/a00003.vtc @@ -5,7 +5,7 @@ server s1 { expect req.method == PUT expect req.proto == HTTP/1.0 expect req.url == "/foo" - txresp -proto HTTP/1.2 -status 201 -msg Foo + txresp -proto HTTP/1.2 -status 201 -reason Foo } server s2 { diff --git a/bin/varnishtest/tests/a00004.vtc b/bin/varnishtest/tests/a00004.vtc index 85659ac..01f3e29 100644 --- a/bin/varnishtest/tests/a00004.vtc +++ b/bin/varnishtest/tests/a00004.vtc @@ -5,7 +5,7 @@ server s1 -repeat 2 { expect req.method == PUT expect req.proto == HTTP/1.0 expect req.url == "/foo" - txresp -proto HTTP/1.2 -status 201 -msg Foo + txresp -proto HTTP/1.2 -status 201 -reason Foo } server s1 -start diff --git a/bin/varnishtest/tests/a00005.vtc b/bin/varnishtest/tests/a00005.vtc index 6c0d634..f8c27ae 100644 --- a/bin/varnishtest/tests/a00005.vtc +++ b/bin/varnishtest/tests/a00005.vtc @@ -5,7 +5,7 @@ server s1 { expect req.method == PUT expect req.proto == HTTP/1.0 expect req.url == "/foo" - txresp -proto HTTP/1.2 -status 201 -msg Foo + txresp -proto HTTP/1.2 -status 201 -reason Foo } server s2 { diff --git a/bin/varnishtest/tests/a00006.vtc b/bin/varnishtest/tests/a00006.vtc index f8a928e..81f7c6e 100644 --- a/bin/varnishtest/tests/a00006.vtc +++ b/bin/varnishtest/tests/a00006.vtc @@ -5,7 +5,7 @@ server s1 { expect req.method == PUT expect req.proto == HTTP/1.0 expect req.url == "/foo" - txresp -proto HTTP/1.2 -status 201 -msg Foo \ + txresp -proto HTTP/1.2 -status 201 -reason Foo \ -body "987654321\n" } diff --git a/bin/varnishtest/tests/r00962.vtc b/bin/varnishtest/tests/r00962.vtc index cc5168c..dc79ae4 100644 --- a/bin/varnishtest/tests/r00962.vtc +++ b/bin/varnishtest/tests/r00962.vtc @@ -38,7 +38,7 @@ varnish v1 -stop server s1 { rxreq - txresp -status 400 -msg "Persistent Object Not Found" + txresp -status 400 -reason "Persistent Object Not Found" } -start diff --git a/bin/varnishtest/tests/r01485.vtc b/bin/varnishtest/tests/r01485.vtc index abfe839..a6b8b60 100644 --- a/bin/varnishtest/tests/r01485.vtc +++ b/bin/varnishtest/tests/r01485.vtc @@ -6,7 +6,7 @@ server s1 { rxreq expect req.http.If-None-Match == {"foo"} - txresp -status 304 -msg "Not Modified" + txresp -status 304 -reason "Not Modified" } -start varnish v1 -vcl+backend { diff --git a/bin/varnishtest/vtc_http.c b/bin/varnishtest/vtc_http.c index 48759de..1503b7e 100644 --- a/bin/varnishtest/vtc_http.c +++ b/bin/varnishtest/vtc_http.c @@ -823,7 +823,7 @@ cmd_http_txresp(CMD_ARGS) struct http *hp; const char *proto = "HTTP/1.1"; const char *status = "200"; - const char *msg = "OK"; + const char *reason = "OK"; char* body = NULL; (void)cmd; @@ -842,15 +842,15 @@ cmd_http_txresp(CMD_ARGS) } else if (!strcmp(*av, "-status")) { status = av[1]; av++; - } else if (!strcmp(*av, "-msg")) { - msg = av[1]; + } else if (!strcmp(*av, "-reason") || !strcmp(*av, "-msg")) { + reason = av[1]; av++; continue; } else break; } - VSB_printf(hp->vsb, "%s %s %s%s", proto, status, msg, nl); + VSB_printf(hp->vsb, "%s %s %s%s", proto, status, reason, nl); /* send a "Content-Length: 0" header unless something else happens */ REPLACE(body, ""); From fgsch at lodoss.net Thu Oct 13 13:53:05 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Thu, 13 Oct 2016 15:53:05 +0200 Subject: [master] bd33cab Only try nuking after we failed to allocate space Message-ID: commit bd33cab5a454c0cac450716c1082ba9bdd36a679 Author: Federico G. Schwindt Date: Thu Oct 13 14:50:50 2016 +0100 Only try nuking after we failed to allocate space diff --git a/bin/varnishd/storage/storage_persistent.c b/bin/varnishd/storage/storage_persistent.c index 2b7bf72..af9e46b 100644 --- a/bin/varnishd/storage/storage_persistent.c +++ b/bin/varnishd/storage/storage_persistent.c @@ -531,12 +531,7 @@ smp_allocobj(struct worker *wrk, const struct stevedore *stv, ltot = sizeof(struct object) + PRNDUP(wsl); ltot = IRNUP(sc, ltot); - while (1) { - if (nuke_limit > 0) { - if (!LRU_NukeOne(wrk, stv->lru)) - return (0); - nuke_limit--; - } + for (; nuke_limit >= 0; nuke_limit--) { st = smp_allocx(stv, ltot, ltot, &so, &objidx, &sg); if (st != NULL && st->space < ltot) { stv->sml_free(st); // NOP @@ -544,7 +539,7 @@ smp_allocobj(struct worker *wrk, const struct stevedore *stv, } if (st != NULL) break; - if (!nuke_limit) + if (!nuke_limit || !LRU_NukeOne(wrk, stv->lru)) return (0); } diff --git a/bin/varnishd/storage/storage_simple.c b/bin/varnishd/storage/storage_simple.c index 1385927..8649c6f 100644 --- a/bin/varnishd/storage/storage_simple.c +++ b/bin/varnishd/storage/storage_simple.c @@ -141,12 +141,7 @@ SML_allocobj(struct worker *wrk, const struct stevedore *stv, CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); assert(nuke_limit >= 0); ltot = sizeof(struct object) + PRNDUP(wsl); - while (1) { - if (nuke_limit > 0) { - if (!LRU_NukeOne(wrk, stv->lru)) - return (0); - nuke_limit--; - } + for (; nuke_limit >= 0; nuke_limit--) { AN(stv->sml_alloc); st = stv->sml_alloc(stv, ltot); if (st != NULL && st->space < ltot) { @@ -155,7 +150,7 @@ SML_allocobj(struct worker *wrk, const struct stevedore *stv, } if (st != NULL) break; - if (!nuke_limit) + if (!nuke_limit || !LRU_NukeOne(wrk, stv->lru)) return (0); } AN(st); From fgsch at lodoss.net Thu Oct 13 17:03:05 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Thu, 13 Oct 2016 19:03:05 +0200 Subject: [master] 8fce2ca Polish Message-ID: commit 8fce2ca77d28e88a0e3fd9fc78a3ac584ff0f0d5 Author: Federico G. Schwindt Date: Thu Oct 13 15:07:22 2016 +0100 Polish diff --git a/bin/varnishd/storage/storage_simple.c b/bin/varnishd/storage/storage_simple.c index 8649c6f..bf0834c 100644 --- a/bin/varnishd/storage/storage_simple.c +++ b/bin/varnishd/storage/storage_simple.c @@ -54,6 +54,7 @@ sml_stv_alloc(const struct stevedore *stv, size_t size, int flags) struct storage *st; CHECK_OBJ_NOTNULL(stv, STEVEDORE_MAGIC); + AN(stv->sml_alloc); if (!(flags & LESS_MEM_ALLOCED_IS_OK)) { if (size > cache_param->fetch_maxchunksize) @@ -70,7 +71,6 @@ sml_stv_alloc(const struct stevedore *stv, size_t size, int flags) for (;;) { /* try to allocate from it */ assert(size > 0); - AN(stv->sml_alloc); st = stv->sml_alloc(stv, size); if (st != NULL) break; @@ -138,11 +138,14 @@ SML_allocobj(struct worker *wrk, const struct stevedore *stv, unsigned ltot; CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); + CHECK_OBJ_NOTNULL(stv, STEVEDORE_MAGIC); CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); + + AN(stv->sml_alloc); assert(nuke_limit >= 0); + ltot = sizeof(struct object) + PRNDUP(wsl); for (; nuke_limit >= 0; nuke_limit--) { - AN(stv->sml_alloc); st = stv->sml_alloc(stv, ltot); if (st != NULL && st->space < ltot) { stv->sml_free(st); @@ -153,7 +156,7 @@ SML_allocobj(struct worker *wrk, const struct stevedore *stv, if (!nuke_limit || !LRU_NukeOne(wrk, stv->lru)) return (0); } - AN(st); + CHECK_OBJ_NOTNULL(st, STORAGE_MAGIC); o = SML_MkObject(stv, oc, st->ptr); CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC); st->len = sizeof(*o); From fgsch at lodoss.net Fri Oct 14 07:04:05 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Fri, 14 Oct 2016 09:04:05 +0200 Subject: [master] 3b9e09f Expose VCL_STEVEDORE to the world Message-ID: commit 3b9e09fcba89caffea0d93f7f45eace1f54e4d69 Author: Federico G. Schwindt Date: Fri Oct 14 07:50:30 2016 +0100 Expose VCL_STEVEDORE to the world diff --git a/doc/sphinx/reference/vmod.rst b/doc/sphinx/reference/vmod.rst index 406c0f3..910d613 100644 --- a/doc/sphinx/reference/vmod.rst +++ b/doc/sphinx/reference/vmod.rst @@ -266,6 +266,11 @@ STRING by using storage allocated from the client or backend workspaces. +STEVEDORE + C-type: ``const struct stevedore *`` + + A storage backend. + STRING_LIST C-type: ``const char *, ...`` diff --git a/lib/libvcc/vmodtool.py b/lib/libvcc/vmodtool.py index 944df58..60415f2 100755 --- a/lib/libvcc/vmodtool.py +++ b/lib/libvcc/vmodtool.py @@ -68,6 +68,7 @@ ctypes = { 'PRIV_TOP': "struct vmod_priv *", 'PROBE': "VCL_PROBE", 'REAL': "VCL_REAL", + 'STEVEDORE': "VCL_STEVEDORE", 'STRING': "VCL_STRING", 'STRING_LIST': "const char *, ...", 'TIME': "VCL_TIME", From fgsch at lodoss.net Fri Oct 14 07:04:05 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Fri, 14 Oct 2016 09:04:05 +0200 Subject: [master] 6db04b7 Add a no_stevedore function for testing Message-ID: commit 6db04b78bc38bc2f5c4a0f2750123341636b5bac Author: Federico G. Schwindt Date: Fri Oct 14 07:51:02 2016 +0100 Add a no_stevedore function for testing diff --git a/lib/libvmod_debug/vmod.vcc b/lib/libvmod_debug/vmod.vcc index 82e264f..4cebb1f 100644 --- a/lib/libvmod_debug/vmod.vcc +++ b/lib/libvmod_debug/vmod.vcc @@ -71,6 +71,10 @@ $Function BACKEND no_backend() Fails at backend selection +$Function STEVEDORE no_stevedore() + +Fails at storage selection. + $Object obj(STRING string="default", ENUM { one, two, three } number="one") Test object diff --git a/lib/libvmod_debug/vmod_debug.c b/lib/libvmod_debug/vmod_debug.c index 3322da7..a0ab290 100644 --- a/lib/libvmod_debug/vmod_debug.c +++ b/lib/libvmod_debug/vmod_debug.c @@ -184,6 +184,14 @@ vmod_no_backend(VRT_CTX) return (NULL); } +VCL_STEVEDORE __match_proto__(td_debug_no_stevedore) +vmod_no_stevedore(VRT_CTX) +{ + + CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + return (NULL); +} + VCL_VOID __match_proto__(td_debug_rot52) vmod_rot52(VRT_CTX, VCL_HTTP hp) { From fgsch at lodoss.net Fri Oct 14 10:22:05 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Fri, 14 Oct 2016 12:22:05 +0200 Subject: [master] 44d5fe9 Kill any temporary vsm files on failure Message-ID: commit 44d5fe9e8ec4d73ea11587dc34010912da3fd848 Author: Federico G. Schwindt Date: Fri Oct 14 11:04:46 2016 +0100 Kill any temporary vsm files on failure Use MGT_complain() to log any errors; we might have a better chance to understand what's going in. Fixes #2115. diff --git a/bin/varnishd/mgt/mgt_shmem.c b/bin/varnishd/mgt/mgt_shmem.c index d760066..a8ec79b 100644 --- a/bin/varnishd/mgt/mgt_shmem.c +++ b/bin/varnishd/mgt/mgt_shmem.c @@ -152,7 +152,7 @@ vsm_zerofile(const char *fn, ssize_t size) fd = VFL_Open(fn, O_RDWR | O_CREAT | O_EXCL | O_NONBLOCK, 0640); if (fd < 0) { - fprintf(stderr, "Could not create %s: %s\n", + MGT_complain(C_ERR, "Could not create %s: %s", fn, strerror(errno)); return (-1); } @@ -162,7 +162,7 @@ vsm_zerofile(const char *fn, ssize_t size) flags &= ~O_NONBLOCK; AZ(fcntl(fd, F_SETFL, flags)); if (VFIL_allocate(fd, (off_t)size, 1)) { - fprintf(stderr, "File allocation error %s: %s\n", + MGT_complain(C_ERR, "File allocation error %s: %s", fn, strerror(errno)); return (-1); } @@ -184,6 +184,17 @@ mgt_shm_size(void) return (size); } +static void +mgt_shm_cleanup(void) +{ + char fnbuf[64]; + + bprintf(fnbuf, "%s.%jd", VSM_FILENAME, (intmax_t)getpid()); + VJ_master(JAIL_MASTER_FILE); + (void)unlink(fnbuf); + VJ_master(JAIL_MASTER_LOW); +} + void mgt_SHM_Create(void) { @@ -200,8 +211,10 @@ mgt_SHM_Create(void) VJ_master(JAIL_MASTER_FILE); vsm_fd = vsm_zerofile(fnbuf, size); VJ_master(JAIL_MASTER_LOW); - if (vsm_fd < 0) + if (vsm_fd < 0) { + mgt_shm_cleanup(); exit(1); + } p = (void *)mmap(NULL, size, PROT_READ|PROT_WRITE, @@ -211,7 +224,9 @@ mgt_SHM_Create(void) AZ(close(vsm_fd)); if (p == MAP_FAILED) { - fprintf(stderr, "Mmap error %s: %s\n", fnbuf, strerror(errno)); + MGT_complain(C_ERR, "Mmap error %s: %s", + fnbuf, strerror(errno)); + mgt_shm_cleanup(); exit(1); } From fgsch at lodoss.net Fri Oct 14 10:22:05 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Fri, 14 Oct 2016 12:22:05 +0200 Subject: [master] 2a55d39 Newline is done by MGT_complain() Message-ID: commit 2a55d3966b18c8f756ce176e954eb5df8c352d0f Author: Federico G. Schwindt Date: Fri Oct 14 11:18:40 2016 +0100 Newline is done by MGT_complain() diff --git a/bin/varnishd/mgt/mgt_shmem.c b/bin/varnishd/mgt/mgt_shmem.c index a8ec79b..e565d06 100644 --- a/bin/varnishd/mgt/mgt_shmem.c +++ b/bin/varnishd/mgt/mgt_shmem.c @@ -275,7 +275,7 @@ mgt_SHM_Commit(void) bprintf(fnbuf, "%s.%jd", VSM_FILENAME, (intmax_t)getpid()); VJ_master(JAIL_MASTER_FILE); if (rename(fnbuf, VSM_FILENAME)) { - MGT_complain(C_ERR, "Rename failed %s -> %s: %s\n", + MGT_complain(C_ERR, "Rename failed %s -> %s: %s", fnbuf, VSM_FILENAME, strerror(errno)); (void)unlink(fnbuf); retval = -1; From varnish-commit at varnish-cache.org Mon Oct 17 04:28:00 2016 From: varnish-commit at varnish-cache.org (omy) Date: Mon, 17 Oct 2016 12:28:00 +0800 Subject: =?utf-8?B?dmFybmlzaC1jb21taXQ6bGsw5aaC5L2V?= =?utf-8?B?5aSE55CG5ZGY5bel6L+d57qq6Zeu6aKYPyA=?= Message-ID: <20161017122811161857@dtvw.com> varnish-commit: ?? 1.???????????????????????????? 2.?????????????????? 3.???????????????????????? 4.?????????????????????????? 5.?????????????? 6.?????????????????????????? 7.?????????????????????????? 8.??????????????????? 9.??????????????????????? ??????????????????? ?????????????????????? 2016/10/17 ???12:28:10 ??? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ??????????????????????.docx Type: application/vnd.openxmlformats-officedocument.wordprocessingml.document Size: 27463 bytes Desc: not available URL: From phk at FreeBSD.org Mon Oct 17 09:05:05 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 17 Oct 2016 11:05:05 +0200 Subject: [master] fe2ff86 Always use a throw-away -n argument with -C and clean it up afterwards. Message-ID: commit fe2ff86ad636bbee0f18cff7f6e34980d0bb4cf7 Author: Poul-Henning Kamp Date: Thu Oct 13 08:15:04 2016 +0000 Always use a throw-away -n argument with -C and clean it up afterwards. diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h index a4d835a..8565f82 100644 --- a/bin/varnishd/mgt/mgt.h +++ b/bin/varnishd/mgt/mgt.h @@ -176,7 +176,7 @@ char *mgt_VccCompile(struct cli *, struct vclprog *, const char *vclname, const char *vclsrc, const char *vclsrcfile, int C_flag); void mgt_vcl_init(void); -void mgt_vcc_startup(struct cli *, const char *b_arg, const char *f_arg, +void mgt_vcl_startup(struct cli *, const char *b_arg, const char *f_arg, const char *vclsrc, int Cflag); int mgt_push_vcls_and_start(struct cli *, unsigned *status, char **p); void mgt_vcl_export_labels(struct vcc *); diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c index 97a814f..29e1cb6 100644 --- a/bin/varnishd/mgt/mgt_main.c +++ b/bin/varnishd/mgt/mgt_main.c @@ -512,6 +512,7 @@ main(int argc, char * const *argv) char **av; unsigned clilim; int jailed = 0; + char Cn_arg[] = "/tmp/varnishd_C_XXXXXXX"; /* Set up the mgt counters */ memset(&static_VSC_C_mgt, 0, sizeof static_VSC_C_mgt); @@ -685,6 +686,13 @@ main(int argc, char * const *argv) } } + if (C_flag) { + if (b_arg == NULL && f_arg == NULL) + ARGV_ERR("-C only good with -b or -f\n"); + AN(mkdtemp(Cn_arg)); + n_arg = Cn_arg; + } + if (!jailed) VJ_Init(NULL); @@ -738,7 +746,6 @@ main(int argc, char * const *argv) ARGV_ERR("Cannot create working directory (%s): %s\n", dirname, strerror(errno)); - /* XXX: should this be relative to the -n arg ? */ VJ_master(JAIL_MASTER_FILE); if (P_arg && (pfh = VPF_Open(P_arg, 0644, NULL)) == NULL) ARGV_ERR("Could not open pid/lock (-P) file (%s): %s\n", @@ -755,16 +762,16 @@ main(int argc, char * const *argv) mgt_vcl_init(); if (b_arg != NULL || f_arg != NULL) { - mgt_vcc_startup(cli, b_arg, f_arg, vcl, C_flag); - if (C_flag && cli->result == CLIS_OK) { - AZ(VSB_finish(cli->sb)); - fprintf(stderr, "%s\n", VSB_data(cli->sb)); + mgt_vcl_startup(cli, b_arg, f_arg, vcl, C_flag); + if (C_flag) { + AZ(rmdir(Cn_arg)); + cli_check(cli); exit(0); } cli_check(cli); free(vcl); - } else if (C_flag) - ARGV_ERR("-C only good with -b or -f\n"); + } + AZ(C_flag); if (VTAILQ_EMPTY(&heritage.socks)) MAC_Arg(":80"); diff --git a/bin/varnishd/mgt/mgt_vcl.c b/bin/varnishd/mgt/mgt_vcl.c index bbf590e..e34b0cb 100644 --- a/bin/varnishd/mgt/mgt_vcl.c +++ b/bin/varnishd/mgt/mgt_vcl.c @@ -369,7 +369,7 @@ mgt_new_vcl(struct cli *cli, const char *vclname, const char *vclsrc, /*--------------------------------------------------------------------*/ void -mgt_vcc_startup(struct cli *cli, const char *b_arg, const char *f_arg, +mgt_vcl_startup(struct cli *cli, const char *b_arg, const char *f_arg, const char *vclsrc, int C_flag) { char buf[BUFSIZ]; From fgsch at lodoss.net Mon Oct 17 09:49:05 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Mon, 17 Oct 2016 11:49:05 +0200 Subject: [master] e8cfcc2 Add beresp.storage Message-ID: commit e8cfcc22b02dd42e38deefba30c3e733d76896cf Author: Federico G. Schwindt Date: Thu Oct 13 13:51:05 2016 +0100 Add beresp.storage This supersedes beresp.storage_hint but is kept around for backward compatibility until the next major release. Setting beresp.storage_hint to a valid storage will set beresp.storage as well. If the storage is invalid, beresp.storage is left untouched. This also reworks how the storage selection is done when multiple storages are defined but none is explicitly set, and what nuke limit we use. Previously we will first try with a nuke limit of 0, and on failure retry with the configured nuke limit. Furthermore, if we did not specify the storage we will go through every one in RR fashion with a nuke limit of 0 before retrying. With this change we initialise beresp.storage with a storage backend from the list before we enter v_b_r{}. At the end of v_b_r{} we will use whatever storage is in beresp.storage. If the storage is NULL we fail the request. In all cases we use the nuke limit. diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 57f260c..9d63a5a 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -507,6 +507,7 @@ struct busyobj { struct acct_bereq acct; const char *storage_hint; + const struct stevedore *storage; const struct director *director_req; const struct director *director_resp; enum director_state_e director_state; @@ -1083,7 +1084,7 @@ void RFC2616_Vary_AE(struct http *hp); /* stevedore.c */ int STV_NewObject(struct worker *, struct objcore *, - const char *hint, unsigned len); + const struct stevedore *, unsigned len); /* * A normal pointer difference is signed, but we never want a negative value diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index 40664b4..9051689 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -33,6 +33,7 @@ #include "cache_director.h" #include "cache_filter.h" #include "hash/hash_slinger.h" +#include "storage/storage.h" #include "vcl.h" #include "vtim.h" @@ -46,7 +47,7 @@ static int vbf_allocobj(struct busyobj *bo, unsigned l) { struct objcore *oc; - const char *storage_hint; + const struct stevedore *stv; double lifetime; CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC); @@ -56,16 +57,20 @@ vbf_allocobj(struct busyobj *bo, unsigned l) lifetime = oc->ttl + oc->grace + oc->keep; if (bo->uncacheable || lifetime < cache_param->shortlived) - storage_hint = TRANSIENT_STORAGE; + stv = stv_transient; else - storage_hint = bo->storage_hint; + stv = bo->storage; + bo->storage = NULL; bo->storage_hint = NULL; - if (STV_NewObject(bo->wrk, bo->fetch_objcore, storage_hint, l)) + if (stv == NULL) + return (0); + + if (STV_NewObject(bo->wrk, bo->fetch_objcore, stv, l)) return (1); - if (storage_hint != NULL && !strcmp(storage_hint, TRANSIENT_STORAGE)) + if (stv == stv_transient) return (0); /* @@ -78,7 +83,7 @@ vbf_allocobj(struct busyobj *bo, unsigned l) oc->grace = 0.0; oc->keep = 0.0; return (STV_NewObject(bo->wrk, bo->fetch_objcore, - TRANSIENT_STORAGE, l)); + stv_transient, l)); } /*-------------------------------------------------------------------- @@ -169,6 +174,7 @@ vbf_stp_mkbereq(struct worker *wrk, struct busyobj *bo) CHECK_OBJ_NOTNULL(bo->req, REQ_MAGIC); assert(bo->fetch_objcore->boc->state == BOS_INVALID); + AZ(bo->storage); AZ(bo->storage_hint); HTTP_Setup(bo->bereq0, bo->ws, bo->vsl, SLT_BereqMethod); @@ -232,6 +238,7 @@ vbf_stp_retry(struct worker *wrk, struct busyobj *bo) assert(bo->director_state == DIR_S_NULL); /* reset other bo attributes - See VBO_GetBusyObj */ + bo->storage = NULL; bo->storage_hint = NULL; bo->do_esi = 0; bo->do_stream = 1; @@ -260,8 +267,11 @@ vbf_stp_startfetch(struct worker *wrk, struct busyobj *bo) CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC); + AZ(bo->storage); AZ(bo->storage_hint); + bo->storage = STV_next(); + if (bo->retries > 0) http_Unset(bo->bereq, "\012X-Varnish:"); diff --git a/bin/varnishd/cache/cache_priv.h b/bin/varnishd/cache/cache_priv.h index 415822c..460d01e 100644 --- a/bin/varnishd/cache/cache_priv.h +++ b/bin/varnishd/cache/cache_priv.h @@ -127,6 +127,8 @@ void V1P_Init(void); /* stevedore.c */ void STV_open(void); void STV_close(void); +const struct stevedore *STV_find(const char *); +const struct stevedore *STV_next(void); int STV_BanInfoDrop(const uint8_t *ban, unsigned len); int STV_BanInfoNew(const uint8_t *ban, unsigned len); void STV_BanExport(const uint8_t *banlist, unsigned len); diff --git a/bin/varnishd/cache/cache_req_body.c b/bin/varnishd/cache/cache_req_body.c index 2858e0f..891cd4c 100644 --- a/bin/varnishd/cache/cache_req_body.c +++ b/bin/varnishd/cache/cache_req_body.c @@ -36,6 +36,7 @@ #include "cache_filter.h" #include "vtim.h" #include "hash/hash_slinger.h" +#include "storage/storage.h" /*---------------------------------------------------------------------- * Pull the req.body in via/into a objcore @@ -60,7 +61,7 @@ vrb_pull(struct req *req, ssize_t maxsize, objiterate_f *func, void *priv) req->body_oc = HSH_Private(req->wrk); AN(req->body_oc); - XXXAN(STV_NewObject(req->wrk, req->body_oc, TRANSIENT_STORAGE, 8)); + XXXAN(STV_NewObject(req->wrk, req->body_oc, stv_transient, 8)); vfc->oc = req->body_oc; diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index bab15ec..e2eb434 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -44,6 +44,7 @@ #include "cache_transport.h" #include "hash/hash_slinger.h" +#include "storage/storage.h" #include "vcl.h" #include "vsha256.h" #include "vtim.h" @@ -194,7 +195,7 @@ cnt_synth(struct worker *wrk, struct req *req) req->objcore = HSH_Private(wrk); CHECK_OBJ_NOTNULL(req->objcore, OBJCORE_MAGIC); szl = -1; - if (STV_NewObject(wrk, req->objcore, TRANSIENT_STORAGE, 1024)) { + if (STV_NewObject(wrk, req->objcore, stv_transient, 1024)) { szl = VSB_len(synth_body); assert(szl >= 0); sz = szl; diff --git a/bin/varnishd/cache/cache_vrt_var.c b/bin/varnishd/cache/cache_vrt_var.c index d7e3be4..7fe8d5a 100644 --- a/bin/varnishd/cache/cache_vrt_var.c +++ b/bin/varnishd/cache/cache_vrt_var.c @@ -333,6 +333,7 @@ VRT_r_beresp_storage_hint(VRT_CTX) void VRT_l_beresp_storage_hint(VRT_CTX, const char *str, ...) { + const struct stevedore *stv; va_list ap; const char *b; @@ -347,6 +348,27 @@ VRT_l_beresp_storage_hint(VRT_CTX, const char *str, ...) return; } ctx->bo->storage_hint = b; + stv = STV_find(b); + if (stv != NULL) + ctx->bo->storage = stv; +} + +/*--------------------------------------------------------------------*/ + +VCL_STEVEDORE +VRT_r_beresp_storage(VRT_CTX) +{ + CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + CHECK_OBJ_NOTNULL(ctx->bo, BUSYOBJ_MAGIC); + return (ctx->bo->storage); +} + +void +VRT_l_beresp_storage(VRT_CTX, VCL_STEVEDORE stv) +{ + CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + CHECK_OBJ_NOTNULL(ctx->bo, BUSYOBJ_MAGIC); + ctx->bo->storage = stv; } /*--------------------------------------------------------------------*/ diff --git a/bin/varnishd/storage/stevedore.c b/bin/varnishd/storage/stevedore.c index f977cdd..072ba26 100644 --- a/bin/varnishd/storage/stevedore.c +++ b/bin/varnishd/storage/stevedore.c @@ -49,24 +49,10 @@ static const struct stevedore * volatile stv_next; * XXX: trust pointer writes to be atomic */ -static struct stevedore * -stv_pick_stevedore(struct vsl_log *vsl, const char **hint) +const struct stevedore * +STV_next() { struct stevedore *stv; - - AN(hint); - if (*hint != NULL && **hint != '\0') { - VTAILQ_FOREACH(stv, &stv_stevedores, list) { - if (!strcmp(stv->ident, *hint)) - return (stv); - } - if (!strcmp(TRANSIENT_STORAGE, *hint)) - return (stv_transient); - - /* Hint was not valid, nuke it */ - VSLb(vsl, SLT_Debug, "Storage hint not usable"); - *hint = NULL; - } if (stv_next == NULL) return (stv_transient); /* pick a stevedore and bump the head along */ @@ -87,31 +73,15 @@ stv_pick_stevedore(struct vsl_log *vsl, const char **hint) int STV_NewObject(struct worker *wrk, struct objcore *oc, - const char *hint, unsigned wsl) + const struct stevedore *stv, unsigned wsl) { - struct stevedore *stv, *stv0; - int j; - CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); + CHECK_OBJ_NOTNULL(stv, STEVEDORE_MAGIC); assert(wsl > 0); - stv = stv0 = stv_pick_stevedore(wrk->vsl, &hint); AN(stv->allocobj); - j = stv->allocobj(wrk, stv, oc, wsl, 0); - if (j == 0 && hint == NULL) { - do { - stv = stv_pick_stevedore(wrk->vsl, &hint); - AN(stv->allocobj); - j = stv->allocobj(wrk, stv, oc, wsl, 0); - } while (j == 0 && stv != stv0); - } - if (j == 0 && cache_param->nuke_limit > 0) { - /* no luck; try to free some space and keep trying */ - j = stv->allocobj(wrk, stv, oc, wsl, cache_param->nuke_limit); - } - - if (j == 0) + if (stv->allocobj(wrk, stv, oc, wsl, cache_param->nuke_limit) == 0) return (0); wrk->stats->n_object++; @@ -216,8 +186,8 @@ STV_BanExport(const uint8_t *bans, unsigned len) * VRT functions for stevedores */ -static const struct stevedore * -stv_find(const char *nm) +const struct stevedore * +STV_find(const char *nm) { const struct stevedore *stv; @@ -233,7 +203,7 @@ int VRT_Stv(const char *nm) { - if (stv_find(nm) != NULL) + if (STV_find(nm) != NULL) return (1); return (0); } @@ -250,7 +220,7 @@ VRT_STEVEDORE_string(VCL_STEVEDORE s) VCL_STEVEDORE VRT_stevedore(const char *nm) { - return (stv_find(nm)); + return (STV_find(nm)); } #define VRTSTVVAR(nm, vtype, ctype, dval) \ @@ -259,7 +229,7 @@ VRT_Stv_##nm(const char *nm) \ { \ const struct stevedore *stv; \ \ - stv = stv_find(nm); \ + stv = STV_find(nm); \ if (stv == NULL) \ return (dval); \ if (stv->var_##nm == NULL) \ diff --git a/bin/varnishtest/tests/c00044.vtc b/bin/varnishtest/tests/c00044.vtc index 1519ff5..46bd2b3 100644 --- a/bin/varnishtest/tests/c00044.vtc +++ b/bin/varnishtest/tests/c00044.vtc @@ -24,7 +24,6 @@ varnish v1 \ -vcl+backend { sub vcl_backend_response { set beresp.do_stream = false; - set beresp.storage_hint = "invalid"; # Unset Date header to not change the object sizes unset beresp.http.Date; } diff --git a/bin/varnishtest/tests/c00045.vtc b/bin/varnishtest/tests/c00045.vtc index 022e904..4d3850f 100644 --- a/bin/varnishtest/tests/c00045.vtc +++ b/bin/varnishtest/tests/c00045.vtc @@ -1,4 +1,4 @@ -varnishtest "Object/LRU/Stevedores with hinting" +varnishtest "Object/LRU/Stevedores with storage set" server s1 { rxreq @@ -16,7 +16,7 @@ varnish v1 \ -vcl+backend { sub vcl_backend_response { set beresp.do_stream = false; - set beresp.storage_hint = "s0"; + set beresp.storage = storage.s0; # Unset Date header to not change the object sizes unset beresp.http.Date; } diff --git a/bin/varnishtest/tests/c00046.vtc b/bin/varnishtest/tests/c00046.vtc index 52872d7..87963dc 100644 --- a/bin/varnishtest/tests/c00046.vtc +++ b/bin/varnishtest/tests/c00046.vtc @@ -1,4 +1,4 @@ -varnishtest "Object/LRU/Stevedores with hinting and body alloc failures" +varnishtest "Object/LRU/Stevedores with storage set and body alloc failures" server s1 { rxreq @@ -11,7 +11,7 @@ varnish v1 \ -arg "-smalloc,1m" \ -vcl+backend { sub vcl_backend_response { - set beresp.storage_hint = "s0"; + set beresp.storage = storage.s0; } } -start diff --git a/bin/varnishtest/tests/c00078.vtc b/bin/varnishtest/tests/c00078.vtc new file mode 100644 index 0000000..e7047ff --- /dev/null +++ b/bin/varnishtest/tests/c00078.vtc @@ -0,0 +1,42 @@ +varnishtest "Stevedores RR, beresp.storage and beresp.storage_hint" + +server s1 -repeat 6 { + rxreq + txresp +} -start + +varnish v1 -arg "-smalloc,1m" -arg "-smalloc,1m" \ + -arg "-smalloc,1m" -vcl+backend { + import debug; + sub vcl_backend_response { + if (bereq.url == "/1") { + set beresp.storage_hint = "invalid"; + } else if (bereq.url == "/2") { + set beresp.storage_hint = "s1"; + } else if (bereq.url == "/6") { + set beresp.storage = debug.no_stevedore(); + } + set beresp.http.storage = beresp.storage; + } +} -start + +client c1 { + txreq -url /1 + rxresp + expect resp.http.storage == "storage.s1" + txreq -url /2 + rxresp + expect resp.http.storage == "storage.s1" + txreq -url /3 + rxresp + expect resp.http.storage == "storage.s0" + txreq -url /4 + rxresp + expect resp.http.storage == "storage.s1" + txreq -url /5 + rxresp + expect resp.http.storage == "storage.s2" + txreq -url /6 + rxresp + expect resp.http.storage == +} -run diff --git a/bin/varnishtest/tests/p00008.vtc b/bin/varnishtest/tests/p00008.vtc index c7a144d..fb55acb 100644 --- a/bin/varnishtest/tests/p00008.vtc +++ b/bin/varnishtest/tests/p00008.vtc @@ -20,9 +20,9 @@ varnish v1 \ -arg "-sper2=deprecated_persistent,${tmpdir}/_.per2,10m" \ -vcl+backend { sub vcl_backend_response { - set beresp.storage_hint = "per1"; + set beresp.storage = storage.per1; if (bereq.url ~ "silo2") { - set beresp.storage_hint = "per2"; + set beresp.storage = storage.per2; } } } -start diff --git a/bin/varnishtest/tests/r00962.vtc b/bin/varnishtest/tests/r00962.vtc index 2b6d03e..9b280ec 100644 --- a/bin/varnishtest/tests/r00962.vtc +++ b/bin/varnishtest/tests/r00962.vtc @@ -16,7 +16,7 @@ varnish v1 \ -arg "-sdeprecated_persistent,${tmpdir}/_.per2,10m" \ -vcl+backend { sub vcl_backend_response { - set beresp.storage_hint = "s0"; + set beresp.storage = storage.s0; } } -start diff --git a/bin/varnishtest/tests/r01175.vtc b/bin/varnishtest/tests/r01175.vtc index 5624c20..42a87a3 100644 --- a/bin/varnishtest/tests/r01175.vtc +++ b/bin/varnishtest/tests/r01175.vtc @@ -7,7 +7,7 @@ server s1 { varnish v1 -arg "-s test=malloc,1M" -vcl+backend { sub vcl_backend_response { - set beresp.storage_hint = "test"; + set beresp.storage = storage.test; set beresp.do_stream = false; } } -start diff --git a/bin/varnishtest/tests/r01284.vtc b/bin/varnishtest/tests/r01284.vtc index 7c0502d..dd8c65f 100644 --- a/bin/varnishtest/tests/r01284.vtc +++ b/bin/varnishtest/tests/r01284.vtc @@ -15,7 +15,7 @@ varnish v1 \ -vcl+backend { sub vcl_backend_response { set beresp.do_stream = false; - set beresp.storage_hint = "Transient"; + set beresp.storage = storage.Transient; # Unset Date header to not change the object sizes unset beresp.http.Date; } diff --git a/bin/varnishtest/tests/v00020.vtc b/bin/varnishtest/tests/v00020.vtc index 1fa3891..513f56c 100644 --- a/bin/varnishtest/tests/v00020.vtc +++ b/bin/varnishtest/tests/v00020.vtc @@ -300,3 +300,11 @@ varnish v1 -errvcl {INT * BLOB not possible.} { set resp.status = 100 * debug.str2blob("a"); } } + +# XXX: should spot nonexistent storage +varnish v1 -errvcl {Symbol not found: 'storage.foo' (expected type STEVEDORE):} { + backend b { .host = "127.0.0.1"; } + sub vcl_backend_response { + set beresp.storage = storage.foo; + } +} diff --git a/bin/varnishtest/tests/v00025.vtc b/bin/varnishtest/tests/v00025.vtc index 62c51e6..3e1acd3 100644 --- a/bin/varnishtest/tests/v00025.vtc +++ b/bin/varnishtest/tests/v00025.vtc @@ -42,6 +42,7 @@ varnish v1 -arg "-i J.F.Nobody" -vcl+backend { set beresp.http.bereq_backend = bereq.backend; set beresp.http.beresp_backend = beresp.backend; set beresp.http.keep = beresp.keep; + set beresp.http.stv = beresp.storage; set beresp.http.hint = beresp.storage_hint; set beresp.http.be_ip = beresp.backend.ip; set beresp.http.be_nm = beresp.backend.name; diff --git a/bin/varnishtest/tests/v00033.vtc b/bin/varnishtest/tests/v00033.vtc index ca51ece..b370dd7 100644 --- a/bin/varnishtest/tests/v00033.vtc +++ b/bin/varnishtest/tests/v00033.vtc @@ -15,7 +15,7 @@ varnish v1 -vcl+backend { storage.Transient.used_space + 1 B + 1 KB + 1 MB + 1GB + 1TB; if (bereq.url == "/foo") { - set beresp.storage_hint = "Transient"; + set beresp.storage = storage.Transient; } } sub vcl_deliver { diff --git a/lib/libvcc/generate.py b/lib/libvcc/generate.py index 92bed74..6c79950 100755 --- a/lib/libvcc/generate.py +++ b/lib/libvcc/generate.py @@ -620,12 +620,20 @@ sp_variables = [ IP of the backend this response was fetched from. """ ), + ('beresp.storage', + 'STEVEDORE', + ('backend_response', 'backend_error'), + ('backend_response', 'backend_error'), """ + The storage backend to use to save this object. + """ + ), ('beresp.storage_hint', 'STRING', ('backend_response', 'backend_error'), ('backend_response', 'backend_error'), """ - Hint to Varnish that you want to save this object to a - particular storage backend. + Deprecated. Hint to Varnish that you want to + save this object to a particular storage backend. + Use beresp.storage instead. """ ), ('obj.proto', From phk at FreeBSD.org Mon Oct 17 10:16:05 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 17 Oct 2016 12:16:05 +0200 Subject: [master] 0410099 Let stderr escape so we can see it in case of failure Message-ID: commit 041009945399fe1c86ee3085aed77ef99c354f5a Author: Poul-Henning Kamp Date: Mon Oct 17 10:15:25 2016 +0000 Let stderr escape so we can see it in case of failure diff --git a/bin/varnishtest/tests/a00009.vtc b/bin/varnishtest/tests/a00009.vtc index b751383..a1bf19e 100644 --- a/bin/varnishtest/tests/a00009.vtc +++ b/bin/varnishtest/tests/a00009.vtc @@ -1,9 +1,9 @@ varnishtest "Code coverage of VCL compiler and RSTdump etc" -shell "varnishd -b 127.0.0.1:80 -C -n ${tmpdir} > /dev/null 2>&1" -shell "varnishd -x dumprstparam > /dev/null 2>&1" -shell "varnishd -x dumprstvsl > /dev/null 2>&1" -shell "varnishd -x dumprstcli > /dev/null 2>&1" +shell "varnishd -b 127.0.0.1:80 -C -n ${tmpdir} > /${tmpdir}/_.c" +shell "varnishd -x dumprstparam > /${tmpdir}/_.param" +shell "varnishd -x dumprstvsl > /${tmpdir}/_.vsl" +shell "varnishd -x dumprstcli > /${tmpdir}/_.cli" err_shell {VCL version declaration missing} "echo 'bad vcl' >${tmpdir}/t.vcl ; varnishd -f ${tmpdir}/t.vcl -n ${tmpdir} 2>&1" err_shell {VCL version declaration missing} "echo 'bad vcl' >${tmpdir}/t.vcl ; varnishd -C -f ${tmpdir}/t.vcl -n ${tmpdir} 2>&1" err_shell {-spersistent has been deprecated} "varnishd -spersistent 2>&1" From phk at FreeBSD.org Mon Oct 17 10:37:04 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 17 Oct 2016 12:37:04 +0200 Subject: [master] 8b6783c Unbreak -C (duH!) Message-ID: commit 8b6783c01f0feef945ce422793bce61991433b0f Author: Poul-Henning Kamp Date: Mon Oct 17 10:36:19 2016 +0000 Unbreak -C (duH!) diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c index 29e1cb6..e9f7c64 100644 --- a/bin/varnishd/mgt/mgt_main.c +++ b/bin/varnishd/mgt/mgt_main.c @@ -689,8 +689,10 @@ main(int argc, char * const *argv) if (C_flag) { if (b_arg == NULL && f_arg == NULL) ARGV_ERR("-C only good with -b or -f\n"); - AN(mkdtemp(Cn_arg)); - n_arg = Cn_arg; + if (n_arg == NULL) { + AN(mkdtemp(Cn_arg)); + n_arg = Cn_arg; + } } if (!jailed) @@ -764,8 +766,10 @@ main(int argc, char * const *argv) if (b_arg != NULL || f_arg != NULL) { mgt_vcl_startup(cli, b_arg, f_arg, vcl, C_flag); if (C_flag) { - AZ(rmdir(Cn_arg)); - cli_check(cli); + if (Cn_arg == n_arg) + AZ(rmdir(Cn_arg)); + AZ(VSB_finish(cli->sb)); + fprintf(stderr, "%s\n", VSB_data(cli->sb)); exit(0); } cli_check(cli); From fgsch at lodoss.net Mon Oct 17 10:40:05 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Mon, 17 Oct 2016 12:40:05 +0200 Subject: [master] 2360b71 Add req.storage Message-ID: commit 2360b71b887073c5434e170f5f71c7af7a812d7a Author: Federico G. Schwindt Date: Fri Oct 14 08:47:50 2016 +0100 Add req.storage This tells Varnish which storage backend it should use when dealing with the request body (if any). Fixes #1914. diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 9d63a5a..ba28151 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -566,6 +566,7 @@ struct req { double d_ttl; ssize_t req_bodybytes; /* Parsed req bodybytes */ + const struct stevedore *storage; const struct director *director_hint; struct vcl *vcl; diff --git a/bin/varnishd/cache/cache_req_body.c b/bin/varnishd/cache/cache_req_body.c index 891cd4c..865b667 100644 --- a/bin/varnishd/cache/cache_req_body.c +++ b/bin/varnishd/cache/cache_req_body.c @@ -52,6 +52,7 @@ vrb_pull(struct req *req, ssize_t maxsize, objiterate_f *func, void *priv) struct vfp_ctx *vfc; uint8_t *ptr; enum vfp_status vfps = VFP_ERROR; + const struct stevedore *stv; CHECK_OBJ_NOTNULL(req, REQ_MAGIC); @@ -61,7 +62,15 @@ vrb_pull(struct req *req, ssize_t maxsize, objiterate_f *func, void *priv) req->body_oc = HSH_Private(req->wrk); AN(req->body_oc); - XXXAN(STV_NewObject(req->wrk, req->body_oc, stv_transient, 8)); + + if (req->storage != NULL) + stv = req->storage; + else + stv = stv_transient; + + req->storage = NULL; + + XXXAN(STV_NewObject(req->wrk, req->body_oc, stv, 8)); vfc->oc = req->body_oc; diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index e2eb434..5798374 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -673,6 +673,7 @@ cnt_recv(struct worker *wrk, struct req *req) req->hash_always_miss = 0; req->hash_ignore_busy = 0; req->client_identity = NULL; + req->storage = NULL; http_CollectHdr(req->http, H_Cache_Control); diff --git a/bin/varnishd/cache/cache_vrt_var.c b/bin/varnishd/cache/cache_vrt_var.c index 7fe8d5a..11eaa22 100644 --- a/bin/varnishd/cache/cache_vrt_var.c +++ b/bin/varnishd/cache/cache_vrt_var.c @@ -356,6 +356,24 @@ VRT_l_beresp_storage_hint(VRT_CTX, const char *str, ...) /*--------------------------------------------------------------------*/ VCL_STEVEDORE +VRT_r_req_storage(VRT_CTX) +{ + CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + CHECK_OBJ_NOTNULL(ctx->req, REQ_MAGIC); + return (ctx->req->storage); +} + +void +VRT_l_req_storage(VRT_CTX, VCL_STEVEDORE stv) +{ + CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + CHECK_OBJ_NOTNULL(ctx->req, REQ_MAGIC); + ctx->req->storage = stv; +} + +/*--------------------------------------------------------------------*/ + +VCL_STEVEDORE VRT_r_beresp_storage(VRT_CTX) { CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); diff --git a/bin/varnishtest/tests/r01914.vtc b/bin/varnishtest/tests/r01914.vtc new file mode 100644 index 0000000..a14f053 --- /dev/null +++ b/bin/varnishtest/tests/r01914.vtc @@ -0,0 +1,44 @@ +varnishtest "Set the storage backend for the request body" + +server s1 { + rxreq + expect req.bodylen == 100 + txresp + rxreq + expect req.bodylen == 100 + txresp +} -start + +varnish v1 -arg "-s malloc,1MB" -arg "-s malloc,1MB" -vcl+backend { + import std; + sub vcl_recv { + if (req.url == "/1") { + set req.storage = storage.s1; + } + std.cache_req_body(1KB); + } + sub vcl_backend_fetch { + return (fetch); + } + sub vcl_backend_response { + set beresp.storage = storage.s0; + } +} -start + +client c1 { + txreq -url /0 -bodylen 100 + rxresp +} -run + +varnish v1 -expect SMA.s0.c_bytes > 0 +varnish v1 -expect SMA.s1.c_bytes == 0 +varnish v1 -expect SMA.Transient.c_bytes > 0 + +client c1 { + txreq -url /1 -bodylen 100 + rxresp +} -run + +varnish v1 -expect SMA.s0.c_bytes > 0 +varnish v1 -expect SMA.s1.c_bytes > 0 +varnish v1 -expect SMA.Transient.c_bytes > 0 diff --git a/lib/libvcc/generate.py b/lib/libvcc/generate.py index 6c79950..0843ee5 100755 --- a/lib/libvcc/generate.py +++ b/lib/libvcc/generate.py @@ -257,6 +257,13 @@ sp_variables = [ A count of how many times this request has been restarted. """ ), + ('req.storage', + 'STEVEDORE', + ('recv',), + ('recv',), """ + The storage backend to use to save this request body. + """ + ), ('req.esi_level', 'INT', ('client',), From phk at FreeBSD.org Mon Oct 17 10:52:05 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 17 Oct 2016 12:52:05 +0200 Subject: [master] 5f37e7a Maybe I should just stay away from my keyboard today. Message-ID: commit 5f37e7a2ea8f5ce2f5431d0d2ee2d137f90751ad Author: Poul-Henning Kamp Date: Mon Oct 17 10:51:07 2016 +0000 Maybe I should just stay away from my keyboard today. diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c index e9f7c64..00bb7e2 100644 --- a/bin/varnishd/mgt/mgt_main.c +++ b/bin/varnishd/mgt/mgt_main.c @@ -769,8 +769,8 @@ main(int argc, char * const *argv) if (Cn_arg == n_arg) AZ(rmdir(Cn_arg)); AZ(VSB_finish(cli->sb)); - fprintf(stderr, "%s\n", VSB_data(cli->sb)); - exit(0); + printf("%s\n", VSB_data(cli->sb)); + exit(cli->result == CLIS_OK ? 0 : 2); } cli_check(cli); free(vcl); From nils.goroll at uplex.de Mon Oct 17 14:55:05 2016 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 17 Oct 2016 16:55:05 +0200 Subject: [master] dc3dfaa make -n argument wording consistent with varnishd usage Message-ID: commit dc3dfaa80e9c8011e475427c7a347e1bc916f59c Author: Nils Goroll Date: Mon Oct 17 16:51:07 2016 +0200 make -n argument wording consistent with varnishd usage diff --git a/include/vut_options.h b/include/vut_options.h index 50881cd..7afc723 100644 --- a/include/vut_options.h +++ b/include/vut_options.h @@ -57,9 +57,10 @@ ) #define VUT_OPT_n \ - VOPT("n:", "[-n ]", "Varnish instance name", \ - "Specify the name of the varnishd instance to get logs" \ - " from. If -n is not specified, the host name is used." \ + VOPT("n:", "[-n ]", "varnishd working directory", \ + "Specify the varnishd working directory (also known as" \ + " instance name) to get logs from. If -n is not specified," \ + " the host name is used." \ ) #define VUT_OPT_N \ From fgsch at lodoss.net Mon Oct 17 21:11:05 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Mon, 17 Oct 2016 23:11:05 +0200 Subject: [master] 351a4e1 Fix building on Linux and update comment Message-ID: commit 351a4e147ec18f4ec4602717dcc7e6785d0db1cb Author: Federico G. Schwindt Date: Mon Oct 17 22:09:08 2016 +0100 Fix building on Linux and update comment diff --git a/lib/libvarnish/vav.c b/lib/libvarnish/vav.c index bbbdaba..e1de1b1 100644 --- a/lib/libvarnish/vav.c +++ b/lib/libvarnish/vav.c @@ -26,7 +26,7 @@ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * - * const char **VAV_Parse(const char *s, int comment) + * const char **VAV_Parse(const char *s, int *argc, int flag) * Parse a command like line into an argv[] * Index zero contains NULL or an error message * "double quotes" and backslash substitution is handled. @@ -242,10 +242,30 @@ Test(const char *str) char **av; printf("Test: <%V>\n", str); - av = VAV_Parse(str, 0); + av = VAV_Parse(str, NULL, 0); VAV_Print(av); } +#if defined __linux__ +int +printf_v(FILE *stream, const struct printf_info *info, + const void *const *args) +{ + const char *v = *((char **)args[0]); + return (fprintf(stream, "%*s", + info->left ? -info->width : info->width, v)); +} + +int +printf_v_info(const struct printf_info *info, size_t n, int *argtypes, + int *size) +{ + if (n > 0) + argtypes[0] = PA_STRING; + return (1); +} +#endif + int main(int argc, char **argv) { @@ -254,7 +274,13 @@ main(int argc, char **argv) (void)argc; (void)argv; +#if defined __FreeBSD__ register_printf_render_std("V"); +#elif defined __linux__ + register_printf_specifier('V', printf_v, printf_v_info); +#else +#error Unsupported platform +#endif while (fgets(buf, sizeof buf, stdin)) Test(buf); From fgsch at lodoss.net Wed Oct 19 13:30:06 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Wed, 19 Oct 2016 15:30:06 +0200 Subject: [master] 978c3e3 Correct output and general cleanup Message-ID: commit 978c3e39c21583e51f850581e502090fe561307f Author: Federico G. Schwindt Date: Wed Oct 19 14:05:42 2016 +0100 Correct output and general cleanup diff --git a/bin/varnishtest/vtc.c b/bin/varnishtest/vtc.c index 60c6ae9..7c24d99 100644 --- a/bin/varnishtest/vtc.c +++ b/bin/varnishtest/vtc.c @@ -512,13 +512,13 @@ cmd_err_shell(CMD_ARGS) VSB_destroy(&vsb); } -/* SECTION: client-server.spec.delay delay +/* SECTION: delay delay * * Take a float as argument and sleep for that number of seconds. */ /* SECTION: stream.spec.delay delay * - * Take a float as argument and sleep for that number of seconds. + * Same as for the top-level delay. */ void cmd_delay(CMD_ARGS) diff --git a/bin/varnishtest/vtc_doc_hdr.rst b/bin/varnishtest/vtc_doc_hdr.rst deleted file mode 100644 index bde6c56..0000000 --- a/bin/varnishtest/vtc_doc_hdr.rst +++ /dev/null @@ -1,33 +0,0 @@ -Syntax RFP for HTTP2 in varnishtest -=================================== - -This document tries to document how varnishtest would work this H/2, adapting to -the introduction of stream. The main idea is the introduction of a new command -(stream) inside client and server specification that naively maps to RFC7540. - -It provides little abstraction, allowwing great control over test scenario, -while still retaining a familiar logic. - -Here's an example of test file:: - - server s1 { - non-fatal - stream 1 { - rxreq - expect req.http.foo == - txgoaway -laststream 0 -err 9 -debug "COMPRESSION_ERROR" - } -run - } -start - - client c1 -connect ${s1_sock} { - stream 1 { - txreq -idxHdr 100 -litHdr inc plain "foo" plain "bar" - rxgoaway - expect goaway.err == 9 - expect goaway.laststream == 0 - expect goaway.debug == "COMPRESSION_ERROR" - } -run - } -run - -.. contents:: - diff --git a/bin/varnishtest/vtc_http.c b/bin/varnishtest/vtc_http.c index e691198..eef2448 100644 --- a/bin/varnishtest/vtc_http.c +++ b/bin/varnishtest/vtc_http.c @@ -92,7 +92,7 @@ static const struct cmds http_cmds[]; * \-run (client only) * Equivalent to "-start -wait". * - * \repeat NUMBER + * \-repeat NUMBER * Instead of processing the specification only once, do it NUMBER times. * * \-break (server only) @@ -709,9 +709,9 @@ cmd_http_rxresp(CMD_ARGS) vtc_log(hp->vl, 4, "bodylen = %s", hp->bodylen); } -/* SECTION: client-server.spec.rxreqhdrs +/* SECTION: client-server.spec.rxresphdrs * - * rxresp (client only) + * rxresphdrs (client only) * Receive and parse a response's headers. */ static void @@ -921,7 +921,7 @@ http_tx_parse_args(char * const *av, struct vtclog *vl, struct http *hp, return (av); } -/* SECTION: client-server.spec.txre +/* SECTION: client-server.spec.txreq * * txreq|txresp [...] * Send a minimal request or response, but overload it if necessary. @@ -1153,7 +1153,7 @@ cmd_http_rxreqbody(CMD_ARGS) /* SECTION: client-server.spec.rxrespbody * - * rxrespbody + * rxrespbody (client only) * Receive a response's body. */ @@ -1178,7 +1178,7 @@ cmd_http_rxrespbody(CMD_ARGS) /* SECTION: client-server.spec.rxchunk * * rxchunk - * Receive an HTTP chunk + * Receive an HTTP chunk. */ static void @@ -1512,13 +1512,7 @@ cmd_http_timeout(CMD_ARGS) /* SECTION: client-server.spec.expect_close * * expect_close - * Wait for the connected client to close the connection. - */ - -/* SECTION: client-server.spec.expect_close expect_close (server) - * - * Reads from the connection, expecting nothing to read but an EOF. - * + * Reads from the connection, expecting nothing to read but an EOF. */ static void cmd_http_expect_close(CMD_ARGS) @@ -1563,14 +1557,8 @@ cmd_http_expect_close(CMD_ARGS) /* SECTION: client-server.spec.close * * close (server only) - * Close the active TCP connection - */ - -/* SECTION: client-server.spec.close close (server) - * - * Close the connection. Not that if operating in H/2 mode, no extra (GOAWAY) - * frame is sent, it's simply a TCP close. - * + * Close the connection. Note that if operating in HTTP/2 mode no + * extra (GOAWAY) frame is sent, it's simply a TCP close. */ static void cmd_http_close(CMD_ARGS) @@ -1580,6 +1568,7 @@ cmd_http_close(CMD_ARGS) (void)cmd; (void)vl; CAST_OBJ_NOTNULL(hp, priv, HTTP_MAGIC); + ONLY_SERVER(hp, av); AZ(av[1]); assert(hp->sfd != NULL); assert(*hp->sfd >= 0); @@ -1589,16 +1578,11 @@ cmd_http_close(CMD_ARGS) vtc_log(vl, 4, "Closed"); } -/* SECTION: client-server.spec.accept_close +/* SECTION: client-server.spec.accept * * accept (server only) - * Close the active connection (if any) and accept a new one. - */ - -/* SECTION: client-server.spec.accept accept (server) - * - * Close the potential current connection, and accept a new one. Note that this - * new connection is H/1. + * Close the current connection, if any, and accept a new one. Note + * that this new connection is HTTP/1.x. */ static void cmd_http_accept(CMD_ARGS) @@ -1608,6 +1592,7 @@ cmd_http_accept(CMD_ARGS) (void)cmd; (void)vl; CAST_OBJ_NOTNULL(hp, priv, HTTP_MAGIC); + ONLY_SERVER(hp, av); AZ(av[1]); assert(hp->sfd != NULL); assert(*hp->sfd >= 0); @@ -1680,10 +1665,11 @@ cmd_http_fatal(CMD_ARGS) const char PREFACE[24] = "PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n"; -/* SECTION: client-server.spec.txpri txpri (client) +/* SECTION: client-server.spec.txpri * - * Send an H/2 preface ("PRI * HTTP/2.0\\r\\n\\r\\nSM\\r\\n\\r\\n") and set - * client to H/2. + * txpri (client only) + * Send an HTTP/2 preface ("PRI * HTTP/2.0\\r\\n\\r\\nSM\\r\\n\\r\\n") + * and set client to HTTP/2. */ static void cmd_http_txpri(CMD_ARGS) @@ -1710,10 +1696,11 @@ cmd_http_txpri(CMD_ARGS) AN(hp->h2); } -/* SECTION: client-server.spec.rxpri rxpri (server) +/* SECTION: client-server.spec.rxpri * - * Receive a preface, and if it matches, sets the server to H/2, aborts - * otherwise. + * rxpri (server only) + * Receive a preface. If valid set the server to HTTP/2, abort + * otherwise. */ static void cmd_http_rxpri(CMD_ARGS) @@ -1735,8 +1722,8 @@ cmd_http_rxpri(CMD_ARGS) /* SECTION: client-server.spec.settings * * settings -dectbl INT - * Force internal H/2 settings to certain values. Currently only - * support setting the decoding table size. + * Force internal HTTP/2 settings to certain values. Currently only + * support setting the decoding table size. */ static void cmd_http_settings(CMD_ARGS) diff --git a/bin/varnishtest/vtc_http2.c b/bin/varnishtest/vtc_http2.c index f1b5360..1f5b8da 100644 --- a/bin/varnishtest/vtc_http2.c +++ b/bin/varnishtest/vtc_http2.c @@ -1294,10 +1294,11 @@ cmd_sendhex(CMD_ARGS) /* SECTION: stream.spec.data_0 txreq, txresp, txcont, txpush * - * These four commands are about sending headers. txreq, txresp will send - * HEADER frames, txcont will send CONTINUATION frames, and txpush PUSH frames. - * The only difference between txreq and txresp are the default headers set by - * each of them. + * These four commands are about sending headers. txreq and txresp + * will send HEADER frames; txcont will send CONTINUATION frames; txpush + * PUSH frames. + * The only difference between txreq and txresp are the default headers + * set by each of them. * * \-noadd * Do not add default headers. Useful to avoid duplicates when sending @@ -2095,7 +2096,7 @@ rxstuff(struct stream *s) wt < TYPE_MAX ? h2_types[wt] : "?", wt); \ } while (0); -/* SECTION: stream.spec.data_12 rxhdrs +/* SECTION: stream.spec.data_11 rxhdrs * * ``rxhdrs`` will expect one HEADER frame, then, depending on the arguments, * zero or more CONTINUATION frame. @@ -2273,7 +2274,7 @@ cmd_rxreqsp(CMD_ARGS) s->frame = f; } -/* SECTION: stream.spec.data_11 rxpush +/* SECTION: stream.spec.data_12 rxpush * * This works like ``rxhdrs``, expecting a PUSH frame and then zero or more * CONTINUATION frames. @@ -2338,37 +2339,37 @@ cmd_rxpush(CMD_ARGS) /* SECTION: stream.spec.prio_rxprio rxprio * - * Receive a PRIORITY frame + * Receive a PRIORITY frame. */ RXFUNC(prio, PRIORITY) /* SECTION: stream.spec.reset_rxrst rxrst * - * Receive a RST_STREAM frame + * Receive a RST_STREAM frame. */ RXFUNC(rst, RST_STREAM) /* SECTION: stream.spec.settings_rxsettings rxsettings * - * Receive a SETTINGS frame + * Receive a SETTINGS frame. */ RXFUNC(settings,SETTINGS) /* SECTION: stream.spec.ping_rxping rxping * - * Receive a PING frame + * Receive a PING frame. */ RXFUNC(ping, PING) /* SECTION: stream.spec.goaway_rxgoaway rxgoaway * - * Receive a GOAWAY frame + * Receive a GOAWAY frame. */ RXFUNC(goaway, GOAWAY) /* SECTION: stream.spec.winup_rxwinup rxwinup * - * Receive a WINDOW_UPDATE frame + * Receive a WINDOW_UPDATE frame. */ RXFUNC(winup, WINDOW_UPDATE) @@ -2607,29 +2608,31 @@ stream_run(struct stream *s) -/* SECTION: client-server.spec.zstream stream +/* SECTION: client-server.spec.stream * - * H/2 introduces the concept of streams, and these come with their own - * specification, and as it's quite big, have bee move to their own chapter. + * stream + * HTTP/2 introduces the concept of streams, and these come with + * their own specification, and as it's quite big, have been moved + * to their own chapter. * * SECTION: stream stream * - * (note: this section is at the top-level for easier navigation, but it's part - * of the client/server specification) + * (note: this section is at the top-level for easier navigation, but + * it's part of the client/server specification) * - * Streams map roughly to a request in H/2, a request is sent on stream N, - * the response too, then the stream is discarded. The main exception is the - * first stream, 0, that serves as coordinator. + * Streams map roughly to a request in HTTP/2, a request is sent on + * stream N, the response too, then the stream is discarded. The main + * exception is the first stream, 0, that serves as coordinator. * * Stream syntax follow the client/server one:: * * stream ID [SPEC] [ACTION] * - * ID is the H/2 stream number, while SPEC describes what will be done in that - * stream. + * ID is the HTTP/2 stream number, while SPEC describes what will be + * done in that stream. * - * Note that, when parsing a stream action, if the entity isn't operating in H/2 - * mode, these spec is ran before:: + * Note that, when parsing a stream action, if the entity isn't operating + * in HTTP/2 mode, these spec is ran before:: * * txpri/rxpri # client/server * stream 0 { @@ -2640,7 +2643,7 @@ stream_run(struct stream *s) * expect settings.ack == true * } -run * - * And H/2 mode is then activated before parsing the specification. + * And HTTP/2 mode is then activated before parsing the specification. * * SECTION: stream.actions Actions * From hermunn at varnish-software.com Thu Oct 20 14:41:05 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Thu, 20 Oct 2016 16:41:05 +0200 Subject: [4.0] e1eb3e5 Fix an off-by-one when setting the first bit exceeding the current bitmap size Message-ID: commit e1eb3e506bc6e0ae0bf68977bb283bce157fd772 Author: Nils Goroll Date: Thu Mar 3 11:56:22 2016 +0100 Fix an off-by-one when setting the first bit exceeding the current bitmap size When setting bit n (the n+1th bit) of an n-bit vbitmap, we'd miss to expand the bitmap and thus overflow our buffer and overwrite the first bit of the next byte in memory. diff --git a/include/vbm.h b/include/vbm.h index 13d2d5f..984a3ea 100644 --- a/include/vbm.h +++ b/include/vbm.h @@ -86,7 +86,7 @@ vbit_set(struct vbitmap *vb, unsigned bit) { if (bit >= vb->nbits) - vbit_expand(vb, bit); + vbit_expand(vb, bit + 1); vb->bits[VBITMAP_IDX(bit)] |= VBITMAP_BIT(bit); } From varnish-commit at varnish-cache.org Thu Oct 20 16:50:57 2016 From: varnish-commit at varnish-cache.org (=?utf-8?B?5LqU5bm85Lqm5bm85Lqm?=) Date: Fri, 21 Oct 2016 00:50:57 +0800 Subject: varnish-commit Message-ID: <20161021005104100046@pd.com> varnish-commit: ??? ????????????? ???????????? ????????????? ??????????????????????? ????????????? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ?????????????.doc Type: application/msword Size: 55808 bytes Desc: not available URL: From phk at FreeBSD.org Fri Oct 21 07:07:04 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Fri, 21 Oct 2016 09:07:04 +0200 Subject: [master] 5b2888f Make vbf_fetch_body_helper() a proper step in the fsm. Message-ID: commit 5b2888f3556a0cffc7d89d5db99e97f467eb7a17 Author: Poul-Henning Kamp Date: Tue Oct 18 20:09:25 2016 +0000 Make vbf_fetch_body_helper() a proper step in the fsm. diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index 9051689..4c6fa7d 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -482,8 +482,8 @@ vbf_stp_startfetch(struct worker *wrk, struct busyobj *bo) /*-------------------------------------------------------------------- */ -static void -vbf_fetch_body_helper(struct busyobj *bo) +static enum fetch_step +vbf_stp_fetchbody(struct busyobj *bo) { ssize_t l; uint8_t *ptr; @@ -542,6 +542,7 @@ vbf_fetch_body_helper(struct busyobj *bo) } ObjTrimStore(bo->wrk, vfc->oc); + return (F_STP_FETCHEND); } /*-------------------------------------------------------------------- @@ -695,8 +696,14 @@ vbf_stp_fetch(struct worker *wrk, struct busyobj *bo) if (bo->htc->body_status != BS_NONE) { assert(bo->htc->body_status != BS_ERROR); - vbf_fetch_body_helper(bo); + return (F_STP_FETCHBODY); } + return (F_STP_FETCHEND); +} + +static enum fetch_step +vbf_stp_fetchend(struct worker *wrk, struct busyobj *bo) +{ VFP_Close(bo->vfc); diff --git a/include/tbl/steps.h b/include/tbl/steps.h index 2d2bf66..62bb7a3 100644 --- a/include/tbl/steps.h +++ b/include/tbl/steps.h @@ -50,6 +50,8 @@ FETCH_STEP(retry, RETRY, (wrk, bo)) FETCH_STEP(startfetch, STARTFETCH, (wrk, bo)) FETCH_STEP(condfetch, CONDFETCH, (wrk, bo)) FETCH_STEP(fetch, FETCH, (wrk, bo)) +FETCH_STEP(fetchbody, FETCHBODY, (bo)) +FETCH_STEP(fetchend, FETCHEND, (wrk, bo)) FETCH_STEP(error, ERROR, (wrk, bo)) FETCH_STEP(fail, FAIL, (wrk, bo)) FETCH_STEP(done, DONE, ()) From phk at FreeBSD.org Fri Oct 21 07:07:05 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Fri, 21 Oct 2016 09:07:05 +0200 Subject: [master] a3b06bb Simplify fetch a little bit more Message-ID: commit a3b06bb2f31a6199f3ea31b44343753bf4a6d55e Author: Poul-Henning Kamp Date: Tue Oct 18 21:11:51 2016 +0000 Simplify fetch a little bit more diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index 4c6fa7d..0d25a8a 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -483,7 +483,7 @@ vbf_stp_startfetch(struct worker *wrk, struct busyobj *bo) */ static enum fetch_step -vbf_stp_fetchbody(struct busyobj *bo) +vbf_stp_fetchbody(struct worker *wrk, struct busyobj *bo) { ssize_t l; uint8_t *ptr; @@ -509,7 +509,7 @@ vbf_stp_fetchbody(struct busyobj *bo) * objects to be created. */ AN(vfc->oc->flags & OC_F_PASS); - VSLb(vfc->wrk->vsl, SLT_FetchError, + VSLb(wrk->vsl, SLT_FetchError, "Pass delivery abandoned"); vfps = VFP_END; bo->htc->doclose = SC_RX_BODY; @@ -535,13 +535,20 @@ vbf_stp_fetchbody(struct busyobj *bo) } } while (vfps == VFP_OK); - if (vfps == VFP_ERROR) { - AN(vfc->failed); + if (vfc->failed) { (void)VFP_Error(vfc, "Fetch pipeline failed to process"); bo->htc->doclose = SC_RX_BODY; + VDI_Finish(wrk, bo); + if (!bo->do_stream) { + assert(bo->fetch_objcore->boc->state < BOS_STREAM); + // XXX: doclose = ? + return (F_STP_ERROR); + } else { + return (F_STP_FAIL); + } } - ObjTrimStore(bo->wrk, vfc->oc); + ObjTrimStore(wrk, vfc->oc); return (F_STP_FETCHEND); } @@ -698,6 +705,7 @@ vbf_stp_fetch(struct worker *wrk, struct busyobj *bo) assert(bo->htc->body_status != BS_ERROR); return (F_STP_FETCHBODY); } + AZ(bo->vfc->failed); return (F_STP_FETCHEND); } @@ -705,19 +713,9 @@ static enum fetch_step vbf_stp_fetchend(struct worker *wrk, struct busyobj *bo) { + AZ(bo->vfc->failed); VFP_Close(bo->vfc); - if (bo->vfc->failed) { - VDI_Finish(bo->wrk, bo); - if (!bo->do_stream) { - assert(bo->fetch_objcore->boc->state < BOS_STREAM); - // XXX: doclose = ? - return (F_STP_ERROR); - } else { - return (F_STP_FAIL); - } - } - AZ(ObjSetU64(wrk, bo->fetch_objcore, OA_LEN, bo->fetch_objcore->boc->len_so_far)); @@ -798,22 +796,7 @@ vbf_stp_condfetch(struct worker *wrk, struct busyobj *bo) VDI_Finish(bo->wrk, bo); return (F_STP_FAIL); } - - AZ(ObjSetU64(wrk, bo->fetch_objcore, OA_LEN, - bo->fetch_objcore->boc->len_so_far)); - - if (!bo->do_stream) - HSH_Unbusy(wrk, bo->fetch_objcore); - - HSH_Kill(bo->stale_oc); - - /* Recycle the backend connection before setting BOS_FINISHED to - give predictable backend reuse behavior for varnishtest */ - VDI_Finish(bo->wrk, bo); - - ObjSetState(wrk, bo->fetch_objcore, BOS_FINISHED); - VSLb_ts_busyobj(bo, "BerespBody", W_TIM_real(wrk)); - return (F_STP_DONE); + return (F_STP_FETCHEND); } /*-------------------------------------------------------------------- diff --git a/include/tbl/steps.h b/include/tbl/steps.h index 62bb7a3..b862a79 100644 --- a/include/tbl/steps.h +++ b/include/tbl/steps.h @@ -50,7 +50,7 @@ FETCH_STEP(retry, RETRY, (wrk, bo)) FETCH_STEP(startfetch, STARTFETCH, (wrk, bo)) FETCH_STEP(condfetch, CONDFETCH, (wrk, bo)) FETCH_STEP(fetch, FETCH, (wrk, bo)) -FETCH_STEP(fetchbody, FETCHBODY, (bo)) +FETCH_STEP(fetchbody, FETCHBODY, (wrk, bo)) FETCH_STEP(fetchend, FETCHEND, (wrk, bo)) FETCH_STEP(error, ERROR, (wrk, bo)) FETCH_STEP(fail, FAIL, (wrk, bo)) From fgsch at lodoss.net Sun Oct 23 22:39:04 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Mon, 24 Oct 2016 00:39:04 +0200 Subject: [master] 911e0ff Test varnishncsa -F error handling Message-ID: commit 911e0ff5a1ac1244a793ac512475308f2ac68be2 Author: Federico G. Schwindt Date: Sun Oct 23 23:33:39 2016 +0100 Test varnishncsa -F error handling This is in preparation for upcoming changes. diff --git a/bin/varnishtest/tests/u00003.vtc b/bin/varnishtest/tests/u00003.vtc new file mode 100644 index 0000000..18e85e9 --- /dev/null +++ b/bin/varnishtest/tests/u00003.vtc @@ -0,0 +1,37 @@ +varnishtest "varnishncsa -F error handling" + +server s1 { + rxreq + txresp +} -start + +varnish v1 -vcl+backend {} -start + +delay 2 + +err_shell "Missing tag in VSL:" \ + {varnishncsa -n ${v1_name} -F "%{VSL:}x" 2>&1} + +err_shell "Unknown log tag: nonexistent" \ + {varnishncsa -n ${v1_name} -F "%{VSL:nonexistent}x" 2>&1} + +err_shell "Tag not unique: Req" \ + {varnishncsa -n ${v1_name} -F "%{VSL:Req}x" 2>&1} + +err_shell "Syntax error: VSL:Begin[" \ + {varnishncsa -n ${v1_name} -F "%{VSL:Begin[}x" 2>&1} + +err_shell "Syntax error: VSL:Begin]" \ + {varnishncsa -n ${v1_name} -F "%{VSL:Begin]}x" 2>&1} + +err_shell "Unknown log tag: Begin[a" \ + {varnishncsa -n ${v1_name} -F "%{VSL:Begin[a}x" 2>&1} + +err_shell "Syntax error: VSL:Begin[a]" \ + {varnishncsa -n ${v1_name} -F "%{VSL:Begin[a]}x" 2>&1} + +err_shell "Syntax error. Field specifier must be positive: Begin[0]" \ + {varnishncsa -n ${v1_name} -F "%{VSL:Begin[0]}x" 2>&1} + +err_shell "Field specifier 999999999999 for the tag VSL:Begin[999999999999] is probably too high" \ + {varnishncsa -n ${v1_name} -F "%{VSL:Begin[999999999999]}x" 2>&1} From fgsch at lodoss.net Sun Oct 23 22:39:04 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Mon, 24 Oct 2016 00:39:04 +0200 Subject: [master] 7d03352 strlcpy no more Message-ID: commit 7d033523b1b9bd17dce44f3d5302e79a0407a7dc Author: Federico G. Schwindt Date: Sun Oct 23 23:35:55 2016 +0100 strlcpy no more diff --git a/flint.lnt b/flint.lnt index 5359c22..9f3f158 100644 --- a/flint.lnt +++ b/flint.lnt @@ -37,7 +37,6 @@ -esym(534, strcat) -esym(534, strcpy) --esym(534, strlcpy) -esym(534, strncpy) -esym(534, sleep) diff --git a/lib/libvarnish/flint.lnt b/lib/libvarnish/flint.lnt index 881e8f8..cc2ed87 100644 --- a/lib/libvarnish/flint.lnt +++ b/lib/libvarnish/flint.lnt @@ -44,7 +44,6 @@ -esym(534, vsb_putc) // Ignoring return value of function -esym(534, strcat) // Ignoring return value of function -esym(534, strcpy) // Ignoring return value of function --esym(534, strlcpy) // Ignoring return value of function -emacro(506, isnan) // constant value boolean -emacro(747, isnan) // significant coersion From fgsch at lodoss.net Sun Oct 23 22:58:04 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Mon, 24 Oct 2016 00:58:04 +0200 Subject: [master] c447221 Use test from the correct branch Message-ID: commit c447221352fd803d24fed297fb23e002843eca00 Author: Federico G. Schwindt Date: Sun Oct 23 23:57:01 2016 +0100 Use test from the correct branch diff --git a/bin/varnishtest/tests/u00003.vtc b/bin/varnishtest/tests/u00003.vtc index 18e85e9..e76ed03 100644 --- a/bin/varnishtest/tests/u00003.vtc +++ b/bin/varnishtest/tests/u00003.vtc @@ -18,7 +18,7 @@ err_shell "Unknown log tag: nonexistent" \ err_shell "Tag not unique: Req" \ {varnishncsa -n ${v1_name} -F "%{VSL:Req}x" 2>&1} -err_shell "Syntax error: VSL:Begin[" \ +err_shell "Unknown log tag: Begin[" \ {varnishncsa -n ${v1_name} -F "%{VSL:Begin[}x" 2>&1} err_shell "Syntax error: VSL:Begin]" \ From guillaume at varnish-software.com Mon Oct 24 09:21:05 2016 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Mon, 24 Oct 2016 11:21:05 +0200 Subject: [master] ee43d2e Make clear that .host_header is a fallback value Message-ID: commit ee43d2e6ca0b665544beae452bf5f21d8548a067 Author: Guillaume Quintard Date: Mon Oct 24 11:08:11 2016 +0200 Make clear that .host_header is a fallback value diff --git a/doc/sphinx/reference/vcl.rst b/doc/sphinx/reference/vcl.rst index 53fda35..d9f10fc 100644 --- a/doc/sphinx/reference/vcl.rst +++ b/doc/sphinx/reference/vcl.rst @@ -221,7 +221,8 @@ are available: The port on the backend that Varnish should connect to. host_header - A host header to add. + A host header to add to probes and regular backend requests if they have no + such header. connect_timeout Timeout for connections. From hermunn at varnish-software.com Mon Oct 24 11:27:05 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Mon, 24 Oct 2016 13:27:05 +0200 Subject: [4.0] 25ce7db Prepare for 4.0.4-beta1 Message-ID: commit 25ce7db20503e7efa841210a7b2021b0605cadd7 Author: P?l Hermunn Johansen Date: Mon Oct 24 11:36:31 2016 +0200 Prepare for 4.0.4-beta1 diff --git a/configure.ac b/configure.ac index 765a8b4..886e645 100644 --- a/configure.ac +++ b/configure.ac @@ -1,8 +1,8 @@ AC_PREREQ(2.59) AC_COPYRIGHT([Copyright (c) 2006 Verdens Gang AS -Copyright (c) 2006-2015 Varnish Software AS]) +Copyright (c) 2006-2016 Varnish Software AS]) AC_REVISION([$Id$]) -AC_INIT([Varnish], [4.0.3], [varnish-dev at varnish-cache.org]) +AC_INIT([Varnish], [4.0.4], [varnish-dev at varnish-cache.org]) AC_CONFIG_SRCDIR(include/miniobj.h) AC_CONFIG_HEADERS([config.h]) AC_CONFIG_MACRO_DIR([m4]) diff --git a/doc/changes.rst b/doc/changes.rst index 8d8932c..2ec9540 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -1,7 +1,7 @@ -================================== -Changes from 4.0.3 to (unreleased) -================================== +============================================== +Changes from 4.0.3 to 4.0.4-beta1 (2016-10-24) +============================================== * Percent-based storage size configuration was broken and has been retired. From hermunn at varnish-software.com Mon Oct 24 12:34:05 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Mon, 24 Oct 2016 14:34:05 +0200 Subject: [4.0] 8eb622f Update changelog Message-ID: commit 8eb622f29afad18683cfbd0360ecb0f0996098c3 Author: P?l Hermunn Johansen Date: Mon Oct 24 14:31:42 2016 +0200 Update changelog Add changelog entry for #1879. diff --git a/doc/changes.rst b/doc/changes.rst index 2ec9540..5fa8414 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -12,6 +12,7 @@ Changes from 4.0.3 to 4.0.4-beta1 (2016-10-24) Bugs fixed ---------- +.. _1879: https://www.varnish-cache.org/trac/ticket/1879 .. _1768: https://www.varnish-cache.org/trac/ticket/1768 .. _1815: https://www.varnish-cache.org/trac/ticket/1815 .. _1862: https://www.varnish-cache.org/trac/ticket/1862 @@ -22,6 +23,7 @@ Bugs fixed .. _1602: https://www.varnish-cache.org/trac/ticket/1602 .. _1598: https://www.varnish-cache.org/trac/ticket/1598 +- 1879_ - Wrong set of duplicate haeaders when backend 304 NOT MODIFIED - 1768_ - Properly encode HTTP headers with weird characters to C identifiers. - 1815_ - Close piped sessions in the absence of backends - 1862_ - Request URL with whitespace is allowed From phk at FreeBSD.org Mon Oct 24 14:55:06 2016 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 24 Oct 2016 16:55:06 +0200 Subject: [master] 107c4a2 Bail out of CLI reads on signals Message-ID: commit 107c4a288187d69a3d9c5e5f406a61d7b2c2cce9 Author: Poul-Henning Kamp Date: Mon Oct 24 14:53:27 2016 +0000 Bail out of CLI reads on signals Spotted & prodded by: Hermunn Closes: #2026 diff --git a/lib/libvarnish/vcli_proto.c b/lib/libvarnish/vcli_proto.c index 7e52fb2..9e94fc6 100644 --- a/lib/libvarnish/vcli_proto.c +++ b/lib/libvarnish/vcli_proto.c @@ -119,6 +119,10 @@ read_tmo(int fd, char *ptr, unsigned len, double tmo) pfd.events = POLLIN; for (j = 0; len > 0; ) { i = poll(&pfd, 1, to); + if (i < 0) { + errno = EINTR; + return (-1); + } if (i == 0) { errno = ETIMEDOUT; return (-1); From hermunn at varnish-software.com Tue Oct 25 08:36:05 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Tue, 25 Oct 2016 10:36:05 +0200 Subject: [4.1] 78be1d7 Kill any temporary vsm files on failure Message-ID: commit 78be1d7d91c1a4e8660824fb620d76ae8484bb00 Author: Federico G. Schwindt Date: Fri Oct 14 11:04:46 2016 +0100 Kill any temporary vsm files on failure Use MGT_complain() to log any errors; we might have a better chance to understand what's going in. Fixes #2115. diff --git a/bin/varnishd/mgt/mgt_shmem.c b/bin/varnishd/mgt/mgt_shmem.c index 7d941ee..c12913f 100644 --- a/bin/varnishd/mgt/mgt_shmem.c +++ b/bin/varnishd/mgt/mgt_shmem.c @@ -152,7 +152,7 @@ vsm_zerofile(const char *fn, ssize_t size) fd = flopen(fn, O_RDWR | O_CREAT | O_EXCL | O_NONBLOCK, 0640); if (fd < 0) { - fprintf(stderr, "Could not create %s: %s\n", + MGT_complain(C_ERR, "Could not create %s: %s", fn, strerror(errno)); return (-1); } @@ -162,7 +162,7 @@ vsm_zerofile(const char *fn, ssize_t size) flags &= ~O_NONBLOCK; AZ(fcntl(fd, F_SETFL, flags)); if (VFIL_allocate(fd, (off_t)size, 1)) { - fprintf(stderr, "File allocation error %s: %s\n", + MGT_complain(C_ERR, "File allocation error %s: %s", fn, strerror(errno)); return (-1); } @@ -184,6 +184,17 @@ mgt_shm_size(void) return (size); } +static void +mgt_shm_cleanup(void) +{ + char fnbuf[64]; + + bprintf(fnbuf, "%s.%jd", VSM_FILENAME, (intmax_t)getpid()); + VJ_master(JAIL_MASTER_FILE); + (void)unlink(fnbuf); + VJ_master(JAIL_MASTER_LOW); +} + void mgt_SHM_Create(void) { @@ -200,8 +211,10 @@ mgt_SHM_Create(void) VJ_master(JAIL_MASTER_FILE); vsm_fd = vsm_zerofile(fnbuf, size); VJ_master(JAIL_MASTER_LOW); - if (vsm_fd < 0) + if (vsm_fd < 0) { + mgt_shm_cleanup(); exit(1); + } p = (void *)mmap(NULL, size, PROT_READ|PROT_WRITE, @@ -211,7 +224,9 @@ mgt_SHM_Create(void) AZ(close(vsm_fd)); if (p == MAP_FAILED) { - fprintf(stderr, "Mmap error %s: %s\n", fnbuf, strerror(errno)); + MGT_complain(C_ERR, "Mmap error %s: %s", + fnbuf, strerror(errno)); + mgt_shm_cleanup(); exit(1); } From hermunn at varnish-software.com Tue Oct 25 09:09:04 2016 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Tue, 25 Oct 2016 11:09:04 +0200 Subject: [4.1] c330106 Bail out of CLI reads on signals Message-ID: commit c3301061ee62ec61f0672630e67ab4fc4bf81076 Author: Poul-Henning Kamp Date: Mon Oct 24 14:53:27 2016 +0000 Bail out of CLI reads on signals Spotted & prodded by: Hermunn Closes: #2026 diff --git a/lib/libvarnish/cli_common.c b/lib/libvarnish/cli_common.c index d14611f..895705f 100644 --- a/lib/libvarnish/cli_common.c +++ b/lib/libvarnish/cli_common.c @@ -155,6 +155,10 @@ read_tmo(int fd, char *ptr, unsigned len, double tmo) pfd.events = POLLIN; for (j = 0; len > 0; ) { i = poll(&pfd, 1, to); + if (i < 0) { + errno = EINTR; + return (-1); + } if (i == 0) { errno = ETIMEDOUT; return (-1); From varnish-commit at varnish-cache.org Tue Oct 25 14:11:21 2016 From: varnish-commit at varnish-cache.org (ocolzitqm) Date: Tue, 25 Oct 2016 14:11:21 -0000 Subject: =?utf-8?B?dmFybmlzaC1jb21taXQtLeWmguS9leWcqOS/nQ==?= =?utf-8?B?6K+B5Lqn5ZOB6LSo6YeP55qE5ZCM5pe25Y+I6KaB6ZmN5L2O5Lqn5ZOB55qE56CU5Y+R6LS555So5ZKM?= =?utf-8?B?6K6+6K6h5oiQ5pys77yf?= Message-ID: <20151026213815523283@xlncvzfok.org> ??????????????????? ??????2016?10?27-28 ???10?31-11?1? ??,11?03-04??? ????????CEO/?????????/???????/???????????/?????PMO???????????????????????? ?????? ???? + ???? + ???? +???? + ???? ?????? 3200?/????????????????????? ?????? 010-5129-9910,021-3100-6787 ,0755-6128-0006 ?QQ???? 320588808 18917870808 ??? ????: ?????????/????????????????????????????? 1.?????????????????????????????? 2.????????????????????????????????????????? 3.???????????????????????????????????? 4.???????????????????????????????????????????????????????? 5.????????????????????????????????????????????????????????????????????????????????????? 6.???????????????????? 7.????????????????????????????? 8.?????????????????????????????????? 9.?? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? ????: 1.????300???????????????????????????????????????? 2.???????????????????????????????????????? 3.??????????????????????????????????? 4.??????????????????????? 5.??????????????????????????????? 6.????????50??????????????????????????????????Action Plan?????????????????????????????? ???? ?????? ?????????? 1???????????????????????????IT???? 2???????????? 3???????????????NPD?CMMI?ISO9000???? 4???????????????? 5???????? 6???????????????????????????????? 7???????????????? 8?????????????????? 9??????????????? 10???????? ???????????? 1?????????????? 2?????????? a)????? b)????? c)????? 3????????????????? 4?????????????????? a)??????????????????????????????? b)????????????????????? c)????????????????? d)????????????????? e)??????????????????????? 5???????????????????? a)?????????? b)???? c)????????????? d)???????????? 6??????????????? 7????????????????????? 8????????????????????????? 9??????????????/????????? 10???????? ???????????????????????????? 1?????????????????? a)?????????? b)???????????? c)??????????????????????? d)?????????????? e)?????????????? 2????????????? 3?????????????????????????? 4??????????????????????????????????? 5????????????????? 6?????????????? 7???????????? 8???????????? 9?????????????? 10??????????????? 11?????????PMO? 12?????????????? 13????????????????????? 14???????? ???????????? 1?????????????? 2????????????????? 3???????????? a??????????? b??????????????????? c??????????????????? d?????????????????? e??????????????????????????? 4???????????????????? a)?????? b)?????? c)??????????? d)??????????????? 5??????????????????????? ? ????????????????? 6????????????????????????????? 7???????? ?????????? 1???????? a) ?????? b) ?????? 2???????? a) ????????????????????? b)????????? c)??????????? d)????????? e)????????? f)??????????????? g)?????????????? h)WBS????????? i)WBS??????? j)PBS?WBS?OBS?RBS??????? k)????????? l)??????????? m)PERT???? n)?????????? ????? ????? o)??????? 3????????????????WBS????????? ?????????? 1??????????????????? 2????????????? 3????? a??????? b???????? c???????? 4?????????????? a) ?????? b) ?????? 5?????????????? a) ?????? b) ??????? 6???????????????? a) ?????? b) ?????? 7?????????????? 8???????????????? 9????????????????? 10?????????????? 11???????????????? 12???????????????? a) ????? b) ????? 13?????????????? 14?????????????? 15?????????????? a) ???? b) ???????? c) ???? 16??????????QA???? 17?????????????? a) ????? b) ??????? c) ???? 18?????????????? 19?????????????? a) ???????? b) ???? c) ???????? 20???????? ??????????????????? 1???????????????????????????????????????? 2???????????????????????? 3???????????? 4???????????? 5??????????? 6?????????????? 7?????????????? ???? Don ???????????INCOSE???????????www.incose.org??? ???? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????BFL?Infosys??????????????CMM??????????????????????????????? ???????? ?????????????10??????????????????????????????????????????????????????????? 1)???? 2)???? 3)???? 4)?? 5)????? 6)???? 7)????? 8)???? 9)???? 10)????? ????????? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????TCL?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????100????????????????????????????????????????????? ?????????? ???????????????2000?????????????????????80?????????????????????????????????????????????????????????????????????????????????????????? ?????: ????????????????????????????????????????????TCL???????????????????????????????????????????????????????TCL??????????????????????????????????PBI?????????????????????????????????????????????????????????? ?????????? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? ??????? ????????????????????TCL???????????????????????????????????????????????????????????????????????????????????????????????????????? ?????? ???????????????????????????????????TCL???????????????????????????????????????????????????????????????????????????????? ?????? ??????????????/???????????????????????????????????????????????????????????????????????????????????? ???????? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????OTIS????????????????????????????? ????????? ????????????????????????????????????????????????????????? ??????????? ???????????????????????????????????????????????????????????????????????????????????????????????????????7?????????43?????????29??????????????????????????????? ?????????? ???????????????????????????????????????????????????????????????????????? ?????? ????????????????????????????????????????????????????????????????????? ?????? ???????????????????????????????????????????????????????????????? ?? ?????????? ????????????????? -------------- next part -------------- An HTML attachment was scrubbed... URL: From nils.goroll at uplex.de Tue Oct 25 14:39:04 2016 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 25 Oct 2016 16:39:04 +0200 Subject: [master] 92b7585 document new fetch sub-states Message-ID: commit 92b7585006fe0a6130454fe27427beb23d8529be Author: Nils Goroll Date: Tue Oct 25 16:36:02 2016 +0200 document new fetch sub-states this reflects changes from 5b2888f3556a0cffc7d89d5db99e97f467eb7a17 and a3b06bb2f31a6199f3ea31b44343753bf4a6d55e diff --git a/doc/graphviz/cache_fetch.dot b/doc/graphviz/cache_fetch.dot index d477a7b..f6fc959 100644 --- a/doc/graphviz/cache_fetch.dot +++ b/doc/graphviz/cache_fetch.dot @@ -56,13 +56,30 @@ digraph cache_fetch { v_b_r_retry [label="RETRY",shape=plaintext] + vbf_stp_fetchbody [ + shape=record + fontcolor=grey + color=grey + label="{vbf_stp_fetchbody:|get storage|read body, run VFPs|{fetch_fail?|error?|ok?}}" + ] + vbf_stp_fetchbody:ok:s -> vbf_stp_fetchend + vbf_stp_fetch [ shape=record fontcolor=grey color=grey - label="{vbf_stp_fetch:|setup VFPs|fetch|{fetch_fail?|error?|ok?}}" + label="{vbf_stp_fetch:|setup VFPs|get object|{error?|body?}}" + ] + vbf_stp_fetch:body:s -> vbf_stp_fetchbody + vbf_stp_fetch:body:s -> vbf_stp_fetchend + + vbf_stp_fetchend [ + shape=record + fontcolor=grey + color=grey + label="{vbf_stp_fetchend:|finalize object and director|done}" ] - vbf_stp_fetch:ok:s -> FETCH_DONE + vbf_stp_fetchend:done:s -> FETCH_DONE vbf_stp_condfetch [ shape=record @@ -71,7 +88,7 @@ digraph cache_fetch { label="{vbf_stp_condfetch:|copy obj attr|steal body|{fetch_fail?|ok?}}" ] - vbf_stp_condfetch:ok:s -> FETCH_DONE + vbf_stp_condfetch:ok:s -> vbf_stp_fetchend error [shape=plaintext] error -> FETCH_FAIL diff --git a/doc/graphviz/cache_fetch.svg b/doc/graphviz/cache_fetch.svg index ecd062a..a79dea9 100644 --- a/doc/graphviz/cache_fetch.svg +++ b/doc/graphviz/cache_fetch.svg @@ -4,237 +4,274 @@ - - + + cache_fetch - + cluster_backend - + RETRY -RETRY +RETRY v_b_f - -vbf_stp_startfetch: - -vcl_backend_fetch{} - -bereq.* - -abandon - -fetch + +vbf_stp_startfetch: + +vcl_backend_fetch{} + +bereq.* + +abandon + +fetch RETRY->v_b_f - - + + v_b_f_BGFETCH - -BGFETCH + +BGFETCH v_b_f_BGFETCH->v_b_f - - + + v_b_f_FETCH - -FETCH + +FETCH v_b_f_FETCH->v_b_f - - + + v_b_f_FETCH->v_b_f - - + + v_b_hdrs - -send bereq, -read beresp (headers) + +send bereq, +read beresp (headers) v_b_f:fetch:s->v_b_hdrs - - + + v_b_r - -vbf_stp_startfetch: - -vcl_backend_response{} - -bereq.* - -beresp.* - -retry - -max? - -ok? - -abandon - -deliver - -304? - -other? + +vbf_stp_startfetch: + +vcl_backend_response{} + +bereq.* + +beresp.* + +retry + +max? + +ok? + +abandon + +deliver + +304? + +other? v_b_hdrs->v_b_r - - + + v_b_e - -vbf_stp_error: - -vcl_backend_error{} - -bereq.* - -beresp.* - -retry - -max? - -ok? - -abandon - -deliver + +vbf_stp_error: + +vcl_backend_error{} + +bereq.* + +beresp.* + +retry + +max? + +ok? + +abandon + +deliver v_b_hdrs->v_b_e - - + + v_b_r:max->v_b_e - - + + v_b_r_retry -RETRY +RETRY v_b_r:retry->v_b_r_retry - - + + vbf_stp_condfetch - -vbf_stp_condfetch: - -copy obj attr - -steal body - -fetch_fail? - -ok? + +vbf_stp_condfetch: + +copy obj attr + +steal body + +fetch_fail? + +ok? v_b_r:fetch_304:s->vbf_stp_condfetch - - + + vbf_stp_fetch - -vbf_stp_fetch: - -setup VFPs - -fetch - -fetch_fail? - -error? - -ok? + +vbf_stp_fetch: + +setup VFPs + +get object + +error? + +body? v_b_r:non_304:s->vbf_stp_fetch - - + + -FETCH_DONE - -FETCH_DONE +FETCH_DONE + +FETCH_DONE -v_b_e:deliver->FETCH_DONE - - -"backend synth" +v_b_e:deliver->FETCH_DONE + + +"backend synth" -FETCH_FAIL - -FETCH_FAIL +FETCH_FAIL + +FETCH_FAIL -v_b_e:max:s->FETCH_FAIL - - +v_b_e:max:s->FETCH_FAIL + + -v_b_e_retry -RETRY +v_b_e_retry +RETRY -v_b_e:retry->v_b_e_retry - - - - -vbf_stp_condfetch:ok:s->FETCH_DONE - - - - -vbf_stp_fetch:ok:s->FETCH_DONE - - +v_b_e:retry->v_b_e_retry + + + + +vbf_stp_fetchend + +vbf_stp_fetchend: + +finalize object and director + +done + + +vbf_stp_condfetch:ok:s->vbf_stp_fetchend + + + + +vbf_stp_fetchbody + +vbf_stp_fetchbody: + +get storage + +read body, run VFPs + +fetch_fail? + +error? + +ok? + + +vbf_stp_fetch:body:s->vbf_stp_fetchbody + + + + +vbf_stp_fetch:body:s->vbf_stp_fetchend + + + + +vbf_stp_fetchbody:ok:s->vbf_stp_fetchend + + + + +vbf_stp_fetchend:done:s->FETCH_DONE + + -error -error +error +error -error->FETCH_FAIL - - +error->FETCH_FAIL + + -abandon -abandon +abandon +abandon -abandon->FETCH_FAIL - - +abandon->FETCH_FAIL + + From fgsch at lodoss.net Tue Oct 25 14:58:05 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Tue, 25 Oct 2016 16:58:05 +0200 Subject: [master] 5db2901 Revert to using stderr for now Message-ID: commit 5db290135305260e7fffffd996ef087d6293eb5d Author: Federico G. Schwindt Date: Tue Oct 25 15:46:57 2016 +0100 Revert to using stderr for now We need to veirfy this apparently innocuous change does not have any side effects. diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c index 00bb7e2..a31d1c7 100644 --- a/bin/varnishd/mgt/mgt_main.c +++ b/bin/varnishd/mgt/mgt_main.c @@ -769,7 +769,7 @@ main(int argc, char * const *argv) if (Cn_arg == n_arg) AZ(rmdir(Cn_arg)); AZ(VSB_finish(cli->sb)); - printf("%s\n", VSB_data(cli->sb)); + fprintf(stderr, "%s\n", VSB_data(cli->sb)); exit(cli->result == CLIS_OK ? 0 : 2); } cli_check(cli); From fgsch at lodoss.net Wed Oct 26 14:14:05 2016 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Wed, 26 Oct 2016 16:14:05 +0200 Subject: [master] b21b0d7 Ident and type are optional Message-ID: commit b21b0d7b326b18a808c26d4687fe87911a94f4fa Author: Federico G. Schwindt Date: Wed Oct 26 14:51:57 2016 +0100 Ident and type are optional Update the code to accommodate this. Originally reported by Reinis Rozitis (r at roze.lv) on varnish-misc. Fixes #2118. diff --git a/bin/varnishtest/tests/u00002.vtc b/bin/varnishtest/tests/u00002.vtc index d6aa4ef..173c9cd 100644 --- a/bin/varnishtest/tests/u00002.vtc +++ b/bin/varnishtest/tests/u00002.vtc @@ -24,3 +24,7 @@ shell "grep -q mempool ${tmpdir}/p2/stdout" err_shell "" "grep -q LCK.vbe.destroy ${tmpdir}/p2/stdout" shell "cmp -s ${tmpdir}/p1/stdout ${tmpdir}/p2/stdout" + +process p3 {varnishstat -1 -n ${v1_name} -f MAIN.cache_hit} -run + +shell "grep -q cache_hit ${tmpdir}/p3/stdout" diff --git a/lib/libvarnishapi/vsc.c b/lib/libvarnishapi/vsc.c index 6af266e..67da9d6 100644 --- a/lib/libvarnishapi/vsc.c +++ b/lib/libvarnishapi/vsc.c @@ -379,10 +379,13 @@ vsc_filter_pt_list(struct VSM_data *vd) VTAILQ_FOREACH_SAFE(pt, &vsc->pt_list, list, pt2) { CHECK_OBJ_NOTNULL(pt, VSC_PT_MAGIC); VSB_clear(vsb); - VSB_printf(vsb, "%s.%s.%s", - pt->point.section->type, - pt->point.section->ident, - pt->point.desc->name); + if (strcmp(pt->point.section->type, "")) + VSB_printf(vsb, "%s.", + pt->point.section->type); + if (strcmp(pt->point.section->ident, "")) + VSB_printf(vsb, "%s.", + pt->point.section->ident); + VSB_printf(vsb, "%s", pt->point.desc->name); VSB_finish(vsb); if (fnmatch(sf->pattern, VSB_data(vsb), 0)) continue; From nils.goroll at uplex.de Wed Oct 26 14:55:05 2016 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 26 Oct 2016 16:55:05 +0200 Subject: [master] 764f988 sync comment with reality Message-ID: commit 764f9887da9e607f1e969840090af2bd0c31119c Author: Nils Goroll Date: Wed Oct 26 16:36:23 2016 +0200 sync comment with reality diff --git a/bin/varnishd/cache/cache_wrk.c b/bin/varnishd/cache/cache_wrk.c index 7176f09..2421b44 100644 --- a/bin/varnishd/cache/cache_wrk.c +++ b/bin/varnishd/cache/cache_wrk.c @@ -416,13 +416,13 @@ pool_breed(struct pool *qp) /*-------------------------------------------------------------------- * Herd a single pool * - * This thread wakes up whenever a pool queues. + * This thread wakes every 5 seconds and whenever a pool queues. * - * The trick here is to not be too aggressive about creating threads. - * We do this by only examining one pool at a time, and by sleeping - * a short while whenever we create a thread and a little while longer - * whenever we fail to, hopefully missing a lot of cond_signals in - * the meantime. + * The trick here is to not be too aggressive about creating threads. In + * pool_breed(), we sleep whenever we create a thread and a little while longer + * whenever we fail to, hopefully missing a lot of cond_signals in the meantime. + * + * Idle threads are destroyed at a rate termined by wthread_destroy_delay * * XXX: probably need a lot more work. * From nils.goroll at uplex.de Wed Oct 26 14:55:05 2016 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 26 Oct 2016 16:55:05 +0200 Subject: [master] 1138f10 fix thread name for pool_poolherder, add it for pool_herder Message-ID: commit 1138f10b695ecc6fad7560df28a03208de9da8ae Author: Nils Goroll Date: Wed Oct 26 16:29:33 2016 +0200 fix thread name for pool_poolherder, add it for pool_herder diff --git a/bin/varnishd/cache/cache_pool.c b/bin/varnishd/cache/cache_pool.c index d3deed2..f337714 100644 --- a/bin/varnishd/cache/cache_pool.c +++ b/bin/varnishd/cache/cache_pool.c @@ -190,7 +190,7 @@ pool_poolherder(void *priv) struct pool *pp; uint64_t u; - THR_SetName("pool_herder"); + THR_SetName("pool_poolherder"); (void)priv; nwq = 0; diff --git a/bin/varnishd/cache/cache_wrk.c b/bin/varnishd/cache/cache_wrk.c index 2421b44..5549408 100644 --- a/bin/varnishd/cache/cache_wrk.c +++ b/bin/varnishd/cache/cache_wrk.c @@ -438,6 +438,8 @@ pool_herder(void *priv) CAST_OBJ_NOTNULL(pp, priv, POOL_MAGIC); + THR_SetName("pool_herder"); + while (1) { /* Make more threads if needed and allowed */ if (pp->nthr < cache_param->wthread_min || From varnish-commit at varnish-cache.org Wed Oct 26 19:23:02 2016 From: varnish-commit at varnish-cache.org (ve) Date: Thu, 27 Oct 2016 03:23:02 +0800 Subject: =?utf-8?B?dmFybmlzaC1jb21taXQtLeS4reWxguS4jeWQjA==?= =?utf-8?B?6Zi25q6155qE5a6a5L2N77yf?= Message-ID: <20161027032311442670@njdcgkm.org> varnish-commit:? ?? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? ???????????? ?????????????? 2016/10/27 ???3:23:10 zrlra -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ????6??.docx Type: application/vnd.openxmlformats-officedocument.wordprocessingml.document Size: 33384 bytes Desc: not available URL: From nils.goroll at uplex.de Thu Oct 27 11:16:04 2016 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 27 Oct 2016 13:16:04 +0200 Subject: [master] 0743434 comment typo Message-ID: commit 074343416071b694d61edc7ee52bb1905ea35140 Author: Nils Goroll Date: Thu Oct 27 06:51:40 2016 +0200 comment typo diff --git a/bin/varnishd/cache/cache_wrk.c b/bin/varnishd/cache/cache_wrk.c index 5549408..4e8af44 100644 --- a/bin/varnishd/cache/cache_wrk.c +++ b/bin/varnishd/cache/cache_wrk.c @@ -422,7 +422,7 @@ pool_breed(struct pool *qp) * pool_breed(), we sleep whenever we create a thread and a little while longer * whenever we fail to, hopefully missing a lot of cond_signals in the meantime. * - * Idle threads are destroyed at a rate termined by wthread_destroy_delay + * Idle threads are destroyed at a rate determined by wthread_destroy_delay * * XXX: probably need a lot more work. * From varnish-commit at varnish-cache.org Sun Oct 30 01:26:45 2016 From: varnish-commit at varnish-cache.org (jkruwyca) Date: Sun, 30 Oct 2016 09:26:45 +0800 Subject: =?utf-8?B?dmFybmlzaC1jb21taXTkuqcg5ZOBIOW4giA=?= =?utf-8?B?5Zy6IOeuoSDnkIY=?= Message-ID: <20161030092655161413@ahkd.org> ? ? ? ? ? ? ?????? 2016?11?03-04??? ???? ?????? ??CEO/???????????????????????/?????/???????????????? ?????? ???? + ???? + ???? +???? + ???? ?????? 3200?/????????????????????? ?????? 021-31006787 18917870808 ??? ?QQ???? 320588808 ????? ???????????????????????????????????????????????? 1????????????????????????? 2???????????????????/??????? 3???????/???????????????? 4??????????????????????????????????? 5??????????????????????? 6?????????????????????????????? 7?????????????????? 8?????????????????????????? 9??????????????? 10???????????????????????? 11??? ??????????????????????????????????????????????????????????????????????????????????????? ?????????????????????????????????????????????????????????????????????? ????? 1)???????????????????????? 2)???????????????????????? 3)?????????????????????????????? 4)????????????????????????????????????????? 5)???????????????????????????????????????? 6)???????????????????????????????? ????? ?????? 1.???? 2.???? 3.???? ?????????? 1.????????? 1)???????? 2)??????????? 3)????????????????? 4)????????????? 5)???????????????????? 2.?????? 1)????????Marketing Management?? 2)????????????? 3)?????/???????Product/Technology Marketing Management?? 4)??????????????????????????? 5)????????????????? 6)?????????????????????? 7)???????????????????? ??????????----???? 1.?????????????/?? 2.???? 1)???? 2)???? 3)???? ???$APPEALS ?$APPEALS???????? ?$APPEALS???? ?????????? ??????$APPEALS????????? 4)?????? 3.?????? 1)????? 2)????? 3)???? 4)?????? 5)?????????????? ??????????----???? 1.?????????????/?? 2.??????? 3.?????????? 4.???????? 1)???? 2)?????? 3)???? 4)??????? 5)?????? 5.?????????????????? ??????????----?????? 1.?????????????/?? 2.?????? 3.????????????? 4.?????? 5.??????? 1)????Ansolf???????? 2)?????? 3)???? 4)??????? 5)?????? 6)?????? 7)??? 8)???? 9)?????????? 6.?????????????? 1)???????????????? 2)??????????? 3)??????????? 4)??????????????? 5)???? 6)???????????????? ??????????? 1.???????? 2.???????????????????????????????????????? 3.??????????? 4.?????????? 1)???????????????????? 2)????????? 3)??????V/R/M?????????????? 4)??????????????????? 5.????????????? 1)???????????????????? 2)???????????? 6.????????????????????? 1)???????????????????? 2)????????????????????? 7.????????????????????? 8.??????????????????? ???? ???? ?Charles ?????????????????????????????????????????????? ????? 16???????????????????????????????????????????????????????????NPI????????????????????????????? ???????????????????97?04????????????????????????????????????????????????????????????????????????????????????? ????????? 7??????????????20??????????????????????????????????????????????????????????????????????????????????IT????????????? 1)???? 2)OPPO 3)TCL????? 4)???? 5)???? 6)???? 7)???? 8)?????????? ????????? ?????????????????TCL????????OPPO??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????500???????????????? ?????????? ????????????2000?????????????????????80?????????????????????????????????????????????????????????????????????????????????????????? ?????: ????????????????????????????????????????????TCL???????????????????????????????????????????????????????TCL??????????????????????????????????PBI?????????????????????????????????????????????????????????? ?????????? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? ??????? ????????????????????TCL???????????????????????????????????????????????????????????????????????????????????????????????????????? ?????? ???????????????????????????????????TCL???????????????????????????????????????????????????????????????????????????????? ?????? ??????????????/???????????????????????????????????????????????????????????????????????????????????? ???????? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????OTIS????????????????????????????? ????????? ????????????????????????????????????????????????????????? ??????????? ???????????????????????????????????????????????????????????????????????????????????????????????????????7?????????43?????????29??????????????????????????????? ?????????? ???????????????????????????????????????????????????????????????????????? ?????? ????????????????????????????????????????????????????????????????????? ?????? ???????????????????????????????????????????????????????????????? ?? -------------- next part -------------- An HTML attachment was scrubbed... URL: From nils.goroll at uplex.de Mon Oct 31 13:12:05 2016 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 31 Oct 2016 14:12:05 +0100 Subject: [master] 66e9ab7 clarify the per-backend ramup parameter magic default value Message-ID: commit 66e9ab78235028cacef1bbfdbb4957e6c92fa8b7 Author: Nils Goroll Date: Mon Oct 31 14:11:32 2016 +0100 clarify the per-backend ramup parameter magic default value diff --git a/lib/libvmod_directors/vmod.vcc b/lib/libvmod_directors/vmod.vcc index c3e2a8a..c0fd1b8 100644 --- a/lib/libvmod_directors/vmod.vcc +++ b/lib/libvmod_directors/vmod.vcc @@ -360,8 +360,9 @@ backend name. `ident` allows to add multiple instances of the same backend. -`rampup`: Optionally specify a rampup time for this backend. Defaults -to the director rampup time. +`rampup`: Optionally specify a specific rampup time for this +backend. The magic default value of `973279260s` instructs the shard +director to use the default rampup time (see :ref:`func_shard.set_rampup`). NOTE: Backend changes need to be finalized with `shard.reconfigure()` and are only supported on one shard director at a time. From accounts at varnish-cache.org Mon Oct 31 13:20:36 2016 From: accounts at varnish-cache.org (REYNALDO MCCLUNG) Date: Mon, 31 Oct 2016 20:20:36 +0700 Subject: Document No 987180938 Message-ID: Thanks for using electronic billing Please find your document attached Regards REYNALDO MCCLUNG -------------- next part -------------- A non-text attachment was scrubbed... Name: Document 987180938.zip Type: application/zip Size: 7911 bytes Desc: not available URL: From accounts at varnish-cache.org Mon Oct 31 13:36:25 2016 From: accounts at varnish-cache.org (NORMA FULCHER) Date: Mon, 31 Oct 2016 19:06:25 +0530 Subject: Document No 33078198 Message-ID: Thanks for using electronic billing Please find your document attached Regards NORMA FULCHER -------------- next part -------------- A non-text attachment was scrubbed... Name: Document No 33078198.zip Type: application/zip Size: 7918 bytes Desc: not available URL: From phk at phk.freebsd.dk Mon Oct 31 13:49:22 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 31 Oct 2016 13:49:22 +0000 Subject: [master] 66e9ab7 clarify the per-backend ramup parameter magic default value In-Reply-To: References: Message-ID: <10309.1477921762@critter.freebsd.dk> -------- In message , Nils Goroll writes: >+`rampup`: Optionally specify a specific rampup time for this >+backend. The magic default value of `973279260s` instructs the shard >+director to use the default rampup time (see :ref:`func_shard.set_rampup`). That is an incredibly ugly way to do it IMO, but I guess it ensures that people read the manual... :-) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From nils.goroll at uplex.de Mon Oct 31 13:51:46 2016 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 31 Oct 2016 14:51:46 +0100 Subject: [master] 66e9ab7 clarify the per-backend ramup parameter magic default value In-Reply-To: <10309.1477921762@critter.freebsd.dk> References: <10309.1477921762@critter.freebsd.dk> Message-ID: <9cd47d1a-1cae-4115-1633-c94873596e5e@uplex.de> On 31/10/16 14:49, Poul-Henning Kamp wrote: >> >+`rampup`: Optionally specify a specific rampup time for this >> >+backend. The magic default value of `973279260s` instructs the shard >> >+director to use the default rampup time (see :ref:`func_shard.set_rampup`). > That is an incredibly ugly way to do it IMO, but I guess it ensures > that people read the manual... :-) The magic value is not intended to be used directly, it is only intended to be just that - the default value. I've changed the explanation because the vcc to rst python exposes the value and someone has asked, what that strange number is. Other than that, if there's a better way, I'm all ears. Nils -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg tel +49 40 28805731 mob +49 170 2723133 fax +49 40 42949753 xmpp://slink at jabber.int.uplex.de/ http://uplex.de/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 473 bytes Desc: OpenPGP digital signature URL: From accounts at varnish-cache.org Mon Oct 31 15:21:52 2016 From: accounts at varnish-cache.org (DAMIAN MARTIN) Date: Mon, 31 Oct 2016 21:21:52 +0600 Subject: Document No 561820405886 Message-ID: Thanks for using electronic billing Please find your document attached Regards DAMIAN MARTIN -------------- next part -------------- A non-text attachment was scrubbed... Name: Document 561820405886.zip Type: application/zip Size: 7937 bytes Desc: not available URL: