From phk at phk.freebsd.dk Mon Jan 5 08:20:39 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 05 Jan 2015 08:20:39 +0000 Subject: [PATCH] For Vary processing, treat an empty header (only whitespace), like a non-existent header[PATCH] For Vary processing, treat an empty header (only whitespace), like a non-existent header In-Reply-To: <549EFB94.6050206@schokola.de> References: <549EFB94.6050206@schokola.de> Message-ID: <5848.1420446039@critter.freebsd.dk> -------- In message <549EFB94.6050206 at schokola.de>, Nils Goroll writes: >Our interpretation of http://tools.ietf.org/html/rfc7234#section-4.1 is that >removing all whitespace from a header field only containing whitespace >equals to removing the header. I do not agree. It depends on the definition of each header if an empty header is the same as an absent header, and I really do not want to add a lot of code to handle that corner-case. Varnish's Vary processing is by design rather simplistic, we do not attempt to normalize headers etc. If you want to wash your headers before vary processing, by all means do so in VCL. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From fgsch at lodoss.net Wed Jan 7 00:12:41 2015 From: fgsch at lodoss.net (Federico Schwindt) Date: Wed, 7 Jan 2015 00:12:41 +0000 Subject: [master] 468a047 Expose is_hit to VCL as resp.is_hit In-Reply-To: References:

Message-ID: I must say I'm not very thrilled about having another variable to check whether it was a hit. How much of an issue this really is and can we fix the code so we still use obj.hits? On Wed, Jan 7, 2015 at 12:10 AM, Federico Schwindt wrote: > I must say I'm not very thrilled about having another variable to check > whether it was a hit. > > How much of an issue this really is and can we fix the code so we still > use obj.hits? > > On Tue, Jan 6, 2015 at 9:41 PM, Nils Goroll wrote: > >> >> commit 468a0472fa313dbff69269afc085957642011abc >> Author: Nils Goroll >> Date: Tue Jan 6 22:24:21 2015 +0100 >> >> Expose is_hit to VCL as resp.is_hit >> >> The common method to check obj.hits is racy: While the update to >> the hits counter happens under a lock, a hit could have happened >> before the missing request gets to read obj.hits. >> >> diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h >> index 2e7fa47..82bf5ed 100644 >> --- a/bin/varnishd/cache/cache.h >> +++ b/bin/varnishd/cache/cache.h >> @@ -540,7 +540,7 @@ struct req { >> int restarts; >> int esi_level; >> >> -#define REQ_FLAG(l, r, w, d) unsigned l:1; >> +#define REQ_FLAG(l, o, r, w, d) unsigned l:1; >> #include "tbl/req_flags.h" >> #undef REQ_FLAG >> >> diff --git a/bin/varnishd/cache/cache_panic.c >> b/bin/varnishd/cache/cache_panic.c >> index d45458e..9bd99a7 100644 >> --- a/bin/varnishd/cache/cache_panic.c >> +++ b/bin/varnishd/cache/cache_panic.c >> @@ -415,7 +415,7 @@ pan_req(const struct req *req) >> } >> >> VSB_printf(pan_vsp, " flags = {\n"); >> -#define REQ_FLAG(l, r, w, d) if(req->l) VSB_printf(pan_vsp, " " #l >> ",\n"); >> +#define REQ_FLAG(l, o, r, w, d) if(req->l) VSB_printf(pan_vsp, " " #l >> ",\n"); >> #include "tbl/req_flags.h" >> #undef REQ_FLAG >> VSB_printf(pan_vsp, " }\n"); >> diff --git a/bin/varnishd/cache/cache_vrt_var.c >> b/bin/varnishd/cache/cache_vrt_var.c >> index 4c6f2c6..5b2e5c2 100644 >> --- a/bin/varnishd/cache/cache_vrt_var.c >> +++ b/bin/varnishd/cache/cache_vrt_var.c >> @@ -566,29 +566,29 @@ VRT_r_bereq_xid(VRT_CTX) >> * req fields >> */ >> >> -#define VREQW0(field) >> -#define VREQW1(field) \ >> +#define VREQW0(obj, field) >> +#define VREQW1(obj, field) \ >> void \ >> -VRT_l_req_##field(VRT_CTX, unsigned a) \ >> +VRT_l_##obj##_##field(VRT_CTX, unsigned a) \ >> { \ >> CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); \ >> CHECK_OBJ_NOTNULL(ctx->req, REQ_MAGIC); \ >> ctx->req->field = a ? 1 : 0; \ >> } >> >> -#define VREQR0(field) >> -#define VREQR1(field) \ >> +#define VREQR0(obj, field) >> +#define VREQR1(obj, field) \ >> unsigned \ >> -VRT_r_req_##field(VRT_CTX) \ >> +VRT_r_##obj##_##field(VRT_CTX) \ >> { \ >> CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); \ >> CHECK_OBJ_NOTNULL(ctx->req, REQ_MAGIC); \ >> return (ctx->req->field); \ >> } >> >> -#define REQ_FLAG(l, r, w, d) \ >> - VREQR##r(l) \ >> - VREQW##w(l) >> +#define REQ_FLAG(l, o, r, w, d) \ >> + VREQR##r(o, l) \ >> + VREQW##w(o, l) >> #include "tbl/req_flags.h" >> #undef REQ_FLAG >> >> diff --git a/bin/varnishtest/tests/v00013.vtc >> b/bin/varnishtest/tests/v00013.vtc >> index c428add..19e9f26 100644 >> --- a/bin/varnishtest/tests/v00013.vtc >> +++ b/bin/varnishtest/tests/v00013.vtc >> @@ -13,6 +13,11 @@ varnish v1 -vcl+backend { >> >> sub vcl_deliver { >> set resp.http.foo = obj.hits; >> + if (resp.is_hit) { >> + set resp.http.X-Hit = "yes"; >> + } else { >> + set resp.http.X-Hit = "no"; >> + } >> } >> } -start >> >> @@ -21,16 +26,19 @@ client c1 { >> rxresp >> expect resp.status == 200 >> expect resp.http.foo == 0 >> + expect resp.http.X-hit == "no" >> >> txreq >> rxresp >> expect resp.status == 200 >> expect resp.http.foo == 1 >> + expect resp.http.X-hit == "yes" >> >> txreq -url /foo >> rxresp >> expect resp.status == 200 >> expect resp.http.foo == 0 >> + expect resp.http.X-hit == "no" >> delay .1 >> >> txreq >> @@ -38,4 +46,5 @@ client c1 { >> expect resp.status == 200 >> expect resp.http.foo == 2 >> expect resp.http.bar >= 0.100 >> + expect resp.http.X-hit == "yes" >> } -run >> diff --git a/bin/varnishtest/tests/v00039.vtc >> b/bin/varnishtest/tests/v00039.vtc >> index 029e114..c248aa8 100644 >> --- a/bin/varnishtest/tests/v00039.vtc >> +++ b/bin/varnishtest/tests/v00039.vtc >> @@ -13,6 +13,11 @@ varnish v1 \ >> -vcl+backend { >> sub vcl_deliver { >> set resp.http.hits = obj.hits; >> + if (resp.is_hit) { >> + set resp.http.X-Hit = "yes"; >> + } else { >> + set resp.http.X-Hit = "no"; >> + } >> } >> } -start >> >> @@ -23,6 +28,7 @@ client c1 { >> expect resp.status == 200 >> expect resp.bodylen == 6 >> expect resp.http.hits == 0 >> + expect resp.http.X-Hit == "no" >> >> # This is a hit -> hits == 1 >> txreq -url "/" -hdr "Bar: 1" >> @@ -30,6 +36,7 @@ client c1 { >> expect resp.status == 200 >> expect resp.bodylen == 6 >> expect resp.http.hits == 1 >> + expect resp.http.X-Hit == "yes" >> >> # This is a miss on different vary -> hits == 0 >> txreq -url "/" -hdr "Bar: 2" >> @@ -37,6 +44,7 @@ client c1 { >> expect resp.status == 200 >> expect resp.bodylen == 4 >> expect resp.http.hits == 0 >> + expect resp.http.X-Hit == "no" >> >> # This is a hit -> hits == 1 >> txreq -url "/" -hdr "Bar: 2" >> @@ -44,6 +52,7 @@ client c1 { >> expect resp.status == 200 >> expect resp.bodylen == 4 >> expect resp.http.hits == 1 >> + expect resp.http.X-Hit == "yes" >> >> } -run >> >> diff --git a/include/tbl/req_flags.h b/include/tbl/req_flags.h >> index 6757d90..55f20f7 100644 >> --- a/include/tbl/req_flags.h >> +++ b/include/tbl/req_flags.h >> @@ -29,11 +29,11 @@ >> >> /*lint -save -e525 -e539 */ >> >> -/* lower, vcl_r, vcl_w, doc */ >> -REQ_FLAG(disable_esi, 0, 0, "") >> -REQ_FLAG(hash_ignore_busy, 1, 1, "") >> -REQ_FLAG(hash_always_miss, 1, 1, "") >> -REQ_FLAG(is_hit, 0, 0, "") >> -REQ_FLAG(wantbody, 0, 0, "") >> -REQ_FLAG(gzip_resp, 0, 0, "") >> +/* lower, vcl_obj, vcl_r, vcl_w, doc */ >> +REQ_FLAG(disable_esi, none, 0, 0, "") >> +REQ_FLAG(hash_ignore_busy, req, 1, 1, "") >> +REQ_FLAG(hash_always_miss, req, 1, 1, "") >> +REQ_FLAG(is_hit, resp, 1, 0, "") >> +REQ_FLAG(wantbody, none, 0, 0, "") >> +REQ_FLAG(gzip_resp, none, 0, 0, "") >> /*lint -restore */ >> diff --git a/lib/libvcc/generate.py b/lib/libvcc/generate.py >> index e582c0f..699bab3 100755 >> --- a/lib/libvcc/generate.py >> +++ b/lib/libvcc/generate.py >> @@ -564,8 +564,10 @@ sp_variables = [ >> 'INT', >> ( 'hit', 'deliver',), >> ( ), """ >> - The count of cache-hits on this object. A value of 0 >> indicates a >> - cache miss. >> + The count of cache-hits on this object. >> + >> + NB: obj.hits is not a reliable way to determine cache >> + misses. See resp.is_hit. >> """ >> ), >> ('obj.http.', >> @@ -645,6 +647,13 @@ sp_variables = [ >> The corresponding HTTP header. >> """ >> ), >> + ('resp.is_hit', >> + 'BOOL', >> + ( 'deliver',), >> + ( ), """ >> + If this response is from a cache hit. >> + """ >> + ), >> ('now', >> 'TIME', >> ( 'all',), >> >> _______________________________________________ >> varnish-commit mailing list >> varnish-commit at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-commit >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fgsch at lodoss.net Wed Jan 7 00:51:06 2015 From: fgsch at lodoss.net (Federico Schwindt) Date: Wed, 7 Jan 2015 00:51:06 +0000 Subject: [master] 468a047 Expose is_hit to VCL as resp.is_hit In-Reply-To: References:

Message-ID: Can we just do: return (ctx->req->is_hit ? ctx->req->objcore->hits : 0); in VRT_r_obj_hits() and revert this commit altogether? I'm not sure whether this was discussed but besides the macros now having the wrong names (REQ_FLAG and VREQ[WR][01] used for other variables than req) adding more knobs is not the way to go IMO. On Wed, Jan 7, 2015 at 12:12 AM, Federico Schwindt wrote: > I must say I'm not very thrilled about having another variable to check > whether it was a hit. > > How much of an issue this really is and can we fix the code so we still > use obj.hits? > > On Wed, Jan 7, 2015 at 12:10 AM, Federico Schwindt > wrote: > >> I must say I'm not very thrilled about having another variable to check >> whether it was a hit. >> >> How much of an issue this really is and can we fix the code so we still >> use obj.hits? >> >> On Tue, Jan 6, 2015 at 9:41 PM, Nils Goroll wrote: >> >>> >>> commit 468a0472fa313dbff69269afc085957642011abc >>> Author: Nils Goroll >>> Date: Tue Jan 6 22:24:21 2015 +0100 >>> >>> Expose is_hit to VCL as resp.is_hit >>> >>> The common method to check obj.hits is racy: While the update to >>> the hits counter happens under a lock, a hit could have happened >>> before the missing request gets to read obj.hits. >>> >>> diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h >>> index 2e7fa47..82bf5ed 100644 >>> --- a/bin/varnishd/cache/cache.h >>> +++ b/bin/varnishd/cache/cache.h >>> @@ -540,7 +540,7 @@ struct req { >>> int restarts; >>> int esi_level; >>> >>> -#define REQ_FLAG(l, r, w, d) unsigned l:1; >>> +#define REQ_FLAG(l, o, r, w, d) unsigned l:1; >>> #include "tbl/req_flags.h" >>> #undef REQ_FLAG >>> >>> diff --git a/bin/varnishd/cache/cache_panic.c >>> b/bin/varnishd/cache/cache_panic.c >>> index d45458e..9bd99a7 100644 >>> --- a/bin/varnishd/cache/cache_panic.c >>> +++ b/bin/varnishd/cache/cache_panic.c >>> @@ -415,7 +415,7 @@ pan_req(const struct req *req) >>> } >>> >>> VSB_printf(pan_vsp, " flags = {\n"); >>> -#define REQ_FLAG(l, r, w, d) if(req->l) VSB_printf(pan_vsp, " " #l >>> ",\n"); >>> +#define REQ_FLAG(l, o, r, w, d) if(req->l) VSB_printf(pan_vsp, " " >>> #l ",\n"); >>> #include "tbl/req_flags.h" >>> #undef REQ_FLAG >>> VSB_printf(pan_vsp, " }\n"); >>> diff --git a/bin/varnishd/cache/cache_vrt_var.c >>> b/bin/varnishd/cache/cache_vrt_var.c >>> index 4c6f2c6..5b2e5c2 100644 >>> --- a/bin/varnishd/cache/cache_vrt_var.c >>> +++ b/bin/varnishd/cache/cache_vrt_var.c >>> @@ -566,29 +566,29 @@ VRT_r_bereq_xid(VRT_CTX) >>> * req fields >>> */ >>> >>> -#define VREQW0(field) >>> -#define VREQW1(field) \ >>> +#define VREQW0(obj, field) >>> +#define VREQW1(obj, field) \ >>> void \ >>> -VRT_l_req_##field(VRT_CTX, unsigned a) \ >>> +VRT_l_##obj##_##field(VRT_CTX, unsigned a) \ >>> { \ >>> CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); \ >>> CHECK_OBJ_NOTNULL(ctx->req, REQ_MAGIC); \ >>> ctx->req->field = a ? 1 : 0; \ >>> } >>> >>> -#define VREQR0(field) >>> -#define VREQR1(field) \ >>> +#define VREQR0(obj, field) >>> +#define VREQR1(obj, field) \ >>> unsigned \ >>> -VRT_r_req_##field(VRT_CTX) \ >>> +VRT_r_##obj##_##field(VRT_CTX) \ >>> { \ >>> CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); \ >>> CHECK_OBJ_NOTNULL(ctx->req, REQ_MAGIC); \ >>> return (ctx->req->field); \ >>> } >>> >>> -#define REQ_FLAG(l, r, w, d) \ >>> - VREQR##r(l) \ >>> - VREQW##w(l) >>> +#define REQ_FLAG(l, o, r, w, d) \ >>> + VREQR##r(o, l) \ >>> + VREQW##w(o, l) >>> #include "tbl/req_flags.h" >>> #undef REQ_FLAG >>> >>> diff --git a/bin/varnishtest/tests/v00013.vtc >>> b/bin/varnishtest/tests/v00013.vtc >>> index c428add..19e9f26 100644 >>> --- a/bin/varnishtest/tests/v00013.vtc >>> +++ b/bin/varnishtest/tests/v00013.vtc >>> @@ -13,6 +13,11 @@ varnish v1 -vcl+backend { >>> >>> sub vcl_deliver { >>> set resp.http.foo = obj.hits; >>> + if (resp.is_hit) { >>> + set resp.http.X-Hit = "yes"; >>> + } else { >>> + set resp.http.X-Hit = "no"; >>> + } >>> } >>> } -start >>> >>> @@ -21,16 +26,19 @@ client c1 { >>> rxresp >>> expect resp.status == 200 >>> expect resp.http.foo == 0 >>> + expect resp.http.X-hit == "no" >>> >>> txreq >>> rxresp >>> expect resp.status == 200 >>> expect resp.http.foo == 1 >>> + expect resp.http.X-hit == "yes" >>> >>> txreq -url /foo >>> rxresp >>> expect resp.status == 200 >>> expect resp.http.foo == 0 >>> + expect resp.http.X-hit == "no" >>> delay .1 >>> >>> txreq >>> @@ -38,4 +46,5 @@ client c1 { >>> expect resp.status == 200 >>> expect resp.http.foo == 2 >>> expect resp.http.bar >= 0.100 >>> + expect resp.http.X-hit == "yes" >>> } -run >>> diff --git a/bin/varnishtest/tests/v00039.vtc >>> b/bin/varnishtest/tests/v00039.vtc >>> index 029e114..c248aa8 100644 >>> --- a/bin/varnishtest/tests/v00039.vtc >>> +++ b/bin/varnishtest/tests/v00039.vtc >>> @@ -13,6 +13,11 @@ varnish v1 \ >>> -vcl+backend { >>> sub vcl_deliver { >>> set resp.http.hits = obj.hits; >>> + if (resp.is_hit) { >>> + set resp.http.X-Hit = "yes"; >>> + } else { >>> + set resp.http.X-Hit = "no"; >>> + } >>> } >>> } -start >>> >>> @@ -23,6 +28,7 @@ client c1 { >>> expect resp.status == 200 >>> expect resp.bodylen == 6 >>> expect resp.http.hits == 0 >>> + expect resp.http.X-Hit == "no" >>> >>> # This is a hit -> hits == 1 >>> txreq -url "/" -hdr "Bar: 1" >>> @@ -30,6 +36,7 @@ client c1 { >>> expect resp.status == 200 >>> expect resp.bodylen == 6 >>> expect resp.http.hits == 1 >>> + expect resp.http.X-Hit == "yes" >>> >>> # This is a miss on different vary -> hits == 0 >>> txreq -url "/" -hdr "Bar: 2" >>> @@ -37,6 +44,7 @@ client c1 { >>> expect resp.status == 200 >>> expect resp.bodylen == 4 >>> expect resp.http.hits == 0 >>> + expect resp.http.X-Hit == "no" >>> >>> # This is a hit -> hits == 1 >>> txreq -url "/" -hdr "Bar: 2" >>> @@ -44,6 +52,7 @@ client c1 { >>> expect resp.status == 200 >>> expect resp.bodylen == 4 >>> expect resp.http.hits == 1 >>> + expect resp.http.X-Hit == "yes" >>> >>> } -run >>> >>> diff --git a/include/tbl/req_flags.h b/include/tbl/req_flags.h >>> index 6757d90..55f20f7 100644 >>> --- a/include/tbl/req_flags.h >>> +++ b/include/tbl/req_flags.h >>> @@ -29,11 +29,11 @@ >>> >>> /*lint -save -e525 -e539 */ >>> >>> -/* lower, vcl_r, vcl_w, doc */ >>> -REQ_FLAG(disable_esi, 0, 0, "") >>> -REQ_FLAG(hash_ignore_busy, 1, 1, "") >>> -REQ_FLAG(hash_always_miss, 1, 1, "") >>> -REQ_FLAG(is_hit, 0, 0, "") >>> -REQ_FLAG(wantbody, 0, 0, "") >>> -REQ_FLAG(gzip_resp, 0, 0, "") >>> +/* lower, vcl_obj, vcl_r, vcl_w, doc */ >>> +REQ_FLAG(disable_esi, none, 0, 0, "") >>> +REQ_FLAG(hash_ignore_busy, req, 1, 1, "") >>> +REQ_FLAG(hash_always_miss, req, 1, 1, "") >>> +REQ_FLAG(is_hit, resp, 1, 0, "") >>> +REQ_FLAG(wantbody, none, 0, 0, "") >>> +REQ_FLAG(gzip_resp, none, 0, 0, "") >>> /*lint -restore */ >>> diff --git a/lib/libvcc/generate.py b/lib/libvcc/generate.py >>> index e582c0f..699bab3 100755 >>> --- a/lib/libvcc/generate.py >>> +++ b/lib/libvcc/generate.py >>> @@ -564,8 +564,10 @@ sp_variables = [ >>> 'INT', >>> ( 'hit', 'deliver',), >>> ( ), """ >>> - The count of cache-hits on this object. A value of 0 >>> indicates a >>> - cache miss. >>> + The count of cache-hits on this object. >>> + >>> + NB: obj.hits is not a reliable way to determine cache >>> + misses. See resp.is_hit. >>> """ >>> ), >>> ('obj.http.', >>> @@ -645,6 +647,13 @@ sp_variables = [ >>> The corresponding HTTP header. >>> """ >>> ), >>> + ('resp.is_hit', >>> + 'BOOL', >>> + ( 'deliver',), >>> + ( ), """ >>> + If this response is from a cache hit. >>> + """ >>> + ), >>> ('now', >>> 'TIME', >>> ( 'all',), >>> >>> _______________________________________________ >>> varnish-commit mailing list >>> varnish-commit at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-commit >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fgsch at lodoss.net Wed Jan 7 03:03:32 2015 From: fgsch at lodoss.net (Federico Schwindt) Date: Wed, 7 Jan 2015 03:03:32 +0000 Subject: [PATCH] Add support to scale fields in varnishstat(1) Message-ID: The following two patches add support for scaling byte fields in varnishstat(1). Fixes #1653. The first one marks byte fields as such using the uppercase variant; bytes counter becomes 'C' and bytes gauge 'G'. The second adds support to scale these byte fields using "e" in the ncurses interface. Comments? OKs? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Add-specific-counters-for-byte-fields.patch Type: text/x-patch Size: 8480 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-Add-support-to-scale-byte-fields-in-varnishstat-1.patch Type: text/x-patch Size: 3722 bytes Desc: not available URL: From slink at schokola.de Wed Jan 7 09:55:12 2015 From: slink at schokola.de (Nils Goroll) Date: Wed, 07 Jan 2015 10:55:12 +0100 Subject: [master] 468a047 Expose is_hit to VCL as resp.is_hit In-Reply-To: References:

Message-ID: <54AD0280.1080600@schokola.de> Hi, On 07/01/15 01:51, Federico Schwindt wrote: > Can we just do: > > return (ctx->req->is_hit ? ctx->req->objcore->hits : 0); Sounds like a good idea. > in VRT_r_obj_hits() and revert this commit altogether? will do. > I'm not sure whether this was discussed but besides the macros now having the > wrong names (REQ_FLAG and VREQ[WR][01] used for other variables than req) adding > more knobs is not the way to go IMO. The REQ in REQ_FLAG and VREQ[WR][01] refers to struct req, not the VCL object. Thanks, Nils From geoff at uplex.de Wed Jan 7 10:43:36 2015 From: geoff at uplex.de (Geoff Simmons) Date: Wed, 07 Jan 2015 11:43:36 +0100 Subject: [master] 468a047 Expose is_hit to VCL as resp.is_hit In-Reply-To: References:

Message-ID: <54AD0DD8.3010105@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 > commit 468a0472fa313dbff69269afc085957642011abc Author: Nils Goroll > > Date: Tue > Jan 6 22:24:21 2015 +0100 > > Expose is_hit to VCL as resp.is_hit A very common idiom in just about every Varnish deployment anywhere is: sub vcl_hit { set req.http.X-Cache = "HIT"; } sub vcl_deliver { # ... if (req.http.X-Cache == "HIT") { # ... mumble ... } # ... } What would we gain from resp.is_hit that couldn't be solved that way? Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQIcBAEBCAAGBQJUrQ3NAAoJEOUwvh9pJNURiyAQAJ9VxasCUxPWMrMFY1pcBRfP do1nuwVXb2YTr8Mxi5SKZnoiDL5/mn5/ZcQKlcw71imCc7OGX193SRHsSBbm4mSc QTtgydWVVUN1h5HkqsRnFBWGpbPijY9BEUYLCgxbI6pH+V6DVy73UAPNxzJ7irWb ahwN/Vs/h68C9qJMkepBJuJgPNBoKXRunPNIqv8lD+EYxxtKaUfAXAZ4B2hZ6TIg ZUK6j3+TgbRsftHDI5nKqSiwIjPEmU+SEBE5hYS/QgUXIhZbrsBa5rQnWvLoLLeZ pAsJN9zCNDi+EWTDSJhJ9hAgDn6KyDMZsUgz6pZx61yELEEm0h8EW+i/uN6LBm34 g2N6j2qVu6Wlggtcp1ZEnaxAENOkU6h/lebayBRvMkIKbus8A4YlKqfkB0j4RoJ+ 2TrBrig5hz2D/UrRwi0UsPihadYhcpRrpw0BiBQFCtVRw+f2w+OJpeqTPa/rrLRG 01VwMhIEKrKpDFchDZaVcppLutcA+CHBLf2h/M0H4eY47vJ8qXkB6u6K1rSNeSWO zUYlVmiFBfHOjTtzmNoYkXtmuk7m+69MokkV+cPefIVJ2joQO3MoTotYHlgIMl+9 g1wWqbc3+B8oC07Z/GbEmt6wxMcbFf5EhSyHAw564cQ9mnn5y/6ZT7EoF2PIbfid FO9PESltP9NVdIiLCk28 =Xb7r -----END PGP SIGNATURE----- From slink at schokola.de Wed Jan 7 11:40:02 2015 From: slink at schokola.de (Nils Goroll) Date: Wed, 07 Jan 2015 12:40:02 +0100 Subject: [master] 468a047 Expose is_hit to VCL as resp.is_hit In-Reply-To: <54AD0DD8.3010105@uplex.de> References:

<54AD0DD8.3010105@uplex.de> Message-ID: <54AD1B12.2080005@schokola.de> On 07/01/15 11:43, Geoff Simmons wrote: > What would we gain from resp.is_hit that couldn't be solved that way? Some efficiency. But I've now fixed obj.hits as fgs has rightly suggested, so this really is not an issue any more. From slink at schokola.de Wed Jan 7 11:49:07 2015 From: slink at schokola.de (Nils Goroll) Date: Wed, 07 Jan 2015 12:49:07 +0100 Subject: [PATCH] Add support to scale fields in varnishstat(1) In-Reply-To: References: Message-ID: <54AD1D33.6010006@schokola.de> Hi, I've reviewed and tested the patches and they look fine to me. But my personal preference would be auto-scaling per value, for instance so that you never get more than 3 digits or using the next unit when reaching 90% of its base value. I'd suggest a -h(uman readable) command line option plus an interactive toggle key. If there is a good use case for manual sc(e)ling is, I don't mind keeping it. Nils From slink at schokola.de Wed Jan 7 11:52:54 2015 From: slink at schokola.de (Nils Goroll) Date: Wed, 07 Jan 2015 12:52:54 +0100 Subject: [PATCH] Add support to scale fields in varnishstat(1) In-Reply-To: <54AD1D33.6010006@schokola.de> References: <54AD1D33.6010006@schokola.de> Message-ID: <54AD1E16.8090806@schokola.de> And I'd consider adding an i to the unit (Ki, Mi, Gi etc) - see http://en.wikipedia.org/wiki/Binary_prefix From fgsch at lodoss.net Wed Jan 7 14:37:40 2015 From: fgsch at lodoss.net (Federico Schwindt) Date: Wed, 7 Jan 2015 14:37:40 +0000 Subject: [PATCH] Add support to scale fields in varnishstat(1) In-Reply-To: <54AD1D33.6010006@schokola.de> References: <54AD1D33.6010006@schokola.de> Message-ID: I prefer to let the user decide (i.e. no auto-scaling). This is what top in linux does as well. The changes required were small enough as well. That said if the consensus is to go for auto-scaling that's fine with me. Regarding adding the i to the unit, I've kept it brief on purpose to maintain the width as wide as possible. I've documented the units so this shouldn't be a problem. Also this is how top displays it. On Wed, Jan 7, 2015 at 11:49 AM, Nils Goroll wrote: > Hi, > > I've reviewed and tested the patches and they look fine to me. > > But my personal preference would be auto-scaling per value, for instance > so that > you never get more than 3 digits or using the next unit when reaching 90% > of its > base value. > > I'd suggest a -h(uman readable) command line option plus an interactive > toggle key. > > If there is a good use case for manual sc(e)ling is, I don't mind keeping > it. > > Nils > -------------- next part -------------- An HTML attachment was scrubbed... URL: From slink at schokola.de Wed Jan 7 15:55:03 2015 From: slink at schokola.de (Nils Goroll) Date: Wed, 07 Jan 2015 16:55:03 +0100 Subject: "expire superseded" in master In-Reply-To: <5493F193.3000205@schokola.de> References: <54906370.4000500@epochtimes.com> <5490C38A.4040309@epochtimes.com> <5492CDBF.7080504@schokola.de> <54937BEF.104@epochtimes.com> <5493F193.3000205@schokola.de> Message-ID: <54AD56D7.3040304@schokola.de> Hi, could you please retest master 0302e16e8888d9213931077b9b331f72dcc018b6 or newer? Thanks, Nils On 19/12/14 10:36, Nils Goroll wrote: > On 19/12/14 02:14, Jingyi Wei wrote: >> I see that >> bo->ims_obj is set in VBF_Fetch() only if Last-Modified or ETag header >> exists in the previous backend response. Otherwise |ims_obj| is NULL and >> the memory will not be freed, so the actual problem is that there's no >> Last-Modified header in our backend response. > > Good point, thank you. LM or ETag are required for issuing a backend conditional > request, but this should be independent of freeing the superseded object. I will > look after this unless someone else does. > > Nils > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > From slink at schokola.de Wed Jan 7 15:57:31 2015 From: slink at schokola.de (Nils Goroll) Date: Wed, 07 Jan 2015 16:57:31 +0100 Subject: [PATCH] Add support to scale fields in varnishstat(1) In-Reply-To: References: <54AD1D33.6010006@schokola.de> Message-ID: <54AD576B.8030606@schokola.de> On 07/01/15 15:37, Federico Schwindt wrote: > I prefer to let the user decide (i.e. no auto-scaling). This is what top in > linux does as well. Then I'd like the option to make that decision please. :) Would you add auto-scaling as an option/toggle key? > Regarding adding the i to the unit, I've kept it brief on purpose to maintain > the width as wide as possible. got it, ok. From jyw at epochtimes.com Wed Jan 7 19:28:06 2015 From: jyw at epochtimes.com (Jingyi Wei) Date: Wed, 07 Jan 2015 14:28:06 -0500 Subject: "expire superseded" in master In-Reply-To: <54AD56D7.3040304@schokola.de> References: <54906370.4000500@epochtimes.com> <5490C38A.4040309@epochtimes.com> <5492CDBF.7080504@schokola.de> <54937BEF.104@epochtimes.com> <5493F193.3000205@schokola.de> <54AD56D7.3040304@schokola.de> Message-ID: <54AD88C6.2030908@epochtimes.com> Hello Nils, Thank you for your patch. I tested it just now and I didn't see any memory increase caused by "expire superseded" objects. Thanks, Jingyi Wei On 01/07/15 10:55, Nils Goroll wrote: > Hi, > > could you please retest master 0302e16e8888d9213931077b9b331f72dcc018b6 or newer? > > Thanks, Nils > > On 19/12/14 10:36, Nils Goroll wrote: >> On 19/12/14 02:14, Jingyi Wei wrote: >>> I see that >>> bo->ims_obj is set in VBF_Fetch() only if Last-Modified or ETag header >>> exists in the previous backend response. Otherwise |ims_obj| is NULL and >>> the memory will not be freed, so the actual problem is that there's no >>> Last-Modified header in our backend response. >> Good point, thank you. LM or ETag are required for issuing a backend conditional >> request, but this should be independent of freeing the superseded object. I will >> look after this unless someone else does. >> >> Nils >> >> _______________________________________________ >> varnish-dev mailing list >> varnish-dev at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev >> > From fgsch at lodoss.net Wed Jan 7 22:43:26 2015 From: fgsch at lodoss.net (Federico Schwindt) Date: Wed, 7 Jan 2015 22:43:26 +0000 Subject: [PATCH] Add support to scale fields in varnishstat(1) In-Reply-To: <54AD576B.8030606@schokola.de> References: <54AD1D33.6010006@schokola.de> <54AD576B.8030606@schokola.de> Message-ID: I consider auto-scaling eye candy but if that's the consensus I might look into. Personally I'd prefer not to bloat varnishstat unnecessarily. On Wed, Jan 7, 2015 at 3:57 PM, Nils Goroll wrote: > On 07/01/15 15:37, Federico Schwindt wrote: > > I prefer to let the user decide (i.e. no auto-scaling). This is what top > in > > linux does as well. > > Then I'd like the option to make that decision please. :) Would you add > auto-scaling as an option/toggle key? > > > Regarding adding the i to the unit, I've kept it brief on purpose to > maintain > > the width as wide as possible. > > got it, ok. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From slink at schokola.de Fri Jan 9 12:36:28 2015 From: slink at schokola.de (Nils Goroll) Date: Fri, 09 Jan 2015 13:36:28 +0100 Subject: solve the pcre stack overflow #1576 - was: stack vs workspace for pcre and others In-Reply-To: <54061BDD.2000104@schokola.de> References: <5405FA81.7050808@schokola.de> <54060625.5060404@schokola.de> <54061983.9040301@schokola.de> <54061BDD.2000104@schokola.de> Message-ID: <54AFCB4C.7090401@schokola.de> Hi, as we (UPLEX) have hit the pcre stack overflow (SIGSEGV) issue again in production, I'd really appreciate if we could get a solution into master and I volunteer to work on this, once we have consensus about which way to go. Unfortunately this is bad timing for myself as I will be away for 16 days starting Monday, but I want to take the opportunity to write up a summary at least. I would like to continue after my holidays, but I really need the plan to be backed by phk to ensure the investment in time pays off. The most important point is that the current default thread_pool_stack (48k) and pcre_match_limit_recursion (10.000) do not match at all: For 10k pcre recursions, we'd need a stack size in the order of 2 MB. So at the very least, we really must use sensible values like 64k stack and pcre_match_limit_recursion 200. When built with pcre jit, we should default to 80k stack. -> @phk OK to change as a quick temporary workaround? But we can do better, here are the aspects I consider relevant based on the discussion back in September (which happened on irc for the main part, primarily phk, fgs, tollef and myself): I will focus on pcre1. pcre2 has been released 2015-01-05 and we should consider migrating later, but I think it's too early at this point in time. memory usage of pcre is primarily determined by whether or not JIT is used: * regular pcre_exec without JIT - stack required per pcre recursion on machine stack by default - alternatively, can use external alloc/free via callbacks, but this can only be enabled at pcre compile time, so it is not an option for varnish, unless we want to pull in pcre or change runtime behavior based on pcre_config(PCRE_CONFIG_STACKRECURSE). Also there is a relevant performance impact (according to the docs, have not checked myself) - pcre with PCRE_CONFIG_STACKRECURSE and without JIT has no built-in stack checks, it will run into a stack overflow unless we take precautions. - estimate of stack frame per recursion can be obtained using magic arguments pcre_exec(NULL, NULL, NULL, -999, -999) (documented in pcreapi(3), as output by ./pcretest -m -C, e.g. "Match recursion uses stack: approximate frame size = 176 bytes" on my install) - we can limit the recursion depth per pcre_exec - in recursions, not in bytes. Viable option I can see based on the work done back in September: * at init time: - get the pcre stack frame size * at init and parameter change time - check if the varnish parameters pcre_match_limit_recursion and thread_pool_stack make any sense at all and either refuse bad values or at least warn and auto-adjust suggestion: allow 48k stack for non-pcre and allow (approx pcre frame size + 16) * recursion_limit in addition * at pcre_exec time - check the available stack (see c522f7af7abaf8d847d62f8edaa46a2f5c0178e3 stack_start/stack_end pointers added to struct worker by phk) - further reduce the recursion_limit passed to pcre_exec based on avail_stack / (approx pcre frame size + 16) [the 16 is an arbitrary safely margin) - separate log/stats for PCRE_ERROR_RECURSIONLIMIT * JIT - uses - 32K on the machine stack - OR - - memory external to the machine stack (mmap(RWX, PRIVATE) with MAP_ANON or /dev/zero mapping) allocated with pcre_jit_stack_alloc(startsize, maxsize) (run time decision) - memory requirements have no strong tie to the notion of "pcre recursion", but do still depend on the pattern and the data (subject) When jit stack is insufficient in size, pcre_exec returns PCRE_ERROR_JIT_STACKLIMIT, so there is no built-in stack overflow nuke Viable options I can see: a) allow additional 32K on the machine stack and live with the fact that REs requiring more memory will fail b) manage a pool of pre-allocated jit stacks of arbitrary (user-specified) size c) re-try with memory external to the stack for PCRE_ERROR_JIT_STACKLIMIT errors * DFA: we don't use atm So I think the main decision point is what do to about JIT: Fist of all, for JIT we should require 8.32 according to Zoltan: http://www.mail-archive.com/pcre-dev at exim.org/msg03448.html Then the pcre guys recommend enabling JIT by default. I'd like to follow this recommendation, as long as pcre has a 'save' minimum version and there is a way to change back to non-JIT at run time in case JIT causes trouble. I'd prefer a per-RE flag in pcre, but we cannot expect this before pcre2, if at all. So, overall, here's my suggestion on how to proceed: - replace the compile time jit flag with a run-time (actually VCC time) parameter to enable pcre jit. On by default for "known good" pcre versions, off by default otherwise - check/set sensible stack and recursion limit parameter defaults and make pcre_exec calls safe as discussed above - add a way to switch jit on/off per RE at VCC time. The best way I can think of is to have options after the RE like vcl_recv { if (req.http.foo ~ "^/bar") { ... } # /J turns off jit # /j turns on if (req.http.foo ~ "jit-unsafe-RE"/J) { ... } } Alternatively, we could only implement a switch for https://code.uplex.de/uplex-varnish/libvmod-re only and keep the builtin match operator with the (run time) default. I'd appreciate any feedback, in particular - your thoughts on the JIT a/b/c options above - your thoughts on the overall plan - better ideas for per-RE flags Thanks, Nils From puiterwijk at redhat.com Wed Jan 21 20:24:49 2015 From: puiterwijk at redhat.com (Patrick Uiterwijk) Date: Wed, 21 Jan 2015 21:24:49 +0100 Subject: [PATCH] Remove need for FOWNER kernel capability Message-ID: <20150121202449.GA5510@bofh.thuis.puiterwijk.org> Hi, As explained in ticket #1663, currently varnish needs the fowner kernel capability, because it first fchowns the compiled policy module and then chmods it to 0755. This is most likely caused by a reconsideration of ticket's #1072 discussions. This is a problem for systems running selinux, as it will deny that capability unless it gets explicitely allowed by policy. Attached is a patch to remove the chown, since the compiled module is already owned by the unprivileged user. Please let me know what you think of this. With kind regards, Patrick Uiterwijk -------------- next part -------------- From 42610801bd1ea72833b6ddc247df6f3a49520153 Mon Sep 17 00:00:00 2001 From: Patrick Uiterwijk Date: Wed, 21 Jan 2015 19:41:16 +0100 Subject: [PATCH] Remove the chown, since the file is already owned by the unprivileged user. --- bin/varnishd/mgt/mgt_vcc.c | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/bin/varnishd/mgt/mgt_vcc.c b/bin/varnishd/mgt/mgt_vcc.c index f18fe41..57f9fe2 100644 --- a/bin/varnishd/mgt/mgt_vcc.c +++ b/bin/varnishd/mgt/mgt_vcc.c @@ -306,16 +306,6 @@ mgt_run_cc(const char *vcl, struct vsb *sb, int C_flag, unsigned *status) if (!subs) subs = VSUB_run(sb, run_dlopen, of, "dlopen", 10); - /* Ensure the file is readable to the unprivileged user */ - if (!subs) { - i = chmod(of, 0755); - if (i) { - VSB_printf(sb, "Failed to set permissions on %s: %s", - of, strerror(errno)); - subs = 2; - } - } - if (subs) { (void)unlink(of); *status = subs; -- 2.1.0 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From daghf at varnish-software.com Fri Jan 30 17:11:24 2015 From: daghf at varnish-software.com (Dag Haavi Finstad) Date: Fri, 30 Jan 2015 18:11:24 +0100 Subject: PROXY protocol - patches Message-ID: Hi Here's a set of patches for PROXY support in Varnish master, as discussed at the meeting in Barcelona last November (https://www.varnish-cache.org/trac/wiki/VDD14Q4#Notestakenduringthemeeting). 0001-Introduce-local.ip-and-remote.ip.patch 0002-Make-client.ip-settable-from-VCL.patch These patches introduce new VCL vars local.ip and remote.ip, and makes client.ip writable from VCL. local.ip and remote.ip are immutable and will always represent the endpoints of the local socket. VCL will then have four different IP vars (client.ip, server.ip, remote.ip, local.ip). By default (no PROXY proto), client.ip and server.ip will be set to the values of remote.ip and client.ip respectively. With PROXY protocol, they will get their values set from the PROXY header. 0003-Add-support-for-specifying-protocol-to-the-listen-so.patch: This adds an optional protocol specifier to the -a command line argument, to let us specify e.g. "-a proxy at 192.168.1.1:8081" to enable PROXY protocol for that listen endpoint. 0004-Set-blocking-mode-prior-to-entering-HTTP-1-FSM.patch: PROXY code needed blocking mode, so this had to be reorganized slightly. 0005-Preliminary-PROXY-v1-v2-support.patch: This is the big one that adds handling and parsing of PROXY v1 and v2. Adds a new fsm step S_STP_PROXY where we end up when a sess comes in from the acceptor on a PROXY protocol listen endpoint. If the sender does not satisfy the exact specifications of the PROXY protocol, we close the connection. With a valid PROXY header, the addresses will be available in client.ip and server.ip in VCL. 0006-Update-graphviz-doc-to-reflect-new-S_STP_PROXY-step.patch: And finally update the dotty diagrams to reflect how the state machine looks with PROXY protocol support. Further work: - Byte counters for PROXY protocol bytes. Is req_hdrbytes an acceptable place to stick it? - Put an example like this somewhere in the docs, to help users be safe from a spoofing attack: acl trusted_remotes { // ... } sub vcl_recv { if (remote.ip !~ trusted_remotes) { return (synth(403, "Forbidden")); } } Comments very much appreciated. -Dag -- Dag Haavi Finstad Software Developer | Varnish Software Mobile: +47 476 64 134 We Make Websites Fly! -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Introduce-local.ip-and-remote.ip.patch Type: text/x-patch Size: 9853 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-Make-client.ip-settable-from-VCL.patch Type: text/x-patch Size: 5881 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0003-Add-support-for-specifying-protocol-to-the-listen-so.patch Type: text/x-patch Size: 8092 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0004-Set-blocking-mode-prior-to-entering-HTTP-1-FSM.patch Type: text/x-patch Size: 2607 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0005-Preliminary-PROXY-v1-v2-support.patch Type: text/x-patch Size: 23398 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0006-Update-graphviz-doc-to-reflect-new-S_STP_PROXY-step.patch Type: text/x-patch Size: 1295 bytes Desc: not available URL: