From slink at schokola.de Wed Feb 1 16:14:40 2017 From: slink at schokola.de (Nils Goroll) Date: Wed, 1 Feb 2017 17:14:40 +0100 Subject: auto-tune ws / headroom poc In-Reply-To: <56D43E68.60606@schokola.de> References: <56D43E68.60606@schokola.de> Message-ID: Hi, this just surfaced again. More than before I think we should not let the admin size workspaces, but just the space available to VCL. Nils On 29/02/16 13:49, Nils Goroll wrote: > Hi, > > following up an attempt to discuss this on irc: > > > The following discussion is, for the time being, regarding workpace_backend > only, but the question is relevant for the other workspaces also: > > Relevant portions of workspace_backend are not available for use in VCL, but are > consumed in core code. Setting workspace_backend too low will trigger an > immediate assertion failure in VBO_GetBusyObj(). > > IMHO, it doesn't make sense to handle these assertions differently, because > space for http headers, vsl and the initial read from the backend are mandatory > for varnish to issue backend requests. > > Instead of trying to better handle nonsense workspace_backend sizes, I'd suggest > to make it impossible to tune it stupidly in the first place - and to give > admins a tunable which is easier to understand. > > The attached patch contains a PoC (this is _not_ a finished patch, see below) > suggesting the following changes: > > - make workspace_backend a protected (read-only) parameter > > - add headroom_backend as a new admin-tunable parameter which roughly equals to > "workspace available to VCL" > > - calculate the size of workspace_backend based on all other relevant parameters > > Notes on the patch: > > - I've kept the workspace parameter as protected so the mempool code can stay > as is (pointing to a single parameter for sizing) > > - in VBO_Init() I've taken a lightweight approach at the consistency issue from > parameter changes, and I'd be curious to hear if others agree or think > that we need to make this safer. > > - what is the best way to cross the border between the cache and mgt code? > > Originally I would have preferred to have the sizing code in cache_busyobj.c, > but then we'd need to make mgt_param visible there - which I didn't like. > > But what I have in the patch now doesn't look clean at all, I am really > unsure at this point. > > Thx, Nils > > > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > From phk at phk.freebsd.dk Thu Feb 2 14:06:42 2017 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 02 Feb 2017 14:06:42 +0000 Subject: thoughts about vcl cleanup around (be)resp.body and abandon/synth In-Reply-To: <4df6a46a-b3c4-1ac0-3e40-032409b135bc@schokola.de> References: <4df6a46a-b3c4-1ac0-3e40-032409b135bc@schokola.de> Message-ID: <1192.1486044402@critter.freebsd.dk> Sorry for being extremely late to this debate. The fundamental problem is that the body is not like headers, and IMO trying to make it look like one is a grave mistake. I am also very surprised that the word "filter" appears nowhere in the emails in this thread? We need to decide how VCL produced fetch bodies work with VFP, and why. If the answer is "They dont", then the "why" part needs a better answer than "I personally wont need that", because we know there are users wanting this. If the answer is "They do", then we need to figure out how VFP's work in VCL in the first place, before we can progress much. I think the correct answer is "They do", but I'm willing to be persuaded. I have been thinking about the VCL/filter issue a lot, and all but one of my ideas becomes a royal mess to implement. The one idea which only becomes a mere ducal mess, is that vcl_backend_response{} gets a new variable named beresp.filter_list which contains a space-separated list of VFP's to use ("gunzip esi gzip") If you want modify the filter stack, you edit that beresp.filter_list, probably with a regsub(). VMODs offering novel filters, must register them, so they can be found when the list is interpreted, which unfortunately leaves a hole wrt. to passing parameters from VCL to that filter instance. As for stuffing actual content through the filters and into the resp.body, that cannot happen in backend_response{} because we have neither object, storage or filterstack yet. The vcl_synth{} vcl_backend_error{} situation is not very satisfying and trying to build on that seems unwise. One possible solution, on both client and backend side, would be to be able to tell what function to call to produce the body, in straw-man form something like: sub my_error { body += ""; body += "

" + bereq.url + "

" body += vmod.foobar(something, something); body += "Because We Can"; body += ""; } sub vcl_backend_response { [...] set beresp.[...]; return (synth(my_error)); } It goes without saying that the "body-functions" would not be able to do anything else etc. Semi-major VCC work in other words... The final thing I want to say at this point is that before we do any of this, we need to change how VCL fails. It used to be that we would try to muddle through when VCL did something counter-productive, but that is increasingly hard and increasingly wrong. I'm not done figuring out how a "VCL botched up" state would look, inputs welcome. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Tue Feb 7 11:15:17 2017 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 07 Feb 2017 11:15:17 +0000 Subject: VRT_fail() Message-ID: <829.1486466117@critter.freebsd.dk> I have worked on the handling of near-fatal errors in VCL. "near-fatal" errors are things like running out of workspace, or doing silly things like: set req.url = ""; Where we cannot continue, but don't need to panic the worker process because no state outside the work at hand have been corrupted. In practice this works by setting ctx->handling to VCL_RET_FAIL and having the VCC compiled code check if that happend after each statement executed (a very cheap check btw.) To trigger a VCL failure use VRT_fail() with a one line usable message. The fmt+args given to the VRT_fail() function will be VSL'ed as VCL_Error or in the case of vcl_init{} spit out in the CLI. And now the footnotes. Right now failure in the backend side of VCL just fails the fetch, with no diagnostic available as to why on the client side. This is a general problem not special to VRT_fail(). On the client side failure sends you to vcl_synth{}, but the bugwash yesterday expressed a preference for a new dedicated vcl method for this, so dont get too attached to this bit. (Failures in vcl_synth{} summarily close the client connection) We need a generic function on top of VRT_fail() for workspace overflows, so the diagnostics don't end up all different. That is yet to come. I've been wondering if we should record the VCL-coords (file,line,char like in VCL_count) of the failures in the VCL_Error() message, input welcome? Finally, because we only check for failure between handling, expression evaluation can chew on for quite some time before it gets there. We need to think/study that a bit, before we decide if something more radical is called for. Otherwise, have at it, (and please use logexpect to check that the messages are indeed in VSL) Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From hermunn at varnish-software.com Wed Feb 8 09:53:15 2017 From: hermunn at varnish-software.com (=?UTF-8?Q?P=C3=A5l_Hermunn_Johansen?=) Date: Wed, 8 Feb 2017 10:53:15 +0100 Subject: The upcomming 4.1.5 release Message-ID: Hello, everyone. I am working towards a 4.1.5 release. I have backported fixes from master, but sometimes things slip through the cracks. Thanks to Federico for pointing out some of the things I have missed. If anyone has specific patches or backports to 4.1.5, please notify me as soon as possible. I will update changelogs and tag a 4.1.5beta2 now, and hopefully we can release 4.1.5 late tomorrow, if nothing comes up. Best, P?l From jonathan.huot at thomsonreuters.com Fri Feb 10 16:33:30 2017 From: jonathan.huot at thomsonreuters.com (jonathan.huot at thomsonreuters.com) Date: Fri, 10 Feb 2017 16:33:30 +0000 Subject: Implementing req.do_gunzip Message-ID: <8E656B642592B942AE317E2AFAE0ABA18BEC465A@C111MFBLMBX07.ERF.thomson.com> Hi varnish-dev, We're currently migrating our vmods from varnish 3, and we have a vmod which is uncompressing request's body to handle situation where backends are very old and not able to handle gzip encoding... It happens sometime. Instead of rewriting this vmod into a V4 module, I was guessing if you see the benefit of integrating into the core instead? It could be done by doing something like set req.do_gunzip = true; VTC could be approx..: $ cat g00008.vtc varnishtest "test basic gunzip for client request body" server s1 { rxreq expect req.http.content-encoding == txresp -status 200 } -start varnish v1 -cliok "param.set http_gzip_support true" -vcl+backend { sub vcl_recv { set req.do_gunzip = true; } } -start client c1 { txreq -gzipbody {FOOBARFOO} rxresp expect resp.status == 200 } -run varnish v1 -expect n_gzip == 0 varnish v1 -expect n_gunzip == 1 If you think it's interesting and any PR are welcomed, I will be glad to do one. Else, I will upgrade our VMOD instead :-) -- Jonathan Huot Thomson Reuters ________________________________ This e-mail is for the sole use of the intended recipient and contains information that may be privileged and/or confidential. If you are not an intended recipient, please notify the sender by return e-mail and delete this e-mail and any attachments. Certain required legal entity disclosures can be accessed on our website. From fgsch at lodoss.net Mon Feb 13 00:28:29 2017 From: fgsch at lodoss.net (Federico Schwindt) Date: Mon, 13 Feb 2017 00:28:29 +0000 Subject: Implementing req.do_gunzip In-Reply-To: <8E656B642592B942AE317E2AFAE0ABA18BEC465A@C111MFBLMBX07.ERF.thomson.com> References: <8E656B642592B942AE317E2AFAE0ABA18BEC465A@C111MFBLMBX07.ERF.thomson.com> Message-ID: Hi, Sounds like a good feature and shouldn't be too hard since we already have the VFPs That said, if we were to implement it I reckon we should have both req.do_gunzip and req.do_gzip. Best. On Fri, Feb 10, 2017 at 4:33 PM, wrote: > Hi varnish-dev, > > We're currently migrating our vmods from varnish 3, and we have a vmod > which > is uncompressing request's body to handle situation where backends are > very old > and not able to handle gzip encoding... It happens sometime. > > Instead of rewriting this vmod into a V4 module, I was guessing if you see > the benefit of integrating into the core instead? It could be done by doing > something like > > set req.do_gunzip = true; > > > VTC could be approx..: > > $ cat g00008.vtc > varnishtest "test basic gunzip for client request body" > > server s1 { > rxreq > expect req.http.content-encoding == > txresp -status 200 > } -start > > varnish v1 -cliok "param.set http_gzip_support true" -vcl+backend { > sub vcl_recv { > set req.do_gunzip = true; > } > } -start > > client c1 { > txreq -gzipbody {FOOBARFOO} > rxresp > expect resp.status == 200 > } -run > > varnish v1 -expect n_gzip == 0 > varnish v1 -expect n_gunzip == 1 > > > If you think it's interesting and any PR are welcomed, I will be glad to > do one. > Else, I will upgrade our VMOD instead :-) > > -- > Jonathan Huot > Thomson Reuters > > > ________________________________ > > This e-mail is for the sole use of the intended recipient and contains > information that may be privileged and/or confidential. If you are not an > intended recipient, please notify the sender by return e-mail and delete > this e-mail and any attachments. Certain required legal entity disclosures > can be accessed on our website. disclosures/> > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Mon Feb 13 07:28:48 2017 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 13 Feb 2017 07:28:48 +0000 Subject: Implementing req.do_gunzip In-Reply-To: References: <8E656B642592B942AE317E2AFAE0ABA18BEC465A@C111MFBLMBX07.ERF.thomson.com> Message-ID: <83419.1486970928@critter.freebsd.dk> -------- In message , Federico Schwindt writes: >Sounds like a good feature and shouldn't be too hard since we already have >the VFPs >That said, if we were to implement it I reckon we should have both >req.do_gunzip and req.do_gzip. This ties into the recent chat about filters and bodies in general. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Wed Feb 22 13:12:32 2017 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 22 Feb 2017 14:12:32 +0100 Subject: [master] cbe8047 Reintroduce hit-for-pass with new and better syntax: In-Reply-To: References: Message-ID: On Wed, Feb 22, 2017 at 2:04 PM, Poul-Henning Kamp wrote: > > commit cbe804760d78b917fbbb8a1d69b2236a2d98a028 > Author: Poul-Henning Kamp > Date: Wed Feb 22 10:40:58 2017 +0000 > > Reintroduce hit-for-pass with new and better syntax: > > sub vcl_backend_response { > return (pass(2s)); > } We need to clear the confusion of turning hitpass into a hitmiss in 5.0, while keeping the MAIN.cache_hitpass counter as is. Especially now that hitpass lives alongside hitmiss. Also, should the built-in VCL make a distinction between hitmiss and hitpass? For example if the response contains a Set-Cookie header it might be a one-time thing and hitmiss would be preferred, otherwise it looks like something reliably not cacheable: so hitpass. Thoughts? Cheers From phk at phk.freebsd.dk Wed Feb 22 13:19:50 2017 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 22 Feb 2017 13:19:50 +0000 Subject: [master] cbe8047 Reintroduce hit-for-pass with new and better syntax: In-Reply-To: References: Message-ID: <34824.1487769590@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >We need to clear the confusion of turning hitpass into a hitmiss in >5.0, while keeping the MAIN.cache_hitpass counter as is. Especially >now that hitpass lives alongside hitmiss. Yes, there are a number of consequences that we need to attend to. >Also, should the built-in VCL make a distinction between hitmiss and >hitpass? IMO built-in VCL should never use hit-for-pass. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence.