From phk at phk.freebsd.dk Mon Oct 3 08:13:44 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 03 Oct 2016 08:13:44 +0000 Subject: suggesting bugwash topic: hit-for-pass vs. hit-for-miss In-Reply-To: References: <9417fd0c-b569-76fe-0103-d4ba67027ce8@schokola.de> Message-ID: <15961.1475482424@critter.freebsd.dk> -------- In message , Nils Goroll writ es: This is agenda item #1 for bugwash today, please everybody prepare. Poul-Henning >Hi, > >Geoff any myself would want to get ahead with the design aspects of this issue, >so we'd appreciate if we could discuss it during the next bugwash. > >It would be great if anyone interested could review the background info - my >initial email, quoted in full below, has the top-level overview. > >For the option of a vcl_hit based decision, >https://github.com/varnishcache/varnish-cache/issues/1799 is relevant also. > >Thank you, Nils > >On 02/09/16 20:10, Nils Goroll wrote: >> (quick brain dump before I need to rush out) >> >> Geoff discovered this interesting consequence of a recent important change of >> phk and we just spent an hour to discuss this: >> >> before commit 9f272127c6fba76e6758d7ab7ba6527d9aad98b0, a hit-for-pass object >> lead to a pass, not it's a miss. IIUC the discussions we had on a trip to >> Amsterdam, phks main motivation was to eliminate the potentially deadly effect >> unintentionally created hfps had on cache efficiency: No matter what, for the >> lifetime of the hfp, all requests hitting that object became passes. >> >> so, in short >> >> - previously: an uncacheable response wins and sticks for its ttl >> - now: an cacheable response wins and sticks for its ttl >> >> or eben shorter: >> >> - previously: hit-for-pass >> - now: hit-for-miss >> >> From the perspective of a cache, the "now" case seems clearly favorable, but now >> Geoff has discovered that the reverse is true for a case which is important to >> one of our projects: >> >> - varnish is running in "do how the backend says" mode >> - backend devs know when to make responses uncacheable >> - a huge (600MB) backend response is uncacheable, but client-validatable >> >> so this is the case for the previous semantics: >> >> - 1st request creates the hfp >> - 2nd request from client carries INM >> - gets passed with INM >> - 304 from backend goes to client >> >> What we have now is: >> >> - 1st request creates the hfm (hit-for-miss) >> - 2nd request is a miss >> - INM gets removed >> - backend sends 600MB unnecessarily >> >> We've thought about a couple of options which I want to write down before they >> expire from my cache: >> >> * decide in vcl_hit >> >> sub vcl_hit { >> if (obj.uncacheable) { >> if (obj.http.Criterium) { >> return (miss); >> } else { >> return (pass); >> } >> } >> } >> >> * Do not strip INM/IMS for miss and have a bereq property if it was a misspass >> >> - core code keeps INM/IMS >> - builtin.vcl strips them in vcl_miss >> - can check for hitpass in vcl_miss >> - any 304 backend response forced as uncacheable >> - interesting detail: can it still create a hfp object ? >> >> BUT: how would we know in vcl_miss if we see >> *client* inm/ims or varnish-generated inm/ims ? >> >> So at this point I only see the YAS option. >> >> Nils >> >> _______________________________________________ >> varnish-dev mailing list >> varnish-dev at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev >> > >_______________________________________________ >varnish-dev mailing list >varnish-dev at varnish-cache.org >https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Wed Oct 5 12:39:02 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 5 Oct 2016 14:39:02 +0200 Subject: libvmod-example moved to /varnishcache/ In-Reply-To: <20160823111319.GA12002@immer.varnish-software.com> References: <20160823111319.GA12002@immer.varnish-software.com> Message-ID: On Tue, Aug 23, 2016 at 1:13 PM, Lasse Karstensen wrote: > Hi all. > > To make collaboration easier, I've moved vmod-example to: > > https://github.com/varnishcache/libvmod-example/ Hi all, Following the move of the repository and the master branch now targeting Varnish 5.0, libvmod-example is making use of the new autoconf macros (also available since 4.1.4-beta1). The result is much simpler configure.ac and Makefile.am files, and thebability to specify constraints for Varnish's version. Technically a dist tarball will work with Varnish 4.1.2+, but you need at least Varnish 4.1.4-beta1 to build from git. Hopefully this will lower the barrier of entry, and lower the burden of maintenance. Best, Dridi From slink at schokola.de Mon Oct 10 11:26:57 2016 From: slink at schokola.de (Nils Goroll) Date: Mon, 10 Oct 2016 13:26:57 +0200 Subject: VDD16Q4 polls Message-ID: Varnish C hackers (core/vmod authors) who are interested in attending a dev meeting, please fill in these doodles so we get an idea of possible options: http://doodle.com/poll/78evzv5m3c9c8rhd#table http://doodle.com/poll/5f34i577x9t7ypc4#table From dridi at varni.sh Mon Oct 10 12:30:10 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 10 Oct 2016 14:30:10 +0200 Subject: VCL storage discussion summary Message-ID: Hello, Following discussions on storage/stevedore literals vs hints in VCL I've tried to sum up the current situation and suggest a direction to take. Currently we have: 1) a STEVEDORE type for VCL (storage namespace) 2) all storages resolved at boot time (typesafe) 3) beresp.storage_hint takes a STRING 4) no/wrong hint means round-robin among storages Today during the bugwash, following two pull requests introducing a storage hint for the request body (instead of systematically using Transient) we discussed the possibility of removing the hint part. The reason being 1) in the list above. After some testing on master, it doesn't seem to be enforced: sub vcl_backend_response { set beresp.storage_hint = "some random junk"; set beresp.storage_hint = beresp.http.x-storage; } However, implicit stevedore conversion to string exists: sub vcl_backend_response { set beresp.storage_hint = storage.Transient; } Suggestions: A) Keep the _hint to allow backend- or vmod-driven _loose_ storage selection. B) Maybe introduce besresp.storage to avoid conversions to and from STRING, but allow NULL to behave the same as 4) for the _hint. C) Be consistent when storage selection is introduced for the request body. Cheers, Dridi From fgsch at lodoss.net Mon Oct 10 16:45:22 2016 From: fgsch at lodoss.net (Federico Schwindt) Date: Mon, 10 Oct 2016 17:45:22 +0100 Subject: VCL storage discussion summary In-Reply-To: References: Message-ID: [ And now replying to all.. ] Hi, If setting or using a storage cannot fail I'd say kill _hint (eventually) and just use storage. If there is a chance of failing, I'd keep the _hint. I'd avoid having both. Having backend_hint and backend is confusing enough already. Regardless of the outcome, I think we should be more explicit on how tell tell Varnish any storage (or the next storage for the matter) is OK. Right now this happens under the hood and I've seen people setting the wrong storage by mistake (typos), not realising it and asking why Transient or some other storage is full. Best. On Mon, Oct 10, 2016 at 1:30 PM, Dridi Boukelmoune wrote: > Hello, > > Following discussions on storage/stevedore literals vs hints in VCL > I've tried to sum up the current situation and suggest a direction to > take. > > Currently we have: > > 1) a STEVEDORE type for VCL (storage namespace) > 2) all storages resolved at boot time (typesafe) > 3) beresp.storage_hint takes a STRING > 4) no/wrong hint means round-robin among storages > > Today during the bugwash, following two pull requests introducing a > storage hint for the request body (instead of systematically using > Transient) we discussed the possibility of removing the hint part. The > reason being 1) in the list above. > > After some testing on master, it doesn't seem to be enforced: > > sub vcl_backend_response { > set beresp.storage_hint = "some random junk"; > set beresp.storage_hint = beresp.http.x-storage; > } > > However, implicit stevedore conversion to string exists: > > sub vcl_backend_response { > set beresp.storage_hint = storage.Transient; > } > > Suggestions: > > A) Keep the _hint to allow backend- or vmod-driven _loose_ storage > selection. > > B) Maybe introduce besresp.storage to avoid conversions to and from > STRING, but allow NULL to behave the same as 4) for the _hint. > > C) Be consistent when storage selection is introduced for the request body. > > Cheers, > Dridi > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Mon Oct 10 16:55:29 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 10 Oct 2016 16:55:29 +0000 Subject: VCL storage discussion summary In-Reply-To: References: Message-ID: <75110.1476118529@critter.freebsd.dk> -------- In message , Federico Schwindt writes: >If setting or using a storage cannot fail I'd say kill _hint (eventually) >and just use storage. > >If there is a chance of failing, I'd keep the _hint. I'd avoid having both. >Having backend_hint and backend is confusing enough already. I agree, and I'd like to kill backend_hint as well. That said, in both cases we need to deal with "no setting" and "vmod returning NULL". For backend, it's the default backend. Is it Transient for storage ? If so, I think we have a decision... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From fgsch at lodoss.net Mon Oct 10 17:07:43 2016 From: fgsch at lodoss.net (Federico Schwindt) Date: Mon, 10 Oct 2016 18:07:43 +0100 Subject: VCL storage discussion summary In-Reply-To: <75110.1476118529@critter.freebsd.dk> References: <75110.1476118529@critter.freebsd.dk> Message-ID: Assuming this will depend on whether this is the client or the backend side (request vs response), yes, for the request body Transient being the default makes sense (current behaviour). On Mon, Oct 10, 2016 at 5:55 PM, Poul-Henning Kamp wrote: > -------- > In message ctATgu2QwFTqvxxA at mail.gmail.com> > , Federico Schwindt writes: > > >If setting or using a storage cannot fail I'd say kill _hint (eventually) > >and just use storage. > > > >If there is a chance of failing, I'd keep the _hint. I'd avoid having > both. > >Having backend_hint and backend is confusing enough already. > > I agree, and I'd like to kill backend_hint as well. > > That said, in both cases we need to deal with "no setting" and "vmod > returning NULL". > > For backend, it's the default backend. > > Is it Transient for storage ? > > If so, I think we have a decision... > > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Mon Oct 10 17:08:46 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 10 Oct 2016 17:08:46 +0000 Subject: VCL storage discussion summary In-Reply-To: References: <75110.1476118529@critter.freebsd.dk> Message-ID: <75176.1476119326@critter.freebsd.dk> -------- In message , Federico Schwindt writes: >Assuming this will depend on whether this is the client or the backend side >(request vs response), yes, > >for the request body Transient being the default makes sense (current >behaviour). You left out what will make sense on backend side ? I don't see an alternative to Transient, and I might even advocate pass... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From fgsch at lodoss.net Mon Oct 10 17:15:06 2016 From: fgsch at lodoss.net (Federico Schwindt) Date: Mon, 10 Oct 2016 18:15:06 +0100 Subject: VCL storage discussion summary In-Reply-To: <75176.1476119326@critter.freebsd.dk> References: <75110.1476118529@critter.freebsd.dk> <75176.1476119326@critter.freebsd.dk> Message-ID: Doesn't that depend on whether is a cache insertion or pass? Are we considering changing that? On Mon, Oct 10, 2016 at 6:08 PM, Poul-Henning Kamp wrote: > -------- > In message cy_rR2Xju87UiXAw at mail.gmail.com> > , Federico Schwindt writes: > > >Assuming this will depend on whether this is the client or the backend > side > >(request vs response), yes, > > > >for the request body Transient being the default makes sense (current > >behaviour). > > You left out what will make sense on backend side ? > > I don't see an alternative to Transient, and I might even advocate pass... > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Mon Oct 10 17:33:13 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 10 Oct 2016 17:33:13 +0000 Subject: VCL storage discussion summary In-Reply-To: References: <75110.1476118529@critter.freebsd.dk> <75176.1476119326@critter.freebsd.dk> Message-ID: <75267.1476120793@critter.freebsd.dk> -------- In message , Federico Schwindt writes: >Doesn't that depend on whether is a cache insertion or pass? > >Are we considering changing that? If we have NULL for stevedore and it is an insert, I would consider making it pass+Transient, to limit the amount of non-placed storage we need. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From fgsch at lodoss.net Mon Oct 10 17:37:56 2016 From: fgsch at lodoss.net (Federico Schwindt) Date: Mon, 10 Oct 2016 18:37:56 +0100 Subject: VCL storage discussion summary In-Reply-To: <75267.1476120793@critter.freebsd.dk> References: <75110.1476118529@critter.freebsd.dk> <75176.1476119326@critter.freebsd.dk> <75267.1476120793@critter.freebsd.dk> Message-ID: This might be obvious but are we considering NULL a "no setting" as well or only when a VMOD returns NULL? If it's the latter I agree. The former would be a breaking change. On Mon, Oct 10, 2016 at 6:33 PM, Poul-Henning Kamp wrote: > -------- > In message gG88CEL9Uv-K74oQ at mail.gmail.com> > , Federico Schwindt writes: > > >Doesn't that depend on whether is a cache insertion or pass? > > > >Are we considering changing that? > > If we have NULL for stevedore and it is an insert, I would consider > making it pass+Transient, to limit the amount of non-placed storage > we need. > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Mon Oct 10 17:42:41 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 10 Oct 2016 17:42:41 +0000 Subject: VCL storage discussion summary In-Reply-To: References: <75110.1476118529@critter.freebsd.dk> <75176.1476119326@critter.freebsd.dk> <75267.1476120793@critter.freebsd.dk> Message-ID: <75289.1476121361@critter.freebsd.dk> -------- In message , Federico Schwindt writes: >This might be obvious but are we considering NULL a "no setting" as well or >only when a VMOD returns NULL? > >If it's the latter I agree. The former would be a breaking change. Well, that's why I'm asking: If vmod returns NULL does it mean A) Failure to select stevedore B) Use default stevedore selection ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From fgsch at lodoss.net Mon Oct 10 19:18:41 2016 From: fgsch at lodoss.net (Federico Schwindt) Date: Mon, 10 Oct 2016 20:18:41 +0100 Subject: VCL storage discussion summary In-Reply-To: <75289.1476121361@critter.freebsd.dk> References: <75110.1476118529@critter.freebsd.dk> <75176.1476119326@critter.freebsd.dk> <75267.1476120793@critter.freebsd.dk> <75289.1476121361@critter.freebsd.dk> Message-ID: What about something like this: - NULL or "no setting" means default storage (this is the same as no backend). - default means the first non-Transient storage for cache insertions, Transient for anything else (pass and request bodies). - No more RR on storages by default. If this is wanted, it should be set explicitly somehow (VMOD?) If we failed to allocate using the selected storage we fallback to Transient, always. If Transient fails because it's capped and we couldn't make space, we fail hard. Comments? On Mon, Oct 10, 2016 at 6:42 PM, Poul-Henning Kamp wrote: > -------- > In message RTqa686AkS4g at mail.gmail.com> > , Federico Schwindt writes: > > >This might be obvious but are we considering NULL a "no setting" as well > or > >only when a VMOD returns NULL? > > > >If it's the latter I agree. The former would be a breaking change. > > Well, that's why I'm asking: > > If vmod returns NULL does it mean > > A) Failure to select stevedore > > B) Use default stevedore selection > > ? > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Mon Oct 10 19:23:40 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 10 Oct 2016 19:23:40 +0000 Subject: VCL storage discussion summary In-Reply-To: References: <75110.1476118529@critter.freebsd.dk> <75176.1476119326@critter.freebsd.dk> <75267.1476120793@critter.freebsd.dk> <75289.1476121361@critter.freebsd.dk> Message-ID: <75710.1476127420@critter.freebsd.dk> -------- In message , Federico Schwindt writes: >- No more RR on storages by default. If this is wanted, it should be set >explicitly somehow (VMOD?) I agree with the rest, but this on I think would be unwise, we should retain the RR behaviour. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From fgsch at lodoss.net Mon Oct 10 19:40:15 2016 From: fgsch at lodoss.net (Federico Schwindt) Date: Mon, 10 Oct 2016 20:40:15 +0100 Subject: VCL storage discussion summary In-Reply-To: <75710.1476127420@critter.freebsd.dk> References: <75110.1476118529@critter.freebsd.dk> <75176.1476119326@critter.freebsd.dk> <75267.1476120793@critter.freebsd.dk> <75289.1476121361@critter.freebsd.dk> <75710.1476127420@critter.freebsd.dk> Message-ID: Why? Is there anyone depending on this feature? When do you want to use it? Wouldn't be easier to visualise and/or explain what is going where if it's done explicitly? Also, if we allow this shouldn't be a way to disable it? My problem with this is two fold: 1, it's not documented AFAICT; and 2. people specifying the wrong storage by mistake ends up using another storage and causing evictions (seen this in a customer). Since the definition and usage is done separately, it is not that difficult to get it wrong, specially when you change one but forget to update the other. This also means that we cannot say that default means the first non-Transient storage for cache insertions. On Mon, Oct 10, 2016 at 8:23 PM, Poul-Henning Kamp wrote: > -------- > In message g5PX0Xcd6ukqn+ACdQ at mail.gmail.com> > , Federico Schwindt writes: > > >- No more RR on storages by default. If this is wanted, it should be set > >explicitly somehow (VMOD?) > > I agree with the rest, but this on I think would be unwise, we should > retain the RR behaviour. > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Mon Oct 10 19:43:48 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 10 Oct 2016 19:43:48 +0000 Subject: VCL storage discussion summary In-Reply-To: References: <75110.1476118529@critter.freebsd.dk> <75176.1476119326@critter.freebsd.dk> <75267.1476120793@critter.freebsd.dk> <75289.1476121361@critter.freebsd.dk> <75710.1476127420@critter.freebsd.dk> Message-ID: <75762.1476128628@critter.freebsd.dk> -------- In message , Federico Schwindt writes: >Why? Is there anyone depending on this feature? Pretty much anyone with two -s arguments are, and they probably dont know it. >Wouldn't be easier to visualise and/or explain what is going where if it's >done explicitly? This doesn't preclude doing it explicitly, it merely maintains existing configs working. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From fgsch at lodoss.net Mon Oct 10 22:19:19 2016 From: fgsch at lodoss.net (Federico Schwindt) Date: Mon, 10 Oct 2016 23:19:19 +0100 Subject: VCL storage discussion summary In-Reply-To: <75762.1476128628@critter.freebsd.dk> References: <75110.1476118529@critter.freebsd.dk> <75176.1476119326@critter.freebsd.dk> <75267.1476120793@critter.freebsd.dk> <75289.1476121361@critter.freebsd.dk> <75710.1476127420@critter.freebsd.dk> <75762.1476128628@critter.freebsd.dk> Message-ID: In my experience setups defining multiple storages are few and they use them explicitly (in VCL). While I'm not necessarily advocating this change I think this will be closer to how someone would expect it to work. Waiting for the next major release and documenting the change might do the trick. On Mon, Oct 10, 2016 at 8:43 PM, Poul-Henning Kamp wrote: > -------- > In message gmail.com> > , Federico Schwindt writes: > > >Why? Is there anyone depending on this feature? > > Pretty much anyone with two -s arguments are, and they probably dont know > it. > > >Wouldn't be easier to visualise and/or explain what is going where if it's > >done explicitly? > > This doesn't preclude doing it explicitly, it merely maintains existing > configs working. > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fgsch at lodoss.net Tue Oct 11 13:28:43 2016 From: fgsch at lodoss.net (Federico Schwindt) Date: Tue, 11 Oct 2016 14:28:43 +0100 Subject: VCL storage discussion summary In-Reply-To: References: <75110.1476118529@critter.freebsd.dk> <75176.1476119326@critter.freebsd.dk> <75267.1476120793@critter.freebsd.dk> <75289.1476121361@critter.freebsd.dk> <75710.1476127420@critter.freebsd.dk> <75762.1476128628@critter.freebsd.dk> Message-ID: Regardless of the RR, any comments or guidelines to move this forward? Should we stick to the _hint and gc in the next major release? On Mon, Oct 10, 2016 at 11:19 PM, Federico Schwindt wrote: > In my experience setups defining multiple storages are few and they use > them explicitly (in VCL). > > While I'm not necessarily advocating this change I think this will be > closer to how someone would expect it to work. > Waiting for the next major release and documenting the change might do the > trick. > > On Mon, Oct 10, 2016 at 8:43 PM, Poul-Henning Kamp > wrote: > >> -------- >> In message > ail.com> >> , Federico Schwindt writes: >> >> >Why? Is there anyone depending on this feature? >> >> Pretty much anyone with two -s arguments are, and they probably dont know >> it. >> >> >Wouldn't be easier to visualise and/or explain what is going where if >> it's >> >done explicitly? >> >> This doesn't preclude doing it explicitly, it merely maintains existing >> configs working. >> >> -- >> Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 >> phk at FreeBSD.ORG | TCP/IP since RFC 956 >> FreeBSD committer | BSD since 4.3-tahoe >> Never attribute to malice what can adequately be explained by >> incompetence. >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Tue Oct 11 20:12:24 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 11 Oct 2016 20:12:24 +0000 Subject: VCL storage discussion summary In-Reply-To: References: <75110.1476118529@critter.freebsd.dk> <75176.1476119326@critter.freebsd.dk> <75267.1476120793@critter.freebsd.dk> <75289.1476121361@critter.freebsd.dk> <75710.1476127420@critter.freebsd.dk> <75762.1476128628@critter.freebsd.dk> Message-ID: <95407.1476216744@critter.freebsd.dk> -------- In message , Federico Schwindt writes: >Regardless of the RR, any comments or guidelines to move this forward? Having read the thread again, I suggest: Before vcl_backend_response{} we point beresp->storage to the next stevedore RR-wise. (Today we do it later, because it is a hint). If at the end of v_b_r{} beresp->storage is non-NULL, we use that stevedore and only that stevedore. If at the end of v_b_r{} beresp->storage is NULL, we take it to mean storage failure and we 50x. That means you can still salvage inside v_b_r{}: beresp->stevedore = vmod.forklift(); if (!beresp->stevedore) { beresp->stevedore = Transient; beresp->uncacheable = True; } For compatibility, if VCL sets beresp->storage_hint, and the string is a stevedore name, we *also* set beresp->storage, but on failure we leave it alone. I belive that gives the semantics you desire in a POLA compliant fashion ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From fgsch at lodoss.net Wed Oct 12 08:29:17 2016 From: fgsch at lodoss.net (Federico Schwindt) Date: Wed, 12 Oct 2016 09:29:17 +0100 Subject: VCL storage discussion summary In-Reply-To: <95407.1476216744@critter.freebsd.dk> References: <75110.1476118529@critter.freebsd.dk> <75176.1476119326@critter.freebsd.dk> <75267.1476120793@critter.freebsd.dk> <75289.1476121361@critter.freebsd.dk> <75710.1476127420@critter.freebsd.dk> <75762.1476128628@critter.freebsd.dk> <95407.1476216744@critter.freebsd.dk> Message-ID: In other words, for cache insertions default / no setting means RR and NULL means failure? I believe that'd work. About this: > If at the end of v_b_r{} beresp->storage is non-NULL, we use that stevedore and only that stevedore. If someone sets the wrong storage, because e.g. they had a typo, we will fail the request or fallback to Transient? On Tue, Oct 11, 2016 at 9:12 PM, Poul-Henning Kamp wrote: > -------- > In message mail.gmail.com> > , Federico Schwindt writes: > > >Regardless of the RR, any comments or guidelines to move this forward? > > Having read the thread again, I suggest: > > Before vcl_backend_response{} we point beresp->storage to the next > stevedore RR-wise. (Today we do it later, because it is a hint). > > If at the end of v_b_r{} beresp->storage is non-NULL, we use that > stevedore and only that stevedore. > > If at the end of v_b_r{} beresp->storage is NULL, we take it to > mean storage failure and we 50x. > > That means you can still salvage inside v_b_r{}: > > beresp->stevedore = vmod.forklift(); > if (!beresp->stevedore) { > beresp->stevedore = Transient; > beresp->uncacheable = True; > } > > For compatibility, if VCL sets beresp->storage_hint, and the string > is a stevedore name, we *also* set beresp->storage, but on failure > we leave it alone. > > I belive that gives the semantics you desire in a POLA compliant fashion ? > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Wed Oct 12 08:32:21 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 12 Oct 2016 08:32:21 +0000 Subject: VCL storage discussion summary In-Reply-To: References: <75110.1476118529@critter.freebsd.dk> <75176.1476119326@critter.freebsd.dk> <75267.1476120793@critter.freebsd.dk> <75289.1476121361@critter.freebsd.dk> <75710.1476127420@critter.freebsd.dk> <75762.1476128628@critter.freebsd.dk> <95407.1476216744@critter.freebsd.dk> Message-ID: <67097.1476261141@critter.freebsd.dk> -------- In message , Federico Schwindt writes: >In other words, for cache insertions default / no setting means RR and NULL >means failure? yes. >I believe that'd work. >About this: > >> If at the end of v_b_r{} beresp->storage is non-NULL, we use that stevedore >and only that stevedore. > >If someone sets the wrong storage, because e.g. they had a typo, we will >fail the request or fallback to Transient? beresp->storage is typed and type-checked, so if you have a typo, the compiler complains unless you happen to name one of your other stevedores with the typo. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Wed Oct 12 08:54:01 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 12 Oct 2016 10:54:01 +0200 Subject: VCL storage discussion summary In-Reply-To: <67097.1476261141@critter.freebsd.dk> References: <75110.1476118529@critter.freebsd.dk> <75176.1476119326@critter.freebsd.dk> <75267.1476120793@critter.freebsd.dk> <75289.1476121361@critter.freebsd.dk> <75710.1476127420@critter.freebsd.dk> <75762.1476128628@critter.freebsd.dk> <95407.1476216744@critter.freebsd.dk> <67097.1476261141@critter.freebsd.dk> Message-ID: Hi, Apologies for coming back late on this topic. After some discussions on IRC I realized I should mention that my suggestions were non-breaking because the next major is supposedly "only" a 5.1 so I figured we'd maintain VCL compatibility. I'm not against removing beresp.storage_hint as long as we have an alternative like STEVEDORE std.storage_hint(STRING[, STEVEDORE]). I believe Nils is on the same page regarding this point. On Mon, Oct 10, 2016 at 9:18 PM, Federico Schwindt wrote: > What about something like this: > > - NULL or "no setting" means default storage (this is the same as no > backend). It may be the case for req.backend_hint, but it isn't for bereq.backend. That would be where I draw the line with a hint: it'd have a graceful fallback. On Wed, Oct 12, 2016 at 10:32 AM, Poul-Henning Kamp wrote: > -------- > In message > , Federico Schwindt writes: > >>In other words, for cache insertions default / no setting means RR and NULL >>means failure? > > yes. How about aligning on backends, and make the first storage (besides Transient) be the default unless there's a stevedore actually called `default'? RR storage selection could be moved to std and be made explicit. That is fine if we rename beresp.storage_hint to just beresp.storage and back the change with proper docs. On Tue, Oct 11, 2016 at 10:12 PM, Poul-Henning Kamp wrote: > Having read the thread again, I suggest: > > Before vcl_backend_response{} we point beresp->storage to the next > stevedore RR-wise. (Today we do it later, because it is a hint). Or to a default, see above. > If at the end of v_b_r{} beresp->storage is non-NULL, we use that > stevedore and only that stevedore. Agreed. > If at the end of v_b_r{} beresp->storage is NULL, we take it to > mean storage failure and we 50x. Agreed. > That means you can still salvage inside v_b_r{}: > > beresp->stevedore = vmod.forklift(); > if (!beresp->stevedore) { > beresp->stevedore = Transient; > beresp->uncacheable = True; > } > > For compatibility, if VCL sets beresp->storage_hint, and the string > is a stevedore name, we *also* set beresp->storage, but on failure > we leave it alone. I would rather retire beresp.storage_hint as soon as beresp.storage lands and avoid confusion between both. > I belive that gives the semantics you desire in a POLA compliant fashion ? Moving RR selection to std would be even more POLA compliant IMHO. Cheers, Dridi From phk at phk.freebsd.dk Wed Oct 12 09:08:30 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 12 Oct 2016 09:08:30 +0000 Subject: VCL storage discussion summary In-Reply-To: References: <75110.1476118529@critter.freebsd.dk> <75176.1476119326@critter.freebsd.dk> <75267.1476120793@critter.freebsd.dk> <75289.1476121361@critter.freebsd.dk> <75710.1476127420@critter.freebsd.dk> <75762.1476128628@critter.freebsd.dk> <95407.1476216744@critter.freebsd.dk> <67097.1476261141@critter.free bsd.dk> Message-ID: <67221.1476263310@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >I would rather retire beresp.storage_hint as soon as beresp.storage >lands and avoid confusion between both. We can do that at the next major rev. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From fgsch at lodoss.net Wed Oct 12 09:22:10 2016 From: fgsch at lodoss.net (Federico Schwindt) Date: Wed, 12 Oct 2016 10:22:10 +0100 Subject: VCL storage discussion summary In-Reply-To: <67097.1476261141@critter.freebsd.dk> References: <75110.1476118529@critter.freebsd.dk> <75176.1476119326@critter.freebsd.dk> <75267.1476120793@critter.freebsd.dk> <75289.1476121361@critter.freebsd.dk> <75710.1476127420@critter.freebsd.dk> <75762.1476128628@critter.freebsd.dk> <95407.1476216744@critter.freebsd.dk> <67097.1476261141@critter.freebsd.dk> Message-ID: OK, all makes sense now. I will work on the PR. On Wed, Oct 12, 2016 at 9:32 AM, Poul-Henning Kamp wrote: > -------- > In message 15wLQ at mail.gmail.com> > , Federico Schwindt writes: > > >In other words, for cache insertions default / no setting means RR and > NULL > >means failure? > > yes. > > >I believe that'd work. > > >About this: > > > >> If at the end of v_b_r{} beresp->storage is non-NULL, we use that > stevedore > >and only that stevedore. > > > >If someone sets the wrong storage, because e.g. they had a typo, we will > >fail the request or fallback to Transient? > > beresp->storage is typed and type-checked, so if you have a typo, > the compiler complains unless you happen to name one of your other > stevedores with the typo. > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > -------------- next part -------------- An HTML attachment was scrubbed... URL: