From slink at schokola.de Mon Dec 1 09:19:23 2014 From: slink at schokola.de (Nils Goroll) Date: Mon, 01 Dec 2014 10:19:23 +0100 Subject: [PATCH] Finish the director (and close the backend connection) also, for retry in backend_error Message-ID: <547C329B.5050408@schokola.de> We should revisit this later to check if we don't close the backend connection unnecessarily. Fixes #1638 -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Finish-the-director-and-close-the-backend-connection.patch Type: text/x-patch Size: 3253 bytes Desc: not available URL: From lkarsten at varnish-software.com Mon Dec 1 10:20:06 2014 From: lkarsten at varnish-software.com (Lasse Karstensen) Date: Mon, 1 Dec 2014 11:20:06 +0100 Subject: [PATCH] Backport a fix for CLI protocol failure to 3.0.6 In-Reply-To: <5470A31F.8030901@uplex.de> References: <5470A31F.8030901@uplex.de> Message-ID: <20141201102005.GA18133@immer.varnish-software.com> On Sat, Nov 22, 2014 at 03:52:15PM +0100, Geoff Simmons wrote: > The patch in the attachment backports commit 2144dc78, a fix for > mgt_cli.c, to version 3.0.6. The fix catches one of conditions for > calling MGT_Child_Cli_Fail() that had previously been missed. The patch > would have to be applied to the 3.0 branch. [..] > So if there's ever a 3.0.7, then I suggest that this fix be included. Applied to 3.0 branch. -- Lasse Karstensen Varnish Software AS From phk at phk.freebsd.dk Mon Dec 1 11:30:12 2014 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 01 Dec 2014 11:30:12 +0000 Subject: [PATCH] Finish the director (and close the backend connection) also, for retry in backend_error In-Reply-To: <547C329B.5050408@schokola.de> References: <547C329B.5050408@schokola.de> Message-ID: <85140.1417433412@critter.freebsd.dk> -------- In message <547C329B.5050408 at schokola.de>, Nils Goroll writes: >We should revisit this later to check if we don't close the backend connection >unnecessarily. > >Fixes #1638 It doesn't seem to do so for me, can you double-check ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From slink at schokola.de Mon Dec 1 18:00:08 2014 From: slink at schokola.de (Nils Goroll) Date: Mon, 01 Dec 2014 19:00:08 +0100 Subject: when to close backend connection for retry? In-Reply-To: <85140.1417433412@critter.freebsd.dk> References: <547C329B.5050408@schokola.de> <85140.1417433412@critter.freebsd.dk> Message-ID: <547CACA8.6000702@schokola.de> >We should revisit this later to check if we don't close the backend connection >unnecessarily. In vbe_dir_finish() via VDI_Finish() we close the backend connection for bo->doclose != SC_NULL For a retry, we force a close via SC_RESP_CLOSE if bo->director_state != DIR_S_NULL Can we rely on the director_state to determine if we need to close or not? Also, instead of closing we could drain-read the body, but I agree that closing should be more efficient for many/most? backends, although, IIUC, I have once come across one which didn't like closing. Nils From slink at schokola.de Mon Dec 1 20:42:17 2014 From: slink at schokola.de (Nils Goroll) Date: Mon, 01 Dec 2014 21:42:17 +0100 Subject: [PATCH] Need to do more work when retrying In-Reply-To: <547C329B.5050408@schokola.de> References: <547C329B.5050408@schokola.de> Message-ID: <547CD2A9.4060204@schokola.de> (supersedes "Finish the director...") - close the backend connection if required - finish the director - reset bo attributes - reset VFPs - signal bo readers that we had a failure if required Fixes #1638 Fixes #1641 -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Need-to-do-more-work-when-retrying.patch Type: text/x-patch Size: 5354 bytes Desc: not available URL: From fgsch at lodoss.net Sun Dec 7 18:19:38 2014 From: fgsch at lodoss.net (Federico Schwindt) Date: Sun, 7 Dec 2014 18:19:38 +0000 Subject: [PATCH] Improve range support Message-ID: Hi, Someone at the SF summit mentioned this so here's a patch for handling not satisfiable ranges (error code 416) and fixing some corner cases. I took the liberty of renaming some variables to follow the spec more closely. I've also changed V1D_Deliver() to only add Accept-Ranges for 200 responses. I believe this makes more sense and it's what another web server does but I'm fine either way. Tests updated. Comments? OKs? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Improve-Range-handling.patch Type: text/x-patch Size: 8198 bytes Desc: not available URL: From geoff at uplex.de Sun Dec 7 19:08:36 2014 From: geoff at uplex.de (Geoff Simmons) Date: Sun, 07 Dec 2014 20:08:36 +0100 Subject: [PATCH] Optionally configure a separate port for health probes Message-ID: <5484A5B4.7050405@uplex.de> Hello all, With the patch in the attachment, you can add a "port" field to a probe definition, so that a backend can respond to health probes from a different port. By default, probes work as before (the port is the same as in the backend definition if you don't have the port field), so existing configs are unchanged. This is a Varnish 4 re-implementation of a solution we've had working in production under Varnish 3 for about a year and a half. Our use case is a third-party app as the backend, for which the vendor supplies a health check under a different port, and our developers can't easily change that. (The app is SD LFredhopper.) The patch adds some vtc tests that are not in order in the numbering (starting at c00100.vtc and v00100.vtc), so as not to overlap with anyone else's tests. But of course they can be easily renumbered. Best, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-add-a-port-field-to-probe-definitions-to-allow-backe.patch Type: text/x-patch Size: 14259 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From fgsch at lodoss.net Mon Dec 8 09:34:09 2014 From: fgsch at lodoss.net (Federico Schwindt) Date: Mon, 8 Dec 2014 09:34:09 +0000 Subject: [PATCH] Improve range support In-Reply-To: References: Message-ID: And here's a minimal version without renames or removing Accept-Ranges from !200 responses. On Sun, Dec 7, 2014 at 6:19 PM, Federico Schwindt wrote: > Hi, > > Someone at the SF summit mentioned this so here's a patch for handling not > satisfiable ranges (error code 416) and fixing some corner cases. > I took the liberty of renaming some variables to follow the spec more > closely. > I've also changed V1D_Deliver() to only add Accept-Ranges for 200 > responses. I believe this makes more sense and it's what another web server > does but I'm fine either way. > Tests updated. > > Comments? OKs? > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Improve-Range-handling.patch Type: text/x-patch Size: 5157 bytes Desc: not available URL: From fgsch at lodoss.net Mon Dec 8 17:18:45 2014 From: fgsch at lodoss.net (Federico Schwindt) Date: Mon, 8 Dec 2014 17:18:45 +0000 Subject: Range support Message-ID: Hi, phk asked me to summarise what's wrong with our range support. - We don't support multiple ranges (e.g. Range: bytes=0-500,5000-). This is regardless of streaming. - If streaming is enabled, the object is not in the cache, we have a range request and it doesn't specify the last-byte-pos (e.g. bytes=F-), it is a suffix range (e.g. bytes=-L) or a full range (e.g. bytes=F-L) we might return the wrong range based on the so-far fetched object. Using #1643 as base for this example, the following cases will be wrong: bytes=0- will return Content-Range: 0-65535/65536 bytes=65536- will return 416 bytes=32768-131072 will return Content-Range: 32768-65535/65536 bytes=131072-262144 will return 416 bytes=-131072 will return Content-Range: 0-65535/65536 We can send * if we don't know the Content-Length but we need to ensure that we have the minimum required to fulfill the range request. As it stands today, for ranges where first-byte-pos is present but not last-byte-pos (first 2 examples above), turning streaming off would be equivalent as we don't support range requests to the backend. For the remaining 3 examples we need to get up to last-byte-pos in case 3 and 4 and use * if C-L is unknown. For the last case we'll need to fetch the whole object. Unless I'm missing something that's all. Salute. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fgsch at lodoss.net Wed Dec 10 20:40:47 2014 From: fgsch at lodoss.net (Federico Schwindt) Date: Wed, 10 Dec 2014 20:40:47 +0000 Subject: [PATCH 1/2] Permit to compare 2 times using standard operators Message-ID: Useful with upcoming std.rfc1123(). See next mail. Comments? OK? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Permit-to-compare-2-times-using-standard-operators.patch Type: text/x-patch Size: 699 bytes Desc: not available URL: From fgsch at lodoss.net Wed Dec 10 20:40:50 2014 From: fgsch at lodoss.net (Federico Schwindt) Date: Wed, 10 Dec 2014 20:40:50 +0000 Subject: [PATCH 2/2] Add std.rfc1123() to convert a formatted datetime to VCL_TIME Message-ID: Depends on previous email "[PATCH 1/2] Permit to compare 2 times using standard operators". Useful to check if e.g. Last-Modified is older than a given time. Comments? OK? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-Add-std.rfc1123-to-convert-a-formatted-datetime-to-V.patch Type: text/x-patch Size: 2529 bytes Desc: not available URL: From fgsch at lodoss.net Sun Dec 14 18:05:52 2014 From: fgsch at lodoss.net (Federico Schwindt) Date: Sun, 14 Dec 2014 18:05:52 +0000 Subject: [PATCH] Add STRING + REAL support Message-ID: Attached. Comments? OK? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Add-STRING-REAL-support.patch Type: text/x-patch Size: 1583 bytes Desc: not available URL: From phk at phk.freebsd.dk Mon Dec 15 06:48:13 2014 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 15 Dec 2014 06:48:13 +0000 Subject: [PATCH 1/2] Permit to compare 2 times using standard operators In-Reply-To: References: Message-ID: <46824.1418626093@critter.freebsd.dk> -------- In message , Fede rico Schwindt writes: >Useful with upcoming std.rfc1123(). > >See next mail. > >Comments? OK? OK. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Mon Dec 15 06:47:08 2014 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 15 Dec 2014 06:47:08 +0000 Subject: [PATCH] Add STRING + REAL support In-Reply-To: References: Message-ID: <46793.1418626028@critter.freebsd.dk> -------- In message , Fede rico Schwindt writes: >Attached. > >Comments? OK? OK. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Mon Dec 15 06:51:28 2014 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 15 Dec 2014 06:51:28 +0000 Subject: [PATCH 2/2] Add std.rfc1123() to convert a formatted datetime to VCL_TIME In-Reply-To: References: Message-ID: <46852.1418626288@critter.freebsd.dk> -------- In message , Fede rico Schwindt writes: >Depends on previous email "[PATCH 1/2] Permit to compare 2 times using >standard operators". > >Useful to check if e.g. Last-Modified is older than a given time. Shouldn't this be named vmod.time() to match the other conversion functions ? rfc1123 certainly seems to be a far too obscure name for the job performed. Otherwise OK -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From fgsch at lodoss.net Mon Dec 15 09:50:15 2014 From: fgsch at lodoss.net (Federico Schwindt) Date: Mon, 15 Dec 2014 09:50:15 +0000 Subject: [PATCH 2/2] Add std.rfc1123() to convert a formatted datetime to VCL_TIME In-Reply-To: <46852.1418626288@critter.freebsd.dk> References: <46852.1418626288@critter.freebsd.dk> Message-ID: I thought about this and decided to go for rfc1123() as time() would indicate a TIME input with .integer(), .ip() and .real(). While I agree that rfc1123() is a bit obscure it hints that the input requires some standard format. Actually I should also document the different formats supported. The other options I had in mind were datetime() and date() but I liked rfc1123() best. On Mon, Dec 15, 2014 at 6:51 AM, Poul-Henning Kamp wrote: > > -------- > In message yomOB-VY6F0CFkxXA at mail.gmail.com>, Fede > rico Schwindt writes: > > >Depends on previous email "[PATCH 1/2] Permit to compare 2 times using > >standard operators". > > > >Useful to check if e.g. Last-Modified is older than a given time. > > Shouldn't this be named vmod.time() to match the other conversion > functions ? > > rfc1123 certainly seems to be a far too obscure name for the job performed. > > Otherwise OK > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fgsch at lodoss.net Mon Dec 15 10:04:54 2014 From: fgsch at lodoss.net (Federico Schwindt) Date: Mon, 15 Dec 2014 10:04:54 +0000 Subject: [PATCH 2/2] Add std.rfc1123() to convert a formatted datetime to VCL_TIME In-Reply-To: References: <46852.1418626288@critter.freebsd.dk> Message-ID: Talking of which, isn't this format in vtim.c "%F %T", /* ISO 8601 */ Missing the T between the date and time and maybe the trailing Z? On Mon, Dec 15, 2014 at 9:50 AM, Federico Schwindt wrote: > > I thought about this and decided to go for rfc1123() as time() would > indicate a TIME input with .integer(), .ip() and .real(). > > While I agree that rfc1123() is a bit obscure it hints that the input > requires some standard format. Actually I should also document the > different formats supported. > > The other options I had in mind were datetime() and date() but I liked > rfc1123() best. > > On Mon, Dec 15, 2014 at 6:51 AM, Poul-Henning Kamp > wrote: >> >> -------- >> In message > yomOB-VY6F0CFkxXA at mail.gmail.com>, Fede >> rico Schwindt writes: >> >> >Depends on previous email "[PATCH 1/2] Permit to compare 2 times using >> >standard operators". >> > >> >Useful to check if e.g. Last-Modified is older than a given time. >> >> Shouldn't this be named vmod.time() to match the other conversion >> functions ? >> >> rfc1123 certainly seems to be a far too obscure name for the job >> performed. >> >> Otherwise OK >> >> -- >> Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 >> phk at FreeBSD.ORG | TCP/IP since RFC 956 >> FreeBSD committer | BSD since 4.3-tahoe >> Never attribute to malice what can adequately be explained by >> incompetence. >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at uplex.de Mon Dec 15 16:11:51 2014 From: geoff at uplex.de (Geoff Simmons) Date: Mon, 15 Dec 2014 17:11:51 +0100 Subject: [PATCH] define AS_COPY_VAR in varnish.m4 if not already defined Message-ID: <548F0847.5000306@uplex.de> Hallo all, This patch makes it possible for autoconf < 2.63b (which is the unfortunate situation on distros such as RHEL 6.5) to interpret the m4 macros needed for VMOD builds. AS_COPY_VAR is left alone if already defined, so the patch should have no effect on systems with a recent autoconf. Best, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-define-AS_COPY_VAR-in-varnish.m4-if-not-already-defi.patch Type: text/x-patch Size: 764 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From phk at phk.freebsd.dk Mon Dec 15 20:12:08 2014 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 15 Dec 2014 20:12:08 +0000 Subject: [PATCH 2/2] Add std.rfc1123() to convert a formatted datetime to VCL_TIME In-Reply-To: References: <46852.1418626288@critter.freebsd.dk> Message-ID: <67700.1418674328@critter.freebsd.dk> -------- In message , Federico Schwindt writes: >I thought about this and decided to go for rfc1123() as time() would >indicate a TIME input with .integer(), .ip() and .real(). I simply don't grok that. The idea behind the conversion functions is std.${typename}(${convert_this}, ${use_this_if_you_cant}) Nothing prevents std.time("Mon Dec 15 20:11:04 UTC 2014", std.now()) and std.time("1418674264", std.now()) from giving the same result ? >The other options I had in mind were datetime() and date() but I liked >rfc1123() best. I really hate rfc1123 as a name... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi.boukelmoune at zenika.com Mon Dec 15 21:51:07 2014 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Mon, 15 Dec 2014 22:51:07 +0100 Subject: [PATCH 2/2] Add std.rfc1123() to convert a formatted datetime to VCL_TIME In-Reply-To: <67700.1418674328@critter.freebsd.dk> References: <46852.1418626288@critter.freebsd.dk> <67700.1418674328@critter.freebsd.dk> Message-ID: Hi, As I've hit the same questions on an unrelated project, and changed my mind even though I reasonned like Federico initlially, I thought I could give my two cents. I'm also referring to the RFC 2616 because I haven't found time to go through the new ones yet, let me know if this particular concern has changed. On Mon, Dec 15, 2014 at 9:12 PM, Poul-Henning Kamp wrote: > -------- > In message > , Federico Schwindt writes: > >>I thought about this and decided to go for rfc1123() as time() would >>indicate a TIME input with .integer(), .ip() and .real(). I would go with "date" because in the RFC's grammar (3.3.1) it's called an "HTTP-date". HTTP-date = rfc1123-date | rfc850-date | asctime-date > I simply don't grok that. > > The idea behind the conversion functions is > > std.${typename}(${convert_this}, ${use_this_if_you_cant}) > > Nothing prevents > > std.time("Mon Dec 15 20:11:04 UTC 2014", std.now()) > and > std.time("1418674264", std.now()) > > from giving the same result ? It actually should (ish) do that. Again, the RFC (3.3.1) follows the robustness principle and accepts all three date formats for parsing, but allows only rfc1123-date when generating. Arguably, this is Varnish Cache and such a function could also accept any Varnish-specific formats. > HTTP/1.1 clients and servers that parse the date value MUST accept > all three formats (for compatibility with HTTP/1.0), though they MUST > only generate the RFC 1123 format for representing HTTP-date values > in header fields. See section 19.3 for further information. My two cents, Dridi >>The other options I had in mind were datetime() and date() but I liked >>rfc1123() best. > > I really hate rfc1123 as a name... > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev From ksorensen at nordija.com Mon Dec 15 22:25:12 2014 From: ksorensen at nordija.com (Kristian =?ISO-8859-1?Q?Gr=F8nfeldt_S=F8rensen?=) Date: Mon, 15 Dec 2014 23:25:12 +0100 Subject: [PATCH] Allow varnishncsa logformat to be read from a file Message-ID: <1418682312.5583.9.camel@nordija.com> Hi all, This patch allows varnishncsa to obtain the logformat string from a file instead of a command line argument. The main usecase is for sysv init scripts where shell expansion causes issues like https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=657449 . Varnishncsa will read a single line from the specified af the "-f" option, use that as the logging format and close the file. I don't really like using -f as an option, as that has a different meaning in varnishncsa 3.0.x (X-Forwarded-For vs client.ip), but I couldn't come up with a better alternative. Comments? BR Kristian S?rensen -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Allow-varnishncsa-logformat-to-be-read-from-a-file.patch Type: text/x-patch Size: 2892 bytes Desc: not available URL: From fgsch at lodoss.net Mon Dec 15 23:13:28 2014 From: fgsch at lodoss.net (Federico Schwindt) Date: Mon, 15 Dec 2014 23:13:28 +0000 Subject: [PATCH 2/2] Add std.rfc1123() to convert a formatted datetime to VCL_TIME In-Reply-To: <67700.1418674328@critter.freebsd.dk> References: <46852.1418626288@critter.freebsd.dk> <67700.1418674328@critter.freebsd.dk> Message-ID: Sure, nothing prevents it but do we want to support it? The time in "Mon Dec 15 20:11:04 UTC 2014" is the representation of a time in a string context. "1418674264" is just a number without any meaning attached to it. What stops us converting "1" to "Thu Jan 1 01:00:01 1970" if we were to support such conversion? When will the fallback be used if passing any number will return a TIME? At the end of the day I just want to have a way to convert a date as being used in the HTTP protocol to something useful. While there might be some people wanting to use a timestamp directly we already have some ways to do it and is not particularly standard. If you still want to use time() fine, modifying the patch is rather trivial. I'm just not fond of using that name because I don't think it should support a timestamp. On Mon, Dec 15, 2014 at 8:12 PM, Poul-Henning Kamp wrote: > > -------- > In message < > CAJV_h0YFuL+OMcFckxAd+D9noX__aPRyLK17WA4m5Cm6f-XdQg at mail.gmail.com> > , Federico Schwindt writes: > > >I thought about this and decided to go for rfc1123() as time() would > >indicate a TIME input with .integer(), .ip() and .real(). > > I simply don't grok that. > > The idea behind the conversion functions is > > std.${typename}(${convert_this}, ${use_this_if_you_cant}) > > Nothing prevents > > std.time("Mon Dec 15 20:11:04 UTC 2014", std.now()) > and > std.time("1418674264", std.now()) > > from giving the same result ? > > >The other options I had in mind were datetime() and date() but I liked > >rfc1123() best. > > I really hate rfc1123 as a name... > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fgsch at lodoss.net Mon Dec 15 23:23:41 2014 From: fgsch at lodoss.net (Federico Schwindt) Date: Mon, 15 Dec 2014 23:23:41 +0000 Subject: [PATCH] Fix the iso 8601 format Message-ID: As per my comment on the std.rfc1123 patch. Should it also handle the trailing Z? Comments? OK? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Fix-the-iso-8601-format.patch Type: text/x-patch Size: 1299 bytes Desc: not available URL: From fgsch at lodoss.net Tue Dec 16 14:35:41 2014 From: fgsch at lodoss.net (Federico Schwindt) Date: Tue, 16 Dec 2014 14:35:41 +0000 Subject: [PATCH] define AS_COPY_VAR in varnish.m4 if not already defined In-Reply-To: <548F0847.5000306@uplex.de> References: <548F0847.5000306@uplex.de> Message-ID: Looks good to me and others are doing the same, e.g. https://github.com/gdnsd/gdnsd/issues/85. On Mon, Dec 15, 2014 at 4:11 PM, Geoff Simmons wrote: > > Hallo all, > > This patch makes it possible for autoconf < 2.63b (which is the > unfortunate situation on distros such as RHEL 6.5) to interpret the m4 > macros needed for VMOD builds. > > AS_COPY_VAR is left alone if already defined, so the patch should have > no effect on systems with a recent autoconf. > > > Best, > Geoff > -- > ** * * UPLEX - Nils Goroll Systemoptimierung > > Scheffelstra?e 32 > 22301 Hamburg > > Tel +49 40 2880 5731 > Mob +49 176 636 90917 > Fax +49 40 42949753 > > http://uplex.de > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at varnish-software.com Tue Dec 16 17:17:20 2014 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 16 Dec 2014 18:17:20 +0100 Subject: [PATCH 1/2] Relax an assertion for the IMS update candidate object Message-ID: <1418750241-13168-1-git-send-email-martin@varnish-software.com> The streaming fetch of an object can fail between lookup time and when we actually use it. Relax this assertion. Fixes: #1647 --- bin/varnishd/cache/cache_req_fsm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index 1078d34..b2e5aef 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -356,7 +356,7 @@ cnt_lookup(struct worker *wrk, struct req *req) switch (wrk->handling) { case VCL_RET_DELIVER: if (boc != NULL) { - AZ(oc->flags & (OC_F_FAILED|OC_F_PASS)); + AZ(oc->flags & OC_F_PASS); AZ(oc->exp_flags & OC_EF_DYING); AZ(boc->busyobj); VBF_Fetch(wrk, req, boc, oc, VBF_BACKGROUND); -- 2.1.3 From martin at varnish-software.com Tue Dec 16 17:17:21 2014 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 16 Dec 2014 18:17:21 +0100 Subject: [PATCH 2/2] Handle streaming fail of the source object during condfetch In-Reply-To: <1418750241-13168-1-git-send-email-martin@varnish-software.com> References: <1418750241-13168-1-git-send-email-martin@varnish-software.com> Message-ID: <1418750241-13168-2-git-send-email-martin@varnish-software.com> Fixes: #1648 --- bin/varnishd/cache/cache_fetch.c | 9 ++++- bin/varnishtest/tests/r01648.vtc | 75 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 83 insertions(+), 1 deletion(-) create mode 100644 bin/varnishtest/tests/r01648.vtc diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index f74a5c0..b171786 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -688,6 +688,9 @@ vbf_stp_condfetch(struct worker *wrk, struct busyobj *bo) oi = ObjIterBegin(wrk, bo->ims_oc); do { ois = ObjIter(bo->ims_oc, oi, &sp, &sl); + if (ois == OIS_ERROR) { + VFP_Error(bo->vfc, "Conditional fetch object failed"); + } while (sl > 0) { l = ObjGetLen(bo->wrk, bo->ims_oc) - al; assert(l > 0); @@ -703,8 +706,12 @@ vbf_stp_condfetch(struct worker *wrk, struct busyobj *bo) } } while (!bo->vfc->failed && (ois == OIS_DATA || ois == OIS_STREAM)); ObjIterEnd(bo->ims_oc, &oi); - if (bo->vfc->failed) + if (bo->ims_oc->flags & OC_F_FAILED) + VFP_Error(bo->vfc, "Conditional fetch object failed"); + if (bo->vfc->failed) { + VDI_Finish(bo->director_resp, bo->wrk, bo); return (F_STP_FAIL); + } if (!bo->do_stream) HSH_Unbusy(wrk, bo->fetch_objcore); diff --git a/bin/varnishtest/tests/r01648.vtc b/bin/varnishtest/tests/r01648.vtc new file mode 100644 index 0000000..de0c364 --- /dev/null +++ b/bin/varnishtest/tests/r01648.vtc @@ -0,0 +1,75 @@ +varnishtest "#1648 - corrupt object in cache through IMS update" + +# This server sends a broken response body +server s1 { + rxreq + txresp -nolen -hdr "Transfer-Encoding: chunked" -hdr "Etag: \"foo\"" -hdr "Server: s1" + sema r1 sync 3 + delay 1 + chunked "abc" +} -start + +# This server validates the streaming response from s1 as it hasn't failed yet +server s2 { + rxreq + expect req.http.If-None-Match == "\"foo\"" + sema r1 sync 3 + txresp -status 304 -nolen -hdr "Server: s2" +} -start + +# This server sends the proper response body +server s3 { + rxreq + txresp -hdr "Server: s3" -body "abcdef" +} -start + +varnish v1 -vcl+backend { + sub vcl_recv { + if (req.http.client == "c1") { + set req.backend_hint = s1; + } else if (req.http.client == "c2") { + set req.backend_hint = s2; + } else if (req.http.client == "c3") { + set req.backend_hint = s3; + } + } + sub vcl_backend_response { + if (bereq.http.client == "c1") { + set beresp.ttl = 0.1s; + set beresp.grace = 0s; + set beresp.keep = 60s; + } + } +} -start + +varnish v1 -cliok "param.set debug +syncvsl" + +# This client gets a streaming failing result from s1 +client c1 { + txreq -hdr "Client: c1" + rxresphdrs + expect resp.status == 200 + expect resp.http.transfer-encoding == "chunked" +} -start + +delay 1 + +# This client gets a streaming failing result from s1 through +# IMS update by s2 +client c2 { + txreq -hdr "Client: c2" + sema r1 sync 3 + rxresphdrs + expect resp.status == 200 + expect resp.http.transfer-encoding == "chunked" +} -run + +delay 1 + +# This client should get a fresh fetch from s3 +client c3 { + txreq -hdr "Client: c3" + rxresp + expect resp.status == 200 + expect resp.body == "abcdef" +} -run -- 2.1.3 From phk at phk.freebsd.dk Tue Dec 16 21:40:13 2014 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 16 Dec 2014 21:40:13 +0000 Subject: [PATCH 1/2] Relax an assertion for the IMS update candidate object In-Reply-To: <1418750241-13168-1-git-send-email-martin@varnish-software.com> References: <1418750241-13168-1-git-send-email-martin@varnish-software.com> Message-ID: <54076.1418766013@critter.freebsd.dk> -------- In message <1418750241-13168-1-git-send-email-martin at varnish-software.com>, Mar tin Blix Grydeland writes: OK. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Tue Dec 16 21:42:31 2014 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 16 Dec 2014 21:42:31 +0000 Subject: [PATCH 2/2] Handle streaming fail of the source object during condfetch In-Reply-To: <1418750241-13168-2-git-send-email-martin@varnish-software.com> References: <1418750241-13168-1-git-send-email-martin@varnish-software.com> <1418750241-13168-2-git-send-email-martin@varnish-software.com> Message-ID: <54603.1418766151@critter.freebsd.dk> -------- In message <1418750241-13168-2-git-send-email-martin at varnish-software.com>, Mar tin Blix Grydeland writes: >Fixes: #1648 >+ if (ois == OIS_ERROR) { >+ VFP_Error(bo->vfc, "Conditional fetch object failed"); >+ } unnecessary {...} >- if (bo->vfc->failed) >+ if (bo->ims_oc->flags & OC_F_FAILED) >+ VFP_Error(bo->vfc, "Conditional fetch object failed"); That text confused me for a moment and I suspect it will have that effect on other people as well. Maybe we should call the object we IMS from "the template object" or some such ? ("model" ? "stencil" ? ...) Otherwise it looks fine. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From jyw at epochtimes.com Tue Dec 16 23:43:06 2014 From: jyw at epochtimes.com (Jingyi Wei) Date: Tue, 16 Dec 2014 18:43:06 -0500 Subject: [PATCH] Purge all expired matching objects except the most recent one during hash lookup In-Reply-To: <54906370.4000500@epochtimes.com> References: <54906370.4000500@epochtimes.com> Message-ID: <5490C38A.4040309@epochtimes.com> Hello, I see that varnish 4.0.2 does not free duplicate objects when ttl is expired but grace is not expired. Every time when ttl expired it will fetch a new object and increase memory usage. I set ttl to 1ms and grace to 1 day and did stress test. Varnish memory usage continued to increase and soon exceeded the max memory limit, but remained stable at a point. I made a testing patch on 4.0.2 source release and tried to purge all expired matching objects except the most recent one during hash lookup, so the memory of duplicate graced objects will be freed. Any comments? Thanks, Jingyi Wei -------------- next part -------------- A non-text attachment was scrubbed... Name: purge.patch Type: text/x-patch Size: 1394 bytes Desc: not available URL: From martin at varnish-software.com Wed Dec 17 13:51:44 2014 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Wed, 17 Dec 2014 14:51:44 +0100 Subject: [PATCH 1/2] Latch VDP error value Message-ID: <1418824305-8548-1-git-send-email-martin@varnish-software.com> Latch error values returned from any VDP function, making sure that all subsequent calls to VDP_bytes() becomes noops. This allows calling code to allow calling VDP_bytes() again even if a previous call returned error. Fixes: #1642 --- bin/varnishd/cache/cache.h | 1 + bin/varnishd/cache/cache_deliver_proc.c | 4 ++++ bin/varnishd/cache/cache_session.c | 1 + 3 files changed, 6 insertions(+) diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 1db9644..f250afe 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -612,6 +612,7 @@ struct req { /* Deliver pipeline */ struct vdp_entry_s vdp; struct vdp_entry *vdp_nxt; + unsigned vdp_errval; /* Transaction VSL buffer */ struct vsl_log vsl[1]; diff --git a/bin/varnishd/cache/cache_deliver_proc.c b/bin/varnishd/cache/cache_deliver_proc.c index a3701a9..a74b41f 100644 --- a/bin/varnishd/cache/cache_deliver_proc.c +++ b/bin/varnishd/cache/cache_deliver_proc.c @@ -40,6 +40,8 @@ VDP_bytes(struct req *req, enum vdp_action act, const void *ptr, ssize_t len) CHECK_OBJ_NOTNULL(req, REQ_MAGIC); assert(act == VDP_NULL || act == VDP_FLUSH); + if (req->vdp_errval) + return (req->vdp_errval); vdp = req->vdp_nxt; CHECK_OBJ_NOTNULL(vdp, VDP_ENTRY_MAGIC); req->vdp_nxt = VTAILQ_NEXT(vdp, list); @@ -47,6 +49,8 @@ VDP_bytes(struct req *req, enum vdp_action act, const void *ptr, ssize_t len) assert(act > VDP_NULL || len > 0); /* Call the present layer, while pointing to the next layer down */ retval = vdp->func(req, act, &vdp->priv, ptr, len); + if (retval) + req->vdp_errval = retval; /* Latch error value */ req->vdp_nxt = vdp; return (retval); } diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c index 60c893e..cf1c4d5 100644 --- a/bin/varnishd/cache/cache_session.c +++ b/bin/varnishd/cache/cache_session.c @@ -373,6 +373,7 @@ SES_GetReq(const struct worker *wrk, struct sess *sp) req->vdp_nxt = 0; VTAILQ_INIT(&req->vdp); + req->vdp_errval = 0; return (req); } -- 2.1.3 From martin at varnish-software.com Wed Dec 17 13:51:45 2014 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Wed, 17 Dec 2014 14:51:45 +0100 Subject: [PATCH 2/2] Improve error handling in ESI_Deliver() In-Reply-To: <1418824305-8548-1-git-send-email-martin@varnish-software.com> References: <1418824305-8548-1-git-send-email-martin@varnish-software.com> Message-ID: <1418824305-8548-2-git-send-email-martin@varnish-software.com> Break out of the ESI handling sooner when there is a delivery problem, minimizing the time and effects of a ESI delivery on a client that has gone away. --- bin/varnishd/cache/cache_esi_deliver.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/bin/varnishd/cache/cache_esi_deliver.c b/bin/varnishd/cache/cache_esi_deliver.c index 39df537..d83504e 100644 --- a/bin/varnishd/cache/cache_esi_deliver.c +++ b/bin/varnishd/cache/cache_esi_deliver.c @@ -305,7 +305,8 @@ ESI_Deliver(struct req *req) if (isgzip) { assert(sizeof gzip_hdr == 10); /* Send out the gzip header */ - (void)VDP_bytes(req, VDP_NULL, gzip_hdr, 10); + if (VDP_bytes(req, VDP_NULL, gzip_hdr, 10)) + return; req->l_crc = 0; req->gzip_resp = 1; req->crc = crc32(0L, Z_NULL, 0); @@ -347,7 +348,10 @@ ESI_Deliver(struct req *req) sl -= l2; l -= l2; - (void)VDP_bytes(req, VDP_NULL, pp, l2); + if (VDP_bytes(req, VDP_NULL, pp, l2)) { + p = e; + break; + } pp += l2; if (sl == 0) { ois = ObjIter(req->objcore, oi, @@ -390,7 +394,6 @@ ESI_Deliver(struct req *req) r = (void*)strchr((const char*)q, '\0'); AN(r); if (VDP_bytes(req, VDP_FLUSH, NULL, 0)) { - SES_Close(req->sp, SC_REM_CLOSE); p = e; break; } -- 2.1.3 From martin at varnish-software.com Wed Dec 17 13:52:59 2014 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Wed, 17 Dec 2014 14:52:59 +0100 Subject: [PATCH 1/2] Latch VDP error value In-Reply-To: <1418824305-8548-1-git-send-email-martin@varnish-software.com> References: <1418824305-8548-1-git-send-email-martin@varnish-software.com> Message-ID: Ooops, small error present in this patch. The vdp_errval should be declared as an 'int', not unsigned. Martin On 17 December 2014 at 14:51, Martin Blix Grydeland < martin at varnish-software.com> wrote: > > Latch error values returned from any VDP function, making sure that > all subsequent calls to VDP_bytes() becomes noops. This allows calling > code to allow calling VDP_bytes() again even if a previous call > returned error. > > Fixes: #1642 > --- > bin/varnishd/cache/cache.h | 1 + > bin/varnishd/cache/cache_deliver_proc.c | 4 ++++ > bin/varnishd/cache/cache_session.c | 1 + > 3 files changed, 6 insertions(+) > > diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h > index 1db9644..f250afe 100644 > --- a/bin/varnishd/cache/cache.h > +++ b/bin/varnishd/cache/cache.h > @@ -612,6 +612,7 @@ struct req { > /* Deliver pipeline */ > struct vdp_entry_s vdp; > struct vdp_entry *vdp_nxt; > + unsigned vdp_errval; > > /* Transaction VSL buffer */ > struct vsl_log vsl[1]; > diff --git a/bin/varnishd/cache/cache_deliver_proc.c > b/bin/varnishd/cache/cache_deliver_proc.c > index a3701a9..a74b41f 100644 > --- a/bin/varnishd/cache/cache_deliver_proc.c > +++ b/bin/varnishd/cache/cache_deliver_proc.c > @@ -40,6 +40,8 @@ VDP_bytes(struct req *req, enum vdp_action act, const > void *ptr, ssize_t len) > > CHECK_OBJ_NOTNULL(req, REQ_MAGIC); > assert(act == VDP_NULL || act == VDP_FLUSH); > + if (req->vdp_errval) > + return (req->vdp_errval); > vdp = req->vdp_nxt; > CHECK_OBJ_NOTNULL(vdp, VDP_ENTRY_MAGIC); > req->vdp_nxt = VTAILQ_NEXT(vdp, list); > @@ -47,6 +49,8 @@ VDP_bytes(struct req *req, enum vdp_action act, const > void *ptr, ssize_t len) > assert(act > VDP_NULL || len > 0); > /* Call the present layer, while pointing to the next layer down */ > retval = vdp->func(req, act, &vdp->priv, ptr, len); > + if (retval) > + req->vdp_errval = retval; /* Latch error value */ > req->vdp_nxt = vdp; > return (retval); > } > diff --git a/bin/varnishd/cache/cache_session.c > b/bin/varnishd/cache/cache_session.c > index 60c893e..cf1c4d5 100644 > --- a/bin/varnishd/cache/cache_session.c > +++ b/bin/varnishd/cache/cache_session.c > @@ -373,6 +373,7 @@ SES_GetReq(const struct worker *wrk, struct sess *sp) > > req->vdp_nxt = 0; > VTAILQ_INIT(&req->vdp); > + req->vdp_errval = 0; > > return (req); > } > -- > 2.1.3 > > -- *Martin Blix Grydeland* Senior Developer | Varnish Software AS Mobile: +47 992 74 756 We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From fgsch at lodoss.net Wed Dec 17 23:28:24 2014 From: fgsch at lodoss.net (Federico Schwindt) Date: Wed, 17 Dec 2014 23:28:24 +0000 Subject: heads-up: migrating to github Message-ID: Hi, I appreciate this is a somewhat controversial topic so I wanted to mention it well in advance. We're planning to migrate the varnish-cache repository to github. The exact date is not defined yet and will depend on a number of factors but I don't expect this to take place before mid February as the earliest. There are many reasons for this but the main reasons are simplicity and manageability, not only for the repository itself but also for the companion tools (issues, wiki, pull requests, etc.) and for integration with third-party services. The plan is not finalised yet but we're aiming to have a seamless migration. If you believe something is missing please let me know. The following series of QAs will hopefully cover most of the questions. While we all have our preferences I will appreciate if we concentrate on the specifics and leave the bike shedding for another day. Q. Why migrating it? A. Simplicity. Less things to manage for us, better tools available, easier for users to collaborate. Q. What will be migrated? A. wiki, issues and repository. Q. Why github and not ? A. Again, the main reason here is simplicity. We're already hosting a mirror there and switching to it, both from the user and our point of view, should be easy. We have several other projects hosted there also so makes for a common place to manage them all. A few less important reasons include the companion tools, the integration with third-party services and the user base. Q. Does this mean that github is better than ? A. Not at all. I haven't done a comparison with other providers and quite frankly neither want nor have the time to do it. If you think it's important and have the time please send us your findings but please ensure we can cover the 3 items I mentioned before. Q. Would this break the repo I cloned off from git.varnish-cache.org? A. No, we will mirror from github for the foreseeable future. Q. What will happen with trac? A. The existing tickets will be migrated to github. Q. What will happen with the wiki? A. We will switch to github. I'm not entirely sure if there is any value on migrating all the existing content but if we decide to keep all/some of it this will be migrated as well. Q. What will happen with the existing link to ? A. Redirects will be put in place for the foreseeable future. Q. When will this happen? A. We don't have a date yet. Not sooner than mid February but probably much later than that. It will likely be done in stages following the kiss principle. Q. How this will affect me? A. It should be transparent for you unless you have something other than read-only access to the repository or trac. You will need a user and github and change the origin for the former or some magic from our side for the latter. Q. Why are you sending this? A. It seems a lot of people care beyond "git remote set-url ...". If you don't please ignore this See you space cowboy! -------------- next part -------------- An HTML attachment was scrubbed... URL: From slink at schokola.de Thu Dec 18 10:43:07 2014 From: slink at schokola.de (Nils Goroll) Date: Thu, 18 Dec 2014 11:43:07 +0100 Subject: [PATCH] Improve application of the default ttl by response status, code Message-ID: <5492AFBB.4000906@schokola.de> Ref: https://tools.ietf.org/html/rfc7231#section-6.1 Changes: * 302 Found * 307 Temporary Redirect Do not apply the default ttl, only set a ttl if Cache-Control or Expires are present. Responses with these status codes are not cacheable by default * 414 Request-URI Too Large Cacheable with "heuristic expiration" -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Improve-application-of-the-default-ttl-by-response-s.patch Type: text/x-patch Size: 2396 bytes Desc: not available URL: From slink at schokola.de Thu Dec 18 12:22:54 2014 From: slink at schokola.de (Nils Goroll) Date: Thu, 18 Dec 2014 13:22:54 +0100 Subject: [PATCH] expose response 304 status to VCL Message-ID: <5492C71E.1090409@schokola.de> A non-text attachment was scrubbed... Name: 0001-expose-response-304-status-to-VCL.patch Type: text/x-patch Size: 4250 bytes Desc: not available URL: From slink at schokola.de Thu Dec 18 12:51:11 2014 From: slink at schokola.de (Nils Goroll) Date: Thu, 18 Dec 2014 13:51:11 +0100 Subject: "expire superseded" in master - Re: [PATCH] Purge all expired matching objects except the most recent one during hash lookup In-Reply-To: <5490C38A.4040309@epochtimes.com> References: <54906370.4000500@epochtimes.com> <5490C38A.4040309@epochtimes.com> Message-ID: <5492CDBF.7080504@schokola.de> Hi On 17/12/14 00:43, Jingyi Wei wrote: > I see that varnish 4.0.2 does not free duplicate objects when ttl is > expired but grace is not expired. Every time when ttl expired it will > fetch a new object and increase memory usage. I set ttl to 1ms and grace > to 1 day and did stress test. Varnish memory usage continued to increase > and soon exceeded the max memory limit, but remained stable at a point. This is an old topic which we had termed "expire superseded" in the past. We had discussed the approach you are suggesting already, but later agreed that the better solution is to expire the superseded object at fetch time. This is exactly what the fix for #1530 commit b61c2305462c9428225a9531e664907db408fd7f should do. This is in 4.0.2, so could you please make sure that the effect you are seeing is really root caused by superseded objects? Nils P.S.: IIUC your patch would throw out old objects before they got refreshed and probably would break IMS (have not checked in detail) From slink at schokola.de Thu Dec 18 13:17:51 2014 From: slink at schokola.de (Nils Goroll) Date: Thu, 18 Dec 2014 14:17:51 +0100 Subject: [PATCH] Allow varnish_reload_vcl to discard old VCL after reload In-Reply-To: <5F530A9242E7F84F999DB40E0E268FBD45EB9CBA@mercalli.lild01.pictime.fr> References: <5F530A9242E7F84F999DB40E0E268FBD45EB9CBA@mercalli.lild01.pictime.fr> Message-ID: <5492D3FF.6090701@schokola.de> We shouldn't exit 1 if vcl.discard failed because this is not an error, but other than that the patch looks good to me without testing. From slink at schokola.de Thu Dec 18 13:25:43 2014 From: slink at schokola.de (Nils Goroll) Date: Thu, 18 Dec 2014 14:25:43 +0100 Subject: [PATCH] Allow varnishncsa logformat to be read from a file In-Reply-To: <1418682312.5583.9.camel@nordija.com> References: <1418682312.5583.9.camel@nordija.com> Message-ID: <5492D5D7.3030503@schokola.de> looks ok to me, I will double check during the next bugwash I participate in. From jyw at epochtimes.com Fri Dec 19 01:14:23 2014 From: jyw at epochtimes.com (Jingyi Wei) Date: Thu, 18 Dec 2014 20:14:23 -0500 Subject: "expire superseded" in master In-Reply-To: <5492CDBF.7080504@schokola.de> References: <54906370.4000500@epochtimes.com> <5490C38A.4040309@epochtimes.com> <5492CDBF.7080504@schokola.de> Message-ID: <54937BEF.104@epochtimes.com> Hello, Thank you for your explanation. In the commit you mentioned, I see that bo->ims_obj is set in VBF_Fetch() only if Last-Modified or ETag header exists in the previous backend response. Otherwise |ims_obj| is NULL and the memory will not be freed, so the actual problem is that there's no Last-Modified header in our backend response. I commented out the check for the existence of two headers in VBF_Fetch() and reverted my patch. The old object is purged correctly in fetch step. Could you please tell me why Last-Modified or ETag must present in order to free the previous object? I looked at the places that used |ims_obj| in varnish 4.0.2, and I found no issues to remove the check. Thanks, Jingyi Wei On 12/18/14 07:51, Nils Goroll wrote: > Hi > > On 17/12/14 00:43, Jingyi Wei wrote: >> I see that varnish 4.0.2 does not free duplicate objects when ttl is >> expired but grace is not expired. Every time when ttl expired it will >> fetch a new object and increase memory usage. I set ttl to 1ms and grace >> to 1 day and did stress test. Varnish memory usage continued to increase >> and soon exceeded the max memory limit, but remained stable at a point. > This is an old topic which we had termed "expire superseded" in the past. We had > discussed the approach you are suggesting already, but later agreed that the > better solution is to expire the superseded object at fetch time. > > This is exactly what the fix for #1530 commit > b61c2305462c9428225a9531e664907db408fd7f should do. This is in 4.0.2, so could > you please make sure that the effect you are seeing is really root caused by > superseded objects? > > Nils > > P.S.: IIUC your patch would throw out old objects before they got refreshed and > probably would break IMS (have not checked in detail) > > From slink at schokola.de Fri Dec 19 09:36:19 2014 From: slink at schokola.de (Nils Goroll) Date: Fri, 19 Dec 2014 10:36:19 +0100 Subject: "expire superseded" in master In-Reply-To: <54937BEF.104@epochtimes.com> References: <54906370.4000500@epochtimes.com> <5490C38A.4040309@epochtimes.com> <5492CDBF.7080504@schokola.de> <54937BEF.104@epochtimes.com> Message-ID: <5493F193.3000205@schokola.de> On 19/12/14 02:14, Jingyi Wei wrote: > I see that > bo->ims_obj is set in VBF_Fetch() only if Last-Modified or ETag header > exists in the previous backend response. Otherwise |ims_obj| is NULL and > the memory will not be freed, so the actual problem is that there's no > Last-Modified header in our backend response. Good point, thank you. LM or ETag are required for issuing a backend conditional request, but this should be independent of freeing the superseded object. I will look after this unless someone else does. Nils From slink at schokola.de Sat Dec 27 18:33:56 2014 From: slink at schokola.de (Nils Goroll) Date: Sat, 27 Dec 2014 19:33:56 +0100 Subject: [PATCH] For Vary processing, treat an empty header (only whitespace), like a non-existent header[PATCH] For Vary processing, treat an empty header (only whitespace), like a non-existent header Message-ID: <549EFB94.6050206@schokola.de> Our interpretation of http://tools.ietf.org/html/rfc7234#section-4.1 is that removing all whitespace from a header field only containing whitespace equals to removing the header. -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-For-Vary-processing-treat-an-empty-header-only-white.patch Type: text/x-patch Size: 2307 bytes Desc: not available URL: From sky at crucially.net Sun Dec 28 22:02:17 2014 From: sky at crucially.net (Artur Bergman) Date: Sun, 28 Dec 2014 15:02:17 -0700 Subject: [PATCH] For Vary processing, treat an empty header (only whitespace), like a non-existent header[PATCH] For Vary processing, treat an empty header (only whitespace), like a non-existent header In-Reply-To: <549EFB94.6050206@schokola.de> References: <549EFB94.6050206@schokola.de> Message-ID: <972663C0-4650-46C6-9910-C05C04EE284F@crucially.net> Apparently mnot disagrees in general but allows for it in certain cases https://twitter.com/crucially/status/541425319920414720 Mark Nottingham ?@mnot Dec 6 @crucially it?s per-header. If you know the semantics of the header don?t change if you make a change, that?s OK Artur > On Dec 27, 2014, at 11:33 AM, Nils Goroll wrote: > > Our interpretation of http://tools.ietf.org/html/rfc7234#section-4.1 is that > removing all whitespace from a header field only containing whitespace > equals to removing the header. > <0001-For-Vary-processing-treat-an-empty-header-only-white.patch>_______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev From slink at schokola.de Mon Dec 29 12:47:33 2014 From: slink at schokola.de (Nils Goroll) Date: Mon, 29 Dec 2014 13:47:33 +0100 Subject: [PATCH] For Vary processing, treat an empty header (only whitespace), like a non-existent header[PATCH] For Vary processing, treat an empty header (only whitespace), like a non-existent header In-Reply-To: <549EFB94.6050206@schokola.de> References: <549EFB94.6050206@schokola.de> Message-ID: <54A14D65.1090501@schokola.de> irc: On 27/12/14 19:33, Nils Goroll wrote: > Our interpretation of http://tools.ietf.org/html/rfc7234#section-4.1 is that > removing all whitespace from a header field only containing whitespace > equals to removing the header. > > > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > From slink at schokola.de Mon Dec 29 12:48:04 2014 From: slink at schokola.de (Nils Goroll) Date: Mon, 29 Dec 2014 13:48:04 +0100 Subject: [PATCH] For Vary processing, treat an empty header (only whitespace), like a non-existent header[PATCH] For Vary processing, treat an empty header (only whitespace), like a non-existent header In-Reply-To: <549EFB94.6050206@schokola.de> References: <549EFB94.6050206@schokola.de> Message-ID: <54A14D84.5050203@schokola.de> (sorry for the previous empty email) irc: (13:09:00) slink: phk: yes, the current vary behaviour regarding no-content headers is "fail-safe" and the case which I think we could change is marginal (read: probably exotic) (13:11:19) phk: so lets leave it as is. On 27/12/14 19:33, Nils Goroll wrote: > Our interpretation of http://tools.ietf.org/html/rfc7234#section-4.1 is that > removing all whitespace from a header field only containing whitespace > equals to removing the header. > > > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > From slink at schokola.de Mon Dec 29 19:04:12 2014 From: slink at schokola.de (Nils Goroll) Date: Mon, 29 Dec 2014 20:04:12 +0100 Subject: [PATCH] For Vary processing, treat an empty header (only whitespace), like a non-existent header[PATCH] For Vary processing, treat an empty header (only whitespace), like a non-existent header In-Reply-To: <972663C0-4650-46C6-9910-C05C04EE284F@crucially.net> References: <549EFB94.6050206@schokola.de> <972663C0-4650-46C6-9910-C05C04EE284F@crucially.net> Message-ID: <54A1A5AC.6070308@schokola.de> On 28/12/14 23:02, Artur Bergman wrote: > Apparently mnot disagrees in general but allows for it in certain cases Hm. What else could he say when asked this way? The interesting point is: How can an intermediate know for which headers "empty" equals "non existent" if the rfcs don't make any statement about this? (or do they and I haven't found the right place?) Tackling it from the other end: Who can name a single example where an empty request header has any other semantics than "same as if it wasn't there"? At any rate, phk is most likely right about treating these as two distinct cases for Vary, but without any example for the contrary, I still think it should be ok to merge them. On the other hand, it's most likely an irrelevant discussion... Nils From sky at crucially.net Mon Dec 29 20:45:09 2014 From: sky at crucially.net (Artur Bergman) Date: Mon, 29 Dec 2014 13:45:09 -0700 Subject: [PATCH] For Vary processing, treat an empty header (only whitespace), like a non-existent header[PATCH] For Vary processing, treat an empty header (only whitespace), like a non-existent header In-Reply-To: <54A1A5AC.6070308@schokola.de> References: <549EFB94.6050206@schokola.de> <972663C0-4650-46C6-9910-C05C04EE284F@crucially.net> <54A1A5AC.6070308@schokola.de> Message-ID: He mentioned SOAP does. > On Dec 29, 2014, at 12:04 PM, Nils Goroll wrote: > > > Tackling it from the other end: Who can name a single example where an empty > request header has any other semantics than "same as if it wasn't there"? >