From slink at schokola.de Thu Dec 1 14:11:16 2016 From: slink at schokola.de (Nils Goroll) Date: Thu, 1 Dec 2016 15:11:16 +0100 Subject: (be)(req|resp).body In-Reply-To: <4df6a46a-b3c4-1ac0-3e40-032409b135bc@schokola.de> References: <4df6a46a-b3c4-1ac0-3e40-032409b135bc@schokola.de> Message-ID: I would like to wrap up the discussion so far and start with a review and outlook on body access. I will prepare a separate email regarding the abandon/backend_error changes proposed. To avoid working around the missing features in core at the moment, I would like to go ahead with the actual implementation ASAP. So I'm asking phk for a review. >From a VCL autor's perspective, (be)?(req|resp).body could be first class citizens in VCL and/or we could have better vmod support for working with them. The basic issue is that bodies are not strings, but stevedore objects made up of several (discontinuous) allocations. So, for example, for a regular expression match, we'd need to use multi segment matching, which has specific issues (see pcrepartial(3)). I see this boiling down to three steps #1 VCL support for set/unset (be)?(req|resp).body #2 Better vmod support for bodies - iterate over bodies? - push fetch processors? #3 more complete support for working with bodies - string operators - reading (cast to STRING_LIST ?) bodies How exactly #2 and #3 are going to look is unclear to me still, but I guess the major change would be that some of the cases would require vcl_recv / vcl_backend_fetch to pull the body into cache early. Nevertheless, I can't see a case where implementing #1 would prevent us from DTRT regarding #2 and #3. To implement #1 we have/need: - HAVE vcl_backend_fetch { unset bereq.body; # in buitlin.vcl } vcl_backend_error { unset beresp.body; # will add a VTC set beresp.body = "..."; # in buitlin.vcl } vcl_synth { unset resp.body; # will add a VTC set resp.body = "..."; # in buitlin.vcl } - NEED # synthetic backend request body vcl_backend_fetch { set bereq.body = "..."; } # synthetic cache object without vcl_backend_error de-tour # will drain-read the backend response vcl_backend_response { unset beresp.body; set beresp.body = "..."; } # synthetic client response without the vcl_synth de-tour vcl_deliver { unset resp.body; set resp.body = "..."; } Nils From slink at schokola.de Thu Dec 1 14:20:29 2016 From: slink at schokola.de (Nils Goroll) Date: Thu, 1 Dec 2016 15:20:29 +0100 Subject: vcl_backend_error changes In-Reply-To: <4df6a46a-b3c4-1ac0-3e40-032409b135bc@schokola.de> References: <4df6a46a-b3c4-1ac0-3e40-032409b135bc@schokola.de> Message-ID: <00d4146c-4c9f-0456-267d-8ed1fd9246d5@schokola.de> So here's the second suggestion update from the "thoughts about vcl cleanup around ..." thread: * add status and reason to abandon which get pre-set for the call to vcl_synth vcl_backend_fetch { return (abandon(901, "coming from vcl_backend_fetch")); } vcl_backend_response { return (abandon(902, "coming from vcl_backend_response")); } vcl_backend_error { return (abandon(903, "coming from vcl_backend_error")); } vcl_synth { # resp.status is now 901, 902 or 903 for the cases above # resp.reason is set to "coming from ..." } * add return(error(status, reason)) to fail to vcl_backend_error and have status and reason pre-set vcl_backend_fetch { return (error(901, "coming from vcl_backend_fetch")); } vcl_backend_response { return (error(902, "coming from vcl_backend_response")); } vcl_backend_error { # beresp.status is now 901 or 902 for the cases above # beresp.reason is set to "coming from ..." } Nils From slink at schokola.de Thu Dec 1 20:52:31 2016 From: slink at schokola.de (Nils Goroll) Date: Thu, 1 Dec 2016 21:52:31 +0100 Subject: (be)(req|resp).body In-Reply-To: References: <4df6a46a-b3c4-1ac0-3e40-032409b135bc@schokola.de> Message-ID: Additional notes from IRC discussion with phk * we also want set (be)(req|resp).body += "..."; to append. This may help save workspace * Setting the body should clear Content-Encoding (can be un-done for instance when a vmod injects compressed data) * Whatever we set the body to should need to go though the (fetch) pipeline to get gzip/esi Processing From slink at schokola.de Wed Dec 7 21:08:28 2016 From: slink at schokola.de (Nils Goroll) Date: Wed, 7 Dec 2016 22:08:28 +0100 Subject: autoconf madness Message-ID: <4e44b962-fa73-10af-8bea-9aa940438c46@schokola.de> I'm stuck with this: to run 32 bit tests on the solaris box, I set CFLAGS externally: export PATH=/opt/local/gcc47/bin:/opt/local/gnu/bin:/opt/local/bin:/opt/local/sbin:/usr/bin:/usr/sbin export MESSAGE=32_gcc4.7 export CPPFLAGS="-I/opt/local32/include" export CFLAGS="-m32" export LDFLAGS="-m32 -L/opt/local32/lib -R/opt/local32/lib" runtest & runtest basically just calls vtest Now for reasons which I do not understand, autoconf seems to create an environment where the second (inner) configure called via make distcheck gets passed the CFLAGS determined during the first (outer) configure run. The effect is that basic configure tests fail because of the clags being to strict for the code used by configure itself. e.g.: checking for gcc... gcc checking whether the C compiler works... no configure: error: in `/tmp/vtest.build.32_gcc4.7/varnish-trunk/_build': configure: error: C compiler cannot create executables See `config.log' for more details Makefile:737: recipe for target 'distcheck' failed make: *** [distcheck] Error 1 caused by conftest.c failing on -Wstrict-prototypes Read more here: http://www.varnish-cache.org/vtest/bugs_distcheck.html#7fb8751 I understand that outer CFLAGS need to be passed to the inner configure for it to work, but why should the other configure need pass the CFLAGS it has itself determined? Any ideas? From phk at phk.freebsd.dk Thu Dec 8 07:08:17 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 08 Dec 2016 07:08:17 +0000 Subject: autoconf madness In-Reply-To: <4e44b962-fa73-10af-8bea-9aa940438c46@schokola.de> References: <4e44b962-fa73-10af-8bea-9aa940438c46@schokola.de> Message-ID: <31156.1481180897@critter.freebsd.dk> -------- In message <4e44b962-fa73-10af-8bea-9aa940438c46 at schokola.de>, Nils Goroll writ es: >I understand that outer CFLAGS need to be passed to the inner configure for it >to work, but why should the other configure need pass the CFLAGS it has itself >determined? About the only thing I have to contribute here is that I picked the "distcheck" target because that's what DES told me ten years ago, and Lasse repeated about five years ago... If somethinge else makes more sense, I'm all for it. Keep in mind that we don't actually need all vtest-contributors to run the same target... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From slink at schokola.de Thu Dec 8 15:31:34 2016 From: slink at schokola.de (Nils Goroll) Date: Thu, 8 Dec 2016 16:31:34 +0100 Subject: autoconf madness In-Reply-To: <4e44b962-fa73-10af-8bea-9aa940438c46@schokola.de> References: <4e44b962-fa73-10af-8bea-9aa940438c46@schokola.de> Message-ID: > Now for reasons which I do not understand, autoconf seems to create an > environment where the second (inner) configure called via make distcheck gets > passed the CFLAGS determined during the first (outer) configure run. Today I've changed configure.ac and Makefile.am to save the external CFLAGS and explicitly pass them to the inner configure explicitly (commits cb00896414ba72c7cc10882c1be7e23dfb796dd3 and d49afc041773f915e74a82c962063edde3e1d9eb). This feels like a workaround to me, in particular because CFLAGS is just one of many relevant environment variables which we touch. IMHO, a good solution would restore all of the original environment for distcheck. Any ideas? Nils From phk at phk.freebsd.dk Thu Dec 8 21:23:01 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 08 Dec 2016 21:23:01 +0000 Subject: autoconf madness In-Reply-To: References: <4e44b962-fa73-10af-8bea-9aa940438c46@schokola.de> Message-ID: <57951.1481232181@critter.freebsd.dk> -------- In message , Nils Goroll writ es: >Any ideas? Not apart from "get rid of autocrap sooner rather than later". -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From reza at varnish-software.com Fri Dec 9 20:37:09 2016 From: reza at varnish-software.com (Reza Naghibi) Date: Fri, 9 Dec 2016 15:37:09 -0500 Subject: Age precision rounding causes premature stale object Message-ID: So in light of the hit-for-pass vs. hit-for-miss change in 5.0 (which I think is great), I have been researching 0s TTL objects coming into cache (and generating the hit-for-XXX marker). In particular, even with grace disabled, I have still seen valid objects from one Varnish come in as a stale 0s TTL object in another Varnish. As in, an object comes into Varnish with a max-age of X and an equal Age of X. Example: Cache-Control: max-age=14 Age: 14 The effective TTL here is 0s, triggering the hit-for-XXX. Looking at the code: http_PrintfHeader(req->resp, "Age: %.0f", fmax(0., req->t_prev - req->objcore->t_origin)); https://github.com/varnishcache/varnish-cache/blob/f4400e0c682c69ba63fc7c00abe7246ea570fbd8/bin/varnishd/cache/cache_req_fsm.c#L95-L96 The issue here is that %.0f will do a standard IEEE floating point numerical rounding to satisfy the precision. I am not sure if this is a C standard, but in my tests, this holds true. In the context of our Age calculation, this means that objects in their last 0.5s of TTL life will get their Age rounded up to the max-age and then anything reading this response will see a stale object. Given that we are losing precision and accepting a 1s rounding error, floor() is the best choice here as it doesn't have the side effect of this premature Age == max-age scenario. So something as simple as: http_PrintfHeader(req->resp, "Age: %.0f", floor(fmax(0., req->t_prev - req->objcore->t_origin))); Will make sure we *always* get an Age < max-age, up until the actual Age >= max-age and the object is infact stale. Thoughts? I am not sure of the history here and if this has been previously considered and there are valid reasons for this logic. If so, I apologize in advance! -- Reza Naghibi Varnish Software -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Sun Dec 11 13:29:30 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sun, 11 Dec 2016 13:29:30 +0000 Subject: busy week & bugwash(es) Message-ID: <10106.1481462970@critter.freebsd.dk> This week is going to be pretty busy for me, and I'll not be able to participate in the monday bugwash. Speaking of bugwashes, I have been pondering how we can make them less EU-centric, and starting after X-mas I will try to do a two-shot bugwash. Unless there are better ideas, we will keep the bugwash we already have, mondays at 13:00 EU time. But as an experiment I will also do a bugwash monday evening (EU time) to make it possible for other continents to chime in. Being an experiment, nothing is nailed down yet, and I'm very open to suggestions as to what time suits people best ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From reza at varnish-software.com Mon Dec 12 17:12:58 2016 From: reza at varnish-software.com (Reza Naghibi) Date: Mon, 12 Dec 2016 12:12:58 -0500 Subject: busy week & bugwash(es) In-Reply-To: <10106.1481462970@critter.freebsd.dk> References: <10106.1481462970@critter.freebsd.dk> Message-ID: I think this is a good idea. However, reading the bugwash IRC logs and the associated tickets is pretty simple and I usually can catch up and get a good idea of things. Personally, I think only 1% of bugwashed tickets matter for me. So for me, since my focus is more on the commercial end of Varnish Software, I dont plan on participating much. But if there is a issue (with a previously discussed ticket or a ticket I introduced), it would make sense to have the ability to do a bugwash on it and hash out the details. So as you said, I, and possible others, would participate in the evening bugwash? Then I guess the AM bugwash people can read up on our bugwash and get all the details. As I said before, I dont plan on regularly participating because reading over the mornings bugwash is good enough and im usually in 99% agreement. So maybe during the day, I or you will ping each other and say, lets have an evening bugwash and talk about tickets X, Y, and Z? Otherwise, if no one speaks up, just assume there will be no evening bugwash? So its opt in with a default of it wont be held if no one opts in. -- Reza Naghibi Varnish Software On Sun, Dec 11, 2016 at 8:29 AM, Poul-Henning Kamp wrote: > This week is going to be pretty busy for me, and I'll not be able > to participate in the monday bugwash. > > Speaking of bugwashes, I have been pondering how we can make them > less EU-centric, and starting after X-mas I will try to do a two-shot > bugwash. > > Unless there are better ideas, we will keep the bugwash we already > have, mondays at 13:00 EU time. > > But as an experiment I will also do a bugwash monday evening (EU time) > to make it possible for other continents to chime in. > > Being an experiment, nothing is nailed down yet, and I'm very open > to suggestions as to what time suits people best ? > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Thu Dec 22 13:18:28 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 22 Dec 2016 14:18:28 +0100 Subject: autoconf madness In-Reply-To: <57951.1481232181@critter.freebsd.dk> References: <4e44b962-fa73-10af-8bea-9aa940438c46@schokola.de> <57951.1481232181@critter.freebsd.dk> Message-ID: On Thu, Dec 8, 2016 at 10:23 PM, Poul-Henning Kamp wrote: > -------- > In message , Nils Goroll writ > es: > >>Any ideas? > > Not apart from "get rid of autocrap sooner rather than later". I had a look at how we manage CFLAGS, including developer flags and it's a big [1] mess. I have a definite plan on how to clean it up, and will submit a pull request. There are unfortunately some special cases that I will leave alone for now. And *then* I can look at Nils' specific problem. Now, my main concern is that we append flags to CFLAGS, overriding presets that should take precedence (passing CFLAGS via the environment or the configure command line). It may be related to Nils' problem, but at this point I think not. As a side note, I ran into E_ANONYMOUS_UNION_DECL (in struct suckaddr) while building with suncc (in C11 but not C99), didn't get any further because I realized I haven't had lunch yet. I think it should be ignored when developer flags are enabled, leaving it to Nils for now. Cheers, Dridi [1] it's a euphemism :p From phk at phk.freebsd.dk Thu Dec 22 22:24:31 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 22 Dec 2016 22:24:31 +0000 Subject: [master] 1f45a46 Turn daemonize() inside out, and don't let the original process die until we have started the child process. In-Reply-To: References: Message-ID: <37781.1482445471@critter.freebsd.dk> -------- This is my attempt to solve all, or at least many, of the issues related to errors during startup. Normal daemonize() immediately throws the original process away after detaching the demonized process. Now we keep the original "daemonize" process around, so that it can exit(3) suitably based on what the detached (= varnishd master) process finds out. Ideally we should wait for even more conclusive evidence of the child process running, but I am not quite sure what the evidence should be ? uptime > 0 ? First request handled ? (Ideas welcome...) It's going to be bear to write test-cases for this, so I would really appreciate it if some of you could spend half an hour stressing it manually. See also tickets 2141 & 2041, >commit 1f45a46516ec265fd983d8a2fca5f97deed96fc6 >Author: Poul-Henning Kamp >Date: Thu Dec 22 22:14:58 2016 +0000 > > Turn daemonize() inside out, and don't let the original process > die until we have started the child process. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Fri Dec 30 16:34:02 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Fri, 30 Dec 2016 17:34:02 +0100 Subject: autoconf madness In-Reply-To: References: <4e44b962-fa73-10af-8bea-9aa940438c46@schokola.de> Message-ID: > Any ideas? Not yet, but I started another thread on github [1] so I suggest we move the discussion there. As far I'm concerned, I think I'm done with Varnish for this year ;) Cheers [1] https://github.com/varnishcache/varnish-cache/pull/2174