From lisachenko.it at gmail.com Mon Aug 3 15:00:27 2015 From: lisachenko.it at gmail.com (Alexander Lisachenko) Date: Mon, 3 Aug 2015 18:00:27 +0300 Subject: Modification of parent response object from the ESI subrequests Message-ID: Hello, dev team! I want to continue the discussion of issue https://www.varnish-cache.org/trac/ticket/1766 I think, that this feature can be very useful for many purposes, like automatic calculation of last-modified dates for conditional responses, propagation of statuses from ESI blocks in addtional headers, for example X-1-Status: 200 OK, X-2-Status: 404 Not Modifed and much more. I know, that the Varnish streams the context as it fetched, but is it possible to organize a temporary buffer for the response to buffer it and to deliver the whole response to the client, while having an access to the parent block from ESI? -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at varnish-software.com Mon Aug 3 15:36:49 2015 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 3 Aug 2015 17:36:49 +0200 Subject: [PATCH] Do not close session in ESI VDP Message-ID: <1438616209-23681-1-git-send-email-martin@varnish-software.com> The ESI_VDP would double-close sessions when a flush error happened before an ESI include at one level, and then again at a higher level. Fixed by not closing the session in this case, as that will be taken care of at the end of delivery anyways when V1L_FlushRelease is called. Fixes: #1737 --- bin/varnishd/cache/cache_esi_deliver.c | 1 - bin/varnishtest/tests/r01737.vtc | 43 ++++++++++++++++++++++++++++++++++ 2 files changed, 43 insertions(+), 1 deletion(-) create mode 100644 bin/varnishtest/tests/r01737.vtc diff --git a/bin/varnishd/cache/cache_esi_deliver.c b/bin/varnishd/cache/cache_esi_deliver.c index aa011b6..72a7763 100644 --- a/bin/varnishd/cache/cache_esi_deliver.c +++ b/bin/varnishd/cache/cache_esi_deliver.c @@ -317,7 +317,6 @@ VDP_ESI(struct req *req, enum vdp_action act, void **priv, r = (void*)strchr((const char*)q, '\0'); AN(r); if (VDP_bytes(req, VDP_FLUSH, NULL, 0)) { - SES_Close(req->sp, SC_REM_CLOSE); ecx->p = ecx->e; break; } diff --git a/bin/varnishtest/tests/r01737.vtc b/bin/varnishtest/tests/r01737.vtc new file mode 100644 index 0000000..66c5aee --- /dev/null +++ b/bin/varnishtest/tests/r01737.vtc @@ -0,0 +1,43 @@ +varnishtest "#1737 - ESI sublevel session close" + +# Build a esi request tree that fails on flush before include at two different +# levels. Synchronize a client close after the response headers have been +# received by the client. This produces write erros for the body parts in all +# fragments. +server s1 { + rxreq + txresp -body {} + sema r1 sync 2 + + rxreq + delay 1 + txresp -body {22} + + rxreq + txresp -body {1} + + rxreq + expect req.url == "/check" + rxresp +} -start + +varnish v1 -vcl+backend { + sub vcl_backend_response { + set beresp.do_esi = true; + } +} -start + +# Send request, read response headers then close connection +client c1 { + txreq + rxresp -no_obj + sema r1 sync 2 +} -run + +delay 3 + +# Check that Varnish is alive +client c1 { + txreq + rxresp +} -run -- 2.1.4 From fgsch at lodoss.net Mon Aug 3 17:16:04 2015 From: fgsch at lodoss.net (Federico Schwindt) Date: Mon, 3 Aug 2015 18:16:04 +0100 Subject: [PATCH] Variable for vcl_dir in startup scripts In-Reply-To: References: <5F530A9242E7F84F999DB40E0E268FBD4B0EB01D@mercalli.lild01.pictime.fr> <5F530A9242E7F84F999DB40E0E268FBD4B0EB0F2@mercalli.lild01.pictime.fr> <5F530A9242E7F84F999DB40E0E268FBD4B1796F7@mercalli.lild01.pictime.fr> Message-ID: Hi Gauthier, We've discussed these on irc today and I've now removed ExecStartPre and committed your diff for the sysv init script. Thanks for the diff. Cheers. On Mon, Jul 27, 2015 at 9:38 PM, Federico Schwindt wrote: > > Seems the prestart is required for reloads. > > I haven't tested RH but this doesn't seem to be the case in Debian - > ExecStartPre is only called before (re)start. > > The sysv init script change might be needed though. Setting vmod_dir will > also affects the test btw. > > On Mon, Jul 27, 2015 at 11:04 AM, Delacroix, Gauthier < > Gauthier.Delacroix at coreye.fr> wrote: > >> Hi Federico, >> >> Seems the prestart is required for reloads. >> >> Here is a new patch, just including DAEMON_OPTS in both tests. >> >> Gauthier >> >> From 4c11e58d6270591f93be58551496412e3118aeda Mon Sep 17 00:00:00 2001 >> From: Gauthier Delacroix >> Date: Mon, 27 Jul 2015 11:55:13 +0200 >> Subject: [PATCH] Include DAEMON_OPTS in service prestart tests >> >> --- >> redhat/varnish.initrc | 2 +- >> redhat/varnish.service | 2 +- >> 2 files changed, 2 insertions(+), 2 deletions(-) >> >> diff --git a/redhat/varnish.initrc b/redhat/varnish.initrc >> index 117e334..03f47fb 100755 >> --- a/redhat/varnish.initrc >> +++ b/redhat/varnish.initrc >> @@ -126,7 +126,7 @@ rh_status_q() { >> >> configtest() { >> if [ -f "$VARNISH_VCL_CONF" ]; then >> - $exec -f "$VARNISH_VCL_CONF" -C -n /tmp > /dev/null && echo >> "Syntax ok" >> + $exec -f "$VARNISH_VCL_CONF" -C -n /tmp $DAEMON_OPTS > /dev/null >> && echo "Syntax ok" >> else >> echo "VARNISH_VCL_CONF is unset or does not point to a file" >> fi >> diff --git a/redhat/varnish.service b/redhat/varnish.service >> index a4f3355..ca93b3d 100644 >> --- a/redhat/varnish.service >> +++ b/redhat/varnish.service >> @@ -27,7 +27,7 @@ EnvironmentFile=/etc/varnish/varnish.params >> Type=forking >> PIDFile=/var/run/varnish.pid >> PrivateTmp=true >> -ExecStartPre=/usr/sbin/varnishd -C -f $VARNISH_VCL_CONF >> +ExecStartPre=/usr/sbin/varnishd -C -f $VARNISH_VCL_CONF $DAEMON_OPTS >> ExecStart=/usr/sbin/varnishd \ >> -P /var/run/varnish.pid \ >> -f $VARNISH_VCL_CONF \ >> -- >> 1.8.3.msysgit.0 >> >> De : Federico Schwindt [mailto:fgsch at lodoss.net] >> Envoy? : mercredi 15 juillet 2015 20:07 >> ? : Delacroix, Gauthier >> Cc : varnish-dev at varnish-cache.org; ssm at redpill-linpro.com >> Objet : Re: [PATCH] Variable for vcl_dir in startup scripts >> >> Hi Gauthier, >> >> Apologies for the delay. >> >> In all fairness I copied this from the Debian systemd files [1] in an >> attempt to unify the service files. >> I don't see any particular reason for having this, specially if it's the >> source of issues. >> >> Stig, is there any reason for the ExecStartPre? The commit message says >> "debian/varnish.service: Test configuration before starting, add reload". >> I don't see any difference in the output for an invalid config with >> ExecStartPre when I run either `systemctl start' or `systemctl status'. >> >> Thanks. >> >> 1. http://anonscm.debian.org/cgit/pkg-varnish/pkg-varnish.git/tree/debian >> >> On Thu, Jun 25, 2015 at 5:12 PM, Delacroix, Gauthier < >> Gauthier.Delacroix at coreye.fr> wrote: >> Federico, >> >> I've initially set vcl_dir in DAEMON_OPTS in varnish.params with 4.0.2, >> but it's not included in the syntax check added in 4.0.3, making my config >> unable to start without changing varnish.service. >> >> I'm not aware of the reason behind this syntax check (won't a syntax >> error make startup fail anyway ?) but I assumed you had a good reason not >> to include DAEMON_OPTS. >> >> As it's better to change varnish.params rather than varnish.service, my >> proposal was to add a way to include parameters needed to make the config >> work in the syntax check, without including the full DAEMON_OPTS. >> >> If you finally think DAEMON_OPTS can be included in the syntax check, >> then I can send a really smaller patch. >> >> Gauthier >> >> De : Federico Schwindt [mailto:fgsch at lodoss.net] >> Envoy? : jeudi 25 juin 2015 17:53 >> ? : Delacroix, Gauthier >> Cc : varnish-dev at varnish-cache.org >> Objet : Re: [PATCH] Variable for vcl_dir in startup scripts >> >> Hi, >> >> My first inclination is to ensure that the ExecStartPre line uses the >> same parameters as ExecStart. >> >> So my questions is how do you set vcl_dir? Do you edit varnish.service or >> varnish.params? >> >> Wouldn't adding DAEMON_OPTS to ExecStartPre (and configtest) do it? >> >> On Thu, Jun 25, 2015 at 3:26 PM, Delacroix, Gauthier < >> Gauthier.Delacroix at coreye.fr> wrote: >> Here is another patch proposal to make syntax check handle parameters >> required to compile the VCL (vcl_dir, etc.) without creating a startup >> variable for each parameter. >> >> It just adds a COMPILE_OPTS that is merged in DAEMON_OPTS to start >> Varnish but is used alone in the syntax check. >> >> Gauthier >> >> From d1567a956d53a489aa4ace66ce0b1c1ef745570b Mon Sep 17 00:00:00 2001 >> From: Gauthier Delacroix >> Date: Thu, 25 Jun 2015 15:06:08 +0200 >> Subject: [PATCH] Add COMPILE_OPTS in startup scripts to make syntax check >> check handle compilation parameters >> >> --- >> redhat/varnish.initrc | 6 ++++-- >> redhat/varnish.params | 8 ++++++++ >> redhat/varnish.service | 3 ++- >> redhat/varnish.sysconfig | 11 ++++++++++- >> 4 files changed, 24 insertions(+), 4 deletions(-) >> >> diff --git a/redhat/varnish.initrc b/redhat/varnish.initrc >> index 117e334..0bde074 100755 >> --- a/redhat/varnish.initrc >> +++ b/redhat/varnish.initrc >> @@ -126,9 +126,11 @@ rh_status_q() { >> >> configtest() { >> if [ -f "$VARNISH_VCL_CONF" ]; then >> - $exec -f "$VARNISH_VCL_CONF" -C -n /tmp > /dev/null && echo >> "Syntax ok" >> + $exec -f "$VARNISH_VCL_CONF" -C -n /tmp $COMPILE_OPTS > >> /dev/null \ >> + && echo "Syntax ok" >> else >> - echo "VARNISH_VCL_CONF is unset or does not point to a file" >> + echo "VARNISH_VCL_CONF is unset or does not point to a file" >> + echo "Also check that COMPILE_OPTS is set depending on the VCL >> config" >> fi >> } >> >> diff --git a/redhat/varnish.params b/redhat/varnish.params >> index 27a14dd..970d088 100644 >> --- a/redhat/varnish.params >> +++ b/redhat/varnish.params >> @@ -31,5 +31,13 @@ VARNISH_TTL=120 >> VARNISH_USER=varnish >> VARNISH_GROUP=varnish >> >> +# Startup options required to compile the configuration. >> +# The following run-time parameters must be defined here, if needed: >> +# cc_command, group_cc, vcc_allow_inline_c, vcc_err_unref, >> vcc_unsafe_path, >> +# vcl_dir, vmod_dir >> +# Defining them in DAEMON_OPTS may result in a syntax check failure. >> +# See the man page varnishd(1). >> +#COMPILE_OPTS="-p vcl_dir=/etc/varnish -p vcc_err_unref=on" >> + >> # Other options, see the man page varnishd(1) >> #DAEMON_OPTS="-p thread_pool_min=5 -p thread_pool_max=500 -p >> thread_pool_timeout=300" >> diff --git a/redhat/varnish.service b/redhat/varnish.service >> index a4f3355..a08db58 100644 >> --- a/redhat/varnish.service >> +++ b/redhat/varnish.service >> @@ -27,7 +27,7 @@ EnvironmentFile=/etc/varnish/varnish.params >> Type=forking >> PIDFile=/var/run/varnish.pid >> PrivateTmp=true >> -ExecStartPre=/usr/sbin/varnishd -C -f $VARNISH_VCL_CONF >> +ExecStartPre=/usr/sbin/varnishd -C -f $VARNISH_VCL_CONF $COMPILE_OPTS >> ExecStart=/usr/sbin/varnishd \ >> -P /var/run/varnish.pid \ >> -f $VARNISH_VCL_CONF \ >> @@ -38,6 +38,7 @@ ExecStart=/usr/sbin/varnishd \ >> -g $VARNISH_GROUP \ >> -S $VARNISH_SECRET_FILE \ >> -s $VARNISH_STORAGE \ >> + $COMPILE_OPTS \ >> $DAEMON_OPTS >> >> ExecReload=/usr/sbin/varnish_reload_vcl >> diff --git a/redhat/varnish.sysconfig b/redhat/varnish.sysconfig >> index 6aa2354..0e376ff 100644 >> --- a/redhat/varnish.sysconfig >> +++ b/redhat/varnish.sysconfig >> @@ -91,6 +91,14 @@ VARNISH_STORAGE="malloc,${VARNISH_STORAGE_SIZE}" >> # # Default TTL used when the backend does not specify one >> VARNISH_TTL=120 >> # >> +# Startup options required to compile the configuration. >> +# The following run-time parameters must be defined here, if needed: >> +# cc_command, group_cc, vcc_allow_inline_c, vcc_err_unref, >> vcc_unsafe_path, >> +# vcl_dir, vmod_dir >> +# Defining them in DAEMON_OPTS may result in a syntax check failure. >> +# See the man page varnishd(1). >> +#COMPILE_OPTS="-p vcl_dir=/etc/varnish -p vcc_err_unref=on" >> +# >> # # DAEMON_OPTS is used by the init script. If you add or remove >> options, make >> # # sure you update this section, too. >> DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \ >> @@ -102,7 +110,8 @@ DAEMON_OPTS="-a >> ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \ >> -p thread_pool_timeout=${VARNISH_THREAD_TIMEOUT} \ >> -u varnish -g varnish \ >> -S ${VARNISH_SECRET_FILE} \ >> - -s ${VARNISH_STORAGE}" >> + -s ${VARNISH_STORAGE}" \ >> + ${COMPILE_OPTS} >> # >> >> >> -- >> 1.8.3.msysgit.0 >> >> >> >> _______________________________________________ >> varnish-dev mailing list >> varnish-dev at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Mon Aug 3 17:36:11 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 3 Aug 2015 19:36:11 +0200 Subject: Modification of parent response object from the ESI subrequests In-Reply-To: References: Message-ID: On Mon, Aug 3, 2015 at 5:00 PM, Alexander Lisachenko wrote: > Hello, dev team! > > I want to continue the discussion of issue > https://www.varnish-cache.org/trac/ticket/1766 FYI I haven't followed this discussion. > I think, that this feature can be very useful for many purposes, like > automatic calculation of last-modified dates for conditional responses, > propagation of statuses from ESI blocks in addtional headers, for example > X-1-Status: 200 OK, X-2-Status: 404 Not Modifed and much more. You can't add headers, because esi tags are monitored while the body is being served. Which means that the headers are already sent by the time an ESI request is created VCL-wise. > I know, that the Varnish streams the context as it fetched, but is it > possible to organize a temporary buffer for the response to buffer it and to > deliver the whole response to the client, while having an access to the > parent block from ESI? That might be a case for a VDP, you could maybe write a VMOD that pushes an action to the delivery pipeline on ESI requests, and there, intercept the response headers and body, buffer them, and get a chance to inject headers during ESI sub-request processing. ESI is not my sweet-spot so I have really no idea whether that can be done at all. Best Regards, Dridi From fgsch at lodoss.net Fri Aug 7 16:01:58 2015 From: fgsch at lodoss.net (Federico Schwindt) Date: Fri, 7 Aug 2015 17:01:58 +0100 Subject: [PATCH] Specify the full path of the logs Message-ID: This makes easier to run multiple instances, each one writing to different files. OK? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Specify-the-full-path-of-the-logs.patch Type: text/x-patch Size: 700 bytes Desc: not available URL: From dridi at varni.sh Mon Aug 10 12:52:02 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 10 Aug 2015 14:52:02 +0200 Subject: Counters in master In-Reply-To: References: Message-ID: Hi Federico, I have comments for one of the counters. On Mon, Jul 13, 2015 at 8:35 PM, Federico Schwindt wrote: > Removed: > > VBE..vcls - no longer needed Yes, this one doesn't make sense any longer in 4.1, a backend can only belong to one VCL (I'm working on documenting that). It could be replaced by VBT counters to get stats on the TCP pools. The VBE ones you are referring to could look like: VBT.(ipv4,ipv6).backends There might be interesting stats on pools worth adding. Best, Dridi From dridi at varni.sh Mon Aug 10 13:14:27 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 10 Aug 2015 15:14:27 +0200 Subject: [PATCH 2/4] Improve client ws overflow handling. In-Reply-To: <20150731074833.GA17875@immer.varnish-software.com> References: <20150731074833.GA17875@immer.varnish-software.com> Message-ID: On Fri, Jul 31, 2015 at 9:48 AM, Lasse Karstensen wrote: > --- > vdp = WS_Alloc(req->ws, sizeof *vdp); > - AN(vdp); > + if (vdp == NULL) { > + WS_MarkOverflow(req->ws); > + return; > + } I don't think you need to call WS_MarkOverflow above, WS_Alloc already does it From dridi at varni.sh Tue Aug 11 08:07:43 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 11 Aug 2015 10:07:43 +0200 Subject: Embed code from the source in sphinx Message-ID: Hi, I'd like to show code snippets (eg. struct definitions) in the documentation, but also be future-proof. Instead of hard-pasting the code and risk divergence over time, I'd like to tell sphinx to extract a piece of code instead. Initially asked on IRC, just in case someone would know because my research tells me that it can't be done. I have already seen struct definitions in the docs so I have little to no hope... Someone who knows better might be on holidays, so asking here instead. Cheers, Dridi From dridi at varni.sh Tue Aug 11 16:08:27 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 11 Aug 2015 18:08:27 +0200 Subject: [master] 00d5576 Start documenting how to write a director In-Reply-To: References: Message-ID: On Tue, Aug 11, 2015 at 6:01 PM, Dridi Boukelmoune wrote: > > commit 00d5576f68383f900b772b1657d3d7010a2f9a44 > Author: Dridi Boukelmoune > Date: Tue Aug 11 17:50:43 2015 +0200 > > Start documenting how to write a director Hi, I have started documenting directors for vmod developers, and related topics. Before I proceed any further, I'd like some feedback to simply know whether I can continue writing like this. I also intend to take care of the TODOs I introduce. Thanks, Dridi From dridi at varni.sh Wed Aug 12 08:03:05 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 12 Aug 2015 10:03:05 +0200 Subject: [PATCH] Specify the full path of the logs In-Reply-To: References: Message-ID: On Fri, Aug 7, 2015 at 6:01 PM, Federico Schwindt wrote: > This makes easier to run multiple instances, each one writing to > different files. Agreed, this won't mess with other files than the ones installed by default. > OK? I wish logrotate'd support multiple instances... That's the best we can do right now... From arianna.aondio at varnish-software.com Wed Aug 12 15:16:20 2015 From: arianna.aondio at varnish-software.com (Arianna Aondio) Date: Wed, 12 Aug 2015 17:16:20 +0200 Subject: [patch]real2int std_vmod function. Message-ID: Hi, as discussed this afternoon on IRC with Federico, here is a patch for a std_vmod function for converting from real to int. we might be interested in pushing this into master. -- Arianna Aondio Software Developer | Varnish Software AS Mobile: +47 98062619 We Make Websites Fly www.varnish-software.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: real2int_std.patch Type: text/x-diff Size: 1762 bytes Desc: not available URL: From geoff at uplex.de Wed Aug 12 21:23:21 2015 From: geoff at uplex.de (Geoff Simmons) Date: Wed, 12 Aug 2015 23:23:21 +0200 Subject: [patch]real2int std_vmod function. In-Reply-To: References: Message-ID: <55CBB949.8030107@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 8/12/15 5:16 PM, Arianna Aondio wrote: > Hi, as discussed this afternoon on IRC with Federico, here is a > patch for a std_vmod function for converting from real to int. +VCL_INT __match_proto__(td_std_time2real) +vmod_real2integer(VRT_CTX, VCL_REAL r) +{ + CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + + return((long)floor(r)); +} Do we have to worry about what happens if r is non-finite (+-Inf or NaN)? (Is that possible for a VCL_REAL?) Best, Geoff - -- UPLEX Systemoptimierung Scheffelstra?e 32 22301 Hamburg http://uplex.de/ Mob: +49-176-63690917 -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) iQIcBAEBCAAGBQJVy7lIAAoJEOUwvh9pJNURnRgP/1sW9vHr3Eq/sG2aicckF+/6 51YEhwtziG70BJHy1ph2b7bB2KXps7WfiT30EOVCCNjr0tvGrp/bvUpZHd0G9MGq h9HhSpk13yZtA0TkNjWjyn89hHy52SSY0ksmTS2hP8EbB0SP7+RPAQUf7jApsvwI DohjnL6We1/cxnnsdojMG1JAAWBJpD6/jHxadgV5wQ1J1yXzW5KxYP3+7Myp+aks MN0XQjLi5bAoAj10Mc4UAeho3IGuSTqbqaI+YHNG+0izgJaXLo56B04bxCZQeG/b mpgOGbCl2xuCWPgYGBV5F6IMGs804BXphAt/U6eG+g+ja5AEsE15dKlE85sPyiql L8JtdRuYOu1d87gIm4ittH1EBrKz+wwD5Im/v8VGev8wC00p2sLZTRxTfpZiRacf AZ5Dg4GPQ25gzl/zk6p+Ym/AhN/hkTMjgB3TzBAP79VHaA2lrT1zHWPML7skHQvS nBu6bA7ElrMjOHYSo0nLGZvWHSf2Lq2UFri2qr+RxTa44puclgcD+BMyTfR3X+aw KEi0iGBk5ejLwpnxQ62cUBWL6vGBXOKjqxf/hhS/5hsbspTQSO9w9kDbieXJhHKV IxkyZ4Zavr4AuuVKc48HcrgdejfOCetM3uzLr7glnGcsIWzmJT5VkWjGzWOqUqms 3BQR1vMXBwZSs/SsAbDY =bvX0 -----END PGP SIGNATURE----- From phk at phk.freebsd.dk Thu Aug 13 08:30:58 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 13 Aug 2015 08:30:58 +0000 Subject: [patch]real2int std_vmod function. In-Reply-To: <55CBB949.8030107@uplex.de> References: <55CBB949.8030107@uplex.de> Message-ID: <31718.1439454658@critter.freebsd.dk> -------- In message <55CBB949.8030107 at uplex.de>, Geoff Simmons writes: >-----BEGIN PGP SIGNED MESSAGE----- >Hash: SHA256 > >On 8/12/15 5:16 PM, Arianna Aondio wrote: >> Hi, as discussed this afternoon on IRC with Federico, here is a >> patch for a std_vmod function for converting from real to int. > >+VCL_INT __match_proto__(td_std_time2real) >+vmod_real2integer(VRT_CTX, VCL_REAL r) >+{ >+ CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); >+ >+ return((long)floor(r)); >+} > >Do we have to worry about what happens if r is non-finite (+-Inf or >NaN)? (Is that possible for a VCL_REAL?) Yes. Proper range checking should be done, and like all other conversions, a default value parameter to use when that fails should be added. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From arianna.aondio at varnish-software.com Thu Aug 13 08:42:27 2015 From: arianna.aondio at varnish-software.com (Arianna Aondio) Date: Thu, 13 Aug 2015 10:42:27 +0200 Subject: [patch]real2int std_vmod function. In-Reply-To: <31718.1439454658@critter.freebsd.dk> References: <55CBB949.8030107@uplex.de> <31718.1439454658@critter.freebsd.dk> Message-ID: On Thu, Aug 13, 2015 at 10:30 AM, Poul-Henning Kamp wrote: > -------- > In message <55CBB949.8030107 at uplex.de>, Geoff Simmons writes: >>-----BEGIN PGP SIGNED MESSAGE----- >>Hash: SHA256 >> >>On 8/12/15 5:16 PM, Arianna Aondio wrote: >>> Hi, as discussed this afternoon on IRC with Federico, here is a >>> patch for a std_vmod function for converting from real to int. >> >>+VCL_INT __match_proto__(td_std_time2real) >>+vmod_real2integer(VRT_CTX, VCL_REAL r) >>+{ >>+ CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); >>+ >>+ return((long)floor(r)); >>+} >> >>Do we have to worry about what happens if r is non-finite (+-Inf or >>NaN)? (Is that possible for a VCL_REAL?) > > Yes. > > Proper range checking should be done, and like all other conversions, > a default value parameter to use when that fails should be added. Range control should also be added to functions such as real2time, time2integer, time2real. I'm also tempted to restrict the range just to positive numbers. -- Arianna Aondio Software Developer | Varnish Software AS Mobile: +47 98062619 We Make Websites Fly www.varnish-software.com From phk at phk.freebsd.dk Mon Aug 17 08:59:56 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 17 Aug 2015 08:59:56 +0000 Subject: WS/http_conn (Lasse & Martins patches) Message-ID: <38515.1439801996@critter.freebsd.dk> Lasse and Martin have sent in a group of patches which hit WS and http_conn from either side and I have tried to see if something sensible could be done before 4.1 with this area. I've cherry-picked some bits from the patches and rearranged some deck-chairs for better Feng-Shui, but what really needs to happen is not something I want to attempt right before 4.1-R. Here are some gory entrails for those interested: We originally had two operations on a workspace: Allocate and Reserve. The original idea was that one could still do allocations while part of the workspace was reserved, but that soon fell by the road-side. The main user of Reserve was the http_conn receive buffer - we didn't want to Alloc a buffer and drag the unused part around, so http_conn would reserve and only hang onto what was actually needed. The secondary user of Reserve is when we don't know how much space we will need, but these not really Reservations as much as time-extended Allocs. Later we got the Snapshot/Reset trick as a means for reusing WS, but that is really just another shape of Reserve once you look hard enough at it. Next we defined the backend-side timeouts (first/between bytes) in a way that makes it very hard to reuse the client side "receive a HTTP header" code for the backend. The http_conn ended up as a sort of H1-proto-state struct which keeps track of the existence of a body to be fetched etc. And then we added PROXY, which uses the http_conn, and has a rather incestuous relationship with the HTTP1 code all in all. And all along we have worked to make out-of-WS a handled error, rather than a panic. Now we're facing H2, which as entirely different wire/mem properties than H1, which pretty much violates the assumptions under http_conn. Some major work will happen here once 4.1 is cut, but to be honest, after mucking with it for a week I'm still not entirely sure how it should look in the long run... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Mon Aug 17 15:20:52 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 17 Aug 2015 17:20:52 +0200 Subject: Dynamic backends revisited, again Message-ID: Hi, Here is an update of 2 previous patches that got old and don't apply anymore. A third patch managed to sneak between them. 0001: should be trivial to review 0002: should be self-explanatory 0003: is hackish and most likely needs more details Best Regards, Dridi -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-VMOD-read-only-access-to-VCL-probe-definitions.patch Type: text/x-patch Size: 7110 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-Allow-dynamic-backend-creation-in-cold-VCLs.patch Type: text/x-patch Size: 1273 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0003-Allow-ACL-references-in-VMODs.patch Type: text/x-patch Size: 8244 bytes Desc: not available URL: From dridi at varni.sh Tue Aug 18 10:56:53 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 18 Aug 2015 12:56:53 +0200 Subject: Dynamic backends revisited, again In-Reply-To: References: Message-ID: Follow-up on the ACL patch after an initial review, the test case doesn't lazily panic on failure. Dridi -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Allow-ACL-references-in-VMODs.patch Type: text/x-patch Size: 8362 bytes Desc: not available URL: From dridi at varni.sh Tue Aug 18 13:22:28 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 18 Aug 2015 15:22:28 +0200 Subject: Dynamic backends revisited, again In-Reply-To: References: Message-ID: On Tue, Aug 18, 2015 at 12:56 PM, Dridi Boukelmoune wrote: > Follow-up on the ACL patch after an initial review, the test case > doesn't lazily panic on failure. Question I got while I was away on IRC: > dridi, so what difference is there between 'if (!debug.match.acl(loopback, "127.0.0.127"))" and "if (std.ip("127.0.0.127", "0.0.0.0") ~ loopback)" ? I'm not testing the ACL operator (~) but the use of ACLs from within a VMOD, both inside and outside of an HTTP transaction. From arianna.aondio at varnish-software.com Tue Aug 18 15:50:43 2015 From: arianna.aondio at varnish-software.com (Arianna Aondio) Date: Tue, 18 Aug 2015 17:50:43 +0200 Subject: [PATCH] new function and range control in std_vmod Message-ID: Hi, this is a follow up from the previous email I sent to this list suggesting a new function for the stc_vmod. The new function is a conversion function: real2integer. I've also added range control to all the other conversion functions. Please find patch(for master branch) attached. -- Arianna Aondio Software Developer | Varnish Software AS Mobile: +47 98062619 We Make Websites Fly www.varnish-software.com -------------- next part -------------- A non-text attachment was scrubbed... Name: std_range_check.patch Type: text/x-diff Size: 4390 bytes Desc: not available URL: From phk at phk.freebsd.dk Wed Aug 19 07:07:18 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 19 Aug 2015 07:07:18 +0000 Subject: [PATCH] new function and range control in std_vmod In-Reply-To: References: Message-ID: <16048.1439968038@critter.freebsd.dk> -------- In message , Arianna Aondio writes: > VCL_REAL __match_proto__(td_std_time2real) > -vmod_time2real(VRT_CTX, VCL_TIME t) > +vmod_time2real(VRT_CTX, VCL_TIME t, VCL_REAL r) > { > CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); > > + if (t > DBL_MAX || t < DBL_MIN || (!isfinite(t))) > + return (r); > return (t); > } Ohhh dear... Have I told you how much I hate floating-point corner cases ? Don't take this review the wrong way, I'm just using it as an educational tool to give you guys an idea how horrible FP-work is: First, if you ran this on a machine with silly ieee default flags and t were NAN of INF, your program would likely core-dump with a FP-trap. You should do the isfinite() test first to avoid this. There is a set of special comparison functions that will explictly not trap: isgreater(3), isless(3) etc. etc. Second, in this particular case (time/real) they're both doubles, so all you need is the isfinite() test. Third, DBL_MIN is not, as many suppose the largest negative number a double can hold, but rather the smallest positive number it can hold without denormalizing. Your code above fails on any argument zero or less. Negative timestamps are allowed, they're moments in time before 1970. > VCL_TIME __match_proto__(td_std_real2time) > -vmod_real2time(VRT_CTX, VCL_REAL r) > +vmod_real2time(VRT_CTX, VCL_REAL r, VCL_TIME t) > { > CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); > > + if (r > LONG_MAX || r < LONG_MIN || (!isfinite(r))) > + return (t); > return (r); > } Why you use LONG_MIN/LONG_MAX here when VCL_TIME is a double ? > +VCL_INT __match_proto__(td_std_real2integer) > +vmod_real2integer(VRT_CTX, VCL_REAL r, VCL_INT i) > +{ > + CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); > + > + if (r > LONG_MAX || r < LONG_MIN || (!isfinite(r))) > + return (i); > + return ((long)floor(r)); > +} > + > VCL_TIME __match_proto__(td_std_real2time) isfinite() should be first. This one adds a new horror to the beastiarium: Rounding. I don't think your choice of floor() is correct, I would expect round(). But we're not done: Imagine the double number is (LONG_MAX + .4), that should round down and pass the <= LONG_MAX test. However, it will not, because C will promote LONG_MAX to a double making the comparison fail. You have to explicitly round the double first, which means that you cannot use the isless() family, since round() would TRAP before isless got called, so something like: if (!isfinite(r)) return (i); r = round(r); if (r > LONG_MAX || r < LONG_MIN) return (i); return ((long)r); > + set resp.http.x-fallback1 = std.real2time(382473843278423748923749238429834.848383, now); > + set resp.http.x-fallback2 = std.real2integer(382473843278423748923749238429834.848383, 2); First you should add a comment about what these magic numbers are, and what limits you expect them to exceed. Second, they're no good: If LONG were a 128bit type the test would fail. Third, this might fail on machines with FP implementations than double=ieeebin64 Fourth, passing *precise* FP values around in decimal format is wrong and imprecise, setting you up for rounding error grief. There is a special printf(3) format for representing FP numbers exactly, the "%a" format which looks like "0x1.921fb54411744p+1" In this case a better strategy might be to use "9e99" -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Thu Aug 20 10:39:20 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 20 Aug 2015 12:39:20 +0200 Subject: [PATCH] reliable name resolution in varnishtest Message-ID: Hi, Yesterday I had an idea to make it possible to reliably test a DNS director, hoping to contribute one soon with a decent coverage. Today I have come back to it and finished my prototype. I have already discussed this patch's contents with phk and it's likely to be rejected unless it manages to stay simple enough. I'm submitting my prototype for a review, it's basically my initial idea with phk's guidelines on how to work around the process isolation in varnish. I haven't polished it yet, I could see a couple areas to slightly improve, but nothing major. Feedback greatly appreciated! Best Regards, Dridi -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Introduce-reliable-name-resolution-in-varnishtest.patch Type: text/x-patch Size: 12921 bytes Desc: not available URL: From dridi at varni.sh Thu Aug 20 10:43:08 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 20 Aug 2015 12:43:08 +0200 Subject: [PATCH] reliable name resolution in varnishtest In-Reply-To: References: Message-ID: On Thu, Aug 20, 2015 at 12:39 PM, Dridi Boukelmoune wrote: > Feedback greatly appreciated! I forgot to mention I've only tested it on Linux, but I believe I've only used portable APIs. From dridi at varni.sh Fri Aug 21 17:12:39 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Fri, 21 Aug 2015 19:12:39 +0200 Subject: Dynamic backends revisited, again In-Reply-To: References: Message-ID: On Mon, Aug 17, 2015 at 5:20 PM, Dridi Boukelmoune wrote: > 0002: should be self-explanatory Patch number 2 was rejected, so I updated my DNS director draft to track VCL events and act accordingly. When a director's VCL emits a COLD event, there's still a race condition if the event is emitted while the director is adding backends. There's no locking workaround, and you can't batch backends additions. The only solution I see so far, would be to check between every insertions whether the VCL emitted a COLD event. I don't like that too much, for it leaves a half updated backends list in a cold VCL that could become WARM again. Currently, I have a background loop that does this: while (not discarded) { if neither WARM nor USE cond_wait if neither WARM nor USE continue i = lookup (blocking call) if i != 0 { log error continue } if neither WARM nor USE continue update_backends cond_timedwait } The cond_wait is signaled whenever a new event occurs, the cond_timedwait is signaled when the VCL is discarded. So basically I need to move the last WARM/USE check inside the update_backends loop, which is OK for a DNS director, once the VCL becomes WARM again it does a lookup straight away. I'm not sure that's enough to prevent the race, I need to take a closer look at the locks involved in both process (adding backends and cooling the VCL). I'm not asking to reconsider the patch, I'm sharing my observations regarding the current constraints. Also, can it be checked with the witness facility? I think that the witness code only works for the Lck_ functions and macros, but I didn't get into the specifics when it was introduced. Best Regards, Dridi From dridi at varni.sh Sun Aug 23 10:41:01 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Sun, 23 Aug 2015 12:41:01 +0200 Subject: Dynamic backends revisited, again In-Reply-To: References: Message-ID: On Fri, Aug 21, 2015 at 7:12 PM, Dridi Boukelmoune wrote: > So basically I need to move the last WARM/USE check inside the > update_backends loop, which is OK for a DNS director, once the VCL > becomes WARM again it does a lookup straight away. I'm not sure that's > enough to prevent the race, I need to take a closer look at the locks > involved in both process (adding backends and cooling the VCL). After a quick glance today, I'm pretty sure there's an inherent race when you call VRT_new_backend from outside the CLI thread. It's not possible to lock the VCL since the structure is opaque and it would regardless conflict with the underlying call to VCL_AddBackend -- locking the VCL too. From dridi at varni.sh Mon Aug 24 09:35:06 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 24 Aug 2015 11:35:06 +0200 Subject: [VDD15Q3] Move timeouts outside of the backend VCL definition Message-ID: Hi, We can currently declare up to three timeouts in a backend definition. --->8-------->8-------->8-------->8-------->8--- backend www { [...] .connect_timeout = ...; .first_byte_timeout = ...; .between_bytes_timeout = ...; } --->8-------->8-------->8-------->8-------->8--- If I have a director for a cluster of similar backends that all share the same timeout values, I can only repeat them for all declared backends. And relying on Varnish's defaults only works when you want to share only one set of timeouts. Instead, one could imagine doing this: --->8-------->8-------->8-------->8-------->8--- timeouts www_timeouts { .connect = ...; .first_byte = ...; .between_bytes = ...; } backend www1 { [...] .timeouts = www_timeouts; } backend www2 { [...] .timeouts = www_timeouts; } backend static { [...] .timeouts = { .connect = ...; .first_byte = ...; .between_bytes = ...; } } --->8-------->8-------->8-------->8-------->8--- You can imagine declaring timeouts based on backends locations (eg. lan vs wan etc) or behaviors (streaming responses vs building them upfront etc). Having standalone VCL symbols could also allow a read-only access to timeouts definitions for VMODs dealing with dynamic backends. Best Regards, Dridi PS. inspired by recent work on probes From fgsch at lodoss.net Mon Aug 24 09:37:36 2015 From: fgsch at lodoss.net (Federico Schwindt) Date: Mon, 24 Aug 2015 10:37:36 +0100 Subject: [master] d771942 Hide backends and directors implementation details In-Reply-To: References: Message-ID: As I mentioned to Dridi I'm not too thrilled about this. This pollutes cache.h and causes artificial breakages. I'd prefer, if we really want to do this, to take the _int.h approach. I also had a chat with Martin not so long ago and one of the things that came in that discussion is the possibility of installing all the includes. This is a step backwards from that. On Sun, Aug 23, 2015 at 11:35 AM, Dridi Boukelmoune < dridi.boukelmoune at gmail.com> wrote: > > commit d771942c0c6f151e267b63bbc548a4d4371b25be > Author: Dridi Boukelmoune > Date: Sun Aug 23 12:29:29 2015 +0200 > > Hide backends and directors implementation details > > Don't install cache_backend.h and cache_director.h, they expose more > than what a VMOD needs to see. > > The VRT API jumps directly from struct vrt_backend to struct director, > which makes struct backend more than simply opaque, it is completely > invisible. However struct director needs to be visible and has been > moved to cache.h along with its functions definitions. > > diff --git a/bin/varnishd/Makefile.am b/bin/varnishd/Makefile.am > index dd2bae2..eb01348 100644 > --- a/bin/varnishd/Makefile.am > +++ b/bin/varnishd/Makefile.am > @@ -99,6 +99,8 @@ varnishd_SOURCES = \ > > noinst_HEADERS = \ > builtin_vcl.h \ > + cache/cache_backend.h \ > + cache/cache_director.h \ > cache/cache_esi.h \ > cache/cache_pool.h \ > common/heritage.h \ > @@ -116,8 +118,6 @@ noinst_HEADERS = \ > nobase_pkginclude_HEADERS = \ > cache/cache.h \ > cache/cache_filter.h \ > - cache/cache_backend.h \ > - cache/cache_director.h \ > common/common.h \ > common/params.h \ > waiter/waiter.h > diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h > index 648dad4..ed72296 100644 > --- a/bin/varnishd/cache/cache.h > +++ b/bin/varnishd/cache/cache.h > @@ -190,6 +190,47 @@ struct http { > }; > > /*-------------------------------------------------------------------- > + * Director public APIs > + */ > + > +typedef unsigned vdi_healthy_f(const struct director *, const struct > busyobj *, > + double *changed); > + > +typedef const struct director *vdi_resolve_f(const struct director *, > + struct worker *, struct busyobj *); > + > +typedef int vdi_gethdrs_f(const struct director *, struct worker *, > + struct busyobj *); > +typedef int vdi_getbody_f(const struct director *, struct worker *, > + struct busyobj *); > +typedef const struct suckaddr *vdi_getip_f(const struct director *, > + struct worker *, struct busyobj *); > +typedef void vdi_finish_f(const struct director *, struct worker *, > + struct busyobj *); > + > +typedef void vdi_http1pipe_f(const struct director *, struct req *, > + struct busyobj *); > + > +typedef void vdi_panic_f(const struct director *, struct vsb *); > + > +struct director { > + unsigned magic; > +#define DIRECTOR_MAGIC 0x3336351d > + const char *name; > + char *vcl_name; > + vdi_http1pipe_f *http1pipe; > + vdi_healthy_f *healthy; > + vdi_resolve_f *resolve; > + vdi_gethdrs_f *gethdrs; > + vdi_getbody_f *getbody; > + vdi_getip_f *getip; > + vdi_finish_f *finish; > + vdi_panic_f *panic; > + void *priv; > + const void *priv2; > +}; > + > +/*-------------------------------------------------------------------- > * VFP filter state > */ > > diff --git a/bin/varnishd/cache/cache_director.h > b/bin/varnishd/cache/cache_director.h > index cf20dc1..3bccf80 100644 > --- a/bin/varnishd/cache/cache_director.h > +++ b/bin/varnishd/cache/cache_director.h > @@ -37,49 +37,6 @@ > * > */ > > -/*-------------------------------------------------------------------- > - * A director is a piece of code which selects one of possibly multiple > - * backends to use. > - */ > - > - > -typedef unsigned vdi_healthy_f(const struct director *, const struct > busyobj *, > - double *changed); > - > -typedef const struct director *vdi_resolve_f(const struct director *, > - struct worker *, struct busyobj *); > - > -typedef int vdi_gethdrs_f(const struct director *, struct worker *, > - struct busyobj *); > -typedef int vdi_getbody_f(const struct director *, struct worker *, > - struct busyobj *); > -typedef const struct suckaddr *vdi_getip_f(const struct director *, > - struct worker *, struct busyobj *); > -typedef void vdi_finish_f(const struct director *, struct worker *, > - struct busyobj *); > - > -typedef void vdi_http1pipe_f(const struct director *, struct req *, > - struct busyobj *); > - > -typedef void vdi_panic_f(const struct director *, struct vsb *); > - > -struct director { > - unsigned magic; > -#define DIRECTOR_MAGIC 0x3336351d > - const char *name; > - char *vcl_name; > - vdi_http1pipe_f *http1pipe; > - vdi_healthy_f *healthy; > - vdi_resolve_f *resolve; > - vdi_gethdrs_f *gethdrs; > - vdi_getbody_f *getbody; > - vdi_getip_f *getip; > - vdi_finish_f *finish; > - vdi_panic_f *panic; > - void *priv; > - const void *priv2; > -}; > - > /* cache_director.c */ > > int VDI_GetHdr(struct worker *, struct busyobj *); > diff --git a/lib/libvmod_directors/vdir.c b/lib/libvmod_directors/vdir.c > index 411df34..1738bcf 100644 > --- a/lib/libvmod_directors/vdir.c > +++ b/lib/libvmod_directors/vdir.c > @@ -31,7 +31,6 @@ > #include > > #include "cache/cache.h" > -#include "cache/cache_director.h" > > #include "vrt.h" > #include "vbm.h" > > _______________________________________________ > varnish-commit mailing list > varnish-commit at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-commit > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Mon Aug 24 09:48:28 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 24 Aug 2015 11:48:28 +0200 Subject: [master] d771942 Hide backends and directors implementation details In-Reply-To: References: Message-ID: On Mon, Aug 24, 2015 at 11:37 AM, Federico Schwindt wrote: > As I mentioned to Dridi I'm not too thrilled about this. As I told Federico, I may have rushed this change yesterday, but my fear is that I'll spend the week running after the white rabbit. > This pollutes cache.h and causes artificial breakages. I'd prefer, if we > really want to do this, to take the _int.h approach. My take on this is that I don't care breaking stuff on a major release that already breaks other areas. If there's a better place for the public APIs (maybe vrt.h) then let's move them there. > I also had a chat with Martin not so long ago and one of the things that > came in that discussion is the possibility of installing all the includes. > This is a step backwards from that. As I also told yesterday, that may make things seemingly simpler, but I think it would be even more confusing wrt what you could actually use or not. Best, Dridi From phk at phk.freebsd.dk Mon Aug 24 10:04:04 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 24 Aug 2015 10:04:04 +0000 Subject: [master] d771942 Hide backends and directors implementation details In-Reply-To: References: Message-ID: <75284.1440410644@critter.freebsd.dk> -------- In message , Federico Schwindt writes: >As I mentioned to Dridi I'm not too thrilled about this. Personally I'm not to thrilled about it being committed without any review... >This pollutes cache.h and causes artificial breakages. I'd prefer, if we >really want to do this, to take the _int.h approach. I have refrained from this myself, because I have a really hard time deciding where the line should be drawn. >I also had a chat with Martin not so long ago and one of the things that >came in that discussion is the possibility of installing all the includes. >This is a step backwards from that. I'm not too thrilled about that either, because that makes everything in varnishd an API seen from a VMOD point of view. (gotta run over at take a tent down...) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From fgsch at lodoss.net Mon Aug 24 11:06:16 2015 From: fgsch at lodoss.net (Federico Schwindt) Date: Mon, 24 Aug 2015 12:06:16 +0100 Subject: [master] d771942 Hide backends and directors implementation details In-Reply-To: <75284.1440410644@critter.freebsd.dk> References: <75284.1440410644@critter.freebsd.dk> Message-ID: On Mon, Aug 24, 2015 at 11:04 AM, Poul-Henning Kamp wrote: > -------- > In message < > CAJV_h0Yj-fLdhR7Qv0PgJU1Kg_XZKR_0Cie3aGXKXJWd2rEckw at mail.gmail.com> > , Federico Schwindt writes: > >[..] > >This pollutes cache.h and causes artificial breakages. I'd prefer, if we > >really want to do this, to take the _int.h approach. > > I have refrained from this myself, because I have a really hard time > deciding where the line should be drawn. > We should try to be consistent while we decide. > >I also had a chat with Martin not so long ago and one of the things that > >came in that discussion is the possibility of installing all the includes. > >This is a step backwards from that. > > I'm not too thrilled about that either, because that makes everything > in varnishd an API seen from a VMOD point of view. > Aren't we there already, e.g. https://www.varnish-cache.org/trac/ticket/1759 ? Not installing everything just makes things harder for not much gain AFAICT. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ingvar at redpill-linpro.com Wed Aug 26 21:45:14 2015 From: ingvar at redpill-linpro.com (Ingvar Hagelund) Date: Wed, 26 Aug 2015 23:45:14 +0200 Subject: Patch: redhat startup scripts should no longer use -u user Message-ID: <55DE336A.8070408@redpill-linpro.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 varnish-4.1.0 does no longer support the -u user option. This changes the various variants of redhat startup scripts to use the unix "jail" variant. Ingvar -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJV3jNhAAoJEB4pl4iKhgvRafEP/0diip1SqH5xZoXqL7R8kUGV OpSpWpC2f/rJsPNVtZP+/ZSUyKg4bKn6eGff0F/2O2KY59a2Cv21KRytA3dptECQ pBJzFw4r2BiG62BmbMHeb95Vdbs81wh/h2EdagAK9J/PwwRPIhAk8jsvoEd6Q7ms sgkfaw30wNDvN1DVxUn6Wwhqs/igi1Ssm2lav5VzhfaLmCnkmkMZ2FG2PJ9R8w/M ekzX1hy/Q6Qh6cPb+OH2CgQN5l5C4XO0ep3Xn/44gU/eA5FqfNKiFpOlhfCPWtSX lwHoKU4LL7DOHK3UnX+elsG0iwjGyXJ70ATKzhSz7sj7e4QBMe2eko6XEzHCiID6 54EqBVwR+zrQfyLiOPy1pGGqc1IcRPiYnwWwkGqRRiRyyG1gUq6NPPb8cOkfBj0W t08Yz/HMrfK5NYDK2pLLW+ZH7mf0vCFSSSzDmIxTLnAgBl6xZ4PrqIkgRioht3xQ gz4FIzyNwSbu1zqLmkYUgIqcs+y68pi+BZvgUjFu8r+Vghvsn51doWh2yfB6fhLu OTdzls4faj58mcu/kKWePAaxKj60OMy45d+3lSaKbcxh0zzdUwJrbD6qYUFLkXA7 56hQF41VMlBniTOUVHgf8k3WEU7srYDzq3XfLG1YGDDD+fHIjchRPinmi6mCEYBU OtB94xkXNdg6Au6YMKVY =gcyH -----END PGP SIGNATURE----- -------------- next part -------------- A non-text attachment was scrubbed... Name: varnish-4.1.0-tp1.no_more_dash_u_for_user.patch Type: text/x-patch Size: 3357 bytes Desc: not available URL: From ingvar at redpill-linpro.com Thu Aug 27 07:05:59 2015 From: ingvar at redpill-linpro.com (Ingvar Hagelund) Date: Thu, 27 Aug 2015 09:05:59 +0200 Subject: [PATCH]: redhat startup scripts should no longer use -u user In-Reply-To: <55DE336A.8070408@redpill-linpro.com> References: <55DE336A.8070408@redpill-linpro.com> Message-ID: <55DEB6D7.8090109@redpill-linpro.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Den 26. aug. 2015 23:45, skrev Ingvar Hagelund: > varnish-4.1.0 does no longer support the -u user option. This > changes the various variants of redhat startup scripts to use the > unix "jail" variant. Or even better, giving users the probably very seldom used option to use some other user than "varnish" by the sysconfig/params files. Ingvar -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJV3rbTAAoJEB4pl4iKhgvREWMQAJJs8wfogMHfyRgED7vrscCL T5JCXN2hv/06LpzgM3c1gGdoAQOBObh83ZSeZCd+XMGlvg1FAuJn11BrV1DbUmmZ +rZWwmOcg5a6oKGBGFVOOAD/GrfrsEyN+3KdHRnfb7mGKPyMKOych8Ir4y9hbx2y 7xfxjmMk8JCNugYyS+F36Umd8uYCDhkr5x1cN50j7P32KWMQEd21XGfahJ5kLN2V rCbKa2EDlWXECFA2u90ZUBc3kI61IT/DqJxvdaO8/MtYp1oz37EF3uu+bnJ1YlDn pQpVO4IdWPif4y7OjHvbA/FNig9gbO/TMVKnyl0Npy+vJpaEcV6Nmo5tMCL5yJjL CgM95PwAhn77HufcFDoUFj8NZUWgg+1B7yUxECEQoSwOmfD/6z9Wd+uxcziu+FG0 8CLxA9dPMYPfYCKdz3ik2/IAvxGmfsZOdSEzPLumFDYOd/FEClZrAlWpLijysAhh Jpezrb5HLCpHM5aDAJuCUN9ug6Endur2KebVj0cYk3qUUf4JTOnkZ6ZpwBWMd5on R0zhbh5ie/VjzcI4alRrNLRvSmhDu/tXgeUB20cphnOVF0h8jW0Oo38DC2yFiS+a gUTmteKTFNvGEKB8TniqiuCWbQ8omURH/YhSShzd7T5FkxAE4Shl3QLqS7c+aF1b Rycpt+/DV0QlHVzu9NNO =3Rmi -----END PGP SIGNATURE----- -------------- next part -------------- A non-text attachment was scrubbed... Name: varnish-4.1.0-tp1.no_more_dash_u_for_user.patch Type: text/x-patch Size: 3556 bytes Desc: not available URL: From arianna.aondio at varnish-software.com Thu Aug 27 13:22:28 2015 From: arianna.aondio at varnish-software.com (Arianna Aondio) Date: Thu, 27 Aug 2015 15:22:28 +0200 Subject: [VDD15Q3] VMOD in main distribution and VFP-based VMOD(s) Message-ID: Hi everyone, two are the points I'd like to discuss during the upcoming VDD: * (HIGH PRIORITY) I'd like libvmod-rstatus to be part of the main distribution. I've already written an email to the dev-list in this regard, please read the linked email in order to have some context(https://www.varnish-cache.org/lists/pipermail/varnish-dev/2015-July/008394.html). The final idea is to have a standalone VMOD, part of the main distribution, that creates a json object containing several Varnish counters, with particular attention to backends' counters. * (LOW PRIORITY)VFP-based VMODs and response body accessibility. We are able to access the request body using a VMOD (https://github.com/aondio/libvmod-bodyaccess), but nothing has been done for the response body. Accessing and modifying the response body is possible during the fetch process, before the object is stored in the cache. Implementing VFP-based VMODs opens up to a lot of different possibilities such as transformations, do regex on response body, images resizing, etc, etc... It would be nice to have VMODs able to insert themselves in th VFP chain. See you all on Tuesday! -- Arianna Aondio Software Developer | Varnish Software Group Mobile: +47 98062619 We Make Websites Fly www.varnish-software.com From ruben at varnish-software.com Fri Aug 28 13:18:10 2015 From: ruben at varnish-software.com (=?UTF-8?Q?Rub=C3=A9n_Romero?=) Date: Fri, 28 Aug 2015 15:18:10 +0200 Subject: [VDD15Q3] - Updated information Message-ID: Hello everyone, I'm happy to announce that next Tuesday 9.00 AM we will be meeting at the Opera HQ in Oslo for the next Varnish Core Developer meeting. You just need to get yourself there and register for entrance (no ID required, but your name should be on the list). You can see details and additional information here: https://www.varnish-cache.org/trac/wiki/VDD15Q3 *In behalf of the Varnish project I would like to thank Opera Software for hosting the meeting and specially to Cosimo for helping us to make this happen and bringing the team back to Oslo!* For those of you that want to join we will have a dinner on Monday. Meet up at Citybox at 19.00 and we will walk from there to our restaurant. If you cannot find us there or want to join later, my cell number is below. Text me. Looking forward to see you all next week! Best regards, -- *Rub?n Romero* Varnish Software Group Cell: +47 95964088 / Office: +47 21989260 Skype, Twitter & IRC: ruben_varnish We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at varnish-software.com Fri Aug 28 15:05:44 2015 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 28 Aug 2015 17:05:44 +0200 Subject: WS/http_conn (Lasse & Martins patches) In-Reply-To: <38515.1439801996@critter.freebsd.dk> References: <38515.1439801996@critter.freebsd.dk> Message-ID: Hi, My apologies for having my vacation in the middle of the execution of these changes. Makes the process halt and context lost. I still hope something can be salvaged though. To recap the story, and prepare the others for this as a discussion topic on the coming VDD, it is to facilitate SSL enabled backends in Varnish. Having good SSL support with minimum hassle is becoming increasingly important. To that effort we have released the Hitch project to terminate the client connections, and using the proxy protocol it interfaces nicely with Varnish. On the backend side however using a proxy is still cumbersome and error prone, and solving it should be a priority. I see PHKs arguments for avoiding libssl in the Varnish source tree, so some other solution should be found. We do have the API available to provide Varnish with arbitrary headers and body bytes through a VMOD backend objects. This is used in libvmod-fsbackend to read static files from the file system. The API is complete from this point of view. Implementing SSL backends using VMOD backend objects would allow us to get "native" to VCL backends without polluting the Varnish source tree with libssl-linkage. I've started writing code for this. The parsing of the HTTP headers and body content doesn't change with SSL, only the reading and writing of bytes on the socket changes with appropriate calls to the SSL library instead of the read()/writev() system calls. But reusing the HTTP implementation of Varnish while changing the way bytes are read/written is impossible in it's current form. To avoid having to reimplement all of the HTTP/1 code in the VMOD, I would like to add a set of function pointers associated with a HTTP connection. The VMOD will then be able to insert it's own read()/writev() implementation using the appropriate means to encrypt/decrypt. The natural place to have these function pointers is the struct http_conn. I sent a patch set to achieve this on dev a while ago. It moved things around a bit, cleaning up some places but muddled the water a bit in others. A new sketch for how to design this was worked out with PHK on IRC, but I couldn't manage to complete the second iteration of the patch set before my vacation. It's mostly done though, and can be brought to completion for 4.1. End of recap. If I understand PHKs conclusion correctly, he has since been looking at the code and how to make it right and future proof at a deeper level than my patch set or the unfinished new sketch attempted. This was found to be too much of an undertaking this late in the release cycle and thus should be postponed. I want to argue that the sketched solution, which is close to completion, should be seriously considered for inclusion in 4.1. I agree that it doesn't deal with the incestuous relationship with proxy protocol in any ways, but that is the case with the current code as well. That patch set would create something that'll be only for the lifetime of 4.x, but these areas that are bound to see big changes happening with H2 anyway. If we don't include this and postpone it for 4.2, we push the SSL backend VMOD forward in time as well. I think this is functionality that it is important for the project to get sooner rather than later and that would be a loss. -Martin On 17 August 2015 at 10:59, Poul-Henning Kamp wrote: > Lasse and Martin have sent in a group of patches which hit WS and > http_conn from either side and I have tried to see if something > sensible could be done before 4.1 with this area. > > I've cherry-picked some bits from the patches and rearranged some > deck-chairs for better Feng-Shui, but what really needs to happen > is not something I want to attempt right before 4.1-R. > > Here are some gory entrails for those interested: > > We originally had two operations on a workspace: Allocate and Reserve. > > The original idea was that one could still do allocations while part of > the workspace was reserved, but that soon fell by the road-side. > > The main user of Reserve was the http_conn receive buffer - we didn't > want to Alloc a buffer and drag the unused part around, so http_conn > would reserve and only hang onto what was actually needed. > > The secondary user of Reserve is when we don't know how much space > we will need, but these not really Reservations as much as time-extended > Allocs. > > Later we got the Snapshot/Reset trick as a means for reusing WS, > but that is really just another shape of Reserve once you look hard > enough at it. > > Next we defined the backend-side timeouts (first/between bytes) in a > way that makes it very hard to reuse the client side "receive a > HTTP header" code for the backend. > > The http_conn ended up as a sort of H1-proto-state struct which keeps > track of the existence of a body to be fetched etc. > > And then we added PROXY, which uses the http_conn, and has a rather > incestuous relationship with the HTTP1 code all in all. > > And all along we have worked to make out-of-WS a handled error, rather > than a panic. > > Now we're facing H2, which as entirely different wire/mem properties > than H1, which pretty much violates the assumptions under http_conn. > > Some major work will happen here once 4.1 is cut, but to be honest, > after mucking with it for a week I'm still not entirely sure how it > should look in the long run... > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -- *Martin Blix Grydeland* Senior Developer | Varnish Software AS Mobile: +47 992 74 756 We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From slink at schokola.de Fri Aug 28 17:09:59 2015 From: slink at schokola.de (Nils Goroll) Date: Fri, 28 Aug 2015 19:09:59 +0200 Subject: WS/http_conn (Lasse & Martins patches) In-Reply-To: References: <38515.1439801996@critter.freebsd.dk> Message-ID: <55E095E7.80704@schokola.de> Without getting into any detail, I fully agree that we should really aim for more of an iterative/incremental cycle, as long as incremental improvements don't seriously hinder the next step. Nils From phk at phk.freebsd.dk Mon Aug 31 09:43:31 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 31 Aug 2015 09:43:31 +0000 Subject: VDD15Q3 agenda Message-ID: <8797.1441014211@critter.freebsd.dk> I'm going over the agenda. I don't want to log into the wiki on this network, but I and going to say this up front: The major topic of this VDD will have to be "How do we get 4.1-R out ?" I am *not* going to entertain *any* patches for new features until we know and have fixed whatever caused Geoffs test to run poorly. If need be, I'm prefectly fine with us converting the VDD into an ad-hoc live test of -trunk on whatever websites we can comandeer for that purpose. I know you all have a list of pet features and requests, and I know they're all very important to you and your job, but you can take it from me: None of that is going *anywhere* until after 4.1R. See you in Oslo... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From slink at schokola.de Mon Aug 31 15:10:18 2015 From: slink at schokola.de (Nils Goroll) Date: Mon, 31 Aug 2015 17:10:18 +0200 Subject: VDD15Q3 agenda In-Reply-To: <8797.1441014211@critter.freebsd.dk> References: <8797.1441014211@critter.freebsd.dk> Message-ID: <55E46E5A.606@schokola.de> Hi, > until we know and have fixed whatever caused Geoffs test to run poorly. I feel unhappy about dedicating a whole VDD to working on issues which Geoff has not yet had the opportunity to reproduce - unfortunately he had to go on sick leave right after he got the first results. Geoff is working on this as I type so we may have more results tomorrow, but until we have excluded other possible causes and at least have a rough idea about where to look, I'd prefer if we made plans for a VDD to be beneficial to everyone. Specific performance issues, should they exist, are, at least to my experience, nothing which can be worked on productively in a larger group. Geoff suggested that he may shall have a closer look at his results with one or two people. And I really don't want a room full of people to chase for an issue which may not even be root caused in Varnish. Thanks, Nils From phk at phk.freebsd.dk Mon Aug 31 15:44:06 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 31 Aug 2015 15:44:06 +0000 Subject: VDD15Q3 agenda In-Reply-To: <55E46E5A.606@schokola.de> References: <8797.1441014211@critter.freebsd.dk> <55E46E5A.606@schokola.de> Message-ID: <10763.1441035846@critter.freebsd.dk> -------- In message <55E46E5A.606 at schokola.de>, Nils Goroll writes: >Hi, > >> until we know and have fixed whatever caused Geoffs test to run poorly. > >I feel unhappy about dedicating a whole VDD to working on issues [...] I don't want to spend the entire VDD on it. But so far, Geoffs test is the only live(-ish) test of -trunk I have heard of in a looong time, so I'm treating the failure(-ish) of the test very seriously. If I had more test-reports and they said differently, I would be more inclined to discount Geoffs data point, but having *only* that data point, I cannot ignore it. >Specific performance issues, should they exist, are, at least to my experience, >nothing which can be worked on productively in a larger group. The issue I want us to talk about is "How do we gain confidence that 4.1 isn't going to be a disaster when we havn't tested it?" I *also* want to talk about 4.2/5.0 and the future, but right here and now our biggest problem is that we don't know if 4.1-R can be released or not, and that should be our top priority. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From geoff at uplex.de Mon Aug 31 15:44:43 2015 From: geoff at uplex.de (Geoff Simmons) Date: Mon, 31 Aug 2015 17:44:43 +0200 Subject: VDD15Q3 agenda In-Reply-To: <8797.1441014211@critter.freebsd.dk> References: <8797.1441014211@critter.freebsd.dk> Message-ID: <55E4766B.7040601@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 08/31/2015 11:43 AM, Poul-Henning Kamp wrote: > > The major topic of this VDD will have to be "How do we get 4.1-R > out ?" > > I am *not* going to entertain *any* patches for new features until > we know and have fixed whatever caused Geoffs test to run poorly. I just got word that we unfortunately can't use the same load test environment tomorrow. It may be possible later in the week. I still don't know if the problems that day were something specific to our deployment; we do know that there was a successful load test with 4.0.3 earlier on the same day. If we can run tests tomorrow in another environment, and similar problems arise, then we'd have something to work with. But if we can't reproduce the issue, the question will still be open as to what happened on the first attempt. (Maybe even if we can reproduce it.) So to echo Nils' thoughts a bit -- if we dedicate the VDD tomorrow to load testing, we may have an inconclusive result no matter how we go about it. Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQIcBAEBCAAGBQJV5HZrAAoJEOUwvh9pJNURfE0QAIh7xUmSd3B7rBf4vluWcnH1 ZoTrq4ungD/j+052GHvLRWC9PChKpqev4KcJWo8PCUaaYO6lrCPx0UZsfmnRVELw y3kjnwB4W/TZtNNqJM8xzWXMLIMQAqHgBmZ4epHVpPuJ4aSKMaR8H6NH1PSXE/cm 5Dblrzp3pyAn4RIch+nBePW6Pqa91BMK82+rMm3nLkTzmSpmN7bEt8eA2YqrnqY7 tSZ6Q9Okq1Tii1kr3813slLo0nCtV3y6WDNfm4kLbAd8vvyYUypoF6ETLgELQpIt lfvgHFqUOi/MVEAipHrZlbFZFiTnZqxRmZ9bNtgLtI9fVa4Zttm6FPO5g9r0Lq3p lESkUJM0V4bjHNC7EujmpnuraW85Kh7wTeDqzAfHidYBqh+YbMb80H0DI8I9+l5B IG914EGKx+ibvyhdg8UNlhqPkPKap83VxnXz0pn6e6sS+H+YqarcIvOLAh8ZkDU/ 52ACCq9+jlLCZOIf+9ZmXBkjX8A5tXYNLLtzWqJBYlChnQzFMI26VWDpYvm7OvGS vKyS5m2mfTxy3t52nBBV5bt3OLgOrxdNORgkXVb2VNza2tsOVbGoYIwyu6QqGmfN 8LTrgu28lRjkqxlDAshLVoJ0YSbBn9iX7ifY8y/h6GavidBT/8aW30tV91o85xTh cVlvLKIf6MB5svyuLWv3 =CxWV -----END PGP SIGNATURE-----