From lkarsten at varnish-software.com Wed Dec 2 13:53:56 2015 From: lkarsten at varnish-software.com (Lasse Karstensen) Date: Wed, 2 Dec 2015 14:53:56 +0100 Subject: VMOD versioning guidelines Message-ID: <20151202135354.GA19950@immer.varnish-software.com> Hi all. I've written up a guideline or recommendation on how to do versioning on Varnish vmods. It can be found on: https://www.varnish-cache.org/trac/wiki/VMOD_versioning_guidelines The problem it looks to solve is how we signal which Varnish version and VRT ABI a distributed vmod package works with. Currently this is a bit of a mess. Digest: libvmod-foo-VMODVER[+GITBRANCH].tar.gz libvmod-foo-1.1.tar.gz (vmod-foo has a single branch called master) libvmod-foo-1.14+4.0.tar.gz (vmod-foo's 4.0 branch, standalone.) libvmod-foo-1.15+v4.0-plus.tar.gz vmod-foo-1.14+4.0-1.x86_64.rpm vmod-foo-1.14+4.0-1.deb The work mostly comes out from the vmod workshop earlier this fall with Stig, Ingvar, Kristian and me. Now that this is in place (or at least a first version up for discussion) we can start releasing vmod source packages that is easily built by third party packagers. -- Lasse Karstensen Varnish Software AS From lkarsten at varnish-software.com Wed Dec 2 14:40:42 2015 From: lkarsten at varnish-software.com (Lasse Karstensen) Date: Wed, 2 Dec 2015 15:40:42 +0100 Subject: Update on vmods and packaging Message-ID: <20151202144041.GB19950@immer.varnish-software.com> Hi all. Varnish Software (VS) is writing and maintaining a fair amount of vmods for Varnish Cache. This includes major ones like vmod-header, which can be said to be borderline core Varnish functionality. We will be doing some changes on how these are maintained and distributed. Currently the normal way of using them (in the community) is to do a git checkout from Github, build it from source and run it. This is all very bleeding edge, and leaves something to be desired. We (VS) will start doing versioned tarball releases of the vmods. They will get changelogs, and the tarballs will be made available for mirroring just as Varnish Cache code is. This will allow distribution packagers to add the vmod packages on their platforms. I'm working with the Fedora/EPEL and Debian packagers to get it in place there. Additionally we will try to write down what the community can expect from VS in this regard, to help everyone make informed decisions on what to run. (on how long) The tarball releases will be adopted to the current latest Varnish Cache release. The development version will continue to live on Github, but will be moved to master branch in each one. Code for 4.0 and 3.0 will be deprecated. Implementation of this plan will be gradual over the next months. Further down the line it is likely that we will be combining the simple vmods (with no third party/library dependencies) into a single collection source package. Updates on that will come later. I'm happy to answer any questions on this, either on-list or privately. -- Lasse Karstensen Varnish Software (with VS hat on.) From ksorensen at nordija.com Fri Dec 4 08:42:23 2015 From: ksorensen at nordija.com (Kristian =?ISO-8859-1?Q?Gr=F8nfeldt_S=F8rensen?=) Date: Fri, 04 Dec 2015 09:42:23 +0100 Subject: Varnish 5.0 changes to defaults Message-ID: <1449218543.12225.58.camel@nordija.com> Hi, After we talked about changes to default values for 5.0 yesterday at VUGX, I realised that I'd like varnishlog to group log entries by request as a default, rather than by session. Grouping by session usually outputs too much data for me, so I've actually developed a habit of including "-g request" in my varnishlog commandline. And when you are behind at TLS-terminator, that multiplexes requests across multiple connections to Varnish, grouping by session becomes mostly useless, so grouping by requests by default makes more sense to me. While on the topic of changing defaults, I think we should include Surrogate-Control and ESI parsing in to the builtin VCL as discussed yesterday. BR Kristian S?rensen From guillaume at varnish-software.com Fri Dec 4 09:20:39 2015 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Fri, 4 Dec 2015 10:20:39 +0100 Subject: H/2 VTC draft Message-ID: Hi all, One step toward H2/ in varnish is having a way to test it. So, here's a draft proposal for the H/2 handling in varnishtest: https://github.com/gquintard/h2vtcspec/blob/master/h2_vtc.rst You'll note that servers are touched, and while varnish may not use H/2 backends for some time, it'll definitely be useful to test varnishtest itself. Some things are definitely not sorted out (eg. headers compression) and of course, opinions are welcomed. -- Guillaume Quintard -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at uplex.de Fri Dec 4 15:19:41 2015 From: geoff at uplex.de (Geoff Simmons) Date: Fri, 04 Dec 2015 16:19:41 +0100 Subject: [PATCH] director VMODs use a rwlock for mutexing, add_backend() uses a wrlock, all other methods use rdlock Message-ID: <5661AF0D.6080708@uplex.de> Hello all, As agreed at VDD today. Best, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-director-VMODs-use-a-rwlock-for-mutexing-add_backend.patch Type: text/x-patch Size: 4670 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From slink at schokola.de Fri Dec 4 16:00:06 2015 From: slink at schokola.de (Nils Goroll) Date: Fri, 04 Dec 2015 17:00:06 +0100 Subject: VCC backend functions should need busyobject argument Message-ID: <5661B886.1020107@schokola.de> Just a quick note on something I noticed working with Dag on vmod_saintmode in the context of other directors (I am dropping it knowing that this is not particularly precise just to not loose it): The director VMOD backend functions can return their own object, in which case the resolve function will choose the "real" backend when an actual backend request is about to be issued. Alternatively, they can resolve (choose a backend) immediately and return a reference to a "real" backend object (or the next level down). For the latter way, there is no struct busyobj passed down, and we probably need to change that - otherwise stacking directors with vmod_saintmode won't work. Nils From phk at phk.freebsd.dk Sat Dec 5 17:34:35 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 05 Dec 2015 17:34:35 +0000 Subject: VCC backend functions should need busyobject argument In-Reply-To: <5661B886.1020107@schokola.de> References: <5661B886.1020107@schokola.de> Message-ID: <65419.1449336875@critter.freebsd.dk> -------- In message <5661B886.1020107 at schokola.de>, Nils Goroll writes: >The director VMOD backend functions can return their own object, in which case >the resolve function will choose the "real" backend when an actual backend >request is about to be issued. Alternatively, they can resolve (choose a >backend) immediately and return a reference to a "real" backend object (or the >next level down). > >For the latter way, there is no struct busyobj passed down, and >we probably need to change that - otherwise stacking directors >with vmod_saintmode won't work. Uhm... both dir->resolve() and dir->gethdrs() take a BO argument: typedef const struct director *vdi_resolve_f(const struct director *, struct worker *, struct busyobj *); typedef int vdi_gethdrs_f(const struct director *, struct worker *, struct busyobj *); -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From geoff at uplex.de Sun Dec 6 14:01:28 2015 From: geoff at uplex.de (Geoff Simmons) Date: Sun, 6 Dec 2015 15:01:28 +0100 Subject: [PATCH] add a remove_backend() method to each of the directors in the VMOD Message-ID: <56643FB8.1070302@uplex.de> Hello all, As agreed at VDD. This patch requires the previous one that I sent to varnish-dev (using rwlocks in the directors VMOD). Best, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-add-a-remove_backend-method-to-each-of-the-directors.patch Type: text/x-patch Size: 9976 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From ksorensen at nordija.com Sun Dec 6 19:52:07 2015 From: ksorensen at nordija.com (=?UTF-8?Q?Kristian_Gr=C3=B8nfeldt_S=C3=B8rensen?=) Date: Sun, 6 Dec 2015 20:52:07 +0100 Subject: Varnish 5.0 changes to defaults In-Reply-To: <1449218543.12225.58.camel@nordija.com> References: <1449218543.12225.58.camel@nordija.com> Message-ID: On 4 December 2015 at 09:42, Kristian Gr?nfeldt S?rensen < ksorensen at nordija.com> wrote: > Hi, > > After we talked about changes to default values for 5.0 yesterday at > VUGX, I realised that I'd like varnishlog to group log entries by > request as a default, rather than by session. > Grouping by session usually outputs too much data for me, so I've > actually developed a habit of including "-g request" in my varnishlog > commandline. > And when you are behind at TLS-terminator, that multiplexes requests > across multiple connections to Varnish, grouping by session becomes > mostly useless, so grouping by requests by default makes more sense to > me. > I realise now, that the default is to group by by XVID rather than by session. I'd still like the default to be grouping by request rather than by XVID though, as grouping by XVID gives you either Req or BeReq, but not both - unless you want varnishlog to display all records (no "-q" specified). Even if you get both records, you still have to stitch BeReq and the Req together using the XVID of the Req in the "Begin" tag of the BeReq. I can't think of a case where i would prefer grouping by XVID over grouping by request. BR Kristian S?rensen -------------- next part -------------- An HTML attachment was scrubbed... URL: From ruben at varnish-software.com Mon Dec 7 13:30:20 2015 From: ruben at varnish-software.com (=?UTF-8?Q?Rub=C3=A9n_Romero?=) Date: Mon, 7 Dec 2015 14:30:20 +0100 Subject: Varnish 5.0 changes to defaults In-Reply-To: <1449218543.12225.58.camel@nordija.com> References: <1449218543.12225.58.camel@nordija.com> Message-ID: On Fri, Dec 4, 2015 at 9:42 AM, Kristian Gr?nfeldt S?rensen < ksorensen at nordija.com> wrote: > Hi, > [...] > While on the topic of changing defaults, I think we should include > Surrogate-Control and ESI parsing in to the builtin VCL as discussed > yesterday. > > This was discussed last year: https://www.varnish-cache.org/trac/wiki/VDD14Q3#Surrogate-Control So we now need to follow-up on that and the discussions during VUGX last week. > BR > Kristian S?rensen > All the best, -- *Rub?n Romero* Varnish Software Group Cell: +47 95964088 / Office: +47 21989260 Skype, Twitter & IRC: ruben_varnish We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at uplex.de Mon Dec 7 14:05:13 2015 From: geoff at uplex.de (Geoff Simmons) Date: Mon, 7 Dec 2015 15:05:13 +0100 Subject: Varnish 5.0 changes to defaults In-Reply-To: References: <1449218543.12225.58.camel@nordija.com> Message-ID: <56659219.7080201@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 12/07/2015 02:30 PM, Rub?n Romero wrote: > > While on the topic of changing defaults, I think we should include > Surrogate-Control and ESI parsing in to the builtin VCL as > discussed yesterday. > > This was discussed last year: > https://www.varnish-cache.org/trac/wiki/VDD14Q3#Surrogate-Control > > So we now need to follow-up on that and the discussions during VUGX > last week. The conversations we've had in the past were more about using Surrogate-Control to set TTLs that are targeted to the proxy, but not for downstream caches. I believe we never got further than "yes we should have that, and a VMOD should do it" -- partially my fault, I should add in fairness. I didn't want to get nitpicky about this at VUG, but there's a bit of a problem with Varnish adding Surrogate-Control:ESI/1.0 to the backend request header (whether by default or not), since Varnish does not in fact support the full 1.0 ESI spec -- only the include and remove tags. If the idea is that backends are informed that they can use ESI in their responses, either for Varnish or, say, a CDN that might use another proxy, "ESI/1.0" would suggest that the backends can use ESI tags that Varnish would ignore. If we really want to do this, the tag might have to be something like "ESI/Varnish" or "ESI/include&remove" or "VESI/x.y". (There's a precedent for the latter -- Oracle Web Cache uses "ORAESI/x.y.z".) Of course, by the time we get to 5.0, fetch processors might be far enough along that the full ESI spec could be supported (but if so, I suspect that it will be left to a VMOD, which of course could also take care of generating Surrogate-Control headers). Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCAAGBQJWZZIZAAoJEOUwvh9pJNURZr4P/0ylF3moKWHaDPD89vY/7dDp 8kKFXs2lqO2z3HD3sqMWn2JejvTbG3abef/0BwKcPnv9LN0ycSe/yYOOWB/FaeIw hJ4XprMJXJcTfGEg6RaJW5EjytGA2+kFNkUm173ms4Y6Z2LA1T9fXLLC3N6clzAm 37oGPCxo2Gud8dLITOVWsCvav0Xk1q9oa+ah6ZKESKgj9w29im5uY6XuIFksbLdG Y5TZTsk17BWSLFAiXFqNxpQFyRw/YHHR+y/nlE2wsrYj7B3mXX4nlVLnGzB6dGsA nZ3XwWohTGia7s+UEKXCEI/eCBlm0wyWyzLgvzuHph4NzM7pVYHSs77Dz2EVZLd9 yWRV0D2ffnKlicHaWBznMhrNMV1igtF+gWSfaFwfLzFd/lusaEfY2CNMgO/0KZn6 sdJ9YWRtLjYAEIOc+5B3NACbSPSvj8SDlS+a4bBm01eZ6DK1rdgQMehOL46Wqq/1 A+qznlnVSL+DAWlS5Wlb8fyWV78+QZ4XK/Y4FVmWm7AxgoL6kXzv7bvNo4oQgyXU s/2iVjld/buE4WgpmNNWA2O7IdeQYUXOFhBgre/nMpHU2UHJtyz3H6pkKKaV6EGo 6b/6v0/SMn+5h/1ORk/3fX2cdo681m9+y7Kf9ne8PwGJwVCsBsnNLTCjUwdAXZPT 4ebwFjPAdNXswSkq327m =sAc4 -----END PGP SIGNATURE----- From phk at phk.freebsd.dk Mon Dec 7 14:16:20 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 07 Dec 2015 14:16:20 +0000 Subject: Varnish 5.0 changes to defaults In-Reply-To: <56659219.7080201@uplex.de> References: <1449218543.12225.58.camel@nordija.com> <56659219.7080201@uplex.de> Message-ID: <66643.1449497780@critter.freebsd.dk> -------- In message <56659219.7080201 at uplex.de>, Geoff Simmons writes: >I didn't want to get nitpicky about this at VUG, but there's a bit of >a problem with Varnish adding Surrogate-Control:ESI/1.0 to the backend >request header (whether by default or not), since Varnish does not in >fact support the full 1.0 ESI spec -- only the include and remove >tags. Yes, I was wondering about that issue too. (We also support esi comments, which is really just another kind of removes) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From martin at varnish-software.com Mon Dec 7 14:33:15 2015 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 7 Dec 2015 15:33:15 +0100 Subject: Varnish 5.0 changes to defaults In-Reply-To: <1449218543.12225.58.camel@nordija.com> References: <1449218543.12225.58.camel@nordija.com> Message-ID: On 4 December 2015 at 09:42, Kristian Gr?nfeldt S?rensen < ksorensen at nordija.com> wrote: > Hi, > > After we talked about changes to default values for 5.0 yesterday at > VUGX, I realised that I'd like varnishlog to group log entries by > request as a default, rather than by session. > Grouping by session usually outputs too much data for me, so I've > actually developed a habit of including "-g request" in my varnishlog > commandline. > And when you are behind at TLS-terminator, that multiplexes requests > across multiple connections to Varnish, grouping by session becomes > mostly useless, so grouping by requests by default makes more sense to > me. I believe my reasoning for choosing VXID as the default grouping comes from choosing the one that would cost the least. Having the -g request will keep more structure, and if the requests are very long lived have the potential of buffering more log data. so I thought that I'd be better to force that as a conscious choice rather than be the default. I'm not sure that argument is very strong though. Most requests are short lived and even the normal miss with one backend request wouldn't cost any more internal structure kept. So it'd probably be just fine to change the default. And the -g request is probably a better default. The leaves the question if we want to subject our users to a default change though... Martin -- *Martin Blix Grydeland* Senior Developer | Varnish Software AS Mobile: +47 992 74 756 We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at uplex.de Mon Dec 7 15:11:20 2015 From: geoff at uplex.de (Geoff Simmons) Date: Mon, 7 Dec 2015 16:11:20 +0100 Subject: Varnish 5.0 changes to defaults In-Reply-To: References: <1449218543.12225.58.camel@nordija.com> Message-ID: <5665A198.5020106@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 12/07/2015 03:33 PM, Martin Blix Grydeland wrote: > > The leaves the question if we want to subject our users to a > default change though... Unless there is a larger hue and cry among users for a new default, I don't yet see compelling reasons to introduce an incompatible change, especially since typing "-g request" isn't really that demanding. Kristian wrote that he can't imagine preferring -g vxid, which is fine, but I for one am much happier with that default. The different preferences might be due to differences between the sites we run -- in my case, the developers love ESI includes, so a single original request with -g request can easily produce dozens of subrequests and backend requests. Sometimes that's the view of the log that helps, but most of the time it's too much information, and the VXID grouping better focuses what I need to see. Since we've managed to teach developers to make a note of the XID (well some of them), I can often jump right to the right place. Those are two anecdotes, which of course do not constitute data. If we can determine that most people like -g request better, I'm willing to go with the flow and take on the burden of typing "-g vxid". But we should find that out, rather than assuming that -g request is always the obviously better choice -- for me at least, it isn't. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCAAGBQJWZaGYAAoJEOUwvh9pJNURjH0P/27ili+4/vJPtxtLwJ8+oPIN mUPTKxk7JcvqlHREFQDpB7h1mgAYsaOgu9TRfnUFfhXZb5aWx2mbaDhKZFE2Ka1M OPmLG1MSAK2Lgr0OQG+tJIrE3v7lGLJOmoQzCMrWtaS09wKU2HSqTXJzumU8PSur oLCVxkJYqBRQpq/LRJu3GoH0GR9vm5Kwb4ENsgB8Hqw64nBXzPdtuioMRonZ5gPl G+nhzIeknIQhmN/LUPj1Btn8yY6kWwtk3+7WnGKePohFYFyqx77Fl/2+rxkX+p4x IrAxwtOlZ7o6gf4LGRCiUrw9uSacTyc3GTRd3QonCQS1tMepd3e+LmGLHVuYaLFX BPDoPyTJZF1bILmXHzwPOZ2s9EpIi3s1cy51abNSGf7KLbqbHvl/W3Rt4xtx3PiR wZlqJ3+oH8MJ710O3QVdlDlnqBLlwI4ANVfl7LHz1P+1fG4/hICcB1uqrieqouKL 7OygkKYB9FPB2bwpytp1xfv7slMxBr/vzhKVifvg1rGIoThabXM9eJ57ok+d7lDb lSaXtupSD77ucJ5pWh8dzaLfI7K025sUUk/eb2A1mV/giITInFSDcUQxQ+hqD9oX +qi3D1B0lF1R9OI68kr4ihJM7C4pWaldhDFlbV5RisJkzCwR/wQPMiSSCpBZ9HN1 8f/btZZdMrunMGfg3xTa =uvMY -----END PGP SIGNATURE----- From daghf at varnish-software.com Mon Dec 7 15:23:15 2015 From: daghf at varnish-software.com (Dag Haavi Finstad) Date: Mon, 7 Dec 2015 16:23:15 +0100 Subject: VCC backend functions should need busyobject argument In-Reply-To: <65419.1449336875@critter.freebsd.dk> References: <5661B886.1020107@schokola.de> <65419.1449336875@critter.freebsd.dk> Message-ID: On Sat, Dec 5, 2015 at 6:34 PM, Poul-Henning Kamp wrote: > > > Uhm... both dir->resolve() and dir->gethdrs() take a BO argument: > > typedef const struct director *vdi_resolve_f(const struct director *, > struct worker *, struct busyobj *); > > typedef int vdi_gethdrs_f(const struct director *, struct worker *, > struct busyobj *); Hi The problem is rather that vmod_directors passes NULL for the bo when invoking the vdi_healthy_f function in vdir_pick_be. I opened a PR for a fix, see https://github.com/varnish/Varnish-Cache/pull/77 -- Dag Haavi Finstad Software Developer | Varnish Software Mobile: +47 476 64 134 We Make Websites Fly! From dridi at varni.sh Mon Dec 7 20:03:46 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 7 Dec 2015 21:03:46 +0100 Subject: [PATCH] VCL temperature improvements Message-ID: Hi, This is hopefully the closure of work I started during VDD15Q3, with a design refined during VDD15Q4. Some of the post-Q3 patches have already been pushed to the master branch, and the remaining have been split and should be easier to review. If a release manager happens to look at this mail, here is a list of commits that would need to be back-ported to 4.1 (I can do that if asked) along with the attached patches: * 513aee7 Add a reminder for a VRT_MINOR_VERSION bump * ae5a342 Polish * 6a6358f Polish * 89ce06a Wording * 502a871 Coverage for the cooling state * 8bf2b0a Hold a VCL reference in the debug vmod * 28eec51 Make event handling more readable in vmod-debug * 57a5b9c Introduce a new "busy" VCL temperature * 2b6bf4f Start documenting that VCL cool down should go fast * 3162873 Don't create backends on cooling VCLs, return NULL * 50aeefc Allow VMODs to hold a reference on a warm VCL * 16b8375 Clarify VRT_delete_backend a bit * c503bb7 Avoid connection attempts to ${bad_ip}:80 * efd5dfd VMODs handling of VCL_EVENT_COLD must be failsafe * fbd1c18 Document that VCL_EVENT_USE is deprecated * 445854c Reserve space for the "cooling" state in vcl.list I'm not sure when I can join on IRC to discuss those patches, but I tried to convey as much information as I could in the commit messages. If the patches are merged, let me know and I'll write changelog entries. Best, Dridi -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Turn-VCL-state-magic-numbers-into-an-enum.patch Type: text/x-patch Size: 4101 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-Make-vcl_set_state-accept-a-ctx-instead-of-a-vcl.patch Type: text/x-patch Size: 3457 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0003-Allow-VMODs-to-fail-a-warm-up.patch Type: text/x-patch Size: 5911 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0004-Make-VMODs-actually-fail-warm-ups.patch Type: text/x-patch Size: 7834 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0005-Catch-a-vcl.state-failure-on-the-manager-side.patch Type: text/x-patch Size: 6732 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0006-Simplify-vcl.use-by-making-it-failsafe.patch Type: text/x-patch Size: 2603 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0007-Replace-the-VCL-refcount-by-a-self-desribing-list.patch Type: text/x-patch Size: 10963 bytes Desc: not available URL: From ksorensen at nordija.com Mon Dec 7 21:40:51 2015 From: ksorensen at nordija.com (=?UTF-8?Q?Kristian_Gr=C3=B8nfeldt_S=C3=B8rensen?=) Date: Mon, 7 Dec 2015 22:40:51 +0100 Subject: Varnish 5.0 changes to defaults In-Reply-To: <56659219.7080201@uplex.de> References: <1449218543.12225.58.camel@nordija.com> <56659219.7080201@uplex.de> Message-ID: On 7 December 2015 at 15:05, Geoff Simmons wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > On 12/07/2015 02:30 PM, Rub?n Romero wrote: > > > > While on the topic of changing defaults, I think we should include > > Surrogate-Control and ESI parsing in to the builtin VCL as > > discussed yesterday. > > > > This was discussed last year: > > https://www.varnish-cache.org/trac/wiki/VDD14Q3#Surrogate-Control > > > > So we now need to follow-up on that and the discussions during VUGX > > last week. > > The conversations we've had in the past were more about using > Surrogate-Control to set TTLs that are targeted to the proxy, but not > for downstream caches. I believe we never got further than "yes we > should have that, and a VMOD should do it" -- partially my fault, I > should add in fairness. > > I didn't want to get nitpicky about this at VUG, but there's a bit of > a problem with Varnish adding Surrogate-Control:ESI/1.0 to the backend > request header (whether by default or not), since Varnish does not in > fact support the full 1.0 ESI spec -- only the include and remove > tags. If the idea is that backends are informed that they can use ESI > in their responses, either for Varnish or, say, a CDN that might use > another proxy, "ESI/1.0" would suggest that the backends can use ESI > tags that Varnish would ignore. > > I forgot that Varnish does not support the full ESI spec. I don't think Varnish should signal ESI/1.0 as long as it doesn't support the full spec. That could give some people an unpleasant surprise. > If we really want to do this, the tag might have to be something like > "ESI/Varnish" or "ESI/include&remove" or "VESI/x.y". (There's a > precedent for the latter -- Oracle Web Cache uses "ORAESI/x.y.z".) > I would prefer if Varnish somehow advertised it's ESI capabilities by default. I don't really have an opinion on how exactly it should be done as long as we don't risk to break anything that correctly implements the full ESI spec . Any of your 3 suggestions would work for me personally. /Kristian -------------- next part -------------- An HTML attachment was scrubbed... URL: From kacperw at gmail.com Mon Dec 7 23:32:15 2015 From: kacperw at gmail.com (Kacper Wysocki) Date: Tue, 8 Dec 2015 00:32:15 +0100 Subject: Varnish 5.0 changes to defaults In-Reply-To: References: <1449218543.12225.58.camel@nordija.com> <56659219.7080201@uplex.de> Message-ID: On Mon, Dec 7, 2015 at 10:40 PM, Kristian Gr?nfeldt S?rensen wrote: > On 7 December 2015 at 15:05, Geoff Simmons wrote: >> On 12/07/2015 02:30 PM, Rub?n Romero wrote: >> > >> > While on the topic of changing defaults, I think we should include >> > Surrogate-Control and ESI parsing in to the builtin VCL as >> > discussed yesterday. [snip] >> If we really want to do this, the tag might have to be something like >> "ESI/Varnish" or "ESI/include&remove" or "VESI/x.y". (There's a >> precedent for the latter -- Oracle Web Cache uses "ORAESI/x.y.z".) > > > I would prefer if Varnish somehow advertised it's ESI capabilities by > default. I don't really have an opinion on how exactly it should be done as > long as we don't risk to break anything that correctly implements the full > ESI spec . Any of your 3 suggestions would work for me personally. Yes to this. It's a nice approach and might get more devs to use ESI when it doesn't involve starting a whole ITIL supply chain to get ESI enabled. -- http://comotion.delta9.pl http://u.delta9.pl http://kacper.doesntexist.org Too much order is its own chaos. Employ no technique to gain supreme enlightment. From fgsch at lodoss.net Thu Dec 10 10:23:39 2015 From: fgsch at lodoss.net (Federico Schwindt) Date: Thu, 10 Dec 2015 10:23:39 +0000 Subject: [PATCH] Use a weak comparison function for If-None-Match Message-ID: As per RFC. Reorganise some of the tests to avoid duplicates. Fixes #1816. OK? Comments? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Use-a-weak-comparison-function-for-If-None-Match.patch Type: text/x-diff Size: 5844 bytes Desc: not available URL: From dridi at varni.sh Thu Dec 10 11:48:46 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 10 Dec 2015 12:48:46 +0100 Subject: [PATCH] VCL temperature improvements In-Reply-To: References: Message-ID: On Mon, Dec 7, 2015 at 9:03 PM, Dridi Boukelmoune wrote: > Hi, > > This is hopefully the closure of work I started during VDD15Q3, with a > design refined during VDD15Q4. Some of the post-Q3 patches have > already been pushed to the master branch, and the remaining have been > split and should be easier to review. Hi, Here is a new batch of patches fix changes requested from a first review. There's one more commit this time, it's number 3. Best, Dridi -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Turn-VCL-state-magic-numbers-into-an-enum.patch Type: text/x-patch Size: 4101 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-Make-vcl_set_state-accept-a-ctx-instead-of-a-vcl.patch Type: text/x-patch Size: 3457 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0003-Wrap-VCL-event-calls-in-a-dedicated-function.patch Type: text/x-patch Size: 3220 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0004-Allow-VMODs-to-fail-a-warm-up.patch Type: text/x-patch Size: 5668 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0005-Make-VMODs-actually-fail-warm-ups.patch Type: text/x-patch Size: 7834 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0006-Catch-a-vcl.state-failure-on-the-manager-side.patch Type: text/x-patch Size: 6650 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0007-Simplify-vcl.use-by-making-it-failsafe.patch Type: text/x-patch Size: 3115 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0008-Replace-the-VCL-refcount-by-a-self-desribing-list.patch Type: text/x-patch Size: 10958 bytes Desc: not available URL: From dridi at varni.sh Thu Dec 10 15:30:42 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 10 Dec 2015 16:30:42 +0100 Subject: [PATCH] VCL temperature improvements In-Reply-To: References: Message-ID: > Hi, > > Here is a new batch of patches fix changes requested from a first > review. There's one more commit this time, it's number 3. One more update for the first review, I forgot to update patch number 1. -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Turn-VCL-state-magic-numbers-into-an-enum.patch Type: text/x-patch Size: 4251 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-Make-vcl_set_state-accept-a-ctx-instead-of-a-vcl.patch Type: text/x-patch Size: 3457 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0003-Wrap-VCL-event-calls-in-a-dedicated-function.patch Type: text/x-patch Size: 3220 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0004-Allow-VMODs-to-fail-a-warm-up.patch Type: text/x-patch Size: 5668 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0005-Make-VMODs-actually-fail-warm-ups.patch Type: text/x-patch Size: 7834 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0006-Catch-a-vcl.state-failure-on-the-manager-side.patch Type: text/x-patch Size: 6642 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0007-Simplify-vcl.use-by-making-it-failsafe.patch Type: text/x-patch Size: 3115 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0008-Replace-the-VCL-refcount-by-a-self-desribing-list.patch Type: text/x-patch Size: 10958 bytes Desc: not available URL: From fgsch at lodoss.net Sat Dec 12 13:48:29 2015 From: fgsch at lodoss.net (Federico Schwindt) Date: Sat, 12 Dec 2015 13:48:29 +0000 Subject: [PATCH] Ignore If-Modified-Since if we have If-None-Match Message-ID: The latter takes precedence as per RFC 7232. Also if we have an I-N-M but no entity don't check I-M-S, just do a non-conditional request. Comments? OKs? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Ignore-If-Modified-Since-if-we-have-If-None-Match.patch Type: text/x-diff Size: 4228 bytes Desc: not available URL: From phk at phk.freebsd.dk Sat Dec 12 17:26:26 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 12 Dec 2015 17:26:26 +0000 Subject: [PATCH] Ignore If-Modified-Since if we have If-None-Match In-Reply-To: References: Message-ID: <21411.1449941186@critter.freebsd.dk> -------- In message , Federico Schwindt writes: >The latter takes precedence as per RFC 7232. Also if we have an I-N-M >but no entity don't check I-M-S, just do a non-conditional request. > >Comments? OKs? OK with me. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From fgsch at lodoss.net Tue Dec 15 15:03:47 2015 From: fgsch at lodoss.net (Federico Schwindt) Date: Tue, 15 Dec 2015 15:03:47 +0000 Subject: [PATCH] VSB_bcat: Don't do a single char at the time Message-ID: Hi, VSB_bcat calls VSB_put_byte for each char that needs to append. Since we know the length it's better to extend the buffer if needed and just memcpy the whole input. Comments? OK? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: vsb.patch Type: text/x-diff Size: 716 bytes Desc: not available URL: From phk at phk.freebsd.dk Tue Dec 15 15:10:00 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 15 Dec 2015 15:10:00 +0000 Subject: [PATCH] VSB_bcat: Don't do a single char at the time In-Reply-To: References: Message-ID: <52668.1450192200@critter.freebsd.dk> -------- In message , Federico Schwindt writes: >VSB_bcat calls VSB_put_byte for each char that needs to append. >Since we know the length it's better to extend the buffer if needed and >just memcpy the whole input. > >Comments? OK? OK. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From daghf at varnish-software.com Tue Dec 22 14:46:36 2015 From: daghf at varnish-software.com (Dag Haavi Finstad) Date: Tue, 22 Dec 2015 15:46:36 +0100 Subject: IPC semaphores in varnishtest Message-ID: Hi guys At the latest dev meeting in Rotterdam it was brought up that it would be a useful addition to be able to sync a varnishtest semaphore with Varnish itself. The current 'sema' construct in varnishtest lives within the same process and is implemented using regular pthread mutexes and condvars. I've hacked together a suggestion for doing IPC semaphores in varnishtest/Varnish, backed by SysV semaphores. The patches introduce a new varnishtest keyword 'ipcsema' and a new function in vmod-debug, 'debug.sema_sync()'. 'ipcsema' in varnishtest follows the syntax and semantics of the usual 'sema' keyword (with an exception described below [1]), ipcsema r1 sync 2 It can be used in both client/server and global scopes in varnishtest, just like the regular 'sema' construct. Further, to sync with this semaphore from VCL, we issue sub vcl_backend_fetch { debug.sema_sync("r1", 2); } How this works under the covers is that varnishtest will on each run generate a random number and stick that in an environment variable ($VTC_IPC_RAND) prior to launching varnishd. This string, along with the semaphore name (e.g. "r1") goes into a hash that is used as the key argument to semget(2). After the test case finishes, the semaphore will be removed (however, see [2] below). There are a couple of open issues: [1] I have yet to find a non-racy way to reset an ipcsema back to its original value after the wait is over. This is something that is supported by the regular sema construct in varnishtest, that allows a sema to be reused multiple times within the same test case. (See for example c00013.vtc). [2] Potential for leaking a semaphore. If the varnishtest process is killed and thus the cleanup code isn't called the semaphore is leaked. It will then have to be cleaned up via ipcrm(1). Active system semaphores can be shown via ipcs(1). Please take a look - any ideas or feedback will be most welcome. Patches attached. Merry X-mas, Dag -- Dag Haavi Finstad Software Developer | Varnish Software Mobile: +47 476 64 134 We Make Websites Fly! -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Prototype-varnishtest-IPC-semaphores.patch Type: text/x-diff Size: 10035 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-Add-.sema_sync-to-libvmod-debug.patch Type: text/x-diff Size: 5576 bytes Desc: not available URL: From dridi at varni.sh Tue Dec 22 16:13:03 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 22 Dec 2015 17:13:03 +0100 Subject: IPC semaphores in varnishtest In-Reply-To: References: Message-ID: On Tue, Dec 22, 2015 at 3:46 PM, Dag Haavi Finstad wrote: > Hi guys > > At the latest dev meeting in Rotterdam it was brought up that it would > be a useful addition to be able to sync a varnishtest semaphore with > Varnish itself. > > snip > > Please take a look - any ideas or feedback will be most welcome. Hi Dag, I sent a similar patch set off-list last night and we had offline discussions on IPC sema in varnishtest. The consensus is to retire semaphores in their current state and introduce non-cyclic barriers that would need to be explicitly initialized and would work in either local (cond) or shared (socket) mode. The syntax will look like this: > # initialization > barrier b1 cond 2 > barrier b2 socket 2 > > # usage in varnishtest, you don't "sync N" anymore > barrier b1 sync > > # usage in VCL > debug.barrier_sync("${b2_sock}"); I'll send patches to the list, that's unfortunate that we worked on the same thing around the same time :( Cheers, Dridi From daghf at varnish-software.com Tue Dec 22 17:06:13 2015 From: daghf at varnish-software.com (Dag Haavi Finstad) Date: Tue, 22 Dec 2015 18:06:13 +0100 Subject: IPC semaphores in varnishtest In-Reply-To: References: Message-ID: Hey Dridi, That looks pretty cool. Having a separate initialisation step and doing the syncing via a socket sounds like a cleaner approach than what I did. I didn't spend all that much time on my ipcsema thing, so not to worry. :-) -Dag On Tue, Dec 22, 2015 at 5:13 PM, Dridi Boukelmoune wrote: > On Tue, Dec 22, 2015 at 3:46 PM, Dag Haavi Finstad > wrote: >> Hi guys >> >> At the latest dev meeting in Rotterdam it was brought up that it would >> be a useful addition to be able to sync a varnishtest semaphore with >> Varnish itself. >> >> snip >> >> Please take a look - any ideas or feedback will be most welcome. > > Hi Dag, > > I sent a similar patch set off-list last night and we had offline > discussions on IPC sema in varnishtest. The consensus is to retire > semaphores in their current state and introduce non-cyclic barriers > that would need to be explicitly initialized and would work in either > local (cond) or shared (socket) mode. > > The syntax will look like this: > >> # initialization >> barrier b1 cond 2 >> barrier b2 socket 2 >> >> # usage in varnishtest, you don't "sync N" anymore >> barrier b1 sync >> >> # usage in VCL >> debug.barrier_sync("${b2_sock}"); > > I'll send patches to the list, that's unfortunate that we worked on > the same thing around the same time :( > > Cheers, > Dridi -- Dag Haavi Finstad Software Developer | Varnish Software Mobile: +47 476 64 134 We Make Websites Fly! From dridi at varni.sh Thu Dec 24 09:56:11 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 24 Dec 2015 10:56:11 +0100 Subject: [PATCH] Teach vmod-debug how to sync with varnishtest Message-ID: Hi, As discussed in Rotterdam, and in Dag's thread on IPC semaphores, we need to sync varnishtest with varnishd. This patch set retires varnishtest semaphores and introduce barriers instead. The main difference is that barriers are not cyclic by default, and separate the initialization from actual syncing. I kept Dag's test case, it was more interesting than mine. The syntax looks like this: > # initialization > barrier b1 cond 2 > barrier b2 sock 2 > > # usage in varnishtest, you don't "sync N" anymore > barrier b1 sync > > # usage in VCL > debug.barrier_sync("${b2_sock}"); I see two choices I made that could be controversial: - a sock barrier doesn't keep track of incoming connections - debug.barrier_sync won't work in vcl_(init|fini) {} Thoughts? Happy syncing, Dridi -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Introduce-barriers-in-varnishtest.patch Type: text/x-patch Size: 10226 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-Barriers-can-be-cyclic-but-not-like-semaphores.patch Type: text/x-patch Size: 3635 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0003-Retire-varnishtest-semaphores-and-use-barriers.patch Type: text/x-patch Size: 37832 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0004-Barriers-can-only-be-created-on-the-top-thread.patch Type: text/x-patch Size: 928 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0005-Implement-socket-barriers-shared-between-processes.patch Type: text/x-patch Size: 7965 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0006-Teach-vmod-debug-how-to-sync-with-varnishtest.patch Type: text/x-patch Size: 3008 bytes Desc: not available URL: From dridi at varni.sh Thu Dec 24 11:48:41 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 24 Dec 2015 12:48:41 +0100 Subject: [PATCH] Teach vmod-debug how to sync with varnishtest In-Reply-To: References: Message-ID: > Thoughts? Do you know this feeling when you press the start button but you forgot something? Then your brain starts a background thread and moment later (like in the shower or during lunch) you realize you forgot to put some of the laundry in the washing machine... Cheers, Dridi -------------- next part -------------- A non-text attachment was scrubbed... Name: 0007-Close-the-socket.patch Type: text/x-patch Size: 1138 bytes Desc: not available URL: From fgsch at lodoss.net Tue Dec 29 01:33:20 2015 From: fgsch at lodoss.net (Federico Schwindt) Date: Tue, 29 Dec 2015 01:33:20 +0000 Subject: [PATCH] Use realloc if possible Message-ID: Avoids memcpy/free if the buffer was dynamic. Comments? OK? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Use-realloc-if-possible.patch Type: text/x-diff Size: 1069 bytes Desc: not available URL: