From slink at schokola.de Tue Mar 1 17:26:08 2016 From: slink at schokola.de (Nils Goroll) Date: Tue, 1 Mar 2016 18:26:08 +0100 Subject: [PATCH] improve the vbm facility and fix an off-by-one error Message-ID: <56D5D0B0.1010606@schokola.de> A non-text attachment was scrubbed... Name: 0001-improve-the-vbm-facility-and-fix-an-off-by-one-error.patch Type: text/x-patch Size: 6892 bytes Desc: not available URL: From guillaume at varnish-software.com Wed Mar 2 08:51:33 2016 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Wed, 2 Mar 2016 09:51:33 +0100 Subject: [PATCH] improve the vbm facility and fix an off-by-one error In-Reply-To: <56D5D0B0.1010606@schokola.de> References: <56D5D0B0.1010606@schokola.de> Message-ID: Can we get rid of the "if ENABLE_TESTS"? Tests should be executed. If you don't want to run them, don't run "make check". -- Guillaume Quintard -------------- next part -------------- An HTML attachment was scrubbed... URL: From slink at schokola.de Wed Mar 2 10:47:10 2016 From: slink at schokola.de (Nils Goroll) Date: Wed, 2 Mar 2016 11:47:10 +0100 Subject: make ENABLE_TESTS default? In-Reply-To: References: <56D5D0B0.1010606@schokola.de> Message-ID: <56D6C4AE.9020409@schokola.de> On 02/03/16 09:51, Guillaume Quintard wrote: > Can we get rid of the "if ENABLE_TESTS"? Tests should be executed. If you don't > want to run them, don't run "make check". This seems to be an ancient addition, I cant' see any use for disabling tests and I agree that tests should just be enabled by default. commit 35b895fd01ee23c4a2889fa3d0fe04c2358baf80 Author: Dag Erling Sm?rgrav Date: Wed Feb 13 13:05:36 2008 +0000 Some source files (especially in libraries) have embedded test programs. Add a configure option and a corresponding automake conditional to enable these tests. git-svn-id: http://www.varnish-cache.org/svn/trunk/varnish-cache at 2453 d4fa192b-c00b-0410-8231-f00ffab90ce4 diff --git a/configure.ac b/configure.ac index 5197c1f..204124f 100644 --- a/configure.ac +++ b/configure.ac @@ -293,6 +293,11 @@ AC_ARG_ENABLE(stack-protector, AS_HELP_STRING([--enable-stack-protector],[enable stack protector (default is NO)]), CFLAGS="${CFLAGS} -fstack-protector-all") +# --enable-tests +AC_ARG_ENABLE(tests, + AS_HELP_STRING([--enable-tests],[build test programs (default is NO)])) +AM_CONDITIONAL([ENABLE_TESTS], [test x$enable_tests = xyes]) + # --enable-werror AC_ARG_ENABLE(werror, AS_HELP_STRING([--enable-werror],[use -Werror (default is NO)]), From guillaume at varnish-software.com Wed Mar 2 16:00:52 2016 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Wed, 2 Mar 2016 17:00:52 +0100 Subject: [PATCH] improve the vbm facility and fix an off-by-one error In-Reply-To: References: <56D5D0B0.1010606@schokola.de> Message-ID: Since I smelled a consensus, here are the patches to remove the "--enable-tests" -- Guillaume Quintard On Wed, Mar 2, 2016 at 9:51 AM, Guillaume Quintard < guillaume at varnish-software.com> wrote: > Can we get rid of the "if ENABLE_TESTS"? Tests should be executed. If you > don't want to run them, don't run "make check". > > -- > Guillaume Quintard > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-Get-rid-of-enable-tests.patch Type: text/x-patch Size: 819 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Always-enable-libvarnish-test.patch Type: text/x-patch Size: 782 bytes Desc: not available URL: From phk at phk.freebsd.dk Wed Mar 2 23:11:59 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 02 Mar 2016 23:11:59 +0000 Subject: [PATCH] improve the vbm facility and fix an off-by-one error In-Reply-To: References: <56D5D0B0.1010606@schokola.de> Message-ID: <84220.1456960319@critter.freebsd.dk> -------- In message , Guillaume Quintard writes: >Since I smelled a consensus, here are the patches to remove the >"--enable-tests" OK with me. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Wed Mar 2 23:26:56 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 02 Mar 2016 23:26:56 +0000 Subject: [PATCH] improve the vbm facility and fix an off-by-one error In-Reply-To: <56D5D0B0.1010606@schokola.de> References: <56D5D0B0.1010606@schokola.de> Message-ID: <88459.1456961216@critter.freebsd.dk> -------- In message <56D5D0B0.1010606 at schokola.de>, Nils Goroll writes: >- fix an off-by-one in vbit_set Please fix this in a stand-alone commit. >- add VBITMAP_INITIALIZE() for on-stack use of struct vbitmap I am *really* alergic to alloca(), so please make a case for here this is a good idea ? >- change the allocation scheme to pow2() I'm not sure I understand what you mean here ? >- add a test program Always a good idea. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Thu Mar 3 17:38:27 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 03 Mar 2016 17:38:27 +0000 Subject: Archiving Trac tickets (Calling HTML wizards) Message-ID: <23441.1457026707@critter.freebsd.dk> I've hacked up some scripts to archive the Trac tickets when we move bug reports to github. It's a rather crude hack consisting of fetching the HTML from Trac and munging it up with sed(1). I would appreciate if somebody with more HTML clue than me could take a peek and see if this is a viable way to archive the pages. You can access the tickets like: http://r.varnish-cache.org/trac/ticket/1029 Attachments are also archived the same way. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From lkarsten at varnish-software.com Fri Mar 4 13:30:40 2016 From: lkarsten at varnish-software.com (Lasse Karstensen) Date: Fri, 4 Mar 2016 14:30:40 +0100 Subject: Varnish modules package released (Was: Re: Update on vmods and packaging) In-Reply-To: <20151202144041.GB19950@immer.varnish-software.com> References: <20151202144041.GB19950@immer.varnish-software.com> Message-ID: <20160304133039.GA32277@immer.varnish-software.com> [crosspost to both -dev and -misc] On Wed, Dec 02, 2015 at 03:40:42PM +0100, Lasse Karstensen wrote: > Varnish Software (VS) is writing and maintaining a fair amount of vmods for > Varnish Cache. This includes major ones like vmod-header, which can be > said to be borderline core Varnish functionality. > We will be doing some changes on how these are maintained and distributed. [snip] > Further down the line it is likely that we will be combining the simple > vmods (with no third party/library dependencies) into a single > collection source package. Updates on that will come later. Hi all. We have now made an initial release of this collection. Tarball distribution is here: http://files.varnish-software.com/vmod/varnish-modules-0.9.0.tar.gz The source trees and distribution of the following vmods have been merged into this single project: cookie, vsthrottle, header, saintmode, softpurge, tcp, var, xkey. More vmods will be added at a later stage. This means that starting from today the authoritative location for development of these vmods is on: https://github.com/varnish/varnish-modules Releases of this tree will be a tarball with a proper changelog, suitable for packaging into rpm/deb/* packages by motivated third parties. It is my hope that this will significantly simplify the installation of these vmods, both for users installing from source, and for users installing pre-packaged versions from third-party yum/apt/* sources. 3.0 and 4.0 versions of the old vmods will be archived and made available in a similar manner. When that has been done, the old github projects for the vmods will be retired. -- Lasse Karstensen Varnish Software AS (VS hat) From hermunn at varnish-software.com Tue Mar 8 10:40:44 2016 From: hermunn at varnish-software.com (=?UTF-8?Q?P=C3=A5l_Hermunn_Johansen?=) Date: Tue, 8 Mar 2016 11:40:44 +0100 Subject: Patch: Use calloc instead of malloc when running out of VSM space (common_vsm.c) Message-ID: Hello, I just started working at Varnish Software (last Thursday), and I am looking forward to working with the Varnish Cache code. After firm guidance from Martin, I am proposing my first patch. Best Regards, P?l Hermunn Johansen -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Use-calloc-instead-of-malloc-to-allocate-extra-VSM-s.patch Type: text/x-patch Size: 1485 bytes Desc: not available URL: From fgsch at lodoss.net Tue Mar 8 12:15:45 2016 From: fgsch at lodoss.net (Federico Schwindt) Date: Tue, 8 Mar 2016 12:15:45 +0000 Subject: Patch: Use calloc instead of malloc when running out of VSM space (common_vsm.c) In-Reply-To: References: Message-ID: Incidentally, this also reminds me of something else I observed some time ago. We use calloc in many places, I do wonder how many of them do really need it. The downside of using calloc when is not really needed is that by zeroing the memory you end up with resident memory and not virtual, which in turn might lead to swapping. On Tue, Mar 8, 2016 at 10:40 AM, P?l Hermunn Johansen < hermunn at varnish-software.com> wrote: > Hello, > I just started working at Varnish Software (last Thursday), and I am > looking forward to working with the Varnish Cache code. After firm > guidance from Martin, I am proposing my first patch. > > Best Regards, > > P?l Hermunn Johansen > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Tue Mar 8 12:26:47 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 08 Mar 2016 12:26:47 +0000 Subject: Patch: Use calloc instead of malloc when running out of VSM space (common_vsm.c) In-Reply-To: References: Message-ID: <13219.1457440007@critter.freebsd.dk> -------- In message , Federico Schwindt writes: >We use calloc in many places, I do wonder how many of them do really need >it. The downside of using calloc when is not really needed is that by >zeroing the memory you end up with resident memory and not virtual, which >in turn might lead to swapping. This is almost always intentional, as we generally do not over-allocate. The exception is the malloc stevedore where we do. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dho at fastly.com Tue Mar 8 15:52:51 2016 From: dho at fastly.com (Devon H. O'Dell) Date: Tue, 08 Mar 2016 15:52:51 +0000 Subject: Patch: Use calloc instead of malloc when running out of VSM space (common_vsm.c) In-Reply-To: <13219.1457440007@critter.freebsd.dk> References: <13219.1457440007@critter.freebsd.dk> Message-ID: Depending on the implementation of the allocator used, calloc may not have any additional overhead. When it does, there are really only a couple cases it shouldn't happen on modern systems: * If an object is allocated and freed in a tight loop. This shouldn't be happening anyway -- reuse / pool objects with this sort of access pattern. * If the object is large. Malloced memory should not be immediately visible to multiple concurrent processes, and objects that consist of only a few cache lines cost very little to zero on modern processors. It may be worth auditing for these situations, but I've done extensive profiling of extremely memory heavy workloads (hundreds of gb) in Varnish over the past few years, and I promise that calloc is not a current limiting factor in terms of latency or throughput. Of course, if you're concerned about swapping, I'd also argue that your cache is not properly sized. On Tue, Mar 8, 2016, 04:28 Poul-Henning Kamp wrote: > -------- > In message 6N3H9jNxnrXM05ht2pQkyJ-FY-LGfu1H_g at mail.gmail.com> > , Federico Schwindt writes: > > >We use calloc in many places, I do wonder how many of them do really need > >it. The downside of using calloc when is not really needed is that by > >zeroing the memory you end up with resident memory and not virtual, which > >in turn might lead to swapping. > > This is almost always intentional, as we generally do not over-allocate. > > The exception is the malloc stevedore where we do. > > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fgsch at lodoss.net Tue Mar 8 20:22:01 2016 From: fgsch at lodoss.net (Federico Schwindt) Date: Tue, 8 Mar 2016 20:22:01 +0000 Subject: Patch: Use calloc instead of malloc when running out of VSM space (common_vsm.c) In-Reply-To: References: <13219.1457440007@critter.freebsd.dk> Message-ID: In the case this would be an issue my concerns are not directly related to performance nor latency. If the configuration, either the default or modified by the user, would be oversized there could be lot of waste, which might have other side effects (as swapping). I doubt this would the case for Fastly since you are likely on the high side but it could be an issue for small to medium setups. On 8 Mar 2016 3:53 p.m., "Devon H. O'Dell" wrote: > Depending on the implementation of the allocator used, calloc may not have > any additional overhead. When it does, there are really only a couple cases > it shouldn't happen on modern systems: > > * If an object is allocated and freed in a tight loop. This shouldn't be > happening anyway -- reuse / pool objects with this sort of access pattern. > * If the object is large. Malloced memory should not be immediately > visible to multiple concurrent processes, and objects that consist of only > a few cache lines cost very little to zero on modern processors. > > It may be worth auditing for these situations, but I've done extensive > profiling of extremely memory heavy workloads (hundreds of gb) in Varnish > over the past few years, and I promise that calloc is not a current > limiting factor in terms of latency or throughput. > > Of course, if you're concerned about swapping, I'd also argue that your > cache is not properly sized. > > On Tue, Mar 8, 2016, 04:28 Poul-Henning Kamp wrote: > >> -------- >> In message > 6N3H9jNxnrXM05ht2pQkyJ-FY-LGfu1H_g at mail.gmail.com> >> , Federico Schwindt writes: >> >> >We use calloc in many places, I do wonder how many of them do really need >> >it. The downside of using calloc when is not really needed is that by >> >zeroing the memory you end up with resident memory and not virtual, which >> >in turn might lead to swapping. >> >> This is almost always intentional, as we generally do not over-allocate. >> >> The exception is the malloc stevedore where we do. >> >> >> -- >> Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 >> phk at FreeBSD.ORG | TCP/IP since RFC 956 >> FreeBSD committer | BSD since 4.3-tahoe >> Never attribute to malice what can adequately be explained by >> incompetence. >> >> _______________________________________________ >> varnish-dev mailing list >> varnish-dev at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Tue Mar 8 21:38:29 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 08 Mar 2016 21:38:29 +0000 Subject: Varnish Project migration and mailing lists future Message-ID: <15457.1457473109@critter.freebsd.dk> As I have warned earlier, we are busy reevaluating all the Varnish Cache projects infrastructure. Starting tomorrow, wednesday march 9th 2016, noon-ish, our main source code repository will live on github: https://github.com/varnishcache/varnish-cache In the future bugreports and patches will happen through githubs facilities for issues and pull-requests. (We have taken a snapshot of the old tickets, so history is preserved.) The new homepage is in the works -- any day or possibly week now. Next up was migrating email and mailing-lists to the new server. As somebody who wrote a sendmail.cf from scratch back in 198x when Denmark got connected to the Internet, I was really not looking forward to having to deal with spam-filtering and all that crap. But like Mr. Prosser, I would argue: "It's email, you got to have email!" Ruben prompted me to take a closer, more critical look, and that was a good idea. As far as I can see, Stackoverflow.com has significantly more Varnish traffic than our own varnish-misc list. Judging from the archives, a lot of the current varnish-dev traffic move to github as part of the pull requests handling. So Ruben has a point: It might be time to lie down in front of the bulldozer. Here is a strawman proposal to down-size our email: -misc autoreply "Please use stackoverflow.com instead" -core alias Incoming contact point for project management. -security alias Incoming contact point for security issues. -comitters mailman, closed subscription Info-channel for sysadm like messages -announce. mailman, open subscription, moderated. For official project information only. -commit mailman, open subscription, no posting Robot sends one email per commit (Do we one such list for each repo ?) -dev mailman, open subscription Technical discussions, warnings, anouncements. I have no idea how much traffic -dev will see in the future, but I I want to keep it as a "technical list" which is open for two-way communication. -dev will be the only "sign up and spam" list we have, and given the usual low level of traffic, I propose we deal with that moderating the list and white-list people as we go. Comments, observations, ideas ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dho at fastly.com Tue Mar 8 23:03:01 2016 From: dho at fastly.com (Devon H. O'Dell) Date: Tue, 8 Mar 2016 15:03:01 -0800 Subject: Patch: Use calloc instead of malloc when running out of VSM space (common_vsm.c) In-Reply-To: References: <13219.1457440007@critter.freebsd.dk> Message-ID: On Tue, Mar 8, 2016 at 12:22 PM, Federico Schwindt wrote: > > In the case this would be an issue my concerns are not directly related to performance nor latency. > > If the configuration, either the default or modified by the user, would be oversized there could be lot of waste, which might have other side effects (as swapping). First of all, I respect your point. I just disagree that it is an argument against using calloc(3). To echo phk, Varnish doesn't tend to allocate memory it isn't going to need. Referring specifically to configurations, most VCL and VMODs do no allocations whatsoever (there are of course exceptions, but those are not really interesting) -- they obtain memory via workspace, which was preallocated specifically for that purpose, and is going to be resident anyway for a peak load. There are tunables to limit the number of preallocated things like sessions, workspace size, allocation size from the malloc storage for chunked and EOF bodies, and more. Varnish is actually very sane in this respect. Effectively this argument states that using calloc(3) is bad IFF you expect your users to overcommit on their system resources. If they do this, then they are unaware of relevant configuration tunables, they don't understand their workload, or both. That's a documentation / education / community issue. It shouldn't be an argument against using a specific standard C API that gives you memory filled with defined values. > On 8 Mar 2016 3:53 p.m., "Devon H. O'Dell" wrote: >> >> Depending on the implementation of the allocator used, calloc may not have any additional overhead. When it does, there are really only a couple cases it shouldn't happen on modern systems: >> >> * If an object is allocated and freed in a tight loop. This shouldn't be happening anyway -- reuse / pool objects with this sort of access pattern. >> * If the object is large. Malloced memory should not be immediately visible to multiple concurrent processes, and objects that consist of only a few cache lines cost very little to zero on modern processors. >> >> It may be worth auditing for these situations, but I've done extensive profiling of extremely memory heavy workloads (hundreds of gb) in Varnish over the past few years, and I promise that calloc is not a current limiting factor in terms of latency or throughput. >> >> Of course, if you're concerned about swapping, I'd also argue that your cache is not properly sized. >> >> >> On Tue, Mar 8, 2016, 04:28 Poul-Henning Kamp wrote: >>> >>> -------- >>> In message >>> , Federico Schwindt writes: >>> >>> >We use calloc in many places, I do wonder how many of them do really need >>> >it. The downside of using calloc when is not really needed is that by >>> >zeroing the memory you end up with resident memory and not virtual, which >>> >in turn might lead to swapping. >>> >>> This is almost always intentional, as we generally do not over-allocate. >>> >>> The exception is the malloc stevedore where we do. >>> >>> >>> -- >>> Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 >>> phk at FreeBSD.ORG | TCP/IP since RFC 956 >>> FreeBSD committer | BSD since 4.3-tahoe >>> Never attribute to malice what can adequately be explained by incompetence. >>> >>> _______________________________________________ >>> varnish-dev mailing list >>> varnish-dev at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev From dho at fastly.com Tue Mar 8 23:13:00 2016 From: dho at fastly.com (Devon H. O'Dell) Date: Tue, 8 Mar 2016 15:13:00 -0800 Subject: Patch: Use calloc instead of malloc when running out of VSM space (common_vsm.c) In-Reply-To: References: Message-ID: WRT this patch, I'm unclear where the implicit uninitialized read mentioned in the commit message occurs. The only members read/written from a `vr` attached to the `r_bogus` list are `len` and `ptr`, which are both assigned immediately after a successful allocation. (And that memory range is not shared between concurrent processes.) Also minor nits in the comment: you typoed "avoid" and "available", and one of the `sc->g_overflow` should be `sc->c_overflow`. On Tue, Mar 8, 2016 at 2:40 AM, P?l Hermunn Johansen wrote: > Hello, > I just started working at Varnish Software (last Thursday), and I am > looking forward to working with the Varnish Cache code. After firm > guidance from Martin, I am proposing my first patch. > > Best Regards, > > P?l Hermunn Johansen > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev From slink at schokola.de Wed Mar 9 08:00:00 2016 From: slink at schokola.de (Nils Goroll) Date: Wed, 9 Mar 2016 09:00:00 +0100 Subject: Varnish Project migration and mailing lists future In-Reply-To: <15457.1457473109@critter.freebsd.dk> References: <15457.1457473109@critter.freebsd.dk> Message-ID: <56DFD800.9050109@schokola.de> On 08/03/16 22:38, Poul-Henning Kamp wrote: > Comments, observations, ideas ? The only concern I have is giving away control over our information flow to private companies which we don't govern. For instance, if stackoverflow was to shut down or had all varnish posts censored (not that that's likely, but just to illustrate the concern), we'd loose an important part of the project's assets. With the move to github a similar step has already been taken and I have agreed due to all the practical aspects. I see the question about where to move the mailinglist traffic as a similar thing, so: I agree for all practical aspects, but overall I dislike the centralism which is happening all over the Internet and now also with the varnish project. Nils From phk at phk.freebsd.dk Wed Mar 9 08:09:51 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 09 Mar 2016 08:09:51 +0000 Subject: Patch: Use calloc instead of malloc when running out of VSM space (common_vsm.c) In-Reply-To: References: Message-ID: <35019.1457510991@critter.freebsd.dk> -------- In message , =?UTF-8?Q?P=C3=A5l_Hermunn_Johansen?= writes: >Subject: [PATCH] Use calloc instead of malloc to allocate extra VSM space. > >With malloc we would read from uninitialized memory (implicitly with ++ or +=). > >Also expanded comment: When we need to allocate extra VSM space >with malloc/calloc, the space will not be available to other >processes. The workaround is to increase VSM space through the >runtime parameter vsm_space. Good catch. The comment isn't going to help anybody very much though, it would be a better idea to improve on the comments on the vsm_overflow[ed] counters where people can see them. Personally the "did/does" difference was too subtle for me to spot, and I spent some time trying to find out why we had two counters with the exact same definition. It does not look like we emit a SLT_Error record when this happens, I think we should. VSM_Alloc() looks like the best place for it, but there is a chicken and egg issue since VSM_Alloc is also used to allocate the VSL buffer. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Wed Mar 9 08:28:42 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 09 Mar 2016 08:28:42 +0000 Subject: Varnish Project migration and mailing lists future In-Reply-To: <56DFD800.9050109@schokola.de> References: <15457.1457473109@critter.freebsd.dk> <56DFD800.9050109@schokola.de> Message-ID: <40313.1457512122@critter.freebsd.dk> -------- In message <56DFD800.9050109 at schokola.de>, Nils Goroll writes: >I agree for all practical aspects, but overall I dislike the >centralism which is happening all over the Internet and now also >with the varnish project. I'm not particularly happy about that aspect either. With respect to github, we are going to keep local copies of everything on github, just in case they explode some day. With respect to -misc/stackoverflow it is not really "our data" in the first place, and if, as it looks to me, stackoverflow is doing a better job, then we'd be stupid not let them. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From slink at schokola.de Wed Mar 9 08:55:53 2016 From: slink at schokola.de (Nils Goroll) Date: Wed, 9 Mar 2016 09:55:53 +0100 Subject: Varnish Project migration and mailing lists future In-Reply-To: <40313.1457512122@critter.freebsd.dk> References: <15457.1457473109@critter.freebsd.dk> <56DFD800.9050109@schokola.de> <40313.1457512122@critter.freebsd.dk> Message-ID: <56DFE519.3090501@schokola.de> On 09/03/16 09:28, Poul-Henning Kamp wrote: > With respect to github, we are going to keep local copies of > everything on github, just in case they explode some day. yes, that's the most relevant practical aspect, but still there is the potential of loosing issues, wiki pages etc. > With respect to -misc/stackoverflow it is not really "our data" > in the first place, and if, as it looks to me, stackoverflow > is doing a better job, then we'd be stupid not let them. Again I agree. I really see no relevant points to veto walking in the direction we're heading, except for the general unhappyness about centralism. Nils P.S.: I've got rid of my i-pocketcomputer three weeks ago. Certain things got more complicated, but my live got significantly better. From phk at phk.freebsd.dk Wed Mar 9 09:03:19 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 09 Mar 2016 09:03:19 +0000 Subject: Varnish Project migration and mailing lists future In-Reply-To: <56DFE519.3090501@schokola.de> References: <15457.1457473109@critter.freebsd.dk> <56DFD800.9050109@schokola.de> <40313.1457512122@critter.freebsd.dk> <56DFE519.3090501@schokola.de> Message-ID: <44783.1457514199@critter.freebsd.dk> -------- In message <56DFE519.3090501 at schokola.de>, Nils Goroll writes: >On 09/03/16 09:28, Poul-Henning Kamp wrote: >> With respect to github, we are going to keep local copies of >> everything on github, just in case they explode some day. > >yes, that's the most relevant practical aspect, but still there >is the potential of loosing issues, wiki pages etc. Which part of "everything on github" didn't you read ? :-) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From slink at schokola.de Wed Mar 9 09:19:00 2016 From: slink at schokola.de (Nils Goroll) Date: Wed, 9 Mar 2016 10:19:00 +0100 Subject: Varnish Project migration and mailing lists future In-Reply-To: <44783.1457514199@critter.freebsd.dk> References: <15457.1457473109@critter.freebsd.dk> <56DFD800.9050109@schokola.de> <40313.1457512122@critter.freebsd.dk> <56DFE519.3090501@schokola.de> <44783.1457514199@critter.freebsd.dk> Message-ID: <56DFEA84.1040000@schokola.de> > Which part of "everything on github" didn't you read ? :-) The "everything" bit. Sorry. From hermunn at varnish-software.com Wed Mar 9 13:01:00 2016 From: hermunn at varnish-software.com (=?UTF-8?Q?P=C3=A5l_Hermunn_Johansen?=) Date: Wed, 9 Mar 2016 14:01:00 +0100 Subject: Patch: Use calloc instead of malloc when running out of VSM space (common_vsm.c) In-Reply-To: <35019.1457510991@critter.freebsd.dk> References: <35019.1457510991@critter.freebsd.dk> Message-ID: > The comment isn't going to help anybody very much though, it would be > a better idea to improve on the comments on the vsm_overflow[ed] > counters where people can see them. I have changed the patch and attached it again. Now the comment is trimmed down a bit, and with this the many typos are hopefully gone. As I say in the commit message, I plan to do a different patch for the documentation of the counters. I would appreciate it if I got a green light on the patch, or some feedback on anything else which should be changed before pushing. > It does not look like we emit a SLT_Error record when this happens, > I think we should. VSM_Alloc() looks like the best place for it, > but there is a chicken and egg issue since VSM_Alloc is also used > to allocate the VSL buffer. I think I understood this, but will not change anything about error reporting now, unless it is completely straightforward. I am just starting working with the code, and I do not want to introduce an infinite recursion during my first week at Varnish Software. -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Use-calloc-instead-of-malloc-to-allocate-extra-VSM-s.patch Type: text/x-patch Size: 1641 bytes Desc: not available URL: From dridi at varni.sh Wed Mar 9 14:28:07 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 9 Mar 2016 15:28:07 +0100 Subject: Varnish Project migration and mailing lists future In-Reply-To: <56DFE519.3090501@schokola.de> References: <15457.1457473109@critter.freebsd.dk> <56DFD800.9050109@schokola.de> <40313.1457512122@critter.freebsd.dk> <56DFE519.3090501@schokola.de> Message-ID: On Wed, Mar 9, 2016 at 9:55 AM, Nils Goroll wrote: > I really see no relevant points to veto walking in the direction we're heading, > except for the general unhappyness about centralism. I'm with Nils but *also* unhappy because github feels very crippling to me, especially its web interface. I sincerely hope we'll never press the merge button of a pull request but instead keep a linear git history as we usually manage to do. Also I will not participate in Stackoverflow Q&As because: 1. I don't want to create yet another account to some random online service 2. I've had bad experiences with SO-driven-developers (copy-paste-madness) 3. I'm currently lagging significantly on the lower traffic -misc list Anyway I still hope this change is for the best :) Dridi From dridi at varni.sh Wed Mar 9 17:24:06 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 9 Mar 2016 18:24:06 +0100 Subject: VIP4: Restrict VMOD function call sites Message-ID: Hi, As instructed last Monday after the bugwash, I have created a new VIP. We started a discussion on restricting VRT functions usage during compilation of VMODs, I suggested a syntax to do that for VMOD functions. I interpreted phk's request for me to do a write up as part of the VIP process and came up with a slightly different suggestion. https://github.com/varnishcache/varnish-cache/wiki/VIP4%3A-Restrict-VMOD-function-call-sites Best Regards, Dridi From dho at fastly.com Wed Mar 9 17:47:41 2016 From: dho at fastly.com (Devon H. O'Dell) Date: Wed, 9 Mar 2016 09:47:41 -0800 Subject: Varnish Project migration and mailing lists future In-Reply-To: References: <15457.1457473109@critter.freebsd.dk> <56DFD800.9050109@schokola.de> <40313.1457512122@critter.freebsd.dk> <56DFE519.3090501@schokola.de> Message-ID: On Wed, Mar 9, 2016 at 6:28 AM, Dridi Boukelmoune wrote: > On Wed, Mar 9, 2016 at 9:55 AM, Nils Goroll wrote: >> I really see no relevant points to veto walking in the direction we're heading, >> except for the general unhappyness about centralism. > > I'm with Nils but *also* unhappy because github feels very crippling > to me, especially its web interface. I sincerely hope we'll never > press the merge button of a pull request but instead keep a linear git > history as we usually manage to do. > > Also I will not participate in Stackoverflow Q&As because: > 1. I don't want to create yet another account to some random online service > 2. I've had bad experiences with SO-driven-developers (copy-paste-madness) > 3. I'm currently lagging significantly on the lower traffic -misc list I also worry about using SO as a support mechanism. Apart from the concerns that Dridi mentioned, there are other problems with the entire SO model of having questions answered. Very fundamentally, it is up to the person who asks the question to accept an answer. The difference between "what works" and "what is correct" is in many cases quite stark. This sometimes has the effect of driving knowledgeable folks away from answering questions, which in turn produces more poor quality answers. Unless a question is extremely popular, it is also unlikely that community involvement results in a better answer becoming the selected answer. People who ask questions also frequently skip the whole part of accepting answers. This can make it confusing as to which answer is correct (upvotes aren't always a great indicator), and for people who take SO karma seriously, this also tends to drive down answer quality (especially when the asker is a guest / has 0 karma). If SO is inevitable for -misc traffic, I would recommend a slightly different approach: "Please ask your question at SO, and then link it here." And then encourage people to be good denizens, seeing questions through to resolution. Digging through the -misc archive, it appears that most questions at least result in discussion, whether or not they are resolved to the satisfaction of the asker. So it isn't a black hole. SO questions tagged with "varnish" currently has 5 pages of questions not marked as answered out of 25 pages of questions with the tag. Without looking at all of them, this illustrates that 20% questions are not being handled according to SO community guidelines. Driving more people to SO who are unfamiliar with its model is likely to make the support experience worse if there is no strategy for actually monitoring the questions. --dho > Anyway I still hope this change is for the best :) > > Dridi > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev From geoff at uplex.de Wed Mar 9 18:11:40 2016 From: geoff at uplex.de (Geoff Simmons) Date: Wed, 9 Mar 2016 19:11:40 +0100 Subject: Varnish Project migration and mailing lists future In-Reply-To: References: <15457.1457473109@critter.freebsd.dk> <56DFD800.9050109@schokola.de> <40313.1457512122@critter.freebsd.dk> <56DFE519.3090501@schokola.de> Message-ID: <56E0675C.7090101@uplex.de> On 03/09/2016 06:47 PM, Devon H. O'Dell wrote: > > I also worry about using SO as a support mechanism. Apart from the > concerns that Dridi mentioned, there are other problems with the > entire SO model of having questions answered. To be honest, I only see a number of downsides to SO as a replacement for -misc, and one upside: it's an alternative to setting up a new mailing list, which is a PITA due to the joys of sendmail.cf and combating spam. And I don't blame phk at all for preferring to avoid that. You want a mailing list to Just Work, and not consume a lot of time for administration. But there are good reasons for concern that it won't turn out that way. I wish I could suggest a clever solution, but this is a bit of a dilemma. Best, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From dho at fastly.com Wed Mar 9 18:48:28 2016 From: dho at fastly.com (Devon H. O'Dell) Date: Wed, 9 Mar 2016 10:48:28 -0800 Subject: Varnish Project migration and mailing lists future In-Reply-To: <56E0675C.7090101@uplex.de> References: <15457.1457473109@critter.freebsd.dk> <56DFD800.9050109@schokola.de> <40313.1457512122@critter.freebsd.dk> <56DFE519.3090501@schokola.de> <56E0675C.7090101@uplex.de> Message-ID: On Wed, Mar 9, 2016 at 10:11 AM, Geoff Simmons wrote: > On 03/09/2016 06:47 PM, Devon H. O'Dell wrote: >> >> I also worry about using SO as a support mechanism. Apart from the >> concerns that Dridi mentioned, there are other problems with the >> entire SO model of having questions answered. > > To be honest, I only see a number of downsides to SO as a replacement > for -misc, and one upside: it's an alternative to setting up a new > mailing list, which is a PITA due to the joys of sendmail.cf and > combating spam. > > And I don't blame phk at all for preferring to avoid that. You want a > mailing list to Just Work, and not consume a lot of time for > administration. But there are good reasons for concern that it won't > turn out that way. Having worked in email space for a while, I totally agree and sympathize with this. Had a brief chat with phk regarding this stuff last week on IRC regarding the overhead of MTA and ML maintenance, and being a "responsible" email generator. I agree that it's worthwhile to reduce the overhead of list management. But I think if one of the lists that gets a chopping block is going to be a support list, there should be some procedures in place to ease that transition, and not put the support burden on an anonymous community that may or may not actually represent Varnish support talent. --dho > I wish I could suggest a clever solution, but this is a bit of a dilemma. > > > Best, > Geoff > -- > ** * * UPLEX - Nils Goroll Systemoptimierung > > Scheffelstra?e 32 > 22301 Hamburg > > Tel +49 40 2880 5731 > Mob +49 176 636 90917 > Fax +49 40 42949753 > > http://uplex.de > > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev From phk at phk.freebsd.dk Wed Mar 9 21:25:32 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 09 Mar 2016 21:25:32 +0000 Subject: Patch: Use calloc instead of malloc when running out of VSM space (common_vsm.c) In-Reply-To: References: <35019.1457510991@critter.freebsd.dk> Message-ID: <88461.1457558732@critter.freebsd.dk> -------- In message , =?UTF-8?Q?P=C3=A5l_Hermunn_Johansen?= writes: I have committed your patch. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From ruben at varnish-software.com Wed Mar 9 21:29:24 2016 From: ruben at varnish-software.com (=?UTF-8?Q?Rub=C3=A9n_Romero?=) Date: Wed, 9 Mar 2016 22:29:24 +0100 Subject: Varnish User Survey 2016 Message-ID: Hello, As PHK wrote recently to this list (yes, both), the Varnish project is undergoing a few adjustments and so it is time for us to get more knowledge on how you, our users, are currently using the software and the project's resources. We have a survey for that and it is now up: * https://www.varnish-cache.org/survey2016 Thank you in advance for helping us make Varnish better for you! Best regards, -- *Rub?n Romero*Community Manager Varnish Software Group Cell: +47 95964088 / Office: +47 21989260 Skype, Twitter & IRC: ruben_varnish We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Wed Mar 9 22:16:57 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 09 Mar 2016 22:16:57 +0000 Subject: Varnish Project migration and mailing lists future In-Reply-To: References: <15457.1457473109@critter.freebsd.dk> <56DFD800.9050109@schokola.de> <40313.1457512122@critter.freebsd.dk> <56DFE519.3090501@schokola.de> <56E0675C.7090101@uplex.de> Message-ID: <12880.1457561817@critter.freebsd.dk> -------- It doesn't look like my idea of abolishing -misc to SO is particular popular around here, so I guess we'll continue the mailing list for now. I've spent a fair bit of time today getting the mail-jail configured and I can't say I'm particularlyk thrilled about the crap-load of code that jail runs[1]. So far I've managed to get emails through postfix via the amavisd and clam-av filters, I consider that a success. Next up is to get mailman configured. Should there, somewhere on the -dev@ list, be somebody who just itches to become the Varnish Project postmaster, but havn't told us about this secret ambition yet: Please don't hesitate any longer. I'd far rather spend my time writing code than (re)learning postmastering. Poul-Henning [1] # pkg install -y git postfix amavisd-new clamav mailman New packages to be INSTALLED: amavisd-new: 2.10.1_1,1 arc: 5.21p arj: 3.10.22_4 ca_root_nss: 3.22.2 cabextract: 1.6 clamav: 0.99 curl: 7.47.0 cvsps: 2.1_1 db5: 5.3.28_3 expat: 2.1.0_3 file: 5.22 freeze: 2.5_2 gettext-runtime: 0.19.6 git: 2.6.4 gnupg1: 1.4.20 indexinfo: 0.2.4 json-c: 0.12_2 lha: 1.14i_6 libffi: 3.2.1 libidn: 1.31 libltdl: 2.4.6 libxml2: 2.9.3 lzo2: 2.09 lzop: 1.03 mailman: 2.1.20_2 p5-Archive-Zip: 1.55 p5-Authen-SASL: 2.16_1 p5-BerkeleyDB: 0.55 p5-Convert-BinHex: 1.125 p5-Convert-TNEF: 0.18_1 p5-Convert-UUlib: 1.50,1 p5-Crypt-OpenSSL-Bignum: 0.06 p5-Crypt-OpenSSL-RSA: 0.28_1 p5-Crypt-OpenSSL-Random: 0.11 p5-Digest-HMAC: 1.03_1 p5-Encode-Detect: 1.01_1 p5-Error: 0.17024 p5-GSSAPI: 0.28_1 p5-HTML-Parser: 3.71_1 p5-HTML-Tagset: 3.20_1 p5-HTTP-Date: 6.02_1 p5-IO-Multiplex: 1.13_1 p5-IO-Socket-INET6: 2.72_1 p5-IO-Socket-IP: 0.37 p5-IO-Socket-SSL: 2.021 p5-IO-stringy: 2.111 p5-MIME-Tools: 5.507,2 p5-Mail-DKIM: 0.40_2 p5-Mail-Tools: 2.14 p5-Mozilla-CA: 20141217 p5-Net-DNS: 1.04,1 p5-Net-LibIDN: 0.12_4 p5-Net-SMTP-SSL: 1.03 p5-Net-SSLeay: 1.72 p5-Net-Server: 2.008_1 p5-NetAddr-IP: 4.075 p5-Socket6: 0.25_2 p5-Socket: 2.021 p5-TimeDate: 2.30_2,1 p5-Unix-Syslog: 1.1_1 p7zip: 15.09 pcre: 8.37_4 perl5: 5.20.3_8 postfix: 2.11.7_1,1 py27-dnspython: 1.12.0 py27-setuptools27: 19.2 python27: 2.7.11_1 python2: 2_3 re2c: 0.14.3 ripole: 0.2.2 rpm2cpio: 1.4_1 spamassassin: 3.4.1_5 unrar: 5.30,5 unzoo: 4.4_2 zoo: 2.10.1_3 -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Fri Mar 11 09:46:28 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 11 Mar 2016 09:46:28 +0000 Subject: Somebody please write this for us... Message-ID: <92008.1457689588@critter.freebsd.dk> I have added a VIP5 for a remote test-reporting facility: https://github.com/varnishcache/varnish-cache/wiki/VIP5:-Remote-test-run-reporting-facility Any thumb-twiddling devops out there looking for a good little project? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Sun Mar 13 22:33:57 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Sun, 13 Mar 2016 23:33:57 +0100 Subject: [PATCH] Add a miniobj macro for pointer acquisitions Message-ID: Hi, In this evening's offline self-hackathon in the train I came up with a macro for a recurring pattern in Varnish Cache that I also happen to use in at least one VMOD so it might benefit others too. I used Coccinelle (and still love it) in order to apply the change, so if the first patch is accepted you remember doing this kind of pointer acquisition in one of your VMOD or VUT, I've attached the semantic patch too. If the name "acquire" is not appropriate (because I know at least one other meaning) please know that I'm short on naming ideas. Best, Dridi -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Add-a-miniobj-macro-for-pointer-acquisitions.patch Type: text/x-patch Size: 1577 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-Use-the-ACQUIRE_OBJ_NOTNULL-macro-where-relevant.patch Type: text/x-patch Size: 8461 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: acquire.cocci Type: application/octet-stream Size: 127 bytes Desc: not available URL: From fgsch at lodoss.net Tue Mar 15 08:29:41 2016 From: fgsch at lodoss.net (Federico Schwindt) Date: Tue, 15 Mar 2016 08:29:41 +0000 Subject: github labels Message-ID: Hi, Continuing the conversation during the bugwash yesterday I think we should have these labels for starts: - 4.0 - 4.1 - master - more-info-needed - needs-backport We can also have labels for different components as we had in trac: - build - documentation - varnishadm - varnishd - varnishhist - varnishlog - varnishncsa - varnishstat - varnishtest - varnishtop - vmod That should cover most if not all our current needs. Thoughts? -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at uplex.de Tue Mar 15 08:56:34 2016 From: geoff at uplex.de (Geoff Simmons) Date: Tue, 15 Mar 2016 09:56:34 +0100 Subject: github labels In-Reply-To: References: Message-ID: <56E7CE42.2020009@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 03/15/2016 09:29 AM, Federico Schwindt wrote: > > - 4.0 - 4.1 - master - more-info-needed - needs-backport ACK > We can also have labels for different components as we had in > trac: > > - build - documentation - varnishadm - varnishd - varnishhist - > varnishlog - varnishncsa - varnishstat - varnishtest - varnishtop - > vmod - - varnishapi I have occasionally brought up issues about VSL, and there's also VSC and the CLI. I may have been the only one, but it never fit well into any of these categories. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCAAGBQJW5842AAoJEOUwvh9pJNURml0P/2tikLJkZ1kElCK7fV9/RFsY FS/RuCl16nLA6bitVt7T+7E2+w9Kh+qeOCPQh3TLMJ6VjCV9Qfuhde3ytT09I7b3 6sP+kYybmKN2mObdoKwkgf6chcIrsZeGvYWiLbZ3VYcRMSwyelSajhhUxWHciczA 1C5aBwXNxCpH0TRoh6yndD4ReDHTiXeS/2/IALkelkNV1wZhD6iQ1dU22URfiAT1 ZIn7ECouiZxSgXLslDsx2oHcjHMvjhCDQEmRmdT3KTzMoS4Oyr5fCk53DgeILqV2 2rJ2mADetd449gcTDMRQkw18PwJFoaGaWN7c4Jx1tOVCCQMPdZn4hue/YK6noqbe 1IcG1gVhjO3aUq4ItKlQxGAdAlDq0AS4E73+KRJpK3GhNn9dZpOvyROlYEJggN5Q tsGV6hBKBeFzdJNd9Ec0/cPJtWOyrigt/OD3mbvj+6YL8qn5v2tGUdBhFk67v4Iu goF8qArIxVDfPt5YLyuumAUZwyWC82tgYfj+aVXkKKYmyj8uOrcdmUsjNC3EwxEi BGgCYySxF3EBtQzn9hGXoM6/cy1zm5A8/rhuroCU2n9XSyWziEfzlcQ0Wj4y0E3H 1tVLb08hOSjNe8z3CvQCowPce1B8kZlfXOEdMI4h09fxwS3GK6r1x5Lu7pEAldGX 2GsZIUg4sKby7PsQj60L =qAre -----END PGP SIGNATURE----- From fgsch at lodoss.net Tue Mar 15 09:23:33 2016 From: fgsch at lodoss.net (Federico Schwindt) Date: Tue, 15 Mar 2016 09:23:33 +0000 Subject: github labels In-Reply-To: <56E7CE42.2020009@uplex.de> References: <56E7CE42.2020009@uplex.de> Message-ID: Good idea. On Tue, Mar 15, 2016 at 8:56 AM, Geoff Simmons wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > On 03/15/2016 09:29 AM, Federico Schwindt wrote: > > > > - 4.0 - 4.1 - master - more-info-needed - needs-backport > > ACK > > > We can also have labels for different components as we had in > > trac: > > > > - build - documentation - varnishadm - varnishd - varnishhist - > > varnishlog - varnishncsa - varnishstat - varnishtest - varnishtop - > > vmod > > - - varnishapi > > I have occasionally brought up issues about VSL, and there's also VSC > and the CLI. I may have been the only one, but it never fit well into > any of these categories. > > > Best, > Geoff > - -- > ** * * UPLEX - Nils Goroll Systemoptimierung > > Scheffelstra?e 32 > 22301 Hamburg > > Tel +49 40 2880 5731 > Mob +49 176 636 90917 > Fax +49 40 42949753 > > http://uplex.de > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1 > > iQIcBAEBCAAGBQJW5842AAoJEOUwvh9pJNURml0P/2tikLJkZ1kElCK7fV9/RFsY > FS/RuCl16nLA6bitVt7T+7E2+w9Kh+qeOCPQh3TLMJ6VjCV9Qfuhde3ytT09I7b3 > 6sP+kYybmKN2mObdoKwkgf6chcIrsZeGvYWiLbZ3VYcRMSwyelSajhhUxWHciczA > 1C5aBwXNxCpH0TRoh6yndD4ReDHTiXeS/2/IALkelkNV1wZhD6iQ1dU22URfiAT1 > ZIn7ECouiZxSgXLslDsx2oHcjHMvjhCDQEmRmdT3KTzMoS4Oyr5fCk53DgeILqV2 > 2rJ2mADetd449gcTDMRQkw18PwJFoaGaWN7c4Jx1tOVCCQMPdZn4hue/YK6noqbe > 1IcG1gVhjO3aUq4ItKlQxGAdAlDq0AS4E73+KRJpK3GhNn9dZpOvyROlYEJggN5Q > tsGV6hBKBeFzdJNd9Ec0/cPJtWOyrigt/OD3mbvj+6YL8qn5v2tGUdBhFk67v4Iu > goF8qArIxVDfPt5YLyuumAUZwyWC82tgYfj+aVXkKKYmyj8uOrcdmUsjNC3EwxEi > BGgCYySxF3EBtQzn9hGXoM6/cy1zm5A8/rhuroCU2n9XSyWziEfzlcQ0Wj4y0E3H > 1tVLb08hOSjNe8z3CvQCowPce1B8kZlfXOEdMI4h09fxwS3GK6r1x5Lu7pEAldGX > 2GsZIUg4sKby7PsQj60L > =qAre > -----END PGP SIGNATURE----- > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Thu Mar 17 11:50:39 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 17 Mar 2016 11:50:39 +0000 Subject: [PATCH] Add a miniobj macro for pointer acquisitions In-Reply-To: References: Message-ID: <31860.1458215439@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >In this evening's offline self-hackathon in the train I came up with a >macro for a recurring pattern in Varnish Cache that I also happen to >use in at least one VMOD so it might benefit others too. I agree that we should beautify this pattern, but I can't get used to the word "ACQUIRE" for it, particular because we are inside a "release" functionality. Could we call it "TAKE_OBJ_NOTMULL()" ? That matches what we are doing to the object reference much better. I think TAKEOVER_OBJ_NOTNULL() would be too clunky? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Thu Mar 17 12:42:37 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 17 Mar 2016 12:42:37 +0000 Subject: Collapse libvarnish and libvarnishapi Message-ID: <32884.1458218557@critter.freebsd.dk> Does it make any sense to have two "API" libraries any more ? A lot of the code is already replicated between them. If we collapse, which name to retain ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Thu Mar 17 13:03:55 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 17 Mar 2016 14:03:55 +0100 Subject: [PATCH] Add a miniobj macro for pointer acquisitions In-Reply-To: <31860.1458215439@critter.freebsd.dk> References: <31860.1458215439@critter.freebsd.dk> Message-ID: On Thu, Mar 17, 2016 at 12:50 PM, Poul-Henning Kamp wrote: > -------- > In message , Dridi Boukelmoune writes: > >>In this evening's offline self-hackathon in the train I came up with a >>macro for a recurring pattern in Varnish Cache that I also happen to >>use in at least one VMOD so it might benefit others too. > > I agree that we should beautify this pattern, but I can't get used > to the word "ACQUIRE" for it, particular because we are inside a > "release" functionality. > > Could we call it "TAKE_OBJ_NOTMULL()" ? > > That matches what we are doing to the object reference much better. > > I think TAKEOVER_OBJ_NOTNULL() would be too clunky? As I said, I'm lacking imagination here. Maybe STEAL_OBJ_NOTNULL? Synonyms to take according to my search engine: Verb: take, occupy, use up, lead, direct, conduct, guide, get hold of, assume, acquire. Synonyms to steal: Verb: slip, advance, gain, gain ground, get ahead, make headway, move, pull ahead, take, win. The ones I could see myself using, in alphabetical order: ASSUME_ (as in "it becomes my responsibility") MOVE_ (as in C++ or Rust move semantics IIUC) OWN_ STEAL_ (stolen from vbc terminology) TAKE_ UNDERTAKE_ (not sure it'd fit better than take over) Thoughts? From geoff at uplex.de Thu Mar 17 13:30:37 2016 From: geoff at uplex.de (Geoff Simmons) Date: Thu, 17 Mar 2016 14:30:37 +0100 Subject: Collapse libvarnish and libvarnishapi In-Reply-To: <32884.1458218557@critter.freebsd.dk> References: <32884.1458218557@critter.freebsd.dk> Message-ID: <56EAB17D.6020203@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 03/17/2016 01:42 PM, Poul-Henning Kamp wrote: > Does it make any sense to have two "API" libraries any more ? Would that mean that all of libvarnish becomes a public API, as VSL and VSC are now? If so, you probably can best answer the question -- commit to keeping the API compatible over time, or take flak when there are incompatible changes? Design the interfaces with this sort of thing in mind? I've developed apps and APIs against VSL and VSC, so IMO it matters a lot. But then if all of libvarnish becomes a public API, *many* things become much easier. - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCAAGBQJW6rF9AAoJEOUwvh9pJNURpiEP/iik0YnQDEf9bjQjd3s/Kbx6 HCPI5lg1MinwGLDXkEMK5dUzpPAaLP4VzDatnv2+5HQEjP4Ar1kjw/aD2QC+c1dl d+FRE4qY1PLpItFoCWM2RO37ZylZkIoWnQQ95vFjxqa/6c9sgJY7QLz4zjwgkB2Z yU654b4no+qxCntg3l5nn9RBYY7zm7EMXEpzM2l86UYhPiwjjMDOZ2wi+cDvtxHP I5hcIPEVtorOOZlttvxhKgVl7FVzZf+GX85vb4xsnhYNy8K2R+paoNC4F3md8CtW 5Bv0mt3hsBV2bhG/Hq56Cz+6491JY8eqSffbT7QmTEHtwULOFJOLXwNXy/N8PaHQ LKrGdPBCR8XslVkrr8x4hdKHwjBXtV9USWIPw3kTjHTnDq/vgTM4LBX3ABLh4Reg D83qzOZZ7FGi8WneBMI+ex23VltLtxGs/sgzoG7MG9kh4ahq523N7W68uv6f4yE4 +QH4ji2QgHoAYZWGQZUO+13YIcH81fP9wUSeXpDgOComGUi8z29q2TGFMeF4xtjI htizWxprDc1mHp4Xdw/GsrOh9eIWHf2UOYjSF2+qTJFgf41GW2OZSNKee8p/rceF lhNCUjDPE7k88AKfiG/zAgnZ8NLibG8fa0u27zPLoQGljRYmD4XfLpTtyLr++HsV 9M4sypHQvXiTrOZBNkPu =bcZ5 -----END PGP SIGNATURE----- From phk at phk.freebsd.dk Thu Mar 17 15:52:26 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 17 Mar 2016 15:52:26 +0000 Subject: Collapse libvarnish and libvarnishapi In-Reply-To: <56EAB17D.6020203@uplex.de> References: <32884.1458218557@critter.freebsd.dk> <56EAB17D.6020203@uplex.de> Message-ID: <33300.1458229946@critter.freebsd.dk> -------- In message <56EAB17D.6020203 at uplex.de>, Geoff Simmons writes: >> Does it make any sense to have two "API" libraries any more ? > >Would that mean that all of libvarnish becomes a public API, as VSL >and VSC are now? Isn't it already a defacto public API for VMODS ? >If so, you probably can best answer the question -- commit to keeping >the API compatible over time, or take flak when there are incompatible >changes? Design the interfaces with this sort of thing in mind? Most of the stuff in the libraries are actually incredibly stable. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From geoff at uplex.de Thu Mar 17 16:07:37 2016 From: geoff at uplex.de (Geoff Simmons) Date: Thu, 17 Mar 2016 17:07:37 +0100 Subject: Collapse libvarnish and libvarnishapi In-Reply-To: <33300.1458229946@critter.freebsd.dk> References: <32884.1458218557@critter.freebsd.dk> <56EAB17D.6020203@uplex.de> <33300.1458229946@critter.freebsd.dk> Message-ID: <56EAD649.1000700@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 03/17/2016 04:52 PM, Poul-Henning Kamp wrote: > >>> Does it make any sense to have two "API" libraries any more ? >> >> Would that mean that all of libvarnish becomes a public API, as >> VSL and VSC are now? > > Isn't it already a defacto public API for VMODS ? Very much so. > Most of the stuff in the libraries are actually incredibly stable. I'm all +1, and can't think of why a developer wouldn't be. Like I said, easy for me to say. - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCAAGBQJW6tZJAAoJEOUwvh9pJNURcXsP/AhUeRLyX4x45lFbSx8SmLSz yYUCcT3Hv+yZA+jB7g+B3SzhWzmYAgXNiCh+/fZBQQlzMiSntLILXZU+Jfj+eHM/ NMPMc5wSlTJcAnvLJF05FdgSTWq0l+NrC7+gfmdKwjkykJyfOUvnGr0pmmIGkypF gHND/wENDA0cfHVpvGKLbg68z9iMtQZnDChtJhIP9hDWjgmNdPJcvMr1e1q0cI4x pFdhvPswd2VrlYsd8R97Lob75HIsnwlXPSfan8sFR49Lutv/suWAMMalwctvI323 Kp9WscEhKOJm+Y/QVvDzPNkQDXtSDbt+J7lT27Gm0B1ulpTNkhhEFR/zYWVaIC5C D5fdkEIdXc0E5dXcu7lXClNyRBn+HvgSgug2aEc6y4vphMpd/u1VGrlkC5p55tVx TLjBeHw8T0Ktk8mtZ0xpfgBe6SybcrMaQ6PixNBmoWWk3dAvxcwgabIzFdrNTmVt 2LpQ0Jnf8uxcawn8+ThxRt6cAwXdtBZotREPbj0CYKRj7F/EL7HMWmdy4H+l9IrG a7mp7W41junwQVtuLz3nRdiHfTalas/h2x3m1jRCDwahS4C1OGSs7Po1NcvHvrRO 1Dyd8OKu3GXFAIF8hVHcJvvxUftWDcvJYkj+cyt24WZjw4q0G7eFWUPhw1qC07Q2 vPHD0Zbp6/3O5/ROKaxh =JHIs -----END PGP SIGNATURE----- From dridi at varni.sh Thu Mar 17 16:16:35 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 17 Mar 2016 17:16:35 +0100 Subject: Collapse libvarnish and libvarnishapi In-Reply-To: <33300.1458229946@critter.freebsd.dk> References: <32884.1458218557@critter.freebsd.dk> <56EAB17D.6020203@uplex.de> <33300.1458229946@critter.freebsd.dk> Message-ID: On Thu, Mar 17, 2016 at 4:52 PM, Poul-Henning Kamp wrote: > -------- > In message <56EAB17D.6020203 at uplex.de>, Geoff Simmons writes: > >>> Does it make any sense to have two "API" libraries any more ? >> >>Would that mean that all of libvarnish becomes a public API, as VSL >>and VSC are now? > > Isn't it already a defacto public API for VMODS ? It is because varnishd links to it, and for the includes that are installed. I know bits were missing in the past (I remember vtim.h at some point) but I can't talk for now. >>If so, you probably can best answer the question -- commit to keeping >>the API compatible over time, or take flak when there are incompatible >>changes? Design the interfaces with this sort of thing in mind? > > Most of the stuff in the libraries are actually incredibly stable. Then it should move from $(libdir)/varnish to $(libdir) and become a first-class citizen. libvarnishapi has a clear scope, but libvarnish OTOH is a catch-all library kind of like libc and contains things that are sometimes completely unrelated. It cover areas like date/time, tcp, string buffers, and so much more so I don't see the benefit of merging them. Dridi From dridi at varni.sh Thu Mar 17 16:19:59 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 17 Mar 2016 17:19:59 +0100 Subject: Somebody please write this for us... In-Reply-To: <92008.1457689588@critter.freebsd.dk> References: <92008.1457689588@critter.freebsd.dk> Message-ID: On Fri, Mar 11, 2016 at 10:46 AM, Poul-Henning Kamp wrote: > I have added a VIP5 for a remote test-reporting facility: > > https://github.com/varnishcache/varnish-cache/wiki/VIP5:-Remote-test-run-reporting-facility > > Any thumb-twiddling devops out there looking for a good little project? Per wrote a varnishtest results aggregator, so maybe we only need machine time? Dridi From phk at phk.freebsd.dk Thu Mar 17 16:23:36 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 17 Mar 2016 16:23:36 +0000 Subject: Collapse libvarnish and libvarnishapi In-Reply-To: References: <32884.1458218557@critter.freebsd.dk> <56EAB17D.6020203@uplex.de> <33300.1458229946@critter.freebsd.dk> Message-ID: <33458.1458231816@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >libvarnishapi has a clear scope, Yes, that's for VSM/CLI only applications. >but libvarnish OTOH is a catch-all library kind of like libc >and contains things that are sometimes completely unrelated. It is our internal library. But if you look carefully, you'll see that we drag a lot libvarnish into libvarnishapi already: ../libvarnish/vas.c \ ../libvarnish/vav.c \ ../libvarnish/version.c \ ../libvarnish/cli_common.c \ ../libvarnish/cli_auth.c \ ../libvarnish/vin.c \ ../libvarnish/vmb.c \ ../libvarnish/vre.c \ ../libvarnish/vsb.c \ ../libvarnish/vtim.c \ ../libvarnish/vnum.c \ ../libvarnish/vsha256.c \ So in practice the libraries are less different than you'd think... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Thu Mar 17 16:39:36 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 17 Mar 2016 17:39:36 +0100 Subject: Collapse libvarnish and libvarnishapi In-Reply-To: <33458.1458231816@critter.freebsd.dk> References: <32884.1458218557@critter.freebsd.dk> <56EAB17D.6020203@uplex.de> <33300.1458229946@critter.freebsd.dk> <33458.1458231816@critter.freebsd.dk> Message-ID: > It is our internal library. Yes, this is why I'm suggesting to turn it into a first-class citizen. By that I meant public and on the default linker's path. > But if you look carefully, you'll see that we drag a lot > libvarnish into libvarnishapi already: > > ../libvarnish/vas.c \ > ../libvarnish/vav.c \ > ../libvarnish/version.c \ > ../libvarnish/cli_common.c \ > ../libvarnish/cli_auth.c \ > ../libvarnish/vin.c \ > ../libvarnish/vmb.c \ > ../libvarnish/vre.c \ > ../libvarnish/vsb.c \ > ../libvarnish/vtim.c \ > ../libvarnish/vnum.c \ > ../libvarnish/vsha256.c \ > > So in practice the libraries are less different than you'd think... Well, I did ask about that once, but didn't get an answer. I don't remember clearly, maybe a year or two ago on IRC. The question was about linking against libvarnish instead of embedding sources files. Last time I checked only varnishd and varnishtest were linking against libvarnish while the rest would embed whatever they use. Why not making libvarnish public and linking against it whenever we need to use it? Dridi From perbu at varnish-software.com Thu Mar 17 17:37:43 2016 From: perbu at varnish-software.com (Per Buer) Date: Thu, 17 Mar 2016 13:37:43 -0400 Subject: Somebody please write this for us... In-Reply-To: References: <92008.1457689588@critter.freebsd.dk> Message-ID: Hi, On Thu, Mar 17, 2016 at 12:19 PM, Dridi Boukelmoune wrote: > On Fri, Mar 11, 2016 at 10:46 AM, Poul-Henning Kamp > wrote: > > I have added a VIP5 for a remote test-reporting facility: > > > > > https://github.com/varnishcache/varnish-cache/wiki/VIP5:-Remote-test-run-reporting-facility > > > > Any thumb-twiddling devops out there looking for a good little project? > > Per wrote a varnishtest results aggregator, so maybe we only need machine > time? > Yes, I did. I stopped working on it as varnishtest needed some outputs for it to be useful. Uname output and uuid for the machine. Is it in place (varnish doesn't seem to build for me now - OS X seems broken)? If the new stuff is in place I can get it working again over easter. I also chose a different transport (HTTP) compared to what I see PHK had specified. Downside of that is that it is synchronous. The upside is that it is much more likely to work as opposed to mail (working email doesn't seem to be a hard requirement on unix anymore). -- *Per Buer* CTO | Varnish Software AS Cell: +47 95839117 We Make Websites Fly! www.varnish-software.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Thu Mar 17 18:18:36 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 17 Mar 2016 18:18:36 +0000 Subject: Somebody please write this for us... In-Reply-To: References: <92008.1457689588@critter.freebsd.dk> Message-ID: <33778.1458238716@critter.freebsd.dk> -------- In message , Per Buer writes: >I also chose a different transport (HTTP) compared to what I see PHK had >specified. Downside of that is that it is synchronous. The upside is that >it is much more likely to work as opposed to mail (working email doesn't >seem to be a hard requirement on unix anymore). Being able to select transport would be nice. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Thu Mar 17 18:48:31 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 17 Mar 2016 18:48:31 +0000 Subject: Collapse libvarnish and libvarnishapi In-Reply-To: References: <32884.1458218557@critter.freebsd.dk> <56EAB17D.6020203@uplex.de> <33300.1458229946@critter.freebsd.dk> <33458.1458231816@critter.freebsd.dk> Message-ID: <33871.1458240511@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >Why not making libvarnish public and linking against it whenever we >need to use it? That was sort of the question I tried to ask :-) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Fri Mar 18 07:53:05 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 18 Mar 2016 07:53:05 +0000 Subject: [PATCH] Add a miniobj macro for pointer acquisitions In-Reply-To: References: <31860.1458215439@critter.freebsd.dk> Message-ID: <36525.1458287585@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >> Could we call it "TAKE_OBJ_NOTMULL()" ? >> >As I said, I'm lacking imagination here. Maybe STEAL_OBJ_NOTNULL? Well, we're not "stealing" it, it's being handed to us. Make it TAKE_OBJ_NOTNULL() -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From arianna.aondio at varnish-software.com Fri Mar 18 10:26:34 2016 From: arianna.aondio at varnish-software.com (Arianna Aondio) Date: Fri, 18 Mar 2016 06:26:34 -0400 Subject: [PATCH] Varnishadm: log last time a backend health changed. Message-ID: Hi everyone, it may be useful to know when a backend's health changed. Please find attached a patch that expose last_changed in "varnishadm backend.list". This is 5.0 material because this patch could break any parser using the actual 4.x varnishadm.(I.E. the varnish-agent) -- Arianna Aondio Field Engineer | Varnish Software Group Mobile: +47 98062619 We Make Websites Fly www.varnish-software.com -------------- next part -------------- A non-text attachment was scrubbed... Name: backend_list.patch Type: text/x-diff Size: 1373 bytes Desc: not available URL: From phk at phk.freebsd.dk Fri Mar 18 11:11:36 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 18 Mar 2016 11:11:36 +0000 Subject: [PATCH] Varnishadm: log last time a backend health changed. In-Reply-To: References: Message-ID: <37371.1458299496@critter.freebsd.dk> -------- In message , Arianna Aondio writes: >it may be useful to know when a backend's health changed. >Please find attached a patch that expose last_changed in "varnishadm >backend.list". OK with me. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From geoff at uplex.de Fri Mar 18 11:20:00 2016 From: geoff at uplex.de (Geoff Simmons) Date: Fri, 18 Mar 2016 12:20:00 +0100 Subject: [PATCH] Varnishadm: log last time a backend health changed. In-Reply-To: References: Message-ID: <56EBE460.8080804@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 03/18/2016 11:26 AM, Arianna Aondio wrote: > > it may be useful to know when a backend's health changed. Please > find attached a patch that expose last_changed in "varnishadm > backend.list". Very minor quibble -- "Last updated" might be mistaken for meaning the last time a probe was performed, rather than the last change in health status. "Last changed" would be IMO less prone to misunderstanding. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCAAGBQJW6+RWAAoJEOUwvh9pJNURxnYP/ikOvQ7oPeuXbcnnsoKcqygC 2XS+aHeipY/j4vBA2kivWxFhSIDjnEXl9UuTNc7xZYkZM4xTQGmGoJQgiA+hHdtP 5suBKpnaNAHFAkD6m9jlqs+zPWXtxh4CpOxHt2qi9uFjS2mJ4VwOHKa3UzNfpyO0 vt7InxVu7ziB7OcNhR237bQg5ki7OYceBbsNwndWtHL5qKUo2cQY/+6iE06v0z02 tYGCEjuDcC6ZGxUz39gjRaprOKOG5pFP7SirKBDVSjKXtdwYpZh1wvywzkMJTwcA ZIu27rh0Zuhquha3xbN1HXDNZ70fTjCFyHuFc2y/w0hD1shCiTaYvoU9tdEpZV2E B0zHQ8IsJxWtEYtrR9JaBfGyDZ9bDGXsCWeI8d4DhDy2qp8ezWGmHmR2D7QIMGeh zGQlWbzSYymCzm1XNvbboWSjmnQ39xFv0C0V2KnvMHNBUoCS5kb+tFM9aUAUv8HL cTWin3sS+cPIN6zV8owBY6rls+XHIFvW9N5XtKEe4mXErDe0UG3R4ArxNplJ/b+f mBSJPWOsSdG2ewfwYKSu8CJkkvvKpbO/1vfhXgKrCalOTiGbCDQWJWeCi2lIUTiL VqfdA+QTyJAHG1F54ZJsqirhEf0E2+SCF3NOwk+6R9DNlWHYspkp7x1KlTiRxQ0p kpJVbqwpuiTmIOvj4JrZ =QACL -----END PGP SIGNATURE----- From dridi at varni.sh Fri Mar 18 11:31:27 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Fri, 18 Mar 2016 12:31:27 +0100 Subject: [PATCH] Add a miniobj macro for pointer acquisitions In-Reply-To: <36525.1458287585@critter.freebsd.dk> References: <31860.1458215439@critter.freebsd.dk> <36525.1458287585@critter.freebsd.dk> Message-ID: > Make it TAKE_OBJ_NOTNULL() Done! Semantic patch attached in case anyone's interested. Should I back-port the macro to the 4.0 and 4.1 branch? So that VMOD authors and libvarnishapi users may start using it with the next maintenance releases. That would be this commit: https://github.com/varnishcache/varnish-cache/commit/4a655c0 Dridi -------------- next part -------------- A non-text attachment was scrubbed... Name: take.cocci Type: application/octet-stream Size: 124 bytes Desc: not available URL: From phk at phk.freebsd.dk Fri Mar 18 11:38:21 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 18 Mar 2016 11:38:21 +0000 Subject: [PATCH] Add a miniobj macro for pointer acquisitions In-Reply-To: References: <31860.1458215439@critter.freebsd.dk> <36525.1458287585@critter.freebsd.dk> Message-ID: <37773.1458301101@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >> Make it TAKE_OBJ_NOTNULL() >Should I back-port the macro to the 4.0 and 4.1 branch? So that VMOD >authors and libvarnishapi users may start using it with the next >maintenance releases. I don't see why not. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From guillaume at varnish-software.com Mon Mar 21 10:39:57 2016 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Mon, 21 Mar 2016 11:39:57 +0100 Subject: [PATCH] Teach vmod-debug how to sync with varnishtest In-Reply-To: References: Message-ID: Looks ok to me. I'd just like we actually send some bytes instead of expecting to read 0 bytes in a buf[32]. I'm not sure we actually need 2 types of barriers, but I get the need to reduce waste when not needed. Rebase and go? -- Guillaume Quintard On Thu, Dec 24, 2015 at 12:48 PM, Dridi Boukelmoune wrote: > > Thoughts? > > Do you know this feeling when you press the start button but you > forgot something? Then your brain starts a background thread and > moment later (like in the shower or during lunch) you realize you > forgot to put some of the laundry in the washing machine... > > Cheers, > Dridi > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Mon Mar 21 17:57:45 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 21 Mar 2016 18:57:45 +0100 Subject: Collapse libvarnish and libvarnishapi In-Reply-To: <33871.1458240511@critter.freebsd.dk> References: <32884.1458218557@critter.freebsd.dk> <56EAB17D.6020203@uplex.de> <33300.1458229946@critter.freebsd.dk> <33458.1458231816@critter.freebsd.dk> <33871.1458240511@critter.freebsd.dk> Message-ID: On Thu, Mar 17, 2016 at 7:48 PM, Poul-Henning Kamp wrote: > -------- > In message , Dridi Boukelmoune writes: > >>Why not making libvarnish public and linking against it whenever we >>need to use it? > > That was sort of the question I tried to ask :-) In this case my answer is no, I'd rather not collapse them but instead turn libvarnish into public API and apply this change in the varnish source tree by linking instead of embedding! A long-awaited change as far as I'm concerned :) Dridi From phk at phk.freebsd.dk Mon Mar 21 22:29:43 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 21 Mar 2016 22:29:43 +0000 Subject: Collapse libvarnish and libvarnishapi In-Reply-To: References: <32884.1458218557@critter.freebsd.dk> <56EAB17D.6020203@uplex.de> <33300.1458229946@critter.freebsd.dk> <33458.1458231816@critter.freebsd.dk> <33871.1458240511@critter.freebsd.dk> Message-ID: <42794.1458599383@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >In this case my answer is no, I'd rather not collapse them but instead >turn libvarnish into public API and apply this change in the varnish >source tree by linking instead of embedding! Why don't you want to collapse them to a single API library ? Half their content is already shared... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Mon Mar 21 23:13:38 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 22 Mar 2016 00:13:38 +0100 Subject: Collapse libvarnish and libvarnishapi In-Reply-To: <42794.1458599383@critter.freebsd.dk> References: <32884.1458218557@critter.freebsd.dk> <56EAB17D.6020203@uplex.de> <33300.1458229946@critter.freebsd.dk> <33458.1458231816@critter.freebsd.dk> <33871.1458240511@critter.freebsd.dk> <42794.1458599383@critter.freebsd.dk> Message-ID: > Why don't you want to collapse them to a single API library ? Why? I believe someone coined the term bikeshedding :) More seriously, that would make varnishd (and consequently VMODs) link against the contents of the current libvarnishapi, which feels backwards at least to me (and maybe Brinch-Hansens=) If libvarnish consumers start linking against it, that means libvarnishapi will link and tools authors will automatically benefit from libvarnish's contents with no extra effort. > Half their content is already shared... They aren't really shared, libvarnishapi consumes half of libvarnish. I don't think libvarnish consumes anything but libc and pcre. But if you collapse them it should be under the name libvarnishapi to avoid breaking everyone's pkgconfig. From slink at schokola.de Fri Mar 25 15:12:29 2016 From: slink at schokola.de (Nils Goroll) Date: Fri, 25 Mar 2016 16:12:29 +0100 Subject: backend PROXY support - thought dump Message-ID: <56F5555D.6050406@schokola.de> Hi, yesterday I have written backend PROXY2 support for an ancient varnish2 fork which we still maintain - and doing so is part of the plan to get rid of it. What I have now is simply support for sub vcl_pipe { return (proxy); } which does what I need. Varnish mainline currently does have PROXY support for frontend connections, but not for backend connections. After a brief irc discussion with phk on irc, I understand that Dag is already working on changing this, so I'd like to dump some thoughts which accumulated in this context: It became apparent to me that having this feature available would facilitate clustered and layered varnish setups and remove quite a bit of additional logic regarding client IP handling: - pipe mode with PROXY would offer a particularly efficient way to load-balance from a (possibly dumb) varnish node terminating client connections to worker nodes for actual request processing. This could eliminate additional L4 loadbalancers in many scenarios. - generic backend PROXY mode would simplify code requiring client.ip in clustered and layered varnish setups. Non-pipe PROXY mode with the protocol as-it would require single-request connections and as such would be highly inefficient. Adding a per-request PROXY header would appear to be a simple solution, but in his spec http://www.haproxy.org/download/1.5/doc/proxy-protocol.txt Willy explicitly forbids such use: (http proxies) MUST use the facilities provided by this protocol to present the client's address. The reason appears simple to me: With a connection-origiented PROXY speaker in front of a request-oriented PROXY backend, a client could insert addtional PROXY headers and thus spoof client socket information. On the other hand, I do not see any reason why we shouldn't use the protocol with per-request semantics for our own HTTP purposes: A new accept socket protocol type (eg named CLUSTER) could require a PROXY2 header with every request. On irc, phk suggested that one way of getting backend PROXY support was though a vmod inserting the header on the backend connection, but with these usage scenarios in mind, I now think that we should really support configuration for PROXY mode by means of a backend protocol property: Whether a socket we connect to supports PROXY (or even CLUSTER as suggested above) is purely a property of that socket and we should avoid the need to duplicate this information in VCL. To summarize, I think we could have: - an additional accept socket protocol type to support and _require_ per-request PROXY2 headers (which could be called CLUSTER type) - support for backend protocol types PROXY and CLUSTER where - PROXY would always fail unless in pipe mode - CLUSTER would fail if in pipe mode Nils From slink at schokola.de Fri Mar 25 15:30:40 2016 From: slink at schokola.de (Nils Goroll) Date: Fri, 25 Mar 2016 16:30:40 +0100 Subject: pipe mode and sendfile? In-Reply-To: <56F5555D.6050406@schokola.de> References: <56F5555D.6050406@schokola.de> Message-ID: <56F559A0.1030102@schokola.de> btw, On 25/03/16 16:12, Nils Goroll wrote: > - pipe mode with PROXY would offer a particularly efficient way to load-balance > from a (possibly dumb) varnish node terminating client connections to worker > nodes for actual request processing. is there a particular reason for pipe mode not using sendfile half the number of syscalls required and avoid copying to/from a userspace buffer? Nils From martin at varnish-software.com Fri Mar 25 16:09:39 2016 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 25 Mar 2016 16:09:39 +0000 Subject: pipe mode and sendfile? In-Reply-To: <56F559A0.1030102@schokola.de> References: <56F5555D.6050406@schokola.de> <56F559A0.1030102@schokola.de> Message-ID: On Fri, 25 Mar 2016 at 16:35 Nils Goroll wrote: > > is there a particular reason for pipe mode not using sendfile half the > number of > syscalls required and avoid copying to/from a userspace buffer? > sendfile() requires the in_fd to be an fd supporting mmap-like operations, ie no network sockets. Martin -------------- next part -------------- An HTML attachment was scrubbed... URL: From slink at schokola.de Fri Mar 25 17:20:17 2016 From: slink at schokola.de (Nils Goroll) Date: Fri, 25 Mar 2016 18:20:17 +0100 Subject: pipe mode and sendfile? In-Reply-To: References: <56F5555D.6050406@schokola.de> <56F559A0.1030102@schokola.de> Message-ID: <56F57351.6080205@schokola.de> > sendfile() requires the in_fd to be an fd supporting mmap-like operations ouch. /me stupid, thx From perbu at varnish-software.com Tue Mar 29 08:34:58 2016 From: perbu at varnish-software.com (Per Buer) Date: Tue, 29 Mar 2016 10:34:58 +0200 Subject: pipe mode and sendfile? In-Reply-To: References: <56F5555D.6050406@schokola.de> <56F559A0.1030102@schokola.de> Message-ID: On Fri, Mar 25, 2016 at 5:09 PM, Martin Blix Grydeland < martin at varnish-software.com> wrote: > On Fri, 25 Mar 2016 at 16:35 Nils Goroll wrote: > >> >> is there a particular reason for pipe mode not using sendfile half the >> number of >> syscalls required and avoid copying to/from a userspace buffer? >> > > sendfile() requires the in_fd to be an fd supporting mmap-like operations, > ie no network sockets. > How about TCP Splicing? Shouldn't we be able to do that? If I'm not mistaken HA-Proxy used TCP Splicing for quite some time to avoid read/write copying to userspace. -- *Per Buer* CTO | Varnish Software AS Cell: +47 95839117 We Make Websites Fly! www.varnish-software.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Tue Mar 29 08:51:17 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 29 Mar 2016 08:51:17 +0000 Subject: pipe mode and sendfile? In-Reply-To: References: <56F5555D.6050406@schokola.de> <56F559A0.1030102@schokola.de> Message-ID: <84525.1459241477@critter.freebsd.dk> -------- In message , Per Buer writes: There is a very big elephant in this room: "what does 'pipe' mean in HTTP/2" One option is to make it connection based, like H1, in which case vcl_recv{} is far too late to decide it. The other option is to make it stream based, in which case vcl_recv{} will only work for streams which look like HTTP, which are probably the least useful ones to do it to. My current thinking is that we need to split vcl_recv{} into the moral equivalent of vcl_sess{} and vcl_req{}. It's tempting to keep the vcl_recv{} name for vcl_req{}, but if history is any guide we will regret that decision later on and biting the bullet is probably the right thing. So vcl_sess{} would allow you to do things with the 4 ip numbers and the protocol type (H1/H2/whatever) but you do not have a request. For H2 vcl_sess{} probably has access to the HEADERS, and if the PROXY offers it, also ALPN and client cert info. This is where you decide to pipe the entire connection. vcl_req{} is where we have a HTTP request. This only allows us to pipe per connection, but not per stream, and in particular it does not allow H1 to switch to piping after the first request. If we want per-stream piping, we need also vcl_http1_stream{} and vcl_http2_stream{} to decide that, and now it gets really messy. Do we give vcl_http1_stream{} access to req.* ? Do we give vcl_http2_stream{} access to the frame that opened the stream or do we attempt to turn it into a req.* ? And how does H2 pipe-per-stream even work in the first place ? Do we even know which shared state we have to synchronize ? So my inclination at this point is to say "pipe is per connection". If we do that, the only remaining question is if we should allow a hac^H^H^Hcat-flap so H1 can still switch to pipe in vcl_req{} ? And it definitively settles the question if pipe should always start on a new backend connection, which again, enables us to send PROXY headers if we want to. Comments ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Tue Mar 29 10:04:56 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 29 Mar 2016 12:04:56 +0200 Subject: pipe mode and sendfile? In-Reply-To: <84525.1459241477@critter.freebsd.dk> References: <56F5555D.6050406@schokola.de> <56F559A0.1030102@schokola.de> <84525.1459241477@critter.freebsd.dk> Message-ID: > If we want per-stream piping, we need also vcl_http1_stream{} and > vcl_http2_stream{} to decide that, and now it gets really messy. Wouldn't the vcl_(recv|req) already map to a stream in H2? > Do we give vcl_http1_stream{} access to req.* ? Do we give > vcl_http2_stream{} access to the frame that opened the stream or > do we attempt to turn it into a req.* ? And how does H2 pipe-per-stream > even work in the first place ? Do we even know which shared state > we have to synchronize ? I don't know how to handle non-standard methods that are currently piped by Varnish but IIRC the CONNECT method has to be piped in a stream. I think looking at how WebSocket should be handled over H2 would give us the answer and unless I have the wrong link [1] it's still an open question. Dridi [1] https://tools.ietf.org/html/draft-hirano-httpbis-websocket-over-http2-01#section-5 From dridi at varni.sh Tue Mar 29 12:21:04 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 29 Mar 2016 14:21:04 +0200 Subject: [PATCH] Teach vmod-debug how to sync with varnishtest In-Reply-To: References: Message-ID: On Mon, Mar 21, 2016 at 11:39 AM, Guillaume Quintard wrote: > Looks ok to me. I'd just like we actually send some bytes instead of > expecting to read 0 bytes in a buf[32]. > > I'm not sure we actually need 2 types of barriers, but I get the need to > reduce waste when not needed. > > Rebase and go? Hello everyone, You no longer have semaphores in master varnishtest, from now on you can use barriers instead ;) Cheers, Dridi From martin at varnish-software.com Tue Mar 29 12:46:44 2016 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 29 Mar 2016 12:46:44 +0000 Subject: pipe mode and sendfile? In-Reply-To: <84525.1459241477@critter.freebsd.dk> References: <56F5555D.6050406@schokola.de> <56F559A0.1030102@schokola.de> <84525.1459241477@critter.freebsd.dk> Message-ID: On Tue, 29 Mar 2016 at 10:51 Poul-Henning Kamp wrote: > So vcl_sess{} would allow you to do things with the 4 ip numbers > and the protocol type (H1/H2/whatever) but you do not have a request. > > So IIUC this is really an implementation of what is known as a dumb proxy? No attempt is made at looking at the contents of the stream at all, and everything is piped through to the other end including the original request even if it happened to be HTTP-alike (which it wouldn't have to be). While a dumb proxy certainly has its uses, it is quite far from what Varnish is, and there are other tools that do the job well. Complementing Varnish with this functionality could be useful, but I guess this would count for <1% of the pipe use-cases. For H2 vcl_sess{} probably has access to the HEADERS, and if the > PROXY offers it, also ALPN and client cert info. This is where you > decide to pipe the entire connection. > Here I don't really follow you. The HEADERS won't come until the first H2 stream is opened and the HEADERS frame read, and we are not talking about piping an entire connection any more. Wouldn't we then need to obviously start with a fresh connection, create a first stream using exactly the same HEADERS (if not the real client and the server would become desynched in the header compression for subsequent streams) and the start piping. While maybe this would work, it sounds dirty. And the same hack would be available to H1 (without the need to deny any header changes). I believe it's important to this discussion to remember that by my account >90% of all pipe uses in Varnish is to deal with very large content. Both from the client (large post) and from a server (.iso download), without using excessive amounts of memory / stevedore space and LRU-purging cached content in the process. We have the strangely named stevedore-bypass-for-pass changes that should take away most of the need for pipe as we know it. So getting that feature in becomes important, and would allow this discussion to take more innovative paths. Martin -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Tue Mar 29 13:06:35 2016 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Tue, 29 Mar 2016 15:06:35 +0200 Subject: pipe mode and sendfile? In-Reply-To: References: <56F5555D.6050406@schokola.de> <56F559A0.1030102@schokola.de> <84525.1459241477@critter.freebsd.dk> Message-ID: On Tue, Mar 29, 2016 at 12:04 PM, Dridi Boukelmoune wrote: > > > If we want per-stream piping, we need also vcl_http1_stream{} and > > vcl_http2_stream{} to decide that, and now it gets really messy. > > Wouldn't the vcl_(recv|req) already map to a stream in H2? I asked the question on IRC, and the headers compression popped up. But actually, the headers we care about will be in about will be in HEADERS frames. So, in case of H/2, piping would be more akin to passing with header parsing. -- Guillaume Quintard -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at varnish-software.com Tue Mar 29 13:10:51 2016 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 29 Mar 2016 13:10:51 +0000 Subject: pipe mode and sendfile? In-Reply-To: References: <56F5555D.6050406@schokola.de> <56F559A0.1030102@schokola.de> Message-ID: On Tue, 29 Mar 2016 at 10:35 Per Buer wrote: > On Fri, Mar 25, 2016 at 5:09 PM, Martin Blix Grydeland < > martin at varnish-software.com> wrote: > >> On Fri, 25 Mar 2016 at 16:35 Nils Goroll wrote: >> >>> >>> is there a particular reason for pipe mode not using sendfile half the >>> number of >>> syscalls required and avoid copying to/from a userspace buffer? >>> >> >> sendfile() requires the in_fd to be an fd supporting mmap-like >> operations, ie no network sockets. >> > > How about TCP Splicing? Shouldn't we be able to do that? If I'm not > mistaken HA-Proxy used TCP Splicing for quite some time to avoid read/write > copying to userspace. > TCP splicing would be an option. I've never played with it, so I'm not sure about the details. But as I understand it, the number of system calls wouldn't go down, and the number of fd's used per piped connection would increase by 4 (2 fd's per pipe, and one pipe per direction). You would avoid the userspace/kernelspace copying though. Martin -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Tue Mar 29 13:17:44 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 29 Mar 2016 15:17:44 +0200 Subject: pipe mode and sendfile? In-Reply-To: References: <56F5555D.6050406@schokola.de> <56F559A0.1030102@schokola.de> <84525.1459241477@critter.freebsd.dk> Message-ID: >> > If we want per-stream piping, we need also vcl_http1_stream{} and >> > vcl_http2_stream{} to decide that, and now it gets really messy. >> >> Wouldn't the vcl_(recv|req) already map to a stream in H2? > > I asked the question on IRC, and the headers compression popped up. But > actually, the headers we care about will be in about will be in HEADERS > frames. So, in case of H/2, piping would be more akin to passing with header > parsing. I'd even go further and say that headers compression is handled by the session and should not be a per-stream concern. Dridi From phk at phk.freebsd.dk Tue Mar 29 13:24:31 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 29 Mar 2016 13:24:31 +0000 Subject: pipe mode and sendfile? In-Reply-To: References: <56F5555D.6050406@schokola.de> <56F559A0.1030102@schokola.de> <84525.1459241477@critter.freebsd.dk> Message-ID: <29137.1459257871@critter.freebsd.dk> -------- In message , Martin Blix Grydeland writes: >While a dumb proxy certainly has its uses, it is quite far from what >Varnish is, and there are other tools that do the job well. Complementing >Varnish with this functionality could be useful, but I guess this would >count for <1% of the pipe use-cases. Part of the H2 job is to get those 99% to stop using pipe, because you cannot pipe from a H2 request to a H1 backend (or vice versa). >For H2 vcl_sess{} probably has access to the HEADERS, and if the >> PROXY offers it, also ALPN and client cert info. This is where you >> decide to pipe the entire connection. > >Here I don't really follow you. The HEADERS [...] Sorry, I meant SETTINGS. >We have the strangely named stevedore-bypass-for-pass changes that should >take away most of the need for pipe as we know it. So getting that feature >in becomes important, and would allow this discussion to take more innovative >paths. I recently added the "delete as we deliver" for the backend->client direction which was the main missing part of this, there may be some tuning left, but please test it. (One XXX: we should alloc from Transient based on huge CL from backend.) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Tue Mar 29 13:27:55 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 29 Mar 2016 13:27:55 +0000 Subject: pipe mode and sendfile? In-Reply-To: References: <56F5555D.6050406@schokola.de> <56F559A0.1030102@schokola.de> <84525.1459241477@critter.freebsd.dk> Message-ID: <29178.1459258075@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >>> > If we want per-stream piping, we need also vcl_http1_stream{} and >>> > vcl_http2_stream{} to decide that, and now it gets really messy. >>> >>> Wouldn't the vcl_(recv|req) already map to a stream in H2? >> >> I asked the question on IRC, and the headers compression popped up. But >> actually, the headers we care about will be in about will be in HEADERS >> frames. So, in case of H/2, piping would be more akin to passing with header >> parsing. > >I'd even go further and say that headers compression is handled by the >session and should not be a per-stream concern. We don't know how arbitrary FOO-over-H2 streams will compress and have no intention of learning about it. It's bad enough to pipe half the HTTP streams to one backend and the other half to the other. Think for instance about how we would merge the two backend->client HPACK state into one. So unless somebody volunteers to write that code *and* gets it past my review, "pipe" in 5.0 is per connection only... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Tue Mar 29 13:49:38 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 29 Mar 2016 15:49:38 +0200 Subject: pipe mode and sendfile? In-Reply-To: <29178.1459258075@critter.freebsd.dk> References: <56F5555D.6050406@schokola.de> <56F559A0.1030102@schokola.de> <84525.1459241477@critter.freebsd.dk> <29178.1459258075@critter.freebsd.dk> Message-ID: > We don't know how arbitrary FOO-over-H2 streams will compress and > have no intention of learning about it. I don't know about other FOOs but for WebSocket for instance, AFAIK it doesn't look possible, so I doubt we could *really* pipe anything besides CONNECT tunnels and H2 streams (and only pretend we are piping the latter). > It's bad enough to pipe half the HTTP streams to one backend and > the other half to the other. Think for instance about how we would > merge the two backend->client HPACK state into one. You're not supposed to do that, HPACK is maintained as a one-way compression between two parties and cannot be shared with third-parties. So I'm afraid we can't share HPACK tables with backends, but fortunately that makes the pipe thing a non problem wrt HPACK. When the session sees HEADER frames it does the HPACK lookups/updates before it handing them to streams. > So unless somebody volunteers to write that code *and* gets it past > my review, "pipe" in 5.0 is per connection only... We need CONNECT piping to comply with the RFC, and as I said in previous comments, I believe that's the only kind of "true" pipe we can do in H2, and it's stream-based. Or we can choose to close all streams that do a pipe transition, I believe it can be done too. Dridi From phk at phk.freebsd.dk Tue Mar 29 14:26:28 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 29 Mar 2016 14:26:28 +0000 Subject: pipe mode and sendfile? In-Reply-To: References: <56F5555D.6050406@schokola.de> <56F559A0.1030102@schokola.de> <84525.1459241477@critter.freebsd.dk> <29178.1459258075@critter.freebsd.dk> Message-ID: <56059.1459261588@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >> It's bad enough to pipe half the HTTP streams to one backend and >> the other half to the other. Think for instance about how we would >> merge the two backend->client HPACK state into one. > >You're not supposed to do that, HPACK is maintained as a one-way >compression between two parties and cannot be shared with >third-parties. That just means that we would have to decompress and recompress all the streams, in which case we might as well return(pass). >We need CONNECT piping to comply with the RFC, and as I said in >previous comments, I believe that's the only kind of "true" pipe we >can do in H2, and it's stream-based. No, we can also do transparent piping, for instance based on ACL matches and similar. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Tue Mar 29 14:43:36 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 29 Mar 2016 16:43:36 +0200 Subject: pipe mode and sendfile? In-Reply-To: <56059.1459261588@critter.freebsd.dk> References: <56F5555D.6050406@schokola.de> <56F559A0.1030102@schokola.de> <84525.1459241477@critter.freebsd.dk> <29178.1459258075@critter.freebsd.dk> <56059.1459261588@critter.freebsd.dk> Message-ID: > That just means that we would have to decompress and recompress > all the streams, in which case we might as well return(pass). Assuming that pipe would be per-connection, then saying that return(pass) would transmit raw H2 frames directly to the backend (as I understand your comment) would put you in the half pipe situation where you decode some HEADER frames but tunnel others. It's no news that H2 is not very friendly to proxies or tunneling and IMO those were afterthoughts as demonstrated by the expired draft on WS/H2. If you put a proxy between to H2 parties, then the proxy has to maintain 2 HPACK tables with both parties and maintain them as it wishes (except for the never-indexed fields). > No, we can also do transparent piping, for instance based on ACL > matches and similar. I don't understand, are we talking about something like vcl_sess -> vcl_pipe here? Dridi From phk at phk.freebsd.dk Tue Mar 29 15:00:51 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 29 Mar 2016 15:00:51 +0000 Subject: pipe mode and sendfile? In-Reply-To: References: <56F5555D.6050406@schokola.de> <56F559A0.1030102@schokola.de> <84525.1459241477@critter.freebsd.dk> <29178.1459258075@critter.freebsd.dk> <56059.1459261588@critter.freebsd.dk> Message-ID: <56165.1459263651@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >> That just means that we would have to decompress and recompress >> all the streams, in which case we might as well return(pass). > >Assuming that pipe would be per-connection, [...] This was in the pipe-per-stream situation. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Tue Mar 29 15:12:18 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 29 Mar 2016 17:12:18 +0200 Subject: pipe mode and sendfile? In-Reply-To: <56165.1459263651@critter.freebsd.dk> References: <56F5555D.6050406@schokola.de> <56F559A0.1030102@schokola.de> <84525.1459241477@critter.freebsd.dk> <29178.1459258075@critter.freebsd.dk> <56059.1459261588@critter.freebsd.dk> <56165.1459263651@critter.freebsd.dk> Message-ID: On Tue, Mar 29, 2016 at 5:00 PM, Poul-Henning Kamp wrote: > -------- > In message > , Dridi Boukelmoune writes: >>> That just means that we would have to decompress and recompress >>> all the streams, in which case we might as well return(pass). >> >>Assuming that pipe would be per-connection, [...] > > This was in the pipe-per-stream situation. Yes, in this situation, H2 blurs the line between a pipe and a pass in Varnish, making them look almost identical but not quite. The difference ultimately boils down to how the beresp is handled, including the possibility to retry or restart. Even in H2 vcl_pipe can still act as a dumb proxy and never bother you with the beresp and only be around until the stream ends. Dridi From phk at phk.freebsd.dk Tue Mar 29 15:17:15 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 29 Mar 2016 15:17:15 +0000 Subject: pipe mode and sendfile? In-Reply-To: References: <56F5555D.6050406@schokola.de> <56F559A0.1030102@schokola.de> <84525.1459241477@critter.freebsd.dk> <29178.1459258075@critter.freebsd.dk> <56059.1459261588@critter.freebsd.dk> <56165.1459263651@critter.freebsd.dk> Message-ID: <56210.1459264635@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >Yes, in this situation, H2 blurs the line between a pipe and a pass >in Varnish, making them look almost identical but not quite. Which is why I think we should make "pipe" mean "pipe the connection" and "pass" mean "pass the request" -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From slink at schokola.de Thu Mar 31 16:32:19 2016 From: slink at schokola.de (Nils Goroll) Date: Thu, 31 Mar 2016 18:32:19 +0200 Subject: [patch] "monotonic wallclock time" Message-ID: <56FD5113.9000707@schokola.de> https://github.com/varnishcache/varnish-cache/issues/1874 somehow caught my attention because years ago I made an attempt to suggest that we should use the monotonic clock almost always. So here's a suggestion for a simple interface which would allow us to still use wallclock time, but make it partially monotonic based on a timestamp: VTIM_ts_now() calculates a new wallclock time based on the monotonic clock delta. I think there are some other places where we could just use monotonic timestamps, but for the remaining timestamps this suggestion could be an option. Nils -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Add-a-simple-interface-to-make-wallclock-time-monoto.patch Type: text/x-patch Size: 19284 bytes Desc: not available URL: