From dridi at varni.sh Wed Jun 1 13:16:01 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 1 Jun 2016 15:16:01 +0200 Subject: [master] 4fadbca Rename to fit with the other thread names. In-Reply-To: References: Message-ID: On Wed, Jun 1, 2016 at 1:59 PM, Lasse Karstensen wrote: > > commit 4fadbca2b035bccebec478a2b486347e23ef4843 > Author: Lasse Karstensen > Date: Wed Jun 1 13:56:39 2016 +0200 > > Rename to fit with the other thread names. > > Before this change: > > hostname:/proc/24139$ cat task/*/comm | sort | uniq > Backend poller > ban-lurker > cache-acceptor > cache-epoll > cache-main > cache-timeout > cache-worker > hcb-cleaner > MPL_busyobj > MPL_req0 > MPL_req1 > MPL_sess0 > MPL_sess1 > pool_herder > vsm_cleaner Hi, For the sake of consistency, should we rename other threads too? What I have in mind: s/_/-/ s/MPL/mempool/ s/hcb/critbit/ (consistent with varnishd -h) Possibly others that didn't show up in Lasse's box too. Dridi From guillaume at varnish-software.com Thu Jun 2 07:06:55 2016 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 2 Jun 2016 09:06:55 +0200 Subject: [PATCH] Simplify r01941.vtc a bit Message-ID: Looks like git send-mail failed me, sending now through gmail... -- Guillaume Quintard -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Simplify-r01941.vtc-a-bit.patch Type: text/x-patch Size: 7680 bytes Desc: not available URL: From phk at phk.freebsd.dk Thu Jun 2 07:17:43 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 02 Jun 2016 07:17:43 +0000 Subject: [PATCH] Simplify r01941.vtc a bit In-Reply-To: References: Message-ID: <10162.1464851863@critter.freebsd.dk> -------- In message , Guillaume Quintard writes: >Looks like git send-mail failed me, sending now through gmail... Unless the Content-Length is important I would have used chunked encoding to avoid having to do math :-) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Fri Jun 10 12:07:22 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 10 Jun 2016 12:07:22 +0000 Subject: Paper to think about... Message-ID: <12945.1465560442@critter.freebsd.dk> http://neverworkintheory.org/2016/06/09/too-many-knobs.html Interesting... Maybe we should weed out some parameters for 5.0... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From kacperw at gmail.com Tue Jun 14 23:29:45 2016 From: kacperw at gmail.com (Kacper Wysocki) Date: Wed, 15 Jun 2016 01:29:45 +0200 Subject: Paper to think about... In-Reply-To: <12945.1465560442@critter.freebsd.dk> References: <12945.1465560442@critter.freebsd.dk> Message-ID: Only a handful params actually need changing from site to site. There are a dusin params that I've neer seen anyone use to any good effect.. in particular, most of the ones labeled "NB: We do not know yet if it is a good idea to change this" are voodoo for users. On the other hand, I was thinking how does language-as-config relate to the too-many-knobs effect? Language constructs aren't knobs per se; they let the (literate) user write their own desired system behavior, but forces them to invest in a steep learning curve. So it's either not-a-knob or infinite-knobs... On Fri, Jun 10, 2016 at 2:07 PM, Poul-Henning Kamp wrote: > http://neverworkintheory.org/2016/06/09/too-many-knobs.html > > Interesting... > > Maybe we should weed out some parameters for 5.0... > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev -- http://comotion.delta9.pl http://u.delta9.pl http://kacper.doesntexist.org Too much order is its own chaos. Employ no technique to gain supreme enlightment. From dridi at varni.sh Wed Jun 15 07:48:16 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 15 Jun 2016 09:48:16 +0200 Subject: Paper to think about... In-Reply-To: References: <12945.1465560442@critter.freebsd.dk> Message-ID: On Wed, Jun 15, 2016 at 1:29 AM, Kacper Wysocki wrote: > Only a handful params actually need changing from site to site. > There are a dusin params that I've neer seen anyone use to any good > effect.. in particular, most of the ones labeled "NB: We do not know > yet if it is a good idea to change this" are voodoo for users. In this case the varnish-cli could be changed so that experimental params don't show up by default with `param.show -l`, requiring a new flag like -x for instance. That would be one more knob, but to hide less common knobs. Experimental flags could be moved to the varnish-cli(7) manual too, in my experience most people don't read it, only really advanced users. > On the other hand, I was thinking how does language-as-config relate > to the too-many-knobs effect? Language constructs aren't knobs per se; > they let the (literate) user write their own desired system behavior, > but forces them to invest in a steep learning curve. So it's either > not-a-knob or infinite-knobs... One could argue that there are too many steps in VCL. For instance, do we really need a `sub vcl_fini{}` when VMODs could always be responsible for cleaning up state? I don't know, but haven't seen it used so far. The two knobs I'd remove from VCL are /* C style */ and // C++ style comments. The remaining one would be the same between vcl and vtc files and has a scripting language feel that fits better IMHO. Dridi From slink at schokola.de Wed Jun 15 08:35:55 2016 From: slink at schokola.de (Nils Goroll) Date: Wed, 15 Jun 2016 10:35:55 +0200 Subject: Paper to think about... In-Reply-To: References: <12945.1465560442@critter.freebsd.dk> Message-ID: <5761136B.7090108@schokola.de> On 15/06/16 09:48, Dridi Boukelmoune wrote: > varnish-cli could be changed so that experimental params > don't show up by default with `param.show -l`, requiring a new flag like -x for > instance. I like this idea. So we'd have - user tunables - expert tunables - param.show : show user tunables and non-default expert tunables - param.show -x : show all - same for -l, but full info Having the knobs can safe your live for edge cases, and not having them built in helps us (devs + poweruser) find the right values - so removing them does not sound like a good idea to me. Nils From guillaume at varnish-software.com Wed Jun 15 08:50:07 2016 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Wed, 15 Jun 2016 10:50:07 +0200 Subject: Paper to think about... In-Reply-To: <5761136B.7090108@schokola.de> References: <12945.1465560442@critter.freebsd.dk> <5761136B.7090108@schokola.de> Message-ID: On Wed, Jun 15, 2016 at 10:35 AM, Nils Goroll wrote: > On 15/06/16 09:48, Dridi Boukelmoune wrote: > > varnish-cli could be changed so that experimental params > > don't show up by default with `param.show -l`, requiring a new flag like > -x for > > instance. > > People tend to think the expert stuff is more powerful, and that they are entitled to it. I would drop the ones labeled "NB: We do not know..." and only enabled them with a compile flag and make it very clear that the plan is to drop them unless there's a relevant use case. I vote against the comment changes, we would unnecessarily break VCL for very little gain. -- Guillaume Quintard -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at uplex.de Wed Jun 15 09:31:53 2016 From: geoff at uplex.de (Geoff Simmons) Date: Wed, 15 Jun 2016 11:31:53 +0200 Subject: Paper to think about... In-Reply-To: References: <12945.1465560442@critter.freebsd.dk> <5761136B.7090108@schokola.de> Message-ID: <57612089.1040102@uplex.de> On 15/06/16 10:50, Guillaume Quintard wrote: > > On 15/06/16 09:48, Dridi Boukelmoune wrote: > > varnish-cli could be changed so that experimental params > > don't show up by default with `param.show -l`, requiring a new flag like -x for > > instance. > > People tend to think the expert stuff is more powerful, and that they > are entitled to it. > > I would drop the ones labeled "NB: We do not know..." and only enabled > them with a compile flag and make it very clear that the plan is to drop > them unless there's a relevant use case. I vote for the classification into expert- and non-expert-level params, and very strongly against removing any of them for no other reason than having fewer of them. My experience with this sort of thing is that we can all agree that there can fewer config options, and that we will never agree on which of them, specifically, can be eliminated. I can name many params whose relevance seems inconceivable, to me, but to many of you, the idea of removing some of my candidates will be horrifying. You'll think I'm out of my mind. However we go about it, for someone somewhere the impact will be very painful. "NB: We do not know..." means just that -- we don't know. Probably the vast majority of those have little impact on how Varnish works, but I'll bet a beer at VDD that some of us have discovered a few that can make a real difference. If we really mean it when we say we don't know, then a blanket assumption that they all can be removed is clearly not warranted. Version changes usually mean that certain params really do become obsolete, that's normal. Documenting the typically important ones should go a long way toward helping less experienced users. But swinging an axe just to have fewer params is IMO the road to madness. > I vote against the comment changes, we would unnecessarily break VCL for > very little gain. Agreed. CU in AMS, Geoff -- UPLEX Systemoptimierung Scheffelstra?e 32 22301 Hamburg http://uplex.de/ Mob: +49-176-63690917 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: OpenPGP digital signature URL: From guillaume at varnish-software.com Wed Jun 15 12:25:58 2016 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Wed, 15 Jun 2016 14:25:58 +0200 Subject: Tweakable madvise Message-ID: I re-read some performance concerns about the file storage and remembered that I said I'd send a patch, probably a year ago. So here we go. The only thing it does is exposing the madvise parameter to the user, in practice, it helps performance for big objects and/or rotational disk when using MADV_SEQUENTIAL on Linux. -- Guillaume Quintard -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Tunable-madvise-for-file-storage.patch Type: text/x-patch Size: 4036 bytes Desc: not available URL: From perbu at varnish-software.com Wed Jun 15 16:24:14 2016 From: perbu at varnish-software.com (Per Buer) Date: Wed, 15 Jun 2016 18:24:14 +0200 Subject: Paper to think about... In-Reply-To: <57612089.1040102@uplex.de> References: <12945.1465560442@critter.freebsd.dk> <5761136B.7090108@schokola.de> <57612089.1040102@uplex.de> Message-ID: On Wed, Jun 15, 2016 at 11:31 AM, Geoff Simmons wrote: > I vote for the classification into expert- and non-expert-level params, > and very strongly against removing any of them for no other reason than > having fewer of them. > How do you feel about making a few more of them compile time options? This would reduce the apparent number of parameters that will confuse the bit-twidlers, shorten the docs and hopefully make people invest more time and effort in figuring out the remaining parameters. -- *Per Buer* CTO | Varnish Software AS Cell: +47 95839117 We Make Websites Fly! www.varnish-software.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Wed Jun 15 16:35:17 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 15 Jun 2016 18:35:17 +0200 Subject: Paper to think about... In-Reply-To: References: <12945.1465560442@critter.freebsd.dk> <5761136B.7090108@schokola.de> <57612089.1040102@uplex.de> Message-ID: >> I vote for the classification into expert- and non-expert-level params, >> and very strongly against removing any of them for no other reason than >> having fewer of them. > > > How do you feel about making a few more of them compile time options? This would reduce the apparent number of parameters that will confuse the bit-twidlers, shorten the docs and hopefully make people invest more time and effort in figuring out the remaining parameters. The great thing about `param.set` is that we can do it at run time. If we take the very extreme end of the expert spectrum, one can have auto-tuning in place depending on monitoring of some sort. >From a less extreme PoV, tweaking a parameter would mean having to recompile, reinstall and restart Varnish. Tweaking becomes a show stopper if that means such long feedback cycles. It may take time to put the system back into a measurable state, and it means that you can't safely tune your production without losing your cache for every single tweak. Dridi From geoff at uplex.de Thu Jun 16 08:09:20 2016 From: geoff at uplex.de (Geoff Simmons) Date: Thu, 16 Jun 2016 10:09:20 +0200 Subject: params in use (was Re: Paper to think about...) In-Reply-To: References: <12945.1465560442@critter.freebsd.dk> <5761136B.7090108@schokola.de> <57612089.1040102@uplex.de>

Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hello all, As phk asked for at VDD, here are varnishd params that I know of being set in production. This is for 4.0.3, so some of them are no longer in use for 4.1.2 and master. - -p waiter - -p vcl_dir - -p feature (to disable esi xml check) - -p vcc_err_unref=off - -p shm_reclen - -p thread_pools - -p thread_pool_min - -p thread_pool_max - -p max_esi_depth - -p workspace_client - -p workspace_backend - -p send_timeout - -p connect_timeout - -p first_byte_timeout - -p between_bytes_timeout - -p cli_timeout - -p pipe_timeout - -p idle_send_timeout - -p thread_pool_timeout - -p max_restarts - -p max_retries - -p vsl_mask - -p vsl_buffer - -p syslog_cli_traffic Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCAAGBQJXYl6kAAoJEOUwvh9pJNURZqEQAIDzAJ0ZqFoyBxL4aNQgcsz1 gtpRv0jAX7gnCcbHFg6fx0PaLK7acJOylZbPRC9Ny15PKusoZWxKXAC48QMNjtbK qKRxT628dpPBMJWuOeC8dH3F5NHPeOc7FbxBD8BPF4rqPdbwd1ZiwP4B+/Ipr9LR paZQ0HE9CFryhVV7ZqZTFRmx/6ozVfkkx5XohoAKv/Fs5L3oSVglhVpl8tQrWY5H 6Dx6t4LwZxNOSx1YaVkSBGbt5WoR+uCsfpp28/UHLuKrO69q3AXU2BaEKpLIEBG4 cWklOLqJSQvt46UroOgTbyHkVQByd7FTcGGROqQuv5+71lk7IIn1f+KJSdkuOB3l TU8QTE3Ucmf9tgPCBHpKI9GiejwhFYMob+5IP+Z0fdyC1y5VnQ+fE3oagC3nJM1d cUyT+aNyHwDSTNZyvfFQMMXvbEf+r9s1gQPZGIgLHahwYX/UNF34GMHnCXJC46z6 H8C9ePVdYEltjxY7jYyp08fkC1TcWCvE/K0VzfZcB3Wcdg1kVS1iEY8WaKU2lmHs a4rf917BsiOvd8lygd/c+Pmtg5OOtsqu5AwciitBNJ0gGLhos53VXfamk2RyOEJK r2Oqa0nsr7SSGazasUxehulLo2WiQlYWdWoQzyCKpxZYdaSVi5TsQHLJ2osZC8DX F5tIG7MvkYLwkjAfBZRl =CUQu -----END PGP SIGNATURE----- From phk at phk.freebsd.dk Sat Jun 18 19:26:35 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 18 Jun 2016 19:26:35 +0000 Subject: Paper to think about... In-Reply-To: References: <12945.1465560442@critter.freebsd.dk> <5761136B.7090108@schokola.de> <57612089.1040102@uplex.de> Message-ID: <81468.1466277995@critter.freebsd.dk> -------- In message , Per Buer writes: >How do you feel about making a few more of them compile time options? There is not really any point in making them compile time options if we just as easily can make them runtime options. In particular the infrastructure for documenting param.set is much better than for compile time params. The VDD sort-of-consensus was to have two or maybe three levels of param.* * no options: workspaces, timeouts, thread-pool + other often used params preferably not enough to scroll out of typical windows * -d debug/developer params * -something (-x for expert ?) all non-wizard params * -a all params -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Thu Jun 23 09:36:27 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 23 Jun 2016 11:36:27 +0200 Subject: Paper to think about... In-Reply-To: <81468.1466277995@critter.freebsd.dk> References: <12945.1465560442@critter.freebsd.dk> <5761136B.7090108@schokola.de> <57612089.1040102@uplex.de> <81468.1466277995@critter.freebsd.dk> Message-ID: > The VDD sort-of-consensus was to have two or maybe three levels of > param.* On the #varnish channel today someone was wondering whether Varnish cleared a panic report that never existed, because sigsegv_handler was off. It is now on by default, which is good because it's one of the must restart parameters. If you're still looking for knobs to retire, this one is a good candidate. It doesn't strike me as useful to omit panic reports. Dridi From dridi at varni.sh Thu Jun 23 15:49:30 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 23 Jun 2016 17:49:30 +0200 Subject: VTAILQ_EMPTY vs -Wparentheses-equality with clang In-Reply-To: <52076.1459781842@critter.freebsd.dk> References: <52076.1459781842@critter.freebsd.dk> Message-ID: On Mon, Apr 4, 2016 at 4:57 PM, Poul-Henning Kamp wrote: > -------- > In message > , Federico Schwindt writes: > >>Weird. I'm using clang 3.7 and I'm not seeing any warnings. > > I havn't compared vqueue.h to FreeBSD's sys/queue.h, but since > FreeBSD is on Clain 3.8 now, that would be a place to look... I just tried running an autogen.des'ed build with clang 3.8 (Fedora 24) and I still get the same problem. Dridi From fgsch at lodoss.net Thu Jun 23 15:57:02 2016 From: fgsch at lodoss.net (Federico Schwindt) Date: Thu, 23 Jun 2016 16:57:02 +0100 Subject: VTAILQ_EMPTY vs -Wparentheses-equality with clang In-Reply-To: References: <52076.1459781842@critter.freebsd.dk> Message-ID: clang-3.8: clang version 3.8.0-2 (tags/RELEASE_380/final) No issues. Looks like it's Fedora specific. On Thu, Jun 23, 2016 at 4:49 PM, Dridi Boukelmoune wrote: > On Mon, Apr 4, 2016 at 4:57 PM, Poul-Henning Kamp > wrote: > > -------- > > In message < > CAJV_h0bcW6MUNHbc8Qy4emHcrFfdvsGiPGSkoUSTQpXXrA64_g at mail.gmail.com> > > , Federico Schwindt writes: > > > >>Weird. I'm using clang 3.7 and I'm not seeing any warnings. > > > > I havn't compared vqueue.h to FreeBSD's sys/queue.h, but since > > FreeBSD is on Clain 3.8 now, that would be a place to look... > > I just tried running an autogen.des'ed build with clang 3.8 (Fedora > 24) and I still get the same problem. > > Dridi > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Thu Jun 23 16:05:40 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 23 Jun 2016 18:05:40 +0200 Subject: VTAILQ_EMPTY vs -Wparentheses-equality with clang In-Reply-To: References: <52076.1459781842@critter.freebsd.dk>

Message-ID: On Thu, Jun 23, 2016 at 5:57 PM, Federico Schwindt wrote: > clang-3.8: clang version 3.8.0-2 (tags/RELEASE_380/final) > > No issues. Looks like it's Fedora specific. I'll assume good faith although Fedora has an official bias towards GCC and file a bug there. From phk at phk.freebsd.dk Mon Jun 27 08:47:37 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 27 Jun 2016 08:47:37 +0000 Subject: Tweakable madvise In-Reply-To: References: Message-ID: <76239.1467017257@critter.freebsd.dk> -------- In message , Guillaume Quintard writes: >I re-read some performance concerns about the file storage and remembered >that I said I'd send a patch, probably a year ago. So here we go. Uhm, can you make this a pull request on github please ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From guillaume at varnish-software.com Mon Jun 27 15:39:22 2016 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Mon, 27 Jun 2016 08:39:22 -0700 Subject: Tweakable madvise In-Reply-To: <76239.1467017257@critter.freebsd.dk> References: <76239.1467017257@critter.freebsd.dk> Message-ID: Sure, old habits die hard. On Jun 27, 2016 01:47, "Poul-Henning Kamp" wrote: > -------- > In message < > CAJ6ZYQyjuCsnnAL6J0NygCNeBvQ9VAccWDk8C2OJYSe6awLKdA at mail.gmail.com> > , Guillaume Quintard writes: > > >I re-read some performance concerns about the file storage and remembered > >that I said I'd send a patch, probably a year ago. So here we go. > > Uhm, can you make this a pull request on github please ? > > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From slink at schokola.de Mon Jun 27 17:05:00 2016 From: slink at schokola.de (Nils Goroll) Date: Mon, 27 Jun 2016 19:05:00 +0200 Subject: Tweakable madvise In-Reply-To: References: Message-ID: <2c4c8ac7-fffc-d001-fe67-ee6d70d7fa06@schokola.de> Allowing to set MADV_DONTDUMP would help make coredumps on systems with huge storage more manageable