From geoff at uplex.de Sun Nov 1 11:23:43 2015 From: geoff at uplex.de (Geoff Simmons) Date: Sun, 01 Nov 2015 12:23:43 +0100 Subject: [PATCH] add VCL_HasBackend() (Re: Preventing dup backend names with dynamic backends in VMODs) In-Reply-To: <74085.1446020238@critter.freebsd.dk> References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> Message-ID: <5635F63F.1060109@uplex.de> On 10/28/2015 09:17 AM, Poul-Henning Kamp wrote: > >>> My current thinking is that we'll name the backend whatever the >>> user/vcl/vmod writer likes (ie: Backend name), and deal with the >>> fall-out. >> >> At the very least, VMODs should have a way to discover if a backend >> name is already in use by a VCL -- the VCL_HasBackend(vcl,name) idea. >> Then a VMOD can choose to raise an error for duplicate names. > > Agreed. This patch adds VCL_HasBackend(). Best, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-add-VCL_HasBackend.patch Type: text/x-patch Size: 3776 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From geoff at uplex.de Sun Nov 1 11:30:22 2015 From: geoff at uplex.de (Geoff Simmons) Date: Sun, 01 Nov 2015 12:30:22 +0100 Subject: [PATCH] add VCL_HasBackend() (Re: Preventing dup backend names with dynamic backends in VMODs) In-Reply-To: <5635F63F.1060109@uplex.de> References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <5635F63F.1060109@uplex.de> Message-ID: <5635F7CE.6000902@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 11/01/2015 12:23 PM, Geoff Simmons wrote: > > This patch adds VCL_HasBackend(). Forgot to mention about this -- it might be necessary to lock the vcl_mtx while traversing the vcl->backend_list, since a backend may be added while the query is executing. - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCAAGBQJWNffOAAoJEOUwvh9pJNURnDQP/ieNXfZGyY4BUg1zVYi8jIh+ aRrn9Fs2hK5XPENxw9bgqVzXAH1FO6H7D5ZSbnSyW0WmqucOROsM4e6WVB8VFVgu UXVornIF3PNNbq+3fsRIkBW0PLo6N0cJjGf9cUY19zjRaNFe0MG0Jta4iAWnAfAY mVts32OGiHQjmce/rU5sJLDWsQKcWZWfrobH83fC+lnLwCJj7FXJrlerffShXlIn P1kl+pxLYJv/O1nPaqeLpf4KZXiCFX+KAMIYLndfj80qrU/GF38/xKRe4DTfP6F/ 7xurYO3TL8c3E4NSP6c+bVk1MsX8IIXeQ5BS9R/U4EHgQOwXwSOWCEVzeLEC38ih 0X4mQEMx7L5xEmhfJw1bfJMHzRpUoLAxuF1vYeV+6QWNibBbQN4rqQpn8phd9nJN NphN6fRxzoxQF9mcCAu3oE8pvD7YPSYRTNl31+QcaDgc7+/Le8JTkKoX6BCTKafX RoMJbsEyYjP5lYLskhxLP98prFB2j5G1f0VrByv4BpH/UZNkCcb27LJlOtO+BQMO qZDNG+FKkD+HO4g7wGicGHymedx/pwZGcV+JlrSyejCR4fo3ZvaXheXMEcVjruv0 KOP6PgdaxgkN9sO3zBxvD9hYzxwcp2GpfgiTuNH3bxf6/zhpqHl6JGII0LSqPqiL wVrE6qrmRH+zu/9hG8q+ =BrxH -----END PGP SIGNATURE----- From geoff at uplex.de Sun Nov 1 14:58:13 2015 From: geoff at uplex.de (Geoff Simmons) Date: Sun, 01 Nov 2015 15:58:13 +0100 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: <74085.1446020238@critter.freebsd.dk> References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> Message-ID: <56362885.1000700@uplex.de> On 10/28/2015 09:17 AM, Poul-Henning Kamp wrote: > >> Maybe add a sentence or two to the "Writing a Director" doc about >> backend names, what they're used for, and why this could be a problem. > > Also a good idea. > > I'm starting to think that maybe we should have a "name-spaces" > section in the reference manual for stuff like this. This patch just adds a section to the "Writing a Director" doc about backend names, and it turned out to be several paragraphs, maybe too long-winded. Suggestions for editing are welcome. Or we could move it out to a more comprehensive doc about namespaces, but that will take some work (and I'm not sure what all it would have to cover). Best, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-add-documentation-about-backend-naming-for-VMOD-auth.patch Type: text/x-patch Size: 3846 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From geoff at uplex.de Sun Nov 1 17:06:18 2015 From: geoff at uplex.de (Geoff Simmons) Date: Sun, 01 Nov 2015 18:06:18 +0100 Subject: Preventing dup backend names with dynamic backends in VMODs In-Reply-To: References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> Message-ID: <5636468A.6060300@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 10/30/2015 05:19 PM, Kacper Wysocki wrote: > > slightly related to this, could we possibly allow accessing the > name in VCL, like > > if(req.backend_hint ~ "zool") { /* do something */ } If you can wait until vcl_backend_response(), you do have beresp.backend.name as a string, and this will work: if (beresp.backend.name ~ "zool") { /* do something */ } > or even > > std.log("backend was " + req.backend_hint); ? That works, try it. Evidently because a STRING_LIST is required in std.log(), and VCC knows to do implicit string conversions for the various data types -- it even works without the concatenation. A regex match against req.backend_hint or bereq.backend doesn't work because a STRING is required. IIUC, VCC does implicit string conversions where a STRING_LIST is expected, but not where a STRING is expected. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCAAGBQJWNkaKAAoJEOUwvh9pJNURxzEP/AuAvXWe0mOYh7XhGfPN25dU 6/EcBRCLbRhG+SXiBiv2rKl8iyU8O8zlH/tLbBsXalg+KMUIgB+qks7Qvk7XU8Y1 2/N4xDW9mwm3e9I1nYuBs2wEbTUmkfXgi4D1TXYHL6PUGZ64oj+px3sntDKHV5Xu jNIpMW3S4/O91csoO/2ittab8H6S5ue3R690ncVMaJCiIm3zyfwNO73Yy7fsivcn m8IXFI7aW1ER0gO9fFso9zthYjoqhsDGZ9BcUQ1RK7uq9YKrUAEexxPHHNaizLk3 vv37m8dgYMu8YSjX8mB4r2sag+SCuwkmSrxvSsuVG5vS/YvsVm6rupSPl3sHP8ix Xu48LEPVNGTySDRVY+oeo0Eqi3R71FUqoe1UFpMGsgxEXUFgRPSpwZG8dl4pa+wc jdOLheP9NkKzUDSd8npidrcn7jEysHXFD9OeiNoUwQpjDhSGtagV/wyU+A/xs8y3 SQuATofqjv4+PhblF90dvEepVZzJBLqrQ4BMRx9hHzHwwtHZOFQhUIMxr2tPAmYa pxq1SpfjS+RuJzsw3Gi5DkyU0XOKcZ1QZg01bNZWdKesijy7mXcHrrzRQq/HQ9Ee KM+Y8ZrF+InSne/YtmCVJFpXG+CncfRkFRSgoYnPZSzpIaVJ1w11cJkP41rlEFE2 BzbJJwNrpDuv2BikqW7y =F6by -----END PGP SIGNATURE----- From dridi at varni.sh Sun Nov 1 18:01:46 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Sun, 1 Nov 2015 19:01:46 +0100 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: <56362885.1000700@uplex.de> References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <56362885.1000700@uplex.de> Message-ID: > This patch just adds a section to the "Writing a Director" doc about > backend names, and it turned out to be several paragraphs, maybe too > long-winded. Suggestions for editing are welcome. Didn't have time to join the discussion earlier, but your patch looks great! > Or we could move it out to a more comprehensive doc about namespaces, > but that will take some work (and I'm not sure what all it would have to > cover). We had sort-of started a discussion on IRC at some point about that, but there's no way AFAIK to enforce namespaces. This is why in this doc I suggest to back backends with a VMOD object, in order to get a vcl_name from the compiler and reuse it. For instance in vmod-named I name backends "()" like in the old 4.0 days. With this snippet: new mydir = named.director("myhost"); You get "myvcl.mydir()" full names (uniqueness on ipv4 and ipv6 addresses). I rely on Varnish for a 1st level of namespace (myvcl.mydir) and then I have my own 2nd level. I hope that helps clarify my initial intent suggesting VMOD objects for directors in the docs. Cheers, Dridi From geoff at uplex.de Sun Nov 1 21:04:28 2015 From: geoff at uplex.de (Geoff Simmons) Date: Sun, 01 Nov 2015 22:04:28 +0100 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <56362885.1000700@uplex.de> Message-ID: <56367E5C.1070603@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 11/01/2015 07:01 PM, Dridi Boukelmoune wrote: > > We had sort-of started a discussion on IRC at some point about > that, but there's no way AFAIK to enforce namespaces. This is why > in this doc I suggest to back backends with a VMOD object, in order > to get a vcl_name from the compiler and reuse it. I should have pointed out that the doc-patch mentions the function VCL_HasBackend(), which would come from another patch earlier in the thread. So that part of the new docs should come if and when the code patch is also added. VCL_HasBackend() tells you if a backend name is already in use by a VCL. The one object/one backend solution can help you with that as well, but it depends on what you want your VMOD to do. I have a VMOD in the works for creating new backends at runtime, and that won't work by creating new objects for every backend, since objects can only be created in vcl_init. But it does have something similar to what you said, Dridi -- the VMOD uses PRIV_VCL arguments to keep track of the backends that it created, and then you can get the backends back (for req.backend_hint etc.) with a function by_name("backend_name") that returns VCL_BACKEND. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCAAGBQJWNn5cAAoJEOUwvh9pJNURGNkP/RtLN4yfHqoKfAfMHOU8FRvp 86l/avbcrfaLEoUFBs3TJrhPQboX0Q0/4HovZJbpg4qVJTIoAixKL3f+P9Pwgd4h EbLxSeQUvn+R2tc7e1Y/6+2OVsBC4gsyx/Sy/9+TJZfAZRth7XDM+X8YoPyQj/8R hOdgQQvQ9xq4IApFtb0BFqkKA6grtNdkvCUrD2RiVpQcQ8yhSME//POBE3rLNRaZ U2dfVJtLko3nULh4M1G2NGkBXcrFdPGE7ai6kB9XkB7X7JwwfZYbFu2j/BhvkgiZ CLGGbMCxkeDy/963fBOxEPkcxKFcTb978yeEkolIm9CfJnLru5P1wtklAjXJekRW eIU6gSz+5G7Azo3jQ1YLuPKUgyxH4zmhJysDwvNkEz4AqwwfP4VMcAlXco84ecv+ D9iq/iEokqNYWvMip4fZK5+sgaK82Qm2eGeMN9K4JocudoHGvUBWoIbTaGfbYK/B +sywHTz7l+sbZ8g1caL3Iyb4BjT1wzB6eUtxEIsivv8pegYZ8DsTkvxqRe6H7F4O BNKBMBcLUYWxKG4d9q5EaikthMkkw1QWCNrC6/q7zBjcEoJowznJpTPV8vb4Qdx8 kgXYTrKMimpVnju5Ofo62nfFw5m/1HcNC6/JNg4+GMeojJ56TBnDrNrD0vC3W1Rz mR8KkdyPNq3/FTCyylAs =b/q3 -----END PGP SIGNATURE----- From phk at phk.freebsd.dk Sun Nov 1 21:49:26 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sun, 01 Nov 2015 21:49:26 +0000 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <56362885.1000700@uplex.de> Message-ID: <52334.1446414566@critter.freebsd.dk> -------- In message , Dridi Bouke lmoune writes: >> Or we could move it out to a more comprehensive doc about namespaces, >> but that will take some work (and I'm not sure what all it would have to >> cover). > >We had sort-of started a discussion on IRC at some point about that, >but there's no way AFAIK to enforce namespaces. Without being to philosophical about it, enforcing namespaces is always a bad idea, unless that is the only way to know which namespace you are dealing with. A good example of this is towns with a namesuffix which identifies them as towns (Karlstadt, New York City, R?dby) because there might be confusion otherwise, but we don't insist that all towns have such suffixes. But I think we will have to think about and document objects and their names and lifetimes in VMOD context RSN, because it's getting complicated. The VMOD_PRIV thing ties heavily into this, (and maybe we should really create backends as VMOD_PRIV's to unify these concerns.) But that reminds me: What was the consensus on my proposal for .%d suffix for colliding backend names ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Sun Nov 1 22:20:08 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Sun, 1 Nov 2015 23:20:08 +0100 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: <52334.1446414566@critter.freebsd.dk> References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> Message-ID: > The VMOD_PRIV thing ties heavily into this, (and maybe we should > really create backends as VMOD_PRIV's to unify these concerns.) I considered using a PRIV_VCL pointer, but that wouldn't give me a "reserved" symbol (vcl_name) like a VMOD object does. And if you use a VMOD object, you already have a struct to manage state and don't need a private pointer. > But that reminds me: What was the consensus on my proposal for .%d suffix > for colliding backend names ? I didn't follow the discussions, I only saw that something was going on with dynamic backends. But adding a suffix to the backend name doesn't seem like a good idea IMHO. I think it would be confusing for end users to deal with dupes and figuring out what's going on. I'd rather shift the responsibility to VMOD writers to follow the POLA and have Varnish enforce rules such as not having two backends with the same name. I don't care how it's handled (panic or return NULL) but I think we shouldn't have some magic behavior. Just like Varnish will panic if a backend is created on a cold VCL. Best, Dridi From phk at phk.freebsd.dk Sun Nov 1 23:42:27 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sun, 01 Nov 2015 23:42:27 +0000 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> Message-ID: <52611.1446421347@critter.freebsd.dk> -------- In message , Dridi Bouke lmoune writes: >> But that reminds me: What was the consensus on my proposal for .%d suffix >> for colliding backend names ? > >I didn't follow the discussions, I only saw that something was going >on with dynamic backends. But adding a suffix to the backend name >doesn't seem like a good idea IMHO. I think it would be confusing for >end users to deal with dupes and figuring out what's going on. > >I'd rather shift the responsibility to VMOD writers to follow the POLA >and have Varnish enforce rules such as not having two backends with >the same name. I don't care how it's handled (panic or return NULL) >but I think we shouldn't have some magic behavior. So what if two different VMODs both try to create a backend with the same name ? In my view, the "magic behaviour" is to fail randomly for reasons the VMOD writer has no control over... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From geoff at uplex.de Mon Nov 2 00:32:31 2015 From: geoff at uplex.de (Geoff Simmons) Date: Mon, 02 Nov 2015 01:32:31 +0100 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: <52611.1446421347@critter.freebsd.dk> References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> <52611.1446421347@critter.freebsd.dk> Message-ID: <5636AF1F.3010907@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 11/02/2015 12:42 AM, Poul-Henning Kamp wrote: > >>> But that reminds me: What was the consensus on my proposal for >>> .%d suffix for colliding backend names ? The previous conversation was just you and me talking, so I don't know if we could call anything "consensus", but one point was that with the suffix, VBE stats would exist for every backend. Otherwise, VBE stats are only set up for one backend with a duplicated name, and all other backends with the same name get no stats at all. But on further consideration, could we always ensure that any backend created by a VMOD might get the suffix if it needs to? The message of the "Writing a Director" doc is that a VMOD creates a struct director, never a struct backend. If you want it to be a backend, then you leave the resolve function as NULL. An easy way to do that is to use VRT_new_backend, which in turn calls VCL_AddBackend, and that's where the suffix can be added if necessary. But you don't have to do it that way. A VMOD author can set up everything in a struct director, implementing all of the functions in the interface (and connection pools, probes, stats, a listener for the event function, the works; all of which are necessary for backends that don't speak HTTP over TCP). Then these objects can be used for everything you can do with VCL_BACKEND -- add it to a director, assign it to req.backend_hint etc. -- with no other part of Varnish intervening. In that case, the VMOD assigns a name to the vcl_name field; and it can assign a suffix if it wants to, but I don't see any way to force it. So unless we add a requirement like "you have to call VCL_Something() to 'register' the director/backend", so that VCL_Something() can change the name, I don't think we can use suffixes or anything else to enforce unique names. (And as of now, I don't see why a VMOD backend wouldn't work even if it doesn't call VCL_Something().) Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCAAGBQJWNq8UAAoJEOUwvh9pJNURFNgP/1unx26k70cVmIEJ3Kh4oR1g a0Aaevbp6gHBkPimlylNKSGHXGnNxzqHKKGT0dqhH6TjExGoH2CeBa+0J0uCRqN/ HMDEbq5wrk1zindR5dSPxoOl5X3YGXZ60sXSeHnNNBYy+VqvhvCctEEsFAL1Ri6k +7SG+0QLF8XRm/cDW7lcLTlyzN5q3Z0cAEeVJ2ffMGIJQCsOG3/k2ilXI6ZXjohC ePeyu5MsLhV7n3t59y/wwkhmDWV3RwUQUJDZBxprmncp/1CHrlkCSIws04Qej4kH nFFKM7eH/GRHKzXKC28ctvMDwTpzROGKsRUVGEMif9+QGoCmWSe2S/E9doFiV5UH KRSnVI8kAwIYALcy6SlPBnjb1a1aMGKpz6gnojUP3nWuDJd4nzsT/CgDfelkIcjS ItzA79SPyJAp5I+OYUIEtKgImWEhP8vg5ojPLg9O43Z6PS4AQ1gVgK4boMlpDuam u0p1PRTg4kxfWeoSd7hshZSCtEf0mcGn8F2MjtdapUExbOemyDmHyKSEEzvfxMof tg9PFVQq+EpuWU2O1RvHyJuqHE6XNxVl2oGfnS/J1Sr+XCr07HI09BmxG+aLE5lP I9LedqxdBTnZzjaz5OK4dSfNlRE/9fHWC3SzVpiAFiA47H0aDqxZTKHYJtN2yFMu yQSijHqnQDi0rwQ2sNug =qGV2 -----END PGP SIGNATURE----- From dridi at varni.sh Mon Nov 2 07:43:29 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 2 Nov 2015 08:43:29 +0100 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: <52611.1446421347@critter.freebsd.dk> References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> <52611.1446421347@critter.freebsd.dk> Message-ID: > So what if two different VMODs both try to create a backend with the > same name ? If both VMODs rely on a vcl_name to name their backends (myvcl.mydir) then they can't get in this situation. That's my whole point. > In my view, the "magic behaviour" is to fail randomly for reasons the > VMOD writer has no control over... In case of a panic, you get a little clue: the VMOD file in the backtrace. If VRT_new_backend returns NULL, the VMOD can log something useful before panicking or not panic at all (if that makes any sense in the VMOD's behavior). Or again, if Varnish panics, but provides the VCL_HasBackend function (shouldn't it be in VRT instead?) the VMOD author can also do this check and log a meaningful error. Best, Dridi From phk at phk.freebsd.dk Mon Nov 2 07:52:11 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 02 Nov 2015 07:52:11 +0000 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> <52611.1446421347@critter.freebsd.dk> Message-ID: <54220.1446450731@critter.freebsd.dk> -------- In message , Dridi Bouke lmoune writes: >> So what if two different VMODs both try to create a backend with the >> same name ? > >If both VMODs rely on a vcl_name to name their backends (myvcl.mydir) >then they can't get in this situation. That's my whole point. And in a VCL which uses two different VMODs which create dynamic backends ? >> In my view, the "magic behaviour" is to fail randomly for reasons the >> VMOD writer has no control over... > >In case of a panic, you get a little clue: the VMOD file in the backtrace. No matter what, I don't think this is panic material, unless the VMOD writer feels that way about it. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Mon Nov 2 08:03:26 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 02 Nov 2015 08:03:26 +0000 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: <5636AF1F.3010907@uplex.de> References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> <52611.1446421347@critter.freebsd.dk> <5636AF1F.3010907@uplex.de> Message-ID: <54274.1446451406@critter.freebsd.dk> -------- In message <5636AF1F.3010907 at uplex.de>, Geoff Simmons writes: >>>> But that reminds me: What was the consensus on my proposal for >>>> .%d suffix for colliding backend names ? > >The previous conversation was just you and me talking, so I don't know >if we could call anything "consensus", but one point was that with the >suffix, VBE stats would exist for every backend. Otherwise, VBE stats >are only set up for one backend with a duplicated name, and all other >backends with the same name get no stats at all. also: CLI commands to set sick/healty etc. >But you don't have to do it that way. A VMOD author can set up >everything in a struct director, implementing all of the functions in >the interface [...] Sure. And it can add it to the data structures to appear in the CLI and add VSM stats chunks. But I'd rather question the sanity of the VMOD writer, than change our architecture to accomodate such code :-) >So unless we add a requirement like "you have to call VCL_Something() >to 'register' the director/backend", so that VCL_Something() can >change the name, I don't think we can use suffixes or anything else to >enforce unique names. (And as of now, I don't see why a VMOD backend >wouldn't work even if it doesn't call VCL_Something().) I think there is merit to this, in the sense that I can see why a VMOD creating dynamic backends wouldn't want them to "pollute" the CLI and VSM. But in the case where the VMOD wants the backend to appear in the CLI/VSM, I think it is perfectly fair to disambiguate the name there, and I think it would be silly to fail the registration, because some other VMOD happened to use the same name already. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Mon Nov 2 08:17:54 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 2 Nov 2015 09:17:54 +0100 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: <54220.1446450731@critter.freebsd.dk> References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> <52611.1446421347@critter.freebsd.dk> <54220.1446450731@critter.freebsd.dk> Message-ID: >>If both VMODs rely on a vcl_name to name their backends (myvcl.mydir) >>then they can't get in this situation. That's my whole point. > > And in a VCL which uses two different VMODs which create dynamic > backends ? We can't have two VMOD objects from different VMODs (or even from the same VMOD) having the same vcl_name for a given VCL. That's a guarantee we get from libvcc. > No matter what, I don't think this is panic material, unless the VMOD > writer feels that way about it. As I said, I don't feel strongly about panicking or returning NULL and letting the VMOD writer deal with it. It's the suffix suggestion I'd rather avoid. Dridi From phk at phk.freebsd.dk Mon Nov 2 08:20:36 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 02 Nov 2015 08:20:36 +0000 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> <52611.1446421347@critter.freebsd.dk> <54220.1446450731@critter.freebsd.dk> Message-ID: <54424.1446452436@critter.freebsd.dk> -------- In message , Dridi Bouke lmoune writes: >>>If both VMODs rely on a vcl_name to name their backends (myvcl.mydir) >>>then they can't get in this situation. That's my whole point. >> >> And in a VCL which uses two different VMODs which create dynamic >> backends ? > >We can't have two VMOD objects from different VMODs (or even from the >same VMOD) having the same vcl_name for a given VCL. That's a >guarantee we get from libvcc. VCC is not involved in dynamic backends. As Geoff said, a VMOD can spit out as many backends as it wants... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Mon Nov 2 08:32:27 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 2 Nov 2015 09:32:27 +0100 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: <54424.1446452436@critter.freebsd.dk> References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> <52611.1446421347@critter.freebsd.dk> <54220.1446450731@critter.freebsd.dk> <54424.1446452436@critter.freebsd.dk> Message-ID: Sorry for the frenchism but it's turning into a "dialogue de sourd" :( > VCC is not involved in dynamic backends. I *know* that. In the "Writing a director" documentation I *recommend* directors writers to back their director with a *VMOD object* because VMOD object come with a *unique* vcl_name and don't outlive the VCL's lifespan. It makes VMOD objects in my opinion the best facility to write a director. My point is that a VMOD writer *should be responsible* for not conflicting with other well-behaving VMODs. In this case, claiming a unique vcl_name as a first level of namespace seems like a good idea to me. > As Geoff said, a VMOD can spit out as many backends as it wants... I *know* that. In vmod-named I use the object's vcl_name as a basis for the backend, and use the resolved IP address to ensure uniqueness within this namespace. Crossing fingers that this time I'm making some sense. Dridi From phk at phk.freebsd.dk Mon Nov 2 08:52:28 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 02 Nov 2015 08:52:28 +0000 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> <52611.1446421347@critter.freebsd.dk> <54220.1446450731@critter.freebsd.dk> <54424.1446452436@critter.freebsd.dk> Message-ID: <54558.1446454348@critter.freebsd.dk> -------- In message , Dridi Bouke lmoune writes: >Sorry for the frenchism but it's turning into a "dialogue de sourd" :( > >> VCC is not involved in dynamic backends. > >I *know* that. > >In the "Writing a director" documentation I *recommend* directors >writers to back their director with a *VMOD object* > >[...] > >Crossing fingers that this time I'm making some sense. Not really. VMOD objects are only usable if you know the number of such backends at the time you write your VCL code. What about VMODs that create an unknown number of backends ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From geoff at uplex.de Mon Nov 2 09:44:24 2015 From: geoff at uplex.de (Geoff Simmons) Date: Mon, 02 Nov 2015 10:44:24 +0100 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: <54274.1446451406@critter.freebsd.dk> References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> <52611.1446421347@critter.freebsd.dk> <5636AF1F.3010907@uplex.de> <54274.1446451406@critter.freebsd.dk> Message-ID: <56373078.3000905@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 11/02/2015 09:03 AM, Poul-Henning Kamp wrote: > >> So unless we add a requirement like "you have to call >> VCL_Something() to 'register' the director/backend", so that >> VCL_Something() can change the name, I don't think we can use >> suffixes or anything else to enforce unique names. (And as of >> now, I don't see why a VMOD backend wouldn't work even if it >> doesn't call VCL_Something().) > > I think there is merit to this, in the sense that I can see why a > VMOD creating dynamic backends wouldn't want them to "pollute" the > CLI and VSM. > > But in the case where the VMOD wants the backend to appear in the > CLI/VSM, I think it is perfectly fair to disambiguate the name > there, and I think it would be silly to fail the registration, > because some other VMOD happened to use the same name already. Hmm, I brought up that idea to be dismissive of it, but now I'm beginning to see the merits as well. So we could have something like: VCL_Register(vcl, director) ... which would do things for you like change the name for uniqueness if necessary, create the VBE stats ... (and whatever else we'd offer to do the right way on behalf of director/backend VMODs). And then the docs can say, "Dear VMOD author, you can call that and then we'll take care of these things for you. Or don't, but then you're on your own, just be aware of the consequences." I'm starting to like it. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCAAGBQJWNzBtAAoJEOUwvh9pJNURfgAP/2v6cf/qg92WYWuWPK6nxDrz iTLkYX0Upaf9fjQ3GI8mZ7Hmzt+mYHdIWUohSxSfatsH1/8nmw2HVtI15r4mtHuv C4Sd+66EGom2tBvVg4BCw1d+snI5WcIEF646taZI3AHWPQ4qKJng2+XKgJU/C8iD kGLh4oM0zh2wxRkouUfx3O5efxQ9v7/Bplv+p0YSXNG+Bo5B5U/Mj0jMbAOKExmu EekQBZA93DYUyBLEH0wNmD7QMruqcCS7QgOfnvTejQR/EN+ThHrxWNyChjy6fWzv sqJ42kW0Wkhis4Eel4ik4jufeCV80zaWrXZoPzBluQXPaFTs88WJjNq0dyK3VFUT Eb5pydCxMDk0y59xDHJD4oL/Y2Ed09rqzdZDg2AKAH6DSehsdp2gcMrB8GEmOk+q gdJBzgCzUSEhbmQTc2kEHbswnLxDTJGbQGj7upMRCUCdmX8p27LlLyYYyFg1aA4i UWJ6y5JQ/YuIR1HvjK2X+4iNKfawydtjZ0tBW5qyId/RHaN6PKEvQhTb0aqiqMXZ IOX2NXtQcYKIs6AKtHzu2yDh0I6IRLdRhmVDewVZc8BW6TmPPO3wpieOmVR3f2fu BlIm7T4ideqk6PBB+YxfzTMKDep9p2xauSjnjqy02ozB6DkjeMVfwjckoQf86jgm cRdzjcB12SRK3PttWxZr =QcUr -----END PGP SIGNATURE----- From dridi at varni.sh Mon Nov 2 09:54:23 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 2 Nov 2015 10:54:23 +0100 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: <54558.1446454348@critter.freebsd.dk> References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> <52611.1446421347@critter.freebsd.dk> <54220.1446450731@critter.freebsd.dk> <54424.1446452436@critter.freebsd.dk> <54558.1446454348@critter.freebsd.dk> Message-ID: >>[...] >> >>Crossing fingers that this time I'm making some sense. > > Not really. > > VMOD objects are only usable if you know the number of such backends > at the time you write your VCL code. > > What about VMODs that create an unknown number of backends ? Last try, after that I give up. I tried to answer that point in the "[...]" part but obviously that didn't work. If we eventually come to mutual understanding by the way, I can later update the documentation to clarify this point and provide better guidance to VMOD writers. If you use the vcl_name of a VMOD object, you have the guarantee that this name won't conflict with an existing backend: **** v1 0.1 CLI RX| Message from VCC-compiler:\n **** v1 0.1 CLI RX| Object name 's1' already used.\n **** v1 0.1 CLI RX| First usage:\n **** v1 0.1 CLI RX| ('input' Line 2 Pos 9)\n **** v1 0.1 CLI RX| backend s1 { .host = "127.0.0.1"; .port = "41338"; }\n **** v1 0.1 CLI RX| --------##------------------------------------------\n **** v1 0.1 CLI RX| \n **** v1 0.1 CLI RX| Redefinition:\n **** v1 0.1 CLI RX| ('input' Line 8 Pos 21)\n **** v1 0.1 CLI RX| new s1 = named.director("localhost", "... **** v1 0.1 CLI RX| --------------------##-------------------------------- I believe we already agree on this bit, but I wanted to emphasize it Backends can only be named using [0-9a-zA-Z_] so a director creating an unknown number of backends (like vmod-named by the way) should implement its own suffix starting with an "illegal" character (so that you don't accidentally end up with the name of another backend). The VMOD author can pick a ".%d" suffix like the one you suggested Varnish could use, or "(%s)" like I do with vmod-named or anything else that may have more meaning to the end-user. If all VMODs use a unique name as the basis, and a conflict-free naming scheme, the problem disappears. Regex-ish pattern: (.+)\.(.+)([^0-9a-zA-Z_])(.+) Groups: - name of the vcl - vmod object's "vcl_name" - "illegal" separator - uniqueness bit Example in vmod-named's test suite: https://github.com/dridi/libvmod-named/blob/43798d7/src/tests/test01.vtc#L26 VMOD authors have enough tools as of today to ensure uniqueness of their dynamic backends names, regardless of the number of backends they create and when they are creating, therefore I want to shift the responsibility to said VMOD authors and properly document it. This is also why I'm fine with a plain panic instead of a new function like VCL_HasBackend. Best, Dridi From phk at phk.freebsd.dk Mon Nov 2 10:06:46 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 02 Nov 2015 10:06:46 +0000 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> <52611.1446421347@critter.freebsd.dk> <54220.1446450731@critter.freebsd.dk> <54424.1446452436@critter.freebsd.dk> <54558.1446454348@critter.freebsd.dk> Message-ID: <55112.1446458806@critter.freebsd.dk> -------- In message , Dridi Bouke lmoune writes: >If you use the vcl_name of a VMOD object, you have the guarantee that >this name won't conflict with an existing backend: > >**** v1 0.1 CLI RX| Message from VCC-compiler:\n >**** v1 0.1 CLI RX| Object name 's1' already used.\n >**** v1 0.1 CLI RX| First usage:\n >**** v1 0.1 CLI RX| ('input' Line 2 Pos 9)\n >**** v1 0.1 CLI RX| backend s1 { .host = "127.0.0.1"; .port = "41338"; }\n >**** v1 0.1 CLI RX| --------##------------------------------------------\n >**** v1 0.1 CLI RX| \n >**** v1 0.1 CLI RX| Redefinition:\n >**** v1 0.1 CLI RX| ('input' Line 8 Pos 21)\n >**** v1 0.1 CLI RX| new s1 = named.director("localhost", "... >**** v1 0.1 CLI RX| --------------------##-------------------------------- That is almost entirely without relevance to the present discussion. When you do "new something = vmod.constructor()" you don't get a backend, you get a VMOD object. This VMOD object may or may not have a member function which returns a backend, and depending on how you call it, you may get the same or different backends each time you call it. That/those backends has no name in the VCL namespace and thus the VCC cannot see them. >Backends can only be named using [0-9a-zA-Z_] so a director creating >an unknown number of backends (like vmod-named by the way) should >implement its own suffix starting with an "illegal" character (so that you >don't accidentally end up with the name of another backend). And what happens if vmod-named and vmod-ldap just happen to pick the same "unique" name ? One of them decides to ignore it, the other panics ? Nope... I think Geoff's last email is on the right track: Backends can be named or unnamed. Only named backends appear in CLI and VSM Backends defined in VCL are always named. VMOD created backends can be named by calling a VRT function. That VRT function will make sure the name is globally unique over CLI/VSM (I have some questions related to lifetime and cleanup, but we'll tackle those later) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From geoff at uplex.de Mon Nov 2 10:31:21 2015 From: geoff at uplex.de (Geoff Simmons) Date: Mon, 02 Nov 2015 11:31:21 +0100 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> <52611.1446421347@critter.freebsd.dk> <54220.1446450731@critter.freebsd.dk> <54424.1446452436@critter.freebsd.dk> Message-ID: <56373B79.10300@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 11/02/2015 09:32 AM, Dridi Boukelmoune wrote: > Sorry for the frenchism but it's turning into a "dialogue de sourd" > :( Dridi, mon ami, ce n`est pas grave. %^) My impression is that we're all insistently agreeing with each other. > In the "Writing a director" documentation I *recommend* directors > writers to back their director with a *VMOD object* because VMOD > object come with a *unique* vcl_name and don't outlive the VCL's > lifespan. It makes VMOD objects in my opinion the best facility to > write a director. Certainly, that's a good solution -- as long as your use case allows for it. You could even have an object generate multiple backends, using the object's name for a "safe namespace", within which it creates unique names (say by adding the ".%d" suffix). And you've made it clear that you're fully aware that using objects is not the only, necessary solution -- and even then, the VMOD has to go through the motions of using the object's vcl_name for backend names. However the VMOD goes about it, uniqueness of naming with all of the consequences is something it has to take care of (or not, and take the consequences). Adding something to the Varnish/VRT interface to make it work right will go a long way to making this painless. D'accord? (Sorry, slink is better at that than I am.) Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCAAGBQJWNzt5AAoJEOUwvh9pJNURPF8QAIj8j5I82lrrlc6H3ipS9JF8 p+tbVP8xjV3NPT0P7UTADT/6+9j/tq/sG7x/oogn9qQKRW/nJHO77kyGb2Yl3XTm smm5FvpxRb0PqIVOZz3RHUjKyTvMGji0k78pIay33SfyPwRW076T3QYzEWVluWfI QZak19Pbmf73Sn+3N+Szr5itQYo2Mw1DeHynOv3MtKVqk/NWNEQRJapvcL5497n6 nZYd105MQd56b/xg+v5x4AUN1XBWAjpCl3OAefMKAgYrWMJ4ceobUqPtfy4yJsxO 6D8uH5Xg4Oth2UKdaTmidzWg47l2CCG/ErJmrHbYaX3k0IPavT5ifKUdRmV3i7ZI GRmSJuJSbUhKP8IKg3j51Ut3B6Az+//uZXR/UhsuehFl06dfy/2ETDoofWUY6vZu lF49vZ70dIa1iQ7h+WLUEGDfI6xCP1dlW8yQ2ptZSCZo2qN0CndufMZU9hhzYewP +zPnQEoFDGx/Nlm/9LFnm/VV0EGtzv3ueGZV6c2UxiwfNZVPMvJw8DK1B75Rpk8O bTOH8LFavmZU46BG53H2g2PDFYDp8dqsEAhZKf5lAKlhDpY0EmZFHq88RMUZ142F eMWdTegAyKrWx9dz/tX7P4TE4kKw3juopSSeNwAUbLaFGjrkXh6OdPuUtTAfhd15 b6PGXhizhR7jJKrZkIQO =prCa -----END PGP SIGNATURE----- From dridi at varni.sh Mon Nov 2 10:42:48 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 2 Nov 2015 11:42:48 +0100 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: <56373B79.10300@uplex.de> References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> <52611.1446421347@critter.freebsd.dk> <54220.1446450731@critter.freebsd.dk> <54424.1446452436@critter.freebsd.dk> <56373B79.10300@uplex.de> Message-ID: >> In the "Writing a director" documentation I *recommend* directors >> writers to back their director with a *VMOD object* because VMOD >> object come with a *unique* vcl_name and don't outlive the VCL's >> lifespan. It makes VMOD objects in my opinion the best facility to >> write a director. > > Certainly, that's a good solution -- as long as your use case allows > for it. You could even have an object generate multiple backends, > using the object's name for a "safe namespace", within which it > creates unique names (say by adding the ".%d" suffix). > > And you've made it clear that you're fully aware that using objects is > not the only, necessary solution -- and even then, the VMOD has to go > through the motions of using the object's vcl_name for backend names. I apparently I haven't made it clear that I fully understand how VMOD objects and backends are totally unrelated, except for the vcl_name on which they can't collide. > However the VMOD goes about it, uniqueness of naming with all of the > consequences is something it has to take care of (or not, and take the > consequences). > > Adding something to the Varnish/VRT interface to make it work right > will go a long way to making this painless. > > D'accord? I don't think it's a good idea to leave the end-user with unnamed backends in the VSL that don't even appear in the CLI and VSM, or confusing names in the CLI/VSM because of name collisions auto-repair. I care way way more about the end-user's convenience rather than the VMOD writer's. Especially since the rules to follow to ensure uniqueness wouldn't even be hard, there isn't that much pain in my opinion. Best, Dridi From phk at phk.freebsd.dk Mon Nov 2 10:47:14 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 02 Nov 2015 10:47:14 +0000 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> <52611.1446421347@critter.freebsd.dk> <54220.1446450731@critter.freebsd.dk> <54424.1446452436@critter.freebsd.dk> <56373B79.10300@uplex.de> Message-ID: <77098.1446461234@critter.freebsd.dk> -------- In message , Dridi Bouke lmoune writes: >I don't think it's a good idea to leave the end-user with unnamed >backends in the VSL that don't even appear in the CLI and VSM, or >confusing names in the CLI/VSM because of name collisions auto-repair. I think you are thinking far too narrow about backend. Imagine vmod-cgi. It would create a backend good only for a single request and it doesn't have an IP#, cannot be polled and cannot in any meaningful way be set healthy/sick. Why would you ever name that backend or show it in the CLI ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Mon Nov 2 11:07:44 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 2 Nov 2015 12:07:44 +0100 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: <77098.1446461234@critter.freebsd.dk> References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> <52611.1446421347@critter.freebsd.dk> <54220.1446450731@critter.freebsd.dk> <54424.1446452436@critter.freebsd.dk> <56373B79.10300@uplex.de> <77098.1446461234@critter.freebsd.dk> Message-ID: >>I don't think it's a good idea to leave the end-user with unnamed >>backends in the VSL that don't even appear in the CLI and VSM, or >>confusing names in the CLI/VSM because of name collisions auto-repair. > > I think you are thinking far too narrow about backend. > > Imagine vmod-cgi. It's actually on my TODO-unless-someone-starts-before-me list :) > It would create a backend good only for a single request and it > doesn't have an IP#, cannot be polled and cannot in any meaningful > way be set healthy/sick. But this wouldn't create a backend, only a director with a custom transport. You wouldn't even get the opportunity for a backend name conflict. > Why would you ever name that backend or show it in the CLI ? That's a director, directors don't show up in the backend.list CLI. They also don't show up anywhere in varnishstat, and this is a question I was saving for later because I haven't studied yet whether you can add out-of-tree VSC fields (at first glance I think not). Best, Dridi From phk at phk.freebsd.dk Mon Nov 2 11:27:37 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 02 Nov 2015 11:27:37 +0000 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> <52611.1446421347@critter.freebsd.dk> <54220.1446450731@critter.freebsd.dk> <54424.1446452436@critter.freebsd.dk> <56373B79.10300@uplex.de> <77098.1446461234@critter.freebsd.dk> Message-ID: <81765.1446463657@critter.freebsd.dk> -------- In message , Dridi Bouke lmoune writes: Let me try to summarize at an even higher level: * All "struct director" (= VCL_BACKEND) have a "vcl_name" which can be queried from VCL. VMODs are responsible for doing something sensible here. * struct director without a struct backend do not appear in CLI/VSM unless the VMOD implements all the stuff for this. * struct director with a struct backend, defined in VCL is always in CLI/VSM, and the vcl_name is the name from the VCL namespace. * struct director with a struct backend, created by VMOD, may or may not be in CLI/VSM, at the VMODs discretion. * If it is in CLI/VSM, the VRT() used to register it there may modify the vcl_name to make sure it is globally unique. (In practice: vcl-unique, because the namespace is $VCL.$BACKEND) * If the VMOD ensures there are no duplicate vcl_names in the first place, the vcl_name will not be touched. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Mon Nov 2 12:00:18 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 02 Nov 2015 12:00:18 +0000 Subject: Continuous integration moves to Travis CI In-Reply-To: <20151031083433.GA5176@immer.varnish-software.com> References: <20151031083433.GA5176@immer.varnish-software.com> Message-ID: <82459.1446465618@critter.freebsd.dk> -------- In message <20151031083433.GA5176 at immer.varnish-software.com>, Lasse Karstensen writes: >Hi all. > >I'm changing continuous integration system for Varnish Cache to the >online Travis CI system. (https://travis-ci.org/ ) > >Travis is integrated with Github, and can build on Linux and OS X. >We've been testing it on vmods and VC for a while, but not as the main >CI system. > >Here is the new build job: > > https://travis-ci.org/varnishcache/varnish-cache It looks like they have a problem with test c00007 ? Any clues ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From geoff at uplex.de Mon Nov 2 12:05:53 2015 From: geoff at uplex.de (Geoff Simmons) Date: Mon, 02 Nov 2015 13:05:53 +0100 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: <81765.1446463657@critter.freebsd.dk> References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> <52611.1446421347@critter.freebsd.dk> <54220.1446450731@critter.freebsd.dk> <54424.1446452436@critter.freebsd.dk> <56373B79.10300@uplex.de> <77098.1446461234@critter.freebsd.dk> <81765.1446463657@critter.freebsd.dk> Message-ID: <563751A1.2000104@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 11/02/2015 12:27 PM, Poul-Henning Kamp wrote: > > * struct director without a struct backend do not appear in > CLI/VSM unless the VMOD implements all the stuff for this. What about a VMOD for backends that don't speak HTTP over TCP? I have assumed that these would be directors without struct backend (or is that assumption wrong?), but I don't see why they shouldn't be allowed to VRT_register() to get the "official" CLI and VSM support. For the rest: d'accord. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCAAGBQJWN1GhAAoJEOUwvh9pJNURwPIP/R/8F8WYPFxnMXW39AhvL1Tb q/T9QBeuFlhjDm6wZf+E/oHT6KGbp6cAL9cgkpBILjP0d5k8H5g5OXKPBYvv3L87 jjjktrUBLyd6EDVD6KHSa3klT4nXsWuS4hH/rbQ28h6C9zzpH2Vv/2Stre5YlnJx nYX6pkY9ETDWGrsdHCMBcdoakzqU9Vc6ztY1PISNbA+dtYgac6cHhWJWrzFt6E0N xDaXIEXYbPPjIiNKTXBhOtBPoj5i5ulVCcP0MRw2sja0GLslYo/Xa4uV8hS4LM0b jYc885aKafk53lqBs4uFQf8+KgytjMtyZzg5XMSEezaCOlNsIBjRrHU0prk3a1XC 1Li5D522zw6jKsiXhpdXMs9Cp4ekSham9TePR9XewBscffo58++wla3zWZtMNzao p1iO3Sg+FsUi3KhNB2k5AxuAkdKsrE9KS8ZhSTYG/kMmgsptxA+K99i+2y9W94sv 82ppNgb2VAeb+rZfxTcNnyM5FuTEtvt/tKtfouDRRg0OM7PYCjk87m9ZKMBHVjJE JV6n4b+zM/jQa8R+0uUgc25a++gMpIYcDth/naNVZTiCojbS6L/Q0k31kqd9N7Up NQRz5kWln+OTgnYAEBOXzg6eUBDF3slwzzOf4yKvNgaSSZfEPI6rDGPSh59+TiBX THPyfqsK1BpdQPeMPbu8 =LLJW -----END PGP SIGNATURE----- From dridi at varni.sh Mon Nov 2 12:06:55 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 2 Nov 2015 13:06:55 +0100 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: <81765.1446463657@critter.freebsd.dk> References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> <52611.1446421347@critter.freebsd.dk> <54220.1446450731@critter.freebsd.dk> <54424.1446452436@critter.freebsd.dk> <56373B79.10300@uplex.de> <77098.1446461234@critter.freebsd.dk> <81765.1446463657@critter.freebsd.dk> Message-ID: > Let me try to summarize at an even higher level: Good idea, I'll use it to summarize my position too. > * All "struct director" (= VCL_BACKEND) have a "vcl_name" which can > be queried from VCL. VMODs are responsible for doing something > sensible here. Sounds good. > * struct director without a struct backend do not appear in CLI/VSM > unless the VMOD implements all the stuff for this. Sounds good, though I don't know how to implement it. > * struct director with a struct backend, defined in VCL is always > in CLI/VSM, and the vcl_name is the name from the VCL namespace. Sounds good, and for static backends defined in plain native vcl, backend.display_name == vcl.loaded_name + "." + director.vcl_name > * struct director with a struct backend, created by VMOD, may or > may not be in CLI/VSM, at the VMODs discretion. They are currently always in the CLI/VSM, and this is controlled by the VCL temperature changes. But I think a struct director with a struct backend should always show up in the CLI/VSM. > * If it is in CLI/VSM, the VRT() used to register it there may > modify the vcl_name to make sure it is globally unique. > (In practice: vcl-unique, because the namespace is $VCL.$BACKEND) Since the CLI/VSM only show backends (not director) that would be the backend's display_name. That is the part I'm not comfortable with, for end-users. > * If the VMOD ensures there are no duplicate vcl_names in the > first place, the vcl_name will not be touched. The VMOD can enforce proper uniqueness, but can't prevent against name collisions from other VMODs. From phk at phk.freebsd.dk Mon Nov 2 12:37:23 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 02 Nov 2015 12:37:23 +0000 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> <52611.1446421347@critter.freebsd.dk> <54220.1446450731@critter.freebsd.dk> <54424.1446452436@critter.freebsd.dk> <56373B79.10300@uplex.de> <77098.1446461234@critter.freebsd.dk> <81765.1446463657@critter.freebsd.dk> Message-ID: <82533.1446467843@critter.freebsd.dk> -------- In message , Dridi Bouke lmoune writes: >> * All "struct director" (= VCL_BACKEND) have a "vcl_name" which can >> be queried from VCL. VMODs are responsible for doing something >> sensible here. > >Sounds good. > >> * struct director without a struct backend do not appear in CLI/VSM >> unless the VMOD implements all the stuff for this. > >Sounds good, though I don't know how to implement it. It would have to be a very magical case before it *should* be implemented. >> * struct director with a struct backend, defined in VCL is always >> in CLI/VSM, and the vcl_name is the name from the VCL namespace. > >Sounds good, and for static backends defined in plain native vcl, >backend.display_name == vcl.loaded_name + "." + director.vcl_name *All* backend names has the from "$VCL.something" >> * struct director with a struct backend, created by VMOD, may or >> may not be in CLI/VSM, at the VMODs discretion. > >They are currently always in the CLI/VSM, and this is controlled by >the VCL temperature changes. But I think a struct director with a >struct backend should always show up in the CLI/VSM. It probably should *almost* always show up, but there are valid schenarios where a struct backend is good for a single request and I can see why you wouldn't want to clutter CLI/VSM with that. >> * If it is in CLI/VSM, the VRT() used to register it there may >> modify the vcl_name to make sure it is globally unique. >> (In practice: vcl-unique, because the namespace is $VCL.$BACKEND) > >Since the CLI/VSM only show backends (not director) that would be the >backend's display_name. That is the part I'm not comfortable with, for >end-users. I understand that, but I think the alternative, failing backend creation unpredictably (as seen from the VMOD) is worse. >> * If the VMOD ensures there are no duplicate vcl_names in the >> first place, the vcl_name will not be touched. > >The VMOD can enforce proper uniqueness, but can't prevent against name >collisions from other VMODs. ... which is why the VRT function will have to step in. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Mon Nov 2 12:52:00 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 2 Nov 2015 13:52:00 +0100 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: <82533.1446467843@critter.freebsd.dk> References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> <52611.1446421347@critter.freebsd.dk> <54220.1446450731@critter.freebsd.dk> <54424.1446452436@critter.freebsd.dk> <56373B79.10300@uplex.de> <77098.1446461234@critter.freebsd.dk> <81765.1446463657@critter.freebsd.dk> <82533.1446467843@critter.freebsd.dk> Message-ID: >>Sounds good, and for static backends defined in plain native vcl, >>backend.display_name == vcl.loaded_name + "." + director.vcl_name > > *All* backend names has the from "$VCL.something" What I meant is that dynamic backends should start with $VCL.something (the whole suffix discussion), while static backends are strictly $VCL.something. I wasn't disagreeing but adding to the summary. >>They are currently always in the CLI/VSM, and this is controlled by >>the VCL temperature changes. But I think a struct director with a >>struct backend should always show up in the CLI/VSM. > > It probably should *almost* always show up, but there are valid > schenarios where a struct backend is good for a single request > and I can see why you wouldn't want to clutter CLI/VSM with that. Just to get rid of ambiguity here, is it your vmod-cgi example? From dridi at varni.sh Mon Nov 2 12:55:24 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 2 Nov 2015 13:55:24 +0100 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: <563751A1.2000104@uplex.de> References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> <52611.1446421347@critter.freebsd.dk> <54220.1446450731@critter.freebsd.dk> <54424.1446452436@critter.freebsd.dk> <56373B79.10300@uplex.de> <77098.1446461234@critter.freebsd.dk> <81765.1446463657@critter.freebsd.dk> <563751A1.2000104@uplex.de> Message-ID: >> * struct director without a struct backend do not appear in >> CLI/VSM unless the VMOD implements all the stuff for this. > > What about a VMOD for backends that don't speak HTTP over TCP? I have > assumed that these would be directors without struct backend (or is > that assumption wrong?), but I don't see why they shouldn't be allowed > to VRT_register() to get the "official" CLI and VSM support. Your assumption is right, AFAICT. From fgsch at lodoss.net Mon Nov 2 13:58:14 2015 From: fgsch at lodoss.net (Federico Schwindt) Date: Mon, 2 Nov 2015 13:58:14 +0000 Subject: IPv[46] parsing In-Reply-To: References: <74327.1446022709@critter.freebsd.dk> Message-ID: Hi, As requested I've provided the details of how this currently works and my proposal. Any comments? On Wed, Oct 28, 2015 at 1:38 PM, Federico Schwindt wrote: > Hi, > > Current functionality > ---------------------------- > > 1. If the string starts with "[" we take everything up to "]" as the host, > excluding the "[]". If there is a ":" following the "]" we > take > > everything after it as the port. > > 2. If the string has a space, take everything up to the space (but > excluding > it) as the host. Anything after the space is the port. > > 3. If the string doesn't have a space, but has a ":" take everything up to > the ":" (but excluding it) as a host, and anything after as the port. > > My proposal > ----------------- > > Change 3 as follows: > > 3. If the string doesn't have a space, but has a single ":" take > everything up to the ":" (but excluding it) as a host, and anything > after as the port. If the string has 2 or more ":", take the whole > string as a host without port. > > Examples > -------------- > > 1.2.3.4 => host = 1.2.3.4 port = > 1.2.3.4:80 => host = 1.2.3.4 port = 80 > 1.2.3.4 80 => host = 1.2.3.4 port = 80 > foo.bar => host = foobar port = > foo.bar:80 => host = foobar port = 80 > foo.bar 80 => host = foobar port = 80 > [::1] => host = ::1 port = > [::1]:80 => host = ::1 port = 80 > ::1 => host = ::1 port = > ::1::2 => host = ::1::2 port = > 1.2.3.4::80 => host = 1.2.3.4::80 port = > > With the current implementation ::1 would fail. It works with my patch. > The last 2 would fail at getaddrinfo(), with or without my proposal. > > f.- > > On Wed, Oct 28, 2015 at 8:58 AM, Poul-Henning Kamp > wrote: > >> -------- >> In message > dCFHBtDL_XMzmGLu0q8mwWnCbSEZw at mail.gmail.com> >> , Federico Schwindt writes: >> >> >While moving from 4.0.x to 4.1 I noticed that std.ip(..., "::1") doesn't >> >work anymore. >> >> I agree this is very far from optimal, but adding special-casing >> IPv6 address by IPv6 address is certainly *not* the way forward. >> >> The real question is, what does "::1:8080" mean ? >> >> Please propose the exact algorithm you prose for turning strings >> into IP numbers, and bear in mind that we need to be able to >> include port numbers. >> >> -- >> Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 >> phk at FreeBSD.ORG | TCP/IP since RFC 956 >> FreeBSD committer | BSD since 4.3-tahoe >> Never attribute to malice what can adequately be explained by >> incompetence. >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kacperw at gmail.com Mon Nov 2 18:10:43 2015 From: kacperw at gmail.com (Kacper Wysocki) Date: Mon, 2 Nov 2015 19:10:43 +0100 Subject: Varnish project autumn cleaning In-Reply-To: <44520.1446285081@critter.freebsd.dk> References: <28767.1445957308@critter.freebsd.dk> <20151027161427.GA1274@immer.varnish-software.com> <20151030094930.GA14933@immer.varnish-software.com> <20151031092506.GB5176@immer.varnish-software.com> <44520.1446285081@critter.freebsd.dk> Message-ID: sorry, building from source is a pretty crap alternative for all the ops guys out there. yay for more V-S-Plus customers? On Sat, Oct 31, 2015 at 10:51 AM, Poul-Henning Kamp wrote: > -------- > In message <20151031092506.GB5176 at immer.varnish-software.com>, Lasse Karstensen writes: > >>With my VS hat on: Varnish Plus customers using community Varnish Cache >>should contact support to find alternatives. > > The obvious alternative is to build from source. > > That also ensures that any bugfixes or security patches are accessible > with no undue delay. > From kacperw at gmail.com Mon Nov 2 18:20:52 2015 From: kacperw at gmail.com (Kacper Wysocki) Date: Mon, 2 Nov 2015 19:20:52 +0100 Subject: Preventing dup backend names with dynamic backends in VMODs In-Reply-To: <5636468A.6060300@uplex.de> References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <5636468A.6060300@uplex.de> Message-ID: On Sun, Nov 1, 2015 at 6:06 PM, Geoff Simmons wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > On 10/30/2015 05:19 PM, Kacper Wysocki wrote: >> >> slightly related to this, could we possibly allow accessing the >> name in VCL, like >> >> if(req.backend_hint ~ "zool") { /* do something */ } > > If you can wait until vcl_backend_response(), you do have > beresp.backend.name as a string, and this will work: > > if (beresp.backend.name ~ "zool") { > /* do something */ > } > >> or even >> >> std.log("backend was " + req.backend_hint); ? > > That works, try it. Evidently because a STRING_LIST is required in > std.log(), and VCC knows to do implicit string conversions for the > various data types -- it even works without the concatenation. > > A regex match against req.backend_hint or bereq.backend doesn't work > because a STRING is required. IIUC, VCC does implicit string > conversions where a STRING_LIST is expected, but not where a STRING is > expected. Oh. I hadn't seen req.backend.name. I guess the type conversion is a little counter to my intuition too. thanks, -kacper From jos at dwim.org Mon Nov 2 18:22:30 2015 From: jos at dwim.org (Jos Boumans) Date: Mon, 2 Nov 2015 10:22:30 -0800 Subject: Varnish project autumn cleaning In-Reply-To: References: <28767.1445957308@critter.freebsd.dk> <20151027161427.GA1274@immer.varnish-software.com> <20151030094930.GA14933@immer.varnish-software.com> <20151031092506.GB5176@immer.varnish-software.com> <44520.1446285081@critter.freebsd.dk> Message-ID: +1 - hosting a build of every blessed varnish release for the popular platforms like redhat & ubuntu would be really helpful. What?s the gating factor here for you? Time, cost, expertise? Is there a way the community can help you? -Jos > On Nov 2, 2015, at 10:10 AM, Kacper Wysocki wrote: > > sorry, building from source is a pretty crap alternative for all the > ops guys out there. yay for more V-S-Plus customers? > > On Sat, Oct 31, 2015 at 10:51 AM, Poul-Henning Kamp wrote: >> -------- >> In message <20151031092506.GB5176 at immer.varnish-software.com>, Lasse Karstensen writes: >> >>> With my VS hat on: Varnish Plus customers using community Varnish Cache >>> should contact support to find alternatives. >> >> The obvious alternative is to build from source. >> >> That also ensures that any bugfixes or security patches are accessible >> with no undue delay. >> > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev ? Entropy isn?t what it used to be... -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 163 bytes Desc: Message signed with OpenPGP using GPGMail URL: From phk at phk.freebsd.dk Mon Nov 2 18:58:51 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 02 Nov 2015 18:58:51 +0000 Subject: Varnish project autumn cleaning In-Reply-To: References: <28767.1445957308@critter.freebsd.dk> <20151027161427.GA1274@immer.varnish-software.com> <20151030094930.GA14933@immer.varnish-software.com> <20151031092506.GB5176@immer.varnish-software.com> <44520.1446285081@critter.freebsd.dk> Message-ID: <83670.1446490731@critter.freebsd.dk> -------- In message , Jos Boumans writes: >+1 - hosting a build of every blessed varnish release for >the popular platforms like redhat & ubuntu would be really >helpful. > >What's the gating factor here for you? Time, cost, expertise? >Is there a way the community can help you? Time. People who are willing to take such tasks of our hands are most welcome. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From geoff at uplex.de Mon Nov 2 21:25:33 2015 From: geoff at uplex.de (Geoff Simmons) Date: Mon, 2 Nov 2015 22:25:33 +0100 Subject: Preventing dup backend names with dynamic backends in VMODs In-Reply-To: References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <5636468A.6060300@uplex.de> Message-ID: <5637D4CD.4060102@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 11/2/15 7:20 PM, Kacper Wysocki wrote: > > Oh. I hadn't seen req.backend.name. It's beresp.backend.name, not available until vcl_backend_response(). - -- UPLEX Systemoptimierung Scheffelstra?e 32 22301 Hamburg http://uplex.de/ Mob: +49-176-63690917 -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) iQIcBAEBCAAGBQJWN9TMAAoJEOUwvh9pJNURKXgP/RuZZ/oFfaaUv4a+qWWXKqHU beE/NHLxeX4IuxJdvmao0UaU1KZWuX5Fn1T7FrUQobZ6ftAGaBOAVUIK2auOvHTR g+BPffzEhRbg1kdDCpcJ8WuY6X1uP5prfg4G0ant26ERWUVjCYibJTCOkKhkfLrF sUSNo4NhG8YA3BbzNoRTmLOD37lmCKftCWhJD8PW05g0vK2jDX429N+7wQ6frkAt q+q41JgdlJDOxCbhriwm3+hK/FdeGpHakzOBDZw/dugtUxxIjeeTgjH+OBapd7ui 0wGuqxeJvCmcaIQr1Agi4MozJcavRk8W4o9sAIXpKDtiDNoaA/M4ZsBlxwaW5QJk XpzrAt5rZxDdiKCWRvSsxaiaL0QQdFu+MSw9gaWSN4J3gK+wztOSCjXEcY1PgDmT DNI6BZWs5xvGK3TG+AX1eZ8618i1pv4TVtQJflbfF1eTZmRLUa5KepH7MdWlvpHe g9uVkLw6Eb7otgoD5bAUwD1xXs0rzoc2JWyXRiGFnozFC23WDEiQmJfJVGjfGdSG q56NavhXuAm2wFtzfTklie9aYVz0NItG9IMgAZiLqVQuvvSsL2/BUJ3baQAHpmDR PDAYICmwzplQhby8yiWRHPlBq9LSqEyP+FG/pJrSOX6TdFhHHiiB5PRTvkvYXk1N hz7Jqmaoivd5yke133Em =s9RT -----END PGP SIGNATURE----- From kly at redpill-linpro.com Tue Nov 3 08:23:37 2015 From: kly at redpill-linpro.com (Kristian Lyngstol) Date: Tue, 3 Nov 2015 09:23:37 +0100 Subject: Varnish project autumn cleaning In-Reply-To: References: <28767.1445957308@critter.freebsd.dk> <20151027161427.GA1274@immer.varnish-software.com> <20151030094930.GA14933@immer.varnish-software.com> <20151031092506.GB5176@immer.varnish-software.com> <44520.1446285081@critter.freebsd.dk> Message-ID: <20151103082337.GA1795@leia.redpill-linpro.com> Top posting... On Mon, Nov 02, 2015 at 10:22:30AM -0800, Jos Boumans wrote: > +1 - hosting a build of every blessed varnish release for > the popular platforms like redhat & ubuntu would be really > helpful. This is what packaging is. > What?s the gating factor here for you? Time, cost, expertise? > Is there a way the community can help you? Who is "you"? phk is not a packager. Kacper doesn't do the packaging either. So who are you addressing here? Packaging for most distros is already taking place, as Lasse pointed out. With the steps Ingvar and ssm are taking, we'll have even better coverage for Debian, Fedora and EPEL. Lasse's mail is mostly about moving responsibility away from V-S over to where it rightfully belongs: With the appropriate distros. > > On Nov 2, 2015, at 10:10 AM, Kacper Wysocki wrote: > > On Sat, Oct 31, 2015 at 10:51 AM, Poul-Henning Kamp wrote: > >> -------- > >> In message <20151031092506.GB5176 at immer.varnish-software.com>, Lasse Karstensen writes: > >> > >>> With my VS hat on: Varnish Plus customers using community Varnish Cache > >>> should contact support to find alternatives. > >> > >> The obvious alternative is to build from source. No. The amount of times I've had to clean up the mess people have created after using this "obvious alternative" far outweighs the times I've had to deal with issues related to packages lagging a bit behind. Building from source is an ops nightmare. If your business is centered around Varnish, it might make sense. If Varnish is just one of many many tools, then instead of telling people to build from source, you might as well just attach the nginx user manual. - Kristian From dridi at varni.sh Tue Nov 3 08:27:48 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 3 Nov 2015 09:27:48 +0100 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: <82533.1446467843@critter.freebsd.dk> References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> <52611.1446421347@critter.freebsd.dk> <54220.1446450731@critter.freebsd.dk> <54424.1446452436@critter.freebsd.dk> <56373B79.10300@uplex.de> <77098.1446461234@critter.freebsd.dk> <81765.1446463657@critter.freebsd.dk> <82533.1446467843@critter.freebsd.dk> Message-ID: >>> * If the VMOD ensures there are no duplicate vcl_names in the >>> first place, the vcl_name will not be touched. >> >>The VMOD can enforce proper uniqueness, but can't prevent against name >>collisions from other VMODs. > > ... which is why the VRT function will have to step in. Hi guys, I just realized that a VCL_HasBackend (or VRT) function for VMODs to "ensure" uniqueness would be racy by nature. In a scenario where the backend name is "stolen" right after we check its availability. Since the intention is to have a safety net in Varnish for deduplication, the VCL_HasBackend becomes obsolete. After a quick glance at the thread again, it seems that non-VCL backends such as for instance unix-domain socket backend should eventually show up in the CLI and VSM. While I have nothing against that (quite the opposite) I don't think it can be done without breaking both ABI and API. The current statistics should be generic enough to suit any kind of backend, except maybe for the probe part. The varnish-cli on the other end is too struct backend-centric, and despite my attempt to restrict the director API to the bare minimum, cache_backend.h made it in 4.1 and cache_director.h wasn't stripped off of the VDI_ functions. I have the feeling that Geoff started a 4.1 thread and that we are also discussing things that belong post-4.1. Best, Dridi From phk at phk.freebsd.dk Tue Nov 3 08:51:51 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 03 Nov 2015 08:51:51 +0000 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> <52611.1446421347@critter.freebsd.dk> <54220.1446450731@critter.freebsd.dk> <54424.1446452436@critter.freebsd.dk> <56373B79.10300@uplex.de> <77098.1446461234@critter.freebsd.dk> <81765.1446463657@critter.freebsd.dk> <82533.1446467843@cri tter.freebsd.dk> Message-ID: <86444.1446540711@critter.freebsd.dk> -------- In message , Dridi Bouke lmoune writes: >Hi guys, > >I just realized that a VCL_HasBackend (or VRT) function for VMODs to >"ensure" uniqueness would be racy by nature. In a scenario where the >backend name is "stolen" right after we check its availability. The way we solve that problem in kernels is to name the VCL_HasBackend() function VCL_CreateBackend() and allowing it to fail :-) >After a quick glance at the thread again, it seems that non-VCL >backends such as for instance unix-domain socket backend should >eventually show up in the CLI and VSM. So far I have kept the CLI away from VMODs, but that is probably not viable in the long term. The backend/director split is, as you point out, not clean, and if nothing else the naming is horrible. So I think 5.0 is going to look quite differently than 4.1 in this area. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Tue Nov 3 09:18:00 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 3 Nov 2015 10:18:00 +0100 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: <86444.1446540711@critter.freebsd.dk> References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> <52611.1446421347@critter.freebsd.dk> <54220.1446450731@critter.freebsd.dk> <54424.1446452436@critter.freebsd.dk> <56373B79.10300@uplex.de> <77098.1446461234@critter.freebsd.dk> <81765.1446463657@critter.freebsd.dk> <86444.1446540711@critter.freebsd.dk> Message-ID: > The way we solve that problem in kernels is to name the VCL_HasBackend() > function VCL_CreateBackend() and allowing it to fail :-) Why not keep VRT_new_backend as the only interface for VMODs and allow this one to fail? (or do the atomic auto-renaming if needed) > So far I have kept the CLI away from VMODs, but that is probably > not viable in the long term. > > The backend/director split is, as you point out, not clean, and > if nothing else the naming is horrible. > > So I think 5.0 is going to look quite differently than 4.1 in > this area. So maybe my narrow thinking is not that narrow if we narrow down the scope of Geoff's proposal to 4.1 :) It occurred to me today that I was replying with 4.1 in mind and that it may not be the case for you. Best, Dridi From lkarsten at varnish-software.com Tue Nov 3 09:43:23 2015 From: lkarsten at varnish-software.com (Lasse Karstensen) Date: Tue, 3 Nov 2015 10:43:23 +0100 Subject: Patchwork sunsetting. Message-ID: <20151103094322.GA27190@immer.varnish-software.com> Hi. We've been running an instance[1] of the Patchwork system to track patches sent to -dev at . As Poul-Henning mentioned in an earlier mail here, it has never really worked well for us. Trying to track down a reason for us running Patchwork, I'm being told it was to avoid patches being sent to -dev@ and forgotten about. Our current future plans involves a pull request based workflow on Github, and that should fulfill the requirement nicely. Patchwork will be shut down as of Monday November 9th 2015. As suggested by Poul-Henning, the plan is to do a final review of the patches in there on bugwash that Monday. When that is done, I'll shut it down. To save us some time on Monday, please take a look at the list and close off any patches you have there that is merged/rejected/completed in some way. 1: https://www.varnish-cache.org/patchwork/project/varnish-cache/list/ -- Lasse Karstensen Varnish Software AS From phk at phk.freebsd.dk Tue Nov 3 09:44:41 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 03 Nov 2015 09:44:41 +0000 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> <52611.1446421347@critter.freebsd.dk> <54220.1446450731@critter.freebsd.dk> <54424.1446452436@critter.freebsd.dk> <56373B79.10300@uplex.de> <77098.1446461234@critter.freebsd.dk> <81765.1446463657@critter.freebsd.dk> Message-ID: <86723.1446543881@critter.freebsd.dk> -------- In message , Dridi Bouke lmoune writes: >> The way we solve that problem in kernels is to name the VCL_HasBackend() >> function VCL_CreateBackend() and allowing it to fail :-) > >Why not keep VRT_new_backend as the only interface for VMODs and allow >this one to fail? (or do the atomic auto-renaming if needed) That may be the eventual outcome, but first we have to figure out how we want this to work, before we can decide how to make it work that way. >It occurred to me today that I was replying with 4.1 in mind and that >it may not be the case for you. I'm trying to think of a way to make 4.1 not be a dead end. I would far prefer that whatever we do to 4.1, it points directly at, rather than away from 5.0. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From geoff at uplex.de Tue Nov 3 11:04:23 2015 From: geoff at uplex.de (Geoff Simmons) Date: Tue, 03 Nov 2015 12:04:23 +0100 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: References: <562E4736.5030708@uplex.de> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> <52611.1446421347@critter.freebsd.dk> <54220.1446450731@critter.freebsd.dk> <54424.1446452436@critter.freebsd.dk> <56373B79.10300@uplex.de> <77098.1446461234@critter.freebsd.dk> <81765.1446463657@critter.freebsd.dk> <82533.1446467843@critter.freebsd.dk> Message-ID: <563894B7.1020904@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 11/03/2015 09:27 AM, Dridi Boukelmoune wrote: > > I just realized that a VCL_HasBackend (or VRT) function for VMODs > to "ensure" uniqueness would be racy by nature. Yep, classic TOCTOU race. In addition to the fact that, if we do have it, it should lock the mutex on the VCL's backend_list (which I neglected in the patch). So for a reliable solution, it looks like there's no way around VRT_Register(). > Why not keep VRT_new_backend as the only interface for VMODs and > allow this one to fail? (or do the atomic auto-renaming if needed) Um, VRT_new_backend creates a conventional backend (HTTP over TCP) from a vrt_backend config. What if you want to do something entirely different? > After a quick glance at the thread again, it seems that non-VCL > backends such as for instance unix-domain socket backend should > eventually show up in the CLI and VSM. While I have nothing > against that (quite the opposite) I don't think it can be done > without breaking both ABI and API. The current statistics should be > generic enough to suit any kind of backend, except maybe for the > probe part. A non-HTTP/TCP backend would have to update the stats itself, but at least VRT_Register() can ensure that they are created. Which is good, because you really need an arcane understanding of VSM to get that right. > The varnish-cli on the other end is too struct backend-centric, > and despite my attempt to restrict the director API to the bare > minimum, cache_backend.h made it in 4.1 and cache_director.h wasn't > stripped off of the VDI_ functions. Really? Didn't know that. So if a VMOD creates a non-stuct-backend backend, it gets no support in the CLI at all? > I have the feeling that Geoff started a 4.1 thread and that we are > also discussing things that belong post-4.1. Sorry about that. %^) I was just trying to get started on a backend VMOD one fine day, then wondered what happens if a name gets duplicated, and then, well ... Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCAAGBQJWOJSsAAoJEOUwvh9pJNUR8MQP/1lml8o7/vi+h92YUBB8Tlsu aYMn1DCyLWmRXJj4fL3/Xeupo9wzwbmCRgOgBSagR13jaYok/oY1cxU8qD44zYBf V30xcUlImLvlV3/Eso8atLlHrMAQwkuZlzybFh2Fk9QrZFX5HoQpUdojJ7ar7/Kw 0yMfUVs7EMD0U1SGRkl35zbis6bgJOF1VwQwBAJRIDQkrZRrPS5kFrdKlVjnGR9m lvdPKBq6GWGCPbEAVEw27eTV544ZqICO0/yzXWQ5pyG4SxQsRBRClsoSxr4C4l1X 3cV1fyzfki3nShOn3HZ6kp/GqeiEZe7k/fHj2fOEk/Os2Z/JMIRb/CwsNCgnJbUv GK1yNFmwD2AY1dmIF3cVlu17q5oV9tQsezrBYz7JSNHbLa2hA9bxZONFJQ1zv2eK gjFzfgDImyzZt5YBRWAIyJ5lP9dS9e6oym0fjXH8tnslfneNp5rokUkYJJNCGdAb GgYAXri6dH7gdJrb1qt6x9KiewzQ5cAiB6WKQGpJRRu1UXj3VWjpKMH2GdteD4PV 54PjpXxBrACEe9z5j3HbI+zaLyuGMyYPk6thOQI75MV5K/WISPfDl/idmc3ncgAm V/lFntYbgwgRJYjLOSk+QXjlcDxDC91dgkHdj/HpDnU0e5voxtJVhOz2i/MDngsg oNj0UCiE65TOfB+Txaa8 =pNwT -----END PGP SIGNATURE----- From francisco at varnish-software.com Tue Nov 3 12:20:34 2015 From: francisco at varnish-software.com (=?utf-8?Q?Francisco_Vel=C3=A1zquez?=) Date: Tue, 3 Nov 2015 13:20:34 +0100 Subject: Patch for ticket 1808 (Error in Varnish Request [Flow]) Message-ID: <194F512A-9095-450D-BF9A-F2401C36988D@varnish-software.com> Hi, Here is the patch to show in the Varnish finite state machine diagram that function cnt_recv calls VCL_hash_method before changing to any other state, including pass, pipe, and synth. https://www.varnish-cache.org/trac/attachment/ticket/1808/0001-Improving-finite-state-machine-diagram-to-show-that-.patch -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Improving-finite-state-machine-diagram-to-show-that-.patch Type: application/octet-stream Size: 3341 bytes Desc: not available URL: -------------- next part -------------- Best regards, Francisco From jos at dwim.org Tue Nov 3 17:13:31 2015 From: jos at dwim.org (Jos Boumans) Date: Tue, 3 Nov 2015 09:13:31 -0800 Subject: Varnish project autumn cleaning In-Reply-To: <20151103082337.GA1795@leia.redpill-linpro.com> References: <28767.1445957308@critter.freebsd.dk> <20151027161427.GA1274@immer.varnish-software.com> <20151030094930.GA14933@immer.varnish-software.com> <20151031092506.GB5176@immer.varnish-software.com> <44520.1446285081@critter.freebsd.dk> <20151103082337.GA1795@leia.redpill-linpro.com> Message-ID: <33994DC0-2129-4F20-8A31-C6F566542D1A@dwim.org> > On Nov 3, 2015, at 12:23 AM, Kristian Lyngstol wrote: >> On Mon, Nov 02, 2015 at 10:22:30AM -0800, Jos Boumans wrote: >> What?s the gating factor here for you? Time, cost, expertise? >> Is there a way the community can help you? > > Who is "you?? That would be those identifying themselves as ?we? in the original post. Presumably the same great folks who have offered & maintained: https://www.varnish-cache.org/installation/debian https://www.varnish-cache.org/installation/redhat See here, for your convenience, and apologies if that caused confusion: >>>> On Fri, Oct 30, 2015 at 10:49:32AM +0100, Lasse Karstensen wrote: >>>>> On Tue, Oct 27, 2015 at 05:37:58PM +0000, Federico Schwindt wrote: >>>> [..] >>>>>> Are we going to continue providing packages? >>>>> Consensus seem to be no on the package building. > Lasse's mail is mostly about moving responsibility away from V-S over to > where it rightfully belongs: With the appropriate distros. The problem with that is that distros don?t update packages as upstream releases newer versions. They only release new versions of the OS, or if needed, security updates. I offer you the below link to substantiate that claim: https://launchpad.net/ubuntu/+source/varnish This means that incentives between the Varnish team & distros are not likely to be aligned; if the Varnish project releases a new version, presumably it would like its userbase to upgrade to that version, as it has bug fixes, feature enhancements and other changes that made it worth while enough to release. Distros will stick to the same major/minor version for the lifetime of their OS release. Most customers will be on long term support releases of the OS (for Ubuntu, those happen every 2 years), so in best case, Varnish users will be using up to 2 year old software, but probably closer to 3 or 4. That doesn?t seem to be a win for the user base, nor for the core dev team & community in terms of supporting that older software. > Building from source is an ops nightmare. > > If your business is centered around Varnish, it might make sense. If > Varnish is just one of many many tools, then instead of telling people to > build from source, you might as well just attach the nginx user manual. We agree, hence my question how the community at large might help the ?we? in the original post to keep providing the awesome repositories of varnish packages. Any thoughts & constructive criticism much appreciated, -Jos ? Entropy isn?t what it used to be... -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 163 bytes Desc: Message signed with OpenPGP using GPGMail URL: From phk at phk.freebsd.dk Wed Nov 4 10:09:26 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 04 Nov 2015 10:09:26 +0000 Subject: Continuous integration moves to Travis CI In-Reply-To: <20151031083433.GA5176@immer.varnish-software.com> References: <20151031083433.GA5176@immer.varnish-software.com> Message-ID: <91175.1446631766@critter.freebsd.dk> -------- In message <20151031083433.GA5176 at immer.varnish-software.com>, Lasse Karstensen writes: >Hi all. > >I'm changing continuous integration system for Varnish Cache to the >online Travis CI system. (https://travis-ci.org/ ) > >Travis is integrated with Github, and can build on Linux and OS X. >We've been testing it on vmods and VC for a while, but not as the main >CI system. I've looked at Travis, and it is basically a Linux only thing which *might* support os/x if they can get it to work. FreeBSD support doesn't even seem to be on a road-map. Varnish is not a linux-only project, so we need something better. Ideally I'd like something which doesn't require me to log into som startup, and which just needs a non-root ssh-accessible account on the slave machines. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Wed Nov 4 10:15:50 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 04 Nov 2015 10:15:50 +0000 Subject: Preventing dup backend names with dynamic backends in VMODs In-Reply-To: References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <5636468A.6060300@uplex.de> Message-ID: <91251.1446632150@critter.freebsd.dk> -------- In message , Kacper Wyso cki writes: >Oh. I hadn't seen req.backend.name. I guess the type conversion is a >little counter to my intuition too. I'm not very happy about "req.backend.name", it is *very* magic and not general to any other backend anywhere else in VCL. The real fix would be to allow standardized (ie: builtin) methods on the VCL_* types themselves, so that one could do things like $backend.ip() $backend.name() etc. Implementing it will take some VCC hacking -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Wed Nov 4 10:40:22 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 04 Nov 2015 10:40:22 +0000 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: <563894B7.1020904@uplex.de> References: <562E4736.5030708@uplex.de> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> <52611.1446421347@critter.freebsd.dk> <54220.1446450731@critter.freebsd.dk> <54424.1446452436@critter.freebsd.dk> <56373B79.10300@uplex.de> <77098.1446461234@critter.freebsd.dk> <81765.1446463657@critter.freebsd.dk> <82533.1446467843@critter.freebsd.dk> <563894B7.1020904 @uplex.de> Message-ID: <91356.1446633622@critter.freebsd.dk> -------- In message <563894B7.1020904 at uplex.de>, Geoff Simmons writes: >> Why not keep VRT_new_backend as the only interface for VMODs and >> allow this one to fail? (or do the atomic auto-renaming if needed) > >Um, VRT_new_backend creates a conventional backend (HTTP over TCP) >from a vrt_backend config. What if you want to do something entirely >different? Ok, I think we need to wrap this discussion up and move on. My conclusion for the 5.x perspective is the following: 1. Our terminology sucks, in particular the director/backend confusion. 2. We will not force all directors/backends to appear in CLI/VSC a) Backends defined in VCL -> always listed b) Backends/directors created by VMODS -> vmod decides. I think it still makes sense to register the be/dir with the VRT, in order to get the same cleanup-behaviour. A flag on the VRT call will mark it as "unlisted" and disable VSC allocation. (Or malloc dummy VSC to reduce special-casing ?) 3. Setting non-HTTP directors sick/healthy from CLI makes sense 4. Directors/backends created by VMODs should be tagged so it can be seen in CLI which VMOD created them. (Possibly also line# ?) 5. The VRT code will take a "unique name" flag argument. When unset, that code will uniqu-ify the backend name with a ".%u" suffix if necessary. When set the call will fail if a backend already exists with that name. 6. We should support health probing for non-http directors. Some of the bitmaps apply only to HTTP based directors. The overkill version would be for the director to tell which and how many bitmaps it wants to record for health-probes. That could massively complicate VSC handling. A "sneaky" way would be to have VSC contain a fixed number of bitmaps, and let the director name these. This "only" require a place to stash the names and varnishstat code to pull that info out (A per director implementation VSC segment?) 7. struct VSC_C_vbe seems to be general enough that it makes sense for any kind of director, so there is no point in decoupling CLI/VSC. The be->vsc pointer needs to be moved up to the director structure. This may be more messy than it sounds. Comments ? (After this is settled, we'll look at which parts we can also put in 4.1.x in ABI/API compatible ways) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Wed Nov 4 10:46:59 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 04 Nov 2015 10:46:59 +0000 Subject: IPv[46] parsing In-Reply-To: References: <74327.1446022709@critter.freebsd.dk> Message-ID: <91451.1446634019@critter.freebsd.dk> -------- In message , Federico Schwindt writes: >Any comments? Sounds sensible, send patch. >> My proposal >> ----------------- >> >> Change 3 as follows: >> >> 3. If the string doesn't have a space, but has a single ":" take >> everything up to the ":" (but excluding it) as a host, and anything >> after as the port. If the string has 2 or more ":", take the whole >> string as a host without port. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From fgsch at lodoss.net Wed Nov 4 10:54:17 2015 From: fgsch at lodoss.net (Federico Schwindt) Date: Wed, 4 Nov 2015 10:54:17 +0000 Subject: IPv[46] parsing In-Reply-To: <91451.1446634019@critter.freebsd.dk> References: <74327.1446022709@critter.freebsd.dk> <91451.1446634019@critter.freebsd.dk> Message-ID: Attached. On Wed, Nov 4, 2015 at 10:46 AM, Poul-Henning Kamp wrote: > -------- > In message < > CAJV_h0ZY3E4t5704qOHQbnQXJGxnh1hDgMneybuBFxN1jhWsgw at mail.gmail.com> > , Federico Schwindt writes: > > >Any comments? > > Sounds sensible, send patch. > > >> My proposal > >> ----------------- > >> > >> Change 3 as follows: > >> > >> 3. If the string doesn't have a space, but has a single ":" take > >> everything up to the ":" (but excluding it) as a host, and anything > >> after as the port. If the string has 2 or more ":", take the whole > >> string as a host without port. > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ip6.patch Type: text/x-diff Size: 504 bytes Desc: not available URL: From phk at phk.freebsd.dk Wed Nov 4 10:58:27 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 04 Nov 2015 10:58:27 +0000 Subject: [PATCH] PRIV pointers available in vmod obj methods In-Reply-To: <562E2362.2050104@uplex.de> References: <562E2362.2050104@uplex.de> Message-ID: <91548.1446634707@critter.freebsd.dk> -------- Ok, wrapping this subject up: Right now PRIV_foo is named with a mix of lifetime and visibility, and we seem to need to be able to control both aspects. The easiest and most backward compatible, seems to be to make the naming PRIV_lifetime[_visibility] With a default visibility of "VMOD" with other possible values being "CALL" and "OBJ" Three of the the current four PRIV's maps naturally: PRIV_VCL -> PRIV_VCL_VMOD PRIV_TOP -> PRIV_TOP_VMOD PRIV_TASK -> PRIV_TASK_VMOD But: PRIV_CALL -> PRIV_VCL_CALL The functionality Geoff requires would be PRIV_TASK_OBJ If we special-case PRIV_CALL, this is backwards compatible. Comments ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Wed Nov 4 11:04:41 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 04 Nov 2015 11:04:41 +0000 Subject: IPv[46] parsing In-Reply-To: References: <74327.1446022709@critter.freebsd.dk> <91451.1446634019@critter.freebsd.dk> Message-ID: <91674.1446635081@critter.freebsd.dk> -------- In message , Federico Schwindt writes: Go for it. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From kly at redpill-linpro.com Wed Nov 4 11:32:33 2015 From: kly at redpill-linpro.com (Kristian Lyngstol) Date: Wed, 4 Nov 2015 12:32:33 +0100 Subject: Varnish project autumn cleaning In-Reply-To: <33994DC0-2129-4F20-8A31-C6F566542D1A@dwim.org> References: <20151027161427.GA1274@immer.varnish-software.com> <20151030094930.GA14933@immer.varnish-software.com> <20151031092506.GB5176@immer.varnish-software.com> <44520.1446285081@critter.freebsd.dk> <20151103082337.GA1795@leia.redpill-linpro.com> <33994DC0-2129-4F20-8A31-C6F566542D1A@dwim.org> Message-ID: <20151104113233.GA1766@leia.redpill-linpro.com> On Tue, Nov 03, 2015 at 09:13:31AM -0800, Jos Boumans wrote: > > On Nov 3, 2015, at 12:23 AM, Kristian Lyngstol wrote: > >> On Mon, Nov 02, 2015 at 10:22:30AM -0800, Jos Boumans wrote: > >> What?s the gating factor here for you? Time, cost, expertise? > >> Is there a way the community can help you? > > > > Who is "you?? > > That would be those identifying themselves as ?we? in the original > post. Presumably the same great folks who have offered & maintained: > > https://www.varnish-cache.org/installation/debian > https://www.varnish-cache.org/installation/redhat > > See here, for your convenience, and apologies if that caused confusion: No worries. It sounded like you suggested that phk should start packaging software for a moment. > >>>> On Fri, Oct 30, 2015 at 10:49:32AM +0100, Lasse Karstensen wrote: > >>>>> On Tue, Oct 27, 2015 at 05:37:58PM +0000, Federico Schwindt wrote: > >>>> [..] > >>>>>> Are we going to continue providing packages? > >>>>> Consensus seem to be no on the package building. > > > Lasse's mail is mostly about moving responsibility away from V-S over to > > where it rightfully belongs: With the appropriate distros. > > The problem with that is that distros don?t update packages as upstream > releases newer versions. They only release new versions of the OS, or > if needed, security updates. (...) Ingvar and ssm represent downstream and are working on exactly this issue, sorry if that wasn't evident in my mail. The plan for Debian is to provide updated packages through backports, and Ingvar is working on the solution[1] for EPEL and Fedora. As both ssm and Ingvar are active users of the software and packages, that should more than cover those distros. That still leaves Ubuntu, where I figure a PPA is in order. No details on who will end up with that responsibility yet, but it'll have to be resolved before repo.varnish-cache.org can be discontinued (or whatever its fate will be). This means that instead of using repo.varnish-cache.org, you will get the same type of updates from backports.debian.org and ingvar's coprs-repo. Nobody will have have to wait for a new Debian stable or RHEL release to get a newer Varnish version. [1] https://copr.fedoraproject.org/coprs/ingvar/varnish41/builds/ > Any thoughts & constructive criticism much appreciated, I think we're pretty much on the same page, just a slight misunderstanding. No change in what's available except for the better, but possible change in where to get it. Relevant guides/pages will obviously be updated. - Kristian From jos at dwim.org Wed Nov 4 17:38:14 2015 From: jos at dwim.org (Jos Boumans) Date: Wed, 4 Nov 2015 09:38:14 -0800 Subject: Varnish project autumn cleaning In-Reply-To: <20151104113233.GA1766@leia.redpill-linpro.com> References: <20151027161427.GA1274@immer.varnish-software.com> <20151030094930.GA14933@immer.varnish-software.com> <20151031092506.GB5176@immer.varnish-software.com> <44520.1446285081@critter.freebsd.dk> <20151103082337.GA1795@leia.redpill-linpro.com> <33994DC0-2129-4F20-8A31-C6F566542D1A@dwim.org> <20151104113233.GA1766@leia.redpill-linpro.com> Message-ID: <2BA2B45F-8DD0-4795-82AB-9E620ADB73C6@dwim.org> > On Nov 4, 2015, at 3:32 AM, Kristian Lyngstol wrote: > > The plan for Debian is to provide updated packages through backports, > and Ingvar is working on the solution[1] for EPEL and Fedora. As both > ssm and Ingvar are active users of the software and packages, that > should more than cover those distros. That still leaves Ubuntu, where I > figure a PPA is in order. For Debian, EPEL & Fedora, that sounds like a perfrect solution. Regarding PPAs, there?s one snag: A PPA can only host 1 version of a package at a time. From https://help.launchpad.net/Packaging/PPA/Draft: >>> Each user and team in Launchpad can have a single public PPA. If you want to have different versions of the same package, testing different features or focused on different use cases, then we would encourage you to create a new team and use the PPA for that team. That way, for example, you can have a team of people interested in "server" issues that has one version of the Apache package, and another interested in "workstation" issues that has a different version of the same package, each in a different PPA. The obvious drawback here is that it forces people to roll forward with Varnish releases, or the Ubuntu maintainer to host lots of teams & PPAs. Perhaps this could be an alternative? They seem to offer OSS packages: https://packagecloud.io/ > I think we're pretty much on the same page, just a slight > misunderstanding. No change in what's available except for the better, > but possible change in where to get it. Relevant guides/pages will > obviously be updated. Awesome - thanks for clarifying. ? Entropy isn?t what it used to be... -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 163 bytes Desc: Message signed with OpenPGP using GPGMail URL: From rafaelfz at taghos.com.br Fri Nov 6 10:40:16 2015 From: rafaelfz at taghos.com.br (Rafael Zalamena) Date: Fri, 6 Nov 2015 08:40:16 -0200 Subject: Proxy protocol v1 client port fix Message-ID: <20151106084016.5de96c85@taghos.com.br> After giving the proxy feature a try I noticed the client port collected in the vpx_proto1() function is wrong. I read the code and noticed it splits the proxy line in tokens and then when reading the client port it picks the wrong token. 0 1 2 3 4 PROXY PROTO_IPVX SRCADDR DSTADDR SRCPORT DSTPORT\r\n The code is partially working because when it resolves the address it uses the right token: cache_proxy_proto.c:line 110 i = getaddrinfo(fld[1], fld[3], &hints, &res); The proxy version 2 part seems to be working correctly. The diff below fixes the problem I just reported. diff --git a/bin/varnishd/proxy/cache_proxy_proto.c b/bin/varnishd/proxy/cache_proxy_proto.c index 604d796..6f64f3a 100644 --- a/bin/varnishd/proxy/cache_proxy_proto.c +++ b/bin/varnishd/proxy/cache_proxy_proto.c @@ -125,7 +125,7 @@ vpx_proto1(const struct worker *wrk, struct req *req) SES_Reserve_client_addr(req->sp, &sa); AN(VSA_Build(sa, res->ai_addr, res->ai_addrlen)); SES_Set_String_Attr(req->sp, SA_CLIENT_IP, fld[1]); - SES_Set_String_Attr(req->sp, SA_CLIENT_PORT, fld[2]); + SES_Set_String_Attr(req->sp, SA_CLIENT_PORT, fld[3]); freeaddrinfo(res); i = getaddrinfo(fld[2], fld[4], &hints, &res); From fgsch at lodoss.net Sat Nov 7 18:47:31 2015 From: fgsch at lodoss.net (Federico Schwindt) Date: Sat, 7 Nov 2015 18:47:31 +0000 Subject: Proxy protocol v1 client port fix In-Reply-To: <20151106084016.5de96c85@taghos.com.br> References: <20151106084016.5de96c85@taghos.com.br> Message-ID: Committed. Thank for the patch. On Fri, Nov 6, 2015 at 10:40 AM, Rafael Zalamena wrote: > After giving the proxy feature a try I noticed the client port collected > in the vpx_proto1() function is wrong. I read the code and noticed it > splits the proxy line in tokens and then when reading the client port it > picks the wrong token. > > 0 1 2 3 4 > PROXY PROTO_IPVX SRCADDR DSTADDR SRCPORT DSTPORT\r\n > > The code is partially working because when it resolves the address it uses > the right token: > cache_proxy_proto.c:line 110 > i = getaddrinfo(fld[1], fld[3], &hints, &res); > > The proxy version 2 part seems to be working correctly. > > The diff below fixes the problem I just reported. > > diff --git a/bin/varnishd/proxy/cache_proxy_proto.c > b/bin/varnishd/proxy/cache_proxy_proto.c > index 604d796..6f64f3a 100644 > --- a/bin/varnishd/proxy/cache_proxy_proto.c > +++ b/bin/varnishd/proxy/cache_proxy_proto.c > @@ -125,7 +125,7 @@ vpx_proto1(const struct worker *wrk, struct req *req) > SES_Reserve_client_addr(req->sp, &sa); > AN(VSA_Build(sa, res->ai_addr, res->ai_addrlen)); > SES_Set_String_Attr(req->sp, SA_CLIENT_IP, fld[1]); > - SES_Set_String_Attr(req->sp, SA_CLIENT_PORT, fld[2]); > + SES_Set_String_Attr(req->sp, SA_CLIENT_PORT, fld[3]); > freeaddrinfo(res); > > i = getaddrinfo(fld[2], fld[4], &hints, &res); > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lkarsten at varnish-software.com Mon Nov 9 13:52:55 2015 From: lkarsten at varnish-software.com (Lasse Karstensen) Date: Mon, 9 Nov 2015 14:52:55 +0100 Subject: Patchwork sunsetting. In-Reply-To: <20151103094322.GA27190@immer.varnish-software.com> References: <20151103094322.GA27190@immer.varnish-software.com> Message-ID: <20151109135254.GA10071@immer.varnish-software.com> On Tue, Nov 03, 2015 at 10:43:23AM +0100, Lasse Karstensen wrote: [..] > Patchwork will be shut down as of Monday November 9th 2015. Patchwork has been disabled. Any requests for the old URLs are redirected to the github project pull request page. -- Lasse Karstensen Varnish Software AS From dridi at varni.sh Thu Nov 12 17:44:56 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 12 Nov 2015 18:44:56 +0100 Subject: Preventing dup backend names with dynamic backends in VMODs In-Reply-To: <91251.1446632150@critter.freebsd.dk> References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <5636468A.6060300@uplex.de> <91251.1446632150@critter.freebsd.dk> Message-ID: On Wed, Nov 4, 2015 at 11:15 AM, Poul-Henning Kamp wrote: > -------- > In message , Kacper Wyso > cki writes: > >>Oh. I hadn't seen req.backend.name. I guess the type conversion is a >>little counter to my intuition too. > > I'm not very happy about "req.backend.name", it is *very* magic and not > general to any other backend anywhere else in VCL. > > The real fix would be to allow standardized (ie: builtin) methods on the > VCL_* types themselves, so that one could do things like > > > $backend.ip() > $backend.name() > > etc. > > Implementing it will take some VCC hacking If needed I can take this one once I get enough bandwidth and more precise requirements, I'm getting used to dealing with VCC and backends :) Dridi From dridi at varni.sh Thu Nov 12 17:49:40 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 12 Nov 2015 18:49:40 +0100 Subject: [PATCH] add VCL_HasBackend() (Re: Preventing dup backend names with dynamic backends in VMODs) In-Reply-To: <5635F7CE.6000902@uplex.de> References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <5635F63F.1060109@uplex.de> <5635F7CE.6000902@uplex.de> Message-ID: On Sun, Nov 1, 2015 at 12:30 PM, Geoff Simmons wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > On 11/01/2015 12:23 PM, Geoff Simmons wrote: >> >> This patch adds VCL_HasBackend(). > > Forgot to mention about this -- it might be necessary to lock the > vcl_mtx while traversing the vcl->backend_list, since a backend may be > added while the query is executing. As discussed in another thread, such interface would be subject to TOCTOU races too. From dridi at varni.sh Thu Nov 12 18:39:59 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 12 Nov 2015 19:39:59 +0100 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: <91356.1446633622@critter.freebsd.dk> References: <562E4736.5030708@uplex.de> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> <52611.1446421347@critter.freebsd.dk> <54220.1446450731@critter.freebsd.dk> <54424.1446452436@critter.freebsd.dk> <56373B79.10300@uplex.de> <77098.1446461234@critter.freebsd.dk> <81765.1446463657@critter.freebsd.dk> <82533.1446467843@critter.freebsd.dk> <563894B7.1020904@uplex.de> <91356.1446633622@critter.freebsd.dk> Message-ID: On Wed, Nov 4, 2015 at 11:40 AM, Poul-Henning Kamp wrote: > -------- > In message <563894B7.1020904 at uplex.de>, Geoff Simmons writes: > >>> Why not keep VRT_new_backend as the only interface for VMODs and >>> allow this one to fail? (or do the atomic auto-renaming if needed) >> >>Um, VRT_new_backend creates a conventional backend (HTTP over TCP) >>from a vrt_backend config. What if you want to do something entirely >>different? > > Ok, I think we need to wrap this discussion up and move on. > > My conclusion for the 5.x perspective is the following: > > 1. Our terminology sucks, in particular the director/backend confusion. Agreed. > 2. We will not force all directors/backends to appear in CLI/VSC > a) Backends defined in VCL -> always listed > b) Backends/directors created by VMODS -> vmod decides. > I think it still makes sense to register the > be/dir with the VRT, in order to get the same > cleanup-behaviour. A flag on the VRT call will > mark it as "unlisted" and disable VSC allocation. > (Or malloc dummy VSC to reduce special-casing ?) Sounds good. > 3. Setting non-HTTP directors sick/healthy from CLI makes sense It does, which means we'd need to make this check in varnishd before asking the backend. That may already be the case. > 4. Directors/backends created by VMODs should be tagged so it can > be seen in CLI which VMOD created them. (Possibly also line# ?) Why not. It could be something carried by a VRT_CTX, set up by the VCC. Of course nothing prevents a vmod author to build a context without the vmod tag/field so it would need to be properly documented. > 5. The VRT code will take a "unique name" flag argument. > When unset, that code will uniqu-ify the backend name with a > ".%u" suffix if necessary. When set the call will fail if a > backend already exists with that name. Why not. It could be useful if combined with 4 for error logging. > 6. We should support health probing for non-http directors. > Some of the bitmaps apply only to HTTP based directors. > The overkill version would be for the director to tell which > and how many bitmaps it wants to record for health-probes. > That could massively complicate VSC handling. > A "sneaky" way would be to have VSC contain a fixed number > of bitmaps, and let the director name these. This "only" > require a place to stash the names and varnishstat code to > pull that info out (A per director implementation VSC segment?) I suggest the probe would use the director's functions, just like a bereq. But we'd need to also tell the director not to try recycling a connection for instance. > 7. struct VSC_C_vbe seems to be general enough that it makes sense > for any kind of director, so there is no point in decoupling > CLI/VSC. The be->vsc pointer needs to be moved up to the > director structure. This may be more messy than it sounds. Small nitpick, it has a conn field and as you pointed out, we could have connection-less directors. > Comments ? > > (After this is settled, we'll look at which parts we can also > put in 4.1.x in ABI/API compatible ways) Sounds like a good plan! Best, Dridi From dridi at varni.sh Thu Nov 12 23:29:10 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Fri, 13 Nov 2015 00:29:10 +0100 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: <91356.1446633622@critter.freebsd.dk> References: <562E4736.5030708@uplex.de> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> <52611.1446421347@critter.freebsd.dk> <54220.1446450731@critter.freebsd.dk> <54424.1446452436@critter.freebsd.dk> <56373B79.10300@uplex.de> <77098.1446461234@critter.freebsd.dk> <81765.1446463657@critter.freebsd.dk> <82533.1446467843@critter.freebsd.dk> <563894B7.1020904@uplex.de> <91356.1446633622@critter.freebsd.dk> Message-ID: > Comments ? > > (After this is settled, we'll look at which parts we can also > put in 4.1.x in ABI/API compatible ways) 8. set the default backend/director in vcl_init set default_backend = rr.backend(); The default backend cannot be removed. If no default backend is set, the first one is picked unless there's one actually named "default". 9. remove the need for at least one native backend As long as a at least one backend is registered when the VCL is loaded, a user should not need to declare a dummy backend when they only use VMOD backends. Best, Dridi From phk at phk.freebsd.dk Thu Nov 12 23:31:34 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 12 Nov 2015 23:31:34 +0000 Subject: [PATCH] add documentation about backend naming for VMOD authors In-Reply-To: References: <562E4736.5030708@uplex.de> <56362885.1000700@uplex.de> <52334.1446414566@critter.freebsd.dk> <52611.1446421347@critter.freebsd.dk> <54220.1446450731@critter.freebsd.dk> <54424.1446452436@critter.freebsd.dk> <56373B79.10300@uplex.de> <77098.1446461234@critter.freebsd.dk> <81765.1446463657@critter.freebsd.dk> <82533.1446467843@critter.freebsd.dk> <563894B7.1020904 @uplex.de> <91356.14466336 Message-ID: <67567.1447371094@critter.freebsd.dk> -------- Just a note that I'm off the clock with some kind of throat infection and I'm in no shape to think or make decisions. Hope to be back monday. In message , Dridi Boukelmoune writes: >> Comments ? >> >> (After this is settled, we'll look at which parts we can also >> put in 4.1.x in ABI/API compatible ways) > >8. set the default backend/director in vcl_init > >set default_backend = rr.backend(); > >The default backend cannot be removed. If no default backend is set, >the first one is picked unless there's one actually named "default". > >9. remove the need for at least one native backend > >As long as a at least one backend is registered when the VCL is >loaded, a user should not need to declare a dummy backend when they >only use VMOD backends. > >Best, >Dridi > -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From geoff at uplex.de Sun Nov 15 17:58:21 2015 From: geoff at uplex.de (Geoff Simmons) Date: Sun, 15 Nov 2015 18:58:21 +0100 Subject: VTC for connect_timeout? Message-ID: <5648C7BD.8070804@uplex.de> Hello all, I noticed that there is no test case for backend connect_timeout among the Varnish *.vtc's, so I tried to cook one up for a VMOD, but I can't seem to get a backend connection to fail due to the timeout when I think it should. The attached VTC just uses standard Varnish, and doesn't pass. I can't see what could be wrong about VTCP_connect(), so I assume that my reasoning about the test case must be flawed. The first request gets "Connection: close" from the backend, and I can see BackendClose in the log. Then the client sends the second request: ** c1 0.6 === txreq **** c1 0.6 txreq| GET / HTTP/1.1\r\n **** c1 0.6 txreq| \r\n ** c1 0.6 === rxresp But the backend delays 1.5 seconds, which is longer than connect_timeout, before accepting again: ** s1 0.6 === delay 1.5 *** s1 0.6 delaying 1.5 second(s) [...] ** s1 2.1 === accept **** s1 2.1 Accepting *** s1 2.1 Accepted socket fd is 4 ** s1 2.1 === rxreq So I thought the timeout should elapse and the response should be 503, but it's 200, no failure. How am I getting this wrong? Thanks, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- varnishtest "test connect_timeout" server s1 { rxreq txresp -hdr "Connection: close" delay 1.5 accept rxreq txresp } -start varnish v1 -vcl { backend default { .host = "${s1_addr}"; .port = "${s1_port}"; .connect_timeout = 1s; } sub vcl_recv { return(pass); } sub vcl_backend_fetch { set bereq.connect_timeout = 1s; } sub vcl_backend_response { set beresp.do_stream = false; } } -start varnish v1 -cliok "param.set connect_timeout 1" client c1 { txreq rxresp expect resp.status == 200 txreq rxresp expect resp.status == 503 } -run server s1 { rxreq txresp -hdr "Connection: close" delay 0.5 accept rxreq txresp } -start client c1 { txreq rxresp expect resp.status == 200 txreq rxresp expect resp.status == 200 } -run -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From geoff at uplex.de Sun Nov 15 22:02:46 2015 From: geoff at uplex.de (Geoff Simmons) Date: Sun, 15 Nov 2015 23:02:46 +0100 Subject: Assertion failure in VCL_DelBackend() Message-ID: <56490106.6020309@uplex.de> Hello all, The patch and test case in the attachment demonstrate VCL_DelBackend() crashing on assertion failure, at least when called from a VMOD. It's failing AN(ptr) on line 288 of common/common_vsm.c, because be->vsc is NULL (VSM_Free(be->vsc) called by VBE_Event()). The crash happens when you attempt to remove a statically declared backend. Oddly, I seem to have no problem at all calling VCL_DelBackend() on a dynamic backend created within the VMOD (using VRT_new_backend()). Bug report? (Are we still using trac for that, or are we into the Brave New World now?) I've noticed a few other strange things about VCL_DelBackend: - n_backend is evidently not decremented - the deleted backend still appears in the output of backend.list Best, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- varnishtest "VCL_DelBackend" server s1 { } -start varnish v1 -vcl { import ${vmod_debug}; backend s1 { .host="${s1_addr}"; .port="${s1_port}"; } sub vcl_recv { if (req.method == "DELETE") { if (debug.delete(s1)) { return(synth(204, "No Content")); } else { return(synth(404, "Not found")); } } } } -start client c1 { txreq -req "DELETE" rxresp expect resp.status == 204 txreq -req "DELETE" rxresp expect resp.status == 404 } -run -------------- next part -------------- A non-text attachment was scrubbed... Name: VCL_DelBackend.patch Type: text/x-patch Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From dridi at varni.sh Sun Nov 15 23:24:50 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 16 Nov 2015 00:24:50 +0100 Subject: Assertion failure in VCL_DelBackend() In-Reply-To: <56490106.6020309@uplex.de> References: <56490106.6020309@uplex.de> Message-ID: > The patch and test case in the attachment demonstrate VCL_DelBackend() > crashing on assertion failure, at least when called from a VMOD. It's > failing AN(ptr) on line 288 of common/common_vsm.c, because be->vsc is > NULL (VSM_Free(be->vsc) called by VBE_Event()). I believe you're not supposed to use VCL_DelBackend directly, and static backends are owned by the VCL. You're not supposed to tamper with them. What is the use case? > Bug report? (Are we still using trac for that, or are we into the Brave > New World now?) I think not. From geoff at uplex.de Mon Nov 16 09:26:58 2015 From: geoff at uplex.de (Geoff Simmons) Date: Mon, 16 Nov 2015 10:26:58 +0100 Subject: Assertion failure in VCL_DelBackend() In-Reply-To: References: <56490106.6020309@uplex.de> Message-ID: <5649A162.104@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 11/16/2015 12:24 AM, Dridi Boukelmoune wrote: > > I believe you're not supposed to use VCL_DelBackend directly, and > static backends are owned by the VCL. You're not supposed to > tamper with them. > > What is the use case? It's for a VMOD to manage backends at runtime, meaning create and delete them. The use case takes us a little far afield, so let me come back to that further down. VCL_DelBackend is declared in $include_dir/cache/cache_backend.h, so it's available to a VMOD, and there's no indication that it's "not supposed to" be used. I've suspected that there are things in $include_dir that aren't really meant to be public, but if we're serious about that, and if Varnish is liable to crash if a VMOD does use them, then perhaps we should separate out the non-public interfaces. Or at least document them as non-public -- we have that in the *_int.h includes in vapi, for example. But it would be a real letdown if VCL_DelBackend is made unavailable with no replacement for removing backends. I don't see any other current interface that can do it. As I mentioned, VCL_DelBackend does work for the dynamic backends that the VMOD creates (with VRT_new_backend), and I haven't investigated why they're different from the static backends. I can live with a VMOD that can only remove its own backends, but the call seemed to me to be general enough for static backends as well, so I thought why not. >> Bug report? (Are we still using trac for that, or are we into the >> Brave New World now?) > > I think not. You think ... there should be no bug report? Or that we're not using trac any more? Or that we're not using github for bug tracking ... ? ... about that use case: It's for "microservices", which are lightweight backend apps running in docker containers that are managed by mesos/marathon, and are rapidly created, removed and migrated. So rapidly that we couldn't get it to work any more by changing the backend config in VCL and running vcl.load/vcl.use. We've been forced to add haproxies as an intervening layer, which are backends for Varnish, and they in turn get the changing backend configs. I'd like to use dynamic backends in 4.1 to be able to get rid of the extra layer, and for that, we need to be able to remove backends at runtime. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCAAGBQJWSaFXAAoJEOUwvh9pJNURVtQP/ibhGWpmvghc+EZ1kbDvrWly GyehuFE7Cq30yMQg4n4ZoH34stI2jV5ZASj8D7RAvDlygAXGr9Ow+ykdACRxrew3 dh2PqF/6ET/vcskyzIGBSC8LKUTu2avWZoKZD86YU++sggIQ+L+KDq4yvLUQyPGC nno29WdMnGFFZxuZb1G085kpq0UTwvpeIjY5NAkHEWezjfr5sQxGlJ4GI2Tku6SR kYO+AbSw8RgXsMczdw6wUheCiwG2bja3ZjYRpnyASPNjUpHkxnXCP7SNxd29EW8m uN+EqnK982dbbSv6UZURUfNsnCIFzgg5X+XvkI1TaA/Sg1QwuqUOYroO+hUvOM23 g+vo1rqeLYKbvSK1ioAhhPQcnPT4m3HHp+TxeeC6Ij7P3OASfBHG8WnzWIcO/lJd LLbhsRdwlt7bAlL8gg/TTtV+HvHQH314iDMLGslUEC1t2bzxOMVUFmYqAfJT7msr HoZxIdE+SPZB0Wyx/F0Xo4K1s8cCviY762govkVRR2rXf8r9aA6ca53BlzNrxMxQ 7NJZi7HRLvUTlHNoGguQe+p0rqsZJx0SdizRSbD6XYBUk5AXT8V7PB3/2cSRw1Sa BhDUeP29crcQAMM97YwfXF+h9uc+e7BHTIkJZT7ySX8u5OYMo/yWy+uMWuQzcHku TJe8VRy3ilxCTpyHBZiU =GR3f -----END PGP SIGNATURE----- From dridi at varni.sh Mon Nov 16 09:53:30 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 16 Nov 2015 10:53:30 +0100 Subject: Assertion failure in VCL_DelBackend() In-Reply-To: <5649A162.104@uplex.de> References: <56490106.6020309@uplex.de> <5649A162.104@uplex.de> Message-ID: > VCL_DelBackend is declared in $include_dir/cache/cache_backend.h, so > it's available to a VMOD, and there's no indication that it's "not > supposed to" be used. I've suspected that there are things in > $include_dir that aren't really meant to be public, but if we're > serious about that, and if Varnish is liable to crash if a VMOD does > use them, then perhaps we should separate out the non-public > interfaces. Or at least document them as non-public -- we have that in > the *_int.h includes in vapi, for example. If it were up to me I'd simply remove everything that doesn't start with VRT_, except of course symbols from libvarnish* and related data structures :) > But it would be a real letdown if VCL_DelBackend is made unavailable > with no replacement for removing backends. I don't see any other > current interface that can do it. VRT_delete_backend ? > As I mentioned, VCL_DelBackend does work for the dynamic backends that > the VMOD creates (with VRT_new_backend), and I haven't investigated > why they're different from the static backends. I can live with a VMOD > that can only remove its own backends, but the call seemed to me to be > general enough for static backends as well, so I thought why not. I don't even know how you would delete the backend without breaking the type system. If you reference it from a VMOD you should get a const struct director and the actual struct backend should be hidden behind the priv pointer. You should get enough hints that you're not supposed to delete them at this point :p >>> Bug report? (Are we still using trac for that, or are we into the >>> Brave New World now?) >> >> I think not. > > You think ... there should be no bug report? Or that we're not using > trac any more? Or that we're not using github for bug tracking ... ? Sorry, I meant there should not be a bug report. > ... about that use case: It's for "microservices", which are > lightweight backend apps running in docker containers that are managed > by mesos/marathon, and are rapidly created, removed and migrated. So > rapidly that we couldn't get it to work any more by changing the > backend config in VCL and running vcl.load/vcl.use. We've been forced > to add haproxies as an intervening layer, which are backends for > Varnish, and they in turn get the changing backend configs. I'd like > to use dynamic backends in 4.1 to be able to get rid of the extra > layer, and for that, we need to be able to remove backends at runtime. Sounds good, I have been doing that with DNS :) If you have some sort of API to get the backends, you can go with dynamic-only backends. You still need at least one static backend to please libvcc, but I had a patch to remove this limitation in the past. I could re-submit if it makes more sense this time. The reason it was rejected initially was because it would defer the backend sanity check to vcl_init{} but since then it has become possible to fail vcl_init{} so maybe it could be reconsidered. Cheers, Dridi From phk at phk.freebsd.dk Mon Nov 16 09:55:54 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 16 Nov 2015 09:55:54 +0000 Subject: Assertion failure in VCL_DelBackend() In-Reply-To: <5649A162.104@uplex.de> References: <56490106.6020309@uplex.de> <5649A162.104@uplex.de> Message-ID: <15918.1447667754@critter.freebsd.dk> -------- In message <5649A162.104 at uplex.de>, Geoff Simmons writes: > VCL_DelBackend is declared in $include_dir/cache/cache_backend.h, so > it's available to a VMOD, and there's no indication that it's "not > supposed to" be used. I've suspected that there are things in > $include_dir that aren't really meant to be public, One of my background-tasks is to put the "hands-off!" items in clearly marked #include files. cache/cache_priv.h is the first part of that. > But it would be a real letdown if VCL_DelBackend is made unavailable > with no replacement for removing backends. I don't see any other > current interface that can do it. It's monday morning. Power has been out twice already. I was down with a flu-ish infection for the last five days and my brain only started working again yesterday. So I can't give you a definitive VCL_DelBackend() answer right now. But what I will say, is that we have a general object-lifetime issue, which so far is related to both VMOD_PRIV's and backends, and it is complicated. One example is: if (foomod.get_me_a_new_backend() == backend_1) { // do something } How long does that backend live ? Who decides ? - and how ? What about if (amod.get_blob() == bmod.get_blob()) { // do something } How long time does those blobs live ? My current thinking is what I outlined last week(?) about VMOD_PRIV, that VMODs will have to become explicit about this. If we do that and make backends (ie: directors) VMOD_PRIV's, then a VMOD would be able to create a backend and mark it for deletion when the transaction dies, and have it happen automatically. That widget may not be a VMOD_PRIV anymore, but rather a "VMOD_OBJ" which has a "Interface Spec" ("I'm a director") and a "Method-Vector" (the ->getfd(), ->gethdr() C-functions etc.) Needless to say that is not back-portable 4.x stuff. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Mon Nov 16 10:27:11 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 16 Nov 2015 10:27:11 +0000 Subject: Preventing dup backend names with dynamic backends in VMODs In-Reply-To: References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <5636468A.6060300@uplex.de> <91251.1446632150@critter.freebsd.dk> Message-ID: <21179.1447669631@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >> The real fix would be to allow standardized (ie: builtin) methods on the >> VCL_* types themselves, so that one could do things like >> >> $backend.ip() >> $backend.name() >> >> Implementing it will take some VCC hacking > >If needed I can take this one once I get enough bandwidth and more >precise requirements, I'm getting used to dealing with VCC and >backends :) Please hold off until we have a design for the entire VMOD_PRIV/VMOD_OBJ/Backend thing. I want a clean future-proof solution rather than more hacks. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From geoff at uplex.de Mon Nov 16 11:00:51 2015 From: geoff at uplex.de (Geoff Simmons) Date: Mon, 16 Nov 2015 12:00:51 +0100 Subject: Assertion failure in VCL_DelBackend() In-Reply-To: References: <56490106.6020309@uplex.de> <5649A162.104@uplex.de> Message-ID: <5649B763.7080405@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 11/16/2015 10:53 AM, Dridi Boukelmoune wrote: > > VRT_delete_backend ? Sure. > I don't even know how you would delete the backend without > breaking the type system. If you reference it from a VMOD you > should get a const struct director and the actual struct backend > should be hidden behind the priv pointer. You should get enough > hints that you're not supposed to delete them at this point :p Yes, I'm taking advantage of the knowledge that dir->priv is a struct backend, which is indeed a clear sign that I'm peeking behind the curtain. (I need the delete, so I confess to cheating.) phk has the public/private separation in the queue, so we'll be OK. But even without separating the #include's, just a clear statement in the comments such as we currently have in vsm_int.h and vsl_int.h is really enough. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCAAGBQJWSbdjAAoJEOUwvh9pJNURzUAP/3IxBYqRWW8rKG2Gxe2UBvg8 X9nueS0FrdE4Ul2xZhcDUqkiSOClmAYCwptyWAlD3ZIc1MaeZ+oMLq2jV6vreJYo HMOQYGvYG/yBgf1jfUUH+Yu3ENoPmtef0uQVXH44UFkv5F/35t0Ud9X6jdqQqInM rM9qxFV9uChkV+54D/Pg4tNOPKam0I6Z71u3hOgjWSP2g5fmgMJaMuX4T2OK7yTc mZIbCrz5x8IXjhYeamHKPFpiq23oafmiTeUlbfZmzB4c3wMSgypgkIoBlkiqGu+P 2E5hnMmMql6Af/58jbg578dW+q4obcdIVnnMagmE5LKkXR85JoIsAMw3/aQko+6F NjV2MgzIM6DDY4CAlEsn7WA3xQTAzXg42RxdjlWoC5bCfxd+jq9spPTraxBcvAp0 9P7XAlS2ATCqhsoTP4sNgixKNBXH7ZWR3UAn+INRfuoVkTk2/OTCYybD1v8bXnM7 IFr1DYdjYLNZrXiJFVc+THOWIOgM4TiRXj1h4ei3FlSwV+5USsYTkzXQgAdqOn00 0+pJTVPLX6mO69maFep/r0xEVI9giPojZqgE94D80me8gMv6MEb5r0X+Z+dtl9V3 A7SRIhQzjOrZEH8A0ZaUQhuJSFYrdQW+M2VK/Kq/dXpl6Jo512GaPT+wCqJ+CIw7 l8Ccoo8RAOAEpl/aeEvI =El29 -----END PGP SIGNATURE----- From dridi at varni.sh Mon Nov 16 11:23:09 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 16 Nov 2015 12:23:09 +0100 Subject: Preventing dup backend names with dynamic backends in VMODs In-Reply-To: <21179.1447669631@critter.freebsd.dk> References: <562E4736.5030708@uplex.de> <8599.1445948729@critter.freebsd.dk> <562F91DA.8010104@uplex.de> <74085.1446020238@critter.freebsd.dk> <5636468A.6060300@uplex.de> <91251.1446632150@critter.freebsd.dk> <21179.1447669631@critter.freebsd.dk> Message-ID: On Mon, Nov 16, 2015 at 11:27 AM, Poul-Henning Kamp wrote: > -------- > In message > , Dridi Boukelmoune writes: > >>> The real fix would be to allow standardized (ie: builtin) methods on the >>> VCL_* types themselves, so that one could do things like >>> >>> $backend.ip() >>> $backend.name() >>> >>> Implementing it will take some VCC hacking >> >>If needed I can take this one once I get enough bandwidth and more >>precise requirements, I'm getting used to dealing with VCC and >>backends :) > > Please hold off until we have a design for the entire > VMOD_PRIV/VMOD_OBJ/Backend thing. I want a clean future-proof solution > rather than more hacks. Don't worry, I want to prepare my VCL temperature fixes first as we discussed in Stockholm. I will remove the ABI breakage from when I submitted it before 4.1, hopefully in time for the next VDD. And I have a probably controversial solution I'd like to propose too :) Dridi From geoff at uplex.de Mon Nov 16 11:39:25 2015 From: geoff at uplex.de (Geoff Simmons) Date: Mon, 16 Nov 2015 12:39:25 +0100 Subject: Assertion failure in VCL_DelBackend() In-Reply-To: <15918.1447667754@critter.freebsd.dk> References: <56490106.6020309@uplex.de> <5649A162.104@uplex.de> <15918.1447667754@critter.freebsd.dk> Message-ID: <5649C06D.7070403@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 11/16/2015 10:55 AM, Poul-Henning Kamp wrote: > > One of my background-tasks is to put the "hands-off!" items in > clearly marked #include files. OK > It's monday morning. [...] All good. I can go with what's possible at the moment. Get well. > But what I will say, is that we have a general object-lifetime > issue, which so far is related to both VMOD_PRIV's and backends, > and it is complicated. > > One example is: > > if (foomod.get_me_a_new_backend() == backend_1) { // do something > } > > How long does that backend live ? > > Who decides ? - and how ? FWIW, the VMODs I have in mind are all based around the idea that backends are owned by VCL, which defines their lifetime, unless you explicitly delete them before the VCL is discarded. (Now that you mention it, though, it would be quite useful for our use case to let them live on until the child process dies ...) > My current thinking is what I outlined last week(?) about > VMOD_PRIV, that VMODs will have to become explicit about this. > > If we do that and make backends (ie: directors) VMOD_PRIV's, then a > VMOD would be able to create a backend and mark it for deletion > when the transaction dies, and have it happen automatically. Well OK, but is there really a use case for backends with anything shorter than PRIV_VCL lifetime? Backends that live and die during a task or a top request ... that sounds pretty far out to me. And I can't even picture what PRIV_CALL for a backend would mean. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCAAGBQJWScBtAAoJEOUwvh9pJNURwdAQAIgL/PQCazVsVVcQhDBCPBh+ HnPMv2JJNCpAQ5X3QKe5VFgVI9BqFooRvUTpG7/U2r8vMwBcEPExc6nX+wyFIcdZ WhJwxgUTT5WIAP8g5Nj4vMZvM0Ay/0XaCVX1VhmjNIawdiOXfjjabLRYVsBzz87C sX170Z2j8hMWvAgdREVAbIY4nTZWolGQvdzK2aul7+oPt9UblIsmPdnDV+w5w27A 9EiwQbUV0MKxMicSdL1F6aYRBKAZB0HOcbxrRkI3R0g8PUxpzz4UgBKBRZD08WRj nhjOjwl4b3tTQVhi/AvZBz/TM25eijVlWDi/3DIy4vviejKnVs+UKWuG5Mdgzogr nw07nMRDG+4QMSFazriVxwYaf7q0HmjoNenRQ9xRm0R7lntlYjJDQPgVIyb2daDI mcOy80axNyvyI5VO2dWcr7/H/0v3JqMOpqMeDjMrYjOyy5d3tSXvLsndcS2t1Uti 5mEpIi6mCeAR++NQn77cTyuFkz/T6Hod6Jj0lr+SQhEEFxn4yS1UJkpkgSqoOzGn Lg20OVVr7BOJ+ZH9SEkyQ2f7tz4iVxqIM/ao5Uie9BlZ5S9MIBrZJApXQqJJdA6n 249ZqChQV9zSfX+51LLRp5cOKSjPBNkaDSz6LeDmAQqJaJ2cmCKGkuk+Yr0+fqzL 6N4kPi7SB5XoCOxN5/3w =1/S5 -----END PGP SIGNATURE----- From phk at phk.freebsd.dk Mon Nov 16 11:41:33 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 16 Nov 2015 11:41:33 +0000 Subject: Assertion failure in VCL_DelBackend() In-Reply-To: <5649C06D.7070403@uplex.de> References: <56490106.6020309@uplex.de> <5649A162.104@uplex.de> <15918.1447667754@critter.freebsd.dk> <5649C06D.7070403@uplex.de> Message-ID: <21453.1447674093@critter.freebsd.dk> -------- In message <5649C06D.7070403 at uplex.de>, Geoff Simmons writes: >Well OK, but is there really a use case for backends with anything >shorter than PRIV_VCL lifetime? A backend when popens a CGI command for this request ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From geoff at uplex.de Mon Nov 16 12:54:21 2015 From: geoff at uplex.de (Geoff Simmons) Date: Mon, 16 Nov 2015 13:54:21 +0100 Subject: Assertion failure in VCL_DelBackend() In-Reply-To: <21453.1447674093@critter.freebsd.dk> References: <56490106.6020309@uplex.de> <5649A162.104@uplex.de> <15918.1447667754@critter.freebsd.dk> <5649C06D.7070403@uplex.de> <21453.1447674093@critter.freebsd.dk> Message-ID: <5649D1FD.7020201@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 11/16/2015 12:41 PM, Poul-Henning Kamp wrote: > >> Well OK, but is there really a use case for backends with >> anything shorter than PRIV_VCL lifetime? > > A backend when popens a CGI command for this request ? I'm sure I wouldn't do that by creating and then destroying a struct director every time. Of course the API can be open for unforeseen ideas, but in this case it seems like a real stretch. And as I said earlier, I have no intuition at all about what a PRIV_CALL-scoped backend could be. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCAAGBQJWSdH8AAoJEOUwvh9pJNURGa4P/j/nJkddPcWwsXALJP9Eh+lX 9d2JDatfR0WXKwjWR6X1foeo2pRBXJQKzX66uAYQiWgKy2rXxrLQXeY5fJFehmKx KxH56wx6wRu8Z734MORs1CNLsiy/6gDT3L01IZAr74VVTihtJby7g377dfEOvNLp MRRqzBImIQOYcR4W2ly5cPVSna5y9I2gyn8fPZEFAsizoj0OgjqF3olZuFk2kgbI D7hBtTu7woEPwMAsyZlTzDow8sOT5eOat6IXz8HNR933IJMBio6me53VIVTOCo+/ ydWnx5JWuEJ7loOWEl94B2BW49DzsE2K1YNBEgyma+ePzwWQok59I+FdAbsrEza2 h22+xuguSRFFKvkZU12i0rXtNwqAYMm/HJbZBwUwaT2Afp0DIuJCaVeXhquK9UMV w/g14KjluxHVKjKJIenEtOJ7JAWMsJKYXw1jGdbAliOypiVGcS8umz/ibBHYBEhx iko/GEUnYFB4T+u8GWA8xRgbK3AywmuhniHWDlOU1W76yOIqyo0vaDfZ8iy+avPR v9huUfH7CVS6fBk2auU1UTeld8/YjkR+AmI1fbPF6BwSIFeOS0Z4d70+6RQ0gU7u 5xjVcSYDWFH47zvGkGupnHZUCFot3akNxqFhSMXbchuBQ77iTZiUhRdXGjFZqex9 D21BLdp4J3yzPwXfLZnW =H9Ek -----END PGP SIGNATURE----- From dridi at varni.sh Mon Nov 16 14:28:05 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 16 Nov 2015 15:28:05 +0100 Subject: Assertion failure in VCL_DelBackend() In-Reply-To: <5649C06D.7070403@uplex.de> References: <56490106.6020309@uplex.de> <5649A162.104@uplex.de> <15918.1447667754@critter.freebsd.dk> <5649C06D.7070403@uplex.de> Message-ID: > FWIW, the VMODs I have in mind are all based around the idea that > backends are owned by VCL, which defines their lifetime, unless you > explicitly delete them before the VCL is discarded. > > (Now that you mention it, though, it would be quite useful for our use > case to let them live on until the child process dies ...) It has been discussed in the past, and while nothing prevents you from creating a backend/director that outlives the VCL, you shouldn't. The semantics changed in 4.1 along with VCL temperature and the idea that a cold VCL should have the least possible footprint and not get in the way of the active VCL. This rule is enforced by native backends, but nothing prevents VMODs not to comply. On the other hand, if you have two backends with the same endpoint they will share the same connection pool, even if they come from different VCLs. From dridi at varni.sh Mon Nov 16 14:31:15 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 16 Nov 2015 15:31:15 +0100 Subject: Assertion failure in VCL_DelBackend() In-Reply-To: <5649D1FD.7020201@uplex.de> References: <56490106.6020309@uplex.de> <5649A162.104@uplex.de> <15918.1447667754@critter.freebsd.dk> <5649C06D.7070403@uplex.de> <21453.1447674093@critter.freebsd.dk> <5649D1FD.7020201@uplex.de> Message-ID: >> A backend when popens a CGI command for this request ? > > I'm sure I wouldn't do that by creating and then destroying a struct > director every time. Of course the API can be open for unforeseen > ideas, but in this case it seems like a real stretch. Agreed, deleting a struct director carelessly could lead to dangling pointers in other transactions. Maybe I should document a rule of thumb such as never deleting a struct director that is made available to VCL code. If you delete the director/backend underneath it's fine. You can simply get a NULL director from the resolve function. > And as I said earlier, I have no intuition at all about what a > PRIV_CALL-scoped backend could be. From geoff at uplex.de Mon Nov 16 15:41:37 2015 From: geoff at uplex.de (Geoff Simmons) Date: Mon, 16 Nov 2015 16:41:37 +0100 Subject: Assertion failure in VCL_DelBackend() In-Reply-To: References: <56490106.6020309@uplex.de> <5649A162.104@uplex.de> <15918.1447667754@critter.freebsd.dk> <5649C06D.7070403@uplex.de> <21453.1447674093@critter.freebsd.dk> <5649D1FD.7020201@uplex.de> Message-ID: <5649F931.8010206@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 11/16/2015 03:31 PM, Dridi Boukelmoune wrote: >>> A backend when popens a CGI command for this request ? >> >> I'm sure I wouldn't do that by creating and then destroying a >> struct director every time. Of course the API can be open for >> unforeseen ideas, but in this case it seems like a real stretch. > > Agreed, deleting a struct director carelessly could lead to > dangling pointers in other transactions. > > Maybe I should document a rule of thumb such as never deleting a > struct director that is made available to VCL code. If you delete > the director/backend underneath it's fine. You can simply get a > NULL director from the resolve function. Mmmm, this may be a misunderstanding, the context was PRIV_* scopes for backends, such as PRIV_TASK or PRIV_TOP. I would argue that creating backends at the beginning of such a scope and removing them at the end is such an unlikely solution that we should consider not supporting it. But it sounds like you're saying backends should not be deleted under any circumstances. That I don't agree with. What should I do about backends that cease to exist, in a context where it's not workable to reload a new static backend configuration? That's what I want the runtime backend delete for. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCAAGBQJWSfkxAAoJEOUwvh9pJNURhDAQAJlpbpZFqOyT3A1+VGFdAXaE A5YTUf7tlrIajQZlHfAlR26XxgEfB2pUsUDuVPo7SUvcht4MrLnBy/sZXOMdNt15 Mz/qv2ilB5oF80Lih6E8TB52aghExPfYUQ26bEtKGRaAVNcB1jYhn8NF5SmU36Nn eTbpbUuo5PdfN+HQy4i5sQSPJQTWPFy7Bfbzhs/8hKVmyG9ZpT5bwQC3InPLj9RY zEYDNrzF/JD797yz+ppdpEscFZHIMVZsr46wmyhD+8ukghttlvH5XWI32gU3nHuy wkLa7r1heIjetBf0Jfz08UHpQXJ+aGIcGB81bhA0jocjSooAFc4dcgs/BkxYME/5 2nkTk1ZiwGsAnNlhV9URl0vF/eHt68a6Fy4ur/HQoBmvMCZs6hRKhZYynatOAtqw gmwR7jSntpDjU28hIpeNuUOR7M52P6OwZxZwh+7eslSfnqlJ6bDlXmA1jbYCyCRq RSEcWtoo4twskEUi4zHfgznWS1QF2CC7ZNRj82QLx9V/cWGuhaHmNXkTQOYMMrWQ iZDC+qpHMTMWVllKcHUi9cJde9x0hO5d+p/0//ysnF/PRnidSHns8JUHra3fpqaC 3Nx8lqEWO88HFtFCO04a+POcjo89X4KH1omDfpk5Oj1HVjof9TMEKmPhkbqUSSra 5tUoOKyes30T/U+PT1MA =lFpm -----END PGP SIGNATURE----- From phk at phk.freebsd.dk Mon Nov 16 20:46:30 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 16 Nov 2015 20:46:30 +0000 Subject: Assertion failure in VCL_DelBackend() In-Reply-To: <5649F931.8010206@uplex.de> References: <56490106.6020309@uplex.de> <5649A162.104@uplex.de> <15918.1447667754@critter.freebsd.dk> <5649C06D.7070403@uplex.de> <21453.1447674093@critter.freebsd.dk> <5649D1FD.7020201@uplex.de> <5649F931.8010206@uplex.de> Message-ID: <48458.1447706790@critter.freebsd.dk> -------- In message <5649F931.8010206 at uplex.de>, Geoff Simmons writes: >>>> A backend when popens a CGI command for this request ? >>> >>> I'm sure I wouldn't do that by creating and then destroying a >>> struct director every time. Of course the API can be open for >>> unforeseen ideas, but in this case it seems like a real stretch. >> >> Agreed, deleting a struct director carelessly could lead to >> dangling pointers in other transactions. You guys are seriously missing my point here. The point is that we should make it simple and safe for a VMOD to say: Here is a director, please destroy it when... [ ] This transaction is over [ ] This VCL becomes cold [ ] (Any other which might make sense) [ ] I'l take care of it myself And the reasone we should make it easy is *EXACTLY* to not hav dangling pointers etc. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Mon Nov 16 20:49:35 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 16 Nov 2015 20:49:35 +0000 Subject: Assertion failure in VCL_DelBackend() In-Reply-To: References: <56490106.6020309@uplex.de> <5649A162.104@uplex.de> Message-ID: <48486.1447706975@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >> VCL_DelBackend is declared in $include_dir/cache/cache_backend.h, so >> it's available to a VMOD, and there's no indication that it's "not >> supposed to" be used. I've suspected that there are things in >> $include_dir that aren't really meant to be public, but if we're >> serious about that, and if Varnish is liable to crash if a VMOD does >> use them, then perhaps we should separate out the non-public >> interfaces. Or at least document them as non-public -- we have that in >> the *_int.h includes in vapi, for example. > >If it were up to me I'd simply remove everything that doesn't start >with VRT_, except of course symbols from libvarnish* and related data >structures :) No. The plan here is to offer two levels of API: If you stick with VRT functions, porting your VMOD should be piece of cake. If you need to do something more advanced, you get what you ask for. >I don't even know how you would delete the backend without breaking >the type system. That is *exactly* what I'm talking about. When VCL types have underlying C types that require manual memory management (ie: BLOB, BACKEND ...) we need to do manual memory management, and we need to have a coherent and *understandable* architecture for it. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Mon Nov 16 22:25:24 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 16 Nov 2015 23:25:24 +0100 Subject: Assertion failure in VCL_DelBackend() In-Reply-To: <5649F931.8010206@uplex.de> References: <56490106.6020309@uplex.de> <5649A162.104@uplex.de> <15918.1447667754@critter.freebsd.dk> <5649C06D.7070403@uplex.de> <21453.1447674093@critter.freebsd.dk> <5649D1FD.7020201@uplex.de> <5649F931.8010206@uplex.de> Message-ID: > Mmmm, this may be a misunderstanding, the context was PRIV_* scopes > for backends, such as PRIV_TASK or PRIV_TOP. I would argue that > creating backends at the beginning of such a scope and removing them > at the end is such an unlikely solution that we should consider not > supporting it. As long as it gets out of VCL (end-user) scope before you delete it, it's fine. > But it sounds like you're saying backends should not be deleted under > any circumstances. That I don't agree with. I'm saying this for static backends. That would make dynamic backends pointless if you couldn't add/delete them dynamically. My point is more that if you made a dynamic round-robin director for instance. You could either return a wrapper director that resolve the right backend when the resolve function is called or simply return the leaf director directly to VCL code. Returning the leaf director would be error-prone if you can delete it concurrently, that's what I'm against. > What should I do about backends that cease to exist, in a context > where it's not workable to reload a new static backend configuration? I don't understand this question. If you need a backend that can be reloaded dynamically, you shouldn't use a static backend. > That's what I want the runtime backend delete for. And you can VRT_delete_backend any backend created with VRT_new_backend. From dridi at varni.sh Mon Nov 16 22:31:40 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 16 Nov 2015 23:31:40 +0100 Subject: Assertion failure in VCL_DelBackend() In-Reply-To: <48486.1447706975@critter.freebsd.dk> References: <56490106.6020309@uplex.de> <5649A162.104@uplex.de> <48486.1447706975@critter.freebsd.dk> Message-ID: >>If it were up to me I'd simply remove everything that doesn't start >>with VRT_, except of course symbols from libvarnish* and related data >>structures :) > > No. > > The plan here is to offer two levels of API: > > If you stick with VRT functions, porting your VMOD should be piece > of cake. > > If you need to do something more advanced, you get what you ask for. I take back what I said, I am well aware of this fact. And I actually have questions about other non-VRT APIs that I will bring up at some point. >>I don't even know how you would delete the backend without breaking >>the type system. > > That is *exactly* what I'm talking about. > > When VCL types have underlying C types that require manual memory > management (ie: BLOB, BACKEND ...) we need to do manual memory > management, and we need to have a coherent and *understandable* > architecture for it. I get this point, and in this case it was about deleting a backend that is already managed by the VCL, which I don't think is coherent. From dridi at varni.sh Mon Nov 16 22:38:05 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 16 Nov 2015 23:38:05 +0100 Subject: Assertion failure in VCL_DelBackend() In-Reply-To: <48458.1447706790@critter.freebsd.dk> References: <56490106.6020309@uplex.de> <5649A162.104@uplex.de> <15918.1447667754@critter.freebsd.dk> <5649C06D.7070403@uplex.de> <21453.1447674093@critter.freebsd.dk> <5649D1FD.7020201@uplex.de> <5649F931.8010206@uplex.de> <48458.1447706790@critter.freebsd.dk> Message-ID: > You guys are seriously missing my point here. > > The point is that we should make it simple and safe for a VMOD > to say: > > Here is a director, please destroy it when... > [ ] This transaction is over > [ ] This VCL becomes cold > [ ] (Any other which might make sense) > [ ] I'l take care of it myself > > And the reasone we should make it easy is *EXACTLY* to not hav > dangling pointers etc. You are right, I did miss most of this point. I did get the feel of having different scopes with the different kinds of PRIV pointers, but not to this extent. In this case I might have to revisit the docs since I initially wrote them with the VCL's lifespan in mind in the "I'll take care of it myself" case before the 4.1 release. From kacperw at gmail.com Tue Nov 17 04:06:43 2015 From: kacperw at gmail.com (Kacper Wysocki) Date: Tue, 17 Nov 2015 05:06:43 +0100 Subject: relative imports and duplicate includes Message-ID: Hi all, attached is a patch to make 'import foo' relative to the importing source file, and a patch to remove the error upon duplicate 'include'. vcl that comes from the cli or f_arg doesn't get a path, but in this case we fallback to vcl_dir. This means the same behavior for absolute paths and one-directory deep relative includes under vcl_dir. Use case: These two patches together make it possible to write and release a self-contained vcl package which you can include into your vcl. PRs are on github fwiw. 0K -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Allow-duplicate-vmod-imports.patch Type: text/x-patch Size: 2128 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-VCL-relative-includes.patch Type: text/x-patch Size: 5861 bytes Desc: not available URL: From phk at phk.freebsd.dk Tue Nov 17 10:16:26 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 17 Nov 2015 10:16:26 +0000 Subject: relative imports and duplicate includes In-Reply-To: References: Message-ID: <3093.1447755386@critter.freebsd.dk> -------- In message , Kacper Wysocki writes: >attached is a patch to make 'import foo' relative to the importing source file, We thought a lot about this originally, and the conclusion back then was to not do it for reasons of predictability and security. First: What is "relative to the importing source file" when you load with vcl.inline ? There are also a host of security/trust issues. What if you include "foo", which comes from vcl_dir, and foo includes "bar" ? Do you look for "bar" in the original "relative to the importing source file" dir first or do you look only in vcl_dir ? What if you include "/foo/bar/barf.vcl" and it includes "murphy.vcl" which lives in vcl_dir, and murphy includes "heisenberg.vcl" ? Where do you look for heisenberg.vcl ? If we want to reopen this issue, I think the much more predictable and sensible way to go about it is to turn vcl_dir (and vmod_dir?) into lists of paths to search, (just like $PATH). >and a patch to remove the error upon duplicate 'include'. You mean "import" right ? What if one is: import "foo"; and the other is: import "foo" from /somewhere/libvmod_foo.so"; Isn't that a problem ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From kacperw at gmail.com Tue Nov 17 12:23:19 2015 From: kacperw at gmail.com (Kacper Wysocki) Date: Tue, 17 Nov 2015 13:23:19 +0100 Subject: relative imports and duplicate includes In-Reply-To: <3093.1447755386@critter.freebsd.dk> References: <3093.1447755386@critter.freebsd.dk> Message-ID: Hi. On Tue, Nov 17, 2015 at 11:16 AM, Poul-Henning Kamp wrote: > -------- > In message , Kacper Wysocki writes: > >>attached is a patch to make 'import foo' relative to the importing source file, > > We thought a lot about this originally, and the conclusion back > then was to not do it for reasons of predictability and security. Main motivator for me hammering out this patch was to make vcl packages possible with a minimal amount of impact on existing code out there. I thought there might be reasons for the current behavior. I am proposing a predictable way to solving this, but regarding security it might make sense to talk threat models. > First: What is "relative to the importing source file" when you > load with vcl.inline ? Relative to vcl_dir. Looking over all the different ways a VCL could enter, if you're inlining or f_arg'ing or if it's builtin and there is an include statement, it'll have to resolve somehow. Without the patch, that is vcl_dir. With patch, that is still vcl_dir. > There are also a host of security/trust issues. I am wondering about those trust issues. If you include untrusted vcl, isn't the game over anyway, includes or nay? Are we talking about insecure paths? I have poignantly avoided any parsing or validating of paths in my code past the existing "does the filename foo actually open" > What if you include "foo", which comes from vcl_dir, and foo includes > "bar" ? > Do you look for "bar" in the original "relative to the importing > source file" dir first or do you look only in vcl_dir ? Assuming you mean "foo.vcl" and "bar.vcl" then they are in the same directory and nothing changes. If foo and bar are in subdirectories, ie "path/to/foo.vcl" which includes "relative/path/to/bar.vcl" then bar will be loaded at path/to/relative/path/to/bar.vcl". As a rule, I only look in vcl_dir for includes that can't be resolved relative to the including file. There has got to be a topdir for includes anyway. f_arg default-vcl gets loaded as either an absolute path or from this directory, and includes are loaded from this directory. Check the testcase provided in the patch for how this should work for the not-so-general cases. > What if you include "/foo/bar/barf.vcl" and it includes "murphy.vcl" > which lives in vcl_dir, and murphy includes "heisenberg.vcl" ? Under the new behavior, we error out because /foo/barf/murphy.vcl does not exist, unless vcl_dir is set to /foo/bar. > Where do you look for heisenberg.vcl ? If /foo/bar/murphy.vcl did exist, we'd look for /foo/barf/heisenberg.vcl, too. > If we want to reopen this issue, I think the much more > predictable and sensible way to go about it is to turn > vcl_dir (and vmod_dir?) into lists of paths to search, > (just like $PATH). It's quite possible that there are more elegant solutions, but I'd like to agree on the problem domain at least. How does the "just like $PATH" behavior solve the original problem of "I wanted to release complete, self contained vcl packages"? An instance of the problem I have out there is a multi-site varnish has vcl snippets site1/main.vcl which includes site1/foo.vcl and site1/bar.vcl and site2/main.vcl which includes site2/foo.vcl site2/zool/baz.vcl. Today these includes have to either be absolute, or relative to vcl_dir. Tomorrow, site1 and site2 could be movable package-like things where I could rename the directory, put them on a different varnish server, make vcl_dir=path/to/site2 and even let site1 include site2 if I so wished. >>and a patch to remove the error upon duplicate 'include'. > > You mean "import" right ? yes. those early-morning emails.. > What if one is: > > import "foo"; > > and the other is: > > import "foo" from /somewhere/libvmod_foo.so"; > > Isn't that a problem ? I don't know. In either case you're assuming something about what the user wants. I would like to do import "foo"; include "some/other/vcl"; in my default vcl and not have some/other/vcl break because it too imports foo. That's behavior which is in line with the "first come first serve" behavior of ie multiple vcl_recv subs. It is a bonus that I can override where some/other/vcl loads its vmod from. Saves us modifying all the vcl to match every operational environment out there. -Kacper From kly at redpill-linpro.com Tue Nov 17 13:36:24 2015 From: kly at redpill-linpro.com (Kristian Lyngstol) Date: Tue, 17 Nov 2015 14:36:24 +0100 Subject: relative imports and duplicate includes In-Reply-To: <3093.1447755386@critter.freebsd.dk> References: <3093.1447755386@critter.freebsd.dk> Message-ID: <20151117133624.GB13482@leia.redpill-linpro.com> On Tue, Nov 17, 2015 at 10:16:26AM +0000, Poul-Henning Kamp wrote: > -------- > In message , Kacper Wysocki writes: > > >attached is a patch to make 'import foo' relative to the importing source file, > > We thought a lot about this originally, and the conclusion back > then was to not do it for reasons of predictability and security. (...) Extensive snipping applied. There are two separate, but related issues here: 1. Duplicate "import" statements. 2. Relative paths for "include". I speak now of expected behavior from a user-perspective, not implementation. (convenient!) For 1., the most obvious issue is the "import foo;" vs "import foo from ...". These should be treated as conflicts. You should not be able to load foo twice from different places. To do so would require name spaces for each included VCL file, and we don't want to open that can of worms. As import statements have nothing to do with VCL search paths, the location or search mechanics of VCLs is irrelevant to import statements. You either get the system default, or you specify the path. Whether vmod_dir should be a list or not is also secondary to this. List or not, it's not defined by the location of the VCL. "import foo;" means the same thing in all VCL files. For 2., we should look to what others have done before us. I briefly revisited Python's way of treating module search paths [1] and gcc/C's way of treating header files[2]. If it wasn't already obvious, both suggest that vcl_dir should in fact be a list of paths. #include "foo" and #include are a bit different in C, but I'm leaning towards treating our include statements as #include "foo", which would also align closer to Python's module search path. That would suggest searching for a file first in the same directory as the input file, then system directories. Assuming vcl_dir is "/usr/share/varnish/vcls/", a file in /etc/varnish/default.vcl stating "include foo/bar.vcl" will then first look for /etc/varnish/foo/bar.vcl, then "/usr/share/varnish/vcls/foo/bar.vcl". If foo/bar.vcl was found in /etc/varnish/foo/bar.vcl and contains an other include statement, say "include blatti/blop.vcl", we will look first for "/etc/varnish/foo/blatti/blop.vcl", then for "/usr/share/varnish/vcls/blatti/blop.vcl". With in-line VCL, there is no current directory. Any include statement would have to be found in the vcl_dir search path if a relative URL is used. This seems fairly predictable, in line with how others have done it, and not particularly dangerous. If one is concerned for security, set vcl_dir to blank. An issue that has not been addressed: #ifndef _FOO_H #define _FOO_H do stuff #endif We need to be able to do similar things in VCL in the long run. You want to be able to build VCL libraries where it's safe to include the same VCL multiple times. This, however, can be a separate discussion. I believe the C-approach would suffice. [1] https://docs.python.org/2/tutorial/modules.html#the-module-search-path [2] https://gcc.gnu.org/onlinedocs/cpp/Include-Syntax.html - Kristian From phk at phk.freebsd.dk Tue Nov 17 14:58:56 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 17 Nov 2015 14:58:56 +0000 Subject: relative imports and duplicate includes In-Reply-To: References: <3093.1447755386@critter.freebsd.dk> Message-ID: <3804.1447772336@critter.freebsd.dk> -------- In message , Kacper Wysocki writes: >How does the "just like $PATH" behavior solve the original problem of >"I wanted to release complete, self contained vcl packages"? So, this is one of those procedural things... It might have been better to open a discussion with the problem you're trying to solve, rather than throw out a patch that does something with only passing mention of the problem and no analysis of it :-) >An instance of the problem I have out there is >a multi-site varnish has vcl snippets site1/main.vcl which includes >site1/foo.vcl and site1/bar.vcl and site2/main.vcl which includes >site2/foo.vcl site2/zool/baz.vcl. Today these includes have to either >be absolute, or relative to vcl_dir. Now we're talking... See below... >> What if one is: >> >> import "foo"; >> >> and the other is: >> >> import "foo" from /somewhere/libvmod_foo.so"; >> >> Isn't that a problem ? > >I don't know. In either case you're assuming something about what the >user wants. I would like to do > >import "foo"; >include "some/other/vcl"; Yes, but if some/other/vcl expects vmod_foo to be one thing and another already imported something different under that name, you are screwed and will get *really* weird error messages. The current "include" facility is 100% textual and you can do stuff like: if (include condition.vcl;) { ... } Which is incredibly powerful and 100% unstructured. Both of the two issues you raise actually sound like something entirely different is needed, some kind of "local namespaces", so that the "include foo_lib.vcl" can have its own vmods and includes, which do not affect or spill out over the rest. Maybe what we really need is VCL libraries: library geoip geoip.vcl [from "path"]; ... sub vcl_recv { geoip.recv_stuff(); } -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From kacperw at gmail.com Tue Nov 17 17:17:53 2015 From: kacperw at gmail.com (Kacper Wysocki) Date: Tue, 17 Nov 2015 18:17:53 +0100 Subject: relative imports and duplicate includes In-Reply-To: <3804.1447772336@critter.freebsd.dk> References: <3093.1447755386@critter.freebsd.dk> <3804.1447772336@critter.freebsd.dk> Message-ID: On Tue, Nov 17, 2015 at 3:58 PM, Poul-Henning Kamp wrote: > So, this is one of those procedural things... > > It might have been better to open a discussion with the problem > you're trying to solve, rather than throw out a patch that does > something with only passing mention of the problem and no > analysis of it :-) Sorry, perhaps I got carried away researching the kinks. Before opening a discussion, I felt I needed to map it to varnish internals, but I'm not emotionally tied to the patch; it is just a product of sleepless nights and an itch to scratch. Making a functional solution to this problem, besides letting me figure out the special cases, also made me realise what limitations&facilities there are in the existing code. It's first now that I've tried it out that I feel we can discuss the issue without me treading my foot into some proverbial salad... >> An instance of the problem I have out there is >> a multi-site varnish has vcl snippets site1/main.vcl which includes >> site1/foo.vcl and site1/bar.vcl and site2/main.vcl which includes >> site2/foo.vcl site2/zool/baz.vcl. Today these includes have to either >> be absolute, or relative to vcl_dir. > Now we're talking... If you like that, consider VSF as another specific instance; to me VSF feels more like a special case important to the few people who use Varnish as a programming platform rather than the more general case of ops people using Varnish to bring some happiness into their ops, so the multi-site problem instance may be more common. >>> What if one is: >>> >>> import "foo"; >>> >>> and the other is: >>> >>> import "foo" from /somewhere/libvmod_foo.so"; [snip is this a problem?] > Yes, but if some/other/vcl expects vmod_foo to be one thing and another > already imported something different under that name, you are screwed > and will get *really* weird error messages. I get what you are saying. With my simplistic patch there is no validation of the sameness of the duplicate imports, and checking sameness seems a little heavy-handed. You'll get messages like "no such symbol bar in vmod foo" which aren't really that weird, and having two different vmods named the same is nearly guaranteed to cause confusion anyway. However, the user might at least expect a "Warning: vmod foo imported twice, previous import here" message. > The current "include" facility is 100% textual and you can do stuff > like: > > if (include condition.vcl;) { ... } > > Which is incredibly powerful and 100% unstructured. Yes, imports and includes are not related except by my problem statement. Includes are simple and powerful and that's the reason I like it. I'd prefer it stay this way. The include patch does not change this use. > Both of the two issues you raise actually sound like something > entirely different is needed, some kind of "local namespaces", > so that the "include foo_lib.vcl" can have its own vmods and includes, > which do not affect or spill out over the rest. Yes, namespaces is another way to go. Kristian even mentions this, and I am assuming he doesn't want to go there because it's a heavy-handed approach to take for such a simple language. I would prefer a solution that mirrors the simplest facility other languages have. Each included source file already has a path, making it intuitive that relative include-statements source from that path. For VCL that doesn't have a path there is the vcl_dir. I won't knock language features with merit though! > Maybe what we really need is VCL libraries: > library geoip geoip.vcl [from "path"]; > ... > sub vcl_recv { > geoip.recv_stuff(); > } Maybe? From a programmer perspective this is certainly cleaner. However, the people who run Varnish out there are primarily concerned with ops problems. Will the library/namespaces approach resonate with the ops folks? From phk at phk.freebsd.dk Tue Nov 17 21:39:19 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 17 Nov 2015 21:39:19 +0000 Subject: relative imports and duplicate includes In-Reply-To: References: <3093.1447755386@critter.freebsd.dk> <3804.1447772336@critter.freebsd.dk> Message-ID: <4982.1447796359@critter.freebsd.dk> -------- In message , Kacper Wysocki writes: >> Maybe what we really need is VCL libraries: >> >> library geoip geoip.vcl [from "path"]; >> ... >> sub vcl_recv { >> geoip.recv_stuff(); >> } > >Maybe? From a programmer perspective this is certainly cleaner. >However, the people who run Varnish out there are primarily concerned >with ops problems. Will the library/namespaces approach resonate with >the ops folks? My experience is that the simpler cleaner solution is always preferable, in particular for the ops people. The above mockup gave me a number of interesting ideas in the subsequent shower. DISCLAIMER: ** I'm not promising ANYTHING ** ...but here are some random notes: I don't think the mechanisms we use to implement typed arguments to VMOD entry points would very hard to adapt also to VCL functions: def STRING foobar(INTEGER a1, BACKEND a2, STRING_LIST a3) { ... } ... sub vcl_recv { set req.http.magic = foobar(remote.ip.port, default_backend, req.http.cookie); } If that is true, it *also* means, that you can do the "reverse": Take a bunch of VCL code, and compile it into a VMOD. That automatically gives us different namespaces for the VCL inside the VMOD and the main VCL. I need to look at how it works if the VMOD compiled from VCL imports a VMOD, but as I remember it, I think it would just Do The Right Thing. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From ruben at varnish-software.com Thu Nov 19 09:14:25 2015 From: ruben at varnish-software.com (=?UTF-8?Q?Rub=C3=A9n_Romero?=) Date: Thu, 19 Nov 2015 10:14:25 +0100 Subject: Welcome to VUG10 in Rotterdam, NL on Dec 3-4 2015 Message-ID: Hello, First of all, please spreading the word, specially if you are in BeNeLux or nearby :o) As you have probably noticed on the Varnish community site, we will be having the 10th Varnish User Group meeting in Rotterdam in a couple of weeks. Agenda and details are on . *Please grab right now your ticket for our User Day Conference on December 3rd from >* and you will enjoy talks from: - Poul-Henning Kamp (Varnish Chief Architect) - Phil Stanhope (Fellow @Dyn Inc) - Lasse Karstensen (Release Manager + Tech Lead, @Varnish Software) - Nils Goroll (Owner/Performance Engineer, Uplex) - Thijs Feryn (Evangelist @Combell + PHPBeNeLux) - Kristian Lyngst?l (Everything Varnish @Redpill-Linpro) - Dag Haavi Finstad (Software Developer @Varnish Software) - And more... We still have some slots left, so if you wish to share with us how you use Varnish, please contact me and I will be happy to give you a hand to make that happen (including help with slides and finding a good title for your talk ;o) ) For those of you who hack on Varnish Cache itself, make VMODs or utilities for the software (integration with a CMS, stats, dashboard, anything) please consider joining us also for the Varnish Dev/Hack Day on Friday 4th. Send me an email or simply add yourself to the wiki: https://www.varnish-cache.org/trac/wiki/VDD15Q4 This meeting would not be possible without our sponsors. Many, many thanks for making this event possible go to Floorplanner.com, Dynamic Network Services, Inc . and Varnish Software < https://www.varnish-software.com/> Hope to see you all in The Netherlands! PD: Congrats go to the Drupal community with the release of 8.0.0 expected today. All the best, -- *Rub?n Romero* Community Cheerleader Hat On | Varnish Software Group Cell: +47 95964088 / Office: +47 21989260 Skype, Twitter & IRC: ruben_varnish We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From kacperw at gmail.com Thu Nov 19 13:54:34 2015 From: kacperw at gmail.com (Kacper Wysocki) Date: Thu, 19 Nov 2015 14:54:34 +0100 Subject: relative imports and duplicate includes In-Reply-To: <4982.1447796359@critter.freebsd.dk> References: <3093.1447755386@critter.freebsd.dk> <3804.1447772336@critter.freebsd.dk> <4982.1447796359@critter.freebsd.dk> Message-ID: On Tue, Nov 17, 2015 at 10:39 PM, Poul-Henning Kamp wrote: > -------- > In message > , Kacper Wysocki writes: >>> Maybe what we really need is VCL libraries: [snip] >>> library geoip geoip.vcl [from "path"]; [snip] >>Maybe? From a programmer perspective this is certainly cleaner. >>However, the people who run Varnish out there are primarily concerned >>with ops problems. Will the library/namespaces approach resonate with >>the ops folks? > > My experience is that the simpler cleaner solution is always preferable, > in particular for the ops people. I'm not clear on how this would work in practice; I if I try to rewrite my example to use this approach; site1/main.vcl site1/foo.vcl site1/bar/baz.vcl ok I'll declare in my default.vcl near the top library site1 main.vcl from "site1/main.vcl"; how does main.vcl refer to foo, bar and baz? are these includes, or some new keyword? Does the "from" statement make a path prefix to resolve includes? > I don't think the mechanisms we use to implement typed arguments > to VMOD entry points would very hard to adapt also to VCL functions: > > def STRING > foobar(INTEGER a1, BACKEND a2, STRING_LIST a3) { > ... > } > > ... > > sub vcl_recv { > set req.http.magic = > foobar(remote.ip.port, default_backend, req.http.cookie); > } having function return values, that's cool, but how does it relate to this? > If that is true, it *also* means, that you can do the "reverse": > > Take a bunch of VCL code, and compile it into a VMOD. > > That automatically gives us different namespaces for the VCL > inside the VMOD and the main VCL. > > I need to look at how it works if the VMOD compiled from VCL > imports a VMOD, but as I remember it, I think it would just Do > The Right Thing. Regarding imports, the problem is that I want to ship a vcl which imports a vmod that you're already using. I most certainly do not want to compile my vcl before shipping it. You've lost me here. -Kacper From dridi at varni.sh Thu Nov 19 14:41:52 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 19 Nov 2015 15:41:52 +0100 Subject: relative imports and duplicate includes In-Reply-To: References: <3093.1447755386@critter.freebsd.dk> <3804.1447772336@critter.freebsd.dk> <4982.1447796359@critter.freebsd.dk> Message-ID: > Regarding imports, the problem is that I want to ship a vcl which > imports a vmod that you're already using. I most certainly do not want > to compile my vcl before shipping it. You've lost me here. I believe this is an idea that popped from the discussions, related to your use case of "packaging" ready-to-use VCL. From geoff at uplex.de Fri Nov 20 11:33:00 2015 From: geoff at uplex.de (Geoff Simmons) Date: Fri, 20 Nov 2015 12:33:00 +0100 Subject: [PATCH] PRIV pointers available in vmod obj methods In-Reply-To: <91548.1446634707@critter.freebsd.dk> References: <562E2362.2050104@uplex.de> <91548.1446634707@critter.freebsd.dk> Message-ID: <564F04EC.80404@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 11/04/2015 11:58 AM, Poul-Henning Kamp wrote: > -------- > > Ok, wrapping this subject up: > > Right now PRIV_foo is named with a mix of lifetime and visibility, > and we seem to need to be able to control both aspects. > > The easiest and most backward compatible, seems to be to make the > naming > > PRIV_lifetime[_visibility] > > With a default visibility of "VMOD" with other possible values > being "CALL" and "OBJ" > > Three of the the current four PRIV's maps naturally: > > PRIV_VCL -> PRIV_VCL_VMOD PRIV_TOP -> PRIV_TOP_VMOD PRIV_TASK -> > PRIV_TASK_VMOD > > But: > > PRIV_CALL -> PRIV_VCL_CALL > > The functionality Geoff requires would be PRIV_TASK_OBJ > > If we special-case PRIV_CALL, this is backwards compatible. > > Comments ? Sounds like the right solution. If you can indulge me for a few minutes, I'd like to try re-phrasing it, to make sure we all understand the same thing. It gets a long-winded further down, because I want to go through all of the permutations. 3 lifetimes * 3 visibilities mean that we'll have 9 PRIV scopes, more than twice of what we have now. That will make documentation to aid VMOD authors important, so this will also be a kind of first draft for that (unless of course I'm getting it all wrong). For practical purposes, the VMOD author will need to know: - - When can the scope be used at all (priv != NULL) - - When is the priv object new (when will priv->priv == NULL) - - When will it be destroyed (when is priv->free called) - - Under what circumstances will a function/method declaring a parameter with this scope see the same object, either in another function/method call from the same VMOD interface, in the same VCL, or in another thread. PRIV_{lifetime}_* controls the second and third conditions, and PRIV_*_{visibility} controls the last one. Is PRIV_*_OBJ legal for function declarations? My guess is no, and that and that priv==NULL for PRIV_*_OBJ in any context that is not an object method or init/free function. (Actually, I think it would be best if generate.py would reject illegal scopes in the first place, by declining to generate vcc_if.*. But we presently have priv==NULL in backend contexts for PRIV_TOP, so I'll just assume that priv==NULL for PRIV_*_OBJ in function calls as well.) Also not clear to me -- do PRIV_TOP_* and PRIV_TASK_* extend visibility over restarts and retries? So if I allocate resources for a PRIV_TOP_* or PRIV_TASK_* scope during a client request, and there's a restart, will I see the same resources after the restart? And the same question for PRIV_TASK_* and retries in backend threads. I'm guessing that the answer is yes in both cases (at least that's what makes sense to me). So for the 9 scopes, this is what I come up with: PRIV_VCL_VMOD (== current PRIV_VCL) - - new when called the first time for any func/method in the VMOD interface after the VCL is loaded - - destroyed when the VCL is unloaded - - the same for any func/method in the interface, in any thread, using the scope during the lifetime of a VCL - - the same object is passed into a VMOD's event function PRIV_TOP_VMOD (== PRIV_TOP) - - new when called the first time by any func/method during a client request - - destroyed when the client request is completed - - the same for any func/method in the interface during a client request and its ESI subrequests, including after restarts - - different for different client requests/subrequests (in different threads) - - priv==NULL always for backend threads PRIV_TASK_VMOD (== PRIV_TASK) - - new when called the first time by any func/method during a client or backend task - - destroyed when the client/backend task is completed - - the same for any func/method in the interface during the same c/b task, including after restarts and retries - - different in calls during different c/b tasks (in different threads) PRIV_VCL_CALL (== PRIV_CALL) - - new when invoked the first time by any distinct VCL statement after the VCL is loaded - - destroyed when the VCL is unloaded - - the same for that particular call, in any thread, during the lifetime of VCL - - different for any other invocation, i.e. by a different VCL statement, even of the same func/method PRIV_TOP_CALL - - new when invoked by a VCL statement during a client request - - destroyed when the client request is completed - - the same for that invocation during any ESI subrequest, and after restarts - - different for other client requests/subrequests, and for other invocations of the same func/method - - priv==NULL in backend threads PRIV_TASK_CALL - - new when invoked by a VCL statement during a client or backend task - - destroyed when the client/backend task is completed - - for the client side: the same for that invocation during any ESI subrequest, and after restarts - - backend side: the same for that invocation after retries - - different for any other invocation (even of the same func/method) - - If I'm seeing this right, the usefulness would be to preserve information for one call over restarts or retries. PRIV_VCL_OBJ - - new when called the first time for any method of a specific object after the VCL is loaded - - destroyed when the VCL is unloaded - - the same for any method call of that object, in any thread, during the lifetime of a VCL - - priv==NULL for functions PRIV_TOP_OBJ - - new when called the first time for any method of a specific object during a client request - - destroyed when the client request is completed - - the same for that object during the client request and its subrequests, and after restarts - - different for other client requests/subrequests, and for different objects - - priv==NULL for backend threads and functions PRIV_TASK_OBJ - - new when called the first time for a specific object during a client or backend task - - destroyed when the client/backend task is completed - - the same for that object during that task, including ESI subrequests and after restarts on the client side, and after retries on the backend side - - different for different objects, and for different c/b tasks in different threads - - priv==NULL for functions Sorry for another long email, and/or if I got it all wrong. I do think we'll eventually need to understand and document the 9 scopes at this level of detail. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCAAGBQJWTwThAAoJEOUwvh9pJNUR6mUQAJXMPurMZ84uy7UgsDFBl24Y m7dfqENa+J7tsO4VLwBR5OJM4Z3T5rk+gBks6t9RI+P+53aPIeBbZNHdbCGZOIDD jIAfVRSt38nW6nYeuU4/5v0pNFI3zeM7p4AOviqc6K3MOWe+dVSGN3S0pTJ+e5qC 2/QlEj2NmQJwfXaqL0I8zHfad2v5syDLKPmsfs60vooyUFsPRc4jyGpYAMNu5Y/a uS2H2hK1XpmKMcz1urKz5k4FbqpCYl/2DaoyRWSWBxav79LwLaNaBeDkrTGdU5Iu vbTjxQpp+de5w/AJ6zpi9bz+iCGiPflSNFzdnUhPpsA86SxeWHT0JZQsD11/ikho emaUU7u5DarWlquHhcqF6HFFOzS8LRRO1UMFceu1Apj5YhrtGlDMGU0SWTQ+KsS4 Pyx4QejKHB/L2I9OoLFO1bl3yP2xoscoTHmW0d9Lmxtx2yuAN1CHAfcq3nSgvXcH wB2KoUEFIpjV3OUlyQvfjOXJdIVApDVyG3fpEXyo6cB3mGGtwfRE4XThhG2g7ZxX osOGGUsYLEi9O+0sCgG7JZkxTlWqgXBgu7Sqb3jSr/YXX6BspfrYBE+7OLg/6zlT oOLrwrtke3+mCJvJGvXtTiDs0x3eFy6YFc6uX4SA4LF8k7tadlEvvo2twzW5R8r4 WChNwb5Rxn+PQZfvxhh0 =Mo4h -----END PGP SIGNATURE----- From phk at phk.freebsd.dk Mon Nov 23 09:40:49 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 23 Nov 2015 09:40:49 +0000 Subject: Varnish Improvement Proposals Message-ID: <52391.1448271649@critter.freebsd.dk> As part of the general transition of the project, I have instigated a new VTLA: VIP. Varnish Improvement Proposals are inspired by Pythons PEP process, for managing significant change proposals, and they live on the github wiki: https://github.com/varnish/Varnish-Cache/wiki/Varnish-Improvement-Proposals The procedure is that we start ideas and discussions here on -dev, and if that gels into something I think is viable, I authorize the creation of a VIP to give the idea "somewhere to live" which is more manageable than the mail-archive. A VIP page is *not* a decision or commitment to implement the proposed improvement, it is simply a tool to work record the idea. Everybody is welcome to add information, insights and observations to the VIP wiki pages, but as architect I retain the final word. VIP's will not be used for small and trivial improvements but only but only for "serious stuff" which has long term and architectural implications. The VIP's will be left behind no matter what, and they should record the reasoning that lead to whatever the outcome ends up being. Presently I have created VIP's 1, 2 & 3: * VIP1: PRIV_* visibility and lifetime control * VIP2: VCL typed functions * VIP3: VCL implemented VMODS We'll try out the new "process" with those before we open too many more. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Thu Nov 26 10:31:06 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 26 Nov 2015 10:31:06 +0000 Subject: objcore/req symmetry and Obj->methods Message-ID: <70945.1448533866@critter.freebsd.dk> (This is rather long and rambling and probably mostly for Martin, but I'm sending it -dev in case anybody else is interested and for completeness of the mail-archive.) In the past, varnishd has been very asymmetric: We had requests going one way and objcore going the other and they were very different. Being asymmetric saved us a lot of code initially, we didn't do req.body unless we piped, that saved a lot of code. Strictly speaking the HTTP protocol does not see requests and responses as particularly different, there's that first "envelope" line, and a lot of detailed semantic rules, but overall they're not different. We have moved towards a higher degree of symmetry in the last release cycle, for instance we use VFP's to slurp in the req.body and we store it in a (private) objcore. In the future we will see more pressure for symmetry, for instance: https://tools.ietf.org/html/draft-thomson-http-encryption (Perfectly sensible proposal and great topic for a (series of?) VMODs) So I have been staring at cache_obj.c and cache_busyobj.c a lot, for the "no-storage-pass", "streaming-end-trim" and #1788 issues, and concluded that something needs to happen, so we don't have to solve these issues separately for both directions. We already have such two-of-a-kind solutions, for instance: By default we 'stream' req.body, but we have a mechanism for "caching" it. How is this different from streaming responses, but having a way to say 'no_stream' ? A stevedore can override the Obj methods, but only at the cost of replicating all the rather incestuous objcore/busyobj code which makes streaming work. That code is highly irregular, for instance ObjIter knows about busyobj, but ObjExtend() does not etc. I don't want to see the necessary logic for busyobj/streaming replicated in all advanced stevedores, so the present API will not do. This raises the interesting question: Should we always use a busyobj to fill an objcore, even for req.body and synthetic ? I'm increasingly leaning that way, for reasons of code simplicity and symmetry. We have people asking for the "send the request to N backends and use the first response you get" functionality - which is the exact same problem as obj.body streaming is, so for req.body there is a plain use-case. Streaming on synthetic seems far out only until you start using weird VMODs to generate the synthetic response body slowly[1]. ...so assume for a moment we always use a busyobj and the streaming mechanism to fill objcores. As long as the busyobj is assembling the objcore, *all* the Obj->methods go to the busyobj's method set, and the busyobj methods call the stevedores methods to get things done. Once the busyobj is done, it calls the stevedore with a "commit" method and once that returns, all future Obj->methods go directly to the stevedore. ... Except the ObjIter()s which were already running, we cannot allow the stevedore to rip storage out or move it under them, and they have to finish in the busyobj state. I can imagine three stevedore stragegies: 1. Can do Trim in place, so allocates final storage right away and trims last segment when we know it. 2. Allocates temp space for each segment, moves into final storage when it is full/finished. 3. Allocates all segments in temp space, and moves them all into final storage only when object is finished. But I'm going to deny #3, because the performance difference between #2 with "large enough" segments and #3 is not relevant. So the stevedore gets to pick between strategy #1 and #2 for each object. The decision, and the #2 segment size, might be based on knowing the length from C-L or having an estimate from objhdr. This means we have at most one "uncommitted" segment at any time, and if we let the busyobj own that one (from Transient), the ObjIter issue is manageable. Fitting the "delete while passing" case into this, is a matter of bookkeeping and calling the stevedores ->delete_segment method. Comments, inputs, thoughts ? Poul-Henning [1] Actually, it's not obvious to me that we should have the "magic" path for synthetic any more, why don't we just push a synthetic backend and use the regular path ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Thu Nov 26 11:04:13 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 26 Nov 2015 12:04:13 +0100 Subject: objcore/req symmetry and Obj->methods In-Reply-To: <70945.1448533866@critter.freebsd.dk> References: <70945.1448533866@critter.freebsd.dk> Message-ID: > Comments, inputs, thoughts ? I don't have a deep knowledge of the stevedore API, so I'll share my cents on the side topic instead... > [1] Actually, it's not obvious to me that we should have the "magic" > path for synthetic any more, why don't we just push a synthetic > backend and use the regular path ? Could you please elaborate on pushing a synthetic backend? Would it be a kind of one-off backend like the cgi example we discussed earlier? It sounds good, but there may be issues like seeing an actual bereq in the log. It may lead to confusing user experience, I need more time to think about it. From stanhope at gmail.com Fri Nov 27 16:24:32 2015 From: stanhope at gmail.com (Phil Stanhope) Date: Fri, 27 Nov 2015 11:24:32 -0500 Subject: varnish-dev Digest, Vol 116, Issue 25 Message-ID: I've not responded in awhile, but this one touches on things close to how I use aspects of Varnish. (This is rather long and rambling and probably mostly for Martin, > but I'm sending it -dev in case anybody else is interested and for > completeness of the mail-archive.) > > [SNIP] This raises the interesting question: Should we always use a busyobj > to fill an objcore, even for req.body and synthetic ? > Anything that slows down synthetic, makes me worry. I like all the benefits that I get for normal use cases, plus the extra boost from "clever" synthetic services. > > I'm increasingly leaning that way, for reasons of code simplicity > and symmetry. > But, I can see the value to the underlying system. > Streaming on synthetic seems far out only until you start using > weird VMODs to generate the synthetic response body slowly[1]. > This seems counter-intuitive. But I get the [a] use case and I've used it before. Blackholing a requester with a very long response. But since I've got a terminator in front of varnish and I'm more concerned about DDoS via TLS attacks, I shifted the blackholing to the terminator and let varnish continue to do what it does best. Alas, a port 80 blackhole could be just as important because my varnish servers also respond there. I'd not considered this before. I'd like to discuss more in Rotterdam. I've got something that could come in very handy in terms of managing and accessing the "db" of prefixes for which you'd want to do something like this. There are other related use cases for how you might decide to blackhole at the app layer (rather than IP filters, etc) and use the same lookup service for other purposes. > ...so assume for a moment we always use a busyobj and the streaming > mechanism to fill objcores. > > As long as the busyobj is assembling the objcore, *all* the Obj->methods > go to the busyobj's method set, and the busyobj methods call the > stevedores methods to get things done. > > Once the busyobj is done, it calls the stevedore with a "commit" > method and once that returns, all future Obj->methods go directly > to the stevedore. > > ... Except the ObjIter()s which were already running, we cannot > allow the stevedore to rip storage out or move it under them, > and they have to finish in the busyobj state. > > I can imagine three stevedore stragegies: > > 1. Can do Trim in place, so allocates final storage right away > and trims last segment when we know it. > A blackhole service would have, in effect infinite storage. But of course, that doesn't happen because of the very very slow chunking of response (e.g. byte per time period). These are aberrant behavior though. You'd not want the response cached because you're really sacrifycing a socket. Thus my desire to really implement this in front. > > 2. Allocates temp space for each segment, moves into final storage > when it is full/finished. > > 3. Allocates all segments in temp space, and moves them all into > final storage only when object is finished. > > But I'm going to deny #3, because the performance difference between > #2 with "large enough" segments and #3 is not relevant. > > So the stevedore gets to pick between strategy #1 and #2 for each > object. The decision, and the #2 segment size, might be based on > knowing the length from C-L or having an estimate from objhdr. > > This means we have at most one "uncommitted" segment at any time, > and if we let the busyobj own that one (from Transient), the ObjIter > issue is manageable. > > Fitting the "delete while passing" case into this, is a matter of > bookkeeping and calling the stevedores ->delete_segment method. > > Comments, inputs, thoughts ? > > Poul-Henning > > > [1] Actually, it's not obvious to me that we should have the "magic" > path for synthetic any more, why don't we just push a synthetic > backend and use the regular path ? > I'm okay with this conceptually. Just need to understand the refactoring necessary and the (I can only assume) decrease in throughput for an intentionally non-caching, synthetic backend. But ... "magic" isn't my conceptual understanding. Awhile back we talked about separating the front-end and back-end state machines. Synthetic living only in the front and having increased performance as a by-product. Isn't this separation also a source of asymmetry? There is no BERESP. There is no BERESP keep/ttl, only the final deliver/resp. No stale serving. These are all arguably things that support this idea and the resulting symmetry that would come of it. > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > > > > ------------------------------ > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > > End of varnish-dev Digest, Vol 116, Issue 25 > ******************************************** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Sun Nov 29 14:34:56 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Sun, 29 Nov 2015 15:34:56 +0100 Subject: objcore/req symmetry and Obj->methods In-Reply-To: <70945.1448533866@critter.freebsd.dk> References: <70945.1448533866@critter.freebsd.dk> Message-ID: > We have people asking for the "send the request to N backends and > use the first response you get" functionality - which is the exact > same problem as obj.body streaming is, so for req.body there is > a plain use-case. Couldn't this functionality be implemented in a VMOD instead? A director could spawn multiple backend requests and pick the first one, but either way, we'd need a way to pin several sub-transactions to a bereq transaction. Dridi From fgsch at lodoss.net Mon Nov 30 09:06:55 2015 From: fgsch at lodoss.net (Federico Schwindt) Date: Mon, 30 Nov 2015 09:06:55 +0000 Subject: VTC for connect_timeout? In-Reply-To: <5648C7BD.8070804@uplex.de> References: <5648C7BD.8070804@uplex.de> Message-ID: Hi, Catching up with my email I came across your post. The problem with your test is that while you're delaying the accept(), the connection is not refused (the socket is in listen state) so you are actually exercising the read timeout (first_byte_timeout). You could use $(bad_ip} to test the connection timeout, that'd work. On Sun, Nov 15, 2015 at 5:58 PM, Geoff Simmons wrote: > Hello all, > > I noticed that there is no test case for backend connect_timeout among > the Varnish *.vtc's, so I tried to cook one up for a VMOD, but I can't > seem to get a backend connection to fail due to the timeout when I think > it should. > > The attached VTC just uses standard Varnish, and doesn't pass. I can't > see what could be wrong about VTCP_connect(), so I assume that my > reasoning about the test case must be flawed. > > The first request gets "Connection: close" from the backend, and I can > see BackendClose in the log. > > Then the client sends the second request: > > ** c1 0.6 === txreq > **** c1 0.6 txreq| GET / HTTP/1.1\r\n > **** c1 0.6 txreq| \r\n > ** c1 0.6 === rxresp > > But the backend delays 1.5 seconds, which is longer than > connect_timeout, before accepting again: > > ** s1 0.6 === delay 1.5 > *** s1 0.6 delaying 1.5 second(s) > [...] > ** s1 2.1 === accept > **** s1 2.1 Accepting > *** s1 2.1 Accepted socket fd is 4 > ** s1 2.1 === rxreq > > So I thought the timeout should elapse and the response should be 503, > but it's 200, no failure. > > How am I getting this wrong? > > > Thanks, > Geoff > -- > ** * * UPLEX - Nils Goroll Systemoptimierung > > Scheffelstra?e 32 > 22301 Hamburg > > Tel +49 40 2880 5731 > Mob +49 176 636 90917 > Fax +49 40 42949753 > > http://uplex.de > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at varnish-software.com Mon Nov 30 11:04:17 2015 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 30 Nov 2015 12:04:17 +0100 Subject: objcore/req symmetry and Obj->methods In-Reply-To: <70945.1448533866@critter.freebsd.dk> References: <70945.1448533866@critter.freebsd.dk> Message-ID: On 26 November 2015 at 11:31, Poul-Henning Kamp wrote: > A stevedore can override the Obj methods, but only at the cost of > replicating all the rather incestuous objcore/busyobj code which > makes streaming work. > That code is highly irregular, for instance ObjIter knows about > busyobj, but ObjExtend() does not etc. > > I don't want to see the necessary logic for busyobj/streaming > replicated in all advanced stevedores, so the present API will > not do. > > This raises the interesting question: Should we always use a busyobj > to fill an objcore, even for req.body and synthetic ? > > I'm increasingly leaning that way, for reasons of code simplicity > and symmetry. > I agree that that would be a good next step. > ...so assume for a moment we always use a busyobj and the streaming > mechanism to fill objcores. > > As long as the busyobj is assembling the objcore, *all* the Obj->methods > go to the busyobj's method set, and the busyobj methods call the > stevedores methods to get things done. What kind stevedore interface are you looking at here? An iterator approach like the current one, only with the busyobj method methods wrapped around enforcing the limits the stream synchronization sets? Once the busyobj is done, it calls the stevedore with a "commit" > method and once that returns, all future Obj->methods go directly > to the stevedore. > ... Except the ObjIter()s which were already running, we cannot > allow the stevedore to rip storage out or move it under them, > and they have to finish in the busyobj state. > More or less like we do it now right? I can imagine three stevedore stragegies: > > 1. Can do Trim in place, so allocates final storage right away > and trims last segment when we know it. > > 2. Allocates temp space for each segment, moves into final storage > when it is full/finished. > > 3. Allocates all segments in temp space, and moves them all into > final storage only when object is finished. > > But I'm going to deny #3, because the performance difference between > #2 with "large enough" segments and #3 is not relevant. > I am uncertain what you mean here. For #2, would the general busyobj methods expect to be able to serve content the stevedore has put in temp space to the streaming clients? Or will only final storage segments be available to client consumption? If the temp space is also available to the clients, I will argue that for a sufficiently popular object #2 and #3 are the same. Given enough clients there will be enough slow clients touching on the temp spaces preventing them to be released, and both the temp spaces and the final storage space would eat up memory resources. If the temp space is not available to clients, then that fits with my #2 below. I believe a more useful way to look at the stevedore capabilities is: 1. Can provide memory pointer to final storage location, allowing the fetch processor to fetch directly into the final storage. 2. Can not provide memory pointer to final storage location. Busyobj would need to have a buffer to fetch into, and pushes the buffered content to the stevedores either when it's full or on a short read (socket read buffer empty, slow backend). Streaming will then be delayed by up to the buffer size. > So the stevedore gets to pick between strategy #1 and #2 for each > object. The decision, and the #2 segment size, might be based on > knowing the length from C-L or having an estimate from objhdr. > > This means we have at most one "uncommitted" segment at any time, > and if we let the busyobj own that one (from Transient), the ObjIter > issue is manageable. > I guess this wouldn't be Transiet, as Transient is a stevedore that might not support that operation. I read this as the busyobj buffer above. > > Fitting the "delete while passing" case into this, is a matter of > bookkeeping and calling the stevedores ->delete_segment method. > -Martin -- *Martin Blix Grydeland* Senior Developer | Varnish Software AS Mobile: +47 992 74 756 We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at uplex.de Mon Nov 30 11:29:39 2015 From: geoff at uplex.de (Geoff Simmons) Date: Mon, 30 Nov 2015 12:29:39 +0100 Subject: VTC for connect_timeout? In-Reply-To: References: <5648C7BD.8070804@uplex.de> Message-ID: <565C3323.1090700@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 11/30/2015 10:06 AM, Federico Schwindt wrote: > > The problem with your test is that while you're delaying the > accept(), the connection is not refused (the socket is in listen > state) so you are actually exercising the read timeout > (first_byte_timeout). OK thx. > You could use $(bad_ip} to test the connection timeout, that'd > work. Good thought, I'll try it. - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCAAGBQJWXDMXAAoJEOUwvh9pJNURIVYP/2dvIIlG4wRtZda8zwthBeoK HZDwhGjslbwVDrqttHhFL8+kTSfOKOK+fKi/aczYFuOgKrSQfer8wd+RqoSg9aNT g4a7E/933ycbP7c3I8KBvezTkhAgf2i4gZD/ih+76p5NLfVuAJVUqho9bMY2Iprq fwOwW56VINnjBCnfSGIK/06kFjfkMPJ6vGkHP1D1MsxXNV9J8lVaN/2BQe0mdOKx MGBhUWWACQ4ZpyvDEVQjoic472ejbW5SQwDAAJNIVetFBWWsVR/2nZ8cyYsvEd/t ac6pIsZY1vsFFpcPhtO/TXai72OMF6KaTNR3XfU/JQab0EWHD20PTY8oHbKSH9m1 5cxPjUgD+Le+D0jJ+ivSHX5svJQElvmvr2oKm7DqLFiAWvqxuQdxTnSaR6K+hIA7 psQurT2JaEmNW8TvsosRDQBQZIPtxC+z73nNkU5wcV9b9JBm0Fc/VG4o1ZS/Mn4z iYRiQh7rCuZ2Na8wFj6BpK3n+AHRMTos4f4KRdbLE5Q1qKThZ8cg1cc40+d0miyg yMqip7St7Qy8fh0KQYDV+ebbabvt/ssSoIo40xfghumuBuctbkFPFs+uF289BFmE nDDoDWFhMLMAgYj+msOZ13YBBWzy8KMkvIDpwMkdCcibh5SmWlvweHBSHwCSQ5hm 2om4Ao81GPqNnLGH9B/v =YWmE -----END PGP SIGNATURE----- From kly at redpill-linpro.com Mon Nov 30 12:27:19 2015 From: kly at redpill-linpro.com (Kristian Lyngstol) Date: Mon, 30 Nov 2015 13:27:19 +0100 Subject: Varnish Improvement Proposals In-Reply-To: <52391.1448271649@critter.freebsd.dk> References: <52391.1448271649@critter.freebsd.dk> Message-ID: <20151130122719.GD4094@leia.redpill-linpro.com> On Mon, Nov 23, 2015 at 09:40:49AM +0000, Poul-Henning Kamp wrote: > As part of the general transition of the project, I have instigated > a new VTLA: VIP. (...) > The procedure is that we start ideas and discussions here on -dev, > and if that gels into something I think is viable, I authorize the > creation of a VIP to give the idea "somewhere to live" which is > more manageable than the mail-archive. I like this. Would it make sense to also have VIP-pages for proposals/features that are rejected? Having to reference what could be a lengthy mail on -dev for old proposals is inconvenient at best. It would save us the trouble of having to go through the same discussions every 1-2 years. I was initially thinking of "stale-if-error" for example. - Kristian From phk at phk.freebsd.dk Mon Nov 30 12:31:51 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 30 Nov 2015 12:31:51 +0000 Subject: Varnish Improvement Proposals In-Reply-To: <20151130122719.GD4094@leia.redpill-linpro.com> References: <52391.1448271649@critter.freebsd.dk> <20151130122719.GD4094@leia.redpill-linpro.com> Message-ID: <64002.1448886711@critter.freebsd.dk> -------- In message <20151130122719.GD4094 at leia.redpill-linpro.com>, Kristian Lyngstol w rites: >On Mon, Nov 23, 2015 at 09:40:49AM +0000, Poul-Henning Kamp wrote: >Would it make sense to also have VIP-pages for proposals/features that >are rejected? That was exactly the plan :-) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence.