From geoff at uplex.de Mon May 8 10:05:55 2017 From: geoff at uplex.de (Geoff Simmons) Date: Mon, 8 May 2017 12:05:55 +0200 Subject: RFC for VIP17: unix domain sockets for listen and backend addresses In-Reply-To: References: <427ae252-6daa-d5ba-c7a5-f0b513ef1f2f@uplex.de> Message-ID: On 04/25/2017 04:42 PM, Dridi Boukelmoune wrote: >> >> # How? ## Address notation I suggest that we require a prefix >> such as ``unix:`` to identify UDS addresses (nginx uses >> ``unix:``, haproxy uses ``unix@``): ``` varnishd -a >> unix:/path/to/uds > > This should be enough: > > varnishd -a /path/to/uds Dridi, I'm sorry I never answered all of this, after you put the effort into responding. On a tight schedule today (and I unfortunately can't make it to bugwash today, May 8th), but I'd like to elaborate on this part a bit. What I forgot to say in WIP17: I would suggest a varnishd parameter uds_path, paralleling vcl_path and vmod_path, which specifies a path in which to search for relative UDS paths in a -a argument and in backend definitions. From that it follows that we couldn't identify a UDS address unambiguously as having a '/' as the first character, which is why I think we should require the 'unix:' prefix (or some other prefix). I think experience has shown that absolute paths in the Varnish configuration, which used to be necessary for "include", for example, leads to awkward problems, and the two *_path parameters have been a relief. And I think we'll find that requiring absolute paths everywhere for UDS addresses will lead to the same kinds of problems -- say, you're running a test instance of your Varnish deployment in an environment where files and directories a laid out differently from the production environment. So then you'd have to get sed or something replace all of the absolute paths, just like we used to have to do with absolute include paths. So I say do it right from the beginning this time, and make it possible to use relative paths and just change the uds_path parameter when you have to. Notice that if we do have relative paths, it's not impossible to have a file named "127.0.0.1", or anything else that looks just like an IP address, as the file that's meant to be a UDS address. Of course that's very unlikely and just asking for trouble. But the point is that, strictly speaking, we *couldn't* have an unambiguous distinction between IP and UDS addresses *unless* we require all UDS paths to be absolute (and begin with a '/'). I say let's do everyone a favor by having uds_path and the 'unix:' prefix. Best, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From geoff at uplex.de Mon May 8 10:16:15 2017 From: geoff at uplex.de (Geoff Simmons) Date: Mon, 8 May 2017 12:16:15 +0200 Subject: RFC for VIP17: unix domain sockets for listen and backend addresses In-Reply-To: References: <427ae252-6daa-d5ba-c7a5-f0b513ef1f2f@uplex.de> Message-ID: <4cc17585-bd12-88a7-e55a-a9f98684072f@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 05/08/2017 12:05 PM, Geoff Simmons wrote: > > But the point is that, strictly speaking, we *couldn't* have an > unambiguous distinction between IP and UDS addresses *unless* we > require all UDS paths to be absolute (and begin with a '/'). Um, or unless we have a special prefix for UDS addresses (my point). - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJZEEVvAAoJEOUwvh9pJNURR/AP/i3No6m84xq6X+H3ph8DIAv3 EPjQTh6EGihj3iYr//FwVqy3rowC89TkvR4yJTQT7MTL3+v3OjucnPtFH5HPgL7O utg5ePkZQVSg0e+jUhr7TJZ3cpfjZaBQUrXqBO8F+o9EZ/JEfoJLVLGFvl+ruheP kgHAQ4UzwtkoVggZ48w6xkKGashWGDjVcijy3H2OWwFLEJIJO+Skg4WBDVbypi3h bpt8cwCQMpgp6/tdIRUW4hA+ZQzUFyD2Jcy60MccSnH1Elc3lhoI+yEccew+qwux +QFVX5wpO4t9uGxfw6/5iQwlb6KJfFdr6QUQ6KER5oF+0a1ltkppISe0mgq9QQYm OdQb8EeS3xYngu1UopY9pEyAWfdKMxh1t8Bez/HUd3t2uxxACWcRYw/kUtpssAub Hj95xp/EFnZAHrHOAJZAxv6tR3Gess6y+m7/BMT3ov2mkF4fa9r4xSDfCbBxNQYB TDjmlT5o3FkLXqqvaeVXnDnenKzVfKEH7sMQihb0e2TKVxZ8xtfIrnD3EIvsZcSt meetVoCBthdB4b0Le3REHYyqckwEjXSY6t2l2XMNqM6kjU7vvjpxTpbpkrWFxneW olwIR9npITnSAOsVLPrpzFKHuKFsTxmc85ktsjOv9V/AknlfuCN/eNpnAv6GsxmX UO2biKuxGhbZN7j7zAkq =GDhV -----END PGP SIGNATURE----- From phk at phk.freebsd.dk Wed May 10 05:57:05 2017 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 10 May 2017 05:57:05 +0000 Subject: VIP18, change .vcc to .rst Message-ID: <89214.1494395825@critter.freebsd.dk> I did some research on this, and created VIP18: https://github.com/varnishcache/varnish-cache/wiki/VIP-18:-RST-specification-for-code-generation The origin of this proposal is that the .vcc file cannot be used as README in a github project, but as far as my research, neither can a .rst file with a custom ".. varnish::" directive ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Wed May 10 07:48:38 2017 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 10 May 2017 09:48:38 +0200 Subject: VIP18, change .vcc to .rst In-Reply-To: <89214.1494395825@critter.freebsd.dk> References: <89214.1494395825@critter.freebsd.dk> Message-ID: On Wed, May 10, 2017 at 7:57 AM, Poul-Henning Kamp wrote: > I did some research on this, and created VIP18: > > https://github.com/varnishcache/varnish-cache/wiki/VIP-18:-RST-specification-for-code-generation At quick glance it seems you forgot the conclusions of our discussion and settled for sphinx integration. I thought we agreed that the sphinx/rst2* integration of the directives would be no-op, and a dedicated rst2vmod python script would actually generate VMOD boilerplate code. The idea was to hook directly into RST, see api/ and howto/: http://docutils.sourceforge.net/docs/index.html#api-api-reference-material-for-client-developers We don't want to hard-require sphinx at build time if we don't build the docs. > The origin of this proposal is that the .vcc file cannot be used as README That would be Geoff's intent, a remnant of the Varnish 3 days I'm not particularly interested into. I don't put the same things in a readme and in a manual. My goal is to instead take control of vmod docs, which is not entirely the case today with vmodtool generating RST. > in a github project, but as far as my research, neither can a .rst file with > a custom ".. varnish::" directive ? Github's support is very limited (and I don't blame them for that) probably on purpose. Try an .. include:: directive too ;) Dridi From dridi at varni.sh Wed May 10 15:19:34 2017 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 10 May 2017 17:19:34 +0200 Subject: RFC for VIP17: unix domain sockets for listen and backend addresses In-Reply-To: References: <427ae252-6daa-d5ba-c7a5-f0b513ef1f2f@uplex.de> Message-ID: > Dridi, I'm sorry I never answered all of this, after you put the > effort into responding. No worries, thanks for the feedback :) > On a tight schedule today (and I unfortunately can't make it to > bugwash today, May 8th), but I'd like to elaborate on this part a bit. We haven't decided on the next occurrence of the v6 planning so it's fine. > What I forgot to say in WIP17: I would suggest a varnishd parameter > uds_path, paralleling vcl_path and vmod_path, which specifies a path > in which to search for relative UDS paths in a -a argument and in > backend definitions. I think uds_path would be awkward. Please note that listen addresses are immutable once the manager setup is over (which wasn't the case up to 4.0) so whatever we put on the command line, we stick to it. It means that a parameter as in varnishd -p of the dynamic persuasion wouldn't fit in the picture. > From that it follows that we couldn't identify a UDS address > unambiguously as having a '/' as the first character, which is why I > think we should require the 'unix:' prefix (or some other prefix). For the reason explained above, I think we would be better off with sticking to absolute paths. Less complexity, no prefix, and no possible collision with IP addresses and port numbers. Also, a named address would look like this: varnishd -a public_https=unix:,PROXY We'd be stacking new syntax on new syntax, I'm not too happy with the idea. > I think experience has shown that absolute paths in the Varnish > configuration, which used to be necessary for "include", for example, > leads to awkward problems, and the two *_path parameters have been a > relief. And I think we'll find that requiring absolute paths > everywhere for UDS addresses will lead to the same kinds of problems > -- say, you're running a test instance of your Varnish deployment in > an environment where files and directories a laid out differently from > the production environment. So then you'd have to get sed or something > replace all of the absolute paths, just like we used to have to do > with absolute include paths. I would be happy to make that someone else's problem (like all those "devops" fancy tools to manage environments). > So I say do it right from the beginning this time, and make it > possible to use relative paths and just change the uds_path parameter > when you have to. Define "when you have to", as I said earlier listen addresses are immutable once the manager is started. That's a condition that we must hold since we are to expose them in VCL. > Notice that if we do have relative paths, it's not impossible to have > a file named "127.0.0.1", or anything else that looks just like an IP > address, as the file that's meant to be a UDS address. Of course > that's very unlikely and just asking for trouble. But the point is > that, strictly speaking, we *couldn't* have an unambiguous distinction > between IP and UDS addresses *unless* we require all UDS paths to be > absolute (and begin with a '/'). > > I say let's do everyone a favor by having uds_path and the 'unix:' prefix. Please let me know whether this opinion still holds. -- As agreed during the v6 planning I'm also updating the draft for VIP 17 according to the outcome of our discussions. This is still work in progress but you can have a look at the current version: https://github.com/varnishcache/varnish-cache/wiki/VIP-17:-Enable-Unix-domain-sockets-for-listen-and-backend-addresses/7c181afe87ef3f3ecb24967906f41d51a8813c5e The draft acknowledges the fact that the "named listen addresses" change has officially been accepted. I used VIP17 as a convenient placeholder for the named listen addresses spec and tried to write the whole thing in a way that could be usable for documentation or release notes. The main part of the draft (How ?) has yet to be written. It will take more time as I need to read the v6 planning minutes more carefully. Best, Dridi From dridi at varni.sh Wed May 10 15:27:09 2017 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 10 May 2017 17:27:09 +0200 Subject: RFC for VIP17: unix domain sockets for listen and backend addresses In-Reply-To: References: <427ae252-6daa-d5ba-c7a5-f0b513ef1f2f@uplex.de> Message-ID: >> So I say do it right from the beginning this time, and make it >> possible to use relative paths and just change the uds_path parameter >> when you have to. > > Define "when you have to", as I said earlier listen addresses are > immutable once the manager is started. That's a condition that > we must hold since we are to expose them in VCL. Just when I decide to move to something else for the rest of the day I get hit hard with this thought: A uds_path would actually make sense for backend definitions, so actually I'm not against the idea. For the reasons exposed previously I still think we should stick to absolute paths on the client/listen side. Cheers, Dridi From phk at phk.freebsd.dk Mon May 15 06:59:25 2017 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 15 May 2017 06:59:25 +0000 Subject: VIP18, change .vcc to .rst In-Reply-To: References: <89214.1494395825@critter.freebsd.dk> Message-ID: <2590.1494831565@critter.freebsd.dk> -------- I'm putting this VIP on hold for now, for the following three reasons: I spent a day playing with sphinx/docutils extensions, and while it is hairy, it certainly has potential for improving the typography of our docs, but it will not materially add to the informative content thereof. The idea that the same file can be used as github README and specification file dies with githubs (wise!) decision to not support RST extension directives. We have a handful of ideas in the pipeline (variables, counters etc.) which requires us to extend the specification for VMODs in as of yet unknown ways. Until we have some idea about what information that needs to capture and what code it needs to produce, I'd prefer to not add another level of complexity to our process. Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From geoff at uplex.de Mon May 15 09:05:43 2017 From: geoff at uplex.de (Geoff Simmons) Date: Mon, 15 May 2017 11:05:43 +0200 Subject: RFC for VIP17: unix domain sockets for listen and backend addresses In-Reply-To: References: <427ae252-6daa-d5ba-c7a5-f0b513ef1f2f@uplex.de> Message-ID: <57362b7c-269a-9588-6952-5d0c816e26ba@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 04/25/2017 04:42 PM, Dridi Boukelmoune wrote: > >> ## Access permissions on the listen address For the ``-a`` >> address, I suggest an optional means of specifying who can access >> the UDS: ``` varnishd -a unix:/path/to/uds:uid=foo,gid=bar ``` > >> If no access restrictions were requested, then don't manipulate >> ownership, let bind create the UDS, and set its permissions to >> 0666. > > Wouldn't it be based on umask instead? This is something that I unfortunately never responded to, but I want to get back to it as a comment on the current draft of VIP17, since the VIP now has this as a kind of example: varnishd -a /path/to/http.sock,proto=,uid=varnish,mode=0600 First of all (a lesser matter): OK, we can go with commas as a separator, with the consequence that UDS paths may not have a comma. There is a precedent for that with -sfile. The more important matter is the mode. *If* we want to do this -- set access restrictions on a listen UDS for UID and/or GID --, then if I'm understanding the various platform quirks correctly, we can't have a mode option. Because if you want to restrict access to a user and/or group, then the management process will have to set permissions a certain way, and the user cannot have any options. For connect(2), BSD-derived systems don't care about permissions on the UDS path at all. That is, the permissions can look as if you couldn't access it, if it were a regular file, but nevertheless connect will succeed. However, you can restrict access by restricting the directory to which the UDS belongs. (This is what I read in the docs, haven't tried it myself.) With Linux, however, the process that calls connect(2) must have both read and write permissions on the path. I believe that covers all the possibilities for the platforms that Varnish supports. So it seems to me that this would have to happen (remember that bind creates the path): - - If uid and/or gid are specified in -a - bind and set read access on the containing directory for uid and/or gid, no access for other - set read/write access on the path for uid and/or gid, no access for other - - otherwise (the UDS should be accessible to everyone) - bind and set read/write access for everyone on the path That would mean that users don't have any options for setting the mode, the management process has to control that. Setting mode in -a, in the best case, would get it just right, but would most likely screw it up. If you don't do anything, then yes, the permissions on the path created by bind depend on umask. But if we want to implement access restrictions, then I believe we'd have to do something like this. Another possibility would leave the "otherwise" case to whatever results from umask. Or that we don't implement this feature at all, at least not in the next release. But in either case, I'm not sure if "leave it to the umask" is a good idea, it might result in other processes being unable to connect, forcing users to set a umask before starting Varnish. They would have to know to do that, for one thing, and they might have to set a wide-open umask that we wouldn't want otherwise. (Incidentally, we'll have to decide what Varnish does if the UDS path already exists before the bind, which causes bind to throw EADDRINUSE. unlink and then bind, or throw the error, requiring users to make sure it doesn't exist before starting Varnish?) Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJZGW9nAAoJEOUwvh9pJNURaxQQAIlpTRr7/oxcytIQRxXd3WfS q/L3OvhuZ2cm0RhylAVscKvse951rFL0yHx1oUJdzxnkCQ7ZRvGwjixBvD8mzgaa R2IWWfQwO/cT4FKB4sYLJOxJMUS1zeWa8Kh9WDa/UOijH67RWsoasNfewvLLjWfX qJjzIoc7YN5WzAXf1j4BJqvsbH5Q8xN38TMMxqVQ6XhyL+QieSxGK3y2hEYTHz27 jgWGxR1iTOknawYYQPXRyRZqnzkgMLjdIQuNTZWZtUKoLqu1x+UuJ/Rs9Sn31YrN Uc6Q00Dz6dYf3WyeoKc2SGZmCmVYcYZf7E0y+KLbJQSBQK0uBafCvU8teXAotLDL PqitZjvgh2EkFn8dVzKubbg0zQesWXMU5I8BHf5zau2NN4nKD+PPs8Fb1sTsiAL2 JZ2MyJfvr47NMUoto4jcgduNupAdqglPuzzir3hmyl45pIgD2PMTprwmcyOYq1j4 05ystoyHWan0KXHO8aWBKF38dIhh1DJjYnF6b1ZU83RlXnhfQEeZBufoN9nl+BzO CojUxCnzIoHF08XJrQp+y8cGBJ6u4bfcfeqdY5CjeTWRhKiJrOZh7/47d2EpOZ7Y ctZ6pQM3/EwdBlMLCb2q6hs2J2agyQ76izDAs+T8pOy08XN1ppc2Poh6Y6zxI9TH 1jNDph4vSKCuP6JlZwpG =b2M/ -----END PGP SIGNATURE----- From dridi at varni.sh Mon May 15 09:41:07 2017 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 15 May 2017 11:41:07 +0200 Subject: RFC for VIP17: unix domain sockets for listen and backend addresses In-Reply-To: <57362b7c-269a-9588-6952-5d0c816e26ba@uplex.de> References: <427ae252-6daa-d5ba-c7a5-f0b513ef1f2f@uplex.de> <57362b7c-269a-9588-6952-5d0c816e26ba@uplex.de> Message-ID: > First of all (a lesser matter): OK, we can go with commas as a > separator, with the consequence that UDS paths may not have a comma. > There is a precedent for that with -sfile. Yes, we can probably get away with no commas in file names. Also see this comment from phk: https://github.com/varnishcache/varnish-cache/issues/2325#issuecomment-301413244 > The more important matter is the mode. *If* we want to do this -- set > access restrictions on a listen UDS for UID and/or GID --, then if I'm > understanding the various platform quirks correctly, we can't have a > mode option. Because if you want to restrict access to a user and/or > group, then the management process will have to set permissions a > certain way, and the user cannot have any options. > > For connect(2), BSD-derived systems don't care about permissions on > the UDS path at all. That is, the permissions can look as if you > couldn't access it, if it were a regular file, but nevertheless > connect will succeed. However, you can restrict access by restricting > the directory to which the UDS belongs. (This is what I read in the > docs, haven't tried it myself.) > > With Linux, however, the process that calls connect(2) must have both > read and write permissions on the path. > > I believe that covers all the possibilities for the platforms that > Varnish supports. Well, we can always set a mode and let users refer to the UDS docs of their platforms. > So it seems to me that this would have to happen (remember that bind > creates the path): > > - - If uid and/or gid are specified in -a > - bind and set read access on the containing directory for uid > and/or gid, no access for other I'm not fond of fiddling with directories we don't own, for example, what if the user picks /tmp/varnish.sock? > - set read/write access on the path for uid and/or gid, no access > for other > - - otherwise (the UDS should be accessible to everyone) > - bind and set read/write access for everyone on the path > > That would mean that users don't have any options for setting the > mode, the management process has to control that. Setting mode in -a, > in the best case, would get it just right, but would most likely screw > it up. > > If you don't do anything, then yes, the permissions on the path > created by bind depend on umask. But if we want to implement access > restrictions, then I believe we'd have to do something like this. > > Another possibility would leave the "otherwise" case to whatever > results from umask. Or that we don't implement this feature at all, at > least not in the next release. But in either case, I'm not sure if > "leave it to the umask" is a good idea, it might result in other > processes being unable to connect, forcing users to set a umask before > starting Varnish. They would have to know to do that, for one thing, > and they might have to set a wide-open umask that we wouldn't want > otherwise. My take on it is do nothing unless a mode was specified when the socket is created. Simple for us, leaving the users in control. > (Incidentally, we'll have to decide what Varnish does if the UDS path > already exists before the bind, which causes bind to throw EADDRINUSE. > unlink and then bind, or throw the error, requiring users to make sure > it doesn't exist before starting Varnish?) If we can't bind, we should just fail the startup. Varnish shouldn't do the sysadmin's job. Cheers, Dridi From sollie at sparkz.no Tue May 16 15:33:39 2017 From: sollie at sparkz.no (sollie) Date: Tue, 16 May 2017 11:33:39 -0400 Subject: =?utf-8?B?4p2kbXkgYXJ0aWNsZQ==?= Message-ID: <1005148673.20170516183339@sparkz.no> Dear! Have you read the latest article written by me? Just take a look, it's very interesting. Please read here http://www.erkeklerdesertlesmeme.com/cache/dump/rohi5l/users/jepm/needle.php?5958 sollie Sent from Mail for Windows 10 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 73AD9608805A98A0107D114E5179730D.jpg Type: image/jpeg Size: 19795 bytes Desc: not available URL: From phk at phk.freebsd.dk Thu May 18 07:15:56 2017 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 18 May 2017 07:15:56 +0000 Subject: VSC self-documentation Message-ID: <12936.1495091756@critter.freebsd.dk> I have been prototyping the extensible VSC-stuff for some days and this is a progress report of sorts. The major issue with extensible VSC's is that they need to document themselves so varnishstat(1) and other VSM users can understand them. The layout of a VSM segment follows directly from that: { uint64_t offset_to_doc; // N * 8 uint64_t counters[N]; char doc[]; } The documentation should have 1+N parts, a "head" which explains what this set of counters is about, and a "stanza" for each counter. The 'stanza' tells the type (counter/gauge/bitmap), the level (info/diag/debug) and maybe also unit of this counter. It must have a "oneliner" and optionally also a multi-paragraph "long" explanation. And then we run into the how do we encode that in the VSM segment thing, and after looking at all the alternatives I could imagine, I end up with JSON ... again, again, again. I have a small, 500 LOC, json parser in C we can use, so that is not going to drag in a dependency. If we're going to bite the JSON bullet, the other places it makes sense to use it, is for the vcc_if.c::Vmod_Spec data structure which explains the contents of a VMOD to VCC. Given that we are going to expand what VMODs can contain, that "minilanguage" needs expansion too, so all in all, I think we are over the threshold were JSON makes sense. ... but I still feel dirty using JSON to pass data structures from one C-program to another, or as it may be, from a python program to another. Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Thu May 18 16:16:53 2017 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 18 May 2017 18:16:53 +0200 Subject: RFC for VIP17: unix domain sockets for listen and backend addresses In-Reply-To: References: <427ae252-6daa-d5ba-c7a5-f0b513ef1f2f@uplex.de> <57362b7c-269a-9588-6952-5d0c816e26ba@uplex.de> Message-ID: On Mon, May 15, 2017 at 11:41 AM, Dridi Boukelmoune wrote: Hello, Making slow progress on the draft, but getting there. While gathering details from the first v6wash and the original draft, I found that named listen addresses could once again solve one of the issues. In this case, that would be ACLs containing paths. With named listen address you could already implement name-based access control (with strong typing) and I figured we could do path matching too (via string conversion). https://github.com/varnishcache/varnish-cache/wiki/VIP-17:-Enable-Unix-domain-sockets-for-listen-and-backend-addresses/_compare/5b5ca04148c0420789af4006f0f1b2bf1815f53c...caa8cf86af2ab1e8c37f2a80bee29f4a16333898 Thoughts? Thanks, Dridi From dridi at varni.sh Thu May 18 16:30:35 2017 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 18 May 2017 18:30:35 +0200 Subject: VSC self-documentation In-Reply-To: <12936.1495091756@critter.freebsd.dk> References: <12936.1495091756@critter.freebsd.dk> Message-ID: > I have a small, 500 LOC, json parser in C we can use, so that is > not going to drag in a dependency. > > If we're going to bite the JSON bullet, the other places it makes > sense to use it, is for the vcc_if.c::Vmod_Spec data structure > which explains the contents of a VMOD to VCC. > > Given that we are going to expand what VMODs can contain, that > "minilanguage" needs expansion too, so all in all, I think we > are over the threshold were JSON makes sense. > > ... but I still feel dirty using JSON to pass data structures > from one C-program to another, or as it may be, from a python > program to another. There are other solutions for this kind of descriptors, and you could probably get a msgpack [1] subset implementation with the same-ish amount of code and get the benefits of actual typing (int, string, etc) and easier parsing. Dridi [1] http://msgpack.org/ From dho at fastly.com Thu May 18 17:24:41 2017 From: dho at fastly.com (Devon H. O'Dell) Date: Thu, 18 May 2017 10:24:41 -0700 Subject: VSC self-documentation In-Reply-To: <12936.1495091756@critter.freebsd.dk> References: <12936.1495091756@critter.freebsd.dk> Message-ID: On Thu, May 18, 2017 at 12:15 AM, Poul-Henning Kamp wrote: > And then we run into the how do we encode that in the VSM segment > thing, and after looking at all the alternatives I could imagine, > I end up with JSON ... again, again, again. > > I have a small, 500 LOC, json parser in C we can use, so that is > not going to drag in a dependency. In case JSON ends up being a thing, I recently contributed to a JSON project, https://github.com/skeeto/pdjson. It's a streaming parser, and is the only C JSON parser that passes / fails all the tests as expected in https://github.com/nst/JSONTestSuite (http://seriot.ch/parsing_json.php). It's ~700 LOC with header files, and supports custom allocators. Unsure whether strict correctness is required, but it certainly can't hurt (especially if folks end up building e.g. VMODs on top of it). --dho From phk at phk.freebsd.dk Thu May 18 18:57:46 2017 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 18 May 2017 18:57:46 +0000 Subject: VSC self-documentation In-Reply-To: References: <12936.1495091756@critter.freebsd.dk> Message-ID: <14784.1495133866@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >> ... but I still feel dirty using JSON to pass data structures >> from one C-program to another, or as it may be, from a python >> program to another. > >There are other solutions for this kind of descriptors, and you could >probably get a msgpack [...] Yes, or CBOR, XDR, N different ASN.1's or ... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Fri May 19 08:08:55 2017 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 19 May 2017 08:08:55 +0000 Subject: VSC self-documentation In-Reply-To: References: <12936.1495091756@critter.freebsd.dk> Message-ID: <25731.1495181335@critter.freebsd.dk> -------- In message , "Devon H. O'Dell" writes: >In case JSON ends up being a thing, [...] >Unsure whether strict correctness is required, but it certainly can't >hurt (especially if folks end up building e.g. VMODs on top of it). Thus inspired I just tested my code[1] agains the testsuite, and after one small tweak[2] it passes all the Yes/No tests. The "i_" and transform testcases gets various results: I don't transform numbers to C numeric types, so I don't find all the ieee64 overflows, and I'm not being anal about unicode either. > It's a streaming parser [...] It's ~700 LOC with header files, > and supports custom allocators. That's not a bad payback for a couple hundred lines of code. For the specific case of VCC/VMOD "symbol tables" and VSC (and VSL?) descriptions, the JSON won't be streaming, and I don't see why varnishstat/log would ever need custom allocators. For this prototyping run I'll hang onto my own code - primarily because it is already written in "varnish style" and uses vqueue etc, but once I get it working I'll circle back and try pdjson also. > In case JSON ends up being a thing, [...] ... we will have to decide if we expose the "raw" JSON parser in libvarnishapi, or if it is just used internally to implement the VSM/VSC/VSL APIs we expose. Like VGZ, I think I am inclined to not expose it, in order to not grow our APIs beyond our support-capacity. Poul-Henning [1] http://phk.freebsd.dk/misc/json.c [2] I my check for control-chars in strings I forgot that char is signed. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Fri May 19 11:38:08 2017 From: dridi at varni.sh (Dridi Boukelmoune) Date: Fri, 19 May 2017 13:38:08 +0200 Subject: RFC for VIP17: unix domain sockets for listen and backend addresses In-Reply-To: References: <427ae252-6daa-d5ba-c7a5-f0b513ef1f2f@uplex.de> <57362b7c-269a-9588-6952-5d0c816e26ba@uplex.de> Message-ID: > Thoughts? I finally completed the draft, modulus open questions: https://github.com/varnishcache/varnish-cache/wiki/VIP-17:-Enable-Unix-domain-sockets-for-listen-and-backend-addresses/_compare/caa8cf86af2ab1e8c37f2a80bee29f4a16333898...295e902cbdef98aa28f5cacf171f433c5a48e8e7 Ready for the next discussion. Dridi From slink at schokola.de Fri May 19 11:56:10 2017 From: slink at schokola.de (Nils Goroll) Date: Fri, 19 May 2017 13:56:10 +0200 Subject: RFC for VIP17: unix domain sockets for listen and backend addresses In-Reply-To: References: <427ae252-6daa-d5ba-c7a5-f0b513ef1f2f@uplex.de> Message-ID: <4584e2a0-b77d-d4a9-354a-67b533d7b5c7@schokola.de> On 25/04/17 16:42, Dridi Boukelmoune wrote: > I would even like > to see UDS support for admin sockets (-T option). +1 I just had a use case for this yesterday, which might also be a general use case: cross-container communication (in docker). Sharing a file system with a UDS (read only) between container is safe and easy, while configuring a shared network between containers is not. I must say though that this use case calls for more finely grained access control for cli connections. Sounds like we could want a cli vcl? Nils From dridi at varni.sh Fri May 19 13:24:57 2017 From: dridi at varni.sh (Dridi Boukelmoune) Date: Fri, 19 May 2017 15:24:57 +0200 Subject: RFC for VIP17: unix domain sockets for listen and backend addresses In-Reply-To: <4584e2a0-b77d-d4a9-354a-67b533d7b5c7@schokola.de> References: <427ae252-6daa-d5ba-c7a5-f0b513ef1f2f@uplex.de> <4584e2a0-b77d-d4a9-354a-67b533d7b5c7@schokola.de> Message-ID: > +1 I just had a use case for this yesterday, which might also be a general use > case: cross-container communication (in docker). Sharing a file system with a > UDS (read only) between container is safe and easy, while configuring a shared > network between containers is not. The VIP now covers both -a -T and -M. > I must say though that this use case calls for more finely grained access > control for cli connections. Sounds like we could want a cli vcl? The varnish-cli is already its own language so I don't see how a "cli vcl" would fit in the picture. I think that loopback+secret or uds[+secret] is already quite fine. You need a local access somehow and local credentials with enough privileges to even use the CLI. Dridi From dho at fastly.com Fri May 19 14:17:14 2017 From: dho at fastly.com (Devon H. O'Dell) Date: Fri, 19 May 2017 07:17:14 -0700 Subject: VSC self-documentation In-Reply-To: <25731.1495181335@critter.freebsd.dk> References: <12936.1495091756@critter.freebsd.dk> <25731.1495181335@critter.freebsd.dk> Message-ID: On Fri, May 19, 2017 at 1:08 AM, Poul-Henning Kamp wrote: > -------- > In message , "Devon H. O'Dell" writes: > >>In case JSON ends up being a thing, [...] > >>Unsure whether strict correctness is required, but it certainly can't >>hurt (especially if folks end up building e.g. VMODs on top of it). > > Thus inspired I just tested my code[1] agains the testsuite, and after > one small tweak[2] it passes all the Yes/No tests. > > The "i_" and transform testcases gets various results: I don't > transform numbers to C numeric types, so I don't find all the ieee64 > overflows, and I'm not being anal about unicode either. Cool! The unspecified stuff is wiggly; I didn't care much about one way or the other on that, either. > [2] I my check for control-chars in strings I forgot that char is signed. Except when it isn't! Whether "plain" char is signed or unsigned is implementation-defined (?6.2.5p3, ?6.3.1.1p3). (But I guess the fix was making the thing unsigned char, so it doesn't matter anyway :)). --dho From phk at phk.freebsd.dk Fri May 19 14:48:36 2017 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 19 May 2017 14:48:36 +0000 Subject: VSC self-documentation In-Reply-To: References: <12936.1495091756@critter.freebsd.dk> <25731.1495181335@critter.freebsd.dk> Message-ID: <26800.1495205316@critter.freebsd.dk> -------- In message , "Devon H. O'Dell" writes: >> [2] I my check for control-chars in strings I forgot that char is signed. > >Except when it isn't! Whether "plain" char is signed or unsigned is >implementation-defined Yeah, compliments to the ISO-C people for that bit of insanity... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From reza at varnish-software.com Fri May 19 17:19:28 2017 From: reza at varnish-software.com (Reza Naghibi) Date: Fri, 19 May 2017 13:19:28 -0400 Subject: VSC self-documentation In-Reply-To: <26800.1495205316@critter.freebsd.dk> References: <12936.1495091756@critter.freebsd.dk> <25731.1495181335@critter.freebsd.dk> <26800.1495205316@critter.freebsd.dk> Message-ID: Looks pretty decent. Definitely makes sense to have some kind of JSON support in Varnish, especially on the parsing and reading end. I too had to implement my own JSON lib just because the options out there just dont hit all requirements, libraries generally optimize for 1 thing and somehow do everything else poorly, like navigation vs performance or strictness vs performance. What I got is ~1600 LOC, but it has very strict JSON grammer parsing. Very fast on both ends too, parsing is done in 1 pass (streamed) and the result is a nice search index which acts very close to a hashtable when searching. Infact, the search index get stored into cache (see below). I could definitely see open sourcing this in the form of a VMOD. Yes, I said VMOD. Check out the following VCL. Sorry for the hijack, and yes, im digging up object support in VCL again :) --- import request; import objcore; import json; import types; sub vcl_recv { new json_req = request.new(); if (req.url == "/") { json_req.copy_headers(); json_req.set_host(local.ip); json_req.set_port(std.port(local.ip)); json_req.set_url("/some/data.json"); # Tell the VMOD if this request goes back into Varnish # safely grab the objcore, if possible # (there is a more elegant way to do this # but I rather get the point across) json_req.want_objcore(); json_req.send(); } } sub vcl_deliver { new oc = objcore.new(); new json_doc = json.new(); new json_field = types.string(); if (json_req.sent()) { json_req.wait(); # Load the objcore from the json request oc.get_reference(json_req.get_objcore()); if (!oc.is_valid()) { return(synth(401)); } # Similar to ESI data json_doc.load_index(oc.get_attribute("json")); if (!json_doc.is_valid()) { # Parse the response text into JSON json_doc.load_text(json_req.get_body()); # Write back the index into cache oc.set_attribute("json", json_doc.get_index()); } json_field.set(json_doc.get("some_json_field")); # We now read a JSON field from the body of an object in cache # Future lookups will hit the JSON search index directly } } --- Basically, make a request back to Varnish, grab the objcore, and then load/store/cache JSON into it. That JSON could have been a cache miss, it gets cached, it could have been a 404, this all plays really nice with how Varnish works. A client can even request that JSON back out (minus the search index, obviously). Dont get all up in arms, this all works, this is pretty much how Edgestash works over at Varnish Software :) Also, remember, the point of VCL is to allow people to write what they want, so if someone wants to write the above VMODs/VCL, good for them. So ya, if I can write a JSON vmod object, then that would be a good case to release JSON code. The flipside is that with objects, JSON parsing is just the tip of the iceburg in terms of providing a nice standard library with all the things dev expect, plus interop with Varnish internals, like objcore above. -- Reza Naghibi Varnish Software On Fri, May 19, 2017 at 10:48 AM, Poul-Henning Kamp wrote: > -------- > In message mail.gmail.com> > , "Devon H. O'Dell" writes: > > >> [2] I my check for control-chars in strings I forgot that char is > signed. > > > >Except when it isn't! Whether "plain" char is signed or unsigned is > >implementation-defined > > Yeah, compliments to the ISO-C people for that bit of insanity... > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at uplex.de Mon May 22 10:12:01 2017 From: geoff at uplex.de (Geoff Simmons) Date: Mon, 22 May 2017 12:12:01 +0200 Subject: RFC for VIP17: unix domain sockets for listen and backend addresses In-Reply-To: References: <427ae252-6daa-d5ba-c7a5-f0b513ef1f2f@uplex.de> <57362b7c-269a-9588-6952-5d0c816e26ba@uplex.de> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 This will be comments on this part of the thread so far, and on the current state of the draft in Wiki. On 05/15/2017 11:41 AM, Dridi Boukelmoune wrote: >> >> For connect(2), BSD-derived systems don't care about permissions >> on the UDS path at all. That is, the permissions can look as if >> you couldn't access it, if it were a regular file, but >> nevertheless connect will succeed. However, you can restrict >> access by restricting the directory to which the UDS belongs. >> (This is what I read in the docs, haven't tried it myself.) >> >> With Linux, however, the process that calls connect(2) must have >> both read and write permissions on the path. > > Well, we can always set a mode and let users refer to the UDS docs > of their platforms. > >> So it seems to me that this would have to happen (remember that >> bind creates the path): >> >> - - If uid and/or gid are specified in -a - bind and set read >> access on the containing directory for uid and/or gid, no access >> for other > > I'm not fond of fiddling with directories we don't own, for > example, what if the user picks /tmp/varnish.sock? > > My take on it is do nothing unless a mode was specified when the > socket is created. Simple for us, leaving the users in control. (BTW the same discussion applies to UDSen for the -T admin address.) /tmp is a good example, and that's probably enough to shoot this idea down. If we do what I was thinking of, we'd have to document loudly: "don't do that with a listen path if you're unwilling to set access restrictions this way". Thing is, a lot of people are likely to use /tmp just for testing (some might do it in production). At the risk of getting long-winded and philosophical again, here's what's been bothering me about the way this has been going. I've had security in mind with respect to UDSen, and I've been looking for ways to express security restrictions in the configuration -- VCL and the command line -- so that Varnish is able to enforce them. That way the config and Varnish could do some things to ensure that the admin set things up the way they were intended. An obvious use case will be connecting Varnish to SSL on- and offloading components, although that isn't the only context in which it's relevant for an admin to be sure that Varnish is talking to whom it's supposed to be talking to. If someone finds away to sandwich a man-in-the-middle between Varnish and a peer component, then as things are now just talking to IP addresses, there's no way for Varnish to know and nothing we can say in the config to stop it. You can't know if the peer at the loopback address, or any other address, is the "right" component. UDSen with access restrictions in the file system and credentials present ways to impose limitations on who Varnish talks to. (Of course those can be undermined as well, but we can make it a little harder.) Dridi, your position appears to me to be: the admin can make use of those means to do tighten things up, and Varnish doesn't have to do a lot of checking if they got it right. But admins famously set things up incorrectly sometimes, and don't always notice until it's too late. Even if they get it right the first time, a change may silently undermine what had been working. So I've been looking for ways to express restrictions, for example in VCL and in this case for the listen address, so that Varnish can enforce them, for example by failing to start, or making it possible to write VCL that sets restrictions. That was my motivation behind expanding ACLs to cover UDS paths. We've dropped the ACL idea, and if we drop the idea of Varnish *enforcing* restrictions on the listen and admin addresses -- instead saying "I'll set uid, gid and mode as you say, admin, you have to know what you're doing" -- well then, if there's no other way. But I think it's worth it to remain open for ways to state restrictions in the config that Varnish can enforce. The same thought motivates some comments further down. At the least we should document some of these considerations. The User's Guide might a good place for that -- something like "for example, on BSD-derived systems it's like this, on Linux like this ..." >> (Incidentally, we'll have to decide what Varnish does if the UDS >> path already exists before the bind, which causes bind to throw >> EADDRINUSE. unlink and then bind, or throw the error, requiring >> users to make sure it doesn't exist before starting Varnish?) > > If we can't bind, we should just fail the startup. Varnish > shouldn't do the sysadmin's job. OK, that should be documented though (there is other software that works with UDSen that does in fact unlink before bind). Now on the VIP17 draft: About std.uid, std.gid etc.: > If local.address is not a UDS, numeric variants could also return > -1 and name variants could return NULL. The functions could also > take fallback parameters, possibly with a default value to the ones > suggested (-1 and NULL). +1, and this would be the solution (the only option) on a platform on which peer credentials cannot be obtained. > The question here is more whether we need something like > beresp.backend.path in addition to the ip field. Same question for > peer credentials, they probably don't make sense for backends (and > that would keep the new std functions limited to the listen > addresses type). For the long-winded reasons stated above, I disagree that it doesn't make sense. %^) Especially since connect(2) doesn't create the path as bind(2) does, getting peer credentials on a backend address is about the only thing that could be expressed in VCL about who the peer is intended to be, that goes beyond assuming that the admin got it right. > Is the question of naming from the original draft still relevant? That was about VTCP_name, _hisname and _myname, so one way or another we'll have to decide what happens with those. We could drop them and have them do what they do some other way, and introduce something else again where they're currently called, when it's a UDS. git grep on those function names currently gets over 30 hits, so that would be a bit of a chore. The suggestion in the original draft was to avoid the pain -- leave them as they are, and have them create names for UDSen in way that doesn't change their interfaces. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJZIrlxAAoJEOUwvh9pJNUREGYP/RZUqcO77vKt0BYVI3bj6/Xp Cp/WNt2H1euu73OLllEDiPlGuw39VoTKD8UBmiiC9TX6ASCYwB2PKtp8mxYIqOzD 1JqyAcjz5FI/cbEAB4b5MtC1keTDqlWTkjK/0HACIhP8hdFAAksf3CIBebm+GzMH PQvY5yqW/78eknpq7CpnGpJY3nyXwVnd5RF2MmO6tmXI0VDk6Z7uqlK1v5VV1i7A F2LWgIam1Oqcsqx02Xqdt8pCYrJT23qmthIDAxBQ9VzUTgK25CNj6EdpeDicfLaC rIDCSKGka5J+6WwDcUpct375D1CZGuQl1qEM+0xul33Z67L5bKijaPgEWYsMjnqW GfZCONf759YvwEj4FOWnO9vYKrD8OUJkLAW2zl4JTz5S9HcxUEcu3nBvqY8FCEJx DqFKfc94fEy/8U2QAhS5ZVaaMqLEexSI65taaRMQqSWka69P0twfoQy/NrN0hgaC CxYwynvgRWrgkhI8vNLNw5H1CKWZD+BaM2w+qmIyWYGWqEygtqbIjne7ToEf9JTr VMLo8ivhCeBtrzYDES40NBDDS0/uRNWnxx21TGrkCtaqkepjt4Vi6t3ZLhto+ZLg bdKr/sDtBWhcRdGdpODBie2HreXDNqyYd0LrGruRHRgEtFLoRIzGLCFsc9iCWPny aR6F0fkrUADgqbn3yUcN =rF98 -----END PGP SIGNATURE----- From dridi at varni.sh Mon May 22 10:37:44 2017 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 22 May 2017 12:37:44 +0200 Subject: RFC for VIP17: unix domain sockets for listen and backend addresses In-Reply-To: References: <427ae252-6daa-d5ba-c7a5-f0b513ef1f2f@uplex.de> <57362b7c-269a-9588-6952-5d0c816e26ba@uplex.de> Message-ID: > OK, that should be documented though (there is other software that > works with UDSen that does in fact unlink before bind). There should be a limit to hand-holding. I have bitter memories of tracking down processes holding unlinked files around and leaving a partition with no space. The upside was that it introduced me to lsof(1) at the time. I would invoke POLA, at least it would feel surprising to me, not being sure what service I'm connecting to when I pick a path. >> The question here is more whether we need something like >> beresp.backend.path in addition to the ip field. Same question for >> peer credentials, they probably don't make sense for backends (and >> that would keep the new std functions limited to the listen >> addresses type). > > For the long-winded reasons stated above, I disagree that it doesn't > make sense. %^) Especially since connect(2) doesn't create the path as > bind(2) does, getting peer credentials on a backend address is about > the only thing that could be expressed in VCL about who the peer is > intended to be, that goes beyond assuming that the admin got it right. Well, I was cautious enough to say probably. I'm basing my reasoning on the fact that Varnish tends to trust the backend. It was recently-ish reconfirmed [1] by phk. >> Is the question of naming from the original draft still relevant? > > That was about VTCP_name, _hisname and _myname, so one way or another > we'll have to decide what happens with those. We could drop them and > have them do what they do some other way, and introduce something else > again where they're currently called, when it's a UDS. git grep on > those function names currently gets over 30 hits, so that would be a > bit of a chore. The suggestion in the original draft was to avoid the > pain -- leave them as they are, and have them create names for UDSen > in way that doesn't change their interfaces. I will confess I haven't researched this part thoroughly, that's why I left it as an open question and linked directly to your original draft. Dridi [1] https://github.com/varnishcache/varnish-cache/issues/2197#issuecomment-275055034