From daghf at varnish-software.com Mon Feb 1 18:58:23 2016 From: daghf at varnish-software.com (Dag Haavi Finstad) Date: Mon, 1 Feb 2016 19:58:23 +0100 Subject: Playing around with address sanitizer Message-ID: Hi Inspired by a talk at FOSDEM this weekend [1], I tried building Varnish with -fsanitize=address enabled. This exposed a few buffer overflow situations, mainly in varnishtest. Patches w/ fixes attached. This was done with ASan's leak checker disabled, since varnishtest has intentional memory leaks and I haven't yet investigated if specific allocations can be annotated to make ASan disregard individual leaks. In addition, there is a use-after-free situation in current master related to ESI (not present in 4.1.1). I'm about to file a bug for that one. In order to build varnish w/ -fsanitize=address, you can do ./autogen.des --without-jemalloc CFLAGS="-fno-omit-frame-pointer -fsanitize=address" export ASAN_OPTIONS=detect_leaks=0 followed by the regular make/make check/etc. This requires clang >= 3.1 or gcc >= 4.8. Disabling jemalloc is required as ASan only works with the system allocator. Note the third patch attached may be a false positive, but I think it could be an issue on platforms where memcmp does 64 bit word comparisons at a time. Regards, Dag [1]: https://fosdem.org/2016/schedule/event/csafecode/attachments/slides/1131/export/events/attachments/csafecode/slides/1131/fosdem_gentoo_asan.pdf -- Dag Haavi Finstad Software Developer | Varnish Software Mobile: +47 476 64 134 We Make Websites Fly! -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Fix-a-buffer-overflow-in-vtc.c-macro_get.patch Type: text/x-diff Size: 893 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-Fix-a-buffer-overflow-in-vtc_http.c-cmd_var_resolve.patch Type: text/x-diff Size: 1005 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0003-Test-lengths-of-both-ban-strings-to-avoid-a-potentia.patch Type: text/x-diff Size: 1258 bytes Desc: not available URL: From phk at phk.freebsd.dk Mon Feb 1 19:24:13 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 01 Feb 2016 19:24:13 +0000 Subject: Playing around with address sanitizer In-Reply-To: References: Message-ID: <84048.1454354653@critter.freebsd.dk> -------- In message , Dag Haavi Finstad writes: >This exposed a few buffer overflow situations, mainly in varnishtest. >Patches w/ fixes attached. Cool! >This was done with ASan's leak checker disabled, since varnishtest has >intentional memory leaks [...] I wouldn't really call them "intentional", it's more like they're tolerated because we know they will not cause trouble. Ideally I like all memory to be accounted for, if for nothing else to prove to yourself that you know what the program actually does :-) >In addition, there is a use-after-free situation in current master >related to ESI (not present in 4.1.1). I'm about to file a bug for >that one. Ok, that's bad. I've put the patches in my queue. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dho at fastly.com Mon Feb 1 19:26:56 2016 From: dho at fastly.com (Devon H. O'Dell) Date: Mon, 1 Feb 2016 11:26:56 -0800 Subject: Playing around with address sanitizer In-Reply-To: References: Message-ID: I have some patches to enable msan / asan / tsan / ubsan, as well as test code coverage metrics. (The latter of these isn't super useful for tsan because total code coverage doesn't guarantee you've caught concurrent interactions between any two tested bits, but it's still a start). I've seen problems with using asan even without jemalloc as some libc functions don't have asan overrides. I forget which; it's been a while since I did a test with it, but there may be cases where it's not useful for some tests unless the libc was built with it. (Alternatively, this may have been fixed in more recent versions). I'll send patches for this later today. --dho On Mon, Feb 1, 2016 at 10:58 AM, Dag Haavi Finstad wrote: > Hi > > Inspired by a talk at FOSDEM this weekend [1], I tried building > Varnish with -fsanitize=address enabled. > > This exposed a few buffer overflow situations, mainly in varnishtest. > Patches w/ fixes attached. > > This was done with ASan's leak checker disabled, since varnishtest has > intentional memory leaks and I haven't yet investigated if specific > allocations can be annotated to make ASan disregard individual leaks. > > In addition, there is a use-after-free situation in current master > related to ESI (not present in 4.1.1). I'm about to file a bug for > that one. > > In order to build varnish w/ -fsanitize=address, you can do > ./autogen.des --without-jemalloc CFLAGS="-fno-omit-frame-pointer > -fsanitize=address" > export ASAN_OPTIONS=detect_leaks=0 > followed by the regular make/make check/etc. > > This requires clang >= 3.1 or gcc >= 4.8. Disabling jemalloc is > required as ASan only works with the system allocator. > > Note the third patch attached may be a false positive, but I think it > could be an issue on platforms where memcmp does 64 bit word > comparisons at a time. > > Regards, > Dag > > > [1]: https://fosdem.org/2016/schedule/event/csafecode/attachments/slides/1131/export/events/attachments/csafecode/slides/1131/fosdem_gentoo_asan.pdf > > -- > Dag Haavi Finstad > Software Developer | Varnish Software > Mobile: +47 476 64 134 > We Make Websites Fly! > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev From dho at fastly.com Mon Feb 1 21:36:52 2016 From: dho at fastly.com (Devon H. O'Dell) Date: Mon, 1 Feb 2016 13:36:52 -0800 Subject: Playing around with address sanitizer In-Reply-To: References: Message-ID: On Mon, Feb 1, 2016 at 11:26 AM, Devon H. O'Dell wrote: > I have some patches to enable msan / asan / tsan / ubsan, as well as > test code coverage metrics. (The latter of these isn't super useful Sorry for repeat replies. I've pushed a branch to github for consideration. If y'all prefer sending patchsets through this ML, I'm happy to oblige. From phk at phk.freebsd.dk Tue Feb 2 00:40:10 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 02 Feb 2016 00:40:10 +0000 Subject: Playing around with address sanitizer In-Reply-To: References: Message-ID: <56010.1454373610@critter.freebsd.dk> -------- In message , Dag Haavi Finstad writes: >Inspired by a talk at FOSDEM this weekend [1], I tried building >Varnish with -fsanitize=address enabled. Good idea! That did find at least one genuine bug. I've fixed the ESI ticket, that one would have been really hard to spot any other way, because the calling convention is rather unorthodox (We don't automatically flush, because we want to get the T-E: chunked trailer into the flush too, and close doesn't either.) The memcmp() issues I've fixed with strncmp(). The varnishtest macro code got a miniobj overhaul at the same time. At this point only c00057 fails here, because AS correctly spots that raise(SIGSEGV) derefs NULL. Yell if you find more. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Tue Feb 2 08:06:24 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 02 Feb 2016 08:06:24 +0000 Subject: [PATCH] Add support for REAL [+-] REAL operations In-Reply-To: References: Message-ID: <76812.1454400384@critter.freebsd.dk> -------- In message , Federico Schwindt writes: >Someone on #varnish mentioned the need for this. > >Personally I can't think of any reason for not allowing it. > >Comments? OK? OK. Also allow multiplication/division, while you're at it: @@ -942,6 +942,7 @@ vcc_expr_mul(struct vcc *tl, struct expr **e, enum var_type fmt) case INT: f2 = INT; break; case DURATION: f2 = REAL; break; case BYTES: f2 = REAL; break; + case REAL: f2 = REAL; break; default: if (tl->t->tok != '*' && tl->t->tok != '/') return; (also add to the test-case). -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Tue Feb 2 09:20:09 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 02 Feb 2016 09:20:09 +0000 Subject: New (be)req fields: path scheme and authority In-Reply-To: References: Message-ID: <49219.1454404809@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >The proposal is to have new request fields: >- req.path (the absolute path) >- req.host (the virtual host) >- req.authority Doing this opens various cans of worms. For instance, if you change req.path, does req.url also change automatically ? And as a general principle, I'd prefer to push "text-processing" like this into VMODs if at all possible. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Tue Feb 2 09:25:04 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 02 Feb 2016 09:25:04 +0000 Subject: Release procedure documented in the wiki. In-Reply-To: <20160128125636.GA5462@immer.varnish-software.com> References: <20160128125636.GA5462@immer.varnish-software.com> Message-ID: <49266.1454405104@critter.freebsd.dk> -------- In message <20160128125636.GA5462 at immer.varnish-software.com>, Lasse Karstensen writes: >As requested, I've written down the steps necessary for doing a release >in the wiki. Thanks a lot, this helps our bus-quotient a fair bit. One of the things we discovered in FreeBSD was that when we automated our release making procedure ("cd /usr/src/release && make") people started using it for their own in-house releases etc. and it became a very popular feature with the devops crowd. I'm not quite sure if the same would make sense for Varnish, but keep it in mind. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From fgsch at lodoss.net Tue Feb 2 11:56:37 2016 From: fgsch at lodoss.net (Federico Schwindt) Date: Tue, 2 Feb 2016 11:56:37 +0000 Subject: New (be)req fields: path scheme and authority In-Reply-To: References: Message-ID: Hi, As I mention to Dridi in a private irc channel, I like the idea of scheme, specially in the context of #1847. req.path is a different story. The meaning of path in HTTP/1.x and HTTP/2 has diverted AFAIU. In the former the path is just the path component of the URI, in the latter (well, the pseudo header :path) it includes the query parts. If we were to implement req.path it will mean different things depending on the protocol which will be very confusing. Also I don't think neither the browser nor the users will make that distinction. Regarding req.authority, I don't think there is any reason to add req.authority or have both req.authority and req.http.host. As long as the latter has the correct value we can use it to express the authority pseudo header and the Host header as we already do for absolute requests. On Thu, Jan 28, 2016 at 11:20 AM, Dridi Boukelmoune wrote: > Hi all, > > I'm starting a discussion that will hopefully lead to a VIP. I have > pondered this one for quite a while now and trac #1847 convinced > me that I should share this with the -dev list. I predict it will be > one of those long emails I'm very good at not synthesizing. > > The proposal is to have new request fields: > - req.path (the absolute path) > - req.host (the virtual host) > - req.authority > > This is based on my understanding of HTTP/1.1 and it seems like it'd > play nicely with HTTP/2 but I'm not done studying the latter. > > In HTTP/1.1 a request starts[1] with a request-line: > > method SP request-target SP HTTP/1.1 CRLF > > The request-target is a URI[2] that can take several forms: > > request-target = origin-form => the path, /something > / absolute-form => [scheme]://[authority][path] > / authority-form => host name for CONNECT > / asterisk-form => an actual * for OPTIONS > > The problem with the absolute-form is that Varnish is not supposed to > receive this kind of request because they are meant for forward > proxies, but it MUST [5] be handled regardless. > > The good thing with the absolute-form is that it integrates nicely with > HTTP/2 [3] since all its components map to pseudo headers. > > The other thing that Varnish should do with absolute-form URIs is to > ignore[4] host headers and use the absolute-form authority as such. > > One may ask where we should store the asterisk-form, and HTTP/2 says > it belongs in the path[3]. > > How would it work? > > For HTTP/1.1 Varnish would dissect the request and populate the > following fields: > - req.url => request-target > - req.path => origin form OR asterisk-form OR path from absolute-form > - req.authority => authority-form OR authority from absolute-form OR host > header > - req.scheme => scheme from absolute-form OR "http" > > For HTTP/2 I suppose we could reconstruct an absolute-form with the > pseudo headers from the new fields, since we'd get them as pseudo > headers[3]. > > Changes in the built-in VCL: > sed -e s/req.url/req.path/ -e s/req.http.host/req.authority/ > > Security concerns: > > Mainly, how to deal with an "https" scheme? And for that I'd shift the > responsibility to the user/documentation. If you have a trusted TLS or > HTTPS proxy you can always route the decrypted traffic to a different > port and check it when req.scheme == "https". > > Breaking changes: > > On top of the breaking changes (only the built-in VCL, i thnik) I > wouldn't mind renaming req.url to req.target but not because I like to > break things for the sake of breaking them (but I do like breaking > stuff). > > The rationale is that since the request-target is not necessarily a > URL, that would be the occasion of getting better semantics wrt to > the RFCs (like req.request that became req.method) and also make > sure that VCL wouldn't compile and give you subtle bugs because of > changes in the built-in VCL. > > Thoughts? > > Best, > Dridi > > [1] https://tools.ietf.org/html/rfc7230#section-3.1.1 > [2] https://tools.ietf.org/html/rfc7230#section-5.3 > [3] https://tools.ietf.org/html/rfc7540#section-8.1.2.3 > [4] https://tools.ietf.org/html/rfc7230#section-5.4 > [5] https://tools.ietf.org/html/rfc7230#section-5.3.2 > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Thu Feb 4 09:14:06 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 04 Feb 2016 09:14:06 +0000 Subject: [master] e0b9077 Add support for JSON CLI responses. Message-ID: <98333.1454577246@critter.freebsd.dk> Kristian prodded me for this, and I decided to get this moving right away, because somebody else? will have to do the work of actually writing the functions to emit JSON for the relevant commands. To get consistency in the output/structure/pathology of the resulting JSON, I'm appointing Kristian as our "JSON-Czar", (Subject to appeal to me, if he starts to demand Unicode emoticons etc.) See VCLS_func_help_json() for some hints. Be aware that the -j is passed to the json functions. Please always use the VCLI_JSON_ver() function. The version number is per-command. (The reason the call is conditional in 'help' is to deal with being called both from the master and the slave. Poul-Henning commit e0b9077c87b683a12af45daaf8c08ab4e7ce4195 Author: Poul-Henning Kamp Date: Thu Feb 4 01:51:05 2016 +0000 Add support for JSON CLI responses. To get JSON back, the first argument must be "-j". Currently only "help -j" is implemented, but that will helpfully tell you which commands support json output. All JSON output has the form: [ version#, [], stuff ] For instance: [ 1, ["help", "-j"], { "request": "help", "syntax": "help []", "help": "\tShow command/protocol help.", "minarg": 0, "maxarg": 1, "flags": "", "json": true }, { "request": "ping", "syntax": "ping []", "help": "\tKeep connection alive.", "minarg": 0, "maxarg": 1, "flags": "", "json": false }, ... ] The string quoting for weird characters in JSON is XXX incomplete. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From lkarsten at varnish-software.com Wed Feb 10 12:10:58 2016 From: lkarsten at varnish-software.com (Lasse Karstensen) Date: Wed, 10 Feb 2016 13:10:58 +0100 Subject: VC4.1.2 merge window open until Thursday 18th. Message-ID: <20160210121056.GA26101@immer.varnish-software.com> Hi. We're planning to do a 4.1.2 release in about two weeks. Merge window closes Thursday next week. Stable branch commit guidelines are in effect. -- Lasse Karstensen From slink at schokola.de Wed Feb 17 09:42:38 2016 From: slink at schokola.de (Nils Goroll) Date: Wed, 17 Feb 2016 10:42:38 +0100 Subject: [PATCH] Fix workspace overglow handling in VFP_Push() and test for it Message-ID: <56C4408E.70502@schokola.de> A non-text attachment was scrubbed... Name: 0001-Fix-workspace-overglow-handling-in-VFP_Push-and-test.patch Type: text/x-patch Size: 3556 bytes Desc: not available URL: From slink at schokola.de Wed Feb 17 14:07:41 2016 From: slink at schokola.de (Nils Goroll) Date: Wed, 17 Feb 2016 15:07:41 +0100 Subject: [PATCH] Error handling for V1F_Setup_Fetch Message-ID: <56C47EAD.2020509@schokola.de> A non-text attachment was scrubbed... Name: 0001-Error-handling-for-V1F_Setup_Fetch.patch Type: text/x-patch Size: 4799 bytes Desc: not available URL: From dridi at varni.sh Wed Feb 17 23:26:20 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 18 Feb 2016 00:26:20 +0100 Subject: [PATCH] Automatically expand vmod_* macros in varnishtest Message-ID: Hi, Trying to make life easier for VMOD writers and users, a macro expansion similar to ${date} for VMOD names. It should be simple enough to hopefully be considered for the 4.1 series too. Best, Dridi -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Automatically-expand-vmod_-macros-in-varnishtest.patch Type: text/x-patch Size: 1507 bytes Desc: not available URL: From phk at phk.freebsd.dk Wed Feb 17 23:45:41 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 17 Feb 2016 23:45:41 +0000 Subject: [PATCH] Automatically expand vmod_* macros in varnishtest In-Reply-To: References: Message-ID: <45240.1455752741@critter.freebsd.dk> -------- I think we decided to change the vcl_dir and vmod_dir parameters to be paths rather than single directories. If we got that in, we could get totally rid of the ${vmod_*} magic instead, and have varnishtest -i set vmod_path apropriately. That sounds like a better idea to me... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Thu Feb 18 08:21:10 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 18 Feb 2016 09:21:10 +0100 Subject: [PATCH] Automatically expand vmod_* macros in varnishtest In-Reply-To: <45240.1455752741@critter.freebsd.dk> References: <45240.1455752741@critter.freebsd.dk> Message-ID: On Thu, Feb 18, 2016 at 12:45 AM, Poul-Henning Kamp wrote: > -------- > > I think we decided to change the vcl_dir and vmod_dir parameters to be > paths rather than single directories. > > If we got that in, we could get totally rid of the ${vmod_*} magic Would it mean that we'd retire the ${vmod_*} macros in varnishtest and the test suite? > instead, and have varnishtest -i set vmod_path apropriately. We also need a way to set a vmod_path for VMOD writers. Something like varnishtest -p /path/to/vmod_dir:$libdir/varnish/vmods when building the VMOD. > That sounds like a better idea to me... varnishtest -i already sets the PATH for the programs, should those macros also be retired and leave it to varnishtest users to set an appropriate PATH? I have already submitted such a change in vmod-example to get all programs from the PATH and not just varnishd. From phk at phk.freebsd.dk Thu Feb 18 08:37:49 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 18 Feb 2016 08:37:49 +0000 Subject: [PATCH] Automatically expand vmod_* macros in varnishtest In-Reply-To: References: <45240.1455752741@critter.freebsd.dk> Message-ID: <47069.1455784669@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >On Thu, Feb 18, 2016 at 12:45 AM, Poul-Henning Kamp wrote: >> -------- >> >> I think we decided to change the vcl_dir and vmod_dir parameters to be >> paths rather than single directories. Actually, I already implemented that: 3fea94618a7ac2b0f13548ab13794586d4bba4b8 (Insert appropriate joke about old people and bad memory here) I didn't rename the parameter names from _dir to _path. Not sure if we should. >> If we got that in, we could get totally rid of the ${vmod_*} magic > >Would it mean that we'd retire the ${vmod_*} macros in varnishtest and >the test suite? yes. >> instead, and have varnishtest -i set vmod_path apropriately. > >We also need a way to set a vmod_path for VMOD writers. Something >like varnishtest -p /path/to/vmod_dir:$libdir/varnish/vmods when building >the VMOD. I think I lean '-m' for vmod_path, that way '-v' could be used similarly for vcl_path But now that you mention... We should use '-p' to set _any_ parameter for the varnishd's we start in a test. That would make it trivial to do what Nils and I bantered about yesterday: Running the test-cases with varying sizes of workspace, to see what explodes. If individual tests rely on certain parameters, they should set them explictly to document that in the first place. >varnishtest -i already sets the PATH for the programs, should those >macros also be retired and leave it to varnishtest users to set an >appropriate PATH? No, -i is shortcut for "I'm running in the build-tree" and it should retain that meaning. In the new world order, it should append appropriately to vmod_path. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Thu Feb 18 08:58:07 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 18 Feb 2016 09:58:07 +0100 Subject: [PATCH] Automatically expand vmod_* macros in varnishtest In-Reply-To: <47069.1455784669@critter.freebsd.dk> References: <45240.1455752741@critter.freebsd.dk> <47069.1455784669@critter.freebsd.dk> Message-ID: >>varnishtest -i already sets the PATH for the programs, should those >>macros also be retired and leave it to varnishtest users to set an >>appropriate PATH? > > No, -i is shortcut for "I'm running in the build-tree" and it should > retain that meaning. Bit of misunderstanding here. I say that varnishtest -i already sets an appropriate PATH, so without a -D for instance ${varnishd} will always expand to varnishd. My question was about dropping the ${varnish*} macros and let varnishtest users do this kind of PATH setting, just like varnishtest -i currently does. I have no intention in changing the -i behavior. In my vmod-example patch, varnishtest is run like this: env PATH=@TEST_PATH@ varnishtest -v ... And the TEST_PATH is created by the configure script: AC_SUBST( [TEST_PATH], [$LIBVARNISHAPI_SBINDIR:$LIBVARNISHAPI_BINDIR:$PATH]) This is what I mean by letting the varnishtest user figure the PATH for the tools, and retire the ${varnish*} macros. > In the new world order, it should append appropriately to vmod_path. s/append/prepend/ ? From phk at phk.freebsd.dk Thu Feb 18 09:30:46 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 18 Feb 2016 09:30:46 +0000 Subject: [PATCH] Automatically expand vmod_* macros in varnishtest In-Reply-To: References: <45240.1455752741@critter.freebsd.dk> <47069.1455784669@critter.freebsd.dk> Message-ID: <47768.1455787846@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >> No, -i is shortcut for "I'm running in the build-tree" and it should >> retain that meaning. > >Bit of misunderstanding here. I say that varnishtest -i already sets >an appropriate PATH, so without a -D for instance ${varnishd} will >always expand to varnishd. > >My question was about dropping the ${varnish*} macros and let >varnishtest users do this kind of PATH setting, just like varnishtest >-i currently does. I have no intention in changing the -i behavior. Ohh, I'd forgotten about that. yes, in that case we should drop the ${varnishstat} macros as well. >> In the new world order, it should append appropriately to vmod_path. > >s/append/prepend/ ? I guess the PATH codes sets the precedent that we prepend, and that is probably the most robust choice -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Thu Feb 18 11:00:51 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 18 Feb 2016 12:00:51 +0100 Subject: [PATCH] Automatically expand vmod_* macros in varnishtest In-Reply-To: <47768.1455787846@critter.freebsd.dk> References: <45240.1455752741@critter.freebsd.dk> <47069.1455784669@critter.freebsd.dk> <47768.1455787846@critter.freebsd.dk> Message-ID: >>My question was about dropping the ${varnish*} macros and let >>varnishtest users do this kind of PATH setting, just like varnishtest >>-i currently does. I have no intention in changing the -i behavior. > > Ohh, I'd forgotten about that. yes, in that case we should > drop the ${varnishstat} macros as well. > >>> In the new world order, it should append appropriately to vmod_path. >> >>s/append/prepend/ ? > > I guess the PATH codes sets the precedent that we prepend, and that > is probably the most robust choice Now that we have reached a consensus I'll take care of this, I think next Sunday :) From dridi at varni.sh Mon Feb 22 09:15:31 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 22 Feb 2016 10:15:31 +0100 Subject: [PATCH] Automatically expand vmod_* macros in varnishtest In-Reply-To: References: <45240.1455752741@critter.freebsd.dk> <47069.1455784669@critter.freebsd.dk> <47768.1455787846@critter.freebsd.dk> Message-ID: > Now that we have reached a consensus I'll take care of this, I think > next Sunday :) Patches attached! -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Retire-varnish-macros-in-varnishtest.patch Type: text/x-patch Size: 9370 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-varnishtest-i-builds-vmod_path-instead-of-macros.patch Type: text/x-patch Size: 4127 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0003-Retire-vmod_-macros-for-varnishtest-i.patch Type: text/x-patch Size: 34326 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0004-Polish.patch Type: text/x-patch Size: 980 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0005-Pass-varnishd-parameters-through-varnishtest.patch Type: text/x-patch Size: 3593 bytes Desc: not available URL: From dridi at varni.sh Tue Feb 23 12:58:07 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 23 Feb 2016 13:58:07 +0100 Subject: [PATCH] Automatically expand vmod_* macros in varnishtest In-Reply-To: References: <45240.1455752741@critter.freebsd.dk> <47069.1455784669@critter.freebsd.dk> <47768.1455787846@critter.freebsd.dk> Message-ID: On Mon, Feb 22, 2016 at 10:15 AM, Dridi Boukelmoune wrote: >> Now that we have reached a consensus I'll take care of this, I think >> next Sunday :) > > Patches attached! This is now in the master branch, with an extra commit for documentation: https://www.varnish-cache.org/docs/trunk/reference/varnishtest.html#testing-a-build-tree Cheers, Dridi From dridi at varni.sh Tue Feb 23 17:51:08 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 23 Feb 2016 18:51:08 +0100 Subject: Make named ACLs available to VMODs Message-ID: Hi, As discussed today offline, a set of patches to allow VMODs to check IP addresses against named VCL ACLs. Cheers, Dridi -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Allow-ACLs-to-be-used-outside-of-transactions.patch Type: text/x-patch Size: 818 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-Introduce-a-new-ACL-type-for-VMODs.patch Type: text/x-patch Size: 1752 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0003-Make-named-ACLs-available-to-VMODs.patch Type: text/x-patch Size: 4098 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0004-Add-a-new-VRT_acl_match-function-to-the-VMODs-ABI.patch Type: text/x-patch Size: 1269 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0005-Test-VMOD-ACLs-using-vmod-debug.patch Type: text/x-patch Size: 2116 bytes Desc: not available URL: From phk at phk.freebsd.dk Wed Feb 24 10:14:24 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 24 Feb 2016 10:14:24 +0000 Subject: Make named ACLs available to VMODs In-Reply-To: References: Message-ID: <81061.1456308864@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: - VSLb(ctx->vsl, SLT_VCL_acl, "%s", msg); + AN(msg); + if (ctx->vsl != NULL) + VSLb(ctx->vsl, SLT_VCL_acl, "%s", msg); else VSL(SLT_VCL_acl, 0, "%s", msg); +#define VRT_ACL_MAGIC 0x78329d96 + int (*match)(VRT_CTX, VCL_IP); +}; + Use a typedef for the function pointer in vrt.h Please have VCC also emit the __match_proto__(name of typedef) for the ACL matchers. -.. TODO document ACL if patchwork #314 is merged +ACL + C-type: ``int(acl_f)(VRT_CTX, VCL_IP)*`` + + A function that checks an IP address against the named ACL declared in + VCL. I don't undstand this bit, shouldn't that be the struct ? + if (fmt == ACL) + sym = VCC_FindSymbol(tl, tl->t, SYM_ACL); if (fmt == BACKEND) sym = VCC_FindSymbol(tl, tl->t, SYM_BACKEND); if (fmt == PROBE) For clarity we should have some 'else' there, or possibly a switch instead. +int +VRT_acl_match(VRT_CTX, VCL_ACL acl, VCL_IP ip) +{ + + CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + CHECK_OBJ_NOTNULL(acl, VRT_ACL_MAGIC); + AN(ip); + return (acl->match(ctx, ip)); +} + Do a VSA_Sane() in the ip, the acl-matcher function doesn't do it as far as I remember. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Wed Feb 24 11:02:48 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 24 Feb 2016 12:02:48 +0100 Subject: Make named ACLs available to VMODs In-Reply-To: <81061.1456308864@critter.freebsd.dk> References: <81061.1456308864@critter.freebsd.dk> Message-ID: On Wed, Feb 24, 2016 at 11:14 AM, Poul-Henning Kamp wrote: > -------- Done! -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Don-t-capture-function-typedefs-in-generate.py.patch Type: text/x-patch Size: 638 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-Allow-ACLs-to-be-used-outside-of-transactions.patch Type: text/x-patch Size: 863 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0003-Introduce-a-new-ACL-type-for-VMODs.patch Type: text/x-patch Size: 2103 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0004-Make-named-ACLs-available-to-VMODs.patch Type: text/x-patch Size: 4577 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0005-Add-a-new-VRT_acl_match-function-to-the-VMODs-ABI.patch Type: text/x-patch Size: 1410 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0006-Test-VMOD-ACLs-using-vmod-debug.patch Type: text/x-patch Size: 2263 bytes Desc: not available URL: From phk at phk.freebsd.dk Wed Feb 24 11:36:52 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 24 Feb 2016 11:36:52 +0000 Subject: Make named ACLs available to VMODs In-Reply-To: References: <81061.1456308864@critter.freebsd.dk> Message-ID: <90294.1456313812@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >On Wed, Feb 24, 2016 at 11:14 AM, Poul-Henning Kamp wrote: >> -------- > > >Done! Go. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From ruben at varnish-software.com Wed Feb 24 19:13:28 2016 From: ruben at varnish-software.com (=?UTF-8?Q?Rub=C3=A9n_Romero?=) Date: Wed, 24 Feb 2016 20:13:28 +0100 Subject: User Survey questions feedback Message-ID: Hello, In order to gather more information about how Varnish is being used nowadays you will see a User Survey popping up soon. Please provide any and all feedback you might have on the current questions: https://docs.google.com/spreadsheets/d/1QvCXqTkE9nAJ1AoK0d2oTtFOZhtXTygfp9DHw1Koy70/edit#gid=108526467 Feel free to add more questions or make any changes you would like to propose. The document should be available for anyone anonymously. Thanks in advance for your help! Best regards, -- *Rub?n Romero* Varnish Software Group Cell: +47 95964088 / Office: +47 21989260 Skype, Twitter & IRC: ruben_varnish We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Thu Feb 25 14:49:12 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 25 Feb 2016 15:49:12 +0100 Subject: [PATCH] Make Lck_AssertHeld trigger in the caller code Message-ID: Hi, With Varnish locks available for VMODs in 4.1 I decided to convert my plain pthread locks on my 4.1-only module. I even found a place where I forgot a lock thanks to being able to assert on it being held :) This patch aims at making this assertion a bit more useful: Assert error in Lck__Assert(), cache/cache_lck.c line 175: Condition(ilck->held) not true. The backtrace doesn't mention static function names or line numbers, so with the patch I got this instead: Assert error in vmod_dns_del(), vmod_named.c line 231: Condition(Lck__Held(&dir->mtx)) not true. Best, Dridi -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Make-Lck_AssertHeld-trigger-in-the-caller-code.patch Type: text/x-patch Size: 2565 bytes Desc: not available URL: From slink at schokola.de Mon Feb 29 12:49:44 2016 From: slink at schokola.de (Nils Goroll) Date: Mon, 29 Feb 2016 13:49:44 +0100 Subject: auto-tune ws / headroom poc Message-ID: <56D43E68.60606@schokola.de> Hi, following up an attempt to discuss this on irc: The following discussion is, for the time being, regarding workpace_backend only, but the question is relevant for the other workspaces also: Relevant portions of workspace_backend are not available for use in VCL, but are consumed in core code. Setting workspace_backend too low will trigger an immediate assertion failure in VBO_GetBusyObj(). IMHO, it doesn't make sense to handle these assertions differently, because space for http headers, vsl and the initial read from the backend are mandatory for varnish to issue backend requests. Instead of trying to better handle nonsense workspace_backend sizes, I'd suggest to make it impossible to tune it stupidly in the first place - and to give admins a tunable which is easier to understand. The attached patch contains a PoC (this is _not_ a finished patch, see below) suggesting the following changes: - make workspace_backend a protected (read-only) parameter - add headroom_backend as a new admin-tunable parameter which roughly equals to "workspace available to VCL" - calculate the size of workspace_backend based on all other relevant parameters Notes on the patch: - I've kept the workspace parameter as protected so the mempool code can stay as is (pointing to a single parameter for sizing) - in VBO_Init() I've taken a lightweight approach at the consistency issue from parameter changes, and I'd be curious to hear if others agree or think that we need to make this safer. - what is the best way to cross the border between the cache and mgt code? Originally I would have preferred to have the sizing code in cache_busyobj.c, but then we'd need to make mgt_param visible there - which I didn't like. But what I have in the patch now doesn't look clean at all, I am really unsure at this point. Thx, Nils -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg tel +49 40 28805731 mob +49 170 2723133 fax +49 40 42949753 xmpp://slink at jabber.ccc.de/ http://uplex.de/ -------------- next part -------------- A non-text attachment was scrubbed... Name: headroom_poc.patch Type: text/x-patch Size: 9211 bytes Desc: not available URL: