From slink at schokola.de Mon Nov 7 16:51:48 2016 From: slink at schokola.de (Nils Goroll) Date: Mon, 7 Nov 2016 17:51:48 +0100 Subject: ban_limit parameter // Re: BAN - Memory consumption exploding under load In-Reply-To: References: Message-ID: <48f0e2bf-7eec-d2e3-6c8e-38b6d4f5a6fb@schokola.de> (discussion started on -misc, added -dev) Hi, On 03/11/16 11:04, Dridi Boukelmoune wrote: > I was thinking (against the too-many-knobs trend) that maybe we should > have some sort of ban_queue_limit parameter to avoid piling up too > many bans. Dridi and myself have briefly discussed this and at this point I think, yes, we should have such a parameter. It seems that the best option when exceeding the ban list maximum would probably be to have (by the ban lurker) all objects removed which are untested at the tail of the ban list. This way, we would loose the long-tail bit of the cache, but not the hot objects (because they will already have been tested up the ban list) while still retaining the correct ban behavior (because all objects which are supposed to be banned will be banned - implicitly). The alternative to refuse additional bans appears problematic because applications will rely on the ability to invalidate the cache for correctness. Nils From dridi at varni.sh Mon Nov 7 17:57:30 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 7 Nov 2016 18:57:30 +0100 Subject: ban_limit parameter // Re: BAN - Memory consumption exploding under load In-Reply-To: <48f0e2bf-7eec-d2e3-6c8e-38b6d4f5a6fb@schokola.de> References: <48f0e2bf-7eec-d2e3-6c8e-38b6d4f5a6fb@schokola.de> Message-ID: > The alternative to refuse additional bans appears problematic because > applications will rely on the ability to invalidate the cache for correctness. To which I suggested that we should maybe have a means to clear a cache completely without relying on a whole-encompassing ban, like a varnish-cli command. Users seeing failing bans could then make the decision to wipe the cache while still maintaining uptime. Same thing when lookups start taking seconds because of the number of bans to test, wiping the cache can be a last-resort solution if turning the hypothetical knob down isn't enough. Dridi From slink at schokola.de Mon Nov 7 19:04:23 2016 From: slink at schokola.de (Nils Goroll) Date: Mon, 7 Nov 2016 20:04:23 +0100 Subject: ban_limit parameter // Re: BAN - Memory consumption exploding under load In-Reply-To: References: <48f0e2bf-7eec-d2e3-6c8e-38b6d4f5a6fb@schokola.de> Message-ID: <67e31074-d4f5-998b-000c-93d7aee095e8@schokola.de> https://github.com/varnishcache/varnish-cache/pull/2131 From reza at varnish-software.com Wed Nov 9 22:02:49 2016 From: reza at varnish-software.com (Reza Naghibi) Date: Wed, 9 Nov 2016 17:02:49 -0500 Subject: VIP9 - Expanding VCL object support Message-ID: This is another discussion for VIP9. The previous mailing list thread is linked below [0]. This allows objects to live in req, req.top, and bereq scopes. Currently objects are global to the VCL and can only be defined in vcl_init. The driver for this is allowing for multiple VMOD objects to exist in a single request, each with their own attributes. There is a usecase where we need to make multiple HTTP requests from VCL to 3rd party services and then build multiple security related digests on several aspects of the request, response, and 3rd party services. So having proper objects here would make the VMOD based solution extremely clean and easy to understand. I have a branch ready for a PR, but it was requested to have another discussion. Branch is located here [1] and based on master, 5.0, as of last week. m00026.vtc shows how the VCL looks [2]. The goal would be to have this available to VMODs in the next major release, Q1 2017. I will briefly address several of the concerns that have been brought up: * Syntax is too verbose. I agree that the first iteration kind of went overboard with the syntax. Currently, the syntax requires a (scope) style cast when you define the object and thats it. Please see [2]. * This implementation is completely backwards compatible with how objects are defined in 4.X and does not require VMOD code to change. Object VMODs will be immediately available to these new scopes and will still go thru the _init() _fini() lifecycle. * Conflicts with VIP1. This VIP is only concerned with exposing PRIV_TASK and PRIV_TOP into existing objects at the VCL level. I believe VIP1 is concerned with PRIV_* at the VMOD level. So I believe these are a bit orthogonal and independent. * VMOD safety. Im pretty confident that VMOD objects actually allow for higher levels of reference and memory safety because objects have explicit _init() and _fini() methods and are passed in a struct for tracking state. So if you wanted to reference objects from other objects, then you could implement your own ref counting algorithm ontop of __init(), __finish(), and the state (or a gc implementation or whatever algorithm you want). I think this argument is a bit arbitrary because if we look at something as simple as VCL_STRING, which all VMODs have access to via req/resp headers, if you were to reference that in a VMOD, it will be freed from under you and leave you with unsafe code and memory. So any kind of unsafe VMOD example probably extends to the entire VMOD universe and should not be limited to this one VIP. Safe coding practices used today in VMODs should not be abandoned in context of this VIP :) It might be more helpful to see the compiled VCL code interact with the VRT to understand how this PR will work. So the C code for m00026.vtc is located here [3]. Just grep for g0, r0, r1, t0, and b0 to better see how this is implemented. [0] https://www.varnish-cache.org/lists/pipermail/varnish-dev/2016-April/008906.html [1] https://github.com/varnishcache/varnish-cache/compare/master...rezan:feature/object_scopes_master [2] https://github.com/rezan/varnish-cache/blob/feature/object_scopes_master/bin/varnishtest/tests/m00026.vtc [3] https://gist.github.com/rezan/76f92f76d31ee2d2105501b63612db43 -- Reza Naghibi Varnish Software -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at varnish-software.com Fri Nov 11 11:06:47 2016 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 11 Nov 2016 11:06:47 +0000 Subject: VIP9 - Expanding VCL object support In-Reply-To: References: Message-ID: Hi Reza, I like this a lot. Though I am not completely sure about the way the scoping is expressed. It seems to me that the scoping is (mostly) implicit from where the new keyword is placed. If it's in vcl_backend_*, the scope is bereq. If it's in vcl_init the scope is global, and if it is in the others the scope is req (except when it's top). This leaves top as the only odd scope, and if we found some other way to deal with that special case, we would not need the scope cast in most places making it prettier. Perhaps only require the scope cast for (top)? Also how the scoping tests are performed in the vcc compiler leaves things to be desired. The first attached test case shows something that fails as a compiler error, but that I believe should be perfectly valid VCL (and the ordering is certainly something I've seen in plenty of messy user VCLs). In the test the vcl_deliver method comes before the vcl_recv in the input text, and the scoping then errors out on seeing the object in vcl_deliver that is defined in vcl_recv. The execution of vcl_recv is always before vcl_deliver, so the code isn't wrong, and would work just fine if it wasn't for the use of the object. I assume that the same issue is also present for vcl_init today. I believe we should address this, and could do so by enforcing an order in which the VCL functions are compiled. So after having read all of the source code, the compiler should in order do vcl_init, vcl_recv ... vcl_deliver, vcl_synth, vcl_backend_*. The 2nd attached test case demonstrates another concern on uninitialised objects. This test case fails with an assertion in the debug vmod for obvious reasons. My concern is that it is tempting to write VCL code like this, and having the Varnish assert itself by a simple VCL mistake is bad. (I do believe that similar issues exist for our current vcl_init-only director objects, though it's less error prone because noone writes if-clauses in vcl_init). One way of looking at this issue is to say that the debug object used here is in the wrong to assert when it's uninitialised, and that the uninitialised state is a valid vmod object state. The vmods then need to define correct behaviour for these cases (e.g. do nothing for the right meaning of nothing, for this case I guess return the empty string). And we should formally define this in the nonexistant vmod developer guide. Another approach is to define that object initialisation can't fail, which is more in-spirit with the VCL language. So make the VCC compiler give compiler errors if it can't say for sure that the new statement for an object has been executed before the object method invocation. E.g. an object method can't be used in vcl_recv for an object that is created in vcl_deliver. This would then drag with it new scopes for every curly bracket opening inside of the VCL functions. So an object can be new'ed and used inside the if-statement, but is out-of-scope outside of it. If we do define that object initialisation can't fail, I would also like to see a proper exception mechanism build in. So vmods can throw exceptions that halt VCL execution and safely call the priv free callbacks to clean up, before presenting an error. Details unclear. Lastly I think we should have the calling scope limits as vmod metadata in place before allowing this. So a vmod object that's only meant to be called in a global scope can declare that in its vcc file. So in conclusion I am concerned about the Pandora's box this is opening, and there are building blocks that are missing today that should be addressed first. Martin On Wed, 9 Nov 2016 at 23:13 Reza Naghibi wrote: > This is another discussion for VIP9. The previous mailing list thread is > linked below [0]. > > This allows objects to live in req, req.top, and bereq scopes. Currently > objects are global to the VCL and can only be defined in vcl_init. > > The driver for this is allowing for multiple VMOD objects to exist in a > single request, each with their own attributes. There is a usecase where we > need to make multiple HTTP requests from VCL to 3rd party services and then > build multiple security related digests on several aspects of the request, > response, and 3rd party services. So having proper objects here would make > the VMOD based solution extremely clean and easy to understand. > > I have a branch ready for a PR, but it was requested to have another > discussion. Branch is located here [1] and based on master, 5.0, as of last > week. m00026.vtc shows how the VCL looks [2]. The goal would be to have > this available to VMODs in the next major release, Q1 2017. > > I will briefly address several of the concerns that have been brought up: > > * Syntax is too verbose. I agree that the first iteration kind of went > overboard with the syntax. Currently, the syntax requires a (scope) style > cast when you define the object and thats it. Please see [2]. > > * This implementation is completely backwards compatible with how objects > are defined in 4.X and does not require VMOD code to change. Object VMODs > will be immediately available to these new scopes and will still go thru > the _init() _fini() lifecycle. > > * Conflicts with VIP1. This VIP is only concerned with exposing PRIV_TASK > and PRIV_TOP into existing objects at the VCL level. I believe VIP1 is > concerned with PRIV_* at the VMOD level. So I believe these are a bit > orthogonal and independent. > > * VMOD safety. Im pretty confident that VMOD objects actually allow for > higher levels of reference and memory safety because objects have explicit > _init() and _fini() methods and are passed in a struct for tracking state. > So if you wanted to reference objects from other objects, then you could > implement your own ref counting algorithm ontop of __init(), __finish(), > and the state (or a gc implementation or whatever algorithm you want). I > think this argument is a bit arbitrary because if we look at something as > simple as VCL_STRING, which all VMODs have access to via req/resp headers, > if you were to reference that in a VMOD, it will be freed from under you > and leave you with unsafe code and memory. So any kind of unsafe VMOD > example probably extends to the entire VMOD universe and should not be > limited to this one VIP. Safe coding practices used today in VMODs should > not be abandoned in context of this VIP :) > > It might be more helpful to see the compiled VCL code interact with the > VRT to understand how this PR will work. So the C code for m00026.vtc is > located here [3]. Just grep for g0, r0, r1, t0, and b0 to better see how > this is implemented. > > [0] > https://www.varnish-cache.org/lists/pipermail/varnish-dev/2016-April/008906.html > > [1] > https://github.com/varnishcache/varnish-cache/compare/master...rezan:feature/object_scopes_master > > [2] > https://github.com/rezan/varnish-cache/blob/feature/object_scopes_master/bin/varnishtest/tests/m00026.vtc > > [3] https://gist.github.com/rezan/76f92f76d31ee2d2105501b63612db43 > > -- > Reza Naghibi > Varnish Software > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: scope.vtc Type: application/octet-stream Size: 340 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: scope2.vtc Type: application/octet-stream Size: 376 bytes Desc: not available URL: From reza at varnish-software.com Fri Nov 11 15:24:06 2016 From: reza at varnish-software.com (Reza Naghibi) Date: Fri, 11 Nov 2016 10:24:06 -0500 Subject: VIP9 - Expanding VCL object support In-Reply-To: References: Message-ID: > scoping is (mostly) implicit I agree here. Not sure why I have been fixated on this ornamental prefixing/casting, but I can do away with it. (top) would still exist and any future scopes that are invented which fall into req/bereq would also have to be casted in this way. > variable uses out of order Right now this seems to be a limitation in the vcc. Im tempted to say this would simply be irksome for the time being. I would have to research how this could be addressed, but im feeling this fix is out of scope right now (and can be fixed independently of this VIP). > uninitialised objects This is a very valid concern and good thing you brought this up. When I first read it, I thought about how other languages tackle this, in particular, Java and C. Both languages either error out or warn at compile time about uninitialized variables. So I think we could use the same approach. If you define a variable inside of a control statement (if/else), then that's a compile error due to it possibly being uninitialized and unsafe. So this definitely needs to be addressed and fixed. There is also some weirdness around defining variables inside of custom subs. This worked on 4.0, but I think something changed in 5.0. So this needs to be fixed too :) > proper exception mechanism Totally agree, right now you can only assert and take down the entire process. There are many times where it would be nice to just have a 'request panic' and kill the request, but leave everything else intact. Not sure if this feature is in scope for this VIP, but its definitely something that VMODs and Varnish code in general needs access to. > scope limits as vmod metadata Ya, I thought about this and it probably makes sense to add this in just to give VMOD authors more assurances that their objects will be used in the way they expect. In theory, it shouldn't matter, but obviously in practice there is a big difference in VCL global vs request vs [insert random scope here]. So I will see if I can address this as well with some new vcc definition syntax. This is good feedback, thank you. -- Reza Naghibi Varnish Software On Fri, Nov 11, 2016 at 6:06 AM, Martin Blix Grydeland < martin at varnish-software.com> wrote: > Hi Reza, > > I like this a lot. Though I am not completely sure about the way the > scoping is expressed. It seems to me that the scoping is (mostly) implicit > from where the new keyword is placed. If it's in vcl_backend_*, the scope > is bereq. If it's in vcl_init the scope is global, and if it is in the > others the scope is req (except when it's top). This leaves top as the only > odd scope, and if we found some other way to deal with that special case, > we would not need the scope cast in most places making it prettier. Perhaps > only require the scope cast for (top)? > > Also how the scoping tests are performed in the vcc compiler leaves things > to be desired. The first attached test case shows something that fails as a > compiler error, but that I believe should be perfectly valid VCL (and the > ordering is certainly something I've seen in plenty of messy user VCLs). In > the test the vcl_deliver method comes before the vcl_recv in the input > text, and the scoping then errors out on seeing the object in vcl_deliver > that is defined in vcl_recv. The execution of vcl_recv is always before > vcl_deliver, so the code isn't wrong, and would work just fine if it wasn't > for the use of the object. I assume that the same issue is also present for > vcl_init today. I believe we should address this, and could do so by > enforcing an order in which the VCL functions are compiled. So after having > read all of the source code, the compiler should in order do vcl_init, > vcl_recv ... vcl_deliver, vcl_synth, vcl_backend_*. > > The 2nd attached test case demonstrates another concern on uninitialised > objects. This test case fails with an assertion in the debug vmod for > obvious reasons. My concern is that it is tempting to write VCL code like > this, and having the Varnish assert itself by a simple VCL mistake is bad. > (I do believe that similar issues exist for our current vcl_init-only > director objects, though it's less error prone because noone writes > if-clauses in vcl_init). > > One way of looking at this issue is to say that the debug object used here > is in the wrong to assert when it's uninitialised, and that the > uninitialised state is a valid vmod object state. The vmods then need to > define correct behaviour for these cases (e.g. do nothing for the right > meaning of nothing, for this case I guess return the empty string). And we > should formally define this in the nonexistant vmod developer guide. > > Another approach is to define that object initialisation can't fail, which > is more in-spirit with the VCL language. So make the VCC compiler give > compiler errors if it can't say for sure that the new statement for an > object has been executed before the object method invocation. E.g. an > object method can't be used in vcl_recv for an object that is created in > vcl_deliver. This would then drag with it new scopes for every curly > bracket opening inside of the VCL functions. So an object can be new'ed and > used inside the if-statement, but is out-of-scope outside of it. > > If we do define that object initialisation can't fail, I would also like > to see a proper exception mechanism build in. So vmods can throw exceptions > that halt VCL execution and safely call the priv free callbacks to clean > up, before presenting an error. Details unclear. > > Lastly I think we should have the calling scope limits as vmod metadata in > place before allowing this. So a vmod object that's only meant to be called > in a global scope can declare that in its vcc file. > > So in conclusion I am concerned about the Pandora's box this is opening, > and there are building blocks that are missing today that should be > addressed first. > > Martin > > On Wed, 9 Nov 2016 at 23:13 Reza Naghibi > wrote: > >> This is another discussion for VIP9. The previous mailing list thread is >> linked below [0]. >> >> This allows objects to live in req, req.top, and bereq scopes. Currently >> objects are global to the VCL and can only be defined in vcl_init. >> >> The driver for this is allowing for multiple VMOD objects to exist in a >> single request, each with their own attributes. There is a usecase where we >> need to make multiple HTTP requests from VCL to 3rd party services and then >> build multiple security related digests on several aspects of the request, >> response, and 3rd party services. So having proper objects here would make >> the VMOD based solution extremely clean and easy to understand. >> >> I have a branch ready for a PR, but it was requested to have another >> discussion. Branch is located here [1] and based on master, 5.0, as of last >> week. m00026.vtc shows how the VCL looks [2]. The goal would be to have >> this available to VMODs in the next major release, Q1 2017. >> >> I will briefly address several of the concerns that have been brought up: >> >> * Syntax is too verbose. I agree that the first iteration kind of went >> overboard with the syntax. Currently, the syntax requires a (scope) style >> cast when you define the object and thats it. Please see [2]. >> >> * This implementation is completely backwards compatible with how objects >> are defined in 4.X and does not require VMOD code to change. Object VMODs >> will be immediately available to these new scopes and will still go thru >> the _init() _fini() lifecycle. >> >> * Conflicts with VIP1. This VIP is only concerned with exposing PRIV_TASK >> and PRIV_TOP into existing objects at the VCL level. I believe VIP1 is >> concerned with PRIV_* at the VMOD level. So I believe these are a bit >> orthogonal and independent. >> >> * VMOD safety. Im pretty confident that VMOD objects actually allow for >> higher levels of reference and memory safety because objects have explicit >> _init() and _fini() methods and are passed in a struct for tracking state. >> So if you wanted to reference objects from other objects, then you could >> implement your own ref counting algorithm ontop of __init(), __finish(), >> and the state (or a gc implementation or whatever algorithm you want). I >> think this argument is a bit arbitrary because if we look at something as >> simple as VCL_STRING, which all VMODs have access to via req/resp headers, >> if you were to reference that in a VMOD, it will be freed from under you >> and leave you with unsafe code and memory. So any kind of unsafe VMOD >> example probably extends to the entire VMOD universe and should not be >> limited to this one VIP. Safe coding practices used today in VMODs should >> not be abandoned in context of this VIP :) >> >> It might be more helpful to see the compiled VCL code interact with the >> VRT to understand how this PR will work. So the C code for m00026.vtc is >> located here [3]. Just grep for g0, r0, r1, t0, and b0 to better see how >> this is implemented. >> >> [0] https://www.varnish-cache.org/lists/pipermail/varnish-dev/ >> 2016-April/008906.html >> >> [1] https://github.com/varnishcache/varnish-cache/compare/master...rezan: >> feature/object_scopes_master >> >> [2] https://github.com/rezan/varnish-cache/blob/feature/ >> object_scopes_master/bin/varnishtest/tests/m00026.vtc >> >> [3] https://gist.github.com/rezan/76f92f76d31ee2d2105501b63612db43 >> >> -- >> Reza Naghibi >> Varnish Software >> _______________________________________________ >> varnish-dev mailing list >> varnish-dev at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Mon Nov 14 10:28:11 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 14 Nov 2016 10:28:11 +0000 Subject: "vtest" script for review/experimentation Message-ID: <20887.1479119291@critter.freebsd.dk> I'm trying to construct a light-weight cross-platform tinderbox facility which doesn't require jvm or other esoterics. First step is this 'vtest.sh' script: http://phk.freebsd.dk/misc/vtest.sh It basically runs in a loop, git pulls, runs autogen.des and make distcheck after which a report is sent of to the varnish project server. I would appreciate if people would take a few minutes to review and possibly test this script to see that it is as portable as I hope. Sending to the varnish project server is done with a ssh key built by the script for the same purpose. Needless to say I need a copy of that key before the scp will work, but the script will tell you that... The script should run fine without root permission, but since we would also like coverage of the JAIL bits I hope we can get some people to run it as root also, and therefore it needs to be reviewed and scrutinized properly. Obviously, running a "make distcheck" can never be "safe" in any meaning of the word, in both theory and practice we can put any kind of shit in our Makefiles, but at the very minimum the vtest.sh script should not increase exposure beyond that. The backend side is very rudimentary still, but I have a script which can output this: 1479111273 718531c +autogen +distcheck FreeBSD_11.0-CURRENT_amd64 phk 1477252091 718531c +autogen +distcheck FreeBSD_11.0-RELEASE-p1_arm phk 1479116010 1a989e2 +autogen +distcheck FreeBSD_11.0-CURRENT_amd64 phk 1477255339 1a989e2 +autogen -distcheck FreeBSD_11.0-RELEASE-p1_arm phk c00049 ---- v1 11.1 Not true: n_object (4) == 3 (3) More to follow on that side as I get some test-data to work with. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Mon Nov 14 11:30:51 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 14 Nov 2016 12:30:51 +0100 Subject: "vtest" script for review/experimentation In-Reply-To: <20887.1479119291@critter.freebsd.dk> References: <20887.1479119291@critter.freebsd.dk> Message-ID: > I would appreciate if people would take a few minutes to > review and possibly test this script to see that it is > as portable as I hope. As the official shell over-enthusiast I will review it. First nitpick: > nice make distclean > /dev/null 2>&1 || true > nice /usr/bin/time sh autogen.des The first distclean is redundant, it is done by autogen.des already, unless you want to avoid timing the distclean too. I would replace `/usr/bin/time` by `command time` to pick the binary from the PATH, or `command time -v` if it's portable. I would log and create the report in $TMPDIR. In failedtests() you assume that all tests are VTCs but a distcheck will include C tests as well, although they fail less often. Further review later, that was a quick skim through the script. > The script should run fine without root permission, but > since we would also like coverage of the JAIL bits I hope > we can get some people to run it as root also, and therefore > it needs to be reviewed and scrutinized properly. Bringing an "esoteric" here, what about capabilities to run as a pseudo-root? Where your process thinks it's run by root, and has root privileges in (let's call it) its container. Can you pretend to be root in a FreeBSD jail and have actual privileges limited to inside the jail? Otherwise there's VMs... Dridi From phk at phk.freebsd.dk Mon Nov 14 11:39:29 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 14 Nov 2016 11:39:29 +0000 Subject: "vtest" script for review/experimentation In-Reply-To: References: <20887.1479119291@critter.freebsd.dk> Message-ID: <21164.1479123569@critter.freebsd.dk> -------- In message , Dridi Boukel moune writes: >As the official shell over-enthusiast I will review it. Cool. >First nitpick: > >> nice make distclean > /dev/null 2>&1 || true >> nice /usr/bin/time sh autogen.des > >The first distclean is redundant, it is done by autogen.des already, >unless you want to avoid timing the distclean too. That was the point. >I would replace `/usr/bin/time` by `command time` to pick the binary >from the PATH, or `command time -v` if it's portable. Right now I don't use the output of the time for anything, the idea was to collect the timing info and include it in the report. >I would log and create the report in $TMPDIR. yeah, probably a good idea as well. >In failedtests() you assume that all tests are VTCs but a distcheck >will include C tests as well, although they fail less often. I've been wondering if we should wrap the C tests in a VTC, for sake of uniformity, but since I can not even remember when a C test has failed last time, it has priority well down the list. >Can you pretend to be root in a FreeBSD jail and have actual >privileges limited to inside the jail? That's basically what jails does... My intent with this script is that it does not know or need to know about its privileges (that's a 100% surefire way to punch holes in your own code). The idea is simply to run make distcheck and report the result. What distcheck investigates, should not affect this script. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From martin at varnish-software.com Mon Nov 14 13:48:16 2016 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 14 Nov 2016 13:48:16 +0000 Subject: VIP9 - Expanding VCL object support In-Reply-To: References: Message-ID: On Fri, 11 Nov 2016 at 16:24 Reza Naghibi wrote: > > variable uses out of order > > Right now this seems to be a limitation in the vcc. Im tempted to say this > would simply be irksome for the time being. I would have to research how > this could be addressed, but im feeling this fix is out of scope right now > (and can be fixed independently of this VIP). > Agreed that it it can be fixed independently, though I do think that the new object powers will make the problem more prevalent. > > uninitialised objects > > This is a very valid concern and good thing you brought this up. When I > first read it, I thought about how other languages tackle this, in > particular, Java and C. Both languages either error out or warn at compile > time about uninitialized variables. So I think we could use the same > approach. If you define a variable inside of a control statement (if/else), > then that's a compile error due to it possibly being uninitialized and > unsafe. So this definitely needs to be addressed and fixed. > We do not correctly handle object initialization inside of control statements today, so disallowing that at compile time seems like a natural first step. Though there are use cases where it would be nice to be able to do that, so having a plan for what it should be like down the line would help to make sure we don't get it wrong in the mean time. Martin -------------- next part -------------- An HTML attachment was scrubbed... URL: From slink at schokola.de Mon Nov 21 13:40:46 2016 From: slink at schokola.de (Nils Goroll) Date: Mon, 21 Nov 2016 14:40:46 +0100 Subject: linux (debian?) epoll Message-ID: <61ed2029-385a-27ec-5d61-6a7235356564@schokola.de> Hi, we've seen #2117 appear about a month ago and #2106 a bit earlier. Martin and myself now believe that #2106 may have the same root cause as #2117, fixed (actually worked around) by #2133 - which would be a suspected regression in Linux epoll(), maybe debian specific. So this is a shout out to those following linux changes closely: If anyone is aware of any known issue or discussion around epoll changes, please let us know. Thanks, Nils * #2117 https://github.com/varnishcache/varnish-cache/issues/2117 * #2133 https://github.com/varnishcache/varnish-cache/pull/2133 * #2106 https://github.com/varnishcache/varnish-cache/issues/2106 From phk at phk.freebsd.dk Mon Nov 21 20:47:54 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 21 Nov 2016 20:47:54 +0000 Subject: Vtest compiler/arch issue Message-ID: <62953.1479761274@critter.freebsd.dk> I spent some time looking at this. I want to avoid, to the fullest extent possible, that people have to supply such information by hand, because they generally suck at it, and we should be able to autodetect the compiler/target/architecture. This patch will do it: (NB: Copy&Pasted) diff --git a/bin/varnishtest/Makefile.am b/bin/varnishtest/Makefile.am index 26563e2..6b7e4eb 100644 --- a/bin/varnishtest/Makefile.am +++ b/bin/varnishtest/Makefile.am @@ -9,7 +9,15 @@ check: check-am check-local check-am: check-local # See if list of checks have changed, recheck check-local: - LC_ALL=C; \ + @echo "VTEST CCVER BEGIN" + @ for i in --version -v -V -qversion ; \ + do \ + if ${CC} ${i} ; then \ + break ; \ + fi \ + done + @echo "VTEST CCVER END" + @LC_ALL=C; \ if [ "$$(cd $(srcdir) && echo tests/*.vtc)" != "@VTC_TESTS@" ]; then \ cd $(top_builddir) && ./config.status --recheck ; \ fi Only, that doesn't work at all in practice. I need to be able to build the "id" of the report also if the "autogen" step fails, which is where the configure script which eventually determins what compiler and target arch will be. Also, if makedistcheck ends up GOOD, the output does not get returned. So I think we are stuck with manual input for this and maybe other cases as well. So I'm going to add a "message" variable in vtest.sh which the owner of the machine can set, and that will be part of the 'id' and displayed at the bottom line at the table header People get to write whatever they want in that message, provided they use max 10 char from the set [A-Za-z0-9/. _-]. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Tue Nov 22 10:42:51 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 22 Nov 2016 11:42:51 +0100 Subject: Vtest compiler/arch issue In-Reply-To: <62953.1479761274@critter.freebsd.dk> References: <62953.1479761274@critter.freebsd.dk> Message-ID: > People get to write whatever they want in that message, provided > they use max 10 char from the set [A-Za-z0-9/. _-]. I'm feeling compelled to point out that yesterday you were complaining about artificial compiler limitations that are no longer relevant in today's computing. And as such with 10 characters I can't even fit "Linux-x86_64": MESSAGE=$(uname -s)-$(uname -m) Cheers, Dridi From phk at phk.freebsd.dk Tue Nov 22 10:55:07 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 22 Nov 2016 10:55:07 +0000 Subject: Vtest compiler/arch issue In-Reply-To: References: <62953.1479761274@critter.freebsd.dk> Message-ID: <67571.1479812107@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >> People get to write whatever they want in that message, provided >> they use max 10 char from the set [A-Za-z0-9/. _-]. > >I'm feeling compelled to point out that yesterday you were complaining >about artificial compiler limitations that are no longer relevant in >today's computing. Correct. But that has absolutly nothing to do with putting random user input into your HTML web-pages without proper data-hygiene. >today's computing. And as such with 10 characters I can't even fit >"Linux-x86_64": You wouldn't need to, that is already derived from the uname and lsb information. This is for writing "gcc-m32" or similar. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From devnexen at gmail.com Tue Nov 22 13:41:14 2016 From: devnexen at gmail.com (David CARLIER) Date: Tue, 22 Nov 2016 13:41:14 +0000 Subject: varnish and pcre2 Message-ID: Hi dear developers, You probably thought of it but is there any plan (next year ?) to implement pcre2 support in varnish ? Thanks in advance. Kind regards. -------------- next part -------------- An HTML attachment was scrubbed... URL: From slink at schokola.de Tue Nov 22 22:22:19 2016 From: slink at schokola.de (Nils Goroll) Date: Tue, 22 Nov 2016 23:22:19 +0100 Subject: varnish and pcre2 In-Reply-To: References: Message-ID: On 22/11/16 14:41, David CARLIER wrote: > You probably thought of it but is there any plan (next year ?) to implement > pcre2 support in varnish ? IIRC Geoff has a pcre2 version of https://code.uplex.de/uplex-varnish/libvmod-re in the works From geoff at uplex.de Wed Nov 23 08:42:56 2016 From: geoff at uplex.de (Geoff Simmons) Date: Wed, 23 Nov 2016 09:42:56 +0100 Subject: varnish and pcre2 In-Reply-To: References: Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 11/22/2016 11:22 PM, Nils Goroll wrote: > On 22/11/16 14:41, David CARLIER wrote: >> You probably thought of it but is there any plan (next year ?) to >> implement pcre2 support in varnish ? > > IIRC Geoff has a pcre2 version of > https://code.uplex.de/uplex-varnish/libvmod-re in the works "In the works" is saying too much, more like kicking around the idea. Real Soon Now in my Copious Free Time. I haven't heard any developers talking about pcre2, and I suspect that support via VMODs for new regex libraries is the most likely path forward. One of the reasons for introducing VMODs was to lessen the burden on the main project to add and maintain new features, and it seems to me that this is a good case for that. This one, for example, supports use of Google re2: https://code.uplex.de/uplex-varnish/libvmod-re2 @David, do you have a use case for pcre2? I haven't heard much about it, for example whether or how much it performs better than original pcre. Just the fact that the pcre project will be going forward with pcre2 is a good reason to support it. Any other arguments for it would help to move the VMOD out of the vaporware column. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJYNVaQAAoJEOUwvh9pJNURu6MP/2DZ3uHwO/XvcbQrWeOM8wsV CnGEn+7fr1pOV94sfhXyVMs4I3YIChzt/ys+Mc72MLkw+mxnyC1y+8zrXeubpO5S 7DQX6uP7iadtoGtBuY0WWGyha1X/dbIGYSsI8B9JRZPZnT7MP5y4AkOh8hnJZrW/ dQxGqIQPkB+WTDivTE/T6BXQAFk+HLJbHZ7sxH2LzddqIV8+Ewb4Ul8QY3hwxLo5 Z/8C3K0nEdLH/Z7BgMTz3ud7CTLVEHowdgEBqo9lR6cP7v4JrGnt6pfdoYBFt340 YDws+mjG1fmNBTc+avV8KNeXST7D8X6FXEPsY+TXegtfU4Ki5O22071rRLswtoj8 s++ERLmYqKP9z+gkp7fZEBiq17JNeKtJZQCpm90i939imeqdmCAFb+qr9m6VXVJJ FCNgFfNj0uea7RQoC8ewzh+kvxj5PNUlOUr+bdx1u2N0NqVe2LE0h49nvt5/LNmd liABQrHhIjE9GvvsLlcaqx95Xz7aXeb0U2irfd2YQKD8Pbc5Lgjq0F9GLGU5G5lD WdeyCve/CutPxKx0lOQRLtsjyWEy/3D9d7QZpKBXd6xe/FpuySj+T/kCWplYMq+C jSHJ+xm/ZIJwqNjmbulMxSdlqqqIPMHLALOp/iNvNz906qorIVrh8k17huy7Eo6c wGBT/hEEMR9fS/sAmrdg =wB9x -----END PGP SIGNATURE----- From devnexen at gmail.com Wed Nov 23 08:47:07 2016 From: devnexen at gmail.com (David CARLIER) Date: Wed, 23 Nov 2016 08:47:07 +0000 Subject: varnish and pcre2 In-Reply-To: References: Message-ID: Hi, @Geoff your answer makes sense, so to explain some people start to install pcre2 over pcre due to pcre's security flaws rather than supposely performance increases (I do not think there is really). Plus I have submitted a pcre2 support this week to haproxy which might land in the next branch. I was just curious how other softwares was doing ;) Kind regards. On 23 November 2016 at 08:42, Geoff Simmons wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > On 11/22/2016 11:22 PM, Nils Goroll wrote: > > On 22/11/16 14:41, David CARLIER wrote: > >> You probably thought of it but is there any plan (next year ?) to > >> implement pcre2 support in varnish ? > > > > IIRC Geoff has a pcre2 version of > > https://code.uplex.de/uplex-varnish/libvmod-re in the works > > "In the works" is saying too much, more like kicking around the idea. > Real Soon Now in my Copious Free Time. > > I haven't heard any developers talking about pcre2, and I suspect that > support via VMODs for new regex libraries is the most likely path > forward. One of the reasons for introducing VMODs was to lessen the > burden on the main project to add and maintain new features, and it > seems to me that this is a good case for that. > > This one, for example, supports use of Google re2: > > https://code.uplex.de/uplex-varnish/libvmod-re2 > > @David, do you have a use case for pcre2? I haven't heard much about > it, for example whether or how much it performs better than original > pcre. Just the fact that the pcre project will be going forward with > pcre2 is a good reason to support it. Any other arguments for it would > help to move the VMOD out of the vaporware column. > > > Best, > Geoff > - -- > ** * * UPLEX - Nils Goroll Systemoptimierung > > Scheffelstra?e 32 > 22301 Hamburg > > Tel +49 40 2880 5731 > Mob +49 176 636 90917 > Fax +49 40 42949753 > > http://uplex.de > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2 > > iQIcBAEBCAAGBQJYNVaQAAoJEOUwvh9pJNURu6MP/2DZ3uHwO/XvcbQrWeOM8wsV > CnGEn+7fr1pOV94sfhXyVMs4I3YIChzt/ys+Mc72MLkw+mxnyC1y+8zrXeubpO5S > 7DQX6uP7iadtoGtBuY0WWGyha1X/dbIGYSsI8B9JRZPZnT7MP5y4AkOh8hnJZrW/ > dQxGqIQPkB+WTDivTE/T6BXQAFk+HLJbHZ7sxH2LzddqIV8+Ewb4Ul8QY3hwxLo5 > Z/8C3K0nEdLH/Z7BgMTz3ud7CTLVEHowdgEBqo9lR6cP7v4JrGnt6pfdoYBFt340 > YDws+mjG1fmNBTc+avV8KNeXST7D8X6FXEPsY+TXegtfU4Ki5O22071rRLswtoj8 > s++ERLmYqKP9z+gkp7fZEBiq17JNeKtJZQCpm90i939imeqdmCAFb+qr9m6VXVJJ > FCNgFfNj0uea7RQoC8ewzh+kvxj5PNUlOUr+bdx1u2N0NqVe2LE0h49nvt5/LNmd > liABQrHhIjE9GvvsLlcaqx95Xz7aXeb0U2irfd2YQKD8Pbc5Lgjq0F9GLGU5G5lD > WdeyCve/CutPxKx0lOQRLtsjyWEy/3D9d7QZpKBXd6xe/FpuySj+T/kCWplYMq+C > jSHJ+xm/ZIJwqNjmbulMxSdlqqqIPMHLALOp/iNvNz906qorIVrh8k17huy7Eo6c > wGBT/hEEMR9fS/sAmrdg > =wB9x > -----END PGP SIGNATURE----- > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Wed Nov 23 09:44:50 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 23 Nov 2016 09:44:50 +0000 Subject: varnish and pcre2 In-Reply-To: References: Message-ID: <73032.1479894290@critter.freebsd.dk> -------- In message , David CARLIER writes: >Hi dear developers, > >You probably thought of it but is there any plan (next year ?) to implement >pcre2 support in varnish ? I will just echo what Geoff said: If we support pcre2 it will be as a VMOD. (In hindsight, it was a mistake to pull in pcre, rather than stick with POSIX regexps, but we didn't have VMODS back then, so it was either or.) If we find that practically everybody "import pcre2" in their VCL, we will probably drag that VMOD into the main project, otherwise it will live independently. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Wed Nov 23 10:38:37 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 23 Nov 2016 10:38:37 +0000 Subject: range/streaming of incomplete objects Message-ID: <80671.1479897517@critter.freebsd.dk> This IETF draft is interesting: https://tools.ietf.org/id/draft-pratt-httpbis-rand-access-live-00.txt The HTTPbis WG is looking for "go/nogo" input if this is something which should be adopted. Input welcome -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From geoff at uplex.de Wed Nov 23 14:15:08 2016 From: geoff at uplex.de (Geoff Simmons) Date: Wed, 23 Nov 2016 15:15:08 +0100 Subject: varnish and pcre2 In-Reply-To: References: Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 11/23/2016 09:47 AM, David CARLIER wrote: > > ... some people start to install pcre2 over pcre due to pcre's > security flaws Is there something to be worried about? I'm looking at the CVEs for pcre, and it looks to me like there's nothing that can't be fixed by updating the libraries. The most recent one (March of this year) applies to both pcre and pcre2. On 11/23/2016 10:44 AM, Poul-Henning Kamp wrote: > > If we find that practically everybody "import pcre2" in their VCL, > we will probably drag that VMOD into the main project, otherwise it > will live independently. Philip Hazel's plan is to continue maintaining the original pcre lib only for bug fixes, and all new features will go into pcre2. At some point in the mid to long term, old pcre might become just too old. I suspect that will be when everyone will be importing a VMOD for regexen. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJYNaRsAAoJEOUwvh9pJNURbHYP/iKBWkJqryFdSHZxxQRxim6U LGtiJbC4JIu9il68sMNn/I+Pqa/7xp1ChWe3GpZLlcBaM9/aOamJrR8WvRaRPHhv dGIILXBotzMPnUJyk3mEott2O6W+4sXbknNJJITizo5gEvNgHU4T7c11GxHL+dp/ o0Fav0eh9pL+5FzLSdBuBx4vQefxfjB8XR1XCVBSplZzbQsfs7xbeN9qWvQYzEw1 qyWlzHBRKz2Ao9e2PUfZU5wfDp56KFQV5kXRGU1cm+HER+Q+0/gU5/w0ze8TaIcx v6QqK5Xxawd0Ju/pf5ve/ujsHHqGdeD5R5ZrG371gE9MsOvtzxaNurZFK9speRd/ v71lSEk+EOvSy76JJM+ggIL3GZIHKlnWRs2FTIu0dtpIyUwvQX8b0VaZzr/q4QGc 0vmRmW53FYApyj37naYJMZVspBikVJaltbiXR9e8gVWikWVsluAgKMx1CLNLzWhg 1LYrGvxwOpOdqg/efkPfP+RwqhEnG5pl7Wbug9fHYmJiwsVhICE1jBaBmiS6GyNF yw+x2Ynh79oxyShBMMmWxkh/hKbAUMVqgY7oTpqUG8v00ynhqR4bZDs6r4l6SULs 2GPFrcknkZiYwO16YlXsNVDC9BxO4OpmB26lyc00C8DhW0HuYmkRKw3hkBJTEiDf dwcejzdIdWbKso8vd9Ue =jY5K -----END PGP SIGNATURE----- From devnexen at gmail.com Wed Nov 23 14:21:43 2016 From: devnexen at gmail.com (David CARLIER) Date: Wed, 23 Nov 2016 14:21:43 +0000 Subject: varnish and pcre2 In-Reply-To: References: Message-ID: @Geoff I agree with you, there s no urgency to implement it, althought the API of pcre2 is way better designed (ie the ovec array resulting from the match is properly sized now for example) but more for the near future indeed, anyway in Haproxy the developement is done but won t be available before next year. On 23 November 2016 at 14:15, Geoff Simmons wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > On 11/23/2016 09:47 AM, David CARLIER wrote: > > > > ... some people start to install pcre2 over pcre due to pcre's > > security flaws > > Is there something to be worried about? I'm looking at the CVEs for > pcre, and it looks to me like there's nothing that can't be fixed by > updating the libraries. The most recent one (March of this year) > applies to both pcre and pcre2. > > On 11/23/2016 10:44 AM, Poul-Henning Kamp wrote: > > > > If we find that practically everybody "import pcre2" in their VCL, > > we will probably drag that VMOD into the main project, otherwise it > > will live independently. > > Philip Hazel's plan is to continue maintaining the original pcre lib > only for bug fixes, and all new features will go into pcre2. At some > point in the mid to long term, old pcre might become just too old. I > suspect that will be when everyone will be importing a VMOD for regexen. > > > Best, > Geoff > - -- > ** * * UPLEX - Nils Goroll Systemoptimierung > > Scheffelstra?e 32 > 22301 Hamburg > > Tel +49 40 2880 5731 > Mob +49 176 636 90917 > Fax +49 40 42949753 > > http://uplex.de > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2 > > iQIcBAEBCAAGBQJYNaRsAAoJEOUwvh9pJNURbHYP/iKBWkJqryFdSHZxxQRxim6U > LGtiJbC4JIu9il68sMNn/I+Pqa/7xp1ChWe3GpZLlcBaM9/aOamJrR8WvRaRPHhv > dGIILXBotzMPnUJyk3mEott2O6W+4sXbknNJJITizo5gEvNgHU4T7c11GxHL+dp/ > o0Fav0eh9pL+5FzLSdBuBx4vQefxfjB8XR1XCVBSplZzbQsfs7xbeN9qWvQYzEw1 > qyWlzHBRKz2Ao9e2PUfZU5wfDp56KFQV5kXRGU1cm+HER+Q+0/gU5/w0ze8TaIcx > v6QqK5Xxawd0Ju/pf5ve/ujsHHqGdeD5R5ZrG371gE9MsOvtzxaNurZFK9speRd/ > v71lSEk+EOvSy76JJM+ggIL3GZIHKlnWRs2FTIu0dtpIyUwvQX8b0VaZzr/q4QGc > 0vmRmW53FYApyj37naYJMZVspBikVJaltbiXR9e8gVWikWVsluAgKMx1CLNLzWhg > 1LYrGvxwOpOdqg/efkPfP+RwqhEnG5pl7Wbug9fHYmJiwsVhICE1jBaBmiS6GyNF > yw+x2Ynh79oxyShBMMmWxkh/hKbAUMVqgY7oTpqUG8v00ynhqR4bZDs6r4l6SULs > 2GPFrcknkZiYwO16YlXsNVDC9BxO4OpmB26lyc00C8DhW0HuYmkRKw3hkBJTEiDf > dwcejzdIdWbKso8vd9Ue > =jY5K > -----END PGP SIGNATURE----- > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Wed Nov 23 14:36:11 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 23 Nov 2016 14:36:11 +0000 Subject: varnish and pcre2 In-Reply-To: References: Message-ID: <88773.1479911771@critter.freebsd.dk> -------- In message , Geoff Simmons writes: >Philip Hazel's plan is to continue maintaining the original pcre lib >only for bug fixes, and all new features will go into pcre2. At some >point in the mid to long term, old pcre might become just too old. I >suspect that will be when everyone will be importing a VMOD for regexen. Name it wisely then, ot they will all curse your name on a daily basis :-) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Fri Nov 25 10:32:55 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 25 Nov 2016 10:32:55 +0000 Subject: autocrap de-cargo-culting Message-ID: <52150.1480069975@critter.freebsd.dk> Does anybody know of any reason not to apply this patch ? diff --git a/configure.ac b/configure.ac index bfc0099..da8d2d9 100644 --- a/configure.ac +++ b/configure.ac @@ -21,12 +21,6 @@ AM_SILENT_RULES([yes]) AC_DISABLE_STATIC AC_PROG_LIBTOOL -# Checks for programs. -AC_GNU_SOURCE -AC_PROG_CC -AC_PROG_CC_STDC -AC_PROG_CPP - AX_PTHREAD(,[AC_MSG_ERROR([Could not configure pthreads support])]) LIBS="$PTHREAD_LIBS $LIBS" @@ -88,13 +82,6 @@ fi save_LIBS="${LIBS}" LIBS="" -AC_SEARCH_LIBS(pthread_create, [thr pthread c_r]) -PTHREAD_LIBS="${LIBS}" -LIBS="${save_LIBS}" -AC_SUBST(PTHREAD_LIBS) - -save_LIBS="${LIBS}" -LIBS="" AC_CHECK_LIB(socket, socket) AC_CHECK_LIB(nsl, getaddrinfo) NET_LIBS="${LIBS}" @@ -198,43 +185,18 @@ AC_CHECK_HEADERS([edit/readline/readline.h], ]) # Checks for header files. -AC_HEADER_STDC -AC_HEADER_SYS_WAIT -AC_HEADER_TIME -AC_CHECK_HEADERS([sys/param.h]) -AC_CHECK_HEADERS([sys/types.h]) AC_CHECK_HEADERS([sys/endian.h]) AC_CHECK_HEADERS([sys/filio.h]) AC_CHECK_HEADERS([sys/mount.h], [], [], [#include ]) -AC_CHECK_HEADERS([sys/socket.h]) AC_CHECK_HEADERS([sys/statvfs.h]) AC_CHECK_HEADERS([sys/vfs.h]) AC_CHECK_HEADERS([endian.h]) AC_CHECK_HEADERS([execinfo.h]) -AC_CHECK_HEADERS([netinet/in.h]) AC_CHECK_HEADERS([pthread_np.h], [], [], [#include ]) -AC_CHECK_HEADERS([stddef.h]) -AC_CHECK_HEADERS([stdlib.h]) -AC_CHECK_HEADERS([unistd.h]) AC_CHECK_HEADERS([priv.h]) -# Checks for typedefs, structures, and compiler characteristics. -AC_C_CONST -AC_CHECK_MEMBERS([struct sockaddr.sa_len],,,[ -#include -#ifdef HAVE_SYS_SOCKET_H -#include -#endif -]) - # Checks for library functions. -AC_TYPE_SIGNAL -AC_TYPE_SIZE_T -AC_FUNC_VPRINTF -AC_CHECK_FUNCS([strerror]) -AC_FUNC_STRERROR_R AC_CHECK_FUNCS([dladdr]) -AC_CHECK_FUNCS([socket]) AC_CHECK_FUNCS([nanosleep]) AC_CHECK_FUNCS([setppriv]) AC_CHECK_FUNCS([fallocate]) @@ -244,8 +206,6 @@ save_LIBS="${LIBS}" LIBS="${PTHREAD_LIBS}" AC_CHECK_FUNCS([pthread_set_name_np]) AC_CHECK_FUNCS([pthread_setname_np]) -AC_CHECK_FUNCS([pthread_mutex_isowned_np]) -AC_CHECK_FUNCS([pthread_timedjoin_np]) LIBS="${save_LIBS}" # Support for visibility attribute -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Fri Nov 25 14:18:49 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Fri, 25 Nov 2016 15:18:49 +0100 Subject: autocrap de-cargo-culting In-Reply-To: <52150.1480069975@critter.freebsd.dk> References: <52150.1480069975@critter.freebsd.dk> Message-ID: On Fri, Nov 25, 2016 at 11:32 AM, Poul-Henning Kamp wrote: > Does anybody know of any reason not to apply this patch ? When I asked whether I could sweep through the build system, it was partly to get rid of the accumulated cruft. But this being Varnish, I'd take a more conservative approach :) Short answer is, no, you can't safely apply this patch. It depends on what you'd consider safe. Dridi From phk at phk.freebsd.dk Fri Nov 25 14:23:33 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 25 Nov 2016 14:23:33 +0000 Subject: autocrap de-cargo-culting In-Reply-To: References: <52150.1480069975@critter.freebsd.dk> Message-ID: <51605.1480083813@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >> Does anybody know of any reason not to apply this patch ? >Short answer is, no, you can't safely apply this patch. It depends on >what you'd consider safe. The above patch was arrived at by spotting things autocrap examined where we subsequently totally ignore the result of the examination. What's unsafe about that ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Sat Nov 26 10:58:40 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Sat, 26 Nov 2016 11:58:40 +0100 Subject: autocrap de-cargo-culting In-Reply-To: <51605.1480083813@critter.freebsd.dk> References: <52150.1480069975@critter.freebsd.dk> <51605.1480083813@critter.freebsd.dk> Message-ID: On Fri, Nov 25, 2016 at 3:23 PM, Poul-Henning Kamp wrote: > -------- > In message > , Dridi Boukelmoune writes: > >>> Does anybody know of any reason not to apply this patch ? > >>Short answer is, no, you can't safely apply this patch. It depends on >>what you'd consider safe. > > The above patch was arrived at by spotting things autocrap examined > where we subsequently totally ignore the result of the examination. > > What's unsafe about that ? There's a reason why *_LIBS variables are created in such a convoluted way, and it's first on my list of things to clean up so I'll pick this de-cargo-culting bit myself if you don't mind. If you don't save the libs, binaries linking to ${NET_LIBS} will also link to all the previous libs. Some commands like AC_PROG_CC will actually check that the compiler works by compiling a file. It possibly does other things that I don't remember of. If you are running out of autocrap patience I can move that up in my todo list and do the same kind of cleaning that I did for VMODs. That would be on the master branch so that maintainers can yell at me soon enough if I break someone else's machine. Dridi From phk at phk.freebsd.dk Sat Nov 26 12:04:56 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 26 Nov 2016 12:04:56 +0000 Subject: autocrap de-cargo-culting In-Reply-To: References: <52150.1480069975@critter.freebsd.dk> <51605.1480083813@critter.freebsd.dk> Message-ID: <53630.1480161896@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >Some commands like AC_PROG_CC will actually check that the compiler >works by compiling a file. It possibly does other things that I don't >remember of. That's high on my list of things we don't need to worry about and consequently doesn't have to test :-) Feel free to de-cargo-cult as you see fit & when you see fit... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Sat Nov 26 13:40:26 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Sat, 26 Nov 2016 14:40:26 +0100 Subject: autocrap de-cargo-culting In-Reply-To: <53630.1480161896@critter.freebsd.dk> References: <52150.1480069975@critter.freebsd.dk> <51605.1480083813@critter.freebsd.dk> <53630.1480161896@critter.freebsd.dk> Message-ID: > That's high on my list of things we don't need to worry about and > consequently doesn't have to test :-) > > Feel free to de-cargo-cult as you see fit & when you see fit... I gave a quick try to libraries detection [1] and I was wondering why we bother with platform detection. For instance, do we care what platform have libumem or jemalloc today? If we don't care, then detection will fail just fine today and start working [2] as soon as the libraries hypothetically appear on other Varnish-supported platforms. Dridi [1] https://github.com/varnishcache/varnish-cache/commit/d6216cc [2] to some uncertain degree of working From guillaume at varnish-software.com Sun Nov 27 19:35:01 2016 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Sun, 27 Nov 2016 20:35:01 +0100 Subject: range/streaming of incomplete objects In-Reply-To: <80671.1479897517@critter.freebsd.dk> References: <80671.1479897517@critter.freebsd.dk> Message-ID: Typo on page 3: "accessable" Did I miss something or is the "Very Large Value" definition a bit fuzzy? Overall, I'd prefer have the client use an open ended range, possibly adding a header to say that it refuse/accept/requires aggregation. And the server answer would just send 200, 416 or 206 + CL (no aggregation) or 206 + chunked (aggregation). For HEAD requests, allow content range to answer "1000-*/*" to say it allows an open-ended range starting at byte 1000. Am I oversimplifying here? On Nov 23, 2016 11:40, "Poul-Henning Kamp" wrote: > This IETF draft is interesting: > > https://tools.ietf.org/id/draft-pratt-httpbis-rand-access-live-00.txt > > The HTTPbis WG is looking for "go/nogo" input if this is something > which should be adopted. > > Input welcome > > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From slink at schokola.de Tue Nov 29 15:30:44 2016 From: slink at schokola.de (Nils Goroll) Date: Tue, 29 Nov 2016 16:30:44 +0100 Subject: thoughts about vcl cleanup around (be)resp.body and abandon/synth Message-ID: <4df6a46a-b3c4-1ac0-3e40-032409b135bc@schokola.de> Hi, it appeared to me that VCL is cumbersome for handling body replacement on the backend side and the behavior of abandon from the client side is confusing. Current status: - generating synthetic content (setting a body) on the backend side is only possible in vcl_backend_error. To get there, we need to reach the max retries or trigger a fetch error. To tell vcl_backend_error what to do, we need markers in bereq.http.* - return(abandon) fails the backend request and ends up on the client side, but we can't tell the client side what to do Suggestion: - retire synthetic and support (un)set (be)resp.body (not directly related to the topic, but something we should do for consistency) - add support for (un)set beresp.body to vcl_backend_response if used: -> short-term: silently abort the backend connection -> better: discard-read the response body - add support for (un)set resp.body to vcl_deliver - rename vcl_backend_error to vcl_backend_synth to be consistent with the client side - add status and reason to abandon which get pre-set for the call to vcl_synth 503 as default - add return(backend_synth(status, reason)) to vcl_backend_fetch as an easy way to return synthetic content at backend request time. More thoughts: - if we want the (un)set beresp.body, should we maybe have set beresp.body = fetch() also? This would be the default if no other beresp.body action happend in vcl_backend_response Thanks, Nils From dridi at varni.sh Tue Nov 29 16:28:11 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 29 Nov 2016 17:28:11 +0100 Subject: thoughts about vcl cleanup around (be)resp.body and abandon/synth In-Reply-To: <4df6a46a-b3c4-1ac0-3e40-032409b135bc@schokola.de> References: <4df6a46a-b3c4-1ac0-3e40-032409b135bc@schokola.de> Message-ID: > - rename vcl_backend_error to vcl_backend_synth to be consistent > with the client side I'd rather have both subroutines and distinguish between actual fetch errors and artificial backend responses. As a side note, at some point I remember discussing (with at least phk) carrying the SLT_Error message in a bereq.error readable only from v_b_e. > - add status and reason to abandon which get pre-set for the call > to vcl_synth > > 503 as default +1, except for the default but I wouldn't mind it. > - add return(backend_synth(status, reason)) to vcl_backend_fetch > as an easy way to return synthetic content at backend request > time. I don't mind namespacing it, but I'd go for just synth. > More thoughts: > > - if we want the (un)set beresp.body, should we maybe have > > set beresp.body = fetch() > > also? What happens if I set the beresp.body several times with and without fetch()? I actually need to check what happens today if you synthetic()/set *resp.body more than once. > This would be the default if no other beresp.body action happend > in vcl_backend_response I have trouble being convinced by the behavior, despite me being previously in favor of being able to set beresp.body in v_b_r. After two mental context switches it no longer seems like such a good idea to me. Cheers From slink at schokola.de Tue Nov 29 16:41:34 2016 From: slink at schokola.de (Nils Goroll) Date: Tue, 29 Nov 2016 17:41:34 +0100 Subject: thoughts about vcl cleanup around (be)resp.body and abandon/synth In-Reply-To: <4df6a46a-b3c4-1ac0-3e40-032409b135bc@schokola.de> References: <4df6a46a-b3c4-1ac0-3e40-032409b135bc@schokola.de> Message-ID: <8596c470-17c6-a611-976c-24d6ebc3ba4d@schokola.de> On 29/11/16 16:30, Nils Goroll wrote: > - generating synthetic content (setting a body) on the backend side is only > possible in vcl_backend_error. [...] Thank you to fgs for pointing our that people who are not following IRC may find it hard to understand the actual use case I am working on. It's dead simple: Replace bodies for backend responses which should not be sent downstream while keeping the original response headers (or at least some). Nils From fgsch at lodoss.net Tue Nov 29 20:45:39 2016 From: fgsch at lodoss.net (Federico Schwindt) Date: Tue, 29 Nov 2016 20:45:39 +0000 Subject: thoughts about vcl cleanup around (be)resp.body and abandon/synth In-Reply-To: <4df6a46a-b3c4-1ac0-3e40-032409b135bc@schokola.de> References: <4df6a46a-b3c4-1ac0-3e40-032409b135bc@schokola.de> Message-ID: On Tue, Nov 29, 2016 at 3:30 PM, Nils Goroll wrote: > [..] > Suggestion: > > - retire synthetic and support (un)set (be)resp.body > (not directly related to the topic, but something we should do > for consistency) > set (be)resp.body is already supported, but restricted to vcl_backend_error / vcl_synth. I did not retire it to maintain compatibility and because I'm was, and still am, unclear as to what the future plans are wrt bodies. > - add support for (un)set beresp.body to vcl_backend_response > if used: > -> short-term: silently abort the backend connection > -> better: discard-read the response body > I will not object "unset beresp.body" but setting the body in the middle of vcl_backend_response{} feels odd to me. > - add support for (un)set resp.body to vcl_deliver > Is this the same use case but for hits? If so, I'm on the opinion of allowing to unset the body but not generating one, same as above in the vcl_backend_response{} case. > - rename vcl_backend_error to vcl_backend_synth to be consistent > with the client side > I'd object to this. I think the current name is fine and self-explanatory. > - add status and reason to abandon which get pre-set for the call > to vcl_synth > > 503 as default > +1 > - add return(backend_synth(status, reason)) to vcl_backend_fetch > as an easy way to return synthetic content at backend request > time. > I'm all for adding return(error(code, reason)) to go to vcl_backend_error{}. I think this subroutine should not be renamed. > More thoughts: > > - if we want the (un)set beresp.body, should we maybe have > > set beresp.body = fetch() > > also? > I'd prefer to have a way to express the opposite, as in "unset beresp.body". > This would be the default if no other beresp.body action happend > in vcl_backend_response > > Thanks, Nils > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From slink at schokola.de Wed Nov 30 08:44:27 2016 From: slink at schokola.de (Nils Goroll) Date: Wed, 30 Nov 2016 09:44:27 +0100 Subject: thoughts about vcl cleanup around (be)resp.body and abandon/synth In-Reply-To: References: <4df6a46a-b3c4-1ac0-3e40-032409b135bc@schokola.de> Message-ID: On 29/11/16 21:45, Federico Schwindt wrote: > I will not object "unset beresp.body" but setting the body in the middle of > vcl_backend_response{} feels odd to me. This seems to be the item with the least consensus. For the use case of replacing the body while keeping headers, the alternative is - copy response headers to bereq or - use a vmod to save them - resurrect them in v_b_e While this is feasible, I sill don't understand why we would need a de-tour over v_b_e just to create a synthetic body. Why would we - want unset beresp.body; - but not set beresp.body; ? Both need to abort the backend connection or drain the request in progress, the only difference would be if we end up with an empty ws non-empty object. Nils P.S.: Actually this is something which worked in varnish2, where one could just call synthetic in vcl_fetch From dridi at varni.sh Wed Nov 30 09:24:46 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 30 Nov 2016 10:24:46 +0100 Subject: thoughts about vcl cleanup around (be)resp.body and abandon/synth In-Reply-To: References: <4df6a46a-b3c4-1ac0-3e40-032409b135bc@schokola.de> Message-ID: On Wed, Nov 30, 2016 at 9:44 AM, Nils Goroll wrote: > On 29/11/16 21:45, Federico Schwindt wrote: >> I will not object "unset beresp.body" but setting the body in the middle of >> vcl_backend_response{} feels odd to me. > > This seems to be the item with the least consensus. > > For the use case of replacing the body while keeping headers, the alternative is > > - copy response headers to bereq or > - use a vmod to save them I will merely point out that you can't iterate over a headers in VCL, and you'd need a way to distinguish bereq headers from beresp headers if you copy them in one place. > - resurrect them in v_b_e > > While this is feasible, I sill don't understand why we would need a de-tour over > v_b_e just to create a synthetic body. The problem with using v_b_e is that you need to trick Varnish to get there on purpose It would work better like this: - in v_b_r - some_vmod.save_headers(beresp) (with some priv_task-ness) - return synth or backend_synth(XXX) - in a v_b_synth - check status is XXX - some_vmod.restore_headers(beresp) - set beresp.body But as discussed on IRC yesterday after the initial use case was reminded, I finally made up my mind and agreed it would make sense having set beresp.body in v_b_r. I also see a case for having synthetic (even cacheable if needed) backend responses, eg via a vmod. This is doable today, but as I said by tricking Varnish before the fetch and from v_b_r you can't jump directly to v_b_e. Very convoluted for a simple need. > Both need to abort the backend connection or drain the request in progress, the > only difference would be if we end up with an empty ws non-empty object. Ideally drain the response (you meant response?) because the backend may not be a native h1/tcp one. > P.S.: Actually this is something which worked in varnish2, where one could just > call synthetic in vcl_fetch Wut? There was a Varnish before 3.0? From fgsch at lodoss.net Wed Nov 30 09:28:18 2016 From: fgsch at lodoss.net (Federico Schwindt) Date: Wed, 30 Nov 2016 09:28:18 +0000 Subject: thoughts about vcl cleanup around (be)resp.body and abandon/synth In-Reply-To: References: <4df6a46a-b3c4-1ac0-3e40-032409b135bc@schokola.de> Message-ID: On Wed, Nov 30, 2016 at 9:24 AM, Dridi Boukelmoune wrote: > [..] > > While this is feasible, I sill don't understand why we would need a > de-tour over > > v_b_e just to create a synthetic body. > > The problem with using v_b_e is that you need to trick Varnish to get > there on purpose > I think we all agree that there should be a way to go to vcl_backend_error from the other 2 backend subroutines. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Wed Nov 30 09:34:58 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 30 Nov 2016 10:34:58 +0100 Subject: thoughts about vcl cleanup around (be)resp.body and abandon/synth In-Reply-To: References: <4df6a46a-b3c4-1ac0-3e40-032409b135bc@schokola.de> Message-ID: > I think we all agree that there should be a way to go to vcl_backend_error > from the other 2 backend subroutines. For completeness, I did not agree to that yesterday because I'd rather keep error handling separate from artificial responses. I don't have fond memories of vcl_error days.