From fgsch at lodoss.net Mon May 4 19:25:29 2015 From: fgsch at lodoss.net (Federico Schwindt) Date: Mon, 4 May 2015 20:25:29 +0100 Subject: PATCH Message-ID: The attached patch fixes #1724 and #1730. A test is included for the latter. I will commit them separately providing they are OK. Yes / No / Maybe ? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: r01724+r01730.patch Type: text/x-patch Size: 1205 bytes Desc: not available URL: From phk at phk.freebsd.dk Mon May 4 19:34:22 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 04 May 2015 19:34:22 +0000 Subject: PATCH In-Reply-To: References: Message-ID: <77566.1430768062@critter.freebsd.dk> -------- In message , Federico Schwindt writes: Looks good, commit it. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From anand at rediff-inc.com Thu May 7 06:31:38 2015 From: anand at rediff-inc.com (Anand Shah) Date: 7 May 2015 06:31:38 -0000 Subject: =?utf-8?B?dmFybmlzaCBjb250ZW50IGxlbmd0aCByZXBsYWNlZCA=?= Message-ID: <20150507063138.27163.qmail@pro-236-32.rediffmailpro.com> Hello,Using Varnish 4, I have a set of backends that're responding with a valid Content-Length header and no Transfer-Encoding header. On the first hit from a client, rather than responding to the client with those headers, Varnish is dropping the Content-Length header and adding Transfer-Encoding: chunked to the response. This is especially happening for fonts. The same data when fetched from apache and nginx works like a charm and continue to be problematic when used varnish in front. I've tried a number of semi-obvious things like below but all in vain:beresp.do_stream = trueberesp.do_gzip = falseunset req.http.Accept-EncodingSample varnishlog <backend> for your reference. *   << BeReq    >> 247282680-   Begin          bereq 247282679 fetch-   Timestamp      Start: 1430979942.850688 0.000000 0.000000-   BereqMethod    GET-   BereqURL       /ajaxprism/js_jq_1_0/css/fonts/roboto-regular-webfont.woff-   BereqProtocol  HTTP/1.1-   BereqHeader    Accept: application/font-woff;q=0.9,*/*;q=0.8-   BereqHeader    Accept-Language: en-US,en;q=0.5-   BereqHeader    Origin: http://xxx.domain.com-   BereqHeader    Pragma: no-cache-   BereqHeader    Referer: http://xxx.domain.com/ajaxprism/js_jq_1_0/css/style_2.css-   BereqHeader    User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:37.0) Gecko/20100101 Firefox/37.0-   BereqHeader    X-Forwarded-For: ::1-   BereqHeader    host: xxx.domain.com-   BereqHeader    Accept-Encoding: gzip-   BereqHeader    X-Varnish: 247282680-   VCL_call       BACKEND_FETCH-   VCL_return     fetch-   BackendClose   24 jdelivery(xxx.xxx.xxx.xxx,,80) toolate-   BackendOpen    24 jdelivery(xxx.xxx.xxx.xxx,,80) xxx.xxx.xxx.xxx 35680-   Backend        24 jdelivery jdelivery(xxx.xxx.xxx.xxx,,80)-   Timestamp      Bereq: 1430979942.852142 0.001454 0.001454-   Timestamp      Beresp: 1430979942.855756 0.005068 0.003614-   BerespProtocol HTTP/1.1-   BerespStatus   200-   BerespReason   OK-   BerespHeader   Date: Thu, 07 May 2015 06:25:43 GMT-   BerespHeader   Server: Apache-   BerespHeader   P3P: CP="ALL DSP COR LAW CUR DEVi TAI PSAi PSD IVA IVD CONo HIS TELo OUR DEL SAM BUS LOC" policyref="http://xxx.domain.com/w3c/p3p.xml"-   BerespHeader   Last-Modified: Wed, 06 May 2015 06:54:03 GMT-   BerespHeader   Accept-Ranges: bytes-   BerespHeader   Content-Length: 25008-   BerespHeader   Content-Type: application/font-woff-   TTL            RFC 120 -1 -1 1430979943 1430979943 1430979943 0 0-   VCL_call       BACKEND_RESPONSE-   TTL            VCL 120 604800 0 1430979943-   VCL_return     deliver-   Storage        file s0-   ObjProtocol    HTTP/1.1-   ObjStatus      200-   ObjReason      OK-   ObjHeader      Date: Thu, 07 May 2015 06:25:43 GMT-   ObjHeader      Server: Apache-   ObjHeader      P3P: CP="ALL DSP COR LAW CUR DEVi TAI PSAi PSD IVA IVD CONo HIS TELo OUR DEL SAM BUS LOC" policyref="http://xxx.domain.com/w3c/p3p.xml"-   ObjHeader      Last-Modified: Wed, 06 May 2015 06:54:03 GMT-   ObjHeader      Content-Type: application/font-woff-   Fetch_Body     3 length stream-   BackendReuse   24 jdelivery(xxx.xxx.xxx.xxx,,80)-   Timestamp      BerespBody: 1430979942.859386 0.008698 0.003630-   Length         25008-   BereqAcct      449 0 449 337 25008 25345-   EndRegards,Anand -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Thu May 14 14:15:30 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 14 May 2015 16:15:30 +0200 Subject: [master] d9c1916 Add a per-method specific pointer to the vrt-context, and use it to slim down req/busyobj a tiny bit. In-Reply-To: References: Message-ID: Hi, Please find attached a patch to amend this commit and replace the void pointer with a union type instead. If you don't like the idea of the union type, there's a typo you can fix with: sed -i s/SHA256ctx/SHA256Context/ include/vrt.h Cheers, Dridi On Thu, May 14, 2015 at 11:43 AM, Poul-Henning Kamp wrote: > > commit d9c19168e544747d0f8d08499a484a883231318c > Author: Poul-Henning Kamp > Date: Thu May 14 09:42:46 2015 +0000 > > Add a per-method specific pointer to the vrt-context, and use > it to slim down req/busyobj a tiny bit. -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Use-a-union-instead-of-void-for-VRT-specific-args.patch Type: text/x-patch Size: 11973 bytes Desc: not available URL: From phk at phk.freebsd.dk Thu May 14 15:12:21 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 14 May 2015 15:12:21 +0000 Subject: [master] d9c1916 Add a per-method specific pointer to the vrt-context, and use it to slim down req/busyobj a tiny bit. In-Reply-To: References: Message-ID: <48424.1431616341@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >If you don't like the idea of the union type, there's a typo you can fix with: I do indeed not like unions. I use them only when they are absolutely necessary to share a memory allocation is a quasi-type-safe way, where the chances of f**kups are well handled in other ways. I never use them to pretend I have type-safety on a single pointer. > sed -i s/SHA256ctx/SHA256Context/ include/vrt.h done. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Thu May 14 15:24:30 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 14 May 2015 17:24:30 +0200 Subject: [master] d9c1916 Add a per-method specific pointer to the vrt-context, and use it to slim down req/busyobj a tiny bit. In-Reply-To: <48424.1431616341@critter.freebsd.dk> References: <48424.1431616341@critter.freebsd.dk> Message-ID: On Thu, May 14, 2015 at 5:12 PM, Poul-Henning Kamp wrote: > -------- > In message , Dridi Boukelmoune writes: > >>If you don't like the idea of the union type, there's a typo you can fix with: > > I do indeed not like unions. > > I use them only when they are absolutely necessary to share a memory allocation > is a quasi-type-safe way, where the chances of f**kups are well handled in other > ways. > > I never use them to pretend I have type-safety on a single pointer. It was never about safety, but readability. With many NULLs in the code, I need to go other places to fully understand a piece of code. Named argument make sometimes things more explicit: - VCL_recv_method(req->vcl, wrk, req, NULL, NULL); + VCL_recv_method(req->vcl, wrk, req, NULL, no_specific); I also just assumed switching to a union type would not add overhead in the compiled binary. Cheers, Dridi >> sed -i s/SHA256ctx/SHA256Context/ include/vrt.h > > done. > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Thu May 14 15:31:09 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 14 May 2015 15:31:09 +0000 Subject: [master] d9c1916 Add a per-method specific pointer to the vrt-context, and use it to slim down req/busyobj a tiny bit. In-Reply-To: References: <48424.1431616341@critter.freebsd.dk> Message-ID: <48586.1431617469@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >> I never use them to pretend I have type-safety on a single pointer. > >It was never about safety, but readability. I don't see how it would change readability for the better ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From slink at schokola.de Mon May 18 10:13:43 2015 From: slink at schokola.de (Nils Goroll) Date: Mon, 18 May 2015 12:13:43 +0200 Subject: [PATCH] The default stack should have room for the 32k jit requires Message-ID: <5559BB57.3070805@schokola.de> As discussed here http://comments.gmane.org/gmane.comp.web.varnish.dev/2400 and on irc today, I am unhappy about the decision to use JIT unconditionally, but if we do, we should at least provide more than 16k stack for the rest of VRT. --- bin/varnishd/mgt/mgt_main.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c index 533ec80..f645245 100644 --- a/bin/varnishd/mgt/mgt_main.c +++ b/bin/varnishd/mgt/mgt_main.c @@ -382,7 +382,13 @@ init_params(struct cli *cli) bprintf(stackmin, "%jd", (intmax_t)low); MCF_SetMinimum("thread_pool_stack", stackmin); +#if defined(USE_PCRE_JIT) && PCRE_MAJOR == 8 && PCRE_MINOR >= 32 + // jit always needs 32k on the stack + def = (48 + 32) * 1024; +#else + // recursive pcre may need less stack but may also trigger a stack overflow def = 48 * 1024; +#endif if (def < low) def = low; bprintf(stackdef, "%jd", (intmax_t)def); -- 2.1.4 From slink at schokola.de Mon May 18 10:29:24 2015 From: slink at schokola.de (Nils Goroll) Date: Mon, 18 May 2015 12:29:24 +0200 Subject: solve the pcre stack overflow #1576 - was: stack vs workspace for pcre and others In-Reply-To: <54AFCB4C.7090401@schokola.de> References: <5405FA81.7050808@schokola.de> <54060625.5060404@schokola.de> <54061983.9040301@schokola.de> <54061BDD.2000104@schokola.de> <54AFCB4C.7090401@schokola.de> Message-ID: <5559BF04.4010500@schokola.de> To conclude this discussion: phk has decided to turn on JIT again unconditionally if compiled against PCRE versions which we believe is safe (see ce89ba7fbdbeb684789db7c2834bdbff4856ea7f and others). As I have explained in previous emails in this thread, I don't agree that this is a good idea. We had already enabled jit by default in the past only to disable it again by default later. Should we ever see JIT issues again, users would need to either recompile varnish or rewrite their regexen to use a vmod like https://code.uplex.de/uplex-varnish/libvmod-re Also for binary distributions I fear that runtime linking against a different pcre version than that used at compile time could be an issue. What I would like to see is - the option to disable jit at runtime (vcl reload) and per re at best - stack boundary checking for recursive (non-jit) pcre For the latter, phk hat added stack boundary info to struct wrk, but today he stated on irc that he does not want to see it used in core code because the check depends on the assumption that the stack is continuous and grows downwards and that there could be platforms for which it does not hold (I am not aware of any platforms for which the assumption wouldn't hold which varnish supports currently). On this basis I don't see what I can do about this topic and will hopefully find the time to look after adding the jit choice and stack checks to libvmod-re. Nils From phk at phk.freebsd.dk Mon May 18 10:48:49 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 18 May 2015 10:48:49 +0000 Subject: solve the pcre stack overflow #1576 - was: stack vs workspace for pcre and others In-Reply-To: <5559BF04.4010500@schokola.de> References: <5405FA81.7050808@schokola.de> <54060625.5060404@schokola.de> <54061983.9040301@schokola.de> <54061BDD.2000104@schokola.de> <54AFCB4C.7090401@schokola.de> <5559BF04.4010500@schokola.de> Message-ID: <52767.1431946129@critter.freebsd.dk> -------- In message <5559BF04.4010500 at schokola.de>, Nils Goroll writes: >To conclude this discussion: For the record: I primarily consider this to be a PCRE problem. Any library which gives rise to potential infinite on-stack recursion, depending on the data input, is not a particularly good library in my view. As far as I can tell, PCRE is aware of this problem, but are facing CS-theoretical constraints on possible solutions (as in: "which part of recursive didn't you understand ?") When we talked about this last time, the assumption was that JIT was not a solution for this. Since then people who are more clued in these details than me told us that JIT is indeed the solution now, and we have enabled it, provided PCRE version is know to be good. I am quite confident that the PCRE people will want JIT to work, so I am not too worried about it failing mysteriously later. So the remining issue is people using a particular subclass of regexps, on systems with old PCRE's or architectures where PCRE does not support JIT. This does not construe "a big problem" in my view. (So far all I have been shown that this subclass of regexps are *convenient*, nobody has shown that they are *indispensable*.) The other side of this debate is the practice of trying to second guess the compiler and runtime environments behaviour with respect to stack use. The C-standards have this to say about it: Don't even think about it. Compilers can do all sorts of weird shit to stacks, and some hardware architectures have really strange stacks. Stacks can run up or down, they can be discontinuous, the can behave in very unpredicatable ways. For instance an arch with register banks can show plenty of stack available, but if you call a function which needs 16 bytes total on the stack you fail, because that exact call spills a register bank of several thousand bytes onto the stack. (The fact that you just called the very same function a few lines earlier and it succeeded is because the compiler chose to inline the function there.) We need *really* good arguments before we overcome that stigma, and in my judgement "a subclass of regexps on non-mainstream architectures or outdated installations" is not enough to overcome that barrier. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From slink at schokola.de Mon May 18 11:10:58 2015 From: slink at schokola.de (Nils Goroll) Date: Mon, 18 May 2015 13:10:58 +0200 Subject: solve the pcre stack overflow #1576 - was: stack vs workspace for pcre and others In-Reply-To: <52767.1431946129@critter.freebsd.dk> References: <5405FA81.7050808@schokola.de> <54060625.5060404@schokola.de> <54061983.9040301@schokola.de> <54061BDD.2000104@schokola.de> <54AFCB4C.7090401@schokola.de> <5559BF04.4010500@schokola.de> <52767.1431946129@critter.freebsd.dk> Message-ID: <5559C8C2.5070503@schokola.de> > "which part of recursive didn't you understand ?" pcre allows to limit the recursion depth. iirc, the original plan was to determine the available space and calculate a safe upper bound for the recursion depth. The remaining issue with JIT, besides bugs, is that the 32k jit stack allocatited from the machine stack may not suffice. See CONTROLLING THE JIT STACK in pcrejit(3) From slink at schokola.de Mon May 18 11:15:45 2015 From: slink at schokola.de (Nils Goroll) Date: Mon, 18 May 2015 13:15:45 +0200 Subject: solve the pcre stack overflow #1576 - was: stack vs workspace for pcre and others In-Reply-To: <5559C8C2.5070503@schokola.de> References: <5405FA81.7050808@schokola.de> <54060625.5060404@schokola.de> <54061983.9040301@schokola.de> <54061BDD.2000104@schokola.de> <54AFCB4C.7090401@schokola.de> <5559BF04.4010500@schokola.de> <52767.1431946129@critter.freebsd.dk> <5559C8C2.5070503@schokola.de> Message-ID: <5559C9E1.8040003@schokola.de> plus also with jit we have no idea if we DO have the 32k available it needs. VCL/vmods could have used any amount of stack when we get around to calling pcre From slink at schokola.de Mon May 18 11:33:00 2015 From: slink at schokola.de (Nils Goroll) Date: Mon, 18 May 2015 13:33:00 +0200 Subject: return(refresh) from vcl_hit / Re: [PATCH] Add "hash_always_expire" option to treat any existing objects as expired In-Reply-To: <5500D746.4070707@epochtimes.com> References: <5500D746.4070707@epochtimes.com> Message-ID: <5559CDEC.6020102@schokola.de> Hi Jingyi, this has been discussed on irc today: You have come up with a useful idea to allow refresh of a cache object despite a cache hit, which we want to implement. We'd prefer a different implementation though and will probably look in the direction of a return(refresh) from vcl_hit. Cheers, Nils From dridi at varni.sh Mon May 18 11:42:13 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 18 May 2015 13:42:13 +0200 Subject: return(refresh) from vcl_hit / Re: [PATCH] Add "hash_always_expire" option to treat any existing objects as expired In-Reply-To: <5559CDEC.6020102@schokola.de> References: <5500D746.4070707@epochtimes.com> <5559CDEC.6020102@schokola.de> Message-ID: Hi Nils, It's actually return(fetch) that will do that from vcl_hit. You can see it in the built-in VCL[1] and I'm afraid it hasn't been documented yet in the 4.0[2] branch, ironically it is documented but deprecated in the trunk[3]. Cheers, Dridi [1] /usr/share/doc/varnish/builtin.vcl [2] https://www.varnish-cache.org/docs/4.0/users-guide/vcl-built-in-subs.html#vcl-hit [3] https://www.varnish-cache.org/docs/trunk/users-guide/vcl-built-in-subs.html#vcl-hit On Mon, May 18, 2015 at 1:33 PM, Nils Goroll wrote: > Hi Jingyi, > > this has been discussed on irc today: > > You have come up with a useful idea to allow refresh of a cache object despite a > cache hit, which we want to implement. We'd prefer a different implementation > though and will probably look in the direction of a return(refresh) from vcl_hit. > > Cheers, Nils > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev From dridi at varni.sh Mon May 18 12:02:05 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 18 May 2015 14:02:05 +0200 Subject: return(refresh) from vcl_hit / Re: [PATCH] Add "hash_always_expire" option to treat any existing objects as expired In-Reply-To: References: <5500D746.4070707@epochtimes.com> <5559CDEC.6020102@schokola.de> Message-ID: Hi Nils, Please ignore my remarks, except that 4.0 docs are incomplete. I misread everything including the documentation, your email and the discussions on IRC. I will document the return(fetch) in 4.0 though. Cheers, Dridi On Mon, May 18, 2015 at 1:42 PM, Dridi Boukelmoune wrote: > Hi Nils, > > It's actually return(fetch) that will do that from vcl_hit. You can > see it in the built-in VCL[1] and I'm afraid it hasn't been documented > yet in the 4.0[2] branch, ironically it is documented but deprecated > in the trunk[3]. > > Cheers, > Dridi > > [1] /usr/share/doc/varnish/builtin.vcl > [2] https://www.varnish-cache.org/docs/4.0/users-guide/vcl-built-in-subs.html#vcl-hit > [3] https://www.varnish-cache.org/docs/trunk/users-guide/vcl-built-in-subs.html#vcl-hit > > On Mon, May 18, 2015 at 1:33 PM, Nils Goroll wrote: >> Hi Jingyi, >> >> this has been discussed on irc today: >> >> You have come up with a useful idea to allow refresh of a cache object despite a >> cache hit, which we want to implement. We'd prefer a different implementation >> though and will probably look in the direction of a return(refresh) from vcl_hit. >> >> Cheers, Nils >> >> _______________________________________________ >> varnish-dev mailing list >> varnish-dev at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev From dridi at varni.sh Tue May 19 04:50:20 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 19 May 2015 06:50:20 +0200 Subject: Add locking around &backends operations Message-ID: Hi, As discussed yesterday on IRC, please find attached a really simple patch to add locking in VBE operations. It was surprisingly easy, so I'm rather suspicious. I have also removed CLI assertions in (soon to be) dynamic operations. Best Regards, Dridi -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Add-locking-around-the-backends-list-operations.patch Type: text/x-patch Size: 3036 bytes Desc: not available URL: From phk at phk.freebsd.dk Tue May 19 08:45:03 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 19 May 2015 08:45:03 +0000 Subject: That guy with the dog-sled... Message-ID: <65049.1432025103@critter.freebsd.dk> It seems that guy with the dog-sled crossed my path once more. I recently changed the code so that a waiter would watch idle backend connections to detect if/when the other end closes (in order to reduce reuse failures). After fighting with ticket 1675 for some time, I've come to the conclusion that I've picked a wrong way to do it, so I'll be spending a couple of days to do it a better way. This is a little bit too close to the 4.1-R release than I'm comfortable with, but I see no choice. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From slink at schokola.de Wed May 20 14:05:10 2015 From: slink at schokola.de (Nils Goroll) Date: Wed, 20 May 2015 16:05:10 +0200 Subject: guards for includes ok? Message-ID: <555C9496.2090404@schokola.de> I had asked this on IRC, but now I need to leave, so quickly moving the question here: Does anyone have objections to adding guards (like #ifdef VARINSH_CACHE_CACHE_H_INCLUDED) to all our includes? This would make it easier for vmod authors because compilers wouldn't barf about double includes. Nils From phk at phk.freebsd.dk Wed May 20 14:53:53 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 20 May 2015 14:53:53 +0000 Subject: guards for includes ok? In-Reply-To: <555C9496.2090404@schokola.de> References: <555C9496.2090404@schokola.de> Message-ID: <1425.1432133633@critter.freebsd.dk> -------- In message <555C9496.2090404 at schokola.de>, Nils Goroll writes: >Does anyone have objections to adding guards (like #ifdef >VARINSH_CACHE_CACHE_H_INCLUDED) to all our includes? This would make it easier >for vmod authors because compilers wouldn't barf about double includes. I had actually thought about doing the opposite: Add a "don't include this more than once" guard: #ifdef COMBINER_H_INCLUDED # error "combiner.h included multiple times" #endif #define COMBINER_H_INCLUDED -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Sat May 23 07:33:28 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 23 May 2015 07:33:28 +0000 Subject: solve the pcre stack overflow #1576 - was: stack vs workspace for pcre and others In-Reply-To: <5559C9E1.8040003@schokola.de> References: <5405FA81.7050808@schokola.de> <54060625.5060404@schokola.de> <54061983.9040301@schokola.de> <54061BDD.2000104@schokola.de> <54AFCB4C.7090401@schokola.de> <5559BF04.4010500@schokola.de> <52767.1431946129@critter.freebsd.dk> <5559C8C2.5070503@schokola.de> <5559C9E1.8040003@schokola.de> Message-ID: <31345.1432366408@critter.freebsd.dk> -------- I have done what I belive I can do about PCRE/recursion/stack before the 4.1-R. I looked into making JIT a parameter, but it would require me to either change the VRE API or add a gross out-of-band hack, which I don't want to do at this time. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From slink at schokola.de Sat May 23 09:33:54 2015 From: slink at schokola.de (Nils Goroll) Date: Sat, 23 May 2015 11:33:54 +0200 Subject: solve the pcre stack overflow #1576 - was: stack vs workspace for pcre and others In-Reply-To: <31345.1432366408@critter.freebsd.dk> References: <5405FA81.7050808@schokola.de> <54060625.5060404@schokola.de> <54061983.9040301@schokola.de> <54061BDD.2000104@schokola.de> <54AFCB4C.7090401@schokola.de> <5559BF04.4010500@schokola.de> <52767.1431946129@critter.freebsd.dk> <5559C8C2.5070503@schokola.de> <5559C9E1.8040003@schokola.de> <31345.1432366408@critter.freebsd.dk> Message-ID: <55604982.2030405@schokola.de> On 23/05/15 09:33, Poul-Henning Kamp wrote: > change the VRE API Wouldn't a minor release be a good opportunity to do this? Nils From phk at phk.freebsd.dk Sat May 23 09:39:20 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 23 May 2015 09:39:20 +0000 Subject: solve the pcre stack overflow #1576 - was: stack vs workspace for pcre and others In-Reply-To: <55604982.2030405@schokola.de> References: <5405FA81.7050808@schokola.de> <54060625.5060404@schokola.de> <54061983.9040301@schokola.de> <54061BDD.2000104@schokola.de> <54AFCB4C.7090401@schokola.de> <5559BF04.4010500@schokola.de> <52767.1431946129@critter.freebsd.dk> <5559C8C2.5070503@schokola.de> <5559C9E1.8040003@schokola.de> <31345.1432366408@critter.freebsd.dk> <55604982.2030405@schokola.de> Message-ID: <31845.1432373960@critter.freebsd.dk> -------- In message <55604982.2030405 at schokola.de>, Nils Goroll writes: >On 23/05/15 09:33, Poul-Henning Kamp wrote: >> change the VRE API > >Wouldn't a minor release be a good opportunity to do this? I'd prefer to avoid it. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From geoff at uplex.de Sun May 24 19:43:51 2015 From: geoff at uplex.de (Geoff Simmons) Date: Sun, 24 May 2015 21:43:51 +0200 Subject: [PATCH] std.ip() emits an error to the log and returns the fallback when there is insufficient workspace Message-ID: <556229F7.6090808@uplex.de> Hello all, The ip() function in VMOD std currently fails an assertion when there is insufficient workspace: p = WS_Alloc(); AN(p) With the attached patch, it emits an error message to the log using VCL_Error and returns the fallback when this happens. Best, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-std.ip-emits-an-error-to-the-log-and-returns-the-fal.patch Type: text/x-patch Size: 887 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From dridi at varni.sh Tue May 26 23:06:15 2015 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 27 May 2015 01:06:15 +0200 Subject: Composable directors proposal Message-ID: Hi, TL;DR today you can't write a custom director and reuse an existing selection strategy, and I would like to make it possible. I have been pondering this for quite a while now, and I thought it was time I'd share my idea on the mailing list. As of today, it is not possible to compose directors. The closest thing to composition is simply stacking them. It was great progress from 3.0 to 4.0, but I think it can be pushed one step further. When I talk about composition, I sort of think about the functional programming-ish higher-order-functions way of doing composition. I would like to separate director selection strategies from their creations. One could refactor all the existing built-in directors as a single one: cluster. A vmod writer could then use an enumeration to pick a strategy: new rr = directors.cluster(round_robin); It also means that weight has to become a property of the (added) director. It could be a property on a vcl backend definition, and a method on vmod directors. Or it could also be set to any director with the std (or directors?) module: directors.set_weight(rr.bacvkend(), 2); It also means that a backend/director cannot belong to 2 different directors with different weights. But is that a great loss? So with this single "cluster" director, we could still compose it with different strategies and achieve similar directors stacking to what we can do today: backend www1 { [...] } [...] sub vcl_init { new active = directors.cluster(round_robin); active.add_backend(www1); [...] new passive = directors.cluster(round_robin); [...] new www = directos.cluster(fallback); www.add_backend(active); www.add_backend(passive); } Now what if we name this "cluster" director "static" instead? After all, once created, it doesn't change. What if we had dynamic backends and directors? They could reuse the same strategies. What if we have a DNS director? Or any other director creating backends using whatever discovery/registry tool people use (etcd, zookeeper, eureka, ...). They can go round robin, or even do sticky sessions out of the box. Also, if Varnish knows the relationship between directors, it can maybe prevent cycles like this: import directors; sub vcl_init { new d1 = directors.round_robin(); d1.add_backend(d1.backend()); } For this we'd need to: - move the backend selection code to VRT - rewrite the directors vmod And if we have dynamic backends: - support backend updates in the selection code - write new built-in dns directors Please note that it doesn't mean having to once again break everyone's VCL making use of directors. We could keep the existing objects, acting as aliases/wrappers to the new cluster director (for a while). Comments? Ideas? Best Regards, Dridi From slink at schokola.de Thu May 28 18:10:29 2015 From: slink at schokola.de (Nils Goroll) Date: Thu, 28 May 2015 20:10:29 +0200 Subject: [PATCH] Remove the waiter interface from cache.h, split, waited/waitfor, session: struct waited on ws Message-ID: <55675A15.5090504@schokola.de> this happens when all you want to do is get vmod_example to compile again -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Remove-the-waiter-interface-from-cache.h-split-waite.patch Type: text/x-patch Size: 15457 bytes Desc: not available URL: From phk at phk.freebsd.dk Fri May 29 07:00:11 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 29 May 2015 07:00:11 +0000 Subject: [PATCH] Remove the waiter interface from cache.h, split, waited/waitfor, session: struct waited on ws In-Reply-To: <55675A15.5090504@schokola.de> References: <55675A15.5090504@schokola.de> Message-ID: <23400.1432882811@critter.freebsd.dk> -------- In message <55675A15.5090504 at schokola.de>, Nils Goroll writes: So before I reply in detail, let me do a quick sketch of what the heck I have been hacking the last couple of weeks. I was taken quite by surprise when various people started talking about backends in multiple of 100k, not *backend connections* but *backends*. In a number of places our code doesn't scale at that level, for instance we have a thread per backend for healthchecks and just weeks before I had added an entire waiter per backend-tcp-pool. Multiplied by 100k that simply doesn't scale. So instead I went the other way. Waiters are now a general purpose event-driver for filedescriptors, and we have a single waiter per thread-pool, which is used both for client connections and backend connections. Next step was to add a global "wait on this in any suitable thread-pool" API, for things which do not have pool-affinity[1]. After that I want to change the backend health-polling to be event driven, so that a single global thread kicks the polls off and the waiters finish them using regular worker threads (still thinking this through). In particular I want the tcp connect(2) to be asynchronous because the healthpolls will see into network problems (in order that backend fetches should not). With that background: >* We should not include waiter.h from cache.h as vmods requiring cache.h > should not need the waiter interface (otherwise we'd need to install waiter.h) I think I may have added waiter.h the wrong place in Makefile.am. The waiter interface is/can/will be useful for VMODS, as it allows them to do event-driven processing, so it should not be hidden from them. >* For sessions, temporarily alloc struct waited on the ws - we don't need > the workspace while we are waiting, and we don't need struct waited unless > we are waiting Good idea. >* Add Waiter_Done to invalidate struct waited and change the signature of > Wait_Call / waiter_handle_f to struct waited ** to allow for invalidation > of the struct waited pointer. Not so sure about this one, seems a tad strict to me. (It's obviously a good idea to clear it if/when it has no permanence (ie: on ws), but I fail to see a particular reason to mandate clearing it ? Poul-Henning [1] This is a NUMA thing: The client fd's we have accepted should and in the future undoubtedly will have NUMA pool affinity. Backend connections and even less so backend health poll connections will not, so we need to distribute those over the thread pools[2] in some fair fashion. [2] I've been really tempted to rename to concept of thread pools to "clans", because the NUMA aspect is about so much more than just the threads. Clans would have a (just a) pool of threads, acceptor sockets, waiter, etc. I may do this for 5.0. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From slink at schokola.de Fri May 29 07:22:10 2015 From: slink at schokola.de (Nils Goroll) Date: Fri, 29 May 2015 09:22:10 +0200 Subject: [PATCH] Remove the waiter interface from cache.h, split, waited/waitfor, session: struct waited on ws In-Reply-To: <23400.1432882811@critter.freebsd.dk> References: <55675A15.5090504@schokola.de> <23400.1432882811@critter.freebsd.dk> Message-ID: <556813A2.1000100@schokola.de> On 29/05/15 09:00, Poul-Henning Kamp wrote: > So before I reply in detail, let me do a quick sketch of what the heck > I have been hacking the last couple of weeks. Thank you, this is really helpful. Some of this I inferred from what you were doing, but it's really good to know goal and the perspective. For all others: We have discussed the rest: - waiter interface should be exposed to vmods - I will push the patch to put the session struct waited onto the ws (later) - Waiter_Done is overdone but we should clear the magic Nils