From martin at varnish-software.com Thu Oct 6 08:21:11 2011 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Thu, 6 Oct 2011 10:21:11 +0200 Subject: Streaming development In-Reply-To: <4E8B2FF6.2020509@smartjog.com> References: <4E8B2FF6.2020509@smartjog.com> Message-ID: Hi Thomas, Range requests and streaming are right now mutually exclusive features. If a range request comes in, it will loose it's do_stream flag. I am working on enabling the range support on streaming requests, and for the streaming case this means the clients will receive the range bytes at the time they become available on the stream (backend request still asks for the whole object). I hope to have something finished for this within a couple of weeks. Allowing range requests against the backend and populating the cache from those bits is not something we are working on right now. It would then require ETAGs matching to guard against the backend object being altered over the life time of the cached object, and will require some heavy thinking to get right. We might look into doing this some time, but it is not in the plans right now. Regards, Martin Blix Grydeland On Tue, Oct 4, 2011 at 18:10, Thomas SOUVIGNET < thomas.souvignet at smartjog.com> wrote: > Hi, > > Your work on the streaming features for varnish is great but I have a > little problem using it for range caching. Is it possible for HTTP range > requests to use the cache while the file was entirely cached ? How about > when it is being retrieved and cached at the same time ? Is it possible to > populate the cache just with range requests (ie. a file goes from 0 to 500, > one client asks for 0-200, another one for 200-400 and another one for > 400-500 and each request will be used to have the file entirely or partially > cached while the requests are being served) ? > > Thanks for your work. And thanks in advance for your answer. > > -- > Thomas SOUVIGNET, R&E Engineer > > SmartJog SAS - http://www.smartjog.com - A TDF Group Company > Office: 27, blvd Hippolyte Marques 94200 Ivry-sur-Seine - France EU > Phone: +33 (0)1 5868 6207 > > -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.souvignet at smartjog.com Tue Oct 11 09:10:51 2011 From: thomas.souvignet at smartjog.com (Thomas SOUVIGNET) Date: Tue, 11 Oct 2011 11:10:51 +0200 Subject: Streaming development In-Reply-To: References: <4E8B2FF6.2020509@smartjog.com> Message-ID: <4E94081B.9020401@smartjog.com> On 10/06/2011 10:21 AM, Martin Blix Grydeland wrote: > Hi Thomas, > > Range requests and streaming are right now mutually exclusive > features. If a range request comes in, it will loose it's do_stream > flag. I am working on enabling the range support on streaming > requests, and for the streaming case this means the clients will > receive the range bytes at the time they become available on the > stream (backend request still asks for the whole object). I hope to > have something finished for this within a couple of weeks. > > Allowing range requests against the backend and populating the cache > from those bits is not something we are working on right now. It would > then require ETAGs matching to guard against the backend object being > altered over the life time of the cached object, and will require some > heavy thinking to get right. We might look into doing this some time, > but it is not in the plans right now. > > Regards, > Hi Martin, We ran our tests on your streaming branch and so far everything is going really fine. For now your solution runs far faster than any other free solution we tested (squid, nginx and traffic server). The only missing thing for us is a working HTTP Range caching. Do you have an estimation about when you can provide a patch for this ? My company is very interested with this feature and with Varnish in general and we should have soon some dedicated time and developers to work on it so we can provide patches for the streaming part of Varnish. If you are interested in us helping you for the range caching, tell us so we can coordinate our efforts to have this working. Thanks for your work. Regards. -- Thomas SOUVIGNET, R&E Engineer SmartJog SAS - http://www.smartjog.com - A TDF Group Company Office: 27, blvd Hippolyte Marques 94200 Ivry-sur-Seine - France EU Phone: +33 (0)1 5868 6207 a From slink at schokola.de Tue Oct 11 12:26:03 2011 From: slink at schokola.de (Nils Goroll) Date: Tue, 11 Oct 2011 14:26:03 +0200 Subject: [PATCH] vmod_digest In-Reply-To: <4A029B1A60B8E340A50D654D2F130DAA2FE8FE41BF@EXCV001.encara.local.ads> References: <4D42F382.5050402@schokola.de> <4E4D1C2C.3000005@schokola.de> <4A029B1A60B8E340A50D654D2F130DAA2FE8FE41BF@EXCV001.encara.local.ads> Message-ID: <4E9435DB.2060000@schokola.de> Hi, > Do you plan to release a newer version soon or do I start with what you posted and try to continue ? Tollef had the same intention. Any news, Tollef? Other than that, yes, this is quite high up on my priority list, but not on top, unfortunately. Nils From kristian at varnish-software.com Tue Oct 11 12:39:03 2011 From: kristian at varnish-software.com (Kristian Lyngstol) Date: Tue, 11 Oct 2011 14:39:03 +0200 Subject: [PATCH] vmod_digest In-Reply-To: <4E9435DB.2060000@schokola.de> References: <4D42F382.5050402@schokola.de> <4E4D1C2C.3000005@schokola.de> <4A029B1A60B8E340A50D654D2F130DAA2FE8FE41BF@EXCV001.encara.local.ads> <4E9435DB.2060000@schokola.de> Message-ID: <20111011123903.GA7766@freud.kly.no> On Tue, Oct 11, 2011 at 02:26:03PM +0200, Nils Goroll wrote: > Hi, > > > Do you plan to release a newer version soon or do I start with what you posted and try to continue ? > > Tollef had the same intention. Any news, Tollef? We (Varnish Software) released a mhash-based libvmod-digest a few weeks ago, which also supply a few base64-dialects. Feature requests are welcome. It's available from https://github.com/varnish/libvmod-digest - Kristian -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: Digital signature URL: From slink at schokola.de Tue Oct 11 16:16:55 2011 From: slink at schokola.de (Nils Goroll) Date: Tue, 11 Oct 2011 18:16:55 +0200 Subject: fix compiler warnings: [PATCH] solaris sandbox / least privileges overhaul In-Reply-To: <28005.1317389918@critter.freebsd.dk> References: <28005.1317389918@critter.freebsd.dk> Message-ID: <4E946BF7.9060501@schokola.de> Hi phk, > I put the solaris sandbox in its own sourcefile for clarity, yell at me > (and send patches) until it works :-) Here's one. Thanks, Nils -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: 0001-Add-Solaris-specific-setuid-setgid-code-which-got-lo.patch URL: From slink at schokola.de Tue Oct 11 19:21:47 2011 From: slink at schokola.de (Nils Goroll) Date: Tue, 11 Oct 2011 21:21:47 +0200 Subject: [PATCH] synchronize ccf_listen_address() on hack_ready Message-ID: <4E94974B.3010702@schokola.de> An embedded and charset-unspecified text was scrubbed... Name: 0001-synchronize-ccf_listen_address-on-hack_ready.patch URL: From slink at schokola.de Wed Oct 12 18:20:11 2011 From: slink at schokola.de (Nils Goroll) Date: Wed, 12 Oct 2011 20:20:11 +0200 Subject: [PATCH] Error handling for VTCP_blocking in CNT_Session Message-ID: <4E95DA5B.5010808@schokola.de> An embedded and charset-unspecified text was scrubbed... Name: 0001-Error-handling-for-VTCP_blocking-in-CNT_Session.patch URL: From slink at schokola.de Wed Oct 12 18:21:01 2011 From: slink at schokola.de (Nils Goroll) Date: Wed, 12 Oct 2011 20:21:01 +0200 Subject: [PATCH] Avoid panicking if setsockopt(SO_RCVTIMEO) fails with EINVAL on Solaris Message-ID: <4E95DA8D.4000803@schokola.de> -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: 0002-Avoid-panicking-if-setsockopt-SO_RCVTIMEO-fails-wit.patch URL: From slink at schokola.de Wed Oct 12 18:29:37 2011 From: slink at schokola.de (Nils Goroll) Date: Wed, 12 Oct 2011 20:29:37 +0200 Subject: [PATCH] (Updated) Solaris: Test for SO_{RCV,SND}TIMEO needs NET_LIBS In-Reply-To: References: <4D7A50D6.4040601@schokola.de> Message-ID: <4E95DC91.3040606@schokola.de> On 03/13/11 12:58 AM, Theo Schlossnagle wrote: > Do you have evidence that they work? both setsockopt and the > SO_RCVTIMEO define have existed for 10+ years, but anytime setsockopt > is called with that, it will simply fail. Meanwhile, I am using this in production and, yes, AFAIK, the socket timeouts now _do_ work on Solaris. phk, could you please include the patch? Thanks, Nils -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: 0001-Solaris-Test-for-SO_-RCV-SND-TIMEO-needs-NET_LIBS.patch URL: From slink at schokola.de Wed Oct 12 19:24:22 2011 From: slink at schokola.de (Nils Goroll) Date: Wed, 12 Oct 2011 21:24:22 +0200 Subject: [PATCH] solaris sandbox setpprivs order (again) In-Reply-To: <4E845A0F.7040809@schokola.de> References: <4E845A0F.7040809@schokola.de> Message-ID: <4E95E966.40606@schokola.de> (sorry) -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: 0001-re-order-setpprivs-again-to-avoid-SNOCD.patch URL: From valyala at gmail.com Wed Oct 12 23:56:26 2011 From: valyala at gmail.com (Aliaksandr Valialkin) Date: Thu, 13 Oct 2011 02:56:26 +0300 Subject: [PATCH] Binheap performance improvements Message-ID: Hi there. Could you look into this pull request: https://github.com/varnish/Varnish-Cache/pull/1 ? I attached patches from the pull request. The first two are essential, while the last two are just small cleanups. I could merge them with the second patch, but didn't do this, since the corresponding commits were already published on github and history rewrites are considered bad :) These patches should improve Varnish performance when its cache don't fit RAM. -- Best Regards, Aliaksandr -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Binheap-test-improvements.patch Type: text/x-patch Size: 18797 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-Major-binheap-refactoring.patch Type: text/x-patch Size: 88818 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0003-added-missing-parentheses.patch Type: text/x-patch Size: 1182 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0004-verify-idx-value-for-pointers-from-malloc_list.patch Type: text/x-patch Size: 1433 bytes Desc: not available URL: From tfheen at varnish-software.com Thu Oct 13 11:30:03 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Thu, 13 Oct 2011 13:30:03 +0200 Subject: [PATCH] (Updated) Solaris: Test for SO_{RCV, SND}TIMEO needs NET_LIBS In-Reply-To: <4E95DC91.3040606@schokola.de> (Nils Goroll's message of "Wed, 12 Oct 2011 20:29:37 +0200") References: <4D7A50D6.4040601@schokola.de> <4E95DC91.3040606@schokola.de> Message-ID: <87zkh5qgck.fsf@qurzaw.varnish-software.com> ]] Nils Goroll | On 03/13/11 12:58 AM, Theo Schlossnagle wrote: | > Do you have evidence that they work? both setsockopt and the | > SO_RCVTIMEO define have existed for 10+ years, but anytime setsockopt | > is called with that, it will simply fail. | | Meanwhile, I am using this in production and, yes, AFAIK, the socket timeouts | now _do_ work on Solaris. | | phk, could you please include the patch? Applied -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From valyala at gmail.com Thu Oct 13 12:53:34 2011 From: valyala at gmail.com (Aliaksandr Valialkin) Date: Thu, 13 Oct 2011 15:53:34 +0300 Subject: [PATCH] Binheap performance improvements In-Reply-To: References: <2399CC17-F2B7-41BF-89F2-6D306ECF7468@crucially.net> Message-ID: cc'ing varnish-dev On Thu, Oct 13, 2011 at 3:51 PM, Aliaksandr Valialkin wrote: > On Thu, Oct 13, 2011 at 3:10 AM, Artur Bergman wrote: >> Just so I follow, it helps when pages that are part of the binheap are swapped out? > > Almost correct, except that the current implementation touches > external pages via cmp(key1, key2) and update(index) callbacks while > traversing and updating the binheap. So the total number of pages, > which can be potentially touched by binheap (;et's call this set of > pages 'working set') can become quite big under certain conditions > such as large-size structures, which embed key and index and high > fragmentation of these structures in memory. For the reference, > current size of objcore, which embeds key and index, is 120 bytes. See > my remarks for test results in the comment to the second patch for > more details. > > The patch significantly reduces working set for binheap and makes the > size of the working set more predictable. This provides two benefits: > - Speedup when pages from binheap working set are swapped out. > - Speedup due to reduced probability of swapping out other pages when > swapping in binheap pages. Imagine that these 'other' pages contain > cache data. > >> Artur >> >> On Oct 12, 2011, at 4:56 PM, Aliaksandr Valialkin wrote: >> >>> Hi there. >>> >>> Could you look into this pull request: >>> https://github.com/varnish/Varnish-Cache/pull/1 ? >>> >>> I attached patches from the pull request. The first two are essential, >>> while the last two are just small cleanups. I could merge them with >>> the second patch, but didn't do this, since the corresponding commits >>> were already published on github and history rewrites are considered >>> bad :) >>> These patches should improve Varnish performance when its cache don't fit RAM. >>> >>> -- >>> Best Regards, >>> >>> Aliaksandr >>> <0001-Binheap-test-improvements.patch><0002-Major-binheap-refactoring.patch><0003-added-missing-parentheses.patch><0004-verify-idx-value-for-pointers-from-malloc_list.patch>_______________________________________________ >>> varnish-dev mailing list >>> varnish-dev at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev >> >> > > > > -- > Best Regards, > > Aliaksandr > -- Best Regards, Aliaksandr From l at lrowe.co.uk Thu Oct 13 17:45:54 2011 From: l at lrowe.co.uk (Laurence Rowe) Date: Thu, 13 Oct 2011 18:45:54 +0100 Subject: [PATCH] vmod_digest In-Reply-To: <20111011123903.GA7766@freud.kly.no> References: <4D42F382.5050402@schokola.de> <4E4D1C2C.3000005@schokola.de> <4A029B1A60B8E340A50D654D2F130DAA2FE8FE41BF@EXCV001.encara.local.ads> <4E9435DB.2060000@schokola.de> <20111011123903.GA7766@freud.kly.no> Message-ID: On 11 October 2011 13:39, Kristian Lyngstol wrote: > On Tue, Oct 11, 2011 at 02:26:03PM +0200, Nils Goroll wrote: >> Hi, >> >> > Do you plan to release a newer version soon or do I start with what you posted and try to continue ? >> >> Tollef had the same intention. Any news, Tollef? > > We (Varnish Software) released a mhash-based libvmod-digest a few weeks > ago, which also supply a few base64-dialects. Feature requests are > welcome. > > It's available from https://github.com/varnish/libvmod-digest Cool! How do you handle null bytes in the strings returned by digest.hash* / digest.hmac* / digest.base64*decode ? I was under the impression varnish used null terminated strings internally. Laurence From slink at schokola.de Thu Oct 13 12:56:01 2011 From: slink at schokola.de (Nils Goroll) Date: Thu, 13 Oct 2011 14:56:01 +0200 Subject: [PATCH] With SO_(RCV|SND)TIMEDOUT, we also see ETIMEO for ioctl() Message-ID: <4E96DFE1.4020606@schokola.de> (hm, or should we rather have specific VTCP_Asserts ?) -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: 0001-With-SO_-RCV-SND-TIMEDOUT-we-also-see-ETIMEO-for-io.patch URL: From martin at varnish-software.com Thu Oct 13 21:21:58 2011 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Thu, 13 Oct 2011 23:21:58 +0200 Subject: Proposed patch to fix #1031 Message-ID: <1318540919-24761-1-git-send-email-martin@varnish-software.com> Find attached a proposed patch to fix #1031 Note: The patch is against 3.0, as I wanted to address this before 3.0.2 and unfortunately master was not building for me this evening. Regards, Martin Blix Grydeland From martin at varnish-software.com Thu Oct 13 21:21:59 2011 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Thu, 13 Oct 2011 23:21:59 +0200 Subject: [PATCH] Make http_PutProtocol() and http_PutResponse() never fail, and increase make the synthetic object's workspace be param->http_resp_size In-Reply-To: <1318540919-24761-1-git-send-email-martin@varnish-software.com> References: <1318540919-24761-1-git-send-email-martin@varnish-software.com> Message-ID: <1318540919-24761-2-git-send-email-martin@varnish-software.com> The http_PutProtocol() and http_PutResponse() would in the case of workspace overflow leave the headers as NULL and log a SLT_LostHeader. This would make Varnish assert correctly later when writing to the wire, as these are mandated by HTTP. This patch changes them to set the fields to static strings instead ("HTTP/1.1" and "Lost Response") when failing to write them to the workspace. This leaves enough information to complete the protocol in the case of overflow. The patch also increases the synthetic object's workspace from static 1024 to param->http_resp_size. This leaves more (and configurable) room for manipulating the headers of the synthetic object in vcl_error. Test case by Kristian Fixes: #1031 --- bin/varnishtest/tests/r01031.vtc | 30 ++++++++++++++++++++++++++++++ 1 files changed, 30 insertions(+), 0 deletions(-) create mode 100644 bin/varnishtest/tests/r01031.vtc diff --git a/bin/varnishtest/tests/r01031.vtc b/bin/varnishtest/tests/r01031.vtc new file mode 100644 index 0000000..435f973 --- /dev/null +++ b/bin/varnishtest/tests/r01031.vtc @@ -0,0 +1,30 @@ +varnishtest "Test overflowing the response through sp->err_reason" + +varnish v1 -vcl { + backend blatti { + .host = "127.0.0.1"; + } + + sub vcl_recv { + error 200 "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"; + } + sub vcl_error { + return(deliver); + } +} -start + +client c1 { + txreq -req GET + rxresp + expect resp.status == 200 + expect resp.msg == "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" +} -run + +varnish v1 -cliok "param.set http_resp_size 256" + +client c2 { + txreq -req GET + rxresp + expect resp.status == 200 + expect resp.msg == "Lost Response" +} -run -- 1.7.4.1 From martin at varnish-software.com Thu Oct 13 21:35:12 2011 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Thu, 13 Oct 2011 23:35:12 +0200 Subject: [PATCH] Make http_PutProtocol() and http_PutResponse() never fail, and increase make the synthetic object's workspace be param->http_resp_size Message-ID: <1318541712-25077-2-git-send-email-martin@varnish-software.com> The http_PutProtocol() and http_PutResponse() would in the case of workspace overflow leave the headers as NULL and log a SLT_LostHeader. This would make Varnish assert correctly later when writing to the wire, as these are mandated by HTTP. This patch changes them to set the fields to static strings instead ("HTTP/1.1" and "Lost Response") when failing to write them to the workspace. This leaves enough information to complete the protocol in the case of overflow. The patch also increases the synthetic object's workspace from static 1024 to param->http_resp_size. This leaves more (and configurable) room for manipulating the headers of the synthetic object in vcl_error. Test case by Kristian Fixes: #1031 --- bin/varnishd/cache_center.c | 8 ++++---- bin/varnishd/cache_http.c | 6 ++++++ bin/varnishtest/tests/r01031.vtc | 30 ++++++++++++++++++++++++++++++ 3 files changed, 40 insertions(+), 4 deletions(-) create mode 100644 bin/varnishtest/tests/r01031.vtc diff --git a/bin/varnishd/cache_center.c b/bin/varnishd/cache_center.c index 39a2104..4d94d88 100644 --- a/bin/varnishd/cache_center.c +++ b/bin/varnishd/cache_center.c @@ -436,13 +436,13 @@ cnt_error(struct sess *sp) w = sp->wrk; if (sp->obj == NULL) { HSH_Prealloc(sp); - /* XXX: 1024 is a pure guess */ EXP_Clr(&w->exp); - sp->obj = STV_NewObject(sp, NULL, 1024, &w->exp, - (uint16_t)params->http_max_hdr); + sp->obj = STV_NewObject(sp, NULL, params->http_resp_size, + &w->exp, (uint16_t)params->http_max_hdr); if (sp->obj == NULL) sp->obj = STV_NewObject(sp, TRANSIENT_STORAGE, - 1024, &w->exp, (uint16_t)params->http_max_hdr); + params->http_resp_size , &w->exp, + (uint16_t)params->http_max_hdr); if (sp->obj == NULL) { sp->doclose = "Out of objects"; sp->director = NULL; diff --git a/bin/varnishd/cache_http.c b/bin/varnishd/cache_http.c index f77ab74..5ce1b6a 100644 --- a/bin/varnishd/cache_http.c +++ b/bin/varnishd/cache_http.c @@ -981,6 +981,9 @@ http_PutProtocol(struct worker *w, int fd, const struct http *to, { http_PutField(w, fd, to, HTTP_HDR_PROTO, protocol); + if (to->hd[HTTP_HDR_PROTO].b == NULL) + http_SetH(to, HTTP_HDR_PROTO, "HTTP/1.1"); + Tcheck(to->hd[HTTP_HDR_PROTO]); } void @@ -997,6 +1000,9 @@ http_PutResponse(struct worker *w, int fd, const struct http *to, { http_PutField(w, fd, to, HTTP_HDR_RESPONSE, response); + if (to->hd[HTTP_HDR_RESPONSE].b == NULL) + http_SetH(to, HTTP_HDR_RESPONSE, "Lost Response"); + Tcheck(to->hd[HTTP_HDR_RESPONSE]); } void diff --git a/bin/varnishtest/tests/r01031.vtc b/bin/varnishtest/tests/r01031.vtc new file mode 100644 index 0000000..435f973 --- /dev/null +++ b/bin/varnishtest/tests/r01031.vtc @@ -0,0 +1,30 @@ +varnishtest "Test overflowing the response through sp->err_reason" + +varnish v1 -vcl { + backend blatti { + .host = "127.0.0.1"; + } + + sub vcl_recv { + error 200 "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"; + } + sub vcl_error { + return(deliver); + } +} -start + +client c1 { + txreq -req GET + rxresp + expect resp.status == 200 + expect resp.msg == "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" +} -run + +varnish v1 -cliok "param.set http_resp_size 256" + +client c2 { + txreq -req GET + rxresp + expect resp.status == 200 + expect resp.msg == "Lost Response" +} -run -- 1.7.4.1 From l at lrowe.co.uk Thu Oct 13 22:24:25 2011 From: l at lrowe.co.uk (Laurence Rowe) Date: Thu, 13 Oct 2011 23:24:25 +0100 Subject: [PATCH] vmod_digest In-Reply-To: References: <4D42F382.5050402@schokola.de> <4E4D1C2C.3000005@schokola.de> <4A029B1A60B8E340A50D654D2F130DAA2FE8FE41BF@EXCV001.encara.local.ads> <4E9435DB.2060000@schokola.de> <20111011123903.GA7766@freud.kly.no> Message-ID: On 13 October 2011 18:45, Laurence Rowe wrote: > On 11 October 2011 13:39, Kristian Lyngstol > wrote: >> On Tue, Oct 11, 2011 at 02:26:03PM +0200, Nils Goroll wrote: >>> Hi, >>> >>> > Do you plan to release a newer version soon or do I start with what you posted and try to continue ? >>> >>> Tollef had the same intention. Any news, Tollef? >> >> We (Varnish Software) released a mhash-based libvmod-digest a few weeks >> ago, which also supply a few base64-dialects. Feature requests are >> welcome. >> >> It's available from https://github.com/varnish/libvmod-digest > > Cool! > > How do you handle null bytes in the strings returned by digest.hash* / > digest.hmac* / digest.base64*decode ? I was under the impression > varnish used null terminated strings internally. >From reading the source I see that the hash and hmac functions return hexdigests. Presumably it's assumed any base64 data you would be interested in reading in Varnish would not contain null characters. Laurence From kristian at varnish-software.com Fri Oct 14 08:42:35 2011 From: kristian at varnish-software.com (Kristian Lyngstol) Date: Fri, 14 Oct 2011 10:42:35 +0200 Subject: [PATCH] vmod_digest In-Reply-To: References: <4D42F382.5050402@schokola.de> <4E4D1C2C.3000005@schokola.de> <4A029B1A60B8E340A50D654D2F130DAA2FE8FE41BF@EXCV001.encara.local.ads> <4E9435DB.2060000@schokola.de> <20111011123903.GA7766@freud.kly.no> Message-ID: <20111014084235.GA4766@freud.kly.no> Hi Laurence, On Thu, Oct 13, 2011 at 11:24:25PM +0100, Laurence Rowe wrote: > On 13 October 2011 18:45, Laurence Rowe wrote: > > How do you handle null bytes in the strings returned by digest.hash* / > > digest.hmac* / digest.base64*decode ? I was under the impression > > varnish used null terminated strings internally. > > From reading the source I see that the hash and hmac functions return > hexdigests. Presumably it's assumed any base64 data you would be > interested in reading in Varnish would not contain null characters. A valid point... I could provide base64 functions that accept a length-argument too. However, since there is no simple way to deal with NULL in VCL itself, I'm curious if you have a use case for it? The only real use case I see is if you combine it with other vmods. I can actually see that happening with the nulldata vmod, now that I think about it. It's not entirely unreasonable to have a base64-encoded constant in your VCL, use the digest-vmod to decode it before the nulldata-vmod sticks it in synthetic. Then the problem becomes that you have to inform the caller of the length. That shouldn't be a big deal either, but not quite as elegant as I'd prefer. Hmm.... - Kristian -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: Digital signature URL: From phk at phk.freebsd.dk Fri Oct 14 09:13:24 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 14 Oct 2011 09:13:24 +0000 Subject: [PATCH] Make http_PutProtocol() and http_PutResponse() never fail, and increase make the synthetic object's workspace be param->http_resp_size In-Reply-To: Your message of "Thu, 13 Oct 2011 23:35:12 +0200." <1318541712-25077-2-git-send-email-martin@varnish-software.com> Message-ID: <60017.1318583604@critter.freebsd.dk> Works for me: Commit it. In message <1318541712-25077-2-git-send-email-martin at varnish-software.com>, Mar tin Blix Grydeland writes: >The http_PutProtocol() and http_PutResponse() would in the case of >workspace overflow leave the headers as NULL and log a >SLT_LostHeader. This would make Varnish assert correctly later when >writing to the wire, as these are mandated by HTTP. This patch changes >them to set the fields to static strings instead ("HTTP/1.1" and "Lost >Response") when failing to write them to the workspace. This leaves >enough information to complete the protocol in the case of overflow. > >The patch also increases the synthetic object's workspace from static >1024 to param->http_resp_size. This leaves more (and configurable) >room for manipulating the headers of the synthetic object in >vcl_error. > >Test case by Kristian > >Fixes: #1031 >--- > bin/varnishd/cache_center.c | 8 ++++---- > bin/varnishd/cache_http.c | 6 ++++++ > bin/varnishtest/tests/r01031.vtc | 30 ++++++++++++++++++++++++++++++ > 3 files changed, 40 insertions(+), 4 deletions(-) > create mode 100644 bin/varnishtest/tests/r01031.vtc > >diff --git a/bin/varnishd/cache_center.c b/bin/varnishd/cache_center.c >index 39a2104..4d94d88 100644 >--- a/bin/varnishd/cache_center.c >+++ b/bin/varnishd/cache_center.c >@@ -436,13 +436,13 @@ cnt_error(struct sess *sp) > w = sp->wrk; > if (sp->obj == NULL) { > HSH_Prealloc(sp); >- /* XXX: 1024 is a pure guess */ > EXP_Clr(&w->exp); >- sp->obj = STV_NewObject(sp, NULL, 1024, &w->exp, >- (uint16_t)params->http_max_hdr); >+ sp->obj = STV_NewObject(sp, NULL, params->http_resp_size, >+ &w->exp, (uint16_t)params->http_max_hdr); > if (sp->obj == NULL) > sp->obj = STV_NewObject(sp, TRANSIENT_STORAGE, >- 1024, &w->exp, (uint16_t)params->http_max_hdr); >+ params->http_resp_size , &w->exp, >+ (uint16_t)params->http_max_hdr); > if (sp->obj == NULL) { > sp->doclose = "Out of objects"; > sp->director = NULL; >diff --git a/bin/varnishd/cache_http.c b/bin/varnishd/cache_http.c >index f77ab74..5ce1b6a 100644 >--- a/bin/varnishd/cache_http.c >+++ b/bin/varnishd/cache_http.c >@@ -981,6 +981,9 @@ http_PutProtocol(struct worker *w, int fd, const struct http *to, > { > > http_PutField(w, fd, to, HTTP_HDR_PROTO, protocol); >+ if (to->hd[HTTP_HDR_PROTO].b == NULL) >+ http_SetH(to, HTTP_HDR_PROTO, "HTTP/1.1"); >+ Tcheck(to->hd[HTTP_HDR_PROTO]); > } > > void >@@ -997,6 +1000,9 @@ http_PutResponse(struct worker *w, int fd, const struct http *to, > { > > http_PutField(w, fd, to, HTTP_HDR_RESPONSE, response); >+ if (to->hd[HTTP_HDR_RESPONSE].b == NULL) >+ http_SetH(to, HTTP_HDR_RESPONSE, "Lost Response"); >+ Tcheck(to->hd[HTTP_HDR_RESPONSE]); > } > > void >diff --git a/bin/varnishtest/tests/r01031.vtc b/bin/varnishtest/tests/r01031.vtc >new file mode 100644 >index 0000000..435f973 >--- /dev/null >+++ b/bin/varnishtest/tests/r01031.vtc >@@ -0,0 +1,30 @@ >+varnishtest "Test overflowing the response through sp->err_reason" >+ >+varnish v1 -vcl { >+ backend blatti { >+ .host = "127.0.0.1"; >+ } >+ >+ sub vcl_recv { >+ error 200 "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"; >+ } >+ sub vcl_error { >+ return(deliver); >+ } >+} -start >+ >+client c1 { >+ txreq -req GET >+ rxresp >+ expect resp.status == 200 >+ expect resp.msg == "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa >aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" >+} -run >+ >+varnish v1 -cliok "param.set http_resp_size 256" >+ >+client c2 { >+ txreq -req GET >+ rxresp >+ expect resp.status == 200 >+ expect resp.msg == "Lost Response" >+} -run >-- >1.7.4.1 > > >_______________________________________________ >varnish-dev mailing list >varnish-dev at varnish-cache.org >https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Fri Oct 14 09:24:18 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 14 Oct 2011 09:24:18 +0000 Subject: [PATCH] vmod_digest In-Reply-To: Your message of "Fri, 14 Oct 2011 10:42:35 +0200." <20111014084235.GA4766@freud.kly.no> Message-ID: <67354.1318584258@critter.freebsd.dk> In message <20111014084235.GA4766 at freud.kly.no>, Kristian Lyngstol writes: >Then the problem becomes that you have to inform the caller of the >length. That shouldn't be a big deal either, but not quite as elegant as >I'd prefer. The "txt" typedef we use primarily in cache_http can contain binary data, but we usually tacitly assume that it contains NUL-terminated data. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From l at lrowe.co.uk Fri Oct 14 13:25:52 2011 From: l at lrowe.co.uk (Laurence Rowe) Date: Fri, 14 Oct 2011 14:25:52 +0100 Subject: [PATCH] vmod_digest In-Reply-To: <20111014084235.GA4766@freud.kly.no> References: <4D42F382.5050402@schokola.de> <4E4D1C2C.3000005@schokola.de> <4A029B1A60B8E340A50D654D2F130DAA2FE8FE41BF@EXCV001.encara.local.ads> <4E9435DB.2060000@schokola.de> <20111011123903.GA7766@freud.kly.no> <20111014084235.GA4766@freud.kly.no> Message-ID: On 14 October 2011 09:42, Kristian Lyngstol wrote: > Hi Laurence, > > On Thu, Oct 13, 2011 at 11:24:25PM +0100, Laurence Rowe wrote: >> On 13 October 2011 18:45, Laurence Rowe wrote: >> > How do you handle null bytes in the strings returned by digest.hash* / >> > digest.hmac* / digest.base64*decode ? I was under the impression >> > varnish used null terminated strings internally. >> >> From reading the source I see that the hash and hmac functions return >> hexdigests. Presumably it's assumed any base64 data you would be >> interested in reading in Varnish would not contain null characters. > > A valid point... > > I could provide base64 functions that accept a length-argument too. > However, since there is no simple way to deal with NULL in VCL itself, > I'm curious if you have a use case for it? The only real use case I see > is if you combine it with other vmods. My particular use case is to validate Plone's signed authentication cookies within VCL. To save space I included a binary version of the hmac sha256 digest in the base64 encoded cookie. I think I'll just change Plone's algorithm to use a hexdigest instead, it's only another 45 characters once base64 encoded. With the binary version I would need comparison tests which were also length aware. Laurence From wwwcmj110 at 126.com Mon Oct 31 10:17:42 2011 From: wwwcmj110 at 126.com (=?GBK?B?s8LD9A==?=) Date: Mon, 31 Oct 2011 18:17:42 +0800 (CST) Subject: About chunked encoding of Varnish 3 Message-ID: <1d6f13a.7ae8.133597c0def.Coremail.wwwcmj110@126.com> Hi folks, I'm a newcomer here.I want to know about varnish 3.0.1 chunked encoding function. The Version of varnish 3 add support for gzip?and default support for Transfer-Encoding:chunked. I want to get the content-length of object by header . Transfer-Encoding is set chunked ,so I can't get content-length. I want to know how can I disable Transfer-Encoding in varnish 3 ? ( modify the setup source code? ) Thanks. Jimmy Chen -------------- next part -------------- An HTML attachment was scrubbed... URL: