From tfheen at varnish-software.com Mon May 6 07:18:23 2013 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Mon, 6 May 2013 09:18:23 +0200 Subject: [PATCH] Return an error on duplicated Host headers Message-ID: <1367824703-12987-1-git-send-email-tfheen@varnish-software.com> --- bin/varnishd/cache/cache.h | 1 + bin/varnishd/cache/cache_http.c | 2 +- bin/varnishd/cache/cache_http1_proto.c | 29 +++++++++++++++++++++++++++++ 3 files changed, 31 insertions(+), 1 deletion(-) diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 865b315..8763669 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -877,6 +877,7 @@ double http_GetHdrQ(const struct http *hp, const char *hdr, const char *field); uint16_t http_GetStatus(const struct http *hp); const char *http_GetReq(const struct http *hp); int http_HdrIs(const struct http *hp, const char *hdr, const char *val); +int http_IsHdr(const txt *hh, const char *hdr); enum sess_close http_DoConnection(const struct http *); void http_CopyHome(const struct http *hp); void http_Unset(struct http *hp, const char *hdr); diff --git a/bin/varnishd/cache/cache_http.c b/bin/varnishd/cache/cache_http.c index e9f9e22..1eb429a 100644 --- a/bin/varnishd/cache/cache_http.c +++ b/bin/varnishd/cache/cache_http.c @@ -154,7 +154,7 @@ http_Teardown(struct http *hp) /*--------------------------------------------------------------------*/ -static int +int http_IsHdr(const txt *hh, const char *hdr) { unsigned l; diff --git a/bin/varnishd/cache/cache_http1_proto.c b/bin/varnishd/cache/cache_http1_proto.c index 00ed9bf..6be14d1 100644 --- a/bin/varnishd/cache/cache_http1_proto.c +++ b/bin/varnishd/cache/cache_http1_proto.c @@ -376,6 +376,30 @@ htc_splitline(struct http *hp, const struct http_conn *htc, int req) /*--------------------------------------------------------------------*/ +static int +htc_request_invalid(struct http *hp) +{ + int u; + int seen_host = 0; + for (u = HTTP_HDR_FIRST; u < hp->nhd; u++) { + if (hp->hd[u].b == NULL) + continue; + AN(hp->hd[u].b); + AN(hp->hd[u].e); + if (http_IsHdr(&hp->hd[u], H_Host)) { + if (seen_host) { + VSLb(hp->vsl, SLT_Error, "Duplicated Host header"); + return (400); + } + seen_host = 1; + } + } + return (0); +} + + +/*--------------------------------------------------------------------*/ + static void htc_proto_ver(struct http *hp) { @@ -412,6 +436,11 @@ HTTP1_DissectRequest(struct req *req) } htc_proto_ver(hp); + retval = htc_request_invalid(hp); + if (retval != 0) { + return (retval); + } + /* RFC2616, section 5.2, point 1 */ if (!strncasecmp(hp->hd[HTTP_HDR_URL].b, "http://", 7)) { b = e = hp->hd[HTTP_HDR_URL].b + 7; -- 1.7.10.4 From martin at varnish-software.com Tue May 7 13:14:34 2013 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 7 May 2013 15:14:34 +0200 Subject: Waitinglist counters Message-ID: Reminder, and request for feedback, to implement log statements when requests are put on waitinglist. If those log statements also included how many items are already on that list, this could be interesting data to feed to varnishhist and help identify buggy VCL situations early. Regards, Martin Blix Grydeland -- *Martin Blix Grydeland* Senior Developer | Varnish Software AS Cell: +47 21 98 92 60 We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at varnish-software.com Mon May 13 12:44:57 2013 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 13 May 2013 14:44:57 +0200 Subject: Varnishapi patches Message-ID: Please find attached patch set for the first parts of new libvarnishapi Regards, Martin Blix Grydeland -- *Martin Blix Grydeland* Senior Developer | Varnish Software AS Cell: +47 21 98 92 60 We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: varnishapi.tar.gz Type: application/x-gzip Size: 88543 bytes Desc: not available URL: From perbu at varnish-software.com Wed May 15 10:34:51 2013 From: perbu at varnish-software.com (Per Buer) Date: Wed, 15 May 2013 12:34:51 +0200 Subject: Make reason-phrase optional? Message-ID: In the web-forum I noticed someone ran into some trouble with the probes declaring the backend sick if the backend dropped the reason-phrase (The "OK" in "HTTP/1.1 200 OK"). According to http://tools.ietf.org/html/rfc2616#section-6.1.1 - the reason phrase is meant for human consumption and you're allowed to say whatever you'd like here. The reason phrase is required by the RFC and the question is whether Varnish should require it or not. After all, it isn't used for anything so I don't see a reason why we should require it. Then again, we cannot be faulted for declaring a backend sick when it is in violation of the protocol. -- *Per Buer* Varnish Software AS Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: make-reason-optional-for-probe.patch Type: application/octet-stream Size: 459 bytes Desc: not available URL: From magnus at hagander.net Wed May 15 10:40:17 2013 From: magnus at hagander.net (Magnus Hagander) Date: Wed, 15 May 2013 12:40:17 +0200 Subject: Make reason-phrase optional? In-Reply-To: References: Message-ID: On Wed, May 15, 2013 at 12:34 PM, Per Buer wrote: > In the web-forum I noticed someone ran into some trouble with the probes > declaring the backend sick if the backend dropped the reason-phrase (The > "OK" in "HTTP/1.1 200 OK"). > > According to http://tools.ietf.org/html/rfc2616#section-6.1.1 - the > reason phrase is meant for human consumption and you're allowed to say > whatever you'd like here. The reason phrase is required by the RFC and the > question is whether Varnish should require it or not. After all, it isn't > used for anything so I don't see a reason why we should require it. Then > again, we cannot be faulted for declaring a backend sick when it is in > violation of the protocol. > > Postels law dictates it should be treated as optional, I guess :) And if Varnish doesn't make use of it anywhere, there's no point in rejecting it being missing. But it looks really weird to me to first test for ==2 and then for ==1. Just seems less natural than to first test for ==1 and then ==2 :) -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Wed May 15 10:44:17 2013 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 15 May 2013 10:44:17 +0000 Subject: Make reason-phrase optional? In-Reply-To: References: Message-ID: <99150.1368614657@critter.freebsd.dk> In message , Per Buer writes: >According to http://tools.ietf.org/html/rfc2616#section-6.1.1 - the reason >phrase is meant for human consumption and you're allowed to say whatever >you'd like here. The reason phrase is required by the RFC and the question >is whether Varnish should require it or not. After all, it isn't used for >anything so I don't see a reason why we should require it. Then again, we >cannot be faulted for declaring a backend sick when it is in violation of >the protocol. I have no opinion on this. Find a consensus and send me a patch... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From perbu at varnish-software.com Wed May 15 11:05:16 2013 From: perbu at varnish-software.com (Per Buer) Date: Wed, 15 May 2013 13:05:16 +0200 Subject: Make reason-phrase optional? In-Reply-To: References: Message-ID: On Wed, May 15, 2013 at 12:40 PM, Magnus Hagander wrote: > > But it looks really weird to me to first test for ==2 and then for ==1. > Just seems less natural than to first test for ==1 and then ==2 :) > > "less" threw me into the gawd awful "GNU nano". I blame it. But, yeah, noted. -- *Per Buer* Varnish Software Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From apj at mutt.dk Wed May 15 11:11:03 2013 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Wed, 15 May 2013 13:11:03 +0200 Subject: Make reason-phrase optional? In-Reply-To: References: Message-ID: <20130515111103.GF19694@nerd.dk> On Wed, May 15, 2013 at 12:34:51PM +0200, Per Buer wrote: > > According to http://tools.ietf.org/html/rfc2616#section-6.1.1 - the reason > phrase is meant for human consumption and you're allowed to say whatever > you'd like here. The reason phrase is required by the RFC and the question > is whether Varnish should require it or not. After all, it isn't used for > anything so I don't see a reason why we should require it. Then again, we > cannot be faulted for declaring a backend sick when it is in violation of > the protocol. I think it's ok to ignore it. Because Postel. -- Andreas From jammy.linux at gmail.com Fri May 17 02:57:41 2013 From: jammy.linux at gmail.com (Jammy) Date: Fri, 17 May 2013 10:57:41 +0800 Subject: varnishlog coredump Message-ID: Hi, there, Recently, we're experiencing the coredump issue of varnishlog, below is the debug information I've got. Does anyone meet with the same problem? any thoughts? Thanks. Core was generated by `/usr/bin/varnishlog'. Program terminated with signal 11, Segmentation fault. #0 0x0000003993c09307 in VSL_Open (vd=0x22b7010, diag=1) at vsl.c:396 (gdb) bt #0 0x0000003993c09307 in VSL_Open (vd=0x22b7010, diag=1) at vsl.c:396 #1 0x0000000000401b0d in main () (gdb) p *vd $1 = {magic = 1849415323, diag = 0x3e7544e9c0 , priv = 0x3e7578c860, n_opt = 0x22b7b80 "", fname = 0x22b7ba0 "/var/lib/varnish/ip-10-38-2-4/_.vsm", fstat = { st_dev = 64769, st_ino = 524219, st_nlink = 1, st_mode = 33188, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 84934656, st_blksize = 4096, st_blocks = 165888, st_atim = {tv_sec = 1368723984, tv_nsec = 800301232}, st_mtim = {tv_sec = 1368724011, tv_nsec = 864301673}, st_ctim = {tv_sec = 1368724011, tv_nsec = 864301673}, __unused = {0, 0, 0}}, vsm_fd = 3, VSM_head = 0x7f7feee8a000, vsm_end = 0x7f7ff3f8a000, alloc_seq = 570313140, vsc = 0x0, vsl = 0x22b7100} (gdb) p *vd $2 = {magic = 1849415323, diag = 0x3e7544e9c0 , priv = 0x3e7578c860, n_opt = 0x22b7b80 "", fname = 0x22b7ba0 "/var/lib/varnish/ip-10-38-2-4/_.vsm", fstat = { st_dev = 64769, st_ino = 524219, st_nlink = 1, st_mode = 33188, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 84934656, st_blksize = 4096, st_blocks = 165888, st_atim = {tv_sec = 1368723984, tv_nsec = 800301232}, st_mtim = {tv_sec = 1368724011, tv_nsec = 864301673}, st_ctim = {tv_sec = 1368724011, tv_nsec = 864301673}, __unused = {0, 0, 0}}, vsm_fd = 3, VSM_head = 0x7f7feee8a000, vsm_end = 0x7f7ff3f8a000, alloc_seq = 570313140, vsc = 0x0, vsl = 0x22b7100} (gdb) p *vd->vsl $3 = {magic = 2050087736, log_start = 0x7f7feee9a5e4, log_end = 0x7f7ff3e9a5e4, log_ptr = 0x7f7feee9a5e8, last_seq = 35066, r_fd = -1, rbuflen = 256, rbuf = 0x22b7770, b_opt = 0, c_opt = 0, d_opt = 0, flags = 0, vbm_client = 0x22b71b0, vbm_backend = 0x22b73e0, vbm_select = 0x22b76c0, vbm_supress = 0x22b7610, regflags = 0, regincl = 0x0, regexcl = 0x0, num_matchers = 0, matchers = {vtqh_first = 0x0, vtqh_last = 0x22b7188}, skip = 0, keep = 0} ---------------------------------- Best wishes, Jammy -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi.boukelmoune at zenika.com Fri May 17 19:01:58 2013 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Fri, 17 May 2013 21:01:58 +0200 Subject: Varnishapi patches In-Reply-To: References: Message-ID: Hi, Those patches have been committed in the master if I'm not mistaken. I've read a lot of it but at some point it became way beyond my understanding/knowledge of Varnish internals. Anyway, with the current master (e16233fd332) varnishd dies (SIGHUP) when I run varnishtest in verbose mode. I'm replying here because I remember seeing something about the logs, varnishlog and varnishtest a some point(s), so I thought it might be related. ** v1 0.7 as expected: cache_hit (2) == 2 * top 0.7 RESETTING after src/tests/regfilter01_regfilter_in_vcl_hash.vtc ** s1 0.7 Waiting for server **** s1 0.7 macro undef s1_addr **** s1 0.7 macro undef s1_port **** s1 0.7 macro undef s1_sock **** v1 0.7 macro undef v1_addr **** v1 0.7 macro undef v1_port **** v1 0.7 macro undef v1_sock ** v1 0.7 Stop **** v1 0.7 CLI TX| stop *** v1 0.7 debug| Stopping Child\n **** v1 0.7 vsl| 0 CLI - EOF on CLI connection, worker stops *** v1 1.7 debug| Child (13868) died status=1\n *** v1 1.7 debug| Child (13868) said Child dies\n *** v1 1.7 debug| Child cleanup complete\n *** v1 1.7 CLI RX 200 It happens at the very end of the test after the last assertion. Best Regards, Dridi On Mon, May 13, 2013 at 2:44 PM, Martin Blix Grydeland wrote: > > Please find attached patch set for the first parts of new libvarnishapi > > Regards, > Martin Blix Grydeland > > -- > Martin Blix Grydeland > Senior Developer | Varnish Software AS > Cell: +47 21 98 92 60 > We Make Websites Fly! > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev From flygoast at 126.com Sun May 19 08:59:17 2013 From: flygoast at 126.com (flygoast) Date: Sun, 19 May 2013 16:59:17 +0800 (CST) Subject: vmod memory allocation problem Message-ID: <5b444c58.3020.13ebc03ab90.Coremail.flygoast@126.com> hi, guys, I'm newbie in varnish vmod development. I'am comfusing when I return a char * pointting to a block of memory malloced from heap, Whether resultting in a memory leak? Such as: char * vmod_foo(struct sess *sp, struct vmod_priv *priv) { return strdup("foo"); } Thank you! -------------- next part -------------- An HTML attachment was scrubbed... URL: From jammy.linux at gmail.com Mon May 20 02:25:04 2013 From: jammy.linux at gmail.com (Jammy) Date: Mon, 20 May 2013 10:25:04 +0800 Subject: vmod memory allocation problem In-Reply-To: <5b444c58.3020.13ebc03ab90.Coremail.flygoast@126.com> References: <5b444c58.3020.13ebc03ab90.Coremail.flygoast@126.com> Message-ID: <4A7CB743-3153-4551-B9C1-CCEF5CE7C2AA@gmail.com> Yes, it will result in memory leakage. I recommend to use WS_Reserve to allocate your memory within your VMOD. Best regards On 19 May, 2013, at 4:59 PM, flygoast wrote: > hi, guys, > > I'm newbie in varnish vmod development. > > I'am comfusing when I return a char * pointting to a block of memory malloced from heap, Whether resultting in a memory leak? > > Such as: > > char * > > vmod_foo(struct sess *sp, struct vmod_priv *priv) > > { > > return strdup("foo"); > > } > > Thank you! > > > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev ---------------------------------- Best wishes, Jammy -------------- next part -------------- An HTML attachment was scrubbed... URL: From drwilco at drwilco.net Mon May 20 05:09:22 2013 From: drwilco at drwilco.net (Rogier 'DocWilco' Mulhuijzen) Date: Mon, 20 May 2013 07:09:22 +0200 Subject: vmod memory allocation problem In-Reply-To: <4A7CB743-3153-4551-B9C1-CCEF5CE7C2AA@gmail.com> References: <5b444c58.3020.13ebc03ab90.Coremail.flygoast@126.com> <4A7CB743-3153-4551-B9C1-CCEF5CE7C2AA@gmail.com> Message-ID: Or WS_Alloc if you know how much you need to allocate beforehand. On May 19, 2013 7:26 PM, "Jammy" wrote: > > Yes, it will result in memory leakage. > > I recommend to use WS_Reserve to allocate your memory within your VMOD. > > Best regards > > > On 19 May, 2013, at 4:59 PM, flygoast wrote: > > hi, guys, > > I'm newbie in varnish vmod development. > > I'am comfusing when I return a char * pointting to a block of memory > malloced from heap, Whether resultting in a memory leak? > > Such as: > > char * > > vmod_foo(struct sess *sp, struct vmod_priv *priv) > > { > > return strdup("foo"); > > } > > Thank you! > > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > > > ---------------------------------- > Best wishes, > Jammy > > > > > > > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at varnish-software.com Tue May 21 09:19:40 2013 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 21 May 2013 11:19:40 +0200 Subject: Varnishapi patches In-Reply-To: References: Message-ID: Hi Dridi, Looking at the varnishtest output attached, that looks completely normal to me, and the expected behavior. None of those lines indicate an error condition. When a varnishtest test case ends, it will always stop the varnishd instance before exiting, which is what the log lines you see describe. Please elaborate on what problems you are experiencing, and why you believe this is a problem. Regards, Martin Blix Grydeland On 17 May 2013 21:01, Dridi Boukelmoune wrote: > Hi, > > Those patches have been committed in the master if I'm not mistaken. > I've read a lot of it but at some point it became way beyond my > understanding/knowledge of Varnish internals. Anyway, with the current > master (e16233fd332) varnishd dies (SIGHUP) when I run varnishtest in > verbose mode. > > I'm replying here because I remember seeing something about the logs, > varnishlog and varnishtest a some point(s), so I thought it might be > related. > > ** v1 0.7 as expected: cache_hit (2) == 2 > * top 0.7 RESETTING after > src/tests/regfilter01_regfilter_in_vcl_hash.vtc > ** s1 0.7 Waiting for server > **** s1 0.7 macro undef s1_addr > **** s1 0.7 macro undef s1_port > **** s1 0.7 macro undef s1_sock > **** v1 0.7 macro undef v1_addr > **** v1 0.7 macro undef v1_port > **** v1 0.7 macro undef v1_sock > ** v1 0.7 Stop > **** v1 0.7 CLI TX| stop > *** v1 0.7 debug| Stopping Child\n > **** v1 0.7 vsl| 0 CLI - EOF on CLI > connection, worker stops > *** v1 1.7 debug| Child (13868) died status=1\n > *** v1 1.7 debug| Child (13868) said Child dies\n > *** v1 1.7 debug| Child cleanup complete\n > *** v1 1.7 CLI RX 200 > > It happens at the very end of the test after the last assertion. > > Best Regards, > Dridi > > > On Mon, May 13, 2013 at 2:44 PM, Martin Blix Grydeland > wrote: > > > > Please find attached patch set for the first parts of new libvarnishapi > > > > Regards, > > Martin Blix Grydeland > > > > -- > > Martin Blix Grydeland > > Senior Developer | Varnish Software AS > > Cell: +47 21 98 92 60 > > We Make Websites Fly! > > > > _______________________________________________ > > varnish-dev mailing list > > varnish-dev at varnish-cache.org > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -- *Martin Blix Grydeland* Senior Developer | Varnish Software AS Cell: +47 21 98 92 60 We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi.boukelmoune at zenika.com Tue May 21 09:53:23 2013 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Tue, 21 May 2013 11:53:23 +0200 Subject: Varnishapi patches In-Reply-To: References: Message-ID: Hi, This wasn't actually brought by your patches, I've tried to rewind to the previous commit I was building my module against and had the same problem. I can't test this until tonight, but I'll dig in my memory instead: - varnishtest has become very slow on the master branch, like a second slower per test. I know this is unrelated but it had to be said :p - I'm used to grep "varnishtest -v" output, but I wrote the previous mail after failing to grep and having varnishtest's ouput in stderr instead. So I assumed there was an error (I don't remember myself checking the exit status). I hope this helps. Best Regards, Dridi On Tue, May 21, 2013 at 11:19 AM, Martin Blix Grydeland wrote: > > Hi Dridi, > > Looking at the varnishtest output attached, that looks completely normal to me, and the expected behavior. None of those lines indicate an error condition. When a varnishtest test case ends, it will always stop the varnishd instance before exiting, which is what the log lines you see describe. Please elaborate on what problems you are experiencing, and why you believe this is a problem. > > Regards, > Martin Blix Grydeland > > > On 17 May 2013 21:01, Dridi Boukelmoune wrote: >> >> Hi, >> >> Those patches have been committed in the master if I'm not mistaken. >> I've read a lot of it but at some point it became way beyond my >> understanding/knowledge of Varnish internals. Anyway, with the current >> master (e16233fd332) varnishd dies (SIGHUP) when I run varnishtest in >> verbose mode. >> >> I'm replying here because I remember seeing something about the logs, >> varnishlog and varnishtest a some point(s), so I thought it might be >> related. >> >> ** v1 0.7 as expected: cache_hit (2) == 2 >> * top 0.7 RESETTING after src/tests/regfilter01_regfilter_in_vcl_hash.vtc >> ** s1 0.7 Waiting for server >> **** s1 0.7 macro undef s1_addr >> **** s1 0.7 macro undef s1_port >> **** s1 0.7 macro undef s1_sock >> **** v1 0.7 macro undef v1_addr >> **** v1 0.7 macro undef v1_port >> **** v1 0.7 macro undef v1_sock >> ** v1 0.7 Stop >> **** v1 0.7 CLI TX| stop >> *** v1 0.7 debug| Stopping Child\n >> **** v1 0.7 vsl| 0 CLI - EOF on CLI >> connection, worker stops >> *** v1 1.7 debug| Child (13868) died status=1\n >> *** v1 1.7 debug| Child (13868) said Child dies\n >> *** v1 1.7 debug| Child cleanup complete\n >> *** v1 1.7 CLI RX 200 >> >> It happens at the very end of the test after the last assertion. >> >> Best Regards, >> Dridi >> >> >> On Mon, May 13, 2013 at 2:44 PM, Martin Blix Grydeland >> wrote: >> > >> > Please find attached patch set for the first parts of new libvarnishapi >> > >> > Regards, >> > Martin Blix Grydeland >> > >> > -- >> > Martin Blix Grydeland >> > Senior Developer | Varnish Software AS >> > Cell: +47 21 98 92 60 >> > We Make Websites Fly! >> > >> > _______________________________________________ >> > varnish-dev mailing list >> > varnish-dev at varnish-cache.org >> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > > > > > -- > Martin Blix Grydeland > Senior Developer | Varnish Software AS > Cell: +47 21 98 92 60 > We Make Websites Fly! From mck at apache.org Tue May 21 11:45:39 2013 From: mck at apache.org (Mck) Date: Tue, 21 May 2013 13:45:39 +0200 Subject: Can experimental-ims be updated off master? Message-ID: <1369136739.10101.116.camel@dragonfly> I'd be keen to test the experimental-ims branch. But the last master->experimental-ims merge is quite old. more than twelve months ago. https://github.com/varnish/Varnish-Cache/commits/experimental-ims Is it possible for someone with commits rights to bring the branch up to date against master? it would be greatly appreciated, and we at finn.no can help in (seriously) testing this feature. ~mck -- "I'm not one of those who think Bill Gates is the devil. I simply suspect that if Microsoft ever met up with the devil, it wouldn't need an interpreter." Nicholas Petreley | http://github.com/finn-no | http://tech.finn.no | From dridi.boukelmoune at zenika.com Tue May 21 18:06:32 2013 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Tue, 21 May 2013 20:06:32 +0200 Subject: vmod memory allocation problem In-Reply-To: References: <5b444c58.3020.13ebc03ab90.Coremail.flygoast@126.com> <4A7CB743-3153-4551-B9C1-CCEF5CE7C2AA@gmail.com> Message-ID: Hi, For some reason I've missed your question. I wrote a bit about that, I hope this will help: http://blog.zenika.com/index.php?post/2012/08/21/Creating-a-Varnish-module Cheers, Dridi On Mon, May 20, 2013 at 7:09 AM, Rogier 'DocWilco' Mulhuijzen wrote: > Or WS_Alloc if you know how much you need to allocate beforehand. > > On May 19, 2013 7:26 PM, "Jammy" wrote: >> >> >> Yes, it will result in memory leakage. >> >> I recommend to use WS_Reserve to allocate your memory within your VMOD. >> >> Best regards >> >> >> On 19 May, 2013, at 4:59 PM, flygoast wrote: >> >> hi, guys, >> >> I'm newbie in varnish vmod development. >> >> I'am comfusing when I return a char * pointting to a block of memory >> malloced from heap, Whether resultting in a memory leak? >> >> Such as: >> >> char * >> >> vmod_foo(struct sess *sp, struct vmod_priv *priv) >> >> { >> >> return strdup("foo"); >> >> } >> >> Thank you! >> >> >> >> _______________________________________________ >> varnish-dev mailing list >> varnish-dev at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev >> >> >> ---------------------------------- >> Best wishes, >> Jammy >> >> >> >> >> >> >> >> _______________________________________________ >> varnish-dev mailing list >> varnish-dev at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev From nwjsmith at gmail.com Wed May 22 20:49:09 2013 From: nwjsmith at gmail.com (Nate Smith) Date: Wed, 22 May 2013 16:49:09 -0400 Subject: Per-backend connection stat Message-ID: <3F036238E26F4217B22B7A3E011C3006@gmail.com> Hey there, I've added a per-backend statistic for the number of current connections made to each backend, which I've missed since switch to Varnish from HAProxy. Would this be useful for varnish core? It's a very simple 4 line patch (attached). Thanks, Nate -------------- next part -------------- A non-text attachment was scrubbed... Name: backend_connections_stat.patch Type: application/octet-stream Size: 1470 bytes Desc: not available URL: From dridi.boukelmoune at zenika.com Thu May 23 11:44:23 2013 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Thu, 23 May 2013 13:44:23 +0200 Subject: Varnishapi patches In-Reply-To: References: Message-ID: Hi Martin, I did some tests regarding this non-issue, and I can't reproduce it. I came back to my source directory and launched the same exact commands ("varnishtest | grep" from my bash history) and it worked. I haven't touched anything in my source tree or my varnish clone, so I'm clueless. I've also tested a non verbose varnishtest execution with an error in the test case, and it writes to stdout, so I can't explain why the output ended up in my tty instead of grep's stdin. Sorry for the disturbance, Dridi On Tue, May 21, 2013 at 11:53 AM, Dridi Boukelmoune wrote: > Hi, > > This wasn't actually brought by your patches, I've tried to rewind to > the previous commit I was building my module against and had the same > problem. > > I can't test this until tonight, but I'll dig in my memory instead: > - varnishtest has become very slow on the master branch, like a second > slower per test. I know this is unrelated but it had to be said :p > > - I'm used to grep "varnishtest -v" output, but I wrote the previous > mail after failing to grep and having varnishtest's ouput in stderr > instead. So I assumed there was an error (I don't remember myself > checking the exit status). > > I hope this helps. > > Best Regards, > Dridi > > On Tue, May 21, 2013 at 11:19 AM, Martin Blix Grydeland > wrote: >> >> Hi Dridi, >> >> Looking at the varnishtest output attached, that looks completely normal to me, and the expected behavior. None of those lines indicate an error condition. When a varnishtest test case ends, it will always stop the varnishd instance before exiting, which is what the log lines you see describe. Please elaborate on what problems you are experiencing, and why you believe this is a problem. >> >> Regards, >> Martin Blix Grydeland >> >> >> On 17 May 2013 21:01, Dridi Boukelmoune wrote: >>> >>> Hi, >>> >>> Those patches have been committed in the master if I'm not mistaken. >>> I've read a lot of it but at some point it became way beyond my >>> understanding/knowledge of Varnish internals. Anyway, with the current >>> master (e16233fd332) varnishd dies (SIGHUP) when I run varnishtest in >>> verbose mode. >>> >>> I'm replying here because I remember seeing something about the logs, >>> varnishlog and varnishtest a some point(s), so I thought it might be >>> related. >>> >>> ** v1 0.7 as expected: cache_hit (2) == 2 >>> * top 0.7 RESETTING after src/tests/regfilter01_regfilter_in_vcl_hash.vtc >>> ** s1 0.7 Waiting for server >>> **** s1 0.7 macro undef s1_addr >>> **** s1 0.7 macro undef s1_port >>> **** s1 0.7 macro undef s1_sock >>> **** v1 0.7 macro undef v1_addr >>> **** v1 0.7 macro undef v1_port >>> **** v1 0.7 macro undef v1_sock >>> ** v1 0.7 Stop >>> **** v1 0.7 CLI TX| stop >>> *** v1 0.7 debug| Stopping Child\n >>> **** v1 0.7 vsl| 0 CLI - EOF on CLI >>> connection, worker stops >>> *** v1 1.7 debug| Child (13868) died status=1\n >>> *** v1 1.7 debug| Child (13868) said Child dies\n >>> *** v1 1.7 debug| Child cleanup complete\n >>> *** v1 1.7 CLI RX 200 >>> >>> It happens at the very end of the test after the last assertion. >>> >>> Best Regards, >>> Dridi >>> >>> >>> On Mon, May 13, 2013 at 2:44 PM, Martin Blix Grydeland >>> wrote: >>> > >>> > Please find attached patch set for the first parts of new libvarnishapi >>> > >>> > Regards, >>> > Martin Blix Grydeland >>> > >>> > -- >>> > Martin Blix Grydeland >>> > Senior Developer | Varnish Software AS >>> > Cell: +47 21 98 92 60 >>> > We Make Websites Fly! >>> > >>> > _______________________________________________ >>> > varnish-dev mailing list >>> > varnish-dev at varnish-cache.org >>> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev >> >> >> >> >> -- >> Martin Blix Grydeland >> Senior Developer | Varnish Software AS >> Cell: +47 21 98 92 60 >> We Make Websites Fly! From aleemb at gmail.com Thu May 23 11:57:22 2013 From: aleemb at gmail.com (Aleem B) Date: Thu, 23 May 2013 16:57:22 +0500 Subject: Varnish as a redirector for a millions URLs Message-ID: Hi, I currently use nginx + redis + lua module to setup redirects for close to a million old URLs that have been ported to a new platform. Is it possible to achieve the same with varnish (maybe replay logs for all redirects and set a max TTL. My cache size is more than enough to prevent LRU from erasing these entries). However, I am not sure if that's a good approach since Varnish would still require a backend through which it populates the redirects. Is this recommended through Varnish and is there any better approach? Aleem -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Thu May 23 13:37:36 2013 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 23 May 2013 13:37:36 +0000 Subject: Varnish as a redirector for a millions URLs In-Reply-To: References: Message-ID: <8616.1369316256@critter.freebsd.dk> In message >I currently use nginx + redis + lua module to setup redirects for close to >a million old URLs that have been ported to a new platform. Are the redirects based on rules ? Ie, something like: /article?ID=##### -> /##### or is it a database lookup to figure out the mapping ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From aleemb at gmail.com Thu May 23 13:41:08 2013 From: aleemb at gmail.com (Aleem B) Date: Thu, 23 May 2013 18:41:08 +0500 Subject: Varnish as a redirector for a millions URLs In-Reply-To: <8616.1369316256@critter.freebsd.dk> References: <8616.1369316256@critter.freebsd.dk> Message-ID: It's based on a DB lookup. Currently I am using NGINX + Redis and there are about a million URLs. Aleem On Thu, May 23, 2013 at 6:37 PM, Poul-Henning Kamp wrote: > In message Uhp_pepXsb6Q_ORQpYe+AhWSh_yytRNjKBzBBA_FxwJb28Q at mail.gmail.com> > > >I currently use nginx + redis + lua module to setup redirects for close to > >a million old URLs that have been ported to a new platform. > > Are the redirects based on rules ? > > Ie, something like: > > /article?ID=##### -> /##### > > or is it a database lookup to figure out the mapping ? > > > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Raul.Rangel at disney.com Thu May 23 13:57:56 2013 From: Raul.Rangel at disney.com (Rangel, Raul) Date: Thu, 23 May 2013 06:57:56 -0700 Subject: Varnish as a redirector for a millions URLs In-Reply-To: References: <8616.1369316256@critter.freebsd.dk> Message-ID: <2465AAEEC8B8A242B26ED5F44BCA805F261AAAEB63@SM-CALA-VXMB04A.swna.wdpr.disney.com> Can you make a backend that takes a url, looks it up in the database and returns a 301? You could then cache the 301 in varnish. Thought tht will only work for GETs. Once you get into POST requests it's another story. Raul From: varnish-dev-bounces at varnish-cache.org [mailto:varnish-dev-bounces at varnish-cache.org] On Behalf Of Aleem B Sent: Thursday, May 23, 2013 7:41 AM To: Poul-Henning Kamp Cc: varnish-dev at varnish-cache.org Subject: Re: Varnish as a redirector for a millions URLs It's based on a DB lookup. Currently I am using NGINX + Redis and there are about a million URLs. Aleem On Thu, May 23, 2013 at 6:37 PM, Poul-Henning Kamp > wrote: In message > >I currently use nginx + redis + lua module to setup redirects for close to >a million old URLs that have been ported to a new platform. Are the redirects based on rules ? Ie, something like: /article?ID=##### -> /##### or is it a database lookup to figure out the mapping ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Thu May 23 14:45:36 2013 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 23 May 2013 14:45:36 +0000 Subject: Varnish as a redirector for a millions URLs In-Reply-To: References: <8616.1369316256@critter.freebsd.dk> Message-ID: <26572.1369320336@critter.freebsd.dk> In message , Aleem B writes: >It's based on a DB lookup. Currently I am using NGINX + Redis and there are >about a million URLs. Then you can either have a backend server do the lookup and have varnish cache the result, or you can write a VMOD for varnish which does the lookup and generate the redirects that way. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From aleemb at gmail.com Thu May 23 15:37:48 2013 From: aleemb at gmail.com (Aleem B) Date: Thu, 23 May 2013 20:37:48 +0500 Subject: Varnish as a redirector for a millions URLs In-Reply-To: <26572.1369320336@critter.freebsd.dk> References: <8616.1369316256@critter.freebsd.dk> <26572.1369320336@critter.freebsd.dk> Message-ID: > > Then you can either have a backend server do the lookup and have varnish > cache the result, or you can write a VMOD for varnish which does the lookup > and generate the redirects that way. The database of URLS won't change (these are legacy domains). Is there any way manually insert a 301 redirect entry to the varnish cache? Can the VMOD do that? If yes, is there anyway for the VMOD to protect entries from getting wiped out by LRU (make them static)? Thanks, Aleem -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at varnish-software.com Mon May 27 11:12:03 2013 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 27 May 2013 13:12:03 +0200 Subject: Reminder to change the default Varnish version in trac bugs from "trunk" to "unknown" Message-ID: Today, the default Varnish version attribute on new trac tickets is "trunk". Most tickets seen are 3.0 specific issues, but they are still reported on "trunk" because people forget to set the field. We should maybe have an "unknown" value by default to make it clear when the reporter hasn't changed the field. Regards, Martin -- *Martin Blix Grydeland* Senior Developer | Varnish Software AS Cell: +47 21 98 92 60 We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark at wikimedia.org Wed May 29 16:02:41 2013 From: mark at wikimedia.org (Mark Bergsma) Date: Wed, 29 May 2013 18:02:41 +0200 Subject: VCL control of Vary processing Message-ID: <7CC5DE73-4BE7-4183-B88B-4D8319775245@wikimedia.org> I'd like to make a case for some additional control of Vary processing by VCL. As everyone knows, the HTTP standard Vary header is pretty limited in terms of flexibility. It only supports specifying header names, and when used in a naive way, i.e. for headers which clients fill with greatly varying content or format, it can be disastrous to your cache efficiency. Good examples of this are the Cookie, Accept-Language, User-Agent headers. Fortunately Varnish provides a powerful way to influence this with VCL, and headers can be filtered, normalized or cleaned up before they hit the cache lookup stage. This works well for headers with well known, predefined keys or values: specific strings can be tested for in VCL, and be removed from the header if not relevant for the cache lookup. With a bit more string (list) processing, this could even be easier and more powerful with list filtering, sorting capability for canonical ordering etc., but that's the sort of thing that could be implemented in a VMOD. What isn't currently very easy and straightforward to do in Varnish, is using information sent by the backend for Vary processing. Sometimes it's not easy or even nearly impossible to put the logic for preprocessing/sanitizing the varied headers in VCL; for example because it varies (no pun intended) a lot across all served content. In that case it could be nice to have the backend provide support for the Vary preprocessing logic in VCL, for example by sending a list of acceptable values for a header that is varied on for that particular URL, in a response header. Take for example the Accept-Language header. Although fortunately not required for the majority of content we serve, we do have a need to Vary on that header for some wikis in the Wikimedia cluster, for language wikis with multiple variants. Obviously, simply adding the header to the Vary list without anything else would simply destroy the caching. Users across the world send wildly different Accept-Language based on their location and browser settings. Doing some regex filtering and sanitization of the header would help a bit, but it's still far from optimal. What we've done in the last 5 years with Squid is to have our backend (MediaWiki) send a header specifying what header content should be varied on, and what should be ignored. It's handled by a patch to Squid, and it's described here: http://www.squid-cache.org/mail-archive/squid-dev/200802/0085.html This has worked very well for us and has resulted in a very good cache efficiency, while still providing some flexibility at the cache level that is being directed by the backend. This was convenient, as the backend is in control of the cache variance when and where it's needed. It knows its content and its needs best, and Squid doesn't allow for much logic and flexibility in its configuration anyway. With our migration to Varnish, we can do a large part of the necessary cache varying processing in VCL, as described above. Accept-Encoding (compression) normalization is already better handled by Varnish internally, most cookie names/values are pretty consistent across our content set (in _our_ application, but YMMV!) and with a few hacks in VCL we can mostly get there. But the Accept-Language issue as described above is a tougher one, as the acceptable values depend on which wiki/language is being served, and which variants are available for that particular URL. We've looked at porting (a variant of) the X-Vary-Options header to Varnish, using it in some way with VCL for the logic behind it, but it seems messy and suboptimal without patching Varnish itself, which isn't a very nice solution either. It seems that at the root of the problem is the current lack of ability in VCL to influence the Vary processing, while having an object from the cache available for lookup of a response header. Something like a VCL hook, between the result of a cache lookup hit, but before additional Vary processing/lookup is done. Conceptually, assuming the Vary response headers by the backend for Vary direction (like Vary and X-Vary-Options) are consistent across all variants, a random (or first) variant object at the hash key location could be used for "obj" in VCL in the hook. The VCL hook can learn about which headers are varied on (Vary resp. header), possibly get the allowed values for each header (e.g. a list of available languages for this url, "Available-Languages: sr-ec,sr-el"), and can filter the Accept-Language header in preparation of the Vary processing by Varnish. Even nicer if it could influence the Vary string generation itself, similar to how hash_data() works. Right now a hook between the cache lookup hit and the Vary processing step doesn't exist, which makes this clunky. You can probably sort of do it with restart today: 1) In vcl_recv() or vcl_lookup(), guess which headers might be in the Vary header, and rename those out of the way 2) in vcl_hit(), check the Vary response header for which headers are varying, sanitize those from the saved values, restore them under original name, restart the request 3) Make sure to skip 1) and 2) in the restarted request. vcl_recv, vcl_lookup, and the cache lookup are repeated although this isn't really necessary. This probably works, but VCL has limited info in step 1), and the necessary request restart makes things more complex and requires the rest of the VCL code to handle that, while it's not really obvious. I think a VCL hook between the lookup hit and vary processing stage would go a long way toward solving this problem in a clean way. This should allow for a lot of flexibility for optimizing cache variance, while not sacrificing performance. It could easily be used for things like the Cookie header as well, when the "allowed" cookies are not easily hardcoded in VCL. One possible catch/frequent mistake I see to such a VCL hook: it does require people to be aware that the "obj" reference they get in that VCL hook would not be the final variant object that will eventually deliver the object... Additionally it could be nice to be able to restart just a lookup in vcl_hit/vcl_miss, to be able to modify the request "to make a cache hit work" without restarting everything, for some other use cases as well. But I can also see problems with introducing such a loop. Right now it looks like VCL has no cycles besides the big "restart" of the request, and this would change that. As far as I can see from phk's flow diagram of last month, this also doesn't seem possible with the Varnish 4.0 vcl_lookup() plan[1] either. Considering the future of the VCL state machine flow and the VCL hooks is now under discussion, I thought I'd throw this in there. :) [1] https://www.varnish-cache.org/lists/pipermail/varnish-dev/2013-April/007529.html -- Mark Bergsma Lead Operations Architect Wikimedia Foundation