From varnish-cache at otoh.org Sun Jan 5 18:12:56 2014 From: varnish-cache at otoh.org (Paul Armstrong) Date: Sun, 5 Jan 2014 18:12:56 +0000 Subject: Disabling connection close in vcl_error Message-ID: <20140105181255.GS19190@suricate.otoh.org> For our use case, we need to disable the default behavior of closing connections whenever vcl_error is called so that we can generate synthetic responses to various error conditions without causing connections to be constantly re-established. We've applied the following patch in an effort to do this, but we keep on seeing leaks of n_sess_mem (it routinely bumps up against the 100,000 limit when we have high levels of traffic responded to synthetically). --- varnish-3.0.5/bin/varnishd/cache_center.c.orig 2013-12-02 08:06:30.000000000 +0000 +++ varnish-3.0.5/bin/varnishd/cache_center.c 2013-12-31 22:54:57.094893222 +0000 @@ -495,13 +495,18 @@ /* We always close when we take this path */ - sp->doclose = "error"; + // sp->doclose = "error"; sp->wantbody = 1; assert(sp->handling == VCL_RET_DELIVER); sp->err_code = 0; sp->err_reason = NULL; http_Setup(sp->wrk->bereq, NULL); + if (sp->obj->objcore != NULL) { + AN(sp->obj->objcore); + AN(sp->obj->objcore->ban); + HSH_Unbusy(sp); + } sp->step = STP_PREPRESP; return (0); } Obviously, we're missing some point in the cleanup process, but we're having trouble identifying it and some assistance would be appreciated. Thanks, Paul From slink at schokola.de Thu Jan 9 14:56:38 2014 From: slink at schokola.de (Nils Goroll) Date: Thu, 09 Jan 2014 15:56:38 +0100 Subject: "officially" supported compilers? Message-ID: <52CEB8A6.1050401@schokola.de> Hi, d9492da540430103b9dcd6844eb59b508c061d66 effectively breaks builds with gcc 3.4. Should we make a statement regarding minimum supported compiler versions? Nils From phk at phk.freebsd.dk Thu Jan 9 10:01:52 2014 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 09 Jan 2014 10:01:52 +0000 Subject: Varnishtop on the new logging API In-Reply-To: <529C9F97.60209@smartjog.com> References: <529C9F97.60209@smartjog.com> Message-ID: <17463.1389261712@critter.freebsd.dk> >As said on IRC, here's a shot at porting varnishtop on the new logging >API. I didn't change much, maybe not enough (RB tree freeing?), but it >seems to work. I'm wading through my inbox, did Martin look at this patch ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Thu Jan 9 15:57:23 2014 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 09 Jan 2014 15:57:23 +0000 Subject: "officially" supported compilers? In-Reply-To: <52CEB8A6.1050401@schokola.de> References: <52CEB8A6.1050401@schokola.de> Message-ID: <1723.1389283043@critter.freebsd.dk> In message <52CEB8A6.1050401 at schokola.de>, Nils Goroll writes: >d9492da540430103b9dcd6844eb59b508c061d66 effectively breaks builds >with gcc 3.4. That was not the intention. Can you find out which particular option it is ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From slink at schokola.de Thu Jan 9 16:08:41 2014 From: slink at schokola.de (Nils Goroll) Date: Thu, 09 Jan 2014 17:08:41 +0100 Subject: "officially" supported compilers? In-Reply-To: <1723.1389283043@critter.freebsd.dk> References: <52CEB8A6.1050401@schokola.de> <1723.1389283043@critter.freebsd.dk> Message-ID: <52CEC989.8020607@schokola.de> On 01/ 9/14 04:57 PM, Poul-Henning Kamp wrote: > Can you find out which particular option it is ? I had already started work on a fix, so now that I know your intention I will finish it. Nils From martin at varnish-software.com Thu Jan 9 16:41:03 2014 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Thu, 9 Jan 2014 17:41:03 +0100 Subject: Varnishtop on the new logging API In-Reply-To: <17463.1389261712@critter.freebsd.dk> References: <529C9F97.60209@smartjog.com> <17463.1389261712@critter.freebsd.dk> Message-ID: Not looked at it in detail yet, though first impression looks good. Will be going in with some tweaks. Thanks for the reminder though :-) Main 9. jan. 2014 17:23 skrev "Poul-Henning Kamp" f?lgende: > > >As said on IRC, here's a shot at porting varnishtop on the new logging > >API. I didn't change much, maybe not enough (RB tree freeing?), but it > >seems to work. > > I'm wading through my inbox, did Martin look at this patch ? > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume.quintard at smartjog.com Thu Jan 9 16:44:10 2014 From: guillaume.quintard at smartjog.com (Guillaume Quintard) Date: Thu, 9 Jan 2014 17:44:10 +0100 Subject: Varnishtop on the new logging API In-Reply-To: References: <529C9F97.60209@smartjog.com> <17463.1389261712@critter.freebsd.dk> Message-ID: <52CED1DA.7090404@smartjog.com> Ah, I was wondering about it too. Martin, feel free to give me pointer about what you want changed, I see a few spots that could be more elegant or consistent, but you'll have probably more to say. On 01/09/2014 05:41 PM, Martin Blix Grydeland wrote: > > Not looked at it in detail yet, though first impression looks good. > Will be going in with some tweaks. Thanks for the reminder though :-) > > Main > > 9. jan. 2014 17:23 skrev "Poul-Henning Kamp" > f?lgende: > > > >As said on IRC, here's a shot at porting varnishtop on the new > logging > >API. I didn't change much, maybe not enough (RB tree freeing?), > but it > >seems to work. > > I'm wading through my inbox, did Martin look at this patch ? > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by > incompetence. > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -- Guillaume Quintard -------------- next part -------------- An HTML attachment was scrubbed... URL: From slink at schokola.de Thu Jan 9 17:31:42 2014 From: slink at schokola.de (Nils Goroll) Date: Thu, 09 Jan 2014 18:31:42 +0100 Subject: [PATCH] [EXPERIMENTAL] autocrap autohardening Message-ID: <52CEDCFE.2090508@schokola.de> Hi, I stumbled over http://mainisusuallyafunction.blogspot.de/2012/05/automatic-binary-hardening-with.html today and thought to give it a try with varnish. Result is attached: I have integrated a slightly modified version of Keegan's configure.ac. Changes: - removed CXX support - replaced -fstack-protector-all with -fstack-protector-strong and fallback to -fstack-protector - removed -Wstack-protector (XXXLATER: disable for specific functions only?) This survives a "make check" on SunOS 5.11 snv_134 # ancient gcc (GCC) 4.3.3 and Debian 6.0.8 Linux debhag 2.6.32-5-xen-amd64 #1 SMP gcc (Debian 4.4.5-8) 4.4.5 I have checked checksec output (attached) and run-times for make check on linux with -fstack-protector (-fstack-protector-strong is TODO) * hardening enabled (default) debhag:~/v/varnish-git/varnish-cache# time make check ... ==================== All 352 tests passed ==================== ... real 12m32.646s user 1m12.137s sys 0m51.791s - * --disable-hardening debhag:~/v/varnish-git/varnish-cache# time make check ... ==================== All 352 tests passed ==================== real 12m21.915s user 1m11.992s sys 0m53.631s Should there be any interest in integrating something like this, we probably would need to do more extensive testing and benchmarking. Also, whether or not varnish could benefit from such hardening is a completely different question - personally I'd consider phk's defensive coding approach much more important than additional stack/buffer overflow protection, load address randomization and page protection. Nils -------------- next part -------------- A non-text attachment was scrubbed... Name: varnish_checksec_disable_hardening.png Type: image/png Size: 18174 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: varnish_checksec_with_hardening.png Type: image/png Size: 18461 bytes Desc: not available URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: 0001-autocrap-auto-hardening.patch URL: From slink at schokola.de Thu Jan 9 18:07:25 2014 From: slink at schokola.de (Nils Goroll) Date: Thu, 09 Jan 2014 19:07:25 +0100 Subject: [PATCH] fix build with --enable-developer-warnings and gcc3.4 In-Reply-To: <52CEB8A6.1050401@schokola.de> References: <52CEB8A6.1050401@schokola.de> Message-ID: <52CEE55D.4050108@schokola.de> > d9492da540430103b9dcd6844eb59b508c061d66 effectively breaks builds with gcc 3.4. ... only when developer warnings are enabled. Patch attached. This also fixes linker issues if the compiler supports -fstack-protector but not the linker and a "the address of ?buf? will never be NULL" on Solaris / gcc 4.3.3 Nils -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: 0001-fix-build-with-enable-developer-warnings-and-gcc3.patch URL: From guillaume.quintard at smartjog.com Fri Jan 10 13:58:03 2014 From: guillaume.quintard at smartjog.com (Guillaume Quintard) Date: Fri, 10 Jan 2014 14:58:03 +0100 Subject: Non-recursive build system Message-ID: <52CFFC6B.7080701@smartjog.com> Good friday all, I took a few couple hour to toy with Poul-Henning build framework and see if I could make it non-recursive (yes, I do have strange obsessions). I attached the result as a git patch in order to get some feedback and see if it may be an interesting direction. A non-recursive solution allows for very efficient parallel builds and clean dependency handling. The first point may not be very important as varnish is rather small and builds are fast, but the second one is essential regarding consistency. However, if you want to do it without generating makefiles (like autools do), while still being able to run from subdirs, and having readable makefiles, well, it gets a bit complicated. And I'm wondering if the gain is worth the complexity. And since we are considering a new framework, I think it's the right time to look at this possibility. A few words about the patch attached: - run "sh config.phk" once first, then "make -f master.gq" - only supports all, clean, varnishd, varnishtest,... for goals (no check, doc, install yet) - dependencies are correctly handled (touching a vgz header will rebuild .o depending on it as well as ar'ing the lib, and varnishd link, for example), and no need to order the subdirs to compile in the right order. - only runs from the top directory, but the plan is to be able to run from subdirectories as well - for those interested, the main part of the code is in master.gq and footer.gq (GNUMake rookies, beware) - I had to touch Makefile.phks and config.phk a wee bit, but not that much As said, feedback is more than welcome. -- Guillaume Quintard -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-first-commit.patch Type: text/x-patch Size: 17503 bytes Desc: not available URL: From martin at varnish-software.com Mon Jan 13 14:55:51 2014 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 13 Jan 2014 15:55:51 +0100 Subject: Varnishtop on the new logging API In-Reply-To: <52CED1DA.7090404@smartjog.com> References: <529C9F97.60209@smartjog.com> <17463.1389261712@critter.freebsd.dk> <52CED1DA.7090404@smartjog.com> Message-ID: Hi Guillaume, I'll come back to you with a proper review and reply to this some time next week. This week I am away on a business trip all week and connectivity will be mostly offline. Martin 9. jan. 2014 16:44 skrev "Guillaume Quintard" < guillaume.quintard at smartjog.com> f?lgende: > Ah, I was wondering about it too. Martin, feel free to give me pointer > about what you want changed, I see a few spots that could be more elegant > or consistent, but you'll have probably more to say. > > On 01/09/2014 05:41 PM, Martin Blix Grydeland wrote: > > Not looked at it in detail yet, though first impression looks good. Will > be going in with some tweaks. Thanks for the reminder though :-) > > Main > 9. jan. 2014 17:23 skrev "Poul-Henning Kamp" > f?lgende: > >> >> >As said on IRC, here's a shot at porting varnishtop on the new logging >> >API. I didn't change much, maybe not enough (RB tree freeing?), but it >> >seems to work. >> >> I'm wading through my inbox, did Martin look at this patch ? >> >> -- >> Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 >> phk at FreeBSD.ORG | TCP/IP since RFC 956 >> FreeBSD committer | BSD since 4.3-tahoe >> Never attribute to malice what can adequately be explained by >> incompetence. >> >> _______________________________________________ >> varnish-dev mailing list >> varnish-dev at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev >> > > > -- > Guillaume Quintard > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at academia.edu Wed Jan 15 02:01:21 2014 From: david at academia.edu (David Judd) Date: Tue, 14 Jan 2014 18:01:21 -0800 Subject: ESI requests don't send modified headers from req to backend? Message-ID: Working with Varnish 3.0.5, in vcl_recv, I set a X-Request-Start header to the current millis since the epoch. I want that header set, with the same value, both in the main request and in a resulting ESI request (in part so I can use it to help match them.) But it doesn't work that way--although as I read the documentation, it sounds like it should. In fact, it seems that the backed only gets the original, client headers when it receives the ESI request, and not the custom headers I added before the ESI was triggered. The provided example in this earlier thread appears to be essentially the same issue: https://www.varnish-cache.org/lists/pipermail/varnish-dev/2012-November/007385.html Thoughts? Is this a bug? Is there a better way to send the same timestamp for the main request and an ESI request? Thanks, David From Gauthier.Delacroix at coreye.fr Mon Jan 20 08:31:53 2014 From: Gauthier.Delacroix at coreye.fr (Delacroix, Gauthier) Date: Mon, 20 Jan 2014 08:31:53 +0000 Subject: 4.0-tp1: obj.hits not incremented Message-ID: <5F530A9242E7F84F999DB40E0E268FBD45ADD6BD@mercalli.lild01.pictime.fr> Hi, I'm trying Varnish 4.0-tp1 on Linux Fedora 20 and I've found that obj.hits doesn't seem to be incremented anymore. Here is a simple example with this VCL : sub vcl_backend_response { unset beresp.http.Set-Cookie; set beresp.ttl = 3600s; } sub vcl_deliver { set resp.http.Hits = obj.hits; } Here's the varnishlog showing that I have a HIT, but obj.hits is zero : * << Request >> 32770 - Begin req 32769 - ReqMethod GET - ReqURL http://XXXX/icons/poweredby.png - ReqProtocol HTTP/1.1 - ReqHeader Host: XXXX - ReqHeader User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0 - ReqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 - ReqHeader Accept-Language: fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3 - ReqHeader Accept-Encoding: gzip, deflate - ReqHeader Connection: keep-alive - ReqHeader Pragma: no-cache - ReqHeader Cache-Control: no-cache - ReqStart 10.0.2.2 51974 - VCL_call RECV - VCL_return hash - VCL_call HASH - VCL_return lookup - Hit 2147483651 - VCL_call HIT - VCL_return deliver - VCL_call DELIVER - VCL_return deliver - Debug "RES_MODE 2%00" - RespProtocol HTTP/1.1 - RespStatus 200 - RespResponse OK - RespHeader Server: Apache/2.4.6 (Fedora) mod_perl/2.0.9-dev Perl/v5.18.2 - RespHeader Last-Modified: Thu, 04 Apr 2013 19:44:33 GMT - RespHeader ETag: "bda-4d98e331d8a40" - RespHeader Content-Type: image/png - RespHeader Date: Fri, 17 Jan 2014 10:21:56 GMT - RespHeader X-Varnish: 32770 3 - RespHeader Age: 3 - RespHeader Via: 1.1 varnish - RespHeader Hits: 0 - RespHeader Content-Length: 3034 - RespHeader Connection: keep-alive - RespHeader Accept-Ranges: bytes - Debug "XXX REF 2%00" - Length 0 - ReqEnd 1389954116.455149174 1389954116.455080271 -0.000174284 0.000105381 -0.000174284 - End My real VCL is using a condition (if (obj.hits > 0)...) that thus doesn't work. I've also tried with the same problem : - obj.hits in vcl_hit - todays master head I didn't find this in known issues. I didn't have this problem with Varnish 3.0.X Is this a bug ? Thanks ! Gauthier From phk at phk.freebsd.dk Mon Jan 20 10:26:04 2014 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 20 Jan 2014 10:26:04 +0000 Subject: 4.0-tp1: obj.hits not incremented In-Reply-To: <5F530A9242E7F84F999DB40E0E268FBD45ADD6BD@mercalli.lild01.pictime.fr> References: <5F530A9242E7F84F999DB40E0E268FBD45ADD6BD@mercalli.lild01.pictime.fr> Message-ID: <11754.1390213564@critter.freebsd.dk> In message <5F530A9242E7F84F999DB40E0E268FBD45ADD6BD at mercalli.lild01.pictime.fr >I'm trying Varnish 4.0-tp1 on Linux Fedora 20 and I've found that >obj.hits doesn't seem to be incremented anymore. See default.vcl. If you want obj.hits to increment, you must do it yourself in VCL. This was changed to make objects "read-only" in memory by default. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From apj at mutt.dk Mon Jan 20 10:29:37 2014 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Mon, 20 Jan 2014 11:29:37 +0100 Subject: 4.0-tp1: obj.hits not incremented In-Reply-To: <5F530A9242E7F84F999DB40E0E268FBD45ADD6BD@mercalli.lild01.pictime.fr> References: <5F530A9242E7F84F999DB40E0E268FBD45ADD6BD@mercalli.lild01.pictime.fr> Message-ID: <20140120102937.GT19870@nerd.dk> On Mon, Jan 20, 2014 at 08:31:53AM +0000, Delacroix, Gauthier wrote: > > I'm trying Varnish 4.0-tp1 on Linux Fedora 20 and I've found that obj.hits doesn't seem to be incremented anymore. Read the default vcl. You have to increment in vcl now. -- Andreas From Gauthier.Delacroix at coreye.fr Mon Jan 20 10:39:27 2014 From: Gauthier.Delacroix at coreye.fr (Delacroix, Gauthier) Date: Mon, 20 Jan 2014 10:39:27 +0000 Subject: 4.0-tp1: obj.hits not incremented In-Reply-To: <11754.1390213564@critter.freebsd.dk> References: <5F530A9242E7F84F999DB40E0E268FBD45ADD6BD@mercalli.lild01.pictime.fr> <11754.1390213564@critter.freebsd.dk> Message-ID: <5F530A9242E7F84F999DB40E0E268FBD45ADFA6A@mercalli.lild01.pictime.fr> Ok, I should have read the new default.vcl...this is often a quick course of the new syntax... Thanks for the answer. Is there a risk in enabling them ? Gauthier -----Message d'origine----- De?: Poul-Henning Kamp [mailto:phk at phk.freebsd.dk] Envoy??: lundi 20 janvier 2014 11:26 ??: Delacroix, Gauthier Cc?: varnish-dev at varnish-cache.org Objet?: Re: 4.0-tp1: obj.hits not incremented In message <5F530A9242E7F84F999DB40E0E268FBD45ADD6BD at mercalli.lild01.pictime.fr >I'm trying Varnish 4.0-tp1 on Linux Fedora 20 and I've found that >obj.hits doesn't seem to be incremented anymore. See default.vcl. If you want obj.hits to increment, you must do it yourself in VCL. This was changed to make objects "read-only" in memory by default. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From fgsch at lodoss.net Mon Jan 20 11:13:22 2014 From: fgsch at lodoss.net (Federico Schwindt) Date: Mon, 20 Jan 2014 11:13:22 +0000 Subject: [PATCH] fix build with --enable-developer-warnings and gcc3.4 In-Reply-To: <52CEE55D.4050108@schokola.de> References: <52CEB8A6.1050401@schokola.de> <52CEE55D.4050108@schokola.de> Message-ID: Hi, I was looking at your diff I don't understand the -Wno-missing-field-initializers vs -Wno-extra. If the compiler doesn't support the former why add the -Wno-extra instead of just not adding -Wno-missing... ? Why are you disabling -Wextra in that case? The linker check seems wrong to me. The config.log output for this part is: configure:16930: checking whether C compiler accepts -fstack-protector configure:16949: gcc -std=gnu99 -c -g -O2 -pthread -Wall -Werror -Wno-error=unused-result -fstack-protector conftest.c >&5 configure:16949: $? = 0 configure:16957: result: yes configure:16960: checking whether the linker accepts -fstack-protector configure:16979: gcc -std=gnu99 -o conftest -g -O2 -pthread -Wall -Werror -Wno-error=unused-result -fstack-protector conftest.c >&5 configure:16979: $? = 0 configure:16988: result: yes So both checks are doing the same. Can you share the config.log? f.- On Thu, Jan 9, 2014 at 6:07 PM, Nils Goroll wrote: > d9492da540430103b9dcd6844eb59b508c061d66 effectively breaks builds with >> gcc 3.4. >> > > ... only when developer warnings are enabled. > > Patch attached. > > This also fixes linker issues if the compiler supports -fstack-protector > but not the linker and a "the address of ?buf? will never be NULL" on > Solaris / gcc 4.3.3 > > Nils > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Mon Jan 20 13:55:55 2014 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 20 Jan 2014 13:55:55 +0000 Subject: 4.0-tp1: obj.hits not incremented In-Reply-To: <5F530A9242E7F84F999DB40E0E268FBD45ADFA6A@mercalli.lild01.pictime.fr> References: <5F530A9242E7F84F999DB40E0E268FBD45ADD6BD@mercalli.lild01.pictime.fr> <11754.1390213564@critter.freebsd.dk> <5F530A9242E7F84F999DB40E0E268FBD45ADFA6A@mercalli.lild01.pictime.fr> Message-ID: <12611.1390226155@critter.freebsd.dk> In message <5F530A9242E7F84F999DB40E0E268FBD45ADFA6A at mercalli.lild01.pictime.fr >, "Delacroix, Gauthier" writes: >Ok, I should have read the new default.vcl...this is often a quick course of >the new syntax... > >Thanks for the answer. Is there a risk in enabling them ? No. It will cost a fraction of your performance, but most people will never notice. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From Gauthier.Delacroix at coreye.fr Mon Jan 20 14:07:39 2014 From: Gauthier.Delacroix at coreye.fr (Delacroix, Gauthier) Date: Mon, 20 Jan 2014 14:07:39 +0000 Subject: 4.0-tp1: obj.hits not incremented In-Reply-To: <12611.1390226155@critter.freebsd.dk> References: <5F530A9242E7F84F999DB40E0E268FBD45ADD6BD@mercalli.lild01.pictime.fr> <11754.1390213564@critter.freebsd.dk> <5F530A9242E7F84F999DB40E0E268FBD45ADFA6A@mercalli.lild01.pictime.fr> <12611.1390226155@critter.freebsd.dk> Message-ID: <5F530A9242E7F84F999DB40E0E268FBD45ADFC1F@mercalli.lild01.pictime.fr> Ok, it works fine for obj.hits, but obj.lastuse gives this : Message from VCC-compiler: Unknown variable 'obj.lastuse' At: ('common_end.vcl' Line 68 Pos 7) set obj.lastuse = now; ------###########------- Running VCC-compiler failed, exit 1 VCL compilation failed Gauthier -----Message d'origine----- De?: Poul-Henning Kamp [mailto:phk at phk.freebsd.dk] Envoy??: lundi 20 janvier 2014 14:56 ??: Delacroix, Gauthier Cc?: varnish-dev at varnish-cache.org Objet?: Re: 4.0-tp1: obj.hits not incremented In message <5F530A9242E7F84F999DB40E0E268FBD45ADFA6A at mercalli.lild01.pictime.fr >, "Delacroix, Gauthier" writes: >Ok, I should have read the new default.vcl...this is often a quick >course of the new syntax... > >Thanks for the answer. Is there a risk in enabling them ? No. It will cost a fraction of your performance, but most people will never notice. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From fgsch at lodoss.net Mon Jan 20 15:29:14 2014 From: fgsch at lodoss.net (Federico Schwindt) Date: Mon, 20 Jan 2014 15:29:14 +0000 Subject: PATCH: fix dependency In-Reply-To: <20131214012228.GA30708@nitzer> References: <20131214012228.GA30708@nitzer> Message-ID: ping On Sat, Dec 14, 2013 at 1:22 AM, Federico G. Schwindt wrote: > Hi, > > The diff below ensures the dependency on vsc_fields.h is always present. > > f.- > > diff --git a/man/Makefile.am b/man/Makefile.am > index 56b0a52..477d37d 100644 > --- a/man/Makefile.am > +++ b/man/Makefile.am > @@ -32,11 +32,11 @@ else > @false > endif > > +varnish-counters.7: vsc2rst \ > + $(top_srcdir)/include/tbl/vsc_fields.h > if HAVE_RST2MAN > -varnish-counters.7: vsc2rst > ./vsc2rst | ${RST2MAN} - $@ > else > -varnish-counters.7: $(top_srcdir)/include/tbl/vsc_fields.h > @echo "========================================" > @echo "You need rst2man installed to make dist" > @echo "========================================" > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fgsch at lodoss.net Mon Jan 20 15:46:42 2014 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Mon, 20 Jan 2014 15:46:42 +0000 Subject: [PATCH] Add missing library dep Message-ID: <20140120154642.GA12024@nitzer> Hi, The diff below adds rt to the libvarnishapi dependencies if rt is required. Makes linking against libvarnishapi happy. OK? f.- objdump output before: Dynamic Section: NEEDED libpcre.so.3 NEEDED libpthread.so.0 NEEDED libc.so.6 SONAME libvarnishapi.so.1 And after: Dynamic Section: NEEDED libpcre.so.3 NEEDED librt.so.1 NEEDED libpthread.so.0 NEEDED libc.so.6 SONAME libvarnishapi.so.1 diff --git a/lib/libvarnishapi/Makefile.am b/lib/libvarnishapi/Makefile.am index fce096d..9c96d39 100644 --- a/lib/libvarnishapi/Makefile.am +++ b/lib/libvarnishapi/Makefile.am @@ -42,7 +42,7 @@ libvarnishapi_la_SOURCES = \ libvarnishapi_la_CFLAGS = \ -DVARNISH_STATE_DIR='"${VARNISH_STATE_DIR}"' -libvarnishapi_la_LIBADD = @PCRE_LIBS@ +libvarnishapi_la_LIBADD = @PCRE_LIBS@ @RT_LIBS@ if HAVE_LD_VERSION_SCRIPT libvarnishapi_la_LDFLAGS += -Wl,--version-script=$(srcdir)/libvarnishapi.map From fgsch at lodoss.net Mon Jan 20 15:50:22 2014 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Mon, 20 Jan 2014 15:50:22 +0000 Subject: [PATCH] Move local functions to vsm_int.h Message-ID: <20140120155022.GA2667@nitzer> VSM__iter0 and VSM__itern are local. Move the prototypes to vsm_int.h. Tests are still happy. OK? f.- diff --git a/include/vapi/vsm.h b/include/vapi/vsm.h index 04679b5..5a14d2e 100644 --- a/include/vapi/vsm.h +++ b/include/vapi/vsm.h @@ -147,18 +147,6 @@ void VSM_Close(struct VSM_data *vd); * record references. */ - -void VSM__iter0(const struct VSM_data *vd, struct VSM_fantom *vf); -int VSM__itern(const struct VSM_data *vd, struct VSM_fantom *vf); - -#define VSM_FOREACH(vf, vd) \ - for(VSM__iter0((vd), (vf)); VSM__itern((vd), (vf));) - /* - * Iterate over all chunks in shared memory - * vf = "struct VSM_fantom *" - * vd = "struct VSM_data *" - */ - enum VSM_valid_e { VSM_invalid, VSM_valid, diff --git a/include/vapi/vsm_int.h b/include/vapi/vsm_int.h index 9305547..a5cbd8c 100644 --- a/include/vapi/vsm_int.h +++ b/include/vapi/vsm_int.h @@ -100,6 +100,9 @@ #define VSM_MARKER_LEN 8 #define VSM_IDENT_LEN 128 +struct VSM_fantom; +struct VSM_data; + struct VSM_chunk { #define VSM_CHUNK_MARKER "VSMCHUNK" char marker[VSM_MARKER_LEN]; @@ -120,4 +123,15 @@ struct VSM_head { uint64_t age; }; +void VSM__iter0(const struct VSM_data *vd, struct VSM_fantom *vf); +int VSM__itern(const struct VSM_data *vd, struct VSM_fantom *vf); + +#define VSM_FOREACH(vf, vd) \ + for(VSM__iter0((vd), (vf)); VSM__itern((vd), (vf));) + /* + * Iterate over all chunks in shared memory + * vf = "struct VSM_fantom *" + * vd = "struct VSM_data *" + */ + #endif /* VSM_INT_H_INCLUDED */ From phk at phk.freebsd.dk Mon Jan 20 16:23:15 2014 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 20 Jan 2014 16:23:15 +0000 Subject: [PATCH] Move local functions to vsm_int.h In-Reply-To: <20140120155022.GA2667@nitzer> References: <20140120155022.GA2667@nitzer> Message-ID: <13460.1390234995@critter.freebsd.dk> In message <20140120155022.GA2667 at nitzer>, "Federico G. Schwindt" writes: > VSM__iter0 and VSM__itern are local. Move the prototypes to vsm_int.h. > Tests are still happy. I think VSM_FOREACH() is part of the official API, so it shold not go in vsm_int.h, and I think we should keep the two functions it use with it. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From bilbo at hobbiton.org Mon Jan 20 16:53:27 2014 From: bilbo at hobbiton.org (Leif Pedersen) Date: Mon, 20 Jan 2014 10:53:27 -0600 Subject: varnish 3.0.4version may be have memory leak In-Reply-To: References: Message-ID: Hi, Varnish uses a memory-mapped file to store its data. Therefore, it is expected to use a lot of memory. However, when the kernel gets low on memory, it will push memory-mapped back to disk. This is similar in mechanism to the kernel's disk cache. Does anything indicate that the memory pressure is actually the problem? Perhaps try lowering the maximum number of threads. Also, perhaps try making the varnish cache file smaller. Hope this helps. - Leif -------------- next part -------------- An HTML attachment was scrubbed... URL: From fgsch at lodoss.net Mon Jan 20 17:16:20 2014 From: fgsch at lodoss.net (Federico Schwindt) Date: Mon, 20 Jan 2014 17:16:20 +0000 Subject: [PATCH] Move local functions to vsm_int.h In-Reply-To: <13460.1390234995@critter.freebsd.dk> References: <20140120155022.GA2667@nitzer> <13460.1390234995@critter.freebsd.dk> Message-ID: Oh ok. If that's the case please disregard this diff. f.- On Mon, Jan 20, 2014 at 4:23 PM, Poul-Henning Kamp wrote: > In message <20140120155022.GA2667 at nitzer>, "Federico G. Schwindt" writes: > > > VSM__iter0 and VSM__itern are local. Move the prototypes to vsm_int.h. > > Tests are still happy. > > I think VSM_FOREACH() is part of the official API, so it > shold not go in vsm_int.h, and I think we should keep > the two functions it use with it. > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From slink at schokola.de Mon Jan 20 18:37:09 2014 From: slink at schokola.de (Nils Goroll) Date: Mon, 20 Jan 2014 19:37:09 +0100 Subject: [PATCH] fix build with --enable-developer-warnings and gcc3.4 In-Reply-To: References: <52CEB8A6.1050401@schokola.de> <52CEE55D.4050108@schokola.de> Message-ID: <52DD6CD5.2040409@schokola.de> Hi Federico, On 01/20/14 12:13 PM, Federico Schwindt wrote: > I was looking at your diff I don't understand the > -Wno-missing-field-initializers vs -Wno-extra. If the compiler doesn't support > the former why add the -Wno-extra instead of just not adding -Wno-missing... ? Hm, your question seems to contain the answer ;) If the compiler does not support -Wno-missing-field-initializers, we can't use -Wno-missing-field-initializers, so we have to use -Wno-extra > The linker check seems wrong to me. The config.log output for this part is: > > configure:16930: checking whether C compiler accepts -fstack-protector > configure:16949: gcc -std=gnu99 -c -g -O2 -pthread -Wall -Werror > -Wno-error=unused-result -fstack-protector conftest.c >&5 > configure:16949: $? = 0 > configure:16957: result: yes > configure:16960: checking whether the linker accepts -fstack-protector > configure:16979: gcc -std=gnu99 -o conftest -g -O2 -pthread -Wall -Werror > -Wno-error=unused-result -fstack-protector conftest.c >&5 > configure:16979: $? = 0 > configure:16988: result: yes > > So both checks are doing the same. Can you share the config.log? Notice that gcc is invoked with -c for the first and -o for the second case. It's no surprise to me that the second test only fails on an exotic OS (read: OpenSolaris) :P Cheers, Nils From fgsch at lodoss.net Mon Jan 20 20:47:50 2014 From: fgsch at lodoss.net (Federico Schwindt) Date: Mon, 20 Jan 2014 20:47:50 +0000 Subject: [PATCH] fix build with --enable-developer-warnings and gcc3.4 In-Reply-To: <52DD6CD5.2040409@schokola.de> References: <52CEB8A6.1050401@schokola.de> <52CEE55D.4050108@schokola.de> <52DD6CD5.2040409@schokola.de> Message-ID: On Mon, Jan 20, 2014 at 6:37 PM, Nils Goroll wrote: > [..] > > I was looking at your diff I don't understand the >> -Wno-missing-field-initializers vs -Wno-extra. If the compiler doesn't >> support >> the former why add the -Wno-extra instead of just not adding >> -Wno-missing... ? >> > > Hm, your question seems to contain the answer ;) > If the compiler does not support -Wno-missing-field-initializers, we > can't use -Wno-missing-field-initializers, so we have to use -Wno-extra And answering to myself: to silence the otherwise warning added by -Wextra. Because -Werror is there. > [..] >> So both checks are doing the same. Can you share the config.log? >> > > Notice that gcc is invoked with -c for the first and -o for the second > case. It's no surprise to me that the second test only fails on an exotic > OS (read: OpenSolaris) :P > You are absolutely right, missed that. f.- -------------- next part -------------- An HTML attachment was scrubbed... URL: From lkarsten at varnish-software.com Tue Jan 21 13:12:22 2014 From: lkarsten at varnish-software.com (Lasse Karstensen) Date: Tue, 21 Jan 2014 14:12:22 +0100 Subject: PATCH: fix dependency In-Reply-To: <20131214012228.GA30708@nitzer> References: <20131214012228.GA30708@nitzer> Message-ID: <20140121131221.GA11667@immer.varnish-software.com> On Sat, Dec 14, 2013 at 01:22:28AM +0000, Federico G. Schwindt wrote: > The diff below ensures the dependency on vsc_fields.h is always present. Merged in b1b777d now. Thanks. -- With regards, Lasse Karstensen Varnish Software AS From lkarsten at varnish-software.com Tue Jan 21 13:14:49 2014 From: lkarsten at varnish-software.com (Lasse Karstensen) Date: Tue, 21 Jan 2014 14:14:49 +0100 Subject: PATCH: fix warning on systems without TCP_KEEPxxx In-Reply-To: <20131214012916.GB30708@nitzer> References: <20131214012916.GB30708@nitzer> Message-ID: <20140121131448.GB11667@immer.varnish-software.com> On Sat, Dec 14, 2013 at 01:29:16AM +0000, Federico G. Schwindt wrote: > Just a reminder. Not all OS support TCP_KEEPxxx, OpenBSD being one of them. This seems to already have made its way into master in git commit 57633531456febf60663562b938f3ff7a40f59d3. Thanks. -- With regards, Lasse Karstensen Varnish Software AS From fgsch at lodoss.net Wed Jan 22 09:40:35 2014 From: fgsch at lodoss.net (Federico Schwindt) Date: Wed, 22 Jan 2014 09:40:35 +0000 Subject: 4.0-tp1: obj.hits not incremented In-Reply-To: <5F530A9242E7F84F999DB40E0E268FBD45ADFC1F@mercalli.lild01.pictime.fr> References: <5F530A9242E7F84F999DB40E0E268FBD45ADD6BD@mercalli.lild01.pictime.fr> <11754.1390213564@critter.freebsd.dk> <5F530A9242E7F84F999DB40E0E268FBD45ADFA6A@mercalli.lild01.pictime.fr> <12611.1390226155@critter.freebsd.dk> <5F530A9242E7F84F999DB40E0E268FBD45ADFC1F@mercalli.lild01.pictime.fr> Message-ID: Hi, lastuse was renamed to last_use. f.- On Mon, Jan 20, 2014 at 2:07 PM, Delacroix, Gauthier < Gauthier.Delacroix at coreye.fr> wrote: > Ok, it works fine for obj.hits, but obj.lastuse gives this : > > Message from VCC-compiler: > Unknown variable 'obj.lastuse' > At: ('common_end.vcl' Line 68 Pos 7) > set obj.lastuse = now; > ------###########------- > > Running VCC-compiler failed, exit 1 > > VCL compilation failed > > Gauthier > > -----Message d'origine----- > De : Poul-Henning Kamp [mailto:phk at phk.freebsd.dk] > Envoy? : lundi 20 janvier 2014 14:56 > ? : Delacroix, Gauthier > Cc : varnish-dev at varnish-cache.org > Objet : Re: 4.0-tp1: obj.hits not incremented > > In message < > 5F530A9242E7F84F999DB40E0E268FBD45ADFA6A at mercalli.lild01.pictime.fr > >, "Delacroix, Gauthier" writes: > > >Ok, I should have read the new default.vcl...this is often a quick > >course of the new syntax... > > > >Thanks for the answer. Is there a risk in enabling them ? > > No. It will cost a fraction of your performance, but most people will > never notice. > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfheen at varnish-software.com Mon Jan 27 13:57:20 2014 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Mon, 27 Jan 2014 14:57:20 +0100 Subject: varnishctl? Message-ID: <20140127135720.GA4089@err.no> Hi all, This got triggered by bug #1415 which asks for init scripts to be able to verify you're not just breaking your cache on a restart. I can see the need for such a tool, but I think it should have a bit wider scope. Problems: - you don't want to restart Varnish when the configuration is obviously broken. - it should be possible to reload configurations without having to fiddle with vcl.load, vcl.use. Coming up with new names is surprisingly annoying. - Duplication of logic between RH and Debian init scripts. Varnish should probably be able to read a parameter file in some reasonable format, which will also make the transition to non-sysvinit inits easier. Suggestion 1: Write something like to apachectl. It knows how to run a config test, start and stop the management process. It makes it easy to run multiple instances (something that's more important to apache than us). It's a bit like a distro-independent init script, but has custom functions (configtest for instance) Suggestion 2: Extend varnishadm to not only be able to talk to varnishd, but also start and stop the management process, do a config test and so on. Probably also extend varnishd to be able to reload the VCL somehow and make it able to read parameters from a file. I'm willing to work on 1. I think it'd be fairly easy to do and keeps the existing tools doing one thing well. Input, opinions, etc? -- Tollef Fog Heen Technical lead | Varnish Software AS ??: +47 21 98 92 64 We Make Websites Fly! From lkarsten at varnish-software.com Mon Jan 27 14:07:30 2014 From: lkarsten at varnish-software.com (Lasse Karstensen) Date: Mon, 27 Jan 2014 15:07:30 +0100 Subject: Set bereq.http.connection:close in vcl_pipe by default Message-ID: <20140127140729.GA27173@immer.varnish-software.com> I'm going through the documentation, and came over our handling of "bereq.http.Connection: close" in vcl_pipe. In the builtin VCL (previously default.vcl) there is a commented out line that the administrator can enable if they want to only do a single request per connection. What I'm struggling with is that we have a good amount of help requests and tickets because of this. I think many people must set connection:close for pipe to work as they expect. (possibly overestimated by me.) I don't know the history here, but I don't quite understand why we don't flip this default and save us/them the trouble? Downside is mainly TCP slow start and extra connection setup work on the backend. The crazy people doing SSH-tunneling or whatever must remember to unset the header if we do. Websocket ("bereq.http.Connection: Upgrade") handling must be done manually anyway, so they'll figure it out. On the upside we get (I think) less odd things to write documentation about. No more wrong-vhost-go-away if the browser sent a subsequent request for a different vhost that went through to the wrong backend. Fewer unecessary backend requests because the client asked a piped session for something that Varnish already could serve from cache. Request coalescing that works. Tollef had some input on how this should be handled in VCL, but I thought I'd run the main idea through the development team first. Any takers on the history here? Why do we do it this way? -- With regards, Lasse Karstensen Varnish Software AS From phk at phk.freebsd.dk Mon Jan 27 14:50:36 2014 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 27 Jan 2014 14:50:36 +0000 Subject: varnishctl? In-Reply-To: <20140127135720.GA4089@err.no> References: <20140127135720.GA4089@err.no> Message-ID: <20055.1390834236@critter.freebsd.dk> My feeling is that I have a hard time seeing the clear difference between the two in practice, and consequently I question why we would need or want two different tools. I don't particular care if the name is -adm or -ctl, but if we're going to add a -ctl, -adm should be renamed to something which more sensibly reflects what it does (-cli ?) As for the sanity-check, varnishd has no present '-c' argument and that could trivially be made a variant of -C which doesn't dump the C source code. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi.boukelmoune at zenika.com Mon Jan 27 15:10:14 2014 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Mon, 27 Jan 2014 16:10:14 +0100 Subject: varnishctl? In-Reply-To: <20140127135720.GA4089@err.no> References: <20140127135720.GA4089@err.no> Message-ID: Hi, On Mon, Jan 27, 2014 at 2:57 PM, Tollef Fog Heen wrote: > Hi all, > > This got triggered by bug #1415 which asks for init scripts to be able > to verify you're not just breaking your cache on a restart. I can see > the need for such a tool, but I think it should have a bit wider scope. > > Problems: > > - you don't want to restart Varnish when the configuration is > obviously broken. > > - it should be possible to reload configurations without having to > fiddle with vcl.load, vcl.use. Coming up with new names is > surprisingly annoying. > > - Duplication of logic between RH and Debian init scripts. Varnish > should probably be able to read a parameter file in some reasonable > format, which will also make the transition to non-sysvinit inits > easier. > > Suggestion 1: > > Write something like to apachectl. It knows how to run a config test, > start and stop the management process. It makes it easy to run > multiple instances (something that's more important to apache than > us). It's a bit like a distro-independent init script, but has custom > functions (configtest for instance) I find the idea compelling, but I've had issues with apachectl. The man page talks about a SysV mode [1], but apachectl uses different semantics. For instance, a restart for SysV [2] will actually stop the process, where apachectl will send a signal [3] to the daemon that will somewhat reload itself. I've found this really misleading, the first time I've had to use it. I really like the idea of a varnishctl that could be used downstream to build services on top of systemd, upstart etc. But I'd go for a simple pass-through command with simple sub-commands such as start/stop/restart/reload/configtest that would just add sugar such as "configtest on restart" and nothing more. On my Fedora laptop, apachectl has been rewritten to use systemd, so this problems would probably not occur. > Suggestion 2: > > Extend varnishadm to not only be able to talk to varnishd, but also > start and stop the management process, do a config test and so on. Would that mean 3 processes in varnishd ? > Probably also extend varnishd to be able to reload the VCL somehow and > make it able to read parameters from a file. > > I'm willing to work on 1. I think it'd be fairly easy to do and keeps > the existing tools doing one thing well. > > Input, opinions, etc? I like the idea of a varnishctl utility, but please don't make it as confusing as apachectl :) Dridi [1] https://httpd.apache.org/docs/2.4/programs/apachectl.html [2] can't find the docs, pretty sure restart must stop the process [3] https://httpd.apache.org/docs/2.4/en/stopping.html > -- > Tollef Fog Heen > Technical lead | Varnish Software AS > ??: +47 21 98 92 64 > We Make Websites Fly! > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev From geoff at uplex.de Mon Jan 27 16:03:45 2014 From: geoff at uplex.de (Geoff Simmons) Date: Mon, 27 Jan 2014 17:03:45 +0100 Subject: Set bereq.http.connection:close in vcl_pipe by default In-Reply-To: <20140127140729.GA27173@immer.varnish-software.com> References: <20140127140729.GA27173@immer.varnish-software.com> Message-ID: <52E68361.6040705@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 01/27/2014 03:07 PM, Lasse Karstensen wrote: > I'm going through the documentation, and came over our handling of > "bereq.http.Connection: close" in vcl_pipe. In the builtin VCL > (previously default.vcl) there is a commented out line that the > administrator can enable if they want to only do a single request > per connection. > > What I'm struggling with is that we have a good amount of help > requests and tickets because of this. I think many people must set > connection:close for pipe to work as they expect. (possibly > overestimated by me.) > > I don't know the history here, but I don't quite understand why we > don't flip this default and save us/them the trouble? [...] > Any takers on the history here? Why do we do it this way? I can't comment on the history, but I agree that 'Connection: close' is the reasonable default. I'm pretty sure that I've never had a situation where I've added return(pipe) but not the connection close. The most common use case is a large, lengthy upload or download for which no caching is necessary, and for that you have to end the session after it's done, if you want to VCL to keep working for whatever comes next. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Icedove - http://www.enigmail.net/ iQIcBAEBCAAGBQJS5oNUAAoJEOUwvh9pJNUR6bQQAId4ZJhgfPWgf2f5pcskpM+6 0aVyOyEyYFJRlK53kSjfLuf12YgnimV9FEzgFlm6ShGiifZ7JWD2vUIGSxZVBGu6 +O3AjrA3ctDmzIw6MC9QyBma1XBLDJADtpDc1Ie6LIrUvD4CxCJIc3166o0J4udk FSaOEjSYyoTGSnxfrH1EVBpQX4osT+xx75C/xkvibOn39M9OCd5RplF9kRYPLf5S gsxYtXJ6E9+h9nSsGE79tTD+Yy/dCpM+fCLKrskcIRcCXzrjJUXpv2lmIH0X//mp sx9yRLufarPvOZg5NlCjd+3gvGGIekTQI4/Yy+5K8QCTSKRQrONv6kUYQKgqo0Un oUCFH4mrlKJ8Nm1q9cac2+eYhKthdntYvv0DHqxKjTSDyJJYIFnzduHgOFOUs/MN pkUMTPnSFAtG5pg3ibS46pYTgxLjj4E9oGAZj3KpRspLvgrE1KUC9rHWwT4hMJ0w b28FH/hdv3ieHWRWQOpu2/EwXdPu+jpXjElbKTiKKJ3ZTW4kJM/3QjpAoJEoYDRF teJBZEJm2+grtD5o8W73rT0Uea3UT0LmHUd/VL48GmiuL8GVJg9AbJB9R0F+/X+t 0O6zu4nPHxkhcnz2HAhJwWq3s3tjucg7dM/s0hxeau4C3T61j6YhgMh8BAWFv7a8 IP8KbAYRawRueAgCAOya =DK0R -----END PGP SIGNATURE----- From bilbo at hobbiton.org Mon Jan 27 17:26:58 2014 From: bilbo at hobbiton.org (Leif Pedersen) Date: Mon, 27 Jan 2014 11:26:58 -0600 Subject: varnishctl? In-Reply-To: References: <20140127135720.GA4089@err.no> Message-ID: I'd like to add 2c if I may, merely as a long-time Apache and Varnish user. Whenever we hire someone who is at entry-level, they're inevitably confused by apachectl's start, stop, and restart commands for two reasons. First, on many OS's (FreeBSD in particular) you never want to use apachectl to start and stop because it's only there for when the OS lacks a service control scheme such as "service apache start". Using apachectl will often load the wrong config or start only one of several instances. I'm guessing the easiest "solution" is to simply not implement start/stop/restart and let it stay a matter of OS integration. Second, most people equate "restart" to "stop then start", and apachectl's restart does something quite different. I like the suggestions in this thread, and I'd just suggest avoiding adding that same type of confusion to Varnish. On Mon, Jan 27, 2014 at 9:10 AM, Dridi Boukelmoune < dridi.boukelmoune at zenika.com> wrote: > Hi, > > On Mon, Jan 27, 2014 at 2:57 PM, Tollef Fog Heen > wrote: > > Hi all, > > > > This got triggered by bug #1415 which asks for init scripts to be able > > to verify you're not just breaking your cache on a restart. I can see > > the need for such a tool, but I think it should have a bit wider scope. > > > > Problems: > > > > - you don't want to restart Varnish when the configuration is > > obviously broken. > > > > - it should be possible to reload configurations without having to > > fiddle with vcl.load, vcl.use. Coming up with new names is > > surprisingly annoying. > > > > - Duplication of logic between RH and Debian init scripts. Varnish > > should probably be able to read a parameter file in some reasonable > > format, which will also make the transition to non-sysvinit inits > > easier. > > > > Suggestion 1: > > > > Write something like to apachectl. It knows how to run a config test, > > start and stop the management process. It makes it easy to run > > multiple instances (something that's more important to apache than > > us). It's a bit like a distro-independent init script, but has custom > > functions (configtest for instance) > > I find the idea compelling, but I've had issues with apachectl. > > The man page talks about a SysV mode [1], but apachectl uses different > semantics. For instance, a restart for SysV [2] will actually stop the > process, where apachectl will send a signal [3] to the daemon that > will somewhat reload itself. I've found this really misleading, the > first time I've had to use it. > > I really like the idea of a varnishctl that could be used downstream > to build services on top of systemd, upstart etc. But I'd go for a > simple pass-through command with simple sub-commands such as > start/stop/restart/reload/configtest that would just add sugar such as > "configtest on restart" and nothing more. > > On my Fedora laptop, apachectl has been rewritten to use systemd, so > this problems would probably not occur. > > > Suggestion 2: > > > > Extend varnishadm to not only be able to talk to varnishd, but also > > start and stop the management process, do a config test and so on. > > Would that mean 3 processes in varnishd ? > > > Probably also extend varnishd to be able to reload the VCL somehow and > > make it able to read parameters from a file. > > > > I'm willing to work on 1. I think it'd be fairly easy to do and keeps > > the existing tools doing one thing well. > > > > Input, opinions, etc? > > I like the idea of a varnishctl utility, but please don't make it as > confusing as apachectl :) > > Dridi > > [1] https://httpd.apache.org/docs/2.4/programs/apachectl.html > [2] can't find the docs, pretty sure restart must stop the process > [3] https://httpd.apache.org/docs/2.4/en/stopping.html > > > -- > > Tollef Fog Heen > > Technical lead | Varnish Software AS > > ??: +47 21 98 92 64 > > We Make Websites Fly! > > > > _______________________________________________ > > varnish-dev mailing list > > varnish-dev at varnish-cache.org > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -- As implied by email protocols, the information in this message is not confidential. Any middle-man or recipient may inspect, modify, copy, forward, reply to, delete, or filter email for any purpose unless said parties are otherwise obligated. As the sender, I acknowledge that I have a lower expectation of the control and privacy of this message than I would a post-card. Further, nothing in this message is legally binding without cryptographic evidence of its integrity. http://bilbo.hobbiton.org/wiki/Eat_My_Sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfheen at varnish-software.com Tue Jan 28 07:03:07 2014 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 28 Jan 2014 08:03:07 +0100 Subject: Set bereq.http.connection:close in vcl_pipe by default In-Reply-To: <20140127140729.GA27173@immer.varnish-software.com> References: <20140127140729.GA27173@immer.varnish-software.com> Message-ID: <20140128070307.GB4089@err.no> ]] Lasse Karstensen > What I'm struggling with is that we have a good amount of help requests and > tickets because of this. I think many people must set connection:close for > pipe to work as they expect. (possibly overestimated by me.) > > I don't know the history here, but I don't quite understand why we don't > flip this default and save us/them the trouble? >From what I see in the wild today, there are basically two valid reasons for using pipe: websockets and working around two bugs in varnish: We don't handle chunked from clients and we don't handle large uploads well (they end up hitting sess_timeout and then we restart with half a POST and it goes rapidly downhill). > Tollef had some input on how this should be handled in VCL, but I thought > I'd run the main idea through the development team first. I'm not opposed to adding the header, but I think it should then be done in the C code running before vcl_pipe so you can remove it if you need to. Adding it to the built-in vcl_pipe means the only way to get rid of it is to return early, something we tell people to avoid for other VCL functions. -- Tollef Fog Heen Technical lead | Varnish Software AS ??: +47 21 98 92 64 We Make Websites Fly! From tfheen at varnish-software.com Tue Jan 28 07:07:53 2014 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 28 Jan 2014 08:07:53 +0100 Subject: varnishctl? In-Reply-To: References: <20140127135720.GA4089@err.no> Message-ID: <20140128070753.GC4089@err.no> ]] Dridi Boukelmoune > Hi, > > On Mon, Jan 27, 2014 at 2:57 PM, Tollef Fog Heen > wrote: > > Hi all, > > > > This got triggered by bug #1415 which asks for init scripts to be able > > to verify you're not just breaking your cache on a restart. I can see > > the need for such a tool, but I think it should have a bit wider scope. > > > > Problems: > > > > - you don't want to restart Varnish when the configuration is > > obviously broken. > > > > - it should be possible to reload configurations without having to > > fiddle with vcl.load, vcl.use. Coming up with new names is > > surprisingly annoying. > > > > - Duplication of logic between RH and Debian init scripts. Varnish > > should probably be able to read a parameter file in some reasonable > > format, which will also make the transition to non-sysvinit inits > > easier. > > > > Suggestion 1: > > > > Write something like to apachectl. It knows how to run a config test, > > start and stop the management process. It makes it easy to run > > multiple instances (something that's more important to apache than > > us). It's a bit like a distro-independent init script, but has custom > > functions (configtest for instance) > > I find the idea compelling, but I've had issues with apachectl. Your problems seem to be "apachectl has bugs", and I wrote "something like apachectl", meaning that we'll implement a different set of bugs. :-) > I really like the idea of a varnishctl that could be used downstream > to build services on top of systemd, upstart etc. But I'd go for a > simple pass-through command with simple sub-commands such as > start/stop/restart/reload/configtest that would just add sugar such as > "configtest on restart" and nothing more. In the stack, I'd want to have the init scripts call varnishctl, not the other way around. > > Suggestion 2: > > > > Extend varnishadm to not only be able to talk to varnishd, but also > > start and stop the management process, do a config test and so on. > > Would that mean 3 processes in varnishd ? No, that wasn't my intention. What would the third one be? -- Tollef Fog Heen Technical lead | Varnish Software AS ??: +47 21 98 92 64 We Make Websites Fly! From dridi.boukelmoune at zenika.com Tue Jan 28 07:18:55 2014 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Tue, 28 Jan 2014 08:18:55 +0100 Subject: varnishctl? In-Reply-To: <20140128070753.GC4089@err.no> References: <20140127135720.GA4089@err.no> <20140128070753.GC4089@err.no> Message-ID: On Tue, Jan 28, 2014 at 8:07 AM, Tollef Fog Heen wrote: > ]] Dridi Boukelmoune > >> Hi, >> >> On Mon, Jan 27, 2014 at 2:57 PM, Tollef Fog Heen >> wrote: >> > Hi all, >> > >> > This got triggered by bug #1415 which asks for init scripts to be able >> > to verify you're not just breaking your cache on a restart. I can see >> > the need for such a tool, but I think it should have a bit wider scope. >> > >> > Problems: >> > >> > - you don't want to restart Varnish when the configuration is >> > obviously broken. >> > >> > - it should be possible to reload configurations without having to >> > fiddle with vcl.load, vcl.use. Coming up with new names is >> > surprisingly annoying. >> > >> > - Duplication of logic between RH and Debian init scripts. Varnish >> > should probably be able to read a parameter file in some reasonable >> > format, which will also make the transition to non-sysvinit inits >> > easier. >> > >> > Suggestion 1: >> > >> > Write something like to apachectl. It knows how to run a config test, >> > start and stop the management process. It makes it easy to run >> > multiple instances (something that's more important to apache than >> > us). It's a bit like a distro-independent init script, but has custom >> > functions (configtest for instance) >> >> I find the idea compelling, but I've had issues with apachectl. > > Your problems seem to be "apachectl has bugs", and I wrote "something > like apachectl", meaning that we'll implement a different set of bugs. > :-) As long as the bugs are well documented ;-) >> I really like the idea of a varnishctl that could be used downstream >> to build services on top of systemd, upstart etc. But I'd go for a >> simple pass-through command with simple sub-commands such as >> start/stop/restart/reload/configtest that would just add sugar such as >> "configtest on restart" and nothing more. > > In the stack, I'd want to have the init scripts call varnishctl, not the > other way around. Sorry, this is what I meant. SysVinit, systemd, upstart and the like that would use varnishctl. By downstream I mean distributions since they mean you as an upstream of their varnish packages. >> > Suggestion 2: >> > >> > Extend varnishadm to not only be able to talk to varnishd, but also >> > start and stop the management process, do a config test and so on. >> >> Would that mean 3 processes in varnishd ? > > No, that wasn't my intention. What would the third one be? I don't know ; if varnishd has only two processes as it currently has, how would you stop the management process ? I understand the feature to be similar to the current stop/start of the child process through varnishadm. Dridi From martin at varnish-software.com Wed Jan 29 16:22:45 2014 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Wed, 29 Jan 2014 11:22:45 -0500 Subject: Varnishtop on the new logging API In-Reply-To: References: <529C9F97.60209@smartjog.com> <17463.1389261712@critter.freebsd.dk> <52CED1DA.7090404@smartjog.com> Message-ID: Hi, This patch has now been merged. Thanks for the contribution! If you'd like to have a look at varnishhist too that would be much appreciated. Have a look at the commit history for some pointers as to missing items in the original patch. Regards, Martin Blix Grydeland On 13 January 2014 09:55, Martin Blix Grydeland wrote: > Hi Guillaume, > > I'll come back to you with a proper review and reply to this some time > next week. This week I am away on a business trip all week and connectivity > will be mostly offline. > > Martin > 9. jan. 2014 16:44 skrev "Guillaume Quintard" < > guillaume.quintard at smartjog.com> f?lgende: > > Ah, I was wondering about it too. Martin, feel free to give me pointer >> about what you want changed, I see a few spots that could be more elegant >> or consistent, but you'll have probably more to say. >> >> On 01/09/2014 05:41 PM, Martin Blix Grydeland wrote: >> >> Not looked at it in detail yet, though first impression looks good. Will >> be going in with some tweaks. Thanks for the reminder though :-) >> >> Main >> 9. jan. 2014 17:23 skrev "Poul-Henning Kamp" >> f?lgende: >> >>> >>> >As said on IRC, here's a shot at porting varnishtop on the new logging >>> >API. I didn't change much, maybe not enough (RB tree freeing?), but it >>> >seems to work. >>> >>> I'm wading through my inbox, did Martin look at this patch ? >>> >>> -- >>> Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 >>> phk at FreeBSD.ORG | TCP/IP since RFC 956 >>> FreeBSD committer | BSD since 4.3-tahoe >>> Never attribute to malice what can adequately be explained by >>> incompetence. >>> >>> _______________________________________________ >>> varnish-dev mailing list >>> varnish-dev at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev >>> >> >> >> -- >> Guillaume Quintard >> >> -- *Martin Blix Grydeland* Senior Developer | Varnish Software AS Cell: +47 21 98 92 60 We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfheen at varnish-software.com Thu Jan 30 07:44:37 2014 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Thu, 30 Jan 2014 08:44:37 +0100 Subject: varnishctl? In-Reply-To: References: <20140127135720.GA4089@err.no> <20140128070753.GC4089@err.no> Message-ID: <20140130074437.GA10577@err.no> ]] Dridi Boukelmoune > On Tue, Jan 28, 2014 at 8:07 AM, Tollef Fog Heen > wrote: > > ]] Dridi Boukelmoune > > > > > I find the idea compelling, but I've had issues with apachectl. > > > > Your problems seem to be "apachectl has bugs", and I wrote "something > > like apachectl", meaning that we'll implement a different set of bugs. > > :-) > > As long as the bugs are well documented ;-) We'll do our best. > >> I really like the idea of a varnishctl that could be used downstream > >> to build services on top of systemd, upstart etc. But I'd go for a > >> simple pass-through command with simple sub-commands such as > >> start/stop/restart/reload/configtest that would just add sugar such as > >> "configtest on restart" and nothing more. > > > > In the stack, I'd want to have the init scripts call varnishctl, not the > > other way around. > > Sorry, this is what I meant. SysVinit, systemd, upstart and the like > that would use varnishctl. By downstream I mean distributions since > they mean you as an upstream of their varnish packages. Ah, right, then we're in agreement where it would fit. > >> > Suggestion 2: > >> > > >> > Extend varnishadm to not only be able to talk to varnishd, but also > >> > start and stop the management process, do a config test and so on. > >> > >> Would that mean 3 processes in varnishd ? > > > > No, that wasn't my intention. What would the third one be? > > I don't know ; if varnishd has only two processes as it currently has, > how would you stop the management process ? I understand the feature > to be similar to the current stop/start of the child process through > varnishadm. In this case, varnishadm would send a sigint to varnishd (the management process) or tell it to exit through the CLI and then wait for it to exit before returning. -- Tollef Fog Heen Technical lead | Varnish Software AS ??: +47 21 98 92 64 We Make Websites Fly! From tfheen at varnish-software.com Thu Jan 30 07:48:48 2014 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Thu, 30 Jan 2014 08:48:48 +0100 Subject: [PATCH] [EXPERIMENTAL] autocrap autohardening In-Reply-To: <52CEDCFE.2090508@schokola.de> References: <52CEDCFE.2090508@schokola.de> Message-ID: <20140130074848.GB10577@err.no> ]] Nils Goroll > I stumbled over http://mainisusuallyafunction.blogspot.de/2012/05/automatic-binary-hardening-with.html > today and thought to give it a try with varnish. I think this is a good idea, just a minor comment: > +# We use the same hardening flags for C. We must check that each flag > +# is supported by both compilers. > +AC_DEFUN([check_cc_flag], > + [AX_CHECK_COMPILE_FLAG([$1], [$2], [$3], [-Werror $4])]) > +AC_DEFUN([check_link_flag], > + [AX_CHECK_LINK_FLAG([$1], [$2], [$3], [-Werror $4])]) auto* function names are generally uppercase, so uppercase those too, please? -- Tollef Fog Heen Technical lead | Varnish Software AS ??: +47 21 98 92 64 We Make Websites Fly! From tfheen at varnish-software.com Thu Jan 30 07:49:50 2014 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Thu, 30 Jan 2014 08:49:50 +0100 Subject: [PATCH] fix build with --enable-developer-warnings and gcc3.4 In-Reply-To: <52CEE55D.4050108@schokola.de> References: <52CEB8A6.1050401@schokola.de> <52CEE55D.4050108@schokola.de> Message-ID: <20140130074950.GC10577@err.no> ]] Nils Goroll > >d9492da540430103b9dcd6844eb59b508c061d66 effectively breaks builds with gcc 3.4. > > ... only when developer warnings are enabled. > > Patch attached. I think this looks reasonable, so feel free to commit. -- Tollef Fog Heen Technical lead | Varnish Software AS ??: +47 21 98 92 64 We Make Websites Fly! From tfheen at varnish-software.com Thu Jan 30 07:53:19 2014 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Thu, 30 Jan 2014 08:53:19 +0100 Subject: varnishtest documentation In-Reply-To: <20131221164129.1509754fm0tueiw4@art.yamanotecorporation.com> References: <20131221140238.61593pibn6wz20w0@art.yamanotecorporation.com> <65579.1387664413@critter.freebsd.dk> <20131221164129.1509754fm0tueiw4@art.yamanotecorporation.com> Message-ID: <20140130075319.GD10577@err.no> ]] varnish at syrota.com > Quoting Poul-Henning Kamp : > > >I trust it you have spotted the 347 test-descriptions in the > >source tree ? They're probably the main documentation for it. > > Yeah, saw them, and that was one of the main sources on how to use > the features. Although some of the arguments I could never find in > those test cases > > >The main reason varnishtest has not been documented, is that it > >we weren't sure if anybody would ever bother read the documentation > >if we wrote it, and given that we have more ideas for things to > >do than time to do them it ended up somewhat low in the pile. > > Yeah, judging by how hard it is to come by any discussion of it, it > is quite rarely used feature, so the priority is understandable :) It's used a bit by vmods, so I think we should aim to have some documentation for it, if nothing else for the "standard" behaviour. People using it are likely to be C developers, so saying "there are more flags, see the source", while not terribly friendly, should not pose too many problems in practice. > >A quick top-level architectural overview: > > Thanks for the overview. That helps with a few things that I had not > understood before. I'll probably put together a pull request for > this architecture overview (I'll try to fit it into the existing > varnishtest page) and vtc commands. And we can go from there, if > anyone will be interested. Did you get anywhere with this? -- Tollef Fog Heen Technical lead | Varnish Software AS ??: +47 21 98 92 64 We Make Websites Fly!