From perbu at varnish-software.com Sat Mar 1 17:06:50 2014 From: perbu at varnish-software.com (Per Buer) Date: Sat, 1 Mar 2014 18:06:50 +0100 Subject: Cleaning up "man vcl" Message-ID: After two years of discussions I've started cleaning up the VCL man page. Since we now have a rather rich user guide the plan is to reduce the VCL man page to something mostly describing syntax - short and sweet. I've started doing this, but then describing the backend declaration I'm struggling finding a list of all the possible attributes. generate.py doesn't seem to deal with this. Where is the magic? -- *Per Buer* CTO | Varnish Software Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Global Award 2013 -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Mon Mar 3 10:23:19 2014 From: perbu at varnish-software.com (Per Buer) Date: Mon, 3 Mar 2014 11:23:19 +0100 Subject: Cleaning up "man vcl" In-Reply-To: References: Message-ID: On Sat, Mar 1, 2014 at 6:06 PM, Per Buer wrote: > After two years of discussions I've started cleaning up the VCL man page. > Since we now have a rather rich user guide the plan is to reduce the VCL > man page to something mostly describing syntax - short and sweet. > > I've started doing this, but then describing the backend declaration I'm > struggling finding a list of all the possible attributes. generate.py > doesn't seem to deal with this. Where is the magic? > Found it. It is set up by the vcc_FldSpec in libvcc. Gone over the structure with Martin. It's gonna be great. -- *Per Buer* CTO | Varnish Software Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Global Award 2013 -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Tue Mar 4 13:55:44 2014 From: perbu at varnish-software.com (Per Buer) Date: Tue, 4 Mar 2014 14:55:44 +0100 Subject: State of the docs Message-ID: I think I'm pretty happy with the state of the docs now. Unless someone can point to something (please do) I think I can declare the docs ready for release. The missing bits might be the release docs, changelog, etc. But that should not be too much work to spit out. -- *Per Buer* CTO | Varnish Software Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Global Award 2013 -------------- next part -------------- An HTML attachment was scrubbed... URL: From thierry.magnien at sfr.com Tue Mar 4 14:10:42 2014 From: thierry.magnien at sfr.com (MAGNIEN, Thierry) Date: Tue, 4 Mar 2014 14:10:42 +0000 Subject: State of the docs In-Reply-To: References: Message-ID: <5D103CE839D50E4CBC62C9FD7B83287C8FB2717B@EXCN013.encara.local.ads> Hi Per, Maybe add a vmod_directors doc (as for vmod_std) ? Regards, Thierry De?: varnish-dev-bounces+thierry.magnien=sfr.com at varnish-cache.org [mailto:varnish-dev-bounces+thierry.magnien=sfr.com at varnish-cache.org] De la part de Per Buer Envoy??: mardi 4 mars 2014 14:56 ??: Varnish Development Objet?: State of the docs I think I'm pretty happy with the state of the docs now. Unless someone can point to something (please do) I think I can declare the docs ready for release. The missing bits might be the release docs, changelog, etc. But that should not be too much work to spit out. -- Per Buer CTO | Varnish Software Phone:?+47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Global Award 2013 From perbu at varnish-software.com Wed Mar 5 13:29:20 2014 From: perbu at varnish-software.com (Per Buer) Date: Wed, 5 Mar 2014 14:29:20 +0100 Subject: State of the docs In-Reply-To: <5D103CE839D50E4CBC62C9FD7B83287C8FB2717B@EXCN013.encara.local.ads> References: <5D103CE839D50E4CBC62C9FD7B83287C8FB2717B@EXCN013.encara.local.ads> Message-ID: Thierry, This is now added. Someone should make sure it also shows up in the reference section of the online docs, not just the man page. Per. On Tue, Mar 4, 2014 at 3:10 PM, MAGNIEN, Thierry wrote: > Hi Per, > > Maybe add a vmod_directors doc (as for vmod_std) ? > > Regards, > Thierry > > De : varnish-dev-bounces+thierry.magnien=sfr.com at varnish-cache.org[mailto: > varnish-dev-bounces+thierry.magnien=sfr.com at varnish-cache.org] De la part > de Per Buer > Envoy? : mardi 4 mars 2014 14:56 > ? : Varnish Development > Objet : State of the docs > > I think I'm pretty happy with the state of the docs now. Unless someone > can point to something (please do) I think I can declare the docs ready for > release. > > The missing bits might be the release docs, changelog, etc. But that > should not be too much work to spit out. > > > -- > > Per Buer > CTO | Varnish Software > Phone: +47 958 39 117 | Skype: per.buer > We Make Websites Fly! > > Winner of the Red Herring Top 100 Global Award 2013 > > > > -- *Per Buer* CTO | Varnish Software Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Global Award 2013 -------------- next part -------------- An HTML attachment was scrubbed... URL: From thierry.magnien at sfr.com Thu Mar 6 10:31:49 2014 From: thierry.magnien at sfr.com (MAGNIEN, Thierry) Date: Thu, 6 Mar 2014 10:31:49 +0000 Subject: [PATCH] Make "done" function available in VCL Message-ID: <5D103CE839D50E4CBC62C9FD7B83287C8FB3024A@EXCN015.encara.local.ads> Hi, Here is a patch proposal in order to make "done" function available in VCL. It adds a "sub vcl_done" in VCL, which is called at beginning of cnt_done function. It allows to call a VMOD at the end of a request, whatever the status is (success, fail, timeout, etc.). Tested on 3.0.5. Best regards, Thierry diff -r 8f2ba576d90c bin/varnishd/cache_center.c --- a/bin/varnishd/cache_center.c Thu Mar 06 11:31:03 2014 +0100 +++ b/bin/varnishd/cache_center.c Thu Mar 06 11:31:09 2014 +0100 @@ -311,6 +311,8 @@ CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); CHECK_OBJ_ORNULL(sp->vcl, VCL_CONF_MAGIC); + VCL_done_method(sp); + AZ(sp->obj); AZ(sp->vbc); sp->director = NULL; diff -r 8f2ba576d90c lib/libvcl/generate.py --- a/lib/libvcl/generate.py Thu Mar 06 11:31:03 2014 +0100 +++ b/lib/libvcl/generate.py Thu Mar 06 11:31:09 2014 +0100 @@ -92,6 +92,7 @@ ('fetch', ('error', 'restart', 'hit_for_pass', 'deliver',)), ('deliver', ('restart', 'deliver',)), ('error', ('restart', 'deliver',)), + ('done', ('ok',)), ('init', ('ok',)), ('fini', ('ok',)), ) From thierry.magnien at sfr.com Fri Mar 7 15:37:41 2014 From: thierry.magnien at sfr.com (MAGNIEN, Thierry) Date: Fri, 7 Mar 2014 15:37:41 +0000 Subject: [PATCH] Make "done" function available in VCL Message-ID: <5D103CE839D50E4CBC62C9FD7B83287C8FB399AD@EXCN015.encara.local.ads> Update : previous patch generates a segfault during varnishtest while building modules. Not sure the fix is perfect but it seems to work. Here is the updated patch: --- varnish-3.0.5/bin/varnishd/cache_center.c +++ varnish-3.0.5.new/bin/varnishd/cache_center.c @@ -332,6 +332,10 @@ cnt_done(struct sess *sp) CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); CHECK_OBJ_ORNULL(sp->vcl, VCL_CONF_MAGIC); + if (sp->fd >= 0) { + VCL_done_method(sp); + } + AZ(sp->obj); AZ(sp->vbc); sp->director = NULL; --- varnish-3.0.5/lib/libvcl/generate.py 2013-12-02 08:47:57.000000000 +0100 +++ varnish-3.0.5.new/lib/libvcl/generate.py 2014-03-06 11:03:48.000000000 +0100 @@ -92,6 +92,7 @@ ('fetch', ('error', 'restart', 'hit_for_pass', 'deliver',)), ('deliver', ('restart', 'deliver',)), ('error', ('restart', 'deliver',)), + ('done', ('ok',)), ('init', ('ok',)), ('fini', ('ok',)), ) --- varnish-3.0.5/bin/varnishd/default.vcl +++ varnish-3.0.5.new/bin/varnishd/default.vcl @@ -147,3 +147,7 @@ sub vcl_init { sub vcl_fini { return (ok); } + +sub vcl_done { + return (ok); +} From ingvar at redpill-linpro.com Mon Mar 10 09:26:13 2014 From: ingvar at redpill-linpro.com (Ingvar Hagelund) Date: Mon, 10 Mar 2014 10:26:13 +0100 Subject: Trivia: Fixing non-existing dates in varnish rpm [patch] Message-ID: <531D8535.9040408@redpill-linpro.com> Ingvar poking his head through the window, scattering a few bits on the floor, and runs: There are a few errnous dates in the old specfile entries that gives warnings on fedora. This trivial patch fixes them, and the dates are checked to be accurate for the actual commits to fedora. Ingvar -------------- next part -------------- A non-text attachment was scrubbed... Name: varnish_fix_rpm_changelog_dates.patch Type: text/x-patch Size: 1562 bytes Desc: not available URL: From fgsch at lodoss.net Tue Mar 11 17:38:04 2014 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Tue, 11 Mar 2014 17:38:04 +0000 Subject: [PATCH] Change vmod headers path Message-ID: <20140311173804.GA29521@nitzer> As discussed on irc this moves the headers under ${prefix}/include/varnish/vmod. Tested with libvmod-curl. f.- bin/varnishd/Makefile.am | 4 ++-- include/Makefile.am | 4 ++-- varnishapi-uninstalled.pc.in | 1 - varnishapi.pc.in | 3 +-- 4 files changed, 5 insertions(+), 7 deletions(-) diff --git a/bin/varnishd/Makefile.am b/bin/varnishd/Makefile.am index d9a8e62..f5d8579 100644 --- a/bin/varnishd/Makefile.am +++ b/bin/varnishd/Makefile.am @@ -100,8 +100,8 @@ noinst_HEADERS = \ waiter/waiter.h # Headers for use with vmods -pkgdataincludedir = $(pkgdatadir)/include -nobase_pkgdatainclude_HEADERS = \ +vmodincludedir = $(pkgincludedir)/vmod +nobase_vmodinclude_HEADERS = \ cache/cache.h \ common/common.h \ common/params.h diff --git a/include/Makefile.am b/include/Makefile.am index 3813836..b8deca6 100644 --- a/include/Makefile.am +++ b/include/Makefile.am @@ -67,8 +67,8 @@ nobase_noinst_HEADERS = \ vut_options.h # Headers for use with vmods -pkgdataincludedir = $(pkgdatadir)/include -nobase_pkgdatainclude_HEADERS = \ +vmodincludedir = $(pkgincludedir)/vmod +nobase_vmodinclude_HEADERS = \ miniobj.h \ vas.h \ vav.h \ diff --git a/varnishapi-uninstalled.pc.in b/varnishapi-uninstalled.pc.in index 3a8f744..d252289 100644 --- a/varnishapi-uninstalled.pc.in +++ b/varnishapi-uninstalled.pc.in @@ -8,7 +8,6 @@ pkgincludedir=${includedir}/@PACKAGE@ datarootdir=@datarootdir@ datadir=@datadir@ pkgdatadir=${datadir}/@PACKAGE@ -pkgdataincludedir=${pkgdatadir}/include vmoddir=${libdir}/@PACKAGE@/vmods builddir=@abs_top_builddir@ srcdir=@abs_top_srcdir@ diff --git a/varnishapi.pc.in b/varnishapi.pc.in index 059725e..7e7603c 100644 --- a/varnishapi.pc.in +++ b/varnishapi.pc.in @@ -8,8 +8,7 @@ pkgincludedir=${includedir}/@PACKAGE@ datarootdir=@datarootdir@ datadir=@datadir@ pkgdatadir=${datadir}/@PACKAGE@ -pkgdataincludedir=${pkgdatadir}/include -vmodincludedir=${pkgdataincludedir} +vmodincludedir=${pkgincludedir}/vmod vmoddir=${libdir}/@PACKAGE@/vmods vmodtool=${pkgdatadir}/vmodtool.py From fgsch at lodoss.net Tue Mar 11 18:28:13 2014 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Tue, 11 Mar 2014 18:28:13 +0000 Subject: [PATCH] Log FetchError on vcl_pipe if no backend is avail Message-ID: <20140311182813.GB29521@nitzer> Log on error rather than silently returning. Same as in V1F_fetch_hdr(). f.- bin/varnishd/cache/cache_pipe.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/bin/varnishd/cache/cache_pipe.c b/bin/varnishd/cache/cache_pipe.c index b4403bb..a74f745 100644 --- a/bin/varnishd/cache/cache_pipe.c +++ b/bin/varnishd/cache/cache_pipe.c @@ -74,8 +74,10 @@ PipeRequest(struct req *req, struct busyobj *bo) CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC); vc = VDI_GetFd(bo); - if (vc == NULL) + if (vc == NULL) { + VSLb(bo->vsl, SLT_FetchError, "no backend connection"); return; + } bo->vbc = vc; /* For panic dumping */ (void)VTCP_blocking(vc->fd); From martin at varnish-software.com Fri Mar 14 14:35:26 2014 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 14 Mar 2014 15:35:26 +0100 Subject: ReqEnd (Varnish 4 timing information revisited) Message-ID: Hi, I'm working on a patch to improve the timing information in the VSL records. Basis is the discussions of last VDD (which is summarized here https://www.varnish-cache.org/trac/wiki/VDD13Q4). When doing this, I found that the ideas from there didn't sufficiently take into account the complexity of restarts in VCL. A single request can cause any number of objects to be fetched into the cache, and each of these will have interesting timing information. From experience I know that timing can be a very valuable debugging tool, and having good timing should be a goal. So instead of the single ReqEnd for all timing data, I believe that it should be split into two records. Timing for anything up until the reply is being sent will be in a new record tentatively named ReqProc (request processing). The fields will include time spent on waiting lists, time spent fetching the request body and time spent fetching objects. The ReqProc record will be logged also on restarts, so the log will show the intermediate timing of the processing happened until that time. SLTM(ReqProc, 0, "Request processing timing", "Contains timing information from the processing of the request.\n\n" "The format is::\n\n" "\t%f %f %f %f %f\n" "\t| | | | |\n" "\t| | | | +- Time spent processing and fetching\n" "\t| | | +---- Time spent reading the request body\n" "\t| | +------- Time spent on waiting lists\n" "\t| +---------- Total time spent processing the request\n" "\t+------------- Timestamp (since epoch) when vcl_recv was called\n" "\n" ) The ReqEnd will be output only once per request processed, and will only be output from the log transaction that finishes the request handling (restarts creates a new log transaction). It will contain the time_to_first_byte (which is equal to the sum of the previous ReqProc records), as well as the transmit time (which would include any ESI processing or fetch time when streaming). The details of ESI processing timing would be found in the ReqProc and ReqEnd log records of the ESI subrequest transactions. Details of background fetch timing would be in the BereqEnd record. SLTM(ReqEnd, 0, "Client request end timing information", "Marks the end of a client request. Contains timing information" " from the complete handling of the request.\n\n" "The format is::\n\n" "\t%f %f %f %f %f\n" "\t| | | | |\n" "\t| | | | +- Time to transmit response (plus ESI processing)\n" "\t| | | +---- Time to process request (sum of ReqProc)\n" "\t| | +------- Total time to finish request handling\n" "\t| +---------- Timestamp (since epoch) when the request ended\n" "\t+------------- Timestamp (since epoch) when the request started\n" "\n" ) Comments much appreciated. Martin -- *Martin Blix Grydeland* Senior Developer | Varnish Software AS Cell: +47 21 98 92 60 We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume.quintard at smartjog.com Fri Mar 14 15:26:16 2014 From: guillaume.quintard at smartjog.com (Guillaume Quintard) Date: Fri, 14 Mar 2014 16:26:16 +0100 Subject: Announcing Varnish Developer Day 2014Q1 (2014-03-18) In-Reply-To: <53047F55.4060404@smartjog.com> References: <20140219092743.GA21990@immer.varnish-software.com> <53047F55.4060404@smartjog.com> Message-ID: <53231F98.6000504@smartjog.com> The day is approaching fast :-) Should you need it, my phone number is +33 6 61 80 45 56. Don't hesitate to call if you have problems in Paris. I think some people will be flying in on monday, so we could have dinner together and possibly a few drinks afterwards, for those willing. See you in a few days1 -- Guillaume Quintard From lkarsten at varnish-software.com Mon Mar 17 09:03:56 2014 From: lkarsten at varnish-software.com (Lasse Karstensen) Date: Mon, 17 Mar 2014 10:03:56 +0100 Subject: Announcing Varnish Developer Day 2014Q1 (2014-03-18) In-Reply-To: <53231F98.6000504@smartjog.com> References: <20140219092743.GA21990@immer.varnish-software.com> <53047F55.4060404@smartjog.com> <53231F98.6000504@smartjog.com> Message-ID: <20140317090355.GB27277@immer.varnish-software.com> On Fri, Mar 14, 2014 at 04:26:16PM +0100, Guillaume Quintard wrote: > The day is approaching fast :-) [..] > I think some people will be flying in on monday, so we could have dinner > together and possibly a few drinks afterwards, for those willing. The Varnish Software AS crowd (martin, daghf and me) are flying tonight and will be at Kyriad Hotel Paris Porte d'Ivry around 21:00. We're happy to meet up for food and drinks with anyone else arriving tonight. -- With regards, Lasse Karstensen From guillaume.quintard at smartjog.com Mon Mar 17 09:17:06 2014 From: guillaume.quintard at smartjog.com (Guillaume Quintard) Date: Mon, 17 Mar 2014 10:17:06 +0100 Subject: Announcing Varnish Developer Day 2014Q1 (2014-03-18) In-Reply-To: <20140317090355.GB27277@immer.varnish-software.com> References: <20140219092743.GA21990@immer.varnish-software.com> <53047F55.4060404@smartjog.com> <53231F98.6000504@smartjog.com> <20140317090355.GB27277@immer.varnish-software.com> Message-ID: <5326BD92.2010705@smartjog.com> On 03/17/2014 10:03 AM, Lasse Karstensen wrote: > The Varnish Software AS crowd (martin, daghf and me) are flying > tonight and will be at Kyriad Hotel Paris Porte d'Ivry around 21:00. > We're happy to meet up for food and drinks with anyone else arriving > tonight. Great, I'll find a restaurant around it (I guess you'll be hungry). By the way, due to pollution concerns, public transportation is free in paris, and I think it includes the airports as well, so you can probably take the train then the tram and save up on taxi fare if you wish, but the taxi should be quicker, considering your time of arrival. -- Guillaume Quintard From fgsch at lodoss.net Mon Mar 17 09:53:36 2014 From: fgsch at lodoss.net (Federico Schwindt) Date: Mon, 17 Mar 2014 09:53:36 +0000 Subject: Announcing Varnish Developer Day 2014Q1 (2014-03-18) In-Reply-To: <20140317090355.GB27277@immer.varnish-software.com> References: <20140219092743.GA21990@immer.varnish-software.com> <53047F55.4060404@smartjog.com> <53231F98.6000504@smartjog.com> <20140317090355.GB27277@immer.varnish-software.com> Message-ID: Same here, count me in for dinner! On Mon, Mar 17, 2014 at 9:03 AM, Lasse Karstensen < lkarsten at varnish-software.com> wrote: > On Fri, Mar 14, 2014 at 04:26:16PM +0100, Guillaume Quintard wrote: > > The day is approaching fast :-) > [..] > > I think some people will be flying in on monday, so we could have dinner > > together and possibly a few drinks afterwards, for those willing. > > The Varnish Software AS crowd (martin, daghf and me) are flying tonight > and will be at Kyriad Hotel Paris Porte d'Ivry around 21:00. > > We're happy to meet up for food and drinks with anyone else arriving > tonight. > > -- > With regards, > Lasse Karstensen > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi.boukelmoune at zenika.com Mon Mar 17 10:09:08 2014 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Mon, 17 Mar 2014 11:09:08 +0100 Subject: Announcing Varnish Developer Day 2014Q1 (2014-03-18) In-Reply-To: References: <20140219092743.GA21990@immer.varnish-software.com> <53047F55.4060404@smartjog.com> <53231F98.6000504@smartjog.com> <20140317090355.GB27277@immer.varnish-software.com> Message-ID: Me too! On Mon, Mar 17, 2014 at 10:53 AM, Federico Schwindt wrote: > Same here, count me in for dinner! > > > On Mon, Mar 17, 2014 at 9:03 AM, Lasse Karstensen > wrote: >> >> On Fri, Mar 14, 2014 at 04:26:16PM +0100, Guillaume Quintard wrote: >> > The day is approaching fast :-) >> [..] >> > I think some people will be flying in on monday, so we could have dinner >> > together and possibly a few drinks afterwards, for those willing. >> >> The Varnish Software AS crowd (martin, daghf and me) are flying tonight >> and will be at Kyriad Hotel Paris Porte d'Ivry around 21:00. >> >> We're happy to meet up for food and drinks with anyone else arriving >> tonight. >> >> -- >> With regards, >> Lasse Karstensen >> >> _______________________________________________ >> varnish-dev mailing list >> varnish-dev at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > > > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev From guillaume.quintard at smartjog.com Mon Mar 17 11:01:12 2014 From: guillaume.quintard at smartjog.com (Guillaume Quintard) Date: Mon, 17 Mar 2014 12:01:12 +0100 Subject: Announcing Varnish Developer Day 2014Q1 (2014-03-18) In-Reply-To: References: <20140219092743.GA21990@immer.varnish-software.com> <53047F55.4060404@smartjog.com> <53231F98.6000504@smartjog.com> <20140317090355.GB27277@immer.varnish-software.com>

Message-ID: <5326D5F8.9060705@smartjog.com> Awesome, I propose Chez Papa 27 Rue de la Colonie 75013 Paris they serve recipes from the south west of France, if it suits everybody. >From the Kyriad hotel, you can take the tram (free!) and hop off at "Poterne des peupliers", or "Stade Charl?ty". Shall we say 21h30? I can wait for our foreign friends at the hotel so we can go there together if they wish so. -- Guillaume Quintard From perbu at varnish-software.com Tue Mar 18 13:47:39 2014 From: perbu at varnish-software.com (Per Buer) Date: Tue, 18 Mar 2014 14:47:39 +0100 Subject: building the directors manpage Message-ID: I don't know shit about Automake and I'm rusty as hell when it comes to make, but I've some basic copy and paste in order to get the man page for vmod_directors to build like the one for the vmod_std. We need this done to release. However, I've failed. Halp. Here is what I've tried (patch attached). It fails with: make[3]: *** No rule to make target `../../doc/sphinx/reference/vmod_directors.rst', needed by `vmod_directors.3'. Stop. -- *Per Buer* CTO | Varnish Software Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Global Award 2013 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: directors-man-page-build.patch Type: application/octet-stream Size: 1086 bytes Desc: not available URL: From slink at schokola.de Tue Mar 18 16:16:45 2014 From: slink at schokola.de (Nils Goroll) Date: Tue, 18 Mar 2014 17:16:45 +0100 Subject: Supported Platforms for Varnish 4 Message-ID: <5328716D.9010808@schokola.de> Hi, we'd like to inform you about one result of today's Varnish Developer Day, see https://www.varnish-cache.org/trac/wiki/VDD14Q1 (currently undergoing editing, so expect frequent updates until 2014-03-19): We've agreed that the following platforms should be "officially" supported with the next Varnish major release 4.0. Supported in this context means that builds from source tarballs should be possible without error provided that all requirements are met on the respective platform: * Debian wheezy, jessie * Ubuntu 12.04 + 14.04 * RHEL 6 * FreeBSD 9 + 10 Platforms we aim to support * Solaris-descendants: OmniOS, SmartOS Joyent Zones, Oracle Solaris Cheers, Nils From slink at schokola.de Wed Mar 19 15:39:25 2014 From: slink at schokola.de (Nils Goroll) Date: Wed, 19 Mar 2014 16:39:25 +0100 Subject: Via Message-ID: <5329BA2D.6030704@schokola.de> this probably is the most irrelevant issue of all, but as we touch this I think we should get it right (or at least know that we don't intentionally): At https://www.varnish-cache.org/trac/wiki/VDD14Q1 we said we should change Via and now I see https://www.varnish-cache.org/trac/changeset/f89015acfc28b6fe301b22a56174e9b85785d29a So according to http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.45, Via: 1.1 varnish (v4) "varnish" is the pseudonym and "(v4)" the comment. Shouldn't we at least abstract from the one example there is in the rfc and make it Via: 1.1 varnish (Varnish/4) ? Other than that: should we respect the rfc * in that we _add_ to any existing Via (which we don't atm)? Each recipient MUST append its information such that the end result is ordered according to the sequence of forwarding applications. * should we add the option to use a real hostname instead of the pseudonym? Personally, I would vote against both to conceal internal infrastructure behind Varnish, but this is a policy decision. Technically we probably should follow the rfc? Nils From slink at schokola.de Wed Mar 19 16:37:01 2014 From: slink at schokola.de (Nils Goroll) Date: Wed, 19 Mar 2014 17:37:01 +0100 Subject: on vcl_error -> vcl_synth : rather have both? Message-ID: <5329C7AD.7010802@schokola.de> editing https://www.varnish-cache.org/trac/wiki/Future_VCL I also removed this comment regarding "synth everywhere": "This eliminates the "error" primitive and reserves vcl_error{} for internally generated errors (like 503)" reflecting on this again: Is this really the implementation of least astonishment? It appears to me that having - vcl_synth for return (synth(701, "FOO")) and - vcl_error for varnish-core-code-induced errors may help clarity? You could never return error in vcl, but you could return synth. synth wouldn't close the connection, error would. In other words, wouldn't it help to know that "I can only have got here by having been called from some fierce varnish-internal error condition"? Also we have discussed the question of how to get a workspace for generating errors, wouldn't it help if we had - vcl_synth live on the original session ws and - vcl_error use a prestine session ws and copy req over? Cheers, Nils From slink at schokola.de Wed Mar 19 17:15:58 2014 From: slink at schokola.de (Nils Goroll) Date: Wed, 19 Mar 2014 18:15:58 +0100 Subject: vdd14q1 notes done Message-ID: <5329D0CE.1040807@schokola.de> Thanks again to Guillaume and all of Arkena / SmartJOG for sponsoring a fun and productive VDD! I have moved everything from yesterday's etherpad over to the wiki and hope to have left https://www.varnish-cache.org/trac/wiki/VDD14Q1 in a usable state now. @phk: I have tried to make your life a bit easier by compiling https://www.varnish-cache.org/trac/wiki/VDD14Q1#PHKsworklist but please take this only as my personal understanding of what your list could be. Cheers, Nils From slink at schokola.de Wed Mar 19 19:15:14 2014 From: slink at schokola.de (Nils Goroll) Date: Wed, 19 Mar 2014 20:15:14 +0100 Subject: [PATCH] create a hard dependency on some curses library Message-ID: <5329ECC2.8070108@schokola.de> what it really boils down to is Tollef's summary well, we require pcre, readline and rst2man to build.. I don't think ncurses is excessive. excerpt from irc: (18:57:53) slink: Also should we make some curses or the other a hard dependency in configure? atm building doc/sphinx fails if ! VARNISH_CURSES - because then there is no varnish(hist|stat|top)_opt2rst (19:15:40) slink: ok, i have fixed the latter, we just should not try to build docs for tools we can't build (19:31:42) fgs: or should we make curses mandatory? (19:31:55) Mithrandir / thfeen: I wouldn't mind. (19:32:20) slink: is there a use case for varnish without interactive consoles? (19:33:59) Kristian: slink: I can see the "I don't really need it"-argument, I just don't think I think that's worth worrying about. (19:34:17) Kristian: I don't see why anyone would *need* to run without curses available. (19:34:29) Mithrandir / thfeen: well, we require pcre, readline and rst2man to build.. I don't think ncurses is excessive. -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-create-a-hard-dependency-on-some-curses-library.patch Type: text/x-patch Size: 3577 bytes Desc: not available URL: From slink at schokola.de Wed Mar 19 19:45:12 2014 From: slink at schokola.de (Nils Goroll) Date: Wed, 19 Mar 2014 20:45:12 +0100 Subject: [PATCH] [EXPERIMENTAL] autocrap autohardening In-Reply-To: <20140130074848.GB10577@err.no> References: <52CEDCFE.2090508@schokola.de> <20140130074848.GB10577@err.no> Message-ID: <5329F3C8.2040707@schokola.de> Here's an update of the patch -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Enable-compiler-and-linker-options-to-frustrate-memo.patch Type: text/x-patch Size: 5532 bytes Desc: not available URL: From phk at phk.freebsd.dk Wed Mar 19 21:06:43 2014 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 19 Mar 2014 21:06:43 +0000 Subject: Via In-Reply-To: <5329BA2D.6030704@schokola.de> References: <5329BA2D.6030704@schokola.de> Message-ID: <2213.1395263203@critter.freebsd.dk> In message <5329BA2D.6030704 at schokola.de>, Nils Goroll writes: > > Via: 1.1 varnish (v4) vs. > Via: 1.1 varnish (Varnish/4) I really don't have an opionion at that level of detail... >Other than that: should we respect the rfc > >* in that we _add_ to any existing Via (which we don't atm)? We append a Via header, if there is one already, that will come first, and I belive Via is one of those headers where you should process all to get the valid result. Obviously, we can append to an existing Via header if we want that instead, but that would make it harder to remove just the Varnish Via again in VCL. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From geoff at uplex.de Thu Mar 20 07:37:47 2014 From: geoff at uplex.de (Geoff Simmons) Date: Thu, 20 Mar 2014 08:37:47 +0100 Subject: Via In-Reply-To: <2213.1395263203@critter.freebsd.dk> References: <5329BA2D.6030704@schokola.de> <2213.1395263203@critter.freebsd.dk> Message-ID: <532A9ACB.2070603@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 At the risk of bikeshedding via Via ... On 3/19/14 10:06 PM, Poul-Henning Kamp wrote: > > We append a Via header, if there is one already, that will come > first, and I belive Via is one of those headers where you should > process all to get the valid result. > > Obviously, we can append to an existing Via header if we want that > instead, but that would make it harder to remove just the Varnish > Via again in VCL. I haven't found a way to read the RFC in such a way that a Via header should be added -- all of my attempted parsings say that a hop should appended to an existing header. Unless of course I've missed it, or unless they use the word "append" in an ambiguous way that encompasses both meanings (which unfortunately I wouldn't rule out with this RFC). That said, separate headers are indeed much easier to work with in VCL, especially now that we have the header VMOD, which could specifically target the Varnish Via (to remove it or change "varnish" into a host name), or collect all the Vias into a single header, all quite trivially. Since this is getting into bike shed territory, I'd say it's easiest to go with the separate header, with the option of changing it in VCL. Best, Geoff - -- UPLEX Systemoptimierung Scheffelstra?e 32 22301 Hamburg http://uplex.de/ Mob: +49-176-63690917 -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJTKprKAAoJEOUwvh9pJNURkkYP/iElGxIOBO8Zd4nP0fljPXcy jZU4eVJY5CGK5lsi1UYjpoPxIcIm9dWV9KRhJsw038Ec7ApDtUfh0Wma+zAqB6vx 68WH+jwJDo61EmwfLMXdjzjbMu5QkhzHP2I1zZqpF3IbFv/j5P5kICNpqb7oRq/v PZwWaNl61MkAm8i8UXfxzgjQn+6M9iA/oDnCyFySvsHb/ddkH19VOcuR+njp/Bxt 3Uo3QDKJtEY0UaJLZ1twespM3uHmr4bK6ShU4OxRuiNujxeIGcJ4apBQXaLzwkzT vKOWJzERybjQvZ94zyCcR0ED1JSFRaPur6C+Nfrkjs8aDQRy/RVbskoBLSZ0lUpz 18p7QwWe3UobNw08iUHMe9G9CjVbGzsPwHFeYUE6r/gALSoOdjweNZdtyvvEnSmW +wKBQVcVjKhTal4ZuLWEu072nCBFXMirMoO6pVmQ8HqX/uFoNKlJh1BXGG8ZrPt0 m6vSssO5o5wWN2MCQEq2YaPsYGMhze4WvMDHNgXerC5I/QmtoluuDLvrxTMy3mZ1 RH57bIImahV/bX7kHIG/Wxk48EQgcYQJbOl9DLWSN4SsNGs44Ajen/17zGdoObwP aNmMqsZ7+r3cVsx/ivPkGgR+zxElfoh7D2gbmsO7IHEDi+PJomWiv6WJQlPQZ3Ao nzbMjxVxrB56ysUnnYv8 =F4SX -----END PGP SIGNATURE----- From tfheen at fastly.com Thu Mar 20 08:14:37 2014 From: tfheen at fastly.com (Tollef Fog Heen) Date: Thu, 20 Mar 2014 09:14:37 +0100 Subject: Via In-Reply-To: <532A9ACB.2070603@uplex.de> References: <5329BA2D.6030704@schokola.de> <2213.1395263203@critter.freebsd.dk> <532A9ACB.2070603@uplex.de> Message-ID: 2014-03-20 8:37 GMT+01:00 Geoff Simmons : > I haven't found a way to read the RFC in such a way that a Via header > should be added -- all of my attempted parsings say that a hop should > appended to an existing header. Unless of course I've missed it, or > unless they use the word "append" in an ambiguous way that encompasses > both meanings (which unfortunately I wouldn't rule out with this RFC). See last paragraph of https://tools.ietf.org/html/rfc2616#section-4.2 . I think it's pretty clear it's allowed. - Tollef From tfheen at fastly.com Fri Mar 21 07:23:52 2014 From: tfheen at fastly.com (Tollef Fog Heen) Date: Fri, 21 Mar 2014 08:23:52 +0100 Subject: [PATCH] create a hard dependency on some curses library In-Reply-To: <5329ECC2.8070108@schokola.de> References: <5329ECC2.8070108@schokola.de> Message-ID: 2014-03-19 20:15 GMT+01:00 Nils Goroll : > what it really boils down to is Tollef's summary > > well, we require pcre, readline and rst2man to build.. I don't think > ncurses is excessive. Generally looks good to me. I think the XXX comments should be kept, they're still relevant. - Tollef From tfheen at fastly.com Fri Mar 21 07:52:00 2014 From: tfheen at fastly.com (Tollef Fog Heen) Date: Fri, 21 Mar 2014 08:52:00 +0100 Subject: [PATCH] [EXPERIMENTAL] autocrap autohardening In-Reply-To: <5329F3C8.2040707@schokola.de> References: <52CEDCFE.2090508@schokola.de> <20140130074848.GB10577@err.no> <5329F3C8.2040707@schokola.de> Message-ID: 2014-03-19 20:45 GMT+01:00 Nils Goroll : > Here's an update of the patch I assume you've smoke-tested this, looks good to me from a quick skim. - Tollef From dridi.boukelmoune at zenika.com Fri Mar 21 08:36:31 2014 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Fri, 21 Mar 2014 09:36:31 +0100 Subject: [PATCH] [EXPERIMENTAL] autocrap autohardening In-Reply-To: <52CEDCFE.2090508@schokola.de> References: <52CEDCFE.2090508@schokola.de> Message-ID: On Thu, Jan 9, 2014 at 6:31 PM, Nils Goroll wrote: > Hi, > > I stumbled over > http://mainisusuallyafunction.blogspot.de/2012/05/automatic-binary-hardening-with.html > today and thought to give it a try with varnish. > > Result is attached: I have integrated a slightly modified version of > Keegan's configure.ac. Changes: > > - removed CXX support > - replaced -fstack-protector-all with -fstack-protector-strong and fallback > to > -fstack-protector > - removed -Wstack-protector (XXXLATER: disable for specific functions only?) > > This survives a "make check" on > > SunOS 5.11 snv_134 # ancient > gcc (GCC) 4.3.3 > > and > > Debian 6.0.8 > Linux debhag 2.6.32-5-xen-amd64 #1 SMP > gcc (Debian 4.4.5-8) 4.4.5 > Hi, You could also have a look at fedora's varnish package. It should already be be hardened [1] since it's a long running program. You can find an outdated (fedora 13) list of flags [2] but on my machine I get this: > $ rpm --eval '%optflags' > -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic > > $ rpm --define '_hardened_build 1' --eval '%optflags' > -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic > > $ cat /usr/lib/rpm/redhat/redhat-hardened-cc1 > *cc1_options: > + %{!fpie:%{!fPIE:%{!fpic:%{!fPIC:%{!fno-pic:-fPIE}}}}} > > $ uname -i > x86_64 Dridi [1] https://fedoraproject.org/wiki/Packaging:Guidelines#PIE [2] https://fedoraproject.org/wiki/Packaging:RPMMacros#Build_flags_macros_and_variables > > I have checked checksec output (attached) and run-times for make check on > linux with -fstack-protector (-fstack-protector-strong is TODO) > > * hardening enabled (default) > > debhag:~/v/varnish-git/varnish-cache# time make check > ... > ==================== > All 352 tests passed > ==================== > ... > > real 12m32.646s > user 1m12.137s > sys 0m51.791s > > - > * --disable-hardening > > debhag:~/v/varnish-git/varnish-cache# time make check > ... > ==================== > All 352 tests passed > ==================== > > real 12m21.915s > user 1m11.992s > sys 0m53.631s > > Should there be any interest in integrating something like this, we probably > would need to do more extensive testing and benchmarking. > > Also, whether or not varnish could benefit from such hardening is a > completely different question - personally I'd consider phk's defensive > coding approach much more important than additional stack/buffer overflow > protection, load address randomization and page protection. > > Nils > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev From martin at varnish-software.com Mon Mar 24 10:32:56 2014 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 24 Mar 2014 11:32:56 +0100 Subject: ReqEnd (Varnish 4 timing information revisited) In-Reply-To: References: Message-ID: After the discussions of the VDD, I have a revised suggestion (patch attached) for how to do timing information in Varnish. The suggestion was given during the VDD that the timing in Varnish should be more general and extendable, instead of having fixed categories of timing. To achieve this, the timing will be based upon timestamps being logged at the time specific events finish. The timestamps logged all have the same information, showing an event label, the absolute time of the event itself, the time spent since the start of the work unit (e.g. request processing or backend fetch), and the time since the last timestamp was logged. Having a limited set of fields that are present on all timestamps makes it easier to remember the format of it and understand the information without having to consult the documentation every time. There will also be less information present in each log entry, making it easier to spot the different fields. Timestamps can also be logged from VCL through a std.timestamp("event-label") call. So e.g. calling this after your curl vmod call, you will be able to show (thorugh the time since last timestamp field) how long time your curl HTTP call took to process. Attached is a patch to implement this for the request side of things. The backend fetch side has not been done yet. Note also that to improve the timing information, the start of time has been moved from after the client headers have been read to before the client headers have been read. The documentation for the Timestamp log record (also showing the events being logged): Timestamp - Timing information Contains timing information for the Varnish worker threads. Time stamps are issued by Varnish on certain events, and show the absolute time of the event, the time spent since the start of the work unit, and the time spent since the last timestamp was logged. The format is: %s: %f %f %f | | | | | | | +- Time since last timestamp | | +---- Time since start of work unit | +------- Absolute time of event +----------- Event label Timestamps are logged automatically by Varnish, or through custom std.timestamp() calls from VCL. The following timestamps are automatically logged: start - The start of a worker task, e.g. request processing or backend fetch. reqheaders - Finished reading the client headers. reqbody - Finished reading the request body. waitinglist - Came off waitinglist. fetch - Fetch processing finished. process - Processing finished, ready to deliver. deliver - Delivery finished. restart - Performing request restart. pipe - Opened pipe to backend. pipesess - Pipe session finished. On 14 March 2014 15:35, Martin Blix Grydeland wrote: > Hi, > > I'm working on a patch to improve the timing information in the VSL > records. Basis is the discussions of last VDD (which is summarized here > https://www.varnish-cache.org/trac/wiki/VDD13Q4). > > When doing this, I found that the ideas from there didn't sufficiently > take into account the complexity of restarts in VCL. A single request can > cause any number of objects to be fetched into the cache, and each of these > will have interesting timing information. From experience I know that > timing can be a very valuable debugging tool, and having good timing should > be a goal. > > So instead of the single ReqEnd for all timing data, I believe that it > should be split into two records. Timing for anything up until the reply is > being sent will be in a new record tentatively named ReqProc (request > processing). The fields will include time spent on waiting lists, time > spent fetching the request body and time spent fetching objects. The > ReqProc record will be logged also on restarts, so the log will show the > intermediate timing of the processing happened until that time. > > SLTM(ReqProc, 0, "Request processing timing", > "Contains timing information from the processing of the request.\n\n" > "The format is::\n\n" > "\t%f %f %f %f %f\n" > "\t| | | | |\n" > "\t| | | | +- Time spent processing and fetching\n" > "\t| | | +---- Time spent reading the request body\n" > "\t| | +------- Time spent on waiting lists\n" > "\t| +---------- Total time spent processing the request\n" > "\t+------------- Timestamp (since epoch) when vcl_recv was called\n" > "\n" > ) > > The ReqEnd will be output only once per request processed, and will only > be output from the log transaction that finishes the request handling > (restarts creates a new log transaction). It will contain the > time_to_first_byte (which is equal to the sum of the previous ReqProc > records), as well as the transmit time (which would include any ESI > processing or fetch time when streaming). The details of ESI processing > timing would be found in the ReqProc and ReqEnd log records of the ESI > subrequest transactions. Details of background fetch timing would be in the > BereqEnd record. > > SLTM(ReqEnd, 0, "Client request end timing information", > "Marks the end of a client request. Contains timing information" > " from the complete handling of the request.\n\n" > "The format is::\n\n" > "\t%f %f %f %f %f\n" > "\t| | | | |\n" > "\t| | | | +- Time to transmit response (plus ESI processing)\n" > "\t| | | +---- Time to process request (sum of ReqProc)\n" > "\t| | +------- Total time to finish request handling\n" > "\t| +---------- Timestamp (since epoch) when the request ended\n" > "\t+------------- Timestamp (since epoch) when the request started\n" > "\n" > ) > > Comments much appreciated. > > Martin > > -- > *Martin Blix Grydeland* > Senior Developer | Varnish Software AS > Cell: +47 21 98 92 60 > We Make Websites Fly! > -- *Martin Blix Grydeland* Senior Developer | Varnish Software AS Cell: +47 21 98 92 60 We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Introduce-SLT_Timestamp-mechanism-for-request-handli.patch Type: text/x-patch Size: 16627 bytes Desc: not available URL: From stanhope at gmail.com Mon Mar 24 11:45:09 2014 From: stanhope at gmail.com (Phil Stanhope) Date: Mon, 24 Mar 2014 07:45:09 -0400 Subject: varnish-dev Digest, Vol 96, Issue 14 In-Reply-To: References: Message-ID: Having been doing a tremendous amount of timestamp related work recently (outside of Varnish), I really like how Martin has done this. My feedback is that i think that the timestamp should always be first and the label second. The reason for me is that you have a natural sort ordering (particularly if the timestamp has 6 digits of precision). When scanning a streaming version of timestamps being emitted your eye can see the labels. But if things start shifting left/right because the labels are not of uniform length it gets very hard to absorb. Alternatively, I could argue that all fixed with fields should be sequential -- and the variable width label be the last field. The other thing that I've done is when serializing this type of information for debugging purposes to actually calculate and include the running delta -- even though it's obvious that it can calculated after the fact. Typically it will be .000003 -- very small number of micro seconds. It's when that delta gets big that it jumps out at you and don't have to do any math in your head -- instead focus on why that particular operation took as long as it did. -phil On Mon, Mar 24, 2014 at 6:33 AM, wrote: > Send varnish-dev mailing list submissions to > varnish-dev at varnish-cache.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > or, via email, send a message with subject or body 'help' to > varnish-dev-request at varnish-cache.org > > You can reach the person managing the list at > varnish-dev-owner at varnish-cache.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of varnish-dev digest..." > > > Today's Topics: > > 1. Re: ReqEnd (Varnish 4 timing information revisited) > (Martin Blix Grydeland) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 24 Mar 2014 11:32:56 +0100 > From: Martin Blix Grydeland > To: Varnish Development > Subject: Re: ReqEnd (Varnish 4 timing information revisited) > Message-ID: > < > CANTn4cpyxEzzfU9bvXgD-U587zB7PSXbyo6EPq0mZPQjLpf4Eg at mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > After the discussions of the VDD, I have a revised suggestion (patch > attached) for how to do timing information in Varnish. > > The suggestion was given during the VDD that the timing in Varnish should > be more general and extendable, instead of having fixed categories of > timing. To achieve this, the timing will be based upon timestamps being > logged at the time specific events finish. The timestamps logged all have > the same information, showing an event label, the absolute time of the > event itself, the time spent since the start of the work unit (e.g. request > processing or backend fetch), and the time since the last timestamp was > logged. > > Having a limited set of fields that are present on all timestamps makes it > easier to remember the format of it and understand the information without > having to consult the documentation every time. There will also be less > information present in each log entry, making it easier to spot the > different fields. > > Timestamps can also be logged from VCL through a > std.timestamp("event-label") call. So e.g. calling this after your curl > vmod call, you will be able to show (thorugh the time since last timestamp > field) how long time your curl HTTP call took to process. > > Attached is a patch to implement this for the request side of things. The > backend fetch side has not been done yet. > > Note also that to improve the timing information, the start of time has > been moved from after the client headers have been read to before the > client headers have been read. > > The documentation for the Timestamp log record (also showing the events > being logged): > > Timestamp - Timing information > > Contains timing information for the Varnish worker threads. > > Time stamps are issued by Varnish on certain events, and show > the absolute time of the event, the time spent since the start > of the work unit, and the time spent since the last timestamp > was logged. > > The format is: > > %s: %f %f %f > | | | | > | | | +- Time since last timestamp > | | +---- Time since start of work unit > | +------- Absolute time of event > +----------- Event label > > Timestamps are logged automatically by Varnish, or through custom > std.timestamp() calls from VCL. The following timestamps are > automatically logged: > > start - The start of a worker task, e.g. request processing or > backend fetch. > > reqheaders - Finished reading the client headers. > > reqbody - Finished reading the request body. > > waitinglist - Came off waitinglist. > > fetch - Fetch processing finished. > > process - Processing finished, ready to deliver. > > deliver - Delivery finished. > > restart - Performing request restart. > > pipe - Opened pipe to backend. > > pipesess - Pipe session finished. > > > > On 14 March 2014 15:35, Martin Blix Grydeland > wrote: > > > Hi, > > > > I'm working on a patch to improve the timing information in the VSL > > records. Basis is the discussions of last VDD (which is summarized here > > https://www.varnish-cache.org/trac/wiki/VDD13Q4). > > > > When doing this, I found that the ideas from there didn't sufficiently > > take into account the complexity of restarts in VCL. A single request can > > cause any number of objects to be fetched into the cache, and each of > these > > will have interesting timing information. From experience I know that > > timing can be a very valuable debugging tool, and having good timing > should > > be a goal. > > > > So instead of the single ReqEnd for all timing data, I believe that it > > should be split into two records. Timing for anything up until the reply > is > > being sent will be in a new record tentatively named ReqProc (request > > processing). The fields will include time spent on waiting lists, time > > spent fetching the request body and time spent fetching objects. The > > ReqProc record will be logged also on restarts, so the log will show the > > intermediate timing of the processing happened until that time. > > > > SLTM(ReqProc, 0, "Request processing timing", > > "Contains timing information from the processing of the request.\n\n" > > "The format is::\n\n" > > "\t%f %f %f %f %f\n" > > "\t| | | | |\n" > > "\t| | | | +- Time spent processing and fetching\n" > > "\t| | | +---- Time spent reading the request body\n" > > "\t| | +------- Time spent on waiting lists\n" > > "\t| +---------- Total time spent processing the request\n" > > "\t+------------- Timestamp (since epoch) when vcl_recv was called\n" > > "\n" > > ) > > > > The ReqEnd will be output only once per request processed, and will only > > be output from the log transaction that finishes the request handling > > (restarts creates a new log transaction). It will contain the > > time_to_first_byte (which is equal to the sum of the previous ReqProc > > records), as well as the transmit time (which would include any ESI > > processing or fetch time when streaming). The details of ESI processing > > timing would be found in the ReqProc and ReqEnd log records of the ESI > > subrequest transactions. Details of background fetch timing would be in > the > > BereqEnd record. > > > > SLTM(ReqEnd, 0, "Client request end timing information", > > "Marks the end of a client request. Contains timing information" > > " from the complete handling of the request.\n\n" > > "The format is::\n\n" > > "\t%f %f %f %f %f\n" > > "\t| | | | |\n" > > "\t| | | | +- Time to transmit response (plus ESI processing)\n" > > "\t| | | +---- Time to process request (sum of ReqProc)\n" > > "\t| | +------- Total time to finish request handling\n" > > "\t| +---------- Timestamp (since epoch) when the request ended\n" > > "\t+------------- Timestamp (since epoch) when the request started\n" > > "\n" > > ) > > > > Comments much appreciated. > > > > Martin > > > > -- > > *Martin Blix Grydeland* > > Senior Developer | Varnish Software AS > > Cell: +47 21 98 92 60 > > We Make Websites Fly! > > > > > > -- > *Martin Blix Grydeland* > Senior Developer | Varnish Software AS > Cell: +47 21 98 92 60 > We Make Websites Fly! > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20140324/077887ef/attachment.html > > > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: 0001-Introduce-SLT_Timestamp-mechanism-for-request-handli.patch > Type: text/x-patch > Size: 16627 bytes > Desc: not available > URL: < > https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20140324/077887ef/attachment.bin > > > > ------------------------------ > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > > End of varnish-dev Digest, Vol 96, Issue 14 > ******************************************* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at varnish-software.com Mon Mar 24 12:18:41 2014 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 24 Mar 2014 13:18:41 +0100 Subject: varnish-dev Digest, Vol 96, Issue 14 In-Reply-To: References: Message-ID: Find reply in line: On 24 March 2014 12:45, Phil Stanhope wrote: > Having been doing a tremendous amount of timestamp related work recently > (outside of Varnish), I really like how Martin has done this. > > My feedback is that i think that the timestamp should always be first and > the label second. The reason for me is that you have a natural sort > ordering (particularly if the timestamp has 6 digits of precision). When > scanning a streaming version of timestamps being emitted your eye can see > the labels. But if things start shifting left/right because the labels are > not of uniform length it gets very hard to absorb. > Agreed on the eye strain, though I chose to put it as the first field followed by a colon specifically to mimic the way we log HTTP headers. This because the log query language has a concept of a key-value record parsing that can then also be used with the timestamps, allowing you to match queries on specific timestamps. So to log transactions that has spent more than half a second on the waitinglist, you can do: varnishlog -q 'Timestamp:waitinglist[3] > 0.5' > Alternatively, I could argue that all fixed with fields should be > sequential -- and the variable width label be the last field. > > The other thing that I've done is when serializing this type of > information for debugging purposes to actually calculate and include the > running delta -- even though it's obvious that it can calculated after the > fact. Typically it will be .000003 -- very small number of micro seconds. > It's when that delta gets big that it jumps out at you and don't have to do > any math in your head -- instead focus on why that particular operation > took as long as it did. > This is what the last field is being used for ("Time since last timestamp"). It will be the delta to the last timestamp issued on this transaction. Martin -- *Martin Blix Grydeland* Senior Developer | Varnish Software AS Cell: +47 21 98 92 60 We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfheen at fastly.com Tue Mar 25 11:25:12 2014 From: tfheen at fastly.com (Tollef Fog Heen) Date: Tue, 25 Mar 2014 12:25:12 +0100 Subject: varnish-dev Digest, Vol 96, Issue 14 In-Reply-To: References:

Message-ID: 2014-03-24 13:18 GMT+01:00 Martin Blix Grydeland : > Agreed on the eye strain, though I chose to put it as the first field followed by a colon specifically to mimic the way we log HTTP headers. This because the log query language has a concept of a key-value record parsing that can then also be used with the timestamps, allowing you to match queries on specific timestamps. I wonder if we should add whitespace there, either in the log itself or reformatting in the varnishlog tool - Tollef From geoff at uplex.de Sun Mar 30 13:07:42 2014 From: geoff at uplex.de (Geoff Simmons) Date: Sun, 30 Mar 2014 15:07:42 +0200 Subject: [PATCH] clarify elements of the public VSL API Message-ID: <5338171E.2030704@uplex.de> Hello all, As discussed with Martin today on IRC, this patch just adds some comments to include/vapi/vsl.h. vsl.h includes vsl_int.h, and the latter defines some things that are essential for an API client to do useful work. But vsl.h identifies itself as the "public API for VSL access", while vsl_int.h says that it is "NOT!", with capital letters and an exclamation point. And there are places in vsl_int.h where an API client really shouldn't mess around. So the idea here is that a client developer can read vsl.h to know what's safe to use. @Martin, to draw your attention to some of the details: the comments now mention VSL_tag_e, SLT__MAX, some of the VSL_* macros, and macros for using VSL_tagflags. The VSL_* macros are only for accessing the same fields that were available in VSL_Dispatch in Varnish <= 3. I'm assuming that a client doesn't need to know about the rest -- VSL_BATCH*, for example, can be left out because IIUC VSLQ_Dispatch takes care of all that. VSL_CDATA is mentioned but not VSL_DATA, the idea being that clients should use a const char* and not a char* for log payloads. No promises if you go writing into the SHM log, rather than just read from it. Please make sure you're comfortable with this, because the implication will be that clients can rely on these things, so it might be painful to change them in the future. Thanks, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: clarify-public-VSL-API.patch Type: text/x-patch Size: 1943 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 897 bytes Desc: OpenPGP digital signature URL: From martin at varnish-software.com Sun Mar 30 18:49:58 2014 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Sun, 30 Mar 2014 20:49:58 +0200 Subject: Varnish 4 byte accounting Message-ID: Hi, Please find attached a set of patches to implement the byte accounting mostly as discussed during VDD13Q4. With hopes for a successful review and no beta blockers prescribed, Martin -- *Martin Blix Grydeland* Senior Developer | Varnish Software AS Cell: +47 21 98 92 60 We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Remove-struct-sess-acct_bit-and-bit-fields-from-Sess.patch Type: text/x-patch Size: 3474 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-Request-handling-byte-accounting.patch Type: text/x-patch Size: 29165 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0003-ESI-byte-accounting.patch Type: text/x-patch Size: 9816 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0004-Pipe-byte-accounting.patch Type: text/x-patch Size: 9859 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0005-Backend-fetch-byte-accounting.patch Type: text/x-patch Size: 11702 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0006-Backend-fetch-counters.patch Type: text/x-patch Size: 10673 bytes Desc: not available URL: From phk at phk.freebsd.dk Mon Mar 31 08:02:40 2014 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 31 Mar 2014 08:02:40 +0000 Subject: [PATCH] clarify elements of the public VSL API In-Reply-To: <5338171E.2030704@uplex.de> References: <5338171E.2030704@uplex.de> Message-ID: <99681.1396252960@critter.freebsd.dk> In message <5338171E.2030704 at uplex.de>, Geoff Simmons writes: >As discussed with Martin today on IRC, this patch just adds some >comments to include/vapi/vsl.h. No objections from me. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence.