From phk at phk.freebsd.dk Tue Oct 1 08:50:38 2013 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 01 Oct 2013 08:50:38 +0000 Subject: [PATCH] Option to choose another root tmp-directory in varnishtest & make check In-Reply-To: <5249CCFE.6070208@uplex.de> References: <5249CCFE.6070208@uplex.de> Message-ID: <51688.1380617438@critter.freebsd.dk> In message <5249CCFE.6070208 at uplex.de>, Geoff Simmons writes: >The Makefile for varnishtest is patched so that if you have TMPDIR >defined in the environment before 'make check' is called, then >varnishtest uses that, otherwise it uses /tmp. Shouldn't we just make varnishtest respect TMPDIR ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From geoff at uplex.de Tue Oct 1 08:57:45 2013 From: geoff at uplex.de (Geoff Simmons) Date: Tue, 01 Oct 2013 10:57:45 +0200 Subject: [PATCH] Option to choose another root tmp-directory in varnishtest & make check In-Reply-To: <51688.1380617438@critter.freebsd.dk> References: <5249CCFE.6070208@uplex.de> <51688.1380617438@critter.freebsd.dk> Message-ID: <524A8E89.1020700@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 10/01/2013 10:50 AM, Poul-Henning Kamp wrote: > >> The Makefile for varnishtest is patched so that if you have TMPDIR >> defined in the environment before 'make check' is called, then >> varnishtest uses that, otherwise it uses /tmp. > > Shouldn't we just make varnishtest respect TMPDIR ? Meaning no CLI option necessary, just use the value of TMPDIR if present, and /tmp otherwise? Sure, works for me. Need a new patch? Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJSSo5+AAoJEOUwvh9pJNURg28QAIXfJ2ZwZK53gj5YZ45WUPzk ikxmARu1ONohJdoj9axW9f+aFJLRQZx82+zFAxKZ3S4EmvKXNPp3uRz8a1lZOQU+ STWKhRpGqgllVfZU7J5xVo1FahqqMKm35T83fnkrXbIb9hKgQgWh1Ob+PMfQ0/B+ NmWDE4TfMDV+tQLFuFDpU+cZWX5KOp+SJlgs0USt5872u+/gazobj0PE1RzzCwcS owe+KJ5GjXmq1Rl7nbNkjbzwEgUqQdxBWVCaovnjiWjXnbN9mIRu771myBCWptrE zefLSJyFYrde5oIDEFBOTv7WqH1fzjNunPiOqKRFhRUTwp4MYQsmg3tcAtx9dm6k Ia2e1aBr5F5yV2aE8nF0bygkT4mUXkaRYgaYmhwtsPQFmFvltG+cQ3Qo5THZCEbz vzeNbTL7MzHyl119a3oI4+fs+9tx6JWRjMmFWrdwNMXi6vZLfSuPLkAd8DEBxxBP Yz1zp8CWkVqFYXoEzrBZpQdgc5Ymu3XyQ7xqpybjuvkYlA2dxG3cM3yWGGXMhFCI /MW5t7oG6LHNtCRVMkUdLo/yXlsd3WhuaUgCQLcFHePGjaClxIicOO40peA10pUw 0waoBQ3ImnS+BLuZUH57qCDy53kgIkSOupMGtpgwSRz2E+HB3eWpjXAv+QjYK8nP 6RpgmkLh4ZCP4r/GMW/x =6A+n -----END PGP SIGNATURE----- From phk at phk.freebsd.dk Tue Oct 1 09:39:57 2013 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 01 Oct 2013 09:39:57 +0000 Subject: [PATCH] Option to choose another root tmp-directory in varnishtest & make check In-Reply-To: <524A8E89.1020700@uplex.de> References: <5249CCFE.6070208@uplex.de> <51688.1380617438@critter.freebsd.dk> <524A8E89.1020700@uplex.de> Message-ID: <67890.1380620397@critter.freebsd.dk> In message <524A8E89.1020700 at uplex.de>, Geoff Simmons writes: >Meaning no CLI option necessary, just use the value of TMPDIR if >present, and /tmp otherwise? Yes. >Sure, works for me. Need a new patch? Yes please ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From geoff at uplex.de Tue Oct 1 11:02:27 2013 From: geoff at uplex.de (Geoff Simmons) Date: Tue, 01 Oct 2013 13:02:27 +0200 Subject: [PATCH] Option to choose another root tmp-directory in varnishtest & make check In-Reply-To: <67890.1380620397@critter.freebsd.dk> References: <5249CCFE.6070208@uplex.de> <51688.1380617438@critter.freebsd.dk> <524A8E89.1020700@uplex.de> <67890.1380620397@critter.freebsd.dk> Message-ID: <524AABC3.8050800@uplex.de> Hello all, On phk's suggestion, the new version of this patch has varnishtest using TMPDIR as the location for temporary vtc directories, or /tmp if TMPDIR is not set. No new CLI option or change to the Makefile necessary. This patch also updates the docs (varnishtest.rst). Best, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-varnishtest-uses-TMPDIR-for-temp-directories-or-tmp-.patch Type: text/x-patch Size: 2918 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 900 bytes Desc: OpenPGP digital signature URL: From phk at phk.freebsd.dk Tue Oct 1 11:44:47 2013 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 01 Oct 2013 11:44:47 +0000 Subject: [PATCH] Option to choose another root tmp-directory in varnishtest & make check In-Reply-To: <524AABC3.8050800@uplex.de> References: <5249CCFE.6070208@uplex.de> <51688.1380617438@critter.freebsd.dk> <524A8E89.1020700@uplex.de> <67890.1380620397@critter.freebsd.dk> <524AABC3.8050800@uplex.de> Message-ID: <68672.1380627887@critter.freebsd.dk> In message <524AABC3.8050800 at uplex.de>, Geoff Simmons writes: Just some nitpickery, fix and commit without further review: >diff --git a/bin/varnishtest/vtc_main.c b/bin/varnishtest/vtc_main.c >index 04aba0d..751a915 100644 >--- a/bin/varnishtest/vtc_main.c >+++ b/bin/varnishtest/vtc_main.c >@@ -85,6 +85,7 @@ static int vtc_verbosity = 1; /* Verbosity Level */ > static int vtc_good; > static int vtc_fail; > static int leave_temp; >+static char *tmp; Can we please call it "tmpdir" instead ? >- fprintf(stderr, FMT, "-l", "Leave /tmp/vtc.* if test fails"); >- fprintf(stderr, FMT, "-L", "Always leave /tmp/vtc.*"); >+ fprintf(stderr, FMT, "-l", "Leave $TMPDIR/vtc.* if test fails"); >+ fprintf(stderr, FMT, "-L", "Always leave $TMPDIR/vtc.*"); I'd prefer to not give the impression TMPDIR is mandatory: >+ fprintf(stderr, FMT, "-l", "Leave temporary vtc.* if test fails"); >+ fprintf(stderr, FMT, "-L", "Always leave temporary vtc.*"); (also in .rst files) > extmacro_def("varnishd", "varnishd"); /* Default to path lookup */ >+ tmp = strdup("/tmp"); >+ if (getenv("TMPDIR") != NULL) >+ tmp = strdup(getenv("TMPDIR")); Don't waste memory :-) >+ if (getenv("TMPDIR") != NULL) >+ tmp = strdup(getenv("TMPDIR")); >+ else >+ tmp = strdup("/tmp"); -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From geoff at uplex.de Tue Oct 1 11:50:21 2013 From: geoff at uplex.de (Geoff Simmons) Date: Tue, 01 Oct 2013 13:50:21 +0200 Subject: [PATCH] Option to choose another root tmp-directory in varnishtest & make check In-Reply-To: <68672.1380627887@critter.freebsd.dk> References: <5249CCFE.6070208@uplex.de> <51688.1380617438@critter.freebsd.dk> <524A8E89.1020700@uplex.de> <67890.1380620397@critter.freebsd.dk> <524AABC3.8050800@uplex.de> <68672.1380627887@critter.freebsd.dk> Message-ID: <524AB6FD.9070500@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 10/01/2013 01:44 PM, Poul-Henning Kamp wrote: > > Just some nitpickery, fix and commit without further review: OK, thx! - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJSSrb8AAoJEOUwvh9pJNURDgUP+gMPnhStOhjwz6SUZWrAPcVE wI+qMtKJldzzRSU0JleK4rnTHKorj4y0IHB32G+Pcjc2zPm5DhTh0EK6tgXu2Dkk uc03g0UadBy0BhLobgx4keJ7ihJg/0GygxFg6agnD+oeZf+ZCC2cag+xBg17lbph iaCz40X8nQ8K9bWY3UcB0mwwve31YtDxBUx4VFCyb/ElqM1xntJdfxdeS+47lQvG 4l5N6ciJhVb6dic6WIIQ80TSUg4c/zZlJKoEYi9ftmL2U5qv/8qOMQf/1SZc94ua vuZOHoe9yw1lgs/kwFJt2fiWqvCsaB315InIuF8r9YtYH77S6Ezqgku/40pcYg/H /yXJEKpyNOJecsx07KMTxV9aCmfjRg4PTn8ULlrCPwEp9uPypu9jkQW0r0WkGlps u11j9hJzGAEThWEqN6ilnXeTJt2XxVwn7NH8pYjnaH2gw7BlZ7YwFFUE2k5PktE/ 2JY3UCgP1wUngO+v/t1p4fj9wJveoZdsHh3GxnSXnkDAo6s3O9DeccK0aJGBwyA4 xHE6Kcwmbh1guu7doGZR1ShVTVa5MV7LMy25TJAG+WVZEdRpFyuJ3S5a6Pr/hL6i glvC+XcwkcsM3Ux2D4ixJXo87eDHAFUmsLpcUg3BgLHk1nMiTbzYhDsk+xXznSy0 eTR+rzJXGGKaPYA0NZpj =47NY -----END PGP SIGNATURE----- From geoff at uplex.de Tue Oct 1 15:03:48 2013 From: geoff at uplex.de (Geoff Simmons) Date: Tue, 01 Oct 2013 17:03:48 +0200 Subject: [PATCH] Option to choose another root tmp-directory in varnishtest & make check In-Reply-To: <68672.1380627887@critter.freebsd.dk> References: <5249CCFE.6070208@uplex.de> <51688.1380617438@critter.freebsd.dk> <524A8E89.1020700@uplex.de> <67890.1380620397@critter.freebsd.dk> <524AABC3.8050800@uplex.de> <68672.1380627887@critter.freebsd.dk> Message-ID: <524AE454.6060409@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Picking a nit in the nitpick: >> +static char *tmp; > > Can we please call it "tmpdir" instead ? tmpdir is already taken in start_test(), so now it's tmppath. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJSSuRUAAoJEOUwvh9pJNUR8/AP/1bVwbJdeH9AKCB/UVq/2EXr vteLRZ6dQ+z7zDAqbfbOAK4yiJ6wc1yGXIbrN2yyp+29IwOOnW6NPtI4rTwJho11 JqcuHUUAfYoolEuTOcWLwElpJ/nS8JFSvwapI7b4OWPl4TG4ivqPZGSnbqS+qfsL Ij3/Ehpz8YDAmLNzjpNWI4YAMf5xjDUbzTgdfaylJlcxoJg969nHHOOaClArzoyi 0/sCb+wCj1vOKtVuPKXYPe6VbdRzeBx95wyv+mQYwsj3uiCsZJ3ZRmuGAzHe5a5P NwxUb4ioT/uB6mmzz5BiVNRwyiHNSYzQgSpuNTz2BF3dFtpTNb9V8eqGvJAxnmm5 hCzcXPHMMF4eN7OkLklB9kqi4+FfzKoP8dlHdNkLD4kkZFMm2LOZBsaKTl+MZizT 5pMJ5S739q8NKOwPiKmI0NzkBXObQjaRw4QSNNR99CeRUfYDgBNnI+fhrAUrGBHo FbIpotF/w59XQbLeoDukDKQ5jzS5Nxn5c57CsyFGq5IYDaJE8+LvJxlawUo1fVu5 96OvnQ5XgtxidG5zBpwlEzih0Dy8fNL9aDbvgN0evaAwAundofuBmbCt5Jgd1IHZ Co4f9MvPmwwaykwmro+IsD1GgEp5j3UaHtKDEW5O5bY0otReEiVGciFuyqnao5hr MzM6axoFtuZOzH6AyDlG =Vi1H -----END PGP SIGNATURE----- From geoff at uplex.de Thu Oct 10 09:37:32 2013 From: geoff at uplex.de (Geoff Simmons) Date: Thu, 10 Oct 2013 11:37:32 +0200 Subject: [PATCH] Add the VSL tag BereqEnd Message-ID: <5256755C.4070405@uplex.de> Hello all, This patch adds the BereqEnd tag, always the last tag emitted for a bereq transaction before the End marker. The format is: BereqEnd Tstart: begin fetch processing (epoch) Tend: end fetch processing (epoch) Thdr: time to receive response header Tbody: time to receive response body Thdr is approximately the delta from just after sending the backend request until just after receiving the header. Tbody is approximately the delta from just after receiving the response header until just after receiving the body. So for example: BereqEnd 1381335026.155810356 1381335026.268885612 0.112095833 0.000851154 The deltas are approximate because the low-level fetch operations do some buffering, so the timings don't correspond strictly to reading the header and body off the network (see the comments in cache_http1_proto.c), but I think will for the most part reflect backend latencies well enough. The rationale is that I think backend timings will be useful for: - measuring backend performance, and in particular finding out which backends are slow - finding out how much time in a fetch is due to work done strictly in varnishd, and how much is due to latencies in systems beyond Varnish's control There is a bit of varnishd overhead in the Thdr and Tbody timings, but they are dominated by backend latencies. That does mean, however, that Tbody is always > 0, even if there is no response body. Probably most people won't be very interested in the difference between Thdr and Tbody (just Thdr+Tbody). We have to take a measurement after the header arrives anyway, since there might not be a body, so I thought we may as well log the delta. But maybe we just want the one value here, just Thdr+Tbody. Tell me what you think. Best, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Add-the-VSL-tag-BereqEnd.patch Type: text/x-patch Size: 4771 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 900 bytes Desc: OpenPGP digital signature URL: From geoff at uplex.de Thu Oct 10 09:51:23 2013 From: geoff at uplex.de (Geoff Simmons) Date: Thu, 10 Oct 2013 11:51:23 +0200 Subject: [PATCH] logexpect only attempts regex matches for matching tags Message-ID: <5256789B.1030303@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJSVnibAAoJEOUwvh9pJNURMioQALHTfUPXn5ZyZ3iqrbguFQIH iKMNjhti7bwghwBjZFRGujAcbYDWcxWKL/yw4vmovTLGX4cgdK48NYgVPBotJR5/ +Cak67UG+PtU+0L1n3U9kRBsN8ETEQMqWZHMrqK85erZs5lfkknHymyuxE+9O5Gd H47ujAOMEiSejVEoY/b4fcTjGdVe09JcPXmbs/nq6S9VqvAogxO7kQJ+VwcLOPcU V9DokK7RRkrq/6v/EouiIkgg1InCa/pUbcOLdbz6OaqaN+SyqKebzmt2C5FOIR9r cCLV4ZqL9UmZhu8KQYWvQu1qz4Ckoeqb3BMUK4skoXoSDTmVG7jQVWmCDx94InGz vpsC7YEQLjS7SMOZJVHK4yxDZjHFkBpmnQlYwew/3aZUH8BS/vQp6NRRYZUEFHcE XBBnLVgPEvFxB5u1UZ1tCUidM6o5RCII9XUrcNE8JCR/5UiPEJNcumIJATkxW99j vyoWGPfbni5ULY6Sl/nsPN2aM/1RQn893a2ys2sTP30APdPxOAyOQv/mJiCbgxFy dZkdWF9lOUlAcY+S0dMHgPssWFXDB2UUtwOpRC+4RnhFTM1AjwgsWBZmbefDTgFD 4QS5z3JuzLI6aKIsbaNFqJNrUMuYgsvfA3JQLGiog6HuSjy+XHn47EAqlU+RoYd7 x9mm2y3kmagk71gyqzhu =pcwB -----END PGP SIGNATURE----- From geoff at uplex.de Thu Oct 10 09:59:45 2013 From: geoff at uplex.de (Geoff Simmons) Date: Thu, 10 Oct 2013 11:59:45 +0200 Subject: [PATCH] logexpect only attempts regex matches for matching tags In-Reply-To: <5256789B.1030303@uplex.de> References: <5256789B.1030303@uplex.de> Message-ID: <52567A91.505@uplex.de> Hello all, Sorry about that empty mail. While trying to learn to use logexpect, I noticed that for expectations like these: expect 0 * FixedTag expect * = FooTag reg.x* ... it would attempt the regex match for every log entry after the FixedTag, regardless of whether the entry has the FooTag. It continues until it hits an entry with a match, which may or may not have the FooTag. I may have misunderstood the intention here, but the attached patch ensures that the regex match is only attempted for log entries with the expected tag. Best, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-varnishtest-logexpect-only-attempts-regex-matches-fo.patch Type: text/x-patch Size: 820 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 900 bytes Desc: OpenPGP digital signature URL: From martin at varnish-software.com Fri Oct 11 10:55:51 2013 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 11 Oct 2013 12:55:51 +0200 Subject: [PATCH] logexpect only attempts regex matches for matching tags In-Reply-To: <52567A91.505@uplex.de> References: <5256789B.1030303@uplex.de> <52567A91.505@uplex.de> Message-ID: Hi Geoff, The intention was exactly as you guess, that the regular expression should only be applied to records that has the tag that goes with the regex. So this is indeed a bug. I will apply the patch. Thanks :) Martin On 10 October 2013 11:59, Geoff Simmons wrote: > Hello all, > > Sorry about that empty mail. > > While trying to learn to use logexpect, I noticed that for expectations > like these: > > expect 0 * FixedTag > expect * = FooTag reg.x* > > ... it would attempt the regex match for every log entry after the > FixedTag, regardless of whether the entry has the FooTag. It continues > until it hits an entry with a match, which may or may not have the FooTag. > > I may have misunderstood the intention here, but the attached patch > ensures that the regex match is only attempted for log entries with the > expected tag. > > > Best, > Geoff > -- > ** * * UPLEX - Nils Goroll Systemoptimierung > > Scheffelstra?e 32 > 22301 Hamburg > > Tel +49 40 2880 5731 > Mob +49 176 636 90917 > Fax +49 40 42949753 > > http://uplex.de > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -- *Martin Blix Grydeland* Senior Developer | Varnish Software AS Cell: +47 21 98 92 60 We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at varnish-software.com Tue Oct 15 11:56:53 2013 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 15 Oct 2013 13:56:53 +0200 Subject: [PATCH] Add the VSL tag BereqEnd In-Reply-To: <5256755C.4070405@uplex.de> References: <5256755C.4070405@uplex.de> Message-ID: Thanks for the patch Geoff! For the format and what is logged I have a couple of comments. I am missing a Treq field, which is the time it took to send the request to the backend in the first place. Would be used to search for network issues (or slow piping of passed client request bodies). Also I think we should have the total backend response time in seconds as a separate field. This because that I guess will be the most frequently used property of the record, and will be useful to query on in terms of finding the ones that take a long time. I realize that the data is there as the sum of Thdr and Tbody, but since the query lang can't test on the sum of two fields it becomes not so usable. The question then becomes if we should add a Tresp field, or replace Tbody with the Tresp (causing Tbody to be calculable but not query-able). My vote goes to add the Tresp field. Lastly, I think mimicking ReqEnd is a goal in itself, and to do that we should think about the vxid that is present on ReqEnd. But in Varnish 4 the vxid is part of every log record always, so here I believe we should remove the vxid from ReqEnd instead. So my suggestion on the fields present is: BereqEnd Tstart: begin fetch processing (epoch) Tend: end fetch processing (epoch) dTreq: time to send the backend request dThdr: time to receive the backend response headers dTbody: time to receive the backend response body dTresp: time to receive the backend response (dThdr + dTbody) Regards, Martin On 10 October 2013 11:37, Geoff Simmons wrote: > Hello all, > > This patch adds the BereqEnd tag, always the last tag emitted for a > bereq transaction before the End marker. > > The format is: > > BereqEnd > > Tstart: begin fetch processing (epoch) > Tend: end fetch processing (epoch) > Thdr: time to receive response header > Tbody: time to receive response body > > Thdr is approximately the delta from just after sending the backend > request until just after receiving the header. Tbody is approximately > the delta from just after receiving the response header until just after > receiving the body. > > So for example: > > BereqEnd 1381335026.155810356 1381335026.268885612 0.112095833 > 0.000851154 > > The deltas are approximate because the low-level fetch operations do > some buffering, so the timings don't correspond strictly to reading the > header and body off the network (see the comments in > cache_http1_proto.c), but I think will for the most part reflect backend > latencies well enough. > > The rationale is that I think backend timings will be useful for: > > - measuring backend performance, and in particular finding out which > backends are slow > > - finding out how much time in a fetch is due to work done strictly in > varnishd, and how much is due to latencies in systems beyond Varnish's > control > > There is a bit of varnishd overhead in the Thdr and Tbody timings, but > they are dominated by backend latencies. That does mean, however, that > Tbody is always > 0, even if there is no response body. > > Probably most people won't be very interested in the difference between > Thdr and Tbody (just Thdr+Tbody). We have to take a measurement after > the header arrives anyway, since there might not be a body, so I thought > we may as well log the delta. But maybe we just want the one value here, > just Thdr+Tbody. Tell me what you think. > > > Best, > Geoff > -- > ** * * UPLEX - Nils Goroll Systemoptimierung > > Scheffelstra?e 32 > 22301 Hamburg > > Tel +49 40 2880 5731 > Mob +49 176 636 90917 > Fax +49 40 42949753 > > http://uplex.de > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -- *Martin Blix Grydeland* Senior Developer | Varnish Software AS Cell: +47 21 98 92 60 We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at uplex.de Tue Oct 15 12:34:23 2013 From: geoff at uplex.de (Geoff Simmons) Date: Tue, 15 Oct 2013 14:34:23 +0200 Subject: [PATCH] Add the VSL tag BereqEnd In-Reply-To: References: <5256755C.4070405@uplex.de> Message-ID: <525D364F.3010808@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 10/15/2013 01:56 PM, Martin Blix Grydeland wrote: > > I am missing a Treq field, which is the time it took to send the request > to the backend in the first place. Would be used to search for network > issues (or slow piping of passed client request bodies). OK. The only concern here (and the only reason I left it out) is that it would require another syscall per backend fetch (VTIM_real just before sending the request). But if we're sure we can afford that, then let's do it. > Also I think we should have the total backend response time in seconds > as a separate field. Short version: agreed %^) Especially since it's just a little more math. > Lastly, I think mimicking ReqEnd is a goal in itself, and to do that we > should think about the vxid that is present on ReqEnd. But in Varnish 4 > the vxid is part of every log record always, so here I believe we should > remove the vxid from ReqEnd instead. ReqEnd also doesn't have the vxid in the log payload any more (at least the last time I tested V4). And I agree, with the new logging API it doesn't have to. > So my suggestion on the fields present is: > BereqEnd > > Tstart: begin fetch processing (epoch) > Tend: end fetch processing (epoch) > dTreq: time to send the backend request > dThdr: time to receive the backend response headers > dTbody: time to receive the backend response body > dTresp: time to receive the backend response (dThdr + dTbody) Will do! I'll send up another patch. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJSXTZPAAoJEOUwvh9pJNUR3GQQAKLLhKlNbru3bjp3W8KcsLkF vFy2/C1ZsOCqM40USMkD7JTq4FzoTOaziXxHSjrCQCSs7WXSUX7OVoObYBWpxSr4 3GQlCrVkp9fSSoarFY7sjnBe3wyrXi+wVoJ/ESsiV0bqBqwS2OfYmkFhcaJ93kJX B8QuxfekEXE7S9xtB2G4q3hh62LcxaPNCTaTZn5vaG1k5D5urYwSiE3PFepikU/e 7XxeFB32xFOQFyA2RyLcti4i9Hefd6tjIptnLYp3dxPJc2f0JO9BJptngoWfeH4r 2QbnqcpHDtJ7FsnWOOxpLaqAHXUUgMoAv2F1F9wnkVmvcinzaj9jXYdWhrCilND8 9Mak1MiTtBWmhCiRrUVT4VSwCTgA08l3RDx9sOgVisVkRaC8wdUI3m6nAbaaD0Gn sfquZlIhUAb8uk9YLYDUWQmLEEdeSh1tm3q20hV6ORbFxs0TOUYA0KIW0Id8LXf5 k4Uq2832qFZFP+n9Pt6gUcqAG40mK0MpeYGKww+hqeFxSOnackf0xJmKOE+PU+wC GVNWABWYEQXRlXsNZaxz3piK00F3p1gH6CyjJxVPvYWg7apV6B21qFxAlAZUcEek fnHoNTgHxd1fuCpjrLsBewuNZxJxpppv0yR30W+OC7NHz8IK1kTmZNoElBJiQDgM l5801XlqSCwqJc4g4LBZ =TwNo -----END PGP SIGNATURE----- From martin at varnish-software.com Tue Oct 15 14:18:56 2013 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 15 Oct 2013 16:18:56 +0200 Subject: [PATCH] Add the VSL tag BereqEnd In-Reply-To: <525D364F.3010808@uplex.de> References: <5256755C.4070405@uplex.de> <525D364F.3010808@uplex.de> Message-ID: On 15 October 2013 14:34, Geoff Simmons wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > On 10/15/2013 01:56 PM, Martin Blix Grydeland wrote: > > > > I am missing a Treq field, which is the time it took to send the request > > to the backend in the first place. Would be used to search for network > > issues (or slow piping of passed client request bodies). > > OK. The only concern here (and the only reason I left it out) is that it > would require another syscall per backend fetch (VTIM_real just before > sending the request). But if we're sure we can afford that, then let's > do it. > Since there's network traffic happening here and backend response time, any additional delays from a VTIM_real will imo be insignificant and not be measurable against the time-to-first-byte of the client reply. But PHK might have an opinion on that? Martin -- *Martin Blix Grydeland* Senior Developer | Varnish Software AS Cell: +47 21 98 92 60 We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at uplex.de Wed Oct 16 10:56:30 2013 From: geoff at uplex.de (Geoff Simmons) Date: Wed, 16 Oct 2013 12:56:30 +0200 Subject: [PATCH] Add the VSL tag BereqEnd In-Reply-To: References: <5256755C.4070405@uplex.de> Message-ID: <525E70DE.8010507@uplex.de> On 10/15/2013 01:56 PM, Martin Blix Grydeland wrote: > > So my suggestion on the fields present is: > BereqEnd > > Tstart: begin fetch processing (epoch) > Tend: end fetch processing (epoch) > dTreq: time to send the backend request > dThdr: time to receive the backend response headers > dTbody: time to receive the backend response body > dTresp: time to receive the backend response (dThdr + dTbody) Attached is a new version of the patch with this implementation of BereqEnd. It also now uses VTIM_mono() to get the measurement points for the deltas, so there are 2 calls to VTIM_real() and 4 to VTIM_mono() for every fetch. Best, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Add-the-VSL-tag-BereqEnd.patch Type: text/x-patch Size: 5714 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 900 bytes Desc: OpenPGP digital signature URL: From ruben at varnish-software.com Wed Oct 16 20:44:11 2013 From: ruben at varnish-software.com (=?UTF-8?Q?Rub=C3=A9n_Romero?=) Date: Wed, 16 Oct 2013 22:44:11 +0200 Subject: Welcome to #VUG8 week in Berlin, Germany - November 25th to 29th, 2013 - http://varnish.org/VUG8 Message-ID: Hallo, Alles gut? Sch?n. Das ist gut. ** Please help us to spread the word to all Varnish & Web Performance enthusiasts you know ** In behalf of the Varnish Team, it is my pleasure to announce to you the next Varnish User Group Meeting, the eight of its kind, which is to be held in Berlin, Germany. #VUG8 is possible thanks to the sponsorship of Axel Springer, Uplex and Varnish Software. *** Short story: Registration is open for the Varnish User Day @ Axel Springer HQ on Thursday, November 28th, 2013. Get your free ticket now: < http://vug8.eventbrite.com> *** *** Long story *** * Varnish User Day is open for everyone and it will be at Axel Springer on Thursday, November 28th, 2013. If you want to share your Varnish & VCL experience with the crowd please let me know; these meetings are, after all, about how YOU use Varnish ;-) For agenda and general information [1] * Varnish Developer meeting: will happen on Wednesday, November 27th, 2013. This event is invite only. See and use the wiki for coordinating details [2] * Hacking Day / Community Roundtable: this is new for #VUG8. It's open for everyone and takes place on Friday, November 29th, 2013. If you want to participate keep an eye on the VUG8 page[3] for updates or reply to this email. Also we need a venue for this one. If you know someone that can help us with a meeting room for that day, please let me know. * For the first time in Berlin: 2-day Varnish Administration Certification Class starting on Monday, November 25th, 2013. More information, pricing and registration [4] We are very much looking forward to meet the Varnish community in Berlin and spending time with you while making Varnish Cache better. And before I let you go: Do not forget to let your colleagues and friends know about #VUG8. Wir sehen uns in Berlin! Links: [1] If you register to the User Day, I reserve the right to ask you if can hold a presentation :-) [2] [3] [4] Mit freundlichen Gr??en,, -- *Rub?n Romero* Global Sales Executive & Community Liaison | Varnish Software AS Cell: +47 95964088 / Office: +47 21989260 Skype & Twitter: ruben_varnish We Make Websites Fly!Winner of the 2013 Red Herring Top 100 Europe Awards -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Wed Oct 23 12:33:59 2013 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 23 Oct 2013 12:33:59 +0000 Subject: [PATCH] Add the VSL tag BereqEnd In-Reply-To: <525E70DE.8010507@uplex.de> References: <5256755C.4070405@uplex.de> <525E70DE.8010507@uplex.de> Message-ID: <1651.1382531639@critter.freebsd.dk> In message <525E70DE.8010507 at uplex.de>, Geoff Simmons writes: >> dTreq: time to send the backend request This one probably does not do what you think. >+ bo->t_send = VTIM_mono(); > (void)HTTP1_Write(wrk, hp, 0); /* XXX: stats ? */ > > /* Deal with any message-body the request might (still) have */ >@@ -241,6 +243,7 @@ V1F_fetch_hdr(struct worker *wrk, struct busyobj *bo, > struct req *req) > if (req->req_body_status == REQ_BODY_DONE) > retry = -1; > } >+ bo->t_sent = VTIM_mono(); > > if (WRW_FlushRelease(wrk) || i != 0) { > VSLb(bo->vsl, SLT_FetchError, "backend write error: %d (%s)", The writev() to the kernel doesn't happen until WRW_FlushRelease(), so at the very least the timestamp should be taken after that. But... The time you measure is only the time to hand the request to the kernels network stack. As far as I know, there is no way to wait on a socket until all queued data has been sent (or even ACKed). -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From geoff at uplex.de Wed Oct 23 15:03:29 2013 From: geoff at uplex.de (Geoff Simmons) Date: Wed, 23 Oct 2013 17:03:29 +0200 Subject: [PATCH] Add the VSL tag BereqEnd In-Reply-To: <1651.1382531639@critter.freebsd.dk> References: <5256755C.4070405@uplex.de> <525E70DE.8010507@uplex.de> <1651.1382531639@critter.freebsd.dk> Message-ID: <5267E541.3060708@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 10/23/2013 02:33 PM, Poul-Henning Kamp wrote: > > The writev() to the kernel doesn't happen until WRW_FlushRelease(), so > at the very least the timestamp should be taken after that. OK > But... > > The time you measure is only the time to hand the request to the kernels > network stack. As far as I know, there is no way to wait on a socket > until all queued data has been sent (or even ACKed). Yes. In my tests, the value was always in the ?s range, altogether dwarfed by the other two deltas. Even in the best case, it will probably be the one that users will be least interested in. Martin suggested adding tReq to help find out if there are network latencies that slow down requests getting to backends in the first place (or slow down pipes). That would be good to know, but if there's no way to actually measure it within varnishd, we should re-consider having it at all. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJSZ+U1AAoJEOUwvh9pJNUR9JsP/28t0XAQyYTYwfsaiePVx0tz +6Dwvd8iPx6nAExEFsReyB01U5m4ck3GtjNyHueiOgFBu/LQ9V/YJmDsieFwq0MY MERWo77P99OQsXgZ0ZVbZaFG0w/2/L0+e9Zw2WqUsGMD1y4bUILoRFfPh+3HUF0N bLmnnniwDaJIxEwYuJgQx84Wi9ox5P2qNxqvqo+v5b0+MgcxqbvHEKXzuaXk83Vd u5ttlys3glv94l/WKRr/q/WcIcMikw+T4RloC/x0kxc6rbf//jJC/KgzFT8uP87G 5Bj9sAsBRLUgfKFfJzvKufbeTUIegqE7JCd8NQljYkhPkRxTfQmqYpAvwAJQZDOt cnhjvfvsd1D+GGbblJSrt9g9jKy5wVUfIigP8ZrGo0/vvusvLUGr8xX2kQMNIWc8 S0+b1kfqHWbDubr2EYm7YtYeX9OvGET1n7ag/lnQsgiK5mPqz1STj+lYOO8GqtHk SHurjiSkb7Hto3cHfIP3N/uc8vUHaYMDOx0oTnP0LAk2UJGB/4YH8e9f6s98YVY7 WewLImKZGgPu3zqUf3vmZHeBd3nzbV3Y2dtAXtfCFHk41p2x+ydHRCuCjpGNd6yF DBZ7+ixB3B0YjAav02jB9Rva/aocSBA2XA2A8/BBFunIkdoTvPXOPF0kDx0LMBda O0Z0f0nFrLQkh0GbA2Lw =KZOn -----END PGP SIGNATURE----- From phk at phk.freebsd.dk Wed Oct 23 15:09:24 2013 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 23 Oct 2013 15:09:24 +0000 Subject: [PATCH] Add the VSL tag BereqEnd In-Reply-To: <5267E541.3060708@uplex.de> References: <5256755C.4070405@uplex.de> <525E70DE.8010507@uplex.de> <1651.1382531639@critter.freebsd.dk> <5267E541.3060708@uplex.de> Message-ID: <1776.1382540964@critter.freebsd.dk> In message <5267E541.3060708 at uplex.de>, Geoff Simmons writes: >Martin suggested adding tReq to help find out if there are network >latencies that slow down requests getting to backends in the first place >(or slow down pipes). That would be good to know, but if there's no way >to actually measure it within varnishd, we should re-consider having it >at all. Yes, it would be interesting, but no, I don't think we can measure it, without polling the kernel with a unportable getsockopt() and stare directly into the protocol control block. Neither the polling nor the non-portability are popular with me. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From carlos.abalde at gmail.com Wed Oct 30 16:33:19 2013 From: carlos.abalde at gmail.com (Carlos Abalde) Date: Wed, 30 Oct 2013 17:33:19 +0100 Subject: Memory management in VMODS: sp->ws vs. sp->wrk->ws Message-ID: <99BDCB7B-580B-4CB9-BEAA-6C988AFCD3E3@gmail.com> Hi there, I hope this is the correct mailing list for a VMOD development question. After reading all the available documentation and also reviewing implementations of almost all VMODs in https://www.varnish-cache.org/vmods, I'm not sure what is the difference between using sp->ws and sp->wrk->ws when allocating memory in a VMOD method. It seems sp->wrk->sp is the way to go when using WS_Reserve() and WS_Release(). However, when using WS_Alloc() or WS_Dup() I've seen modules using both sp->ws and sp->wrk->sp. I guess it's not a random choice, but I haven't been able to identify any rule. Is there any difference? Any hint on when to use sp->ws and when sp->wrk->sp? Thanks! -- Carlos Abalde. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From kacperw at gmail.com Fri Oct 11 18:38:36 2013 From: kacperw at gmail.com (Kacper Wysocki) Date: Fri, 11 Oct 2013 18:38:36 -0000 Subject: [PATCH] Backport #1337: weird http status codes Message-ID: Hia all, this is a backport of out-of-range backend status codes handling to varnish-3.0.4. Martin mentioned maybe that fix needed backporting, and I couldn't wait. Still can't get the test to pass, but if I break the two requests into two tests, they pass. A case of the noobs? -- http://u.delta9.pl Too much order is its own chaos. Employ no technique to gain supreme enlightment. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: backport-statuscodes.patch Type: application/octet-stream Size: 3342 bytes Desc: not available URL: