From phk at phk.freebsd.dk Sun Mar 1 09:05:37 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sun, 01 Mar 2015 09:05:37 +0000 Subject: [PATCH] Instruct the kernel to reset the connection for SC_RX_TIMEOUT, and others In-Reply-To: <54F0B1FE.3040600@schokola.de> References: <54EEF4F9.9050208@schokola.de> <54F0B1FE.3040600@schokola.de> Message-ID: <21409.1425200737@critter.freebsd.dk> -------- In message <54F0B1FE.3040600 at schokola.de>, Nils Goroll writes: >Closing TCP connections with SO_LINGER unset [...] I pressume you mean "set" ? The problem with !SO_LINGER (we tried it some years back) is that all queued data is discarded the moment you call close(2), and we have no way to know if our last response is still lingering in the kernel's tcp-buffers and dribbling out to a slow client with a bad mobile phone connection. The option we need is "Try reasonably to send what you have, then close, but let userland continue right away." Architectually it's amazing how different kernel/TCP semantics are from what HTTP needs... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Sun Mar 1 09:07:40 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sun, 01 Mar 2015 09:07:40 +0000 Subject: [PATCH] Instruct the kernel to reset the connection for SC_RX_TIMEOUT, and others In-Reply-To: <21409.1425200737@critter.freebsd.dk> References: <54EEF4F9.9050208@schokola.de> <54F0B1FE.3040600@schokola.de> <21409.1425200737@critter.freebsd.dk> Message-ID: <47527.1425200860@critter.freebsd.dk> -------- In message <21409.1425200737 at critter.freebsd.dk>, "Poul-Henning Kamp" writes: >The problem with !SO_LINGER (we tried it some years back) is that >all queued data is discarded the moment you call close(2), and we >have no way to know if our last response is still lingering in the >kernel's tcp-buffers and dribbling out to a slow client with a bad >mobile phone connection. Just to clarify: This was not meant to say "No Way!" but to point out the level of real-life testing that will be necessary for any change to this aspect. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From slink at schokola.de Sun Mar 1 16:30:20 2015 From: slink at schokola.de (Nils Goroll) Date: Sun, 01 Mar 2015 17:30:20 +0100 Subject: [PATCH] Instruct the kernel to reset the connection for SC_RX_TIMEOUT, and others In-Reply-To: <21409.1425200737@critter.freebsd.dk> References: <54EEF4F9.9050208@schokola.de> <54F0B1FE.3040600@schokola.de> <21409.1425200737@critter.freebsd.dk> Message-ID: <54F33E9C.50100@schokola.de> On 01/03/15 10:05, Poul-Henning Kamp wrote: >> >Closing TCP connections with SO_LINGER unset [...] > I pressume you mean "set" ? > > The problem with !SO_LINGER (we tried it some years back) is that > all queued data is discarded the moment you call close(2) I fail to find a mistake in the commitmsg of the patch I proposed: My understanding is that we get these semantics with linger on and linger timeout == 0. From Linux net/ipv4/tcp.c : } else if (sock_flag(sk, SOCK_LINGER) && !sk->sk_lingertime) { /* Check zero linger _after_ checking for unread data. */ sk->sk_prot->disconnect(sk, 0); NET_INC_STATS_USER(sock_net(sk), LINUX_MIB_TCPABORTONDATA); Whether or not data gets discarded depends on whether the linger timeout is set and reached before all outstanding data is sent. I think what we want for Varnish is: - attempt an orderly shutdown (3way FIN) for connections where we have actually sent data - reset (RST) the connection as quickly and efficiently as possible otherwise. In particular, a 1-way FIN only closes our write direction and we need a cooperating client to complete the duplex close, while sending an RST implies a full close. In the patch I suggest to limit sending RST to client read timeouts and other fatal conditions. We might want to consider using it for more scenarios should this turn out to be a good idea. If we wanted to be on the safe side, we could also limit the change to RX_TIMEOUT for the time being (which also would be sufficient for the particular real world case I am working on). On 01/03/15 10:07, Poul-Henning Kamp wrote: > real-life testing that will be necessary for any change to this aspect. I'd volunteer to test this on production systems. Some tcpdumps with linux 3.13.1 below. === current varnish origin master waiter timeout (5s default) === 16:47:40.982533 IP6 ::1.50785 > ::1.8080: Flags [S], seq 1964376644, win 43690, options [mss 65476,sackOK,TS val 576054 ecr 0,nop,wscale 7], length 0 16:47:40.982541 IP6 ::1.8080 > ::1.50785: Flags [R.], seq 0, ack 1964376645, win 0, length 0 16:47:40.982590 IP 127.0.0.1.54336 > 127.0.0.1.8080: Flags [S], seq 1857531233, win 43690, options [mss 65495,sackOK,TS val 576054 ecr 0,nop,wscale 7], length 0 16:47:40.982600 IP 127.0.0.1.8080 > 127.0.0.1.54336: Flags [S.], seq 2353711226, ack 1857531234, win 43690, options [mss 65495,sackOK,TS val 576054 ecr 576054,nop,wscale 7], length 0 16:47:40.982617 IP 127.0.0.1.54336 > 127.0.0.1.8080: Flags [.], ack 1, win 342, options [nop,nop,TS val 576054 ecr 576054], length 0 16:47:46.407450 IP 127.0.0.1.8080 > 127.0.0.1.54336: Flags [F.], seq 1, ack 1, win 342, options [nop,nop,TS val 577410 ecr 576054], length 0 16:47:46.407517 IP 127.0.0.1.54336 > 127.0.0.1.8080: Flags [F.], seq 1, ack 2, win 342, options [nop,nop,TS val 577410 ecr 577410], length 0 16:47:46.407537 IP 127.0.0.1.8080 > 127.0.0.1.54336: Flags [.], ack 2, win 342, options [nop,nop,TS val 577410 ecr 577410], length 0 === current varnish origin master request read timeout (2s default) === note: I suggested to increase this timeout to 7 seconds 16:48:44.162945 IP6 ::1.50794 > ::1.8080: Flags [S], seq 3924075046, win 43690, options [mss 65476,sackOK,TS val 591849 ecr 0,nop,wscale 7], length 0 16:48:44.162954 IP6 ::1.8080 > ::1.50794: Flags [R.], seq 0, ack 3924075047, win 0, length 0 16:48:44.163001 IP 127.0.0.1.54345 > 127.0.0.1.8080: Flags [S], seq 3268495574, win 43690, options [mss 65495,sackOK,TS val 591849 ecr 0,nop,wscale 7], length 0 16:48:44.163011 IP 127.0.0.1.8080 > 127.0.0.1.54345: Flags [S.], seq 1673884791, ack 3268495575, win 43690, options [mss 65495,sackOK,TS val 591849 ecr 591849,nop,wscale 7], length 0 16:48:44.163019 IP 127.0.0.1.54345 > 127.0.0.1.8080: Flags [.], ack 1, win 342, options [nop,nop,TS val 591849 ecr 591849], length 0 16:48:46.035540 IP 127.0.0.1.54345 > 127.0.0.1.8080: Flags [P.], seq 1:8, ack 1, win 342, options [nop,nop,TS val 592317 ecr 591849], length 7 16:48:46.035566 IP 127.0.0.1.8080 > 127.0.0.1.54345: Flags [.], ack 8, win 342, options [nop,nop,TS val 592317 ecr 592317], length 0 16:48:48.037686 IP 127.0.0.1.8080 > 127.0.0.1.54345: Flags [F.], seq 1, ack 8, win 342, options [nop,nop,TS val 592817 ecr 592317], length 0 16:48:48.037751 IP 127.0.0.1.54345 > 127.0.0.1.8080: Flags [F.], seq 8, ack 2, win 342, options [nop,nop,TS val 592817 ecr 592817], length 0 16:48:48.037773 IP 127.0.0.1.8080 > 127.0.0.1.54345: Flags [.], ack 9, win 342, options [nop,nop,TS val 592817 ecr 592817], length 0 === with SO_LINGER proposed patch - waiter timeout (5s default) === 16:51:59.473915 IP6 ::1.50797 > ::1.8080: Flags [S], seq 3546360798, win 43690, options [mss 65476,sackOK,TS val 640676 ecr 0,nop,wscale 7], length 0 16:51:59.473936 IP6 ::1.8080 > ::1.50797: Flags [R.], seq 0, ack 3546360799, win 0, length 0 16:51:59.474059 IP 127.0.0.1.54348 > 127.0.0.1.8080: Flags [S], seq 2664129049, win 43690, options [mss 65495,sackOK,TS val 640677 ecr 0,nop,wscale 7], length 0 16:51:59.474073 IP 127.0.0.1.8080 > 127.0.0.1.54348: Flags [S.], seq 3919584645, ack 2664129050, win 43690, options [mss 65495,sackOK,TS val 640677 ecr 640677,nop,wscale 7], length 0 16:51:59.474090 IP 127.0.0.1.54348 > 127.0.0.1.8080: Flags [.], ack 1, win 342, options [nop,nop,TS val 640677 ecr 640677], length 0 16:52:04.689346 IP 127.0.0.1.8080 > 127.0.0.1.54348: Flags [R.], seq 1, ack 1, win 342, options [nop,nop,TS val 641980 ecr 640677], length 0 === with SO_LINGER proposed patch - request read timeout (2s default) === 16:52:11.788384 IP6 ::1.50799 > ::1.8080: Flags [S], seq 1598190622, win 43690, options [mss 65476,sackOK,TS val 643755 ecr 0,nop,wscale 7], length 0 16:52:11.788401 IP6 ::1.8080 > ::1.50799: Flags [R.], seq 0, ack 1598190623, win 0, length 0 16:52:11.788474 IP 127.0.0.1.54350 > 127.0.0.1.8080: Flags [S], seq 510724766, win 43690, options [mss 65495,sackOK,TS val 643755 ecr 0,nop,wscale 7], length 0 16:52:11.788494 IP 127.0.0.1.8080 > 127.0.0.1.54350: Flags [S.], seq 1351958833, ack 510724767, win 43690, options [mss 65495,sackOK,TS val 643755 ecr 643755,nop,wscale 7], length 0 16:52:11.788513 IP 127.0.0.1.54350 > 127.0.0.1.8080: Flags [.], ack 1, win 342, options [nop,nop,TS val 643755 ecr 643755], length 0 16:52:14.144393 IP 127.0.0.1.54350 > 127.0.0.1.8080: Flags [P.], seq 1:7, ack 1, win 342, options [nop,nop,TS val 644344 ecr 643755], length 6 16:52:14.144424 IP 127.0.0.1.8080 > 127.0.0.1.54350: Flags [.], ack 7, win 342, options [nop,nop,TS val 644344 ecr 644344], length 0 16:52:16.146596 IP 127.0.0.1.8080 > 127.0.0.1.54350: Flags [R.], seq 1, ack 7, win 342, options [nop,nop,TS val 644845 ecr 644344], length 0 From slink at schokola.de Sun Mar 1 16:34:47 2015 From: slink at schokola.de (Nils Goroll) Date: Sun, 01 Mar 2015 17:34:47 +0100 Subject: [PATCH] Instruct the kernel to reset the connection for SC_RX_TIMEOUT, and others In-Reply-To: <54F33E9C.50100@schokola.de> References: <54EEF4F9.9050208@schokola.de> <54F0B1FE.3040600@schokola.de> <21409.1425200737@critter.freebsd.dk> <54F33E9C.50100@schokola.de> Message-ID: <54F33FA7.7050302@schokola.de> please disregard the SYN/RST cycle to ipv6 localhost ::1 from my tcpdumps On 01/03/15 17:30, Nils Goroll wrote: > 16:47:40.982533 IP6 ::1.50785 > ::1.8080: Flags [S], seq 1964376644, win 43690, > options [mss 65476,sackOK,TS val 576054 ecr 0,nop,wscale 7], length 0 > 16:47:40.982541 IP6 ::1.8080 > ::1.50785: Flags [R.], seq 0, ack 1964376645, win > 0, length 0 From slink at schokola.de Mon Mar 2 20:05:19 2015 From: slink at schokola.de (Nils Goroll) Date: Mon, 02 Mar 2015 21:05:19 +0100 Subject: Update2: [PATCH] Instruct the kernel to reset the connection for SC_RX_TIMEOUT, and others In-Reply-To: <21409.1425200737@critter.freebsd.dk> References: <54EEF4F9.9050208@schokola.de> <54F0B1FE.3040600@schokola.de> <21409.1425200737@critter.freebsd.dk> Message-ID: <54F4C27F.8020609@schokola.de> On 01/03/15 10:05, Poul-Henning Kamp wrote: > The problem with !SO_LINGER (we tried it some years back) is that > all queued data is discarded the moment you call close(2), and we > have no way to know if our last response is still lingering in the > kernel's tcp-buffers and dribbling out to a slow client with a bad > mobile phone connection. I finally have understood the issue with http keepalive here and that kernels behave differently than how I thought they would. Also I have now understood that phk had already invested significant effort into this and I basically also ended up "out-of-ideas". Except for this: As long was we know that we cannot possibly have written anything, we should we OK to close with RST. The attached updated patch takes a simplistic approach to this. Nils -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Instruct-the-kernel-to-reset-the-connection-for-SC_R.patch Type: text/x-patch Size: 7134 bytes Desc: not available URL: From slink at schokola.de Mon Mar 2 20:33:00 2015 From: slink at schokola.de (Nils Goroll) Date: Mon, 02 Mar 2015 21:33:00 +0100 Subject: Update3: [PATCH] Instruct the kernel to reset the connection for SC_RX_TIMEOUT, and others In-Reply-To: <54F4C27F.8020609@schokola.de> References: <54EEF4F9.9050208@schokola.de> <54F0B1FE.3040600@schokola.de> <21409.1425200737@critter.freebsd.dk> <54F4C27F.8020609@schokola.de> Message-ID: <54F4C8FC.7000203@schokola.de> phk had the great idea to add a "quick close" for SC_RX_BAD and SC_RX_JUNK. Another updated patch attached. On 02/03/15 21:05, Nils Goroll wrote: > On 01/03/15 10:05, Poul-Henning Kamp wrote: >> The problem with !SO_LINGER (we tried it some years back) is that >> all queued data is discarded the moment you call close(2), and we >> have no way to know if our last response is still lingering in the >> kernel's tcp-buffers and dribbling out to a slow client with a bad >> mobile phone connection. > > I finally have understood the issue with http keepalive here and that kernels > behave differently than how I thought they would. Also I have now understood > that phk had already invested significant effort into this and I basically also > ended up "out-of-ideas". > > Except for this: As long was we know that we cannot possibly have written > anything, we should we OK to close with RST. > > The attached updated patch takes a simplistic approach to this. > > Nils > > > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Instruct-the-kernel-to-reset-the-connection-for-SC_R.patch Type: text/x-patch Size: 8844 bytes Desc: not available URL: From phk at phk.freebsd.dk Tue Mar 3 10:08:33 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 03 Mar 2015 10:08:33 +0000 Subject: Update3: [PATCH] Instruct the kernel to reset the connection for SC_RX_TIMEOUT, and others In-Reply-To: <54F4C8FC.7000203@schokola.de> References: <54EEF4F9.9050208@schokola.de> <54F0B1FE.3040600@schokola.de> <21409.1425200737@critter.freebsd.dk> <54F4C27F.8020609@schokola.de> <54F4C8FC.7000203@schokola.de> Message-ID: <81311.1425377313@critter.freebsd.dk> -------- In message <54F4C8FC.7000203 at schokola.de>, Nils Goroll writes: >phk had the great idea to add a "quick close" for SC_RX_BAD and SC_RX_JUNK. > >Another updated patch attached. Just to reiterate: I want to see evindence of serious real-world testing before I will consider this patch for inclusion. We tried stuff in this area once before and overlooked a lot of corner-cases, I want to be sure we don't repeat such mistakes. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Tue Mar 3 10:39:12 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 03 Mar 2015 10:39:12 +0000 Subject: Cache request body and user-accesible functions. In-Reply-To: References: Message-ID: <6728.1425379152@critter.freebsd.dk> -------- In message , Federico Schwindt writes: As an initial observation I think req.body can only have type BLOB. Anything else is just asking for problems I don't want to have, or involves so much special-casing that it's painful to even think about. >hash_data(req.body); > >In this case hash_data() will internally know what (length) to use. This >might work in Varnish core but will require specific handling outside >though. So one way to do that would be to make req.body be a BLOB, and make it hash_data(STRING|BLOB). That requires serious VCC hacking and will cause code duplication (ie: hash_data__string(), hash_data__blob() etc.) Another way of getting the same effect, which may need less VCC hacking, makes it hash_data(BLOB), and adds a general (automatic) conversion from STRING to BLOB (just like we have for -> STRING today.) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From fgsch at lodoss.net Tue Mar 3 23:40:38 2015 From: fgsch at lodoss.net (Federico Schwindt) Date: Tue, 3 Mar 2015 23:40:38 +0000 Subject: Cache request body and user-accesible functions. In-Reply-To: <6728.1425379152@critter.freebsd.dk> References: <6728.1425379152@critter.freebsd.dk> Message-ID: That sounds good and will work for hash_data() and things like hashing. Any thoughts on regex operations? Also, is this 4.1 material? On Tue, Mar 3, 2015 at 10:39 AM, Poul-Henning Kamp wrote: > -------- > In message < > CAJV_h0ZHZtm50LXZZ6pNJVJRVpnmYKuWWHw+xL3t+C4v8ryMAw at mail.gmail.com> > , Federico Schwindt writes: > > As an initial observation I think req.body can only have type BLOB. > > Anything else is just asking for problems I don't want to have, or > involves so much special-casing that it's painful to even think about. > > >hash_data(req.body); > > > >In this case hash_data() will internally know what (length) to use. This > >might work in Varnish core but will require specific handling outside > >though. > > So one way to do that would be to make req.body be a BLOB, and make it > hash_data(STRING|BLOB). That requires serious VCC hacking and will > cause code duplication (ie: hash_data__string(), hash_data__blob() etc.) > > Another way of getting the same effect, which may need less VCC > hacking, makes it hash_data(BLOB), and adds a general (automatic) > conversion from STRING to BLOB (just like we have for -> STRING > today.) > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Wed Mar 4 15:32:06 2015 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 04 Mar 2015 15:32:06 +0000 Subject: NB: new test j00000 is special... Message-ID: <10010.1425483126@critter.freebsd.dk> I have just added a new test "j00000" which does a basic run-through test of the jail_unix code. The test will be skipped unless A) it is run as root (geteuid() == 0) B) The system has a user named "varnish" C) The system has a group named "varnish" Let me know if this causes any trouble... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From slink at schokola.de Fri Mar 6 17:11:03 2015 From: slink at schokola.de (Nils Goroll) Date: Fri, 06 Mar 2015 18:11:03 +0100 Subject: [PATCH] Add a flag to mark 304 backend response processing (aka, Backend IMS/INM) Message-ID: <54F9DFA7.3050006@schokola.de> This is a compromise discussed at VDD15Q1 after a previous suggestion to expose the 304 response status directly to VCL. This way, VCL can differenciate between 200 and 304 responses, but simpler processing logic just checking for status 200 can be preserved where differenciating is not required. -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Add-a-flag-to-mark-304-backend-response-processing-a.patch Type: text/x-patch Size: 5069 bytes Desc: not available URL: From Rayson.MT.Chan at pccw.com Mon Mar 9 05:43:55 2015 From: Rayson.MT.Chan at pccw.com (Chan, Rayson MT) Date: Mon, 9 Mar 2015 13:43:55 +0800 Subject: Porting vmod to version 4 Message-ID: Hi all, I am trying to port this vmod: libvmod-parsereq (found here: https://github.com/xcir/libvmod-parsereq) to Varnish 4.0, which has useful features such as inspecting POST variables. However, I am stumped with a couple of things in the vmod: 1) The vmod calls a function called VRT_panic() in Varnish 3.0 vrt.h but that's not present in Varnish 4.0. What is the proper substitute for that in V4? 2) One part of vmod calls a custom function named vmod_Hook_Miss_opt_post_lookup() when encountered a R_STP_MISS. The inside the custom function vmod_Hook...() has a couple of lines like this: ... //send body sp->sendbody = 1; //modify request method VRT_l_bereq_request(sp,"POST",vrt_magic_string_end); ... But there are no equivalent of sp->sendbody and VRT_l_bereq_request() in V4. Does anybody know what they meant to do? Any kind of help is appreciated. Regards, Rayson Chan - HKT From arianna.aondio at varnish-software.com Mon Mar 9 09:44:08 2015 From: arianna.aondio at varnish-software.com (Arianna Aondio) Date: Mon, 9 Mar 2015 10:44:08 +0100 Subject: Cache_req_body patch for bug #1664 Message-ID: Hi everyone, before Hamburg VDD I had sent an email regarding req.body handling and user-accesible functions. I attach a patch for bug #1664: std.cache_req_body(BYTES size)doesn't have errors handling on the provided size limitation. The proposed solution let Varnish either buffer the whole request body or fail the request. If this patch is ok, I can then start digging into user-accesible functions. Comments much appreciated. -- Arianna Aondio Software Developer | Varnish Software AS Mobile: +47 980 62 619 We Make Websites Fly www.varnish-software.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Cache_req_body-error-handling.patch Type: text/x-diff Size: 3312 bytes Desc: not available URL: From perbu at varnish-software.com Mon Mar 9 10:18:32 2015 From: perbu at varnish-software.com (Per Buer) Date: Mon, 9 Mar 2015 11:18:32 +0100 Subject: On tests Message-ID: Hi. Intermittently failing tests seems to be an issue. The way the current tests are scheduled and run (autocrap and make) are not perfect for handling this. I've written a replacement to give us a bit more flexibility wrt to how tests are run. The below gisted program is a multithreaded python program that run tests. The tests are typically run in parallel (-j n). If a test fails it is scheduled for single threaded execution and these are executed after the rest of the tests are done. This should hopefully reduce the amount of intermittent failures quite with only a small increase in time taken to execute. If we want to gather long term statistics having this program interface with a database of sorts should be reasonably trivial. Another option would be to start giving hints on how the test should be executed within the test itself. By having a header on the test that the testrunner could read one can set up certain parameters. # parallel=0,load<1,timeout=15s,platforms=!netbsd,retries=3 Let me know if I should commit this and start replacement of the current way tests are run. Draft code available at: https://gist.github.com/perbu/3a722df0b168ac8f73bb Syntax: Usage: testrunner