From tfheen at varnish-cache.org  Tue Nov  1 08:43:54 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Tue, 01 Nov 2011 09:43:54 +0100
Subject: [master] fe425ce Avoid infinite loop in regsuball
Message-ID: <E1RL9wo-0002rR-Ee@project.varnish-software.com>

commit fe425ce63a03b0de1a9c257c4e27cf1defe6a862
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Tue Nov 1 09:25:22 2011 +0100

    Avoid infinite loop in regsuball
    
    Also optimise out a few strlen calls in favour of just tracking the
    lengths.
    
    Fixes: #1047

diff --git a/bin/varnishd/cache_vrt_re.c b/bin/varnishd/cache_vrt_re.c
index e6f854b..955b7bf 100644
--- a/bin/varnishd/cache_vrt_re.c
+++ b/bin/varnishd/cache_vrt_re.c
@@ -90,13 +90,16 @@ VRT_regsub(const struct sess *sp, int all, const char *str, void *re,
 	char *b0;
 	const char *s;
 	unsigned u, x;
+	int options = 0;
+	size_t len;
 
 	AN(re);
 	if (str == NULL)
 		str = "";
 	t = re;
 	memset(ovector, 0, sizeof(ovector));
-	i = VRE_exec(t, str, strlen(str), 0, 0, ovector, 30,
+	len = strlen(str);
+	i = VRE_exec(t, str, len, 0, options, ovector, 30,
 	    &params->vre_limits);
 
 	/* If it didn't match, we can return the original string */
@@ -132,10 +135,12 @@ VRT_regsub(const struct sess *sp, int all, const char *str, void *re,
 			}
 		}
 		str += ovector[1];
+		len -= ovector[1];
 		if (!all)
 			break;
 		memset(&ovector, 0, sizeof(ovector));
-		i = VRE_exec(t, str, strlen(str), 0, 0, ovector, 30,
+		options |= VRE_NOTEMPTY_ATSTART;
+		i = VRE_exec(t, str, len, 0, options, ovector, 30,
 		    &params->vre_limits);
 		if (i < VRE_ERROR_NOMATCH ) {
 			WSP(sp, SLT_VCL_Error,
@@ -145,8 +150,7 @@ VRT_regsub(const struct sess *sp, int all, const char *str, void *re,
 	} while (i != VRE_ERROR_NOMATCH);
 
 	/* Copy suffix to match */
-	l = strlen(str) + 1;
-	Tadd(&res, str, l);
+	Tadd(&res, str, len+1);
 	if (res.b >= res.e) {
 		WS_Release(sp->http->ws, 0);
 		return (str);
diff --git a/bin/varnishtest/tests/c00047.vtc b/bin/varnishtest/tests/c00047.vtc
new file mode 100644
index 0000000..4b23194
--- /dev/null
+++ b/bin/varnishtest/tests/c00047.vtc
@@ -0,0 +1,33 @@
+varnishtest "Test VCL regsuball()"
+
+server s1 {
+	rxreq 
+	txresp \
+		-hdr "foo: barbar" \
+		-hdr "bar: bbbar"
+} -start
+
+varnish v1 -vcl+backend { 
+	sub vcl_fetch {
+		set beresp.http.baz1 = regsuball(beresp.http.foo, "barb", "zz");
+		set beresp.http.baz2 = regsuball(beresp.http.foo, "ar", "zz");
+		set beresp.http.baz3 = regsuball(beresp.http.foo, "^", "baz");
+		set beresp.http.baz4 = regsuball(beresp.http.foo, "^[;]*", "baz");
+		set beresp.http.baz5 = regsuball(beresp.http.bar, "^b*", "b");
+		set beresp.http.baz6 = regsuball(beresp.http.foo, "^b*", "z");
+		set beresp.http.baz7 = regsuball(beresp.http.foo, "ping", "pong");
+	}
+} -start 
+
+client c1 {
+	txreq -url "/"
+	rxresp
+	expect resp.status == 200
+	expect resp.http.baz1 == "zzar"
+	expect resp.http.baz2 == "bzzbzz"
+	expect resp.http.baz3 == "bazbarbar"
+	expect resp.http.baz4 == "bazbarbar"
+	expect resp.http.baz5 == "bar"
+	expect resp.http.baz6 == "zarbar"
+	expect resp.http.baz7 == "barbar"
+} -run
diff --git a/include/vre.h b/include/vre.h
index 29c1b08..a1206e5 100644
--- a/include/vre.h
+++ b/include/vre.h
@@ -49,6 +49,7 @@ typedef struct vre vre_t;
 
 /* And those to PCRE options */
 #define VRE_CASELESS           0x00000001
+#define VRE_NOTEMPTY_ATSTART   0x10000000
 
 vre_t *VRE_compile(const char *, int, const char **, int *);
 int VRE_exec(const vre_t *code, const char *subject, int length,



From phk at varnish-cache.org  Tue Nov  1 09:41:44 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 01 Nov 2011 10:41:44 +0100
Subject: [master] f6536dac Use scalbn(3) rather than exp2(3),
	it should be faster and more portable.
Message-ID: <E1RLAqm-0000Qn-F6@project.varnish-software.com>

commit f6536dac9a6c9fdda4236e0ad92cfc4cdd6dd34a
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 1 09:41:00 2011 +0000

    Use scalbn(3) rather than exp2(3), it should be faster and more portable.
    
    Fixes	#1046

diff --git a/bin/varnishd/cache_dir_random.c b/bin/varnishd/cache_dir_random.c
index f18c7d7..d6570ed 100644
--- a/bin/varnishd/cache_dir_random.c
+++ b/bin/varnishd/cache_dir_random.c
@@ -91,7 +91,7 @@ vdi_random_sha(const char *input, ssize_t len)
 	SHA256_Init(&ctx);
 	SHA256_Update(&ctx, input, len);
 	SHA256_Final(sign, &ctx);
-	return (vle32dec(sign) / exp2(32));
+	return (scalbn(vle32dec(sign), -32));
 }
 
 /*
@@ -113,11 +113,11 @@ vdi_random_init_seed(const struct vdi_random *vs, const struct sess *sp)
 		break;
 	case c_hash:
 		AN(sp->digest);
-		retval = vle32dec(sp->digest) / exp2(32);
+		retval = scalbn(vle32dec(sp->digest), -32);
 		break;
 	case c_random:
 	default:
-		retval = random() / exp2(31);
+		retval = scalbn(random(), -31);
 		break;
 	}
 	return (retval);



From phk at varnish-cache.org  Tue Nov  1 09:41:46 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 01 Nov 2011 10:41:46 +0100
Subject: [master] bced3f5 Merge branch 'master' of
	ssh://git.varnish-cache.org/git/varnish-cache
Message-ID: <E1RLAqo-0000Qs-OH@project.varnish-software.com>

commit bced3f5bdadee976bf8208ee61d4b6d293ba07be
Merge: f6536dac fe425ce
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 1 09:41:40 2011 +0000

    Merge branch 'master' of ssh://git.varnish-cache.org/git/varnish-cache



From tfheen at varnish-cache.org  Tue Nov  1 09:54:07 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Tue, 01 Nov 2011 10:54:07 +0100
Subject: [master] aef8337 Use new-style backend naming in output
Message-ID: <E1RLB2l-0001Ur-A4@project.varnish-software.com>

commit aef83370f8ad506dbca5b9df0494ca9b04712945
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Tue Nov 1 10:43:02 2011 +0100

    Use new-style backend naming in output
    
    Prefer vcl_name(ipv4,ipv6,port) to the old vcl_name in output.

diff --git a/bin/varnishd/cache_backend.c b/bin/varnishd/cache_backend.c
index 7bacd26..1a3ae5b 100644
--- a/bin/varnishd/cache_backend.c
+++ b/bin/varnishd/cache_backend.c
@@ -143,8 +143,8 @@ bes_conn_try(const struct sess *sp, struct vbc *vc, const struct vdi_simple *vs)
 {
 	int s;
 	struct backend *bp = vs->backend;
-	char abuf1[VTCP_ADDRBUFSIZE], abuf2[VTCP_ADDRBUFSIZE];
-	char pbuf1[VTCP_PORTBUFSIZE], pbuf2[VTCP_PORTBUFSIZE];
+	char abuf1[VTCP_ADDRBUFSIZE];
+	char pbuf1[VTCP_PORTBUFSIZE];
 
 	CHECK_OBJ_NOTNULL(vs, VDI_SIMPLE_MAGIC);
 
@@ -185,10 +185,8 @@ bes_conn_try(const struct sess *sp, struct vbc *vc, const struct vdi_simple *vs)
 	} else {
 		vc->vsl_id = s | VSL_BACKENDMARKER;
 		VTCP_myname(s, abuf1, sizeof abuf1, pbuf1, sizeof pbuf1);
-		VTCP_name(vc->addr, vc->addrlen,
-		    abuf2, sizeof abuf2, pbuf2, sizeof pbuf2);
-		WSL(sp->wrk, SLT_BackendOpen, vc->vsl_id, "%s %s %s %s %s",
-		    vs->backend->vcl_name, abuf1, pbuf1, abuf2, pbuf2);
+		WSL(sp->wrk, SLT_BackendOpen, vc->vsl_id, "%s %s %s ",
+		    vs->backend->display_name, abuf1, pbuf1);
 	}
 
 }
@@ -347,13 +345,13 @@ vbe_GetVbe(const struct sess *sp, struct vdi_simple *vs)
 			/* XXX locking of stats */
 			VSC_C_main->backend_reuse += 1;
 			WSP(sp, SLT_Backend, "%d %s %s",
-			    vc->fd, sp->director->vcl_name, bp->vcl_name);
+			    vc->fd, sp->director->vcl_name, bp->display_name);
 			vc->vdis = vs;
 			vc->recycled = 1;
 			return (vc);
 		}
 		VSC_C_main->backend_toolate++;
-		WSL(sp->wrk, SLT_BackendClose, vc->vsl_id, "%s", bp->vcl_name);
+		WSL(sp->wrk, SLT_BackendClose, vc->vsl_id, "%s", bp->display_name);
 
 		/* Checkpoint log to flush all info related to this connection
 		   before the OS reuses the FD */
@@ -388,7 +386,7 @@ vbe_GetVbe(const struct sess *sp, struct vdi_simple *vs)
 	vc->backend = bp;
 	VSC_C_main->backend_conn++;
 	WSP(sp, SLT_Backend, "%d %s %s",
-	    vc->fd, sp->director->vcl_name, bp->vcl_name);
+	    vc->fd, sp->director->vcl_name, bp->display_name);
 	vc->vdis = vs;
 	return (vc);
 }
diff --git a/bin/varnishd/cache_backend.h b/bin/varnishd/cache_backend.h
index 47e709d..8937673 100644
--- a/bin/varnishd/cache_backend.h
+++ b/bin/varnishd/cache_backend.h
@@ -115,6 +115,7 @@ struct backend {
 	struct lock		mtx;
 
 	char			*vcl_name;
+	char			*display_name;
 	char			*ipv4_addr;
 	char			*ipv6_addr;
 	char			*port;
diff --git a/bin/varnishd/cache_backend_cfg.c b/bin/varnishd/cache_backend_cfg.c
index b10512d..a4f3285 100644
--- a/bin/varnishd/cache_backend_cfg.c
+++ b/bin/varnishd/cache_backend_cfg.c
@@ -213,6 +213,7 @@ VBE_AddBackend(struct cli *cli, const struct vrt_backend *vb)
 	 * so we cannot simply reference the VCL's copy of things.
 	 */
 	REPLACE(b->vcl_name, vb->vcl_name);
+	REPLACE(b->display_name, buf);
 	REPLACE(b->ipv4_addr, vb->ipv4_addr);
 	REPLACE(b->ipv6_addr, vb->ipv6_addr);
 	REPLACE(b->port, vb->port);
@@ -283,9 +284,9 @@ cli_debug_backend(struct cli *cli, const char * const *av, void *priv)
 	ASSERT_CLI();
 	VTAILQ_FOREACH(b, &backends, list) {
 		CHECK_OBJ_NOTNULL(b, BACKEND_MAGIC);
-		VCLI_Out(cli, "%p %s(%s,%s,:%s) %d %d\n",
-		    b, b->vcl_name, b->ipv4_addr, b->ipv6_addr, b->port,
-		    b->refcount, b->n_conn);
+		VCLI_Out(cli, "%p %s %d %d\n",
+			 b, b->display_name,
+			 b->refcount, b->n_conn);
 	}
 }
 
diff --git a/bin/varnishd/cache_dir.c b/bin/varnishd/cache_dir.c
index e95b015..59ef5fc 100644
--- a/bin/varnishd/cache_dir.c
+++ b/bin/varnishd/cache_dir.c
@@ -50,7 +50,7 @@ VDI_CloseFd(struct worker *wrk)
 
 	bp = wrk->vbc->backend;
 
-	WSL(wrk, SLT_BackendClose, wrk->vbc->vsl_id, "%s", bp->vcl_name);
+	WSL(wrk, SLT_BackendClose, wrk->vbc->vsl_id, "%s", bp->display_name);
 
 	/* Checkpoint log to flush all info related to this connection
 	   before the OS reuses the FD */
@@ -78,7 +78,7 @@ VDI_RecycleFd(struct worker *wrk)
 
 	bp = wrk->vbc->backend;
 
-	WSL(wrk, SLT_BackendReuse, wrk->vbc->vsl_id, "%s", bp->vcl_name);
+	WSL(wrk, SLT_BackendReuse, wrk->vbc->vsl_id, "%s", bp->display_name);
 	/*
 	 * Flush the shmlog, so that another session reusing this backend
 	 * will log chronologically later than our use of it.
diff --git a/bin/varnishd/cache_panic.c b/bin/varnishd/cache_panic.c
index 231adf1..4076d26 100644
--- a/bin/varnishd/cache_panic.c
+++ b/bin/varnishd/cache_panic.c
@@ -96,7 +96,7 @@ pan_vbc(const struct vbc *vbc)
 	be = vbc->backend;
 
 	VSB_printf(vsp, "  backend = %p fd = %d {\n", be, vbc->fd);
-	VSB_printf(vsp, "    vcl_name = \"%s\",\n", be->vcl_name);
+	VSB_printf(vsp, "    display_name = \"%s\",\n", be->display_name);
 	VSB_printf(vsp, "  },\n");
 }
 



From phk at varnish-cache.org  Tue Nov  1 10:49:43 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 01 Nov 2011 11:49:43 +0100
Subject: [master] 374c3a2 Add a missing case: ESI parent document gunzip'ed
	but included document gzip'ed.
Message-ID: <E1RLBuZ-0002yO-32@project.varnish-software.com>

commit 374c3a235fec63ee4f213696fc43731fd329ac2e
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 1 10:49:09 2011 +0000

    Add a missing case:  ESI parent document gunzip'ed but included
    document gzip'ed.
    
    Fixes	#1029

diff --git a/bin/varnishd/cache_esi_parse.c b/bin/varnishd/cache_esi_parse.c
index 663894a..f6eb367 100644
--- a/bin/varnishd/cache_esi_parse.c
+++ b/bin/varnishd/cache_esi_parse.c
@@ -801,7 +801,7 @@ VEP_Parse(const struct worker *w, const char *p, size_t l)
 				vep->state = VEP_ATTR;
 			} else if (p < e) {
 				vep_error(vep,
-				    "XML 1.0 Illegal attribute tart char");
+				    "XML 1.0 Illegal attribute start char");
 				vep->state = VEP_TAGERROR;
 			}
 		} else if (vep->state == VEP_TAGERROR) {
diff --git a/bin/varnishd/cache_response.c b/bin/varnishd/cache_response.c
index 04d2da5..87b0919 100644
--- a/bin/varnishd/cache_response.c
+++ b/bin/varnishd/cache_response.c
@@ -307,6 +307,9 @@ RES_WriteObj(struct sess *sp)
 		ESI_Deliver(sp);
 	} else if (sp->wrk->res_mode & RES_ESI_CHILD && sp->wrk->gzip_resp) {
 		ESI_DeliverChild(sp);
+	} else if (sp->wrk->res_mode & RES_ESI_CHILD &&
+	    !sp->wrk->gzip_resp && sp->obj->gziped) {
+		res_WriteGunzipObj(sp);
 	} else if (sp->wrk->res_mode & RES_GUNZIP) {
 		res_WriteGunzipObj(sp);
 	} else {
diff --git a/bin/varnishtest/tests/r01029.vtc b/bin/varnishtest/tests/r01029.vtc
new file mode 100644
index 0000000..2817bba
--- /dev/null
+++ b/bin/varnishtest/tests/r01029.vtc
@@ -0,0 +1,34 @@
+varnishtest "#1029"
+
+server s1 {
+	rxreq
+	expect req.url == "/bar"
+	txresp -gzipbody {[bar]}
+
+	rxreq
+	expect req.url == "/foo"
+	txresp -body {<h1>FOO<esi:include src="/bar"/>BARF</h1>}
+
+} -start
+
+varnish v1 -vcl+backend {
+	sub vcl_fetch {
+		set beresp.do_esi = true;
+		if (req.url == "/foo") {
+			set beresp.ttl = 0s;
+		} else {
+			set beresp.ttl = 10m;
+		}
+	}
+} -start
+
+client c1 {
+	txreq -url "/bar" -hdr "Accept-Encoding: gzip"
+	rxresp
+	gunzip
+	expect resp.bodylen == 5
+
+	txreq -url "/foo" -hdr "Accept-Encoding: gzip"
+	rxresp
+	expect resp.bodylen == 21
+} -run



From perbu at varnish-cache.org  Thu Nov  3 09:22:18 2011
From: perbu at varnish-cache.org (Per Buer)
Date: Thu, 03 Nov 2011 10:22:18 +0100
Subject: [master] 2c28590 more on hit_for_pass
Message-ID: <E1RLtV4-0004Ty-Pk@project.varnish-software.com>

commit 2c28590561557aa2cb181296e0b63255e81fb24b
Author: Per Buer <perbu at varnish-software.com>
Date:   Thu Sep 1 15:41:39 2011 +0200

    more on hit_for_pass

diff --git a/doc/sphinx/reference/vcl.rst b/doc/sphinx/reference/vcl.rst
index 4f005d0..14d0450 100644
--- a/doc/sphinx/reference/vcl.rst
+++ b/doc/sphinx/reference/vcl.rst
@@ -532,12 +532,18 @@ vcl_fetch
   error code [reason]
     Return the specified error code to the client and abandon the request.
 
-  hit_for_pass
-    Pass in fetch. This will create a hit_for_pass object. Note that
-    the TTL for the hit_for_pass object will be set to what the
-    current value of beresp.ttl. Control will be handled to
-    vcl_deliver on the current request, but subsequent requests will
-    go directly to vcl_pass based on the hit_for_pass object.
+  hit_for_pass 
+    Pass in fetch. Passes the object without caching it. This will
+    create a socalled hit_for_pass object which has the side effect
+    that the decision not to cache will be cached. This is to allow
+    would-be uncachable requests to be passed to the backend at the
+    same time. The same logic is not necessary in vcl_recv because
+    this happens before any potential queueing for an object takes
+    place.  Note that the TTL for the hit_for_pass object will be set
+    to what the current value of beresp.ttl is. Control will be
+    handled to vcl_deliver on the current request, but subsequent
+    requests will go directly to vcl_pass based on the hit_for_pass
+    object.
 
   restart
     Restart the transaction. Increases the restart counter. If the number 



From perbu at varnish-cache.org  Thu Nov  3 09:22:19 2011
From: perbu at varnish-cache.org (Per Buer)
Date: Thu, 03 Nov 2011 10:22:19 +0100
Subject: [master] e14ade5 Merge branch 'master' of
	ssh://git.varnish-cache.org/git/varnish-cache
Message-ID: <E1RLtV5-0004U3-AT@project.varnish-software.com>

commit e14ade5cf2d370c737122f1fc63b8a6883016fa2
Merge: 2c28590 e000786
Author: Per Buer <perbu at varnish-software.com>
Date:   Mon Sep 5 13:01:44 2011 +0200

    Merge branch 'master' of ssh://git.varnish-cache.org/git/varnish-cache



From perbu at varnish-cache.org  Thu Nov  3 09:22:19 2011
From: perbu at varnish-cache.org (Per Buer)
Date: Thu, 03 Nov 2011 10:22:19 +0100
Subject: [master] 82344cb Merge branch 'master' of
	ssh://git.varnish-cache.org/git/varnish-cache
Message-ID: <E1RLtV5-0004UD-F5@project.varnish-software.com>

commit 82344cb91b320ae7720895fc636cd6cbd34c388c
Merge: e14ade5 87ae60f
Author: Per Buer <perbu at varnish-software.com>
Date:   Mon Sep 12 10:37:47 2011 +0200

    Merge branch 'master' of ssh://git.varnish-cache.org/git/varnish-cache



From perbu at varnish-cache.org  Thu Nov  3 09:22:19 2011
From: perbu at varnish-cache.org (Per Buer)
Date: Thu, 03 Nov 2011 10:22:19 +0100
Subject: [master] d8b670d Merge branch 'master' of
	ssh://git.varnish-cache.org/git/varnish-cache
Message-ID: <E1RLtV5-0004UO-LB@project.varnish-software.com>

commit d8b670d15ac31cebd1d58030468791a0868678a3
Merge: 82344cb 6084802
Author: Per Buer <perbu at varnish-software.com>
Date:   Wed Sep 21 15:08:01 2011 +0200

    Merge branch 'master' of ssh://git.varnish-cache.org/git/varnish-cache



From perbu at varnish-cache.org  Thu Nov  3 09:22:20 2011
From: perbu at varnish-cache.org (Per Buer)
Date: Thu, 03 Nov 2011 10:22:20 +0100
Subject: [master] dd4d545 Merge branch 'master' of
	ssh://git.varnish-cache.org/git/varnish-cache
Message-ID: <E1RLtV6-0004Ua-SX@project.varnish-software.com>

commit dd4d54557c7b862f762bbf8eabac3f9fe54c64b4
Merge: d8b670d 374c3a2
Author: Per Buer <perbu at varnish-software.com>
Date:   Thu Nov 3 09:24:03 2011 +0100

    Merge branch 'master' of ssh://git.varnish-cache.org/git/varnish-cache



From perbu at varnish-cache.org  Thu Nov  3 09:22:21 2011
From: perbu at varnish-cache.org (Per Buer)
Date: Thu, 03 Nov 2011 10:22:21 +0100
Subject: [master] 2672b7a document the compression behaviour in Varnish 3.0
Message-ID: <E1RLtV7-0004Ur-78@project.varnish-software.com>

commit 2672b7ab7b11fa4bf4484759062ec3c15f46d2ef
Author: Per Buer <perbu at varnish-software.com>
Date:   Thu Nov 3 10:22:13 2011 +0100

    document the compression behaviour in Varnish 3.0

diff --git a/doc/sphinx/tutorial/compression.rst b/doc/sphinx/tutorial/compression.rst
new file mode 100644
index 0000000..0b8d1e8
--- /dev/null
+++ b/doc/sphinx/tutorial/compression.rst
@@ -0,0 +1,75 @@
+.. _tutorial-compression:
+
+Compression
+~~~~~~~~~~~
+
+New in Varnish 3.0 was native support for compression, using gzip
+encoding. *Before* 3.0, Varnish would never compress objects. 
+
+In Varnish 3.0 compression defaults to "on", meaning that it tries to
+be smart and do the sensible thing.
+
+If you don't want Varnish tampering with the encoding you can disable
+compression all together by setting the parameter http_gzip_support to
+*false*. Please see man :ref:`ref-varnishd` for details.
+
+
+Default behaviour
+~~~~~~~~~~~~~~~~~
+
+The default for Varnish is to check if the client supports our
+compression scheme (gzip) and if it does it will override the
+Accept-Encoding header and set it to "gzip".
+
+When Varnish then issues a backend request the Accept-Encoding will
+then only consist of "gzip". If the server responds with gzip'ed
+content it will be stored in memory in its compressed form. If the
+backend sends content in clear text it will be stored like that.
+
+You can make Varnish compress content before storing it in cache in
+vcl_fetch by setting do_gzip to true, like this::
+
+   sub vcl_fetch {
+        if (beresp.http.content-type ~ "text") {
+                set beresp.do_gzip = true;
+        }
+  }
+
+Please make sure that you don't try to compress content that is
+incompressable, like jpgs, gifs and mp3. You'll only waste CPU
+cycles. You can also uncompress objects before storing it in memory by
+setting do_gunzip to *true* but I have no idea why anybody would want
+to do that.
+
+Generally, Varnish doesn't use much CPU so it might make more sense to
+have Varnish spend CPU cycles compressing content than doing it in
+your web- or application servers, which are more likely to be
+CPU-bound.
+
+GZIP and ESI
+~~~~~~~~~~~~
+
+If you are using Edge Side Includes you'll be happy to note that ESI
+and GZIP work together really well. Varnish will magically decompress
+the content to do the ESI-processing, then recompress it for efficient
+storage and delivery. 
+
+
+Clients that don't support gzip
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If the client does not support gzip the Accept-Encoding header is left
+alone and we'll end up serving whatever we get from the backend
+server. Remember that the Backend might tell Varnish to *Vary* on the
+Accept-Encoding.
+
+If the client does not support gzip but we've already got a compressed
+version of the page in memory Varnish will automatically decompress
+the page while delivering it.
+
+
+A random outburst
+~~~~~~~~~~~~~~~~~
+
+Poul has written :ref:`phk_gzip` which talks abit more about how the
+implementation works. 
diff --git a/doc/sphinx/tutorial/index.rst b/doc/sphinx/tutorial/index.rst
index 59fd94e..32c2264 100644
--- a/doc/sphinx/tutorial/index.rst
+++ b/doc/sphinx/tutorial/index.rst
@@ -25,6 +25,7 @@ separate topic. Good luck.
 	cookies
 	vary
 	purging
+	compression
 	esi
 	virtualised
 	advanced_backend_servers



From tfheen at varnish-cache.org  Thu Nov  3 12:22:04 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Thu, 03 Nov 2011 13:22:04 +0100
Subject: [master] 41c39c8 Add health control of backends from CLI
Message-ID: <E1RLwJ2-0006nH-BR@project.varnish-software.com>

commit 41c39c857907f3b26f8ce540863121b27288649c
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Thu Nov 3 13:03:51 2011 +0100

    Add health control of backends from CLI
    
    Make it possible to mark backends as sick or healthy by way of
    
      backend.set_health $backendname (sick|healthy|auto)

diff --git a/bin/varnishd/cache_backend.c b/bin/varnishd/cache_backend.c
index 1a3ae5b..cad637b 100644
--- a/bin/varnishd/cache_backend.c
+++ b/bin/varnishd/cache_backend.c
@@ -256,7 +256,10 @@ vbe_Healthy(const struct vdi_simple *vs, const struct sess *sp)
 	backend = vs->backend;
 	CHECK_OBJ_NOTNULL(backend, BACKEND_MAGIC);
 
-	if (!backend->healthy)
+	if (backend->admin_health == from_probe && !backend->healthy)
+		return (0);
+
+	if (backend->admin_health == sick)
 		return (0);
 
 	/* VRT/VCC sets threshold to UINT_MAX to mark that it's not
@@ -267,6 +270,9 @@ vbe_Healthy(const struct vdi_simple *vs, const struct sess *sp)
 	else
 		threshold = vs->vrt->saintmode_threshold;
 
+	if (backend->admin_health == healthy)
+		threshold = UINT_MAX;
+
 	/* Saintmode is disabled */
 	if (threshold == 0)
 		return (1);
diff --git a/bin/varnishd/cache_backend.h b/bin/varnishd/cache_backend.h
index 8937673..880c429 100644
--- a/bin/varnishd/cache_backend.h
+++ b/bin/varnishd/cache_backend.h
@@ -106,6 +106,12 @@ struct trouble {
  * An instance of a backend from a VCL program.
  */
 
+enum health_status {
+	healthy,
+	sick,
+	from_probe
+};
+
 struct backend {
 	unsigned		magic;
 #define BACKEND_MAGIC		0x64c4c7c6
@@ -130,6 +136,7 @@ struct backend {
 
 	struct vbp_target	*probe;
 	unsigned		healthy;
+	enum health_status	admin_health;
 	VTAILQ_HEAD(, trouble)	troublelist;
 
 	struct VSC_C_vbe	*vsc;
diff --git a/bin/varnishd/cache_backend_cfg.c b/bin/varnishd/cache_backend_cfg.c
index a4f3285..8168be0 100644
--- a/bin/varnishd/cache_backend_cfg.c
+++ b/bin/varnishd/cache_backend_cfg.c
@@ -38,6 +38,7 @@
 #include "cache.h"
 
 #include "cache_backend.h"
+#include "vcli.h"
 #include "vcli_priv.h"
 #include "vrt.h"
 
@@ -229,6 +230,7 @@ VBE_AddBackend(struct cli *cli, const struct vrt_backend *vb)
 	assert(b->ipv4 != NULL || b->ipv6 != NULL);
 
 	b->healthy = 1;
+	b->admin_health = from_probe;
 
 	VTAILQ_INSERT_TAIL(&backends, b, list);
 	VSC_C_main->n_backend++;
@@ -274,25 +276,151 @@ VRT_fini_dir(struct cli *cli, struct director *b)
 
 /*--------------------------------------------------------------------*/
 
+static int
+backend_find(const char *matcher, struct backend **r, int n)
+{
+	struct backend *b;
+	char *vcl_name;
+	char *s;
+	char *match_ip = NULL;
+	char *match_port = NULL;
+	int found = 0;
+
+	s = strchr(matcher, '(');
+
+	if (s == NULL) {
+		/* Simple match, max one hit */
+		VTAILQ_FOREACH(b, &backends, list) {
+			CHECK_OBJ_NOTNULL(b, BACKEND_MAGIC);
+			if (strcmp(b->vcl_name, matcher) == 0) {
+				if (r && found < n)
+					r[found] = b;
+				found++;
+			}
+		}
+		return found;
+	}
+
+	vcl_name = strndup(matcher, s - matcher);
+	AN(vcl_name);
+	s++;
+	while (*s != ')') {
+		if (*s == ':') {
+			/* Port */
+			s++;
+			match_port = s;
+			if (!(s = strchr(match_port, ','))) {
+				s = strchr(match_port, ')');
+			}
+			XXXAN(s);
+			match_port = strndup(match_port, s - match_port);
+			AN(match_port);
+			if (*s == ',')
+				s++;
+		} else {
+			/* IP */
+			match_ip = s;
+			if (!(s = strchr(match_ip, ','))) {
+				s = strchr(match_ip, ')');
+			}
+			XXXAN(s);
+			match_ip = strndup(match_ip, s - match_ip);
+			AN(match_ip);
+			if (*s == ',')
+				s++;
+		}
+	}
+	VTAILQ_FOREACH(b, &backends, list) {
+		CHECK_OBJ_NOTNULL(b, BACKEND_MAGIC);
+		if (match_port && strcmp(b->port, match_port) != 0)
+			continue;
+		if (match_ip &&
+		    (strcmp(b->ipv4_addr, match_ip) != 0) &&
+		    (strcmp(b->ipv6_addr, match_ip) != 0))
+			continue;
+		if (strcmp(b->vcl_name, vcl_name) == 0) {
+			if (r && found < n)
+				r[found] = b;
+			found++;
+		}
+	}
+	return found;
+}
+
 static void
-cli_debug_backend(struct cli *cli, const char * const *av, void *priv)
+cli_backend_list(struct cli *cli, const char * const *av, void *priv)
 {
 	struct backend *b;
+	const char *ah;
 
 	(void)av;
 	(void)priv;
 	ASSERT_CLI();
+	VCLI_Out(cli, "%-30s %10s %15s %15s", "Backend name",
+		 "Conns", "Probed healthy", "Admin health");
 	VTAILQ_FOREACH(b, &backends, list) {
 		CHECK_OBJ_NOTNULL(b, BACKEND_MAGIC);
-		VCLI_Out(cli, "%p %s %d %d\n",
-			 b, b->display_name,
-			 b->refcount, b->n_conn);
+		if (b->admin_health == from_probe) {
+			ah = "Auto";
+		} else if (b->admin_health == sick) {
+			ah = "Sick";
+		} else {
+			ah = "Healthy";
+		}
+		VCLI_Out(cli, "\n%-30s %10d %15s %15s",
+			 b->display_name,
+			 b->refcount,
+			 (b ->healthy ? "Yes" : "No"),
+			 ah);
+	}
+}
+
+static void
+cli_backend_set_health(struct cli *cli, const char * const *av, void *priv)
+{
+	struct backend **b;
+	enum health_status state;
+	int n;
+	const char *wstate;
+
+	(void)av;
+	(void)priv;
+	ASSERT_CLI();
+	wstate = av[3];
+	if (strcmp(wstate, "healthy") == 0) {
+		state = healthy;
+	} else if (strcmp(wstate, "sick") == 0) {
+		state = sick;
+	} else if (strcmp(wstate, "auto") == 0) {
+		state = from_probe;
+	} else {
+		VCLI_Out(cli, "Invalid state %s", wstate);
+		VCLI_SetResult(cli, CLIS_CANT);
+		return;
+	}
+	n = backend_find(av[2], NULL, 0);
+	if (n == 0) {
+		VCLI_Out(cli, "No matching backends");
+		VCLI_SetResult(cli, CLIS_CANT);
+		return;
+	}
+
+	b = calloc(n, sizeof(struct backend *));
+	AN(b);
+	n = backend_find(av[2], b, n);
+
+	VCLI_Out(cli, "Set state to %s for the following backends:", wstate);
+	for (int i = 0; i < n; i++) {
+		b[i]->admin_health = state;
+		VCLI_Out(cli, "\n\t%s", b[i]->display_name);
 	}
 }
 
-static struct cli_proto debug_cmds[] = {
-	{ "debug.backend", "debug.backend",
-	    "\tExamine Backend internals\n", 0, 0, "d", cli_debug_backend },
+static struct cli_proto backend_cmds[] = {
+	{ "backend.list", "backend.list",
+	    "\tList all backends\n", 0, 0, "d", cli_backend_list },
+	{ "backend.set_health", "backend.set_health matcher state",
+	    "\tShow a backend\n", 2, 2, "d", cli_backend_set_health },
 	{ NULL }
 };
 
@@ -303,5 +431,5 @@ VBE_Init(void)
 {
 
 	Lck_New(&VBE_mtx, lck_vbe);
-	CLI_AddFuncs(debug_cmds);
+	CLI_AddFuncs(backend_cmds);
 }
diff --git a/bin/varnishtest/tests/c00048.vtc b/bin/varnishtest/tests/c00048.vtc
new file mode 100644
index 0000000..ed8671f
--- /dev/null
+++ b/bin/varnishtest/tests/c00048.vtc
@@ -0,0 +1,55 @@
+varnishtest "Forcing health of backends"
+
+server s1 -repeat 3 {
+	rxreq
+	txresp
+} -start
+
+varnish v1 -vcl {
+	backend s1 {
+		.host = "${s1_addr}";
+		.port = "${s1_port}";
+		.probe = {
+			.window = 8;
+			.initial = 7;
+			.threshold = 8;
+			.interval = 10s;
+		}
+	}
+
+	sub vcl_recv {
+		return(pass);
+	}
+
+} -start
+
+delay 1
+
+varnish v1 -cliok "backend.set_health s1 auto"
+
+client c1 {
+	txreq
+	rxresp
+	expect resp.status == 200
+} -run
+
+varnish v1 -cliok "backend.set_health s1 sick"
+
+client c1 {
+	txreq
+	rxresp
+	expect resp.status == 503
+} -run
+
+varnish v1 -cliok "backend.set_health s1 healthy"
+
+client c1 {
+	txreq
+	rxresp
+	expect resp.status == 200
+} -run
+
+varnish v1 -clierr 300 "backend.set_health s1 foo"
+varnish v1 -clierr 300 "backend.set_health s2 foo"
+varnish v1 -clierr 300 "backend.set_health s2 auto"
+



From lkarsten at varnish-cache.org  Thu Nov  3 18:02:11 2011
From: lkarsten at varnish-cache.org (Lasse Karstensen)
Date: Thu, 03 Nov 2011 19:02:11 +0100
Subject: [3.0] 831318a tempcommit: works for single lines
Message-ID: <E1RM1cB-0006y3-Mt@project.varnish-software.com>

commit 831318ada192681f356f4ef2293be29e4da31882
Author: Lasse Karstensen <lasse at varnish-software.com>
Date:   Fri Oct 28 12:59:44 2011 +0200

    tempcommit: works for single lines

diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c
index ad9249d..59167fc 100644
--- a/bin/varnishncsa/varnishncsa.c
+++ b/bin/varnishncsa/varnishncsa.c
@@ -105,6 +105,8 @@ static struct logline {
 	uint64_t bitmap;		/* Bitmap for regex matches */
 	VTAILQ_HEAD(, hdr) req_headers; /* Request headers */
 	VTAILQ_HEAD(, hdr) resp_headers; /* Response headers */
+	char *log1;			/* How the request was handled (hit/miss/pass/pipe) */
+//	VTAILQ_HEAD(, hdr) vcl_log;     /* vcl.log() entries */
 } **ll;
 
 struct VSM_data *vd;
@@ -216,6 +218,23 @@ resp_header(struct logline *l, const char *name)
 	return NULL;
 }
 
+static char *
+vcl_log(struct logline *l, const char *name)
+{
+	struct hdr *h;
+	// JEJE
+	VTAILQ_FOREACH(h, &l->resp_headers, list) {
+		if (strcasecmp(h->key, name) == 0) {
+			return h->value;
+			break;
+		}
+	}
+	return NULL;
+}
+
+
+
+
 static void
 clean_logline(struct logline *lp)
 {
@@ -462,6 +481,28 @@ collect_client(struct logline *lp, enum VSL_tag_e tag, unsigned spec,
 		}
 		break;
 
+	case SLT_VCL_Log:
+		if(!lp->active)
+			break;
+ 		lp->log1 = trimline(ptr, end);
+/*
+		if (strncmp(ptr, "hit", len) == 0) {
+			lp->df_hitmiss = "hit";
+			lp->df_handling = "hit";
+		} else if (strncmp(ptr, "miss", len) == 0) {
+			lp->df_hitmiss = "miss";
+			lp->df_handling = "miss";
+		} else if (strncmp(ptr, "pass", len) == 0) {
+			lp->df_hitmiss = "miss";
+			lp->df_handling = "pass";
+		} else if (strncmp(ptr, "pipe", len) == 0) {
+			clean_logline(lp);
+			break;
+		}
+*/
+		break;
+
+
 	case SLT_VCL_call:
 		if(!lp->active)
 			break;
@@ -690,6 +731,7 @@ h_ncsa(void *priv, enum VSL_tag_e tag, unsigned fd,
 				memcpy(fname, p+1, tmp-p-2);
 				fname[tmp-p-2] = 0;
 			}
+//			printf("fname is: %s", fname);
 
 			switch (type) {
 			case 'i':
@@ -715,7 +757,12 @@ h_ncsa(void *priv, enum VSL_tag_e tag, unsigned fd,
 					VSB_cat(os, (lp->df_handling ? lp->df_handling : "-"));
 					p = tmp;
 					break;
+				} else if (strcmp(fname, "Varnish:vcllog") == 0) {
+					VSB_cat(os, (lp->log1 ? lp->log1 : "-"));
+					p = tmp;
+					break;
 				}
+
 			default:
 				fprintf(stderr, "Unknown format starting at: %s\n", --p);
 				exit(1);



From lkarsten at varnish-cache.org  Thu Nov  3 18:02:14 2011
From: lkarsten at varnish-cache.org (Lasse Karstensen)
Date: Thu, 03 Nov 2011 19:02:14 +0100
Subject: [3.0] 15e4f62 the somewhat correct parts
Message-ID: <E1RM1cE-0006yD-5P@project.varnish-software.com>

commit 15e4f6209415600f5a82f909af2314e859800893
Author: Lasse Karstensen <lasse at varnish-software.com>
Date:   Fri Oct 28 14:30:40 2011 +0200

    the somewhat correct parts

diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c
index 59167fc..9a047f9 100644
--- a/bin/varnishncsa/varnishncsa.c
+++ b/bin/varnishncsa/varnishncsa.c
@@ -105,8 +105,7 @@ static struct logline {
 	uint64_t bitmap;		/* Bitmap for regex matches */
 	VTAILQ_HEAD(, hdr) req_headers; /* Request headers */
 	VTAILQ_HEAD(, hdr) resp_headers; /* Response headers */
-	char *log1;			/* How the request was handled (hit/miss/pass/pipe) */
-//	VTAILQ_HEAD(, hdr) vcl_log;     /* vcl.log() entries */
+	VTAILQ_HEAD(, hdr) vcl_log;     /* VLC_Log entries */
 } **ll;
 
 struct VSM_data *vd;
@@ -222,8 +221,7 @@ static char *
 vcl_log(struct logline *l, const char *name)
 {
 	struct hdr *h;
-	// JEJE
-	VTAILQ_FOREACH(h, &l->resp_headers, list) {
+	VTAILQ_FOREACH(h, &l->vcl_log, list) {
 		if (strcasecmp(h->key, name) == 0) {
 			return h->value;
 			break;
@@ -261,6 +259,14 @@ clean_logline(struct logline *lp)
 		freez(h->value);
 		freez(h);
 	}
+	VTAILQ_FOREACH_SAFE(h, &lp->vcl_log, list, h2) {
+		VTAILQ_REMOVE(&lp->vcl_log, h, list);
+		freez(h->key);
+		freez(h->value);
+		freez(h);
+	}
+
+
 #undef freez
 	memset(lp, 0, sizeof *lp);
 }
@@ -484,22 +490,24 @@ collect_client(struct logline *lp, enum VSL_tag_e tag, unsigned spec,
 	case SLT_VCL_Log:
 		if(!lp->active)
 			break;
- 		lp->log1 = trimline(ptr, end);
-/*
-		if (strncmp(ptr, "hit", len) == 0) {
-			lp->df_hitmiss = "hit";
-			lp->df_handling = "hit";
-		} else if (strncmp(ptr, "miss", len) == 0) {
-			lp->df_hitmiss = "miss";
-			lp->df_handling = "miss";
-		} else if (strncmp(ptr, "pass", len) == 0) {
-			lp->df_hitmiss = "miss";
-			lp->df_handling = "pass";
-		} else if (strncmp(ptr, "pipe", len) == 0) {
-			clean_logline(lp);
+ 		//lp->log1 = trimline(ptr, end);
+
+		// Extract key, value from logline.
+		split = strchr(ptr, ':');
+		if (split == NULL)
 			break;
-		}
-*/
+
+		struct hdr *h;
+		h = malloc(sizeof(struct hdr));
+		// XXX: what is this AN() business? probably important. :)
+		AN(h);
+		AN(split);
+
+		h->key = trimline(ptr, split);
+		h->value = trimline(split+1, end);
+
+		// put onto the linked list that is log entries.
+		VTAILQ_INSERT_HEAD(&lp->vcl_log, h, list);
 		break;
 
 



From lkarsten at varnish-cache.org  Thu Nov  3 18:02:17 2011
From: lkarsten at varnish-cache.org (Lasse Karstensen)
Date: Thu, 03 Nov 2011 19:02:17 +0100
Subject: [3.0] 9902494 cleanup
Message-ID: <E1RM1cH-0006yR-Ks@project.varnish-software.com>

commit 9902494d3579e29637e0952925904232dfc95e31
Author: Lasse Karstensen <lasse at varnish-software.com>
Date:   Fri Oct 28 14:32:36 2011 +0200

    cleanup

diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c
index 9a047f9..dbde1b4 100644
--- a/bin/varnishncsa/varnishncsa.c
+++ b/bin/varnishncsa/varnishncsa.c
@@ -490,23 +490,19 @@ collect_client(struct logline *lp, enum VSL_tag_e tag, unsigned spec,
 	case SLT_VCL_Log:
 		if(!lp->active)
 			break;
- 		//lp->log1 = trimline(ptr, end);
 
-		// Extract key, value from logline.
 		split = strchr(ptr, ':');
 		if (split == NULL)
 			break;
 
 		struct hdr *h;
 		h = malloc(sizeof(struct hdr));
-		// XXX: what is this AN() business? probably important. :)
 		AN(h);
 		AN(split);
 
 		h->key = trimline(ptr, split);
 		h->value = trimline(split+1, end);
 
-		// put onto the linked list that is log entries.
 		VTAILQ_INSERT_HEAD(&lp->vcl_log, h, list);
 		break;
 



From lkarsten at varnish-cache.org  Thu Nov  3 18:02:20 2011
From: lkarsten at varnish-cache.org (Lasse Karstensen)
Date: Thu, 03 Nov 2011 19:02:20 +0100
Subject: [3.0] 2b1905e works
Message-ID: <E1RM1cK-0006yp-Pq@project.varnish-software.com>

commit 2b1905e67eb6b9d4011555cb71268ae7ea398d79
Author: Lasse Karstensen <lasse at varnish-software.com>
Date:   Fri Oct 28 15:48:31 2011 +0200

    works

diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c
index dbde1b4..bb954eb 100644
--- a/bin/varnishncsa/varnishncsa.c
+++ b/bin/varnishncsa/varnishncsa.c
@@ -725,6 +725,10 @@ h_ncsa(void *priv, enum VSL_tag_e tag, unsigned fd,
 		case '{': {
 			const char *h, *tmp;
 			char fname[100], type;
+
+			// maybe
+			const char *key2;
+
 			tmp = p;
 			type = 0;
 			while (*tmp != '\0' && *tmp != '}')
@@ -761,11 +765,50 @@ h_ncsa(void *priv, enum VSL_tag_e tag, unsigned fd,
 					VSB_cat(os, (lp->df_handling ? lp->df_handling : "-"));
 					p = tmp;
 					break;
-				} else if (strcmp(fname, "Varnish:vcllog") == 0) {
-					VSB_cat(os, (lp->log1 ? lp->log1 : "-"));
-					p = tmp;
-					break;
+//				} else if (strcmp(fname, "Varnish:vcllog") == 0) {
+//					VSB_cat(os, (lp->log1 ? lp->log1 : "-"));
+//					p = tmp;
+//					break;
 				}
+			case 'l':
+                                // std.log("Foo: bar")
+                                // %{Foo}l 
+			        printf("case l was hit with fname: %s\n", fname);
+				// Extract "Foo" from fname
+				char *delim;
+			        char keyword[100];
+				char *kwptr;
+				char *startpos;
+
+				// keyword = "init";
+
+                                delim = strchr(fname, ':');
+				printf(": found at %d\n", delim - fname);
+				// startpos = delim - fname;
+
+				// buffer overflow all the way.
+				startpos = strncpy(keyword, delim+1, sizeof keyword);
+				printf("meh");
+				printf("startpos is: %d\n", startpos);
+
+				printf("rest is: %s\n", delim+1); // startpos);
+				printf("keyword is: %s\n", keyword);
+
+				//keyword = delim + 1;
+
+//				printf("split2: %d\n", *split2);
+
+ //                               key2 = trimline(fname, split2);
+				//&&printf("key: %s\n", key2);
+                                //keyword = strcpy("bar"; // trimline(fname+1, split);
+				//keyword = "foo";
+
+                                //strcpy(key2, "bar"); // trimline(fname+1, split);
+                                h = vcl_log(lp, keyword); 
+                                VSB_cat(os, h ? h : "-");
+                                p = tmp;
+                                break;
+///////
 
 			default:
 				fprintf(stderr, "Unknown format starting at: %s\n", --p);



From lkarsten at varnish-cache.org  Thu Nov  3 18:02:24 2011
From: lkarsten at varnish-cache.org (Lasse Karstensen)
Date: Thu, 03 Nov 2011 19:02:24 +0100
Subject: [3.0] 4aa876e works
Message-ID: <E1RM1cO-0006zJ-3s@project.varnish-software.com>

commit 4aa876ee8fdfe496491fc0d5260071dbc5e0ce88
Author: Lasse Karstensen <lasse at varnish-software.com>
Date:   Mon Oct 31 10:23:40 2011 +0100

    works

diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c
index bb954eb..3591eb5 100644
--- a/bin/varnishncsa/varnishncsa.c
+++ b/bin/varnishncsa/varnishncsa.c
@@ -57,6 +57,8 @@
  * TODO:		- Maybe rotate/compress log
  */
 
+#define MAX_VCLLOG_KEYLENGTH 100
+
 #include "config.h"
 
 #include <ctype.h>
@@ -726,9 +728,6 @@ h_ncsa(void *priv, enum VSL_tag_e tag, unsigned fd,
 			const char *h, *tmp;
 			char fname[100], type;
 
-			// maybe
-			const char *key2;
-
 			tmp = p;
 			type = 0;
 			while (*tmp != '\0' && *tmp != '}')
@@ -739,7 +738,6 @@ h_ncsa(void *priv, enum VSL_tag_e tag, unsigned fd,
 				memcpy(fname, p+1, tmp-p-2);
 				fname[tmp-p-2] = 0;
 			}
-//			printf("fname is: %s", fname);
 
 			switch (type) {
 			case 'i':
@@ -765,53 +763,27 @@ h_ncsa(void *priv, enum VSL_tag_e tag, unsigned fd,
 					VSB_cat(os, (lp->df_handling ? lp->df_handling : "-"));
 					p = tmp;
 					break;
-//				} else if (strcmp(fname, "Varnish:vcllog") == 0) {
-//					VSB_cat(os, (lp->log1 ? lp->log1 : "-"));
-//					p = tmp;
-//					break;
 				}
-			case 'l':
-                                // std.log("Foo: bar")
-                                // %{Foo}l 
-			        printf("case l was hit with fname: %s\n", fname);
-				// Extract "Foo" from fname
-				char *delim;
-			        char keyword[100];
-				char *kwptr;
-				char *startpos;
-
-				// keyword = "init";
-
-                                delim = strchr(fname, ':');
-				printf(": found at %d\n", delim - fname);
-				// startpos = delim - fname;
-
-				// buffer overflow all the way.
-				startpos = strncpy(keyword, delim+1, sizeof keyword);
-				printf("meh");
-				printf("startpos is: %d\n", startpos);
-
-				printf("rest is: %s\n", delim+1); // startpos);
-				printf("keyword is: %s\n", keyword);
-
-				//keyword = delim + 1;
-
-//				printf("split2: %d\n", *split2);
-
- //                               key2 = trimline(fname, split2);
-				//&&printf("key: %s\n", key2);
-                                //keyword = strcpy("bar"; // trimline(fname+1, split);
-				//keyword = "foo";
-
-                                //strcpy(key2, "bar"); // trimline(fname+1, split);
-                                h = vcl_log(lp, keyword); 
-                                VSB_cat(os, h ? h : "-");
-                                p = tmp;
-                                break;
-///////
+			case 'L': {
+					char *delim;
+					char keyword[MAX_VCLLOG_KEYLENGTH];
+					char *startpos;
+
+					// Extract "key" from fname
+					delim = strchr(fname, ':');
+					if (delim == NULL) {
+					    break;
+					}
+					startpos = strncpy(keyword, delim+1, MAX_VCLLOG_KEYLENGTH);
+					
+					h = vcl_log(lp, keyword); 
+					VSB_cat(os, h ? h : "-");
+					p = tmp;
+					break;
+				}
 
 			default:
-				fprintf(stderr, "Unknown format starting at: %s\n", --p);
+				fprintf(stderr, "Unknown extended format starting at: %s\n", --p);
 				exit(1);
 			}
 			break;
@@ -821,8 +793,8 @@ h_ncsa(void *priv, enum VSL_tag_e tag, unsigned fd,
 		default:
 			fprintf(stderr, "Unknown format starting at: %s\n", --p);
 			exit(1);
-		}
-	}
+		} // switch
+	} // for 
 	VSB_putc(os, '\n');
 
 	/* flush the stream */



From lkarsten at varnish-cache.org  Thu Nov  3 18:02:27 2011
From: lkarsten at varnish-cache.org (Lasse Karstensen)
Date: Thu, 03 Nov 2011 19:02:27 +0100
Subject: [3.0] 9d693b0 remove extra linefeed
Message-ID: <E1RM1cR-0006zt-4N@project.varnish-software.com>

commit 9d693b0ddbf93dda6a6c22400da93df9ce2ee32e
Author: Lasse Karstensen <lasse at varnish-software.com>
Date:   Mon Oct 31 10:30:46 2011 +0100

    remove extra linefeed

diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c
index 3591eb5..064d49a 100644
--- a/bin/varnishncsa/varnishncsa.c
+++ b/bin/varnishncsa/varnishncsa.c
@@ -727,7 +727,6 @@ h_ncsa(void *priv, enum VSL_tag_e tag, unsigned fd,
 		case '{': {
 			const char *h, *tmp;
 			char fname[100], type;
-
 			tmp = p;
 			type = 0;
 			while (*tmp != '\0' && *tmp != '}')



From lkarsten at varnish-cache.org  Thu Nov  3 18:02:31 2011
From: lkarsten at varnish-cache.org (Lasse Karstensen)
Date: Thu, 03 Nov 2011 19:02:31 +0100
Subject: [3.0] f6195d5 remove compile errors with -Werror
Message-ID: <E1RM1cV-00070X-6n@project.varnish-software.com>

commit f6195d5cc503cc081a04533bfa526b03837fcede
Author: Lasse Karstensen <lasse at varnish-software.com>
Date:   Mon Oct 31 10:39:48 2011 +0100

    remove compile errors with -Werror

diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c
index 064d49a..dc7fb6f 100644
--- a/bin/varnishncsa/varnishncsa.c
+++ b/bin/varnishncsa/varnishncsa.c
@@ -766,14 +766,13 @@ h_ncsa(void *priv, enum VSL_tag_e tag, unsigned fd,
 			case 'L': {
 					char *delim;
 					char keyword[MAX_VCLLOG_KEYLENGTH];
-					char *startpos;
 
 					// Extract "key" from fname
 					delim = strchr(fname, ':');
 					if (delim == NULL) {
 					    break;
 					}
-					startpos = strncpy(keyword, delim+1, MAX_VCLLOG_KEYLENGTH);
+					strncpy(keyword, delim+1, MAX_VCLLOG_KEYLENGTH);
 					
 					h = vcl_log(lp, keyword); 
 					VSB_cat(os, h ? h : "-");



From lkarsten at varnish-cache.org  Thu Nov  3 18:25:16 2011
From: lkarsten at varnish-cache.org (Lasse Karstensen)
Date: Thu, 03 Nov 2011 19:25:16 +0100
Subject: [master] 878036c Revert "Add support for JSON output format". Work in
	progress, pushed in wrong direction.
Message-ID: <E1RM1yW-00076Q-7h@project.varnish-software.com>

commit 878036c77425393f58a53d05b5de97d063a7cddf
Author: Lasse Karstensen <lasse at varnish-software.com>
Date:   Thu Nov 3 19:24:37 2011 +0100

    Revert "Add support for JSON output format". Work in progress, pushed in wrong direction.
    
    This reverts commit 5f45e4ca4996027638a0b28949aed68ffe7eaf8c.

diff --git a/bin/varnishstat/varnishstat.c b/bin/varnishstat/varnishstat.c
index 00ddda8..3c6c4b5 100644
--- a/bin/varnishstat/varnishstat.c
+++ b/bin/varnishstat/varnishstat.c
@@ -82,60 +82,6 @@ do_xml(struct VSM_data *vd)
 	printf("</varnishstat>\n");
 }
 
-
-/*--------------------------------------------------------------------*/
-
-struct json_priv {
-	int first;
-};
-
-static int
-do_json_cb(void *priv, const struct VSC_point * const pt)
-{
-	uint64_t val;
-	struct json_priv *jp;
-	jp = priv;
-
-	assert(!strcmp(pt->fmt, "uint64_t"));
-	val = *(const volatile uint64_t*)pt->ptr;
-
-	if (jp->first ) jp->first = 0; else printf(",\n");
-	printf("\t\"%s\": {", pt->name);
-	if (strcmp(pt->class, ""))
-		printf("\t\t\"type\": \"%s\",\n", pt->class);
-	if (strcmp(pt->ident, ""))
-		printf("\t\t\"ident\": \"%s\",\n", pt->ident);
-	printf("\"value\": %ju, ", val);
-
-	printf("\"flag\": \"%c\",", pt->flag);
-	printf("\"description\": \"%s\"", pt->desc);
-	printf("}"); // closes name section.
-	if (jp->first) printf("\n");
-	return (0);
-}
-
-static void
-do_json(struct VSM_data *vd)
-{
-	/// TODO: support jsonp?
-	char time_stamp[20];
-	time_t now;
-
-	struct json_priv jp;
-
-	memset(&jp, 0, sizeof jp);
-	jp.first = 1;
-
-	printf("{\n"); 
-	now = time(NULL);
-
-	(void)strftime(time_stamp, 20, "%Y-%m-%dT%H:%M:%S", localtime(&now));
-	printf("\t\"timestamp\": \"%s\",\n", time_stamp);
-	(void)VSC_Iter(vd, do_json_cb, &jp);
-	printf("\n}");
-}
-
-
 /*--------------------------------------------------------------------*/
 
 struct once_priv {
@@ -143,7 +89,6 @@ struct once_priv {
 	int pad;
 };
 
-
 static int
 do_once_cb(void *priv, const struct VSC_point * const pt)
 {
@@ -246,12 +191,12 @@ main(int argc, char * const *argv)
 	int c;
 	struct VSM_data *vd;
 	const struct VSC_C_main *VSC_C_main;
-	int delay = 1, once = 0, xml = 0, json = 0;
+	int delay = 1, once = 0, xml = 0;
 
 	vd = VSM_New();
 	VSC_Setup(vd);
 
-	while ((c = getopt(argc, argv, VSC_ARGS "1f:lVw:xj")) != -1) {
+	while ((c = getopt(argc, argv, VSC_ARGS "1f:lVw:x")) != -1) {
 		switch (c) {
 		case '1':
 			once = 1;
@@ -270,9 +215,6 @@ main(int argc, char * const *argv)
 		case 'x':
 			xml = 1;
 			break;
-		case 'j':
-			json = 1;
-			break;
 		default:
 			if (VSC_Arg(vd, c, optarg) > 0)
 				break;
@@ -287,8 +229,6 @@ main(int argc, char * const *argv)
 
 	if (xml)
 		do_xml(vd);
-	else if (json)
-		do_json(vd);
 	else if (once)
 		do_once(vd, VSC_C_main);
 	else



From lkarsten at varnish-cache.org  Thu Nov  3 18:30:09 2011
From: lkarsten at varnish-cache.org (Lasse Karstensen)
Date: Thu, 03 Nov 2011 19:30:09 +0100
Subject: [3.0] 17b750a Reverting 6 commits seems like more confusion. This is
	varnishncsa.c from d18336e59ed66512f5af2f2d26bb0a7261d012c8.
Message-ID: <E1RM23F-0007R0-C6@project.varnish-software.com>

commit 17b750a485685ab96472a341e98416a8dd55a08f
Author: Lasse Karstensen <lasse at varnish-software.com>
Date:   Thu Nov 3 19:30:02 2011 +0100

    Reverting 6 commits seems like more confusion. This is varnishncsa.c from d18336e59ed66512f5af2f2d26bb0a7261d012c8.

diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c
index dc7fb6f..ad9249d 100644
--- a/bin/varnishncsa/varnishncsa.c
+++ b/bin/varnishncsa/varnishncsa.c
@@ -57,8 +57,6 @@
  * TODO:		- Maybe rotate/compress log
  */
 
-#define MAX_VCLLOG_KEYLENGTH 100
-
 #include "config.h"
 
 #include <ctype.h>
@@ -107,7 +105,6 @@ static struct logline {
 	uint64_t bitmap;		/* Bitmap for regex matches */
 	VTAILQ_HEAD(, hdr) req_headers; /* Request headers */
 	VTAILQ_HEAD(, hdr) resp_headers; /* Response headers */
-	VTAILQ_HEAD(, hdr) vcl_log;     /* VLC_Log entries */
 } **ll;
 
 struct VSM_data *vd;
@@ -219,22 +216,6 @@ resp_header(struct logline *l, const char *name)
 	return NULL;
 }
 
-static char *
-vcl_log(struct logline *l, const char *name)
-{
-	struct hdr *h;
-	VTAILQ_FOREACH(h, &l->vcl_log, list) {
-		if (strcasecmp(h->key, name) == 0) {
-			return h->value;
-			break;
-		}
-	}
-	return NULL;
-}
-
-
-
-
 static void
 clean_logline(struct logline *lp)
 {
@@ -261,14 +242,6 @@ clean_logline(struct logline *lp)
 		freez(h->value);
 		freez(h);
 	}
-	VTAILQ_FOREACH_SAFE(h, &lp->vcl_log, list, h2) {
-		VTAILQ_REMOVE(&lp->vcl_log, h, list);
-		freez(h->key);
-		freez(h->value);
-		freez(h);
-	}
-
-
 #undef freez
 	memset(lp, 0, sizeof *lp);
 }
@@ -489,26 +462,6 @@ collect_client(struct logline *lp, enum VSL_tag_e tag, unsigned spec,
 		}
 		break;
 
-	case SLT_VCL_Log:
-		if(!lp->active)
-			break;
-
-		split = strchr(ptr, ':');
-		if (split == NULL)
-			break;
-
-		struct hdr *h;
-		h = malloc(sizeof(struct hdr));
-		AN(h);
-		AN(split);
-
-		h->key = trimline(ptr, split);
-		h->value = trimline(split+1, end);
-
-		VTAILQ_INSERT_HEAD(&lp->vcl_log, h, list);
-		break;
-
-
 	case SLT_VCL_call:
 		if(!lp->active)
 			break;
@@ -763,25 +716,8 @@ h_ncsa(void *priv, enum VSL_tag_e tag, unsigned fd,
 					p = tmp;
 					break;
 				}
-			case 'L': {
-					char *delim;
-					char keyword[MAX_VCLLOG_KEYLENGTH];
-
-					// Extract "key" from fname
-					delim = strchr(fname, ':');
-					if (delim == NULL) {
-					    break;
-					}
-					strncpy(keyword, delim+1, MAX_VCLLOG_KEYLENGTH);
-					
-					h = vcl_log(lp, keyword); 
-					VSB_cat(os, h ? h : "-");
-					p = tmp;
-					break;
-				}
-
 			default:
-				fprintf(stderr, "Unknown extended format starting at: %s\n", --p);
+				fprintf(stderr, "Unknown format starting at: %s\n", --p);
 				exit(1);
 			}
 			break;
@@ -791,8 +727,8 @@ h_ncsa(void *priv, enum VSL_tag_e tag, unsigned fd,
 		default:
 			fprintf(stderr, "Unknown format starting at: %s\n", --p);
 			exit(1);
-		} // switch
-	} // for 
+		}
+	}
 	VSB_putc(os, '\n');
 
 	/* flush the stream */



From tfheen at varnish-cache.org  Fri Nov  4 11:04:58 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Fri, 04 Nov 2011 12:04:58 +0100
Subject: [master] c3fb33c Expose VSL_Name2Tag in libvarnishapi
Message-ID: <E1RMHZy-0002PG-5t@project.varnish-software.com>

commit c3fb33c70e8a92ec2cd18e8314458d48a630a47f
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Fri Nov 4 12:02:31 2011 +0100

    Expose VSL_Name2Tag in libvarnishapi

diff --git a/lib/libvarnishapi/libvarnishapi.map b/lib/libvarnishapi/libvarnishapi.map
index 4df8577..83f5169 100644
--- a/lib/libvarnishapi/libvarnishapi.map
+++ b/lib/libvarnishapi/libvarnishapi.map
@@ -68,3 +68,10 @@ LIBVARNISHAPI_1.0 {
   local:
 	*;
 };
+
+LIBVARNISHAPI_1.1 {
+  global:
+	# Functions:
+	VSL_Name2Tag;
+	# Variables:
+} LIBVARNISHAPI_1.0;



From tfheen at varnish-cache.org  Fri Nov  4 11:04:58 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Fri, 04 Nov 2011 12:04:58 +0100
Subject: [master] 98a29cb Implement support for custom output and profiles in
	varnishhist
Message-ID: <E1RMHZy-0002PL-B8@project.varnish-software.com>

commit 98a29cbc3e345f008a3c169e522992445a95365d
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Fri Nov 4 12:00:47 2011 +0100

    Implement support for custom output and profiles in varnishhist
    
    Drop varnishsizes, since it's pointless now (you can get the same
    functionality with varnishhist -p size)

diff --git a/bin/Makefile.am b/bin/Makefile.am
index 7bdddbb..9906c5e 100644
--- a/bin/Makefile.am
+++ b/bin/Makefile.am
@@ -3,5 +3,5 @@
 SUBDIRS = varnishadm varnishd varnishlog varnishncsa varnishreplay varnishtest
 
 if HAVE_CURSES
-SUBDIRS += varnishhist varnishstat varnishtop varnishsizes
+SUBDIRS += varnishhist varnishstat varnishtop
 endif
diff --git a/bin/varnishhist/varnishhist.c b/bin/varnishhist/varnishhist.c
index 72bc84b..692eb81 100644
--- a/bin/varnishhist/varnishhist.c
+++ b/bin/varnishhist/varnishhist.c
@@ -52,11 +52,12 @@
 #include "vcs.h"
 
 #define HIST_N 2000 /* how far back we remember */
-#define HIST_LOW -6 /* low end of log range */
-#define HIST_HIGH 3 /* high end of log range */
-#define HIST_RANGE (HIST_HIGH - HIST_LOW)
 #define HIST_RES 100 /* bucket resolution */
-#define HIST_BUCKETS (HIST_RANGE * HIST_RES)
+
+static int hist_low;
+static int hist_high;
+static int hist_range;
+static int hist_buckets;
 
 static pthread_mutex_t mtx = PTHREAD_MUTEX_INITIALIZER;
 
@@ -64,10 +65,13 @@ static int delay = 1;
 static unsigned rr_hist[HIST_N];
 static unsigned nhist;
 static unsigned next_hist;
-static unsigned bucket_miss[HIST_BUCKETS];
-static unsigned bucket_hit[HIST_BUCKETS];
+static unsigned *bucket_miss;
+static unsigned *bucket_hit;
 static unsigned char hh[FD_SETSIZE];
 static uint64_t bitmap[FD_SETSIZE];
+static double  values[FD_SETSIZE];
+static char *format;
+static int match_tag;
 
 static double log_ten;
 
@@ -95,11 +99,25 @@ static int scales[] = {
 	INT_MAX
 };
 
+struct profile {
+	const char *name;
+	enum VSL_tag_e tag;
+	int field;
+	int hist_low;
+	int hist_high;
+} profiles[] = {
+	{ .name = "responsetime", .tag = SLT_ReqEnd, .field = 5, .hist_low = -6, .hist_high = 3 },
+	{ .name = "size", .tag = SLT_Length, .field = 1, .hist_low = 1, .hist_high = 8 },
+	{ 0 }
+};
+
+static struct profile *active_profile;
+
 static void
 update(struct VSM_data *vd)
 {
-	int w = COLS / HIST_RANGE;
-	int n = w * HIST_RANGE;
+	int w = COLS / hist_range;
+	int n = w * hist_range;
 	unsigned bm[n], bh[n];
 	unsigned max;
 	int i, j, scale;
@@ -107,11 +125,11 @@ update(struct VSM_data *vd)
 	erase();
 
 	/* Draw horizontal axis */
-	w = COLS / HIST_RANGE;
-	n = w * HIST_RANGE;
+	w = COLS / hist_range;
+	n = w * hist_range;
 	for (i = 0; i < n; ++i)
 		(void)mvaddch(LINES - 2, i, '-');
-	for (i = 0, j = HIST_LOW; i < HIST_RANGE; ++i, ++j) {
+	for (i = 0, j = hist_low; i < hist_range; ++i, ++j) {
 		(void)mvaddch(LINES - 2, w * i, '+');
 		mvprintw(LINES - 1, w * i, "|1e%d", j);
 	}
@@ -121,8 +139,8 @@ update(struct VSM_data *vd)
 	/* count our flock */
 	for (i = 0; i < n; ++i)
 		bm[i] = bh[i] = 0;
-	for (i = 0, max = 1; i < HIST_BUCKETS; ++i) {
-		j = i * n / HIST_BUCKETS;
+	for (i = 0, max = 1; i < hist_buckets; ++i) {
+		j = i * n / hist_buckets;
 		bm[j] += bucket_miss[i];
 		bh[j] += bucket_hit[i];
 		if (bm[j] + bh[j] > max)
@@ -151,11 +169,8 @@ static int
 h_hist(void *priv, enum VSL_tag_e tag, unsigned fd, unsigned len,
     unsigned spec, const char *ptr, uint64_t bm)
 {
-	double b;
 	int i, j;
 	struct VSM_data *vd = priv;
-
-	(void)len;
 	(void)spec;
 
 	if (fd >= FD_SETSIZE)
@@ -168,6 +183,15 @@ h_hist(void *priv, enum VSL_tag_e tag, unsigned fd, unsigned len,
 		hh[fd] = 1;
 		return (0);
 	}
+	if (tag == match_tag) {
+		char buf[1024]; /* size? */
+		assert(len < sizeof(buf));
+		memcpy(buf, ptr, len);
+		buf[len] = '\0';
+		i = sscanf(buf, format, &values[fd]);
+		assert(i == 1);
+	}
+
 	if (tag != SLT_ReqEnd)
 		return (0);
 
@@ -177,24 +201,15 @@ h_hist(void *priv, enum VSL_tag_e tag, unsigned fd, unsigned len,
 		return (0);
 	}
 
-	/* determine processing time */
-#if 1
-	i = sscanf(ptr, "%*d %*f %*f %*f %lf", &b);
-#else
-	i = sscanf(ptr, "%*d %*f %*f %lf", &b);
-#endif
-	assert(i == 1);
-
 	/* select bucket */
-	i = HIST_RES * (log(b) / log_ten);
-	if (i < HIST_LOW * HIST_RES)
-		i = HIST_LOW * HIST_RES;
-	if (i >= HIST_HIGH * HIST_RES)
-		i = HIST_HIGH * HIST_RES - 1;
-	i -= HIST_LOW * HIST_RES;
+	i = HIST_RES * (log(values[fd]) / log_ten);
+	if (i < hist_low * HIST_RES)
+		i = hist_low * HIST_RES;
+	if (i >= hist_high * HIST_RES)
+		i = hist_high * HIST_RES - 1;
+	i -= hist_low * HIST_RES;
 	assert(i >= 0);
-	assert(i < HIST_BUCKETS);
-
+	assert(i < hist_buckets);
 	pthread_mutex_lock(&mtx);
 
 	/* phase out old data */
@@ -320,33 +335,94 @@ static void
 usage(void)
 {
 	fprintf(stderr, "usage: varnishhist "
-	    "%s [-n varnish_name] [-V] [-w delay]\n", VSL_USAGE);
+	    "%s [-p profile] [-f field_num] [-R max] [-r min] [-V] [-w delay]\n", VSL_USAGE);
 	exit(1);
 }
 
 int
 main(int argc, char **argv)
 {
-	int o;
+	int o, i;
 	struct VSM_data *vd;
+	const char *profile = "responsetime";
+	int fnum = -1;
+	hist_low = -1;
+	hist_high = -1;
+	match_tag = -1;
 
 	vd = VSM_New();
 	VSL_Setup(vd);
 
-	while ((o = getopt(argc, argv, VSL_ARGS "Vw:")) != -1) {
+	while ((o = getopt(argc, argv, VSL_ARGS "Vw:r:R:f:p:")) != -1) {
 		switch (o) {
 		case 'V':
 			VCS_Message("varnishhist");
 			exit(0);
+		case 'i':
+			match_tag = VSL_Name2Tag(optarg, -1);
+			if (match_tag < 0) {
+				fprintf(stderr, "No such tag %s\n", optarg);
+				exit(1);
+			}
+			break;
 		case 'w':
 			delay = atoi(optarg);
 			break;
+		case 'f':
+			fnum = atoi(optarg);
+			break;
+		case 'R':
+			hist_high = atoi(optarg);
+			break;
+		case 'r':
+			hist_low = atoi(optarg);
+			break;
+		case 'p':
+			profile = optarg;
+			break;
 		default:
 			if (VSL_Arg(vd, o, optarg) > 0)
 				break;
 			usage();
 		}
 	}
+	if (profile) {
+		for (active_profile = profiles; active_profile->name; active_profile++) {
+			if (strcmp(active_profile->name, profile) == 0) {
+				break;
+			}
+		}
+	}
+	if (! active_profile->name) {
+		fprintf(stderr, "No such profile %s\n", profile);
+		exit(1);
+	}
+	if (match_tag < 0) {
+		match_tag = active_profile->tag;
+	}
+
+	if (fnum < 0) {
+		fnum = active_profile->field;
+	}
+
+	if (hist_low < 0) {
+		hist_low = active_profile->hist_low;
+	}
+
+	if (hist_high < 0) {
+		hist_high = active_profile->hist_high;
+	}
+
+	hist_range = hist_high - hist_low;
+	hist_buckets = hist_range * HIST_RES;
+	bucket_hit = calloc(sizeof bucket_hit, hist_buckets);
+	bucket_miss = calloc(sizeof bucket_miss, hist_buckets);
+
+	format = malloc(4 * fnum);
+	for (i = 0; i < fnum-1; i++) {
+		strcpy(format + 4*i, "%*s ");
+	}
+	strcpy(format + 4*(fnum-1), "%lf");
 
 	if (VSL_Open(vd, 1))
 		exit(1);
diff --git a/bin/varnishsizes/Makefile.am b/bin/varnishsizes/Makefile.am
deleted file mode 100644
index 8029741..0000000
--- a/bin/varnishsizes/Makefile.am
+++ /dev/null
@@ -1,27 +0,0 @@
-#
-
-INCLUDES = -I$(top_srcdir)/include
-
-bin_PROGRAMS = varnishsizes
-
-dist_man_MANS = varnishsizes.1
-
-varnishsizes_SOURCES = varnishsizes.c \
-	$(top_builddir)/lib/libvarnish/vas.c \
-	$(top_builddir)/lib/libvarnish/version.c
-
-varnishsizes_LDADD = \
-	$(top_builddir)/lib/libvarnishcompat/libvarnishcompat.la \
-	$(top_builddir)/lib/libvarnishapi/libvarnishapi.la \
-	-lm \
-	${CURSES_LIBS} ${PTHREAD_LIBS}
-
-varnishsizes.1: $(top_srcdir)/doc/sphinx/reference/varnishsizes.rst
-if HAVE_RST2MAN
-	${RST2MAN} $? $@
-else
-	@echo "========================================"
-	@echo "You need rst2man installed to make dist"
-	@echo "========================================"
-	@false
-endif
diff --git a/bin/varnishsizes/varnishsizes.c b/bin/varnishsizes/varnishsizes.c
deleted file mode 100644
index ce68a9d..0000000
--- a/bin/varnishsizes/varnishsizes.c
+++ /dev/null
@@ -1,360 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2010 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- * Author: Dag-Erling Sm?rgrav <des at des.no>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * Log tailer for Varnish
- */
-
-#include "config.h"
-
-#include <sys/types.h>
-
-#include <curses.h>
-#include <errno.h>
-#include <limits.h>
-#include <math.h>
-#include <pthread.h>
-#include <regex.h>
-#include <signal.h>
-#include <stdint.h>
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-#include <unistd.h>
-
-#include "vapi/vsl.h"
-#include "vapi/vsm.h"
-#include "vas.h"
-#include "vcs.h"
-
-#define HIST_N 2000 /* how far back we remember */
-#define HIST_LOW 1 /* low end of log range */
-#define HIST_HIGH 8 /* high end of log range */
-#define HIST_RANGE (HIST_HIGH - HIST_LOW)
-#define HIST_RES 100 /* bucket resolution */
-#define HIST_BUCKETS (HIST_RANGE * HIST_RES)
-
-static pthread_mutex_t mtx = PTHREAD_MUTEX_INITIALIZER;
-
-static int delay = 1;
-static unsigned rr_hist[HIST_N];
-static unsigned nhist;
-static unsigned next_hist;
-static unsigned bucket_miss[HIST_BUCKETS];
-static unsigned bucket_hit[HIST_BUCKETS];
-static unsigned char hh[FD_SETSIZE];
-static uint64_t bitmap[FD_SETSIZE];
-
-static double log_ten;
-
-static int scales[] = {
-	1,
-	2,
-	3,
-	4,
-	5,
-	10,
-	15,
-	20,
-	25,
-	50,
-	100,
-	250,
-	500,
-	1000,
-	2500,
-	5000,
-	10000,
-	25000,
-	50000,
-	100000,
-	INT_MAX
-};
-
-static void
-update(struct VSM_data *vd)
-{
-	int w = COLS / HIST_RANGE;
-	int n = w * HIST_RANGE;
-	unsigned bm[n], bh[n];
-	unsigned max;
-	int i, j, scale;
-
-	erase();
-
-	/* Draw horizontal axis */
-	w = COLS / HIST_RANGE;
-	n = w * HIST_RANGE;
-	for (i = 0; i < n; ++i)
-		(void)mvaddch(LINES - 2, i, '-');
-	for (i = 0, j = HIST_LOW; i < HIST_RANGE; ++i, ++j) {
-		(void)mvaddch(LINES - 2, w * i, '+');
-		mvprintw(LINES - 1, w * i, "|1e%d", j);
-	}
-
-	mvprintw(0, 0, "%*s", COLS - 1, VSM_Name(vd));
-
-	/* count our flock */
-	for (i = 0; i < n; ++i)
-		bm[i] = bh[i] = 0;
-	for (i = 0, max = 1; i < HIST_BUCKETS; ++i) {
-		j = i * n / HIST_BUCKETS;
-		bm[j] += bucket_miss[i];
-		bh[j] += bucket_hit[i];
-		if (bm[j] + bh[j] > max)
-			max = bm[j] + bh[j];
-	}
-
-	/* scale */
-	for (i = 0; max / scales[i] > LINES - 3; ++i)
-		/* nothing */ ;
-	scale = scales[i];
-
-	mvprintw(0, 0, "1:%d, n = %d", scale, nhist);
-
-	/* show them */
-	for (i = 0; i < n; ++i) {
-		for (j = 0; j < bm[i] / scale; ++j)
-			(void)mvaddch(LINES - 3 - j, i, '#');
-		for (; j < (bm[i] + bh[i]) / scale; ++j)
-			(void)mvaddch(LINES - 3 - j, i, '|');
-	}
-
-	refresh();
-}
-
-static int
-h_hist(void *priv, enum VSL_tag_e tag, unsigned fd, unsigned len,
-    unsigned spec, const char *ptr, uint64_t bm)
-{
-	double b;
-	int i, j, tmp;
-	struct VSM_data *vd = priv;
-
-	(void)len;
-	(void)spec;
-
-	if (fd >= FD_SETSIZE)
-		/* oops */
-		return (0);
-
-	bitmap[fd] |= bm;
-
-	if (tag == SLT_Hit) {
-		hh[fd] = 1;
-		return (0);
-	}
-	if (tag != SLT_Length)
-		return (0);
-
-	if (!VSL_Matched(vd, bitmap[fd])) {
-		bitmap[fd] = 0;
-		hh[fd] = 0;
-		return (0);
-	}
-
-	/* determine processing time */
-	i = sscanf(ptr, "%d", &tmp);
-	assert(i == 1);
-
-	/* Typically 304s and tend to throw the scaling off */
-	if (tmp == 0)
-		return 0;
-
-	b = tmp;
-	/* select bucket */
-	i = HIST_RES * (log(b) / log_ten);
-	if (i < HIST_LOW * HIST_RES)
-		i = HIST_LOW * HIST_RES;
-	if (i >= HIST_HIGH * HIST_RES)
-		i = HIST_HIGH * HIST_RES - 1;
-	i -= HIST_LOW * HIST_RES;
-	assert(i >= 0);
-	assert(i < HIST_BUCKETS);
-
-	pthread_mutex_lock(&mtx);
-
-	/* phase out old data */
-	if (nhist == HIST_N) {
-		j = rr_hist[next_hist];
-		if (j < 0)  {
-			assert(bucket_miss[-j] > 0);
-			bucket_miss[-j]--;
-		} else {
-			assert(bucket_hit[j] > 0);
-			bucket_hit[j]--;
-		}
-	} else {
-		++nhist;
-	}
-
-	/* phase in new data */
-	if (hh[fd] || i == 0) {
-		bucket_hit[i]++;
-		rr_hist[next_hist] = i;
-	} else {
-		bucket_miss[i]++;
-		rr_hist[next_hist] = -i;
-	}
-	if (++next_hist == HIST_N) {
-		next_hist = 0;
-	}
-	hh[fd] = 0;
-	bitmap[fd] = 0;
-
-	pthread_mutex_unlock(&mtx);
-
-	return (0);
-}
-
-static void *
-accumulate_thread(void *arg)
-{
-	struct VSM_data *vd = arg;
-	int i;
-
-	for (;;) {
-		i = VSL_Dispatch(vd, h_hist, vd);
-		if (i < 0)
-			break;
-		if (i == 0)
-			usleep(50000);
-	}
-	return (arg);
-}
-
-static void
-do_curses(struct VSM_data *vd)
-{
-	pthread_t thr;
-	int ch;
-
-	if (pthread_create(&thr, NULL, accumulate_thread, vd) != 0) {
-		fprintf(stderr, "pthread_create(): %s\n", strerror(errno));
-		exit(1);
-	}
-
-	initscr();
-	raw();
-	noecho();
-	nonl();
-	intrflush(stdscr, FALSE);
-	curs_set(0);
-	erase();
-	for (;;) {
-		pthread_mutex_lock(&mtx);
-		update(vd);
-		pthread_mutex_unlock(&mtx);
-
-		timeout(delay * 1000);
-		switch ((ch = getch())) {
-		case ERR:
-			break;
-#ifdef KEY_RESIZE
-		case KEY_RESIZE:
-			erase();
-			break;
-#endif
-		case '\014': /* Ctrl-L */
-		case '\024': /* Ctrl-T */
-			redrawwin(stdscr);
-			refresh();
-			break;
-		case '\003': /* Ctrl-C */
-			raise(SIGINT);
-			break;
-		case '\032': /* Ctrl-Z */
-			endwin();
-			raise(SIGTSTP);
-			break;
-		case '\021': /* Ctrl-Q */
-		case 'Q':
-		case 'q':
-			endwin();
-			return;
-		case '0':
-		case '1':
-		case '2':
-		case '3':
-		case '4':
-		case '5':
-		case '6':
-		case '7':
-		case '8':
-		case '9':
-			delay = 1 << (ch - '0');
-			break;
-		default:
-			beep();
-			break;
-		}
-	}
-}
-
-/*--------------------------------------------------------------------*/
-
-static void
-usage(void)
-{
-	fprintf(stderr, "usage: varnishsizes "
-	    "%s [-n varnish_name] [-V] [-w delay]\n", VSL_USAGE);
-	exit(1);
-}
-
-int
-main(int argc, char **argv)
-{
-	int o;
-	struct VSM_data *vd;
-
-	vd = VSM_New();
-	VSL_Setup(vd);
-
-	while ((o = getopt(argc, argv, VSL_ARGS "Vw:")) != -1) {
-		switch (o) {
-		case 'V':
-			VCS_Message("varnishsizes");
-			exit(0);
-		case 'w':
-			delay = atoi(optarg);
-			break;
-		default:
-			if (VSL_Arg(vd, o, optarg) > 0)
-				break;
-			usage();
-		}
-	}
-
-	if (VSL_Open(vd, 1))
-		exit(1);
-
-	log_ten = log(10.0);
-
-	do_curses(vd);
-	exit(0);
-}



From phk at varnish-cache.org  Mon Nov  7 11:11:48 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 07 Nov 2011 12:11:48 +0100
Subject: [master] fa2af30 FlexeLint spotted a memory leak in Tollefs
	"admin-health" code, so I started fixing that,
	and then I fixed another couple of nits and polished the naming a bit
	and ...
Message-ID: <E1RNN7E-0000EG-E0@project.varnish-software.com>

commit fa2af30fc399792a5a92676db75241964d9ac154
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Nov 7 11:06:42 2011 +0000

    FlexeLint spotted a memory leak in Tollefs "admin-health" code, so I
    started fixing that, and then I fixed another couple of nits and polished
    the naming a bit and ...
    
    Well, you know how it is.
    
    Changed the backend matching function to take a call-back function
    to eliminate the need for the memory which got leaked.
    
    This meant that backend.list could also use it, if only the parsing
    code accepted a blank input.
    
    So I rewrote the parsing code to also allow empty names, so you can:
    	backend.list (:80)
    	backend.list (192.168)
    etc.
    
    And then I added a summary report of the probing, if it is happening,
    and explicitly say that no probing is happening if it isn't.

diff --git a/bin/varnishd/cache_backend.c b/bin/varnishd/cache_backend.c
index cad637b..c94d819 100644
--- a/bin/varnishd/cache_backend.c
+++ b/bin/varnishd/cache_backend.c
@@ -256,10 +256,10 @@ vbe_Healthy(const struct vdi_simple *vs, const struct sess *sp)
 	backend = vs->backend;
 	CHECK_OBJ_NOTNULL(backend, BACKEND_MAGIC);
 
-	if (backend->admin_health == from_probe && !backend->healthy)
+	if (backend->admin_health == ah_probe && !backend->healthy)
 		return (0);
 
-	if (backend->admin_health == sick)
+	if (backend->admin_health == ah_sick)
 		return (0);
 
 	/* VRT/VCC sets threshold to UINT_MAX to mark that it's not
@@ -270,7 +270,7 @@ vbe_Healthy(const struct vdi_simple *vs, const struct sess *sp)
 	else
 		threshold = vs->vrt->saintmode_threshold;
 
-	if (backend->admin_health == healthy)
+	if (backend->admin_health == ah_healthy)
 		threshold = UINT_MAX;
 
 	/* Saintmode is disabled */
diff --git a/bin/varnishd/cache_backend.h b/bin/varnishd/cache_backend.h
index 880c429..91bff1e 100644
--- a/bin/varnishd/cache_backend.h
+++ b/bin/varnishd/cache_backend.h
@@ -106,10 +106,11 @@ struct trouble {
  * An instance of a backend from a VCL program.
  */
 
-enum health_status {
-	healthy,
-	sick,
-	from_probe
+enum admin_health {
+	ah_invalid = 0,
+	ah_healthy,
+	ah_sick,
+	ah_probe
 };
 
 struct backend {
@@ -136,7 +137,7 @@ struct backend {
 
 	struct vbp_target	*probe;
 	unsigned		healthy;
-	enum health_status	admin_health;
+	enum admin_health	admin_health;
 	VTAILQ_HEAD(, trouble)	troublelist;
 
 	struct VSC_C_vbe	*vsc;
@@ -178,6 +179,7 @@ void VBE_DropRefLocked(struct backend *b);
 void VBP_Insert(struct backend *b, struct vrt_backend_probe const *p, const char *hosthdr);
 void VBP_Remove(struct backend *b, struct vrt_backend_probe const *p);
 void VBP_Use(const struct backend *b, const struct vrt_backend_probe const *p);
+void VBP_Summary(struct cli *cli, const struct vbp_target *vt);
 
 /* Init functions for directors */
 typedef void dir_init_f(struct cli *, struct director **, int , const void*);
diff --git a/bin/varnishd/cache_backend_cfg.c b/bin/varnishd/cache_backend_cfg.c
index 8168be0..e87584a 100644
--- a/bin/varnishd/cache_backend_cfg.c
+++ b/bin/varnishd/cache_backend_cfg.c
@@ -32,6 +32,7 @@
 
 #include "config.h"
 
+#include <ctype.h>
 #include <stdio.h>
 #include <stdlib.h>
 
@@ -44,6 +45,7 @@
 
 struct lock VBE_mtx;
 
+
 /*
  * The list of backends is not locked, it is only ever accessed from
  * the CLI thread, so there is no need.
@@ -79,6 +81,11 @@ VBE_Poll(void)
 
 	ASSERT_CLI();
 	VTAILQ_FOREACH_SAFE(b, &backends, list, b2) {
+		assert(
+			b->admin_health == ah_healthy ||
+			b->admin_health == ah_sick ||
+			b->admin_health == ah_probe
+		);
 		if (b->refcount == 0 && b->probe == NULL)
 			VBE_Nuke(b);
 	}
@@ -230,7 +237,7 @@ VBE_AddBackend(struct cli *cli, const struct vrt_backend *vb)
 	assert(b->ipv4 != NULL || b->ipv6 != NULL);
 
 	b->healthy = 1;
-	b->admin_health = from_probe;
+	b->admin_health = ah_probe;
 
 	VTAILQ_INSERT_TAIL(&backends, b, list);
 	VSC_C_main->n_backend++;
@@ -274,157 +281,222 @@ VRT_fini_dir(struct cli *cli, struct director *b)
 	b->priv = NULL;
 }
 
-/*--------------------------------------------------------------------*/
+/*---------------------------------------------------------------------
+ * String to admin_health
+ */
+
+static enum admin_health
+vbe_str2adminhealth(const char *wstate)
+{
+
+	if (strcasecmp(wstate, "healthy") == 0)
+		return(ah_healthy);
+	if (strcasecmp(wstate, "sick") == 0)
+		return(ah_sick);
+	if (strcmp(wstate, "auto") == 0)
+		return(ah_probe);
+	return (ah_invalid);
+}
+
+/*---------------------------------------------------------------------
+ * A general function for finding backends and doing things with them.
+ *
+ * Return -1 on match-argument parse errors.
+ *
+ * If the call-back function returns non-zero, the search is terminated
+ * and we relay that return value.
+ *
+ * Otherwise we return the number of matches.
+ */
+
+typedef int bf_func(struct cli *cli, struct backend *b, void *priv);
 
 static int
-backend_find(const char *matcher, struct backend **r, int n)
+backend_find(struct cli *cli, const char *matcher, bf_func *func, void *priv)
 {
 	struct backend *b;
-	char *vcl_name;
-	char *s;
-	char *match_ip = NULL;
-	char *match_port = NULL;
+	const char *s;
+	const char *name_b;
+	ssize_t name_l = 0;
+	const char *ip_b = NULL;
+	ssize_t ip_l = 0;
+	const char *port_b = NULL;
+	ssize_t port_l = 0;
 	int found = 0;
+	int i;
 
-	s = strchr(matcher, '(');
+	name_b = matcher;
+	if (matcher != NULL) {
+		s = strchr(matcher,'(');
 
-	if (s == NULL) {
-		/* Simple match, max one hit */
-		VTAILQ_FOREACH(b, &backends, list) {
-			CHECK_OBJ_NOTNULL(b, BACKEND_MAGIC);
-			if (strcmp(b->vcl_name, matcher) == 0) {
-				if (r && found < n)
-					r[found] = b;
-				found++;
-			}
-		}
-		return found;
-	}
+		if (s != NULL)
+			name_l = s - name_b;
+		else
+			name_l = strlen(name_b);
 
-	vcl_name = strndup(matcher, s - matcher);
-	AN(vcl_name);
-	s++;
-	while (*s != ')') {
-		if (*s == ':') {
-			/* Port */
+		if (s != NULL) {
 			s++;
-			match_port = s;
-			if (!(s = strchr(match_port, ','))) {
-				s = strchr(match_port, ')');
+			while (isspace(*s))
+				s++;
+			ip_b = s;
+			while (*s != '\0' &&
+			    *s != ')' &&
+			    *s != ':' &&
+			    !isspace(*s))
+				s++;
+			ip_l = s - ip_b;
+			while (isspace(*s))
+				s++;
+			if (*s == ':') {
+				s++;
+				while (isspace(*s))
+					s++;
+				port_b = s;
+				while (*s != '\0' && *s != ')' && !isspace(*s))
+					s++;
+				port_l = s - port_b;
 			}
-			XXXAN(s);
-			match_port = strndup(match_port, s - match_port);
-			AN(match_port);
-			if (*s == ',')
+			while (isspace(*s))
 				s++;
-		} else {
-			/* IP */
-			match_ip = s;
-			if (!(s = strchr(match_ip, ','))) {
-				s = strchr(match_ip, ')');
+			if (*s != ')') {
+				VCLI_Out(cli,
+				    "Match string syntax error:"
+				    " ')' not found.");
+				VCLI_SetResult(cli, CLIS_CANT);
+				return (-1);
 			}
-			XXXAN(s);
-			match_ip = strndup(match_ip, s - match_ip);
-			AN(match_ip);
-			if (*s == ',')
+			s++;
+			while (isspace(*s))
 				s++;
+			if (*s != '\0') {
+				VCLI_Out(cli,
+				    "Match string syntax error:"
+				    " junk after ')'");
+				VCLI_SetResult(cli, CLIS_CANT);
+				return (-1);
+			}
 		}
 	}
 	VTAILQ_FOREACH(b, &backends, list) {
 		CHECK_OBJ_NOTNULL(b, BACKEND_MAGIC);
-		if (match_port && strcmp(b->port, match_port) != 0)
+		if (port_b != NULL && strncmp(b->port, port_b, port_l) != 0)
 			continue;
-		if (match_ip &&
-		    (strcmp(b->ipv4_addr, match_ip) != 0) &&
-		    (strcmp(b->ipv6_addr, match_ip) != 0))
+		if (name_b != NULL && strncmp(b->vcl_name, name_b, name_l) != 0)
 			continue;
-		if (strcmp(b->vcl_name, vcl_name) == 0) {
-			if (r && found < n)
-				r[found] = b;
-			found++;
-		}
+		if (ip_b != NULL &&
+		    (b->ipv4_addr == NULL ||
+		      strncmp(b->ipv4_addr, ip_b, ip_l)) &&
+		    (b->ipv6_addr == NULL ||
+		      strncmp(b->ipv6_addr, ip_b, ip_l)))
+			continue;
+		found++;
+		i = func(cli, b, priv);
+		if (i)
+			return(i);
+	}
+	return (found);
+}
+
+/*---------------------------------------------------------------------*/
+
+static int __match_proto__()
+do_list(struct cli *cli, struct backend *b, void *priv)
+{
+	int *hdr;
+
+	AN(priv);
+	hdr = priv;
+	if (!*hdr) {
+		VCLI_Out(cli, "%-30s %-6s %-10s %s",
+		    "Backend name", "Refs", "Admin", "Probe");
+		*hdr = 1;
+	}
+	CHECK_OBJ_NOTNULL(b, BACKEND_MAGIC);
+
+	VCLI_Out(cli, "\n%-30s %-6d", b->display_name, b->refcount);
+
+	if (b->admin_health == ah_probe)
+		VCLI_Out(cli, " %-10s", "probe");
+	else if (b->admin_health == ah_sick)
+		VCLI_Out(cli, " %-10s", "sick");
+	else if (b->admin_health == ah_healthy)
+		VCLI_Out(cli, " %-10s", "healthy");
+	else
+		VCLI_Out(cli, " %-10s", "invalid");
+
+	if (b->probe == NULL)
+		VCLI_Out(cli, " %s", "Healthy (no probe)");
+	else {
+		if (b->healthy)
+			VCLI_Out(cli, " %s", "Healthy ");
+		else
+			VCLI_Out(cli, " %s", "Sick ");
+		VBP_Summary(cli, b->probe);
 	}
-	return found;
+
+	return (0);
 }
 
 static void
 cli_backend_list(struct cli *cli, const char * const *av, void *priv)
 {
-	struct backend *b;
-	const char *ah;
+	int hdr = 0;
 
 	(void)av;
 	(void)priv;
 	ASSERT_CLI();
-	VCLI_Out(cli, "%-30s %10s %15s %15s", "Backend name",
-		 "Conns", "Probed healthy", "Admin health");
-	VTAILQ_FOREACH(b, &backends, list) {
-		CHECK_OBJ_NOTNULL(b, BACKEND_MAGIC);
-		if (b->admin_health == from_probe) {
-			ah = "Auto";
-		} else if (b->admin_health == sick) {
-			ah = "Sick";
-		} else {
-			ah = "Healthy";
-		}
-		VCLI_Out(cli, "\n%-30s %10d %15s %15s",
-			 b->display_name,
-			 b->refcount,
-			 (b ->healthy ? "Yes" : "No"),
-			 ah);
-	}
+	(void)backend_find(cli, av[2], do_list, &hdr);
+}
+
+/*---------------------------------------------------------------------*/
+
+static int __match_proto__()
+do_set_health(struct cli *cli, struct backend *b, void *priv)
+{
+	enum admin_health state;
+
+	(void)cli;
+	state = *(enum admin_health*)priv;
+	CHECK_OBJ_NOTNULL(b, BACKEND_MAGIC);
+	b->admin_health = state;
+	return (0);
 }
 
 static void
 cli_backend_set_health(struct cli *cli, const char * const *av, void *priv)
 {
-	struct backend **b;
-	enum health_status state;
+	enum admin_health state;
 	int n;
-	const char *wstate;
 
 	(void)av;
 	(void)priv;
 	ASSERT_CLI();
-	wstate = av[3];
-	if (strcmp(wstate, "healthy") == 0) {
-		state = healthy;
-	} else if (strcmp(wstate, "sick") == 0) {
-		state = sick;
-	} else if (strcmp(wstate, "auto") == 0) {
-		state = from_probe;
-	} else {
-		VCLI_Out(cli, "Invalid state %s", wstate);
-		VCLI_SetResult(cli, CLIS_CANT);
+	AN(av[2]);
+	AN(av[3]);
+	state = vbe_str2adminhealth(av[3]);
+	if (state == ah_invalid) {
+		VCLI_Out(cli, "Invalid state %s", av[3]);
+		VCLI_SetResult(cli, CLIS_PARAM);
 		return;
 	}
-	n = backend_find(av[2], NULL, 0);
+	n = backend_find(cli, av[2], do_set_health, &state);
 	if (n == 0) {
-		VCLI_Out(cli, "No matching backends");
-		VCLI_SetResult(cli, CLIS_CANT);
-		return;
-	}
-
-	b = calloc(n, sizeof(struct backend *));
-	AN(b);
-	n = backend_find(av[2], b, n);
-
-	VCLI_Out(cli, "Set state to %s for the following backends:", wstate);
-	for (int i = 0; i < n; i++) {
-		b[i]->admin_health = state;
-		VCLI_Out(cli, "\n\t%s", b[i]->display_name);
+		VCLI_Out(cli, "No Backends matches");
+		VCLI_SetResult(cli, CLIS_PARAM);
 	}
 }
 
+/*---------------------------------------------------------------------*/
+
 static struct cli_proto backend_cmds[] = {
 	{ "backend.list", "backend.list",
-	    "\tList all backends\n", 0, 0, "d", cli_backend_list },
+	    "\tList all backends\n", 0, 1, "d", cli_backend_list },
 	{ "backend.set_health", "backend.set_health matcher state",
 	    "\tShow a backend\n", 2, 2, "d", cli_backend_set_health },
 	{ NULL }
 };
 
-/*--------------------------------------------------------------------*/
+/*---------------------------------------------------------------------*/
 
 void
 VBE_Init(void)
diff --git a/bin/varnishd/cache_backend_poll.c b/bin/varnishd/cache_backend_poll.c
index d79a806..eb6cc45 100644
--- a/bin/varnishd/cache_backend_poll.c
+++ b/bin/varnishd/cache_backend_poll.c
@@ -361,6 +361,14 @@ vbp_wrk_poll_backend(void *priv)
  * Cli functions
  */
 
+void
+VBP_Summary(struct cli *cli, const struct vbp_target *vt)
+{
+
+	CHECK_OBJ_NOTNULL(vt, VBP_TARGET_MAGIC);
+	VCLI_Out(cli, "%d/%d", vt->good, vt->probe.window);
+}
+
 static void
 vbp_bitmap(struct cli *cli, char c, uint64_t map, const char *lbl)
 {
diff --git a/bin/varnishtest/tests/c00048.vtc b/bin/varnishtest/tests/c00048.vtc
index ed8671f..a4d0847 100644
--- a/bin/varnishtest/tests/c00048.vtc
+++ b/bin/varnishtest/tests/c00048.vtc
@@ -49,7 +49,7 @@ client c1 {
 	expect resp.status == 200
 } -run
 
-varnish v1 -clierr 300 "backend.set_health s1 foo"
-varnish v1 -clierr 300 "backend.set_health s2 foo"
-varnish v1 -clierr 300 "backend.set_health s2 auto"
+varnish v1 -clierr 106 "backend.set_health s1 foo"
+varnish v1 -clierr 106 "backend.set_health s2 foo"
+varnish v1 -clierr 106 "backend.set_health s2 auto"
 



From phk at varnish-cache.org  Mon Nov  7 11:53:47 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 07 Nov 2011 12:53:47 +0100
Subject: [master] b32cfd0 Add some backend.list commands to exercise that code
	also
Message-ID: <E1RNNlr-0004ol-5z@project.varnish-software.com>

commit b32cfd034291dd46157eb5f65146a00144485336
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Nov 7 11:44:06 2011 +0000

    Add some backend.list commands to exercise that code also

diff --git a/bin/varnishtest/tests/c00048.vtc b/bin/varnishtest/tests/c00048.vtc
index a4d0847..bf1c3c9 100644
--- a/bin/varnishtest/tests/c00048.vtc
+++ b/bin/varnishtest/tests/c00048.vtc
@@ -25,7 +25,9 @@ varnish v1 -vcl {
 
 delay 1
 
+varnish v1 -cliok "backend.list"
 varnish v1 -cliok "backend.set_health s1 auto"
+varnish v1 -cliok "backend.list s"
 
 client c1 {
 	txreq
@@ -33,7 +35,9 @@ client c1 {
 	expect resp.status == 200
 } -run
 
+varnish v1 -cliok "backend.list (:)"
 varnish v1 -cliok "backend.set_health s1 sick"
+varnish v1 -cliok "backend.list (1:)"
 
 client c1 {
 	txreq
@@ -41,7 +45,9 @@ client c1 {
 	expect resp.status == 503
 } -run
 
+varnish v1 -cliok "backend.list"
 varnish v1 -cliok "backend.set_health s1 healthy"
+varnish v1 -cliok "backend.list"
 
 client c1 {
 	txreq
@@ -52,4 +58,5 @@ client c1 {
 varnish v1 -clierr 106 "backend.set_health s1 foo"
 varnish v1 -clierr 106 "backend.set_health s2 foo"
 varnish v1 -clierr 106 "backend.set_health s2 auto"
+varnish v1 -cliok "backend.list (foo)"
 



From phk at varnish-cache.org  Mon Nov  7 12:27:36 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 07 Nov 2011 13:27:36 +0100
Subject: [master] eab857a Test more of the backend matching code.
Message-ID: <E1RNOIa-00049c-BL@project.varnish-software.com>

commit eab857aca974ea2546171d5a12b5c21f64fa9d2b
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Nov 7 12:27:24 2011 +0000

    Test more of the backend matching code.

diff --git a/bin/varnishtest/tests/c00048.vtc b/bin/varnishtest/tests/c00048.vtc
index bf1c3c9..4ddb888 100644
--- a/bin/varnishtest/tests/c00048.vtc
+++ b/bin/varnishtest/tests/c00048.vtc
@@ -59,4 +59,6 @@ varnish v1 -clierr 106 "backend.set_health s1 foo"
 varnish v1 -clierr 106 "backend.set_health s2 foo"
 varnish v1 -clierr 106 "backend.set_health s2 auto"
 varnish v1 -cliok "backend.list (foo)"
+varnish v1 -clierr 300 "backend.list ("
+varnish v1 -clierr 300 {backend.list " ( : ) -"}
 



From tfheen at varnish-cache.org  Mon Nov  7 12:47:20 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Mon, 07 Nov 2011 13:47:20 +0100
Subject: [master] 513c4a2 Remove more traces of varnishsizes
Message-ID: <E1RNObg-000139-79@project.varnish-software.com>

commit 513c4a28655e9f38035bf0a1bb82f1efc3fd7786
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Mon Nov 7 13:34:51 2011 +0100

    Remove more traces of varnishsizes

diff --git a/.gitignore b/.gitignore
index eb2db8c..14c5393 100644
--- a/.gitignore
+++ b/.gitignore
@@ -70,8 +70,6 @@ TAGS
 /bin/varnishncsa/varnishncsa.1
 /bin/varnishreplay/varnishreplay
 /bin/varnishreplay/varnishreplay.1
-/bin/varnishsizes/varnishsizes
-/bin/varnishsizes/varnishsizes.1
 /bin/varnishstat/varnishstat
 /bin/varnishstat/varnishstat.1
 /bin/varnishtest/varnishtest
diff --git a/configure.ac b/configure.ac
index ebea42b..bbe44b5 100644
--- a/configure.ac
+++ b/configure.ac
@@ -569,7 +569,6 @@ AC_CONFIG_FILES([
     bin/varnishncsa/Makefile
     bin/varnishreplay/Makefile
     bin/varnishstat/Makefile
-    bin/varnishsizes/Makefile
     bin/varnishtest/Makefile
     bin/varnishtop/Makefile
     doc/Makefile



From tfheen at varnish-cache.org  Mon Nov  7 12:47:20 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Mon, 07 Nov 2011 13:47:20 +0100
Subject: [master] ac387cf varnishstat: Add json output and continous mode
Message-ID: <E1RNObg-00013F-De@project.varnish-software.com>

commit ac387cf066aead9e0c0aae63dc0802417d1ca306
Author: Lasse Karstensen <lasse at varnish-software.com>
Date:   Mon Nov 7 13:14:35 2011 +0100

    varnishstat: Add json output and continous mode

diff --git a/bin/varnishstat/varnishstat.c b/bin/varnishstat/varnishstat.c
index 3c6c4b5..c20b87f 100644
--- a/bin/varnishstat/varnishstat.c
+++ b/bin/varnishstat/varnishstat.c
@@ -82,6 +82,69 @@ do_xml(struct VSM_data *vd)
 	printf("</varnishstat>\n");
 }
 
+
+/*--------------------------------------------------------------------*/
+
+static int
+do_json_cb(void *priv, const struct VSC_point * const pt)
+{
+	uint64_t val;
+	int *jp;
+	char jsonkey[255];
+	char jsontmp[255];
+
+	jp = priv;
+
+	assert(!strcmp(pt->fmt, "uint64_t"));
+	val = *(const volatile uint64_t*)pt->ptr;
+
+	if (*jp) *jp = 0; else printf(",\n");
+	jsonkey[0] = '\0';
+
+	/* build the JSON key name.  */
+	if (strcmp(pt->ident, "")  && strcmp(pt->class, ""))  sprintf(jsonkey, "%s.%s", pt->class, pt->ident);
+	if (strcmp(pt->ident, "")  && !strcmp(pt->class, "")) sprintf(jsonkey, "%s", pt->ident);
+	if (!strcmp(pt->ident, "") && strcmp(pt->class, ""))  sprintf(jsonkey, "%s", pt->class);
+
+	strcpy(jsontmp, jsonkey);
+	if (strcmp(jsonkey, "")) sprintf(jsonkey, "%s.%s", jsontmp, pt->name);
+	else strcpy(jsonkey, pt->name);
+
+	printf("\t\"%s\": {", jsonkey);
+
+	if (strcmp(pt->class, "")) printf("\"type\": \"%s\", ",  pt->class);
+	if (strcmp(pt->ident, "")) printf("\"ident\": \"%s\", ", pt->ident);
+
+	printf("\"value\": %ju, ", val);
+
+	printf("\"flag\": \"%c\", ", pt->flag);
+	printf("\"description\": \"%s\"", pt->desc);
+	printf("}");
+
+	if (*jp) printf("\n");
+	return (0);
+}
+
+static void
+do_json(struct VSM_data *vd)
+{
+	char time_stamp[20];
+	time_t now;
+	int jp;
+
+	jp = 1;
+
+	printf("{\n");
+	now = time(NULL);
+
+	(void)strftime(time_stamp, 20, "%Y-%m-%dT%H:%M:%S", localtime(&now));
+	printf("\t\"timestamp\": \"%s\",\n", time_stamp);
+	(void)VSC_Iter(vd, do_json_cb, &jp);
+	printf("\n}\n");
+	fflush(stdout);
+}
+
+
 /*--------------------------------------------------------------------*/
 
 struct once_priv {
@@ -167,7 +230,7 @@ usage(void)
 	    "[-1lV] [-f field_list] "
 	    VSC_n_USAGE " "
 	    "[-w delay]\n");
-	fprintf(stderr, FMT, "-1", "Print the statistics once and exit");
+	fprintf(stderr, FMT, "-1", "Print the statistics to stdout.");
 	fprintf(stderr, FMT, "-f field_list",
 	    "Comma separated list of fields to display. ");
 	fprintf(stderr, FMT, "",
@@ -178,9 +241,11 @@ usage(void)
 	    "The varnishd instance to get logs from");
 	fprintf(stderr, FMT, "-V", "Display the version number and exit");
 	fprintf(stderr, FMT, "-w delay",
-	    "Wait delay seconds between updates.  The default is 1.");
+	    "Wait delay seconds between updates.  Default is 1 second. Can also be be used with -1, -x or -j for repeated output.");
 	fprintf(stderr, FMT, "-x",
-	    "Print statistics once as XML and exit.");
+	    "Print statistics to stdout as XML.");
+	fprintf(stderr, FMT, "-j",
+	    "Print statistics to stdout as JSON.");
 #undef FMT
 	exit(1);
 }
@@ -191,12 +256,12 @@ main(int argc, char * const *argv)
 	int c;
 	struct VSM_data *vd;
 	const struct VSC_C_main *VSC_C_main;
-	int delay = 1, once = 0, xml = 0;
+	int delay = 1, once = 0, xml = 0, json = 0, do_repeat = 0;
 
 	vd = VSM_New();
 	VSC_Setup(vd);
 
-	while ((c = getopt(argc, argv, VSC_ARGS "1f:lVw:x")) != -1) {
+	while ((c = getopt(argc, argv, VSC_ARGS "1f:lVw:xjt:")) != -1) {
 		switch (c) {
 		case '1':
 			once = 1;
@@ -210,11 +275,15 @@ main(int argc, char * const *argv)
 			VCS_Message("varnishstat");
 			exit(0);
 		case 'w':
+			do_repeat = 1;
 			delay = atoi(optarg);
 			break;
 		case 'x':
 			xml = 1;
 			break;
+		case 'j':
+			json = 1;
+			break;
 		default:
 			if (VSC_Arg(vd, c, optarg) > 0)
 				break;
@@ -227,12 +296,27 @@ main(int argc, char * const *argv)
 
 	VSC_C_main = VSC_Main(vd);
 
-	if (xml)
-		do_xml(vd);
-	else if (once)
-		do_once(vd, VSC_C_main);
-	else
+	if (!(xml || json || once)) {
 		do_curses(vd, VSC_C_main, delay);
+		exit(0);
+	}
+
+	while (1) {
+		if (xml)
+			do_xml(vd);
+		else if (json)
+			do_json(vd);
+		else if (once)
+			do_once(vd, VSC_C_main);
+		else {
+			assert(0);
+		}
+		if (!do_repeat) break;
+
+		// end of output block marker.
+		printf("\n");
 
+		sleep(delay);
+	} 
 	exit(0);
 }
diff --git a/doc/sphinx/reference/varnishstat.rst b/doc/sphinx/reference/varnishstat.rst
index 09c4989..3c5d541 100644
--- a/doc/sphinx/reference/varnishstat.rst
+++ b/doc/sphinx/reference/varnishstat.rst
@@ -10,15 +10,16 @@ Varnish Cache statistics
 
 :Author: Dag-Erling Sm?rgrav
 :Author: Per Buer
-:Date:   2010-06-1
-:Version: 1.0
+:Author: Lasse Karstensen
+:Date:   2011-11-07
+:Version: 1.1
 :Manual section: 1
 
 
 SYNOPSIS
 ========
 
-varnishstat [-1] [-x] [-f field_list] [-l] [-n varnish_name] [-V] [-w delay]
+varnishstat [-1] [-x] [-j] [-f field_list] [-l] [-n varnish_name] [-V] [-w delay]
 
 DESCRIPTION
 ===========
@@ -27,7 +28,7 @@ The varnishstat utility displays statistics from a running varnishd(1) instance.
 
 The following options are available:
 
--1          Instead of presenting of a continuously updated display, print the statistics once and exit.
+-1          Instead of presenting of a continuously updated display, print the statistics to stdout.
 
 -f          A comma separated list of the fields to display.  If it starts with '^' it is used as an exclusion
 	    list.
@@ -39,9 +40,11 @@ The following options are available:
 
 -V          Display the version number and exit.
 
--w delay    Wait delay seconds between updates.  The default is 1.
+-w delay    Wait delay seconds between updates.  The default is 1. Can also be used with -1, -x or -j for repeated output.
 
--x          Displays the result as XML once.
+-x          Displays the result as XML.
+
+-j          Displays the result as JSON.
 
 The columns in the main display are, from left to right:
 
@@ -65,6 +68,29 @@ When using the -x option, the output is::
     <description>FIELD DESCRIPTION</description> 
   </stat> 
 
+With -j the output format is::
+
+  {
+    "timestamp": "YYYY-MM-DDTHH:mm:SS",
+    "client_conn": {
+	"value": 0, "flag": "a", 
+	"description": "Client connections accepted"
+    },
+    "client_drop": {
+	"value": 0, "flag": "a", 
+	"description": "Connection dropped, no sess/wrk"
+    },
+    "LCK.backend.creat": {
+	"type": "LCK", "ident": "backend", "value": 1, 
+	"flag": "a", "description": "Created locks"
+    },
+    [..]
+  }
+
+Timestamp is the time when the report was generated by varnishstat.
+
+Repeated output with -1, -x or -j will have a single empty line (\\n) between each block of output.
+
 
 SEE ALSO
 ========
@@ -91,4 +117,4 @@ This document is licensed under the same licence as Varnish
 itself. See LICENCE for details.
 
 * Copyright (c) 2006 Verdens Gang AS
-* Copyright (c) 2006-2008 Varnish Software AS
+* Copyright (c) 2006-2011 Varnish Software AS



From tfheen at varnish-cache.org  Mon Nov  7 12:47:20 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Mon, 07 Nov 2011 13:47:20 +0100
Subject: [master] ae14008 Simplify json callback
Message-ID: <E1RNObg-00013P-Hv@project.varnish-software.com>

commit ae14008200db8e4a5574af34a07d0ea517b12183
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Mon Nov 7 13:26:51 2011 +0100

    Simplify json callback

diff --git a/bin/varnishstat/varnishstat.c b/bin/varnishstat/varnishstat.c
index c20b87f..b5d6df3 100644
--- a/bin/varnishstat/varnishstat.c
+++ b/bin/varnishstat/varnishstat.c
@@ -90,8 +90,6 @@ do_json_cb(void *priv, const struct VSC_point * const pt)
 {
 	uint64_t val;
 	int *jp;
-	char jsonkey[255];
-	char jsontmp[255];
 
 	jp = priv;
 
@@ -99,18 +97,14 @@ do_json_cb(void *priv, const struct VSC_point * const pt)
 	val = *(const volatile uint64_t*)pt->ptr;
 
 	if (*jp) *jp = 0; else printf(",\n");
-	jsonkey[0] = '\0';
 
+	printf("\t\"");
 	/* build the JSON key name.  */
-	if (strcmp(pt->ident, "")  && strcmp(pt->class, ""))  sprintf(jsonkey, "%s.%s", pt->class, pt->ident);
-	if (strcmp(pt->ident, "")  && !strcmp(pt->class, "")) sprintf(jsonkey, "%s", pt->ident);
-	if (!strcmp(pt->ident, "") && strcmp(pt->class, ""))  sprintf(jsonkey, "%s", pt->class);
-
-	strcpy(jsontmp, jsonkey);
-	if (strcmp(jsonkey, "")) sprintf(jsonkey, "%s.%s", jsontmp, pt->name);
-	else strcpy(jsonkey, pt->name);
-
-	printf("\t\"%s\": {", jsonkey);
+	if (pt->class[0])
+		printf("%s.", pt->class);
+	if (pt->ident[0])
+		printf("%s.", pt->ident);
+	printf("%s\": {", pt->name);
 
 	if (strcmp(pt->class, "")) printf("\"type\": \"%s\", ",  pt->class);
 	if (strcmp(pt->ident, "")) printf("\"ident\": \"%s\", ", pt->ident);



From phk at varnish-cache.org  Mon Nov  7 12:54:12 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 07 Nov 2011 13:54:12 +0100
Subject: [master] 6afe314 One more backend.list test to get all of the code.
Message-ID: <E1RNOiK-00038h-HE@project.varnish-software.com>

commit 6afe3140441f0b81723f2c78bf2cb26080df508b
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Nov 7 12:51:14 2011 +0000

    One more backend.list test to get all of the code.

diff --git a/bin/varnishtest/tests/c00048.vtc b/bin/varnishtest/tests/c00048.vtc
index 4ddb888..2502275 100644
--- a/bin/varnishtest/tests/c00048.vtc
+++ b/bin/varnishtest/tests/c00048.vtc
@@ -61,4 +61,5 @@ varnish v1 -clierr 106 "backend.set_health s2 auto"
 varnish v1 -cliok "backend.list (foo)"
 varnish v1 -clierr 300 "backend.list ("
 varnish v1 -clierr 300 {backend.list " ( : ) -"}
+varnish v1 -cliok {backend.list "a ( b : c )"}
 



From phk at varnish-cache.org  Mon Nov  7 12:54:12 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 07 Nov 2011 13:54:12 +0100
Subject: [master] 8f304ab Merge branch 'master' of
	ssh://git.varnish-cache.org/git/varnish-cache
Message-ID: <E1RNOiK-00038n-NN@project.varnish-software.com>

commit 8f304abc63cf0d3e88fa634c8e62ead5014a02f4
Merge: 6afe314 ae14008
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Nov 7 12:54:09 2011 +0000

    Merge branch 'master' of ssh://git.varnish-cache.org/git/varnish-cache



From tfheen at varnish-cache.org  Mon Nov  7 13:37:40 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Mon, 07 Nov 2011 14:37:40 +0100
Subject: [master] 35f8ce5 Add proper handling of TCP timeouts on client side
Message-ID: <E1RNPOO-0001Pj-EK@project.varnish-software.com>

commit 35f8ce5db8b3cf1025369bf4dbf6cd71a7556748
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Mon Nov 7 14:05:36 2011 +0100

    Add proper handling of TCP timeouts on client side
    
    Retry if we hit a TCP timeout if some data was sent and only error out
    when we get an error or no data was sent in the time period.

diff --git a/bin/varnishd/cache_wrw.c b/bin/varnishd/cache_wrw.c
index 0790ab2..5673b54 100644
--- a/bin/varnishd/cache_wrw.c
+++ b/bin/varnishd/cache_wrw.c
@@ -123,7 +123,37 @@ WRW_Flush(struct worker *w)
 			wrw->iov[wrw->ciov].iov_len = 0;
 		}
 		i = writev(*wrw->wfd, wrw->iov, wrw->niov);
-		if (i != wrw->liov) {
+		while (i != wrw->liov && i > 0) {
+			/* Remove sent data from start of I/O vector,
+			 * then retry; we hit a timeout, but some data
+			 * was sent.
+
+			 XXX: Add a "minimum sent data per timeout
+			 counter to prevent slowlaris attacks
+			*/
+			size_t used = 0;
+
+			WSL(w, SLT_Debug, *wrw->wfd,
+			    "Hit send timeout, wrote = %ld/%ld; retrying",
+			    i, wrw->liov);
+
+			for (int j = 0; j < wrw->niov; j++) {
+				if (used + wrw->iov[j].iov_len > i) {
+					/* Cutoff is in this iov */
+					int used_here = i - used;
+					wrw->iov[j].iov_len -= used_here;
+					wrw->iov[j].iov_base = (char*)wrw->iov[j].iov_base + used_here;
+					memmove(wrw->iov, &wrw->iov[j],
+						(wrw->niov - j) * sizeof(struct iovec));
+					wrw->niov -= j;
+					wrw->liov -= i;
+					break;
+				}
+				used += wrw->iov[j].iov_len;
+			}
+			i = writev(*wrw->wfd, wrw->iov, wrw->niov);
+		}
+		if (i <= 0) {
 			wrw->werr++;
 			WSL(w, SLT_Debug, *wrw->wfd,
 			    "Write error, retval = %d, len = %d, errno = %s",



From tfheen at varnish-cache.org  Mon Nov  7 13:37:44 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Mon, 07 Nov 2011 14:37:44 +0100
Subject: [master] 561be2e Make it possible to limit the total transfer time.
Message-ID: <E1RNPOS-0001Pr-T0@project.varnish-software.com>

commit 561be2ef8a694fc39b34cf577011a549ba17d417
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Mon Nov 7 14:20:06 2011 +0100

    Make it possible to limit the total transfer time.
    
    Set the default total transfer time to 600s.

diff --git a/bin/varnishd/cache_acceptor.c b/bin/varnishd/cache_acceptor.c
index f92bace..03a1a34 100644
--- a/bin/varnishd/cache_acceptor.c
+++ b/bin/varnishd/cache_acceptor.c
@@ -318,9 +318,9 @@ vca_acct(void *arg)
 	while (1) {
 		(void)sleep(1);
 #ifdef SO_SNDTIMEO_WORKS
-		if (params->send_timeout != send_timeout) {
+		if (params->idle_send_timeout != send_timeout) {
 			need_test = 1;
-			send_timeout = params->send_timeout;
+			send_timeout = params->idle_send_timeout;
 			tv_sndtimeo = VTIM_timeval(send_timeout);
 			VTAILQ_FOREACH(ls, &heritage.socks, list) {
 				if (ls->sock < 0)
diff --git a/bin/varnishd/cache_wrw.c b/bin/varnishd/cache_wrw.c
index 5673b54..3e2de4a 100644
--- a/bin/varnishd/cache_wrw.c
+++ b/bin/varnishd/cache_wrw.c
@@ -51,6 +51,7 @@
 #include <stdio.h>
 
 #include "cache.h"
+#include "vtim.h"
 
 /*--------------------------------------------------------------------
  */
@@ -133,6 +134,14 @@ WRW_Flush(struct worker *w)
 			*/
 			size_t used = 0;
 
+			if (VTIM_real() - w->sp->t_resp > params->send_timeout) {
+				WSL(w, SLT_Debug, *wrw->wfd,
+				    "Hit total send timeout, wrote = %ld/%ld; not retrying",
+				    i, wrw->liov);
+				i = -1;
+				break;
+			}
+
 			WSL(w, SLT_Debug, *wrw->wfd,
 			    "Hit send timeout, wrote = %ld/%ld; retrying",
 			    i, wrw->liov);
diff --git a/bin/varnishd/heritage.h b/bin/varnishd/heritage.h
index bb684b3..3732f18 100644
--- a/bin/varnishd/heritage.h
+++ b/bin/varnishd/heritage.h
@@ -112,6 +112,7 @@ struct params {
 	unsigned		sess_timeout;
 	unsigned		pipe_timeout;
 	unsigned		send_timeout;
+	unsigned		idle_send_timeout;
 
 	/* Management hints */
 	unsigned		auto_restart;
diff --git a/bin/varnishd/mgt/mgt_param.c b/bin/varnishd/mgt/mgt_param.c
index 8665ff5..3169914 100644
--- a/bin/varnishd/mgt/mgt_param.c
+++ b/bin/varnishd/mgt/mgt_param.c
@@ -613,6 +613,13 @@ static const struct parspec input_parspec[] = {
                 "seconds the session is closed. \n"
 		"See setsockopt(2) under SO_SNDTIMEO for more information.",
 		DELAYED_EFFECT,
+		"600", "seconds" },
+	{ "idle_send_timeout", tweak_timeout, &master.idle_send_timeout, 0, 0,
+		"Time to wait with no data sent. "
+		"If no data has been transmitted in this many\n"
+                "seconds the session is closed. \n"
+		"See setsockopt(2) under SO_SNDTIMEO for more information.",
+		DELAYED_EFFECT,
 		"60", "seconds" },
 	{ "auto_restart", tweak_bool, &master.auto_restart, 0, 0,
 		"Restart child process automatically if it dies.\n",



From phk at varnish-cache.org  Tue Nov  8 10:04:05 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 08 Nov 2011 11:04:05 +0100
Subject: [master] b4aef74 A minor preemptive cleanup before the ban-lurker
	gets remodelled.
Message-ID: <E1RNiXF-00062p-36@project.varnish-software.com>

commit b4aef74f44a1b712ab5a0fb016c1ba8d9a5565f9
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 8 10:03:43 2011 +0000

    A minor preemptive cleanup before the ban-lurker gets remodelled.

diff --git a/bin/varnishd/cache.h b/bin/varnishd/cache.h
index 1875ece..31db957 100644
--- a/bin/varnishd/cache.h
+++ b/bin/varnishd/cache.h
@@ -445,6 +445,7 @@ struct objcore {
 #define OC_F_PASS		(1<<2)
 #define OC_F_LRUDONTMOVE	(1<<4)
 #define OC_F_PRIV		(1<<5)		/* Stevedore private flag */
+#define OC_F_LURK		(3<<6)		/* Ban-lurker-color */
 	unsigned		timer_idx;
 	VTAILQ_ENTRY(objcore)	list;
 	VTAILQ_ENTRY(objcore)	lru_list;
diff --git a/bin/varnishd/cache_ban.c b/bin/varnishd/cache_ban.c
index 5b7d457..e7478a0 100644
--- a/bin/varnishd/cache_ban.c
+++ b/bin/varnishd/cache_ban.c
@@ -81,6 +81,7 @@ struct ban {
 	unsigned		flags;
 #define BAN_F_GONE		(1 << 0)
 #define BAN_F_REQ		(1 << 2)
+#define BAN_F_LURK		(3 << 3)	/* ban-lurker-color */
 	VTAILQ_HEAD(,objcore)	objcore;
 	struct vsb		*vsb;
 	uint8_t			*spec;
@@ -164,23 +165,6 @@ BAN_Free(struct ban *b)
 	FREE_OBJ(b);
 }
 
-static struct ban *
-ban_CheckLast(void)
-{
-	struct ban *b;
-
-	Lck_AssertHeld(&ban_mtx);
-	b = VTAILQ_LAST(&ban_head, banhead_s);
-	if (b != VTAILQ_FIRST(&ban_head) && b->refcount == 0) {
-		VSC_C_main->n_ban--;
-		VSC_C_main->n_ban_retire++;
-		VTAILQ_REMOVE(&ban_head, b, list);
-	} else {
-		b = NULL;
-	}
-	return (b);
-}
-
 /*--------------------------------------------------------------------
  * Get & Release a tail reference, used to hold the list stable for
  * traversals etc.
@@ -473,7 +457,6 @@ BAN_NewObjCore(struct objcore *oc)
 void
 BAN_DestroyObj(struct objcore *oc)
 {
-	struct ban *b;
 
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 	if (oc->ban == NULL)
@@ -484,12 +467,7 @@ BAN_DestroyObj(struct objcore *oc)
 	oc->ban->refcount--;
 	VTAILQ_REMOVE(&oc->ban->objcore, oc, ban_list);
 	oc->ban = NULL;
-
-	/* Attempt to purge last ban entry */
-	b = ban_CheckLast();
 	Lck_Unlock(&ban_mtx);
-	if (b != NULL)
-		BAN_Free(b);
 }
 
 /*--------------------------------------------------------------------
@@ -750,6 +728,23 @@ BAN_CheckObject(struct object *o, const struct sess *sp)
 	return (ban_check_object(o, sp, 1));
 }
 
+static struct ban *
+ban_CheckLast(void)
+{
+	struct ban *b;
+
+	Lck_AssertHeld(&ban_mtx);
+	b = VTAILQ_LAST(&ban_head, banhead_s);
+	if (b != VTAILQ_FIRST(&ban_head) && b->refcount == 0) {
+		VSC_C_main->n_ban--;
+		VSC_C_main->n_ban_retire++;
+		VTAILQ_REMOVE(&ban_head, b, list);
+	} else {
+		b = NULL;
+	}
+	return (b);
+}
+
 /*--------------------------------------------------------------------
  * Ban tail lurker thread
  */
@@ -763,9 +758,6 @@ ban_lurker_work(const struct sess *sp)
 	struct object *o;
 	int i;
 
-	WSL_Flush(sp->wrk, 0);
-	WRK_SumStat(sp->wrk);
-
 	Lck_Lock(&ban_mtx);
 
 	/* First try to route the last ban */
@@ -831,21 +823,34 @@ ban_lurker_work(const struct sess *sp)
 static void * __match_proto__(bgthread_t)
 ban_lurker(struct sess *sp, void *priv)
 {
+	struct ban *bf;
+
 	int i = 0;
 	(void)priv;
 	while (1) {
-		if (params->ban_lurker_sleep == 0.0) {
-			/* Lurker is disabled.  */
-			VTIM_sleep(1.0);
-			continue;
+
+		while (params->ban_lurker_sleep == 0.0) {
+			/*
+			 * Ban-lurker is disabled:
+			 * Clean the last ban, if possible, and sleep
+			 */
+			Lck_Lock(&ban_mtx);
+			bf = ban_CheckLast();
+			Lck_Unlock(&ban_mtx);
+			if (bf != NULL)
+				BAN_Free(bf);
+			else
+				VTIM_sleep(1.0);
 		}
+
+		i = ban_lurker_work(sp);
+		WSL_Flush(sp->wrk, 0);
+		WRK_SumStat(sp->wrk);
+
 		if (i != 0)
 			VTIM_sleep(params->ban_lurker_sleep);
 		else
 			VTIM_sleep(1.0);
-		i = ban_lurker_work(sp);
-		WSL_Flush(sp->wrk, 0);
-		WRK_SumStat(sp->wrk);
 	}
 	NEEDLESS_RETURN(NULL);
 }
@@ -957,10 +962,11 @@ ccf_ban_list(struct cli *cli, const char * const *av, void *priv)
 	/* Get a reference so we are safe to traverse the list */
 	bl = BAN_TailRef();
 
+	VCLI_Out(cli, "Present bans:\n");
 	VTAILQ_FOREACH(b, &ban_head, list) {
 		if (b == bl)
 			break;
-		VCLI_Out(cli, "%p %10.6f %5u%s\t", b, ban_time(b->spec),
+		VCLI_Out(cli, "%10.6f %5u%s\t", ban_time(b->spec),
 		    bl == b ? b->refcount - 1 : b->refcount,
 		    b->flags & BAN_F_GONE ? "G" : " ");
 		ban_render(cli, b->spec);



From phk at varnish-cache.org  Tue Nov  8 13:08:52 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 08 Nov 2011 14:08:52 +0100
Subject: [master] 0dead8f Rewrite the ban-lurker.
Message-ID: <E1RNlQ4-0001Ym-O5@project.varnish-software.com>

commit 0dead8f0bd7b69d6d1530f824406c31f9df7b3ee
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 8 13:07:52 2011 +0000

    Rewrite the ban-lurker.
    
    Use a mark&mark strategy to test all obj.* bans even if there are
    req.* bans in the way.

diff --git a/bin/varnishd/cache_ban.c b/bin/varnishd/cache_ban.c
index e7478a0..8dc7cb4 100644
--- a/bin/varnishd/cache_ban.c
+++ b/bin/varnishd/cache_ban.c
@@ -81,12 +81,14 @@ struct ban {
 	unsigned		flags;
 #define BAN_F_GONE		(1 << 0)
 #define BAN_F_REQ		(1 << 2)
-#define BAN_F_LURK		(3 << 3)	/* ban-lurker-color */
+#define BAN_F_LURK		(3 << 6)	/* ban-lurker-color */
 	VTAILQ_HEAD(,objcore)	objcore;
 	struct vsb		*vsb;
 	uint8_t			*spec;
 };
 
+#define LURK_SHIFT 6
+
 struct ban_test {
 	uint8_t			arg1;
 	const char		*arg1_spec;
@@ -417,6 +419,7 @@ BAN_Insert(struct ban *b)
 	/* Hunt down duplicates, and mark them as gone */
 	bi = b;
 	pcount = 0;
+	Lck_Lock(&ban_mtx);
 	while(bi != be) {
 		bi = VTAILQ_NEXT(bi, list);
 		if (bi->flags & BAN_F_GONE)
@@ -425,9 +428,9 @@ BAN_Insert(struct ban *b)
 		if (memcmp(b->spec + 8, bi->spec + 8, ln - 8))
 			continue;
 		bi->flags |= BAN_F_GONE;
+		VSC_C_main->n_ban_gone++;
 		pcount++;
 	}
-	Lck_Lock(&ban_mtx);
 	be->refcount--;
 	VSC_C_main->n_ban_dups += pcount;
 	Lck_Unlock(&ban_mtx);
@@ -643,7 +646,13 @@ ban_evaluate(const uint8_t *bs, const struct http *objhttp,
 }
 
 /*--------------------------------------------------------------------
- * Check an object any fresh bans
+ * Check an object against all applicable bans
+ *
+ * Return:
+ *	-1 not all bans checked, but none of the checked matched
+ *		Only if !has_req
+ *	0 No bans matched, object moved to ban_start.
+ *	1 Ban matched, object removed from ban list.
  */
 
 static int
@@ -677,6 +686,12 @@ ban_check_object(struct object *o, const struct sess *sp, int has_req)
 		CHECK_OBJ_NOTNULL(b, BAN_MAGIC);
 		if (b->flags & BAN_F_GONE)
 			continue;
+		if ((b->flags & BAN_F_LURK) &&
+		    (b->flags & BAN_F_LURK) == (oc->flags & OC_F_LURK)) {
+			AZ(b->flags & BAN_F_REQ);
+			/* Lurker already tested this */
+			continue;
+		}
 		if (!has_req && (b->flags & BAN_F_REQ)) {
 			/*
 			 * We cannot test this one, but there might
@@ -687,23 +702,27 @@ ban_check_object(struct object *o, const struct sess *sp, int has_req)
 			break;
 	}
 
+	Lck_Lock(&ban_mtx);
+	VSC_C_main->n_ban_obj_test++;
+	VSC_C_main->n_ban_re_test += tests;
+
 	if (b == oc->ban && skipped > 0) {
+		AZ(has_req);
+		Lck_Unlock(&ban_mtx);
 		/*
 		 * Not banned, but some tests were skipped, so we cannot know
 		 * for certain that it cannot be, so we just have to give up.
 		 */
-		return (0);
+		return (-1);
 	}
 
-	Lck_Lock(&ban_mtx);
 	oc->ban->refcount--;
 	VTAILQ_REMOVE(&oc->ban->objcore, oc, ban_list);
 	if (b == oc->ban) {	/* not banned */
+		b->flags &= ~BAN_F_LURK;
 		VTAILQ_INSERT_TAIL(&b0->objcore, oc, ban_list);
 		b0->refcount++;
 	}
-	VSC_C_main->n_ban_obj_test++;
-	VSC_C_main->n_ban_re_test += tests;
 	Lck_Unlock(&ban_mtx);
 
 	if (b == oc->ban) {	/* not banned */
@@ -725,7 +744,7 @@ int
 BAN_CheckObject(struct object *o, const struct sess *sp)
 {
 
-	return (ban_check_object(o, sp, 1));
+	return (ban_check_object(o, sp, 1) > 0);
 }
 
 static struct ban *
@@ -736,6 +755,8 @@ ban_CheckLast(void)
 	Lck_AssertHeld(&ban_mtx);
 	b = VTAILQ_LAST(&ban_head, banhead_s);
 	if (b != VTAILQ_FIRST(&ban_head) && b->refcount == 0) {
+		if (b->flags & BAN_F_GONE)
+			VSC_C_main->n_ban_gone--;
 		VSC_C_main->n_ban--;
 		VSC_C_main->n_ban_retire++;
 		VTAILQ_REMOVE(&ban_head, b, list);
@@ -746,84 +767,144 @@ ban_CheckLast(void)
 }
 
 /*--------------------------------------------------------------------
- * Ban tail lurker thread
+ * Ban lurker thread
  */
 
 static int
-ban_lurker_work(const struct sess *sp)
+ban_lurker_work(const struct sess *sp, unsigned pass)
 {
-	struct ban *b, *bf;
+	struct ban *b, *b0, *b2;
 	struct objhead *oh;
 	struct objcore *oc, *oc2;
 	struct object *o;
 	int i;
 
-	Lck_Lock(&ban_mtx);
+	AN(pass & BAN_F_LURK);
+	AZ(pass & ~BAN_F_LURK);
 
-	/* First try to route the last ban */
-	bf = ban_CheckLast();
-	if (bf != NULL) {
+	/* First route the last ban(s) */
+	do {
+		Lck_Lock(&ban_mtx);
+		b2 = ban_CheckLast();
 		Lck_Unlock(&ban_mtx);
-		BAN_Free(bf);
-		return (0);
-	}
+		if (b2 != NULL)
+			BAN_Free(b2);
+	} while (b2 != NULL);
 
-	/* Find the last ban give up, if we have only one */
-	b = VTAILQ_LAST(&ban_head, banhead_s);
-	if (b == ban_start) {
-		Lck_Unlock(&ban_mtx);
-		return (0);
+	/*
+	 * Find out if we have any bans we can do something about
+	 * If we find any, tag them with our pass number.
+	 */
+	i = 0;
+	b0 = NULL;
+	VTAILQ_FOREACH(b, &ban_head, list) {
+		if (b->flags & BAN_F_GONE)
+			continue;
+		if (b->flags & BAN_F_REQ)
+			continue;
+		if (b == VTAILQ_LAST(&ban_head, banhead_s))
+			continue;
+		if (b0 == NULL)
+			b0 = b;
+		i++;
+		b->flags &= ~BAN_F_LURK;
+		b->flags |= pass;
 	}
-
-	/* Find the first object on it, if any */
-	oc = VTAILQ_FIRST(&b->objcore);
-	if (oc == NULL) {
-		Lck_Unlock(&ban_mtx);
+	if (params->diag_bitmap & 0x80000)
+		VSL(SLT_Debug, 0, "lurker: %d actionable bans", i);
+	if (i == 0)
 		return (0);
-	}
 
-	/* Try to lock the objhead */
-	oh = oc->objhead;
-	CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC);
-	if (Lck_Trylock(&oh->mtx)) {
+	VTAILQ_FOREACH_REVERSE(b, &ban_head, banhead_s, list) {
+		if (params->diag_bitmap & 0x80000)
+			VSL(SLT_Debug, 0, "lurker doing %f %d",
+			    ban_time(b->spec), b->refcount);
+		while (1) {
+			Lck_Lock(&ban_mtx);
+			oc = VTAILQ_FIRST(&b->objcore);
+			if (oc == NULL)
+				break;
+			CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+			if (params->diag_bitmap & 0x80000)
+				VSL(SLT_Debug, 0, "test: %p %d %d",
+				    oc, oc->flags & OC_F_LURK, pass);
+			if ((oc->flags & OC_F_LURK) == pass)
+				break;
+			oh = oc->objhead;
+			CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC);
+			if (Lck_Trylock(&oh->mtx)) {
+				Lck_Unlock(&ban_mtx);
+				VTIM_sleep(params->ban_lurker_sleep);
+				continue;
+			}
+			/*
+			 * See if the objcore is still on the objhead since
+			 * we race against HSH_Deref() which comes in the
+			 * opposite locking order.
+			 */
+			VTAILQ_FOREACH(oc2, &oh->objcs, list)
+				if (oc == oc2)
+					break;
+			if (oc2 == NULL) {
+				Lck_Unlock(&oh->mtx);
+				Lck_Unlock(&ban_mtx);
+				VTIM_sleep(params->ban_lurker_sleep);
+				continue;
+			}
+			/*
+			 * Grab a reference to the OC and we can let go of
+			 * the BAN mutex
+			 */
+			AN(oc->refcnt);
+			oc->refcnt++;
+			oc->flags &= ~OC_F_LURK;
+			Lck_Unlock(&ban_mtx);
+			/*
+			 * Get the object and check it against all relevant bans
+			 */
+			o = oc_getobj(sp->wrk, oc);
+			i = ban_check_object(o, sp, 0);
+			if (params->diag_bitmap & 0x80000)
+				VSL(SLT_Debug, 0, "lurker got: %p %d",
+				    oc, i);
+			if (i == -1) {
+				/* Not banned, not moved */
+				oc->flags |= pass;
+				Lck_Lock(&ban_mtx);
+				VTAILQ_REMOVE(&b->objcore, oc, ban_list);
+				VTAILQ_INSERT_TAIL(&b->objcore, oc, ban_list);
+				Lck_Unlock(&ban_mtx);
+			}
+			Lck_Unlock(&oh->mtx);
+			if (params->diag_bitmap & 0x80000)
+				VSL(SLT_Debug, 0, "lurker done: %p %d %d",
+				    oc, oc->flags & OC_F_LURK, pass);
+			(void)HSH_Deref(sp->wrk, NULL, &o);
+			VTIM_sleep(params->ban_lurker_sleep);
+		}
+		Lck_AssertHeld(&ban_mtx);
+		if (!(b->flags & BAN_F_REQ)) {
+			if (!(b->flags & BAN_F_GONE)) {
+				b->flags |= BAN_F_GONE;
+				VSC_C_main->n_ban_gone++;
+			}
+			if (params->diag_bitmap & 0x80000)
+				VSL(SLT_Debug, 0, "lurker BAN %f now gone",
+				    ban_time(b->spec));
+		}
 		Lck_Unlock(&ban_mtx);
-		return (0);
-	}
-
-	/*
-	 * See if the objcore is still on the objhead since we race against
-	 * HSH_Deref() which comes in the opposite locking order.
-	 */
-	VTAILQ_FOREACH(oc2, &oh->objcs, list)
-		if (oc == oc2)
+		VTIM_sleep(params->ban_lurker_sleep);
+		if (b == b0)
 			break;
-	if (oc2 == NULL) {
-		Lck_Unlock(&oh->mtx);
-		Lck_Unlock(&ban_mtx);
-		return (0);
 	}
-	/*
-	 * Grab a reference to the OC and we can let go of the BAN mutex
-	 */
-	AN(oc->refcnt);
-	oc->refcnt++;
-	Lck_Unlock(&ban_mtx);
-
-	/*
-	 * Get the object and check it against all relevant bans
-	 */
-	o = oc_getobj(sp->wrk, oc);
-	i = ban_check_object(o, sp, 0);
-	Lck_Unlock(&oh->mtx);
-	WSP(sp, SLT_Debug, "lurker: %p %g %d", oc, o->exp.ttl, i);
-	(void)HSH_Deref(sp->wrk, NULL, &o);
-	return (i);
+	return (1);
 }
 
 static void * __match_proto__(bgthread_t)
 ban_lurker(struct sess *sp, void *priv)
 {
 	struct ban *bf;
+	unsigned pass = (1 << LURK_SHIFT);
 
 	int i = 0;
 	(void)priv;
@@ -843,14 +924,18 @@ ban_lurker(struct sess *sp, void *priv)
 				VTIM_sleep(1.0);
 		}
 
-		i = ban_lurker_work(sp);
+		i = ban_lurker_work(sp, pass);
 		WSL_Flush(sp->wrk, 0);
 		WRK_SumStat(sp->wrk);
-
-		if (i != 0)
+		if (i) {
+			pass += (1 << LURK_SHIFT);
+			pass &= BAN_F_LURK;
+			if (pass == 0)
+				pass += (1 << LURK_SHIFT);
 			VTIM_sleep(params->ban_lurker_sleep);
-		else
+		} else {
 			VTIM_sleep(1.0);
+		}
 	}
 	NEEDLESS_RETURN(NULL);
 }
@@ -964,13 +1049,20 @@ ccf_ban_list(struct cli *cli, const char * const *av, void *priv)
 
 	VCLI_Out(cli, "Present bans:\n");
 	VTAILQ_FOREACH(b, &ban_head, list) {
-		if (b == bl)
+		if (b == bl && !(params->diag_bitmap & 0x80000))
 			break;
 		VCLI_Out(cli, "%10.6f %5u%s\t", ban_time(b->spec),
 		    bl == b ? b->refcount - 1 : b->refcount,
 		    b->flags & BAN_F_GONE ? "G" : " ");
 		ban_render(cli, b->spec);
 		VCLI_Out(cli, "\n");
+		if (params->diag_bitmap & 0x80000) {
+			Lck_Lock(&ban_mtx);
+			struct objcore *oc;
+			VTAILQ_FOREACH(oc, &b->objcore, ban_list)
+				VCLI_Out(cli, "     %p\n", oc);
+			Lck_Unlock(&ban_mtx);
+		}
 	}
 
 	BAN_TailDeref(&bl);
@@ -989,10 +1081,14 @@ BAN_Init(void)
 
 	Lck_New(&ban_mtx, lck_ban);
 	CLI_AddFuncs(ban_cmds);
+	assert(BAN_F_LURK == OC_F_LURK);
+	AN((1 << LURK_SHIFT) & BAN_F_LURK);
+	AN((2 << LURK_SHIFT) & BAN_F_LURK);
 
 	ban_magic = BAN_New();
 	AN(ban_magic);
 	ban_magic->flags |= BAN_F_GONE;
+	VSC_C_main->n_ban_gone++;
 	BAN_Insert(ban_magic);
 	WRK_BgThread(&ban_thread, "ban-lurker", ban_lurker, NULL);
 }
diff --git a/bin/varnishd/mgt/mgt_param.c b/bin/varnishd/mgt/mgt_param.c
index 3169914..e2eb16b 100644
--- a/bin/varnishd/mgt/mgt_param.c
+++ b/bin/varnishd/mgt/mgt_param.c
@@ -841,6 +841,7 @@ static const struct parspec input_parspec[] = {
 		"  0x00010000 - synchronize shmlog.\n"
 		"  0x00020000 - synchronous start of persistence.\n"
 		"  0x00040000 - release VCL early.\n"
+		"  0x00080000 - ban-lurker debugging.\n"
 		"  0x80000000 - do edge-detection on digest.\n"
 		"Use 0x notation and do the bitor in your head :-)\n",
 		0,
diff --git a/bin/varnishtest/tests/c00049.vtc b/bin/varnishtest/tests/c00049.vtc
new file mode 100644
index 0000000..facd979
--- /dev/null
+++ b/bin/varnishtest/tests/c00049.vtc
@@ -0,0 +1,131 @@
+varnishtest "ban lurker test"
+
+
+server s1 {
+	rxreq
+	expect req.url == "/alpha"
+	txresp -hdr "Foo: /alpha"
+
+	rxreq
+	expect req.url == "/beta"
+	txresp -hdr "Foo: /beta"
+
+	rxreq
+	expect req.url == "/gamma"
+	txresp -hdr "Foo: /gamma"
+
+	rxreq
+	expect req.url == "/delta"
+	txresp -hdr "Foo: /delta"
+
+	rxreq
+	expect req.url == "/alpha"
+	txresp -hdr "Foo: /alpha2"
+
+	rxreq
+	expect req.url == "/beta"
+	txresp -hdr "Foo: /beta2"
+
+	rxreq
+	expect req.url == "/delta"
+	txresp -hdr "Foo: /delta2"
+
+} -start
+
+varnish v1 -vcl+backend {
+} -start
+
+varnish v1 -cliok "param.set ban_lurker_sleep 0"
+varnish v1 -cliok "param.set diag_bitmap 0x80000"
+
+varnish v1 -cliok "ban.list"
+
+client c1 {
+	txreq -url "/alpha"
+	rxresp
+	expect resp.http.foo == /alpha
+} -run
+
+delay 0.1
+varnish v1 -cliok "ban req.url == /alpha"
+varnish v1 -cliok "ban.list"
+varnish v1 -expect n_ban == 2
+varnish v1 -expect n_ban_gone == 1
+
+client c1 {
+	txreq -url "/beta"
+	rxresp
+	expect resp.http.foo == /beta
+} -run
+
+delay 0.1
+varnish v1 -cliok "ban obj.http.foo == /beta"
+varnish v1 -cliok "ban.list"
+varnish v1 -expect n_ban == 3
+
+client c1 {
+	txreq -url "/gamma"
+	rxresp
+	expect resp.http.foo == /gamma
+} -run
+
+delay 0.1
+varnish v1 -cliok "ban obj.http.foo == /gamma"
+varnish v1 -cliok "ban.list"
+varnish v1 -expect n_ban == 4
+
+client c1 {
+	txreq -url "/delta"
+	rxresp
+	expect resp.http.foo == /delta
+} -run
+
+delay 0.1
+varnish v1 -cliok "ban req.url == /delta"
+
+varnish v1 -expect n_ban_gone == 1
+varnish v1 -cliok "ban obj.http.foo == /gamma"
+# Dup-check should have added one
+varnish v1 -expect n_ban_gone == 2
+
+varnish v1 -cliok "ban req.url == /epsilon"
+varnish v1 -cliok "ban.list"
+varnish v1 -expect n_ban == 7
+varnish v1 -expect n_ban_gone == 2
+
+varnish v1 -cliok "param.set ban_lurker_sleep .01"
+delay 1
+varnish v1 -cliok "param.set ban_lurker_sleep .00"
+varnish v1 -cliok "ban.list"
+varnish v1 -expect n_ban == 7
+varnish v1 -expect n_ban_gone == 4
+
+client c1 {
+	txreq -url "/alpha"
+	rxresp
+	expect resp.http.foo == /alpha2
+} -run
+
+delay 1
+varnish v1 -cliok "ban.list"
+varnish v1 -expect n_ban == 4
+
+client c1 {
+	txreq -url "/beta"
+	rxresp
+	expect resp.http.foo == /beta2
+} -run
+
+varnish v1 -cliok "ban.list"
+
+
+client c1 {
+	txreq -url "/delta"
+	rxresp
+	expect resp.http.foo == /delta2
+} -run
+
+delay 1
+varnish v1 -cliok "ban.list"
+varnish v1 -expect n_ban == 1
+varnish v1 -expect n_ban_gone == 0



From phk at varnish-cache.org  Tue Nov  8 13:10:36 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 08 Nov 2011 14:10:36 +0100
Subject: [master] 78e7b84 Add a new stats counter for "gone" marked bans
Message-ID: <E1RNlRk-0001t2-PG@project.varnish-software.com>

commit 78e7b84010fc8db7a1b9ad35ff76b80f8323d1c1
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 8 13:10:24 2011 +0000

    Add a new stats counter for "gone" marked bans

diff --git a/include/tbl/vsc_fields.h b/include/tbl/vsc_fields.h
index 0fd7e46..904a9e2 100644
--- a/include/tbl/vsc_fields.h
+++ b/include/tbl/vsc_fields.h
@@ -297,12 +297,13 @@ VSC_F(n_vcl,		uint64_t, 0, 'a', "N vcl total", "")
 VSC_F(n_vcl_avail,		uint64_t, 0, 'a', "N vcl available", "")
 VSC_F(n_vcl_discard,	uint64_t, 0, 'a', "N vcl discarded", "")
 
-VSC_F(n_ban,		uint64_t, 0, 'i', "N total active bans", "")
+VSC_F(n_ban,			uint64_t, 0, 'i', "N total active bans", "")
+VSC_F(n_ban_gone,		uint64_t, 0, 'i', "N total gone bans", "")
 VSC_F(n_ban_add,		uint64_t, 0, 'a', "N new bans added", "")
-VSC_F(n_ban_retire,	uint64_t, 0, 'a', "N old bans deleted", "")
-VSC_F(n_ban_obj_test,	uint64_t, 0, 'a', "N objects tested", "")
-VSC_F(n_ban_re_test,	uint64_t, 0, 'a', "N regexps tested against", "")
-VSC_F(n_ban_dups,	uint64_t, 0, 'a', "N duplicate bans removed", "")
+VSC_F(n_ban_retire,		uint64_t, 0, 'a', "N old bans deleted", "")
+VSC_F(n_ban_obj_test,		uint64_t, 0, 'a', "N objects tested", "")
+VSC_F(n_ban_re_test,		uint64_t, 0, 'a', "N regexps tested against", "")
+VSC_F(n_ban_dups,		uint64_t, 0, 'a', "N duplicate bans removed", "")
 
 VSC_F(hcb_nolock,		uint64_t, 0, 'a', "HCB Lookups without lock", "")
 VSC_F(hcb_lock,		uint64_t, 0, 'a', "HCB Lookups with lock", "")



From phk at varnish-cache.org  Tue Nov  8 13:44:42 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 08 Nov 2011 14:44:42 +0100
Subject: [master] 94f4464 Don't start the ban-lurker until -spersistent is
	done loading.
Message-ID: <E1RNlyk-0006EU-Cc@project.varnish-software.com>

commit 94f4464792d5b57041bd8e9b8d089f2bd8bd9bb7
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 8 13:44:29 2011 +0000

    Don't start the ban-lurker until -spersistent is done loading.

diff --git a/bin/varnishd/cache_ban.c b/bin/varnishd/cache_ban.c
index 8dc7cb4..c01a3f6 100644
--- a/bin/varnishd/cache_ban.c
+++ b/bin/varnishd/cache_ban.c
@@ -102,6 +102,7 @@ static struct lock ban_mtx;
 static struct ban *ban_magic;
 static pthread_t ban_thread;
 static struct ban * volatile ban_start;
+static bgthread_t ban_lurker;
 
 /*--------------------------------------------------------------------
  * BAN string magic markers
@@ -583,6 +584,7 @@ BAN_Compile(void)
 
 	SMP_NewBan(ban_magic->spec, ban_len(ban_magic->spec));
 	ban_start = VTAILQ_FIRST(&ban_head);
+	WRK_BgThread(&ban_thread, "ban-lurker", ban_lurker, NULL);
 }
 
 /*--------------------------------------------------------------------
@@ -1090,5 +1092,4 @@ BAN_Init(void)
 	ban_magic->flags |= BAN_F_GONE;
 	VSC_C_main->n_ban_gone++;
 	BAN_Insert(ban_magic);
-	WRK_BgThread(&ban_thread, "ban-lurker", ban_lurker, NULL);
 }



From phk at varnish-cache.org  Tue Nov  8 14:50:20 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 08 Nov 2011 15:50:20 +0100
Subject: [master] 2befbaf Add a missing WS_Release()
Message-ID: <E1RNn0G-0003Cj-G9@project.varnish-software.com>

commit 2befbaf97ce358dcb8d74c5e7e5ff8b3b38f87ff
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 8 14:50:11 2011 +0000

    Add a missing WS_Release()

diff --git a/bin/varnishd/cache_vrt_re.c b/bin/varnishd/cache_vrt_re.c
index 955b7bf..d6dc5d5 100644
--- a/bin/varnishd/cache_vrt_re.c
+++ b/bin/varnishd/cache_vrt_re.c
@@ -143,6 +143,7 @@ VRT_regsub(const struct sess *sp, int all, const char *str, void *re,
 		i = VRE_exec(t, str, len, 0, options, ovector, 30,
 		    &params->vre_limits);
 		if (i < VRE_ERROR_NOMATCH ) {
+			WS_Release(sp->http->ws, 0);
 			WSP(sp, SLT_VCL_Error,
 			    "Regexp matching returned %d", i);
 			return(str);



From phk at varnish-cache.org  Tue Nov  8 15:05:06 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 08 Nov 2011 16:05:06 +0100
Subject: [master] 992ebc9 Implement VRE options with hard linkage to PCRE
	options instead of maintaining magic hex-bit values that must match.
Message-ID: <E1RNnEY-0008RB-15@project.varnish-software.com>

commit 992ebc91708a50ebf383f8ec58ac7b402c092617
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 8 15:04:04 2011 +0000

    Implement VRE options with hard linkage to PCRE options instead
    of maintaining magic hex-bit values that must match.

diff --git a/include/vre.h b/include/vre.h
index a1206e5..59ffeb0 100644
--- a/include/vre.h
+++ b/include/vre.h
@@ -48,8 +48,8 @@ typedef struct vre vre_t;
 #define VRE_ERROR_NOMATCH         (-1)
 
 /* And those to PCRE options */
-#define VRE_CASELESS           0x00000001
-#define VRE_NOTEMPTY_ATSTART   0x10000000
+extern const unsigned VRE_CASELESS;
+extern const unsigned VRE_NOTEMPTY_ATSTART;
 
 vre_t *VRE_compile(const char *, int, const char **, int *);
 int VRE_exec(const vre_t *code, const char *subject, int length,
diff --git a/lib/libvarnish/vre.c b/lib/libvarnish/vre.c
index 9b1d911..cffe396 100644
--- a/lib/libvarnish/vre.c
+++ b/lib/libvarnish/vre.c
@@ -43,6 +43,19 @@ struct vre {
 	pcre *re;
 };
 
+/*
+ * We don't want to spread or even expose the majority of PCRE options
+ * so we establish our own options and implement hard linkage to PCRE
+ * here.
+ */
+const unsigned VRE_CASELESS = PCRE_CASELESS;
+const unsigned VRE_NOTEMPTY_ATSTART =
+#ifdef PCRE_NOTEMPTY_ATSTART
+	PCRE_NOTEMPTY_ATSTART;
+#else
+	0;
+#endif
+
 vre_t *
 VRE_compile(const char *pattern, int options,
 		    const char **errptr, int *erroffset)



From phk at varnish-cache.org  Wed Nov  9 09:47:31 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 09 Nov 2011 10:47:31 +0100
Subject: [master] 0fe28b3 Polish whitespace and update copyrights.
Message-ID: <E1RO4kl-0008Ek-VS@project.varnish-software.com>

commit 0fe28b30be96ca4232b6d037598202ba1272290e
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Nov 9 09:47:15 2011 +0000

    Polish whitespace and update copyrights.

diff --git a/bin/varnishd/cache_backend.h b/bin/varnishd/cache_backend.h
index 91bff1e..72a1283 100644
--- a/bin/varnishd/cache_backend.h
+++ b/bin/varnishd/cache_backend.h
@@ -1,6 +1,6 @@
 /*-
  * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2010 Varnish Software AS
+ * Copyright (c) 2006-2011 Varnish Software AS
  * All rights reserved.
  *
  * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
diff --git a/bin/varnishd/cache_backend_cfg.c b/bin/varnishd/cache_backend_cfg.c
index e87584a..cbb8c85 100644
--- a/bin/varnishd/cache_backend_cfg.c
+++ b/bin/varnishd/cache_backend_cfg.c
@@ -1,6 +1,6 @@
 /*-
  * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2010 Varnish Software AS
+ * Copyright (c) 2006-2011 Varnish Software AS
  * All rights reserved.
  *
  * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
diff --git a/bin/varnishd/cache_ban.c b/bin/varnishd/cache_ban.c
index c01a3f6..648997c 100644
--- a/bin/varnishd/cache_ban.c
+++ b/bin/varnishd/cache_ban.c
@@ -51,7 +51,7 @@
  *	(lump) - compiled regexp
  * A lump is:
  *	4 bytes - be32: length
- * 	n bytes - content
+ *	n bytes - content
  *
  * In a perfect world, we should vector through VRE to get to PCRE,
  * but since we rely on PCRE's ability to encode the regexp into a
diff --git a/bin/varnishd/cache_dir.c b/bin/varnishd/cache_dir.c
index 59ef5fc..d4794f9 100644
--- a/bin/varnishd/cache_dir.c
+++ b/bin/varnishd/cache_dir.c
@@ -1,6 +1,6 @@
 /*-
  * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2010 Varnish Software AS
+ * Copyright (c) 2006-2011 Varnish Software AS
  * All rights reserved.
  *
  * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
diff --git a/bin/varnishd/cache_esi_fetch.c b/bin/varnishd/cache_esi_fetch.c
index 728436c..f414e7b 100644
--- a/bin/varnishd/cache_esi_fetch.c
+++ b/bin/varnishd/cache_esi_fetch.c
@@ -240,7 +240,7 @@ vfp_esi_bytes_ug(struct worker *w, struct http_conn *htc, ssize_t bytes)
  * We receive a gzip'ed object, and want to store it gzip'ed.
  */
 
-static int 
+static int
 vfp_esi_bytes_gg(struct worker *w, struct http_conn *htc, size_t bytes)
 {
 	ssize_t wl;
@@ -391,7 +391,7 @@ vfp_esi_end(struct worker *w)
 		w->vef_priv = NULL;
 		VGZ_UpdateObj(vef->vgz, w->fetch_obj);
 		if (VGZ_Destroy(&vef->vgz,  -1) != VGZ_END)
-			retval = FetchError(w, 
+			retval = FetchError(w,
 			    "ESI+Gzip Failed at the very end");
 		FREE_OBJ(vef);
 	}
diff --git a/bin/varnishd/cache_fetch.c b/bin/varnishd/cache_fetch.c
index ed1e5c8..67bf970 100644
--- a/bin/varnishd/cache_fetch.c
+++ b/bin/varnishd/cache_fetch.c
@@ -277,7 +277,7 @@ fetch_chunked(struct worker *w, struct http_conn *htc)
 				break;
 		}
 
-		if (u >= sizeof buf) 
+		if (u >= sizeof buf)
 			return (FetchError(w,"chunked header too long"));
 
 		/* Skip trailing white space */
@@ -285,7 +285,7 @@ fetch_chunked(struct worker *w, struct http_conn *htc)
 			if (HTC_Read(w, htc, buf + u, 1) <= 0)
 				return (-1);
 
-		if (buf[u] != '\n') 
+		if (buf[u] != '\n')
 			return (FetchError(w,"chunked header no NL"));
 
 		buf[u] = '\0';
@@ -316,7 +316,7 @@ fetch_eof(struct worker *w, struct http_conn *htc)
 
 	assert(w->body_status == BS_EOF);
 	i = w->vfp->bytes(w, htc, SSIZE_MAX);
-	if (i < 0) 
+	if (i < 0)
 		return (-1);
 	return (0);
 }
diff --git a/bin/varnishd/cache_gzip.c b/bin/varnishd/cache_gzip.c
index 1820ffc..6da1595 100644
--- a/bin/varnishd/cache_gzip.c
+++ b/bin/varnishd/cache_gzip.c
@@ -257,7 +257,7 @@ VGZ_ObufStorage(struct worker *w, struct vgz *vg)
 	struct storage *st;
 
 	st = FetchStorage(w, 0);
-	if (st == NULL) 
+	if (st == NULL)
 		return (-1);
 
 	vg->obuf = st;
@@ -487,10 +487,10 @@ vfp_gunzip_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
 			bytes -= wl;
 		}
 
-		if (VGZ_ObufStorage(w, vg)) 
+		if (VGZ_ObufStorage(w, vg))
 			return(-1);
 		i = VGZ_Gunzip(vg, &dp, &dl);
-		if (i != VGZ_OK && i != VGZ_END) 
+		if (i != VGZ_OK && i != VGZ_END)
 			return(FetchError(w, "Gunzip data error"));
 		w->fetch_obj->len += dl;
 		if (w->do_stream)
@@ -564,7 +564,7 @@ vfp_gzip_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
 			VGZ_Ibuf(vg, ibuf, wl);
 			bytes -= wl;
 		}
-		if (VGZ_ObufStorage(w, vg)) 
+		if (VGZ_ObufStorage(w, vg))
 			return(-1);
 		i = VGZ_Gzip(vg, &dp, &dl, VGZ_NORMAL);
 		assert(i == Z_OK);
@@ -661,7 +661,7 @@ vfp_testgzip_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
 		while (!VGZ_IbufEmpty(vg)) {
 			VGZ_Obuf(vg, obuf, sizeof obuf);
 			i = VGZ_Gunzip(vg, &dp, &dl);
-			if (i == VGZ_END && !VGZ_IbufEmpty(vg)) 
+			if (i == VGZ_END && !VGZ_IbufEmpty(vg))
 				return(FetchError(w, "Junk after gzip data"));
 			if (i != VGZ_OK && i != VGZ_END)
 				return(FetchError2(w,
diff --git a/bin/varnishd/cache_httpconn.c b/bin/varnishd/cache_httpconn.c
index 1b718e5..9e0a052 100644
--- a/bin/varnishd/cache_httpconn.c
+++ b/bin/varnishd/cache_httpconn.c
@@ -1,6 +1,6 @@
 /*-
  * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2009 Varnish Software AS
+ * Copyright (c) 2006-2011 Varnish Software AS
  * All rights reserved.
  *
  * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
diff --git a/bin/varnishd/cache_pipe.c b/bin/varnishd/cache_pipe.c
index 31dcf1a..be17ac9 100644
--- a/bin/varnishd/cache_pipe.c
+++ b/bin/varnishd/cache_pipe.c
@@ -1,6 +1,6 @@
 /*-
  * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2010 Varnish Software AS
+ * Copyright (c) 2006-2011 Varnish Software AS
  * All rights reserved.
  *
  * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
diff --git a/bin/varnishd/cache_pool.c b/bin/varnishd/cache_pool.c
index 141a0c7..aed596e 100644
--- a/bin/varnishd/cache_pool.c
+++ b/bin/varnishd/cache_pool.c
@@ -451,13 +451,13 @@ pool_herder(void *priv)
 		}
 
 		pool_breed(pp, &tp_attr);
-		
+
 		if (pp->nthr < params->wthread_min)
 			continue;
 
 		AZ(clock_gettime(CLOCK_MONOTONIC, &ts));
 		ts.tv_sec += params->wthread_purge_delay / 1000;
-		ts.tv_nsec += 
+		ts.tv_nsec +=
 		    (params->wthread_purge_delay % 1000) * 1000000;
 		if (ts.tv_nsec >= 1000000000) {
 			ts.tv_sec++;
@@ -470,7 +470,7 @@ pool_herder(void *priv)
 		if (!i)
 			continue;
 
-		if (pp->nthr <= params->wthread_min) 
+		if (pp->nthr <= params->wthread_min)
 			continue;
 
 		t_idle = VTIM_real() - params->wthread_timeout;
@@ -480,7 +480,7 @@ pool_herder(void *priv)
 		VSC_C_main->sess_dropped += pp->ndropped;
 		pp->nqueued = pp->ndropped = 0;
 		w = VTAILQ_LAST(&pp->idle, workerhead);
-		if (w != NULL &&  
+		if (w != NULL &&
 		    (w->lastused < t_idle || pp->nthr > params->wthread_max)) {
 			VTAILQ_REMOVE(&pp->idle, w, list);
 		} else
@@ -567,12 +567,12 @@ pool_poolherder(void *priv)
 				VSC_C_main->pools++;
 				nwq++;
 				continue;
-			} 
+			}
 		}
 		/* XXX: remove pools */
 		(void)sleep(1);
 		u = 0;
-		VTAILQ_FOREACH(pp, &pools, list) 
+		VTAILQ_FOREACH(pp, &pools, list)
 			u += pp->lqueue;
 		VSC_C_main->thread_queue_len = u;
 	}
diff --git a/bin/varnishd/mgt/mgt_sandbox_solaris.c b/bin/varnishd/mgt/mgt_sandbox_solaris.c
index f243938..becaa2e 100644
--- a/bin/varnishd/mgt/mgt_sandbox_solaris.c
+++ b/bin/varnishd/mgt/mgt_sandbox_solaris.c
@@ -48,7 +48,7 @@
 
 /*--------------------------------------------------------------------
  * SOLARIS PRIVILEGES: Note on use of symbolic PRIV_* constants
- * 
+ *
  * For privileges which existed in Solaris 10 FCS, we may use the constants from
  * sys/priv_names.h
  *
@@ -140,7 +140,7 @@ mgt_sandbox_solaris_init(void)
 		    errno, strerror(errno));
 		return;
 	}
-	
+
 	priv_emptyset(priv_all);
 
 	mgt_sandbox_solaris_add_effective(priv_all);
@@ -168,7 +168,7 @@ mgt_sandbox_solaris_privsep(void)
         }
 }
 
-/* 
+/*
  * Waive most privileges in the child
  *
  * as of onnv_151a, we should end up with:
@@ -206,7 +206,7 @@ mgt_sandbox_solaris_fini(void)
 	priv_copyset(effective, permitted);
 	mgt_sandbox_solaris_add_permitted(permitted);
 
-	/* 
+	/*
 	 * invert the sets and clear privileges such that setppriv will always
 	 * succeed
 	 */
diff --git a/bin/varnishhist/varnishhist.c b/bin/varnishhist/varnishhist.c
index 692eb81..a4bc6b7 100644
--- a/bin/varnishhist/varnishhist.c
+++ b/bin/varnishhist/varnishhist.c
@@ -1,6 +1,6 @@
 /*-
  * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2010 Varnish Software AS
+ * Copyright (c) 2006-2011 Varnish Software AS
  * All rights reserved.
  *
  * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
diff --git a/bin/varnishstat/varnishstat.c b/bin/varnishstat/varnishstat.c
index b5d6df3..d0ef409 100644
--- a/bin/varnishstat/varnishstat.c
+++ b/bin/varnishstat/varnishstat.c
@@ -311,6 +311,6 @@ main(int argc, char * const *argv)
 		printf("\n");
 
 		sleep(delay);
-	} 
+	}
 	exit(0);
 }
diff --git a/bin/varnishtest/vtc_log.c b/bin/varnishtest/vtc_log.c
index 57fa27a..890a98b 100644
--- a/bin/varnishtest/vtc_log.c
+++ b/bin/varnishtest/vtc_log.c
@@ -150,7 +150,7 @@ vtc_log(struct vtclog *vl, int lvl, const char *fmt, ...)
 	AZ(pthread_mutex_unlock(&vl->mtx));
 	if (lvl > 0)
 		return;
-	if (lvl == 0) 
+	if (lvl == 0)
 		vtc_error = 1;
 	if (pthread_self() != vtc_thread)
 		pthread_exit(NULL);
diff --git a/include/tbl/vsc_fields.h b/include/tbl/vsc_fields.h
index 904a9e2..abc5d18 100644
--- a/include/tbl/vsc_fields.h
+++ b/include/tbl/vsc_fields.h
@@ -81,48 +81,48 @@ VSC_F(sess_fail,		uint64_t, 1, 'c',
 
 /*---------------------------------------------------------------------*/
 
-VSC_F(client_req,		uint64_t, 1, 'a', 
-      "Client requests received", 
+VSC_F(client_req,		uint64_t, 1, 'a',
+      "Client requests received",
       "")
 
-VSC_F(cache_hit,		uint64_t, 1, 'a', 
-      "Cache hits", 
+VSC_F(cache_hit,		uint64_t, 1, 'a',
+      "Cache hits",
       "Count of cache hits. "
       "  A cache hit indicates that an object has been delivered to a"
       "  client without fetching it from a backend server."
 )
 
-VSC_F(cache_hitpass,	uint64_t, 1, 'a', 
-      "Cache hits for pass", 
+VSC_F(cache_hitpass,	uint64_t, 1, 'a',
+      "Cache hits for pass",
       "Count of hits for pass"
       "  A cache hit for pass indicates that Varnish is going to"
       "  pass the request to the backend and this decision has been "
       "  cached in it self. This counts how many times the cached "
       "  decision is being used."
 )
-VSC_F(cache_miss,		uint64_t, 1, 'a', 
-      "Cache misses", 
+VSC_F(cache_miss,		uint64_t, 1, 'a',
+      "Cache misses",
       "Count of misses"
       "  A cache miss indicates the object was fetched from the"
       "  backend before delivering it to the backend.")
 
-VSC_F(backend_conn,	uint64_t, 0, 'a', 
-      "Backend conn. success", 
+VSC_F(backend_conn,	uint64_t, 0, 'a',
+      "Backend conn. success",
       "")
 
-VSC_F(backend_unhealthy,	uint64_t, 0, 'a', 
-      "Backend conn. not attempted", 
+VSC_F(backend_unhealthy,	uint64_t, 0, 'a',
+      "Backend conn. not attempted",
       ""
 )
 VSC_F(backend_busy,	uint64_t, 0, 'a', "Backend conn. too many", "")
 VSC_F(backend_fail,	uint64_t, 0, 'a', "Backend conn. failures", "")
-VSC_F(backend_reuse,	uint64_t, 0, 'a', 
-      "Backend conn. reuses", 
+VSC_F(backend_reuse,	uint64_t, 0, 'a',
+      "Backend conn. reuses",
       "Count of backend connection reuses"
       "  This counter is increased whenever we reuse a recycled connection.")
 VSC_F(backend_toolate,	uint64_t, 0, 'a', "Backend conn. was closed", "")
-VSC_F(backend_recycle,	uint64_t, 0, 'a', 
-      "Backend conn. recycles", 
+VSC_F(backend_recycle,	uint64_t, 0, 'a',
+      "Backend conn. recycles",
       "Count of backend connection recycles"
       "  This counter is increased whenever we have a keep-alive"
       "  connection that is put back into the pool of connections."
@@ -255,10 +255,10 @@ VSC_F(n_lru_moved,		uint64_t, 0, 'i', "N LRU moved objects", "")
 
 VSC_F(losthdr,		uint64_t, 0, 'a', "HTTP header overflows", "")
 
-VSC_F(n_objsendfile,	uint64_t, 0, 'a', "Objects sent with sendfile", 
+VSC_F(n_objsendfile,	uint64_t, 0, 'a', "Objects sent with sendfile",
       "The number of objects sent with the sendfile system call. If enabled "
       "sendfile will be used on object larger than a certain size.")
-VSC_F(n_objwrite,		uint64_t, 0, 'a', "Objects sent with write", 
+VSC_F(n_objwrite,		uint64_t, 0, 'a', "Objects sent with write",
       "The number of objects sent with regular write calls."
       "Writes are used when the objects are too small for sendfile "
       "or if the sendfile call has been disabled")
diff --git a/lib/libvarnish/vre.c b/lib/libvarnish/vre.c
index cffe396..eac8121 100644
--- a/lib/libvarnish/vre.c
+++ b/lib/libvarnish/vre.c
@@ -1,5 +1,5 @@
 /*-
- * Copyright (c) 2006-2009 Varnish Software AS
+ * Copyright (c) 2006-2011 Varnish Software AS
  * All rights reserved.
  *
  * Author: Tollef Fog Heen <tfheen at redpill-linpro.com>
diff --git a/lib/libvarnish/vtcp.c b/lib/libvarnish/vtcp.c
index c0f7a95..21de7f2 100644
--- a/lib/libvarnish/vtcp.c
+++ b/lib/libvarnish/vtcp.c
@@ -265,7 +265,7 @@ VTCP_set_read_timeout(int s, double seconds)
 	timeout.tv_sec = (int)floor(seconds);
 	timeout.tv_usec = (int)(1e6 * (seconds - timeout.tv_sec));
 #ifdef SO_RCVTIMEO_WORKS
-	/* 
+	/*
 	 * Solaris bug (present at least in snv_151 and older): If this fails
 	 * with EINVAL, the socket is half-closed (SS_CANTSENDMORE) and the
 	 * timeout does not get set. Needs to be fixed in Solaris, there is



From phk at varnish-cache.org  Wed Nov  9 10:59:44 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 09 Nov 2011 11:59:44 +0100
Subject: [master] a2dbc59 Renovate ban related statistics,
	add long descriptions etc.
Message-ID: <E1RO5se-0001o6-S8@project.varnish-software.com>

commit a2dbc597f6bb7a9bc16345432cdd753caef16986
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Nov 9 10:59:26 2011 +0000

    Renovate ban related statistics, add long descriptions etc.

diff --git a/bin/varnishd/cache_ban.c b/bin/varnishd/cache_ban.c
index 648997c..ac45144 100644
--- a/bin/varnishd/cache_ban.c
+++ b/bin/varnishd/cache_ban.c
@@ -376,7 +376,6 @@ void
 BAN_Insert(struct ban *b)
 {
 	struct ban  *bi, *be;
-	unsigned pcount;
 	ssize_t ln;
 	double t0;
 
@@ -402,8 +401,10 @@ BAN_Insert(struct ban *b)
 	Lck_Lock(&ban_mtx);
 	VTAILQ_INSERT_HEAD(&ban_head, b, list);
 	ban_start = b;
-	VSC_C_main->n_ban++;
-	VSC_C_main->n_ban_add++;
+	VSC_C_main->bans++;
+	VSC_C_main->bans_added++;
+	if (b->flags & BAN_F_REQ)
+		VSC_C_main->bans_req++;
 
 	be = VTAILQ_LAST(&ban_head, banhead_s);
 	if (params->ban_dups && be != b)
@@ -419,7 +420,6 @@ BAN_Insert(struct ban *b)
 
 	/* Hunt down duplicates, and mark them as gone */
 	bi = b;
-	pcount = 0;
 	Lck_Lock(&ban_mtx);
 	while(bi != be) {
 		bi = VTAILQ_NEXT(bi, list);
@@ -429,11 +429,10 @@ BAN_Insert(struct ban *b)
 		if (memcmp(b->spec + 8, bi->spec + 8, ln - 8))
 			continue;
 		bi->flags |= BAN_F_GONE;
-		VSC_C_main->n_ban_gone++;
-		pcount++;
+		VSC_C_main->bans_gone++;
+		VSC_C_main->bans_dups++;
 	}
 	be->refcount--;
-	VSC_C_main->n_ban_dups += pcount;
 	Lck_Unlock(&ban_mtx);
 }
 
@@ -520,20 +519,28 @@ BAN_Reload(const uint8_t *ban, unsigned len)
 
 	t0 = ban_time(ban);
 	assert(len == ban_len(ban));
+
+	Lck_Lock(&ban_mtx);
+
 	VTAILQ_FOREACH(b, &ban_head, list) {
 		t1 = ban_time(b->spec);
 		assert (t1 < t2);
 		t2 = t1;
-		if (t1 == t0)
+		if (t1 == t0) {
+			Lck_Unlock(&ban_mtx);
 			return;
+		}
 		if (t1 < t0)
 			break;
-		if (!memcmp(b->spec + 8, ban + 8, len - 8))
+		if (!memcmp(b->spec + 8, ban + 8, len - 8)) {
 			gone |= BAN_F_GONE;
+			VSC_C_main->bans_dups++;
+			VSC_C_main->bans_gone++;
+		}
 	}
 
-	VSC_C_main->n_ban++;
-	VSC_C_main->n_ban_add++;
+	VSC_C_main->bans++;
+	VSC_C_main->bans_added++;
 
 	b2 = BAN_New();
 	AN(b2);
@@ -552,9 +559,13 @@ BAN_Reload(const uint8_t *ban, unsigned len)
 	for (b = VTAILQ_NEXT(b2, list); b != NULL; b = VTAILQ_NEXT(b, list)) {
 		if (b->flags & BAN_F_GONE)
 			continue;
-		if (!memcmp(b->spec + 8, ban + 8, len - 8))
+		if (!memcmp(b->spec + 8, ban + 8, len - 8)) {
 			b->flags |= BAN_F_GONE;
+			VSC_C_main->bans_dups++;
+			VSC_C_main->bans_gone++;
+		}
 	}
+	Lck_Unlock(&ban_mtx);
 }
 
 /*--------------------------------------------------------------------
@@ -705,8 +716,8 @@ ban_check_object(struct object *o, const struct sess *sp, int has_req)
 	}
 
 	Lck_Lock(&ban_mtx);
-	VSC_C_main->n_ban_obj_test++;
-	VSC_C_main->n_ban_re_test += tests;
+	VSC_C_main->bans_tested++;
+	VSC_C_main->bans_tests_tested += tests;
 
 	if (b == oc->ban && skipped > 0) {
 		AZ(has_req);
@@ -758,9 +769,11 @@ ban_CheckLast(void)
 	b = VTAILQ_LAST(&ban_head, banhead_s);
 	if (b != VTAILQ_FIRST(&ban_head) && b->refcount == 0) {
 		if (b->flags & BAN_F_GONE)
-			VSC_C_main->n_ban_gone--;
-		VSC_C_main->n_ban--;
-		VSC_C_main->n_ban_retire++;
+			VSC_C_main->bans_gone--;
+		if (b->flags & BAN_F_REQ)
+			VSC_C_main->bans_req--;
+		VSC_C_main->bans--;
+		VSC_C_main->bans_deleted++;
 		VTAILQ_REMOVE(&ban_head, b, list);
 	} else {
 		b = NULL;
@@ -888,7 +901,7 @@ ban_lurker_work(const struct sess *sp, unsigned pass)
 		if (!(b->flags & BAN_F_REQ)) {
 			if (!(b->flags & BAN_F_GONE)) {
 				b->flags |= BAN_F_GONE;
-				VSC_C_main->n_ban_gone++;
+				VSC_C_main->bans_gone++;
 			}
 			if (params->diag_bitmap & 0x80000)
 				VSL(SLT_Debug, 0, "lurker BAN %f now gone",
@@ -1090,6 +1103,6 @@ BAN_Init(void)
 	ban_magic = BAN_New();
 	AN(ban_magic);
 	ban_magic->flags |= BAN_F_GONE;
-	VSC_C_main->n_ban_gone++;
+	VSC_C_main->bans_gone++;
 	BAN_Insert(ban_magic);
 }
diff --git a/bin/varnishtest/tests/c00019.vtc b/bin/varnishtest/tests/c00019.vtc
index a27bb8e..5cc5973 100644
--- a/bin/varnishtest/tests/c00019.vtc
+++ b/bin/varnishtest/tests/c00019.vtc
@@ -16,7 +16,7 @@ varnish v1 -vcl+backend {} -start
 varnish v1 -cliok "ban.url FOO"
 
 # There is one "magic" ban from boot
-varnish v1 -expect n_ban_add == 2
+varnish v1 -expect bans_added == 2
 varnish v1 -cliok "ban.list"
 
 # Our fetch is not affected by the ban
@@ -31,12 +31,12 @@ client c1 {
 } -run
 
 varnish v1 -cliok "ban.list"
-varnish v1 -expect n_ban_obj_test == 0
-varnish v1 -expect n_ban_re_test == 0
+varnish v1 -expect bans_tested == 0
+varnish v1 -expect bans_tests_tested == 0
 
 # Add another ban
 varnish v1 -cliok "ban.url FOO"
-varnish v1 -expect n_ban_add == 3
+varnish v1 -expect bans_added == 3
 varnish v1 -cliok "ban.list"
 
 # The cached object will be band, and a new
@@ -47,8 +47,8 @@ client c1 {
 	expect resp.http.foo == 2
 } -run
 
-varnish v1 -expect n_ban_obj_test == 1
-varnish v1 -expect n_ban_re_test == 1
+varnish v1 -expect bans_tested == 1
+varnish v1 -expect bans_tests_tested == 1
 varnish v1 -cliok "ban.list"
 
 # Fetch the cached copy, just for grins
@@ -62,15 +62,15 @@ client c1 {
 # Now add another two bans, Kilroy should not be hit
 varnish v1 -cliok "ban.url KILROY"
 varnish v1 -cliok "ban.url FOO"
-varnish v1 -expect n_ban_add == 5
+varnish v1 -expect bans_added == 5
 
 # Enable dup removal of bans
 varnish v1 -cliok "param.set ban_dups on"
 
 # This should incapacitate the two previous FOO bans.
 varnish v1 -cliok "ban.url FOO"
-varnish v1 -expect n_ban_add == 6
-varnish v1 -expect n_ban_dups == 3
+varnish v1 -expect bans_added == 6
+varnish v1 -expect bans_dups == 3
 varnish v1 -cliok "ban.list"
 
 # And we should get a fresh object from backend
@@ -81,8 +81,8 @@ client c1 {
 } -run
 
 # With only two objects having ever been compared
-varnish v1 -expect n_ban_obj_test == 2
-varnish v1 -expect n_ban_re_test == 2
+varnish v1 -expect bans_tested == 2
+varnish v1 -expect bans_tests_tested == 2
 varnish v1 -cliok "ban.list"
 
 # Test a bogus regexp
diff --git a/bin/varnishtest/tests/c00049.vtc b/bin/varnishtest/tests/c00049.vtc
index facd979..59e6c14 100644
--- a/bin/varnishtest/tests/c00049.vtc
+++ b/bin/varnishtest/tests/c00049.vtc
@@ -49,8 +49,8 @@ client c1 {
 delay 0.1
 varnish v1 -cliok "ban req.url == /alpha"
 varnish v1 -cliok "ban.list"
-varnish v1 -expect n_ban == 2
-varnish v1 -expect n_ban_gone == 1
+varnish v1 -expect bans == 2
+varnish v1 -expect bans_gone == 1
 
 client c1 {
 	txreq -url "/beta"
@@ -61,7 +61,7 @@ client c1 {
 delay 0.1
 varnish v1 -cliok "ban obj.http.foo == /beta"
 varnish v1 -cliok "ban.list"
-varnish v1 -expect n_ban == 3
+varnish v1 -expect bans == 3
 
 client c1 {
 	txreq -url "/gamma"
@@ -72,7 +72,7 @@ client c1 {
 delay 0.1
 varnish v1 -cliok "ban obj.http.foo == /gamma"
 varnish v1 -cliok "ban.list"
-varnish v1 -expect n_ban == 4
+varnish v1 -expect bans == 4
 
 client c1 {
 	txreq -url "/delta"
@@ -83,22 +83,22 @@ client c1 {
 delay 0.1
 varnish v1 -cliok "ban req.url == /delta"
 
-varnish v1 -expect n_ban_gone == 1
+varnish v1 -expect bans_gone == 1
 varnish v1 -cliok "ban obj.http.foo == /gamma"
 # Dup-check should have added one
-varnish v1 -expect n_ban_gone == 2
+varnish v1 -expect bans_gone == 2
 
 varnish v1 -cliok "ban req.url == /epsilon"
 varnish v1 -cliok "ban.list"
-varnish v1 -expect n_ban == 7
-varnish v1 -expect n_ban_gone == 2
+varnish v1 -expect bans == 7
+varnish v1 -expect bans_gone == 2
 
 varnish v1 -cliok "param.set ban_lurker_sleep .01"
 delay 1
 varnish v1 -cliok "param.set ban_lurker_sleep .00"
 varnish v1 -cliok "ban.list"
-varnish v1 -expect n_ban == 7
-varnish v1 -expect n_ban_gone == 4
+varnish v1 -expect bans == 7
+varnish v1 -expect bans_gone == 4
 
 client c1 {
 	txreq -url "/alpha"
@@ -108,7 +108,7 @@ client c1 {
 
 delay 1
 varnish v1 -cliok "ban.list"
-varnish v1 -expect n_ban == 4
+varnish v1 -expect bans == 4
 
 client c1 {
 	txreq -url "/beta"
@@ -127,5 +127,7 @@ client c1 {
 
 delay 1
 varnish v1 -cliok "ban.list"
-varnish v1 -expect n_ban == 1
-varnish v1 -expect n_ban_gone == 0
+varnish v1 -expect bans == 1
+varnish v1 -expect bans_gone == 0
+varnish v1 -expect bans_added == 7
+varnish v1 -expect bans_deleted == 6
diff --git a/bin/varnishtest/tests/p00002.vtc b/bin/varnishtest/tests/p00002.vtc
index cef8f36..e3ea052 100644
--- a/bin/varnishtest/tests/p00002.vtc
+++ b/bin/varnishtest/tests/p00002.vtc
@@ -36,4 +36,4 @@ varnish v1 -start
 varnish v1 -cliok ban.list
 
 # Count of 3 here, because two "magic" bans are also there"
-varnish v1 -expect n_ban == 3
+varnish v1 -expect bans == 3
diff --git a/bin/varnishtest/tests/r00917.vtc b/bin/varnishtest/tests/r00917.vtc
index 6591079..9791034 100644
--- a/bin/varnishtest/tests/r00917.vtc
+++ b/bin/varnishtest/tests/r00917.vtc
@@ -21,4 +21,4 @@ foo
 
 varnish v1 -cliok ban.list
 
-varnish v1 -expect n_ban_add == 2
+varnish v1 -expect bans_added == 2
diff --git a/bin/varnishtest/tests/r01030.vtc b/bin/varnishtest/tests/r01030.vtc
index 97ef3d7..8351d81 100644
--- a/bin/varnishtest/tests/r01030.vtc
+++ b/bin/varnishtest/tests/r01030.vtc
@@ -26,7 +26,7 @@ varnish v1 -vcl+backend {
 } -start
 
 varnish v1 -cliok "param.set ban_lurker_sleep 0.01"
-varnish v1 -expect n_ban_obj_test == 0
+varnish v1 -expect bans_tests_tested == 0
 
 delay 0.01
 client c1 {
@@ -40,10 +40,10 @@ client c1 {
 } -run
 
 delay 0.1
-varnish v1 -expect n_ban_obj_test == 0
+varnish v1 -expect bans_tests_tested == 0
 
 delay 1.0
-varnish v1 -expect n_ban_obj_test == 1
+varnish v1 -expect bans_tests_tested == 1
 
 varnish v1 -cliok "param.set ban_lurker_sleep 5.01"
 
@@ -58,7 +58,7 @@ client c2 {
 } -run
 
 delay 0.1
-varnish v1 -expect n_ban_obj_test == 1
+varnish v1 -expect bans_tests_tested == 1
 
 delay 1.1
-varnish v1 -expect n_ban_obj_test == 2
+varnish v1 -expect bans_tests_tested == 2
diff --git a/include/tbl/vsc_fields.h b/include/tbl/vsc_fields.h
index abc5d18..738703c 100644
--- a/include/tbl/vsc_fields.h
+++ b/include/tbl/vsc_fields.h
@@ -41,6 +41,10 @@
  *    e - Explantion:	Short explanation of field (for screen use)
  *    d - Description:	Long explanation of field (for doc use)
  *
+ * Please describe Gauge variables as "Number of..." to indicate that
+ * this is a snapshot, and Counter variables as "Count of" to indicate
+ * accumulative count.
+ *
  * -----------------------
  * NB: Cleanup in progress
  * -----------------------
@@ -297,13 +301,49 @@ VSC_F(n_vcl,		uint64_t, 0, 'a', "N vcl total", "")
 VSC_F(n_vcl_avail,		uint64_t, 0, 'a', "N vcl available", "")
 VSC_F(n_vcl_discard,	uint64_t, 0, 'a', "N vcl discarded", "")
 
-VSC_F(n_ban,			uint64_t, 0, 'i', "N total active bans", "")
-VSC_F(n_ban_gone,		uint64_t, 0, 'i', "N total gone bans", "")
-VSC_F(n_ban_add,		uint64_t, 0, 'a', "N new bans added", "")
-VSC_F(n_ban_retire,		uint64_t, 0, 'a', "N old bans deleted", "")
-VSC_F(n_ban_obj_test,		uint64_t, 0, 'a', "N objects tested", "")
-VSC_F(n_ban_re_test,		uint64_t, 0, 'a', "N regexps tested against", "")
-VSC_F(n_ban_dups,		uint64_t, 0, 'a', "N duplicate bans removed", "")
+/**********************************************************************/
+
+VSC_F(bans,			uint64_t, 0, 'g',
+   "Count of bans",
+	"Number of all bans in system, including bans superseded"
+	" by newer bans and bans already checked by the ban-lurker."
+)
+VSC_F(bans_gone,		uint64_t, 0, 'g',
+    "Number of bans marked 'gone'",
+	"Number of bans which are no longer active, either because they"
+	" got checked by the ban-lurker or superseded by newer identical bans."
+)
+VSC_F(bans_req,			uint64_t, 0, 'g',
+    "Number of bans using req.*",
+	"Number of bans which use req.* variables.  These bans can not"
+	" be washed by the ban-lurker."
+)
+VSC_F(bans_added,		uint64_t, 0, 'c',
+    "Bans added",
+	"Counter of bans added to ban list."
+)
+VSC_F(bans_deleted,		uint64_t, 0, 'c',
+    "Bans deleted",
+	"Counter of bans deleted from ban list."
+)
+
+VSC_F(bans_tested,		uint64_t, 0, 'c',
+    "Bans tested against objects",
+	"Count of how many bans and objects have been tested against"
+	" each other."
+)
+VSC_F(bans_tests_tested,	uint64_t, 0, 'c',
+    "Ban tests tested against objects",
+	"Count of how many tests and objects have been tested against"
+	" each other.  'ban req.url == foo && req.http.host == bar'"
+	" counts as one in 'bans_tested' and as two in 'bans_tests_tested'"
+)
+VSC_F(bans_dups,		uint64_t, 0, 'c',
+    "Bans superseded by other bans",
+	"Count of bans replaced by later identical bans."
+)
+
+/**********************************************************************/
 
 VSC_F(hcb_nolock,		uint64_t, 0, 'a', "HCB Lookups without lock", "")
 VSC_F(hcb_lock,		uint64_t, 0, 'a', "HCB Lookups with lock", "")



From phk at varnish-cache.org  Thu Nov 10 10:08:41 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Thu, 10 Nov 2011 11:08:41 +0100
Subject: [master] 8241149 Draw the line between the managers and the childs
	copy of the struct params much clearer. Manager should never
	read the childs copy.
Message-ID: <E1RORYn-0006Or-Er@project.varnish-software.com>

commit 8241149492e5649c8a3126ee363f089f33491655
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Nov 10 08:57:41 2011 +0000

    Draw the line between the managers and the childs copy of the struct
    params much clearer.  Manager should never read the childs copy.

diff --git a/bin/varnishd/cache_main.c b/bin/varnishd/cache_main.c
index 8293d38..c610bc3 100644
--- a/bin/varnishd/cache_main.c
+++ b/bin/varnishd/cache_main.c
@@ -37,6 +37,8 @@
 #include "waiter/cache_waiter.h"
 #include "hash/hash_slinger.h"
 
+volatile struct params	*params;
+
 /*--------------------------------------------------------------------
  * Per thread storage for the session currently being processed by
  * the thread.  This is used for panic messages.
diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h
index 1656af8..61493b3 100644
--- a/bin/varnishd/mgt/mgt.h
+++ b/bin/varnishd/mgt/mgt.h
@@ -58,10 +58,11 @@ void mgt_cli_secret(const char *S_arg);
 void mgt_cli_close_all(void);
 
 /* mgt_param.c */
-void MCF_ParamSync(void);
 void MCF_ParamInit(struct cli *);
 void MCF_ParamSet(struct cli *, const char *param, const char *val);
 void MCF_DumpRst(void);
+extern struct params mgt_param;
+
 
 /* mgt_sandbox.c */
 void mgt_sandbox(void);
diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c
index ff8a776..9c6d37d 100644
--- a/bin/varnishd/mgt/mgt_child.c
+++ b/bin/varnishd/mgt/mgt_child.c
@@ -214,7 +214,7 @@ MGT_Child_Cli_Fail(void)
 		return;
 	REPORT(LOG_ERR, "Child (%jd) not responding to CLI, killing it.",
 	    (intmax_t)child_pid);
-	if (params->diag_bitmap & 0x1000)
+	if (mgt_param.diag_bitmap & 0x1000)
 		(void)kill(child_pid, SIGKILL);
 	else
 		(void)kill(child_pid, SIGQUIT);
@@ -315,7 +315,7 @@ start_child(struct cli *cli)
 	heritage.std_fd = cp[1];
 	child_output = cp[0];
 
-	MCF_ParamSync();
+	AN(params);
 	if ((pid = fork()) < 0) {
 		perror("Could not fork child");
 		exit(1);
@@ -374,10 +374,10 @@ start_child(struct cli *cli)
 	AZ(vev_add(mgt_evb, e));
 	ev_listen = e;
 	AZ(ev_poker);
-	if (params->ping_interval > 0) {
+	if (mgt_param.ping_interval > 0) {
 		e = vev_new();
 		XXXAN(e);
-		e->timeout = params->ping_interval;
+		e->timeout = mgt_param.ping_interval;
 		e->callback = child_poker;
 		e->name = "child poker";
 		AZ(vev_add(mgt_evb, e));
@@ -522,7 +522,7 @@ mgt_sigchld(const struct vev *e, int what)
 
 	REPORT0(LOG_DEBUG, "Child cleanup complete");
 
-	if (child_state == CH_DIED && params->auto_restart)
+	if (child_state == CH_DIED && mgt_param.auto_restart)
 		start_child(NULL);
 	else if (child_state == CH_DIED) {
 		child_state = CH_STOPPED;
diff --git a/bin/varnishd/mgt/mgt_cli.c b/bin/varnishd/mgt/mgt_cli.c
index c8f22c3..ec8fd4d 100644
--- a/bin/varnishd/mgt/mgt_cli.c
+++ b/bin/varnishd/mgt/mgt_cli.c
@@ -170,7 +170,7 @@ mcf_askchild(struct cli *cli, const char * const *av, void *priv)
 		return;
 	}
 	VSB_delete(vsb);
-	(void)VCLI_ReadResult(cli_i, &u, &q, params->cli_timeout);
+	(void)VCLI_ReadResult(cli_i, &u, &q, mgt_param.cli_timeout);
 	VCLI_SetResult(cli, u);
 	VCLI_Out(cli, "%s", q);
 	free(q);
@@ -192,7 +192,7 @@ mgt_cli_askchild(unsigned *status, char **resp, const char *fmt, ...) {
 	int i, j;
 	va_list ap;
 	unsigned u;
-	char buf[params->cli_buffer], *p;
+	char buf[mgt_param.cli_buffer], *p;
 
 	if (resp != NULL)
 		*resp = NULL;
@@ -219,7 +219,7 @@ mgt_cli_askchild(unsigned *status, char **resp, const char *fmt, ...) {
 		return (CLIS_COMMS);
 	}
 
-	(void)VCLI_ReadResult(cli_i, &u, resp, params->cli_timeout);
+	(void)VCLI_ReadResult(cli_i, &u, resp, mgt_param.cli_timeout);
 	if (status != NULL)
 		*status = u;
 	if (u == CLIS_COMMS)
@@ -316,7 +316,7 @@ static void
 mgt_cli_cb_before(const struct cli *cli)
 {
 
-	if (params->syslog_cli_traffic)
+	if (mgt_param.syslog_cli_traffic)
 		syslog(LOG_NOTICE, "CLI %s Rd %s", cli->ident, cli->cmd);
 }
 
@@ -324,7 +324,7 @@ static void
 mgt_cli_cb_after(const struct cli *cli)
 {
 
-	if (params->syslog_cli_traffic)
+	if (mgt_param.syslog_cli_traffic)
 		syslog(LOG_NOTICE, "CLI %s Wr %03u %s",
 		    cli->ident, cli->result, VSB_data(cli->sb));
 }
@@ -335,7 +335,7 @@ static void
 mgt_cli_init_cls(void)
 {
 
-	cls = VCLS_New(mgt_cli_cb_before, mgt_cli_cb_after, params->cli_buffer);
+	cls = VCLS_New(mgt_cli_cb_before, mgt_cli_cb_after, mgt_param.cli_buffer);
 	AN(cls);
 	AZ(VCLS_AddFunc(cls, MCF_NOAUTH, cli_auth));
 	AZ(VCLS_AddFunc(cls, MCF_AUTH, cli_proto));
diff --git a/bin/varnishd/mgt/mgt_param.c b/bin/varnishd/mgt/mgt_param.c
index e2eb16b..20456d7 100644
--- a/bin/varnishd/mgt/mgt_param.c
+++ b/bin/varnishd/mgt/mgt_param.c
@@ -51,7 +51,7 @@
 #include "mgt_cli.h"
 
 #define MAGIC_INIT_STRING	"\001"
-struct params master;
+struct params mgt_param;
 static int nparspec;
 static struct parspec const ** parspec;
 static int margin;
@@ -262,7 +262,7 @@ tweak_user(struct cli *cli, const struct parspec *par, const char *arg)
 		if (!strcmp(arg, MAGIC_INIT_STRING)) {
 			pw = getpwnam("nobody");
 			if (pw == NULL) {
-				master.uid = getuid();
+				mgt_param.uid = getuid();
 				return;
 			}
 		} else
@@ -272,19 +272,19 @@ tweak_user(struct cli *cli, const struct parspec *par, const char *arg)
 			VCLI_SetResult(cli, CLIS_PARAM);
 			return;
 		}
-		REPLACE(master.user, pw->pw_name);
-		master.uid = pw->pw_uid;
-		master.gid = pw->pw_gid;
+		REPLACE(mgt_param.user, pw->pw_name);
+		mgt_param.uid = pw->pw_uid;
+		mgt_param.gid = pw->pw_gid;
 
 		/* set group to user's primary group */
 		if ((gr = getgrgid(pw->pw_gid)) != NULL &&
 		    (gr = getgrnam(gr->gr_name)) != NULL &&
 		    gr->gr_gid == pw->pw_gid)
-			REPLACE(master.group, gr->gr_name);
-	} else if (master.user) {
-		VCLI_Out(cli, "%s (%d)", master.user, (int)master.uid);
+			REPLACE(mgt_param.group, gr->gr_name);
+	} else if (mgt_param.user) {
+		VCLI_Out(cli, "%s (%d)", mgt_param.user, (int)mgt_param.uid);
 	} else {
-		VCLI_Out(cli, "%d", (int)master.uid);
+		VCLI_Out(cli, "%d", (int)mgt_param.uid);
 	}
 }
 
@@ -303,8 +303,8 @@ tweak_group(struct cli *cli, const struct parspec *par, const char *arg)
 			gr = getgrnam("nogroup");
 			if (gr == NULL) {
 				/* Only replace if tweak_user didn't */
-				if (master.gid == 0)
-					master.gid = getgid();
+				if (mgt_param.gid == 0)
+					mgt_param.gid = getgid();
 				return;
 			}
 		} else
@@ -314,12 +314,12 @@ tweak_group(struct cli *cli, const struct parspec *par, const char *arg)
 			VCLI_SetResult(cli, CLIS_PARAM);
 			return;
 		}
-		REPLACE(master.group, gr->gr_name);
-		master.gid = gr->gr_gid;
-	} else if (master.group) {
-		VCLI_Out(cli, "%s (%d)", master.group, (int)master.gid);
+		REPLACE(mgt_param.group, gr->gr_name);
+		mgt_param.gid = gr->gr_gid;
+	} else if (mgt_param.group) {
+		VCLI_Out(cli, "%s (%d)", mgt_param.group, (int)mgt_param.gid);
 	} else {
-		VCLI_Out(cli, "%d", (int)master.gid);
+		VCLI_Out(cli, "%d", (int)mgt_param.gid);
 	}
 }
 
@@ -350,7 +350,7 @@ tweak_listen_address(struct cli *cli, const struct parspec *par,
 
 	(void)par;
 	if (arg == NULL) {
-		VCLI_Quote(cli, master.listen_address);
+		VCLI_Quote(cli, mgt_param.listen_address);
 		return;
 	}
 
@@ -401,7 +401,7 @@ tweak_listen_address(struct cli *cli, const struct parspec *par,
 		return;
 	}
 
-	REPLACE(master.listen_address, arg);
+	REPLACE(mgt_param.listen_address, arg);
 
 	clean_listen_sock_head(&heritage.socks);
 	heritage.nsocks = 0;
@@ -452,9 +452,9 @@ tweak_diag_bitmap(struct cli *cli, const struct parspec *par, const char *arg)
 	(void)par;
 	if (arg != NULL) {
 		u = strtoul(arg, NULL, 0);
-		master.diag_bitmap = u;
+		mgt_param.diag_bitmap = u;
 	} else {
-		VCLI_Out(cli, "0x%x", master.diag_bitmap);
+		VCLI_Out(cli, "0x%x", mgt_param.diag_bitmap);
 	}
 }
 
@@ -500,7 +500,7 @@ static const struct parspec input_parspec[] = {
 		"The unprivileged group to run as.",
 		MUST_RESTART,
 		MAGIC_INIT_STRING },
-	{ "default_ttl", tweak_timeout_double, &master.default_ttl,
+	{ "default_ttl", tweak_timeout_double, &mgt_param.default_ttl,
 		0, UINT_MAX,
 		"The TTL assigned to objects if neither the backend nor "
 		"the VCL code assigns one.\n"
@@ -510,7 +510,7 @@ static const struct parspec input_parspec[] = {
 		"flush of the cache use \"ban.url .\"",
 		0,
 		"120", "seconds" },
-	{ "sess_workspace", tweak_uint, &master.sess_workspace, 1024, UINT_MAX,
+	{ "sess_workspace", tweak_uint, &mgt_param.sess_workspace, 1024, UINT_MAX,
 		"Bytes of HTTP protocol workspace allocated for sessions. "
 		"This space must be big enough for the entire HTTP protocol "
 		"header and any edits done to it in the VCL code.\n"
@@ -518,13 +518,13 @@ static const struct parspec input_parspec[] = {
 		DELAYED_EFFECT,
 		"65536",
 		"bytes" },
-	{ "http_req_hdr_len", tweak_uint, &master.http_req_hdr_len,
+	{ "http_req_hdr_len", tweak_uint, &mgt_param.http_req_hdr_len,
 		40, UINT_MAX,
 		"Maximum length of any HTTP client request header we will "
 		"allow.  The limit is inclusive its continuation lines.\n",
 		0,
 		"8192", "bytes" },
-	{ "http_req_size", tweak_uint, &master.http_req_size,
+	{ "http_req_size", tweak_uint, &mgt_param.http_req_size,
 		256, UINT_MAX,
 		"Maximum number of bytes of HTTP client request we will deal "
 		"with.  This is a limit on all bytes up to the double blank "
@@ -534,13 +534,13 @@ static const struct parspec input_parspec[] = {
 		"how much of that the request is allowed to take up.",
 		0,
 		"32768", "bytes" },
-	{ "http_resp_hdr_len", tweak_uint, &master.http_resp_hdr_len,
+	{ "http_resp_hdr_len", tweak_uint, &mgt_param.http_resp_hdr_len,
 		40, UINT_MAX,
 		"Maximum length of any HTTP backend response header we will "
 		"allow.  The limit is inclusive its continuation lines.\n",
 		0,
 		"8192", "bytes" },
-	{ "http_resp_size", tweak_uint, &master.http_resp_size,
+	{ "http_resp_size", tweak_uint, &mgt_param.http_resp_size,
 		256, UINT_MAX,
 		"Maximum number of bytes of HTTP backend resonse we will deal "
 		"with.  This is a limit on all bytes up to the double blank "
@@ -550,7 +550,7 @@ static const struct parspec input_parspec[] = {
 		"how much of that the request is allowed to take up.",
 		0,
 		"32768", "bytes" },
-	{ "http_max_hdr", tweak_uint, &master.http_max_hdr, 32, 65535,
+	{ "http_max_hdr", tweak_uint, &mgt_param.http_max_hdr, 32, 65535,
 		"Maximum number of HTTP headers we will deal with in "
 		"client request or backend reponses.  "
 		"Note that the first line occupies five header fields.\n"
@@ -558,7 +558,7 @@ static const struct parspec input_parspec[] = {
 		"objects allocate exact space for the headers they store.\n",
 		0,
 		"64", "header lines" },
-	{ "shm_workspace", tweak_uint, &master.shm_workspace, 4096, UINT_MAX,
+	{ "shm_workspace", tweak_uint, &mgt_param.shm_workspace, 4096, UINT_MAX,
 		"Bytes of shmlog workspace allocated for worker threads. "
 		"If too big, it wastes some ram, if too small it causes "
 		"needless flushes of the SHM workspace.\n"
@@ -567,12 +567,12 @@ static const struct parspec input_parspec[] = {
 		"Minimum is 4096 bytes.",
 		DELAYED_EFFECT,
 		"8192", "bytes" },
-	{ "shm_reclen", tweak_uint, &master.shm_reclen, 16, 65535,
+	{ "shm_reclen", tweak_uint, &mgt_param.shm_reclen, 16, 65535,
 		"Maximum number of bytes in SHM log record.\n"
 		"Maximum is 65535 bytes.",
 		0,
 		"255", "bytes" },
-	{ "default_grace", tweak_timeout_double, &master.default_grace,
+	{ "default_grace", tweak_timeout_double, &mgt_param.default_grace,
 		0, UINT_MAX,
 		"Default grace period.  We will deliver an object "
 		"this long after it has expired, provided another thread "
@@ -581,7 +581,7 @@ static const struct parspec input_parspec[] = {
 		"made until they are fetched from the backend again.\n",
 		DELAYED_EFFECT,
 		"10", "seconds" },
-	{ "default_keep", tweak_timeout_double, &master.default_keep,
+	{ "default_keep", tweak_timeout_double, &mgt_param.default_keep,
 		0, UINT_MAX,
 		"Default keep period.  We will keep a useless object "
 		"around this long, making it available for conditional "
@@ -590,49 +590,49 @@ static const struct parspec input_parspec[] = {
 		"cache at the end of ttl+grace+keep.",
 		DELAYED_EFFECT,
 		"0", "seconds" },
-	{ "sess_timeout", tweak_timeout, &master.sess_timeout, 0, 0,
+	{ "sess_timeout", tweak_timeout, &mgt_param.sess_timeout, 0, 0,
 		"Idle timeout for persistent sessions. "
 		"If a HTTP request has not been received in this many "
 		"seconds, the session is closed.",
 		0,
 		"5", "seconds" },
-	{ "expiry_sleep", tweak_timeout_double, &master.expiry_sleep, 0, 60,
+	{ "expiry_sleep", tweak_timeout_double, &mgt_param.expiry_sleep, 0, 60,
 		"How long the expiry thread sleeps when there is nothing "
 		"for it to do.\n",
 		0,
 		"1", "seconds" },
-	{ "pipe_timeout", tweak_timeout, &master.pipe_timeout, 0, 0,
+	{ "pipe_timeout", tweak_timeout, &mgt_param.pipe_timeout, 0, 0,
 		"Idle timeout for PIPE sessions. "
 		"If nothing have been received in either direction for "
 		"this many seconds, the session is closed.\n",
 		0,
 		"60", "seconds" },
-	{ "send_timeout", tweak_timeout, &master.send_timeout, 0, 0,
+	{ "send_timeout", tweak_timeout, &mgt_param.send_timeout, 0, 0,
 		"Send timeout for client connections. "
 		"If the HTTP response hasn't been transmitted in this many\n"
                 "seconds the session is closed. \n"
 		"See setsockopt(2) under SO_SNDTIMEO for more information.",
 		DELAYED_EFFECT,
 		"600", "seconds" },
-	{ "idle_send_timeout", tweak_timeout, &master.idle_send_timeout, 0, 0,
+	{ "idle_send_timeout", tweak_timeout, &mgt_param.idle_send_timeout, 0, 0,
 		"Time to wait with no data sent. "
 		"If no data has been transmitted in this many\n"
                 "seconds the session is closed. \n"
 		"See setsockopt(2) under SO_SNDTIMEO for more information.",
 		DELAYED_EFFECT,
 		"60", "seconds" },
-	{ "auto_restart", tweak_bool, &master.auto_restart, 0, 0,
+	{ "auto_restart", tweak_bool, &mgt_param.auto_restart, 0, 0,
 		"Restart child process automatically if it dies.\n",
 		0,
 		"on", "bool" },
 	{ "nuke_limit",
-		tweak_uint, &master.nuke_limit, 0, UINT_MAX,
+		tweak_uint, &mgt_param.nuke_limit, 0, UINT_MAX,
 		"Maximum number of objects we attempt to nuke in order"
 		"to make space for a object body.",
 		EXPERIMENTAL,
 		"50", "allocations" },
 	{ "fetch_chunksize",
-		tweak_uint, &master.fetch_chunksize, 4, UINT_MAX / 1024.,
+		tweak_uint, &mgt_param.fetch_chunksize, 4, UINT_MAX / 1024.,
 		"The default chunksize used by fetcher. "
 		"This should be bigger than the majority of objects with "
 		"short TTLs.\n"
@@ -641,7 +641,7 @@ static const struct parspec input_parspec[] = {
 		EXPERIMENTAL,
 		"128", "kilobytes" },
 	{ "fetch_maxchunksize",
-		tweak_uint, &master.fetch_maxchunksize, 64, UINT_MAX / 1024.,
+		tweak_uint, &mgt_param.fetch_maxchunksize, 64, UINT_MAX / 1024.,
 		"The maximum chunksize we attempt to allocate from storage. "
 		"Making this too large may cause delays and storage "
 		"fragmentation.\n",
@@ -649,12 +649,12 @@ static const struct parspec input_parspec[] = {
 		"262144", "kilobytes" },
 #ifdef SENDFILE_WORKS
 	{ "sendfile_threshold",
-		tweak_uint, &master.sendfile_threshold, 0, UINT_MAX,
+		tweak_uint, &mgt_param.sendfile_threshold, 0, UINT_MAX,
 		"The minimum size of objects transmitted with sendfile.",
 		EXPERIMENTAL,
 		"-1", "bytes" },
 #endif /* SENDFILE_WORKS */
-	{ "vcl_trace", tweak_bool,  &master.vcl_trace, 0, 0,
+	{ "vcl_trace", tweak_bool,  &mgt_param.vcl_trace, 0, 0,
 		"Trace VCL execution in the shmlog.\n"
 		"Enabling this will allow you to see the path each "
 		"request has taken through the VCL program.\n"
@@ -668,22 +668,22 @@ static const struct parspec input_parspec[] = {
 		"Possible formats: host, host:port, :port",
 		MUST_RESTART,
 		":80" },
-	{ "listen_depth", tweak_uint, &master.listen_depth, 0, UINT_MAX,
+	{ "listen_depth", tweak_uint, &mgt_param.listen_depth, 0, UINT_MAX,
 		"Listen queue depth.",
 		MUST_RESTART,
 		"1024", "connections" },
-	{ "cli_timeout", tweak_timeout, &master.cli_timeout, 0, 0,
+	{ "cli_timeout", tweak_timeout, &mgt_param.cli_timeout, 0, 0,
 		"Timeout for the childs replies to CLI requests from "
-		"the master.",
+		"the mgt_param.",
 		0,
 		"10", "seconds" },
-	{ "ping_interval", tweak_uint, &master.ping_interval, 0, UINT_MAX,
+	{ "ping_interval", tweak_uint, &mgt_param.ping_interval, 0, UINT_MAX,
 		"Interval between pings from parent to child.\n"
 		"Zero will disable pinging entirely, which makes "
 		"it possible to attach a debugger to the child.",
 		MUST_RESTART,
 		"3", "seconds" },
-	{ "lru_interval", tweak_timeout, &master.lru_timeout, 0, 0,
+	{ "lru_interval", tweak_timeout, &mgt_param.lru_timeout, 0, 0,
 		"Grace period before object moves on LRU list.\n"
 		"Objects are only moved to the front of the LRU "
 		"list if they have not been moved there already inside "
@@ -698,14 +698,14 @@ static const struct parspec input_parspec[] = {
 		"and %o will be replaced with the output file name.",
 		MUST_RELOAD,
 		VCC_CC , NULL },
-	{ "max_restarts", tweak_uint, &master.max_restarts, 0, UINT_MAX,
+	{ "max_restarts", tweak_uint, &mgt_param.max_restarts, 0, UINT_MAX,
 		"Upper limit on how many times a request can restart."
 		"\nBe aware that restarts are likely to cause a hit against "
 		"the backend, so don't increase thoughtlessly.\n",
 		0,
 		"4", "restarts" },
 	{ "esi_syntax",
-		tweak_uint, &master.esi_syntax, 0, UINT_MAX,
+		tweak_uint, &mgt_param.esi_syntax, 0, UINT_MAX,
 		"Bitmap controlling ESI parsing code:\n"
 		"  0x00000001 - Don't check if it looks like XML\n"
 		"  0x00000002 - Ignore non-esi elements\n"
@@ -715,12 +715,12 @@ static const struct parspec input_parspec[] = {
 		0,
 		"0", "bitmap" },
 	{ "max_esi_depth",
-		tweak_uint, &master.max_esi_depth, 0, UINT_MAX,
+		tweak_uint, &mgt_param.max_esi_depth, 0, UINT_MAX,
 		"Maximum depth of esi:include processing.\n",
 		0,
 		"5", "levels" },
 	{ "connect_timeout", tweak_timeout_double,
-		&master.connect_timeout,0, UINT_MAX,
+		&mgt_param.connect_timeout,0, UINT_MAX,
 		"Default connection timeout for backend connections. "
 		"We only try to connect to the backend for this many "
 		"seconds before giving up. "
@@ -729,7 +729,7 @@ static const struct parspec input_parspec[] = {
 		0,
 		"0.7", "s" },
 	{ "first_byte_timeout", tweak_timeout_double,
-		&master.first_byte_timeout,0, UINT_MAX,
+		&mgt_param.first_byte_timeout,0, UINT_MAX,
 		"Default timeout for receiving first byte from backend. "
 		"We only wait for this many seconds for the first "
 		"byte before giving up. A value of 0 means it will never time "
@@ -739,7 +739,7 @@ static const struct parspec input_parspec[] = {
 		0,
 		"60", "s" },
 	{ "between_bytes_timeout", tweak_timeout_double,
-		&master.between_bytes_timeout,0, UINT_MAX,
+		&mgt_param.between_bytes_timeout,0, UINT_MAX,
 		"Default timeout between bytes when receiving data from "
 		"backend. "
 		"We only wait for this many seconds between bytes "
@@ -749,7 +749,7 @@ static const struct parspec input_parspec[] = {
 		0,
 		"60", "s" },
 	{ "acceptor_sleep_max", tweak_timeout_double,
-		&master.acceptor_sleep_max, 0,  10,
+		&mgt_param.acceptor_sleep_max, 0,  10,
 		"If we run out of resources, such as file descriptors or "
 		"worker threads, the acceptor will sleep between accepts.\n"
 		"This parameter limits how long it can sleep between "
@@ -757,7 +757,7 @@ static const struct parspec input_parspec[] = {
 		EXPERIMENTAL,
 		"0.050", "s" },
 	{ "acceptor_sleep_incr", tweak_timeout_double,
-		&master.acceptor_sleep_incr, 0,  1,
+		&mgt_param.acceptor_sleep_incr, 0,  1,
 		"If we run out of resources, such as file descriptors or "
 		"worker threads, the acceptor will sleep between accepts.\n"
 		"This parameter control how much longer we sleep, each time "
@@ -765,25 +765,25 @@ static const struct parspec input_parspec[] = {
 		EXPERIMENTAL,
 		"0.001", "s" },
 	{ "acceptor_sleep_decay", tweak_generic_double,
-		&master.acceptor_sleep_decay, 0,  1,
+		&mgt_param.acceptor_sleep_decay, 0,  1,
 		"If we run out of resources, such as file descriptors or "
 		"worker threads, the acceptor will sleep between accepts.\n"
 		"This parameter (multiplicatively) reduce the sleep duration "
 		"for each succesfull accept. (ie: 0.9 = reduce by 10%)",
 		EXPERIMENTAL,
 		"0.900", "" },
-	{ "clock_skew", tweak_uint, &master.clock_skew, 0, UINT_MAX,
+	{ "clock_skew", tweak_uint, &mgt_param.clock_skew, 0, UINT_MAX,
 		"How much clockskew we are willing to accept between the "
 		"backend and our own clock.",
 		0,
 		"10", "s" },
-	{ "prefer_ipv6", tweak_bool, &master.prefer_ipv6, 0, 0,
+	{ "prefer_ipv6", tweak_bool, &mgt_param.prefer_ipv6, 0, 0,
 		"Prefer IPv6 address when connecting to backends which "
 		"have both IPv4 and IPv6 addresses.",
 		0,
 		"off", "bool" },
 	{ "session_max", tweak_uint,
-		&master.max_sess, 1000, UINT_MAX,
+		&mgt_param.max_sess, 1000, UINT_MAX,
 		"Maximum number of sessions we will allocate from one pool "
 		"before just dropping connections.\n"
 		"This is mostly an anti-DoS measure, and setting it plenty "
@@ -792,7 +792,7 @@ static const struct parspec input_parspec[] = {
 		0,
 		"100000", "sessions" },
 	{ "session_linger", tweak_uint,
-		&master.session_linger,0, UINT_MAX,
+		&mgt_param.session_linger,0, UINT_MAX,
 		"How long time the workerthread lingers on the session "
 		"to see if a new request appears right away.\n"
 		"If sessions are reused, as much as half of all reuses "
@@ -803,18 +803,18 @@ static const struct parspec input_parspec[] = {
 		"more sessions take a detour around the waiter.",
 		EXPERIMENTAL,
 		"50", "ms" },
-	{ "cli_buffer", tweak_uint, &master.cli_buffer, 4096, UINT_MAX,
+	{ "cli_buffer", tweak_uint, &mgt_param.cli_buffer, 4096, UINT_MAX,
 		"Size of buffer for CLI input."
 		"\nYou may need to increase this if you have big VCL files "
 		"and use the vcl.inline CLI command.\n"
 		"NB: Must be specified with -p to have effect.\n",
 		0,
 		"8192", "bytes" },
-	{ "log_hashstring", tweak_bool, &master.log_hash, 0, 0,
+	{ "log_hashstring", tweak_bool, &mgt_param.log_hash, 0, 0,
 		"Log the hash string components to shared memory log.\n",
 		0,
 		"on", "bool" },
-	{ "log_local_address", tweak_bool, &master.log_local_addr, 0, 0,
+	{ "log_local_address", tweak_bool, &mgt_param.log_local_addr, 0, 0,
 		"Log the local address on the TCP connection in the "
 		"SessionOpen shared memory record.\n",
 		0,
@@ -846,16 +846,16 @@ static const struct parspec input_parspec[] = {
 		"Use 0x notation and do the bitor in your head :-)\n",
 		0,
 		"0", "bitmap" },
-	{ "ban_dups", tweak_bool, &master.ban_dups, 0, 0,
+	{ "ban_dups", tweak_bool, &mgt_param.ban_dups, 0, 0,
 		"Detect and eliminate duplicate bans.\n",
 		0,
 		"on", "bool" },
-	{ "syslog_cli_traffic", tweak_bool, &master.syslog_cli_traffic, 0, 0,
+	{ "syslog_cli_traffic", tweak_bool, &mgt_param.syslog_cli_traffic, 0, 0,
 		"Log all CLI traffic to syslog(LOG_INFO).\n",
 		0,
 		"on", "bool" },
 	{ "ban_lurker_sleep", tweak_timeout_double,
-		&master.ban_lurker_sleep, 0, UINT_MAX,
+		&mgt_param.ban_lurker_sleep, 0, UINT_MAX,
 		"How long time does the ban lurker thread sleeps between "
 		"successful attempts to push the last item up the ban "
 		" list.  It always sleeps a second when nothing can be done.\n"
@@ -863,17 +863,17 @@ static const struct parspec input_parspec[] = {
 		0,
 		"0.01", "s" },
 	{ "saintmode_threshold", tweak_uint,
-		&master.saintmode_threshold, 0, UINT_MAX,
+		&mgt_param.saintmode_threshold, 0, UINT_MAX,
 		"The maximum number of objects held off by saint mode before "
 		"no further will be made to the backend until one times out.  "
 		"A value of 0 disables saintmode.",
 		EXPERIMENTAL,
 		"10", "objects" },
-	{ "http_range_support", tweak_bool, &master.http_range_support, 0, 0,
+	{ "http_range_support", tweak_bool, &mgt_param.http_range_support, 0, 0,
 		"Enable support for HTTP Range headers.\n",
 		EXPERIMENTAL,
 		"on", "bool" },
-	{ "http_gzip_support", tweak_bool, &master.http_gzip_support, 0, 0,
+	{ "http_gzip_support", tweak_bool, &mgt_param.http_gzip_support, 0, 0,
 		"Enable gzip support. When enabled Varnish will compress "
 		"uncompressed objects before they are stored in the cache. "
 		"If a client does not support gzip encoding Varnish will "
@@ -887,7 +887,7 @@ static const struct parspec input_parspec[] = {
 		"Varnish reference.",
 		EXPERIMENTAL,
 		"on", "bool" },
-	{ "gzip_tmp_space", tweak_uint, &master.gzip_tmp_space, 0, 2,
+	{ "gzip_tmp_space", tweak_uint, &mgt_param.gzip_tmp_space, 0, 2,
 		"Where temporary space for gzip/gunzip is allocated:\n"
 		"  0 - malloc\n"
 		"  2 - thread workspace\n"
@@ -897,21 +897,21 @@ static const struct parspec input_parspec[] = {
 		" needs 32+KB of workspace (64+KB if ESI processing).",
 		EXPERIMENTAL,
 		"0", "" },
-	{ "gzip_level", tweak_uint, &master.gzip_level, 0, 9,
+	{ "gzip_level", tweak_uint, &mgt_param.gzip_level, 0, 9,
 		"Gzip compression level: 0=debug, 1=fast, 9=best",
 		0,
 		"6", ""},
-	{ "gzip_window", tweak_uint, &master.gzip_window, 8, 15,
+	{ "gzip_window", tweak_uint, &mgt_param.gzip_window, 8, 15,
 		"Gzip window size 8=least, 15=most compression.\n"
 		"Memory impact is 8=1k, 9=2k, ... 15=128k.",
 		0,
 		"15", ""},
-	{ "gzip_memlevel", tweak_uint, &master.gzip_memlevel, 1, 9,
+	{ "gzip_memlevel", tweak_uint, &mgt_param.gzip_memlevel, 1, 9,
 		"Gzip memory level 1=slow/least, 9=fast/most compression.\n"
 		"Memory impact is 1=1k, 2=2k, ... 9=256k.",
 		0,
 		"8", ""},
-	{ "gzip_stack_buffer", tweak_uint, &master.gzip_stack_buffer,
+	{ "gzip_stack_buffer", tweak_uint, &mgt_param.gzip_stack_buffer,
 	        2048, UINT_MAX,
 		"Size of stack buffer used for gzip processing.\n"
 		"The stack buffers are used for in-transit data,"
@@ -922,13 +922,13 @@ static const struct parspec input_parspec[] = {
 		EXPERIMENTAL,
 		"32768", "Bytes" },
 	{ "shortlived", tweak_timeout_double,
-		&master.shortlived, 0, UINT_MAX,
+		&mgt_param.shortlived, 0, UINT_MAX,
 		"Objects created with TTL shorter than this are always "
 		"put in transient storage.\n",
 		0,
 		"10.0", "s" },
 	{ "critbit_cooloff", tweak_timeout_double,
-		&master.critbit_cooloff, 60, 254,
+		&mgt_param.critbit_cooloff, 60, 254,
 		"How long time the critbit hasher keeps deleted objheads "
 		"on the cooloff list.\n",
 		WIZARD,
@@ -959,7 +959,7 @@ static const struct parspec input_parspec[] = {
 
 
 	{ "pcre_match_limit", tweak_uint,
-		&master.vre_limits.match,
+		&mgt_param.vre_limits.match,
 		1, UINT_MAX,
 		"The limit for the  number of internal matching function"
 		" calls in a pcre_exec() execution.",
@@ -967,7 +967,7 @@ static const struct parspec input_parspec[] = {
 		"10000", ""},
 
 	{ "pcre_match_limit_recursion", tweak_uint,
-		&master.vre_limits.match_recursion,
+		&mgt_param.vre_limits.match_recursion,
 		1, UINT_MAX,
 		"The limit for the  number of internal matching function"
 		" recursions in a pcre_exec() execution.",
@@ -1062,37 +1062,31 @@ mcf_param_show(struct cli *cli, const char * const *av, void *priv)
 /*--------------------------------------------------------------------*/
 
 void
-MCF_ParamSync(void)
-{
-	if (params != &master)
-		*params = master;
-}
-
-/*--------------------------------------------------------------------*/
-
-void
 MCF_ParamSet(struct cli *cli, const char *param, const char *val)
 {
 	const struct parspec *pp;
 
 	pp = mcf_findpar(param);
-	if (pp != NULL) {
-		pp->func(cli, pp, val);
-		if (cli->result != CLIS_OK) {
-			VCLI_Out(cli, "(attempting to set param %s to %s)\n",
-			    pp->name, val);
-		} else if (child_pid >= 0 && pp->flags & MUST_RESTART) {
-			VCLI_Out(cli, "Change will take effect"
-			    " when child is restarted");
-		} else if (pp->flags & MUST_RELOAD) {
-			VCLI_Out(cli, "Change will take effect"
-			    " when VCL script is reloaded");
-		}
-		MCF_ParamSync();
+	if (pp == NULL) {
+		VCLI_SetResult(cli, CLIS_PARAM);
+		VCLI_Out(cli, "Unknown parameter \"%s\".", param);
 		return;
 	}
-	VCLI_SetResult(cli, CLIS_PARAM);
-	VCLI_Out(cli, "Unknown parameter \"%s\".", param);
+	pp->func(cli, pp, val);
+
+	if (cli->result == CLIS_OK && params != NULL)
+		*params = mgt_param;
+
+	if (cli->result != CLIS_OK) {
+		VCLI_Out(cli, "(attempting to set param %s to %s)\n",
+		    pp->name, val);
+	} else if (child_pid >= 0 && pp->flags & MUST_RESTART) {
+		VCLI_Out(cli, "Change will take effect"
+		    " when child is restarted");
+	} else if (pp->flags & MUST_RELOAD) {
+		VCLI_Out(cli, "Change will take effect"
+		    " when VCL script is reloaded");
+	}
 }
 
 
@@ -1173,8 +1167,6 @@ MCF_ParamInit(struct cli *cli)
 	/* XXX: We do this twice, to get past any interdependencies */
 	MCF_SetDefaults(NULL);
 	MCF_SetDefaults(cli);
-
-	params = &master;
 }
 
 /*--------------------------------------------------------------------*/
diff --git a/bin/varnishd/mgt/mgt_pool.c b/bin/varnishd/mgt/mgt_pool.c
index 978e60a..94459b4 100644
--- a/bin/varnishd/mgt/mgt_pool.c
+++ b/bin/varnishd/mgt/mgt_pool.c
@@ -59,8 +59,8 @@ tweak_thread_pool_min(struct cli *cli, const struct parspec *par,
     const char *arg)
 {
 
-	tweak_generic_uint(cli, &master.wthread_min, arg,
-	    (unsigned)par->min, master.wthread_max);
+	tweak_generic_uint(cli, &mgt_param.wthread_min, arg,
+	    (unsigned)par->min, mgt_param.wthread_max);
 }
 
 /*--------------------------------------------------------------------
@@ -86,7 +86,7 @@ tweak_stack_size(struct cli *cli, const struct parspec *par,
 		arg = buf;
 	}
 
-	tweak_generic_uint(cli, &master.wthread_stacksize, arg,
+	tweak_generic_uint(cli, &mgt_param.wthread_stacksize, arg,
 	    low, (uint)par->max);
 }
 
@@ -98,14 +98,14 @@ tweak_thread_pool_max(struct cli *cli, const struct parspec *par,
 {
 
 	(void)par;
-	tweak_generic_uint(cli, &master.wthread_max, arg,
-	    master.wthread_min, UINT_MAX);
+	tweak_generic_uint(cli, &mgt_param.wthread_max, arg,
+	    mgt_param.wthread_min, UINT_MAX);
 }
 
 /*--------------------------------------------------------------------*/
 
 const struct parspec WRK_parspec[] = {
-	{ "thread_pools", tweak_uint, &master.wthread_pools, 1, UINT_MAX,
+	{ "thread_pools", tweak_uint, &mgt_param.wthread_pools, 1, UINT_MAX,
 		"Number of worker thread pools.\n"
 		"\n"
 		"Increasing number of worker pools decreases lock "
@@ -135,7 +135,7 @@ const struct parspec WRK_parspec[] = {
 		"Minimum is 2 threads.",
 		EXPERIMENTAL | DELAYED_EFFECT,
 		"5", "threads" },
-	{ "thread_pool_timeout", tweak_timeout, &master.wthread_timeout, 1, 0,
+	{ "thread_pool_timeout", tweak_timeout, &mgt_param.wthread_timeout, 1, 0,
 		"Thread idle threshold.\n"
 		"\n"
 		"Threads in excess of thread_pool_min, which have been idle "
@@ -145,7 +145,7 @@ const struct parspec WRK_parspec[] = {
 		EXPERIMENTAL | DELAYED_EFFECT,
 		"300", "seconds" },
 	{ "thread_pool_purge_delay",
-		tweak_timeout, &master.wthread_purge_delay, 100, 0,
+		tweak_timeout, &mgt_param.wthread_purge_delay, 100, 0,
 		"Wait this long between purging threads.\n"
 		"\n"
 		"This controls the decay of thread pools when idle(-ish).\n"
@@ -154,7 +154,7 @@ const struct parspec WRK_parspec[] = {
 		EXPERIMENTAL | DELAYED_EFFECT,
 		"1000", "milliseconds" },
 	{ "thread_pool_add_threshold",
-		tweak_uint, &master.wthread_add_threshold, 0, UINT_MAX,
+		tweak_uint, &mgt_param.wthread_add_threshold, 0, UINT_MAX,
 		"Overflow threshold for worker thread creation.\n"
 		"\n"
 		"Setting this too low, will result in excess worker threads, "
@@ -164,7 +164,7 @@ const struct parspec WRK_parspec[] = {
 		EXPERIMENTAL,
 		"2", "requests" },
 	{ "thread_pool_add_delay",
-		tweak_timeout, &master.wthread_add_delay, 0, UINT_MAX,
+		tweak_timeout, &mgt_param.wthread_add_delay, 0, UINT_MAX,
 		"Wait at least this long between creating threads.\n"
 		"\n"
 		"Setting this too long results in insuffient worker threads.\n"
@@ -174,7 +174,7 @@ const struct parspec WRK_parspec[] = {
 		0,
 		"2", "milliseconds" },
 	{ "thread_pool_fail_delay",
-		tweak_timeout, &master.wthread_fail_delay, 100, UINT_MAX,
+		tweak_timeout, &mgt_param.wthread_fail_delay, 100, UINT_MAX,
 		"Wait at least this long after a failed thread creation "
 		"before trying to create another thread.\n"
 		"\n"
@@ -192,7 +192,7 @@ const struct parspec WRK_parspec[] = {
 		EXPERIMENTAL,
 		"200", "milliseconds" },
 	{ "thread_stats_rate",
-		tweak_uint, &master.wthread_stats_rate, 0, UINT_MAX,
+		tweak_uint, &mgt_param.wthread_stats_rate, 0, UINT_MAX,
 		"Worker threads accumulate statistics, and dump these into "
 		"the global stats counters if the lock is free when they "
 		"finish a request.\n"
@@ -201,14 +201,14 @@ const struct parspec WRK_parspec[] = {
 		"its accumulated stats into the global counters.\n",
 		EXPERIMENTAL,
 		"10", "requests" },
-	{ "queue_max", tweak_uint, &master.queue_max, 0, UINT_MAX,
+	{ "queue_max", tweak_uint, &mgt_param.queue_max, 0, UINT_MAX,
 		"Percentage permitted queue length.\n"
 		"\n"
 		"This sets the ratio of queued requests to worker threads, "
 		"above which sessions will be dropped instead of queued.\n",
 		EXPERIMENTAL,
 		"100", "%" },
-	{ "rush_exponent", tweak_uint, &master.rush_exponent, 2, UINT_MAX,
+	{ "rush_exponent", tweak_uint, &mgt_param.rush_exponent, 2, UINT_MAX,
 		"How many parked request we start for each completed "
 		"request on the object.\n"
 		"NB: Even with the implict delay of delivery, "
@@ -217,13 +217,13 @@ const struct parspec WRK_parspec[] = {
 		EXPERIMENTAL,
 		"3", "requests per request" },
 	{ "thread_pool_stack",
-		tweak_stack_size, &master.wthread_stacksize, 0, UINT_MAX,
+		tweak_stack_size, &mgt_param.wthread_stacksize, 0, UINT_MAX,
 		"Worker thread stack size.\n"
 		"On 32bit systems you may need to tweak this down to fit "
 		"many threads into the limited address space.\n",
 		EXPERIMENTAL,
 		"-1", "bytes" },
-	{ "thread_pool_workspace", tweak_uint, &master.wthread_workspace,
+	{ "thread_pool_workspace", tweak_uint, &mgt_param.wthread_workspace,
 		1024, UINT_MAX,
 		"Bytes of HTTP protocol workspace allocated for worker "
 		"threads. "
diff --git a/bin/varnishd/mgt/mgt_sandbox.c b/bin/varnishd/mgt/mgt_sandbox.c
index b01d243..e5fe99c 100644
--- a/bin/varnishd/mgt/mgt_sandbox.c
+++ b/bin/varnishd/mgt/mgt_sandbox.c
@@ -68,8 +68,8 @@ mgt_sandbox(void)
 	mgt_sandbox_solaris_privsep();
 #else
 	if (geteuid() == 0) {
-		XXXAZ(setgid(params->gid));
-		XXXAZ(setuid(params->uid));
+		XXXAZ(setgid(mgt_param.gid));
+		XXXAZ(setuid(mgt_param.uid));
 	} else {
 		REPORT0(LOG_INFO, "Not running as root, no priv-sep");
 	}
diff --git a/bin/varnishd/mgt/mgt_sandbox_solaris.c b/bin/varnishd/mgt/mgt_sandbox_solaris.c
index becaa2e..8c26d69 100644
--- a/bin/varnishd/mgt/mgt_sandbox_solaris.c
+++ b/bin/varnishd/mgt/mgt_sandbox_solaris.c
@@ -158,10 +158,10 @@ void
 mgt_sandbox_solaris_privsep(void)
 {
 	if (priv_ineffect(PRIV_PROC_SETID)) {
-                if (getgid() != params->gid)
-                        XXXAZ(setgid(params->gid));
-                if (getuid() != params->uid)
-                        XXXAZ(setuid(params->uid));
+                if (getgid() != mgt_param.gid)
+                        XXXAZ(setgid(mgt_param.gid));
+                if (getuid() != mgt_param.uid)
+                        XXXAZ(setuid(mgt_param.uid));
         } else {
                 REPORT(LOG_INFO, "Privilege %s missing, will not change uid/gid",
 		    PRIV_PROC_SETID);
diff --git a/bin/varnishd/mgt/mgt_shmem.c b/bin/varnishd/mgt/mgt_shmem.c
index 630865d..37083df 100644
--- a/bin/varnishd/mgt/mgt_shmem.c
+++ b/bin/varnishd/mgt/mgt_shmem.c
@@ -229,7 +229,6 @@ void
 mgt_SHM_Init(const char *l_arg)
 {
 	int i, fill;
-	struct params *pp;
 	const char *q;
 	uintmax_t size, s1, s2, ps;
 	char **av, **ap;
@@ -321,10 +320,10 @@ mgt_SHM_Init(const char *l_arg)
 	    VSC_CLASS, VSC_TYPE_MAIN, "");
 	AN(VSC_C_main);
 
-	pp = VSM_Alloc(sizeof *pp, VSM_CLASS_PARAM, "", "");
-	AN(pp);
-	*pp = *params;
-	params = pp;
+	/* XXX: We need to zero params if we dealloc/clean/wash */
+	params = VSM_Alloc(sizeof *params, VSM_CLASS_PARAM, "", "");
+	AN(params);
+	*params = mgt_param;
 
 	vsl_log_start = VSM_Alloc(s1, VSL_CLASS, "", "");
 	AN(vsl_log_start);
diff --git a/bin/varnishd/varnishd.c b/bin/varnishd/varnishd.c
index e0ba759..6ae1842 100644
--- a/bin/varnishd/varnishd.c
+++ b/bin/varnishd/varnishd.c
@@ -64,7 +64,6 @@
 #endif
 
 struct heritage		heritage;
-volatile struct params	*params;
 unsigned		d_flag = 0;
 pid_t			mgt_pid;
 struct vev_base		*mgt_evb;
@@ -190,17 +189,17 @@ tackle_warg(const char *argv)
 	u = arg_ul(av[1]);
 	if (u < 1)
 		usage();
-	params->wthread_max = params->wthread_min = u;
+	mgt_param.wthread_max = mgt_param.wthread_min = u;
 
 	if (av[2] != NULL) {
 		u = arg_ul(av[2]);
-		if (u < params->wthread_min)
+		if (u < mgt_param.wthread_min)
 			usage();
-		params->wthread_max = u;
+		mgt_param.wthread_max = u;
 
 		if (av[3] != NULL) {
 			u = arg_ul(av[3]);
-			params->wthread_timeout = u;
+			mgt_param.wthread_timeout = u;
 		}
 	}
 	VAV_Free(av);
diff --git a/bin/varnishd/vparam.h b/bin/varnishd/vparam.h
index 2a83de1..2d4a97f 100644
--- a/bin/varnishd/vparam.h
+++ b/bin/varnishd/vparam.h
@@ -55,7 +55,5 @@ void tweak_uint(struct cli *cli, const struct parspec *par, const char *arg);
 void tweak_timeout(struct cli *cli,
     const struct parspec *par, const char *arg);
 
-extern struct params master;
-
 /* mgt_pool.c */
 extern const struct parspec WRK_parspec[];



From phk at varnish-cache.org  Thu Nov 10 10:08:41 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Thu, 10 Nov 2011 11:08:41 +0100
Subject: [master] e4fa4c7 Rename the childs copy of params to cache_param so
	it is clear that it belongs there.
Message-ID: <E1RORYn-0006Oy-S6@project.varnish-software.com>

commit e4fa4c7ab40a7514395645cbe6e1227d580d4882
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Nov 10 09:05:18 2011 +0000

    Rename the childs copy of params to cache_param so it is clear that
    it belongs there.

diff --git a/bin/varnishd/cache.h b/bin/varnishd/cache.h
index 31db957..5941fcc 100644
--- a/bin/varnishd/cache.h
+++ b/bin/varnishd/cache.h
@@ -873,7 +873,7 @@ void WSL_Flush(struct worker *w, int overflow);
 
 #define DSL(flag, tag, id, ...)					\
 	do {							\
-		if (params->diag_bitmap & (flag))		\
+		if (cache_param->diag_bitmap & (flag))		\
 			VSL((tag), (id), __VA_ARGS__);		\
 	} while (0)
 
diff --git a/bin/varnishd/cache_acceptor.c b/bin/varnishd/cache_acceptor.c
index 03a1a34..550a53f 100644
--- a/bin/varnishd/cache_acceptor.c
+++ b/bin/varnishd/cache_acceptor.c
@@ -121,7 +121,7 @@ VCA_Prep(struct sess *sp)
 	    addr, sizeof addr, port, sizeof port);
 	sp->addr = WS_Dup(sp->ws, addr);
 	sp->port = WS_Dup(sp->ws, port);
-	if (params->log_local_addr) {
+	if (cache_param->log_local_addr) {
 		AZ(getsockname(sp->fd, (void*)&sp->mysockaddr, &sp->mysockaddrlen));
 		VTCP_name(&sp->mysockaddr, sp->mysockaddrlen,
 		    addr, sizeof addr, port, sizeof port);
@@ -176,9 +176,9 @@ vca_pace_bad(void)
 {
 
 	Lck_Lock(&pace_mtx);
-	vca_pace += params->acceptor_sleep_incr;
-	if (vca_pace > params->acceptor_sleep_max)
-		vca_pace = params->acceptor_sleep_max;
+	vca_pace += cache_param->acceptor_sleep_incr;
+	if (vca_pace > cache_param->acceptor_sleep_max)
+		vca_pace = cache_param->acceptor_sleep_max;
 	Lck_Unlock(&pace_mtx);
 }
 
@@ -189,8 +189,8 @@ vca_pace_good(void)
 	if (vca_pace == 0.0)
 		return;
 	Lck_Lock(&pace_mtx);
-	vca_pace *= params->acceptor_sleep_decay;
-	if (vca_pace < params->acceptor_sleep_incr)
+	vca_pace *= cache_param->acceptor_sleep_decay;
+	if (vca_pace < cache_param->acceptor_sleep_incr)
 		vca_pace = 0.0;
 	Lck_Unlock(&pace_mtx);
 }
@@ -306,7 +306,7 @@ vca_acct(void *arg)
 	VTAILQ_FOREACH(ls, &heritage.socks, list) {
 		if (ls->sock < 0)
 			continue;
-		AZ(listen(ls->sock, params->listen_depth));
+		AZ(listen(ls->sock, cache_param->listen_depth));
 		AZ(setsockopt(ls->sock, SOL_SOCKET, SO_LINGER,
 		    &linger, sizeof linger));
 	}
@@ -318,9 +318,9 @@ vca_acct(void *arg)
 	while (1) {
 		(void)sleep(1);
 #ifdef SO_SNDTIMEO_WORKS
-		if (params->idle_send_timeout != send_timeout) {
+		if (cache_param->idle_send_timeout != send_timeout) {
 			need_test = 1;
-			send_timeout = params->idle_send_timeout;
+			send_timeout = cache_param->idle_send_timeout;
 			tv_sndtimeo = VTIM_timeval(send_timeout);
 			VTAILQ_FOREACH(ls, &heritage.socks, list) {
 				if (ls->sock < 0)
@@ -332,9 +332,9 @@ vca_acct(void *arg)
 		}
 #endif
 #ifdef SO_RCVTIMEO_WORKS
-		if (params->sess_timeout != sess_timeout) {
+		if (cache_param->sess_timeout != sess_timeout) {
 			need_test = 1;
-			sess_timeout = params->sess_timeout;
+			sess_timeout = cache_param->sess_timeout;
 			tv_rcvtimeo = VTIM_timeval(sess_timeout);
 			VTAILQ_FOREACH(ls, &heritage.socks, list) {
 				if (ls->sock < 0)
diff --git a/bin/varnishd/cache_backend.c b/bin/varnishd/cache_backend.c
index c94d819..e9d0323 100644
--- a/bin/varnishd/cache_backend.c
+++ b/bin/varnishd/cache_backend.c
@@ -96,7 +96,7 @@ VBE_ReleaseConn(struct vbc *vc)
 		if (dst == 0.0)			\
 			dst = be->tmx;		\
 		if (dst == 0.0)			\
-			dst = params->tmx;	\
+			dst = cache_param->tmx;	\
 	} while (0)
 
 /*--------------------------------------------------------------------
@@ -158,7 +158,7 @@ bes_conn_try(const struct sess *sp, struct vbc *vc, const struct vdi_simple *vs)
 
 	/* release lock during stuff that can take a long time */
 
-	if (params->prefer_ipv6 && bp->ipv6 != NULL) {
+	if (cache_param->prefer_ipv6 && bp->ipv6 != NULL) {
 		s = vbe_TryConnect(sp, PF_INET6, bp->ipv6, bp->ipv6len, vs);
 		vc->addr = bp->ipv6;
 		vc->addrlen = bp->ipv6len;
@@ -168,7 +168,7 @@ bes_conn_try(const struct sess *sp, struct vbc *vc, const struct vdi_simple *vs)
 		vc->addr = bp->ipv4;
 		vc->addrlen = bp->ipv4len;
 	}
-	if (s == -1 && !params->prefer_ipv6 && bp->ipv6 != NULL) {
+	if (s == -1 && !cache_param->prefer_ipv6 && bp->ipv6 != NULL) {
 		s = vbe_TryConnect(sp, PF_INET6, bp->ipv6, bp->ipv6len, vs);
 		vc->addr = bp->ipv6;
 		vc->addrlen = bp->ipv6len;
@@ -232,7 +232,7 @@ vbe_NewConn(void)
  * It evaluates if a backend is healthy _for_a_specific_object_.
  * That means that it relies on sp->objcore->objhead. This is mainly for
  * saint-mode, but also takes backend->healthy into account. If
- * params->saintmode_threshold is 0, this is basically just a test of
+ * cache_param->saintmode_threshold is 0, this is basically just a test of
  * backend->healthy.
  *
  * The threshold has to be evaluated _after_ the timeout check, otherwise
@@ -266,7 +266,7 @@ vbe_Healthy(const struct vdi_simple *vs, const struct sess *sp)
 	 * specified by VCL (thus use param).
 	 */
 	if (vs->vrt->saintmode_threshold == UINT_MAX)
-		threshold = params->saintmode_threshold;
+		threshold = cache_param->saintmode_threshold;
 	else
 		threshold = vs->vrt->saintmode_threshold;
 
diff --git a/bin/varnishd/cache_backend_poll.c b/bin/varnishd/cache_backend_poll.c
index eb6cc45..efd64cb 100644
--- a/bin/varnishd/cache_backend_poll.c
+++ b/bin/varnishd/cache_backend_poll.c
@@ -138,7 +138,7 @@ vbp_poke(struct vbp_target *vt)
 	tmo = (int)round((t_end - t_now) * 1e3);
 
 	s = -1;
-	if (params->prefer_ipv6 && bp->ipv6 != NULL) {
+	if (cache_param->prefer_ipv6 && bp->ipv6 != NULL) {
 		s = vbp_connect(PF_INET6, bp->ipv6, bp->ipv6len, tmo);
 		t_now = VTIM_real();
 		tmo = (int)round((t_end - t_now) * 1e3);
diff --git a/bin/varnishd/cache_ban.c b/bin/varnishd/cache_ban.c
index ac45144..768e631 100644
--- a/bin/varnishd/cache_ban.c
+++ b/bin/varnishd/cache_ban.c
@@ -407,7 +407,7 @@ BAN_Insert(struct ban *b)
 		VSC_C_main->bans_req++;
 
 	be = VTAILQ_LAST(&ban_head, banhead_s);
-	if (params->ban_dups && be != b)
+	if (cache_param->ban_dups && be != b)
 		be->refcount++;
 	else
 		be = NULL;
@@ -825,13 +825,13 @@ ban_lurker_work(const struct sess *sp, unsigned pass)
 		b->flags &= ~BAN_F_LURK;
 		b->flags |= pass;
 	}
-	if (params->diag_bitmap & 0x80000)
+	if (cache_param->diag_bitmap & 0x80000)
 		VSL(SLT_Debug, 0, "lurker: %d actionable bans", i);
 	if (i == 0)
 		return (0);
 
 	VTAILQ_FOREACH_REVERSE(b, &ban_head, banhead_s, list) {
-		if (params->diag_bitmap & 0x80000)
+		if (cache_param->diag_bitmap & 0x80000)
 			VSL(SLT_Debug, 0, "lurker doing %f %d",
 			    ban_time(b->spec), b->refcount);
 		while (1) {
@@ -840,7 +840,7 @@ ban_lurker_work(const struct sess *sp, unsigned pass)
 			if (oc == NULL)
 				break;
 			CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-			if (params->diag_bitmap & 0x80000)
+			if (cache_param->diag_bitmap & 0x80000)
 				VSL(SLT_Debug, 0, "test: %p %d %d",
 				    oc, oc->flags & OC_F_LURK, pass);
 			if ((oc->flags & OC_F_LURK) == pass)
@@ -849,7 +849,7 @@ ban_lurker_work(const struct sess *sp, unsigned pass)
 			CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC);
 			if (Lck_Trylock(&oh->mtx)) {
 				Lck_Unlock(&ban_mtx);
-				VTIM_sleep(params->ban_lurker_sleep);
+				VTIM_sleep(cache_param->ban_lurker_sleep);
 				continue;
 			}
 			/*
@@ -863,7 +863,7 @@ ban_lurker_work(const struct sess *sp, unsigned pass)
 			if (oc2 == NULL) {
 				Lck_Unlock(&oh->mtx);
 				Lck_Unlock(&ban_mtx);
-				VTIM_sleep(params->ban_lurker_sleep);
+				VTIM_sleep(cache_param->ban_lurker_sleep);
 				continue;
 			}
 			/*
@@ -879,7 +879,7 @@ ban_lurker_work(const struct sess *sp, unsigned pass)
 			 */
 			o = oc_getobj(sp->wrk, oc);
 			i = ban_check_object(o, sp, 0);
-			if (params->diag_bitmap & 0x80000)
+			if (cache_param->diag_bitmap & 0x80000)
 				VSL(SLT_Debug, 0, "lurker got: %p %d",
 				    oc, i);
 			if (i == -1) {
@@ -891,11 +891,11 @@ ban_lurker_work(const struct sess *sp, unsigned pass)
 				Lck_Unlock(&ban_mtx);
 			}
 			Lck_Unlock(&oh->mtx);
-			if (params->diag_bitmap & 0x80000)
+			if (cache_param->diag_bitmap & 0x80000)
 				VSL(SLT_Debug, 0, "lurker done: %p %d %d",
 				    oc, oc->flags & OC_F_LURK, pass);
 			(void)HSH_Deref(sp->wrk, NULL, &o);
-			VTIM_sleep(params->ban_lurker_sleep);
+			VTIM_sleep(cache_param->ban_lurker_sleep);
 		}
 		Lck_AssertHeld(&ban_mtx);
 		if (!(b->flags & BAN_F_REQ)) {
@@ -903,12 +903,12 @@ ban_lurker_work(const struct sess *sp, unsigned pass)
 				b->flags |= BAN_F_GONE;
 				VSC_C_main->bans_gone++;
 			}
-			if (params->diag_bitmap & 0x80000)
+			if (cache_param->diag_bitmap & 0x80000)
 				VSL(SLT_Debug, 0, "lurker BAN %f now gone",
 				    ban_time(b->spec));
 		}
 		Lck_Unlock(&ban_mtx);
-		VTIM_sleep(params->ban_lurker_sleep);
+		VTIM_sleep(cache_param->ban_lurker_sleep);
 		if (b == b0)
 			break;
 	}
@@ -925,7 +925,7 @@ ban_lurker(struct sess *sp, void *priv)
 	(void)priv;
 	while (1) {
 
-		while (params->ban_lurker_sleep == 0.0) {
+		while (cache_param->ban_lurker_sleep == 0.0) {
 			/*
 			 * Ban-lurker is disabled:
 			 * Clean the last ban, if possible, and sleep
@@ -947,7 +947,7 @@ ban_lurker(struct sess *sp, void *priv)
 			pass &= BAN_F_LURK;
 			if (pass == 0)
 				pass += (1 << LURK_SHIFT);
-			VTIM_sleep(params->ban_lurker_sleep);
+			VTIM_sleep(cache_param->ban_lurker_sleep);
 		} else {
 			VTIM_sleep(1.0);
 		}
@@ -1064,14 +1064,14 @@ ccf_ban_list(struct cli *cli, const char * const *av, void *priv)
 
 	VCLI_Out(cli, "Present bans:\n");
 	VTAILQ_FOREACH(b, &ban_head, list) {
-		if (b == bl && !(params->diag_bitmap & 0x80000))
+		if (b == bl && !(cache_param->diag_bitmap & 0x80000))
 			break;
 		VCLI_Out(cli, "%10.6f %5u%s\t", ban_time(b->spec),
 		    bl == b ? b->refcount - 1 : b->refcount,
 		    b->flags & BAN_F_GONE ? "G" : " ");
 		ban_render(cli, b->spec);
 		VCLI_Out(cli, "\n");
-		if (params->diag_bitmap & 0x80000) {
+		if (cache_param->diag_bitmap & 0x80000) {
 			Lck_Lock(&ban_mtx);
 			struct objcore *oc;
 			VTAILQ_FOREACH(oc, &b->objcore, ban_list)
diff --git a/bin/varnishd/cache_center.c b/bin/varnishd/cache_center.c
index 37c4b7b..e42fac8 100644
--- a/bin/varnishd/cache_center.c
+++ b/bin/varnishd/cache_center.c
@@ -94,11 +94,11 @@ cnt_wait(struct sess *sp)
 	assert(sp->xid == 0);
 
 	i = HTC_Complete(sp->htc);
-	if (i == 0 && params->session_linger > 0) {
+	if (i == 0 && cache_param->session_linger > 0) {
 		pfd[0].fd = sp->fd;
 		pfd[0].events = POLLIN;
 		pfd[0].revents = 0;
-		i = poll(pfd, 1, params->session_linger);
+		i = poll(pfd, 1, cache_param->session_linger);
 		if (i)
 			i = HTC_Rx(sp->htc);
 	}
@@ -183,7 +183,7 @@ cnt_prepresp(struct sess *sp)
 		sp->wrk->res_mode |= RES_ESI_CHILD;
 	}
 
-	if (params->http_gzip_support && sp->obj->gziped &&
+	if (cache_param->http_gzip_support && sp->obj->gziped &&
 	    !RFC2616_Req_Gzip(sp)) {
 		/*
 		 * We don't know what it uncompresses to
@@ -212,7 +212,7 @@ cnt_prepresp(struct sess *sp)
 
 	sp->t_resp = VTIM_real();
 	if (sp->obj->objcore != NULL) {
-		if ((sp->t_resp - sp->obj->last_lru) > params->lru_timeout &&
+		if ((sp->t_resp - sp->obj->last_lru) > cache_param->lru_timeout &&
 		    EXP_Touch(sp->obj->objcore))
 			sp->obj->last_lru = sp->t_resp;
 		sp->obj->last_use = sp->t_resp;	/* XXX: locking ? */
@@ -224,7 +224,7 @@ cnt_prepresp(struct sess *sp)
 	case VCL_RET_DELIVER:
 		break;
 	case VCL_RET_RESTART:
-		if (sp->restarts >= params->max_restarts)
+		if (sp->restarts >= cache_param->max_restarts)
 			break;
 		if (sp->wrk->do_stream) {
 			VDI_CloseFd(sp->wrk);
@@ -373,7 +373,7 @@ cnt_done(struct sess *sp)
 		return (1);
 	}
 
-	if (sp->wrk->stats.client_req >= params->wthread_stats_rate)
+	if (sp->wrk->stats.client_req >= cache_param->wthread_stats_rate)
 		WRK_SumStat(sp->wrk);
 	/* Reset the workspace to the session-watermark */
 	WS_Reset(sp->ws, sp->ws_ses);
@@ -390,7 +390,7 @@ cnt_done(struct sess *sp)
 		sp->step = STP_WAIT;
 		return (0);
 	}
-	if (params->session_linger > 0) {
+	if (cache_param->session_linger > 0) {
 		sp->wrk->stats.sess_linger++;
 		sp->step = STP_WAIT;
 		return (0);
@@ -436,12 +436,12 @@ cnt_error(struct sess *sp)
 	if (sp->obj == NULL) {
 		HSH_Prealloc(sp);
 		EXP_Clr(&w->exp);
-		sp->obj = STV_NewObject(sp, NULL, params->http_resp_size,
-		     &w->exp, (uint16_t)params->http_max_hdr);
+		sp->obj = STV_NewObject(sp, NULL, cache_param->http_resp_size,
+		     &w->exp, (uint16_t)cache_param->http_max_hdr);
 		if (sp->obj == NULL)
 			sp->obj = STV_NewObject(sp, TRANSIENT_STORAGE,
-			    params->http_resp_size, &w->exp,
-			    (uint16_t)params->http_max_hdr);
+			    cache_param->http_resp_size, &w->exp,
+			    (uint16_t)cache_param->http_max_hdr);
 		if (sp->obj == NULL) {
 			sp->doclose = "Out of objects";
 			sp->director = NULL;
@@ -477,7 +477,7 @@ cnt_error(struct sess *sp)
 	VCL_error_method(sp);
 
 	if (sp->handling == VCL_RET_RESTART &&
-	    sp->restarts <  params->max_restarts) {
+	    sp->restarts <  cache_param->max_restarts) {
 		HSH_Drop(sp);
 		sp->director = NULL;
 		sp->restarts++;
@@ -701,7 +701,7 @@ cnt_fetchbody(struct sess *sp)
 	AZ(sp->wrk->vfp);
 
 	/* We do nothing unless the param is set */
-	if (!params->http_gzip_support)
+	if (!cache_param->http_gzip_support)
 		sp->wrk->do_gzip = sp->wrk->do_gunzip = 0;
 
 	sp->wrk->is_gzip =
@@ -768,7 +768,7 @@ cnt_fetchbody(struct sess *sp)
 	 */
 	l += strlen("Content-Length: XxxXxxXxxXxxXxxXxx") + sizeof(void *);
 
-	if (sp->wrk->exp.ttl < params->shortlived || sp->objcore == NULL)
+	if (sp->wrk->exp.ttl < cache_param->shortlived || sp->objcore == NULL)
 		sp->wrk->storage_hint = TRANSIENT_STORAGE;
 
 	sp->obj = STV_NewObject(sp, sp->wrk->storage_hint, l,
@@ -780,8 +780,8 @@ cnt_fetchbody(struct sess *sp)
 		 */
 		sp->obj = STV_NewObject(sp, TRANSIENT_STORAGE, l,
 		    &sp->wrk->exp, nhttp);
-		if (sp->wrk->exp.ttl > params->shortlived)
-			sp->wrk->exp.ttl = params->shortlived;
+		if (sp->wrk->exp.ttl > cache_param->shortlived)
+			sp->wrk->exp.ttl = cache_param->shortlived;
 		sp->wrk->exp.grace = 0.0;
 		sp->wrk->exp.keep = 0.0;
 	}
@@ -893,7 +893,7 @@ cnt_streambody(struct sess *sp)
 	int i;
 	struct stream_ctx sctx;
 	uint8_t obuf[sp->wrk->res_mode & RES_GUNZIP ?
-	    params->gzip_stack_buffer : 1];
+	    cache_param->gzip_stack_buffer : 1];
 
 	memset(&sctx, 0, sizeof sctx);
 	sctx.magic = STREAM_CTX_MAGIC;
@@ -967,8 +967,8 @@ cnt_first(struct sess *sp)
 	sp->ws_ses = WS_Snapshot(sp->ws);
 
 	/* Receive a HTTP protocol request */
-	HTC_Init(sp->htc, sp->ws, sp->fd, sp->vsl_id, params->http_req_size,
-	    params->http_req_hdr_len);
+	HTC_Init(sp->htc, sp->ws, sp->fd, sp->vsl_id, cache_param->http_req_size,
+	    cache_param->http_req_hdr_len);
 	sp->wrk->lastused = sp->t_open;
 	sp->wrk->acct_tmp.sess++;
 
@@ -1186,7 +1186,7 @@ cnt_miss(struct sess *sp)
 	http_Setup(sp->wrk->bereq, sp->wrk->ws);
 	http_FilterHeader(sp, HTTPH_R_FETCH);
 	http_ForceGet(sp->wrk->bereq);
-	if (params->http_gzip_support) {
+	if (cache_param->http_gzip_support) {
 		/*
 		 * We always ask the backend for gzip, even if the
 		 * client doesn't grok it.  We will uncompress for
@@ -1377,7 +1377,7 @@ cnt_recv(struct sess *sp)
 	VCL_recv_method(sp);
 	recv_handling = sp->handling;
 
-	if (sp->restarts >= params->max_restarts) {
+	if (sp->restarts >= cache_param->max_restarts) {
 		if (sp->err_code == 0)
 			sp->err_code = 503;
 		sp->step = STP_ERROR;
@@ -1392,7 +1392,7 @@ cnt_recv(struct sess *sp)
 	sp->wrk->do_gunzip = 0;
 	sp->wrk->do_stream = 0;
 
-	if (params->http_gzip_support &&
+	if (cache_param->http_gzip_support &&
 	     (recv_handling != VCL_RET_PIPE) &&
 	     (recv_handling != VCL_RET_PASS)) {
 		if (RFC2616_Req_Gzip(sp)) {
@@ -1607,7 +1607,7 @@ CNT_Session(struct sess *sp)
 		switch (sp->step) {
 #define STEP(l,u) \
 		    case STP_##u: \
-			if (params->diag_bitmap & 0x01) \
+			if (cache_param->diag_bitmap & 0x01) \
 				cnt_diag(sp, #u); \
 			done = cnt_##l(sp); \
 		        break;
diff --git a/bin/varnishd/cache_cli.c b/bin/varnishd/cache_cli.c
index 30a12e3..ac2afc1 100644
--- a/bin/varnishd/cache_cli.c
+++ b/bin/varnishd/cache_cli.c
@@ -235,7 +235,7 @@ CLI_Init(void)
 	Lck_New(&cli_mtx, lck_cli);
 	cli_thread = pthread_self();
 
-	cls = VCLS_New(cli_cb_before, cli_cb_after, params->cli_buffer);
+	cls = VCLS_New(cli_cb_before, cli_cb_after, cache_param->cli_buffer);
 	AN(cls);
 
 	CLI_AddFuncs(master_cmds);
diff --git a/bin/varnishd/cache_esi_deliver.c b/bin/varnishd/cache_esi_deliver.c
index 2c62bb7..4051027 100644
--- a/bin/varnishd/cache_esi_deliver.c
+++ b/bin/varnishd/cache_esi_deliver.c
@@ -52,7 +52,7 @@ ved_include(struct sess *sp, const char *src, const char *host)
 
 	w = sp->wrk;
 
-	if (sp->esi_level >= params->max_esi_depth)
+	if (sp->esi_level >= cache_param->max_esi_depth)
 		return;
 	sp->esi_level++;
 
@@ -231,7 +231,7 @@ ESI_Deliver(struct sess *sp)
 	uint8_t tailbuf[8 + 5];
 	int isgzip;
 	struct vgz *vgz = NULL;
-	char obuf[params->gzip_stack_buffer];
+	char obuf[cache_param->gzip_stack_buffer];
 	ssize_t obufl = 0;
 	size_t dl;
 	const void *dp;
diff --git a/bin/varnishd/cache_esi_fetch.c b/bin/varnishd/cache_esi_fetch.c
index f414e7b..5ec8f6b 100644
--- a/bin/varnishd/cache_esi_fetch.c
+++ b/bin/varnishd/cache_esi_fetch.c
@@ -51,7 +51,7 @@ vef_read(struct worker *w, struct http_conn *htc, void *buf, ssize_t buflen,
 
 	if (buflen < bytes)
 		bytes = buflen;
-	if (params->esi_syntax & 0x8) {
+	if (cache_param->esi_syntax & 0x8) {
 		d = (random() & 3) + 1;
 		if (d < bytes)
 			bytes = d;
@@ -96,7 +96,7 @@ vfp_esi_bytes_gu(struct worker *w, struct http_conn *htc, ssize_t bytes)
 {
 	struct vgz *vg;
 	ssize_t wl;
-	uint8_t	ibuf[params->gzip_stack_buffer];
+	uint8_t	ibuf[cache_param->gzip_stack_buffer];
 	int i;
 	size_t dl;
 	const void *dp;
@@ -207,7 +207,7 @@ static int
 vfp_esi_bytes_ug(struct worker *w, struct http_conn *htc, ssize_t bytes)
 {
 	ssize_t wl;
-	char ibuf[params->gzip_stack_buffer];
+	char ibuf[cache_param->gzip_stack_buffer];
 	struct vef_priv *vef;
 
 	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
@@ -244,8 +244,8 @@ static int
 vfp_esi_bytes_gg(struct worker *w, struct http_conn *htc, size_t bytes)
 {
 	ssize_t wl;
-	char ibuf[params->gzip_stack_buffer];
-	char ibuf2[params->gzip_stack_buffer];
+	char ibuf[cache_param->gzip_stack_buffer];
+	char ibuf2[cache_param->gzip_stack_buffer];
 	struct vef_priv *vef;
 	size_t dl;
 	const void *dp;
diff --git a/bin/varnishd/cache_esi_parse.c b/bin/varnishd/cache_esi_parse.c
index f6eb367..9e2b4f6 100644
--- a/bin/varnishd/cache_esi_parse.c
+++ b/bin/varnishd/cache_esi_parse.c
@@ -269,7 +269,7 @@ static void
 vep_emit_skip(const struct vep_state *vep, ssize_t l)
 {
 
-	if (params->esi_syntax & 0x20) {
+	if (cache_param->esi_syntax & 0x20) {
 		Debug("---> SKIP(%jd)\n", (intmax_t)l);
 	}
 	vep_emit_len(vep, l, VEC_S1, VEC_S2, VEC_S8);
@@ -280,7 +280,7 @@ vep_emit_verbatim(const struct vep_state *vep, ssize_t l, ssize_t l_crc)
 {
 	uint8_t buf[4];
 
-	if (params->esi_syntax & 0x20) {
+	if (cache_param->esi_syntax & 0x20) {
 		Debug("---> VERBATIM(%jd)\n", (intmax_t)l);
 	}
 	vep_emit_len(vep, l, VEC_V1, VEC_V2, VEC_V8);
@@ -585,7 +585,7 @@ VEP_Parse(const struct worker *w, const char *p, size_t l)
 		 */
 
 		if (vep->state == VEP_START) {
-			if (params->esi_syntax & 0x1)
+			if (cache_param->esi_syntax & 0x1)
 				vep->state = VEP_NEXTTAG;
 			else
 				vep->state = VEP_TESTXML;
@@ -618,7 +618,7 @@ VEP_Parse(const struct worker *w, const char *p, size_t l)
 		 */
 
 		} else if (vep->state == VEP_NOTMYTAG) {
-			if (params->esi_syntax & 0x2) {
+			if (cache_param->esi_syntax & 0x2) {
 				p++;
 				vep->state = VEP_NEXTTAG;
 			} else {
diff --git a/bin/varnishd/cache_expire.c b/bin/varnishd/cache_expire.c
index f7f779d..23e3fc6 100644
--- a/bin/varnishd/cache_expire.c
+++ b/bin/varnishd/cache_expire.c
@@ -113,7 +113,7 @@ EXP_Keep(const struct sess *sp, const struct object *o)
 {
 	double r;
 
-	r = (double)params->default_keep;
+	r = (double)cache_param->default_keep;
 	if (o->exp.keep > 0.)
 		r = o->exp.keep;
 	if (sp != NULL && sp->exp.keep > 0. && sp->exp.keep < r)
@@ -126,7 +126,7 @@ EXP_Grace(const struct sess *sp, const struct object *o)
 {
 	double r;
 
-	r = (double)params->default_grace;
+	r = (double)cache_param->default_grace;
 	if (o->exp.grace >= 0.)
 		r = o->exp.grace;
 	if (sp != NULL && sp->exp.grace > 0. && sp->exp.grace < r)
@@ -344,7 +344,7 @@ exp_timer(struct sess *sp, void *priv)
 		if (oc == NULL) {
 			WSL_Flush(sp->wrk, 0);
 			WRK_SumStat(sp->wrk);
-			VTIM_sleep(params->expiry_sleep);
+			VTIM_sleep(cache_param->expiry_sleep);
 			t = VTIM_real();
 		}
 
diff --git a/bin/varnishd/cache_fetch.c b/bin/varnishd/cache_fetch.c
index 67bf970..a678dcc 100644
--- a/bin/varnishd/cache_fetch.c
+++ b/bin/varnishd/cache_fetch.c
@@ -188,7 +188,7 @@ FetchStorage(struct worker *w, ssize_t sz)
 	if (l == 0)
 		l = sz;
 	if (l == 0)
-		l = params->fetch_chunksize * 1024LL;
+		l = cache_param->fetch_chunksize * 1024LL;
 	st = STV_alloc(w, l);
 	if (st == NULL) {
 		(void)FetchError(w, "Could not get storage");
@@ -437,8 +437,8 @@ FetchHdr(struct sess *sp)
 
 	/* Receive response */
 
-	HTC_Init(w->htc, w->ws, vc->fd, vc->vsl_id, params->http_resp_size,
-	    params->http_resp_hdr_len);
+	HTC_Init(w->htc, w->ws, vc->fd, vc->vsl_id, cache_param->http_resp_size,
+	    cache_param->http_resp_hdr_len);
 
 	VTCP_set_read_timeout(vc->fd, vc->first_byte_timeout);
 
diff --git a/bin/varnishd/cache_gzip.c b/bin/varnishd/cache_gzip.c
index 6da1595..32a7413 100644
--- a/bin/varnishd/cache_gzip.c
+++ b/bin/varnishd/cache_gzip.c
@@ -131,7 +131,7 @@ vgz_alloc_vgz(struct worker *wrk, const char *id)
 	vg->wrk = wrk;
 	vg->id = id;
 
-	switch (params->gzip_tmp_space) {
+	switch (cache_param->gzip_tmp_space) {
 	case 0:
 	case 1:
 		/* malloc, the default */
@@ -196,10 +196,10 @@ VGZ_NewGzip(struct worker *wrk, const char *id)
 	 * XXX: too many worker threads grow the stacks.
 	 */
 	i = deflateInit2(&vg->vz,
-	    params->gzip_level,		/* Level */
+	    cache_param->gzip_level,		/* Level */
 	    Z_DEFLATED,			/* Method */
-	    16 + params->gzip_window,	/* Window bits (16=gzip + 15) */
-	    params->gzip_memlevel,	/* memLevel */
+	    16 + cache_param->gzip_window,	/* Window bits (16=gzip + 15) */
+	    cache_param->gzip_memlevel,	/* memLevel */
 	    Z_DEFAULT_STRATEGY);
 	assert(Z_OK == i);
 	return (vg);
@@ -467,7 +467,7 @@ vfp_gunzip_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
 	struct vgz *vg;
 	ssize_t l, wl;
 	int i = -100;
-	uint8_t	ibuf[params->gzip_stack_buffer];
+	uint8_t	ibuf[cache_param->gzip_stack_buffer];
 	size_t dl;
 	const void *dp;
 
@@ -545,7 +545,7 @@ vfp_gzip_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
 	struct vgz *vg;
 	ssize_t l, wl;
 	int i = -100;
-	uint8_t ibuf[params->gzip_stack_buffer];
+	uint8_t ibuf[cache_param->gzip_stack_buffer];
 	size_t dl;
 	const void *dp;
 
@@ -632,7 +632,7 @@ vfp_testgzip_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
 	struct vgz *vg;
 	ssize_t l, wl;
 	int i = -100;
-	uint8_t	obuf[params->gzip_stack_buffer];
+	uint8_t	obuf[cache_param->gzip_stack_buffer];
 	size_t dl;
 	const void *dp;
 	struct storage *st;
diff --git a/bin/varnishd/cache_hash.c b/bin/varnishd/cache_hash.c
index ecce40d..669fd40 100644
--- a/bin/varnishd/cache_hash.c
+++ b/bin/varnishd/cache_hash.c
@@ -169,7 +169,7 @@ HSH_AddString(const struct sess *sp, const char *str)
 	SHA256_Update(sp->wrk->sha256ctx, str, l);
 	SHA256_Update(sp->wrk->sha256ctx, "#", 1);
 
-	if (params->log_hash)
+	if (cache_param->log_hash)
 		WSP(sp, SLT_Hash, "%s", str);
 }
 
@@ -267,7 +267,7 @@ HSH_Insert(const struct sess *sp)
 	w = sp->wrk;
 
 	HSH_Prealloc(sp);
-	if (params->diag_bitmap & 0x80000000)
+	if (cache_param->diag_bitmap & 0x80000000)
 		hsh_testmagic(sp->wrk->nobjhead->digest);
 
 	AZ(sp->hash_objhead);
@@ -316,7 +316,7 @@ HSH_Lookup(struct sess *sp, struct objhead **poh)
 
 	HSH_Prealloc(sp);
 	memcpy(sp->wrk->nobjhead->digest, sp->digest, sizeof sp->digest);
-	if (params->diag_bitmap & 0x80000000)
+	if (cache_param->diag_bitmap & 0x80000000)
 		hsh_testmagic(sp->wrk->nobjhead->digest);
 
 	if (sp->hash_objhead != NULL) {
@@ -436,7 +436,7 @@ HSH_Lookup(struct sess *sp, struct objhead **poh)
 			}
 			VTAILQ_INSERT_TAIL(&oh->waitinglist->list, sp, list);
 		}
-		if (params->diag_bitmap & 0x20)
+		if (cache_param->diag_bitmap & 0x20)
 			WSP(sp, SLT_Debug,
 				"on waiting list <%p>", oh);
 		SES_Charge(sp);
@@ -492,7 +492,7 @@ hsh_rush(struct objhead *oh)
 	Lck_AssertHeld(&oh->mtx);
 	wl = oh->waitinglist;
 	CHECK_OBJ_NOTNULL(wl, WAITINGLIST_MAGIC);
-	for (u = 0; u < params->rush_exponent; u++) {
+	for (u = 0; u < cache_param->rush_exponent; u++) {
 		sp = VTAILQ_FIRST(&wl->list);
 		if (sp == NULL)
 			break;
@@ -616,7 +616,7 @@ HSH_Unbusy(const struct sess *sp)
 	assert(oh->refcnt > 0);
 	if (o->ws_o->overflow)
 		sp->wrk->stats.n_objoverflow++;
-	if (params->diag_bitmap & 0x40)
+	if (cache_param->diag_bitmap & 0x40)
 		WSP(sp, SLT_Debug,
 		    "Object %u workspace free %u", o->xid, WS_Free(o->ws_o));
 
diff --git a/bin/varnishd/cache_lck.c b/bin/varnishd/cache_lck.c
index ee93b9f..2aef6dc 100644
--- a/bin/varnishd/cache_lck.c
+++ b/bin/varnishd/cache_lck.c
@@ -64,7 +64,7 @@ Lck__Lock(struct lock *lck, const char *p, const char *f, int l)
 	int r;
 
 	CAST_OBJ_NOTNULL(ilck, lck->priv, ILCK_MAGIC);
-	if (!(params->diag_bitmap & 0x18)) {
+	if (!(cache_param->diag_bitmap & 0x18)) {
 		AZ(pthread_mutex_lock(&ilck->mtx));
 		AZ(ilck->held);
 		ilck->stat->locks++;
@@ -76,11 +76,11 @@ Lck__Lock(struct lock *lck, const char *p, const char *f, int l)
 	assert(r == 0 || r == EBUSY);
 	if (r) {
 		ilck->stat->colls++;
-		if (params->diag_bitmap & 0x8)
+		if (cache_param->diag_bitmap & 0x8)
 			VSL(SLT_Debug, 0, "MTX_CONTEST(%s,%s,%d,%s)",
 			    p, f, l, ilck->w);
 		AZ(pthread_mutex_lock(&ilck->mtx));
-	} else if (params->diag_bitmap & 0x8) {
+	} else if (cache_param->diag_bitmap & 0x8) {
 		VSL(SLT_Debug, 0, "MTX_LOCK(%s,%s,%d,%s)", p, f, l, ilck->w);
 	}
 	AZ(ilck->held);
@@ -99,7 +99,7 @@ Lck__Unlock(struct lock *lck, const char *p, const char *f, int l)
 	AN(ilck->held);
 	ilck->held = 0;
 	AZ(pthread_mutex_unlock(&ilck->mtx));
-	if (params->diag_bitmap & 0x8)
+	if (cache_param->diag_bitmap & 0x8)
 		VSL(SLT_Debug, 0, "MTX_UNLOCK(%s,%s,%d,%s)", p, f, l, ilck->w);
 }
 
@@ -112,7 +112,7 @@ Lck__Trylock(struct lock *lck, const char *p, const char *f, int l)
 	CAST_OBJ_NOTNULL(ilck, lck->priv, ILCK_MAGIC);
 	r = pthread_mutex_trylock(&ilck->mtx);
 	assert(r == 0 || r == EBUSY);
-	if (params->diag_bitmap & 0x8)
+	if (cache_param->diag_bitmap & 0x8)
 		VSL(SLT_Debug, 0,
 		    "MTX_TRYLOCK(%s,%s,%d,%s) = %d", p, f, l, ilck->w);
 	if (r == 0) {
diff --git a/bin/varnishd/cache_main.c b/bin/varnishd/cache_main.c
index c610bc3..5dc2d00 100644
--- a/bin/varnishd/cache_main.c
+++ b/bin/varnishd/cache_main.c
@@ -37,7 +37,7 @@
 #include "waiter/cache_waiter.h"
 #include "hash/hash_slinger.h"
 
-volatile struct params	*params;
+volatile struct params	*cache_param;
 
 /*--------------------------------------------------------------------
  * Per thread storage for the session currently being processed by
@@ -134,7 +134,7 @@ child_main(void)
 	BAN_Compile();
 
 	/* Wait for persistent storage to load if asked to */
-	if (params->diag_bitmap & 0x00020000)
+	if (cache_param->diag_bitmap & 0x00020000)
 		SMP_Ready();
 
 	CLI_Run();
diff --git a/bin/varnishd/cache_panic.c b/bin/varnishd/cache_panic.c
index 4076d26..f69e90b 100644
--- a/bin/varnishd/cache_panic.c
+++ b/bin/varnishd/cache_panic.c
@@ -346,7 +346,7 @@ pan_ic(const char *func, const char *file, int line, const char *cond,
 
 	pan_backtrace();
 
-	if (!(params->diag_bitmap & 0x2000)) {
+	if (!(cache_param->diag_bitmap & 0x2000)) {
 		sp = THR_GetSession();
 		if (sp != NULL)
 			pan_sess(sp);
@@ -354,11 +354,11 @@ pan_ic(const char *func, const char *file, int line, const char *cond,
 	VSB_printf(vsp, "\n");
 	VSB_bcat(vsp, "", 1);	/* NUL termination */
 
-	if (params->diag_bitmap & 0x4000)
+	if (cache_param->diag_bitmap & 0x4000)
 		(void)fputs(VSM_head->panicstr, stderr);
 
 #ifdef HAVE_ABORT2
-	if (params->diag_bitmap & 0x8000) {
+	if (cache_param->diag_bitmap & 0x8000) {
 		void *arg[1];
 		char *p;
 
@@ -369,7 +369,7 @@ pan_ic(const char *func, const char *file, int line, const char *cond,
 		abort2(VSM_head->panicstr, 1, arg);
 	}
 #endif
-	if (params->diag_bitmap & 0x1000)
+	if (cache_param->diag_bitmap & 0x1000)
 		exit(4);
 	else
 		abort();
diff --git a/bin/varnishd/cache_pipe.c b/bin/varnishd/cache_pipe.c
index be17ac9..4180d39 100644
--- a/bin/varnishd/cache_pipe.c
+++ b/bin/varnishd/cache_pipe.c
@@ -108,7 +108,7 @@ PipeSession(struct sess *sp)
 	while (fds[0].fd > -1 || fds[1].fd > -1) {
 		fds[0].revents = 0;
 		fds[1].revents = 0;
-		i = poll(fds, 2, params->pipe_timeout * 1000);
+		i = poll(fds, 2, cache_param->pipe_timeout * 1000);
 		if (i < 1)
 			break;
 		if (fds[0].revents && rdf(vc->fd, sp->fd)) {
diff --git a/bin/varnishd/cache_pool.c b/bin/varnishd/cache_pool.c
index aed596e..62d58aa 100644
--- a/bin/varnishd/cache_pool.c
+++ b/bin/varnishd/cache_pool.c
@@ -272,7 +272,7 @@ Pool_Work_Thread(void *priv, struct worker *w)
 			AZ(w->wrw.wfd);
 			AZ(w->storage_hint);
 			assert(w->wlp == w->wlb);
-			if (params->diag_bitmap & 0x00040000) {
+			if (cache_param->diag_bitmap & 0x00040000) {
 				if (w->vcl != NULL)
 					VCL_Rel(&w->vcl);
 			}
@@ -308,7 +308,7 @@ pool_queue(struct pool *pp, struct sess *sp)
 	}
 
 	/* If we have too much in the queue already, refuse. */
-	if (pp->lqueue > (params->queue_max * pp->nthr) / 100) {
+	if (pp->lqueue > (cache_param->queue_max * pp->nthr) / 100) {
 		pp->ndropped++;
 		Lck_Unlock(&pp->mtx);
 		return (-1);
@@ -385,10 +385,10 @@ pool_breed(struct pool *qp, const pthread_attr_t *tp_attr)
 	 * If we need more threads, and have space, create
 	 * one more thread.
 	 */
-	if (qp->nthr < params->wthread_min ||	/* Not enough threads yet */
-	    (qp->lqueue > params->wthread_add_threshold && /* more needed */
+	if (qp->nthr < cache_param->wthread_min ||	/* Not enough threads yet */
+	    (qp->lqueue > cache_param->wthread_add_threshold && /* more needed */
 	    qp->lqueue > qp->last_lqueue)) {	/* not getting better since last */
-		if (qp->nthr > params->wthread_max) {
+		if (qp->nthr > cache_param->wthread_max) {
 			Lck_Lock(&pool_mtx);
 			VSC_C_main->threads_limited++;
 			Lck_Unlock(&pool_mtx);
@@ -398,10 +398,10 @@ pool_breed(struct pool *qp, const pthread_attr_t *tp_attr)
 			Lck_Lock(&pool_mtx);
 			VSC_C_main->threads_limited++;
 			Lck_Unlock(&pool_mtx);
-			VTIM_sleep(params->wthread_fail_delay * 1e-3);
+			VTIM_sleep(cache_param->wthread_fail_delay * 1e-3);
 		} else {
 			AZ(pthread_detach(tp));
-			VTIM_sleep(params->wthread_add_delay * 1e-3);
+			VTIM_sleep(cache_param->wthread_add_delay * 1e-3);
 			qp->nthr++;
 			Lck_Lock(&pool_mtx);
 			VSC_C_main->threads++;
@@ -442,9 +442,9 @@ pool_herder(void *priv)
 
 	while (1) {
 		/* Set the stacksize for worker threads we create */
-		if (params->wthread_stacksize != UINT_MAX)
+		if (cache_param->wthread_stacksize != UINT_MAX)
 			AZ(pthread_attr_setstacksize(&tp_attr,
-			    params->wthread_stacksize));
+			    cache_param->wthread_stacksize));
 		else {
 			AZ(pthread_attr_destroy(&tp_attr));
 			AZ(pthread_attr_init(&tp_attr));
@@ -452,13 +452,13 @@ pool_herder(void *priv)
 
 		pool_breed(pp, &tp_attr);
 
-		if (pp->nthr < params->wthread_min)
+		if (pp->nthr < cache_param->wthread_min)
 			continue;
 
 		AZ(clock_gettime(CLOCK_MONOTONIC, &ts));
-		ts.tv_sec += params->wthread_purge_delay / 1000;
+		ts.tv_sec += cache_param->wthread_purge_delay / 1000;
 		ts.tv_nsec +=
-		    (params->wthread_purge_delay % 1000) * 1000000;
+		    (cache_param->wthread_purge_delay % 1000) * 1000000;
 		if (ts.tv_nsec >= 1000000000) {
 			ts.tv_sec++;
 			ts.tv_nsec -= 1000000000;
@@ -470,10 +470,10 @@ pool_herder(void *priv)
 		if (!i)
 			continue;
 
-		if (pp->nthr <= params->wthread_min)
+		if (pp->nthr <= cache_param->wthread_min)
 			continue;
 
-		t_idle = VTIM_real() - params->wthread_timeout;
+		t_idle = VTIM_real() - cache_param->wthread_timeout;
 
 		Lck_Lock(&pp->mtx);
 		VSC_C_main->sess_queued += pp->nqueued;
@@ -481,7 +481,7 @@ pool_herder(void *priv)
 		pp->nqueued = pp->ndropped = 0;
 		w = VTAILQ_LAST(&pp->idle, workerhead);
 		if (w != NULL &&
-		    (w->lastused < t_idle || pp->nthr > params->wthread_max)) {
+		    (w->lastused < t_idle || pp->nthr > cache_param->wthread_max)) {
 			VTAILQ_REMOVE(&pp->idle, w, list);
 		} else
 			w = NULL;
@@ -560,7 +560,7 @@ pool_poolherder(void *priv)
 
 	nwq = 0;
 	while (1) {
-		if (nwq < params->wthread_pools) {
+		if (nwq < cache_param->wthread_pools) {
 			pp = pool_mkpool();
 			if (pp != NULL) {
 				VTAILQ_INSERT_TAIL(&pools, pp, list);
diff --git a/bin/varnishd/cache_response.c b/bin/varnishd/cache_response.c
index 87b0919..487a514 100644
--- a/bin/varnishd/cache_response.c
+++ b/bin/varnishd/cache_response.c
@@ -118,7 +118,7 @@ RES_BuildHttp(const struct sess *sp)
 
 	if (!(sp->wrk->res_mode & RES_LEN)) {
 		http_Unset(sp->wrk->resp, H_Content_Length);
-	} else if (params->http_range_support) {
+	} else if (cache_param->http_range_support) {
 		/* We only accept ranges if we know the length */
 		http_SetHeader(sp->wrk, sp->vsl_id, sp->wrk->resp,
 		    "Accept-Ranges: bytes");
@@ -156,7 +156,7 @@ res_WriteGunzipObj(const struct sess *sp)
 	struct storage *st;
 	unsigned u = 0;
 	struct vgz *vg;
-	char obuf[params->gzip_stack_buffer];
+	char obuf[cache_param->gzip_stack_buffer];
 	ssize_t obufl = 0;
 	int i;
 
@@ -230,7 +230,7 @@ res_WriteDirObj(const struct sess *sp, ssize_t low, ssize_t high)
 		 * XXX: Should use getpagesize() ?
 		 */
 		if (st->fd >= 0 &&
-		    st->len >= params->sendfile_threshold) {
+		    st->len >= cache_param->sendfile_threshold) {
 			VSC_C_main->n_objsendfile++;
 			WRW_Sendfile(sp->wrk, st->fd, st->where + off, len);
 			continue;
@@ -275,7 +275,7 @@ RES_WriteObj(struct sess *sp)
 	    sp->wantbody &&
 	    (sp->wrk->res_mode & RES_LEN) &&
 	    !(sp->wrk->res_mode & (RES_ESI|RES_ESI_CHILD|RES_GUNZIP)) &&
-	    params->http_range_support &&
+	    cache_param->http_range_support &&
 	    sp->obj->response == 200 &&
 	    http_GetHdr(sp->http, H_Range, &r))
 		res_dorange(sp, r, &low, &high);
diff --git a/bin/varnishd/cache_session.c b/bin/varnishd/cache_session.c
index 5db418c..7befbcc 100644
--- a/bin/varnishd/cache_session.c
+++ b/bin/varnishd/cache_session.c
@@ -107,8 +107,8 @@ ses_sm_alloc(void)
 	 * cache them locally, to make sure we get a consistent
 	 * view of the value.
 	 */
-	nws = params->sess_workspace;
-	nhttp = (uint16_t)params->http_max_hdr;
+	nws = cache_param->sess_workspace;
+	nhttp = (uint16_t)cache_param->http_max_hdr;
 
 	hl = HTTP_estimate(nhttp);
 	l = sizeof *sm + nws + 2 * hl;
@@ -192,7 +192,7 @@ SES_New(struct worker *wrk, struct sesspool *pp)
 	sm = VTAILQ_FIRST(&pp->freelist);
 	if (sm != NULL) {
 		VTAILQ_REMOVE(&pp->freelist, sm, list);
-	} else if (pp->nsess < params->max_sess) {
+	} else if (pp->nsess < cache_param->max_sess) {
 		pp->nsess++;
 		do_alloc = 1;
 	}
@@ -355,9 +355,9 @@ SES_Delete(struct sess *sp, const char *reason)
 	    b->sess, b->req, b->pipe, b->pass,
 	    b->fetch, b->hdrbytes, b->bodybytes);
 
-	if (sm->workspace != params->sess_workspace ||
-	    sm->nhttp != (uint16_t)params->http_max_hdr ||
-	    pp->nsess > params->max_sess) {
+	if (sm->workspace != cache_param->sess_workspace ||
+	    sm->nhttp != (uint16_t)cache_param->http_max_hdr ||
+	    pp->nsess > cache_param->max_sess) {
 		free(sm);
 		Lck_Lock(&pp->mtx);
 		if (wrk != NULL)
diff --git a/bin/varnishd/cache_shmlog.c b/bin/varnishd/cache_shmlog.c
index 5601463..1252fa3 100644
--- a/bin/varnishd/cache_shmlog.c
+++ b/bin/varnishd/cache_shmlog.c
@@ -135,7 +135,7 @@ VSLR(enum VSL_tag_e tag, int id, const char *b, unsigned len)
 	uint32_t *p;
 	unsigned mlen;
 
-	mlen = params->shm_reclen;
+	mlen = cache_param->shm_reclen;
 
 	/* Truncate */
 	if (len > mlen)
@@ -153,7 +153,7 @@ void
 VSL(enum VSL_tag_e tag, int id, const char *fmt, ...)
 {
 	va_list ap;
-	unsigned n, mlen = params->shm_reclen;
+	unsigned n, mlen = cache_param->shm_reclen;
 	char buf[mlen];
 
 	/*
@@ -205,7 +205,7 @@ WSLR(struct worker *w, enum VSL_tag_e tag, int id, txt t)
 	unsigned l, mlen;
 
 	Tcheck(t);
-	mlen = params->shm_reclen;
+	mlen = cache_param->shm_reclen;
 
 	/* Truncate */
 	l = Tlen(t);
@@ -225,7 +225,7 @@ WSLR(struct worker *w, enum VSL_tag_e tag, int id, txt t)
 	w->wlp = VSL_END(w->wlp, l);
 	assert(w->wlp < w->wle);
 	w->wlr++;
-	if (params->diag_bitmap & 0x10000)
+	if (cache_param->diag_bitmap & 0x10000)
 		WSL_Flush(w, 0);
 }
 
@@ -239,7 +239,7 @@ wsl(struct worker *w, enum VSL_tag_e tag, int id, const char *fmt, va_list ap)
 	txt t;
 
 	AN(fmt);
-	mlen = params->shm_reclen;
+	mlen = cache_param->shm_reclen;
 
 	if (strchr(fmt, '%') == NULL) {
 		t.b = TRUST_ME(fmt);
@@ -261,7 +261,7 @@ wsl(struct worker *w, enum VSL_tag_e tag, int id, const char *fmt, va_list ap)
 		assert(w->wlp < w->wle);
 		w->wlr++;
 	}
-	if (params->diag_bitmap & 0x10000)
+	if (cache_param->diag_bitmap & 0x10000)
 		WSL_Flush(w, 0);
 }
 
diff --git a/bin/varnishd/cache_vary.c b/bin/varnishd/cache_vary.c
index c528fb8..026f937 100644
--- a/bin/varnishd/cache_vary.c
+++ b/bin/varnishd/cache_vary.c
@@ -155,7 +155,7 @@ vry_cmp(const uint8_t * const *v1, uint8_t * const *v2)
 	} else if (memcmp((*v1) + 2, (*v2) + 2, (*v1)[2] + 2)) {
 		/* Different header */
 		retval = 1;
-	} else if (params->http_gzip_support &&
+	} else if (cache_param->http_gzip_support &&
 	    !strcasecmp(H_Accept_Encoding, (const char*)((*v1)+2))) {
 		/*
 		 * If we do gzip processing, we do not vary on Accept-Encoding,
diff --git a/bin/varnishd/cache_vrt.c b/bin/varnishd/cache_vrt.c
index 16258ec..3851de7 100644
--- a/bin/varnishd/cache_vrt.c
+++ b/bin/varnishd/cache_vrt.c
@@ -73,7 +73,7 @@ VRT_count(const struct sess *sp, unsigned u)
 	if (sp == NULL)
 		return;
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	if (params->vcl_trace)
+	if (cache_param->vcl_trace)
 		WSP(sp, SLT_VCL_trace, "%u %d.%d", u,
 		    sp->vcl->ref[u].line, sp->vcl->ref[u].pos);
 }
diff --git a/bin/varnishd/cache_vrt_re.c b/bin/varnishd/cache_vrt_re.c
index d6dc5d5..7759d0a 100644
--- a/bin/varnishd/cache_vrt_re.c
+++ b/bin/varnishd/cache_vrt_re.c
@@ -71,7 +71,7 @@ VRT_re_match(const struct sess *sp, const char *s, void *re)
 		s = "";
 	AN(re);
 	t = re;
-	i = VRE_exec(t, s, strlen(s), 0, 0, NULL, 0, &params->vre_limits);
+	i = VRE_exec(t, s, strlen(s), 0, 0, NULL, 0, &cache_param->vre_limits);
 	if (i >= 0)
 		return (1);
 	if (i < VRE_ERROR_NOMATCH )
@@ -100,7 +100,7 @@ VRT_regsub(const struct sess *sp, int all, const char *str, void *re,
 	memset(ovector, 0, sizeof(ovector));
 	len = strlen(str);
 	i = VRE_exec(t, str, len, 0, options, ovector, 30,
-	    &params->vre_limits);
+	    &cache_param->vre_limits);
 
 	/* If it didn't match, we can return the original string */
 	if (i == VRE_ERROR_NOMATCH)
@@ -141,7 +141,7 @@ VRT_regsub(const struct sess *sp, int all, const char *str, void *re,
 		memset(&ovector, 0, sizeof(ovector));
 		options |= VRE_NOTEMPTY_ATSTART;
 		i = VRE_exec(t, str, len, 0, options, ovector, 30,
-		    &params->vre_limits);
+		    &cache_param->vre_limits);
 		if (i < VRE_ERROR_NOMATCH ) {
 			WS_Release(sp->http->ws, 0);
 			WSP(sp, SLT_VCL_Error,
diff --git a/bin/varnishd/cache_wrk.c b/bin/varnishd/cache_wrk.c
index e416eb9..bfee84c 100644
--- a/bin/varnishd/cache_wrk.c
+++ b/bin/varnishd/cache_wrk.c
@@ -185,15 +185,15 @@ WRK_thread(void *priv)
 	uint16_t nhttp;
 	unsigned siov;
 
-	assert(params->http_max_hdr <= 65535);
+	assert(cache_param->http_max_hdr <= 65535);
 	/* We need to snapshot these two for consistency */
-	nhttp = (uint16_t)params->http_max_hdr;
+	nhttp = (uint16_t)cache_param->http_max_hdr;
 	siov = nhttp * 2;
 	if (siov > IOV_MAX)
 		siov = IOV_MAX;
 	return (wrk_thread_real(priv,
-	    params->shm_workspace,
-	    params->wthread_workspace,
+	    cache_param->shm_workspace,
+	    cache_param->wthread_workspace,
 	    nhttp, HTTP_estimate(nhttp), siov));
 }
 
diff --git a/bin/varnishd/cache_wrw.c b/bin/varnishd/cache_wrw.c
index 3e2de4a..2160f69 100644
--- a/bin/varnishd/cache_wrw.c
+++ b/bin/varnishd/cache_wrw.c
@@ -134,7 +134,7 @@ WRW_Flush(struct worker *w)
 			*/
 			size_t used = 0;
 
-			if (VTIM_real() - w->sp->t_resp > params->send_timeout) {
+			if (VTIM_real() - w->sp->t_resp > cache_param->send_timeout) {
 				WSL(w, SLT_Debug, *wrw->wfd,
 				    "Hit total send timeout, wrote = %ld/%ld; not retrying",
 				    i, wrw->liov);
@@ -309,7 +309,7 @@ WRW_Sendfile(struct worker *w, int fd, off_t off, unsigned len)
 	} while (0);
 #elif defined(__sun) && defined(HAVE_SENDFILEV)
 	do {
-		sendfilevec_t svvec[params->http_headers * 2 + 1];
+		sendfilevec_t svvec[cache_param->http_headers * 2 + 1];
 		size_t xferred = 0, expected = 0;
 		int i;
 		for (i = 0; i < wrw->niov; i++) {
diff --git a/bin/varnishd/hash/hash_critbit.c b/bin/varnishd/hash/hash_critbit.c
index 56cc5c0..b1c9cf3 100644
--- a/bin/varnishd/hash/hash_critbit.c
+++ b/bin/varnishd/hash/hash_critbit.c
@@ -369,7 +369,7 @@ hcb_cleaner(void *priv)
 		VTAILQ_CONCAT(&dead_h, &cool_h, hoh_list);
 		Lck_Unlock(&hcb_mtx);
 		WRK_SumStat(&ww);
-		VTIM_sleep(params->critbit_cooloff);
+		VTIM_sleep(cache_param->critbit_cooloff);
 	}
 	NEEDLESS_RETURN(NULL);
 }
diff --git a/bin/varnishd/heritage.h b/bin/varnishd/heritage.h
index 3732f18..39fc03e 100644
--- a/bin/varnishd/heritage.h
+++ b/bin/varnishd/heritage.h
@@ -222,7 +222,7 @@ struct params {
  * We declare this a volatile pointer, so that reads of parameters
  * become atomic, leaving the CLI thread lattitude to change the values
  */
-extern volatile struct params * params;
+extern volatile struct params * cache_param;
 extern struct heritage heritage;
 
 void child_main(void);
diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c
index 9c6d37d..9503e5b 100644
--- a/bin/varnishd/mgt/mgt_child.c
+++ b/bin/varnishd/mgt/mgt_child.c
@@ -315,7 +315,7 @@ start_child(struct cli *cli)
 	heritage.std_fd = cp[1];
 	child_output = cp[0];
 
-	AN(params);
+	AN(cache_param);
 	if ((pid = fork()) < 0) {
 		perror("Could not fork child");
 		exit(1);
diff --git a/bin/varnishd/mgt/mgt_param.c b/bin/varnishd/mgt/mgt_param.c
index 20456d7..dc6b708 100644
--- a/bin/varnishd/mgt/mgt_param.c
+++ b/bin/varnishd/mgt/mgt_param.c
@@ -1074,8 +1074,8 @@ MCF_ParamSet(struct cli *cli, const char *param, const char *val)
 	}
 	pp->func(cli, pp, val);
 
-	if (cli->result == CLIS_OK && params != NULL)
-		*params = mgt_param;
+	if (cli->result == CLIS_OK && cache_param != NULL)
+		*cache_param = mgt_param;
 
 	if (cli->result != CLIS_OK) {
 		VCLI_Out(cli, "(attempting to set param %s to %s)\n",
diff --git a/bin/varnishd/mgt/mgt_shmem.c b/bin/varnishd/mgt/mgt_shmem.c
index 37083df..3953bdb 100644
--- a/bin/varnishd/mgt/mgt_shmem.c
+++ b/bin/varnishd/mgt/mgt_shmem.c
@@ -321,9 +321,9 @@ mgt_SHM_Init(const char *l_arg)
 	AN(VSC_C_main);
 
 	/* XXX: We need to zero params if we dealloc/clean/wash */
-	params = VSM_Alloc(sizeof *params, VSM_CLASS_PARAM, "", "");
-	AN(params);
-	*params = mgt_param;
+	cache_param = VSM_Alloc(sizeof *cache_param, VSM_CLASS_PARAM, "", "");
+	AN(cache_param);
+	*cache_param = mgt_param;
 
 	vsl_log_start = VSM_Alloc(s1, VSL_CLASS, "", "");
 	AN(vsl_log_start);
diff --git a/bin/varnishd/rfc2616.c b/bin/varnishd/rfc2616.c
index d4d5a05..4041f45 100644
--- a/bin/varnishd/rfc2616.c
+++ b/bin/varnishd/rfc2616.c
@@ -74,7 +74,7 @@ RFC2616_Ttl(const struct sess *sp)
 
 	assert(sp->wrk->exp.entered != 0.0 && !isnan(sp->wrk->exp.entered));
 	/* If all else fails, cache using default ttl */
-	sp->wrk->exp.ttl = params->default_ttl;
+	sp->wrk->exp.ttl = cache_param->default_ttl;
 
 	max_age = age = 0;
 	h_expires = 0;
@@ -140,7 +140,7 @@ RFC2616_Ttl(const struct sess *sp)
 		}
 
 		if (h_date == 0 ||
-		    fabs(h_date - sp->wrk->exp.entered) < params->clock_skew) {
+		    fabs(h_date - sp->wrk->exp.entered) < cache_param->clock_skew) {
 			/*
 			 * If we have no Date: header or if it is
 			 * sufficiently close to our clock we will
diff --git a/bin/varnishd/storage/stevedore.c b/bin/varnishd/storage/stevedore.c
index 4048836..b119c06 100644
--- a/bin/varnishd/storage/stevedore.c
+++ b/bin/varnishd/storage/stevedore.c
@@ -169,8 +169,8 @@ stv_alloc(struct worker *w, const struct object *obj, size_t size)
 	stv = obj->objstore->stevedore;
 	CHECK_OBJ_NOTNULL(stv, STEVEDORE_MAGIC);
 
-	if (size > (size_t)(params->fetch_maxchunksize) << 10)
-		size = (size_t)(params->fetch_maxchunksize) << 10;
+	if (size > (size_t)(cache_param->fetch_maxchunksize) << 10)
+		size = (size_t)(cache_param->fetch_maxchunksize) << 10;
 
 	for (;;) {
 		/* try to allocate from it */
@@ -179,7 +179,7 @@ stv_alloc(struct worker *w, const struct object *obj, size_t size)
 		if (st != NULL)
 			break;
 
-		if (size > params->fetch_chunksize * 1024LL) {
+		if (size > cache_param->fetch_chunksize * 1024LL) {
 			size >>= 1;
 			continue;
 		}
@@ -189,7 +189,7 @@ stv_alloc(struct worker *w, const struct object *obj, size_t size)
 			break;
 
 		/* Enough is enough: try another if we have one */
-		if (++fail >= params->nuke_limit)
+		if (++fail >= cache_param->nuke_limit)
 			break;
 	}
 	if (st != NULL)
@@ -336,7 +336,7 @@ STV_NewObject(struct sess *sp, const char *hint, unsigned wsl, struct exp *ep,
 	}
 	if (o == NULL) {
 		/* no luck; try to free some space and keep trying */
-		for (i = 0; o == NULL && i < params->nuke_limit; i++) {
+		for (i = 0; o == NULL && i < cache_param->nuke_limit; i++) {
 			if (EXP_NukeOne(sp->wrk, stv->lru) == -1)
 				break;
 			o = stv->allocobj(stv, sp, ltot, &soc);
diff --git a/bin/varnishd/waiter/cache_waiter_epoll.c b/bin/varnishd/waiter/cache_waiter_epoll.c
index e700676..b5068a6 100644
--- a/bin/varnishd/waiter/cache_waiter_epoll.c
+++ b/bin/varnishd/waiter/cache_waiter_epoll.c
@@ -188,7 +188,7 @@ vwe_thread(void *priv)
 			continue;
 
 		/* check for timeouts */
-		deadline = VTIM_real() - params->sess_timeout;
+		deadline = VTIM_real() - cache_param->sess_timeout;
 		for (;;) {
 			sp = VTAILQ_FIRST(&vwe->sesshead);
 			if (sp == NULL)
diff --git a/bin/varnishd/waiter/cache_waiter_kqueue.c b/bin/varnishd/waiter/cache_waiter_kqueue.c
index b300fae..a631606 100644
--- a/bin/varnishd/waiter/cache_waiter_kqueue.c
+++ b/bin/varnishd/waiter/cache_waiter_kqueue.c
@@ -186,7 +186,7 @@ vwk_thread(void *priv)
 		 * would not know we meant "the old fd of this number".
 		 */
 		vwk_kq_flush(vwk);
-		deadline = VTIM_real() - params->sess_timeout;
+		deadline = VTIM_real() - cache_param->sess_timeout;
 		for (;;) {
 			sp = VTAILQ_FIRST(&vwk->sesshead);
 			if (sp == NULL)
diff --git a/bin/varnishd/waiter/cache_waiter_poll.c b/bin/varnishd/waiter/cache_waiter_poll.c
index 2617365..5f8dbd7 100644
--- a/bin/varnishd/waiter/cache_waiter_poll.c
+++ b/bin/varnishd/waiter/cache_waiter_poll.c
@@ -140,7 +140,7 @@ vwp_main(void *priv)
 		assert(vwp->pollfd[vwp->pipes[1]].fd == -1);
 		v = poll(vwp->pollfd, vwp->hpoll + 1, 100);
 		assert(v >= 0);
-		deadline = VTIM_real() - params->sess_timeout;
+		deadline = VTIM_real() - cache_param->sess_timeout;
 		v2 = v;
 		VTAILQ_FOREACH_SAFE(sp, &vwp->sesshead, list, sp2) {
 			if (v != 0 && v2 == 0)
diff --git a/bin/varnishd/waiter/cache_waiter_ports.c b/bin/varnishd/waiter/cache_waiter_ports.c
index 022131b..aa9f86d 100644
--- a/bin/varnishd/waiter/cache_waiter_ports.c
+++ b/bin/varnishd/waiter/cache_waiter_ports.c
@@ -197,7 +197,7 @@ vws_thread(void *priv)
 
 		/* check for timeouts */
 		now = VTIM_real();
-		deadline = now - params->sess_timeout;
+		deadline = now - cache_param->sess_timeout;
 
 		/*
 		 * This loop assumes that the oldest sessions are always at the
@@ -225,7 +225,7 @@ vws_thread(void *priv)
 		 */
 
 		if (sp) {
-			double tmo = (sp->t_open + params->sess_timeout) - now;
+			double tmo = (sp->t_open + cache_param->sess_timeout) - now;
 
 			/* we should have removed all sps whose timeout has passed */
 			assert(tmo > 0.0);



From phk at varnish-cache.org  Thu Nov 10 10:08:42 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Thu, 10 Nov 2011 11:08:42 +0100
Subject: [master] a37d3cc Fix a struct member hit by a global name-space edit
Message-ID: <E1RORYo-0006P8-1E@project.varnish-software.com>

commit a37d3cc15b5fa9b5f381481a8c95581c3b29bf2c
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Nov 10 09:12:04 2011 +0000

    Fix a struct member hit by a global name-space edit

diff --git a/bin/varnishd/cache_cli.c b/bin/varnishd/cache_cli.c
index ac2afc1..1e09d23 100644
--- a/bin/varnishd/cache_cli.c
+++ b/bin/varnishd/cache_cli.c
@@ -102,7 +102,7 @@ CLI_Run(void)
 
 	add_check = 1;
 
-	AN(VCLS_AddFd(cls, heritage.cli_in, heritage.VCLI_Out, NULL, NULL));
+	AN(VCLS_AddFd(cls, heritage.cli_in, heritage.cli_out, NULL, NULL));
 
 	do {
 		i = VCLS_Poll(cls, -1);
diff --git a/bin/varnishd/heritage.h b/bin/varnishd/heritage.h
index 39fc03e..e6e6558 100644
--- a/bin/varnishd/heritage.h
+++ b/bin/varnishd/heritage.h
@@ -46,7 +46,7 @@ struct heritage {
 
 	/* Two pipe(2)'s for CLI connection between cache and mgt.  */
 	int				cli_in;
-	int				VCLI_Out;
+	int				cli_out;
 
 	/* File descriptor for stdout/stderr */
 	int				std_fd;
diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c
index 9503e5b..cfe0b2f 100644
--- a/bin/varnishd/mgt/mgt_child.c
+++ b/bin/varnishd/mgt/mgt_child.c
@@ -303,8 +303,8 @@ start_child(struct cli *cli)
 
 	/* Open pipe for child->mgr CLI */
 	AZ(pipe(cp));
-	heritage.VCLI_Out = cp[1];
-	mgt_child_inherit(heritage.VCLI_Out, "VCLI_Out");
+	heritage.cli_out = cp[1];
+	mgt_child_inherit(heritage.cli_out, "cli_out");
 	child_cli_in = cp[0];
 
 	/*
@@ -356,8 +356,8 @@ start_child(struct cli *cli)
 	mgt_child_inherit(heritage.cli_in, NULL);
 	closex(&heritage.cli_in);
 
-	mgt_child_inherit(heritage.VCLI_Out, NULL);
-	closex(&heritage.VCLI_Out);
+	mgt_child_inherit(heritage.cli_out, NULL);
+	closex(&heritage.cli_out);
 
 	close_sockets();
 



From phk at varnish-cache.org  Thu Nov 10 10:08:42 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Thu, 10 Nov 2011 11:08:42 +0100
Subject: [master] 3bf99f6 Firmly split manager and child views of hashing.
Message-ID: <E1RORYo-0006PJ-7G@project.varnish-software.com>

commit 3bf99f6c093d87b49cdec143138c98c3cb4e14f6
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Nov 10 09:33:39 2011 +0000

    Firmly split manager and child views of hashing.

diff --git a/bin/varnishd/Makefile.am b/bin/varnishd/Makefile.am
index c95b09a..f7c7620 100644
--- a/bin/varnishd/Makefile.am
+++ b/bin/varnishd/Makefile.am
@@ -54,6 +54,7 @@ varnishd_SOURCES = \
 	cache_ws.c \
 	hash/hash_classic.c \
 	hash/hash_critbit.c \
+	hash/hash_mgt.c \
 	hash/hash_simple_list.c \
 	mgt/mgt_child.c \
 	mgt/mgt_cli.c \
diff --git a/bin/varnishd/cache_hash.c b/bin/varnishd/cache_hash.c
index 669fd40..db865de 100644
--- a/bin/varnishd/cache_hash.c
+++ b/bin/varnishd/cache_hash.c
@@ -59,7 +59,6 @@
 #include "cache.h"
 
 #include "hash/hash_slinger.h"
-#include "vav.h"
 #include "vsha256.h"
 
 static const struct hash_slinger *hash;
@@ -743,53 +742,11 @@ HSH_Deref(struct worker *w, struct objcore *oc, struct object **oo)
 }
 
 void
-HSH_Init(void)
+HSH_Init(const struct hash_slinger *slinger)
 {
 
 	assert(DIGEST_LEN == SHA256_LEN);	/* avoid #include pollution */
-	hash = heritage.hash;
+	hash = slinger;
 	if (hash->start != NULL)
 		hash->start();
 }
-
-static const struct choice hsh_choice[] = {
-	{ "classic",		&hcl_slinger },
-	{ "simple",		&hsl_slinger },
-	{ "simple_list",	&hsl_slinger },	/* backwards compat */
-	{ "critbit",		&hcb_slinger },
-	{ NULL,			NULL }
-};
-
-/*--------------------------------------------------------------------*/
-
-void
-HSH_config(const char *h_arg)
-{
-	char **av;
-	int ac;
-	const struct hash_slinger *hp;
-
-	ASSERT_MGT();
-	av = VAV_Parse(h_arg, NULL, ARGV_COMMA);
-	AN(av);
-
-	if (av[0] != NULL)
-		ARGV_ERR("%s\n", av[0]);
-
-	if (av[1] == NULL)
-		ARGV_ERR("-h argument is empty\n");
-
-	for (ac = 0; av[ac + 2] != NULL; ac++)
-		continue;
-
-	hp = pick(hsh_choice, av[1], "hash");
-	CHECK_OBJ_NOTNULL(hp, SLINGER_MAGIC);
-	VSB_printf(vident, ",-h%s", av[1]);
-	heritage.hash = hp;
-	if (hp->init != NULL)
-		hp->init(ac, av + 2);
-	else if (ac > 0)
-		ARGV_ERR("Hash method \"%s\" takes no arguments\n",
-		    hp->name);
-}
-
diff --git a/bin/varnishd/cache_main.c b/bin/varnishd/cache_main.c
index 5dc2d00..f8f06e1 100644
--- a/bin/varnishd/cache_main.c
+++ b/bin/varnishd/cache_main.c
@@ -120,7 +120,7 @@ child_main(void)
 	Pool_Init();
 
 	EXP_Init();
-	HSH_Init();
+	HSH_Init(heritage.hash);
 	BAN_Init();
 
 	VCA_Init();
diff --git a/bin/varnishd/hash/hash_mgt.c b/bin/varnishd/hash/hash_mgt.c
new file mode 100644
index 0000000..6226749
--- /dev/null
+++ b/bin/varnishd/hash/hash_mgt.c
@@ -0,0 +1,84 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ */
+
+#include "config.h"
+
+#include <math.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+#include "mgt/mgt.h"
+#include "heritage.h"
+
+#include "hash/hash_slinger.h"
+#include "vav.h"
+
+static const struct choice hsh_choice[] = {
+	{ "classic",		&hcl_slinger },
+	{ "simple",		&hsl_slinger },
+	{ "simple_list",	&hsl_slinger },	/* backwards compat */
+	{ "critbit",		&hcb_slinger },
+	{ NULL,			NULL }
+};
+
+/*--------------------------------------------------------------------*/
+
+void
+HSH_config(const char *h_arg)
+{
+	char **av;
+	int ac;
+	const struct hash_slinger *hp;
+
+	ASSERT_MGT();
+	av = VAV_Parse(h_arg, NULL, ARGV_COMMA);
+	AN(av);
+
+	if (av[0] != NULL)
+		ARGV_ERR("%s\n", av[0]);
+
+	if (av[1] == NULL)
+		ARGV_ERR("-h argument is empty\n");
+
+	for (ac = 0; av[ac + 2] != NULL; ac++)
+		continue;
+
+	hp = pick(hsh_choice, av[1], "hash");
+	CHECK_OBJ_NOTNULL(hp, SLINGER_MAGIC);
+	VSB_printf(vident, ",-h%s", av[1]);
+	heritage.hash = hp;
+	if (hp->init != NULL)
+		hp->init(ac, av + 2);
+	else if (ac > 0)
+		ARGV_ERR("Hash method \"%s\" takes no arguments\n",
+		    hp->name);
+}
+
diff --git a/bin/varnishd/hash/hash_slinger.h b/bin/varnishd/hash/hash_slinger.h
index 2c142cf..23b892d 100644
--- a/bin/varnishd/hash/hash_slinger.h
+++ b/bin/varnishd/hash/hash_slinger.h
@@ -57,7 +57,7 @@ struct objcore *HSH_Lookup(struct sess *sp, struct objhead **poh);
 void HSH_Unbusy(const struct sess *sp);
 void HSH_Ref(struct objcore *o);
 void HSH_Drop(struct sess *sp);
-void HSH_Init(void);
+void HSH_Init(const struct hash_slinger *slinger);
 void HSH_AddString(const struct sess *sp, const char *str);
 struct objcore *HSH_Insert(const struct sess *sp);
 void HSH_Purge(const struct sess *, struct objhead *, double ttl, double grace);



From lkarsten at varnish-cache.org  Thu Nov 10 10:16:08 2011
From: lkarsten at varnish-cache.org (Lasse Karstensen)
Date: Thu, 10 Nov 2011 11:16:08 +0100
Subject: [master] 4805cac Clean up some formatting leading to error messages
	in rst2man that made it into the man pages
Message-ID: <E1RORg0-00072I-M2@project.varnish-software.com>

commit 4805cacd6166d8b7f57d61aa016ab61d0556b1ee
Author: Lasse Karstensen <lasse at varnish-software.com>
Date:   Thu Nov 10 11:00:45 2011 +0100

    Clean up some formatting leading to error messages in rst2man that made it into the man pages
    
    Conflicts:
    
    	bin/varnishd/mgt/mgt_param.c

diff --git a/bin/varnishd/mgt/mgt_param.c b/bin/varnishd/mgt/mgt_param.c
index e2eb16b..44f3b9b 100644
--- a/bin/varnishd/mgt/mgt_param.c
+++ b/bin/varnishd/mgt/mgt_param.c
@@ -711,6 +711,7 @@ static const struct parspec input_parspec[] = {
 		"  0x00000002 - Ignore non-esi elements\n"
 		"  0x00000004 - Emit parsing debug records\n"
 		"  0x00000008 - Force-split parser input (debugging)\n"
+		"\n"
 		"Use 0x notation and do the bitor in your head :-)\n",
 		0,
 		"0", "bitmap" },
@@ -843,6 +844,7 @@ static const struct parspec input_parspec[] = {
 		"  0x00040000 - release VCL early.\n"
 		"  0x00080000 - ban-lurker debugging.\n"
 		"  0x80000000 - do edge-detection on digest.\n"
+		"\n"
 		"Use 0x notation and do the bitor in your head :-)\n",
 		0,
 		"0", "bitmap" },
@@ -880,7 +882,7 @@ static const struct parspec input_parspec[] = {
 		"uncompress compressed objects on demand. Varnish will also "
 		"rewrite the Accept-Encoding header of clients indicating "
 		"support for gzip to:\n"
-		"Accept-Encoding: gzip\n\n"
+		"  Accept-Encoding: gzip\n\n"
 		"Clients that do not support gzip will have their "
 		"Accept-Encoding header removed. For more information on how "
 		"gzip is implemented please see the chapter on gzip in the "
@@ -891,6 +893,7 @@ static const struct parspec input_parspec[] = {
 		"Where temporary space for gzip/gunzip is allocated:\n"
 		"  0 - malloc\n"
 		"  2 - thread workspace\n"
+		"\n"
 		"If you have much gzip/gunzip activity, it may be an"
 		" advantage to use workspace for these allocations to reduce"
 		" malloc activity.  Be aware that gzip needs 256+KB and gunzip"
@@ -1186,6 +1189,7 @@ MCF_DumpRst(void)
 	const char *p, *q;
 	int i;
 
+	printf("\n.. The following is the autogenerated output from varnishd -x dumprst\n\n");
 	for (i = 0; i < nparspec; i++) {
 		pp = parspec[i];
 		printf("%s\n", pp->name);



From lkarsten at varnish-cache.org  Thu Nov 10 10:16:09 2011
From: lkarsten at varnish-cache.org (Lasse Karstensen)
Date: Thu, 10 Nov 2011 11:16:09 +0100
Subject: [master] f72bf05 Merge branch 'master' of
	ssh://git.varnish-cache.org/git/varnish-cache
Message-ID: <E1RORg1-00072O-Nu@project.varnish-software.com>

commit f72bf057df68e6b984f6c5aff87ebcc0eaf6f750
Merge: 4805cac 3bf99f6
Author: Lasse Karstensen <lasse at varnish-software.com>
Date:   Thu Nov 10 11:15:45 2011 +0100

    Merge branch 'master' of ssh://git.varnish-cache.org/git/varnish-cache



From apj at varnish-cache.org  Thu Nov 10 10:29:05 2011
From: apj at varnish-cache.org (Andreas Plesner Jacobsen)
Date: Thu, 10 Nov 2011 11:29:05 +0100
Subject: [master] 9b8598d Align error message with new parameter name
Message-ID: <E1RORsX-0007jN-47@project.varnish-software.com>

commit 9b8598d31d0218e5f8da8ff686a699970be8209e
Author: Andreas Plesner Jacobsen <apj at mutt.dk>
Date:   Thu Nov 10 11:13:52 2011 +0100

    Align error message with new parameter name

diff --git a/lib/libvarnishapi/vsl_arg.c b/lib/libvarnishapi/vsl_arg.c
index 56f93e2..4bcd44a 100644
--- a/lib/libvarnishapi/vsl_arg.c
+++ b/lib/libvarnishapi/vsl_arg.c
@@ -182,7 +182,7 @@ vsl_m_arg(const struct VSM_data *vd, const char *opt)
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
 
 	if (!strchr(opt, ':')) {
-		fprintf(stderr, "No : found in -o option %s\n", opt);
+		fprintf(stderr, "No : found in -m option %s\n", opt);
 		return (-1);
 	}
 



From apj at varnish-cache.org  Thu Nov 10 10:29:05 2011
From: apj at varnish-cache.org (Andreas Plesner Jacobsen)
Date: Thu, 10 Nov 2011 11:29:05 +0100
Subject: [master] 3ef111c Merge branch 'master' of
	ssh://git.varnish-cache.org/git/varnish-cache
Message-ID: <E1RORsX-0007jT-BP@project.varnish-software.com>

commit 3ef111cdef48a81e9a69da59764c5a796872cbc6
Merge: 9b8598d f72bf05
Author: Andreas Plesner Jacobsen <apj at mutt.dk>
Date:   Thu Nov 10 11:28:30 2011 +0100

    Merge branch 'master' of ssh://git.varnish-cache.org/git/varnish-cache



From phk at varnish-cache.org  Thu Nov 10 15:55:51 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Thu, 10 Nov 2011 16:55:51 +0100
Subject: [master] dd74a96 OCD patch:  return (x)
Message-ID: <E1ROWyl-0002q5-17@project.varnish-software.com>

commit dd74a963e2a1ea84275ff67aef905c33cc6edc39
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Nov 10 10:54:49 2011 +0000

    OCD patch:  return (x)

diff --git a/bin/varnishd/cache_backend.c b/bin/varnishd/cache_backend.c
index e9d0323..1f290c3 100644
--- a/bin/varnishd/cache_backend.c
+++ b/bin/varnishd/cache_backend.c
@@ -412,7 +412,7 @@ vdi_get_backend_if_simple(const struct director *d)
 
 	vs2 = d->priv;
 	if (vs2->magic != VDI_SIMPLE_MAGIC)
-		return NULL;
+		return (NULL);
 	CAST_OBJ_NOTNULL(vs, d->priv, VDI_SIMPLE_MAGIC);
 	return (vs->backend);
 }
diff --git a/bin/varnishd/cache_vrt.c b/bin/varnishd/cache_vrt.c
index 3851de7..c20b552 100644
--- a/bin/varnishd/cache_vrt.c
+++ b/bin/varnishd/cache_vrt.c
@@ -342,7 +342,7 @@ VRT_time_string(const struct sess *sp, double t)
 
 	AN(p = WS_Alloc(sp->http->ws, VTIM_FORMAT_SIZE));
 	VTIM_format(t, p);
-	return p;
+	return (p);
 }
 
 const char *
diff --git a/bin/varnishd/cache_vrt_var.c b/bin/varnishd/cache_vrt_var.c
index 92d8451..7a5aed0 100644
--- a/bin/varnishd/cache_vrt_var.c
+++ b/bin/varnishd/cache_vrt_var.c
@@ -485,9 +485,9 @@ VRT_r_server_identity(struct sess *sp)
 	(void)sp;
 
 	if (heritage.identity[0] != '\0')
-		return heritage.identity;
+		return (heritage.identity);
 	else
-		return heritage.name;
+		return (heritage.name);
 }
 
 
diff --git a/bin/varnishd/hash/hash_critbit.c b/bin/varnishd/hash/hash_critbit.c
index b1c9cf3..fd50b26 100644
--- a/bin/varnishd/hash/hash_critbit.c
+++ b/bin/varnishd/hash/hash_critbit.c
@@ -57,7 +57,7 @@ static unsigned char hcb_bittbl[256];
 static unsigned char
 hcb_bits(unsigned char x, unsigned char y)
 {
-	return hcb_bittbl[x ^ y];
+	return (hcb_bittbl[x ^ y]);
 }
 
 static void
diff --git a/bin/varnishd/waiter/cache_waiter_epoll.c b/bin/varnishd/waiter/cache_waiter_epoll.c
index b5068a6..46cb288 100644
--- a/bin/varnishd/waiter/cache_waiter_epoll.c
+++ b/bin/varnishd/waiter/cache_waiter_epoll.c
@@ -200,7 +200,7 @@ vwe_thread(void *priv)
 			SES_Delete(sp, "timeout");
 		}
 	}
-	return NULL;
+	return (NULL);
 }
 
 /*--------------------------------------------------------------------*/
@@ -219,7 +219,7 @@ vwe_sess_timeout_ticker(void *priv)
 		assert(write(vwe->timer_pipes[1], &ticker, 1));
 		VTIM_sleep(100 * 1e-3);
 	}
-	return NULL;
+	return (NULL);
 }
 
 /*--------------------------------------------------------------------*/
diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c
index 8d969eb..9765796 100644
--- a/bin/varnishncsa/varnishncsa.c
+++ b/bin/varnishncsa/varnishncsa.c
@@ -200,11 +200,11 @@ req_header(struct logline *l, const char *name)
 	struct hdr *h;
 	VTAILQ_FOREACH(h, &l->req_headers, list) {
 		if (strcasecmp(h->key, name) == 0) {
-			return h->value;
+			return (h->value);
 			break;
 		}
 	}
-	return NULL;
+	return (NULL);
 }
 
 static char *
@@ -213,11 +213,11 @@ resp_header(struct logline *l, const char *name)
 	struct hdr *h;
 	VTAILQ_FOREACH(h, &l->resp_headers, list) {
 		if (strcasecmp(h->key, name) == 0) {
-			return h->value;
+			return (h->value);
 			break;
 		}
 	}
-	return NULL;
+	return (NULL);
 }
 
 static char *
@@ -226,11 +226,11 @@ vcl_log(struct logline *l, const char *name)
 	struct hdr *h;
 	VTAILQ_FOREACH(h, &l->vcl_log, list) {
 		if (strcasecmp(h->key, name) == 0) {
-			return h->value;
+			return (h->value);
 			break;
 		}
 	}
-	return NULL;
+	return (NULL);
 }
 
 static void
diff --git a/lib/libvarnish/vtcp.c b/lib/libvarnish/vtcp.c
index 21de7f2..d15f516 100644
--- a/lib/libvarnish/vtcp.c
+++ b/lib/libvarnish/vtcp.c
@@ -61,11 +61,11 @@ VTCP_port(const struct sockaddr_storage *addr)
 
 	if (addr->ss_family == AF_INET) {
 		const struct sockaddr_in *ain = (const void *)addr;
-		return ntohs((ain->sin_port));
+		return (ntohs((ain->sin_port)));
 	}
 	if (addr->ss_family == AF_INET6) {
 		const struct sockaddr_in6 *ain = (const void *)addr;
-		return ntohs((ain->sin6_port));
+		return (ntohs((ain->sin6_port)));
 	}
 	return (-1);
 }
diff --git a/man/vsc2rst.c b/man/vsc2rst.c
index bf4091f..18250d8 100644
--- a/man/vsc2rst.c
+++ b/man/vsc2rst.c
@@ -64,6 +64,6 @@ int main(int argc, char **argv)
 #include "tbl/vsc_fields.h"
 #undef VSC_DO_VBE
 
-	return 0;
+	return (0);
 }
 



From phk at varnish-cache.org  Thu Nov 10 15:55:57 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Thu, 10 Nov 2011 16:55:57 +0100
Subject: [master] 6c290ee Merge branch 'master' of
	ssh://git.varnish-cache.org/git/varnish-cache
Message-ID: <E1ROWyr-0002qC-6M@project.varnish-software.com>

commit 6c290ee38a280d5c9726ce1b7e10006122429e22
Merge: dd74a96 3ef111c
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Nov 10 13:53:04 2011 +0000

    Merge branch 'master' of ssh://git.varnish-cache.org/git/varnish-cache



From phk at varnish-cache.org  Thu Nov 10 15:55:59 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Thu, 10 Nov 2011 16:55:59 +0100
Subject: [master] 921f14b Split params from heritage,
	they have been different beasts for a long time.
Message-ID: <E1ROWyt-0002rn-FK@project.varnish-software.com>

commit 921f14b2df75f9f285f24a1975d06a1876d82372
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Nov 10 15:55:22 2011 +0000

    Split params from heritage, they have been different beasts for a long time.

diff --git a/bin/varnishd/params.h b/bin/varnishd/params.h
new file mode 100644
index 0000000..e6e6558
--- /dev/null
+++ b/bin/varnishd/params.h
@@ -0,0 +1,228 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * This file contains the heritage passed when mgt forks cache
+ */
+
+#include "vre.h"
+
+struct listen_sock {
+	unsigned			magic;
+#define LISTEN_SOCK_MAGIC		0x999e4b57
+	VTAILQ_ENTRY(listen_sock)	list;
+	int				sock;
+	char				*name;
+	struct vss_addr			*addr;
+};
+
+VTAILQ_HEAD(listen_sock_head, listen_sock);
+
+struct heritage {
+
+	/* Two pipe(2)'s for CLI connection between cache and mgt.  */
+	int				cli_in;
+	int				cli_out;
+
+	/* File descriptor for stdout/stderr */
+	int				std_fd;
+
+	/* Sockets from which to accept connections */
+	struct listen_sock_head		socks;
+	unsigned			nsocks;
+
+	/* Hash method */
+	const struct hash_slinger	*hash;
+
+	char				*name;
+	char                            identity[1024];
+};
+
+struct params {
+
+	/* Unprivileged user / group */
+	char			*user;
+	uid_t			uid;
+	char			*group;
+	gid_t			gid;
+
+	/* TTL used for lack of anything better */
+	double			default_ttl;
+
+	/* Default grace period */
+	double			default_grace;
+
+	/* Default keep period */
+	double			default_keep;
+
+	/* Maximum concurrent sessions */
+	unsigned		max_sess;
+
+	/* Worker threads and pool */
+	unsigned		wthread_min;
+	unsigned		wthread_max;
+	unsigned		wthread_timeout;
+	unsigned		wthread_pools;
+	unsigned		wthread_add_threshold;
+	unsigned		wthread_add_delay;
+	unsigned		wthread_fail_delay;
+	unsigned		wthread_purge_delay;
+	unsigned		wthread_stats_rate;
+	unsigned		wthread_stacksize;
+	unsigned		wthread_workspace;
+
+	unsigned		queue_max;
+
+	/* Memory allocation hints */
+	unsigned		sess_workspace;
+	unsigned		shm_workspace;
+	unsigned		http_req_size;
+	unsigned		http_req_hdr_len;
+	unsigned		http_resp_size;
+	unsigned		http_resp_hdr_len;
+	unsigned		http_max_hdr;
+
+	unsigned		shm_reclen;
+
+	/* Acceptor hints */
+	unsigned		sess_timeout;
+	unsigned		pipe_timeout;
+	unsigned		send_timeout;
+	unsigned		idle_send_timeout;
+
+	/* Management hints */
+	unsigned		auto_restart;
+
+	/* Fetcher hints */
+	unsigned		fetch_chunksize;
+	unsigned		fetch_maxchunksize;
+	unsigned		nuke_limit;
+
+#ifdef SENDFILE_WORKS
+	/* Sendfile object minimum size */
+	unsigned		sendfile_threshold;
+#endif
+
+	/* VCL traces */
+	unsigned		vcl_trace;
+
+	/* Listen address */
+	char			*listen_address;
+
+	/* Listen depth */
+	unsigned		listen_depth;
+
+	/* CLI related */
+	unsigned		cli_timeout;
+	unsigned		ping_interval;
+
+	/* LRU list ordering interval */
+	unsigned		lru_timeout;
+
+	/* Maximum restarts allowed */
+	unsigned		max_restarts;
+
+	/* Maximum esi:include depth allowed */
+	unsigned		max_esi_depth;
+
+	/* ESI parser hints */
+	unsigned		esi_syntax;
+
+	/* Rush exponent */
+	unsigned		rush_exponent;
+
+	/* Default connection_timeout */
+	double			connect_timeout;
+
+	/* Read timeouts for backend */
+	double			first_byte_timeout;
+	double			between_bytes_timeout;
+
+	/* How long to linger on sessions */
+	unsigned		session_linger;
+
+	/* CLI buffer size */
+	unsigned		cli_buffer;
+
+	/* Control diagnostic code */
+	unsigned		diag_bitmap;
+
+	/* Log hash string to shm */
+	unsigned		log_hash;
+
+	/* Log local socket address to shm */
+	unsigned		log_local_addr;
+
+	/* Prefer IPv6 connections to backend*/
+	unsigned		prefer_ipv6;
+
+	/* Acceptable clockskew with backends */
+	unsigned		clock_skew;
+
+	/* Expiry pacer parameters */
+	double			expiry_sleep;
+
+	/* Acceptor pacer parameters */
+	double			acceptor_sleep_max;
+	double			acceptor_sleep_incr;
+	double			acceptor_sleep_decay;
+
+	/* Get rid of duplicate bans */
+	unsigned		ban_dups;
+
+	/* How long time does the ban lurker sleep */
+	double			ban_lurker_sleep;
+
+	/* Max size of the saintmode list. 0 == no saint mode. */
+	unsigned		saintmode_threshold;
+
+	unsigned		syslog_cli_traffic;
+
+	unsigned		http_range_support;
+
+	unsigned		http_gzip_support;
+	unsigned		gzip_stack_buffer;
+	unsigned		gzip_tmp_space;
+	unsigned		gzip_level;
+	unsigned		gzip_window;
+	unsigned		gzip_memlevel;
+
+	double			critbit_cooloff;
+
+	double			shortlived;
+
+	struct vre_limits	vre_limits;
+};
+
+/*
+ * We declare this a volatile pointer, so that reads of parameters
+ * become atomic, leaving the CLI thread lattitude to change the values
+ */
+extern volatile struct params * cache_param;
+extern struct heritage heritage;
+
+void child_main(void);



From apj at varnish-cache.org  Thu Nov 10 21:50:49 2011
From: apj at varnish-cache.org (Andreas Plesner Jacobsen)
Date: Thu, 10 Nov 2011 22:50:49 +0100
Subject: [master] 4f9e61d Explicitly document that concatenation is only
	supported for the builtins.
Message-ID: <E1ROcWH-0007IR-TO@project.varnish-software.com>

commit 4f9e61d652c42f9a7b8650fd4c2ecbb204db031a
Author: Andreas Plesner Jacobsen <apj at mutt.dk>
Date:   Thu Nov 10 22:49:57 2011 +0100

    Explicitly document that concatenation is only supported for the builtins.
    
    Fixes #1042

diff --git a/doc/sphinx/reference/vcl.rst b/doc/sphinx/reference/vcl.rst
index 743d5f2..016e84f 100644
--- a/doc/sphinx/reference/vcl.rst
+++ b/doc/sphinx/reference/vcl.rst
@@ -640,8 +640,11 @@ default code.
 
 Multiple subroutines
 ~~~~~~~~~~~~~~~~~~~~
-If multiple subroutines with the same name are defined, they are
-concatenated in the order in which the appear in the source.
+If multiple subroutines with the the name of one of the builtin
+ones are defined, they are concatenated in the order in which they
+appear in the source.
+The default versions distributed with Varnish will be implicitly
+concatenated as a last resort at the end.
 
 Example:::
 
@@ -667,8 +670,6 @@ Example:::
 	  }
 	}
 
-The builtin default subroutines are implicitly appended in this way.
-
 Variables
 ~~~~~~~~~
 



From tfheen at varnish-cache.org  Fri Nov 11 15:08:50 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Fri, 11 Nov 2011 16:08:50 +0100
Subject: [2.1] 5e1cf02 Fix missing time in beresp.ttl statement.
Message-ID: <E1ROsio-0006wy-I0@project.varnish-software.com>

commit 5e1cf02670e2fe008dbd98f1b451b0cb080646c1
Author: Gavin Sandie <beach at vicecity.co.uk>
Date:   Fri Nov 11 11:15:33 2011 +0000

    Fix missing time in beresp.ttl statement.

diff --git a/doc/sphinx/tutorial/vcl.rst b/doc/sphinx/tutorial/vcl.rst
index 7e5d4aa..133c54e 100644
--- a/doc/sphinx/tutorial/vcl.rst
+++ b/doc/sphinx/tutorial/vcl.rst
@@ -154,7 +154,7 @@ matches certain criteria:::
   sub vcl_fetch {
      if (req.url ~ "\.(png|gif|jpg)$") {
        unset beresp.http.set-cookie;
-       set beresp.ttl = 3600;
+       set beresp.ttl = 3600s;
     }
   }
 



From phk at varnish-cache.org  Sun Nov 13 09:30:16 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Sun, 13 Nov 2011 10:30:16 +0100
Subject: [master] 9699424 Split params.h from heritage.h,
	they are different beasts now. Reduce scope of heritage.h Reduce
	scope of libvcl.h
Message-ID: <E1RPWOG-00018G-Kc@project.varnish-software.com>

commit 9699424ccd9f7fc51a8d1c651889433fdd200ce5
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Nov 13 09:29:46 2011 +0000

    Split params.h from heritage.h, they are different beasts now.
    Reduce scope of heritage.h
    Reduce scope of libvcl.h

diff --git a/bin/varnishd/Makefile.am b/bin/varnishd/Makefile.am
index f7c7620..002447f 100644
--- a/bin/varnishd/Makefile.am
+++ b/bin/varnishd/Makefile.am
@@ -89,6 +89,7 @@ noinst_HEADERS = \
 	heritage.h \
 	mgt/mgt.h \
 	mgt/mgt_cli.h \
+	params.h \
 	storage/storage.h \
 	storage/storage_persistent.h \
 	vparam.h
diff --git a/bin/varnishd/cache.h b/bin/varnishd/cache.h
index 5941fcc..b21f2d4 100644
--- a/bin/varnishd/cache.h
+++ b/bin/varnishd/cache.h
@@ -55,7 +55,7 @@
 #endif
 
 
-#include "heritage.h"
+#include "params.h"
 
 enum body_status {
 #define BODYSTATUS(U,l)	BS_##U,
@@ -909,6 +909,7 @@ void VCL_Init(void);
 void VCL_Refresh(struct VCL_conf **vcc);
 void VCL_Rel(struct VCL_conf **vcc);
 void VCL_Poll(void);
+const char *VCL_Return_Name(unsigned method);
 
 #define VCL_MET_MAC(l,u,b) void VCL_##l##_method(struct sess *);
 #include "tbl/vcl_returns.h"
diff --git a/bin/varnishd/cache_acceptor.c b/bin/varnishd/cache_acceptor.c
index 550a53f..5161b09 100644
--- a/bin/varnishd/cache_acceptor.c
+++ b/bin/varnishd/cache_acceptor.c
@@ -31,6 +31,7 @@
 #include "config.h"
 
 #include "cache.h"
+#include "heritage.h"
 
 #include "vcli.h"
 #include "vcli_priv.h"
diff --git a/bin/varnishd/cache_cli.c b/bin/varnishd/cache_cli.c
index 1e09d23..0bbf94b 100644
--- a/bin/varnishd/cache_cli.c
+++ b/bin/varnishd/cache_cli.c
@@ -39,6 +39,7 @@
 #include <stddef.h>			// offsetof
 
 #include "cache.h"
+#include "heritage.h"
 
 #include "cache_backend.h"		// struct vbc
 #include "hash/hash_slinger.h"		// struct objhead
diff --git a/bin/varnishd/cache_main.c b/bin/varnishd/cache_main.c
index f8f06e1..7a3bcd9 100644
--- a/bin/varnishd/cache_main.c
+++ b/bin/varnishd/cache_main.c
@@ -33,6 +33,7 @@
 #include <stdlib.h>
 
 #include "cache.h"
+#include "heritage.h"
 
 #include "waiter/cache_waiter.h"
 #include "hash/hash_slinger.h"
diff --git a/bin/varnishd/cache_panic.c b/bin/varnishd/cache_panic.c
index f69e90b..c626ae3 100644
--- a/bin/varnishd/cache_panic.c
+++ b/bin/varnishd/cache_panic.c
@@ -44,7 +44,6 @@
 
 #include "cache_backend.h"
 #include "waiter/cache_waiter.h"
-#include "libvcl.h"
 #include "vcl.h"
 
 /*
@@ -226,7 +225,7 @@ pan_sess(const struct sess *sp)
 #undef STEP
 		default: stp = NULL;
 	}
-	hand = VCC_Return_Name(sp->handling);
+	hand = VCL_Return_Name(sp->handling);
 	if (stp != NULL)
 		VSB_printf(vsp, "  step = %s,\n", stp);
 	else
diff --git a/bin/varnishd/cache_pool.c b/bin/varnishd/cache_pool.c
index 62d58aa..eaf7602 100644
--- a/bin/varnishd/cache_pool.c
+++ b/bin/varnishd/cache_pool.c
@@ -46,6 +46,7 @@
 #include <stdlib.h>
 
 #include "cache.h"
+#include "heritage.h"
 
 #include "waiter/cache_waiter.h"
 #include "vtcp.h"
diff --git a/bin/varnishd/cache_vcl.c b/bin/varnishd/cache_vcl.c
index cc67a7f..068b482 100644
--- a/bin/varnishd/cache_vcl.c
+++ b/bin/varnishd/cache_vcl.c
@@ -38,7 +38,6 @@
 
 #include "cache.h"
 
-#include "libvcl.h"
 #include "vcl.h"
 #include "vcli.h"
 #include "vcli_priv.h"
@@ -65,6 +64,21 @@ static struct vcls		*vcl_active; /* protected by vcl_mtx */
 
 /*--------------------------------------------------------------------*/
 
+const char *
+VCL_Return_Name(unsigned method)
+{
+
+	switch (method) {
+#define VCL_RET_MAC(l, U, B) case VCL_RET_##U: return(#l);
+#include "tbl/vcl_returns.h"
+#undef VCL_RET_MAC
+	default:
+		return (NULL);
+	}
+}
+
+/*--------------------------------------------------------------------*/
+
 static void
 VCL_Get(struct VCL_conf **vcc)
 {
@@ -323,7 +337,7 @@ VCL_##func##_method(struct sess *sp)					\
 	sp->cur_method = VCL_MET_ ## upper;				\
 	WSP(sp, SLT_VCL_call, "%s", #func);				\
 	(void)sp->vcl->func##_func(sp);					\
-	WSP(sp, SLT_VCL_return, "%s", VCC_Return_Name(sp->handling));	\
+	WSP(sp, SLT_VCL_return, "%s", VCL_Return_Name(sp->handling));	\
 	sp->cur_method = 0;						\
 	assert((1U << sp->handling) & bitmap);				\
 	assert(!((1U << sp->handling) & ~bitmap));			\
diff --git a/bin/varnishd/cache_vrt_var.c b/bin/varnishd/cache_vrt_var.c
index 7a5aed0..a9bf87f 100644
--- a/bin/varnishd/cache_vrt_var.c
+++ b/bin/varnishd/cache_vrt_var.c
@@ -34,6 +34,7 @@
 #include <stdlib.h>
 
 #include "cache.h"
+#include "heritage.h"
 
 #include "cache_backend.h"
 #include "vrt_obj.h"
diff --git a/bin/varnishd/heritage.h b/bin/varnishd/heritage.h
index e6e6558..36433bb 100644
--- a/bin/varnishd/heritage.h
+++ b/bin/varnishd/heritage.h
@@ -29,8 +29,6 @@
  * This file contains the heritage passed when mgt forks cache
  */
 
-#include "vre.h"
-
 struct listen_sock {
 	unsigned			magic;
 #define LISTEN_SOCK_MAGIC		0x999e4b57
@@ -62,167 +60,6 @@ struct heritage {
 	char                            identity[1024];
 };
 
-struct params {
-
-	/* Unprivileged user / group */
-	char			*user;
-	uid_t			uid;
-	char			*group;
-	gid_t			gid;
-
-	/* TTL used for lack of anything better */
-	double			default_ttl;
-
-	/* Default grace period */
-	double			default_grace;
-
-	/* Default keep period */
-	double			default_keep;
-
-	/* Maximum concurrent sessions */
-	unsigned		max_sess;
-
-	/* Worker threads and pool */
-	unsigned		wthread_min;
-	unsigned		wthread_max;
-	unsigned		wthread_timeout;
-	unsigned		wthread_pools;
-	unsigned		wthread_add_threshold;
-	unsigned		wthread_add_delay;
-	unsigned		wthread_fail_delay;
-	unsigned		wthread_purge_delay;
-	unsigned		wthread_stats_rate;
-	unsigned		wthread_stacksize;
-	unsigned		wthread_workspace;
-
-	unsigned		queue_max;
-
-	/* Memory allocation hints */
-	unsigned		sess_workspace;
-	unsigned		shm_workspace;
-	unsigned		http_req_size;
-	unsigned		http_req_hdr_len;
-	unsigned		http_resp_size;
-	unsigned		http_resp_hdr_len;
-	unsigned		http_max_hdr;
-
-	unsigned		shm_reclen;
-
-	/* Acceptor hints */
-	unsigned		sess_timeout;
-	unsigned		pipe_timeout;
-	unsigned		send_timeout;
-	unsigned		idle_send_timeout;
-
-	/* Management hints */
-	unsigned		auto_restart;
-
-	/* Fetcher hints */
-	unsigned		fetch_chunksize;
-	unsigned		fetch_maxchunksize;
-	unsigned		nuke_limit;
-
-#ifdef SENDFILE_WORKS
-	/* Sendfile object minimum size */
-	unsigned		sendfile_threshold;
-#endif
-
-	/* VCL traces */
-	unsigned		vcl_trace;
-
-	/* Listen address */
-	char			*listen_address;
-
-	/* Listen depth */
-	unsigned		listen_depth;
-
-	/* CLI related */
-	unsigned		cli_timeout;
-	unsigned		ping_interval;
-
-	/* LRU list ordering interval */
-	unsigned		lru_timeout;
-
-	/* Maximum restarts allowed */
-	unsigned		max_restarts;
-
-	/* Maximum esi:include depth allowed */
-	unsigned		max_esi_depth;
-
-	/* ESI parser hints */
-	unsigned		esi_syntax;
-
-	/* Rush exponent */
-	unsigned		rush_exponent;
-
-	/* Default connection_timeout */
-	double			connect_timeout;
-
-	/* Read timeouts for backend */
-	double			first_byte_timeout;
-	double			between_bytes_timeout;
-
-	/* How long to linger on sessions */
-	unsigned		session_linger;
-
-	/* CLI buffer size */
-	unsigned		cli_buffer;
-
-	/* Control diagnostic code */
-	unsigned		diag_bitmap;
-
-	/* Log hash string to shm */
-	unsigned		log_hash;
-
-	/* Log local socket address to shm */
-	unsigned		log_local_addr;
-
-	/* Prefer IPv6 connections to backend*/
-	unsigned		prefer_ipv6;
-
-	/* Acceptable clockskew with backends */
-	unsigned		clock_skew;
-
-	/* Expiry pacer parameters */
-	double			expiry_sleep;
-
-	/* Acceptor pacer parameters */
-	double			acceptor_sleep_max;
-	double			acceptor_sleep_incr;
-	double			acceptor_sleep_decay;
-
-	/* Get rid of duplicate bans */
-	unsigned		ban_dups;
-
-	/* How long time does the ban lurker sleep */
-	double			ban_lurker_sleep;
-
-	/* Max size of the saintmode list. 0 == no saint mode. */
-	unsigned		saintmode_threshold;
-
-	unsigned		syslog_cli_traffic;
-
-	unsigned		http_range_support;
-
-	unsigned		http_gzip_support;
-	unsigned		gzip_stack_buffer;
-	unsigned		gzip_tmp_space;
-	unsigned		gzip_level;
-	unsigned		gzip_window;
-	unsigned		gzip_memlevel;
-
-	double			critbit_cooloff;
-
-	double			shortlived;
-
-	struct vre_limits	vre_limits;
-};
-
-/*
- * We declare this a volatile pointer, so that reads of parameters
- * become atomic, leaving the CLI thread lattitude to change the values
- */
-extern volatile struct params * cache_param;
 extern struct heritage heritage;
 
 void child_main(void);
diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c
index cfe0b2f..d2e0abd 100644
--- a/bin/varnishd/mgt/mgt_child.c
+++ b/bin/varnishd/mgt/mgt_child.c
@@ -44,8 +44,9 @@
 #include <unistd.h>
 
 #include "mgt/mgt.h"
-
 #include "heritage.h"
+#include "params.h"
+
 #include "vapi/vsm_int.h"
 #include "vbm.h"
 #include "vcli.h"
diff --git a/bin/varnishd/mgt/mgt_cli.c b/bin/varnishd/mgt/mgt_cli.c
index ec8fd4d..3fdb432 100644
--- a/bin/varnishd/mgt/mgt_cli.c
+++ b/bin/varnishd/mgt/mgt_cli.c
@@ -43,8 +43,9 @@
 #include <unistd.h>
 
 #include "mgt/mgt.h"
-
 #include "heritage.h"
+#include "params.h"
+
 #include "vcli.h"
 #include "vcli_common.h"
 #include "vcli_priv.h"
diff --git a/bin/varnishd/mgt/mgt_param.c b/bin/varnishd/mgt/mgt_param.c
index dd898ee..cd96e8b 100644
--- a/bin/varnishd/mgt/mgt_param.c
+++ b/bin/varnishd/mgt/mgt_param.c
@@ -38,9 +38,10 @@
 #include <unistd.h>
 
 #include "mgt/mgt.h"
+#include "heritage.h"
+#include "params.h"
 
 #include "waiter/cache_waiter.h"
-#include "heritage.h"
 #include "vav.h"
 #include "vcli.h"
 #include "vcli_common.h"
diff --git a/bin/varnishd/mgt/mgt_pool.c b/bin/varnishd/mgt/mgt_pool.c
index 94459b4..548ad78 100644
--- a/bin/varnishd/mgt/mgt_pool.c
+++ b/bin/varnishd/mgt/mgt_pool.c
@@ -48,8 +48,9 @@
 #include <unistd.h>
 
 #include "mgt/mgt.h"
-
 #include "heritage.h"
+#include "params.h"
+
 #include "vparam.h"
 
 /*--------------------------------------------------------------------*/
diff --git a/bin/varnishd/mgt/mgt_sandbox.c b/bin/varnishd/mgt/mgt_sandbox.c
index e5fe99c..a9fce93 100644
--- a/bin/varnishd/mgt/mgt_sandbox.c
+++ b/bin/varnishd/mgt/mgt_sandbox.c
@@ -53,8 +53,8 @@
 #include <unistd.h>
 
 #include "mgt/mgt.h"
-
 #include "heritage.h"
+#include "params.h"
 
 /*--------------------------------------------------------------------*/
 
diff --git a/bin/varnishd/mgt/mgt_shmem.c b/bin/varnishd/mgt/mgt_shmem.c
index 3953bdb..bb87a8f 100644
--- a/bin/varnishd/mgt/mgt_shmem.c
+++ b/bin/varnishd/mgt/mgt_shmem.c
@@ -98,9 +98,10 @@
 #include <unistd.h>
 
 #include "mgt/mgt.h"
+#include "heritage.h"
+#include "params.h"
 
 #include "flopen.h"
-#include "heritage.h"
 #include "vapi/vsc_int.h"
 #include "vapi/vsl_int.h"
 #include "vapi/vsm_int.h"
diff --git a/bin/varnishd/params.h b/bin/varnishd/params.h
index e6e6558..fb71e33 100644
--- a/bin/varnishd/params.h
+++ b/bin/varnishd/params.h
@@ -31,37 +31,6 @@
 
 #include "vre.h"
 
-struct listen_sock {
-	unsigned			magic;
-#define LISTEN_SOCK_MAGIC		0x999e4b57
-	VTAILQ_ENTRY(listen_sock)	list;
-	int				sock;
-	char				*name;
-	struct vss_addr			*addr;
-};
-
-VTAILQ_HEAD(listen_sock_head, listen_sock);
-
-struct heritage {
-
-	/* Two pipe(2)'s for CLI connection between cache and mgt.  */
-	int				cli_in;
-	int				cli_out;
-
-	/* File descriptor for stdout/stderr */
-	int				std_fd;
-
-	/* Sockets from which to accept connections */
-	struct listen_sock_head		socks;
-	unsigned			nsocks;
-
-	/* Hash method */
-	const struct hash_slinger	*hash;
-
-	char				*name;
-	char                            identity[1024];
-};
-
 struct params {
 
 	/* Unprivileged user / group */
@@ -223,6 +192,3 @@ struct params {
  * become atomic, leaving the CLI thread lattitude to change the values
  */
 extern volatile struct params * cache_param;
-extern struct heritage heritage;
-
-void child_main(void);
diff --git a/bin/varnishd/varnishd.c b/bin/varnishd/varnishd.c
index 6ae1842..ccda130 100644
--- a/bin/varnishd/varnishd.c
+++ b/bin/varnishd/varnishd.c
@@ -44,9 +44,10 @@
 #include <unistd.h>
 
 #include "mgt/mgt.h"
+#include "heritage.h"
+#include "params.h"
 
 #include "hash/hash_slinger.h"
-#include "heritage.h"
 #include "vav.h"
 #include "vcli.h"
 #include "vcli_common.h"
diff --git a/include/libvcl.h b/include/libvcl.h
index a92906f..e046db9 100644
--- a/include/libvcl.h
+++ b/include/libvcl.h
@@ -37,4 +37,3 @@ void VCC_VMOD_dir(struct vcc *, const char *str);
 void VCC_Err_Unref(struct vcc *tl, unsigned u);
 
 char *VCC_Compile(const struct vcc *, struct vsb *sb, const char *b);
-const char *VCC_Return_Name(unsigned action);
diff --git a/lib/libvcl/vcc_compile.c b/lib/libvcl/vcc_compile.c
index a0c6232..5719cbf 100644
--- a/lib/libvcl/vcc_compile.c
+++ b/lib/libvcl/vcc_compile.c
@@ -710,21 +710,6 @@ VCC_Compile(const struct vcc *tl, struct vsb *sb, const char *b)
 	return (r);
 }
 
-/*--------------------------------------------------------------------*/
-
-const char *
-VCC_Return_Name(unsigned method)
-{
-
-	switch (method) {
-#define VCL_RET_MAC(l, U, B) case VCL_RET_##U: return(#l);
-#include "tbl/vcl_returns.h"
-#undef VCL_RET_MAC
-	default:
-		return (NULL);
-	}
-}
-
 /*--------------------------------------------------------------------
  * Allocate a compiler instance
  */



From phk at varnish-cache.org  Sun Nov 13 09:40:23 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Sun, 13 Nov 2011 10:40:23 +0100
Subject: [master] 9393154 Remove some unused #includes
Message-ID: <E1RPWY3-0002Uo-Br@project.varnish-software.com>

commit 9393154348e1f28b7815ba9c7d9078243ea43645
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Nov 13 09:40:14 2011 +0000

    Remove some unused #includes

diff --git a/lib/libvarnish/binary_heap.c b/lib/libvarnish/binary_heap.c
index ed99f13..33616b5 100644
--- a/lib/libvarnish/binary_heap.c
+++ b/lib/libvarnish/binary_heap.c
@@ -38,12 +38,10 @@
 #include <errno.h>
 #include <limits.h>
 #include <stdint.h>
-#include <stdio.h>		// for testcase
 #include <stdlib.h>
 #include <unistd.h>
 
 #include "binary_heap.h"
-#include "miniobj.h"		// for testcase
 #include "vas.h"
 
 /* Parameters --------------------------------------------------------*/
@@ -457,6 +455,11 @@ binheap_reorder(const struct binheap *bh, unsigned idx)
 }
 
 #ifdef TEST_DRIVER
+
+#include <stdio.h>
+
+#include "miniobj.h"
+
 /* Test driver -------------------------------------------------------*/
 
 static void
diff --git a/lib/libvarnish/vmb.c b/lib/libvarnish/vmb.c
index 8f130f1..692aab2 100644
--- a/lib/libvarnish/vmb.c
+++ b/lib/libvarnish/vmb.c
@@ -58,4 +58,4 @@ vmb_pthread(void)
 	AZ(pthread_mutex_unlock(&mb_mtx));
 }
 
-#endif
+#endif /* VMB_NEEDS_PTHREAD_WORKAROUND_THIS_IS_BAD_FOR_PERFORMANCE */
diff --git a/lib/libvarnish/vnum.c b/lib/libvarnish/vnum.c
index d2f423b..605947e 100644
--- a/lib/libvarnish/vnum.c
+++ b/lib/libvarnish/vnum.c
@@ -35,7 +35,6 @@
 #include <stdio.h>
 #include <stdlib.h>
 
-#include "vas.h"
 #include "vnum.h"
 
 static const char err_miss_num[] = "Missing number";
diff --git a/lib/libvmod_std/vmod_std.c b/lib/libvmod_std/vmod_std.c
index 79cbf02..f54958f 100644
--- a/lib/libvmod_std/vmod_std.c
+++ b/lib/libvmod_std/vmod_std.c
@@ -33,7 +33,6 @@
 #include <ctype.h>
 #include <errno.h>
 #include <stdarg.h>
-#include <stdint.h>
 #include <stdlib.h>
 #include <syslog.h>
 
diff --git a/lib/libvmod_std/vmod_std_conversions.c b/lib/libvmod_std/vmod_std_conversions.c
index 156cf13..fe8ae96 100644
--- a/lib/libvmod_std/vmod_std_conversions.c
+++ b/lib/libvmod_std/vmod_std_conversions.c
@@ -30,9 +30,7 @@
 #include "config.h"
 
 #include <ctype.h>
-#include <errno.h>
 #include <math.h>
-#include <stdint.h>
 #include <stdlib.h>
 
 #include "../../bin/varnishd/cache.h"



From phk at varnish-cache.org  Sun Nov 13 09:46:15 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Sun, 13 Nov 2011 10:46:15 +0100
Subject: [master] 43b009a Teach varnishtest about VRE_NOTEMPTY_ATSTART and
	skip c00047 which just hangs if we don't have it
Message-ID: <E1RPWdj-0002mG-40@project.varnish-software.com>

commit 43b009a8454f18c7aaaa211b19d2e54004dce24e
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Nov 13 09:45:49 2011 +0000

    Teach varnishtest about VRE_NOTEMPTY_ATSTART and skip c00047
    which just hangs if we don't have it

diff --git a/bin/varnishtest/tests/c00047.vtc b/bin/varnishtest/tests/c00047.vtc
index 4b23194..dd1d256 100644
--- a/bin/varnishtest/tests/c00047.vtc
+++ b/bin/varnishtest/tests/c00047.vtc
@@ -1,5 +1,7 @@
 varnishtest "Test VCL regsuball()"
 
+feature VRE_NOTEMPTY_ATSTART
+
 server s1 {
 	rxreq 
 	txresp \
diff --git a/bin/varnishtest/vtc.c b/bin/varnishtest/vtc.c
index 9581b3c..d6c3b35 100644
--- a/bin/varnishtest/vtc.c
+++ b/bin/varnishtest/vtc.c
@@ -43,6 +43,7 @@
 #include "vtc.h"
 
 #include "vav.h"
+#include "vre.h"
 #include "vtim.h"
 
 
@@ -494,6 +495,9 @@ cmd_feature(CMD_ARGS)
 			continue;
 #endif
 		}
+		if (!strcmp(av[i], "VRE_NOTEMPTY_ATSTART") &&
+		    VRE_NOTEMPTY_ATSTART)
+			continue;
 
 		vtc_log(vl, 1, "SKIPPING test, missing feature: %s", av[i]);
 		vtc_stop = 1;



From phk at varnish-cache.org  Sun Nov 13 10:10:10 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Sun, 13 Nov 2011 11:10:10 +0100
Subject: [master] 91ac88d Silence a Flexelint warning
Message-ID: <E1RPX0s-0003dv-Bl@project.varnish-software.com>

commit 91ac88d789af4bdd1f546e96b8e7601abf7f04f3
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Nov 13 10:05:39 2011 +0000

    Silence a Flexelint warning

diff --git a/bin/varnishd/storage/storage_file.c b/bin/varnishd/storage/storage_file.c
index 58e1d3d..981d43e 100644
--- a/bin/varnishd/storage/storage_file.c
+++ b/bin/varnishd/storage/storage_file.c
@@ -227,7 +227,8 @@ alloc_smf(struct smf_sc *sc, size_t bytes)
 	b = bytes / sc->pagesize;
 	if (b >= NBUCKET)
 		b = NBUCKET - 1;
-	for (sp = NULL; b < NBUCKET - 1; b++) {
+	sp = NULL;
+	for (; b < NBUCKET - 1; b++) {
 		sp = VTAILQ_FIRST(&sc->free[b]);
 		if (sp != NULL)
 			break;



From phk at varnish-cache.org  Sun Nov 13 10:10:12 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Sun, 13 Nov 2011 11:10:12 +0100
Subject: [master] 5c0eee3 rename varnishd.c to mgt/mgt_main.c
Message-ID: <E1RPX0u-0003e8-CI@project.varnish-software.com>

commit 5c0eee322b6ad646781f66f40d26f01dd7e0079d
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Nov 13 10:09:28 2011 +0000

    rename varnishd.c to mgt/mgt_main.c

diff --git a/bin/varnishd/Makefile.am b/bin/varnishd/Makefile.am
index 002447f..94b52a5 100644
--- a/bin/varnishd/Makefile.am
+++ b/bin/varnishd/Makefile.am
@@ -58,6 +58,7 @@ varnishd_SOURCES = \
 	hash/hash_simple_list.c \
 	mgt/mgt_child.c \
 	mgt/mgt_cli.c \
+	mgt/mgt_main.c \
 	mgt/mgt_param.c \
 	mgt/mgt_pool.c \
 	mgt/mgt_sandbox.c \
@@ -75,7 +76,6 @@ varnishd_SOURCES = \
 	storage/storage_persistent_subr.c \
 	storage/storage_synth.c \
 	storage/storage_umem.c \
-	varnishd.c \
 	vsm.c
 
 noinst_HEADERS = \
diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c
new file mode 100644
index 0000000..ccda130
--- /dev/null
+++ b/bin/varnishd/mgt/mgt_main.c
@@ -0,0 +1,652 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * The management process and CLI handling
+ */
+
+#include "config.h"
+
+#include <sys/stat.h>
+#include <sys/utsname.h>
+
+#include <ctype.h>
+#include <fcntl.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <syslog.h>
+#include <time.h>
+#include <unistd.h>
+
+#include "mgt/mgt.h"
+#include "heritage.h"
+#include "params.h"
+
+#include "hash/hash_slinger.h"
+#include "vav.h"
+#include "vcli.h"
+#include "vcli_common.h"
+#include "vev.h"
+#include "vfil.h"
+#include "vin.h"
+#include "vpf.h"
+#include "vsha256.h"
+#include "vtim.h"
+
+#include "compat/daemon.h"
+
+#ifndef HAVE_SRANDOMDEV
+#include "compat/srandomdev.h"
+#endif
+
+struct heritage		heritage;
+unsigned		d_flag = 0;
+pid_t			mgt_pid;
+struct vev_base		*mgt_evb;
+int			exit_status = 0;
+struct vsb		*vident;
+
+static void
+build_vident(void)
+{
+	struct utsname uts;
+
+	vident = VSB_new_auto();
+	AN(vident);
+	if (!uname(&uts)) {
+		VSB_printf(vident, ",%s", uts.sysname);
+		VSB_printf(vident, ",%s", uts.release);
+		VSB_printf(vident, ",%s", uts.machine);
+	}
+}
+
+/*--------------------------------------------------------------------*/
+
+const void *
+pick(const struct choice *cp, const char *which, const char *kind)
+{
+
+	for(; cp->name != NULL; cp++) {
+		if (!strcmp(cp->name, which))
+			return (cp->ptr);
+	}
+	ARGV_ERR("Unknown %s method \"%s\"\n", kind, which);
+}
+
+/*--------------------------------------------------------------------*/
+
+static unsigned long
+arg_ul(const char *p)
+{
+	char *q;
+	unsigned long ul;
+
+	ul = strtoul(p, &q, 0);
+	if (*q != '\0')
+		ARGV_ERR("Invalid number: \"%s\"\n", p);
+	return (ul);
+}
+
+/*--------------------------------------------------------------------*/
+
+static void
+usage(void)
+{
+#define FMT "    %-28s # %s\n"
+
+	fprintf(stderr, "usage: varnishd [options]\n");
+	fprintf(stderr, FMT, "-a address:port", "HTTP listen address and port");
+	fprintf(stderr, FMT, "-b address:port", "backend address and port");
+	fprintf(stderr, FMT, "", "   -b <hostname_or_IP>");
+	fprintf(stderr, FMT, "", "   -b '<hostname_or_IP>:<port_or_service>'");
+	fprintf(stderr, FMT, "-C", "print VCL code compiled to C language");
+	fprintf(stderr, FMT, "-d", "debug");
+	fprintf(stderr, FMT, "-f file", "VCL script");
+	fprintf(stderr, FMT, "-F", "Run in foreground");
+	fprintf(stderr, FMT, "-h kind[,hashoptions]", "Hash specification");
+	fprintf(stderr, FMT, "", "  -h critbit [default]");
+	fprintf(stderr, FMT, "", "  -h simple_list");
+	fprintf(stderr, FMT, "", "  -h classic");
+	fprintf(stderr, FMT, "", "  -h classic,<buckets>");
+	fprintf(stderr, FMT, "-i identity", "Identity of varnish instance");
+	fprintf(stderr, FMT, "-l shl,free,fill", "Size of shared memory file");
+	fprintf(stderr, FMT, "", "  shl: space for SHL records [80m]");
+	fprintf(stderr, FMT, "", "  free: space for other allocations [1m]");
+	fprintf(stderr, FMT, "", "  fill: prefill new file [+]");
+	fprintf(stderr, FMT, "-M address:port", "Reverse CLI destination.");
+	fprintf(stderr, FMT, "-n dir", "varnishd working directory");
+	fprintf(stderr, FMT, "-P file", "PID file");
+	fprintf(stderr, FMT, "-p param=value", "set parameter");
+	fprintf(stderr, FMT,
+	    "-s kind[,storageoptions]", "Backend storage specification");
+	fprintf(stderr, FMT, "", "  -s malloc");
+#ifdef HAVE_LIBUMEM
+	fprintf(stderr, FMT, "", "  -s umem");
+#endif
+	fprintf(stderr, FMT, "", "  -s file  [default: use /tmp]");
+	fprintf(stderr, FMT, "", "  -s file,<dir_or_file>");
+	fprintf(stderr, FMT, "", "  -s file,<dir_or_file>,<size>");
+	fprintf(stderr, FMT, "", "  -s persist{experimenta}");
+	fprintf(stderr, FMT, "",
+	    "  -s file,<dir_or_file>,<size>,<granularity>");
+	fprintf(stderr, FMT, "-t", "Default TTL");
+	fprintf(stderr, FMT, "-S secret-file",
+	    "Secret file for CLI authentication");
+	fprintf(stderr, FMT, "-T address:port",
+	    "Telnet listen address and port");
+	fprintf(stderr, FMT, "-V", "version");
+	fprintf(stderr, FMT, "-w int[,int[,int]]", "Number of worker threads");
+	fprintf(stderr, FMT, "", "  -w <fixed_count>");
+	fprintf(stderr, FMT, "", "  -w min,max");
+	fprintf(stderr, FMT, "", "  -w min,max,timeout [default: -w2,500,300]");
+	fprintf(stderr, FMT, "-u user", "Priviledge separation user id");
+#undef FMT
+	exit(1);
+}
+
+
+/*--------------------------------------------------------------------*/
+
+static void
+tackle_warg(const char *argv)
+{
+	char **av;
+	unsigned int u;
+
+	av = VAV_Parse(argv, NULL, ARGV_COMMA);
+	AN(av);
+
+	if (av[0] != NULL)
+		ARGV_ERR("%s\n", av[0]);
+
+	if (av[1] == NULL)
+		usage();
+
+	u = arg_ul(av[1]);
+	if (u < 1)
+		usage();
+	mgt_param.wthread_max = mgt_param.wthread_min = u;
+
+	if (av[2] != NULL) {
+		u = arg_ul(av[2]);
+		if (u < mgt_param.wthread_min)
+			usage();
+		mgt_param.wthread_max = u;
+
+		if (av[3] != NULL) {
+			u = arg_ul(av[3]);
+			mgt_param.wthread_timeout = u;
+		}
+	}
+	VAV_Free(av);
+}
+
+/*--------------------------------------------------------------------*/
+
+static void
+cli_check(const struct cli *cli)
+{
+	if (cli->result == CLIS_OK) {
+		VSB_clear(cli->sb);
+		return;
+	}
+	AZ(VSB_finish(cli->sb));
+	fprintf(stderr, "Error:\n%s\n", VSB_data(cli->sb));
+	exit (2);
+}
+
+/*--------------------------------------------------------------------
+ * All praise POSIX!  Thanks to our glorious standards there are no
+ * standard way to get a back-trace of the stack, and even if we hack
+ * that together from spit and pieces of string, there is no way no
+ * standard way to translate a pointer to a symbol, which returns anything
+ * usable.  (See for instance FreeBSD PR-134391).
+ *
+ * Attempt to run nm(1) on our binary during startup, hoping it will
+ * give us a usable list of symbols.
+ */
+
+struct symbols {
+	uintptr_t		a;
+	char			*n;
+	VTAILQ_ENTRY(symbols)	list;
+};
+
+static VTAILQ_HEAD(,symbols) symbols = VTAILQ_HEAD_INITIALIZER(symbols);
+
+int
+Symbol_Lookup(struct vsb *vsb, void *ptr)
+{
+	struct symbols *s, *s0;
+	uintptr_t pp;
+
+	pp = (uintptr_t)ptr;
+	s0 = NULL;
+	VTAILQ_FOREACH(s, &symbols, list) {
+		if (s->a > pp)
+			continue;
+		if (s0 == NULL || s->a > s0->a)
+			s0 = s;
+	}
+	if (s0 == NULL)
+		return (-1);
+	VSB_printf(vsb, "%p: %s+%jx", ptr, s0->n, (uintmax_t)pp - s0->a);
+	return (0);
+}
+
+static void
+Symbol_hack(const char *a0)
+{
+	char buf[BUFSIZ], *p, *e;
+	FILE *fi;
+	uintptr_t a;
+	struct symbols *s;
+
+	bprintf(buf, "nm -an %s 2>/dev/null", a0);
+	fi = popen(buf, "r");
+	if (fi == NULL)
+		return;
+	while (fgets(buf, sizeof buf, fi)) {
+		if (buf[0] == ' ')
+			continue;
+		p = NULL;
+		a = strtoul(buf, &p, 16);
+		if (p == NULL)
+			continue;
+		if (a == 0)
+			continue;
+		if (*p++ != ' ')
+			continue;
+		p++;
+		if (*p++ != ' ')
+			continue;
+		if (*p <= ' ')
+			continue;
+		e = strchr(p, '\0');
+		AN(e);
+		while (e > p && isspace(e[-1]))
+			e--;
+		*e = '\0';
+		s = malloc(sizeof *s + strlen(p) + 1);
+		AN(s);
+		s->a = a;
+		s->n = (void*)(s + 1);
+		strcpy(s->n, p);
+		VTAILQ_INSERT_TAIL(&symbols, s, list);
+	}
+	(void)pclose(fi);
+}
+
+/*--------------------------------------------------------------------
+ * This function is called when the CLI on stdin is closed.
+ */
+
+static void
+cli_stdin_close(void *priv)
+{
+
+	(void)priv;
+	(void)close(0);
+	(void)close(1);
+	(void)close(2);
+	assert(open("/dev/null", O_RDONLY) == 0);
+	assert(open("/dev/null", O_WRONLY) == 1);
+	assert(open("/dev/null", O_WRONLY) == 2);
+
+	if (d_flag) {
+		mgt_stop_child();
+		mgt_cli_close_all();
+		exit(0);
+	}
+}
+
+/*--------------------------------------------------------------------*/
+
+int
+main(int argc, char * const *argv)
+{
+	int o;
+	unsigned C_flag = 0;
+	unsigned F_flag = 0;
+	const char *b_arg = NULL;
+	const char *f_arg = NULL;
+	const char *i_arg = NULL;
+	const char *l_arg = NULL;	/* default in mgt_shmem.c */
+	const char *h_arg = "critbit";
+	const char *M_arg = NULL;
+	const char *n_arg = NULL;
+	const char *P_arg = NULL;
+	const char *S_arg = NULL;
+	const char *s_arg = "file";
+	int s_arg_given = 0;
+	const char *T_arg = NULL;
+	char *p, *vcl = NULL;
+	struct cli cli[1];
+	struct vpf_fh *pfh = NULL;
+	char *dirname;
+
+	/*
+	 * Start out by closing all unwanted file descriptors we might
+	 * have inherited from sloppy process control daemons.
+	 */
+	for (o = getdtablesize(); o > STDERR_FILENO; o--)
+		(void)close(o);
+
+	srandomdev();
+
+	mgt_got_fd(STDERR_FILENO);
+
+	setbuf(stdout, NULL);
+	setbuf(stderr, NULL);
+
+	build_vident();
+
+	Symbol_hack(argv[0]);
+
+	/* for ASSERT_MGT() */
+	mgt_pid = getpid();
+
+	/*
+	 * Run in UTC timezone, on the off-chance that this operating
+	 * system does not have a timegm() function, and translates
+	 * timestamps on the local timescale.
+	 * See lib/libvarnish/time.c
+	 */
+	AZ(setenv("TZ", "UTC", 1));
+	tzset();
+	assert(VTIM_parse("Sun, 06 Nov 1994 08:49:37 GMT") == 784111777);
+	assert(VTIM_parse("Sunday, 06-Nov-94 08:49:37 GMT") == 784111777);
+	assert(VTIM_parse("Sun Nov  6 08:49:37 1994") == 784111777);
+
+	/*
+	 * Check that our SHA256 works
+	 */
+	SHA256_Test();
+
+	memset(cli, 0, sizeof cli);
+	cli[0].sb = VSB_new_auto();
+	XXXAN(cli[0].sb);
+	cli[0].result = CLIS_OK;
+
+	VTAILQ_INIT(&heritage.socks);
+
+	MCF_ParamInit(cli);
+
+	if (sizeof(void *) < 8) {
+		/*
+		 * Adjust default parameters for 32 bit systems to conserve
+		 * VM space.
+		 */
+		MCF_ParamSet(cli, "sess_workspace", "16384");
+		cli_check(cli);
+
+		MCF_ParamSet(cli, "thread_pool_workspace", "16384");
+		cli_check(cli);
+
+		MCF_ParamSet(cli, "http_resp_size", "8192");
+		cli_check(cli);
+
+		MCF_ParamSet(cli, "http_req_size", "12288");
+		cli_check(cli);
+
+		MCF_ParamSet(cli, "thread_pool_stack", "32bit");
+		cli_check(cli);
+
+		MCF_ParamSet(cli, "gzip_stack_buffer", "4096");
+		cli_check(cli);
+	}
+
+	cli_check(cli);
+
+	while ((o = getopt(argc, argv,
+	    "a:b:Cdf:Fg:h:i:l:L:M:n:P:p:S:s:T:t:u:Vx:w:")) != -1)
+		switch (o) {
+		case 'a':
+			MCF_ParamSet(cli, "listen_address", optarg);
+			cli_check(cli);
+			break;
+		case 'b':
+			b_arg = optarg;
+			break;
+		case 'C':
+			C_flag = 1 - C_flag;
+			break;
+		case 'd':
+			d_flag++;
+			break;
+		case 'f':
+			f_arg = optarg;
+			break;
+		case 'F':
+			F_flag = 1 - F_flag;
+			break;
+		case 'g':
+			MCF_ParamSet(cli, "group", optarg);
+			break;
+		case 'h':
+			h_arg = optarg;
+			break;
+		case 'i':
+			i_arg = optarg;
+			break;
+		case 'l':
+			l_arg = optarg;
+			break;
+		case 'M':
+			M_arg = optarg;
+			break;
+		case 'n':
+			n_arg = optarg;
+			break;
+		case 'P':
+			P_arg = optarg;
+			break;
+		case 'p':
+			p = strchr(optarg, '=');
+			if (p == NULL)
+				usage();
+			AN(p);
+			*p++ = '\0';
+			MCF_ParamSet(cli, optarg, p);
+			cli_check(cli);
+			break;
+		case 's':
+			s_arg_given = 1;
+			STV_Config(optarg);
+			break;
+		case 't':
+			MCF_ParamSet(cli, "default_ttl", optarg);
+			break;
+		case 'S':
+			S_arg = optarg;
+			break;
+		case 'T':
+			T_arg = optarg;
+			break;
+		case 'u':
+			MCF_ParamSet(cli, "user", optarg);
+			break;
+		case 'V':
+			/* XXX: we should print the ident here */
+			VCS_Message("varnishd");
+			exit(0);
+		case 'x':
+			if (!strcmp(optarg, "dumprst")) {
+				MCF_DumpRst();
+				exit (0);
+			}
+			usage();
+			break;
+		case 'w':
+			tackle_warg(optarg);
+			break;
+		default:
+			usage();
+		}
+
+	argc -= optind;
+	argv += optind;
+
+	mgt_vcc_init();
+
+	if (argc != 0) {
+		fprintf(stderr, "Too many arguments (%s...)\n", argv[0]);
+		usage();
+	}
+
+	/* XXX: we can have multiple CLI actions above, is this enough ? */
+	if (cli[0].result != CLIS_OK) {
+		fprintf(stderr, "Parameter errors:\n");
+		AZ(VSB_finish(cli[0].sb));
+		fprintf(stderr, "%s\n", VSB_data(cli[0].sb));
+		exit(1);
+	}
+
+	if (d_flag && F_flag) {
+		fprintf(stderr, "Only one of -d or -F can be specified\n");
+		usage();
+	}
+
+	if (b_arg != NULL && f_arg != NULL) {
+		fprintf(stderr, "Only one of -b or -f can be specified\n");
+		usage();
+	}
+	if (S_arg == NULL && T_arg == NULL && d_flag == 0 && b_arg == NULL &&
+	    f_arg == NULL && M_arg == NULL) {
+		fprintf(stderr,
+		    "At least one of -d, -b, -f, -M, -S or -T "
+		    "must be specified\n");
+		usage();
+	}
+
+	if (f_arg != NULL) {
+		vcl = VFIL_readfile(NULL, f_arg, NULL);
+		if (vcl == NULL) {
+			fprintf(stderr, "Cannot read '%s': %s\n",
+			    f_arg, strerror(errno));
+			exit(1);
+		}
+	}
+
+	if (VIN_N_Arg(n_arg, &heritage.name, &dirname, NULL) != 0) {
+		fprintf(stderr, "Invalid instance name: %s\n",
+		    strerror(errno));
+		exit(1);
+	}
+
+	if (i_arg != NULL) {
+		if (snprintf(heritage.identity, sizeof heritage.identity,
+		    "%s", i_arg) > sizeof heritage.identity) {
+			fprintf(stderr, "Invalid identity name: %s\n",
+			    strerror(ENAMETOOLONG));
+			exit(1);
+		}
+	}
+
+	if (n_arg != NULL)
+		openlog(n_arg, LOG_PID, LOG_LOCAL0);
+	else
+		openlog("varnishd", LOG_PID, LOG_LOCAL0);
+
+	if (mkdir(dirname, 0755) < 0 && errno != EEXIST) {
+		fprintf(stderr, "Cannot create working directory '%s': %s\n",
+		    dirname, strerror(errno));
+		exit(1);
+	}
+
+	if (chdir(dirname) < 0) {
+		fprintf(stderr, "Cannot change to working directory '%s': %s\n",
+		    dirname, strerror(errno));
+		exit(1);
+	}
+
+	/* XXX: should this be relative to the -n arg ? */
+	if (P_arg && (pfh = VPF_Open(P_arg, 0644, NULL)) == NULL) {
+		perror(P_arg);
+		exit(1);
+	}
+
+	if (b_arg != NULL || f_arg != NULL)
+		if (mgt_vcc_default(b_arg, f_arg, vcl, C_flag))
+			exit (2);
+
+	if (C_flag)
+		exit (0);
+
+	/* If no -s argument specified, process default -s argument */
+	if (!s_arg_given)
+		STV_Config(s_arg);
+
+	/* Configure Transient storage, if user did not */
+	STV_Config_Transient();
+
+	HSH_config(h_arg);
+
+	mgt_SHM_Init(l_arg);
+
+	AZ(VSB_finish(vident));
+
+	if (!d_flag && !F_flag)
+		AZ(varnish_daemon(1, 0));
+
+	mgt_SHM_Pid();
+
+	if (pfh != NULL && VPF_Write(pfh))
+		fprintf(stderr, "NOTE: Could not write PID file\n");
+
+	if (d_flag)
+		fprintf(stderr, "Platform: %s\n", VSB_data(vident) + 1);
+	syslog(LOG_NOTICE, "Platform: %s\n", VSB_data(vident) + 1);
+
+	/* Do this again after debugstunt and daemon has run */
+	mgt_pid = getpid();
+
+	mgt_evb = vev_new_base();
+	XXXAN(mgt_evb);
+
+	if (d_flag)
+		mgt_cli_setup(0, 1, 1, "debug", cli_stdin_close, NULL);
+	if (S_arg != NULL)
+		mgt_cli_secret(S_arg);
+	if (M_arg != NULL)
+		mgt_cli_master(M_arg);
+	if (T_arg != NULL)
+		mgt_cli_telnet(T_arg);
+
+	AN(VSM_Alloc(0, VSM_CLASS_MARK, "", ""));
+
+	MGT_Run();
+
+	if (pfh != NULL)
+		(void)VPF_Remove(pfh);
+	exit(exit_status);
+}
+
+#if defined(PTHREAD_CANCELED) || defined(PTHREAD_MUTEX_DEFAULT)
+#error "Keep pthreads out of in manager process"
+#endif
diff --git a/bin/varnishd/varnishd.c b/bin/varnishd/varnishd.c
deleted file mode 100644
index ccda130..0000000
--- a/bin/varnishd/varnishd.c
+++ /dev/null
@@ -1,652 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * The management process and CLI handling
- */
-
-#include "config.h"
-
-#include <sys/stat.h>
-#include <sys/utsname.h>
-
-#include <ctype.h>
-#include <fcntl.h>
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-#include <syslog.h>
-#include <time.h>
-#include <unistd.h>
-
-#include "mgt/mgt.h"
-#include "heritage.h"
-#include "params.h"
-
-#include "hash/hash_slinger.h"
-#include "vav.h"
-#include "vcli.h"
-#include "vcli_common.h"
-#include "vev.h"
-#include "vfil.h"
-#include "vin.h"
-#include "vpf.h"
-#include "vsha256.h"
-#include "vtim.h"
-
-#include "compat/daemon.h"
-
-#ifndef HAVE_SRANDOMDEV
-#include "compat/srandomdev.h"
-#endif
-
-struct heritage		heritage;
-unsigned		d_flag = 0;
-pid_t			mgt_pid;
-struct vev_base		*mgt_evb;
-int			exit_status = 0;
-struct vsb		*vident;
-
-static void
-build_vident(void)
-{
-	struct utsname uts;
-
-	vident = VSB_new_auto();
-	AN(vident);
-	if (!uname(&uts)) {
-		VSB_printf(vident, ",%s", uts.sysname);
-		VSB_printf(vident, ",%s", uts.release);
-		VSB_printf(vident, ",%s", uts.machine);
-	}
-}
-
-/*--------------------------------------------------------------------*/
-
-const void *
-pick(const struct choice *cp, const char *which, const char *kind)
-{
-
-	for(; cp->name != NULL; cp++) {
-		if (!strcmp(cp->name, which))
-			return (cp->ptr);
-	}
-	ARGV_ERR("Unknown %s method \"%s\"\n", kind, which);
-}
-
-/*--------------------------------------------------------------------*/
-
-static unsigned long
-arg_ul(const char *p)
-{
-	char *q;
-	unsigned long ul;
-
-	ul = strtoul(p, &q, 0);
-	if (*q != '\0')
-		ARGV_ERR("Invalid number: \"%s\"\n", p);
-	return (ul);
-}
-
-/*--------------------------------------------------------------------*/
-
-static void
-usage(void)
-{
-#define FMT "    %-28s # %s\n"
-
-	fprintf(stderr, "usage: varnishd [options]\n");
-	fprintf(stderr, FMT, "-a address:port", "HTTP listen address and port");
-	fprintf(stderr, FMT, "-b address:port", "backend address and port");
-	fprintf(stderr, FMT, "", "   -b <hostname_or_IP>");
-	fprintf(stderr, FMT, "", "   -b '<hostname_or_IP>:<port_or_service>'");
-	fprintf(stderr, FMT, "-C", "print VCL code compiled to C language");
-	fprintf(stderr, FMT, "-d", "debug");
-	fprintf(stderr, FMT, "-f file", "VCL script");
-	fprintf(stderr, FMT, "-F", "Run in foreground");
-	fprintf(stderr, FMT, "-h kind[,hashoptions]", "Hash specification");
-	fprintf(stderr, FMT, "", "  -h critbit [default]");
-	fprintf(stderr, FMT, "", "  -h simple_list");
-	fprintf(stderr, FMT, "", "  -h classic");
-	fprintf(stderr, FMT, "", "  -h classic,<buckets>");
-	fprintf(stderr, FMT, "-i identity", "Identity of varnish instance");
-	fprintf(stderr, FMT, "-l shl,free,fill", "Size of shared memory file");
-	fprintf(stderr, FMT, "", "  shl: space for SHL records [80m]");
-	fprintf(stderr, FMT, "", "  free: space for other allocations [1m]");
-	fprintf(stderr, FMT, "", "  fill: prefill new file [+]");
-	fprintf(stderr, FMT, "-M address:port", "Reverse CLI destination.");
-	fprintf(stderr, FMT, "-n dir", "varnishd working directory");
-	fprintf(stderr, FMT, "-P file", "PID file");
-	fprintf(stderr, FMT, "-p param=value", "set parameter");
-	fprintf(stderr, FMT,
-	    "-s kind[,storageoptions]", "Backend storage specification");
-	fprintf(stderr, FMT, "", "  -s malloc");
-#ifdef HAVE_LIBUMEM
-	fprintf(stderr, FMT, "", "  -s umem");
-#endif
-	fprintf(stderr, FMT, "", "  -s file  [default: use /tmp]");
-	fprintf(stderr, FMT, "", "  -s file,<dir_or_file>");
-	fprintf(stderr, FMT, "", "  -s file,<dir_or_file>,<size>");
-	fprintf(stderr, FMT, "", "  -s persist{experimenta}");
-	fprintf(stderr, FMT, "",
-	    "  -s file,<dir_or_file>,<size>,<granularity>");
-	fprintf(stderr, FMT, "-t", "Default TTL");
-	fprintf(stderr, FMT, "-S secret-file",
-	    "Secret file for CLI authentication");
-	fprintf(stderr, FMT, "-T address:port",
-	    "Telnet listen address and port");
-	fprintf(stderr, FMT, "-V", "version");
-	fprintf(stderr, FMT, "-w int[,int[,int]]", "Number of worker threads");
-	fprintf(stderr, FMT, "", "  -w <fixed_count>");
-	fprintf(stderr, FMT, "", "  -w min,max");
-	fprintf(stderr, FMT, "", "  -w min,max,timeout [default: -w2,500,300]");
-	fprintf(stderr, FMT, "-u user", "Priviledge separation user id");
-#undef FMT
-	exit(1);
-}
-
-
-/*--------------------------------------------------------------------*/
-
-static void
-tackle_warg(const char *argv)
-{
-	char **av;
-	unsigned int u;
-
-	av = VAV_Parse(argv, NULL, ARGV_COMMA);
-	AN(av);
-
-	if (av[0] != NULL)
-		ARGV_ERR("%s\n", av[0]);
-
-	if (av[1] == NULL)
-		usage();
-
-	u = arg_ul(av[1]);
-	if (u < 1)
-		usage();
-	mgt_param.wthread_max = mgt_param.wthread_min = u;
-
-	if (av[2] != NULL) {
-		u = arg_ul(av[2]);
-		if (u < mgt_param.wthread_min)
-			usage();
-		mgt_param.wthread_max = u;
-
-		if (av[3] != NULL) {
-			u = arg_ul(av[3]);
-			mgt_param.wthread_timeout = u;
-		}
-	}
-	VAV_Free(av);
-}
-
-/*--------------------------------------------------------------------*/
-
-static void
-cli_check(const struct cli *cli)
-{
-	if (cli->result == CLIS_OK) {
-		VSB_clear(cli->sb);
-		return;
-	}
-	AZ(VSB_finish(cli->sb));
-	fprintf(stderr, "Error:\n%s\n", VSB_data(cli->sb));
-	exit (2);
-}
-
-/*--------------------------------------------------------------------
- * All praise POSIX!  Thanks to our glorious standards there are no
- * standard way to get a back-trace of the stack, and even if we hack
- * that together from spit and pieces of string, there is no way no
- * standard way to translate a pointer to a symbol, which returns anything
- * usable.  (See for instance FreeBSD PR-134391).
- *
- * Attempt to run nm(1) on our binary during startup, hoping it will
- * give us a usable list of symbols.
- */
-
-struct symbols {
-	uintptr_t		a;
-	char			*n;
-	VTAILQ_ENTRY(symbols)	list;
-};
-
-static VTAILQ_HEAD(,symbols) symbols = VTAILQ_HEAD_INITIALIZER(symbols);
-
-int
-Symbol_Lookup(struct vsb *vsb, void *ptr)
-{
-	struct symbols *s, *s0;
-	uintptr_t pp;
-
-	pp = (uintptr_t)ptr;
-	s0 = NULL;
-	VTAILQ_FOREACH(s, &symbols, list) {
-		if (s->a > pp)
-			continue;
-		if (s0 == NULL || s->a > s0->a)
-			s0 = s;
-	}
-	if (s0 == NULL)
-		return (-1);
-	VSB_printf(vsb, "%p: %s+%jx", ptr, s0->n, (uintmax_t)pp - s0->a);
-	return (0);
-}
-
-static void
-Symbol_hack(const char *a0)
-{
-	char buf[BUFSIZ], *p, *e;
-	FILE *fi;
-	uintptr_t a;
-	struct symbols *s;
-
-	bprintf(buf, "nm -an %s 2>/dev/null", a0);
-	fi = popen(buf, "r");
-	if (fi == NULL)
-		return;
-	while (fgets(buf, sizeof buf, fi)) {
-		if (buf[0] == ' ')
-			continue;
-		p = NULL;
-		a = strtoul(buf, &p, 16);
-		if (p == NULL)
-			continue;
-		if (a == 0)
-			continue;
-		if (*p++ != ' ')
-			continue;
-		p++;
-		if (*p++ != ' ')
-			continue;
-		if (*p <= ' ')
-			continue;
-		e = strchr(p, '\0');
-		AN(e);
-		while (e > p && isspace(e[-1]))
-			e--;
-		*e = '\0';
-		s = malloc(sizeof *s + strlen(p) + 1);
-		AN(s);
-		s->a = a;
-		s->n = (void*)(s + 1);
-		strcpy(s->n, p);
-		VTAILQ_INSERT_TAIL(&symbols, s, list);
-	}
-	(void)pclose(fi);
-}
-
-/*--------------------------------------------------------------------
- * This function is called when the CLI on stdin is closed.
- */
-
-static void
-cli_stdin_close(void *priv)
-{
-
-	(void)priv;
-	(void)close(0);
-	(void)close(1);
-	(void)close(2);
-	assert(open("/dev/null", O_RDONLY) == 0);
-	assert(open("/dev/null", O_WRONLY) == 1);
-	assert(open("/dev/null", O_WRONLY) == 2);
-
-	if (d_flag) {
-		mgt_stop_child();
-		mgt_cli_close_all();
-		exit(0);
-	}
-}
-
-/*--------------------------------------------------------------------*/
-
-int
-main(int argc, char * const *argv)
-{
-	int o;
-	unsigned C_flag = 0;
-	unsigned F_flag = 0;
-	const char *b_arg = NULL;
-	const char *f_arg = NULL;
-	const char *i_arg = NULL;
-	const char *l_arg = NULL;	/* default in mgt_shmem.c */
-	const char *h_arg = "critbit";
-	const char *M_arg = NULL;
-	const char *n_arg = NULL;
-	const char *P_arg = NULL;
-	const char *S_arg = NULL;
-	const char *s_arg = "file";
-	int s_arg_given = 0;
-	const char *T_arg = NULL;
-	char *p, *vcl = NULL;
-	struct cli cli[1];
-	struct vpf_fh *pfh = NULL;
-	char *dirname;
-
-	/*
-	 * Start out by closing all unwanted file descriptors we might
-	 * have inherited from sloppy process control daemons.
-	 */
-	for (o = getdtablesize(); o > STDERR_FILENO; o--)
-		(void)close(o);
-
-	srandomdev();
-
-	mgt_got_fd(STDERR_FILENO);
-
-	setbuf(stdout, NULL);
-	setbuf(stderr, NULL);
-
-	build_vident();
-
-	Symbol_hack(argv[0]);
-
-	/* for ASSERT_MGT() */
-	mgt_pid = getpid();
-
-	/*
-	 * Run in UTC timezone, on the off-chance that this operating
-	 * system does not have a timegm() function, and translates
-	 * timestamps on the local timescale.
-	 * See lib/libvarnish/time.c
-	 */
-	AZ(setenv("TZ", "UTC", 1));
-	tzset();
-	assert(VTIM_parse("Sun, 06 Nov 1994 08:49:37 GMT") == 784111777);
-	assert(VTIM_parse("Sunday, 06-Nov-94 08:49:37 GMT") == 784111777);
-	assert(VTIM_parse("Sun Nov  6 08:49:37 1994") == 784111777);
-
-	/*
-	 * Check that our SHA256 works
-	 */
-	SHA256_Test();
-
-	memset(cli, 0, sizeof cli);
-	cli[0].sb = VSB_new_auto();
-	XXXAN(cli[0].sb);
-	cli[0].result = CLIS_OK;
-
-	VTAILQ_INIT(&heritage.socks);
-
-	MCF_ParamInit(cli);
-
-	if (sizeof(void *) < 8) {
-		/*
-		 * Adjust default parameters for 32 bit systems to conserve
-		 * VM space.
-		 */
-		MCF_ParamSet(cli, "sess_workspace", "16384");
-		cli_check(cli);
-
-		MCF_ParamSet(cli, "thread_pool_workspace", "16384");
-		cli_check(cli);
-
-		MCF_ParamSet(cli, "http_resp_size", "8192");
-		cli_check(cli);
-
-		MCF_ParamSet(cli, "http_req_size", "12288");
-		cli_check(cli);
-
-		MCF_ParamSet(cli, "thread_pool_stack", "32bit");
-		cli_check(cli);
-
-		MCF_ParamSet(cli, "gzip_stack_buffer", "4096");
-		cli_check(cli);
-	}
-
-	cli_check(cli);
-
-	while ((o = getopt(argc, argv,
-	    "a:b:Cdf:Fg:h:i:l:L:M:n:P:p:S:s:T:t:u:Vx:w:")) != -1)
-		switch (o) {
-		case 'a':
-			MCF_ParamSet(cli, "listen_address", optarg);
-			cli_check(cli);
-			break;
-		case 'b':
-			b_arg = optarg;
-			break;
-		case 'C':
-			C_flag = 1 - C_flag;
-			break;
-		case 'd':
-			d_flag++;
-			break;
-		case 'f':
-			f_arg = optarg;
-			break;
-		case 'F':
-			F_flag = 1 - F_flag;
-			break;
-		case 'g':
-			MCF_ParamSet(cli, "group", optarg);
-			break;
-		case 'h':
-			h_arg = optarg;
-			break;
-		case 'i':
-			i_arg = optarg;
-			break;
-		case 'l':
-			l_arg = optarg;
-			break;
-		case 'M':
-			M_arg = optarg;
-			break;
-		case 'n':
-			n_arg = optarg;
-			break;
-		case 'P':
-			P_arg = optarg;
-			break;
-		case 'p':
-			p = strchr(optarg, '=');
-			if (p == NULL)
-				usage();
-			AN(p);
-			*p++ = '\0';
-			MCF_ParamSet(cli, optarg, p);
-			cli_check(cli);
-			break;
-		case 's':
-			s_arg_given = 1;
-			STV_Config(optarg);
-			break;
-		case 't':
-			MCF_ParamSet(cli, "default_ttl", optarg);
-			break;
-		case 'S':
-			S_arg = optarg;
-			break;
-		case 'T':
-			T_arg = optarg;
-			break;
-		case 'u':
-			MCF_ParamSet(cli, "user", optarg);
-			break;
-		case 'V':
-			/* XXX: we should print the ident here */
-			VCS_Message("varnishd");
-			exit(0);
-		case 'x':
-			if (!strcmp(optarg, "dumprst")) {
-				MCF_DumpRst();
-				exit (0);
-			}
-			usage();
-			break;
-		case 'w':
-			tackle_warg(optarg);
-			break;
-		default:
-			usage();
-		}
-
-	argc -= optind;
-	argv += optind;
-
-	mgt_vcc_init();
-
-	if (argc != 0) {
-		fprintf(stderr, "Too many arguments (%s...)\n", argv[0]);
-		usage();
-	}
-
-	/* XXX: we can have multiple CLI actions above, is this enough ? */
-	if (cli[0].result != CLIS_OK) {
-		fprintf(stderr, "Parameter errors:\n");
-		AZ(VSB_finish(cli[0].sb));
-		fprintf(stderr, "%s\n", VSB_data(cli[0].sb));
-		exit(1);
-	}
-
-	if (d_flag && F_flag) {
-		fprintf(stderr, "Only one of -d or -F can be specified\n");
-		usage();
-	}
-
-	if (b_arg != NULL && f_arg != NULL) {
-		fprintf(stderr, "Only one of -b or -f can be specified\n");
-		usage();
-	}
-	if (S_arg == NULL && T_arg == NULL && d_flag == 0 && b_arg == NULL &&
-	    f_arg == NULL && M_arg == NULL) {
-		fprintf(stderr,
-		    "At least one of -d, -b, -f, -M, -S or -T "
-		    "must be specified\n");
-		usage();
-	}
-
-	if (f_arg != NULL) {
-		vcl = VFIL_readfile(NULL, f_arg, NULL);
-		if (vcl == NULL) {
-			fprintf(stderr, "Cannot read '%s': %s\n",
-			    f_arg, strerror(errno));
-			exit(1);
-		}
-	}
-
-	if (VIN_N_Arg(n_arg, &heritage.name, &dirname, NULL) != 0) {
-		fprintf(stderr, "Invalid instance name: %s\n",
-		    strerror(errno));
-		exit(1);
-	}
-
-	if (i_arg != NULL) {
-		if (snprintf(heritage.identity, sizeof heritage.identity,
-		    "%s", i_arg) > sizeof heritage.identity) {
-			fprintf(stderr, "Invalid identity name: %s\n",
-			    strerror(ENAMETOOLONG));
-			exit(1);
-		}
-	}
-
-	if (n_arg != NULL)
-		openlog(n_arg, LOG_PID, LOG_LOCAL0);
-	else
-		openlog("varnishd", LOG_PID, LOG_LOCAL0);
-
-	if (mkdir(dirname, 0755) < 0 && errno != EEXIST) {
-		fprintf(stderr, "Cannot create working directory '%s': %s\n",
-		    dirname, strerror(errno));
-		exit(1);
-	}
-
-	if (chdir(dirname) < 0) {
-		fprintf(stderr, "Cannot change to working directory '%s': %s\n",
-		    dirname, strerror(errno));
-		exit(1);
-	}
-
-	/* XXX: should this be relative to the -n arg ? */
-	if (P_arg && (pfh = VPF_Open(P_arg, 0644, NULL)) == NULL) {
-		perror(P_arg);
-		exit(1);
-	}
-
-	if (b_arg != NULL || f_arg != NULL)
-		if (mgt_vcc_default(b_arg, f_arg, vcl, C_flag))
-			exit (2);
-
-	if (C_flag)
-		exit (0);
-
-	/* If no -s argument specified, process default -s argument */
-	if (!s_arg_given)
-		STV_Config(s_arg);
-
-	/* Configure Transient storage, if user did not */
-	STV_Config_Transient();
-
-	HSH_config(h_arg);
-
-	mgt_SHM_Init(l_arg);
-
-	AZ(VSB_finish(vident));
-
-	if (!d_flag && !F_flag)
-		AZ(varnish_daemon(1, 0));
-
-	mgt_SHM_Pid();
-
-	if (pfh != NULL && VPF_Write(pfh))
-		fprintf(stderr, "NOTE: Could not write PID file\n");
-
-	if (d_flag)
-		fprintf(stderr, "Platform: %s\n", VSB_data(vident) + 1);
-	syslog(LOG_NOTICE, "Platform: %s\n", VSB_data(vident) + 1);
-
-	/* Do this again after debugstunt and daemon has run */
-	mgt_pid = getpid();
-
-	mgt_evb = vev_new_base();
-	XXXAN(mgt_evb);
-
-	if (d_flag)
-		mgt_cli_setup(0, 1, 1, "debug", cli_stdin_close, NULL);
-	if (S_arg != NULL)
-		mgt_cli_secret(S_arg);
-	if (M_arg != NULL)
-		mgt_cli_master(M_arg);
-	if (T_arg != NULL)
-		mgt_cli_telnet(T_arg);
-
-	AN(VSM_Alloc(0, VSM_CLASS_MARK, "", ""));
-
-	MGT_Run();
-
-	if (pfh != NULL)
-		(void)VPF_Remove(pfh);
-	exit(exit_status);
-}
-
-#if defined(PTHREAD_CANCELED) || defined(PTHREAD_MUTEX_DEFAULT)
-#error "Keep pthreads out of in manager process"
-#endif



From phk at varnish-cache.org  Sun Nov 13 10:10:11 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Sun, 13 Nov 2011 11:10:11 +0100
Subject: [master] ac37476 Add a dead reference to SES_DeletePool() to remind
	ourselves where it will be used from.
Message-ID: <E1RPX0t-0003e1-M3@project.varnish-software.com>

commit ac3747664187fc3487107aee51a776be343745c5
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Nov 13 10:09:00 2011 +0000

    Add a dead reference to SES_DeletePool() to remind ourselves where
    it will be used from.

diff --git a/bin/varnishd/cache_pool.c b/bin/varnishd/cache_pool.c
index eaf7602..be49548 100644
--- a/bin/varnishd/cache_pool.c
+++ b/bin/varnishd/cache_pool.c
@@ -571,6 +571,8 @@ pool_poolherder(void *priv)
 			}
 		}
 		/* XXX: remove pools */
+		if (0)
+			SES_DeletePool(NULL, NULL);
 		(void)sleep(1);
 		u = 0;
 		VTAILQ_FOREACH(pp, &pools, list)



From phk at varnish-cache.org  Sun Nov 13 10:22:23 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Sun, 13 Nov 2011 11:22:23 +0100
Subject: [master] 94c79bd Rename rfc2616.c to cache_rfc2616.c where it
	belongs. Move vparam.h to mgt/mgt_param.h
Message-ID: <E1RPXCh-00041l-Sw@project.varnish-software.com>

commit 94c79bd35de7b47178ac670d9a80d7004168cbc2
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Nov 13 10:21:45 2011 +0000

    Rename rfc2616.c to cache_rfc2616.c where it belongs.
    Move vparam.h to mgt/mgt_param.h

diff --git a/bin/varnishd/Makefile.am b/bin/varnishd/Makefile.am
index 94b52a5..0460303 100644
--- a/bin/varnishd/Makefile.am
+++ b/bin/varnishd/Makefile.am
@@ -36,6 +36,7 @@ varnishd_SOURCES = \
 	cache_pipe.c \
 	cache_pool.c \
 	cache_response.c \
+	cache_rfc2616.c \
 	cache_session.c \
 	cache_shmlog.c \
 	cache_vary.c \
@@ -65,7 +66,6 @@ varnishd_SOURCES = \
 	mgt/mgt_sandbox_solaris.c \
 	mgt/mgt_shmem.c \
 	mgt/mgt_vcc.c \
-	rfc2616.c \
 	storage/stevedore.c \
 	storage/stevedore_utils.c \
 	storage/storage_file.c \
@@ -89,10 +89,10 @@ noinst_HEADERS = \
 	heritage.h \
 	mgt/mgt.h \
 	mgt/mgt_cli.h \
+	mgt/mgt_param.h \
 	params.h \
 	storage/storage.h \
-	storage/storage_persistent.h \
-	vparam.h
+	storage/storage_persistent.h
 
 varnishd_CFLAGS = \
 	@PCRE_CFLAGS@ \
diff --git a/bin/varnishd/cache_rfc2616.c b/bin/varnishd/cache_rfc2616.c
new file mode 100644
index 0000000..4041f45
--- /dev/null
+++ b/bin/varnishd/cache_rfc2616.c
@@ -0,0 +1,336 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include "config.h"
+
+#include <math.h>
+#include <stdlib.h>
+
+#include "cache.h"
+
+#include "vtim.h"
+
+/*--------------------------------------------------------------------
+ * TTL and Age calculation in Varnish
+ *
+ * RFC2616 has a lot to say about how caches should calculate the TTL
+ * and expiry times of objects, but it sort of misses the case that
+ * applies to Varnish:  the server-side cache.
+ *
+ * A normal cache, shared or single-client, has no symbiotic relationship
+ * with the server, and therefore must take a very defensive attitude
+ * if the Data/Expiry/Age/max-age data does not make sense.  Overall
+ * the policy described in section 13 of RFC 2616 results in no caching
+ * happening on the first little sign of trouble.
+ *
+ * Varnish on the other hand tries to offload as many transactions from
+ * the backend as possible, and therefore just passing through everything
+ * if there is a clock-skew between backend and Varnish is not a workable
+ * choice.
+ *
+ * Varnish implements a policy which is RFC2616 compliant when there
+ * is no clockskew, and falls as gracefully as possible otherwise.
+ * Our "clockless cache" model is syntehsized from the bits of RFC2616
+ * that talks about how a cache should react to a clockless origin server,
+ * and more or less uses the inverse logic for the opposite relationship.
+ *
+ */
+
+void
+RFC2616_Ttl(const struct sess *sp)
+{
+	unsigned max_age, age;
+	double h_date, h_expires;
+	char *p;
+	const struct http *hp;
+
+	hp = sp->wrk->beresp;
+
+	assert(sp->wrk->exp.entered != 0.0 && !isnan(sp->wrk->exp.entered));
+	/* If all else fails, cache using default ttl */
+	sp->wrk->exp.ttl = cache_param->default_ttl;
+
+	max_age = age = 0;
+	h_expires = 0;
+	h_date = 0;
+
+	/*
+	 * Initial cacheability determination per [RFC2616, 13.4]
+	 * We do not support ranges yet, so 206 is out.
+	 */
+
+	if (http_GetHdr(hp, H_Age, &p)) {
+		age = strtoul(p, NULL, 0);
+		sp->wrk->exp.age = age;
+	}
+	if (http_GetHdr(hp, H_Expires, &p))
+		h_expires = VTIM_parse(p);
+
+	if (http_GetHdr(hp, H_Date, &p))
+		h_date = VTIM_parse(p);
+
+	switch (sp->err_code) {
+	default:
+		sp->wrk->exp.ttl = -1.;
+		break;
+	case 200: /* OK */
+	case 203: /* Non-Authoritative Information */
+	case 300: /* Multiple Choices */
+	case 301: /* Moved Permanently */
+	case 302: /* Moved Temporarily */
+	case 307: /* Temporary Redirect */
+	case 410: /* Gone */
+	case 404: /* Not Found */
+		/*
+		 * First find any relative specification from the backend
+		 * These take precedence according to RFC2616, 13.2.4
+		 */
+
+		if ((http_GetHdrField(hp, H_Cache_Control, "s-maxage", &p) ||
+		    http_GetHdrField(hp, H_Cache_Control, "max-age", &p)) &&
+		    p != NULL) {
+
+			if (*p == '-')
+				max_age = 0;
+			else
+				max_age = strtoul(p, NULL, 0);
+
+			if (age > max_age)
+				sp->wrk->exp.ttl = 0;
+			else
+				sp->wrk->exp.ttl = max_age - age;
+			break;
+		}
+
+		/* No expire header, fall back to default */
+		if (h_expires == 0)
+			break;
+
+
+		/* If backend told us it is expired already, don't cache. */
+		if (h_expires < h_date) {
+			sp->wrk->exp.ttl = 0;
+			break;
+		}
+
+		if (h_date == 0 ||
+		    fabs(h_date - sp->wrk->exp.entered) < cache_param->clock_skew) {
+			/*
+			 * If we have no Date: header or if it is
+			 * sufficiently close to our clock we will
+			 * trust Expires: relative to our own clock.
+			 */
+			if (h_expires < sp->wrk->exp.entered)
+				sp->wrk->exp.ttl = 0;
+			else
+				sp->wrk->exp.ttl = h_expires -
+				    sp->wrk->exp.entered;
+			break;
+		} else {
+			/*
+			 * But even if the clocks are out of whack we can still
+			 * derive a relative time from the two headers.
+			 * (the negative ttl case is caught above)
+			 */
+			sp->wrk->exp.ttl = (int)(h_expires - h_date);
+		}
+
+	}
+
+	/* calculated TTL, Our time, Date, Expires, max-age, age */
+	WSP(sp, SLT_TTL,
+	    "%u RFC %.0f %.0f %.0f %.0f %.0f %.0f %.0f %u",
+	    sp->xid, sp->wrk->exp.ttl, -1., -1., sp->wrk->exp.entered,
+	    sp->wrk->exp.age, h_date, h_expires, max_age);
+}
+
+/*--------------------------------------------------------------------
+ * Body existence, fetch method and close policy.
+ */
+
+enum body_status
+RFC2616_Body(const struct sess *sp)
+{
+	struct http *hp;
+	char *b;
+
+	hp = sp->wrk->beresp;
+
+	if (hp->protover < 11 && !http_HdrIs(hp, H_Connection, "keep-alive"))
+		sp->wrk->do_close = 1;
+	else if (http_HdrIs(hp, H_Connection, "close"))
+		sp->wrk->do_close = 1;
+	else
+		sp->wrk->do_close = 0;
+
+	if (!strcasecmp(http_GetReq(sp->wrk->bereq), "head")) {
+		/*
+		 * A HEAD request can never have a body in the reply,
+		 * no matter what the headers might say.
+		 * [RFC2516 4.3 p33]
+		 */
+		sp->wrk->stats.fetch_head++;
+		return (BS_NONE);
+	}
+
+	if (hp->status <= 199) {
+		/*
+		 * 1xx responses never have a body.
+		 * [RFC2616 4.3 p33]
+		 */
+		sp->wrk->stats.fetch_1xx++;
+		return (BS_NONE);
+	}
+
+	if (hp->status == 204) {
+		/*
+		 * 204 is "No Content", obviously don't expect a body.
+		 * [RFC2616 10.2.5 p60]
+		 */
+		sp->wrk->stats.fetch_204++;
+		return (BS_NONE);
+	}
+
+	if (hp->status == 304) {
+		/*
+		 * 304 is "Not Modified" it has no body.
+		 * [RFC2616 10.3.5 p63]
+		 */
+		sp->wrk->stats.fetch_304++;
+		return (BS_NONE);
+	}
+
+	if (http_HdrIs(hp, H_Transfer_Encoding, "chunked")) {
+		 sp->wrk->stats.fetch_chunked++;
+		return (BS_CHUNKED);
+	}
+
+	if (http_GetHdr(hp, H_Transfer_Encoding, &b)) {
+		sp->wrk->stats.fetch_bad++;
+		return (BS_ERROR);
+	}
+
+	if (http_GetHdr(hp, H_Content_Length, &sp->wrk->h_content_length)) {
+		sp->wrk->stats.fetch_length++;
+		return (BS_LENGTH);
+	}
+
+	if (http_HdrIs(hp, H_Connection, "keep-alive")) {
+		/*
+		 * Keep alive with neither TE=Chunked or C-Len is impossible.
+		 * We assume a zero length body.
+		 */
+		sp->wrk->stats.fetch_zero++;
+		return (BS_ZERO);
+	}
+
+	if (http_HdrIs(hp, H_Connection, "close")) {
+		/*
+		 * In this case, it is safe to just read what comes.
+		 */
+		sp->wrk->stats.fetch_close++;
+		return (BS_EOF);
+	}
+
+	if (hp->protover < 11) {
+		/*
+		 * With no Connection header, assume EOF.
+		 */
+		sp->wrk->stats.fetch_oldhttp++;
+		return (BS_EOF);
+	}
+
+	/*
+	 * Fall back to EOF transfer.
+	 */
+	sp->wrk->stats.fetch_eof++;
+	return (BS_EOF);
+}
+
+/*--------------------------------------------------------------------
+ * Find out if the request can receive a gzip'ed response
+ */
+
+unsigned
+RFC2616_Req_Gzip(const struct sess *sp)
+{
+
+
+	/*
+	 * "x-gzip" is for http/1.0 backwards compat, final note in 14.3
+	 * p104 says to not do q values for x-gzip, so we just test
+	 * for its existence.
+	 */
+	if (http_GetHdrData(sp->http, H_Accept_Encoding, "x-gzip", NULL))
+		return (1);
+
+	/*
+	 * "gzip" is the real thing, but the 'q' value must be nonzero.
+	 * We do not care a hoot if the client prefers some other
+	 * compression more than gzip: Varnish only does gzip.
+	 */
+	if (http_GetHdrQ(sp->http, H_Accept_Encoding, "gzip") > 0.)
+		return (1);
+
+	/* Bad client, no gzip. */
+	return (0);
+}
+
+/*--------------------------------------------------------------------*/
+
+int
+RFC2616_Do_Cond(const struct sess *sp)
+{
+	char *p, *e;
+	double ims;
+	int do_cond = 0;
+
+	/* RFC 2616 13.3.4 states we need to match both ETag
+	   and If-Modified-Since if present*/
+
+	if (http_GetHdr(sp->http, H_If_Modified_Since, &p) ) {
+		if (!sp->obj->last_modified)
+			return (0);
+		ims = VTIM_parse(p);
+		if (ims > sp->t_req)	/* [RFC2616 14.25] */
+			return (0);
+		if (sp->obj->last_modified > ims)
+			return (0);
+		do_cond = 1;
+	}
+
+	if (http_GetHdr(sp->http, H_If_None_Match, &p) &&
+	    http_GetHdr(sp->obj->http, H_ETag, &e)) {
+		if (strcmp(p,e) != 0)
+			return (0);
+		do_cond = 1;
+	}
+
+	return (do_cond);
+}
diff --git a/bin/varnishd/mgt/mgt_param.c b/bin/varnishd/mgt/mgt_param.c
index cd96e8b..8fff352 100644
--- a/bin/varnishd/mgt/mgt_param.c
+++ b/bin/varnishd/mgt/mgt_param.c
@@ -41,12 +41,12 @@
 #include "heritage.h"
 #include "params.h"
 
+#include "mgt/mgt_param.h"
 #include "waiter/cache_waiter.h"
 #include "vav.h"
 #include "vcli.h"
 #include "vcli_common.h"
 #include "vcli_priv.h"
-#include "vparam.h"
 #include "vss.h"
 
 #include "mgt_cli.h"
diff --git a/bin/varnishd/mgt/mgt_param.h b/bin/varnishd/mgt/mgt_param.h
new file mode 100644
index 0000000..2d4a97f
--- /dev/null
+++ b/bin/varnishd/mgt/mgt_param.h
@@ -0,0 +1,59 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ */
+
+struct parspec;
+
+typedef void tweak_t(struct cli *, const struct parspec *, const char *arg);
+
+struct parspec {
+	const char	*name;
+	tweak_t		*func;
+	volatile void	*priv;
+	double		min;
+	double		max;
+	const char	*descr;
+	int		 flags;
+#define DELAYED_EFFECT	(1<<0)
+#define EXPERIMENTAL	(1<<1)
+#define MUST_RESTART	(1<<2)
+#define MUST_RELOAD	(1<<3)
+#define WIZARD		(1<<4)
+	const char	*def;
+	const char	*units;
+};
+
+void tweak_generic_uint(struct cli *cli,
+    volatile unsigned *dest, const char *arg, unsigned min, unsigned max);
+void tweak_uint(struct cli *cli, const struct parspec *par, const char *arg);
+void tweak_timeout(struct cli *cli,
+    const struct parspec *par, const char *arg);
+
+/* mgt_pool.c */
+extern const struct parspec WRK_parspec[];
diff --git a/bin/varnishd/mgt/mgt_pool.c b/bin/varnishd/mgt/mgt_pool.c
index 548ad78..6f61553 100644
--- a/bin/varnishd/mgt/mgt_pool.c
+++ b/bin/varnishd/mgt/mgt_pool.c
@@ -51,7 +51,7 @@
 #include "heritage.h"
 #include "params.h"
 
-#include "vparam.h"
+#include "mgt/mgt_param.h"
 
 /*--------------------------------------------------------------------*/
 
diff --git a/bin/varnishd/rfc2616.c b/bin/varnishd/rfc2616.c
deleted file mode 100644
index 4041f45..0000000
--- a/bin/varnishd/rfc2616.c
+++ /dev/null
@@ -1,336 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- */
-
-#include "config.h"
-
-#include <math.h>
-#include <stdlib.h>
-
-#include "cache.h"
-
-#include "vtim.h"
-
-/*--------------------------------------------------------------------
- * TTL and Age calculation in Varnish
- *
- * RFC2616 has a lot to say about how caches should calculate the TTL
- * and expiry times of objects, but it sort of misses the case that
- * applies to Varnish:  the server-side cache.
- *
- * A normal cache, shared or single-client, has no symbiotic relationship
- * with the server, and therefore must take a very defensive attitude
- * if the Data/Expiry/Age/max-age data does not make sense.  Overall
- * the policy described in section 13 of RFC 2616 results in no caching
- * happening on the first little sign of trouble.
- *
- * Varnish on the other hand tries to offload as many transactions from
- * the backend as possible, and therefore just passing through everything
- * if there is a clock-skew between backend and Varnish is not a workable
- * choice.
- *
- * Varnish implements a policy which is RFC2616 compliant when there
- * is no clockskew, and falls as gracefully as possible otherwise.
- * Our "clockless cache" model is syntehsized from the bits of RFC2616
- * that talks about how a cache should react to a clockless origin server,
- * and more or less uses the inverse logic for the opposite relationship.
- *
- */
-
-void
-RFC2616_Ttl(const struct sess *sp)
-{
-	unsigned max_age, age;
-	double h_date, h_expires;
-	char *p;
-	const struct http *hp;
-
-	hp = sp->wrk->beresp;
-
-	assert(sp->wrk->exp.entered != 0.0 && !isnan(sp->wrk->exp.entered));
-	/* If all else fails, cache using default ttl */
-	sp->wrk->exp.ttl = cache_param->default_ttl;
-
-	max_age = age = 0;
-	h_expires = 0;
-	h_date = 0;
-
-	/*
-	 * Initial cacheability determination per [RFC2616, 13.4]
-	 * We do not support ranges yet, so 206 is out.
-	 */
-
-	if (http_GetHdr(hp, H_Age, &p)) {
-		age = strtoul(p, NULL, 0);
-		sp->wrk->exp.age = age;
-	}
-	if (http_GetHdr(hp, H_Expires, &p))
-		h_expires = VTIM_parse(p);
-
-	if (http_GetHdr(hp, H_Date, &p))
-		h_date = VTIM_parse(p);
-
-	switch (sp->err_code) {
-	default:
-		sp->wrk->exp.ttl = -1.;
-		break;
-	case 200: /* OK */
-	case 203: /* Non-Authoritative Information */
-	case 300: /* Multiple Choices */
-	case 301: /* Moved Permanently */
-	case 302: /* Moved Temporarily */
-	case 307: /* Temporary Redirect */
-	case 410: /* Gone */
-	case 404: /* Not Found */
-		/*
-		 * First find any relative specification from the backend
-		 * These take precedence according to RFC2616, 13.2.4
-		 */
-
-		if ((http_GetHdrField(hp, H_Cache_Control, "s-maxage", &p) ||
-		    http_GetHdrField(hp, H_Cache_Control, "max-age", &p)) &&
-		    p != NULL) {
-
-			if (*p == '-')
-				max_age = 0;
-			else
-				max_age = strtoul(p, NULL, 0);
-
-			if (age > max_age)
-				sp->wrk->exp.ttl = 0;
-			else
-				sp->wrk->exp.ttl = max_age - age;
-			break;
-		}
-
-		/* No expire header, fall back to default */
-		if (h_expires == 0)
-			break;
-
-
-		/* If backend told us it is expired already, don't cache. */
-		if (h_expires < h_date) {
-			sp->wrk->exp.ttl = 0;
-			break;
-		}
-
-		if (h_date == 0 ||
-		    fabs(h_date - sp->wrk->exp.entered) < cache_param->clock_skew) {
-			/*
-			 * If we have no Date: header or if it is
-			 * sufficiently close to our clock we will
-			 * trust Expires: relative to our own clock.
-			 */
-			if (h_expires < sp->wrk->exp.entered)
-				sp->wrk->exp.ttl = 0;
-			else
-				sp->wrk->exp.ttl = h_expires -
-				    sp->wrk->exp.entered;
-			break;
-		} else {
-			/*
-			 * But even if the clocks are out of whack we can still
-			 * derive a relative time from the two headers.
-			 * (the negative ttl case is caught above)
-			 */
-			sp->wrk->exp.ttl = (int)(h_expires - h_date);
-		}
-
-	}
-
-	/* calculated TTL, Our time, Date, Expires, max-age, age */
-	WSP(sp, SLT_TTL,
-	    "%u RFC %.0f %.0f %.0f %.0f %.0f %.0f %.0f %u",
-	    sp->xid, sp->wrk->exp.ttl, -1., -1., sp->wrk->exp.entered,
-	    sp->wrk->exp.age, h_date, h_expires, max_age);
-}
-
-/*--------------------------------------------------------------------
- * Body existence, fetch method and close policy.
- */
-
-enum body_status
-RFC2616_Body(const struct sess *sp)
-{
-	struct http *hp;
-	char *b;
-
-	hp = sp->wrk->beresp;
-
-	if (hp->protover < 11 && !http_HdrIs(hp, H_Connection, "keep-alive"))
-		sp->wrk->do_close = 1;
-	else if (http_HdrIs(hp, H_Connection, "close"))
-		sp->wrk->do_close = 1;
-	else
-		sp->wrk->do_close = 0;
-
-	if (!strcasecmp(http_GetReq(sp->wrk->bereq), "head")) {
-		/*
-		 * A HEAD request can never have a body in the reply,
-		 * no matter what the headers might say.
-		 * [RFC2516 4.3 p33]
-		 */
-		sp->wrk->stats.fetch_head++;
-		return (BS_NONE);
-	}
-
-	if (hp->status <= 199) {
-		/*
-		 * 1xx responses never have a body.
-		 * [RFC2616 4.3 p33]
-		 */
-		sp->wrk->stats.fetch_1xx++;
-		return (BS_NONE);
-	}
-
-	if (hp->status == 204) {
-		/*
-		 * 204 is "No Content", obviously don't expect a body.
-		 * [RFC2616 10.2.5 p60]
-		 */
-		sp->wrk->stats.fetch_204++;
-		return (BS_NONE);
-	}
-
-	if (hp->status == 304) {
-		/*
-		 * 304 is "Not Modified" it has no body.
-		 * [RFC2616 10.3.5 p63]
-		 */
-		sp->wrk->stats.fetch_304++;
-		return (BS_NONE);
-	}
-
-	if (http_HdrIs(hp, H_Transfer_Encoding, "chunked")) {
-		 sp->wrk->stats.fetch_chunked++;
-		return (BS_CHUNKED);
-	}
-
-	if (http_GetHdr(hp, H_Transfer_Encoding, &b)) {
-		sp->wrk->stats.fetch_bad++;
-		return (BS_ERROR);
-	}
-
-	if (http_GetHdr(hp, H_Content_Length, &sp->wrk->h_content_length)) {
-		sp->wrk->stats.fetch_length++;
-		return (BS_LENGTH);
-	}
-
-	if (http_HdrIs(hp, H_Connection, "keep-alive")) {
-		/*
-		 * Keep alive with neither TE=Chunked or C-Len is impossible.
-		 * We assume a zero length body.
-		 */
-		sp->wrk->stats.fetch_zero++;
-		return (BS_ZERO);
-	}
-
-	if (http_HdrIs(hp, H_Connection, "close")) {
-		/*
-		 * In this case, it is safe to just read what comes.
-		 */
-		sp->wrk->stats.fetch_close++;
-		return (BS_EOF);
-	}
-
-	if (hp->protover < 11) {
-		/*
-		 * With no Connection header, assume EOF.
-		 */
-		sp->wrk->stats.fetch_oldhttp++;
-		return (BS_EOF);
-	}
-
-	/*
-	 * Fall back to EOF transfer.
-	 */
-	sp->wrk->stats.fetch_eof++;
-	return (BS_EOF);
-}
-
-/*--------------------------------------------------------------------
- * Find out if the request can receive a gzip'ed response
- */
-
-unsigned
-RFC2616_Req_Gzip(const struct sess *sp)
-{
-
-
-	/*
-	 * "x-gzip" is for http/1.0 backwards compat, final note in 14.3
-	 * p104 says to not do q values for x-gzip, so we just test
-	 * for its existence.
-	 */
-	if (http_GetHdrData(sp->http, H_Accept_Encoding, "x-gzip", NULL))
-		return (1);
-
-	/*
-	 * "gzip" is the real thing, but the 'q' value must be nonzero.
-	 * We do not care a hoot if the client prefers some other
-	 * compression more than gzip: Varnish only does gzip.
-	 */
-	if (http_GetHdrQ(sp->http, H_Accept_Encoding, "gzip") > 0.)
-		return (1);
-
-	/* Bad client, no gzip. */
-	return (0);
-}
-
-/*--------------------------------------------------------------------*/
-
-int
-RFC2616_Do_Cond(const struct sess *sp)
-{
-	char *p, *e;
-	double ims;
-	int do_cond = 0;
-
-	/* RFC 2616 13.3.4 states we need to match both ETag
-	   and If-Modified-Since if present*/
-
-	if (http_GetHdr(sp->http, H_If_Modified_Since, &p) ) {
-		if (!sp->obj->last_modified)
-			return (0);
-		ims = VTIM_parse(p);
-		if (ims > sp->t_req)	/* [RFC2616 14.25] */
-			return (0);
-		if (sp->obj->last_modified > ims)
-			return (0);
-		do_cond = 1;
-	}
-
-	if (http_GetHdr(sp->http, H_If_None_Match, &p) &&
-	    http_GetHdr(sp->obj->http, H_ETag, &e)) {
-		if (strcmp(p,e) != 0)
-			return (0);
-		do_cond = 1;
-	}
-
-	return (do_cond);
-}
diff --git a/bin/varnishd/vparam.h b/bin/varnishd/vparam.h
deleted file mode 100644
index 2d4a97f..0000000
--- a/bin/varnishd/vparam.h
+++ /dev/null
@@ -1,59 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- */
-
-struct parspec;
-
-typedef void tweak_t(struct cli *, const struct parspec *, const char *arg);
-
-struct parspec {
-	const char	*name;
-	tweak_t		*func;
-	volatile void	*priv;
-	double		min;
-	double		max;
-	const char	*descr;
-	int		 flags;
-#define DELAYED_EFFECT	(1<<0)
-#define EXPERIMENTAL	(1<<1)
-#define MUST_RESTART	(1<<2)
-#define MUST_RELOAD	(1<<3)
-#define WIZARD		(1<<4)
-	const char	*def;
-	const char	*units;
-};
-
-void tweak_generic_uint(struct cli *cli,
-    volatile unsigned *dest, const char *arg, unsigned min, unsigned max);
-void tweak_uint(struct cli *cli, const struct parspec *par, const char *arg);
-void tweak_timeout(struct cli *cli,
-    const struct parspec *par, const char *arg);
-
-/* mgt_pool.c */
-extern const struct parspec WRK_parspec[];



From phk at varnish-cache.org  Sun Nov 13 10:56:48 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Sun, 13 Nov 2011 11:56:48 +0100
Subject: [master] 26cfc94 Move cache/mgt common stuff under,
	you guessed it: common/
Message-ID: <E1RPXk0-0005j7-JX@project.varnish-software.com>

commit 26cfc94ebbff9946dd9dbdabc83382abb9d77e86
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Nov 13 10:38:37 2011 +0000

    Move cache/mgt common stuff under, you guessed it: common/

diff --git a/bin/varnishd/Makefile.am b/bin/varnishd/Makefile.am
index 0460303..245bc6b 100644
--- a/bin/varnishd/Makefile.am
+++ b/bin/varnishd/Makefile.am
@@ -45,14 +45,10 @@ varnishd_SOURCES = \
 	cache_vrt_re.c \
 	cache_vrt_var.c \
 	cache_vrt_vmod.c \
-	waiter/cache_waiter.c \
-	waiter/cache_waiter_epoll.c \
-	waiter/cache_waiter_kqueue.c \
-	waiter/cache_waiter_poll.c \
-	waiter/cache_waiter_ports.c \
 	cache_wrk.c \
 	cache_wrw.c \
 	cache_ws.c \
+	common/common_vsm.c \
 	hash/hash_classic.c \
 	hash/hash_critbit.c \
 	hash/hash_mgt.c \
@@ -76,23 +72,27 @@ varnishd_SOURCES = \
 	storage/storage_persistent_subr.c \
 	storage/storage_synth.c \
 	storage/storage_umem.c \
-	vsm.c
+	waiter/cache_waiter.c \
+	waiter/cache_waiter_epoll.c \
+	waiter/cache_waiter_kqueue.c \
+	waiter/cache_waiter_poll.c \
+	waiter/cache_waiter_ports.c 
 
 noinst_HEADERS = \
 	cache.h \
 	cache_backend.h \
 	cache_esi.h \
-	waiter/cache_waiter.h \
-	common.h \
+	common/common.h \
+	common/heritage.h \
+	common/params.h \
 	default_vcl.h \
 	hash/hash_slinger.h \
-	heritage.h \
 	mgt/mgt.h \
 	mgt/mgt_cli.h \
 	mgt/mgt_param.h \
-	params.h \
 	storage/storage.h \
-	storage/storage_persistent.h
+	storage/storage_persistent.h \
+	waiter/cache_waiter.h
 
 varnishd_CFLAGS = \
 	@PCRE_CFLAGS@ \
diff --git a/bin/varnishd/cache.h b/bin/varnishd/cache.h
index b21f2d4..653fe77 100644
--- a/bin/varnishd/cache.h
+++ b/bin/varnishd/cache.h
@@ -34,7 +34,7 @@
  */
 #define VARNISH_CACHE_CHILD	1
 
-#include "common.h"
+#include "common/common.h"
 
 #include "vapi/vsc_int.h"
 #include "vapi/vsl_int.h"
@@ -55,7 +55,7 @@
 #endif
 
 
-#include "params.h"
+#include "common/params.h"
 
 enum body_status {
 #define BODYSTATUS(U,l)	BS_##U,
diff --git a/bin/varnishd/cache_acceptor.c b/bin/varnishd/cache_acceptor.c
index 5161b09..8c83121 100644
--- a/bin/varnishd/cache_acceptor.c
+++ b/bin/varnishd/cache_acceptor.c
@@ -31,7 +31,7 @@
 #include "config.h"
 
 #include "cache.h"
-#include "heritage.h"
+#include "common/heritage.h"
 
 #include "vcli.h"
 #include "vcli_priv.h"
diff --git a/bin/varnishd/cache_cli.c b/bin/varnishd/cache_cli.c
index 0bbf94b..f29f86a 100644
--- a/bin/varnishd/cache_cli.c
+++ b/bin/varnishd/cache_cli.c
@@ -39,7 +39,7 @@
 #include <stddef.h>			// offsetof
 
 #include "cache.h"
-#include "heritage.h"
+#include "common/heritage.h"
 
 #include "cache_backend.h"		// struct vbc
 #include "hash/hash_slinger.h"		// struct objhead
diff --git a/bin/varnishd/cache_main.c b/bin/varnishd/cache_main.c
index 7a3bcd9..eb3fa1d 100644
--- a/bin/varnishd/cache_main.c
+++ b/bin/varnishd/cache_main.c
@@ -33,7 +33,7 @@
 #include <stdlib.h>
 
 #include "cache.h"
-#include "heritage.h"
+#include "common/heritage.h"
 
 #include "waiter/cache_waiter.h"
 #include "hash/hash_slinger.h"
diff --git a/bin/varnishd/cache_pool.c b/bin/varnishd/cache_pool.c
index be49548..79a5fcd 100644
--- a/bin/varnishd/cache_pool.c
+++ b/bin/varnishd/cache_pool.c
@@ -46,7 +46,7 @@
 #include <stdlib.h>
 
 #include "cache.h"
-#include "heritage.h"
+#include "common/heritage.h"
 
 #include "waiter/cache_waiter.h"
 #include "vtcp.h"
diff --git a/bin/varnishd/cache_vrt_var.c b/bin/varnishd/cache_vrt_var.c
index a9bf87f..860c7aa 100644
--- a/bin/varnishd/cache_vrt_var.c
+++ b/bin/varnishd/cache_vrt_var.c
@@ -34,7 +34,7 @@
 #include <stdlib.h>
 
 #include "cache.h"
-#include "heritage.h"
+#include "common/heritage.h"
 
 #include "cache_backend.h"
 #include "vrt_obj.h"
diff --git a/bin/varnishd/common.h b/bin/varnishd/common.h
deleted file mode 100644
index 4e56a40..0000000
--- a/bin/varnishd/common.h
+++ /dev/null
@@ -1,113 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- */
-
-#include <errno.h>
-#include <stdint.h>
-
-#include <sys/types.h>
-
-#include "miniobj.h"
-#include "vas.h"
-#include "vcs.h"
-#include "vdef.h"
-#include "vqueue.h"
-#include "vsb.h"
-
-struct cli;
-
-extern pid_t mgt_pid;
-#define ASSERT_MGT() do { assert(getpid() == mgt_pid);} while (0)
-
-/* mgt_shmem.c */
-extern struct VSC_C_main *VSC_C_main;
-
-/* varnishd.c */
-struct vsb;
-extern struct vsb *vident;
-int Symbol_Lookup(struct vsb *vsb, void *ptr);
-
-#define TRUST_ME(ptr)	((void*)(uintptr_t)(ptr))
-
-
-/* Help shut up FlexeLint */
-#define __match_proto__(xxx) /*lint -e{818} */
-
-/* Really belongs in mgt.h, but storage_file chokes on both */
-void mgt_child_inherit(int fd, const char *what);
-
-#define ARGV_ERR(...)						\
-	do {							\
-		fprintf(stderr, "Error: " __VA_ARGS__);		\
-		exit(2);					\
-	} while (0);
-
-/* A tiny helper for choosing hash/storage modules */
-struct choice {
-	const char      *name;
-	const void	*ptr;
-};
-const void *pick(const struct choice *cp, const char *which, const char *kind);
-
-#define NEEDLESS_RETURN(foo)	return (foo)
-
-/* stevedore.c */
-void STV_Config(const char *spec);
-void STV_Config_Transient(void);
-
-/* vsm.c */
-// extern struct VSM_head		*VSM_head;
-// extern const struct VSM_chunk	*vsm_end;
-
-/*
- * These three should not be called directly, but only through
- * proper vectors in mgt.h/cache.h, hence the __
- */
-void *VSM__Alloc(unsigned size, const char *class, const char *type,
-    const char *ident);
-void VSM__Free(const void *ptr);
-void VSM__Clean(void);
-
-/* These classes are opaque to other programs, so we define the here */
-#define VSM_CLASS_FREE	"Free"
-#define VSM_CLASS_COOL	"Cool"
-#define VSM_CLASS_PARAM	"Params"
-#define VSM_CLASS_MARK	"MgrCld"
-#define VSM_COOL_TIME	5
-
-/* cache_lck.c */
-struct lock { void *priv; };		// Opaque
-
-/*---------------------------------------------------------------------
- * Generic power-2 rounding macros
- */
-
-#define PWR2(x)     ((((x)-1)&(x))==0)		/* Is a power of two */
-#define RDN2(x, y)  ((x)&(~((y)-1)))		/* if y is powers of two */
-#define RUP2(x, y)  (((x)+((y)-1))&(~((y)-1)))	/* if y is powers of two */
diff --git a/bin/varnishd/common/common.h b/bin/varnishd/common/common.h
new file mode 100644
index 0000000..4e56a40
--- /dev/null
+++ b/bin/varnishd/common/common.h
@@ -0,0 +1,113 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ */
+
+#include <errno.h>
+#include <stdint.h>
+
+#include <sys/types.h>
+
+#include "miniobj.h"
+#include "vas.h"
+#include "vcs.h"
+#include "vdef.h"
+#include "vqueue.h"
+#include "vsb.h"
+
+struct cli;
+
+extern pid_t mgt_pid;
+#define ASSERT_MGT() do { assert(getpid() == mgt_pid);} while (0)
+
+/* mgt_shmem.c */
+extern struct VSC_C_main *VSC_C_main;
+
+/* varnishd.c */
+struct vsb;
+extern struct vsb *vident;
+int Symbol_Lookup(struct vsb *vsb, void *ptr);
+
+#define TRUST_ME(ptr)	((void*)(uintptr_t)(ptr))
+
+
+/* Help shut up FlexeLint */
+#define __match_proto__(xxx) /*lint -e{818} */
+
+/* Really belongs in mgt.h, but storage_file chokes on both */
+void mgt_child_inherit(int fd, const char *what);
+
+#define ARGV_ERR(...)						\
+	do {							\
+		fprintf(stderr, "Error: " __VA_ARGS__);		\
+		exit(2);					\
+	} while (0);
+
+/* A tiny helper for choosing hash/storage modules */
+struct choice {
+	const char      *name;
+	const void	*ptr;
+};
+const void *pick(const struct choice *cp, const char *which, const char *kind);
+
+#define NEEDLESS_RETURN(foo)	return (foo)
+
+/* stevedore.c */
+void STV_Config(const char *spec);
+void STV_Config_Transient(void);
+
+/* vsm.c */
+// extern struct VSM_head		*VSM_head;
+// extern const struct VSM_chunk	*vsm_end;
+
+/*
+ * These three should not be called directly, but only through
+ * proper vectors in mgt.h/cache.h, hence the __
+ */
+void *VSM__Alloc(unsigned size, const char *class, const char *type,
+    const char *ident);
+void VSM__Free(const void *ptr);
+void VSM__Clean(void);
+
+/* These classes are opaque to other programs, so we define the here */
+#define VSM_CLASS_FREE	"Free"
+#define VSM_CLASS_COOL	"Cool"
+#define VSM_CLASS_PARAM	"Params"
+#define VSM_CLASS_MARK	"MgrCld"
+#define VSM_COOL_TIME	5
+
+/* cache_lck.c */
+struct lock { void *priv; };		// Opaque
+
+/*---------------------------------------------------------------------
+ * Generic power-2 rounding macros
+ */
+
+#define PWR2(x)     ((((x)-1)&(x))==0)		/* Is a power of two */
+#define RDN2(x, y)  ((x)&(~((y)-1)))		/* if y is powers of two */
+#define RUP2(x, y)  (((x)+((y)-1))&(~((y)-1)))	/* if y is powers of two */
diff --git a/bin/varnishd/common/common_vsm.c b/bin/varnishd/common/common_vsm.c
new file mode 100644
index 0000000..5298381
--- /dev/null
+++ b/bin/varnishd/common/common_vsm.c
@@ -0,0 +1,238 @@
+/*-
+ * Copyright (c) 2010-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * VSM stuff common to manager and child.
+ *
+ * We have three potential conflicts we need to lock against here:
+ *
+ * VSM-studying programs (varnishstat...) vs. everybody else
+ *	The VSM studying programs only have read-only access to the VSM
+ *	so everybody else must use memory barriers, stable storage and
+ *	similar tricks to keep the VSM image in sync (long enough) for
+ *	the studying programs.
+ *
+ * Manager process vs child process.
+ *	Will only muck about in VSM when child process is not running
+ *	Responsible for cleaning up any mess left behind by dying child.
+ *
+ * Child process threads
+ *	Pthread locking necessary.
+ *
+ * XXX: not all of this is in place yet.
+ */
+
+#include "config.h"
+
+#include <stdint.h>
+#include <stdio.h>
+#include <string.h>
+#include <unistd.h>
+
+#include "common.h"
+
+#include "vapi/vsm_int.h"
+#include "vmb.h"
+#include "vtim.h"
+
+/* These two come from beyond (mgt_shmem.c actually) */
+struct VSM_head		*VSM_head;
+const struct VSM_chunk	*vsm_end;
+
+static unsigned
+vsm_mark(void)
+{
+	unsigned seq;
+
+	seq = VSM_head->alloc_seq;
+	VSM_head->alloc_seq = 0;
+	VWMB();
+	return (seq);
+}
+
+static void
+vsm_release(unsigned seq)
+{
+
+	if (seq == 0)
+		return;
+	VWMB();
+	do
+		VSM_head->alloc_seq = ++seq;
+	while (VSM_head->alloc_seq == 0);
+}
+
+/*--------------------------------------------------------------------*/
+
+static void
+vsm_cleanup(void)
+{
+	unsigned now = (unsigned)VTIM_mono();
+	struct VSM_chunk *sha, *sha2;
+	unsigned seq;
+
+	CHECK_OBJ_NOTNULL(VSM_head, VSM_HEAD_MAGIC);
+	VSM_ITER(sha) {
+		if (strcmp(sha->class, VSM_CLASS_COOL))
+			continue;
+		if (sha->state + VSM_COOL_TIME < now)
+			break;
+	}
+	if (sha == NULL)
+		return;
+	seq = vsm_mark();
+	/* First pass, free, and collapse with next if applicable */
+	VSM_ITER(sha) {
+		if (strcmp(sha->class, VSM_CLASS_COOL))
+			continue;
+		if (sha->state + VSM_COOL_TIME >= now)
+			continue;
+
+		bprintf(sha->class, "%s", VSM_CLASS_FREE);
+		bprintf(sha->type, "%s", "");
+		bprintf(sha->ident, "%s", "");
+		sha2 = VSM_NEXT(sha);
+		assert(sha2 <= vsm_end);
+		if (sha2 == vsm_end)
+			break;
+		CHECK_OBJ_NOTNULL(sha2, VSM_CHUNK_MAGIC);
+		if (!strcmp(sha2->class, VSM_CLASS_FREE)) {
+			sha->len += sha2->len;
+			memset(sha2, 0, sizeof *sha2);
+		}
+		sha->state = 0;
+	}
+	/* Second pass, collaps with prev if applicable */
+	VSM_ITER(sha) {
+		if (strcmp(sha->class, VSM_CLASS_FREE))
+			continue;
+		sha2 = VSM_NEXT(sha);
+		assert(sha2 <= vsm_end);
+		if (sha2 == vsm_end)
+			break;
+		CHECK_OBJ_NOTNULL(sha2, VSM_CHUNK_MAGIC);
+		if (!strcmp(sha2->class, VSM_CLASS_FREE)) {
+			sha->len += sha2->len;
+			memset(sha2, 0, sizeof *sha2);
+		}
+	}
+	vsm_release(seq);
+}
+
+/*--------------------------------------------------------------------*/
+
+void *
+VSM__Alloc(unsigned size, const char *class, const char *type, const char *ident)
+{
+	struct VSM_chunk *sha, *sha2;
+	unsigned seq;
+
+	CHECK_OBJ_NOTNULL(VSM_head, VSM_HEAD_MAGIC);
+
+	vsm_cleanup();
+
+	/* Round up to pointersize */
+	size = RUP2(size, sizeof(void*));
+
+	size += sizeof *sha;		/* Make space for the header */
+
+	VSM_ITER(sha) {
+		CHECK_OBJ_NOTNULL(sha, VSM_CHUNK_MAGIC);
+
+		if (strcmp(sha->class, VSM_CLASS_FREE))
+			continue;
+
+		if (size > sha->len)
+			continue;
+
+		/* Mark as inconsistent while we write string fields */
+		seq = vsm_mark();
+
+		if (size + sizeof (*sha) < sha->len) {
+			sha2 = (void*)((uintptr_t)sha + size);
+
+			memset(sha2, 0, sizeof *sha2);
+			sha2->magic = VSM_CHUNK_MAGIC;
+			sha2->len = sha->len - size;
+			bprintf(sha2->class, "%s", VSM_CLASS_FREE);
+			sha->len = size;
+		}
+
+		bprintf(sha->class, "%s", class);
+		bprintf(sha->type, "%s", type);
+		bprintf(sha->ident, "%s", ident);
+
+		vsm_release(seq);
+		return (VSM_PTR(sha));
+	}
+	return (NULL);
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+VSM__Free(const void *ptr)
+{
+	struct VSM_chunk *sha;
+	unsigned seq;
+
+	CHECK_OBJ_NOTNULL(VSM_head, VSM_HEAD_MAGIC);
+	VSM_ITER(sha)
+		if (VSM_PTR(sha) == ptr)
+			break;
+	AN(sha);
+	seq = vsm_mark();
+	bprintf(sha->class, "%s", VSM_CLASS_COOL);
+	sha->state = (unsigned)VTIM_mono();
+	vsm_release(seq);
+}
+
+/*--------------------------------------------------------------------
+ * Free all allocations after the mark (ie: allocated by child).
+ */
+
+void
+VSM__Clean(void)
+{
+	struct VSM_chunk *sha;
+	unsigned f, seq;
+
+	CHECK_OBJ_NOTNULL(VSM_head, VSM_HEAD_MAGIC);
+	f = 0;
+	seq = vsm_mark();
+	VSM_ITER(sha) {
+		if (f == 0 && !strcmp(sha->class, VSM_CLASS_MARK)) {
+			f = 1;
+			continue;
+		}
+		if (f == 0)
+			continue;
+		if (strcmp(sha->class, VSM_CLASS_FREE) &&
+		    strcmp(sha->class, VSM_CLASS_COOL))
+			VSM__Free(VSM_PTR(sha));
+	}
+	vsm_release(seq);
+}
diff --git a/bin/varnishd/common/heritage.h b/bin/varnishd/common/heritage.h
new file mode 100644
index 0000000..36433bb
--- /dev/null
+++ b/bin/varnishd/common/heritage.h
@@ -0,0 +1,65 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * This file contains the heritage passed when mgt forks cache
+ */
+
+struct listen_sock {
+	unsigned			magic;
+#define LISTEN_SOCK_MAGIC		0x999e4b57
+	VTAILQ_ENTRY(listen_sock)	list;
+	int				sock;
+	char				*name;
+	struct vss_addr			*addr;
+};
+
+VTAILQ_HEAD(listen_sock_head, listen_sock);
+
+struct heritage {
+
+	/* Two pipe(2)'s for CLI connection between cache and mgt.  */
+	int				cli_in;
+	int				cli_out;
+
+	/* File descriptor for stdout/stderr */
+	int				std_fd;
+
+	/* Sockets from which to accept connections */
+	struct listen_sock_head		socks;
+	unsigned			nsocks;
+
+	/* Hash method */
+	const struct hash_slinger	*hash;
+
+	char				*name;
+	char                            identity[1024];
+};
+
+extern struct heritage heritage;
+
+void child_main(void);
diff --git a/bin/varnishd/common/params.h b/bin/varnishd/common/params.h
new file mode 100644
index 0000000..fb71e33
--- /dev/null
+++ b/bin/varnishd/common/params.h
@@ -0,0 +1,194 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * This file contains the heritage passed when mgt forks cache
+ */
+
+#include "vre.h"
+
+struct params {
+
+	/* Unprivileged user / group */
+	char			*user;
+	uid_t			uid;
+	char			*group;
+	gid_t			gid;
+
+	/* TTL used for lack of anything better */
+	double			default_ttl;
+
+	/* Default grace period */
+	double			default_grace;
+
+	/* Default keep period */
+	double			default_keep;
+
+	/* Maximum concurrent sessions */
+	unsigned		max_sess;
+
+	/* Worker threads and pool */
+	unsigned		wthread_min;
+	unsigned		wthread_max;
+	unsigned		wthread_timeout;
+	unsigned		wthread_pools;
+	unsigned		wthread_add_threshold;
+	unsigned		wthread_add_delay;
+	unsigned		wthread_fail_delay;
+	unsigned		wthread_purge_delay;
+	unsigned		wthread_stats_rate;
+	unsigned		wthread_stacksize;
+	unsigned		wthread_workspace;
+
+	unsigned		queue_max;
+
+	/* Memory allocation hints */
+	unsigned		sess_workspace;
+	unsigned		shm_workspace;
+	unsigned		http_req_size;
+	unsigned		http_req_hdr_len;
+	unsigned		http_resp_size;
+	unsigned		http_resp_hdr_len;
+	unsigned		http_max_hdr;
+
+	unsigned		shm_reclen;
+
+	/* Acceptor hints */
+	unsigned		sess_timeout;
+	unsigned		pipe_timeout;
+	unsigned		send_timeout;
+	unsigned		idle_send_timeout;
+
+	/* Management hints */
+	unsigned		auto_restart;
+
+	/* Fetcher hints */
+	unsigned		fetch_chunksize;
+	unsigned		fetch_maxchunksize;
+	unsigned		nuke_limit;
+
+#ifdef SENDFILE_WORKS
+	/* Sendfile object minimum size */
+	unsigned		sendfile_threshold;
+#endif
+
+	/* VCL traces */
+	unsigned		vcl_trace;
+
+	/* Listen address */
+	char			*listen_address;
+
+	/* Listen depth */
+	unsigned		listen_depth;
+
+	/* CLI related */
+	unsigned		cli_timeout;
+	unsigned		ping_interval;
+
+	/* LRU list ordering interval */
+	unsigned		lru_timeout;
+
+	/* Maximum restarts allowed */
+	unsigned		max_restarts;
+
+	/* Maximum esi:include depth allowed */
+	unsigned		max_esi_depth;
+
+	/* ESI parser hints */
+	unsigned		esi_syntax;
+
+	/* Rush exponent */
+	unsigned		rush_exponent;
+
+	/* Default connection_timeout */
+	double			connect_timeout;
+
+	/* Read timeouts for backend */
+	double			first_byte_timeout;
+	double			between_bytes_timeout;
+
+	/* How long to linger on sessions */
+	unsigned		session_linger;
+
+	/* CLI buffer size */
+	unsigned		cli_buffer;
+
+	/* Control diagnostic code */
+	unsigned		diag_bitmap;
+
+	/* Log hash string to shm */
+	unsigned		log_hash;
+
+	/* Log local socket address to shm */
+	unsigned		log_local_addr;
+
+	/* Prefer IPv6 connections to backend*/
+	unsigned		prefer_ipv6;
+
+	/* Acceptable clockskew with backends */
+	unsigned		clock_skew;
+
+	/* Expiry pacer parameters */
+	double			expiry_sleep;
+
+	/* Acceptor pacer parameters */
+	double			acceptor_sleep_max;
+	double			acceptor_sleep_incr;
+	double			acceptor_sleep_decay;
+
+	/* Get rid of duplicate bans */
+	unsigned		ban_dups;
+
+	/* How long time does the ban lurker sleep */
+	double			ban_lurker_sleep;
+
+	/* Max size of the saintmode list. 0 == no saint mode. */
+	unsigned		saintmode_threshold;
+
+	unsigned		syslog_cli_traffic;
+
+	unsigned		http_range_support;
+
+	unsigned		http_gzip_support;
+	unsigned		gzip_stack_buffer;
+	unsigned		gzip_tmp_space;
+	unsigned		gzip_level;
+	unsigned		gzip_window;
+	unsigned		gzip_memlevel;
+
+	double			critbit_cooloff;
+
+	double			shortlived;
+
+	struct vre_limits	vre_limits;
+};
+
+/*
+ * We declare this a volatile pointer, so that reads of parameters
+ * become atomic, leaving the CLI thread lattitude to change the values
+ */
+extern volatile struct params * cache_param;
diff --git a/bin/varnishd/flint.sh b/bin/varnishd/flint.sh
index 4beb9d5..756f65a 100755
--- a/bin/varnishd/flint.sh
+++ b/bin/varnishd/flint.sh
@@ -16,6 +16,7 @@ flexelint \
 	-I/usr/local/include \
 	-DVARNISH_STATE_DIR=\"foo\" \
 	*.c \
+	common/*.c \
 	storage/*.c \
 	waiter/*.c \
 	hash/*.c \
diff --git a/bin/varnishd/hash/hash_mgt.c b/bin/varnishd/hash/hash_mgt.c
index 6226749..ffbba29 100644
--- a/bin/varnishd/hash/hash_mgt.c
+++ b/bin/varnishd/hash/hash_mgt.c
@@ -36,7 +36,7 @@
 #include <unistd.h>
 
 #include "mgt/mgt.h"
-#include "heritage.h"
+#include "common/heritage.h"
 
 #include "hash/hash_slinger.h"
 #include "vav.h"
diff --git a/bin/varnishd/heritage.h b/bin/varnishd/heritage.h
deleted file mode 100644
index 36433bb..0000000
--- a/bin/varnishd/heritage.h
+++ /dev/null
@@ -1,65 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * This file contains the heritage passed when mgt forks cache
- */
-
-struct listen_sock {
-	unsigned			magic;
-#define LISTEN_SOCK_MAGIC		0x999e4b57
-	VTAILQ_ENTRY(listen_sock)	list;
-	int				sock;
-	char				*name;
-	struct vss_addr			*addr;
-};
-
-VTAILQ_HEAD(listen_sock_head, listen_sock);
-
-struct heritage {
-
-	/* Two pipe(2)'s for CLI connection between cache and mgt.  */
-	int				cli_in;
-	int				cli_out;
-
-	/* File descriptor for stdout/stderr */
-	int				std_fd;
-
-	/* Sockets from which to accept connections */
-	struct listen_sock_head		socks;
-	unsigned			nsocks;
-
-	/* Hash method */
-	const struct hash_slinger	*hash;
-
-	char				*name;
-	char                            identity[1024];
-};
-
-extern struct heritage heritage;
-
-void child_main(void);
diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h
index 61493b3..54352a0 100644
--- a/bin/varnishd/mgt/mgt.h
+++ b/bin/varnishd/mgt/mgt.h
@@ -30,7 +30,7 @@
 
 #include <stdint.h>
 
-#include "common.h"
+#include "common/common.h"
 
 struct cli;
 
diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c
index d2e0abd..a2c0b58 100644
--- a/bin/varnishd/mgt/mgt_child.c
+++ b/bin/varnishd/mgt/mgt_child.c
@@ -44,8 +44,8 @@
 #include <unistd.h>
 
 #include "mgt/mgt.h"
-#include "heritage.h"
-#include "params.h"
+#include "common/heritage.h"
+#include "common/params.h"
 
 #include "vapi/vsm_int.h"
 #include "vbm.h"
diff --git a/bin/varnishd/mgt/mgt_cli.c b/bin/varnishd/mgt/mgt_cli.c
index 3fdb432..3a6f365 100644
--- a/bin/varnishd/mgt/mgt_cli.c
+++ b/bin/varnishd/mgt/mgt_cli.c
@@ -43,8 +43,8 @@
 #include <unistd.h>
 
 #include "mgt/mgt.h"
-#include "heritage.h"
-#include "params.h"
+#include "common/heritage.h"
+#include "common/params.h"
 
 #include "vcli.h"
 #include "vcli_common.h"
diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c
index ccda130..981841b 100644
--- a/bin/varnishd/mgt/mgt_main.c
+++ b/bin/varnishd/mgt/mgt_main.c
@@ -44,8 +44,8 @@
 #include <unistd.h>
 
 #include "mgt/mgt.h"
-#include "heritage.h"
-#include "params.h"
+#include "common/heritage.h"
+#include "common/params.h"
 
 #include "hash/hash_slinger.h"
 #include "vav.h"
diff --git a/bin/varnishd/mgt/mgt_param.c b/bin/varnishd/mgt/mgt_param.c
index 8fff352..8050d08 100644
--- a/bin/varnishd/mgt/mgt_param.c
+++ b/bin/varnishd/mgt/mgt_param.c
@@ -38,8 +38,8 @@
 #include <unistd.h>
 
 #include "mgt/mgt.h"
-#include "heritage.h"
-#include "params.h"
+#include "common/heritage.h"
+#include "common/params.h"
 
 #include "mgt/mgt_param.h"
 #include "waiter/cache_waiter.h"
diff --git a/bin/varnishd/mgt/mgt_pool.c b/bin/varnishd/mgt/mgt_pool.c
index 6f61553..e8c4f27 100644
--- a/bin/varnishd/mgt/mgt_pool.c
+++ b/bin/varnishd/mgt/mgt_pool.c
@@ -48,8 +48,8 @@
 #include <unistd.h>
 
 #include "mgt/mgt.h"
-#include "heritage.h"
-#include "params.h"
+#include "common/heritage.h"
+#include "common/params.h"
 
 #include "mgt/mgt_param.h"
 
diff --git a/bin/varnishd/mgt/mgt_sandbox.c b/bin/varnishd/mgt/mgt_sandbox.c
index a9fce93..cdba825 100644
--- a/bin/varnishd/mgt/mgt_sandbox.c
+++ b/bin/varnishd/mgt/mgt_sandbox.c
@@ -53,8 +53,8 @@
 #include <unistd.h>
 
 #include "mgt/mgt.h"
-#include "heritage.h"
-#include "params.h"
+#include "common/heritage.h"
+#include "common/params.h"
 
 /*--------------------------------------------------------------------*/
 
diff --git a/bin/varnishd/mgt/mgt_sandbox_solaris.c b/bin/varnishd/mgt/mgt_sandbox_solaris.c
index 8c26d69..79f6650 100644
--- a/bin/varnishd/mgt/mgt_sandbox_solaris.c
+++ b/bin/varnishd/mgt/mgt_sandbox_solaris.c
@@ -44,7 +44,7 @@
 
 #include "mgt/mgt.h"
 
-#include "heritage.h"
+#include "common/heritage.h"
 
 /*--------------------------------------------------------------------
  * SOLARIS PRIVILEGES: Note on use of symbolic PRIV_* constants
diff --git a/bin/varnishd/mgt/mgt_shmem.c b/bin/varnishd/mgt/mgt_shmem.c
index bb87a8f..d4346df 100644
--- a/bin/varnishd/mgt/mgt_shmem.c
+++ b/bin/varnishd/mgt/mgt_shmem.c
@@ -98,8 +98,8 @@
 #include <unistd.h>
 
 #include "mgt/mgt.h"
-#include "heritage.h"
-#include "params.h"
+#include "common/heritage.h"
+#include "common/params.h"
 
 #include "flopen.h"
 #include "vapi/vsc_int.h"
diff --git a/bin/varnishd/params.h b/bin/varnishd/params.h
deleted file mode 100644
index fb71e33..0000000
--- a/bin/varnishd/params.h
+++ /dev/null
@@ -1,194 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * This file contains the heritage passed when mgt forks cache
- */
-
-#include "vre.h"
-
-struct params {
-
-	/* Unprivileged user / group */
-	char			*user;
-	uid_t			uid;
-	char			*group;
-	gid_t			gid;
-
-	/* TTL used for lack of anything better */
-	double			default_ttl;
-
-	/* Default grace period */
-	double			default_grace;
-
-	/* Default keep period */
-	double			default_keep;
-
-	/* Maximum concurrent sessions */
-	unsigned		max_sess;
-
-	/* Worker threads and pool */
-	unsigned		wthread_min;
-	unsigned		wthread_max;
-	unsigned		wthread_timeout;
-	unsigned		wthread_pools;
-	unsigned		wthread_add_threshold;
-	unsigned		wthread_add_delay;
-	unsigned		wthread_fail_delay;
-	unsigned		wthread_purge_delay;
-	unsigned		wthread_stats_rate;
-	unsigned		wthread_stacksize;
-	unsigned		wthread_workspace;
-
-	unsigned		queue_max;
-
-	/* Memory allocation hints */
-	unsigned		sess_workspace;
-	unsigned		shm_workspace;
-	unsigned		http_req_size;
-	unsigned		http_req_hdr_len;
-	unsigned		http_resp_size;
-	unsigned		http_resp_hdr_len;
-	unsigned		http_max_hdr;
-
-	unsigned		shm_reclen;
-
-	/* Acceptor hints */
-	unsigned		sess_timeout;
-	unsigned		pipe_timeout;
-	unsigned		send_timeout;
-	unsigned		idle_send_timeout;
-
-	/* Management hints */
-	unsigned		auto_restart;
-
-	/* Fetcher hints */
-	unsigned		fetch_chunksize;
-	unsigned		fetch_maxchunksize;
-	unsigned		nuke_limit;
-
-#ifdef SENDFILE_WORKS
-	/* Sendfile object minimum size */
-	unsigned		sendfile_threshold;
-#endif
-
-	/* VCL traces */
-	unsigned		vcl_trace;
-
-	/* Listen address */
-	char			*listen_address;
-
-	/* Listen depth */
-	unsigned		listen_depth;
-
-	/* CLI related */
-	unsigned		cli_timeout;
-	unsigned		ping_interval;
-
-	/* LRU list ordering interval */
-	unsigned		lru_timeout;
-
-	/* Maximum restarts allowed */
-	unsigned		max_restarts;
-
-	/* Maximum esi:include depth allowed */
-	unsigned		max_esi_depth;
-
-	/* ESI parser hints */
-	unsigned		esi_syntax;
-
-	/* Rush exponent */
-	unsigned		rush_exponent;
-
-	/* Default connection_timeout */
-	double			connect_timeout;
-
-	/* Read timeouts for backend */
-	double			first_byte_timeout;
-	double			between_bytes_timeout;
-
-	/* How long to linger on sessions */
-	unsigned		session_linger;
-
-	/* CLI buffer size */
-	unsigned		cli_buffer;
-
-	/* Control diagnostic code */
-	unsigned		diag_bitmap;
-
-	/* Log hash string to shm */
-	unsigned		log_hash;
-
-	/* Log local socket address to shm */
-	unsigned		log_local_addr;
-
-	/* Prefer IPv6 connections to backend*/
-	unsigned		prefer_ipv6;
-
-	/* Acceptable clockskew with backends */
-	unsigned		clock_skew;
-
-	/* Expiry pacer parameters */
-	double			expiry_sleep;
-
-	/* Acceptor pacer parameters */
-	double			acceptor_sleep_max;
-	double			acceptor_sleep_incr;
-	double			acceptor_sleep_decay;
-
-	/* Get rid of duplicate bans */
-	unsigned		ban_dups;
-
-	/* How long time does the ban lurker sleep */
-	double			ban_lurker_sleep;
-
-	/* Max size of the saintmode list. 0 == no saint mode. */
-	unsigned		saintmode_threshold;
-
-	unsigned		syslog_cli_traffic;
-
-	unsigned		http_range_support;
-
-	unsigned		http_gzip_support;
-	unsigned		gzip_stack_buffer;
-	unsigned		gzip_tmp_space;
-	unsigned		gzip_level;
-	unsigned		gzip_window;
-	unsigned		gzip_memlevel;
-
-	double			critbit_cooloff;
-
-	double			shortlived;
-
-	struct vre_limits	vre_limits;
-};
-
-/*
- * We declare this a volatile pointer, so that reads of parameters
- * become atomic, leaving the CLI thread lattitude to change the values
- */
-extern volatile struct params * cache_param;
diff --git a/bin/varnishd/vsm.c b/bin/varnishd/vsm.c
deleted file mode 100644
index 5298381..0000000
--- a/bin/varnishd/vsm.c
+++ /dev/null
@@ -1,238 +0,0 @@
-/*-
- * Copyright (c) 2010-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * VSM stuff common to manager and child.
- *
- * We have three potential conflicts we need to lock against here:
- *
- * VSM-studying programs (varnishstat...) vs. everybody else
- *	The VSM studying programs only have read-only access to the VSM
- *	so everybody else must use memory barriers, stable storage and
- *	similar tricks to keep the VSM image in sync (long enough) for
- *	the studying programs.
- *
- * Manager process vs child process.
- *	Will only muck about in VSM when child process is not running
- *	Responsible for cleaning up any mess left behind by dying child.
- *
- * Child process threads
- *	Pthread locking necessary.
- *
- * XXX: not all of this is in place yet.
- */
-
-#include "config.h"
-
-#include <stdint.h>
-#include <stdio.h>
-#include <string.h>
-#include <unistd.h>
-
-#include "common.h"
-
-#include "vapi/vsm_int.h"
-#include "vmb.h"
-#include "vtim.h"
-
-/* These two come from beyond (mgt_shmem.c actually) */
-struct VSM_head		*VSM_head;
-const struct VSM_chunk	*vsm_end;
-
-static unsigned
-vsm_mark(void)
-{
-	unsigned seq;
-
-	seq = VSM_head->alloc_seq;
-	VSM_head->alloc_seq = 0;
-	VWMB();
-	return (seq);
-}
-
-static void
-vsm_release(unsigned seq)
-{
-
-	if (seq == 0)
-		return;
-	VWMB();
-	do
-		VSM_head->alloc_seq = ++seq;
-	while (VSM_head->alloc_seq == 0);
-}
-
-/*--------------------------------------------------------------------*/
-
-static void
-vsm_cleanup(void)
-{
-	unsigned now = (unsigned)VTIM_mono();
-	struct VSM_chunk *sha, *sha2;
-	unsigned seq;
-
-	CHECK_OBJ_NOTNULL(VSM_head, VSM_HEAD_MAGIC);
-	VSM_ITER(sha) {
-		if (strcmp(sha->class, VSM_CLASS_COOL))
-			continue;
-		if (sha->state + VSM_COOL_TIME < now)
-			break;
-	}
-	if (sha == NULL)
-		return;
-	seq = vsm_mark();
-	/* First pass, free, and collapse with next if applicable */
-	VSM_ITER(sha) {
-		if (strcmp(sha->class, VSM_CLASS_COOL))
-			continue;
-		if (sha->state + VSM_COOL_TIME >= now)
-			continue;
-
-		bprintf(sha->class, "%s", VSM_CLASS_FREE);
-		bprintf(sha->type, "%s", "");
-		bprintf(sha->ident, "%s", "");
-		sha2 = VSM_NEXT(sha);
-		assert(sha2 <= vsm_end);
-		if (sha2 == vsm_end)
-			break;
-		CHECK_OBJ_NOTNULL(sha2, VSM_CHUNK_MAGIC);
-		if (!strcmp(sha2->class, VSM_CLASS_FREE)) {
-			sha->len += sha2->len;
-			memset(sha2, 0, sizeof *sha2);
-		}
-		sha->state = 0;
-	}
-	/* Second pass, collaps with prev if applicable */
-	VSM_ITER(sha) {
-		if (strcmp(sha->class, VSM_CLASS_FREE))
-			continue;
-		sha2 = VSM_NEXT(sha);
-		assert(sha2 <= vsm_end);
-		if (sha2 == vsm_end)
-			break;
-		CHECK_OBJ_NOTNULL(sha2, VSM_CHUNK_MAGIC);
-		if (!strcmp(sha2->class, VSM_CLASS_FREE)) {
-			sha->len += sha2->len;
-			memset(sha2, 0, sizeof *sha2);
-		}
-	}
-	vsm_release(seq);
-}
-
-/*--------------------------------------------------------------------*/
-
-void *
-VSM__Alloc(unsigned size, const char *class, const char *type, const char *ident)
-{
-	struct VSM_chunk *sha, *sha2;
-	unsigned seq;
-
-	CHECK_OBJ_NOTNULL(VSM_head, VSM_HEAD_MAGIC);
-
-	vsm_cleanup();
-
-	/* Round up to pointersize */
-	size = RUP2(size, sizeof(void*));
-
-	size += sizeof *sha;		/* Make space for the header */
-
-	VSM_ITER(sha) {
-		CHECK_OBJ_NOTNULL(sha, VSM_CHUNK_MAGIC);
-
-		if (strcmp(sha->class, VSM_CLASS_FREE))
-			continue;
-
-		if (size > sha->len)
-			continue;
-
-		/* Mark as inconsistent while we write string fields */
-		seq = vsm_mark();
-
-		if (size + sizeof (*sha) < sha->len) {
-			sha2 = (void*)((uintptr_t)sha + size);
-
-			memset(sha2, 0, sizeof *sha2);
-			sha2->magic = VSM_CHUNK_MAGIC;
-			sha2->len = sha->len - size;
-			bprintf(sha2->class, "%s", VSM_CLASS_FREE);
-			sha->len = size;
-		}
-
-		bprintf(sha->class, "%s", class);
-		bprintf(sha->type, "%s", type);
-		bprintf(sha->ident, "%s", ident);
-
-		vsm_release(seq);
-		return (VSM_PTR(sha));
-	}
-	return (NULL);
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-VSM__Free(const void *ptr)
-{
-	struct VSM_chunk *sha;
-	unsigned seq;
-
-	CHECK_OBJ_NOTNULL(VSM_head, VSM_HEAD_MAGIC);
-	VSM_ITER(sha)
-		if (VSM_PTR(sha) == ptr)
-			break;
-	AN(sha);
-	seq = vsm_mark();
-	bprintf(sha->class, "%s", VSM_CLASS_COOL);
-	sha->state = (unsigned)VTIM_mono();
-	vsm_release(seq);
-}
-
-/*--------------------------------------------------------------------
- * Free all allocations after the mark (ie: allocated by child).
- */
-
-void
-VSM__Clean(void)
-{
-	struct VSM_chunk *sha;
-	unsigned f, seq;
-
-	CHECK_OBJ_NOTNULL(VSM_head, VSM_HEAD_MAGIC);
-	f = 0;
-	seq = vsm_mark();
-	VSM_ITER(sha) {
-		if (f == 0 && !strcmp(sha->class, VSM_CLASS_MARK)) {
-			f = 1;
-			continue;
-		}
-		if (f == 0)
-			continue;
-		if (strcmp(sha->class, VSM_CLASS_FREE) &&
-		    strcmp(sha->class, VSM_CLASS_COOL))
-			VSM__Free(VSM_PTR(sha));
-	}
-	vsm_release(seq);
-}



From phk at varnish-cache.org  Sun Nov 13 10:56:49 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Sun, 13 Nov 2011 11:56:49 +0100
Subject: [master] f3a414f Stuff all of the worker process into a cache/
	subdirectory.
Message-ID: <E1RPXk1-0005jq-Ul@project.varnish-software.com>

commit f3a414fbf75eb4e4927aa7d46bd7e92a8abf5fd4
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Nov 13 10:55:14 2011 +0000

    Stuff all of the worker process into a cache/ subdirectory.
    
    Yes, I should have done this five years ago...

diff --git a/bin/varnishd/Makefile.am b/bin/varnishd/Makefile.am
index 245bc6b..850a18a 100644
--- a/bin/varnishd/Makefile.am
+++ b/bin/varnishd/Makefile.am
@@ -3,6 +3,7 @@
 INCLUDES = \
 	-I$(top_srcdir)/include \
 	-I$(top_srcdir)/lib/libvgz \
+	-I$(top_builddir)/bin/varnishd \
 	-I$(top_builddir)/include
 
 sbin_PROGRAMS = varnishd
@@ -10,44 +11,44 @@ sbin_PROGRAMS = varnishd
 dist_man_MANS = varnishd.1
 
 varnishd_SOURCES = \
-	cache_acceptor.c \
-	cache_backend.c \
-	cache_backend_cfg.c \
-	cache_backend_poll.c \
-	cache_ban.c \
-	cache_center.c \
-	cache_cli.c \
-	cache_dir.c \
-	cache_dir_dns.c \
-	cache_dir_random.c \
-	cache_dir_round_robin.c \
-	cache_esi_deliver.c \
-	cache_esi_fetch.c \
-	cache_esi_parse.c \
-	cache_expire.c \
-	cache_fetch.c \
-	cache_gzip.c \
-	cache_hash.c \
-	cache_http.c \
-	cache_httpconn.c \
-	cache_lck.c \
-	cache_main.c \
-	cache_panic.c \
-	cache_pipe.c \
-	cache_pool.c \
-	cache_response.c \
-	cache_rfc2616.c \
-	cache_session.c \
-	cache_shmlog.c \
-	cache_vary.c \
-	cache_vcl.c \
-	cache_vrt.c \
-	cache_vrt_re.c \
-	cache_vrt_var.c \
-	cache_vrt_vmod.c \
-	cache_wrk.c \
-	cache_wrw.c \
-	cache_ws.c \
+	cache/cache_acceptor.c \
+	cache/cache_backend.c \
+	cache/cache_backend_cfg.c \
+	cache/cache_backend_poll.c \
+	cache/cache_ban.c \
+	cache/cache_center.c \
+	cache/cache_cli.c \
+	cache/cache_dir.c \
+	cache/cache_dir_dns.c \
+	cache/cache_dir_random.c \
+	cache/cache_dir_round_robin.c \
+	cache/cache_esi_deliver.c \
+	cache/cache_esi_fetch.c \
+	cache/cache_esi_parse.c \
+	cache/cache_expire.c \
+	cache/cache_fetch.c \
+	cache/cache_gzip.c \
+	cache/cache_hash.c \
+	cache/cache_http.c \
+	cache/cache_httpconn.c \
+	cache/cache_lck.c \
+	cache/cache_main.c \
+	cache/cache_panic.c \
+	cache/cache_pipe.c \
+	cache/cache_pool.c \
+	cache/cache_response.c \
+	cache/cache_rfc2616.c \
+	cache/cache_session.c \
+	cache/cache_shmlog.c \
+	cache/cache_vary.c \
+	cache/cache_vcl.c \
+	cache/cache_vrt.c \
+	cache/cache_vrt_re.c \
+	cache/cache_vrt_var.c \
+	cache/cache_vrt_vmod.c \
+	cache/cache_wrk.c \
+	cache/cache_wrw.c \
+	cache/cache_ws.c \
 	common/common_vsm.c \
 	hash/hash_classic.c \
 	hash/hash_critbit.c \
@@ -79,9 +80,9 @@ varnishd_SOURCES = \
 	waiter/cache_waiter_ports.c 
 
 noinst_HEADERS = \
-	cache.h \
-	cache_backend.h \
-	cache_esi.h \
+	cache/cache.h \
+	cache/cache_backend.h \
+	cache/cache_esi.h \
 	common/common.h \
 	common/heritage.h \
 	common/params.h \
diff --git a/bin/varnishd/cache.h b/bin/varnishd/cache.h
deleted file mode 100644
index 653fe77..0000000
--- a/bin/varnishd/cache.h
+++ /dev/null
@@ -1,1039 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- */
-
-/*
- * This macro can be used in .h files to isolate bits that the manager
- * should not (need to) see, such as pthread mutexes etc.
- */
-#define VARNISH_CACHE_CHILD	1
-
-#include "common/common.h"
-
-#include "vapi/vsc_int.h"
-#include "vapi/vsl_int.h"
-
-#include <sys/socket.h>
-
-#include <pthread.h>
-#ifdef HAVE_PTHREAD_NP_H
-#include <pthread_np.h>
-#endif
-#include <stdarg.h>
-#include <string.h>
-#include <limits.h>
-#include <unistd.h>
-
-#if defined(HAVE_EPOLL_CTL)
-#include <sys/epoll.h>
-#endif
-
-
-#include "common/params.h"
-
-enum body_status {
-#define BODYSTATUS(U,l)	BS_##U,
-#include "tbl/body_status.h"
-#undef BODYSTATUS
-};
-
-static inline const char *
-body_status(enum body_status e)
-{
-	switch(e) {
-#define BODYSTATUS(U,l)	case BS_##U: return (#l);
-#include "tbl/body_status.h"
-#undef BODYSTATUS
-	default:
-		return ("?");
-	}
-}
-
-/*
- * NB: HDR_STATUS is only used in cache_http.c, everybody else uses the
- * http->status integer field.
- */
-
-enum {
-	/* Fields from the first line of HTTP proto */
-	HTTP_HDR_REQ,
-	HTTP_HDR_URL,
-	HTTP_HDR_PROTO,
-	HTTP_HDR_STATUS,
-	HTTP_HDR_RESPONSE,
-	/* HTTP header lines */
-	HTTP_HDR_FIRST,
-};
-
-struct SHA256Context;
-struct VSC_C_lck;
-struct ban;
-struct busyobj;
-struct cli;
-struct cli_proto;
-struct director;
-struct iovec;
-struct objcore;
-struct object;
-struct objhead;
-struct pool;
-struct sess;
-struct sesspool;
-struct vbc;
-struct vef_priv;
-struct vrt_backend;
-struct vsb;
-struct waitinglist;
-struct worker;
-
-#define DIGEST_LEN		32
-
-/* Name of transient storage */
-#define TRANSIENT_STORAGE	"Transient"
-
-/*--------------------------------------------------------------------
- * Pointer aligment magic
- */
-
-#define PALGN		(sizeof(void *) - 1)
-#define PAOK(p)		(((uintptr_t)(p) & PALGN) == 0)
-#define PRNDDN(p)	((uintptr_t)(p) & ~PALGN)
-#define PRNDUP(p)	(((uintptr_t)(p) + PALGN) & ~PALGN)
-
-/*--------------------------------------------------------------------*/
-
-typedef struct {
-	char			*b;
-	char			*e;
-} txt;
-
-/*--------------------------------------------------------------------*/
-
-enum step {
-#define STEP(l, u)	STP_##u,
-#include "tbl/steps.h"
-#undef STEP
-};
-
-/*--------------------------------------------------------------------
- * Workspace structure for quick memory allocation.
- */
-
-struct ws {
-	unsigned		magic;
-#define WS_MAGIC		0x35fac554
-	unsigned		overflow;	/* workspace overflowed */
-	const char		*id;		/* identity */
-	char			*s;		/* (S)tart of buffer */
-	char			*f;		/* (F)ree pointer */
-	char			*r;		/* (R)eserved length */
-	char			*e;		/* (E)nd of buffer */
-};
-
-/*--------------------------------------------------------------------
- * HTTP Request/Response/Header handling structure.
- */
-
-enum httpwhence {
-	HTTP_Rx	 = 1,
-	HTTP_Tx  = 2,
-	HTTP_Obj = 3
-};
-
-/* NB: remember to update http_Copy() if you add fields */
-struct http {
-	unsigned		magic;
-#define HTTP_MAGIC		0x6428b5c9
-
-	enum httpwhence		logtag;
-
-	struct ws		*ws;
-	txt			*hd;
-	unsigned char		*hdf;
-#define HDF_FILTER		(1 << 0)	/* Filtered by Connection */
-	uint16_t		shd;		/* Size of hd space */
-	uint16_t		nhd;		/* Next free hd */
-	uint16_t		status;
-	uint8_t			protover;
-	uint8_t			conds;		/* If-* headers present */
-};
-
-/*--------------------------------------------------------------------
- * HTTP Protocol connection structure
- */
-
-struct http_conn {
-	unsigned		magic;
-#define HTTP_CONN_MAGIC		0x3e19edd1
-
-	int			fd;
-	unsigned		vsl_id;
-	unsigned		maxbytes;
-	unsigned		maxhdr;
-	struct ws		*ws;
-	txt			rxbuf;
-	txt			pipeline;
-};
-
-/*--------------------------------------------------------------------*/
-
-struct acct {
-	double			first;
-#define ACCT(foo)	uint64_t	foo;
-#include "tbl/acct_fields.h"
-#undef ACCT
-};
-
-/*--------------------------------------------------------------------*/
-
-#define L0(t, n)
-#define L1(t, n)		t n;
-#define VSC_F(n, t, l, f, e,d)	L##l(t, n)
-#define VSC_DO_MAIN
-struct dstat {
-#include "tbl/vsc_fields.h"
-};
-#undef VSC_F
-#undef VSC_DO_MAIN
-#undef L0
-#undef L1
-
-/* Fetch processors --------------------------------------------------*/
-
-typedef void vfp_begin_f(struct worker *, size_t );
-typedef int vfp_bytes_f(struct worker *, struct http_conn *, ssize_t);
-typedef int vfp_end_f(struct worker *);
-
-struct vfp {
-	vfp_begin_f	*begin;
-	vfp_bytes_f	*bytes;
-	vfp_end_f	*end;
-};
-
-extern struct vfp vfp_gunzip;
-extern struct vfp vfp_gzip;
-extern struct vfp vfp_testgzip;
-extern struct vfp vfp_esi;
-
-/*--------------------------------------------------------------------*/
-
-struct exp {
-	double			ttl;
-	double			grace;
-	double			keep;
-	double			age;
-	double			entered;
-};
-
-/*--------------------------------------------------------------------*/
-
-struct wrw {
-	int			*wfd;
-	unsigned		werr;	/* valid after WRW_Flush() */
-	struct iovec		*iov;
-	unsigned		siov;
-	unsigned		niov;
-	ssize_t			liov;
-	ssize_t			cliov;
-	unsigned		ciov;	/* Chunked header marker */
-};
-
-/*--------------------------------------------------------------------*/
-
-struct stream_ctx {
-	unsigned		magic;
-#define STREAM_CTX_MAGIC	0x8213728b
-
-	struct vgz		*vgz;
-	void			*obuf;
-	ssize_t			obuf_len;
-	ssize_t			obuf_ptr;
-
-	/* Next byte we will take from storage */
-	ssize_t			stream_next;
-
-	/* First byte of storage if we free it as we go (pass) */
-	ssize_t			stream_front;
-};
-
-/*--------------------------------------------------------------------*/
-
-struct wrk_accept {
-	unsigned		magic;
-#define WRK_ACCEPT_MAGIC	0x8c4b4d59
-
-	/* Accept stuff */
-	struct sockaddr_storage	acceptaddr;
-	socklen_t		acceptaddrlen;
-	int			acceptsock;
-	struct listen_sock	*acceptlsock;
-};
-
-/*--------------------------------------------------------------------*/
-
-struct worker {
-	unsigned		magic;
-#define WORKER_MAGIC		0x6391adcf
-	struct pool		*pool;
-	struct objhead		*nobjhead;
-	struct objcore		*nobjcore;
-	struct waitinglist	*nwaitinglist;
-	struct busyobj		*nbusyobj;
-	void			*nhashpriv;
-	struct dstat		stats;
-
-	/* Pool stuff */
-	double			lastused;
-
-	struct wrw		wrw;
-
-	pthread_cond_t		cond;
-
-	VTAILQ_ENTRY(worker)	list;
-	struct sess		*sp;
-
-	struct VCL_conf		*vcl;
-
-	uint32_t		*wlb, *wlp, *wle;
-	unsigned		wlr;
-
-	/* Lookup stuff */
-	struct SHA256Context	*sha256ctx;
-
-	struct http_conn	htc[1];
-	struct ws		ws[1];
-	struct http		*bereq;
-	struct http		*beresp;
-	struct http		*resp;
-
-	struct exp		exp;
-
-	/* This is only here so VRT can find it */
-	const char		*storage_hint;
-
-	/* Fetch stuff */
-	struct vbc		*vbc;
-	struct object		*fetch_obj;
-	enum body_status	body_status;
-	struct vfp		*vfp;
-	struct vgz		*vgz_rx;
-	struct vef_priv		*vef_priv;
-	unsigned		fetch_failed;
-	unsigned		do_stream;
-	unsigned		do_esi;
-	unsigned		do_gzip;
-	unsigned		is_gzip;
-	unsigned		do_gunzip;
-	unsigned		is_gunzip;
-	unsigned		do_close;
-	char			*h_content_length;
-
-	/* Stream state */
-	struct stream_ctx	*sctx;
-
-	/* ESI stuff */
-	struct vep_state	*vep;
-	int			gzip_resp;
-	ssize_t			l_crc;
-	uint32_t		crc;
-
-	/* Timeouts */
-	double			connect_timeout;
-	double			first_byte_timeout;
-	double			between_bytes_timeout;
-
-	/* Delivery mode */
-	unsigned		res_mode;
-#define RES_LEN			(1<<1)
-#define RES_EOF			(1<<2)
-#define RES_CHUNKED		(1<<3)
-#define RES_ESI			(1<<4)
-#define RES_ESI_CHILD		(1<<5)
-#define RES_GUNZIP		(1<<6)
-
-	/* Temporary accounting */
-	struct acct		acct_tmp;
-};
-
-/* LRU ---------------------------------------------------------------*/
-
-struct lru {
-	unsigned		magic;
-#define LRU_MAGIC		0x3fec7bb0
-	VTAILQ_HEAD(,objcore)	lru_head;
-	struct lock		mtx;
-};
-
-/* Storage -----------------------------------------------------------*/
-
-struct storage {
-	unsigned		magic;
-#define STORAGE_MAGIC		0x1a4e51c0
-
-#ifdef SENDFILE_WORKS
-	int			fd;
-	off_t			where;
-#endif
-
-	VTAILQ_ENTRY(storage)	list;
-	struct stevedore	*stevedore;
-	void			*priv;
-
-	unsigned char		*ptr;
-	unsigned		len;
-	unsigned		space;
-};
-
-/* Object core structure ---------------------------------------------
- * Objects have sideways references in the binary heap and the LRU list
- * and we want to avoid paging in a lot of objects just to move them up
- * or down the binheap or to move a unrelated object on the LRU list.
- * To avoid this we use a proxy object, objcore, to hold the relevant
- * housekeeping fields parts of an object.
- */
-
-typedef struct object *getobj_f(struct worker *wrk, struct objcore *oc);
-typedef void updatemeta_f(struct objcore *oc);
-typedef void freeobj_f(struct objcore *oc);
-typedef struct lru *getlru_f(const struct objcore *oc);
-
-struct objcore_methods {
-	getobj_f	*getobj;
-	updatemeta_f	*updatemeta;
-	freeobj_f	*freeobj;
-	getlru_f	*getlru;
-};
-
-struct objcore {
-	unsigned		magic;
-#define OBJCORE_MAGIC		0x4d301302
-	unsigned		refcnt;
-	struct objcore_methods	*methods;
-	void			*priv;
-	unsigned		priv2;
-	struct objhead		*objhead;
-	struct busyobj		*busyobj;
-	double			timer_when;
-	unsigned		flags;
-#define OC_F_BUSY		(1<<1)
-#define OC_F_PASS		(1<<2)
-#define OC_F_LRUDONTMOVE	(1<<4)
-#define OC_F_PRIV		(1<<5)		/* Stevedore private flag */
-#define OC_F_LURK		(3<<6)		/* Ban-lurker-color */
-	unsigned		timer_idx;
-	VTAILQ_ENTRY(objcore)	list;
-	VTAILQ_ENTRY(objcore)	lru_list;
-	VTAILQ_ENTRY(objcore)	ban_list;
-	struct ban		*ban;
-};
-
-static inline struct object *
-oc_getobj(struct worker *wrk, struct objcore *oc)
-{
-
-	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-	AZ(oc->flags & OC_F_BUSY);
-	AN(oc->methods);
-	AN(oc->methods->getobj);
-	return (oc->methods->getobj(wrk, oc));
-}
-
-static inline void
-oc_updatemeta(struct objcore *oc)
-{
-
-	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-	AN(oc->methods);
-	if (oc->methods->updatemeta != NULL)
-		oc->methods->updatemeta(oc);
-}
-
-static inline void
-oc_freeobj(struct objcore *oc)
-{
-
-	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-	AN(oc->methods);
-	AN(oc->methods->freeobj);
-	oc->methods->freeobj(oc);
-}
-
-static inline struct lru *
-oc_getlru(const struct objcore *oc)
-{
-
-	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-	AN(oc->methods);
-	AN(oc->methods->getlru);
-	return (oc->methods->getlru(oc));
-}
-
-/* Busy Object structure ---------------------------------------------*/
-
-struct busyobj {
-	unsigned		magic;
-#define BUSYOBJ_MAGIC		0x23b95567
-	uint8_t			*vary;
-};
-
-/* Object structure --------------------------------------------------*/
-
-VTAILQ_HEAD(storagehead, storage);
-
-struct object {
-	unsigned		magic;
-#define OBJECT_MAGIC		0x32851d42
-	unsigned		xid;
-	struct storage		*objstore;
-	struct objcore		*objcore;
-
-	struct ws		ws_o[1];
-
-	uint8_t			*vary;
-	unsigned		hits;
-	uint16_t		response;
-
-	/* XXX: make bitmap */
-	uint8_t			gziped;
-	/* Bit positions in the gzip stream */
-	ssize_t			gzip_start;
-	ssize_t			gzip_last;
-	ssize_t			gzip_stop;
-
-	ssize_t			len;
-
-	struct exp		exp;
-
-	double			last_modified;
-	double			last_lru;
-
-	struct http		*http;
-
-	struct storagehead	store;
-
-	struct storage		*esidata;
-
-	double			last_use;
-
-};
-
-/* -------------------------------------------------------------------*/
-
-struct sess {
-	unsigned		magic;
-#define SESS_MAGIC		0x2c2f9c5a
-	int			fd;
-	unsigned		vsl_id;
-	unsigned		xid;
-
-	int			restarts;
-	int			esi_level;
-	int			disable_esi;
-
-	uint8_t			hash_ignore_busy;
-	uint8_t			hash_always_miss;
-
-	struct worker		*wrk;
-
-	socklen_t		sockaddrlen;
-	socklen_t		mysockaddrlen;
-	struct sockaddr_storage	sockaddr;
-	struct sockaddr_storage	mysockaddr;
-	struct listen_sock	*mylsock;
-
-	/* formatted ascii client address */
-	char			*addr;
-	char			*port;
-	char			*client_identity;
-
-	/* HTTP request */
-	const char		*doclose;
-	struct http		*http;
-	struct http		*http0;
-
-	struct ws		ws[1];
-	char			*ws_ses;	/* WS above session data */
-	char			*ws_req;	/* WS above request data */
-
-	unsigned char		digest[DIGEST_LEN];
-
-	/* Built Vary string */
-	uint8_t			*vary_b;
-	uint8_t			*vary_l;
-	uint8_t			*vary_e;
-
-	struct http_conn	htc[1];
-
-	/* Timestamps, all on TIM_real() timescale */
-	double			t_open;
-	double			t_req;
-	double			t_resp;
-	double			t_end;
-
-	/* Acceptable grace period */
-	struct exp		exp;
-
-	enum step		step;
-	unsigned		cur_method;
-	unsigned		handling;
-	unsigned char		sendbody;
-	unsigned char		wantbody;
-	uint16_t		err_code;
-	const char		*err_reason;
-
-	VTAILQ_ENTRY(sess)	list;
-
-	struct director		*director;
-	struct object		*obj;
-	struct objcore		*objcore;
-	struct VCL_conf		*vcl;
-
-	/* The busy objhead we sleep on */
-	struct objhead		*hash_objhead;
-
-	/* Various internal stuff */
-	struct sessmem		*mem;
-
-	VTAILQ_ENTRY(sess)	poollist;
-	uint64_t		req_bodybytes;
-	struct acct		acct_ses;
-
-#if defined(HAVE_EPOLL_CTL)
-	struct epoll_event ev;
-#endif
-};
-
-/* Prototypes etc ----------------------------------------------------*/
-
-/* cache_acceptor.c */
-void VCA_Prep(struct sess *sp);
-void VCA_Init(void);
-void VCA_Shutdown(void);
-int VCA_Accept(struct listen_sock *ls, struct wrk_accept *wa);
-void VCA_SetupSess(struct worker *w);
-void VCA_FailSess(struct worker *w);
-
-/* cache_backend.c */
-void VBE_UseHealth(const struct director *vdi);
-
-struct vbc *VDI_GetFd(const struct director *, struct sess *sp);
-int VDI_Healthy(const struct director *, const struct sess *sp);
-void VDI_CloseFd(struct worker *wrk);
-void VDI_RecycleFd(struct worker *wrk);
-void VDI_AddHostHeader(const struct sess *sp);
-void VBE_Poll(void);
-
-/* cache_backend_cfg.c */
-void VBE_Init(void);
-struct backend *VBE_AddBackend(struct cli *cli, const struct vrt_backend *vb);
-
-/* cache_backend_poll.c */
-void VBP_Init(void);
-
-/* cache_ban.c */
-struct ban *BAN_New(void);
-int BAN_AddTest(struct cli *, struct ban *, const char *, const char *,
-    const char *);
-void BAN_Free(struct ban *b);
-void BAN_Insert(struct ban *b);
-void BAN_Init(void);
-void BAN_NewObjCore(struct objcore *oc);
-void BAN_DestroyObj(struct objcore *oc);
-int BAN_CheckObject(struct object *o, const struct sess *sp);
-void BAN_Reload(const uint8_t *ban, unsigned len);
-struct ban *BAN_TailRef(void);
-void BAN_Compile(void);
-struct ban *BAN_RefBan(struct objcore *oc, double t0, const struct ban *tail);
-void BAN_TailDeref(struct ban **ban);
-double BAN_Time(const struct ban *ban);
-
-/* cache_center.c [CNT] */
-void CNT_Session(struct sess *sp);
-void CNT_Init(void);
-
-/* cache_cli.c [CLI] */
-void CLI_Init(void);
-void CLI_Run(void);
-void CLI_AddFuncs(struct cli_proto *p);
-extern pthread_t cli_thread;
-#define ASSERT_CLI() do {assert(pthread_self() == cli_thread);} while (0)
-
-/* cache_expiry.c */
-void EXP_Clr(struct exp *e);
-double EXP_Get_ttl(const struct exp *e);
-double EXP_Get_grace(const struct exp *e);
-double EXP_Get_keep(const struct exp *e);
-void EXP_Set_ttl(struct exp *e, double v);
-void EXP_Set_grace(struct exp *e, double v);
-void EXP_Set_keep(struct exp *e, double v);
-
-double EXP_Ttl(const struct sess *, const struct object*);
-double EXP_Grace(const struct sess *, const struct object*);
-void EXP_Insert(struct object *o);
-void EXP_Inject(struct objcore *oc, struct lru *lru, double when);
-void EXP_Init(void);
-void EXP_Rearm(const struct object *o);
-int EXP_Touch(struct objcore *oc);
-int EXP_NukeOne(struct worker *w, struct lru *lru);
-
-/* cache_fetch.c */
-struct storage *FetchStorage(struct worker *w, ssize_t sz);
-int FetchError(struct worker *w, const char *error);
-int FetchError2(struct worker *w, const char *error, const char *more);
-int FetchHdr(struct sess *sp);
-int FetchBody(struct worker *w, struct object *obj);
-int FetchReqBody(struct sess *sp);
-void Fetch_Init(void);
-
-/* cache_gzip.c */
-struct vgz;
-
-enum vgz_flag { VGZ_NORMAL, VGZ_ALIGN, VGZ_RESET, VGZ_FINISH };
-struct vgz *VGZ_NewUngzip(struct worker *wrk, const char *id);
-struct vgz *VGZ_NewGzip(struct worker *wrk, const char *id);
-void VGZ_Ibuf(struct vgz *, const void *, ssize_t len);
-int VGZ_IbufEmpty(const struct vgz *vg);
-void VGZ_Obuf(struct vgz *, void *, ssize_t len);
-int VGZ_ObufFull(const struct vgz *vg);
-int VGZ_ObufStorage(struct worker *w, struct vgz *vg);
-int VGZ_Gzip(struct vgz *, const void **, size_t *len, enum vgz_flag);
-int VGZ_Gunzip(struct vgz *, const void **, size_t *len);
-int VGZ_Destroy(struct vgz **, int vsl_id);
-void VGZ_UpdateObj(const struct vgz*, struct object *);
-int VGZ_WrwGunzip(struct worker *w, struct vgz *, const void *ibuf,
-    ssize_t ibufl, char *obuf, ssize_t obufl, ssize_t *obufp);
-
-/* Return values */
-#define VGZ_ERROR	-1
-#define VGZ_OK		0
-#define VGZ_END		1
-#define VGZ_STUCK	2
-
-/* cache_http.c */
-unsigned HTTP_estimate(unsigned nhttp);
-void HTTP_Copy(struct http *to, const struct http * const fm);
-struct http *HTTP_create(void *p, uint16_t nhttp);
-const char *http_StatusMessage(unsigned);
-unsigned http_EstimateWS(const struct http *fm, unsigned how, uint16_t *nhd);
-void HTTP_Init(void);
-void http_ClrHeader(struct http *to);
-unsigned http_Write(struct worker *w, unsigned vsl_id, const struct http *hp,
-    int resp);
-void http_CopyResp(struct http *to, const struct http *fm);
-void http_SetResp(struct http *to, const char *proto, uint16_t status,
-    const char *response);
-void http_FilterFields(struct worker *w, unsigned vsl_id, struct http *to,
-    const struct http *fm, unsigned how);
-void http_FilterHeader(const struct sess *sp, unsigned how);
-void http_PutProtocol(struct worker *w, unsigned vsl_id, const struct http *to,
-    const char *protocol);
-void http_PutStatus(struct http *to, uint16_t status);
-void http_PutResponse(struct worker *w, unsigned vsl_id, const struct http *to,
-    const char *response);
-void http_PrintfHeader(struct worker *w, unsigned vsl_id, struct http *to,
-    const char *fmt, ...);
-void http_SetHeader(struct worker *w, unsigned vsl_id, struct http *to,
-    const char *hdr);
-void http_SetH(const struct http *to, unsigned n, const char *fm);
-void http_ForceGet(const struct http *to);
-void http_Setup(struct http *ht, struct ws *ws);
-int http_GetHdr(const struct http *hp, const char *hdr, char **ptr);
-int http_GetHdrData(const struct http *hp, const char *hdr,
-    const char *field, char **ptr);
-int http_GetHdrField(const struct http *hp, const char *hdr,
-    const char *field, char **ptr);
-double http_GetHdrQ(const struct http *hp, const char *hdr, const char *field);
-uint16_t http_GetStatus(const struct http *hp);
-const char *http_GetReq(const struct http *hp);
-int http_HdrIs(const struct http *hp, const char *hdr, const char *val);
-uint16_t http_DissectRequest(struct sess *sp);
-uint16_t http_DissectResponse(struct worker *w, const struct http_conn *htc,
-    struct http *sp);
-const char *http_DoConnection(const struct http *hp);
-void http_CopyHome(struct worker *w, unsigned vsl_id, const struct http *hp);
-void http_Unset(struct http *hp, const char *hdr);
-void http_CollectHdr(struct http *hp, const char *hdr);
-
-/* cache_httpconn.c */
-void HTC_Init(struct http_conn *htc, struct ws *ws, int fd, unsigned vsl_id,
-    unsigned maxbytes, unsigned maxhdr);
-int HTC_Reinit(struct http_conn *htc);
-int HTC_Rx(struct http_conn *htc);
-ssize_t HTC_Read(struct worker *w, struct http_conn *htc, void *d, size_t len);
-int HTC_Complete(struct http_conn *htc);
-
-#define HTTPH(a, b, c, d, e, f, g) extern char b[];
-#include "tbl/http_headers.h"
-#undef HTTPH
-
-/* cache_main.c */
-void THR_SetName(const char *name);
-const char* THR_GetName(void);
-void THR_SetSession(const struct sess *sp);
-const struct sess * THR_GetSession(void);
-
-/* cache_lck.c */
-
-/* Internal functions, call only through macros below */
-void Lck__Lock(struct lock *lck, const char *p, const char *f, int l);
-void Lck__Unlock(struct lock *lck, const char *p, const char *f, int l);
-int Lck__Trylock(struct lock *lck, const char *p, const char *f, int l);
-void Lck__New(struct lock *lck, struct VSC_C_lck *, const char *);
-void Lck__Assert(const struct lock *lck, int held);
-
-/* public interface: */
-void LCK_Init(void);
-void Lck_Delete(struct lock *lck);
-int Lck_CondWait(pthread_cond_t *cond, struct lock *lck, struct timespec *ts);
-
-#define Lck_New(a, b) Lck__New(a, b, #b)
-#define Lck_Lock(a) Lck__Lock(a, __func__, __FILE__, __LINE__)
-#define Lck_Unlock(a) Lck__Unlock(a, __func__, __FILE__, __LINE__)
-#define Lck_Trylock(a) Lck__Trylock(a, __func__, __FILE__, __LINE__)
-#define Lck_AssertHeld(a) Lck__Assert(a, 1)
-
-#define LOCK(nam) extern struct VSC_C_lck *lck_##nam;
-#include "tbl/locks.h"
-#undef LOCK
-
-/* cache_panic.c */
-void PAN_Init(void);
-
-/* cache_pipe.c */
-void PipeSession(struct sess *sp);
-
-/* cache_pool.c */
-void Pool_Init(void);
-void Pool_Work_Thread(void *priv, struct worker *w);
-void Pool_Wait(struct sess *sp);
-int Pool_Schedule(struct pool *pp, struct sess *sp);
-
-#define WRW_IsReleased(w)	((w)->wrw.wfd == NULL)
-int WRW_Error(const struct worker *w);
-void WRW_Chunked(struct worker *w);
-void WRW_EndChunk(struct worker *w);
-void WRW_Reserve(struct worker *w, int *fd);
-unsigned WRW_Flush(struct worker *w);
-unsigned WRW_FlushRelease(struct worker *w);
-unsigned WRW_Write(struct worker *w, const void *ptr, int len);
-unsigned WRW_WriteH(struct worker *w, const txt *hh, const char *suf);
-#ifdef SENDFILE_WORKS
-void WRW_Sendfile(struct worker *w, int fd, off_t off, unsigned len);
-#endif  /* SENDFILE_WORKS */
-
-/* cache_session.c [SES] */
-struct sess *SES_New(struct worker *wrk, struct sesspool *pp);
-struct sess *SES_Alloc(void);
-void SES_Close(struct sess *sp, const char *reason);
-void SES_Delete(struct sess *sp, const char *reason);
-void SES_Charge(struct sess *sp);
-struct sesspool *SES_NewPool(struct pool *pp);
-void SES_DeletePool(struct sesspool *sp, struct worker *wrk);
-int SES_Schedule(struct sess *sp);
-
-
-/* cache_shmlog.c */
-void VSL_Init(void);
-void *VSM_Alloc(unsigned size, const char *class, const char *type,
-    const char *ident);
-void VSM_Free(const void *ptr);
-#ifdef VSL_ENDMARKER
-void VSL(enum VSL_tag_e tag, int id, const char *fmt, ...);
-void WSLR(struct worker *w, enum VSL_tag_e tag, int id, txt t);
-void WSL(struct worker *w, enum VSL_tag_e tag, int id, const char *fmt, ...);
-void WSLB(struct worker *w, enum VSL_tag_e tag, const char *fmt, ...);
-
-void WSL_Flush(struct worker *w, int overflow);
-
-#define DSL(flag, tag, id, ...)					\
-	do {							\
-		if (cache_param->diag_bitmap & (flag))		\
-			VSL((tag), (id), __VA_ARGS__);		\
-	} while (0)
-
-#define WSP(sess, tag, ...)					\
-	WSL((sess)->wrk, tag, (sess)->vsl_id, __VA_ARGS__)
-
-#define WSPR(sess, tag, txt)					\
-	WSLR((sess)->wrk, tag, (sess)->vsl_id, txt)
-
-#define INCOMPL() do {							\
-	VSL(SLT_Debug, 0, "INCOMPLETE AT: %s(%d)", __func__, __LINE__); \
-	fprintf(stderr,							\
-	    "INCOMPLETE AT: %s(%d)\n",					\
-	    (const char *)__func__, __LINE__);				\
-	abort();							\
-	} while (0)
-#endif
-
-/* cache_response.c */
-void RES_BuildHttp(const struct sess *sp);
-void RES_WriteObj(struct sess *sp);
-void RES_StreamStart(struct sess *sp);
-void RES_StreamEnd(struct sess *sp);
-void RES_StreamPoll(struct worker *);
-
-/* cache_vary.c */
-struct vsb *VRY_Create(const struct sess *sp, const struct http *hp);
-int VRY_Match(struct sess *sp, const uint8_t *vary);
-void VRY_Validate(const uint8_t *vary);
-
-/* cache_vcl.c */
-void VCL_Init(void);
-void VCL_Refresh(struct VCL_conf **vcc);
-void VCL_Rel(struct VCL_conf **vcc);
-void VCL_Poll(void);
-const char *VCL_Return_Name(unsigned method);
-
-#define VCL_MET_MAC(l,u,b) void VCL_##l##_method(struct sess *);
-#include "tbl/vcl_returns.h"
-#undef VCL_MET_MAC
-
-/* cache_vrt.c */
-
-char *VRT_String(struct ws *ws, const char *h, const char *p, va_list ap);
-char *VRT_StringList(char *d, unsigned dl, const char *p, va_list ap);
-
-void ESI_Deliver(struct sess *);
-void ESI_DeliverChild(const struct sess *);
-
-/* cache_vrt_vmod.c */
-void VMOD_Init(void);
-
-/* cache_wrk.c */
-
-void WRK_Init(void);
-int WRK_TrySumStat(struct worker *w);
-void WRK_SumStat(struct worker *w);
-void *WRK_thread(void *priv);
-typedef void *bgthread_t(struct sess *, void *priv);
-void WRK_BgThread(pthread_t *thr, const char *name, bgthread_t *func,
-    void *priv);
-
-/* cache_ws.c */
-
-void WS_Init(struct ws *ws, const char *id, void *space, unsigned len);
-unsigned WS_Reserve(struct ws *ws, unsigned bytes);
-void WS_Release(struct ws *ws, unsigned bytes);
-void WS_ReleaseP(struct ws *ws, char *ptr);
-void WS_Assert(const struct ws *ws);
-void WS_Reset(struct ws *ws, char *p);
-char *WS_Alloc(struct ws *ws, unsigned bytes);
-char *WS_Dup(struct ws *ws, const char *);
-char *WS_Snapshot(struct ws *ws);
-unsigned WS_Free(const struct ws *ws);
-
-/* rfc2616.c */
-void RFC2616_Ttl(const struct sess *sp);
-enum body_status RFC2616_Body(const struct sess *sp);
-unsigned RFC2616_Req_Gzip(const struct sess *sp);
-int RFC2616_Do_Cond(const struct sess *sp);
-
-/* stevedore.c */
-struct object *STV_NewObject(struct sess *sp, const char *hint, unsigned len,
-    struct exp *, uint16_t nhttp);
-struct storage *STV_alloc(struct worker *w, size_t size);
-void STV_trim(struct storage *st, size_t size);
-void STV_free(struct storage *st);
-void STV_open(void);
-void STV_close(void);
-void STV_Freestore(struct object *o);
-
-/* storage_synth.c */
-struct vsb *SMS_Makesynth(struct object *obj);
-void SMS_Finish(struct object *obj);
-void SMS_Init(void);
-
-/* storage_persistent.c */
-void SMP_Init(void);
-void SMP_Ready(void);
-void SMP_NewBan(const uint8_t *ban, unsigned len);
-
-/*
- * A normal pointer difference is signed, but we never want a negative value
- * so this little tool will make sure we don't get that.
- */
-
-static inline unsigned
-pdiff(const void *b, const void *e)
-{
-
-	assert(b <= e);
-	return
-	    ((unsigned)((const unsigned char *)e - (const unsigned char *)b));
-}
-
-static inline void
-Tcheck(const txt t)
-{
-
-	AN(t.b);
-	AN(t.e);
-	assert(t.b <= t.e);
-}
-
-/*
- * unsigned length of a txt
- */
-
-static inline unsigned
-Tlen(const txt t)
-{
-
-	Tcheck(t);
-	return ((unsigned)(t.e - t.b));
-}
-
-static inline void
-Tadd(txt *t, const char *p, int l)
-{
-	Tcheck(*t);
-
-	if (l <= 0) {
-	} if (t->b + l < t->e) {
-		memcpy(t->b, p, l);
-		t->b += l;
-	} else {
-		t->b = t->e;
-	}
-}
-
-static inline void
-AssertObjBusy(const struct object *o)
-{
-	AN(o->objcore);
-	AN (o->objcore->flags & OC_F_BUSY);
-}
-
-static inline void
-AssertObjCorePassOrBusy(const struct objcore *oc)
-{
-	if (oc != NULL)
-		AN (oc->flags & OC_F_BUSY);
-}
diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
new file mode 100644
index 0000000..653fe77
--- /dev/null
+++ b/bin/varnishd/cache/cache.h
@@ -0,0 +1,1039 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ */
+
+/*
+ * This macro can be used in .h files to isolate bits that the manager
+ * should not (need to) see, such as pthread mutexes etc.
+ */
+#define VARNISH_CACHE_CHILD	1
+
+#include "common/common.h"
+
+#include "vapi/vsc_int.h"
+#include "vapi/vsl_int.h"
+
+#include <sys/socket.h>
+
+#include <pthread.h>
+#ifdef HAVE_PTHREAD_NP_H
+#include <pthread_np.h>
+#endif
+#include <stdarg.h>
+#include <string.h>
+#include <limits.h>
+#include <unistd.h>
+
+#if defined(HAVE_EPOLL_CTL)
+#include <sys/epoll.h>
+#endif
+
+
+#include "common/params.h"
+
+enum body_status {
+#define BODYSTATUS(U,l)	BS_##U,
+#include "tbl/body_status.h"
+#undef BODYSTATUS
+};
+
+static inline const char *
+body_status(enum body_status e)
+{
+	switch(e) {
+#define BODYSTATUS(U,l)	case BS_##U: return (#l);
+#include "tbl/body_status.h"
+#undef BODYSTATUS
+	default:
+		return ("?");
+	}
+}
+
+/*
+ * NB: HDR_STATUS is only used in cache_http.c, everybody else uses the
+ * http->status integer field.
+ */
+
+enum {
+	/* Fields from the first line of HTTP proto */
+	HTTP_HDR_REQ,
+	HTTP_HDR_URL,
+	HTTP_HDR_PROTO,
+	HTTP_HDR_STATUS,
+	HTTP_HDR_RESPONSE,
+	/* HTTP header lines */
+	HTTP_HDR_FIRST,
+};
+
+struct SHA256Context;
+struct VSC_C_lck;
+struct ban;
+struct busyobj;
+struct cli;
+struct cli_proto;
+struct director;
+struct iovec;
+struct objcore;
+struct object;
+struct objhead;
+struct pool;
+struct sess;
+struct sesspool;
+struct vbc;
+struct vef_priv;
+struct vrt_backend;
+struct vsb;
+struct waitinglist;
+struct worker;
+
+#define DIGEST_LEN		32
+
+/* Name of transient storage */
+#define TRANSIENT_STORAGE	"Transient"
+
+/*--------------------------------------------------------------------
+ * Pointer aligment magic
+ */
+
+#define PALGN		(sizeof(void *) - 1)
+#define PAOK(p)		(((uintptr_t)(p) & PALGN) == 0)
+#define PRNDDN(p)	((uintptr_t)(p) & ~PALGN)
+#define PRNDUP(p)	(((uintptr_t)(p) + PALGN) & ~PALGN)
+
+/*--------------------------------------------------------------------*/
+
+typedef struct {
+	char			*b;
+	char			*e;
+} txt;
+
+/*--------------------------------------------------------------------*/
+
+enum step {
+#define STEP(l, u)	STP_##u,
+#include "tbl/steps.h"
+#undef STEP
+};
+
+/*--------------------------------------------------------------------
+ * Workspace structure for quick memory allocation.
+ */
+
+struct ws {
+	unsigned		magic;
+#define WS_MAGIC		0x35fac554
+	unsigned		overflow;	/* workspace overflowed */
+	const char		*id;		/* identity */
+	char			*s;		/* (S)tart of buffer */
+	char			*f;		/* (F)ree pointer */
+	char			*r;		/* (R)eserved length */
+	char			*e;		/* (E)nd of buffer */
+};
+
+/*--------------------------------------------------------------------
+ * HTTP Request/Response/Header handling structure.
+ */
+
+enum httpwhence {
+	HTTP_Rx	 = 1,
+	HTTP_Tx  = 2,
+	HTTP_Obj = 3
+};
+
+/* NB: remember to update http_Copy() if you add fields */
+struct http {
+	unsigned		magic;
+#define HTTP_MAGIC		0x6428b5c9
+
+	enum httpwhence		logtag;
+
+	struct ws		*ws;
+	txt			*hd;
+	unsigned char		*hdf;
+#define HDF_FILTER		(1 << 0)	/* Filtered by Connection */
+	uint16_t		shd;		/* Size of hd space */
+	uint16_t		nhd;		/* Next free hd */
+	uint16_t		status;
+	uint8_t			protover;
+	uint8_t			conds;		/* If-* headers present */
+};
+
+/*--------------------------------------------------------------------
+ * HTTP Protocol connection structure
+ */
+
+struct http_conn {
+	unsigned		magic;
+#define HTTP_CONN_MAGIC		0x3e19edd1
+
+	int			fd;
+	unsigned		vsl_id;
+	unsigned		maxbytes;
+	unsigned		maxhdr;
+	struct ws		*ws;
+	txt			rxbuf;
+	txt			pipeline;
+};
+
+/*--------------------------------------------------------------------*/
+
+struct acct {
+	double			first;
+#define ACCT(foo)	uint64_t	foo;
+#include "tbl/acct_fields.h"
+#undef ACCT
+};
+
+/*--------------------------------------------------------------------*/
+
+#define L0(t, n)
+#define L1(t, n)		t n;
+#define VSC_F(n, t, l, f, e,d)	L##l(t, n)
+#define VSC_DO_MAIN
+struct dstat {
+#include "tbl/vsc_fields.h"
+};
+#undef VSC_F
+#undef VSC_DO_MAIN
+#undef L0
+#undef L1
+
+/* Fetch processors --------------------------------------------------*/
+
+typedef void vfp_begin_f(struct worker *, size_t );
+typedef int vfp_bytes_f(struct worker *, struct http_conn *, ssize_t);
+typedef int vfp_end_f(struct worker *);
+
+struct vfp {
+	vfp_begin_f	*begin;
+	vfp_bytes_f	*bytes;
+	vfp_end_f	*end;
+};
+
+extern struct vfp vfp_gunzip;
+extern struct vfp vfp_gzip;
+extern struct vfp vfp_testgzip;
+extern struct vfp vfp_esi;
+
+/*--------------------------------------------------------------------*/
+
+struct exp {
+	double			ttl;
+	double			grace;
+	double			keep;
+	double			age;
+	double			entered;
+};
+
+/*--------------------------------------------------------------------*/
+
+struct wrw {
+	int			*wfd;
+	unsigned		werr;	/* valid after WRW_Flush() */
+	struct iovec		*iov;
+	unsigned		siov;
+	unsigned		niov;
+	ssize_t			liov;
+	ssize_t			cliov;
+	unsigned		ciov;	/* Chunked header marker */
+};
+
+/*--------------------------------------------------------------------*/
+
+struct stream_ctx {
+	unsigned		magic;
+#define STREAM_CTX_MAGIC	0x8213728b
+
+	struct vgz		*vgz;
+	void			*obuf;
+	ssize_t			obuf_len;
+	ssize_t			obuf_ptr;
+
+	/* Next byte we will take from storage */
+	ssize_t			stream_next;
+
+	/* First byte of storage if we free it as we go (pass) */
+	ssize_t			stream_front;
+};
+
+/*--------------------------------------------------------------------*/
+
+struct wrk_accept {
+	unsigned		magic;
+#define WRK_ACCEPT_MAGIC	0x8c4b4d59
+
+	/* Accept stuff */
+	struct sockaddr_storage	acceptaddr;
+	socklen_t		acceptaddrlen;
+	int			acceptsock;
+	struct listen_sock	*acceptlsock;
+};
+
+/*--------------------------------------------------------------------*/
+
+struct worker {
+	unsigned		magic;
+#define WORKER_MAGIC		0x6391adcf
+	struct pool		*pool;
+	struct objhead		*nobjhead;
+	struct objcore		*nobjcore;
+	struct waitinglist	*nwaitinglist;
+	struct busyobj		*nbusyobj;
+	void			*nhashpriv;
+	struct dstat		stats;
+
+	/* Pool stuff */
+	double			lastused;
+
+	struct wrw		wrw;
+
+	pthread_cond_t		cond;
+
+	VTAILQ_ENTRY(worker)	list;
+	struct sess		*sp;
+
+	struct VCL_conf		*vcl;
+
+	uint32_t		*wlb, *wlp, *wle;
+	unsigned		wlr;
+
+	/* Lookup stuff */
+	struct SHA256Context	*sha256ctx;
+
+	struct http_conn	htc[1];
+	struct ws		ws[1];
+	struct http		*bereq;
+	struct http		*beresp;
+	struct http		*resp;
+
+	struct exp		exp;
+
+	/* This is only here so VRT can find it */
+	const char		*storage_hint;
+
+	/* Fetch stuff */
+	struct vbc		*vbc;
+	struct object		*fetch_obj;
+	enum body_status	body_status;
+	struct vfp		*vfp;
+	struct vgz		*vgz_rx;
+	struct vef_priv		*vef_priv;
+	unsigned		fetch_failed;
+	unsigned		do_stream;
+	unsigned		do_esi;
+	unsigned		do_gzip;
+	unsigned		is_gzip;
+	unsigned		do_gunzip;
+	unsigned		is_gunzip;
+	unsigned		do_close;
+	char			*h_content_length;
+
+	/* Stream state */
+	struct stream_ctx	*sctx;
+
+	/* ESI stuff */
+	struct vep_state	*vep;
+	int			gzip_resp;
+	ssize_t			l_crc;
+	uint32_t		crc;
+
+	/* Timeouts */
+	double			connect_timeout;
+	double			first_byte_timeout;
+	double			between_bytes_timeout;
+
+	/* Delivery mode */
+	unsigned		res_mode;
+#define RES_LEN			(1<<1)
+#define RES_EOF			(1<<2)
+#define RES_CHUNKED		(1<<3)
+#define RES_ESI			(1<<4)
+#define RES_ESI_CHILD		(1<<5)
+#define RES_GUNZIP		(1<<6)
+
+	/* Temporary accounting */
+	struct acct		acct_tmp;
+};
+
+/* LRU ---------------------------------------------------------------*/
+
+struct lru {
+	unsigned		magic;
+#define LRU_MAGIC		0x3fec7bb0
+	VTAILQ_HEAD(,objcore)	lru_head;
+	struct lock		mtx;
+};
+
+/* Storage -----------------------------------------------------------*/
+
+struct storage {
+	unsigned		magic;
+#define STORAGE_MAGIC		0x1a4e51c0
+
+#ifdef SENDFILE_WORKS
+	int			fd;
+	off_t			where;
+#endif
+
+	VTAILQ_ENTRY(storage)	list;
+	struct stevedore	*stevedore;
+	void			*priv;
+
+	unsigned char		*ptr;
+	unsigned		len;
+	unsigned		space;
+};
+
+/* Object core structure ---------------------------------------------
+ * Objects have sideways references in the binary heap and the LRU list
+ * and we want to avoid paging in a lot of objects just to move them up
+ * or down the binheap or to move a unrelated object on the LRU list.
+ * To avoid this we use a proxy object, objcore, to hold the relevant
+ * housekeeping fields parts of an object.
+ */
+
+typedef struct object *getobj_f(struct worker *wrk, struct objcore *oc);
+typedef void updatemeta_f(struct objcore *oc);
+typedef void freeobj_f(struct objcore *oc);
+typedef struct lru *getlru_f(const struct objcore *oc);
+
+struct objcore_methods {
+	getobj_f	*getobj;
+	updatemeta_f	*updatemeta;
+	freeobj_f	*freeobj;
+	getlru_f	*getlru;
+};
+
+struct objcore {
+	unsigned		magic;
+#define OBJCORE_MAGIC		0x4d301302
+	unsigned		refcnt;
+	struct objcore_methods	*methods;
+	void			*priv;
+	unsigned		priv2;
+	struct objhead		*objhead;
+	struct busyobj		*busyobj;
+	double			timer_when;
+	unsigned		flags;
+#define OC_F_BUSY		(1<<1)
+#define OC_F_PASS		(1<<2)
+#define OC_F_LRUDONTMOVE	(1<<4)
+#define OC_F_PRIV		(1<<5)		/* Stevedore private flag */
+#define OC_F_LURK		(3<<6)		/* Ban-lurker-color */
+	unsigned		timer_idx;
+	VTAILQ_ENTRY(objcore)	list;
+	VTAILQ_ENTRY(objcore)	lru_list;
+	VTAILQ_ENTRY(objcore)	ban_list;
+	struct ban		*ban;
+};
+
+static inline struct object *
+oc_getobj(struct worker *wrk, struct objcore *oc)
+{
+
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	AZ(oc->flags & OC_F_BUSY);
+	AN(oc->methods);
+	AN(oc->methods->getobj);
+	return (oc->methods->getobj(wrk, oc));
+}
+
+static inline void
+oc_updatemeta(struct objcore *oc)
+{
+
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	AN(oc->methods);
+	if (oc->methods->updatemeta != NULL)
+		oc->methods->updatemeta(oc);
+}
+
+static inline void
+oc_freeobj(struct objcore *oc)
+{
+
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	AN(oc->methods);
+	AN(oc->methods->freeobj);
+	oc->methods->freeobj(oc);
+}
+
+static inline struct lru *
+oc_getlru(const struct objcore *oc)
+{
+
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	AN(oc->methods);
+	AN(oc->methods->getlru);
+	return (oc->methods->getlru(oc));
+}
+
+/* Busy Object structure ---------------------------------------------*/
+
+struct busyobj {
+	unsigned		magic;
+#define BUSYOBJ_MAGIC		0x23b95567
+	uint8_t			*vary;
+};
+
+/* Object structure --------------------------------------------------*/
+
+VTAILQ_HEAD(storagehead, storage);
+
+struct object {
+	unsigned		magic;
+#define OBJECT_MAGIC		0x32851d42
+	unsigned		xid;
+	struct storage		*objstore;
+	struct objcore		*objcore;
+
+	struct ws		ws_o[1];
+
+	uint8_t			*vary;
+	unsigned		hits;
+	uint16_t		response;
+
+	/* XXX: make bitmap */
+	uint8_t			gziped;
+	/* Bit positions in the gzip stream */
+	ssize_t			gzip_start;
+	ssize_t			gzip_last;
+	ssize_t			gzip_stop;
+
+	ssize_t			len;
+
+	struct exp		exp;
+
+	double			last_modified;
+	double			last_lru;
+
+	struct http		*http;
+
+	struct storagehead	store;
+
+	struct storage		*esidata;
+
+	double			last_use;
+
+};
+
+/* -------------------------------------------------------------------*/
+
+struct sess {
+	unsigned		magic;
+#define SESS_MAGIC		0x2c2f9c5a
+	int			fd;
+	unsigned		vsl_id;
+	unsigned		xid;
+
+	int			restarts;
+	int			esi_level;
+	int			disable_esi;
+
+	uint8_t			hash_ignore_busy;
+	uint8_t			hash_always_miss;
+
+	struct worker		*wrk;
+
+	socklen_t		sockaddrlen;
+	socklen_t		mysockaddrlen;
+	struct sockaddr_storage	sockaddr;
+	struct sockaddr_storage	mysockaddr;
+	struct listen_sock	*mylsock;
+
+	/* formatted ascii client address */
+	char			*addr;
+	char			*port;
+	char			*client_identity;
+
+	/* HTTP request */
+	const char		*doclose;
+	struct http		*http;
+	struct http		*http0;
+
+	struct ws		ws[1];
+	char			*ws_ses;	/* WS above session data */
+	char			*ws_req;	/* WS above request data */
+
+	unsigned char		digest[DIGEST_LEN];
+
+	/* Built Vary string */
+	uint8_t			*vary_b;
+	uint8_t			*vary_l;
+	uint8_t			*vary_e;
+
+	struct http_conn	htc[1];
+
+	/* Timestamps, all on TIM_real() timescale */
+	double			t_open;
+	double			t_req;
+	double			t_resp;
+	double			t_end;
+
+	/* Acceptable grace period */
+	struct exp		exp;
+
+	enum step		step;
+	unsigned		cur_method;
+	unsigned		handling;
+	unsigned char		sendbody;
+	unsigned char		wantbody;
+	uint16_t		err_code;
+	const char		*err_reason;
+
+	VTAILQ_ENTRY(sess)	list;
+
+	struct director		*director;
+	struct object		*obj;
+	struct objcore		*objcore;
+	struct VCL_conf		*vcl;
+
+	/* The busy objhead we sleep on */
+	struct objhead		*hash_objhead;
+
+	/* Various internal stuff */
+	struct sessmem		*mem;
+
+	VTAILQ_ENTRY(sess)	poollist;
+	uint64_t		req_bodybytes;
+	struct acct		acct_ses;
+
+#if defined(HAVE_EPOLL_CTL)
+	struct epoll_event ev;
+#endif
+};
+
+/* Prototypes etc ----------------------------------------------------*/
+
+/* cache_acceptor.c */
+void VCA_Prep(struct sess *sp);
+void VCA_Init(void);
+void VCA_Shutdown(void);
+int VCA_Accept(struct listen_sock *ls, struct wrk_accept *wa);
+void VCA_SetupSess(struct worker *w);
+void VCA_FailSess(struct worker *w);
+
+/* cache_backend.c */
+void VBE_UseHealth(const struct director *vdi);
+
+struct vbc *VDI_GetFd(const struct director *, struct sess *sp);
+int VDI_Healthy(const struct director *, const struct sess *sp);
+void VDI_CloseFd(struct worker *wrk);
+void VDI_RecycleFd(struct worker *wrk);
+void VDI_AddHostHeader(const struct sess *sp);
+void VBE_Poll(void);
+
+/* cache_backend_cfg.c */
+void VBE_Init(void);
+struct backend *VBE_AddBackend(struct cli *cli, const struct vrt_backend *vb);
+
+/* cache_backend_poll.c */
+void VBP_Init(void);
+
+/* cache_ban.c */
+struct ban *BAN_New(void);
+int BAN_AddTest(struct cli *, struct ban *, const char *, const char *,
+    const char *);
+void BAN_Free(struct ban *b);
+void BAN_Insert(struct ban *b);
+void BAN_Init(void);
+void BAN_NewObjCore(struct objcore *oc);
+void BAN_DestroyObj(struct objcore *oc);
+int BAN_CheckObject(struct object *o, const struct sess *sp);
+void BAN_Reload(const uint8_t *ban, unsigned len);
+struct ban *BAN_TailRef(void);
+void BAN_Compile(void);
+struct ban *BAN_RefBan(struct objcore *oc, double t0, const struct ban *tail);
+void BAN_TailDeref(struct ban **ban);
+double BAN_Time(const struct ban *ban);
+
+/* cache_center.c [CNT] */
+void CNT_Session(struct sess *sp);
+void CNT_Init(void);
+
+/* cache_cli.c [CLI] */
+void CLI_Init(void);
+void CLI_Run(void);
+void CLI_AddFuncs(struct cli_proto *p);
+extern pthread_t cli_thread;
+#define ASSERT_CLI() do {assert(pthread_self() == cli_thread);} while (0)
+
+/* cache_expiry.c */
+void EXP_Clr(struct exp *e);
+double EXP_Get_ttl(const struct exp *e);
+double EXP_Get_grace(const struct exp *e);
+double EXP_Get_keep(const struct exp *e);
+void EXP_Set_ttl(struct exp *e, double v);
+void EXP_Set_grace(struct exp *e, double v);
+void EXP_Set_keep(struct exp *e, double v);
+
+double EXP_Ttl(const struct sess *, const struct object*);
+double EXP_Grace(const struct sess *, const struct object*);
+void EXP_Insert(struct object *o);
+void EXP_Inject(struct objcore *oc, struct lru *lru, double when);
+void EXP_Init(void);
+void EXP_Rearm(const struct object *o);
+int EXP_Touch(struct objcore *oc);
+int EXP_NukeOne(struct worker *w, struct lru *lru);
+
+/* cache_fetch.c */
+struct storage *FetchStorage(struct worker *w, ssize_t sz);
+int FetchError(struct worker *w, const char *error);
+int FetchError2(struct worker *w, const char *error, const char *more);
+int FetchHdr(struct sess *sp);
+int FetchBody(struct worker *w, struct object *obj);
+int FetchReqBody(struct sess *sp);
+void Fetch_Init(void);
+
+/* cache_gzip.c */
+struct vgz;
+
+enum vgz_flag { VGZ_NORMAL, VGZ_ALIGN, VGZ_RESET, VGZ_FINISH };
+struct vgz *VGZ_NewUngzip(struct worker *wrk, const char *id);
+struct vgz *VGZ_NewGzip(struct worker *wrk, const char *id);
+void VGZ_Ibuf(struct vgz *, const void *, ssize_t len);
+int VGZ_IbufEmpty(const struct vgz *vg);
+void VGZ_Obuf(struct vgz *, void *, ssize_t len);
+int VGZ_ObufFull(const struct vgz *vg);
+int VGZ_ObufStorage(struct worker *w, struct vgz *vg);
+int VGZ_Gzip(struct vgz *, const void **, size_t *len, enum vgz_flag);
+int VGZ_Gunzip(struct vgz *, const void **, size_t *len);
+int VGZ_Destroy(struct vgz **, int vsl_id);
+void VGZ_UpdateObj(const struct vgz*, struct object *);
+int VGZ_WrwGunzip(struct worker *w, struct vgz *, const void *ibuf,
+    ssize_t ibufl, char *obuf, ssize_t obufl, ssize_t *obufp);
+
+/* Return values */
+#define VGZ_ERROR	-1
+#define VGZ_OK		0
+#define VGZ_END		1
+#define VGZ_STUCK	2
+
+/* cache_http.c */
+unsigned HTTP_estimate(unsigned nhttp);
+void HTTP_Copy(struct http *to, const struct http * const fm);
+struct http *HTTP_create(void *p, uint16_t nhttp);
+const char *http_StatusMessage(unsigned);
+unsigned http_EstimateWS(const struct http *fm, unsigned how, uint16_t *nhd);
+void HTTP_Init(void);
+void http_ClrHeader(struct http *to);
+unsigned http_Write(struct worker *w, unsigned vsl_id, const struct http *hp,
+    int resp);
+void http_CopyResp(struct http *to, const struct http *fm);
+void http_SetResp(struct http *to, const char *proto, uint16_t status,
+    const char *response);
+void http_FilterFields(struct worker *w, unsigned vsl_id, struct http *to,
+    const struct http *fm, unsigned how);
+void http_FilterHeader(const struct sess *sp, unsigned how);
+void http_PutProtocol(struct worker *w, unsigned vsl_id, const struct http *to,
+    const char *protocol);
+void http_PutStatus(struct http *to, uint16_t status);
+void http_PutResponse(struct worker *w, unsigned vsl_id, const struct http *to,
+    const char *response);
+void http_PrintfHeader(struct worker *w, unsigned vsl_id, struct http *to,
+    const char *fmt, ...);
+void http_SetHeader(struct worker *w, unsigned vsl_id, struct http *to,
+    const char *hdr);
+void http_SetH(const struct http *to, unsigned n, const char *fm);
+void http_ForceGet(const struct http *to);
+void http_Setup(struct http *ht, struct ws *ws);
+int http_GetHdr(const struct http *hp, const char *hdr, char **ptr);
+int http_GetHdrData(const struct http *hp, const char *hdr,
+    const char *field, char **ptr);
+int http_GetHdrField(const struct http *hp, const char *hdr,
+    const char *field, char **ptr);
+double http_GetHdrQ(const struct http *hp, const char *hdr, const char *field);
+uint16_t http_GetStatus(const struct http *hp);
+const char *http_GetReq(const struct http *hp);
+int http_HdrIs(const struct http *hp, const char *hdr, const char *val);
+uint16_t http_DissectRequest(struct sess *sp);
+uint16_t http_DissectResponse(struct worker *w, const struct http_conn *htc,
+    struct http *sp);
+const char *http_DoConnection(const struct http *hp);
+void http_CopyHome(struct worker *w, unsigned vsl_id, const struct http *hp);
+void http_Unset(struct http *hp, const char *hdr);
+void http_CollectHdr(struct http *hp, const char *hdr);
+
+/* cache_httpconn.c */
+void HTC_Init(struct http_conn *htc, struct ws *ws, int fd, unsigned vsl_id,
+    unsigned maxbytes, unsigned maxhdr);
+int HTC_Reinit(struct http_conn *htc);
+int HTC_Rx(struct http_conn *htc);
+ssize_t HTC_Read(struct worker *w, struct http_conn *htc, void *d, size_t len);
+int HTC_Complete(struct http_conn *htc);
+
+#define HTTPH(a, b, c, d, e, f, g) extern char b[];
+#include "tbl/http_headers.h"
+#undef HTTPH
+
+/* cache_main.c */
+void THR_SetName(const char *name);
+const char* THR_GetName(void);
+void THR_SetSession(const struct sess *sp);
+const struct sess * THR_GetSession(void);
+
+/* cache_lck.c */
+
+/* Internal functions, call only through macros below */
+void Lck__Lock(struct lock *lck, const char *p, const char *f, int l);
+void Lck__Unlock(struct lock *lck, const char *p, const char *f, int l);
+int Lck__Trylock(struct lock *lck, const char *p, const char *f, int l);
+void Lck__New(struct lock *lck, struct VSC_C_lck *, const char *);
+void Lck__Assert(const struct lock *lck, int held);
+
+/* public interface: */
+void LCK_Init(void);
+void Lck_Delete(struct lock *lck);
+int Lck_CondWait(pthread_cond_t *cond, struct lock *lck, struct timespec *ts);
+
+#define Lck_New(a, b) Lck__New(a, b, #b)
+#define Lck_Lock(a) Lck__Lock(a, __func__, __FILE__, __LINE__)
+#define Lck_Unlock(a) Lck__Unlock(a, __func__, __FILE__, __LINE__)
+#define Lck_Trylock(a) Lck__Trylock(a, __func__, __FILE__, __LINE__)
+#define Lck_AssertHeld(a) Lck__Assert(a, 1)
+
+#define LOCK(nam) extern struct VSC_C_lck *lck_##nam;
+#include "tbl/locks.h"
+#undef LOCK
+
+/* cache_panic.c */
+void PAN_Init(void);
+
+/* cache_pipe.c */
+void PipeSession(struct sess *sp);
+
+/* cache_pool.c */
+void Pool_Init(void);
+void Pool_Work_Thread(void *priv, struct worker *w);
+void Pool_Wait(struct sess *sp);
+int Pool_Schedule(struct pool *pp, struct sess *sp);
+
+#define WRW_IsReleased(w)	((w)->wrw.wfd == NULL)
+int WRW_Error(const struct worker *w);
+void WRW_Chunked(struct worker *w);
+void WRW_EndChunk(struct worker *w);
+void WRW_Reserve(struct worker *w, int *fd);
+unsigned WRW_Flush(struct worker *w);
+unsigned WRW_FlushRelease(struct worker *w);
+unsigned WRW_Write(struct worker *w, const void *ptr, int len);
+unsigned WRW_WriteH(struct worker *w, const txt *hh, const char *suf);
+#ifdef SENDFILE_WORKS
+void WRW_Sendfile(struct worker *w, int fd, off_t off, unsigned len);
+#endif  /* SENDFILE_WORKS */
+
+/* cache_session.c [SES] */
+struct sess *SES_New(struct worker *wrk, struct sesspool *pp);
+struct sess *SES_Alloc(void);
+void SES_Close(struct sess *sp, const char *reason);
+void SES_Delete(struct sess *sp, const char *reason);
+void SES_Charge(struct sess *sp);
+struct sesspool *SES_NewPool(struct pool *pp);
+void SES_DeletePool(struct sesspool *sp, struct worker *wrk);
+int SES_Schedule(struct sess *sp);
+
+
+/* cache_shmlog.c */
+void VSL_Init(void);
+void *VSM_Alloc(unsigned size, const char *class, const char *type,
+    const char *ident);
+void VSM_Free(const void *ptr);
+#ifdef VSL_ENDMARKER
+void VSL(enum VSL_tag_e tag, int id, const char *fmt, ...);
+void WSLR(struct worker *w, enum VSL_tag_e tag, int id, txt t);
+void WSL(struct worker *w, enum VSL_tag_e tag, int id, const char *fmt, ...);
+void WSLB(struct worker *w, enum VSL_tag_e tag, const char *fmt, ...);
+
+void WSL_Flush(struct worker *w, int overflow);
+
+#define DSL(flag, tag, id, ...)					\
+	do {							\
+		if (cache_param->diag_bitmap & (flag))		\
+			VSL((tag), (id), __VA_ARGS__);		\
+	} while (0)
+
+#define WSP(sess, tag, ...)					\
+	WSL((sess)->wrk, tag, (sess)->vsl_id, __VA_ARGS__)
+
+#define WSPR(sess, tag, txt)					\
+	WSLR((sess)->wrk, tag, (sess)->vsl_id, txt)
+
+#define INCOMPL() do {							\
+	VSL(SLT_Debug, 0, "INCOMPLETE AT: %s(%d)", __func__, __LINE__); \
+	fprintf(stderr,							\
+	    "INCOMPLETE AT: %s(%d)\n",					\
+	    (const char *)__func__, __LINE__);				\
+	abort();							\
+	} while (0)
+#endif
+
+/* cache_response.c */
+void RES_BuildHttp(const struct sess *sp);
+void RES_WriteObj(struct sess *sp);
+void RES_StreamStart(struct sess *sp);
+void RES_StreamEnd(struct sess *sp);
+void RES_StreamPoll(struct worker *);
+
+/* cache_vary.c */
+struct vsb *VRY_Create(const struct sess *sp, const struct http *hp);
+int VRY_Match(struct sess *sp, const uint8_t *vary);
+void VRY_Validate(const uint8_t *vary);
+
+/* cache_vcl.c */
+void VCL_Init(void);
+void VCL_Refresh(struct VCL_conf **vcc);
+void VCL_Rel(struct VCL_conf **vcc);
+void VCL_Poll(void);
+const char *VCL_Return_Name(unsigned method);
+
+#define VCL_MET_MAC(l,u,b) void VCL_##l##_method(struct sess *);
+#include "tbl/vcl_returns.h"
+#undef VCL_MET_MAC
+
+/* cache_vrt.c */
+
+char *VRT_String(struct ws *ws, const char *h, const char *p, va_list ap);
+char *VRT_StringList(char *d, unsigned dl, const char *p, va_list ap);
+
+void ESI_Deliver(struct sess *);
+void ESI_DeliverChild(const struct sess *);
+
+/* cache_vrt_vmod.c */
+void VMOD_Init(void);
+
+/* cache_wrk.c */
+
+void WRK_Init(void);
+int WRK_TrySumStat(struct worker *w);
+void WRK_SumStat(struct worker *w);
+void *WRK_thread(void *priv);
+typedef void *bgthread_t(struct sess *, void *priv);
+void WRK_BgThread(pthread_t *thr, const char *name, bgthread_t *func,
+    void *priv);
+
+/* cache_ws.c */
+
+void WS_Init(struct ws *ws, const char *id, void *space, unsigned len);
+unsigned WS_Reserve(struct ws *ws, unsigned bytes);
+void WS_Release(struct ws *ws, unsigned bytes);
+void WS_ReleaseP(struct ws *ws, char *ptr);
+void WS_Assert(const struct ws *ws);
+void WS_Reset(struct ws *ws, char *p);
+char *WS_Alloc(struct ws *ws, unsigned bytes);
+char *WS_Dup(struct ws *ws, const char *);
+char *WS_Snapshot(struct ws *ws);
+unsigned WS_Free(const struct ws *ws);
+
+/* rfc2616.c */
+void RFC2616_Ttl(const struct sess *sp);
+enum body_status RFC2616_Body(const struct sess *sp);
+unsigned RFC2616_Req_Gzip(const struct sess *sp);
+int RFC2616_Do_Cond(const struct sess *sp);
+
+/* stevedore.c */
+struct object *STV_NewObject(struct sess *sp, const char *hint, unsigned len,
+    struct exp *, uint16_t nhttp);
+struct storage *STV_alloc(struct worker *w, size_t size);
+void STV_trim(struct storage *st, size_t size);
+void STV_free(struct storage *st);
+void STV_open(void);
+void STV_close(void);
+void STV_Freestore(struct object *o);
+
+/* storage_synth.c */
+struct vsb *SMS_Makesynth(struct object *obj);
+void SMS_Finish(struct object *obj);
+void SMS_Init(void);
+
+/* storage_persistent.c */
+void SMP_Init(void);
+void SMP_Ready(void);
+void SMP_NewBan(const uint8_t *ban, unsigned len);
+
+/*
+ * A normal pointer difference is signed, but we never want a negative value
+ * so this little tool will make sure we don't get that.
+ */
+
+static inline unsigned
+pdiff(const void *b, const void *e)
+{
+
+	assert(b <= e);
+	return
+	    ((unsigned)((const unsigned char *)e - (const unsigned char *)b));
+}
+
+static inline void
+Tcheck(const txt t)
+{
+
+	AN(t.b);
+	AN(t.e);
+	assert(t.b <= t.e);
+}
+
+/*
+ * unsigned length of a txt
+ */
+
+static inline unsigned
+Tlen(const txt t)
+{
+
+	Tcheck(t);
+	return ((unsigned)(t.e - t.b));
+}
+
+static inline void
+Tadd(txt *t, const char *p, int l)
+{
+	Tcheck(*t);
+
+	if (l <= 0) {
+	} if (t->b + l < t->e) {
+		memcpy(t->b, p, l);
+		t->b += l;
+	} else {
+		t->b = t->e;
+	}
+}
+
+static inline void
+AssertObjBusy(const struct object *o)
+{
+	AN(o->objcore);
+	AN (o->objcore->flags & OC_F_BUSY);
+}
+
+static inline void
+AssertObjCorePassOrBusy(const struct objcore *oc)
+{
+	if (oc != NULL)
+		AN (oc->flags & OC_F_BUSY);
+}
diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c
new file mode 100644
index 0000000..8c83121
--- /dev/null
+++ b/bin/varnishd/cache/cache_acceptor.c
@@ -0,0 +1,430 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ */
+
+#include "config.h"
+
+#include "cache.h"
+#include "common/heritage.h"
+
+#include "vcli.h"
+#include "vcli_priv.h"
+#include "vtcp.h"
+#include "vtim.h"
+
+static pthread_t	VCA_thread;
+static struct timeval	tv_sndtimeo;
+static struct timeval	tv_rcvtimeo;
+
+/*--------------------------------------------------------------------
+ * We want to get out of any kind of trouble-hit TCP connections as fast
+ * as absolutely possible, so we set them LINGER enabled with zero timeout,
+ * so that even if there are outstanding write data on the socket, a close(2)
+ * will return immediately.
+ */
+static const struct linger linger = {
+	.l_onoff	=	0,
+};
+
+static unsigned char	need_sndtimeo, need_rcvtimeo, need_linger, need_test;
+
+static void
+sock_test(int fd)
+{
+	struct linger lin;
+	struct timeval tv;
+	socklen_t l;
+	int i;
+
+	l = sizeof lin;
+	i = getsockopt(fd, SOL_SOCKET, SO_LINGER, &lin, &l);
+	if (i) {
+		VTCP_Assert(i);
+		return;
+	}
+	assert(l == sizeof lin);
+	if (memcmp(&lin, &linger, l))
+		need_linger = 1;
+
+#ifdef SO_SNDTIMEO_WORKS
+	l = sizeof tv;
+	i = getsockopt(fd, SOL_SOCKET, SO_SNDTIMEO, &tv, &l);
+	if (i) {
+		VTCP_Assert(i);
+		return;
+	}
+	assert(l == sizeof tv);
+	if (memcmp(&tv, &tv_sndtimeo, l))
+		need_sndtimeo = 1;
+#else
+	(void)tv;
+	(void)tv_sndtimeo;
+	(void)need_sndtimeo;
+#endif
+
+#ifdef SO_RCVTIMEO_WORKS
+	l = sizeof tv;
+	i = getsockopt(fd, SOL_SOCKET, SO_RCVTIMEO, &tv, &l);
+	if (i) {
+		VTCP_Assert(i);
+		return;
+	}
+	assert(l == sizeof tv);
+	if (memcmp(&tv, &tv_rcvtimeo, l))
+		need_rcvtimeo = 1;
+#else
+	(void)tv;
+	(void)tv_rcvtimeo;
+	(void)need_rcvtimeo;
+#endif
+
+	need_test = 0;
+}
+
+/*--------------------------------------------------------------------
+ * Called once the workerthread gets hold of the session, to do setup
+ * setup overhead, we don't want to bother the acceptor thread with.
+ */
+
+void
+VCA_Prep(struct sess *sp)
+{
+	char addr[VTCP_ADDRBUFSIZE];
+	char port[VTCP_PORTBUFSIZE];
+
+	VTCP_name(&sp->sockaddr, sp->sockaddrlen,
+	    addr, sizeof addr, port, sizeof port);
+	sp->addr = WS_Dup(sp->ws, addr);
+	sp->port = WS_Dup(sp->ws, port);
+	if (cache_param->log_local_addr) {
+		AZ(getsockname(sp->fd, (void*)&sp->mysockaddr, &sp->mysockaddrlen));
+		VTCP_name(&sp->mysockaddr, sp->mysockaddrlen,
+		    addr, sizeof addr, port, sizeof port);
+		WSP(sp, SLT_SessionOpen, "%s %s %s %s",
+		    sp->addr, sp->port, addr, port);
+	} else {
+		WSP(sp, SLT_SessionOpen, "%s %s %s",
+		    sp->addr, sp->port, sp->mylsock->name);
+	}
+	sp->acct_ses.first = sp->t_open;
+	if (need_test)
+		sock_test(sp->fd);
+	if (need_linger)
+		VTCP_Assert(setsockopt(sp->fd, SOL_SOCKET, SO_LINGER,
+		    &linger, sizeof linger));
+#ifdef SO_SNDTIMEO_WORKS
+	if (need_sndtimeo)
+		VTCP_Assert(setsockopt(sp->fd, SOL_SOCKET, SO_SNDTIMEO,
+		    &tv_sndtimeo, sizeof tv_sndtimeo));
+#endif
+#ifdef SO_RCVTIMEO_WORKS
+	if (need_rcvtimeo)
+		VTCP_Assert(setsockopt(sp->fd, SOL_SOCKET, SO_RCVTIMEO,
+		    &tv_rcvtimeo, sizeof tv_rcvtimeo));
+#endif
+}
+
+/*--------------------------------------------------------------------
+ * If accept(2)'ing fails, we pace ourselves to relive any resource
+ * shortage if possible.
+ */
+
+static double vca_pace = 0.0;
+static struct lock pace_mtx;
+
+static void
+vca_pace_check(void)
+{
+	double p;
+
+	if (vca_pace == 0.0)
+		return;
+	Lck_Lock(&pace_mtx);
+	p = vca_pace;
+	Lck_Unlock(&pace_mtx);
+	if (p > 0.0)
+		VTIM_sleep(p);
+}
+
+static void
+vca_pace_bad(void)
+{
+
+	Lck_Lock(&pace_mtx);
+	vca_pace += cache_param->acceptor_sleep_incr;
+	if (vca_pace > cache_param->acceptor_sleep_max)
+		vca_pace = cache_param->acceptor_sleep_max;
+	Lck_Unlock(&pace_mtx);
+}
+
+static void
+vca_pace_good(void)
+{
+
+	if (vca_pace == 0.0)
+		return;
+	Lck_Lock(&pace_mtx);
+	vca_pace *= cache_param->acceptor_sleep_decay;
+	if (vca_pace < cache_param->acceptor_sleep_incr)
+		vca_pace = 0.0;
+	Lck_Unlock(&pace_mtx);
+}
+
+/*--------------------------------------------------------------------
+ * Accept on a listen socket, and handle error returns.
+ */
+
+static int hack_ready;
+
+int
+VCA_Accept(struct listen_sock *ls, struct wrk_accept *wa)
+{
+	int i;
+
+	CHECK_OBJ_NOTNULL(ls, LISTEN_SOCK_MAGIC);
+	vca_pace_check();
+
+	while(!hack_ready)
+		(void)usleep(100*1000);
+
+	wa->acceptaddrlen = sizeof wa->acceptaddr;
+	i = accept(ls->sock, (void*)&wa->acceptaddr, &wa->acceptaddrlen);
+
+	if (i < 0) {
+		switch (errno) {
+		case ECONNABORTED:
+			break;
+		case EMFILE:
+			VSL(SLT_Debug, ls->sock, "Too many open files");
+			vca_pace_bad();
+			break;
+		default:
+			VSL(SLT_Debug, ls->sock, "Accept failed: %s",
+			    strerror(errno));
+			vca_pace_bad();
+			break;
+		}
+	}
+	wa->acceptlsock = ls;
+	wa->acceptsock = i;
+	return (i);
+}
+
+/*--------------------------------------------------------------------
+ * Fail a session
+ *
+ * This happens if we accept the socket, but cannot get a session
+ * structure.
+ *
+ * We consider this a DoS situation (false positive:  Extremely popular
+ * busy objects) and silently close the connection with minimum effort
+ * and fuzz, rather than try to send an intelligent message back.
+ */
+
+void
+VCA_FailSess(struct worker *w)
+{
+	struct wrk_accept *wa;
+
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	CAST_OBJ_NOTNULL(wa, (void*)w->ws->f, WRK_ACCEPT_MAGIC);
+	AZ(w->sp);
+	AZ(close(wa->acceptsock));
+	w->stats.sess_drop++;
+	vca_pace_bad();
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+VCA_SetupSess(struct worker *w)
+{
+	struct sess *sp;
+	struct wrk_accept *wa;
+
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	CAST_OBJ_NOTNULL(wa, (void*)w->ws->f, WRK_ACCEPT_MAGIC);
+	sp = w->sp;
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	sp->fd = wa->acceptsock;
+	sp->vsl_id = wa->acceptsock | VSL_CLIENTMARKER ;
+	wa->acceptsock = -1;
+	sp->t_open = VTIM_real();
+	sp->t_end = sp->t_open;
+	sp->mylsock = wa->acceptlsock;
+	CHECK_OBJ_NOTNULL(sp->mylsock, LISTEN_SOCK_MAGIC);
+	assert(wa->acceptaddrlen <= sp->sockaddrlen);
+	memcpy(&sp->sockaddr, &wa->acceptaddr, wa->acceptaddrlen);
+	sp->sockaddrlen = wa->acceptaddrlen;
+	sp->step = STP_FIRST;
+	vca_pace_good();
+	w->stats.sess_conn++;
+}
+
+/*--------------------------------------------------------------------*/
+
+static void *
+vca_acct(void *arg)
+{
+#ifdef SO_RCVTIMEO_WORKS
+	double sess_timeout = 0;
+#endif
+#ifdef SO_SNDTIMEO_WORKS
+	double send_timeout = 0;
+#endif
+	struct listen_sock *ls;
+	double t0, now;
+
+	THR_SetName("cache-acceptor");
+	(void)arg;
+
+	VTAILQ_FOREACH(ls, &heritage.socks, list) {
+		if (ls->sock < 0)
+			continue;
+		AZ(listen(ls->sock, cache_param->listen_depth));
+		AZ(setsockopt(ls->sock, SOL_SOCKET, SO_LINGER,
+		    &linger, sizeof linger));
+	}
+
+	hack_ready = 1;
+
+	need_test = 1;
+	t0 = VTIM_real();
+	while (1) {
+		(void)sleep(1);
+#ifdef SO_SNDTIMEO_WORKS
+		if (cache_param->idle_send_timeout != send_timeout) {
+			need_test = 1;
+			send_timeout = cache_param->idle_send_timeout;
+			tv_sndtimeo = VTIM_timeval(send_timeout);
+			VTAILQ_FOREACH(ls, &heritage.socks, list) {
+				if (ls->sock < 0)
+					continue;
+				AZ(setsockopt(ls->sock, SOL_SOCKET,
+				    SO_SNDTIMEO,
+				    &tv_sndtimeo, sizeof tv_sndtimeo));
+			}
+		}
+#endif
+#ifdef SO_RCVTIMEO_WORKS
+		if (cache_param->sess_timeout != sess_timeout) {
+			need_test = 1;
+			sess_timeout = cache_param->sess_timeout;
+			tv_rcvtimeo = VTIM_timeval(sess_timeout);
+			VTAILQ_FOREACH(ls, &heritage.socks, list) {
+				if (ls->sock < 0)
+					continue;
+				AZ(setsockopt(ls->sock, SOL_SOCKET,
+				    SO_RCVTIMEO,
+				    &tv_rcvtimeo, sizeof tv_rcvtimeo));
+			}
+		}
+#endif
+		now = VTIM_real();
+		VSC_C_main->uptime = (uint64_t)(now - t0);
+	}
+	NEEDLESS_RETURN(NULL);
+}
+
+
+/*--------------------------------------------------------------------*/
+
+static void
+ccf_start(struct cli *cli, const char * const *av, void *priv)
+{
+
+	(void)cli;
+	(void)av;
+	(void)priv;
+
+	AZ(pthread_create(&VCA_thread, NULL, vca_acct, NULL));
+}
+
+/*--------------------------------------------------------------------*/
+
+static void
+ccf_listen_address(struct cli *cli, const char * const *av, void *priv)
+{
+	struct listen_sock *ls;
+	char h[32], p[32];
+
+	(void)cli;
+	(void)av;
+	(void)priv;
+
+	/*
+	 * This CLI command is primarily used by varnishtest.  Don't
+	 * respond until liste(2) has been called, in order to avoid
+	 * a race where varnishtest::client would attempt to connect(2)
+	 * before listen(2) has been called.
+	 */
+	while(!hack_ready)
+		(void)usleep(100*1000);
+
+	VTAILQ_FOREACH(ls, &heritage.socks, list) {
+		if (ls->sock < 0)
+			continue;
+		VTCP_myname(ls->sock, h, sizeof h, p, sizeof p);
+		VCLI_Out(cli, "%s %s\n", h, p);
+	}
+}
+
+/*--------------------------------------------------------------------*/
+
+static struct cli_proto vca_cmds[] = {
+	{ CLI_SERVER_START,	"i", ccf_start },
+	{ "debug.listen_address",
+	    "debug.listen_address",
+	    "Report the actual listen address\n", 0, 0,
+	    "d", ccf_listen_address, NULL },
+	{ NULL }
+};
+
+void
+VCA_Init(void)
+{
+
+	CLI_AddFuncs(vca_cmds);
+	Lck_New(&pace_mtx, lck_vcapace);
+}
+
+void
+VCA_Shutdown(void)
+{
+	struct listen_sock *ls;
+	int i;
+
+	VTAILQ_FOREACH(ls, &heritage.socks, list) {
+		if (ls->sock < 0)
+			continue;
+		i = ls->sock;
+		ls->sock = -1;
+		(void)close(i);
+	}
+}
diff --git a/bin/varnishd/cache/cache_backend.c b/bin/varnishd/cache/cache_backend.c
new file mode 100644
index 0000000..1f290c3
--- /dev/null
+++ b/bin/varnishd/cache/cache_backend.c
@@ -0,0 +1,517 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * Handle backend connections and backend request structures.
+ *
+ */
+
+#include "config.h"
+
+#include <poll.h>
+#include <stdlib.h>
+
+#include "cache.h"
+
+#include "cache_backend.h"
+#include "vrt.h"
+#include "vtcp.h"
+
+/*--------------------------------------------------------------------
+ * The "simple" director really isn't, since thats where all the actual
+ * connections happen.  Nontheless, pretend it is simple by sequestering
+ * the directoricity of it under this line.
+ */
+
+struct vdi_simple {
+	unsigned		magic;
+#define VDI_SIMPLE_MAGIC	0x476d25b7
+	struct director		dir;
+	struct backend		*backend;
+	const struct vrt_backend *vrt;
+};
+
+/*--------------------------------------------------------------------
+ * Create default Host: header for backend request
+ */
+void
+VDI_AddHostHeader(const struct sess *sp)
+{
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->wrk->bereq, HTTP_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->wrk->vbc, VBC_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->wrk->vbc->vdis, VDI_SIMPLE_MAGIC);
+	http_PrintfHeader(sp->wrk, sp->vsl_id, sp->wrk->bereq,
+	    "Host: %s", sp->wrk->vbc->vdis->vrt->hosthdr);
+}
+
+/*--------------------------------------------------------------------*/
+
+/* Private interface from backend_cfg.c */
+void
+VBE_ReleaseConn(struct vbc *vc)
+{
+
+	CHECK_OBJ_NOTNULL(vc, VBC_MAGIC);
+	assert(vc->backend == NULL);
+	assert(vc->fd < 0);
+
+	vc->addr = NULL;
+	vc->addrlen = 0;
+	vc->recycled = 0;
+	Lck_Lock(&VBE_mtx);
+	VSC_C_main->n_vbc--;
+	Lck_Unlock(&VBE_mtx);
+	FREE_OBJ(vc);
+}
+
+#define FIND_TMO(tmx, dst, sp, be)		\
+	do {					\
+		dst = sp->wrk->tmx;		\
+		if (dst == 0.0)			\
+			dst = be->tmx;		\
+		if (dst == 0.0)			\
+			dst = cache_param->tmx;	\
+	} while (0)
+
+/*--------------------------------------------------------------------
+ * Attempt to connect to a given addrinfo entry.
+ *
+ * Must be called with locked backend, but will release the backend
+ * lock during the slow/sleeping stuff, so that other worker threads
+ * can have a go, while we ponder.
+ *
+ */
+
+static int
+vbe_TryConnect(const struct sess *sp, int pf, const struct sockaddr_storage *sa,
+    socklen_t salen, const struct vdi_simple *vs)
+{
+	int s, i, tmo;
+	double tmod;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(vs, VDI_SIMPLE_MAGIC);
+
+	s = socket(pf, SOCK_STREAM, 0);
+	if (s < 0)
+		return (s);
+
+	FIND_TMO(connect_timeout, tmod, sp, vs->vrt);
+
+	tmo = (int)(tmod * 1000.0);
+
+	i = VTCP_connect(s, sa, salen, tmo);
+
+	if (i != 0) {
+		AZ(close(s));
+		return (-1);
+	}
+
+	return (s);
+}
+
+/*--------------------------------------------------------------------*/
+
+static void
+bes_conn_try(const struct sess *sp, struct vbc *vc, const struct vdi_simple *vs)
+{
+	int s;
+	struct backend *bp = vs->backend;
+	char abuf1[VTCP_ADDRBUFSIZE];
+	char pbuf1[VTCP_PORTBUFSIZE];
+
+	CHECK_OBJ_NOTNULL(vs, VDI_SIMPLE_MAGIC);
+
+	Lck_Lock(&bp->mtx);
+	bp->refcount++;
+	bp->n_conn++;		/* It mostly works */
+	Lck_Unlock(&bp->mtx);
+
+	s = -1;
+	assert(bp->ipv6 != NULL || bp->ipv4 != NULL);
+
+	/* release lock during stuff that can take a long time */
+
+	if (cache_param->prefer_ipv6 && bp->ipv6 != NULL) {
+		s = vbe_TryConnect(sp, PF_INET6, bp->ipv6, bp->ipv6len, vs);
+		vc->addr = bp->ipv6;
+		vc->addrlen = bp->ipv6len;
+	}
+	if (s == -1 && bp->ipv4 != NULL) {
+		s = vbe_TryConnect(sp, PF_INET, bp->ipv4, bp->ipv4len, vs);
+		vc->addr = bp->ipv4;
+		vc->addrlen = bp->ipv4len;
+	}
+	if (s == -1 && !cache_param->prefer_ipv6 && bp->ipv6 != NULL) {
+		s = vbe_TryConnect(sp, PF_INET6, bp->ipv6, bp->ipv6len, vs);
+		vc->addr = bp->ipv6;
+		vc->addrlen = bp->ipv6len;
+	}
+
+	vc->fd = s;
+	if (s < 0) {
+		Lck_Lock(&bp->mtx);
+		bp->n_conn--;
+		bp->refcount--;		/* Only keep ref on success */
+		Lck_Unlock(&bp->mtx);
+		vc->addr = NULL;
+		vc->addrlen = 0;
+	} else {
+		vc->vsl_id = s | VSL_BACKENDMARKER;
+		VTCP_myname(s, abuf1, sizeof abuf1, pbuf1, sizeof pbuf1);
+		WSL(sp->wrk, SLT_BackendOpen, vc->vsl_id, "%s %s %s ",
+		    vs->backend->display_name, abuf1, pbuf1);
+	}
+
+}
+
+/*--------------------------------------------------------------------
+ * Check that there is still something at the far end of a given socket.
+ * We poll the fd with instant timeout, if there are any events we can't
+ * use it (backends are not allowed to pipeline).
+ */
+
+static int
+vbe_CheckFd(int fd)
+{
+	struct pollfd pfd;
+
+	pfd.fd = fd;
+	pfd.events = POLLIN;
+	pfd.revents = 0;
+	return(poll(&pfd, 1, 0) == 0);
+}
+
+/*--------------------------------------------------------------------
+ * Manage a pool of vbc structures.
+ * XXX: as an experiment, make this caching controled by a parameter
+ * XXX: so we can see if it has any effect.
+ */
+
+static struct vbc *
+vbe_NewConn(void)
+{
+	struct vbc *vc;
+
+	ALLOC_OBJ(vc, VBC_MAGIC);
+	XXXAN(vc);
+	vc->fd = -1;
+	Lck_Lock(&VBE_mtx);
+	VSC_C_main->n_vbc++;
+	Lck_Unlock(&VBE_mtx);
+	return (vc);
+}
+
+/*--------------------------------------------------------------------
+ * It evaluates if a backend is healthy _for_a_specific_object_.
+ * That means that it relies on sp->objcore->objhead. This is mainly for
+ * saint-mode, but also takes backend->healthy into account. If
+ * cache_param->saintmode_threshold is 0, this is basically just a test of
+ * backend->healthy.
+ *
+ * The threshold has to be evaluated _after_ the timeout check, otherwise
+ * items would never time out once the threshold is reached.
+ */
+
+static unsigned int
+vbe_Healthy(const struct vdi_simple *vs, const struct sess *sp)
+{
+	struct trouble *tr;
+	struct trouble *tr2;
+	struct trouble *old;
+	unsigned i = 0, retval;
+	unsigned int threshold;
+	struct backend *backend;
+	uintptr_t target;
+	double now;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(vs, VDI_SIMPLE_MAGIC);
+	backend = vs->backend;
+	CHECK_OBJ_NOTNULL(backend, BACKEND_MAGIC);
+
+	if (backend->admin_health == ah_probe && !backend->healthy)
+		return (0);
+
+	if (backend->admin_health == ah_sick)
+		return (0);
+
+	/* VRT/VCC sets threshold to UINT_MAX to mark that it's not
+	 * specified by VCL (thus use param).
+	 */
+	if (vs->vrt->saintmode_threshold == UINT_MAX)
+		threshold = cache_param->saintmode_threshold;
+	else
+		threshold = vs->vrt->saintmode_threshold;
+
+	if (backend->admin_health == ah_healthy)
+		threshold = UINT_MAX;
+
+	/* Saintmode is disabled */
+	if (threshold == 0)
+		return (1);
+
+	if (sp->objcore == NULL)
+		return (1);
+
+	now = sp->t_req;
+	target = (uintptr_t)(sp->objcore->objhead);
+
+	old = NULL;
+	retval = 1;
+	Lck_Lock(&backend->mtx);
+	VTAILQ_FOREACH_SAFE(tr, &backend->troublelist, list, tr2) {
+		CHECK_OBJ_NOTNULL(tr, TROUBLE_MAGIC);
+
+		if (tr->timeout < now) {
+			VTAILQ_REMOVE(&backend->troublelist, tr, list);
+			old = tr;
+			retval = 1;
+			break;
+		}
+
+		if (tr->target == target) {
+			retval = 0;
+			break;
+		}
+
+		/* If the threshold is at 1, a single entry on the list
+		 * will disable the backend. Since 0 is disable, ++i
+		 * instead of i++ to allow this behavior.
+		 */
+		if (++i >= threshold) {
+			retval = 0;
+			break;
+		}
+	}
+	Lck_Unlock(&backend->mtx);
+
+	if (old != NULL)
+		FREE_OBJ(old);
+
+	return (retval);
+}
+
+/*--------------------------------------------------------------------
+ * Get a connection to a particular backend.
+ */
+
+static struct vbc *
+vbe_GetVbe(const struct sess *sp, struct vdi_simple *vs)
+{
+	struct vbc *vc;
+	struct backend *bp;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(vs, VDI_SIMPLE_MAGIC);
+	bp = vs->backend;
+	CHECK_OBJ_NOTNULL(bp, BACKEND_MAGIC);
+
+	/* first look for vbc's we can recycle */
+	while (1) {
+		Lck_Lock(&bp->mtx);
+		vc = VTAILQ_FIRST(&bp->connlist);
+		if (vc != NULL) {
+			bp->refcount++;
+			assert(vc->backend == bp);
+			assert(vc->fd >= 0);
+			AN(vc->addr);
+			VTAILQ_REMOVE(&bp->connlist, vc, list);
+		}
+		Lck_Unlock(&bp->mtx);
+		if (vc == NULL)
+			break;
+		if (vbe_CheckFd(vc->fd)) {
+			/* XXX locking of stats */
+			VSC_C_main->backend_reuse += 1;
+			WSP(sp, SLT_Backend, "%d %s %s",
+			    vc->fd, sp->director->vcl_name, bp->display_name);
+			vc->vdis = vs;
+			vc->recycled = 1;
+			return (vc);
+		}
+		VSC_C_main->backend_toolate++;
+		WSL(sp->wrk, SLT_BackendClose, vc->vsl_id, "%s", bp->display_name);
+
+		/* Checkpoint log to flush all info related to this connection
+		   before the OS reuses the FD */
+		WSL_Flush(sp->wrk, 0);
+
+		VTCP_close(&vc->fd);
+		VBE_DropRefConn(bp);
+		vc->backend = NULL;
+		VBE_ReleaseConn(vc);
+	}
+
+	if (!vbe_Healthy(vs, sp)) {
+		VSC_C_main->backend_unhealthy++;
+		return (NULL);
+	}
+
+	if (vs->vrt->max_connections > 0 &&
+	    bp->n_conn >= vs->vrt->max_connections) {
+		VSC_C_main->backend_busy++;
+		return (NULL);
+	}
+
+	vc = vbe_NewConn();
+	assert(vc->fd == -1);
+	AZ(vc->backend);
+	bes_conn_try(sp, vc, vs);
+	if (vc->fd < 0) {
+		VBE_ReleaseConn(vc);
+		VSC_C_main->backend_fail++;
+		return (NULL);
+	}
+	vc->backend = bp;
+	VSC_C_main->backend_conn++;
+	WSP(sp, SLT_Backend, "%d %s %s",
+	    vc->fd, sp->director->vcl_name, bp->display_name);
+	vc->vdis = vs;
+	return (vc);
+}
+
+/*--------------------------------------------------------------------
+ * Returns the backend if and only if the this is a simple director.
+ * XXX: Needs a better name and possibly needs a better general approach.
+ * XXX: This is mainly used by the DNS director to fetch the actual backend
+ * XXX: so it can compare DNS lookups with the actual IP.
+ */
+
+struct backend *
+vdi_get_backend_if_simple(const struct director *d)
+{
+	CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC);
+	struct vdi_simple *vs, *vs2;
+
+	vs2 = d->priv;
+	if (vs2->magic != VDI_SIMPLE_MAGIC)
+		return (NULL);
+	CAST_OBJ_NOTNULL(vs, d->priv, VDI_SIMPLE_MAGIC);
+	return (vs->backend);
+}
+
+/*--------------------------------------------------------------------
+ *
+ */
+
+void
+VBE_UseHealth(const struct director *vdi)
+{
+	struct vdi_simple *vs;
+
+	ASSERT_CLI();
+
+	if (strcmp(vdi->name, "simple"))
+		return;
+	CAST_OBJ_NOTNULL(vs, vdi->priv, VDI_SIMPLE_MAGIC);
+	if (vs->vrt->probe == NULL)
+		return;
+	VBP_Use(vs->backend, vs->vrt->probe);
+}
+
+/*--------------------------------------------------------------------
+ *
+ */
+
+static struct vbc * __match_proto__(vdi_getfd_f)
+vdi_simple_getfd(const struct director *d, struct sess *sp)
+{
+	struct vdi_simple *vs;
+	struct vbc *vc;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC);
+	CAST_OBJ_NOTNULL(vs, d->priv, VDI_SIMPLE_MAGIC);
+	vc = vbe_GetVbe(sp, vs);
+	if (vc != NULL) {
+		FIND_TMO(first_byte_timeout,
+		    vc->first_byte_timeout, sp, vs->vrt);
+		FIND_TMO(between_bytes_timeout,
+		    vc->between_bytes_timeout, sp, vs->vrt);
+	}
+	return (vc);
+}
+
+static unsigned
+vdi_simple_healthy(const struct director *d, const struct sess *sp)
+{
+	struct vdi_simple *vs;
+
+	CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC);
+	CAST_OBJ_NOTNULL(vs, d->priv, VDI_SIMPLE_MAGIC);
+	return (vbe_Healthy(vs, sp));
+}
+
+static void
+vdi_simple_fini(const struct director *d)
+{
+	struct vdi_simple *vs;
+
+	ASSERT_CLI();
+	CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC);
+	CAST_OBJ_NOTNULL(vs, d->priv, VDI_SIMPLE_MAGIC);
+
+	if (vs->vrt->probe != NULL)
+		VBP_Remove(vs->backend, vs->vrt->probe);
+	VBE_DropRefVcl(vs->backend);
+	free(vs->dir.vcl_name);
+	vs->dir.magic = 0;
+	FREE_OBJ(vs);
+}
+
+void
+VRT_init_dir_simple(struct cli *cli, struct director **bp, int idx,
+    const void *priv)
+{
+	const struct vrt_backend *t;
+	struct vdi_simple *vs;
+
+	ASSERT_CLI();
+	(void)cli;
+	t = priv;
+
+	ALLOC_OBJ(vs, VDI_SIMPLE_MAGIC);
+	XXXAN(vs);
+	vs->dir.magic = DIRECTOR_MAGIC;
+	vs->dir.priv = vs;
+	vs->dir.name = "simple";
+	REPLACE(vs->dir.vcl_name, t->vcl_name);
+	vs->dir.getfd = vdi_simple_getfd;
+	vs->dir.fini = vdi_simple_fini;
+	vs->dir.healthy = vdi_simple_healthy;
+
+	vs->vrt = t;
+
+	vs->backend = VBE_AddBackend(cli, t);
+	if (vs->vrt->probe != NULL)
+		VBP_Insert(vs->backend, vs->vrt->probe, vs->vrt->hosthdr);
+
+	bp[idx] = &vs->dir;
+}
diff --git a/bin/varnishd/cache/cache_backend.h b/bin/varnishd/cache/cache_backend.h
new file mode 100644
index 0000000..72a1283
--- /dev/null
+++ b/bin/varnishd/cache/cache_backend.h
@@ -0,0 +1,192 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * This is the central switch-board for backend connections and it is
+ * slightly complicated by a number of optimizations.
+ *
+ * The data structures:
+ *
+ *    A vrt_backend is a definition of a backend in a VCL program.
+ *
+ *    A backend is a TCP destination, possibly multi-homed and it has a
+ *    number of associated properties and statistics.
+ *
+ *    A vbc is an open TCP connection to a backend.
+ *
+ *    A bereq is a memory carrier for handling a HTTP transaction with
+ *    a backend over a vbc.
+ *
+ *    A director is a piece of code that selects which backend to use,
+ *    by whatever method or metric it chooses.
+ *
+ * The relationships:
+ *
+ *    Backends and directors get instantiated when VCL's are loaded,
+ *    and this always happen in the CLI thread.
+ *
+ *    When a VCL tries to instantiate a backend, any existing backend
+ *    with the same identity (== definition in VCL) will be used instead
+ *    so that vbc's can be reused across VCL changes.
+ *
+ *    Directors disapper with the VCL that created them.
+ *
+ *    Backends disappear when their reference count drop to zero.
+ *
+ *    Backends have their host/port name looked up to addrinfo structures
+ *    when they are instantiated, and we just cache that result and cycle
+ *    through the entries (for multihomed backends) on failure only.
+ *    XXX: add cli command to redo lookup.
+ *
+ *    bereq is sort of a step-child here, we just manage the pool of them.
+ *
+ */
+
+struct vbp_target;
+struct vbc;
+struct vrt_backend_probe;
+
+/*--------------------------------------------------------------------
+ * A director is a piece of code which selects one of possibly multiple
+ * backends to use.
+ */
+
+typedef struct vbc *vdi_getfd_f(const struct director *, struct sess *sp);
+typedef void vdi_fini_f(const struct director *);
+typedef unsigned vdi_healthy(const struct director *, const struct sess *sp);
+
+struct director {
+	unsigned		magic;
+#define DIRECTOR_MAGIC		0x3336351d
+	const char		*name;
+	char			*vcl_name;
+	vdi_getfd_f		*getfd;
+	vdi_fini_f		*fini;
+	vdi_healthy		*healthy;
+	void			*priv;
+};
+
+/*--------------------------------------------------------------------
+ * List of objectheads that have recently been rejected by VCL.
+ */
+
+struct trouble {
+	unsigned		magic;
+#define TROUBLE_MAGIC		0x4211ab21
+	uintptr_t		target;
+	double			timeout;
+	VTAILQ_ENTRY(trouble)	list;
+};
+
+/*--------------------------------------------------------------------
+ * An instance of a backend from a VCL program.
+ */
+
+enum admin_health {
+	ah_invalid = 0,
+	ah_healthy,
+	ah_sick,
+	ah_probe
+};
+
+struct backend {
+	unsigned		magic;
+#define BACKEND_MAGIC		0x64c4c7c6
+
+	VTAILQ_ENTRY(backend)	list;
+	int			refcount;
+	struct lock		mtx;
+
+	char			*vcl_name;
+	char			*display_name;
+	char			*ipv4_addr;
+	char			*ipv6_addr;
+	char			*port;
+
+	struct sockaddr_storage	*ipv4;
+	socklen_t		ipv4len;
+	struct sockaddr_storage	*ipv6;
+	socklen_t		ipv6len;
+
+	unsigned		n_conn;
+	VTAILQ_HEAD(, vbc)	connlist;
+
+	struct vbp_target	*probe;
+	unsigned		healthy;
+	enum admin_health	admin_health;
+	VTAILQ_HEAD(, trouble)	troublelist;
+
+	struct VSC_C_vbe	*vsc;
+};
+
+/* -------------------------------------------------------------------*/
+
+/* Backend connection */
+struct vbc {
+	unsigned		magic;
+#define VBC_MAGIC		0x0c5e6592
+	VTAILQ_ENTRY(vbc)	list;
+	struct backend		*backend;
+	struct vdi_simple	*vdis;
+	unsigned		vsl_id;
+	int			fd;
+
+	struct sockaddr_storage	*addr;
+	socklen_t		addrlen;
+
+	uint8_t			recycled;
+
+	/* Timeouts */
+	double			first_byte_timeout;
+	double			between_bytes_timeout;
+};
+
+/* cache_backend.c */
+void VBE_ReleaseConn(struct vbc *vc);
+struct backend *vdi_get_backend_if_simple(const struct director *d);
+
+/* cache_backend_cfg.c */
+extern struct lock VBE_mtx;
+void VBE_DropRefConn(struct backend *);
+void VBE_DropRefVcl(struct backend *);
+void VBE_DropRefLocked(struct backend *b);
+
+/* cache_backend_poll.c */
+void VBP_Insert(struct backend *b, struct vrt_backend_probe const *p, const char *hosthdr);
+void VBP_Remove(struct backend *b, struct vrt_backend_probe const *p);
+void VBP_Use(const struct backend *b, const struct vrt_backend_probe const *p);
+void VBP_Summary(struct cli *cli, const struct vbp_target *vt);
+
+/* Init functions for directors */
+typedef void dir_init_f(struct cli *, struct director **, int , const void*);
+dir_init_f VRT_init_dir_simple;
+dir_init_f VRT_init_dir_dns;
+dir_init_f VRT_init_dir_hash;
+dir_init_f VRT_init_dir_random;
+dir_init_f VRT_init_dir_round_robin;
+dir_init_f VRT_init_dir_fallback;
+dir_init_f VRT_init_dir_client;
diff --git a/bin/varnishd/cache/cache_backend_cfg.c b/bin/varnishd/cache/cache_backend_cfg.c
new file mode 100644
index 0000000..cbb8c85
--- /dev/null
+++ b/bin/varnishd/cache/cache_backend_cfg.c
@@ -0,0 +1,507 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * Handle configuration of backends from VCL programs.
+ *
+ */
+
+#include "config.h"
+
+#include <ctype.h>
+#include <stdio.h>
+#include <stdlib.h>
+
+#include "cache.h"
+
+#include "cache_backend.h"
+#include "vcli.h"
+#include "vcli_priv.h"
+#include "vrt.h"
+
+struct lock VBE_mtx;
+
+
+/*
+ * The list of backends is not locked, it is only ever accessed from
+ * the CLI thread, so there is no need.
+ */
+static VTAILQ_HEAD(, backend) backends = VTAILQ_HEAD_INITIALIZER(backends);
+
+/*--------------------------------------------------------------------
+ */
+
+static void
+VBE_Nuke(struct backend *b)
+{
+
+	ASSERT_CLI();
+	VTAILQ_REMOVE(&backends, b, list);
+	free(b->ipv4);
+	free(b->ipv4_addr);
+	free(b->ipv6);
+	free(b->ipv6_addr);
+	free(b->port);
+	VSM_Free(b->vsc);
+	FREE_OBJ(b);
+	VSC_C_main->n_backend--;
+}
+
+/*--------------------------------------------------------------------
+ */
+
+void
+VBE_Poll(void)
+{
+	struct backend *b, *b2;
+
+	ASSERT_CLI();
+	VTAILQ_FOREACH_SAFE(b, &backends, list, b2) {
+		assert(
+			b->admin_health == ah_healthy ||
+			b->admin_health == ah_sick ||
+			b->admin_health == ah_probe
+		);
+		if (b->refcount == 0 && b->probe == NULL)
+			VBE_Nuke(b);
+	}
+}
+
+/*--------------------------------------------------------------------
+ * Drop a reference to a backend.
+ * The last reference must come from the watcher in the CLI thread,
+ * as only that thread is allowed to clean up the backend list.
+ */
+
+void
+VBE_DropRefLocked(struct backend *b)
+{
+	int i;
+	struct vbc *vbe, *vbe2;
+
+	CHECK_OBJ_NOTNULL(b, BACKEND_MAGIC);
+	assert(b->refcount > 0);
+
+	i = --b->refcount;
+	Lck_Unlock(&b->mtx);
+	if (i > 0)
+		return;
+
+	ASSERT_CLI();
+	VTAILQ_FOREACH_SAFE(vbe, &b->connlist, list, vbe2) {
+		VTAILQ_REMOVE(&b->connlist, vbe, list);
+		if (vbe->fd >= 0) {
+			AZ(close(vbe->fd));
+			vbe->fd = -1;
+		}
+		vbe->backend = NULL;
+		VBE_ReleaseConn(vbe);
+	}
+	VBE_Nuke(b);
+}
+
+void
+VBE_DropRefVcl(struct backend *b)
+{
+
+	CHECK_OBJ_NOTNULL(b, BACKEND_MAGIC);
+
+	Lck_Lock(&b->mtx);
+	b->vsc->vcls--;
+	VBE_DropRefLocked(b);
+}
+
+void
+VBE_DropRefConn(struct backend *b)
+{
+
+	CHECK_OBJ_NOTNULL(b, BACKEND_MAGIC);
+
+	Lck_Lock(&b->mtx);
+	assert(b->n_conn > 0);
+	b->n_conn--;
+	VBE_DropRefLocked(b);
+}
+
+/*--------------------------------------------------------------------
+ * See lib/libvcl/vcc_backend.c::emit_sockaddr()
+ */
+
+static void
+copy_sockaddr(struct sockaddr_storage **sa, socklen_t *len,
+    const unsigned char *src)
+{
+
+	assert(*src > 0);
+	*sa = calloc(sizeof **sa, 1);
+	XXXAN(*sa);
+	memcpy(*sa, src + 1, *src);
+	*len = *src;
+}
+
+/*--------------------------------------------------------------------
+ * Add a backend/director instance when loading a VCL.
+ * If an existing backend is matched, grab a refcount and return.
+ * Else create a new backend structure with reference initialized to one.
+ */
+
+struct backend *
+VBE_AddBackend(struct cli *cli, const struct vrt_backend *vb)
+{
+	struct backend *b;
+	char buf[128];
+
+	AN(vb->vcl_name);
+	assert(vb->ipv4_sockaddr != NULL || vb->ipv6_sockaddr != NULL);
+	(void)cli;
+	ASSERT_CLI();
+
+	/* Run through the list and see if we already have this backend */
+	VTAILQ_FOREACH(b, &backends, list) {
+		CHECK_OBJ_NOTNULL(b, BACKEND_MAGIC);
+		if (strcmp(b->vcl_name, vb->vcl_name))
+			continue;
+		if (vb->ipv4_sockaddr != NULL && (
+		    b->ipv4len != vb->ipv4_sockaddr[0] ||
+		    memcmp(b->ipv4, vb->ipv4_sockaddr + 1, b->ipv4len)))
+			continue;
+		if (vb->ipv6_sockaddr != NULL && (
+		    b->ipv6len != vb->ipv6_sockaddr[0] ||
+		    memcmp(b->ipv6, vb->ipv6_sockaddr + 1, b->ipv6len)))
+			continue;
+		b->refcount++;
+		b->vsc->vcls++;
+		return (b);
+	}
+
+	/* Create new backend */
+	ALLOC_OBJ(b, BACKEND_MAGIC);
+	XXXAN(b);
+	Lck_New(&b->mtx, lck_backend);
+	b->refcount = 1;
+
+	bprintf(buf, "%s(%s,%s,%s)",
+	    vb->vcl_name,
+	    vb->ipv4_addr == NULL ? "" : vb->ipv4_addr,
+	    vb->ipv6_addr == NULL ? "" : vb->ipv6_addr, vb->port);
+
+	b->vsc = VSM_Alloc(sizeof *b->vsc, VSC_CLASS, VSC_TYPE_VBE, buf);
+	b->vsc->vcls++;
+
+	VTAILQ_INIT(&b->connlist);
+
+	VTAILQ_INIT(&b->troublelist);
+
+	/*
+	 * This backend may live longer than the VCL that instantiated it
+	 * so we cannot simply reference the VCL's copy of things.
+	 */
+	REPLACE(b->vcl_name, vb->vcl_name);
+	REPLACE(b->display_name, buf);
+	REPLACE(b->ipv4_addr, vb->ipv4_addr);
+	REPLACE(b->ipv6_addr, vb->ipv6_addr);
+	REPLACE(b->port, vb->port);
+
+	/*
+	 * Copy over the sockaddrs
+	 */
+	if (vb->ipv4_sockaddr != NULL)
+		copy_sockaddr(&b->ipv4, &b->ipv4len, vb->ipv4_sockaddr);
+	if (vb->ipv6_sockaddr != NULL)
+		copy_sockaddr(&b->ipv6, &b->ipv6len, vb->ipv6_sockaddr);
+
+	assert(b->ipv4 != NULL || b->ipv6 != NULL);
+
+	b->healthy = 1;
+	b->admin_health = ah_probe;
+
+	VTAILQ_INSERT_TAIL(&backends, b, list);
+	VSC_C_main->n_backend++;
+	return (b);
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+VRT_init_dir(struct cli *cli, struct director **dir, const char *name,
+    int idx, const void *priv)
+{
+
+	ASSERT_CLI();
+	if (!strcmp(name, "simple"))
+		VRT_init_dir_simple(cli, dir, idx, priv);
+	else if (!strcmp(name, "hash"))
+		VRT_init_dir_hash(cli, dir, idx, priv);
+	else if (!strcmp(name, "random"))
+		VRT_init_dir_random(cli, dir, idx, priv);
+	else if (!strcmp(name, "dns"))
+		VRT_init_dir_dns(cli, dir, idx, priv);
+	else if (!strcmp(name, "round-robin"))
+		VRT_init_dir_round_robin(cli, dir, idx, priv);
+	else if (!strcmp(name, "fallback"))
+		VRT_init_dir_fallback(cli, dir, idx, priv);
+	else if (!strcmp(name, "client"))
+		VRT_init_dir_client(cli, dir, idx, priv);
+	else
+		INCOMPL();
+}
+
+void
+VRT_fini_dir(struct cli *cli, struct director *b)
+{
+
+	(void)cli;
+	ASSERT_CLI();
+	CHECK_OBJ_NOTNULL(b, DIRECTOR_MAGIC);
+	b->fini(b);
+	b->priv = NULL;
+}
+
+/*---------------------------------------------------------------------
+ * String to admin_health
+ */
+
+static enum admin_health
+vbe_str2adminhealth(const char *wstate)
+{
+
+	if (strcasecmp(wstate, "healthy") == 0)
+		return(ah_healthy);
+	if (strcasecmp(wstate, "sick") == 0)
+		return(ah_sick);
+	if (strcmp(wstate, "auto") == 0)
+		return(ah_probe);
+	return (ah_invalid);
+}
+
+/*---------------------------------------------------------------------
+ * A general function for finding backends and doing things with them.
+ *
+ * Return -1 on match-argument parse errors.
+ *
+ * If the call-back function returns non-zero, the search is terminated
+ * and we relay that return value.
+ *
+ * Otherwise we return the number of matches.
+ */
+
+typedef int bf_func(struct cli *cli, struct backend *b, void *priv);
+
+static int
+backend_find(struct cli *cli, const char *matcher, bf_func *func, void *priv)
+{
+	struct backend *b;
+	const char *s;
+	const char *name_b;
+	ssize_t name_l = 0;
+	const char *ip_b = NULL;
+	ssize_t ip_l = 0;
+	const char *port_b = NULL;
+	ssize_t port_l = 0;
+	int found = 0;
+	int i;
+
+	name_b = matcher;
+	if (matcher != NULL) {
+		s = strchr(matcher,'(');
+
+		if (s != NULL)
+			name_l = s - name_b;
+		else
+			name_l = strlen(name_b);
+
+		if (s != NULL) {
+			s++;
+			while (isspace(*s))
+				s++;
+			ip_b = s;
+			while (*s != '\0' &&
+			    *s != ')' &&
+			    *s != ':' &&
+			    !isspace(*s))
+				s++;
+			ip_l = s - ip_b;
+			while (isspace(*s))
+				s++;
+			if (*s == ':') {
+				s++;
+				while (isspace(*s))
+					s++;
+				port_b = s;
+				while (*s != '\0' && *s != ')' && !isspace(*s))
+					s++;
+				port_l = s - port_b;
+			}
+			while (isspace(*s))
+				s++;
+			if (*s != ')') {
+				VCLI_Out(cli,
+				    "Match string syntax error:"
+				    " ')' not found.");
+				VCLI_SetResult(cli, CLIS_CANT);
+				return (-1);
+			}
+			s++;
+			while (isspace(*s))
+				s++;
+			if (*s != '\0') {
+				VCLI_Out(cli,
+				    "Match string syntax error:"
+				    " junk after ')'");
+				VCLI_SetResult(cli, CLIS_CANT);
+				return (-1);
+			}
+		}
+	}
+	VTAILQ_FOREACH(b, &backends, list) {
+		CHECK_OBJ_NOTNULL(b, BACKEND_MAGIC);
+		if (port_b != NULL && strncmp(b->port, port_b, port_l) != 0)
+			continue;
+		if (name_b != NULL && strncmp(b->vcl_name, name_b, name_l) != 0)
+			continue;
+		if (ip_b != NULL &&
+		    (b->ipv4_addr == NULL ||
+		      strncmp(b->ipv4_addr, ip_b, ip_l)) &&
+		    (b->ipv6_addr == NULL ||
+		      strncmp(b->ipv6_addr, ip_b, ip_l)))
+			continue;
+		found++;
+		i = func(cli, b, priv);
+		if (i)
+			return(i);
+	}
+	return (found);
+}
+
+/*---------------------------------------------------------------------*/
+
+static int __match_proto__()
+do_list(struct cli *cli, struct backend *b, void *priv)
+{
+	int *hdr;
+
+	AN(priv);
+	hdr = priv;
+	if (!*hdr) {
+		VCLI_Out(cli, "%-30s %-6s %-10s %s",
+		    "Backend name", "Refs", "Admin", "Probe");
+		*hdr = 1;
+	}
+	CHECK_OBJ_NOTNULL(b, BACKEND_MAGIC);
+
+	VCLI_Out(cli, "\n%-30s %-6d", b->display_name, b->refcount);
+
+	if (b->admin_health == ah_probe)
+		VCLI_Out(cli, " %-10s", "probe");
+	else if (b->admin_health == ah_sick)
+		VCLI_Out(cli, " %-10s", "sick");
+	else if (b->admin_health == ah_healthy)
+		VCLI_Out(cli, " %-10s", "healthy");
+	else
+		VCLI_Out(cli, " %-10s", "invalid");
+
+	if (b->probe == NULL)
+		VCLI_Out(cli, " %s", "Healthy (no probe)");
+	else {
+		if (b->healthy)
+			VCLI_Out(cli, " %s", "Healthy ");
+		else
+			VCLI_Out(cli, " %s", "Sick ");
+		VBP_Summary(cli, b->probe);
+	}
+
+	return (0);
+}
+
+static void
+cli_backend_list(struct cli *cli, const char * const *av, void *priv)
+{
+	int hdr = 0;
+
+	(void)av;
+	(void)priv;
+	ASSERT_CLI();
+	(void)backend_find(cli, av[2], do_list, &hdr);
+}
+
+/*---------------------------------------------------------------------*/
+
+static int __match_proto__()
+do_set_health(struct cli *cli, struct backend *b, void *priv)
+{
+	enum admin_health state;
+
+	(void)cli;
+	state = *(enum admin_health*)priv;
+	CHECK_OBJ_NOTNULL(b, BACKEND_MAGIC);
+	b->admin_health = state;
+	return (0);
+}
+
+static void
+cli_backend_set_health(struct cli *cli, const char * const *av, void *priv)
+{
+	enum admin_health state;
+	int n;
+
+	(void)av;
+	(void)priv;
+	ASSERT_CLI();
+	AN(av[2]);
+	AN(av[3]);
+	state = vbe_str2adminhealth(av[3]);
+	if (state == ah_invalid) {
+		VCLI_Out(cli, "Invalid state %s", av[3]);
+		VCLI_SetResult(cli, CLIS_PARAM);
+		return;
+	}
+	n = backend_find(cli, av[2], do_set_health, &state);
+	if (n == 0) {
+		VCLI_Out(cli, "No Backends matches");
+		VCLI_SetResult(cli, CLIS_PARAM);
+	}
+}
+
+/*---------------------------------------------------------------------*/
+
+static struct cli_proto backend_cmds[] = {
+	{ "backend.list", "backend.list",
+	    "\tList all backends\n", 0, 1, "d", cli_backend_list },
+	{ "backend.set_health", "backend.set_health matcher state",
+	    "\tShow a backend\n", 2, 2, "d", cli_backend_set_health },
+	{ NULL }
+};
+
+/*---------------------------------------------------------------------*/
+
+void
+VBE_Init(void)
+{
+
+	Lck_New(&VBE_mtx, lck_vbe);
+	CLI_AddFuncs(backend_cmds);
+}
diff --git a/bin/varnishd/cache/cache_backend_poll.c b/bin/varnishd/cache/cache_backend_poll.c
new file mode 100644
index 0000000..efd64cb
--- /dev/null
+++ b/bin/varnishd/cache/cache_backend_poll.c
@@ -0,0 +1,597 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * Poll backends for collection of health statistics
+ *
+ * We co-opt threads from the worker pool for probing the backends,
+ * but we want to avoid a potentially messy cleanup operation when we
+ * retire the backend, so the thread owns the health information, which
+ * the backend references, rather than the other way around.
+ *
+ */
+
+#include "config.h"
+
+#include <math.h>
+#include <poll.h>
+#include <stdio.h>
+#include <stdlib.h>
+
+#include "cache.h"
+
+#include "cache_backend.h"
+#include "vcli_priv.h"
+#include "vrt.h"
+#include "vtcp.h"
+#include "vtim.h"
+
+/* Default averaging rate, we want something pretty responsive */
+#define AVG_RATE			4
+
+struct vbp_vcl {
+	unsigned			magic;
+#define VBP_VCL_MAGIC			0x70829764
+
+	VTAILQ_ENTRY(vbp_vcl)		list;
+	const struct vrt_backend_probe	*probep;
+	struct vrt_backend_probe	probe;
+	const char			*hosthdr;
+};
+
+struct vbp_target {
+	unsigned			magic;
+#define VBP_TARGET_MAGIC		0x6b7cb656
+
+	struct backend			*backend;
+	VTAILQ_HEAD( ,vbp_vcl)		vcls;
+
+	struct vrt_backend_probe	probe;
+	int				stop;
+	struct vsb			*vsb;
+	char				*req;
+	int				req_len;
+
+	char				resp_buf[128];
+	unsigned			good;
+
+	/* Collected statistics */
+#define BITMAP(n, c, t, b)	uint64_t	n;
+#include "tbl/backend_poll.h"
+#undef BITMAP
+
+	double				last;
+	double				avg;
+	double				rate;
+
+	VTAILQ_ENTRY(vbp_target)	list;
+	pthread_t			thread;
+};
+
+static VTAILQ_HEAD(, vbp_target)	vbp_list =
+    VTAILQ_HEAD_INITIALIZER(vbp_list);
+
+static struct lock			vbp_mtx;
+
+/*--------------------------------------------------------------------
+ * Poke one backend, once, but possibly at both IPv4 and IPv6 addresses.
+ *
+ * We do deliberately not use the stuff in cache_backend.c, because we
+ * want to measure the backends response without local distractions.
+ */
+
+static int
+vbp_connect(int pf, const struct sockaddr_storage *sa, socklen_t salen, int tmo)
+{
+	int s, i;
+
+	s = socket(pf, SOCK_STREAM, 0);
+	if (s < 0)
+		return (s);
+
+	i = VTCP_connect(s, sa, salen, tmo);
+	if (i == 0)
+		return (s);
+	VTCP_close(&s);
+	return (-1);
+}
+
+static void
+vbp_poke(struct vbp_target *vt)
+{
+	int s, tmo, i;
+	double t_start, t_now, t_end;
+	unsigned rlen, resp;
+	struct backend *bp;
+	char buf[8192], *p;
+	struct pollfd pfda[1], *pfd = pfda;
+
+	bp = vt->backend;
+	CHECK_OBJ_NOTNULL(bp, BACKEND_MAGIC);
+
+	t_start = t_now = VTIM_real();
+	t_end = t_start + vt->probe.timeout;
+	tmo = (int)round((t_end - t_now) * 1e3);
+
+	s = -1;
+	if (cache_param->prefer_ipv6 && bp->ipv6 != NULL) {
+		s = vbp_connect(PF_INET6, bp->ipv6, bp->ipv6len, tmo);
+		t_now = VTIM_real();
+		tmo = (int)round((t_end - t_now) * 1e3);
+		if (s >= 0)
+			vt->good_ipv6 |= 1;
+	}
+	if (tmo > 0 && s < 0 && bp->ipv4 != NULL) {
+		s = vbp_connect(PF_INET, bp->ipv4, bp->ipv4len, tmo);
+		t_now = VTIM_real();
+		tmo = (int)round((t_end - t_now) * 1e3);
+		if (s >= 0)
+			vt->good_ipv4 |= 1;
+	}
+	if (tmo > 0 && s < 0 && bp->ipv6 != NULL) {
+		s = vbp_connect(PF_INET6, bp->ipv6, bp->ipv6len, tmo);
+		t_now = VTIM_real();
+		tmo = (int)round((t_end - t_now) * 1e3);
+		if (s >= 0)
+			vt->good_ipv6 |= 1;
+	}
+	if (s < 0) {
+		/* Got no connection: failed */
+		return;
+	}
+	if (tmo <= 0) {
+		/* Spent too long time getting it */
+		VTCP_close(&s);
+		return;
+	}
+
+	/* Send the request */
+	i = write(s, vt->req, vt->req_len);
+	if (i != vt->req_len) {
+		if (i < 0)
+			vt->err_xmit |= 1;
+		VTCP_close(&s);
+		return;
+	}
+	vt->good_xmit |= 1;
+
+	pfd->fd = s;
+	rlen = 0;
+	do {
+		pfd->events = POLLIN;
+		pfd->revents = 0;
+		tmo = (int)round((t_end - t_now) * 1e3);
+		if (tmo > 0)
+			i = poll(pfd, 1, tmo);
+		if (i == 0 || tmo <= 0) {
+			VTCP_close(&s);
+			return;
+		}
+		if (rlen < sizeof vt->resp_buf)
+			i = read(s, vt->resp_buf + rlen,
+			    sizeof vt->resp_buf - rlen);
+		else
+			i = read(s, buf, sizeof buf);
+		rlen += i;
+	} while (i > 0);
+
+	VTCP_close(&s);
+
+	if (i < 0) {
+		vt->err_recv |= 1;
+		return;
+	}
+
+	if (rlen == 0)
+		return;
+
+	/* So we have a good receive ... */
+	t_now = VTIM_real();
+	vt->last = t_now - t_start;
+	vt->good_recv |= 1;
+
+	/* Now find out if we like the response */
+	vt->resp_buf[sizeof vt->resp_buf - 1] = '\0';
+	p = strchr(vt->resp_buf, '\r');
+	if (p != NULL)
+		*p = '\0';
+	p = strchr(vt->resp_buf, '\n');
+	if (p != NULL)
+		*p = '\0';
+
+	i = sscanf(vt->resp_buf, "HTTP/%*f %u %s", &resp, buf);
+
+	if (i == 2 && resp == vt->probe.exp_status)
+		vt->happy |= 1;
+}
+
+/*--------------------------------------------------------------------
+ * Record pokings...
+ */
+
+static void
+vbp_start_poke(struct vbp_target *vt)
+{
+	CHECK_OBJ_NOTNULL(vt, VBP_TARGET_MAGIC);
+
+#define BITMAP(n, c, t, b)	vt->n <<= 1;
+#include "tbl/backend_poll.h"
+#undef BITMAP
+
+	vt->last = 0;
+	vt->resp_buf[0] = '\0';
+}
+
+static void
+vbp_has_poked(struct vbp_target *vt)
+{
+	unsigned i, j;
+	uint64_t u;
+	const char *logmsg;
+	char bits[10];
+
+	CHECK_OBJ_NOTNULL(vt, VBP_TARGET_MAGIC);
+
+	/* Calculate exponential average */
+	if (vt->happy & 1) {
+		if (vt->rate < AVG_RATE)
+			vt->rate += 1.0;
+		vt->avg += (vt->last - vt->avg) / vt->rate;
+	}
+
+	i = 0;
+#define BITMAP(n, c, t, b)	bits[i++] = (vt->n & 1) ? c : '-';
+#include "tbl/backend_poll.h"
+#undef BITMAP
+	bits[i] = '\0';
+
+	u = vt->happy;
+	for (i = j = 0; i < vt->probe.window; i++) {
+		if (u & 1)
+			j++;
+		u >>= 1;
+	}
+	vt->good = j;
+
+	if (vt->good >= vt->probe.threshold) {
+		if (vt->backend->healthy)
+			logmsg = "Still healthy";
+		else
+			logmsg = "Back healthy";
+		vt->backend->healthy = 1;
+	} else {
+		if (vt->backend->healthy)
+			logmsg = "Went sick";
+		else
+			logmsg = "Still sick";
+		vt->backend->healthy = 0;
+	}
+	VSL(SLT_Backend_health, 0, "%s %s %s %u %u %u %.6f %.6f %s",
+	    vt->backend->vcl_name, logmsg, bits,
+	    vt->good, vt->probe.threshold, vt->probe.window,
+	    vt->last, vt->avg, vt->resp_buf);
+	vt->backend->vsc->happy = vt->happy;
+}
+
+/*--------------------------------------------------------------------
+ * Build request from probe spec
+ */
+
+static void
+vbp_build_req(struct vsb *vsb, const struct vbp_vcl *vcl)
+{
+
+	XXXAN(vsb);
+	XXXAN(vcl);
+	VSB_clear(vsb);
+	if(vcl->probe.request != NULL) {
+		VSB_cat(vsb, vcl->probe.request);
+	} else {
+		VSB_printf(vsb, "GET %s HTTP/1.1\r\n",
+		    vcl->probe.url != NULL ?  vcl->probe.url : "/");
+		if (vcl->hosthdr != NULL)
+			VSB_printf(vsb, "Host: %s\r\n", vcl->hosthdr);
+		VSB_printf(vsb, "Connection: close\r\n");
+		VSB_printf(vsb, "\r\n");
+	}
+	AZ(VSB_finish(vsb));
+}
+
+/*--------------------------------------------------------------------
+ * One thread per backend to be poked.
+ */
+
+static void *
+vbp_wrk_poll_backend(void *priv)
+{
+	struct vbp_target *vt;
+	struct vbp_vcl *vcl = NULL;
+
+	THR_SetName("backend poll");
+
+	CAST_OBJ_NOTNULL(vt, priv, VBP_TARGET_MAGIC);
+
+	while (!vt->stop) {
+		Lck_Lock(&vbp_mtx);
+		if (VTAILQ_FIRST(&vt->vcls) != vcl) {
+			vcl = VTAILQ_FIRST(&vt->vcls);
+			vbp_build_req(vt->vsb, vcl);
+			vt->probe = vcl->probe;
+		}
+		Lck_Unlock(&vbp_mtx);
+
+		vt->req = VSB_data(vt->vsb);
+		vt->req_len = VSB_len(vt->vsb);
+
+		vbp_start_poke(vt);
+		vbp_poke(vt);
+		vbp_has_poked(vt);
+
+		if (!vt->stop)
+			VTIM_sleep(vt->probe.interval);
+	}
+	return (NULL);
+}
+
+/*--------------------------------------------------------------------
+ * Cli functions
+ */
+
+void
+VBP_Summary(struct cli *cli, const struct vbp_target *vt)
+{
+
+	CHECK_OBJ_NOTNULL(vt, VBP_TARGET_MAGIC);
+	VCLI_Out(cli, "%d/%d", vt->good, vt->probe.window);
+}
+
+static void
+vbp_bitmap(struct cli *cli, char c, uint64_t map, const char *lbl)
+{
+	int i;
+	uint64_t u = (1ULL << 63);
+
+	for (i = 0; i < 64; i++) {
+		if (map & u)
+			VCLI_Out(cli, "%c", c);
+		else
+			VCLI_Out(cli, "-");
+		map <<= 1;
+	}
+	VCLI_Out(cli, " %s\n", lbl);
+}
+
+/*lint -e{506} constant value boolean */
+/*lint -e{774} constant value boolean */
+static void
+vbp_health_one(struct cli *cli, const struct vbp_target *vt)
+{
+
+	VCLI_Out(cli, "Backend %s is %s\n",
+	    vt->backend->vcl_name,
+	    vt->backend->healthy ? "Healthy" : "Sick");
+	VCLI_Out(cli, "Current states  good: %2u threshold: %2u window: %2u\n",
+	    vt->good, vt->probe.threshold, vt->probe.window);
+	VCLI_Out(cli, "Average responsetime of good probes: %.6f\n", vt->avg);
+	VCLI_Out(cli,
+	    "Oldest                       "
+	    "                             Newest\n");
+	VCLI_Out(cli,
+	    "============================="
+	    "===================================\n");
+
+#define BITMAP(n, c, t, b)					\
+		if ((vt->n != 0) || (b))			\
+			vbp_bitmap(cli, (c), vt->n, (t));
+#include "tbl/backend_poll.h"
+#undef BITMAP
+}
+
+static void
+vbp_health(struct cli *cli, const char * const *av, void *priv)
+{
+	struct vbp_target *vt;
+
+	ASSERT_CLI();
+	(void)av;
+	(void)priv;
+
+	VTAILQ_FOREACH(vt, &vbp_list, list)
+		vbp_health_one(cli, vt);
+}
+
+static struct cli_proto debug_cmds[] = {
+	{ "debug.health", "debug.health",
+		"\tDump backend health stuff\n",
+		0, 0, "d", vbp_health },
+	{ NULL }
+};
+
+/*--------------------------------------------------------------------
+ * A new VCL wants to probe this backend,
+ */
+
+static struct vbp_vcl *
+vbp_new_vcl(const struct vrt_backend_probe *p, const char *hosthdr)
+{
+	struct vbp_vcl *vcl;
+
+	ALLOC_OBJ(vcl, VBP_VCL_MAGIC);
+	XXXAN(vcl);
+	vcl->probep = p;
+	vcl->probe = *p;
+	vcl->hosthdr = hosthdr;
+
+	/*
+	 * Sanitize and insert defaults
+	 * XXX: we could make these defaults parameters
+	 */
+	if (vcl->probe.timeout == 0.0)
+		vcl->probe.timeout = 2.0;
+	if (vcl->probe.interval == 0.0)
+		vcl->probe.interval = 5.0;
+	if (vcl->probe.window == 0)
+		vcl->probe.window = 8;
+	if (vcl->probe.threshold == 0)
+		vcl->probe.threshold = 3;
+	if (vcl->probe.exp_status == 0)
+		vcl->probe.exp_status = 200;
+
+	if (vcl->probe.threshold == ~0U)
+		vcl->probe.initial = vcl->probe.threshold - 1;
+
+	if (vcl->probe.initial > vcl->probe.threshold)
+		vcl->probe.initial = vcl->probe.threshold;
+	return (vcl);
+}
+
+/*--------------------------------------------------------------------
+ * Insert/Remove/Use called from cache_backend.c
+ */
+
+void
+VBP_Insert(struct backend *b, const struct vrt_backend_probe *p, const char *hosthdr)
+{
+	struct vbp_target *vt;
+	struct vbp_vcl *vcl;
+	int startthread = 0;
+	unsigned u;
+
+	ASSERT_CLI();
+	AN(p);
+
+	if (b->probe == NULL) {
+		ALLOC_OBJ(vt, VBP_TARGET_MAGIC);
+		XXXAN(vt);
+		VTAILQ_INIT(&vt->vcls);
+		vt->backend = b;
+		vt->vsb = VSB_new_auto();
+		XXXAN(vt->vsb);
+		b->probe = vt;
+		startthread = 1;
+		VTAILQ_INSERT_TAIL(&vbp_list, vt, list);
+	} else {
+		vt = b->probe;
+	}
+
+	VTAILQ_FOREACH(vcl, &vt->vcls, list)
+		assert (vcl->probep != p);
+
+	vcl = vbp_new_vcl(p, hosthdr);
+	Lck_Lock(&vbp_mtx);
+	VTAILQ_INSERT_TAIL(&vt->vcls, vcl, list);
+	Lck_Unlock(&vbp_mtx);
+
+	if (startthread) {
+		for (u = 0; u < vcl->probe.initial; u++) {
+			vbp_start_poke(vt);
+			vt->happy |= 1;
+			vbp_has_poked(vt);
+		}
+		AZ(pthread_create(&vt->thread, NULL, vbp_wrk_poll_backend, vt));
+	}
+}
+
+void
+VBP_Use(const struct backend *b, const struct vrt_backend_probe *p)
+{
+	struct vbp_target *vt;
+	struct vbp_vcl *vcl;
+
+	ASSERT_CLI();
+	AN(p);
+	CHECK_OBJ_NOTNULL(b, BACKEND_MAGIC);
+	AN(b->probe);
+	vt = b->probe;
+
+	VTAILQ_FOREACH(vcl, &vt->vcls, list) {
+		if (vcl->probep != p)
+			continue;
+
+		Lck_Lock(&vbp_mtx);
+		VTAILQ_REMOVE(&vt->vcls, vcl, list);
+		VTAILQ_INSERT_HEAD(&vt->vcls, vcl, list);
+		Lck_Unlock(&vbp_mtx);
+		return;
+	}
+}
+
+void
+VBP_Remove(struct backend *b, struct vrt_backend_probe const *p)
+{
+	struct vbp_target *vt;
+	struct vbp_vcl *vcl;
+	void *ret;
+
+	ASSERT_CLI();
+	AN(p);
+	CHECK_OBJ_NOTNULL(b, BACKEND_MAGIC);
+	AN(b->probe);
+	vt = b->probe;
+
+	VTAILQ_FOREACH(vcl, &vt->vcls, list)
+		if (vcl->probep == p)
+			break;
+
+	AN(vcl);
+
+	Lck_Lock(&vbp_mtx);
+	VTAILQ_REMOVE(&vt->vcls, vcl, list);
+	Lck_Unlock(&vbp_mtx);
+
+	FREE_OBJ(vcl);
+
+	if (!VTAILQ_EMPTY(&vt->vcls))
+		return;
+
+	/* No more polling for this backend */
+
+	b->healthy = 1;
+
+	vt->stop = 1;
+	AZ(pthread_cancel(vt->thread));
+	AZ(pthread_join(vt->thread, &ret));
+
+	b->healthy = 1;
+
+	VTAILQ_REMOVE(&vbp_list, vt, list);
+	b->probe = NULL;
+	VSB_delete(vt->vsb);
+	FREE_OBJ(vt);
+}
+
+/*--------------------------------------------------------------------
+ * Initialize the backend probe subsystem
+ */
+
+void
+VBP_Init(void)
+{
+
+	Lck_New(&vbp_mtx, lck_vbp);
+	CLI_AddFuncs(debug_cmds);
+}
diff --git a/bin/varnishd/cache/cache_ban.c b/bin/varnishd/cache/cache_ban.c
new file mode 100644
index 0000000..768e631
--- /dev/null
+++ b/bin/varnishd/cache/cache_ban.c
@@ -0,0 +1,1108 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * Ban processing
+ *
+ * A ban consists of a number of conditions (or tests), all of which must be
+ * satisfied.  Here are some potential bans we could support:
+ *
+ *	req.url == "/foo"
+ *	req.url ~ ".iso" && obj.size > 10MB
+ *	req.http.host ~ "web1.com" && obj.set-cookie ~ "USER=29293"
+ *
+ * We make the "&&" mandatory from the start, leaving the syntax space
+ * for latter handling of "||" as well.
+ *
+ * Bans are compiled into bytestrings as follows:
+ *	8 bytes	- double: timestamp		XXX: Byteorder ?
+ *	4 bytes - be32: length
+ *	1 byte - flags: 0x01: BAN_F_REQ
+ *	N tests
+ * A test have this form:
+ *	1 byte - arg (see ban_vars.h col 3 "BAN_ARG_XXX")
+ *	(n bytes) - http header name, canonical encoding
+ *	lump - comparison arg
+ *	1 byte - operation (BAN_OPER_)
+ *	(lump) - compiled regexp
+ * A lump is:
+ *	4 bytes - be32: length
+ *	n bytes - content
+ *
+ * In a perfect world, we should vector through VRE to get to PCRE,
+ * but since we rely on PCRE's ability to encode the regexp into a
+ * byte string, that would be a little bit artificial, so this is
+ * the exception that confirmes the rule.
+ *
+ */
+
+#include "config.h"
+
+#include <pcre.h>
+#include <stdio.h>
+
+#include "cache.h"
+
+#include "hash/hash_slinger.h"
+#include "vcli.h"
+#include "vcli_priv.h"
+#include "vend.h"
+#include "vtim.h"
+
+struct ban {
+	unsigned		magic;
+#define BAN_MAGIC		0x700b08ea
+	VTAILQ_ENTRY(ban)	list;
+	unsigned		refcount;
+	unsigned		flags;
+#define BAN_F_GONE		(1 << 0)
+#define BAN_F_REQ		(1 << 2)
+#define BAN_F_LURK		(3 << 6)	/* ban-lurker-color */
+	VTAILQ_HEAD(,objcore)	objcore;
+	struct vsb		*vsb;
+	uint8_t			*spec;
+};
+
+#define LURK_SHIFT 6
+
+struct ban_test {
+	uint8_t			arg1;
+	const char		*arg1_spec;
+	uint8_t			oper;
+	const char		*arg2;
+	const void		*arg2_spec;
+};
+
+static VTAILQ_HEAD(banhead_s,ban) ban_head = VTAILQ_HEAD_INITIALIZER(ban_head);
+static struct lock ban_mtx;
+static struct ban *ban_magic;
+static pthread_t ban_thread;
+static struct ban * volatile ban_start;
+static bgthread_t ban_lurker;
+
+/*--------------------------------------------------------------------
+ * BAN string magic markers
+ */
+
+#define	BAN_OPER_EQ	0x10
+#define	BAN_OPER_NEQ	0x11
+#define	BAN_OPER_MATCH	0x12
+#define	BAN_OPER_NMATCH	0x13
+
+#define BAN_ARG_URL	0x18
+#define BAN_ARG_REQHTTP	0x19
+#define BAN_ARG_OBJHTTP	0x1a
+
+/*--------------------------------------------------------------------
+ * Variables we can purge on
+ */
+
+static const struct pvar {
+	const char		*name;
+	unsigned		flag;
+	uint8_t			tag;
+} pvars[] = {
+#define PVAR(a, b, c)	{ (a), (b), (c) },
+#include "tbl/ban_vars.h"
+#undef PVAR
+	{ 0, 0, 0}
+};
+
+/*--------------------------------------------------------------------
+ * Storage handling of bans
+ */
+
+struct ban *
+BAN_New(void)
+{
+	struct ban *b;
+
+	ALLOC_OBJ(b, BAN_MAGIC);
+	if (b == NULL)
+		return (b);
+	b->vsb = VSB_new_auto();
+	if (b->vsb == NULL) {
+		FREE_OBJ(b);
+		return (NULL);
+	}
+	VTAILQ_INIT(&b->objcore);
+	return (b);
+}
+
+void
+BAN_Free(struct ban *b)
+{
+
+	CHECK_OBJ_NOTNULL(b, BAN_MAGIC);
+	AZ(b->refcount);
+	assert(VTAILQ_EMPTY(&b->objcore));
+
+	if (b->vsb != NULL)
+		VSB_delete(b->vsb);
+	if (b->spec != NULL)
+		free(b->spec);
+	FREE_OBJ(b);
+}
+
+/*--------------------------------------------------------------------
+ * Get & Release a tail reference, used to hold the list stable for
+ * traversals etc.
+ */
+
+struct ban *
+BAN_TailRef(void)
+{
+	struct ban *b;
+
+	ASSERT_CLI();
+	Lck_Lock(&ban_mtx);
+	b = VTAILQ_LAST(&ban_head, banhead_s);
+	AN(b);
+	b->refcount++;
+	Lck_Unlock(&ban_mtx);
+	return (b);
+}
+
+void
+BAN_TailDeref(struct ban **bb)
+{
+	struct ban *b;
+
+	b = *bb;
+	*bb = NULL;
+	Lck_Lock(&ban_mtx);
+	b->refcount--;
+	Lck_Unlock(&ban_mtx);
+}
+
+/*--------------------------------------------------------------------
+ * Extract time and length from ban-spec
+ */
+
+static double
+ban_time(const uint8_t *banspec)
+{
+	double t;
+
+	assert(sizeof t == 8);
+	memcpy(&t, banspec, sizeof t);
+	return (t);
+}
+
+static unsigned
+ban_len(const uint8_t *banspec)
+{
+	unsigned u;
+
+	u = vbe32dec(banspec + 8);
+	return (u);
+}
+
+/*--------------------------------------------------------------------
+ * Access a lump of bytes in a ban test spec
+ */
+
+static void
+ban_add_lump(const struct ban *b, const void *p, uint32_t len)
+{
+	uint8_t buf[sizeof len];
+
+	vbe32enc(buf, len);
+	VSB_bcat(b->vsb, buf, sizeof buf);
+	VSB_bcat(b->vsb, p, len);
+}
+
+static const void *
+ban_get_lump(const uint8_t **bs)
+{
+	const void *r;
+	unsigned ln;
+
+	ln = vbe32dec(*bs);
+	*bs += 4;
+	r = (const void*)*bs;
+	*bs += ln;
+	return (r);
+}
+
+/*--------------------------------------------------------------------
+ * Pick a test apart from a spec string
+ */
+
+static void
+ban_iter(const uint8_t **bs, struct ban_test *bt)
+{
+
+	memset(bt, 0, sizeof *bt);
+	bt->arg1 = *(*bs)++;
+	if (bt->arg1 == BAN_ARG_REQHTTP || bt->arg1 == BAN_ARG_OBJHTTP) {
+		bt->arg1_spec = (const char *)*bs;
+		(*bs) += (*bs)[0] + 2;
+	}
+	bt->arg2 = ban_get_lump(bs);
+	bt->oper = *(*bs)++;
+	if (bt->oper == BAN_OPER_MATCH || bt->oper == BAN_OPER_NMATCH)
+		bt->arg2_spec = ban_get_lump(bs);
+}
+
+/*--------------------------------------------------------------------
+ * Parse and add a http argument specification
+ * Output something which HTTP_GetHdr understands
+ */
+
+static void
+ban_parse_http(const struct ban *b, const char *a1)
+{
+	int l;
+
+	l = strlen(a1) + 1;
+	assert(l <= 127);
+	VSB_putc(b->vsb, (char)l);
+	VSB_cat(b->vsb, a1);
+	VSB_putc(b->vsb, ':');
+	VSB_putc(b->vsb, '\0');
+}
+
+/*--------------------------------------------------------------------
+ * Parse and add a ban test specification
+ */
+
+static int
+ban_parse_regexp(struct cli *cli, const struct ban *b, const char *a3)
+{
+	const char *error;
+	int erroroffset, rc;
+	size_t sz;
+	pcre *re;
+
+	re = pcre_compile(a3, 0, &error, &erroroffset, NULL);
+	if (re == NULL) {
+		VSL(SLT_Debug, 0, "REGEX: <%s>", error);
+		VCLI_Out(cli, "%s", error);
+		VCLI_SetResult(cli, CLIS_PARAM);
+		return (-1);
+	}
+	rc = pcre_fullinfo(re, NULL, PCRE_INFO_SIZE, &sz);
+	AZ(rc);
+	ban_add_lump(b, re, sz);
+	return (0);
+}
+
+/*--------------------------------------------------------------------
+ * Add a (and'ed) test-condition to a ban
+ */
+
+int
+BAN_AddTest(struct cli *cli, struct ban *b, const char *a1, const char *a2,
+    const char *a3)
+{
+	const struct pvar *pv;
+	int i;
+
+	CHECK_OBJ_NOTNULL(b, BAN_MAGIC);
+
+	for (pv = pvars; pv->name != NULL; pv++)
+		if (!strncmp(a1, pv->name, strlen(pv->name)))
+			break;
+	if (pv->name == NULL) {
+		VCLI_Out(cli, "unknown or unsupported field \"%s\"", a1);
+		VCLI_SetResult(cli, CLIS_PARAM);
+		return (-1);
+	}
+
+	if (pv->flag & PVAR_REQ)
+		b->flags |= BAN_F_REQ;
+
+	VSB_putc(b->vsb, pv->tag);
+	if (pv->flag & PVAR_HTTP)
+		ban_parse_http(b, a1 + strlen(pv->name));
+
+	ban_add_lump(b, a3, strlen(a3) + 1);
+	if (!strcmp(a2, "~")) {
+		VSB_putc(b->vsb, BAN_OPER_MATCH);
+		i = ban_parse_regexp(cli, b, a3);
+		if (i)
+			return (i);
+	} else if (!strcmp(a2, "!~")) {
+		VSB_putc(b->vsb, BAN_OPER_NMATCH);
+		i = ban_parse_regexp(cli, b, a3);
+		if (i)
+			return (i);
+	} else if (!strcmp(a2, "==")) {
+		VSB_putc(b->vsb, BAN_OPER_EQ);
+	} else if (!strcmp(a2, "!=")) {
+		VSB_putc(b->vsb, BAN_OPER_NEQ);
+	} else {
+		VCLI_Out(cli,
+		    "expected conditional (~, !~, == or !=) got \"%s\"", a2);
+		VCLI_SetResult(cli, CLIS_PARAM);
+		return (-1);
+	}
+	return (0);
+}
+
+/*--------------------------------------------------------------------
+ * We maintain ban_start as a pointer to the first element of the list
+ * as a separate variable from the VTAILQ, to avoid depending on the
+ * internals of the VTAILQ macros.  We tacitly assume that a pointer
+ * write is always atomic in doing so.
+ */
+
+void
+BAN_Insert(struct ban *b)
+{
+	struct ban  *bi, *be;
+	ssize_t ln;
+	double t0;
+
+	CHECK_OBJ_NOTNULL(b, BAN_MAGIC);
+
+	AZ(VSB_finish(b->vsb));
+	ln = VSB_len(b->vsb);
+	assert(ln >= 0);
+
+	b->spec = malloc(ln + 13L);	/* XXX */
+	XXXAN(b->spec);
+
+	t0 = VTIM_real();
+	memcpy(b->spec, &t0, sizeof t0);
+	b->spec[12] = (b->flags & BAN_F_REQ) ? 1 : 0;
+	memcpy(b->spec + 13, VSB_data(b->vsb), ln);
+	ln += 13;
+	vbe32enc(b->spec + 8, ln);
+
+	VSB_delete(b->vsb);
+	b->vsb = NULL;
+
+	Lck_Lock(&ban_mtx);
+	VTAILQ_INSERT_HEAD(&ban_head, b, list);
+	ban_start = b;
+	VSC_C_main->bans++;
+	VSC_C_main->bans_added++;
+	if (b->flags & BAN_F_REQ)
+		VSC_C_main->bans_req++;
+
+	be = VTAILQ_LAST(&ban_head, banhead_s);
+	if (cache_param->ban_dups && be != b)
+		be->refcount++;
+	else
+		be = NULL;
+
+	SMP_NewBan(b->spec, ln);
+	Lck_Unlock(&ban_mtx);
+
+	if (be == NULL)
+		return;
+
+	/* Hunt down duplicates, and mark them as gone */
+	bi = b;
+	Lck_Lock(&ban_mtx);
+	while(bi != be) {
+		bi = VTAILQ_NEXT(bi, list);
+		if (bi->flags & BAN_F_GONE)
+			continue;
+		/* Safe because the length is part of the fixed size hdr */
+		if (memcmp(b->spec + 8, bi->spec + 8, ln - 8))
+			continue;
+		bi->flags |= BAN_F_GONE;
+		VSC_C_main->bans_gone++;
+		VSC_C_main->bans_dups++;
+	}
+	be->refcount--;
+	Lck_Unlock(&ban_mtx);
+}
+
+/*--------------------------------------------------------------------
+ * A new object is created, grab a reference to the newest ban
+ */
+
+void
+BAN_NewObjCore(struct objcore *oc)
+{
+
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	AZ(oc->ban);
+	Lck_Lock(&ban_mtx);
+	oc->ban = ban_start;
+	ban_start->refcount++;
+	VTAILQ_INSERT_TAIL(&ban_start->objcore, oc, ban_list);
+	Lck_Unlock(&ban_mtx);
+}
+
+/*--------------------------------------------------------------------
+ * An object is destroyed, release its ban reference
+ */
+
+void
+BAN_DestroyObj(struct objcore *oc)
+{
+
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	if (oc->ban == NULL)
+		return;
+	CHECK_OBJ_NOTNULL(oc->ban, BAN_MAGIC);
+	Lck_Lock(&ban_mtx);
+	assert(oc->ban->refcount > 0);
+	oc->ban->refcount--;
+	VTAILQ_REMOVE(&oc->ban->objcore, oc, ban_list);
+	oc->ban = NULL;
+	Lck_Unlock(&ban_mtx);
+}
+
+/*--------------------------------------------------------------------
+ * Find and/or Grab a reference to an objects ban based on timestamp
+ * Assume we hold a TailRef, so list traversal is safe.
+ */
+
+struct ban *
+BAN_RefBan(struct objcore *oc, double t0, const struct ban *tail)
+{
+	struct ban *b;
+	double t1 = 0;
+
+	VTAILQ_FOREACH(b, &ban_head, list) {
+		t1 = ban_time(b->spec);
+		if (t1 <= t0)
+			break;
+		if (b == tail)
+			break;
+	}
+	AN(b);
+	assert(t1 == t0);
+	Lck_Lock(&ban_mtx);
+	b->refcount++;
+	VTAILQ_INSERT_TAIL(&b->objcore, oc, ban_list);
+	Lck_Unlock(&ban_mtx);
+	return (b);
+}
+
+/*--------------------------------------------------------------------
+ * Put a skeleton ban in the list, unless there is an identical,
+ * time & condition, ban already in place.
+ *
+ * If a newer ban has same condition, mark the new ban GONE.
+ * mark any older bans, with the same condition, GONE as well.
+ */
+
+void
+BAN_Reload(const uint8_t *ban, unsigned len)
+{
+	struct ban *b, *b2;
+	int gone = 0;
+	double t0, t1, t2 = 9e99;
+
+	ASSERT_CLI();
+
+	t0 = ban_time(ban);
+	assert(len == ban_len(ban));
+
+	Lck_Lock(&ban_mtx);
+
+	VTAILQ_FOREACH(b, &ban_head, list) {
+		t1 = ban_time(b->spec);
+		assert (t1 < t2);
+		t2 = t1;
+		if (t1 == t0) {
+			Lck_Unlock(&ban_mtx);
+			return;
+		}
+		if (t1 < t0)
+			break;
+		if (!memcmp(b->spec + 8, ban + 8, len - 8)) {
+			gone |= BAN_F_GONE;
+			VSC_C_main->bans_dups++;
+			VSC_C_main->bans_gone++;
+		}
+	}
+
+	VSC_C_main->bans++;
+	VSC_C_main->bans_added++;
+
+	b2 = BAN_New();
+	AN(b2);
+	b2->spec = malloc(len);
+	AN(b2->spec);
+	memcpy(b2->spec, ban, len);
+	b2->flags |= gone;
+	if (ban[12])
+		b2->flags |= BAN_F_REQ;
+	if (b == NULL)
+		VTAILQ_INSERT_TAIL(&ban_head, b2, list);
+	else
+		VTAILQ_INSERT_BEFORE(b, b2, list);
+
+	/* Hunt down older duplicates */
+	for (b = VTAILQ_NEXT(b2, list); b != NULL; b = VTAILQ_NEXT(b, list)) {
+		if (b->flags & BAN_F_GONE)
+			continue;
+		if (!memcmp(b->spec + 8, ban + 8, len - 8)) {
+			b->flags |= BAN_F_GONE;
+			VSC_C_main->bans_dups++;
+			VSC_C_main->bans_gone++;
+		}
+	}
+	Lck_Unlock(&ban_mtx);
+}
+
+/*--------------------------------------------------------------------
+ * Get a bans timestamp
+ */
+
+double
+BAN_Time(const struct ban *b)
+{
+
+	if (b == NULL)
+		return (0.0);
+
+	CHECK_OBJ_NOTNULL(b, BAN_MAGIC);
+	return (ban_time(b->spec));
+}
+
+/*--------------------------------------------------------------------
+ * All silos have read their bans, ready for action
+ */
+
+void
+BAN_Compile(void)
+{
+
+	ASSERT_CLI();
+
+	SMP_NewBan(ban_magic->spec, ban_len(ban_magic->spec));
+	ban_start = VTAILQ_FIRST(&ban_head);
+	WRK_BgThread(&ban_thread, "ban-lurker", ban_lurker, NULL);
+}
+
+/*--------------------------------------------------------------------
+ * Evaluate ban-spec
+ */
+
+static int
+ban_evaluate(const uint8_t *bs, const struct http *objhttp,
+    const struct http *reqhttp, unsigned *tests)
+{
+	struct ban_test bt;
+	const uint8_t *be;
+	char *arg1;
+
+	be = bs + ban_len(bs);
+	bs += 13;
+	while (bs < be) {
+		(*tests)++;
+		ban_iter(&bs, &bt);
+		arg1 = NULL;
+		switch (bt.arg1) {
+		case BAN_ARG_URL:
+			arg1 = reqhttp->hd[HTTP_HDR_URL].b;
+			break;
+		case BAN_ARG_REQHTTP:
+			(void)http_GetHdr(reqhttp, bt.arg1_spec, &arg1);
+			break;
+		case BAN_ARG_OBJHTTP:
+			(void)http_GetHdr(objhttp, bt.arg1_spec, &arg1);
+			break;
+		default:
+			INCOMPL();
+		}
+
+		switch (bt.oper) {
+		case BAN_OPER_EQ:
+			if (arg1 == NULL || strcmp(arg1, bt.arg2))
+				return (0);
+			break;
+		case BAN_OPER_NEQ:
+			if (arg1 != NULL && !strcmp(arg1, bt.arg2))
+				return (0);
+			break;
+		case BAN_OPER_MATCH:
+			if (arg1 == NULL ||
+			    pcre_exec(bt.arg2_spec, NULL, arg1, strlen(arg1),
+			    0, 0, NULL, 0) < 0)
+				return (0);
+			break;
+		case BAN_OPER_NMATCH:
+			if (arg1 != NULL &&
+			    pcre_exec(bt.arg2_spec, NULL, arg1, strlen(arg1),
+			    0, 0, NULL, 0) >= 0)
+				return (0);
+			break;
+		default:
+			INCOMPL();
+		}
+	}
+	return (1);
+}
+
+/*--------------------------------------------------------------------
+ * Check an object against all applicable bans
+ *
+ * Return:
+ *	-1 not all bans checked, but none of the checked matched
+ *		Only if !has_req
+ *	0 No bans matched, object moved to ban_start.
+ *	1 Ban matched, object removed from ban list.
+ */
+
+static int
+ban_check_object(struct object *o, const struct sess *sp, int has_req)
+{
+	struct ban *b;
+	struct objcore *oc;
+	struct ban * volatile b0;
+	unsigned tests, skipped;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
+	oc = o->objcore;
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	CHECK_OBJ_NOTNULL(oc->ban, BAN_MAGIC);
+
+	b0 = ban_start;
+	CHECK_OBJ_NOTNULL(b0, BAN_MAGIC);
+
+	if (b0 == oc->ban)
+		return (0);
+
+	/*
+	 * This loop is safe without locks, because we know we hold
+	 * a refcount on a ban somewhere in the list and we do not
+	 * inspect the list past that ban.
+	 */
+	tests = 0;
+	skipped = 0;
+	for (b = b0; b != oc->ban; b = VTAILQ_NEXT(b, list)) {
+		CHECK_OBJ_NOTNULL(b, BAN_MAGIC);
+		if (b->flags & BAN_F_GONE)
+			continue;
+		if ((b->flags & BAN_F_LURK) &&
+		    (b->flags & BAN_F_LURK) == (oc->flags & OC_F_LURK)) {
+			AZ(b->flags & BAN_F_REQ);
+			/* Lurker already tested this */
+			continue;
+		}
+		if (!has_req && (b->flags & BAN_F_REQ)) {
+			/*
+			 * We cannot test this one, but there might
+			 * be other bans that match, so we soldier on
+			 */
+			skipped++;
+		} else if (ban_evaluate(b->spec, o->http, sp->http, &tests))
+			break;
+	}
+
+	Lck_Lock(&ban_mtx);
+	VSC_C_main->bans_tested++;
+	VSC_C_main->bans_tests_tested += tests;
+
+	if (b == oc->ban && skipped > 0) {
+		AZ(has_req);
+		Lck_Unlock(&ban_mtx);
+		/*
+		 * Not banned, but some tests were skipped, so we cannot know
+		 * for certain that it cannot be, so we just have to give up.
+		 */
+		return (-1);
+	}
+
+	oc->ban->refcount--;
+	VTAILQ_REMOVE(&oc->ban->objcore, oc, ban_list);
+	if (b == oc->ban) {	/* not banned */
+		b->flags &= ~BAN_F_LURK;
+		VTAILQ_INSERT_TAIL(&b0->objcore, oc, ban_list);
+		b0->refcount++;
+	}
+	Lck_Unlock(&ban_mtx);
+
+	if (b == oc->ban) {	/* not banned */
+		oc->ban = b0;
+		oc_updatemeta(oc);
+		return (0);
+	} else {
+		EXP_Clr(&o->exp);
+		oc->ban = NULL;
+		oc_updatemeta(oc);
+		/* BAN also changed, but that is not important any more */
+		WSP(sp, SLT_ExpBan, "%u was banned", o->xid);
+		EXP_Rearm(o);
+		return (1);
+	}
+}
+
+int
+BAN_CheckObject(struct object *o, const struct sess *sp)
+{
+
+	return (ban_check_object(o, sp, 1) > 0);
+}
+
+static struct ban *
+ban_CheckLast(void)
+{
+	struct ban *b;
+
+	Lck_AssertHeld(&ban_mtx);
+	b = VTAILQ_LAST(&ban_head, banhead_s);
+	if (b != VTAILQ_FIRST(&ban_head) && b->refcount == 0) {
+		if (b->flags & BAN_F_GONE)
+			VSC_C_main->bans_gone--;
+		if (b->flags & BAN_F_REQ)
+			VSC_C_main->bans_req--;
+		VSC_C_main->bans--;
+		VSC_C_main->bans_deleted++;
+		VTAILQ_REMOVE(&ban_head, b, list);
+	} else {
+		b = NULL;
+	}
+	return (b);
+}
+
+/*--------------------------------------------------------------------
+ * Ban lurker thread
+ */
+
+static int
+ban_lurker_work(const struct sess *sp, unsigned pass)
+{
+	struct ban *b, *b0, *b2;
+	struct objhead *oh;
+	struct objcore *oc, *oc2;
+	struct object *o;
+	int i;
+
+	AN(pass & BAN_F_LURK);
+	AZ(pass & ~BAN_F_LURK);
+
+	/* First route the last ban(s) */
+	do {
+		Lck_Lock(&ban_mtx);
+		b2 = ban_CheckLast();
+		Lck_Unlock(&ban_mtx);
+		if (b2 != NULL)
+			BAN_Free(b2);
+	} while (b2 != NULL);
+
+	/*
+	 * Find out if we have any bans we can do something about
+	 * If we find any, tag them with our pass number.
+	 */
+	i = 0;
+	b0 = NULL;
+	VTAILQ_FOREACH(b, &ban_head, list) {
+		if (b->flags & BAN_F_GONE)
+			continue;
+		if (b->flags & BAN_F_REQ)
+			continue;
+		if (b == VTAILQ_LAST(&ban_head, banhead_s))
+			continue;
+		if (b0 == NULL)
+			b0 = b;
+		i++;
+		b->flags &= ~BAN_F_LURK;
+		b->flags |= pass;
+	}
+	if (cache_param->diag_bitmap & 0x80000)
+		VSL(SLT_Debug, 0, "lurker: %d actionable bans", i);
+	if (i == 0)
+		return (0);
+
+	VTAILQ_FOREACH_REVERSE(b, &ban_head, banhead_s, list) {
+		if (cache_param->diag_bitmap & 0x80000)
+			VSL(SLT_Debug, 0, "lurker doing %f %d",
+			    ban_time(b->spec), b->refcount);
+		while (1) {
+			Lck_Lock(&ban_mtx);
+			oc = VTAILQ_FIRST(&b->objcore);
+			if (oc == NULL)
+				break;
+			CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+			if (cache_param->diag_bitmap & 0x80000)
+				VSL(SLT_Debug, 0, "test: %p %d %d",
+				    oc, oc->flags & OC_F_LURK, pass);
+			if ((oc->flags & OC_F_LURK) == pass)
+				break;
+			oh = oc->objhead;
+			CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC);
+			if (Lck_Trylock(&oh->mtx)) {
+				Lck_Unlock(&ban_mtx);
+				VTIM_sleep(cache_param->ban_lurker_sleep);
+				continue;
+			}
+			/*
+			 * See if the objcore is still on the objhead since
+			 * we race against HSH_Deref() which comes in the
+			 * opposite locking order.
+			 */
+			VTAILQ_FOREACH(oc2, &oh->objcs, list)
+				if (oc == oc2)
+					break;
+			if (oc2 == NULL) {
+				Lck_Unlock(&oh->mtx);
+				Lck_Unlock(&ban_mtx);
+				VTIM_sleep(cache_param->ban_lurker_sleep);
+				continue;
+			}
+			/*
+			 * Grab a reference to the OC and we can let go of
+			 * the BAN mutex
+			 */
+			AN(oc->refcnt);
+			oc->refcnt++;
+			oc->flags &= ~OC_F_LURK;
+			Lck_Unlock(&ban_mtx);
+			/*
+			 * Get the object and check it against all relevant bans
+			 */
+			o = oc_getobj(sp->wrk, oc);
+			i = ban_check_object(o, sp, 0);
+			if (cache_param->diag_bitmap & 0x80000)
+				VSL(SLT_Debug, 0, "lurker got: %p %d",
+				    oc, i);
+			if (i == -1) {
+				/* Not banned, not moved */
+				oc->flags |= pass;
+				Lck_Lock(&ban_mtx);
+				VTAILQ_REMOVE(&b->objcore, oc, ban_list);
+				VTAILQ_INSERT_TAIL(&b->objcore, oc, ban_list);
+				Lck_Unlock(&ban_mtx);
+			}
+			Lck_Unlock(&oh->mtx);
+			if (cache_param->diag_bitmap & 0x80000)
+				VSL(SLT_Debug, 0, "lurker done: %p %d %d",
+				    oc, oc->flags & OC_F_LURK, pass);
+			(void)HSH_Deref(sp->wrk, NULL, &o);
+			VTIM_sleep(cache_param->ban_lurker_sleep);
+		}
+		Lck_AssertHeld(&ban_mtx);
+		if (!(b->flags & BAN_F_REQ)) {
+			if (!(b->flags & BAN_F_GONE)) {
+				b->flags |= BAN_F_GONE;
+				VSC_C_main->bans_gone++;
+			}
+			if (cache_param->diag_bitmap & 0x80000)
+				VSL(SLT_Debug, 0, "lurker BAN %f now gone",
+				    ban_time(b->spec));
+		}
+		Lck_Unlock(&ban_mtx);
+		VTIM_sleep(cache_param->ban_lurker_sleep);
+		if (b == b0)
+			break;
+	}
+	return (1);
+}
+
+static void * __match_proto__(bgthread_t)
+ban_lurker(struct sess *sp, void *priv)
+{
+	struct ban *bf;
+	unsigned pass = (1 << LURK_SHIFT);
+
+	int i = 0;
+	(void)priv;
+	while (1) {
+
+		while (cache_param->ban_lurker_sleep == 0.0) {
+			/*
+			 * Ban-lurker is disabled:
+			 * Clean the last ban, if possible, and sleep
+			 */
+			Lck_Lock(&ban_mtx);
+			bf = ban_CheckLast();
+			Lck_Unlock(&ban_mtx);
+			if (bf != NULL)
+				BAN_Free(bf);
+			else
+				VTIM_sleep(1.0);
+		}
+
+		i = ban_lurker_work(sp, pass);
+		WSL_Flush(sp->wrk, 0);
+		WRK_SumStat(sp->wrk);
+		if (i) {
+			pass += (1 << LURK_SHIFT);
+			pass &= BAN_F_LURK;
+			if (pass == 0)
+				pass += (1 << LURK_SHIFT);
+			VTIM_sleep(cache_param->ban_lurker_sleep);
+		} else {
+			VTIM_sleep(1.0);
+		}
+	}
+	NEEDLESS_RETURN(NULL);
+}
+
+/*--------------------------------------------------------------------
+ * CLI functions to add bans
+ */
+
+static void
+ccf_ban(struct cli *cli, const char * const *av, void *priv)
+{
+	int narg, i;
+	struct ban *b;
+
+	(void)priv;
+
+	/* First do some cheap checks on the arguments */
+	for (narg = 0; av[narg + 2] != NULL; narg++)
+		continue;
+	if ((narg % 4) != 3) {
+		VCLI_Out(cli, "Wrong number of arguments");
+		VCLI_SetResult(cli, CLIS_PARAM);
+		return;
+	}
+	for (i = 3; i < narg; i += 4) {
+		if (strcmp(av[i + 2], "&&")) {
+			VCLI_Out(cli, "Found \"%s\" expected &&", av[i + 2]);
+			VCLI_SetResult(cli, CLIS_PARAM);
+			return;
+		}
+	}
+
+	b = BAN_New();
+	if (b == NULL) {
+		VCLI_Out(cli, "Out of Memory");
+		VCLI_SetResult(cli, CLIS_CANT);
+		return;
+	}
+	for (i = 0; i < narg; i += 4)
+		if (BAN_AddTest(cli, b, av[i + 2], av[i + 3], av[i + 4])) {
+			BAN_Free(b);
+			return;
+		}
+	BAN_Insert(b);
+}
+
+static void
+ccf_ban_url(struct cli *cli, const char * const *av, void *priv)
+{
+	const char *aav[6];
+
+	(void)priv;
+	aav[0] = NULL;
+	aav[1] = "ban";
+	aav[2] = "req.url";
+	aav[3] = "~";
+	aav[4] = av[2];
+	aav[5] = NULL;
+	ccf_ban(cli, aav, priv);
+}
+
+static void
+ban_render(struct cli *cli, const uint8_t *bs)
+{
+	struct ban_test bt;
+	const uint8_t *be;
+
+	be = bs + ban_len(bs);
+	bs += 13;
+	while (bs < be) {
+		ban_iter(&bs, &bt);
+		switch (bt.arg1) {
+		case BAN_ARG_URL:
+			VCLI_Out(cli, "req.url");
+			break;
+		case BAN_ARG_REQHTTP:
+			VCLI_Out(cli, "req.http.%.*s",
+			    bt.arg1_spec[0] - 1, bt.arg1_spec + 1);
+			break;
+		case BAN_ARG_OBJHTTP:
+			VCLI_Out(cli, "obj.http.%.*s",
+			    bt.arg1_spec[0] - 1, bt.arg1_spec + 1);
+			break;
+		default:
+			INCOMPL();
+		}
+		switch (bt.oper) {
+		case BAN_OPER_EQ:	VCLI_Out(cli, " == "); break;
+		case BAN_OPER_NEQ:	VCLI_Out(cli, " != "); break;
+		case BAN_OPER_MATCH:	VCLI_Out(cli, " ~ "); break;
+		case BAN_OPER_NMATCH:	VCLI_Out(cli, " !~ "); break;
+		default:
+			INCOMPL();
+		}
+		VCLI_Out(cli, "%s", bt.arg2);
+		if (bs < be)
+			VCLI_Out(cli, " && ");
+	}
+}
+
+static void
+ccf_ban_list(struct cli *cli, const char * const *av, void *priv)
+{
+	struct ban *b, *bl;
+
+	(void)av;
+	(void)priv;
+
+	/* Get a reference so we are safe to traverse the list */
+	bl = BAN_TailRef();
+
+	VCLI_Out(cli, "Present bans:\n");
+	VTAILQ_FOREACH(b, &ban_head, list) {
+		if (b == bl && !(cache_param->diag_bitmap & 0x80000))
+			break;
+		VCLI_Out(cli, "%10.6f %5u%s\t", ban_time(b->spec),
+		    bl == b ? b->refcount - 1 : b->refcount,
+		    b->flags & BAN_F_GONE ? "G" : " ");
+		ban_render(cli, b->spec);
+		VCLI_Out(cli, "\n");
+		if (cache_param->diag_bitmap & 0x80000) {
+			Lck_Lock(&ban_mtx);
+			struct objcore *oc;
+			VTAILQ_FOREACH(oc, &b->objcore, ban_list)
+				VCLI_Out(cli, "     %p\n", oc);
+			Lck_Unlock(&ban_mtx);
+		}
+	}
+
+	BAN_TailDeref(&bl);
+}
+
+static struct cli_proto ban_cmds[] = {
+	{ CLI_BAN_URL,				"", ccf_ban_url },
+	{ CLI_BAN,				"", ccf_ban },
+	{ CLI_BAN_LIST,				"", ccf_ban_list },
+	{ NULL }
+};
+
+void
+BAN_Init(void)
+{
+
+	Lck_New(&ban_mtx, lck_ban);
+	CLI_AddFuncs(ban_cmds);
+	assert(BAN_F_LURK == OC_F_LURK);
+	AN((1 << LURK_SHIFT) & BAN_F_LURK);
+	AN((2 << LURK_SHIFT) & BAN_F_LURK);
+
+	ban_magic = BAN_New();
+	AN(ban_magic);
+	ban_magic->flags |= BAN_F_GONE;
+	VSC_C_main->bans_gone++;
+	BAN_Insert(ban_magic);
+}
diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c
new file mode 100644
index 0000000..e42fac8
--- /dev/null
+++ b/bin/varnishd/cache/cache_center.c
@@ -0,0 +1,1691 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * This file contains the central state machine for pushing requests.
+ *
+ * We cannot just use direct calls because it is possible to kick a
+ * request back to the lookup stage (usually after a rewrite).  The
+ * state engine also allows us to break the processing up into some
+ * logical chunks which improves readability a little bit.
+ *
+ * Since the states are rather nasty in detail, I have decided to embedd
+ * a dot(1) graph in the source code comments.  So to see the big picture,
+ * extract the DOT lines and run though dot(1), for instance with the
+ * command:
+ *	sed -n '/^DOT/s///p' cache_center.c | dot -Tps > /tmp/_.ps
+ */
+
+/*
+DOT digraph vcl_center {
+xDOT	page="8.2,11.5"
+DOT	size="7.2,10.5"
+DOT	margin="0.5"
+DOT	center="1"
+DOT acceptor [
+DOT	shape=hexagon
+DOT	label="Request received"
+DOT ]
+DOT ERROR [shape=plaintext]
+DOT RESTART [shape=plaintext]
+DOT acceptor -> start [style=bold,color=green]
+ */
+
+#include "config.h"
+
+#include <math.h>
+#include <poll.h>
+#include <stdio.h>
+#include <stdlib.h>
+
+#include "cache.h"
+
+#include "hash/hash_slinger.h"
+#include "vcl.h"
+#include "vcli_priv.h"
+#include "vsha256.h"
+#include "vtcp.h"
+#include "vtim.h"
+
+#ifndef HAVE_SRANDOMDEV
+#include "compat/srandomdev.h"
+#endif
+
+static unsigned xids;
+
+/*--------------------------------------------------------------------
+ * WAIT
+ * Wait (briefly) until we have a full request in our htc.
+ */
+
+static int
+cnt_wait(struct sess *sp)
+{
+	int i;
+	struct pollfd pfd[1];
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	AZ(sp->vcl);
+	AZ(sp->obj);
+	assert(sp->xid == 0);
+
+	i = HTC_Complete(sp->htc);
+	if (i == 0 && cache_param->session_linger > 0) {
+		pfd[0].fd = sp->fd;
+		pfd[0].events = POLLIN;
+		pfd[0].revents = 0;
+		i = poll(pfd, 1, cache_param->session_linger);
+		if (i)
+			i = HTC_Rx(sp->htc);
+	}
+	if (i == 0) {
+		WSP(sp, SLT_Debug, "herding");
+		sp->wrk->stats.sess_herd++;
+		SES_Charge(sp);
+		sp->wrk = NULL;
+		Pool_Wait(sp);
+		return (1);
+	}
+	if (i == 1) {
+		sp->step = STP_START;
+		return (0);
+	}
+	if (i == -2) {
+		SES_Close(sp, "overflow");
+		return (0);
+	}
+	if (i == -1 && Tlen(sp->htc->rxbuf) == 0 &&
+	    (errno == 0 || errno == ECONNRESET))
+		SES_Close(sp, "EOF");
+	else
+		SES_Close(sp, "error");
+	sp->step = STP_DONE;
+	return (0);
+}
+
+/*--------------------------------------------------------------------
+ * We have a refcounted object on the session, now deliver it.
+ *
+DOT subgraph xcluster_prepresp {
+DOT	prepresp [
+DOT		shape=ellipse
+DOT		label="Filter obj.->resp."
+DOT	]
+DOT	vcl_deliver [
+DOT		shape=record
+DOT		label="vcl_deliver()|resp."
+DOT	]
+DOT	prepresp -> vcl_deliver [style=bold,color=green]
+DOT	prepresp -> vcl_deliver [style=bold,color=cyan]
+DOT	prepresp -> vcl_deliver [style=bold,color=red]
+DOT	prepresp -> vcl_deliver [style=bold,color=blue,]
+DOT	vcl_deliver -> deliver [style=bold,color=green,label=deliver]
+DOT	vcl_deliver -> deliver [style=bold,color=red]
+DOT	vcl_deliver -> deliver [style=bold,color=blue]
+DOT     vcl_deliver -> errdeliver [label="error"]
+DOT     errdeliver [label="ERROR",shape=plaintext]
+DOT     vcl_deliver -> rstdeliver [label="restart",color=purple]
+DOT     rstdeliver [label="RESTART",shape=plaintext]
+DOT     vcl_deliver -> streambody [style=bold,color=cyan,label="deliver"]
+DOT }
+ *
+ */
+
+static int
+cnt_prepresp(struct sess *sp)
+{
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
+
+	if (sp->wrk->do_stream)
+		AssertObjCorePassOrBusy(sp->obj->objcore);
+
+	sp->wrk->res_mode = 0;
+
+	if ((sp->wrk->h_content_length != NULL || !sp->wrk->do_stream) &&
+	    !sp->wrk->do_gzip && !sp->wrk->do_gunzip)
+		sp->wrk->res_mode |= RES_LEN;
+
+	if (!sp->disable_esi && sp->obj->esidata != NULL) {
+		/* In ESI mode, we don't know the aggregate length */
+		sp->wrk->res_mode &= ~RES_LEN;
+		sp->wrk->res_mode |= RES_ESI;
+	}
+
+	if (sp->esi_level > 0) {
+		sp->wrk->res_mode &= ~RES_LEN;
+		sp->wrk->res_mode |= RES_ESI_CHILD;
+	}
+
+	if (cache_param->http_gzip_support && sp->obj->gziped &&
+	    !RFC2616_Req_Gzip(sp)) {
+		/*
+		 * We don't know what it uncompresses to
+		 * XXX: we could cache that
+		 */
+		sp->wrk->res_mode &= ~RES_LEN;
+		sp->wrk->res_mode |= RES_GUNZIP;
+	}
+
+	if (!(sp->wrk->res_mode & (RES_LEN|RES_CHUNKED|RES_EOF))) {
+		if (sp->obj->len == 0 && !sp->wrk->do_stream)
+			/*
+			 * If the object is empty, neither ESI nor GUNZIP
+			 * can make it any different size
+			 */
+			sp->wrk->res_mode |= RES_LEN;
+		else if (!sp->wantbody) {
+			/* Nothing */
+		} else if (sp->http->protover >= 11) {
+			sp->wrk->res_mode |= RES_CHUNKED;
+		} else {
+			sp->wrk->res_mode |= RES_EOF;
+			sp->doclose = "EOF mode";
+		}
+	}
+
+	sp->t_resp = VTIM_real();
+	if (sp->obj->objcore != NULL) {
+		if ((sp->t_resp - sp->obj->last_lru) > cache_param->lru_timeout &&
+		    EXP_Touch(sp->obj->objcore))
+			sp->obj->last_lru = sp->t_resp;
+		sp->obj->last_use = sp->t_resp;	/* XXX: locking ? */
+	}
+	http_Setup(sp->wrk->resp, sp->wrk->ws);
+	RES_BuildHttp(sp);
+	VCL_deliver_method(sp);
+	switch (sp->handling) {
+	case VCL_RET_DELIVER:
+		break;
+	case VCL_RET_RESTART:
+		if (sp->restarts >= cache_param->max_restarts)
+			break;
+		if (sp->wrk->do_stream) {
+			VDI_CloseFd(sp->wrk);
+			HSH_Drop(sp);
+		} else {
+			(void)HSH_Deref(sp->wrk, NULL, &sp->obj);
+		}
+		AZ(sp->obj);
+		sp->restarts++;
+		sp->director = NULL;
+		sp->wrk->h_content_length = NULL;
+		http_Setup(sp->wrk->bereq, NULL);
+		http_Setup(sp->wrk->beresp, NULL);
+		http_Setup(sp->wrk->resp, NULL);
+		sp->step = STP_RECV;
+		return (0);
+	default:
+		WRONG("Illegal action in vcl_deliver{}");
+	}
+	if (sp->wrk->do_stream) {
+		AssertObjCorePassOrBusy(sp->obj->objcore);
+		sp->step = STP_STREAMBODY;
+	} else {
+		sp->step = STP_DELIVER;
+	}
+	return (0);
+}
+
+/*--------------------------------------------------------------------
+ * Deliver an already stored object
+ *
+DOT subgraph xcluster_deliver {
+DOT	deliver [
+DOT		shape=ellipse
+DOT		label="Send body"
+DOT	]
+DOT }
+DOT deliver -> DONE [style=bold,color=green]
+DOT deliver -> DONE [style=bold,color=red]
+DOT deliver -> DONE [style=bold,color=blue]
+ *
+ */
+
+static int
+cnt_deliver(struct sess *sp)
+{
+
+	sp->director = NULL;
+	sp->restarts = 0;
+
+	RES_WriteObj(sp);
+
+	assert(WRW_IsReleased(sp->wrk));
+	assert(sp->wrk->wrw.ciov == sp->wrk->wrw.siov);
+	(void)HSH_Deref(sp->wrk, NULL, &sp->obj);
+	http_Setup(sp->wrk->resp, NULL);
+	sp->step = STP_DONE;
+	return (0);
+}
+
+/*--------------------------------------------------------------------
+ * This is the final state, figure out if we should close or recycle
+ * the client connection
+ *
+DOT	DONE [
+DOT		shape=hexagon
+DOT		label="Request completed"
+DOT	]
+ */
+
+static int
+cnt_done(struct sess *sp)
+{
+	double dh, dp, da;
+	int i;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_ORNULL(sp->vcl, VCL_CONF_MAGIC);
+
+	AZ(sp->obj);
+	AZ(sp->wrk->vbc);
+	sp->director = NULL;
+	sp->restarts = 0;
+
+	sp->wrk->do_esi = 0;
+	sp->wrk->do_gunzip = 0;
+	sp->wrk->do_gzip = 0;
+	sp->wrk->do_stream = 0;
+	sp->wrk->is_gunzip = 0;
+	sp->wrk->is_gzip = 0;
+
+	if (sp->vcl != NULL && sp->esi_level == 0) {
+		if (sp->wrk->vcl != NULL)
+			VCL_Rel(&sp->wrk->vcl);
+		sp->wrk->vcl = sp->vcl;
+		sp->vcl = NULL;
+	}
+
+	SES_Charge(sp);
+
+	sp->t_end = VTIM_real();
+	sp->wrk->lastused = sp->t_end;
+	if (sp->xid == 0) {
+		sp->t_req = sp->t_end;
+		sp->t_resp = sp->t_end;
+	} else if (sp->esi_level == 0) {
+		dp = sp->t_resp - sp->t_req;
+		da = sp->t_end - sp->t_resp;
+		dh = sp->t_req - sp->t_open;
+		/* XXX: Add StatReq == StatSess */
+		/* XXX: Workaround for pipe */
+		if (sp->fd >= 0) {
+			WSP(sp, SLT_Length, "%ju",
+			    (uintmax_t)sp->req_bodybytes);
+		}
+		WSP(sp, SLT_ReqEnd, "%u %.9f %.9f %.9f %.9f %.9f",
+		    sp->xid, sp->t_req, sp->t_end, dh, dp, da);
+	}
+	sp->xid = 0;
+	sp->t_open = sp->t_end;
+	sp->t_resp = NAN;
+	WSL_Flush(sp->wrk, 0);
+
+	/* If we did an ESI include, don't mess up our state */
+	if (sp->esi_level > 0)
+		return (1);
+
+	sp->req_bodybytes = 0;
+
+	sp->t_req = NAN;
+	sp->hash_always_miss = 0;
+	sp->hash_ignore_busy = 0;
+
+	if (sp->fd >= 0 && sp->doclose != NULL) {
+		/*
+		 * This is an orderly close of the connection; ditch nolinger
+		 * before we close, to get queued data transmitted.
+		 */
+		// XXX: not yet (void)VTCP_linger(sp->fd, 0);
+		SES_Close(sp, sp->doclose);
+	}
+
+	if (sp->fd < 0) {
+		sp->wrk->stats.sess_closed++;
+		SES_Delete(sp, NULL);
+		return (1);
+	}
+
+	if (sp->wrk->stats.client_req >= cache_param->wthread_stats_rate)
+		WRK_SumStat(sp->wrk);
+	/* Reset the workspace to the session-watermark */
+	WS_Reset(sp->ws, sp->ws_ses);
+	WS_Reset(sp->wrk->ws, NULL);
+
+	i = HTC_Reinit(sp->htc);
+	if (i == 1) {
+		sp->wrk->stats.sess_pipeline++;
+		sp->step = STP_START;
+		return (0);
+	}
+	if (Tlen(sp->htc->rxbuf)) {
+		sp->wrk->stats.sess_readahead++;
+		sp->step = STP_WAIT;
+		return (0);
+	}
+	if (cache_param->session_linger > 0) {
+		sp->wrk->stats.sess_linger++;
+		sp->step = STP_WAIT;
+		return (0);
+	}
+	sp->wrk->stats.sess_herd++;
+	sp->wrk = NULL;
+	Pool_Wait(sp);
+	return (1);
+}
+
+/*--------------------------------------------------------------------
+ * Emit an error
+ *
+DOT subgraph xcluster_error {
+DOT	vcl_error [
+DOT		shape=record
+DOT		label="vcl_error()|resp."
+DOT	]
+DOT	ERROR -> vcl_error
+DOT	vcl_error-> prepresp [label=deliver]
+DOT }
+DOT vcl_error-> rsterr [label="restart",color=purple]
+DOT rsterr [label="RESTART",shape=plaintext]
+ */
+
+static int
+cnt_error(struct sess *sp)
+{
+	struct worker *w;
+	struct http *h;
+	char date[40];
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+
+	sp->wrk->do_esi = 0;
+	sp->wrk->is_gzip = 0;
+	sp->wrk->is_gunzip = 0;
+	sp->wrk->do_gzip = 0;
+	sp->wrk->do_gunzip = 0;
+	sp->wrk->do_stream = 0;
+
+	w = sp->wrk;
+	if (sp->obj == NULL) {
+		HSH_Prealloc(sp);
+		EXP_Clr(&w->exp);
+		sp->obj = STV_NewObject(sp, NULL, cache_param->http_resp_size,
+		     &w->exp, (uint16_t)cache_param->http_max_hdr);
+		if (sp->obj == NULL)
+			sp->obj = STV_NewObject(sp, TRANSIENT_STORAGE,
+			    cache_param->http_resp_size, &w->exp,
+			    (uint16_t)cache_param->http_max_hdr);
+		if (sp->obj == NULL) {
+			sp->doclose = "Out of objects";
+			sp->director = NULL;
+			sp->wrk->h_content_length = NULL;
+			http_Setup(sp->wrk->beresp, NULL);
+			http_Setup(sp->wrk->bereq, NULL);
+			sp->step = STP_DONE;
+			return(0);
+		}
+		AN(sp->obj);
+		sp->obj->xid = sp->xid;
+		sp->obj->exp.entered = sp->t_req;
+	} else {
+		/* XXX: Null the headers ? */
+	}
+	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);
+	h = sp->obj->http;
+
+	if (sp->err_code < 100 || sp->err_code > 999)
+		sp->err_code = 501;
+
+	http_PutProtocol(w, sp->vsl_id, h, "HTTP/1.1");
+	http_PutStatus(h, sp->err_code);
+	VTIM_format(VTIM_real(), date);
+	http_PrintfHeader(w, sp->vsl_id, h, "Date: %s", date);
+	http_SetHeader(w, sp->vsl_id, h, "Server: Varnish");
+
+	if (sp->err_reason != NULL)
+		http_PutResponse(w, sp->vsl_id, h, sp->err_reason);
+	else
+		http_PutResponse(w, sp->vsl_id, h,
+		    http_StatusMessage(sp->err_code));
+	VCL_error_method(sp);
+
+	if (sp->handling == VCL_RET_RESTART &&
+	    sp->restarts <  cache_param->max_restarts) {
+		HSH_Drop(sp);
+		sp->director = NULL;
+		sp->restarts++;
+		sp->step = STP_RECV;
+		return (0);
+	} else if (sp->handling == VCL_RET_RESTART)
+		sp->handling = VCL_RET_DELIVER;
+
+
+	/* We always close when we take this path */
+	sp->doclose = "error";
+	sp->wantbody = 1;
+
+	assert(sp->handling == VCL_RET_DELIVER);
+	sp->err_code = 0;
+	sp->err_reason = NULL;
+	http_Setup(sp->wrk->bereq, NULL);
+	sp->step = STP_PREPRESP;
+	return (0);
+}
+
+/*--------------------------------------------------------------------
+ * Fetch response headers from the backend
+ *
+DOT subgraph xcluster_fetch {
+DOT	fetch [
+DOT		shape=ellipse
+DOT		label="fetch hdr\nfrom backend\n(find obj.ttl)"
+DOT	]
+DOT	vcl_fetch [
+DOT		shape=record
+DOT		label="vcl_fetch()|req.\nbereq.\nberesp."
+DOT	]
+DOT	fetch -> vcl_fetch [style=bold,color=blue]
+DOT	fetch -> vcl_fetch [style=bold,color=red]
+DOT	fetch_pass [
+DOT		shape=ellipse
+DOT		label="obj.f.pass=true"
+DOT	]
+DOT	vcl_fetch -> fetch_pass [label="hit_for_pass",style=bold,color=red]
+DOT }
+DOT fetch_pass -> fetchbody [style=bold,color=red]
+DOT vcl_fetch -> fetchbody [label="deliver",style=bold,color=blue]
+DOT vcl_fetch -> rstfetch [label="restart",color=purple]
+DOT rstfetch [label="RESTART",shape=plaintext]
+DOT fetch -> errfetch
+DOT vcl_fetch -> errfetch [label="error"]
+DOT errfetch [label="ERROR",shape=plaintext]
+ */
+
+static int
+cnt_fetch(struct sess *sp)
+{
+	int i;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
+
+	AN(sp->director);
+	AZ(sp->wrk->vbc);
+	AZ(sp->wrk->h_content_length);
+	AZ(sp->wrk->do_close);
+	AZ(sp->wrk->storage_hint);
+
+	http_Setup(sp->wrk->beresp, sp->wrk->ws);
+
+	i = FetchHdr(sp);
+	/*
+	 * If we recycle a backend connection, there is a finite chance
+	 * that the backend closed it before we get a request to it.
+	 * Do a single retry in that case.
+	 */
+	if (i == 1) {
+		VSC_C_main->backend_retry++;
+		i = FetchHdr(sp);
+	}
+
+	if (i) {
+		sp->handling = VCL_RET_ERROR;
+		sp->err_code = 503;
+	} else {
+		/*
+		 * These two headers can be spread over multiple actual headers
+		 * and we rely on their content outside of VCL, so collect them
+		 * into one line here.
+		 */
+		http_CollectHdr(sp->wrk->beresp, H_Cache_Control);
+		http_CollectHdr(sp->wrk->beresp, H_Vary);
+
+		/*
+		 * Figure out how the fetch is supposed to happen, before the
+		 * headers are adultered by VCL
+		 * NB: Also sets other sp->wrk variables
+		 */
+		sp->wrk->body_status = RFC2616_Body(sp);
+
+		sp->err_code = http_GetStatus(sp->wrk->beresp);
+
+		/*
+		 * What does RFC2616 think about TTL ?
+		 */
+		EXP_Clr(&sp->wrk->exp);
+		sp->wrk->exp.entered = VTIM_real();
+		RFC2616_Ttl(sp);
+
+		/* pass from vclrecv{} has negative TTL */
+		if (sp->objcore == NULL)
+			sp->wrk->exp.ttl = -1.;
+
+		AZ(sp->wrk->do_esi);
+
+		VCL_fetch_method(sp);
+
+		switch (sp->handling) {
+		case VCL_RET_HIT_FOR_PASS:
+			if (sp->objcore != NULL)
+				sp->objcore->flags |= OC_F_PASS;
+			sp->step = STP_FETCHBODY;
+			return (0);
+		case VCL_RET_DELIVER:
+			AssertObjCorePassOrBusy(sp->objcore);
+			sp->step = STP_FETCHBODY;
+			return (0);
+		default:
+			break;
+		}
+
+		/* We are not going to fetch the body, Close the connection */
+		VDI_CloseFd(sp->wrk);
+	}
+
+	/* Clean up partial fetch */
+	AZ(sp->wrk->vbc);
+
+	if (sp->objcore != NULL) {
+		CHECK_OBJ_NOTNULL(sp->objcore, OBJCORE_MAGIC);
+		AZ(HSH_Deref(sp->wrk, sp->objcore, NULL));
+		sp->objcore = NULL;
+	}
+	http_Setup(sp->wrk->bereq, NULL);
+	http_Setup(sp->wrk->beresp, NULL);
+	sp->wrk->h_content_length = NULL;
+	sp->director = NULL;
+	sp->wrk->storage_hint = NULL;
+
+	switch (sp->handling) {
+	case VCL_RET_RESTART:
+		sp->restarts++;
+		sp->step = STP_RECV;
+		return (0);
+	case VCL_RET_ERROR:
+		sp->step = STP_ERROR;
+		return (0);
+	default:
+		WRONG("Illegal action in vcl_fetch{}");
+	}
+}
+
+/*--------------------------------------------------------------------
+ * Fetch response body from the backend
+ *
+DOT subgraph xcluster_body {
+DOT	fetchbody [
+DOT		shape=diamond
+DOT		label="stream ?"
+DOT	]
+DOT	fetchbody2 [
+DOT		shape=ellipse
+DOT		label="fetch body\nfrom backend\n"
+DOT	]
+DOT }
+DOT fetchbody -> fetchbody2 [label=no,style=bold,color=red]
+DOT fetchbody -> fetchbody2 [style=bold,color=blue]
+DOT fetchbody -> prepresp [label=yes,style=bold,color=cyan]
+DOT fetchbody2 -> prepresp [style=bold,color=red]
+DOT fetchbody2 -> prepresp [style=bold,color=blue]
+ */
+
+
+static int
+cnt_fetchbody(struct sess *sp)
+{
+	int i;
+	struct http *hp, *hp2;
+	char *b;
+	uint16_t nhttp;
+	unsigned l;
+	struct vsb *vary = NULL;
+	int varyl = 0, pass;
+
+	assert(sp->handling == VCL_RET_HIT_FOR_PASS ||
+	    sp->handling == VCL_RET_DELIVER);
+
+	if (sp->objcore == NULL) {
+		/* This is a pass from vcl_recv */
+		pass = 1;
+		/* VCL may have fiddled this, but that doesn't help */
+		sp->wrk->exp.ttl = -1.;
+	} else if (sp->handling == VCL_RET_HIT_FOR_PASS) {
+		/* pass from vcl_fetch{} -> hit-for-pass */
+		/* XXX: the bereq was not filtered pass... */
+		pass = 1;
+	} else {
+		/* regular object */
+		pass = 0;
+	}
+
+	/*
+	 * The VCL variables beresp.do_g[un]zip tells us how we want the
+	 * object processed before it is stored.
+	 *
+	 * The backend Content-Encoding header tells us what we are going
+	 * to receive, which we classify in the following three classes:
+	 *
+	 *	"Content-Encoding: gzip"	--> object is gzip'ed.
+	 *	no Content-Encoding		--> object is not gzip'ed.
+	 *	anything else			--> do nothing wrt gzip
+	 *
+	 */
+
+	AZ(sp->wrk->vfp);
+
+	/* We do nothing unless the param is set */
+	if (!cache_param->http_gzip_support)
+		sp->wrk->do_gzip = sp->wrk->do_gunzip = 0;
+
+	sp->wrk->is_gzip =
+	    http_HdrIs(sp->wrk->beresp, H_Content_Encoding, "gzip");
+
+	sp->wrk->is_gunzip =
+	    !http_GetHdr(sp->wrk->beresp, H_Content_Encoding, NULL);
+
+	/* It can't be both */
+	assert(sp->wrk->is_gzip == 0 || sp->wrk->is_gunzip == 0);
+
+	/* We won't gunzip unless it is gzip'ed */
+	if (sp->wrk->do_gunzip && !sp->wrk->is_gzip)
+		sp->wrk->do_gunzip = 0;
+
+	/* If we do gunzip, remove the C-E header */
+	if (sp->wrk->do_gunzip)
+		http_Unset(sp->wrk->beresp, H_Content_Encoding);
+
+	/* We wont gzip unless it is ungziped */
+	if (sp->wrk->do_gzip && !sp->wrk->is_gunzip)
+		sp->wrk->do_gzip = 0;
+
+	/* If we do gzip, add the C-E header */
+	if (sp->wrk->do_gzip)
+		http_SetHeader(sp->wrk, sp->vsl_id, sp->wrk->beresp,
+		    "Content-Encoding: gzip");
+
+	/* But we can't do both at the same time */
+	assert(sp->wrk->do_gzip == 0 || sp->wrk->do_gunzip == 0);
+
+	/* ESI takes precedence and handles gzip/gunzip itself */
+	if (sp->wrk->do_esi)
+		sp->wrk->vfp = &vfp_esi;
+	else if (sp->wrk->do_gunzip)
+		sp->wrk->vfp = &vfp_gunzip;
+	else if (sp->wrk->do_gzip)
+		sp->wrk->vfp = &vfp_gzip;
+	else if (sp->wrk->is_gzip)
+		sp->wrk->vfp = &vfp_testgzip;
+
+	if (sp->wrk->do_esi || sp->esi_level > 0)
+		sp->wrk->do_stream = 0;
+	if (!sp->wantbody)
+		sp->wrk->do_stream = 0;
+
+	l = http_EstimateWS(sp->wrk->beresp,
+	    pass ? HTTPH_R_PASS : HTTPH_A_INS, &nhttp);
+
+	/* Create Vary instructions */
+	if (sp->objcore != NULL) {
+		CHECK_OBJ_NOTNULL(sp->objcore, OBJCORE_MAGIC);
+		vary = VRY_Create(sp, sp->wrk->beresp);
+		if (vary != NULL) {
+			varyl = VSB_len(vary);
+			assert(varyl > 0);
+			l += varyl;
+		}
+	}
+
+	/*
+	 * Space for producing a Content-Length: header including padding
+	 * A billion gigabytes is enough for anybody.
+	 */
+	l += strlen("Content-Length: XxxXxxXxxXxxXxxXxx") + sizeof(void *);
+
+	if (sp->wrk->exp.ttl < cache_param->shortlived || sp->objcore == NULL)
+		sp->wrk->storage_hint = TRANSIENT_STORAGE;
+
+	sp->obj = STV_NewObject(sp, sp->wrk->storage_hint, l,
+	    &sp->wrk->exp, nhttp);
+	if (sp->obj == NULL) {
+		/*
+		 * Try to salvage the transaction by allocating a
+		 * shortlived object on Transient storage.
+		 */
+		sp->obj = STV_NewObject(sp, TRANSIENT_STORAGE, l,
+		    &sp->wrk->exp, nhttp);
+		if (sp->wrk->exp.ttl > cache_param->shortlived)
+			sp->wrk->exp.ttl = cache_param->shortlived;
+		sp->wrk->exp.grace = 0.0;
+		sp->wrk->exp.keep = 0.0;
+	}
+	if (sp->obj == NULL) {
+		sp->err_code = 503;
+		sp->step = STP_ERROR;
+		VDI_CloseFd(sp->wrk);
+		return (0);
+	}
+	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);
+
+	sp->wrk->storage_hint = NULL;
+
+	if (sp->wrk->do_gzip || (sp->wrk->is_gzip && !sp->wrk->do_gunzip))
+		sp->obj->gziped = 1;
+
+	if (vary != NULL) {
+		sp->obj->vary =
+		    (void *)WS_Alloc(sp->obj->http->ws, varyl);
+		AN(sp->obj->vary);
+		memcpy(sp->obj->vary, VSB_data(vary), varyl);
+		VRY_Validate(sp->obj->vary);
+		VSB_delete(vary);
+	}
+
+	sp->obj->xid = sp->xid;
+	sp->obj->response = sp->err_code;
+	WS_Assert(sp->obj->ws_o);
+
+	/* Filter into object */
+	hp = sp->wrk->beresp;
+	hp2 = sp->obj->http;
+
+	hp2->logtag = HTTP_Obj;
+	http_CopyResp(hp2, hp);
+	http_FilterFields(sp->wrk, sp->vsl_id, hp2, hp,
+	    pass ? HTTPH_R_PASS : HTTPH_A_INS);
+	http_CopyHome(sp->wrk, sp->vsl_id, hp2);
+
+	if (http_GetHdr(hp, H_Last_Modified, &b))
+		sp->obj->last_modified = VTIM_parse(b);
+	else
+		sp->obj->last_modified = floor(sp->wrk->exp.entered);
+
+	assert(WRW_IsReleased(sp->wrk));
+
+	/*
+	 * If we can deliver a 304 reply, we don't bother streaming.
+	 * Notice that vcl_deliver{} could still nuke the headers
+	 * that allow the 304, in which case we return 200 non-stream.
+	 */
+	if (sp->obj->response == 200 &&
+	    sp->http->conds &&
+	    RFC2616_Do_Cond(sp))
+		sp->wrk->do_stream = 0;
+
+	AssertObjCorePassOrBusy(sp->obj->objcore);
+
+	if (sp->wrk->do_stream) {
+		sp->step = STP_PREPRESP;
+		return (0);
+	}
+
+	/* Use unmodified headers*/
+	i = FetchBody(sp->wrk, sp->obj);
+
+	sp->wrk->h_content_length = NULL;
+
+	http_Setup(sp->wrk->bereq, NULL);
+	http_Setup(sp->wrk->beresp, NULL);
+	sp->wrk->vfp = NULL;
+	assert(WRW_IsReleased(sp->wrk));
+	AZ(sp->wrk->vbc);
+	AN(sp->director);
+
+	if (i) {
+		HSH_Drop(sp);
+		AZ(sp->obj);
+		sp->err_code = 503;
+		sp->step = STP_ERROR;
+		return (0);
+	}
+
+	if (sp->obj->objcore != NULL) {
+		EXP_Insert(sp->obj);
+		AN(sp->obj->objcore);
+		AN(sp->obj->objcore->ban);
+		HSH_Unbusy(sp);
+	}
+	sp->wrk->acct_tmp.fetch++;
+	sp->step = STP_PREPRESP;
+	return (0);
+}
+
+/*--------------------------------------------------------------------
+ * Stream the body as we fetch it
+DOT subgraph xstreambody {
+DOT	streambody [
+DOT		shape=ellipse
+DOT		label="streaming\nfetch/deliver"
+DOT	]
+DOT }
+DOT streambody -> DONE [style=bold,color=cyan]
+ */
+
+static int
+cnt_streambody(struct sess *sp)
+{
+	int i;
+	struct stream_ctx sctx;
+	uint8_t obuf[sp->wrk->res_mode & RES_GUNZIP ?
+	    cache_param->gzip_stack_buffer : 1];
+
+	memset(&sctx, 0, sizeof sctx);
+	sctx.magic = STREAM_CTX_MAGIC;
+	AZ(sp->wrk->sctx);
+	sp->wrk->sctx = &sctx;
+
+	if (sp->wrk->res_mode & RES_GUNZIP) {
+		sctx.vgz = VGZ_NewUngzip(sp->wrk, "U S -");
+		sctx.obuf = obuf;
+		sctx.obuf_len = sizeof (obuf);
+	}
+
+	RES_StreamStart(sp);
+
+	AssertObjCorePassOrBusy(sp->obj->objcore);
+
+	i = FetchBody(sp->wrk, sp->obj);
+
+	sp->wrk->h_content_length = NULL;
+
+	http_Setup(sp->wrk->bereq, NULL);
+	http_Setup(sp->wrk->beresp, NULL);
+	sp->wrk->vfp = NULL;
+	AZ(sp->wrk->vbc);
+	AN(sp->director);
+
+	if (!i && sp->obj->objcore != NULL) {
+		EXP_Insert(sp->obj);
+		AN(sp->obj->objcore);
+		AN(sp->obj->objcore->ban);
+		HSH_Unbusy(sp);
+	} else {
+		sp->doclose = "Stream error";
+	}
+	sp->wrk->acct_tmp.fetch++;
+	sp->director = NULL;
+	sp->restarts = 0;
+
+	RES_StreamEnd(sp);
+	if (sp->wrk->res_mode & RES_GUNZIP)
+		(void)VGZ_Destroy(&sctx.vgz, sp->vsl_id);
+
+	sp->wrk->sctx = NULL;
+	assert(WRW_IsReleased(sp->wrk));
+	assert(sp->wrk->wrw.ciov == sp->wrk->wrw.siov);
+	(void)HSH_Deref(sp->wrk, NULL, &sp->obj);
+	http_Setup(sp->wrk->resp, NULL);
+	sp->step = STP_DONE;
+	return (0);
+}
+
+/*--------------------------------------------------------------------
+ * The very first request
+ */
+static int
+cnt_first(struct sess *sp)
+{
+
+	/*
+	 * XXX: If we don't have acceptfilters we are somewhat subject
+	 * XXX: to DoS'ing here.  One remedy would be to set a shorter
+	 * XXX: SO_RCVTIMEO and once we have received something here
+	 * XXX: increase it to the normal value.
+	 */
+
+	assert(sp->xid == 0);
+	assert(sp->restarts == 0);
+	VCA_Prep(sp);
+
+	/* Record the session watermark */
+	sp->ws_ses = WS_Snapshot(sp->ws);
+
+	/* Receive a HTTP protocol request */
+	HTC_Init(sp->htc, sp->ws, sp->fd, sp->vsl_id, cache_param->http_req_size,
+	    cache_param->http_req_hdr_len);
+	sp->wrk->lastused = sp->t_open;
+	sp->wrk->acct_tmp.sess++;
+
+	sp->step = STP_WAIT;
+	return (0);
+}
+
+/*--------------------------------------------------------------------
+ * HIT
+ * We had a cache hit.  Ask VCL, then march off as instructed.
+ *
+DOT subgraph xcluster_hit {
+DOT	hit [
+DOT		shape=record
+DOT		label="vcl_hit()|req.\nobj."
+DOT	]
+DOT }
+DOT hit -> err_hit [label="error"]
+DOT err_hit [label="ERROR",shape=plaintext]
+DOT hit -> rst_hit [label="restart",color=purple]
+DOT rst_hit [label="RESTART",shape=plaintext]
+DOT hit -> pass [label=pass,style=bold,color=red]
+DOT hit -> prepresp [label="deliver",style=bold,color=green]
+ */
+
+static int
+cnt_hit(struct sess *sp)
+{
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
+
+	assert(!(sp->obj->objcore->flags & OC_F_PASS));
+
+	AZ(sp->wrk->do_stream);
+
+	VCL_hit_method(sp);
+
+	if (sp->handling == VCL_RET_DELIVER) {
+		/* Dispose of any body part of the request */
+		(void)FetchReqBody(sp);
+		AZ(sp->wrk->bereq->ws);
+		AZ(sp->wrk->beresp->ws);
+		sp->step = STP_PREPRESP;
+		return (0);
+	}
+
+	/* Drop our object, we won't need it */
+	(void)HSH_Deref(sp->wrk, NULL, &sp->obj);
+	sp->objcore = NULL;
+
+	switch(sp->handling) {
+	case VCL_RET_PASS:
+		sp->step = STP_PASS;
+		return (0);
+	case VCL_RET_ERROR:
+		sp->step = STP_ERROR;
+		return (0);
+	case VCL_RET_RESTART:
+		sp->director = NULL;
+		sp->restarts++;
+		sp->step = STP_RECV;
+		return (0);
+	default:
+		WRONG("Illegal action in vcl_hit{}");
+	}
+}
+
+/*--------------------------------------------------------------------
+ * LOOKUP
+ * Hash things together and look object up in hash-table.
+ *
+ * LOOKUP consists of two substates so that we can reenter if we
+ * encounter a busy object.
+ *
+DOT subgraph xcluster_lookup {
+DOT	hash [
+DOT		shape=record
+DOT		label="vcl_hash()|req."
+DOT	]
+DOT	lookup [
+DOT		shape=diamond
+DOT		label="obj in cache ?\ncreate if not"
+DOT	]
+DOT	lookup2 [
+DOT		shape=diamond
+DOT		label="obj.f.pass ?"
+DOT	]
+DOT	hash -> lookup [label="hash",style=bold,color=green]
+DOT	lookup -> lookup2 [label="yes",style=bold,color=green]
+DOT }
+DOT lookup2 -> hit [label="no", style=bold,color=green]
+DOT lookup2 -> pass [label="yes",style=bold,color=red]
+DOT lookup -> miss [label="no",style=bold,color=blue]
+ */
+
+static int
+cnt_lookup(struct sess *sp)
+{
+	struct objcore *oc;
+	struct object *o;
+	struct objhead *oh;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
+
+	if (sp->hash_objhead == NULL) {
+		/* Not a waiting list return */
+		AZ(sp->vary_b);
+		AZ(sp->vary_l);
+		AZ(sp->vary_e);
+		(void)WS_Reserve(sp->ws, 0);
+	} else {
+		AN(sp->ws->r);
+	}
+	sp->vary_b = (void*)sp->ws->f;
+	sp->vary_e = (void*)sp->ws->r;
+	sp->vary_b[2] = '\0';
+
+	oc = HSH_Lookup(sp, &oh);
+
+	if (oc == NULL) {
+		/*
+		 * We lost the session to a busy object, disembark the
+		 * worker thread.   The hash code to restart the session,
+		 * still in STP_LOOKUP, later when the busy object isn't.
+		 * NB:  Do not access sp any more !
+		 */
+		return (1);
+	}
+
+
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC);
+
+	/* If we inserted a new object it's a miss */
+	if (oc->flags & OC_F_BUSY) {
+		sp->wrk->stats.cache_miss++;
+
+		if (sp->vary_l != NULL) {
+			assert(oc->busyobj->vary == sp->vary_b);
+			VRY_Validate(oc->busyobj->vary);
+			WS_ReleaseP(sp->ws, (void*)sp->vary_l);
+		} else {
+			AZ(oc->busyobj->vary);
+			WS_Release(sp->ws, 0);
+		}
+		sp->vary_b = NULL;
+		sp->vary_l = NULL;
+		sp->vary_e = NULL;
+
+		sp->objcore = oc;
+		sp->step = STP_MISS;
+		return (0);
+	}
+
+	o = oc_getobj(sp->wrk, oc);
+	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
+	sp->obj = o;
+
+	WS_Release(sp->ws, 0);
+	sp->vary_b = NULL;
+	sp->vary_l = NULL;
+	sp->vary_e = NULL;
+
+	if (oc->flags & OC_F_PASS) {
+		sp->wrk->stats.cache_hitpass++;
+		WSP(sp, SLT_HitPass, "%u", sp->obj->xid);
+		(void)HSH_Deref(sp->wrk, NULL, &sp->obj);
+		sp->objcore = NULL;
+		sp->step = STP_PASS;
+		return (0);
+	}
+
+	sp->wrk->stats.cache_hit++;
+	WSP(sp, SLT_Hit, "%u", sp->obj->xid);
+	sp->step = STP_HIT;
+	return (0);
+}
+
+/*--------------------------------------------------------------------
+ * We had a miss, ask VCL, proceed as instructed
+ *
+DOT subgraph xcluster_miss {
+DOT	miss [
+DOT		shape=ellipse
+DOT		label="filter req.->bereq."
+DOT	]
+DOT	vcl_miss [
+DOT		shape=record
+DOT		label="vcl_miss()|req.\nbereq."
+DOT	]
+DOT	miss -> vcl_miss [style=bold,color=blue]
+DOT }
+DOT vcl_miss -> rst_miss [label="restart",color=purple]
+DOT rst_miss [label="RESTART",shape=plaintext]
+DOT vcl_miss -> err_miss [label="error"]
+DOT err_miss [label="ERROR",shape=plaintext]
+DOT vcl_miss -> fetch [label="fetch",style=bold,color=blue]
+DOT vcl_miss -> pass [label="pass",style=bold,color=red]
+DOT
+ */
+
+static int
+cnt_miss(struct sess *sp)
+{
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
+
+	AZ(sp->obj);
+	AN(sp->objcore);
+	WS_Reset(sp->wrk->ws, NULL);
+	http_Setup(sp->wrk->bereq, sp->wrk->ws);
+	http_FilterHeader(sp, HTTPH_R_FETCH);
+	http_ForceGet(sp->wrk->bereq);
+	if (cache_param->http_gzip_support) {
+		/*
+		 * We always ask the backend for gzip, even if the
+		 * client doesn't grok it.  We will uncompress for
+		 * the minority of clients which don't.
+		 */
+		http_Unset(sp->wrk->bereq, H_Accept_Encoding);
+		http_SetHeader(sp->wrk, sp->vsl_id, sp->wrk->bereq,
+		    "Accept-Encoding: gzip");
+	}
+	sp->wrk->connect_timeout = 0;
+	sp->wrk->first_byte_timeout = 0;
+	sp->wrk->between_bytes_timeout = 0;
+	VCL_miss_method(sp);
+	switch(sp->handling) {
+	case VCL_RET_ERROR:
+		AZ(HSH_Deref(sp->wrk, sp->objcore, NULL));
+		sp->objcore = NULL;
+		http_Setup(sp->wrk->bereq, NULL);
+		sp->step = STP_ERROR;
+		return (0);
+	case VCL_RET_PASS:
+		AZ(HSH_Deref(sp->wrk, sp->objcore, NULL));
+		sp->objcore = NULL;
+		sp->step = STP_PASS;
+		return (0);
+	case VCL_RET_FETCH:
+		sp->step = STP_FETCH;
+		return (0);
+	case VCL_RET_RESTART:
+		AZ(HSH_Deref(sp->wrk, sp->objcore, NULL));
+		sp->objcore = NULL;
+		INCOMPL();
+	default:
+		WRONG("Illegal action in vcl_miss{}");
+	}
+}
+
+/*--------------------------------------------------------------------
+ * Start pass processing by getting headers from backend, then
+ * continue in passbody.
+ *
+DOT subgraph xcluster_pass {
+DOT	pass [
+DOT		shape=ellipse
+DOT		label="deref obj."
+DOT	]
+DOT	pass2 [
+DOT		shape=ellipse
+DOT		label="filter req.->bereq."
+DOT	]
+DOT	vcl_pass [
+DOT		shape=record
+DOT		label="vcl_pass()|req.\nbereq."
+DOT	]
+DOT	pass_do [
+DOT		shape=ellipse
+DOT		label="create anon object\n"
+DOT	]
+DOT	pass -> pass2 [style=bold, color=red]
+DOT	pass2 -> vcl_pass [style=bold, color=red]
+DOT	vcl_pass -> pass_do [label="pass"] [style=bold, color=red]
+DOT }
+DOT pass_do -> fetch [style=bold, color=red]
+DOT vcl_pass -> rst_pass [label="restart",color=purple]
+DOT rst_pass [label="RESTART",shape=plaintext]
+DOT vcl_pass -> err_pass [label="error"]
+DOT err_pass [label="ERROR",shape=plaintext]
+ */
+
+static int
+cnt_pass(struct sess *sp)
+{
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
+	AZ(sp->obj);
+
+	WS_Reset(sp->wrk->ws, NULL);
+	http_Setup(sp->wrk->bereq, sp->wrk->ws);
+	http_FilterHeader(sp, HTTPH_R_PASS);
+
+	sp->wrk->connect_timeout = 0;
+	sp->wrk->first_byte_timeout = 0;
+	sp->wrk->between_bytes_timeout = 0;
+	VCL_pass_method(sp);
+	if (sp->handling == VCL_RET_ERROR) {
+		http_Setup(sp->wrk->bereq, NULL);
+		sp->step = STP_ERROR;
+		return (0);
+	}
+	assert(sp->handling == VCL_RET_PASS);
+	sp->wrk->acct_tmp.pass++;
+	sp->sendbody = 1;
+	sp->step = STP_FETCH;
+	return (0);
+}
+
+/*--------------------------------------------------------------------
+ * Ship the request header to the backend unchanged, then pipe
+ * until one of the ends close the connection.
+ *
+DOT subgraph xcluster_pipe {
+DOT	pipe [
+DOT		shape=ellipse
+DOT		label="Filter req.->bereq."
+DOT	]
+DOT	vcl_pipe [
+DOT		shape=record
+DOT		label="vcl_pipe()|req.\nbereq\."
+DOT	]
+DOT	pipe_do [
+DOT		shape=ellipse
+DOT		label="send bereq.\npipe until close"
+DOT	]
+DOT	vcl_pipe -> pipe_do [label="pipe",style=bold,color=orange]
+DOT	pipe -> vcl_pipe [style=bold,color=orange]
+DOT }
+DOT pipe_do -> DONE [style=bold,color=orange]
+DOT vcl_pipe -> err_pipe [label="error"]
+DOT err_pipe [label="ERROR",shape=plaintext]
+ */
+
+static int
+cnt_pipe(struct sess *sp)
+{
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
+
+	sp->wrk->acct_tmp.pipe++;
+	WS_Reset(sp->wrk->ws, NULL);
+	http_Setup(sp->wrk->bereq, sp->wrk->ws);
+	http_FilterHeader(sp, HTTPH_R_PIPE);
+
+	VCL_pipe_method(sp);
+
+	if (sp->handling == VCL_RET_ERROR)
+		INCOMPL();
+	assert(sp->handling == VCL_RET_PIPE);
+
+	PipeSession(sp);
+	assert(WRW_IsReleased(sp->wrk));
+	http_Setup(sp->wrk->bereq, NULL);
+	sp->step = STP_DONE;
+	return (0);
+}
+
+/*--------------------------------------------------------------------
+ * RECV
+ * We have a complete request, set everything up and start it.
+ *
+DOT subgraph xcluster_recv {
+DOT	recv [
+DOT		shape=record
+DOT		label="vcl_recv()|req."
+DOT	]
+DOT }
+DOT RESTART -> recv
+DOT recv -> pipe [label="pipe",style=bold,color=orange]
+DOT recv -> pass2 [label="pass",style=bold,color=red]
+DOT recv -> err_recv [label="error"]
+DOT err_recv [label="ERROR",shape=plaintext]
+DOT recv -> hash [label="lookup",style=bold,color=green]
+ */
+
+static int
+cnt_recv(struct sess *sp)
+{
+	unsigned recv_handling;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
+	AZ(sp->obj);
+	assert(sp->wrk->wrw.ciov == sp->wrk->wrw.siov);
+
+	/* By default we use the first backend */
+	AZ(sp->director);
+	sp->director = sp->vcl->director[0];
+	AN(sp->director);
+
+	sp->disable_esi = 0;
+	sp->hash_always_miss = 0;
+	sp->hash_ignore_busy = 0;
+	sp->client_identity = NULL;
+
+	http_CollectHdr(sp->http, H_Cache_Control);
+
+	VCL_recv_method(sp);
+	recv_handling = sp->handling;
+
+	if (sp->restarts >= cache_param->max_restarts) {
+		if (sp->err_code == 0)
+			sp->err_code = 503;
+		sp->step = STP_ERROR;
+		return (0);
+	}
+
+	/* Zap these, in case we came here through restart */
+	sp->wrk->do_esi = 0;
+	sp->wrk->is_gzip = 0;
+	sp->wrk->is_gunzip = 0;
+	sp->wrk->do_gzip = 0;
+	sp->wrk->do_gunzip = 0;
+	sp->wrk->do_stream = 0;
+
+	if (cache_param->http_gzip_support &&
+	     (recv_handling != VCL_RET_PIPE) &&
+	     (recv_handling != VCL_RET_PASS)) {
+		if (RFC2616_Req_Gzip(sp)) {
+			http_Unset(sp->http, H_Accept_Encoding);
+			http_SetHeader(sp->wrk, sp->vsl_id, sp->http,
+			    "Accept-Encoding: gzip");
+		} else {
+			http_Unset(sp->http, H_Accept_Encoding);
+		}
+	}
+
+	SHA256_Init(sp->wrk->sha256ctx);
+	VCL_hash_method(sp);
+	assert(sp->handling == VCL_RET_HASH);
+	SHA256_Final(sp->digest, sp->wrk->sha256ctx);
+
+	if (!strcmp(sp->http->hd[HTTP_HDR_REQ].b, "HEAD"))
+		sp->wantbody = 0;
+	else
+		sp->wantbody = 1;
+
+	sp->sendbody = 0;
+	switch(recv_handling) {
+	case VCL_RET_LOOKUP:
+		/* XXX: discard req body, if any */
+		sp->step = STP_LOOKUP;
+		return (0);
+	case VCL_RET_PIPE:
+		if (sp->esi_level > 0) {
+			/* XXX: VSL something */
+			INCOMPL();
+			/* sp->step = STP_DONE; */
+			return (1);
+		}
+		sp->step = STP_PIPE;
+		return (0);
+	case VCL_RET_PASS:
+		sp->step = STP_PASS;
+		return (0);
+	case VCL_RET_ERROR:
+		/* XXX: discard req body, if any */
+		sp->step = STP_ERROR;
+		return (0);
+	default:
+		WRONG("Illegal action in vcl_recv{}");
+	}
+}
+
+/*--------------------------------------------------------------------
+ * START
+ * Handle a request, wherever it came from recv/restart.
+ *
+DOT start [shape=box,label="Dissect request"]
+DOT start -> recv [style=bold,color=green]
+ */
+
+static int
+cnt_start(struct sess *sp)
+{
+	uint16_t done;
+	char *p;
+	const char *r = "HTTP/1.1 100 Continue\r\n\r\n";
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	AZ(sp->restarts);
+	AZ(sp->obj);
+	AZ(sp->vcl);
+
+	/* Update stats of various sorts */
+	sp->wrk->stats.client_req++;
+	sp->t_req = VTIM_real();
+	sp->wrk->lastused = sp->t_req;
+	sp->wrk->acct_tmp.req++;
+
+	/* Assign XID and log */
+	sp->xid = ++xids;				/* XXX not locked */
+	WSP(sp, SLT_ReqStart, "%s %s %u", sp->addr, sp->port,  sp->xid);
+
+	/* Borrow VCL reference from worker thread */
+	VCL_Refresh(&sp->wrk->vcl);
+	sp->vcl = sp->wrk->vcl;
+	sp->wrk->vcl = NULL;
+
+	http_Setup(sp->http, sp->ws);
+	done = http_DissectRequest(sp);
+
+	/* If we could not even parse the request, just close */
+	if (done == 400) {
+		sp->step = STP_DONE;
+		SES_Close(sp, "junk");
+		return (0);
+	}
+
+	/* Catch request snapshot */
+	sp->ws_req = WS_Snapshot(sp->ws);
+
+	/* Catch original request, before modification */
+	HTTP_Copy(sp->http0, sp->http);
+
+	if (done != 0) {
+		sp->err_code = done;
+		sp->step = STP_ERROR;
+		return (0);
+	}
+
+	sp->doclose = http_DoConnection(sp->http);
+
+	/* XXX: Handle TRACE & OPTIONS of Max-Forwards = 0 */
+
+	/*
+	 * Handle Expect headers
+	 */
+	if (http_GetHdr(sp->http, H_Expect, &p)) {
+		if (strcasecmp(p, "100-continue")) {
+			sp->err_code = 417;
+			sp->step = STP_ERROR;
+			return (0);
+		}
+
+		/* XXX: Don't bother with write failures for now */
+		(void)write(sp->fd, r, strlen(r));
+		/* XXX: When we do ESI includes, this is not removed
+		 * XXX: because we use http0 as our basis.  Believed
+		 * XXX: safe, but potentially confusing.
+		 */
+		http_Unset(sp->http, H_Expect);
+	}
+
+	sp->step = STP_RECV;
+	return (0);
+}
+
+/*--------------------------------------------------------------------
+ * Central state engine dispatcher.
+ *
+ * Kick the session around until it has had enough.
+ *
+ */
+
+static void
+cnt_diag(struct sess *sp, const char *state)
+{
+	if (sp->wrk != NULL) {
+		WSP(sp, SLT_Debug, "thr %p STP_%s sp %p obj %p vcl %p",
+		    pthread_self(), state, sp, sp->obj, sp->vcl);
+		WSL_Flush(sp->wrk, 0);
+	} else {
+		VSL(SLT_Debug, sp->vsl_id,
+		    "thr %p STP_%s sp %p obj %p vcl %p",
+		    pthread_self(), state, sp, sp->obj, sp->vcl);
+	}
+}
+
+void
+CNT_Session(struct sess *sp)
+{
+	int done;
+	struct worker *w;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	w = sp->wrk;
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+
+	/*
+	 * Possible entrance states
+	 */
+	assert(
+	    sp->step == STP_FIRST ||
+	    sp->step == STP_START ||
+	    sp->step == STP_LOOKUP ||
+	    sp->step == STP_RECV);
+
+	AZ(w->do_stream);
+	AZ(w->is_gzip);
+	AZ(w->do_gzip);
+	AZ(w->is_gunzip);
+	AZ(w->do_gunzip);
+	AZ(w->do_esi);
+
+	/*
+	 * Whenever we come in from the acceptor or waiter, we need to set
+	 * blocking mode, but there is no point in setting it when we come from
+	 * ESI or when a parked sessions returns.
+	 * It would be simpler to do this in the acceptor or waiter, but we'd
+	 * rather do the syscall in the worker thread.
+	 * On systems which return errors for ioctl, we close early
+	 */
+	if ((sp->step == STP_FIRST || sp->step == STP_START) &&
+	    VTCP_blocking(sp->fd)) {
+		if (errno == ECONNRESET)
+			SES_Close(sp, "remote closed");
+		else
+			SES_Close(sp, "error");
+		sp->step = STP_DONE;
+	}
+
+	/*
+	 * NB: Once done is set, we can no longer touch sp!
+	 */
+	for (done = 0; !done; ) {
+		assert(sp->wrk == w);
+		/*
+		 * This is a good place to be paranoid about the various
+		 * pointers still pointing to the things we expect.
+		 */
+		CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+		CHECK_OBJ_ORNULL(sp->obj, OBJECT_MAGIC);
+		CHECK_OBJ_NOTNULL(sp->wrk, WORKER_MAGIC);
+		CHECK_OBJ_ORNULL(w->nobjhead, OBJHEAD_MAGIC);
+		WS_Assert(w->ws);
+
+		switch (sp->step) {
+#define STEP(l,u) \
+		    case STP_##u: \
+			if (cache_param->diag_bitmap & 0x01) \
+				cnt_diag(sp, #u); \
+			done = cnt_##l(sp); \
+		        break;
+#include "tbl/steps.h"
+#undef STEP
+		default:
+			WRONG("State engine misfire");
+		}
+		WS_Assert(w->ws);
+		CHECK_OBJ_ORNULL(w->nobjhead, OBJHEAD_MAGIC);
+	}
+	WSL_Flush(w, 0);
+	AZ(w->do_stream);
+	AZ(w->is_gzip);
+	AZ(w->do_gzip);
+	AZ(w->is_gunzip);
+	AZ(w->do_gunzip);
+	AZ(w->do_esi);
+#define ACCT(foo)	AZ(w->acct_tmp.foo);
+#include "tbl/acct_fields.h"
+#undef ACCT
+	assert(WRW_IsReleased(w));
+}
+
+/*
+DOT }
+*/
+
+/*--------------------------------------------------------------------
+ * Debugging aids
+ */
+
+static void
+cli_debug_xid(struct cli *cli, const char * const *av, void *priv)
+{
+	(void)priv;
+	if (av[2] != NULL)
+		xids = strtoul(av[2], NULL, 0);
+	VCLI_Out(cli, "XID is %u", xids);
+}
+
+/*
+ * Default to seed=1, this is the only seed value POSIXl guarantees will
+ * result in a reproducible random number sequence.
+ */
+static void
+cli_debug_srandom(struct cli *cli, const char * const *av, void *priv)
+{
+	(void)priv;
+	unsigned seed = 1;
+
+	if (av[2] != NULL)
+		seed = strtoul(av[2], NULL, 0);
+	srandom(seed);
+	srand48(random());
+	VCLI_Out(cli, "Random(3) seeded with %lu", seed);
+}
+
+static struct cli_proto debug_cmds[] = {
+	{ "debug.xid", "debug.xid",
+		"\tExamine or set XID\n", 0, 1, "d", cli_debug_xid },
+	{ "debug.srandom", "debug.srandom",
+		"\tSeed the random(3) function\n", 0, 1, "d", cli_debug_srandom },
+	{ NULL }
+};
+
+/*--------------------------------------------------------------------
+ *
+ */
+
+void
+CNT_Init(void)
+{
+
+	srandomdev();
+	srand48(random());
+	xids = random();
+	CLI_AddFuncs(debug_cmds);
+}
+
+
diff --git a/bin/varnishd/cache/cache_cli.c b/bin/varnishd/cache/cache_cli.c
new file mode 100644
index 0000000..f29f86a
--- /dev/null
+++ b/bin/varnishd/cache/cache_cli.c
@@ -0,0 +1,243 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * Caching process CLI handling.
+ *
+ * We only have one CLI source, the stdin/stdout pipes from the manager
+ * process, but we complicate things by having undocumented commands that
+ * we do not want to show in a plain help, and by having commands that the
+ * manager has already shown in help before asking us.
+ */
+
+#include "config.h"
+
+#include <stddef.h>			// offsetof
+
+#include "cache.h"
+#include "common/heritage.h"
+
+#include "cache_backend.h"		// struct vbc
+#include "hash/hash_slinger.h"		// struct objhead
+#include "vcli.h"
+#include "vcli_common.h"
+#include "vcli_priv.h"
+#include "vcli_serve.h"
+
+pthread_t		cli_thread;
+static struct lock	cli_mtx;
+static int		add_check;
+static struct VCLS	*cls;
+
+/*
+ * The CLI commandlist is split in three:
+ *  - Commands we get from/share with the manager, we don't show these
+ *	in help, as the manager already did that.
+ *  - Cache process commands, show in help
+ *  - Undocumented debug commands, show in undocumented "help -d"
+ */
+
+/*--------------------------------------------------------------------
+ * Add CLI functions to the appropriate command set
+ */
+
+void
+CLI_AddFuncs(struct cli_proto *p)
+{
+
+	AZ(add_check);
+	Lck_Lock(&cli_mtx);
+	AZ(VCLS_AddFunc(cls, 0, p));
+	Lck_Unlock(&cli_mtx);
+}
+
+static void
+cli_cb_before(const struct cli *cli)
+{
+
+	ASSERT_CLI();
+	VSL(SLT_CLI, 0, "Rd %s", cli->cmd);
+	VCL_Poll();
+	VBE_Poll();
+	Lck_Lock(&cli_mtx);
+}
+
+static void
+cli_cb_after(const struct cli *cli)
+{
+
+	ASSERT_CLI();
+	Lck_Unlock(&cli_mtx);
+	VSL(SLT_CLI, 0, "Wr %03u %u %s",
+	    cli->result, VSB_len(cli->sb), VSB_data(cli->sb));
+}
+
+void
+CLI_Run(void)
+{
+	int i;
+
+	add_check = 1;
+
+	AN(VCLS_AddFd(cls, heritage.cli_in, heritage.cli_out, NULL, NULL));
+
+	do {
+		i = VCLS_Poll(cls, -1);
+	} while(i > 0);
+	VSL(SLT_CLI, 0, "EOF on CLI connection, worker stops");
+	VCA_Shutdown();
+}
+
+/*--------------------------------------------------------------------*/
+
+static void
+cli_debug_sizeof(struct cli *cli, const char * const *av, void *priv)
+{
+	(void)av;
+	(void)priv;
+
+#define SZOF(foo)       VCLI_Out(cli, \
+    "sizeof(%s) = %zd = 0x%zx\n", #foo, sizeof(foo), sizeof(foo))
+	SZOF(struct ws);
+	SZOF(struct http);
+	SZOF(struct http_conn);
+	SZOF(struct acct);
+	SZOF(struct worker);
+	SZOF(struct storage);
+	SZOF(struct object);
+	SZOF(struct objcore);
+	SZOF(struct objhead);
+	SZOF(struct sess);
+	SZOF(struct vbc);
+	SZOF(struct VSC_C_main);
+	SZOF(struct lock);
+#if 0
+#define OFOF(foo, bar)	{ foo __foo; VCLI_Out(cli, \
+    "%-30s = 0x%4zx @ 0x%4zx\n", \
+	#foo "." #bar, sizeof(__foo.bar), offsetof(foo, bar)); }
+#if 0
+	OFOF(struct objhead, magic);
+	OFOF(struct objhead, refcnt);
+	OFOF(struct objhead, mtx);
+	OFOF(struct objhead, objcs);
+	OFOF(struct objhead, digest);
+	OFOF(struct objhead, waitinglist);
+	OFOF(struct objhead, _u);
+#endif
+#if 0
+	OFOF(struct http, magic);
+	OFOF(struct http, logtag);
+	OFOF(struct http, ws);
+	OFOF(struct http, hd);
+	OFOF(struct http, hdf);
+	OFOF(struct http, shd);
+	OFOF(struct http, nhd);
+	OFOF(struct http, status);
+	OFOF(struct http, protover);
+	OFOF(struct http, conds);
+#endif
+#if 0
+	OFOF(struct storage, magic);
+	OFOF(struct storage, fd);
+	OFOF(struct storage, where);
+	OFOF(struct storage, list);
+	OFOF(struct storage, stevedore);
+	OFOF(struct storage, priv);
+	OFOF(struct storage, ptr);
+	OFOF(struct storage, len);
+	OFOF(struct storage, space);
+#endif
+#if 0
+	OFOF(struct object, magic);
+	OFOF(struct object, xid);
+	OFOF(struct object, objstore);
+	OFOF(struct object, objcore);
+	OFOF(struct object, ws_o);
+	OFOF(struct object, vary);
+	OFOF(struct object, hits);
+	OFOF(struct object, response);
+	OFOF(struct object, gziped);
+	OFOF(struct object, gzip_start);
+	OFOF(struct object, gzip_last);
+	OFOF(struct object, gzip_stop);
+	OFOF(struct object, len);
+	OFOF(struct object, age);
+	OFOF(struct object, entered);
+	OFOF(struct object, exp);
+	OFOF(struct object, last_modified);
+	OFOF(struct object, last_lru);
+	OFOF(struct object, http);
+	OFOF(struct object, store);
+	OFOF(struct object, esidata);
+	OFOF(struct object, last_use);
+#endif
+#undef OFOF
+#endif
+}
+
+/*--------------------------------------------------------------------*/
+
+static void
+ccf_panic(struct cli *cli, const char * const *av, void *priv)
+{
+
+	(void)cli;
+	(void)av;
+	(void)priv;
+	assert(!strcmp("", "You asked for it"));
+}
+
+/*--------------------------------------------------------------------*/
+
+static struct cli_proto master_cmds[] = {
+	{ CLI_PING,		"i", VCLS_func_ping },
+	{ CLI_HELP,             "i", VCLS_func_help },
+	{ "debug.sizeof", "debug.sizeof",
+		"\tDump sizeof various data structures\n",
+		0, 0, "d", cli_debug_sizeof },
+	{ "debug.panic.worker", "debug.panic.worker",
+		"\tPanic the worker process.\n",
+		0, 0, "d", ccf_panic },
+	{ NULL }
+};
+
+/*--------------------------------------------------------------------
+ * Initialize the CLI subsystem
+ */
+
+void
+CLI_Init(void)
+{
+
+	Lck_New(&cli_mtx, lck_cli);
+	cli_thread = pthread_self();
+
+	cls = VCLS_New(cli_cb_before, cli_cb_after, cache_param->cli_buffer);
+	AN(cls);
+
+	CLI_AddFuncs(master_cmds);
+}
diff --git a/bin/varnishd/cache/cache_dir.c b/bin/varnishd/cache/cache_dir.c
new file mode 100644
index 0000000..d4794f9
--- /dev/null
+++ b/bin/varnishd/cache/cache_dir.c
@@ -0,0 +1,121 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * Handle backend connections and backend request structures.
+ *
+ */
+
+#include "config.h"
+
+#include "cache.h"
+
+#include "cache_backend.h"
+#include "vtcp.h"
+
+/* Close a connection ------------------------------------------------*/
+
+void
+VDI_CloseFd(struct worker *wrk)
+{
+	struct backend *bp;
+
+	CHECK_OBJ_NOTNULL(wrk->vbc, VBC_MAGIC);
+	CHECK_OBJ_NOTNULL(wrk->vbc->backend, BACKEND_MAGIC);
+	assert(wrk->vbc->fd >= 0);
+
+	bp = wrk->vbc->backend;
+
+	WSL(wrk, SLT_BackendClose, wrk->vbc->vsl_id, "%s", bp->display_name);
+
+	/* Checkpoint log to flush all info related to this connection
+	   before the OS reuses the FD */
+	WSL_Flush(wrk, 0);
+
+	VTCP_close(&wrk->vbc->fd);
+	VBE_DropRefConn(bp);
+	wrk->vbc->backend = NULL;
+	VBE_ReleaseConn(wrk->vbc);
+	wrk->vbc = NULL;
+	wrk->do_close = 0;
+}
+
+/* Recycle a connection ----------------------------------------------*/
+
+void
+VDI_RecycleFd(struct worker *wrk)
+{
+	struct backend *bp;
+
+	CHECK_OBJ_NOTNULL(wrk->vbc, VBC_MAGIC);
+	CHECK_OBJ_NOTNULL(wrk->vbc->backend, BACKEND_MAGIC);
+	assert(wrk->vbc->fd >= 0);
+	AZ(wrk->do_close);
+
+	bp = wrk->vbc->backend;
+
+	WSL(wrk, SLT_BackendReuse, wrk->vbc->vsl_id, "%s", bp->display_name);
+	/*
+	 * Flush the shmlog, so that another session reusing this backend
+	 * will log chronologically later than our use of it.
+	 */
+	WSL_Flush(wrk, 0);
+	Lck_Lock(&bp->mtx);
+	VSC_C_main->backend_recycle++;
+	VTAILQ_INSERT_HEAD(&bp->connlist, wrk->vbc, list);
+	wrk->vbc = NULL;
+	VBE_DropRefLocked(bp);
+}
+
+/* Get a connection --------------------------------------------------*/
+
+struct vbc *
+VDI_GetFd(const struct director *d, struct sess *sp)
+{
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	if (d == NULL)
+		d = sp->director;
+	CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC);
+	return (d->getfd(d, sp));
+}
+
+/* Check health ------------------------------------------------------
+ *
+ * The target is really an objhead pointer, but since it can not be
+ * dereferenced during health-checks, we pass it as uintptr_t, which
+ * hopefully will make people investigate, before mucking about with it.
+ */
+
+int
+VDI_Healthy(const struct director *d, const struct sess *sp)
+{
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC);
+	return (d->healthy(d, sp));
+}
diff --git a/bin/varnishd/cache/cache_dir_dns.c b/bin/varnishd/cache/cache_dir_dns.c
new file mode 100644
index 0000000..ae96dbb
--- /dev/null
+++ b/bin/varnishd/cache/cache_dir_dns.c
@@ -0,0 +1,469 @@
+/*-
+ * Copyright (c) 2009-2010 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Kristian Lyngstol <kristian at redpill-linpro.com>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ */
+
+#include "config.h"
+
+#include <netinet/in.h>
+
+#include <netdb.h>
+#include <stdio.h>
+#include <stdlib.h>
+
+#include "cache.h"
+
+#include "cache_backend.h"
+#include "vrt.h"
+
+/*--------------------------------------------------------------------*/
+
+/* FIXME: Should eventually be a configurable variable. */
+#define VDI_DNS_MAX_CACHE		1024
+#define VDI_DNS_GROUP_MAX_BACKENDS	1024
+
+/* DNS Cache entry
+ */
+struct vdi_dns_hostgroup {
+	unsigned			magic;
+#define VDI_DNSDIR_MAGIC		0x1bacab21
+	char				*hostname;
+	struct director			*hosts[VDI_DNS_GROUP_MAX_BACKENDS];
+	unsigned			nhosts;
+	unsigned			next_host; /* Next to use...*/
+	double				ttl;
+	VTAILQ_ENTRY(vdi_dns_hostgroup)	list;
+};
+
+struct vdi_dns {
+	unsigned			magic;
+#define VDI_DNS_MAGIC			0x1337a178
+	struct director			dir;
+	struct director			**hosts;
+	unsigned			nhosts;
+	VTAILQ_HEAD(_cachelist,vdi_dns_hostgroup)	cachelist;
+	unsigned			ncachelist;
+	pthread_rwlock_t		rwlock;
+	const char			*suffix;
+	double			ttl;
+};
+
+/* Compare an IPv4 backend to a IPv4 addr/len */
+static int
+vdi_dns_comp_addrinfo4(const struct backend *bp,
+		       const struct sockaddr_storage *addr,
+		       const socklen_t len)
+{
+	uint32_t u, p;
+	const struct sockaddr_in *bps = (const void *)bp->ipv4;
+	const struct sockaddr_in *bpd = (const void *)addr;
+
+	if (bp->ipv4len != len || len <= 0)
+		return (0);
+
+	u = bpd->sin_addr.s_addr;
+	p = bps->sin_addr.s_addr;
+
+	return (u == p);
+}
+
+/* Compare an IPv6 backend to a IPv6 addr/len */
+static int
+vdi_dns_comp_addrinfo6(const struct backend *bp,
+		       const struct sockaddr_storage *addr,
+		       const socklen_t len)
+{
+	const uint8_t *u, *p;
+	const struct sockaddr_in6 *bps = (const void *)bp->ipv6;
+	const struct sockaddr_in6 *bpd = (const void *)addr;
+
+	if (bp->ipv6len != len || len <= 0)
+		return (0);
+
+	u = bpd->sin6_addr.s6_addr;
+	p = bps->sin6_addr.s6_addr;
+
+	return (!memcmp(u, p, 16));
+}
+
+/* Check if a backends socket is the same as addr */
+static int
+vdi_dns_comp_addrinfo(const struct director *dir,
+		      const struct sockaddr_storage *addr,
+		      const socklen_t len)
+{
+	struct backend *bp;
+
+	bp = vdi_get_backend_if_simple(dir);
+	AN(bp);
+	if (addr->ss_family == PF_INET && bp->ipv4) {
+		return (vdi_dns_comp_addrinfo4(bp, addr, len));
+	} else if (addr->ss_family == PF_INET6 && bp->ipv6) {
+		return (vdi_dns_comp_addrinfo6(bp, addr, len));
+	}
+	return (0);
+}
+
+/* Pick a host from an existing hostgroup.
+ * Balance on round-robin if multiple backends are available and only pick
+ * healthy ones.
+ */
+static struct director *
+vdi_dns_pick_host(const struct sess *sp, struct vdi_dns_hostgroup *group) {
+	int initial, i, nhosts, current;
+	if (group->nhosts == 0)
+		return (NULL); // In case of error.
+	if (group->next_host >= group->nhosts)
+		group->next_host = 0;
+
+	/* Pick a healthy backend */
+	initial = group->next_host;
+	nhosts = group->nhosts;
+	for (i=0; i < nhosts; i++) {
+		if (i + initial >= nhosts)
+			current = i + initial - nhosts;
+		else
+			current = i + initial;
+		if (VDI_Healthy(group->hosts[current], sp)) {
+			group->next_host = current+1;
+			return (group->hosts[current]);
+		}
+	}
+
+	return (NULL);
+}
+
+/* Remove an item from the dns cache.
+ * If *group is NULL, the head is popped.
+ * Remember locking.
+ */
+static void
+vdi_dns_pop_cache(struct vdi_dns *vs,
+		  struct vdi_dns_hostgroup *group)
+{
+	if (group == NULL)
+		group = VTAILQ_LAST( &vs->cachelist, _cachelist );
+	assert(group != NULL);
+	free(group->hostname);
+	VTAILQ_REMOVE(&vs->cachelist, group, list);
+	FREE_OBJ(group);
+	vs->ncachelist--;
+}
+
+/* Dummy in case someone feels like optimizing it? meh...
+ */
+static inline int
+vdi_dns_groupmatch(const struct vdi_dns_hostgroup *group, const char *hostname)
+{
+	return (!strcmp(group->hostname, hostname));
+}
+
+/* Search the cache for 'hostname' and put a backend-pointer as necessary,
+ * return true for cache hit. This could still be a NULL backend if we did
+ * a lookup earlier and didn't find a host (ie: cache failed too)
+ *
+ * if rwlock is true, the first timed out object found (if any) is popped
+ * and freed.
+ */
+static int
+vdi_dns_cache_has(const struct sess *sp,
+		  struct vdi_dns *vs,
+		  const char *hostname,
+		  struct director **backend,
+		  int rwlock)
+{
+	struct director *ret;
+	struct vdi_dns_hostgroup *hostgr;
+	struct vdi_dns_hostgroup *hostgr2;
+	VTAILQ_FOREACH_SAFE(hostgr, &vs->cachelist, list, hostgr2) {
+		CHECK_OBJ_NOTNULL(hostgr, VDI_DNSDIR_MAGIC);
+		if (hostgr->ttl <= sp->t_req) {
+			if (rwlock)
+				vdi_dns_pop_cache(vs, hostgr);
+			return (0);
+		}
+		if (vdi_dns_groupmatch(hostgr, hostname)) {
+			ret = (vdi_dns_pick_host(sp, hostgr));
+			*backend = ret;
+			if (*backend != NULL)
+				CHECK_OBJ_NOTNULL(*backend, DIRECTOR_MAGIC);
+			return (1);
+		}
+	}
+	return (0);
+}
+
+/* Add a newly cached item to the dns cache list.
+ * (Sorry for the list_add/_add confusion...)
+ */
+static void
+vdi_dns_cache_list_add(const struct sess *sp,
+		       struct vdi_dns *vs,
+		       struct vdi_dns_hostgroup *new)
+{
+	if (vs->ncachelist >= VDI_DNS_MAX_CACHE) {
+		VSC_C_main->dir_dns_cache_full++;
+		vdi_dns_pop_cache(vs, NULL);
+	}
+	CHECK_OBJ_NOTNULL(new, VDI_DNSDIR_MAGIC);
+	assert(new->hostname != 0);
+	new->ttl = sp->t_req + vs->ttl;
+	VTAILQ_INSERT_HEAD(&vs->cachelist, new, list);
+	vs->ncachelist++;
+}
+
+/* Add an item to the dns cache.
+ * XXX: Might want to factor the getaddrinfo() out of the lock and do the
+ * cache_has() afterwards to do multiple dns lookups in parallel...
+ */
+static int
+vdi_dns_cache_add(const struct sess *sp,
+		  struct vdi_dns *vs,
+		  const char *hostname,
+		  struct director **backend)
+{
+	int error, i, host = 0;
+	struct addrinfo *res0, *res, hint;
+	struct vdi_dns_hostgroup *new;
+
+	/* Due to possible race while upgrading the lock, we have to
+	 * recheck if the result is already looked up. The overhead for
+	 * this is insignificant unless dns isn't cached properly (all
+	 * unique names or something equally troublesome).
+	 */
+
+	if (vdi_dns_cache_has(sp, vs, hostname, backend, 1))
+		return (1);
+
+	memset(&hint, 0, sizeof hint);
+	hint.ai_family = PF_UNSPEC;
+	hint.ai_socktype = SOCK_STREAM;
+
+	ALLOC_OBJ(new, VDI_DNSDIR_MAGIC);
+	XXXAN(new);
+
+	REPLACE(new->hostname, hostname);
+
+	error = getaddrinfo(hostname, "80", &hint, &res0);
+	VSC_C_main->dir_dns_lookups++;
+	if (error) {
+		vdi_dns_cache_list_add(sp, vs, new);
+		VSC_C_main->dir_dns_failed++;
+		return (0);
+	}
+
+	for (res = res0; res; res = res->ai_next) {
+		if (res->ai_family != PF_INET && res->ai_family != PF_INET6)
+			continue;
+
+		for (i = 0; i < vs->nhosts; i++) {
+			struct sockaddr_storage ss_hack;
+			memcpy(&ss_hack, res->ai_addr, res->ai_addrlen);
+			if (vdi_dns_comp_addrinfo(vs->hosts[i],
+			    &ss_hack, res->ai_addrlen)) {
+				new->hosts[host] = vs->hosts[i];
+				CHECK_OBJ_NOTNULL(new->hosts[host],
+				    DIRECTOR_MAGIC);
+				host++;
+			}
+		}
+	}
+	freeaddrinfo(res0);
+
+	new->nhosts = host;
+	vdi_dns_cache_list_add(sp, vs, new);
+	*backend = vdi_dns_pick_host(sp, new);
+	return (1);
+}
+
+/* Walk through the cached lookups looking for the relevant host, add one
+ * if it isn't already cached.
+ *
+ * Returns a backend or NULL.
+ */
+static struct director *
+vdi_dns_walk_cache(const struct sess *sp,
+		   struct vdi_dns *vs,
+		   const char *hostname)
+{
+	struct director *backend = NULL;
+	int ret;
+
+	AZ(pthread_rwlock_rdlock(&vs->rwlock));
+	ret = vdi_dns_cache_has(sp, vs, hostname, &backend, 0);
+	AZ(pthread_rwlock_unlock(&vs->rwlock));
+	if (!ret) {
+		/*
+		 * XXX: Isn't there a race here where another thread
+		 * XXX: could grab the lock and add it before we do ?
+		 * XXX: Should 'ret' be checked for that ?
+		 */
+		AZ(pthread_rwlock_wrlock(&vs->rwlock));
+		ret = vdi_dns_cache_add(sp, vs, hostname, &backend);
+		AZ(pthread_rwlock_unlock(&vs->rwlock));
+	} else
+		VSC_C_main->dir_dns_hit++;
+
+	/* Bank backend == cached a failure, so to speak */
+	if (backend != NULL)
+		CHECK_OBJ_NOTNULL(backend, DIRECTOR_MAGIC);
+	return (backend);
+}
+
+/* Parses the Host:-header and heads out to find a backend.
+ */
+static struct director *
+vdi_dns_find_backend(const struct sess *sp, struct vdi_dns *vs)
+{
+	struct director *ret;
+	struct http *hp;
+	char *p, *q;
+	char hostname[NI_MAXHOST];
+
+	/* bereq is only present after recv et. al, otherwise use req (ie:
+	 * use req for health checks in vcl_recv and such).
+	 */
+	if (sp->wrk->bereq)
+		hp = sp->wrk->bereq;
+	else
+		hp = sp->http;
+
+
+	CHECK_OBJ_NOTNULL(hp, HTTP_MAGIC);
+	if (http_GetHdr(hp, H_Host, &p) == 0)
+		return (NULL);
+
+	q = strchr(p, ':');
+	if (q == NULL)
+		q = strchr(p, '\0');
+	AN(q);
+
+	bprintf(hostname, "%.*s%s", (int)(q - p), p,
+	    vs->suffix ? vs->suffix : "");
+
+	ret = vdi_dns_walk_cache(sp, vs, hostname);
+	return (ret);
+}
+
+static struct vbc *
+vdi_dns_getfd(const struct director *director, struct sess *sp)
+{
+	struct vdi_dns *vs;
+	struct director *dir;
+	struct vbc *vbe;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(director, DIRECTOR_MAGIC);
+	CAST_OBJ_NOTNULL(vs, director->priv, VDI_DNS_MAGIC);
+
+	dir = vdi_dns_find_backend(sp, vs);
+	if (!dir || !VDI_Healthy(dir, sp))
+		return (NULL);
+
+	vbe = VDI_GetFd(dir, sp);
+	return (vbe);
+}
+
+static unsigned
+vdi_dns_healthy(const struct director *dir, const struct sess *sp)
+{
+	/* XXX: Fooling -Werror for a bit until it's actually implemented.
+	 */
+	(void)dir;
+	(void)sp;
+	return (1);
+
+	/*
+	struct vdi_dns *vs;
+	struct director *dir;
+	int i;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->director, DIRECTOR_MAGIC);
+	CAST_OBJ_NOTNULL(vs, sp->director->priv, VDI_DNS_MAGIC);
+
+	dir = vdi_dns_find_backend(sp, vs);
+
+	if (dir)
+		return (1);
+	return (0);
+	*/
+}
+
+static void
+vdi_dns_fini(const struct director *d)
+{
+	struct vdi_dns *vs;
+
+	CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC);
+	CAST_OBJ_NOTNULL(vs, d->priv, VDI_DNS_MAGIC);
+
+	free(vs->hosts);
+	free(vs->dir.vcl_name);
+	vs->dir.magic = 0;
+	/* FIXME: Free the cache */
+	AZ(pthread_rwlock_destroy(&vs->rwlock));
+	FREE_OBJ(vs);
+}
+
+void
+VRT_init_dir_dns(struct cli *cli, struct director **bp, int idx,
+    const void *priv)
+{
+	const struct vrt_dir_dns *t;
+	struct vdi_dns *vs;
+	const struct vrt_dir_dns_entry *te;
+	int i;
+
+	ASSERT_CLI();
+	(void)cli;
+	t = priv;
+	ALLOC_OBJ(vs, VDI_DNS_MAGIC);
+	XXXAN(vs);
+	vs->hosts = calloc(sizeof(struct director *), t->nmember);
+	XXXAN(vs->hosts);
+
+	vs->dir.magic = DIRECTOR_MAGIC;
+	vs->dir.priv = vs;
+	vs->dir.name = "dns";
+	REPLACE(vs->dir.vcl_name, t->name);
+	vs->dir.getfd = vdi_dns_getfd;
+	vs->dir.fini = vdi_dns_fini;
+	vs->dir.healthy = vdi_dns_healthy;
+
+	vs->suffix = t->suffix;
+	vs->ttl = t->ttl;
+
+	te = t->members;
+	for (i = 0; i < t->nmember; i++, te++)
+		vs->hosts[i] = bp[te->host];
+	vs->nhosts = t->nmember;
+	vs->ttl = t->ttl;
+	VTAILQ_INIT(&vs->cachelist);
+	AZ(pthread_rwlock_init(&vs->rwlock, NULL));
+	bp[idx] = &vs->dir;
+}
diff --git a/bin/varnishd/cache/cache_dir_random.c b/bin/varnishd/cache/cache_dir_random.c
new file mode 100644
index 0000000..d6570ed
--- /dev/null
+++ b/bin/varnishd/cache/cache_dir_random.c
@@ -0,0 +1,285 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * This code is shared between the random, client and hash directors, because
+ * they share the same properties and most of the same selection logic.
+ *
+ * The random director picks a backend on random.
+ *
+ * The hash director picks based on the hash from vcl_hash{}
+ *
+ * The client director picks based on client identity or IP-address
+ *
+ * In all cases, the choice is by weight of the healthy subset of
+ * configured backends.
+ *
+ * Failures to get a connection are retried, here all three policies
+ * fall back to a deterministically random choice, by weight in the
+ * healthy subset.
+ *
+ */
+
+#include "config.h"
+
+#include <math.h>
+#include <stdlib.h>
+
+#include "cache.h"
+
+#include "cache_backend.h"
+#include "vend.h"
+#include "vrt.h"
+#include "vsha256.h"
+
+/*--------------------------------------------------------------------*/
+
+struct vdi_random_host {
+	struct director		*backend;
+	double			weight;
+};
+
+enum crit_e {c_random, c_hash, c_client};
+
+struct vdi_random {
+	unsigned		magic;
+#define VDI_RANDOM_MAGIC	0x3771ae23
+	struct director		dir;
+
+	enum crit_e		criteria;
+	unsigned		retries;
+	double			tot_weight;
+	struct vdi_random_host	*hosts;
+	unsigned		nhosts;
+};
+
+/*
+ * Applies sha256 using the given context and input/length, and returns
+ * a double in the range [0...1[ based on the hash.
+ */
+static double
+vdi_random_sha(const char *input, ssize_t len)
+{
+	struct SHA256Context ctx;
+	uint8_t sign[SHA256_LEN];
+
+	AN(input);
+	SHA256_Init(&ctx);
+	SHA256_Update(&ctx, input, len);
+	SHA256_Final(sign, &ctx);
+	return (scalbn(vle32dec(sign), -32));
+}
+
+/*
+ * Sets up the initial seed for picking a backend according to policy.
+ */
+static double
+vdi_random_init_seed(const struct vdi_random *vs, const struct sess *sp)
+{
+	const char *p;
+	double retval;
+
+	switch (vs->criteria) {
+	case c_client:
+		if (sp->client_identity != NULL)
+			p = sp->client_identity;
+		else
+			p = sp->addr;
+		retval = vdi_random_sha(p, strlen(p));
+		break;
+	case c_hash:
+		AN(sp->digest);
+		retval = scalbn(vle32dec(sp->digest), -32);
+		break;
+	case c_random:
+	default:
+		retval = scalbn(random(), -31);
+		break;
+	}
+	return (retval);
+}
+
+/*
+ * Find the healthy backend corresponding to the weight r [0...1[
+ */
+static struct vbc *
+vdi_random_pick_one(struct sess *sp, const struct vdi_random *vs, double r)
+{
+	double w[vs->nhosts];
+	int i;
+	double s1;
+
+	assert(r >= 0.0 && r < 1.0);
+
+	memset(w, 0, sizeof w);
+	/* Sum up the weights of healty backends */
+	s1 = 0.0;
+	for (i = 0; i < vs->nhosts; i++) {
+		if (VDI_Healthy(vs->hosts[i].backend, sp))
+			w[i] = vs->hosts[i].weight;
+		s1 += w[i];
+	}
+
+	if (s1 == 0.0)
+		return (NULL);
+
+	r *= s1;
+	s1 = 0.0;
+	for (i = 0; i < vs->nhosts; i++)  {
+		s1 += w[i];
+		if (r < s1)
+			return(VDI_GetFd(vs->hosts[i].backend, sp));
+	}
+	return (NULL);
+}
+
+/*
+ * Try the specified number of times to get a backend.
+ * First one according to policy, after that, deterministically
+ * random by rehashing the key.
+ */
+static struct vbc *
+vdi_random_getfd(const struct director *d, struct sess *sp)
+{
+	int k;
+	struct vdi_random *vs;
+	double r;
+	struct vbc *vbe;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC);
+	CAST_OBJ_NOTNULL(vs, d->priv, VDI_RANDOM_MAGIC);
+
+	r = vdi_random_init_seed(vs, sp);
+
+	for (k = 0; k < vs->retries; k++) {
+		vbe = vdi_random_pick_one(sp, vs, r);
+		if (vbe != NULL)
+			return (vbe);
+		r = vdi_random_sha((void *)&r, sizeof(r));
+	}
+	return (NULL);
+}
+
+/*
+ * Healthy if just a single backend is...
+ */
+static unsigned
+vdi_random_healthy(const struct director *d, const struct sess *sp)
+{
+	struct vdi_random *vs;
+	int i;
+
+	CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC);
+	CAST_OBJ_NOTNULL(vs, d->priv, VDI_RANDOM_MAGIC);
+
+	for (i = 0; i < vs->nhosts; i++) {
+		if (VDI_Healthy(vs->hosts[i].backend, sp))
+			return (1);
+	}
+	return (0);
+}
+
+static void
+vdi_random_fini(const struct director *d)
+{
+	struct vdi_random *vs;
+
+	CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC);
+	CAST_OBJ_NOTNULL(vs, d->priv, VDI_RANDOM_MAGIC);
+
+	free(vs->hosts);
+	free(vs->dir.vcl_name);
+	vs->dir.magic = 0;
+	FREE_OBJ(vs);
+}
+
+static void
+vrt_init(struct cli *cli, struct director **bp, int idx,
+    const void *priv, enum crit_e criteria)
+{
+	const struct vrt_dir_random *t;
+	struct vdi_random *vs;
+	const struct vrt_dir_random_entry *te;
+	struct vdi_random_host *vh;
+	int i;
+
+	ASSERT_CLI();
+	(void)cli;
+	t = priv;
+
+	ALLOC_OBJ(vs, VDI_RANDOM_MAGIC);
+	XXXAN(vs);
+	vs->hosts = calloc(sizeof *vh, t->nmember);
+	XXXAN(vs->hosts);
+
+	vs->dir.magic = DIRECTOR_MAGIC;
+	vs->dir.priv = vs;
+	vs->dir.name = "random";
+	REPLACE(vs->dir.vcl_name, t->name);
+	vs->dir.getfd = vdi_random_getfd;
+	vs->dir.fini = vdi_random_fini;
+	vs->dir.healthy = vdi_random_healthy;
+
+	vs->criteria = criteria;
+	vs->retries = t->retries;
+	if (vs->retries == 0)
+		vs->retries = t->nmember;
+	vh = vs->hosts;
+	te = t->members;
+	vs->tot_weight = 0.;
+	for (i = 0; i < t->nmember; i++, vh++, te++) {
+		assert(te->weight > 0.0);
+		vh->weight = te->weight;
+		vs->tot_weight += vh->weight;
+		vh->backend = bp[te->host];
+		AN(vh->backend);
+	}
+	vs->nhosts = t->nmember;
+	bp[idx] = &vs->dir;
+}
+
+void
+VRT_init_dir_random(struct cli *cli, struct director **bp, int idx,
+    const void *priv)
+{
+	vrt_init(cli, bp, idx, priv, c_random);
+}
+
+void
+VRT_init_dir_hash(struct cli *cli, struct director **bp, int idx,
+    const void *priv)
+{
+	vrt_init(cli, bp, idx, priv, c_hash);
+}
+
+void
+VRT_init_dir_client(struct cli *cli, struct director **bp, int idx,
+    const void *priv)
+{
+	vrt_init(cli, bp, idx, priv, c_client);
+}
diff --git a/bin/varnishd/cache/cache_dir_round_robin.c b/bin/varnishd/cache/cache_dir_round_robin.c
new file mode 100644
index 0000000..7d75473
--- /dev/null
+++ b/bin/varnishd/cache/cache_dir_round_robin.c
@@ -0,0 +1,176 @@
+/*-
+ * Copyright (c) 2008-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Petter Knudsen <petter at linpro.no>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ */
+
+#include "config.h"
+
+#include <stdlib.h>
+
+#include "cache.h"
+
+#include "cache_backend.h"
+#include "vrt.h"
+
+/*--------------------------------------------------------------------*/
+
+struct vdi_round_robin_host {
+	struct director			*backend;
+};
+
+enum mode_e { m_round_robin, m_fallback };
+
+struct vdi_round_robin {
+	unsigned			magic;
+#define VDI_ROUND_ROBIN_MAGIC		0x2114a178
+	struct director			dir;
+	enum mode_e			mode;
+	struct vdi_round_robin_host	*hosts;
+	unsigned			nhosts;
+	unsigned			next_host;
+};
+
+static struct vbc *
+vdi_round_robin_getfd(const struct director *d, struct sess *sp)
+{
+	int i;
+	struct vdi_round_robin *vs;
+	struct director *backend;
+	struct vbc *vbe;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC);
+	CAST_OBJ_NOTNULL(vs, d->priv, VDI_ROUND_ROBIN_MAGIC);
+
+	/*
+	 * In fallback mode we ignore the next_host and always grab the
+	 * first healthy backend we can find.
+	 */
+	for (i = 0; i < vs->nhosts; i++) {
+		if (vs->mode == m_round_robin) {
+			backend = vs->hosts[vs->next_host].backend;
+			vs->next_host = (vs->next_host + 1) % vs->nhosts;
+		} else /* m_fallback */ {
+			backend = vs->hosts[i].backend;
+		}
+		if (!VDI_Healthy(backend, sp))
+			continue;
+		vbe = VDI_GetFd(backend, sp);
+		if (vbe != NULL)
+			return (vbe);
+	}
+
+	return (NULL);
+}
+
+static unsigned
+vdi_round_robin_healthy(const struct director *d, const struct sess *sp)
+{
+	struct vdi_round_robin *vs;
+	struct director *backend;
+	int i;
+
+	CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC);
+	CAST_OBJ_NOTNULL(vs, d->priv, VDI_ROUND_ROBIN_MAGIC);
+
+	for (i = 0; i < vs->nhosts; i++) {
+		backend = vs->hosts[i].backend;
+		if (VDI_Healthy(backend, sp))
+			return (1);
+	}
+	return (0);
+}
+
+static void
+vdi_round_robin_fini(const struct director *d)
+{
+	struct vdi_round_robin *vs;
+
+	CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC);
+	CAST_OBJ_NOTNULL(vs, d->priv, VDI_ROUND_ROBIN_MAGIC);
+
+	free(vs->hosts);
+	free(vs->dir.vcl_name);
+	vs->dir.magic = 0;
+	vs->next_host = 0;
+	FREE_OBJ(vs);
+}
+
+static void
+vrt_init_dir(struct cli *cli, struct director **bp, int idx,
+    const void *priv, enum mode_e mode)
+{
+	const struct vrt_dir_round_robin *t;
+	struct vdi_round_robin *vs;
+	const struct vrt_dir_round_robin_entry *te;
+	struct vdi_round_robin_host *vh;
+	int i;
+
+	ASSERT_CLI();
+	(void)cli;
+	t = priv;
+
+	ALLOC_OBJ(vs, VDI_ROUND_ROBIN_MAGIC);
+	XXXAN(vs);
+	vs->hosts = calloc(sizeof *vh, t->nmember);
+	XXXAN(vs->hosts);
+
+	vs->dir.magic = DIRECTOR_MAGIC;
+	vs->dir.priv = vs;
+	vs->dir.name = "round_robin";
+	REPLACE(vs->dir.vcl_name, t->name);
+	vs->dir.getfd = vdi_round_robin_getfd;
+	vs->dir.fini = vdi_round_robin_fini;
+	vs->dir.healthy = vdi_round_robin_healthy;
+
+	vs->mode = mode;
+	vh = vs->hosts;
+	te = t->members;
+	for (i = 0; i < t->nmember; i++, vh++, te++) {
+		vh->backend = bp[te->host];
+		AN (vh->backend);
+	}
+	vs->nhosts = t->nmember;
+	vs->next_host = 0;
+
+	bp[idx] = &vs->dir;
+}
+
+void
+VRT_init_dir_round_robin(struct cli *cli, struct director **bp, int idx,
+    const void *priv)
+{
+	vrt_init_dir(cli, bp, idx, priv, m_round_robin);
+}
+
+void
+VRT_init_dir_fallback(struct cli *cli, struct director **bp, int idx,
+    const void *priv)
+{
+	vrt_init_dir(cli, bp, idx, priv, m_fallback);
+}
+
diff --git a/bin/varnishd/cache/cache_esi.h b/bin/varnishd/cache/cache_esi.h
new file mode 100644
index 0000000..ff84d41
--- /dev/null
+++ b/bin/varnishd/cache/cache_esi.h
@@ -0,0 +1,48 @@
+/*-
+ * Copyright (c) 2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ */
+
+#define	VEC_GZ	(0x21)
+#define	VEC_V1	(0x40 + 1)
+#define	VEC_V2	(0x40 + 2)
+#define	VEC_V8	(0x40 + 8)
+#define	VEC_C1	(0x50 + 1)
+#define	VEC_C2	(0x50 + 2)
+#define	VEC_C8	(0x50 + 8)
+#define	VEC_S1	(0x60 + 1)
+#define	VEC_S2	(0x60 + 2)
+#define	VEC_S8	(0x60 + 8)
+#define	VEC_INCL	'I'
+
+typedef ssize_t vep_callback_t(struct worker *w, ssize_t l, enum vgz_flag flg);
+
+void VEP_Init(struct worker *w, vep_callback_t *cb);
+void VEP_Parse(const struct worker *w, const char *p, size_t l);
+struct vsb *VEP_Finish(struct worker *w);
+
+
diff --git a/bin/varnishd/cache/cache_esi_deliver.c b/bin/varnishd/cache/cache_esi_deliver.c
new file mode 100644
index 0000000..4051027
--- /dev/null
+++ b/bin/varnishd/cache/cache_esi_deliver.c
@@ -0,0 +1,570 @@
+/*-
+ * Copyright (c) 2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * VED - Varnish Esi Delivery
+ */
+
+#include "config.h"
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#include "cache.h"
+
+#include "cache_esi.h"
+#include "vend.h"
+#include "vgz.h"
+
+/*--------------------------------------------------------------------*/
+
+static void
+ved_include(struct sess *sp, const char *src, const char *host)
+{
+	struct object *obj;
+	struct worker *w;
+	char *sp_ws_wm;
+	char *wrk_ws_wm;
+	unsigned sxid, res_mode;
+
+	w = sp->wrk;
+
+	if (sp->esi_level >= cache_param->max_esi_depth)
+		return;
+	sp->esi_level++;
+
+	(void)WRW_FlushRelease(w);
+
+	obj = sp->obj;
+	sp->obj = NULL;
+	res_mode = sp->wrk->res_mode;
+
+	/* Reset request to status before we started messing with it */
+	HTTP_Copy(sp->http, sp->http0);
+
+	/* Take a workspace snapshot */
+	sp_ws_wm = WS_Snapshot(sp->ws);
+	wrk_ws_wm = WS_Snapshot(w->ws);
+
+	http_SetH(sp->http, HTTP_HDR_URL, src);
+	if (host != NULL && *host != '\0')  {
+		http_Unset(sp->http, H_Host);
+		http_Unset(sp->http, H_If_Modified_Since);
+		http_SetHeader(w, sp->vsl_id, sp->http, host);
+	}
+	/*
+	 * XXX: We should decide if we should cache the director
+	 * XXX: or not (for session/backend coupling).  Until then
+	 * XXX: make sure we don't trip up the check in vcl_recv.
+	 */
+	sp->director = NULL;
+	sp->step = STP_RECV;
+	http_ForceGet(sp->http);
+
+	/* Don't do conditionals */
+	sp->http->conds = 0;
+	http_Unset(sp->http, H_If_Modified_Since);
+
+	/* Client content already taken care of */
+	http_Unset(sp->http, H_Content_Length);
+
+	sp->wrk->do_esi = 0;
+	sp->wrk->is_gzip = 0;
+	sp->wrk->is_gunzip = 0;
+	sp->wrk->do_gzip = 0;
+	sp->wrk->do_gunzip = 0;
+	sp->wrk->do_stream = 0;
+
+	sxid = sp->xid;
+	while (1) {
+		sp->wrk = w;
+		CNT_Session(sp);
+		if (sp->step == STP_DONE)
+			break;
+		AZ(sp->wrk);
+		WSL_Flush(w, 0);
+		DSL(0x20, SLT_Debug, sp->vsl_id, "loop waiting for ESI");
+		(void)usleep(10000);
+	}
+	sp->xid = sxid;
+	AN(sp->wrk);
+	assert(sp->step == STP_DONE);
+	sp->esi_level--;
+	sp->obj = obj;
+	sp->wrk->res_mode = res_mode;
+
+	/* Reset the workspace */
+	WS_Reset(sp->ws, sp_ws_wm);
+	WS_Reset(w->ws, wrk_ws_wm);
+
+	WRW_Reserve(sp->wrk, &sp->fd);
+	if (sp->wrk->res_mode & RES_CHUNKED)
+		WRW_Chunked(sp->wrk);
+}
+
+/*--------------------------------------------------------------------*/
+
+
+//#define Debug(fmt, ...) printf(fmt, __VA_ARGS__)
+#define Debug(fmt, ...) /**/
+
+static ssize_t
+ved_decode_len(uint8_t **pp)
+{
+	uint8_t *p;
+	ssize_t l;
+
+	p = *pp;
+	switch (*p & 15) {
+	case 1:
+		l = p[1];
+		p += 2;
+		break;
+	case 2:
+		l = vbe16dec(p + 1);
+		p += 3;
+		break;
+	case 8:
+		l = vbe64dec(p + 1);
+		p += 9;
+		break;
+	default:
+		printf("Illegal Length %d %d\n", *p, (*p & 15));
+		INCOMPL();
+	}
+	*pp = p;
+	assert(l > 0);
+	return (l);
+}
+
+/*---------------------------------------------------------------------
+ * If a gzip'ed ESI object includes a ungzip'ed object, we need to make
+ * it looked like a gzip'ed data stream.  The official way to do so would
+ * be to fire up libvgz and gzip it, but we don't, we fake it.
+ *
+ * First, we cannot know if it is ungzip'ed on purpose, the admin may
+ * know something we don't.
+ *
+ * What do you mean "BS ?"
+ *
+ * All right then...
+ *
+ * The matter of the fact is that we simply will not fire up a gzip in
+ * the output path because it costs too much memory and CPU, so we simply
+ * wrap the data in very convenient "gzip copy-blocks" and send it down
+ * the stream with a bit more overhead.
+ */
+
+static void
+ved_pretend_gzip(const struct sess *sp, const uint8_t *p, ssize_t l)
+{
+	uint8_t buf1[5], buf2[5];
+	uint16_t lx;
+
+	lx = 65535;
+	buf1[0] = 0;
+	vle16enc(buf1 + 1, lx);
+	vle16enc(buf1 + 3, ~lx);
+
+	while (l > 0) {
+		if (l >= 65535) {
+			lx = 65535;
+			(void)WRW_Write(sp->wrk, buf1, sizeof buf1);
+		} else {
+			lx = (uint16_t)l;
+			buf2[0] = 0;
+			vle16enc(buf2 + 1, lx);
+			vle16enc(buf2 + 3, ~lx);
+			(void)WRW_Write(sp->wrk, buf2, sizeof buf2);
+		}
+		(void)WRW_Write(sp->wrk, p, lx);
+		sp->wrk->crc = crc32(sp->wrk->crc, p, lx);
+		sp->wrk->l_crc += lx;
+		l -= lx;
+		p += lx;
+	}
+	/* buf2 is local, have to flush */
+	(void)WRW_Flush(sp->wrk);
+}
+
+/*---------------------------------------------------------------------
+ */
+
+static const uint8_t gzip_hdr[] = {
+	0x1f, 0x8b, 0x08,
+	0x00, 0x00, 0x00, 0x00,
+	0x00,
+	0x02, 0x03
+};
+
+void
+ESI_Deliver(struct sess *sp)
+{
+	struct storage *st;
+	uint8_t *p, *e, *q, *r;
+	unsigned off;
+	ssize_t l, l2, l_icrc = 0;
+	uint32_t icrc = 0;
+	uint8_t tailbuf[8 + 5];
+	int isgzip;
+	struct vgz *vgz = NULL;
+	char obuf[cache_param->gzip_stack_buffer];
+	ssize_t obufl = 0;
+	size_t dl;
+	const void *dp;
+	int i;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	st = sp->obj->esidata;
+	AN(st);
+	assert(sizeof obuf >= 1024);
+
+	obuf[0] = 0;	/* For flexelint */
+
+	p = st->ptr;
+	e = st->ptr + st->len;
+
+	if (*p == VEC_GZ) {
+		isgzip = 1;
+		p++;
+	} else {
+		isgzip = 0;
+	}
+
+	if (sp->esi_level == 0) {
+		/*
+		 * Only the top level document gets to decide this.
+		 */
+		sp->wrk->gzip_resp = 0;
+		if (isgzip && !(sp->wrk->res_mode & RES_GUNZIP)) {
+			assert(sizeof gzip_hdr == 10);
+			/* Send out the gzip header */
+			(void)WRW_Write(sp->wrk, gzip_hdr, 10);
+			sp->wrk->l_crc = 0;
+			sp->wrk->gzip_resp = 1;
+			sp->wrk->crc = crc32(0L, Z_NULL, 0);
+		}
+	}
+
+	if (isgzip && !sp->wrk->gzip_resp) {
+		vgz = VGZ_NewUngzip(sp->wrk, "U D E");
+
+		/* Feed a gzip header to gunzip to make it happy */
+		VGZ_Ibuf(vgz, gzip_hdr, sizeof gzip_hdr);
+		VGZ_Obuf(vgz, obuf, sizeof obuf);
+		i = VGZ_Gunzip(vgz, &dp, &dl);
+		assert(i == VGZ_OK);
+		assert(VGZ_IbufEmpty(vgz));
+		assert(dl == 0);
+
+		obufl = 0;
+	}
+
+	st = VTAILQ_FIRST(&sp->obj->store);
+	off = 0;
+
+	while (p < e) {
+		switch (*p) {
+		case VEC_V1:
+		case VEC_V2:
+		case VEC_V8:
+			l = ved_decode_len(&p);
+			if (isgzip) {
+				assert(*p == VEC_C1 || *p == VEC_C2 ||
+				    *p == VEC_C8);
+				l_icrc = ved_decode_len(&p);
+				icrc = vbe32dec(p);
+				p += 4;
+				if (sp->wrk->gzip_resp) {
+					sp->wrk->crc = crc32_combine(
+					    sp->wrk->crc, icrc, l_icrc);
+					sp->wrk->l_crc += l_icrc;
+				}
+			}
+			/*
+			 * There is no guarantee that the 'l' bytes are all
+			 * in the same storage segment, so loop over storage
+			 * until we have processed them all.
+			 */
+			while (l > 0) {
+				l2 = l;
+				if (l2 > st->len - off)
+					l2 = st->len - off;
+				l -= l2;
+
+				if (sp->wrk->gzip_resp && isgzip) {
+					/*
+					 * We have a gzip'ed VEC and delivers
+					 * a gzip'ed ESI response.
+					 */
+					(void)WRW_Write(sp->wrk, st->ptr + off, l2);
+				} else if (sp->wrk->gzip_resp) {
+					/*
+					 * A gzip'ed ESI response, but the VEC
+					 * was not gzip'ed.
+					 */
+					ved_pretend_gzip(sp, st->ptr + off, l2);
+				} else if (isgzip) {
+					/*
+					 * A gzip'ed VEC, but ungzip'ed ESI
+					 * response
+					 */
+					AN(vgz);
+					i = VGZ_WrwGunzip(sp->wrk, vgz,
+						st->ptr + off, l2,
+						obuf, sizeof obuf, &obufl);
+					if (WRW_Error(sp->wrk)) {
+						SES_Close(sp, "remote closed");
+						p = e;
+						break;
+					}
+					assert (i == VGZ_OK || i == VGZ_END);
+				} else {
+					/*
+					 * Ungzip'ed VEC, ungzip'ed ESI response
+					 */
+					(void)WRW_Write(sp->wrk, st->ptr + off, l2);
+				}
+				off += l2;
+				if (off == st->len) {
+					st = VTAILQ_NEXT(st, list);
+					off = 0;
+				}
+			}
+			break;
+		case VEC_S1:
+		case VEC_S2:
+		case VEC_S8:
+			l = ved_decode_len(&p);
+			Debug("SKIP1(%d)\n", (int)l);
+			/*
+			 * There is no guarantee that the 'l' bytes are all
+			 * in the same storage segment, so loop over storage
+			 * until we have processed them all.
+			 */
+			while (l > 0) {
+				l2 = l;
+				if (l2 > st->len - off)
+					l2 = st->len - off;
+				l -= l2;
+				off += l2;
+				if (off == st->len) {
+					st = VTAILQ_NEXT(st, list);
+					off = 0;
+				}
+			}
+			break;
+		case VEC_INCL:
+			p++;
+			q = (void*)strchr((const char*)p, '\0');
+			AN(q);
+			q++;
+			r = (void*)strchr((const char*)q, '\0');
+			AN(r);
+			if (obufl > 0) {
+				(void)WRW_Write(sp->wrk, obuf, obufl);
+				obufl = 0;
+			}
+			if (WRW_Flush(sp->wrk)) {
+				SES_Close(sp, "remote closed");
+				p = e;
+				break;
+			}
+			Debug("INCL [%s][%s] BEGIN\n", q, p);
+			ved_include(sp, (const char*)q, (const char*)p);
+			Debug("INCL [%s][%s] END\n", q, p);
+			p = r + 1;
+			break;
+		default:
+			printf("XXXX 0x%02x [%s]\n", *p, p);
+			INCOMPL();
+		}
+	}
+	if (vgz != NULL) {
+		if (obufl > 0)
+			(void)WRW_Write(sp->wrk, obuf, obufl);
+		(void)VGZ_Destroy(&vgz, sp->vsl_id);
+	}
+	if (sp->wrk->gzip_resp && sp->esi_level == 0) {
+		/* Emit a gzip literal block with finish bit set */
+		tailbuf[0] = 0x01;
+		tailbuf[1] = 0x00;
+		tailbuf[2] = 0x00;
+		tailbuf[3] = 0xff;
+		tailbuf[4] = 0xff;
+
+		/* Emit CRC32 */
+		vle32enc(tailbuf + 5, sp->wrk->crc);
+
+		/* MOD(2^32) length */
+		vle32enc(tailbuf + 9, sp->wrk->l_crc);
+
+		(void)WRW_Write(sp->wrk, tailbuf, 13);
+	}
+	(void)WRW_Flush(sp->wrk);
+}
+
+/*---------------------------------------------------------------------
+ * Include an object in a gzip'ed ESI object delivery
+ */
+
+static uint8_t
+ved_deliver_byterange(const struct sess *sp, ssize_t low, ssize_t high)
+{
+	struct storage *st;
+	ssize_t l, lx;
+	u_char *p;
+
+//printf("BR %jd %jd\n", low, high);
+	lx = 0;
+	VTAILQ_FOREACH(st, &sp->obj->store, list) {
+		p = st->ptr;
+		l = st->len;
+//printf("[0-] %jd %jd\n", lx, lx + l);
+		if (lx + l < low) {
+			lx += l;
+			continue;
+		}
+		if (lx == high)
+			return (p[0]);
+		assert(lx < high);
+		if (lx < low) {
+			p += (low - lx);
+			l -= (low - lx);
+			lx = low;
+		}
+//printf("[1-] %jd %jd\n", lx, lx + l);
+		if (lx + l >= high)
+			l = high - lx;
+//printf("[2-] %jd %jd\n", lx, lx + l);
+		assert(lx >= low && lx + l <= high);
+		if (l != 0)
+			(void)WRW_Write(sp->wrk, p, l);
+		if (lx + st->len > high)
+			return(p[l]);
+		lx += st->len;
+	}
+	INCOMPL();
+}
+
+void
+ESI_DeliverChild(const struct sess *sp)
+{
+	struct storage *st;
+	struct object *obj;
+	ssize_t start, last, stop, lpad;
+	u_char *p, cc;
+	uint32_t icrc;
+	uint32_t ilen;
+	uint8_t *dbits;
+
+	if (!sp->obj->gziped) {
+		VTAILQ_FOREACH(st, &sp->obj->store, list)
+			ved_pretend_gzip(sp, st->ptr, st->len);
+		return;
+	}
+	/*
+	 * This is the interesting case: Deliver all the deflate
+	 * blocks, stripping the "LAST" bit of the last one and
+	 * padding it, as necessary, to a byte boundary.
+	 */
+
+	dbits = (void*)WS_Alloc(sp->wrk->ws, 8);
+	AN(dbits);
+	obj = sp->obj;
+	CHECK_OBJ_NOTNULL(obj, OBJECT_MAGIC);
+	start = obj->gzip_start;
+	last = obj->gzip_last;
+	stop = obj->gzip_stop;
+	assert(start > 0 && start < obj->len * 8);
+	assert(last > 0 && last < obj->len * 8);
+	assert(stop > 0 && stop < obj->len * 8);
+	assert(last >= start);
+	assert(last < stop);
+
+	/* The start bit must be byte aligned. */
+	AZ(start & 7);
+
+	/*
+	 * XXX: optimize for the case where the 'last'
+	 * XXX: bit is in a empty copy block
+	 */
+	*dbits = ved_deliver_byterange(sp, start/8, last/8);
+	*dbits &= ~(1U << (last & 7));
+	(void)WRW_Write(sp->wrk, dbits, 1);
+	cc = ved_deliver_byterange(sp, 1 + last/8, stop/8);
+	switch((int)(stop & 7)) {
+	case 0: /* xxxxxxxx */
+		/* I think we have an off by one here, but that's OK */
+		lpad = 0;
+		break;
+	case 1: /* x000.... 00000000 00000000 11111111 11111111 */
+	case 3: /* xxx000.. 00000000 00000000 11111111 11111111 */
+	case 5: /* xxxxx000 00000000 00000000 11111111 11111111 */
+		dbits[1] = cc | 0x00;
+		dbits[2] = 0x00; dbits[3] = 0x00;
+	        dbits[4] = 0xff; dbits[5] = 0xff;
+		lpad = 5;
+		break;
+	case 2: /* xx010000 00000100 00000001 00000000 */
+		dbits[1] = cc | 0x08;
+		dbits[2] = 0x20;
+		dbits[3] = 0x80;
+		dbits[4] = 0x00;
+		lpad = 4;
+		break;
+	case 4: /* xxxx0100 00000001 00000000 */
+		dbits[1] = cc | 0x20;
+		dbits[2] = 0x80;
+		dbits[3] = 0x00;
+		lpad = 3;
+		break;
+	case 6: /* xxxxxx01 00000000 */
+		dbits[1] = cc | 0x80;
+		dbits[2] = 0x00;
+		lpad = 2;
+		break;
+	case 7:	/* xxxxxxx0 00...... 00000000 00000000 11111111 11111111 */
+		dbits[1] = cc | 0x00;
+		dbits[2] = 0x00;
+		dbits[3] = 0x00; dbits[4] = 0x00;
+	        dbits[5] = 0xff; dbits[6] = 0xff;
+		lpad = 6;
+		break;
+	default:
+		INCOMPL();
+	}
+	if (lpad > 0)
+		(void)WRW_Write(sp->wrk, dbits + 1, lpad);
+	st = VTAILQ_LAST(&sp->obj->store, storagehead);
+	assert(st->len > 8);
+
+	p = st->ptr + st->len - 8;
+	icrc = vle32dec(p);
+	ilen = vle32dec(p + 4);
+	sp->wrk->crc = crc32_combine(sp->wrk->crc, icrc, ilen);
+	sp->wrk->l_crc += ilen;
+}
diff --git a/bin/varnishd/cache/cache_esi_fetch.c b/bin/varnishd/cache/cache_esi_fetch.c
new file mode 100644
index 0000000..5ec8f6b
--- /dev/null
+++ b/bin/varnishd/cache/cache_esi_fetch.c
@@ -0,0 +1,405 @@
+/*-
+ * Copyright (c) 2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * VEF Varnish Esi Fetching
+ */
+
+#include "config.h"
+
+#include <stdlib.h>
+
+#include "cache.h"
+
+#include "cache_esi.h"
+
+/*---------------------------------------------------------------------
+ * Read some bytes.
+ *
+ * If the esi_syntax&8 bit is set, we read only a couple of bytes at
+ * a time, in order to stress the parse/pending/callback code.
+ */
+
+static ssize_t
+vef_read(struct worker *w, struct http_conn *htc, void *buf, ssize_t buflen,
+    ssize_t bytes)
+{
+	ssize_t d;
+
+	if (buflen < bytes)
+		bytes = buflen;
+	if (cache_param->esi_syntax & 0x8) {
+		d = (random() & 3) + 1;
+		if (d < bytes)
+			bytes = d;
+	}
+	return (HTC_Read(w, htc, buf, bytes));
+}
+
+/*---------------------------------------------------------------------
+ * We receive a ungzip'ed object, and want to store it ungzip'ed.
+ */
+
+static int
+vfp_esi_bytes_uu(struct worker *w, struct http_conn *htc, ssize_t bytes)
+{
+	ssize_t wl;
+	struct storage *st;
+
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+
+	while (bytes > 0) {
+		st = FetchStorage(w, 0);
+		if (st == NULL)
+			return (-1);
+		wl = vef_read(w, htc,
+		    st->ptr + st->len, st->space - st->len, bytes);
+		if (wl <= 0)
+			return (wl);
+		VEP_Parse(w, (const char *)st->ptr + st->len, wl);
+		st->len += wl;
+		w->fetch_obj->len += wl;
+		bytes -= wl;
+	}
+	return (1);
+}
+
+/*---------------------------------------------------------------------
+ * We receive a gzip'ed object, and want to store it ungzip'ed.
+ */
+
+static int
+vfp_esi_bytes_gu(struct worker *w, struct http_conn *htc, ssize_t bytes)
+{
+	struct vgz *vg;
+	ssize_t wl;
+	uint8_t	ibuf[cache_param->gzip_stack_buffer];
+	int i;
+	size_t dl;
+	const void *dp;
+
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	vg = w->vgz_rx;
+
+	while (bytes > 0) {
+		if (VGZ_IbufEmpty(vg) && bytes > 0) {
+			wl = vef_read(w, htc, ibuf, sizeof ibuf, bytes);
+			if (wl <= 0)
+				return (wl);
+			VGZ_Ibuf(vg, ibuf, wl);
+			bytes -= wl;
+		}
+		if (VGZ_ObufStorage(w, vg))
+			return(-1);
+		i = VGZ_Gunzip(vg, &dp, &dl);
+		xxxassert(i == VGZ_OK || i == VGZ_END);
+		VEP_Parse(w, dp, dl);
+		w->fetch_obj->len += dl;
+	}
+	return (1);
+}
+
+/*---------------------------------------------------------------------
+ */
+
+struct vef_priv {
+	unsigned		magic;
+#define VEF_MAGIC		0xf104b51f
+	struct vgz		*vgz;
+
+	char			*bufp;
+	ssize_t			tot;
+	int			error;
+	char			pending[20];
+	ssize_t			npend;
+};
+
+/*---------------------------------------------------------------------
+ * We receive a [un]gzip'ed object, and want to store it gzip'ed.
+ */
+
+static ssize_t
+vfp_vep_callback(struct worker *w, ssize_t l, enum vgz_flag flg)
+{
+	struct vef_priv *vef;
+	size_t dl, px;
+	const void *dp;
+	int i;
+
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	vef = w->vef_priv;
+	CHECK_OBJ_NOTNULL(vef, VEF_MAGIC);
+	assert(l >= 0);
+
+	if (vef->error) {
+		vef->tot += l;
+		return (vef->tot);
+	}
+
+	/*
+	 * l == 0 is valid when 'flg' calls for action, but in the
+	 * normal case we can just ignore a l==0 request.
+	 * (It would cause Z_BUF_ERROR anyway)
+	 */
+	if (l == 0 && flg == VGZ_NORMAL)
+		return (vef->tot);
+
+	do {
+		px = vef->npend;
+		if (l < px)
+			px = l;
+		if (px != 0) {
+			VGZ_Ibuf(vef->vgz, vef->pending, px);
+			l -= px;
+		} else {
+			VGZ_Ibuf(vef->vgz, vef->bufp, l);
+			vef->bufp += l;
+			l = 0;
+		}
+		do {
+			if (VGZ_ObufStorage(w, vef->vgz)) {
+				vef->error = ENOMEM;
+				vef->tot += l;
+				return (vef->tot);
+			}
+			i = VGZ_Gzip(vef->vgz, &dp, &dl, flg);
+			vef->tot += dl;
+			w->fetch_obj->len += dl;
+		} while (!VGZ_IbufEmpty(vef->vgz) ||
+		    (flg != VGZ_NORMAL && VGZ_ObufFull(vef->vgz)));
+		if (px != 0) {
+			memmove(vef->pending, vef->pending + px,
+			    vef->npend - px);
+			vef->npend -= px;
+		}
+	} while (l > 0);
+	if (flg == VGZ_FINISH)
+		assert(i == 1);			/* XXX */
+	else
+		assert(i == 0);			/* XXX */
+	return (vef->tot);
+}
+
+static int
+vfp_esi_bytes_ug(struct worker *w, struct http_conn *htc, ssize_t bytes)
+{
+	ssize_t wl;
+	char ibuf[cache_param->gzip_stack_buffer];
+	struct vef_priv *vef;
+
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	vef = w->vef_priv;
+	CHECK_OBJ_NOTNULL(vef, VEF_MAGIC);
+
+	while (bytes > 0) {
+		wl = vef_read(w, htc, ibuf, sizeof ibuf, bytes);
+		if (wl <= 0)
+			return (wl);
+		bytes -= wl;
+		vef->bufp = ibuf;
+		VEP_Parse(w, ibuf, wl);
+		assert(vef->bufp >= ibuf && vef->bufp <= ibuf + wl);
+		if (vef->error) {
+			errno = vef->error;
+			return (-1);
+		}
+		if (vef->bufp < ibuf + wl) {
+			wl = (ibuf + wl) - vef->bufp;
+			assert(wl + vef->npend < sizeof vef->pending);
+			memmove(vef->pending + vef->npend, vef->bufp, wl);
+			vef->npend += wl;
+		}
+	}
+	return (1);
+}
+
+/*---------------------------------------------------------------------
+ * We receive a gzip'ed object, and want to store it gzip'ed.
+ */
+
+static int
+vfp_esi_bytes_gg(struct worker *w, struct http_conn *htc, size_t bytes)
+{
+	ssize_t wl;
+	char ibuf[cache_param->gzip_stack_buffer];
+	char ibuf2[cache_param->gzip_stack_buffer];
+	struct vef_priv *vef;
+	size_t dl;
+	const void *dp;
+	int i;
+
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	vef = w->vef_priv;
+	CHECK_OBJ_NOTNULL(vef, VEF_MAGIC);
+	assert(sizeof ibuf >= 1024);
+	ibuf2[0] = 0; /* For Flexelint */
+
+	while (bytes > 0) {
+		wl = vef_read(w, htc, ibuf, sizeof ibuf, bytes);
+		if (wl <= 0)
+			return (wl);
+		bytes -= wl;
+
+		vef->bufp = ibuf;
+		VGZ_Ibuf(w->vgz_rx, ibuf, wl);
+		do {
+			VGZ_Obuf(w->vgz_rx, ibuf2, sizeof ibuf2);
+			i = VGZ_Gunzip(w->vgz_rx, &dp, &dl);
+			/* XXX: check i */
+			assert(i >= VGZ_OK);
+			vef->bufp = ibuf2;
+			if (dl > 0)
+				VEP_Parse(w, ibuf2, dl);
+			if (vef->error) {
+				errno = vef->error;
+				return (-1);
+			}
+			if (vef->bufp < ibuf2 + dl) {
+				dl = (ibuf2 + dl) - vef->bufp;
+				assert(dl + vef->npend < sizeof vef->pending);
+				memmove(vef->pending + vef->npend,
+				    vef->bufp, dl);
+				vef->npend += dl;
+			}
+		} while (!VGZ_IbufEmpty(w->vgz_rx));
+	}
+	return (1);
+}
+
+
+/*---------------------------------------------------------------------*/
+
+static void __match_proto__()
+vfp_esi_begin(struct worker *w, size_t estimate)
+{
+	struct vef_priv *vef;
+
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+
+	AZ(w->vgz_rx);
+	if (w->is_gzip && w->do_gunzip) {
+		w->vgz_rx = VGZ_NewUngzip(w, "U F E");
+		VEP_Init(w, NULL);
+	} else if (w->is_gunzip && w->do_gzip) {
+		ALLOC_OBJ(vef, VEF_MAGIC);
+		AN(vef);
+		vef->vgz = VGZ_NewGzip(w, "G F E");
+		AZ(w->vef_priv);
+		w->vef_priv = vef;
+		VEP_Init(w, vfp_vep_callback);
+	} else if (w->is_gzip) {
+		w->vgz_rx = VGZ_NewUngzip(w, "U F E");
+		ALLOC_OBJ(vef, VEF_MAGIC);
+		AN(vef);
+		vef->vgz = VGZ_NewGzip(w, "G F E");
+		AZ(w->vef_priv);
+		w->vef_priv = vef;
+		VEP_Init(w, vfp_vep_callback);
+	} else {
+		AZ(w->vef_priv);
+		VEP_Init(w, NULL);
+	}
+
+	(void)estimate;
+	AN(w->vep);
+}
+
+static int __match_proto__()
+vfp_esi_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
+{
+	int i;
+
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	AZ(w->fetch_failed);
+	AN(w->vep);
+	assert(w->htc == htc);
+	if (w->is_gzip && w->do_gunzip)
+		i = vfp_esi_bytes_gu(w, htc, bytes);
+	else if (w->is_gunzip && w->do_gzip)
+		i = vfp_esi_bytes_ug(w, htc, bytes);
+	else if (w->is_gzip)
+		i = vfp_esi_bytes_gg(w, htc, bytes);
+	else
+		i = vfp_esi_bytes_uu(w, htc, bytes);
+	AN(w->vep);
+	return (i);
+}
+
+static int __match_proto__()
+vfp_esi_end(struct worker *w)
+{
+	struct vsb *vsb;
+	struct vef_priv *vef;
+	ssize_t l;
+	int retval;
+
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	AN(w->vep);
+
+	retval = w->fetch_failed;
+
+	if (w->vgz_rx != NULL && VGZ_Destroy(&w->vgz_rx, -1) != VGZ_END)
+		retval = FetchError(w,
+		    "Gunzip+ESI Failed at the very end");
+
+	vsb = VEP_Finish(w);
+
+	if (vsb != NULL) {
+		if (!retval) {
+			l = VSB_len(vsb);
+			assert(l > 0);
+			/* XXX: This is a huge waste of storage... */
+			w->fetch_obj->esidata = STV_alloc(w, l);
+			if (w->fetch_obj->esidata != NULL) {
+				memcpy(w->fetch_obj->esidata->ptr,
+				    VSB_data(vsb), l);
+				w->fetch_obj->esidata->len = l;
+			} else {
+				retval = FetchError(w,
+				    "Could not allocate storage for esidata");
+			}
+		}
+		VSB_delete(vsb);
+	}
+
+	if (w->vef_priv != NULL) {
+		vef = w->vef_priv;
+		CHECK_OBJ_NOTNULL(vef, VEF_MAGIC);
+		w->vef_priv = NULL;
+		VGZ_UpdateObj(vef->vgz, w->fetch_obj);
+		if (VGZ_Destroy(&vef->vgz,  -1) != VGZ_END)
+			retval = FetchError(w,
+			    "ESI+Gzip Failed at the very end");
+		FREE_OBJ(vef);
+	}
+	return (retval);
+}
+
+struct vfp vfp_esi = {
+        .begin  =       vfp_esi_begin,
+        .bytes  =       vfp_esi_bytes,
+        .end    =       vfp_esi_end,
+};
diff --git a/bin/varnishd/cache/cache_esi_parse.c b/bin/varnishd/cache/cache_esi_parse.c
new file mode 100644
index 0000000..9e2b4f6
--- /dev/null
+++ b/bin/varnishd/cache/cache_esi_parse.c
@@ -0,0 +1,1189 @@
+/*-
+ * Copyright (c) 2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * VEP Varnish Esi Parsing
+ */
+
+#include "config.h"
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#include "cache.h"
+
+#include "cache_esi.h"
+#include "vct.h"
+#include "vend.h"
+#include "vgz.h"
+
+//#define Debug(fmt, ...) printf(fmt, __VA_ARGS__)
+#define Debug(fmt, ...) /**/
+
+struct vep_state;
+
+enum dowhat {DO_ATTR, DO_TAG};
+typedef void dostuff_f(struct vep_state *, enum dowhat);
+
+struct vep_match {
+	const char	*match;
+	const char	* const *state;
+};
+
+enum vep_mark { VERBATIM = 0, SKIP };
+
+struct vep_state {
+	unsigned		magic;
+#define VEP_MAGIC		0x55cb9b82
+	struct vsb		*vsb;
+
+	struct worker		*wrk;
+	int			dogzip;
+	vep_callback_t		*cb;
+
+	/* Internal Counter for default call-back function */
+	ssize_t			cb_x;
+
+	/* parser state */
+	const char		*state;
+	unsigned		startup;
+	unsigned		esi_found;
+
+	unsigned		endtag;
+	unsigned		emptytag;
+	unsigned		canattr;
+
+	unsigned		remove;
+
+	ssize_t			o_wait;
+	ssize_t			o_pending;
+	ssize_t			o_total;
+	uint32_t		crc;
+	ssize_t			o_crc;
+	uint32_t		crcp;
+	ssize_t			o_last;
+
+const char		*hack_p;
+	const char		*ver_p;
+
+	const char		*until;
+	const char		*until_p;
+	const char		*until_s;
+
+	int			in_esi_tag;
+
+	const char		*esicmt;
+	const char		*esicmt_p;
+
+	struct vep_match	*attr;
+	struct vsb		*attr_vsb;
+	int			attr_delim;
+
+	struct vep_match	*match;
+	struct vep_match	*match_hit;
+
+	char			tag[10];
+	int			tag_i;
+
+	dostuff_f		*dostuff;
+
+	struct vsb		*include_src;
+
+	unsigned		nm_skip;
+	unsigned		nm_verbatim;
+	unsigned		nm_pending;
+	enum vep_mark		last_mark;
+};
+
+/*---------------------------------------------------------------------*/
+
+static const char * const VEP_START =		"[Start]";
+static const char * const VEP_TESTXML =		"[TestXml]";
+static const char * const VEP_NOTXML =		"[NotXml]";
+
+static const char * const VEP_NEXTTAG =		"[NxtTag]";
+static const char * const VEP_NOTMYTAG =	"[NotMyTag]";
+
+static const char * const VEP_STARTTAG =	"[StartTag]";
+static const char * const VEP_COMMENT =		"[Comment]";
+static const char * const VEP_CDATA =		"[CDATA]";
+static const char * const VEP_ESITAG =		"[ESITag]";
+
+static const char * const VEP_ESIREMOVE =	"[ESI:Remove]";
+static const char * const VEP_ESIINCLUDE =	"[ESI:Include]";
+static const char * const VEP_ESICOMMENT =	"[ESI:Comment]";
+static const char * const VEP_ESIBOGON =	"[ESI:Bogon]";
+
+static const char * const VEP_INTAG =		"[InTag]";
+static const char * const VEP_TAGERROR =	"[TagError]";
+
+static const char * const VEP_ATTR =		"[Attribute]";
+static const char * const VEP_SKIPATTR =	"[SkipAttribute]";
+static const char * const VEP_ATTRDELIM =	"[AttrDelim]";
+static const char * const VEP_ATTRGETVAL =	"[AttrGetValue]";
+static const char * const VEP_ATTRVAL =		"[AttrValue]";
+
+static const char * const VEP_UNTIL =		"[Until]";
+static const char * const VEP_MATCHBUF =	"[MatchBuf]";
+static const char * const VEP_MATCH =		"[Match]";
+
+/*---------------------------------------------------------------------*/
+
+static struct vep_match vep_match_starttag[] = {
+	{ "!--",	&VEP_COMMENT },
+	{ "esi:",	&VEP_ESITAG },
+	{ "![CDATA[",	&VEP_CDATA },
+	{ NULL,		&VEP_NOTMYTAG }
+};
+
+/*---------------------------------------------------------------------*/
+
+static struct vep_match vep_match_esi[] = {
+	{ "include",	&VEP_ESIINCLUDE },
+	{ "remove",	&VEP_ESIREMOVE },
+	{ "comment",	&VEP_ESICOMMENT },
+	{ NULL,		&VEP_ESIBOGON }
+};
+
+/*---------------------------------------------------------------------*/
+
+static struct vep_match vep_match_attr_include[] = {
+	{ "src=",	&VEP_ATTRGETVAL },
+	{ NULL,		&VEP_SKIPATTR }
+};
+
+/*--------------------------------------------------------------------
+ * Report a parsing error
+ */
+
+static void
+vep_error(const struct vep_state *vep, const char *p)
+{
+	intmax_t l;
+
+	VSC_C_main->esi_errors++;
+	l = (intmax_t)(vep->ver_p - vep->hack_p);
+	WSLB(vep->wrk, SLT_ESI_xmlerror, "ERR at %jd %s", l, p);
+
+}
+
+/*--------------------------------------------------------------------
+ * Report a parsing warning
+ */
+
+static void
+vep_warn(const struct vep_state *vep, const char *p)
+{
+	intmax_t l;
+
+	VSC_C_main->esi_warnings++;
+	l = (intmax_t)(vep->ver_p - vep->hack_p);
+	printf("WARNING at %jd %s\n", l, p);
+	WSLB(vep->wrk, SLT_ESI_xmlerror, "WARN at %jd %s", l, p);
+
+}
+
+/*---------------------------------------------------------------------
+ * return match or NULL if more input needed.
+ */
+
+static struct vep_match *
+vep_match(struct vep_state *vep, const char *b, const char *e)
+{
+	struct vep_match *vm;
+	const char *q, *r;
+	ssize_t l;
+
+	for (vm = vep->match; vm->match; vm++) {
+		r = b;
+		for (q = vm->match; *q && r < e; q++, r++)
+			if (*q != *r)
+				break;
+		if (*q != '\0' && r == e) {
+			if (b != vep->tag) {
+				l = e - b;
+				assert(l < sizeof vep->tag);
+				memmove(vep->tag, b, l);
+				vep->tag_i = l;
+			}
+			return (NULL);
+		}
+		if (*q == '\0')
+			return (vm);
+	}
+	return (vm);
+}
+
+/*---------------------------------------------------------------------
+ *
+ */
+
+static void
+vep_emit_len(const struct vep_state *vep, ssize_t l, int m8, int m16, int m64)
+{
+	uint8_t buf[9];
+
+	assert(l > 0);
+	if (l < 256) {
+		buf[0] = (uint8_t)m8;
+		buf[1] = (uint8_t)l;
+		assert((ssize_t)buf[1] == l);
+		VSB_bcat(vep->vsb, buf, 2);
+	} else if (l < 65536) {
+		buf[0] = (uint8_t)m16;
+		vbe16enc(buf + 1, (uint16_t)l);
+		assert((ssize_t)vbe16dec(buf + 1) == l);
+		VSB_bcat(vep->vsb, buf, 3);
+	} else {
+		buf[0] = (uint8_t)m64;
+		vbe64enc(buf + 1, l);
+		assert((ssize_t)vbe64dec(buf + 1) == l);
+		VSB_bcat(vep->vsb, buf, 9);
+	}
+}
+
+static void
+vep_emit_skip(const struct vep_state *vep, ssize_t l)
+{
+
+	if (cache_param->esi_syntax & 0x20) {
+		Debug("---> SKIP(%jd)\n", (intmax_t)l);
+	}
+	vep_emit_len(vep, l, VEC_S1, VEC_S2, VEC_S8);
+}
+
+static void
+vep_emit_verbatim(const struct vep_state *vep, ssize_t l, ssize_t l_crc)
+{
+	uint8_t buf[4];
+
+	if (cache_param->esi_syntax & 0x20) {
+		Debug("---> VERBATIM(%jd)\n", (intmax_t)l);
+	}
+	vep_emit_len(vep, l, VEC_V1, VEC_V2, VEC_V8);
+	if (vep->dogzip) {
+		vep_emit_len(vep, l_crc, VEC_C1, VEC_C2, VEC_C8);
+		vbe32enc(buf, vep->crc);
+		VSB_bcat(vep->vsb, buf, sizeof buf);
+	}
+}
+
+static void
+vep_emit_common(struct vep_state *vep, ssize_t l, enum vep_mark mark)
+{
+
+	assert(l > 0);
+	assert(mark == SKIP || mark == VERBATIM);
+	if (mark == SKIP)
+		vep_emit_skip(vep, l);
+	else
+		vep_emit_verbatim(vep, l, vep->o_crc);
+
+	vep->crc = crc32(0L, Z_NULL, 0);
+	vep->o_crc = 0;
+	vep->o_total += l;
+}
+
+/*---------------------------------------------------------------------
+ *
+ */
+
+static void
+vep_mark_common(struct vep_state *vep, const char *p, enum vep_mark mark)
+{
+	ssize_t l, lcb;
+
+	assert(mark == SKIP || mark == VERBATIM);
+
+	/* The NO-OP case, no data, no pending data & no change of mode */
+	if (vep->last_mark == mark && p == vep->ver_p && vep->o_pending == 0)
+		return;
+
+	/*
+	 * If we changed mode, emit whatever the opposite mode
+	 * assembled before the pending bytes.
+	 */
+
+	if (vep->last_mark != mark && (vep->o_wait > 0 || vep->startup)) {
+		lcb = vep->cb(vep->wrk, 0,
+		    mark == VERBATIM ? VGZ_RESET : VGZ_ALIGN);
+		if (lcb - vep->o_last > 0)
+			vep_emit_common(vep, lcb - vep->o_last, vep->last_mark);
+		vep->o_last = lcb;
+		vep->o_wait = 0;
+	}
+
+	/* Transfer pending bytes CRC into active mode CRC */
+	if (vep->o_pending) {
+		(void)vep->cb(vep->wrk, vep->o_pending, VGZ_NORMAL);
+		if (vep->o_crc == 0) {
+			vep->crc = vep->crcp;
+			vep->o_crc = vep->o_pending;
+		} else {
+			vep->crc = crc32_combine(vep->crc,
+			    vep->crcp, vep->o_pending);
+			vep->o_crc += vep->o_pending;
+		}
+		vep->crcp = crc32(0L, Z_NULL, 0);
+		vep->o_wait += vep->o_pending;
+		vep->o_pending = 0;
+	}
+
+	/* * Process this bit of input */
+	AN(vep->ver_p);
+	l = p - vep->ver_p;
+	assert(l >= 0);
+	vep->crc = crc32(vep->crc, (const void*)vep->ver_p, l);
+	vep->o_crc += l;
+	vep->ver_p = p;
+
+	vep->o_wait += l;
+	vep->last_mark = mark;
+	(void)vep->cb(vep->wrk, l, VGZ_NORMAL);
+}
+
+static void
+vep_mark_verbatim(struct vep_state *vep, const char *p)
+{
+
+	vep_mark_common(vep, p, VERBATIM);
+	vep->nm_verbatim++;
+}
+
+static void
+vep_mark_skip(struct vep_state *vep, const char *p)
+{
+
+	vep_mark_common(vep, p, SKIP);
+	vep->nm_skip++;
+}
+
+static void
+vep_mark_pending(struct vep_state *vep, const char *p)
+{
+	ssize_t l;
+
+	AN(vep->ver_p);
+	l = p - vep->ver_p;
+	assert(l > 0);
+	assert(l >= 0);
+	vep->crcp = crc32(vep->crcp, (const void *)vep->ver_p, l);
+	vep->ver_p = p;
+
+	vep->o_pending += l;
+	vep->nm_pending++;
+}
+
+/*---------------------------------------------------------------------
+ */
+
+static void __match_proto__()
+vep_do_comment(struct vep_state *vep, enum dowhat what)
+{
+	Debug("DO_COMMENT(%d)\n", what);
+	assert(what == DO_TAG);
+	if (!vep->emptytag)
+		vep_error(vep, "ESI 1.0 <esi:comment> needs final '/'");
+}
+
+/*---------------------------------------------------------------------
+ */
+
+static void __match_proto__()
+vep_do_remove(struct vep_state *vep, enum dowhat what)
+{
+	Debug("DO_REMOVE(%d, end %d empty %d remove %d)\n",
+	    what, vep->endtag, vep->emptytag, vep->remove);
+	assert(what == DO_TAG);
+	if (vep->emptytag) {
+		vep_error(vep,
+		    "ESI 1.0 <esi:remove/> not legal");
+	} else {
+		if (vep->remove && !vep->endtag)
+			vep_error(vep,
+			    "ESI 1.0 <esi:remove> already open");
+		else if (!vep->remove && vep->endtag)
+			vep_error(vep,
+			    "ESI 1.0 <esi:remove> not open");
+		else
+			vep->remove = !vep->endtag;
+	}
+}
+
+/*---------------------------------------------------------------------
+ */
+
+static void __match_proto__()
+vep_do_include(struct vep_state *vep, enum dowhat what)
+{
+	char *p, *q, *h;
+	ssize_t l;
+	txt url;
+
+	Debug("DO_INCLUDE(%d)\n", what);
+	if (what == DO_ATTR) {
+		Debug("ATTR (%s) (%s)\n", vep->match_hit->match,
+			VSB_data(vep->attr_vsb));
+		if (vep->include_src != NULL) {
+			vep_error(vep,
+			    "ESI 1.0 <esi:include> "
+			    "has multiple src= attributes");
+			vep->state = VEP_TAGERROR;
+			VSB_delete(vep->attr_vsb);
+			VSB_delete(vep->include_src);
+			vep->attr_vsb = NULL;
+			vep->include_src = NULL;
+			return;
+		}
+		XXXAZ(vep->include_src);	/* multiple src= */
+		vep->include_src = vep->attr_vsb;
+		return;
+	}
+	assert(what == DO_TAG);
+	if (!vep->emptytag)
+		vep_warn(vep,
+		    "ESI 1.0 <esi:include> lacks final '/'");
+	if (vep->include_src == NULL) {
+		vep_error(vep,
+		    "ESI 1.0 <esi:include> lacks src attr");
+		return;
+	}
+
+	/*
+	 * Strictly speaking, we ought to spit out any piled up skip before
+	 * emitting the VEC for the include, but objectively that makes no
+	 * difference and robs us of a chance to collapse another skip into
+	 * this on so we don't do that.
+	 * However, we cannot tolerate any verbatim stuff piling up.
+	 * The mark_skip() before calling dostuff should have taken
+	 * care of that.  Make sure.
+	 */
+	assert(vep->o_wait == 0 || vep->last_mark == SKIP);
+	/* XXX: what if it contains NUL bytes ?? */
+	p = VSB_data(vep->include_src);
+	l = VSB_len(vep->include_src);
+	h = 0;
+
+	VSB_printf(vep->vsb, "%c", VEC_INCL);
+	if (l > 7 && !memcmp(p, "http://", 7)) {
+		h = p + 7;
+		p = strchr(h, '/');
+		AN(p);
+		Debug("HOST <%.*s> PATH <%s>\n", (int)(p-h),h, p);
+		VSB_printf(vep->vsb, "Host: %.*s%c",
+		    (int)(p-h), h, 0);
+	} else if (*p == '/') {
+		VSB_printf(vep->vsb, "%c", 0);
+	} else {
+		VSB_printf(vep->vsb, "%c", 0);
+		url = vep->wrk->bereq->hd[HTTP_HDR_URL];
+		/* Look for the last / before a '?' */
+		h = NULL;
+		for (q = url.b; q < url.e && *q != '?'; q++)
+			if (*q == '/')
+				h = q;
+		if (h == NULL)
+			h = q + 1;
+
+		Debug("INCL:: [%.*s]/[%s]\n",
+		    (int)(h - url.b), url.b, p);
+		VSB_printf(vep->vsb, "%.*s/", (int)(h - url.b), url.b);
+	}
+	l -= (p - VSB_data(vep->include_src));
+	for (q = p; *q != '\0'; ) {
+		if (*q == '&') {
+#define R(w,f,r)							\
+			if (q + w <= p + l && !memcmp(q, f, w)) { \
+				VSB_printf(vep->vsb, "%c", r);	\
+				q += w;				\
+				continue;			\
+			}
+			R(6, "&apos;", '\'');
+			R(6, "&quot;", '"');
+			R(4, "&lt;", '<');
+			R(4, "&gt;", '>');
+			R(5, "&amp;", '&');
+		}
+		VSB_printf(vep->vsb, "%c", *q++);
+	}
+#undef R
+	VSB_printf(vep->vsb, "%c", 0);
+
+	VSB_delete(vep->include_src);
+	vep->include_src = NULL;
+}
+
+/*---------------------------------------------------------------------
+ * Lex/Parse object for ESI instructions
+ *
+ * This function is called with the input object piecemal so do not
+ * assume that we have more than one char available at at time, but
+ * optimize for getting huge chunks.
+ *
+ * NB: At the bottom of this source-file, there is a dot-diagram matching
+ * NB: the state-machine.  Please maintain it along with the code.
+ */
+
+void
+VEP_Parse(const struct worker *w, const char *p, size_t l)
+{
+	struct vep_state *vep;
+	const char *e;
+	struct vep_match *vm;
+	int i;
+
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	vep = w->vep;
+	CHECK_OBJ_NOTNULL(vep, VEP_MAGIC);
+	assert(l > 0);
+
+	/* XXX: Really need to fix this */
+	if (vep->hack_p == NULL)
+		vep->hack_p = p;
+
+	vep->ver_p = p;
+
+	e = p + l;
+
+	while (p < e) {
+		AN(vep->state);
+		i = e - p;
+		if (i > 10)
+			i = 10;
+		Debug("EP %s %d (%.*s) [%.*s]\n",
+		    vep->state,
+		    vep->remove,
+		    vep->tag_i, vep->tag,
+		    i, p);
+		assert(p >= vep->ver_p);
+
+		/******************************************************
+		 * SECTION A
+		 */
+
+		if (vep->state == VEP_START) {
+			if (cache_param->esi_syntax & 0x1)
+				vep->state = VEP_NEXTTAG;
+			else
+				vep->state = VEP_TESTXML;
+		} else if (vep->state == VEP_TESTXML) {
+			/*
+			 * If the first non-whitespace char is different
+			 * from '<' we assume this is not XML.
+			 */
+			while (p < e && vct_islws(*p))
+				p++;
+			vep_mark_verbatim(vep, p);
+			if (p < e && *p == '<') {
+				p++;
+				vep->state = VEP_STARTTAG;
+			} else if (p < e) {
+				WSLB(vep->wrk, SLT_ESI_xmlerror,
+				    "No ESI processing, first char not '<'");
+				vep->state = VEP_NOTXML;
+			}
+		} else if (vep->state == VEP_NOTXML) {
+			/*
+			 * This is not recognized as XML, just skip thru
+			 * vfp_esi_end() will handle the rest
+			 */
+			p = e;
+			vep_mark_verbatim(vep, p);
+
+		/******************************************************
+		 * SECTION B
+		 */
+
+		} else if (vep->state == VEP_NOTMYTAG) {
+			if (cache_param->esi_syntax & 0x2) {
+				p++;
+				vep->state = VEP_NEXTTAG;
+			} else {
+				vep->tag_i = 0;
+				while (p < e) {
+					if (*p++ == '>') {
+						vep->state = VEP_NEXTTAG;
+						break;
+					}
+				}
+			}
+			if (p == e && !vep->remove)
+				vep_mark_verbatim(vep, p);
+		} else if (vep->state == VEP_NEXTTAG) {
+			/*
+			 * Hunt for start of next tag and keep an eye
+			 * out for end of EsiCmt if armed.
+			 */
+			vep->emptytag = 0;
+			vep->endtag = 0;
+			vep->attr = NULL;
+			vep->dostuff = NULL;
+			while (p < e && *p != '<') {
+				if (vep->esicmt_p == NULL) {
+					p++;
+					continue;
+				}
+				if (*p != *vep->esicmt_p) {
+					p++;
+					vep->esicmt_p = vep->esicmt;
+					continue;
+				}
+				if (!vep->remove &&
+				    vep->esicmt_p == vep->esicmt)
+					vep_mark_verbatim(vep, p);
+				p++;
+				if (*++vep->esicmt_p == '\0') {
+					vep->esi_found = 1;
+					vep->esicmt = NULL;
+					vep->esicmt_p = NULL;
+					/*
+					 * The end of the esicmt
+					 * should not be emitted.
+					 * But the stuff before should
+					 */
+					vep_mark_skip(vep, p);
+				}
+			}
+			if (p < e) {
+				if (!vep->remove)
+					vep_mark_verbatim(vep, p);
+				assert(*p == '<');
+				p++;
+				vep->state = VEP_STARTTAG;
+			} else if (vep->esicmt_p == vep->esicmt && !vep->remove)
+				vep_mark_verbatim(vep, p);
+
+		/******************************************************
+		 * SECTION C
+		 */
+
+		} else if (vep->state == VEP_STARTTAG) {
+			/*
+			 * Start of tag, set up match table
+			 */
+			if (p < e) {
+				if (*p == '/') {
+					vep->endtag = 1;
+					p++;
+				}
+				vep->match = vep_match_starttag;
+				vep->state = VEP_MATCH;
+			}
+		} else if (vep->state == VEP_COMMENT) {
+			/*
+			 * We are in a comment, find out if it is an
+			 * ESI comment or a regular comment
+			 */
+			if (vep->esicmt == NULL)
+				vep->esicmt_p = vep->esicmt = "esi";
+			while (p < e) {
+				if (*p != *vep->esicmt_p) {
+					vep->esicmt_p = vep->esicmt = NULL;
+					vep->until_p = vep->until = "-->";
+					vep->until_s = VEP_NEXTTAG;
+					vep->state = VEP_UNTIL;
+					vep_mark_verbatim(vep, p);
+					break;
+				}
+				p++;
+				if (*++vep->esicmt_p != '\0')
+					continue;
+				if (vep->remove)
+					vep_error(vep,
+					    "ESI 1.0 Nested <!--esi"
+					    " element in <esi:remove>");
+				vep->esicmt_p = vep->esicmt = "-->";
+				vep->state = VEP_NEXTTAG;
+				vep_mark_skip(vep, p);
+				break;
+			}
+		} else if (vep->state == VEP_CDATA) {
+			/*
+			 * Easy: just look for the end of CDATA
+			 */
+			vep->until_p = vep->until = "]]>";
+			vep->until_s = VEP_NEXTTAG;
+			vep->state = VEP_UNTIL;
+		} else if (vep->state == VEP_ESITAG) {
+			vep->in_esi_tag = 1;
+			vep->esi_found = 1;
+			vep_mark_skip(vep, p);
+			vep->match = vep_match_esi;
+			vep->state = VEP_MATCH;
+		} else if (vep->state == VEP_ESIINCLUDE) {
+			if (vep->remove) {
+				vep_error(vep,
+				    "ESI 1.0 <esi:include> element"
+				    " nested in <esi:remove>");
+				vep->state = VEP_TAGERROR;
+			} else if (vep->endtag) {
+				vep_error(vep,
+				    "ESI 1.0 </esi:include> illegal end-tag");
+				vep->state = VEP_TAGERROR;
+			} else {
+				vep->dostuff = vep_do_include;
+				vep->state = VEP_INTAG;
+				vep->attr = vep_match_attr_include;
+			}
+		} else if (vep->state == VEP_ESIREMOVE) {
+			vep->dostuff = vep_do_remove;
+			vep->state = VEP_INTAG;
+		} else if (vep->state == VEP_ESICOMMENT) {
+			if (vep->remove) {
+				vep_error(vep,
+				    "ESI 1.0 <esi:comment> element"
+				    " nested in <esi:remove>");
+				vep->state = VEP_TAGERROR;
+			} else if (vep->endtag) {
+				vep_error(vep,
+				    "ESI 1.0 </esi:comment> illegal end-tag");
+				vep->state = VEP_TAGERROR;
+			} else {
+				vep->dostuff = vep_do_comment;
+				vep->state = VEP_INTAG;
+			}
+		} else if (vep->state == VEP_ESIBOGON) {
+			vep_error(vep,
+			    "ESI 1.0 <esi:bogus> element");
+			vep->state = VEP_TAGERROR;
+
+		/******************************************************
+		 * SECTION D
+		 */
+
+		} else if (vep->state == VEP_INTAG) {
+			vep->tag_i = 0;
+			while (p < e && vct_islws(*p) && !vep->emptytag) {
+				p++;
+				vep->canattr = 1;
+			}
+			if (p < e && *p == '/' && !vep->emptytag) {
+				p++;
+				vep->emptytag = 1;
+				vep->canattr = 0;
+			}
+			if (p < e && *p == '>') {
+				p++;
+				AN(vep->dostuff);
+				vep_mark_skip(vep, p);
+				vep->dostuff(vep, DO_TAG);
+				vep->in_esi_tag = 0;
+				vep->state = VEP_NEXTTAG;
+			} else if (p < e && vep->emptytag) {
+				vep_error(vep,
+				    "XML 1.0 '>' does not follow '/' in tag");
+				vep->state = VEP_TAGERROR;
+			} else if (p < e && vep->canattr &&
+			    vct_isxmlnamestart(*p)) {
+				vep->state = VEP_ATTR;
+			} else if (p < e) {
+				vep_error(vep,
+				    "XML 1.0 Illegal attribute start char");
+				vep->state = VEP_TAGERROR;
+			}
+		} else if (vep->state == VEP_TAGERROR) {
+			while (p < e && *p != '>')
+				p++;
+			if (p < e) {
+				p++;
+				vep_mark_skip(vep, p);
+				vep->in_esi_tag = 0;
+				vep->state = VEP_NEXTTAG;
+			}
+
+		/******************************************************
+		 * SECTION E
+		 */
+
+		} else if (vep->state == VEP_ATTR) {
+			AZ(vep->attr_delim);
+			if (vep->attr == NULL) {
+				p++;
+				AZ(vep->attr_vsb);
+				vep->state = VEP_SKIPATTR;
+			} else {
+				vep->match = vep->attr;
+				vep->state = VEP_MATCH;
+			}
+		} else if (vep->state == VEP_SKIPATTR) {
+			while (p < e && vct_isxmlname(*p))
+				p++;
+			if (p < e && *p == '=') {
+				p++;
+				vep->state = VEP_ATTRDELIM;
+			} else if (p < e && *p == '>') {
+				vep->state = VEP_INTAG;
+			} else if (p < e && *p == '/') {
+				vep->state = VEP_INTAG;
+			} else if (p < e && vct_issp(*p)) {
+				vep->state = VEP_INTAG;
+			} else if (p < e) {
+				vep_error(vep,
+				    "XML 1.0 Illegal attr char");
+				vep->state = VEP_TAGERROR;
+			}
+		} else if (vep->state == VEP_ATTRGETVAL) {
+			vep->attr_vsb = VSB_new_auto();
+			vep->state = VEP_ATTRDELIM;
+		} else if (vep->state == VEP_ATTRDELIM) {
+			AZ(vep->attr_delim);
+			if (*p == '"' || *p == '\'') {
+				vep->attr_delim = *p++;
+				vep->state = VEP_ATTRVAL;
+			} else if (!vct_issp(*p)) {
+				vep->attr_delim = ' ';
+				vep->state = VEP_ATTRVAL;
+			} else {
+				vep_error(vep,
+				    "XML 1.0 Illegal attribute delimiter");
+				vep->state = VEP_TAGERROR;
+			}
+
+		} else if (vep->state == VEP_ATTRVAL) {
+			while (p < e && *p != '>' && *p != vep->attr_delim &&
+			   (vep->attr_delim != ' ' || !vct_issp(*p))) {
+				if (vep->attr_vsb != NULL)
+					VSB_bcat(vep->attr_vsb, p, 1);
+				p++;
+			}
+			if (p < e && *p == '>') {
+				vep_error(vep,
+				    "XML 1.0 Missing end attribute delimiter");
+				vep->state = VEP_TAGERROR;
+				vep->attr_delim = 0;
+				if (vep->attr_vsb != NULL) {
+					AZ(VSB_finish(vep->attr_vsb));
+					VSB_delete(vep->attr_vsb);
+					vep->attr_vsb = NULL;
+				}
+			} else if (p < e) {
+				vep->attr_delim = 0;
+				p++;
+				vep->state = VEP_INTAG;
+				if (vep->attr_vsb != NULL) {
+					AZ(VSB_finish(vep->attr_vsb));
+					AN(vep->dostuff);
+					vep->dostuff(vep, DO_ATTR);
+					vep->attr_vsb = NULL;
+				}
+			}
+
+		/******************************************************
+		 * Utility Section
+		 */
+
+		} else if (vep->state == VEP_MATCH) {
+			/*
+			 * Match against a table
+			 */
+			vm = vep_match(vep, p, e);
+			vep->match_hit = vm;
+			if (vm != NULL) {
+				if (vm->match != NULL)
+					p += strlen(vm->match);
+				vep->state = *vm->state;
+				vep->match = NULL;
+				vep->tag_i = 0;
+			} else {
+				memcpy(vep->tag, p, e - p);
+				vep->tag_i = e - p;
+				vep->state = VEP_MATCHBUF;
+				p = e;
+			}
+		} else if (vep->state == VEP_MATCHBUF) {
+			/*
+			 * Match against a table while split over input
+			 * sections.
+			 */
+			do {
+				if (*p == '>') {
+					for (vm = vep->match;
+					    vm->match != NULL; vm++)
+						continue;
+					AZ(vm->match);
+				} else {
+					vep->tag[vep->tag_i++] = *p++;
+					vm = vep_match(vep,
+					    vep->tag, vep->tag + vep->tag_i);
+					if (vm && vm->match == NULL) {
+						vep->tag_i--;
+						p--;
+					}
+				}
+			} while (vm == NULL && p < e);
+			vep->match_hit = vm;
+			if (vm == NULL) {
+				assert(p == e);
+			} else {
+				vep->state = *vm->state;
+				vep->match = NULL;
+			}
+		} else if (vep->state == VEP_UNTIL) {
+			/*
+			 * Skip until we see magic string
+			 */
+			while (p < e) {
+				if (*p++ != *vep->until_p++) {
+					vep->until_p = vep->until;
+				} else if (*vep->until_p == '\0') {
+					vep->state = vep->until_s;
+					break;
+				}
+			}
+			if (p == e && !vep->remove)
+				vep_mark_verbatim(vep, p);
+		} else {
+			Debug("*** Unknown state %s\n", vep->state);
+			INCOMPL();
+		}
+	}
+	/*
+	 * We must always mark up the storage we got, try to do so
+	 * in the most efficient way, in particular with respect to
+	 * minimizing and limiting use of pending.
+	 */
+	if (p == vep->ver_p)
+		;
+	else if (vep->in_esi_tag)
+		vep_mark_skip(vep, p);
+	else if (vep->remove)
+		vep_mark_skip(vep, p);
+	else
+		vep_mark_pending(vep, p);
+}
+
+/*---------------------------------------------------------------------
+ */
+
+static ssize_t __match_proto__()
+vep_default_cb(struct worker *w, ssize_t l, enum vgz_flag flg)
+{
+
+	(void)flg;
+	AN(w->vep);
+	w->vep->cb_x += l;
+Debug("CB(%jd,%d) = %jd\n", (intmax_t)l, flg, (intmax_t)w->vep->cb_x);
+	return (w->vep->cb_x);
+}
+
+/*---------------------------------------------------------------------
+ */
+
+void
+VEP_Init(struct worker *w, vep_callback_t *cb)
+{
+	struct vep_state *vep;
+
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	AZ(w->vep);
+	w->vep = (void*)WS_Alloc(w->ws, sizeof *vep);
+	AN(w->vep);
+
+	vep = w->vep;
+	memset(vep, 0, sizeof *vep);
+	vep->magic = VEP_MAGIC;
+	vep->wrk = w;
+	vep->vsb = VSB_new_auto();
+	AN(vep->vsb);
+
+	if (cb != NULL) {
+		vep->dogzip = 1;
+		/* XXX */
+		VSB_printf(vep->vsb, "%c", VEC_GZ);
+		vep->cb = cb;
+	} else {
+		vep->cb = vep_default_cb;
+	}
+
+	vep->state = VEP_START;
+	vep->crc = crc32(0L, Z_NULL, 0);
+	vep->crcp = crc32(0L, Z_NULL, 0);
+
+	/*
+	 * We must force the GZIP header out as a SKIP string, otherwise
+	 * an object starting with <esi:include would have its GZIP header
+	 * appear after the included object (e000026.vtc)
+	 */
+	vep->startup = 1;
+	vep->ver_p = "";
+	vep->last_mark = SKIP;
+	vep_mark_common(vep, vep->ver_p, VERBATIM);
+	vep->startup = 0;
+}
+
+/*---------------------------------------------------------------------
+ */
+
+struct vsb *
+VEP_Finish(struct worker *w)
+{
+	struct vep_state *vep;
+	ssize_t l, lcb;
+
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	vep = w->vep;
+	CHECK_OBJ_NOTNULL(vep, VEP_MAGIC);
+
+	if (vep->o_pending)
+		vep_mark_common(vep, vep->ver_p, vep->last_mark);
+	if (vep->o_wait > 0) {
+		lcb = vep->cb(vep->wrk, 0, VGZ_ALIGN);
+		vep_emit_common(vep, lcb - vep->o_last, vep->last_mark);
+	}
+	(void)vep->cb(vep->wrk, 0, VGZ_FINISH);
+
+	w->vep = NULL;
+
+	AZ(VSB_finish(vep->vsb));
+	l = VSB_len(vep->vsb);
+	if (vep->esi_found && l > 0)
+		return (vep->vsb);
+	VSB_delete(vep->vsb);
+	return (NULL);
+}
+
+#if 0
+
+digraph xml {
+	rankdir="LR"
+	size="7,10"
+#################################################################
+# SECTION A
+#
+
+START		[shape=ellipse]
+TESTXML		[shape=ellipse]
+NOTXML		[shape=ellipse]
+NEXTTAGa	[shape=hexagon, label="NEXTTAG"]
+STARTTAGa	[shape=hexagon, label="STARTTAG"]
+START		-> TESTXML
+START		-> NEXTTAGa	[style=dotted, label="syntax:1"]
+TESTXML		-> TESTXML	[label="lws"]
+TESTXML		-> NOTXML
+TESTXML		-> STARTTAGa	[label="'<'"]
+
+#################################################################
+# SECTION B
+
+NOTMYTAG	[shape=ellipse]
+NEXTTAG		[shape=ellipse]
+NOTMYTAG	-> NEXTTAG	[style=dotted, label="syntax:2"]
+STARTTAGb	[shape=hexagon, label="STARTTAG"]
+NOTMYTAG	-> NEXTTAG	[label="'>'"]
+NOTMYTAG	-> NOTMYTAG	[label="*"]
+NEXTTAG		-> NEXTTAG	[label="'-->'"]
+NEXTTAG		-> NEXTTAG	[label="*"]
+NEXTTAG		-> STARTTAGb	[label="'<'"]
+
+#################################################################
+# SECTION C
+
+STARTTAG	[shape=ellipse]
+COMMENT		[shape=ellipse]
+CDATA		[shape=ellipse]
+ESITAG		[shape=ellipse]
+ESIETAG		[shape=ellipse]
+ESIINCLUDE	[shape=ellipse]
+ESIREMOVE	[shape=ellipse]
+ESICOMMENT	[shape=ellipse]
+ESIBOGON	[shape=ellipse]
+INTAGc		[shape=hexagon, label="INTAG"]
+NOTMYTAGc	[shape=hexagon, label="NOTMYTAG"]
+NEXTTAGc	[shape=hexagon, label="NEXTTAG"]
+TAGERRORc	[shape=hexagon, label="TAGERROR"]
+C1		[shape=circle,label=""]
+STARTTAG	-> COMMENT	[label="'<!--'"]
+STARTTAG	-> ESITAG	[label="'<esi'"]
+STARTTAG	-> CDATA	[label="'<![CDATA['"]
+STARTTAG	-> NOTMYTAGc	[label="'*'"]
+COMMENT		-> NEXTTAGc	[label="'esi'"]
+COMMENT		-> C1		[label="*"]
+C1		-> C1		[label="*"]
+C1		-> NEXTTAGc	[label="-->"]
+CDATA		-> CDATA	[label="*"]
+CDATA		-> NEXTTAGc	[label="]]>"]
+ESITAG		-> ESIINCLUDE	[label="'include'"]
+ESITAG		-> ESIREMOVE	[label="'remove'"]
+ESITAG		-> ESICOMMENT	[label="'comment'"]
+ESITAG		-> ESIBOGON	[label="*"]
+ESICOMMENT	-> INTAGc
+ESICOMMENT	-> TAGERRORc
+ESICOMMENT	-> TAGERRORc	[style=dotted, label="nested\nin\nremove"]
+ESIREMOVE	-> INTAGc
+ESIREMOVE	-> TAGERRORc
+ESIINCLUDE	-> INTAGc
+ESIINCLUDE	-> TAGERRORc
+ESIINCLUDE	-> TAGERRORc	[style=dotted, label="nested\nin\nremove"]
+ESIBOGON	-> TAGERRORc
+
+#################################################################
+# SECTION D
+
+INTAG		[shape=ellipse]
+TAGERROR	[shape=ellipse]
+NEXTTAGd	[shape=hexagon, label="NEXTTAG"]
+ATTRd		[shape=hexagon, label="ATTR"]
+D1		[shape=circle, label=""]
+D2		[shape=circle, label=""]
+INTAG		-> D1		[label="lws"]
+D1		-> D2		[label="/"]
+INTAG		-> D2		[label="/"]
+INTAG		-> NEXTTAGd	[label=">"]
+D1		-> NEXTTAGd	[label=">"]
+D2		-> NEXTTAGd	[label=">"]
+D1		-> ATTRd	[label="XMLstartchar"]
+D1		-> TAGERROR	[label="*"]
+D2		-> TAGERROR	[label="*"]
+TAGERROR	-> TAGERROR	[label="*"]
+TAGERROR	-> NEXTTAGd	[label="'>'"]
+
+#################################################################
+# SECTION E
+
+ATTR		[shape=ellipse]
+SKIPATTR	[shape=ellipse]
+ATTRGETVAL	[shape=ellipse]
+ATTRDELIM	[shape=ellipse]
+ATTRVAL		[shape=ellipse]
+TAGERRORe	[shape=hexagon, label="TAGERROR"]
+INTAGe		[shape=hexagon, label="INTAG"]
+ATTR		-> SKIPATTR	[label="*"]
+ATTR		-> ATTRGETVAL	[label="wanted attr"]
+SKIPATTR	-> SKIPATTR	[label="XMLname"]
+SKIPATTR	-> ATTRDELIM	[label="'='"]
+SKIPATTR	-> TAGERRORe	[label="*"]
+ATTRGETVAL	-> ATTRDELIM
+ATTRDELIM	-> ATTRVAL	[label="\""]
+ATTRDELIM	-> ATTRVAL	[label="\'"]
+ATTRDELIM	-> ATTRVAL	[label="*"]
+ATTRDELIM	-> TAGERRORe	[label="lws"]
+ATTRVAL		-> TAGERRORe	[label="'>'"]
+ATTRVAL		-> INTAGe	[label="delim"]
+ATTRVAL		-> ATTRVAL	[label="*"]
+
+}
+
+#endif
diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c
new file mode 100644
index 0000000..23e3fc6
--- /dev/null
+++ b/bin/varnishd/cache/cache_expire.c
@@ -0,0 +1,490 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * LRU and object timer handling.
+ *
+ * We have two data structures, a LRU-list and a binary heap for the timers
+ * and two ways to kill objects: TTL-timeouts and LRU cleanups.
+ *
+ * Any object on the LRU is also on the binheap and vice versa.
+ *
+ * We hold a single object reference for both data structures.
+ *
+ * An attempted overview:
+ *
+ *	                        EXP_Ttl()      EXP_Grace()   EXP_Keep()
+ *				   |                |            |
+ *      entered                    v                v            |
+ *         |                       +--------------->+            |
+ *         v                       |      grace                  |
+ *         +---------------------->+                             |
+ *                  ttl            |                             v
+ *                                 +---------------------------->+
+ *                                     keep
+ *
+ */
+
+#include "config.h"
+
+#include <math.h>
+
+#include "cache.h"
+
+#include "binary_heap.h"
+#include "hash/hash_slinger.h"
+#include "vtim.h"
+
+static pthread_t exp_thread;
+static struct binheap *exp_heap;
+static struct lock exp_mtx;
+
+/*--------------------------------------------------------------------
+ * struct exp manipulations
+ *
+ * The Get/Set functions encapsulate the mutual magic between the
+ * fields in one single place.
+ */
+
+void
+EXP_Clr(struct exp *e)
+{
+
+	e->ttl = -1;
+	e->grace = -1;
+	e->keep = -1;
+	e->age = 0;
+	e->entered = 0;
+}
+
+#define EXP_ACCESS(fld, low_val, extra)				\
+	double							\
+	EXP_Get_##fld(const struct exp *e)			\
+	{							\
+		return (e->fld > 0. ? e->fld : low_val);	\
+	}							\
+								\
+	void							\
+	EXP_Set_##fld(struct exp *e, double v)			\
+	{							\
+		if (v > 0.)					\
+			e->fld = v;				\
+		else {						\
+			e->fld = -1.;				\
+			extra;					\
+		}						\
+	}							\
+
+EXP_ACCESS(ttl, -1., (e->grace = e->keep = -1.))
+EXP_ACCESS(grace, 0., )
+EXP_ACCESS(keep, 0.,)
+
+/*--------------------------------------------------------------------
+ * Calculate an objects effective keep, grace or ttl time, suitably
+ * adjusted for defaults and by per-session limits.
+ */
+
+static double
+EXP_Keep(const struct sess *sp, const struct object *o)
+{
+	double r;
+
+	r = (double)cache_param->default_keep;
+	if (o->exp.keep > 0.)
+		r = o->exp.keep;
+	if (sp != NULL && sp->exp.keep > 0. && sp->exp.keep < r)
+		r = sp->exp.keep;
+	return (EXP_Ttl(sp, o) + r);
+}
+
+double
+EXP_Grace(const struct sess *sp, const struct object *o)
+{
+	double r;
+
+	r = (double)cache_param->default_grace;
+	if (o->exp.grace >= 0.)
+		r = o->exp.grace;
+	if (sp != NULL && sp->exp.grace > 0. && sp->exp.grace < r)
+		r = sp->exp.grace;
+	return (EXP_Ttl(sp, o) + r);
+}
+
+double
+EXP_Ttl(const struct sess *sp, const struct object *o)
+{
+	double r;
+
+	r = o->exp.ttl;
+	if (sp != NULL && sp->exp.ttl > 0. && sp->exp.ttl < r)
+		r = sp->exp.ttl;
+	return (o->exp.entered + r);
+}
+
+/*--------------------------------------------------------------------
+ * When & why does the timer fire for this object ?
+ */
+
+static int
+update_object_when(const struct object *o)
+{
+	struct objcore *oc;
+	double when, w2;
+
+	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
+	oc = o->objcore;
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	Lck_AssertHeld(&exp_mtx);
+
+	when = EXP_Keep(NULL, o);
+	w2 = EXP_Grace(NULL, o);
+	if (w2 > when)
+		when = w2;
+	assert(!isnan(when));
+	if (when == oc->timer_when)
+		return (0);
+	oc->timer_when = when;
+	return (1);
+}
+
+/*--------------------------------------------------------------------*/
+
+static void
+exp_insert(struct objcore *oc, struct lru *lru)
+{
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
+
+	Lck_AssertHeld(&lru->mtx);
+	Lck_AssertHeld(&exp_mtx);
+	assert(oc->timer_idx == BINHEAP_NOIDX);
+	binheap_insert(exp_heap, oc);
+	assert(oc->timer_idx != BINHEAP_NOIDX);
+	VTAILQ_INSERT_TAIL(&lru->lru_head, oc, lru_list);
+}
+
+/*--------------------------------------------------------------------
+ * Object has been added to cache, record in lru & binheap.
+ *
+ * The objcore comes with a reference, which we inherit.
+ */
+
+void
+EXP_Inject(struct objcore *oc, struct lru *lru, double when)
+{
+
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
+
+	Lck_Lock(&lru->mtx);
+	Lck_Lock(&exp_mtx);
+	oc->timer_when = when;
+	exp_insert(oc, lru);
+	Lck_Unlock(&exp_mtx);
+	Lck_Unlock(&lru->mtx);
+}
+
+/*--------------------------------------------------------------------
+ * Object has been added to cache, record in lru & binheap.
+ *
+ * We grab a reference to the object, which will keep it around until
+ * we decide its time to let it go.
+ */
+
+void
+EXP_Insert(struct object *o)
+{
+	struct objcore *oc;
+	struct lru *lru;
+
+	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
+	oc = o->objcore;
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	AssertObjBusy(o);
+	HSH_Ref(oc);
+
+	assert(o->exp.entered != 0 && !isnan(o->exp.entered));
+	o->last_lru = o->exp.entered;
+
+	lru = oc_getlru(oc);
+	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
+	Lck_Lock(&lru->mtx);
+	Lck_Lock(&exp_mtx);
+	(void)update_object_when(o);
+	exp_insert(oc, lru);
+	Lck_Unlock(&exp_mtx);
+	Lck_Unlock(&lru->mtx);
+	oc_updatemeta(oc);
+}
+
+/*--------------------------------------------------------------------
+ * Object was used, move to tail of LRU list.
+ *
+ * To avoid the exp_mtx becoming a hotspot, we only attempt to move
+ * objects if they have not been moved recently and if the lock is available.
+ * This optimization obviously leaves the LRU list imperfectly sorted.
+ */
+
+int
+EXP_Touch(struct objcore *oc)
+{
+	struct lru *lru;
+
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+
+	/*
+	 * For -spersistent we don't move objects on the lru list.  Each
+	 * segment has its own LRU list, and the order on it is not material
+	 * for anything.  The code below would move the objects to the
+	 * LRU list of the currently open segment, which would prevent
+	 * the cleaner from doing its job.
+	 */
+	if (oc->flags & OC_F_LRUDONTMOVE)
+		return (0);
+
+	lru = oc_getlru(oc);
+	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
+
+	/*
+	 * We only need the LRU lock here.  The locking order is LRU->EXP
+	 * so we can trust the content of the oc->timer_idx without the
+	 * EXP lock.   Since each lru list has its own lock, this should
+	 * reduce contention a fair bit
+	 */
+	if (Lck_Trylock(&lru->mtx))
+		return (0);
+
+	if (oc->timer_idx != BINHEAP_NOIDX) {
+		VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
+		VTAILQ_INSERT_TAIL(&lru->lru_head, oc, lru_list);
+		VSC_C_main->n_lru_moved++;
+	}
+	Lck_Unlock(&lru->mtx);
+	return (1);
+}
+
+/*--------------------------------------------------------------------
+ * We have changed one or more of the object timers, shuffle it
+ * accordingly in the binheap
+ *
+ * The VCL code can send us here on a non-cached object, just return.
+ *
+ * XXX: special case check for ttl = 0 ?
+ */
+
+void
+EXP_Rearm(const struct object *o)
+{
+	struct objcore *oc;
+	struct lru *lru;
+
+	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
+	oc = o->objcore;
+	if (oc == NULL)
+		return;
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	lru = oc_getlru(oc);
+	Lck_Lock(&lru->mtx);
+	Lck_Lock(&exp_mtx);
+	/*
+	 * The hang-man might have this object of the binheap while
+	 * tending to a timer.  If so, we do not muck with it here.
+	 */
+	if (oc->timer_idx != BINHEAP_NOIDX && update_object_when(o)) {
+		assert(oc->timer_idx != BINHEAP_NOIDX);
+		binheap_reorder(exp_heap, oc->timer_idx);
+		assert(oc->timer_idx != BINHEAP_NOIDX);
+	}
+	Lck_Unlock(&exp_mtx);
+	Lck_Unlock(&lru->mtx);
+	oc_updatemeta(oc);
+}
+
+/*--------------------------------------------------------------------
+ * This thread monitors the root of the binary heap and whenever an
+ * object expires, accounting also for graceability, it is killed.
+ */
+
+static void * __match_proto__(void *start_routine(void *))
+exp_timer(struct sess *sp, void *priv)
+{
+	struct objcore *oc;
+	struct lru *lru;
+	double t;
+	struct object *o;
+
+	(void)priv;
+	t = VTIM_real();
+	oc = NULL;
+	while (1) {
+		if (oc == NULL) {
+			WSL_Flush(sp->wrk, 0);
+			WRK_SumStat(sp->wrk);
+			VTIM_sleep(cache_param->expiry_sleep);
+			t = VTIM_real();
+		}
+
+		Lck_Lock(&exp_mtx);
+		oc = binheap_root(exp_heap);
+		if (oc == NULL) {
+			Lck_Unlock(&exp_mtx);
+			continue;
+		}
+		CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+
+		/*
+		 * We may have expired so many objects that our timestamp
+		 * got out of date, refresh it and check again.
+		 */
+		if (oc->timer_when > t)
+			t = VTIM_real();
+		if (oc->timer_when > t) {
+			Lck_Unlock(&exp_mtx);
+			oc = NULL;
+			continue;
+		}
+
+		/*
+		 * It's time...
+		 * Technically we should drop the exp_mtx, get the lru->mtx
+		 * get the exp_mtx again and then check that the oc is still
+		 * on the binheap.  We take the shorter route and try to
+		 * get the lru->mtx and punt if we fail.
+		 */
+
+		lru = oc_getlru(oc);
+		CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
+		if (Lck_Trylock(&lru->mtx)) {
+			Lck_Unlock(&exp_mtx);
+			oc = NULL;
+			continue;
+		}
+
+		/* Remove from binheap */
+		assert(oc->timer_idx != BINHEAP_NOIDX);
+		binheap_delete(exp_heap, oc->timer_idx);
+		assert(oc->timer_idx == BINHEAP_NOIDX);
+
+		/* And from LRU */
+		lru = oc_getlru(oc);
+		VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
+
+		Lck_Unlock(&exp_mtx);
+		Lck_Unlock(&lru->mtx);
+
+		VSC_C_main->n_expired++;
+
+		CHECK_OBJ_NOTNULL(oc->objhead, OBJHEAD_MAGIC);
+		o = oc_getobj(sp->wrk, oc);
+		WSL(sp->wrk, SLT_ExpKill, 0, "%u %.0f",
+		    o->xid, EXP_Ttl(NULL, o) - t);
+		(void)HSH_Deref(sp->wrk, oc, NULL);
+	}
+	NEEDLESS_RETURN(NULL);
+}
+
+/*--------------------------------------------------------------------
+ * Attempt to make space by nuking the oldest object on the LRU list
+ * which isn't in use.
+ * Returns: 1: did, 0: didn't, -1: can't
+ */
+
+int
+EXP_NukeOne(struct worker *w, struct lru *lru)
+{
+	struct objcore *oc;
+	struct object *o;
+
+	/* Find the first currently unused object on the LRU.  */
+	Lck_Lock(&lru->mtx);
+	Lck_Lock(&exp_mtx);
+	VTAILQ_FOREACH(oc, &lru->lru_head, lru_list) {
+		CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+		assert (oc->timer_idx != BINHEAP_NOIDX);
+		/*
+		 * It wont release any space if we cannot release the last
+		 * reference, besides, if somebody else has a reference,
+		 * it's a bad idea to nuke this object anyway.
+		 */
+		if (oc->refcnt == 1)
+			break;
+	}
+	if (oc != NULL) {
+		VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
+		binheap_delete(exp_heap, oc->timer_idx);
+		assert(oc->timer_idx == BINHEAP_NOIDX);
+		VSC_C_main->n_lru_nuked++;
+	}
+	Lck_Unlock(&exp_mtx);
+	Lck_Unlock(&lru->mtx);
+
+	if (oc == NULL)
+		return (-1);
+
+	/* XXX: bad idea for -spersistent */
+	o = oc_getobj(w, oc);
+	WSL(w, SLT_ExpKill, 0, "%u LRU", o->xid);
+	(void)HSH_Deref(w, NULL, &o);
+	return (1);
+}
+
+/*--------------------------------------------------------------------
+ * BinHeap helper functions for objcore.
+ */
+
+static int
+object_cmp(void *priv, void *a, void *b)
+{
+	struct objcore *aa, *bb;
+
+	(void)priv;
+	CAST_OBJ_NOTNULL(aa, a, OBJCORE_MAGIC);
+	CAST_OBJ_NOTNULL(bb, b, OBJCORE_MAGIC);
+	return (aa->timer_when < bb->timer_when);
+}
+
+static void
+object_update(void *priv, void *p, unsigned u)
+{
+	struct objcore *oc;
+
+	(void)priv;
+	CAST_OBJ_NOTNULL(oc, p, OBJCORE_MAGIC);
+	oc->timer_idx = u;
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+EXP_Init(void)
+{
+
+	Lck_New(&exp_mtx, lck_exp);
+	exp_heap = binheap_new(NULL, object_cmp, object_update);
+	XXXAN(exp_heap);
+	WRK_BgThread(&exp_thread, "cache-timeout", exp_timer, NULL);
+}
diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
new file mode 100644
index 0000000..a678dcc
--- /dev/null
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -0,0 +1,645 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include "config.h"
+
+#include <inttypes.h>
+#include <stdio.h>
+#include <stdlib.h>
+
+#include "cache.h"
+
+#include "cache_backend.h"
+#include "vcli_priv.h"
+#include "vct.h"
+#include "vtcp.h"
+
+static unsigned fetchfrag;
+
+/*--------------------------------------------------------------------
+ * We want to issue the first error we encounter on fetching and
+ * supress the rest.  This function does that.
+ *
+ * Other code is allowed to look at w->fetch_failed to bail out
+ *
+ * For convenience, always return -1
+ */
+
+int
+FetchError2(struct worker *w, const char *error, const char *more)
+{
+
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	if (!w->fetch_failed) {
+		if (more == NULL)
+			WSLB(w, SLT_FetchError, "%s", error);
+		else
+			WSLB(w, SLT_FetchError, "%s: %s", error, more);
+	}
+	w->fetch_failed = 1;
+	return (-1);
+}
+
+int
+FetchError(struct worker *w, const char *error)
+{
+	return(FetchError2(w, error, NULL));
+}
+
+/*--------------------------------------------------------------------
+ * VFP_NOP
+ *
+ * This fetch-processor does nothing but store the object.
+ * It also documents the API
+ */
+
+/*--------------------------------------------------------------------
+ * VFP_BEGIN
+ *
+ * Called to set up stuff.
+ *
+ * 'estimate' is the estimate of the number of bytes we expect to receive,
+ * as seen on the socket, or zero if unknown.
+ */
+static void __match_proto__()
+vfp_nop_begin(struct worker *w, size_t estimate)
+{
+
+	if (estimate > 0)
+		(void)FetchStorage(w, estimate);
+}
+
+/*--------------------------------------------------------------------
+ * VFP_BYTES
+ *
+ * Process (up to) 'bytes' from the socket.
+ *
+ * Return -1 on error, issue FetchError()
+ *	will not be called again, once error happens.
+ * Return 0 on EOF on socket even if bytes not reached.
+ * Return 1 when 'bytes' have been processed.
+ */
+
+static int __match_proto__()
+vfp_nop_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
+{
+	ssize_t l, wl;
+	struct storage *st;
+
+	AZ(w->fetch_failed);
+	while (bytes > 0) {
+		st = FetchStorage(w, 0);
+		if (st == NULL)
+			return(-1);
+		l = st->space - st->len;
+		if (l > bytes)
+			l = bytes;
+		wl = HTC_Read(w, htc, st->ptr + st->len, l);
+		if (wl <= 0)
+			return (wl);
+		st->len += wl;
+		w->fetch_obj->len += wl;
+		bytes -= wl;
+		if (w->do_stream)
+			RES_StreamPoll(w);
+	}
+	return (1);
+}
+
+/*--------------------------------------------------------------------
+ * VFP_END
+ *
+ * Finish & cleanup
+ *
+ * Return -1 for error
+ * Return 0 for OK
+ */
+
+static int __match_proto__()
+vfp_nop_end(struct worker *w)
+{
+	struct storage *st;
+
+	st = VTAILQ_LAST(&w->fetch_obj->store, storagehead);
+	if (st == NULL)
+		return (0);
+
+	if (st->len == 0) {
+		VTAILQ_REMOVE(&w->fetch_obj->store, st, list);
+		STV_free(st);
+		return (0);
+	}
+	if (st->len < st->space)
+		STV_trim(st, st->len);
+	return (0);
+}
+
+static struct vfp vfp_nop = {
+	.begin	=	vfp_nop_begin,
+	.bytes	=	vfp_nop_bytes,
+	.end	=	vfp_nop_end,
+};
+
+/*--------------------------------------------------------------------
+ * Fetch Storage to put object into.
+ *
+ */
+
+struct storage *
+FetchStorage(struct worker *w, ssize_t sz)
+{
+	ssize_t l;
+	struct storage *st;
+	struct object *obj;
+
+	obj = w->fetch_obj;
+	CHECK_OBJ_NOTNULL(obj, OBJECT_MAGIC);
+	st = VTAILQ_LAST(&obj->store, storagehead);
+	if (st != NULL && st->len < st->space)
+		return (st);
+
+	l = fetchfrag;
+	if (l == 0)
+		l = sz;
+	if (l == 0)
+		l = cache_param->fetch_chunksize * 1024LL;
+	st = STV_alloc(w, l);
+	if (st == NULL) {
+		(void)FetchError(w, "Could not get storage");
+		return (NULL);
+	}
+	AZ(st->len);
+	VTAILQ_INSERT_TAIL(&obj->store, st, list);
+	return (st);
+}
+
+/*--------------------------------------------------------------------
+ * Convert a string to a size_t safely
+ */
+
+static ssize_t
+fetch_number(const char *nbr, int radix)
+{
+	uintmax_t cll;
+	ssize_t cl;
+	char *q;
+
+	if (*nbr == '\0')
+		return (-1);
+	cll = strtoumax(nbr, &q, radix);
+	if (q == NULL || *q != '\0')
+		return (-1);
+
+	cl = (ssize_t)cll;
+	if((uintmax_t)cl != cll) /* Protect against bogusly large values */
+		return (-1);
+	return (cl);
+}
+
+/*--------------------------------------------------------------------*/
+
+static int
+fetch_straight(struct worker *w, struct http_conn *htc, ssize_t cl)
+{
+	int i;
+
+	assert(w->body_status == BS_LENGTH);
+
+	if (cl < 0) {
+		return (FetchError(w, "straight length field bogus"));
+	} else if (cl == 0)
+		return (0);
+
+	i = w->vfp->bytes(w, htc, cl);
+	if (i <= 0)
+		return (FetchError(w, "straight insufficient bytes"));
+	return (0);
+}
+
+/*--------------------------------------------------------------------
+ * Read a chunked HTTP object.
+ *
+ * XXX: Reading one byte at a time is pretty pessimal.
+ */
+
+static int
+fetch_chunked(struct worker *w, struct http_conn *htc)
+{
+	int i;
+	char buf[20];		/* XXX: 20 is arbitrary */
+	unsigned u;
+	ssize_t cl;
+
+	assert(w->body_status == BS_CHUNKED);
+	do {
+		/* Skip leading whitespace */
+		do {
+			if (HTC_Read(w, htc, buf, 1) <= 0)
+				return (-1);
+		} while (vct_islws(buf[0]));
+
+		if (!vct_ishex(buf[0]))
+			return (FetchError(w,"chunked header non-hex"));
+
+		/* Collect hex digits, skipping leading zeros */
+		for (u = 1; u < sizeof buf; u++) {
+			do {
+				if (HTC_Read(w, htc, buf + u, 1) <= 0)
+					return (-1);
+			} while (u == 1 && buf[0] == '0' && buf[u] == '0');
+			if (!vct_ishex(buf[u]))
+				break;
+		}
+
+		if (u >= sizeof buf)
+			return (FetchError(w,"chunked header too long"));
+
+		/* Skip trailing white space */
+		while(vct_islws(buf[u]) && buf[u] != '\n')
+			if (HTC_Read(w, htc, buf + u, 1) <= 0)
+				return (-1);
+
+		if (buf[u] != '\n')
+			return (FetchError(w,"chunked header no NL"));
+
+		buf[u] = '\0';
+		cl = fetch_number(buf, 16);
+		if (cl < 0)
+			return (FetchError(w,"chunked header number syntax"));
+
+		if (cl > 0 && w->vfp->bytes(w, htc, cl) <= 0)
+			return (-1);
+
+		i = HTC_Read(w, htc, buf, 1);
+		if (i <= 0)
+			return (-1);
+		if (buf[0] == '\r' && HTC_Read(w, htc, buf, 1) <= 0)
+			return (-1);
+		if (buf[0] != '\n')
+			return (FetchError(w,"chunked tail no NL"));
+	} while (cl > 0);
+	return (0);
+}
+
+/*--------------------------------------------------------------------*/
+
+static int
+fetch_eof(struct worker *w, struct http_conn *htc)
+{
+	int i;
+
+	assert(w->body_status == BS_EOF);
+	i = w->vfp->bytes(w, htc, SSIZE_MAX);
+	if (i < 0)
+		return (-1);
+	return (0);
+}
+
+/*--------------------------------------------------------------------
+ * Fetch any body attached to the incoming request, and either write it
+ * to the backend (if we pass) or discard it (anything else).
+ * This is mainly a separate function to isolate the stack buffer and
+ * to contain the complexity when we start handling chunked encoding.
+ */
+
+int
+FetchReqBody(struct sess *sp)
+{
+	unsigned long content_length;
+	char buf[8192];
+	char *ptr, *endp;
+	int rdcnt;
+
+	if (http_GetHdr(sp->http, H_Content_Length, &ptr)) {
+
+		content_length = strtoul(ptr, &endp, 10);
+		/* XXX should check result of conversion */
+		while (content_length) {
+			if (content_length > sizeof buf)
+				rdcnt = sizeof buf;
+			else
+				rdcnt = content_length;
+			rdcnt = HTC_Read(sp->wrk, sp->htc, buf, rdcnt);
+			if (rdcnt <= 0)
+				return (1);
+			content_length -= rdcnt;
+			if (!sp->sendbody)
+				continue;
+			(void)WRW_Write(sp->wrk, buf, rdcnt); /* XXX: stats ? */
+			if (WRW_Flush(sp->wrk))
+				return (2);
+		}
+	}
+	if (http_GetHdr(sp->http, H_Transfer_Encoding, NULL)) {
+		/* XXX: Handle chunked encoding. */
+		WSP(sp, SLT_Debug, "Transfer-Encoding in request");
+		return (1);
+	}
+	return (0);
+}
+
+/*--------------------------------------------------------------------
+ * Send request, and receive the HTTP protocol response, but not the
+ * response body.
+ *
+ * Return value:
+ *	-1 failure, not retryable
+ *	 0 success
+ *	 1 failure which can be retried.
+ */
+
+int
+FetchHdr(struct sess *sp)
+{
+	struct vbc *vc;
+	struct worker *w;
+	char *b;
+	struct http *hp;
+	int retry = -1;
+	int i;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->wrk, WORKER_MAGIC);
+	w = sp->wrk;
+
+	AN(sp->director);
+	AZ(sp->obj);
+
+	if (sp->objcore != NULL) {		/* pass has no objcore */
+		CHECK_OBJ_NOTNULL(sp->objcore, OBJCORE_MAGIC);
+		AN(sp->objcore->flags & OC_F_BUSY);
+	}
+
+	hp = w->bereq;
+
+	sp->wrk->vbc = VDI_GetFd(NULL, sp);
+	if (sp->wrk->vbc == NULL) {
+		WSP(sp, SLT_FetchError, "no backend connection");
+		return (-1);
+	}
+	vc = sp->wrk->vbc;
+	if (vc->recycled)
+		retry = 1;
+
+	/*
+	 * Now that we know our backend, we can set a default Host:
+	 * header if one is necessary.  This cannot be done in the VCL
+	 * because the backend may be chosen by a director.
+	 */
+	if (!http_GetHdr(hp, H_Host, &b))
+		VDI_AddHostHeader(sp);
+
+	(void)VTCP_blocking(vc->fd);	/* XXX: we should timeout instead */
+	WRW_Reserve(w, &vc->fd);
+	(void)http_Write(w, vc->vsl_id, hp, 0);	/* XXX: stats ? */
+
+	/* Deal with any message-body the request might have */
+	i = FetchReqBody(sp);
+	if (WRW_FlushRelease(w) || i > 0) {
+		WSP(sp, SLT_FetchError, "backend write error: %d (%s)",
+		    errno, strerror(errno));
+		VDI_CloseFd(sp->wrk);
+		/* XXX: other cleanup ? */
+		return (retry);
+	}
+
+	/* Checkpoint the vsl.here */
+	WSL_Flush(w, 0);
+
+	/* XXX is this the right place? */
+	VSC_C_main->backend_req++;
+
+	/* Receive response */
+
+	HTC_Init(w->htc, w->ws, vc->fd, vc->vsl_id, cache_param->http_resp_size,
+	    cache_param->http_resp_hdr_len);
+
+	VTCP_set_read_timeout(vc->fd, vc->first_byte_timeout);
+
+	i = HTC_Rx(w->htc);
+
+	if (i < 0) {
+		WSP(sp, SLT_FetchError, "http first read error: %d %d (%s)",
+		    i, errno, strerror(errno));
+		VDI_CloseFd(sp->wrk);
+		/* XXX: other cleanup ? */
+		/* Retryable if we never received anything */
+		return (i == -1 ? retry : -1);
+	}
+
+	VTCP_set_read_timeout(vc->fd, vc->between_bytes_timeout);
+
+	while (i == 0) {
+		i = HTC_Rx(w->htc);
+		if (i < 0) {
+			WSP(sp, SLT_FetchError,
+			    "http first read error: %d %d (%s)",
+			    i, errno, strerror(errno));
+			VDI_CloseFd(sp->wrk);
+			/* XXX: other cleanup ? */
+			return (-1);
+		}
+	}
+
+	hp = w->beresp;
+
+	if (http_DissectResponse(w, w->htc, hp)) {
+		WSP(sp, SLT_FetchError, "http format error");
+		VDI_CloseFd(sp->wrk);
+		/* XXX: other cleanup ? */
+		return (-1);
+	}
+	return (0);
+}
+
+/*--------------------------------------------------------------------*/
+
+int
+FetchBody(struct worker *w, struct object *obj)
+{
+	int cls;
+	struct storage *st;
+	int mklen;
+	ssize_t cl;
+
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	AZ(w->fetch_obj);
+	CHECK_OBJ_NOTNULL(w->vbc, VBC_MAGIC);
+	CHECK_OBJ_NOTNULL(obj, OBJECT_MAGIC);
+	CHECK_OBJ_NOTNULL(obj->http, HTTP_MAGIC);
+
+	if (w->vfp == NULL)
+		w->vfp = &vfp_nop;
+
+	AssertObjCorePassOrBusy(obj->objcore);
+
+	AZ(w->vgz_rx);
+	AZ(VTAILQ_FIRST(&obj->store));
+
+	w->fetch_obj = obj;
+	w->fetch_failed = 0;
+
+	/* XXX: pick up estimate from objdr ? */
+	cl = 0;
+	switch (w->body_status) {
+	case BS_NONE:
+		cls = 0;
+		mklen = 0;
+		break;
+	case BS_ZERO:
+		cls = 0;
+		mklen = 1;
+		break;
+	case BS_LENGTH:
+		cl = fetch_number( w->h_content_length, 10);
+		w->vfp->begin(w, cl > 0 ? cl : 0);
+		cls = fetch_straight(w, w->htc, cl);
+		mklen = 1;
+		if (w->vfp->end(w))
+			cls = -1;
+		break;
+	case BS_CHUNKED:
+		w->vfp->begin(w, cl);
+		cls = fetch_chunked(w, w->htc);
+		mklen = 1;
+		if (w->vfp->end(w))
+			cls = -1;
+		break;
+	case BS_EOF:
+		w->vfp->begin(w, cl);
+		cls = fetch_eof(w, w->htc);
+		mklen = 1;
+		if (w->vfp->end(w))
+			cls = -1;
+		break;
+	case BS_ERROR:
+		cls = 1;
+		mklen = 0;
+		break;
+	default:
+		cls = 0;
+		mklen = 0;
+		INCOMPL();
+	}
+	AZ(w->vgz_rx);
+
+	/*
+	 * It is OK for ->end to just leave the last storage segment
+	 * sitting on w->storage, we will always call vfp_nop_end()
+	 * to get it trimmed or thrown out if empty.
+	 */
+	AZ(vfp_nop_end(w));
+
+	w->fetch_obj = NULL;
+
+	WSLB(w, SLT_Fetch_Body, "%u(%s) cls %d mklen %u",
+	    w->body_status, body_status(w->body_status),
+	    cls, mklen);
+
+	if (w->body_status == BS_ERROR) {
+		VDI_CloseFd(w);
+		return (__LINE__);
+	}
+
+	if (cls < 0) {
+		w->stats.fetch_failed++;
+		/* XXX: Wouldn't this store automatically be released ? */
+		while (!VTAILQ_EMPTY(&obj->store)) {
+			st = VTAILQ_FIRST(&obj->store);
+			VTAILQ_REMOVE(&obj->store, st, list);
+			STV_free(st);
+		}
+		VDI_CloseFd(w);
+		obj->len = 0;
+		return (__LINE__);
+	}
+	AZ(w->fetch_failed);
+
+	if (cls == 0 && w->do_close)
+		cls = 1;
+
+	WSLB(w, SLT_Length, "%u", obj->len);
+
+	{
+	/* Sanity check fetch methods accounting */
+		ssize_t uu;
+
+		uu = 0;
+		VTAILQ_FOREACH(st, &obj->store, list)
+			uu += st->len;
+		if (w->do_stream)
+			/* Streaming might have started freeing stuff */
+			assert (uu <= obj->len);
+
+		else
+			assert(uu == obj->len);
+	}
+
+	if (mklen > 0) {
+		http_Unset(obj->http, H_Content_Length);
+		http_PrintfHeader(w, w->vbc->vsl_id, obj->http,
+		    "Content-Length: %jd", (intmax_t)obj->len);
+	}
+
+	if (cls)
+		VDI_CloseFd(w);
+	else
+		VDI_RecycleFd(w);
+
+	return (0);
+}
+
+/*--------------------------------------------------------------------
+ * Debugging aids
+ */
+
+static void
+debug_fragfetch(struct cli *cli, const char * const *av, void *priv)
+{
+	(void)priv;
+	(void)cli;
+	fetchfrag = strtoul(av[2], NULL, 0);
+}
+
+static struct cli_proto debug_cmds[] = {
+	{ "debug.fragfetch", "debug.fragfetch",
+		"\tEnable fetch fragmentation\n", 1, 1, "d", debug_fragfetch },
+	{ NULL }
+};
+
+/*--------------------------------------------------------------------
+ *
+ */
+
+void
+Fetch_Init(void)
+{
+
+	CLI_AddFuncs(debug_cmds);
+}
diff --git a/bin/varnishd/cache/cache_gzip.c b/bin/varnishd/cache/cache_gzip.c
new file mode 100644
index 0000000..32a7413
--- /dev/null
+++ b/bin/varnishd/cache/cache_gzip.c
@@ -0,0 +1,697 @@
+/*-
+ * Copyright (c) 2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * Interaction with the linvgz (zlib) library.
+ *
+ * The zlib library pollutes namespace a LOT when you include the "vgz.h"
+ * (aka (zlib.h") file so we contain the damage by vectoring all access
+ * to libz through this source file.
+ *
+ * The API defined by this file, will also insulate the rest of the code,
+ * should we find a better gzip library at a later date.
+ *
+ * The absolutely worst case gzip processing path, once we have pipe-lining,
+ * will be the following, so we need to be a bit careful with the scratch
+ * space we use:
+ *
+ *	Backend		Tmp	Input	Output
+ *         |		----------------------
+ *	   v
+ *	 gunzip		wrk	stack	?
+ *         |
+ *	   v
+ *	  esi
+ *         |
+ *	   v
+ *	  gzip		wrk	?	storage
+ *         |
+ *	   v
+ *	  cache
+ *         |
+ *	   v
+ *	 gunzip		wrk	storage	stack
+ *         |
+ *	   v
+ *	 client
+ *
+ * XXXX: The two '?' are obviously the same memory, but I have yet to decide
+ * where it goes.   As usual we try to avoid the session->ws if we can but
+ * I may have to use that.
+ *
+ */
+
+#include "config.h"
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#include "cache.h"
+
+#include "vgz.h"
+
+struct vgz {
+	unsigned		magic;
+#define VGZ_MAGIC		0x162df0cb
+	enum {VGZ_GZ,VGZ_UN}	dir;
+	struct worker		*wrk;
+	const char		*id;
+	struct ws		*tmp;
+	char			*tmp_snapshot;
+	int			last_i;
+
+	struct storage		*obuf;
+
+	z_stream		vz;
+};
+
+/*--------------------------------------------------------------------*/
+
+static voidpf
+vgz_alloc(voidpf opaque, uInt items, uInt size)
+{
+	struct vgz *vg;
+
+	CAST_OBJ_NOTNULL(vg, opaque, VGZ_MAGIC);
+
+	return (WS_Alloc(vg->tmp, items * size));
+}
+
+static void
+vgz_free(voidpf opaque, voidpf address)
+{
+	struct vgz *vg;
+
+	CAST_OBJ_NOTNULL(vg, opaque, VGZ_MAGIC);
+	(void)address;
+}
+
+/*--------------------------------------------------------------------
+ * Set up a gunzip instance
+ */
+
+static struct vgz *
+vgz_alloc_vgz(struct worker *wrk, const char *id)
+{
+	struct vgz *vg;
+	struct ws *ws;
+
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
+	ws = wrk->ws;
+	WS_Assert(ws);
+	// XXX: we restore workspace in esi:include
+	// vg = (void*)WS_Alloc(ws, sizeof *vg);
+	ALLOC_OBJ(vg, VGZ_MAGIC);
+	AN(vg);
+	memset(vg, 0, sizeof *vg);
+	vg->magic = VGZ_MAGIC;
+	vg->wrk = wrk;
+	vg->id = id;
+
+	switch (cache_param->gzip_tmp_space) {
+	case 0:
+	case 1:
+		/* malloc, the default */
+		break;
+	case 2:
+		vg->tmp = wrk->ws;
+		vg->tmp_snapshot = WS_Snapshot(vg->tmp);
+		vg->vz.zalloc = vgz_alloc;
+		vg->vz.zfree = vgz_free;
+		vg->vz.opaque = vg;
+		break;
+	default:
+		assert(0 == __LINE__);
+	}
+	return (vg);
+}
+
+struct vgz *
+VGZ_NewUngzip(struct worker *wrk, const char *id)
+{
+	struct vgz *vg;
+
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
+	vg = vgz_alloc_vgz(wrk, id);
+	vg->dir = VGZ_UN;
+	VSC_C_main->n_gunzip++;
+
+	/*
+	 * Max memory usage according to zonf.h:
+	 *	mem_needed = "a few kb" + (1 << (windowBits))
+	 * Since we don't control windowBits, we have to assume
+	 * it is 15, so 34-35KB or so.
+	 */
+	assert(Z_OK == inflateInit2(&vg->vz, 31));
+	return (vg);
+}
+
+struct vgz *
+VGZ_NewGzip(struct worker *wrk, const char *id)
+{
+	struct vgz *vg;
+	int i;
+
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
+	vg = vgz_alloc_vgz(wrk, id);
+	vg->dir = VGZ_GZ;
+	VSC_C_main->n_gzip++;
+
+	/*
+	 * From zconf.h:
+	 *
+	 *	mem_needed = "a few kb"
+	 *		+ (1 << (windowBits+2))
+	 *		+  (1 << (memLevel+9))
+	 *
+	 * windowBits [8..15] (-> 1K..128K)
+	 * memLevel [1..9] (-> 1K->256K)
+	 *
+	 * XXX: They probably needs to be params...
+	 *
+	 * XXX: It may be more efficent to malloc them, rather than have
+	 * XXX: too many worker threads grow the stacks.
+	 */
+	i = deflateInit2(&vg->vz,
+	    cache_param->gzip_level,		/* Level */
+	    Z_DEFLATED,			/* Method */
+	    16 + cache_param->gzip_window,	/* Window bits (16=gzip + 15) */
+	    cache_param->gzip_memlevel,	/* memLevel */
+	    Z_DEFAULT_STRATEGY);
+	assert(Z_OK == i);
+	return (vg);
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+VGZ_Ibuf(struct vgz *vg, const void *ptr, ssize_t len)
+{
+
+	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
+
+	AZ(vg->vz.avail_in);
+	vg->vz.next_in = TRUST_ME(ptr);
+	vg->vz.avail_in = len;
+}
+
+int
+VGZ_IbufEmpty(const struct vgz *vg)
+{
+
+	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
+	return (vg->vz.avail_in == 0);
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+VGZ_Obuf(struct vgz *vg, void *ptr, ssize_t len)
+{
+
+	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
+
+	vg->vz.next_out = TRUST_ME(ptr);
+	vg->vz.avail_out = len;
+}
+
+int
+VGZ_ObufFull(const struct vgz *vg)
+{
+
+	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
+	return (vg->vz.avail_out == 0);
+}
+
+/*--------------------------------------------------------------------
+ * Keep the outbuffer supplied with storage and file it under the
+ * sp->obj as it fills.
+ */
+
+int
+VGZ_ObufStorage(struct worker *w, struct vgz *vg)
+{
+	struct storage *st;
+
+	st = FetchStorage(w, 0);
+	if (st == NULL)
+		return (-1);
+
+	vg->obuf = st;
+	VGZ_Obuf(vg, st->ptr + st->len, st->space - st->len);
+
+	return (0);
+}
+
+/*--------------------------------------------------------------------*/
+
+int
+VGZ_Gunzip(struct vgz *vg, const void **pptr, size_t *plen)
+{
+	int i;
+	ssize_t l;
+	const uint8_t *before;
+
+	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
+
+	*pptr = NULL;
+	*plen = 0;
+	AN(vg->vz.next_out);
+	AN(vg->vz.avail_out);
+	before = vg->vz.next_out;
+	i = inflate(&vg->vz, 0);
+	if (i == Z_OK || i == Z_STREAM_END) {
+		*pptr = before;
+		l = (const uint8_t *)vg->vz.next_out - before;
+		*plen = l;
+		if (vg->obuf != NULL)
+			vg->obuf->len += l;
+	}
+	vg->last_i = i;
+	if (i == Z_OK)
+		return (VGZ_OK);
+	if (i == Z_STREAM_END)
+		return (VGZ_END);
+	if (i == Z_BUF_ERROR)
+		return (VGZ_STUCK);
+	VSL(SLT_Debug, 0, "Unknown INFLATE=%d (%s)\n", i, vg->vz.msg);
+	return (VGZ_ERROR);
+}
+
+/*--------------------------------------------------------------------*/
+
+int
+VGZ_Gzip(struct vgz *vg, const void **pptr, size_t *plen, enum vgz_flag flags)
+{
+	int i;
+	int zflg;
+	ssize_t l;
+	const uint8_t *before;
+
+	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
+
+	*pptr = NULL;
+	*plen = 0;
+	AN(vg->vz.next_out);
+	AN(vg->vz.avail_out);
+	before = vg->vz.next_out;
+	switch(flags) {
+	case VGZ_NORMAL:	zflg = Z_NO_FLUSH; break;
+	case VGZ_ALIGN:		zflg = Z_SYNC_FLUSH; break;
+	case VGZ_RESET:		zflg = Z_FULL_FLUSH; break;
+	case VGZ_FINISH:	zflg = Z_FINISH; break;
+	default:		INCOMPL();
+	}
+	i = deflate(&vg->vz, zflg);
+	if (i == Z_OK || i == Z_STREAM_END) {
+		*pptr = before;
+		l = (const uint8_t *)vg->vz.next_out - before;
+		*plen = l;
+		if (vg->obuf != NULL)
+			vg->obuf->len += l;
+	}
+	vg->last_i = i;
+	if (i == Z_OK)
+		return (0);
+	if (i == Z_STREAM_END)
+		return (1);
+	if (i == Z_BUF_ERROR)
+		return (2);
+	return (-1);
+}
+
+/*--------------------------------------------------------------------
+ * Gunzip ibuf into outb, if it runs full, emit it with WRW.
+ * Leave flushing to caller, more data may be coming.
+ */
+
+int
+VGZ_WrwGunzip(struct worker *w, struct vgz *vg, const void *ibuf,
+    ssize_t ibufl, char *obuf, ssize_t obufl, ssize_t *obufp)
+{
+	int i;
+	size_t dl;
+	const void *dp;
+
+	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
+	assert(obufl > 16);
+	VGZ_Ibuf(vg, ibuf, ibufl);
+	if (ibufl == 0)
+		return (VGZ_OK);
+	VGZ_Obuf(vg, obuf + *obufp, obufl - *obufp);
+	do {
+		if (obufl == *obufp)
+			i = VGZ_STUCK;
+		else {
+			i = VGZ_Gunzip(vg, &dp, &dl);
+			*obufp += dl;
+		}
+		if (i < VGZ_OK) {
+			/* XXX: VSL ? */
+			return (-1);
+		}
+		if (obufl == *obufp || i == VGZ_STUCK) {
+			w->acct_tmp.bodybytes += *obufp;
+			(void)WRW_Write(w, obuf, *obufp);
+			(void)WRW_Flush(w);
+			*obufp = 0;
+			VGZ_Obuf(vg, obuf + *obufp, obufl - *obufp);
+		}
+	} while (!VGZ_IbufEmpty(vg));
+	if (i == VGZ_STUCK)
+		i = VGZ_OK;
+	return (i);
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+VGZ_UpdateObj(const struct vgz *vg, struct object *obj)
+{
+
+	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
+	CHECK_OBJ_NOTNULL(obj, OBJECT_MAGIC);
+	obj->gzip_start	= vg->vz.start_bit;
+	obj->gzip_last	= vg->vz.last_bit;
+	obj->gzip_stop	= vg->vz.stop_bit;
+}
+
+/*--------------------------------------------------------------------
+ * Passing a vsl_id of -1 means "use w->vbc->vsl_id"
+ */
+
+int
+VGZ_Destroy(struct vgz **vgp, int vsl_id)
+{
+	struct vgz *vg;
+	int i;
+
+	vg = *vgp;
+	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
+	*vgp = NULL;
+
+	if (vsl_id < 0)
+		WSLB(vg->wrk, SLT_Gzip, "%s %jd %jd %jd %jd %jd",
+		    vg->id,
+		    (intmax_t)vg->vz.total_in,
+		    (intmax_t)vg->vz.total_out,
+		    (intmax_t)vg->vz.start_bit,
+		    (intmax_t)vg->vz.last_bit,
+		    (intmax_t)vg->vz.stop_bit);
+	else
+		WSL(vg->wrk, SLT_Gzip, vsl_id, "%s %jd %jd %jd %jd %jd",
+		    vg->id,
+		    (intmax_t)vg->vz.total_in,
+		    (intmax_t)vg->vz.total_out,
+		    (intmax_t)vg->vz.start_bit,
+		    (intmax_t)vg->vz.last_bit,
+		    (intmax_t)vg->vz.stop_bit);
+	if (vg->tmp != NULL)
+		WS_Reset(vg->tmp, vg->tmp_snapshot);
+	if (vg->dir == VGZ_GZ)
+		i = deflateEnd(&vg->vz);
+	else
+		i = inflateEnd(&vg->vz);
+	if (vg->last_i == Z_STREAM_END && i == Z_OK)
+		i = Z_STREAM_END;
+	FREE_OBJ(vg);
+	if (i == Z_OK)
+		return (VGZ_OK);
+	if (i == Z_STREAM_END)
+		return (VGZ_END);
+	if (i == Z_BUF_ERROR)
+		return (VGZ_STUCK);
+	return (VGZ_ERROR);
+}
+
+/*--------------------------------------------------------------------
+ * VFP_GUNZIP
+ *
+ * A VFP for gunzip'ing an object as we receive it from the backend
+ */
+
+static void __match_proto__()
+vfp_gunzip_begin(struct worker *w, size_t estimate)
+{
+	(void)estimate;
+	AZ(w->vgz_rx);
+	w->vgz_rx = VGZ_NewUngzip(w, "U F -");
+}
+
+static int __match_proto__()
+vfp_gunzip_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
+{
+	struct vgz *vg;
+	ssize_t l, wl;
+	int i = -100;
+	uint8_t	ibuf[cache_param->gzip_stack_buffer];
+	size_t dl;
+	const void *dp;
+
+	AZ(w->fetch_failed);
+	vg = w->vgz_rx;
+	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
+	AZ(vg->vz.avail_in);
+	while (bytes > 0 || vg->vz.avail_in > 0) {
+		if (vg->vz.avail_in == 0 && bytes > 0) {
+			l = sizeof ibuf;
+			if (l > bytes)
+				l = bytes;
+			wl = HTC_Read(w, htc, ibuf, l);
+			if (wl <= 0)
+				return (wl);
+			VGZ_Ibuf(vg, ibuf, wl);
+			bytes -= wl;
+		}
+
+		if (VGZ_ObufStorage(w, vg))
+			return(-1);
+		i = VGZ_Gunzip(vg, &dp, &dl);
+		if (i != VGZ_OK && i != VGZ_END)
+			return(FetchError(w, "Gunzip data error"));
+		w->fetch_obj->len += dl;
+		if (w->do_stream)
+			RES_StreamPoll(w);
+	}
+	assert(i == Z_OK || i == Z_STREAM_END);
+	return (1);
+}
+
+static int __match_proto__()
+vfp_gunzip_end(struct worker *w)
+{
+	struct vgz *vg;
+
+	vg = w->vgz_rx;
+	w->vgz_rx = NULL;
+	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
+	if (w->fetch_failed) {
+		(void)VGZ_Destroy(&vg, -1);
+		return(0);
+	}
+	if (VGZ_Destroy(&vg, -1) != VGZ_END)
+		return(FetchError(w, "Gunzip error at the very end"));
+	return (0);
+}
+
+struct vfp vfp_gunzip = {
+        .begin  =       vfp_gunzip_begin,
+        .bytes  =       vfp_gunzip_bytes,
+        .end    =       vfp_gunzip_end,
+};
+
+
+/*--------------------------------------------------------------------
+ * VFP_GZIP
+ *
+ * A VFP for gzip'ing an object as we receive it from the backend
+ */
+
+static void __match_proto__()
+vfp_gzip_begin(struct worker *w, size_t estimate)
+{
+	(void)estimate;
+
+	AZ(w->vgz_rx);
+	w->vgz_rx = VGZ_NewGzip(w, "G F -");
+}
+
+static int __match_proto__()
+vfp_gzip_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
+{
+	struct vgz *vg;
+	ssize_t l, wl;
+	int i = -100;
+	uint8_t ibuf[cache_param->gzip_stack_buffer];
+	size_t dl;
+	const void *dp;
+
+	AZ(w->fetch_failed);
+	vg = w->vgz_rx;
+	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
+	AZ(vg->vz.avail_in);
+	while (bytes > 0 || !VGZ_IbufEmpty(vg)) {
+		if (VGZ_IbufEmpty(vg) && bytes > 0) {
+			l = sizeof ibuf;
+			if (l > bytes)
+				l = bytes;
+			wl = HTC_Read(w, htc, ibuf, l);
+			if (wl <= 0)
+				return (wl);
+			VGZ_Ibuf(vg, ibuf, wl);
+			bytes -= wl;
+		}
+		if (VGZ_ObufStorage(w, vg))
+			return(-1);
+		i = VGZ_Gzip(vg, &dp, &dl, VGZ_NORMAL);
+		assert(i == Z_OK);
+		w->fetch_obj->len += dl;
+		if (w->do_stream)
+			RES_StreamPoll(w);
+	}
+	return (1);
+}
+
+static int __match_proto__()
+vfp_gzip_end(struct worker *w)
+{
+	struct vgz *vg;
+	size_t dl;
+	const void *dp;
+	int i;
+
+	vg = w->vgz_rx;
+	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
+	w->vgz_rx = NULL;
+	if (w->fetch_failed) {
+		(void)VGZ_Destroy(&vg, -1);
+		return(0);
+	}
+	do {
+		VGZ_Ibuf(vg, "", 0);
+		if (VGZ_ObufStorage(w, vg))
+			return(-1);
+		i = VGZ_Gzip(vg, &dp, &dl, VGZ_FINISH);
+		w->fetch_obj->len += dl;
+	} while (i != Z_STREAM_END);
+	if (w->do_stream)
+		RES_StreamPoll(w);
+	VGZ_UpdateObj(vg, w->fetch_obj);
+	if (VGZ_Destroy(&vg, -1) != VGZ_END)
+		return(FetchError(w, "Gzip error at the very end"));
+	return (0);
+}
+
+struct vfp vfp_gzip = {
+        .begin  =       vfp_gzip_begin,
+        .bytes  =       vfp_gzip_bytes,
+        .end    =       vfp_gzip_end,
+};
+
+/*--------------------------------------------------------------------
+ * VFP_TESTGZIP
+ *
+ * A VFP for testing that received gzip data is valid, and for
+ * collecting the magic bits while we're at it.
+ */
+
+static void __match_proto__()
+vfp_testgzip_begin(struct worker *w, size_t estimate)
+{
+	(void)estimate;
+	w->vgz_rx = VGZ_NewUngzip(w, "u F -");
+	CHECK_OBJ_NOTNULL(w->vgz_rx, VGZ_MAGIC);
+}
+
+static int __match_proto__()
+vfp_testgzip_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
+{
+	struct vgz *vg;
+	ssize_t l, wl;
+	int i = -100;
+	uint8_t	obuf[cache_param->gzip_stack_buffer];
+	size_t dl;
+	const void *dp;
+	struct storage *st;
+
+	AZ(w->fetch_failed);
+	vg = w->vgz_rx;
+	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
+	AZ(vg->vz.avail_in);
+	while (bytes > 0) {
+		st = FetchStorage(w, 0);
+		if (st == NULL)
+			return(-1);
+		l = st->space - st->len;
+		if (l > bytes)
+			l = bytes;
+		wl = HTC_Read(w, htc, st->ptr + st->len, l);
+		if (wl <= 0)
+			return (wl);
+		bytes -= wl;
+		VGZ_Ibuf(vg, st->ptr + st->len, wl);
+		st->len += wl;
+		w->fetch_obj->len += wl;
+		if (w->do_stream)
+			RES_StreamPoll(w);
+
+		while (!VGZ_IbufEmpty(vg)) {
+			VGZ_Obuf(vg, obuf, sizeof obuf);
+			i = VGZ_Gunzip(vg, &dp, &dl);
+			if (i == VGZ_END && !VGZ_IbufEmpty(vg))
+				return(FetchError(w, "Junk after gzip data"));
+			if (i != VGZ_OK && i != VGZ_END)
+				return(FetchError2(w,
+				    "Invalid Gzip data", vg->vz.msg));
+		}
+	}
+	assert(i == VGZ_OK || i == VGZ_END);
+	return (1);
+}
+
+static int __match_proto__()
+vfp_testgzip_end(struct worker *w)
+{
+	struct vgz *vg;
+
+	vg = w->vgz_rx;
+	w->vgz_rx = NULL;
+	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
+	if (w->fetch_failed) {
+		(void)VGZ_Destroy(&vg, -1);
+		return(0);
+	}
+	VGZ_UpdateObj(vg, w->fetch_obj);
+	if (VGZ_Destroy(&vg, -1) != VGZ_END)
+		return(FetchError(w, "TestGunzip error at the very end"));
+	return (0);
+}
+
+struct vfp vfp_testgzip = {
+        .begin  =       vfp_testgzip_begin,
+        .bytes  =       vfp_testgzip_bytes,
+        .end    =       vfp_testgzip_end,
+};
diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
new file mode 100644
index 0000000..db865de
--- /dev/null
+++ b/bin/varnishd/cache/cache_hash.c
@@ -0,0 +1,752 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * This is the central hash-table code, it relies on a chosen hash
+ * implementation only for the actual hashing, all the housekeeping
+ * happens here.
+ *
+ * We have two kinds of structures, objecthead and object.  An objecthead
+ * corresponds to a given (Host:, URL) tupple, and the objects hung from
+ * the objecthead may represent various variations (ie: Vary: header,
+ * different TTL etc) instances of that web-entity.
+ *
+ * Each objecthead has a mutex which locks both its own fields, the
+ * list of objects and fields in the objects.
+ *
+ * The hash implementation must supply a reference count facility on
+ * the objecthead, and return with a reference held after a lookup.
+ *
+ * Lookups in the hash implementation returns with a ref held and each
+ * object hung from the objhead holds a ref as well.
+ *
+ * Objects have refcounts which are locked by the objecthead mutex.
+ *
+ * New objects are always marked busy, and they can go from busy to
+ * not busy only once.
+ */
+
+#include "config.h"
+
+#include <math.h>
+#include <stdio.h>
+#include <stdlib.h>
+
+#include "cache.h"
+
+#include "hash/hash_slinger.h"
+#include "vsha256.h"
+
+static const struct hash_slinger *hash;
+
+/*---------------------------------------------------------------------*/
+/* Precreate an objhead and object for later use */
+void
+HSH_Prealloc(const struct sess *sp)
+{
+	struct worker *w;
+	struct objhead *oh;
+	struct objcore *oc;
+	struct waitinglist *wl;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->wrk, WORKER_MAGIC);
+	w = sp->wrk;
+
+	if (w->nobjcore == NULL) {
+		ALLOC_OBJ(oc, OBJCORE_MAGIC);
+		XXXAN(oc);
+		w->nobjcore = oc;
+		w->stats.n_objectcore++;
+		oc->flags |= OC_F_BUSY;
+	}
+	CHECK_OBJ_NOTNULL(w->nobjcore, OBJCORE_MAGIC);
+
+	if (w->nobjhead == NULL) {
+		ALLOC_OBJ(oh, OBJHEAD_MAGIC);
+		XXXAN(oh);
+		oh->refcnt = 1;
+		VTAILQ_INIT(&oh->objcs);
+		Lck_New(&oh->mtx, lck_objhdr);
+		w->nobjhead = oh;
+		w->stats.n_objecthead++;
+	}
+	CHECK_OBJ_NOTNULL(w->nobjhead, OBJHEAD_MAGIC);
+
+	if (w->nwaitinglist == NULL) {
+		ALLOC_OBJ(wl, WAITINGLIST_MAGIC);
+		XXXAN(wl);
+		VTAILQ_INIT(&wl->list);
+		w->nwaitinglist = wl;
+		w->stats.n_waitinglist++;
+	}
+	CHECK_OBJ_NOTNULL(w->nwaitinglist, WAITINGLIST_MAGIC);
+
+	if (w->nbusyobj == NULL) {
+		ALLOC_OBJ(w->nbusyobj, BUSYOBJ_MAGIC);
+		XXXAN(w->nbusyobj);
+	}
+
+	if (hash->prep != NULL)
+		hash->prep(sp);
+}
+
+void
+HSH_Cleanup(struct worker *w)
+{
+
+	if (w->nobjcore != NULL) {
+		FREE_OBJ(w->nobjcore);
+		w->stats.n_objectcore--;
+		w->nobjcore = NULL;
+	}
+	if (w->nobjhead != NULL) {
+		Lck_Delete(&w->nobjhead->mtx);
+		FREE_OBJ(w->nobjhead);
+		w->nobjhead = NULL;
+		w->stats.n_objecthead--;
+	}
+	if (w->nwaitinglist != NULL) {
+		FREE_OBJ(w->nwaitinglist);
+		w->nwaitinglist = NULL;
+	}
+	if (w->nhashpriv != NULL) {
+		/* XXX: If needed, add slinger method for this */
+		free(w->nhashpriv);
+		w->nhashpriv = NULL;
+	}
+	if (w->nbusyobj != NULL) {
+		FREE_OBJ(w->nbusyobj);
+		w->nbusyobj = NULL;
+	}
+}
+
+void
+HSH_DeleteObjHead(struct worker *w, struct objhead *oh)
+{
+
+	AZ(oh->refcnt);
+	assert(VTAILQ_EMPTY(&oh->objcs));
+	Lck_Delete(&oh->mtx);
+	w->stats.n_objecthead--;
+	FREE_OBJ(oh);
+}
+
+void
+HSH_AddString(const struct sess *sp, const char *str)
+{
+	int l;
+
+	if (str == NULL)
+		str = "";
+	l = strlen(str);
+
+	SHA256_Update(sp->wrk->sha256ctx, str, l);
+	SHA256_Update(sp->wrk->sha256ctx, "#", 1);
+
+	if (cache_param->log_hash)
+		WSP(sp, SLT_Hash, "%s", str);
+}
+
+/*---------------------------------------------------------------------
+ * This is a debugging hack to enable testing of boundary conditions
+ * in the hash algorithm.
+ * We trap the first 9 different digests and translate them to different
+ * digests with edge bit conditions
+ */
+
+static struct hsh_magiclist {
+	unsigned char was[SHA256_LEN];
+	unsigned char now[SHA256_LEN];
+} hsh_magiclist[] = {
+	{ .now = {	0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 } },
+	{ .now = {	0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01 } },
+	{ .now = {	0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02 } },
+	{ .now = {	0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x40 } },
+	{ .now = {	0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x80 } },
+	{ .now = {	0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 } },
+	{ .now = {	0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 } },
+	{ .now = {	0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 } },
+	{ .now = {	0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 } },
+};
+
+#define HSH_NMAGIC (sizeof hsh_magiclist / sizeof hsh_magiclist[0])
+
+static void
+hsh_testmagic(void *result)
+{
+	int i, j;
+	static int nused = 0;
+
+	for (i = 0; i < nused; i++)
+		if (!memcmp(hsh_magiclist[i].was, result, SHA256_LEN))
+			break;
+	if (i == nused && i < HSH_NMAGIC)
+		memcpy(hsh_magiclist[nused++].was, result, SHA256_LEN);
+	if (i == nused)
+		return;
+	assert(i < HSH_NMAGIC);
+	fprintf(stderr, "HASHMAGIC: <");
+	for (j = 0; j < SHA256_LEN; j++)
+		fprintf(stderr, "%02x", ((unsigned char*)result)[j]);
+	fprintf(stderr, "> -> <");
+	memcpy(result, hsh_magiclist[i].now, SHA256_LEN);
+	for (j = 0; j < SHA256_LEN; j++)
+		fprintf(stderr, "%02x", ((unsigned char*)result)[j]);
+	fprintf(stderr, ">\n");
+}
+
+/*---------------------------------------------------------------------
+ * Insert an object which magically appears out of nowhere or, more likely,
+ * comes off some persistent storage device.
+ * Return it with a reference held.
+ */
+
+struct objcore *
+HSH_Insert(const struct sess *sp)
+{
+	struct worker *w;
+	struct objhead *oh;
+	struct objcore *oc;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->wrk, WORKER_MAGIC);
+	AN(hash);
+	w = sp->wrk;
+
+	HSH_Prealloc(sp);
+	if (cache_param->diag_bitmap & 0x80000000)
+		hsh_testmagic(sp->wrk->nobjhead->digest);
+
+	AZ(sp->hash_objhead);
+	AN(w->nobjhead);
+	oh = hash->lookup(sp, w->nobjhead);
+	CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC);
+	if (oh == w->nobjhead)
+		w->nobjhead = NULL;
+	Lck_Lock(&oh->mtx);
+	assert(oh->refcnt > 0);
+
+	/* Insert (precreated) objcore in objecthead */
+	oc = w->nobjcore;
+	w->nobjcore = NULL;
+	oc->refcnt = 1;
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	AZ(oc->flags & OC_F_BUSY);
+
+	VTAILQ_INSERT_HEAD(&oh->objcs, oc, list);
+	/* NB: do not deref objhead the new object inherits our reference */
+	oc->objhead = oh;
+	Lck_Unlock(&oh->mtx);
+	sp->wrk->stats.n_vampireobject++;
+	return (oc);
+}
+
+/*---------------------------------------------------------------------
+ */
+
+struct objcore *
+HSH_Lookup(struct sess *sp, struct objhead **poh)
+{
+	struct worker *w;
+	struct objhead *oh;
+	struct objcore *oc;
+	struct objcore *busy_oc, *grace_oc;
+	struct object *o;
+	double grace_ttl;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->wrk, WORKER_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->http, HTTP_MAGIC);
+	AN(sp->director);
+	AN(hash);
+	w = sp->wrk;
+
+	HSH_Prealloc(sp);
+	memcpy(sp->wrk->nobjhead->digest, sp->digest, sizeof sp->digest);
+	if (cache_param->diag_bitmap & 0x80000000)
+		hsh_testmagic(sp->wrk->nobjhead->digest);
+
+	if (sp->hash_objhead != NULL) {
+		/*
+		 * This sess came off the waiting list, and brings a
+		 * oh refcnt with it.
+		 */
+		CHECK_OBJ_NOTNULL(sp->hash_objhead, OBJHEAD_MAGIC);
+		oh = sp->hash_objhead;
+		sp->hash_objhead = NULL;
+	} else {
+		AN(w->nobjhead);
+		oh = hash->lookup(sp, w->nobjhead);
+		if (oh == w->nobjhead)
+			w->nobjhead = NULL;
+	}
+
+	CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC);
+	Lck_Lock(&oh->mtx);
+	assert(oh->refcnt > 0);
+	busy_oc = NULL;
+	grace_oc = NULL;
+	grace_ttl = NAN;
+	VTAILQ_FOREACH(oc, &oh->objcs, list) {
+		/* Must be at least our own ref + the objcore we examine */
+		assert(oh->refcnt > 1);
+		CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+		assert(oc->objhead == oh);
+
+		if (oc->flags & OC_F_BUSY) {
+			CHECK_OBJ_NOTNULL(oc->busyobj, BUSYOBJ_MAGIC);
+			if (sp->hash_ignore_busy)
+				continue;
+
+			if (oc->busyobj->vary != NULL &&
+			    !VRY_Match(sp, oc->busyobj->vary))
+				continue;
+
+			busy_oc = oc;
+			continue;
+		}
+
+		o = oc_getobj(sp->wrk, oc);
+		CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
+
+		if (o->exp.ttl <= 0.)
+			continue;
+		if (BAN_CheckObject(o, sp))
+			continue;
+		if (o->vary != NULL && !VRY_Match(sp, o->vary))
+			continue;
+
+		/* If still valid, use it */
+		if (EXP_Ttl(sp, o) >= sp->t_req)
+			break;
+
+		/*
+		 * Remember any matching objects inside their grace period
+		 * and if there are several, use the least expired one.
+		 */
+		if (EXP_Grace(sp, o) >= sp->t_req) {
+			if (grace_oc == NULL ||
+			    grace_ttl < o->exp.entered + o->exp.ttl) {
+				grace_oc = oc;
+				grace_ttl = o->exp.entered + o->exp.ttl;
+			}
+		}
+	}
+
+	/*
+	 * If we have seen a busy object or the backend is unhealthy, and
+	 * we have an object in grace, use it, if req.grace is also
+	 * satisified.
+	 * XXX: Interesting footnote:  The busy object might be for a
+	 * XXX: different "Vary:" than we sought.  We have no way of knowing
+	 * XXX: this until the object is unbusy'ed, so in practice we
+	 * XXX: serialize fetch of all Vary's if grace is possible.
+	 */
+
+	AZ(sp->objcore);
+	sp->objcore = grace_oc;		/* XXX: Hack-ish */
+	if (oc == NULL			/* We found no live object */
+	    && grace_oc != NULL		/* There is a grace candidate */
+	    && (busy_oc != NULL		/* Somebody else is already busy */
+	    || !VDI_Healthy(sp->director, sp))) {
+					/* Or it is impossible to fetch */
+		o = oc_getobj(sp->wrk, grace_oc);
+		CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
+		oc = grace_oc;
+	}
+	sp->objcore = NULL;
+
+	if (oc != NULL && !sp->hash_always_miss) {
+		o = oc_getobj(sp->wrk, oc);
+		CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
+		assert(oc->objhead == oh);
+
+		/* We found an object we like */
+		oc->refcnt++;
+		if (o->hits < INT_MAX)
+			o->hits++;
+		assert(oh->refcnt > 1);
+		Lck_Unlock(&oh->mtx);
+		assert(hash->deref(oh));
+		*poh = oh;
+		return (oc);
+	}
+
+	if (busy_oc != NULL) {
+		/* There are one or more busy objects, wait for them */
+		if (sp->esi_level == 0) {
+			CHECK_OBJ_NOTNULL(sp->wrk->nwaitinglist,
+			    WAITINGLIST_MAGIC);
+			if (oh->waitinglist == NULL) {
+				oh->waitinglist = sp->wrk->nwaitinglist;
+				sp->wrk->nwaitinglist = NULL;
+			}
+			VTAILQ_INSERT_TAIL(&oh->waitinglist->list, sp, list);
+		}
+		if (cache_param->diag_bitmap & 0x20)
+			WSP(sp, SLT_Debug,
+				"on waiting list <%p>", oh);
+		SES_Charge(sp);
+		/*
+		 * The objhead reference transfers to the sess, we get it
+		 * back when the sess comes off the waiting list and
+		 * calls us again
+		 */
+		sp->hash_objhead = oh;
+		sp->wrk = NULL;
+		Lck_Unlock(&oh->mtx);
+		return (NULL);
+	}
+
+	/* Insert (precreated) objcore in objecthead */
+	oc = w->nobjcore;
+	w->nobjcore = NULL;
+	AN(oc->flags & OC_F_BUSY);
+	oc->refcnt = 1;
+
+	/* XXX: clear w->nbusyobj before use */
+	VRY_Validate(sp->vary_b);
+	if (sp->vary_l != NULL)
+		w->nbusyobj->vary = sp->vary_b;
+	else
+		w->nbusyobj->vary = NULL;
+	oc->busyobj = w->nbusyobj;
+	w->nbusyobj = NULL;
+
+	/*
+	 * Busy objects go on the tail, so they will not trip up searches.
+	 * HSH_Unbusy() will move them to the front.
+	 */
+	VTAILQ_INSERT_TAIL(&oh->objcs, oc, list);
+	oc->objhead = oh;
+	/* NB: do not deref objhead the new object inherits our reference */
+	Lck_Unlock(&oh->mtx);
+	*poh = oh;
+	return (oc);
+}
+
+/*---------------------------------------------------------------------
+ */
+
+static void
+hsh_rush(struct objhead *oh)
+{
+	unsigned u;
+	struct sess *sp;
+	struct waitinglist *wl;
+
+	CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC);
+	Lck_AssertHeld(&oh->mtx);
+	wl = oh->waitinglist;
+	CHECK_OBJ_NOTNULL(wl, WAITINGLIST_MAGIC);
+	for (u = 0; u < cache_param->rush_exponent; u++) {
+		sp = VTAILQ_FIRST(&wl->list);
+		if (sp == NULL)
+			break;
+		CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+		AZ(sp->wrk);
+		VTAILQ_REMOVE(&wl->list, sp, list);
+		DSL(0x20, SLT_Debug, sp->vsl_id, "off waiting list");
+		if (SES_Schedule(sp)) {
+			/*
+			 * We could not schedule the session, leave the
+			 * rest on the busy list.
+			 */
+			break;
+		}
+	}
+	if (VTAILQ_EMPTY(&wl->list)) {
+		oh->waitinglist = NULL;
+		FREE_OBJ(wl);
+	}
+}
+
+/*---------------------------------------------------------------------
+ * Purge an entire objhead
+ */
+
+void
+HSH_Purge(const struct sess *sp, struct objhead *oh, double ttl, double grace)
+{
+	struct objcore *oc, **ocp;
+	unsigned spc, nobj, n;
+	struct object *o;
+
+	CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC);
+	spc = WS_Reserve(sp->wrk->ws, 0);
+	ocp = (void*)sp->wrk->ws->f;
+	Lck_Lock(&oh->mtx);
+	assert(oh->refcnt > 0);
+	nobj = 0;
+	VTAILQ_FOREACH(oc, &oh->objcs, list) {
+		CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+		assert(oc->objhead == oh);
+		if (oc->flags & OC_F_BUSY) {
+			/*
+			 * We cannot purge busy objects here, because their
+			 * owners have special rights to them, and may nuke
+			 * them without concern for the refcount, which by
+			 * definition always must be one, so they don't check.
+			 */
+			continue;
+		}
+
+		(void)oc_getobj(sp->wrk, oc); /* XXX: still needed ? */
+
+		xxxassert(spc >= sizeof *ocp);
+		oc->refcnt++;
+		spc -= sizeof *ocp;
+		ocp[nobj++] = oc;
+	}
+	Lck_Unlock(&oh->mtx);
+
+	/* NB: inverse test to catch NAN also */
+	if (!(ttl > 0.))
+		ttl = -1.;
+	if (!(grace > 0.))
+		grace = -1.;
+	for (n = 0; n < nobj; n++) {
+		oc = ocp[n];
+		CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+		o = oc_getobj(sp->wrk, oc);
+		if (o == NULL)
+			continue;
+		CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
+		o->exp.ttl = ttl;
+		o->exp.grace = grace;
+		EXP_Rearm(o);
+		(void)HSH_Deref(sp->wrk, NULL, &o);
+	}
+	WS_Release(sp->wrk->ws, 0);
+}
+
+
+/*---------------------------------------------------------------------
+ * Kill a busy object we don't need anyway.
+ * There may be sessions on the waiting list, so we cannot just blow
+ * it out of the water.
+ */
+
+void
+HSH_Drop(struct sess *sp)
+{
+	struct object *o;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	o = sp->obj;
+	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
+	AssertObjCorePassOrBusy(o->objcore);
+	o->exp.ttl = -1.;
+	if (o->objcore != NULL)		/* Pass has no objcore */
+		HSH_Unbusy(sp);
+	(void)HSH_Deref(sp->wrk, NULL, &sp->obj);
+}
+
+void
+HSH_Unbusy(const struct sess *sp)
+{
+	struct object *o;
+	struct objhead *oh;
+	struct objcore *oc;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	o = sp->obj;
+	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
+	oc = o->objcore;
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	oh = oc->objhead;
+	CHECK_OBJ(oh, OBJHEAD_MAGIC);
+
+	AssertObjBusy(o);
+	AN(oc->ban);
+	assert(oc->refcnt > 0);
+	assert(oh->refcnt > 0);
+	if (o->ws_o->overflow)
+		sp->wrk->stats.n_objoverflow++;
+	if (cache_param->diag_bitmap & 0x40)
+		WSP(sp, SLT_Debug,
+		    "Object %u workspace free %u", o->xid, WS_Free(o->ws_o));
+
+	/* XXX: pretouch neighbors on oh->objcs to prevent page-on under mtx */
+	Lck_Lock(&oh->mtx);
+	assert(oh->refcnt > 0);
+	/* XXX: strictly speaking, we should sort in Date: order. */
+	VTAILQ_REMOVE(&oh->objcs, oc, list);
+	VTAILQ_INSERT_HEAD(&oh->objcs, oc, list);
+	oc->flags &= ~OC_F_BUSY;
+	AZ(sp->wrk->nbusyobj);
+	sp->wrk->nbusyobj = oc->busyobj;
+	oc->busyobj = NULL;
+	if (oh->waitinglist != NULL)
+		hsh_rush(oh);
+	AN(oc->ban);
+	Lck_Unlock(&oh->mtx);
+	assert(oc_getobj(sp->wrk, oc) == o);
+}
+
+void
+HSH_Ref(struct objcore *oc)
+{
+	struct objhead *oh;
+
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	oh = oc->objhead;
+	CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC);
+	Lck_Lock(&oh->mtx);
+	assert(oc->refcnt > 0);
+	oc->refcnt++;
+	Lck_Unlock(&oh->mtx);
+}
+
+/*--------------------------------------------------------------------
+ * Dereference objcore and or object
+ *
+ * Can deal with:
+ *	bare objcore (incomplete fetch)
+ *	bare object (pass)
+ *	object with objcore
+ *	XXX later:  objcore with object (?)
+ *
+ * But you can only supply one of the two arguments at a time.
+ *
+ * Returns zero if target was destroyed.
+ */
+
+int
+HSH_Deref(struct worker *w, struct objcore *oc, struct object **oo)
+{
+	struct object *o = NULL;
+	struct objhead *oh;
+	unsigned r;
+
+	/* Only one arg at a time */
+	assert(oc == NULL || oo == NULL);
+
+	if (oo != NULL) {
+		o = *oo;
+		*oo = NULL;
+		CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
+		oc = o->objcore;
+	}
+
+	if (o != NULL && oc == NULL) {
+		/*
+		 * A pass object with neither objcore nor objhdr reference.
+		 * -> simply free the (Transient) storage
+		 */
+		STV_Freestore(o);
+		STV_free(o->objstore);
+		w->stats.n_object--;
+		return (0);
+	}
+
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+
+	oh = oc->objhead;
+	CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC);
+
+	Lck_Lock(&oh->mtx);
+	assert(oh->refcnt > 0);
+	assert(oc->refcnt > 0);
+	r = --oc->refcnt;
+	if (!r)
+		VTAILQ_REMOVE(&oh->objcs, oc, list);
+	else {
+		/* Must have an object */
+		AN(oc->methods);
+	}
+	if (oh->waitinglist != NULL)
+		hsh_rush(oh);
+	Lck_Unlock(&oh->mtx);
+	if (r != 0)
+		return (r);
+
+	BAN_DestroyObj(oc);
+	AZ(oc->ban);
+
+	if (oc->flags & OC_F_BUSY) {
+		CHECK_OBJ_NOTNULL(oc->busyobj, BUSYOBJ_MAGIC);
+		if (w->nbusyobj == NULL)
+			w->nbusyobj = oc->busyobj;
+		else
+			FREE_OBJ(oc->busyobj);
+		oc->busyobj = NULL;
+	}
+	AZ(oc->busyobj);
+
+	if (oc->methods != NULL) {
+		oc_freeobj(oc);
+		w->stats.n_object--;
+	}
+	FREE_OBJ(oc);
+
+	w->stats.n_objectcore--;
+	/* Drop our ref on the objhead */
+	assert(oh->refcnt > 0);
+	if (hash->deref(oh))
+		return (0);
+	HSH_DeleteObjHead(w, oh);
+	return (0);
+}
+
+void
+HSH_Init(const struct hash_slinger *slinger)
+{
+
+	assert(DIGEST_LEN == SHA256_LEN);	/* avoid #include pollution */
+	hash = slinger;
+	if (hash->start != NULL)
+		hash->start();
+}
diff --git a/bin/varnishd/cache/cache_http.c b/bin/varnishd/cache/cache_http.c
new file mode 100644
index 0000000..784eb28
--- /dev/null
+++ b/bin/varnishd/cache/cache_http.c
@@ -0,0 +1,1119 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * HTTP request storage and manipulation
+ */
+
+#include "config.h"
+
+#include <stdio.h>
+
+#include "cache.h"
+
+#include "vct.h"
+
+#define HTTPH(a, b, c, d, e, f, g) char b[] = "*" a ":";
+#include "tbl/http_headers.h"
+#undef HTTPH
+
+/*lint -save -e773 not () */
+#define LOGMTX2(ax, bx, cx)	[bx] = SLT_##ax##cx
+
+#define LOGMTX1(ax) {					\
+	LOGMTX2(ax, HTTP_HDR_REQ,	Request),	\
+	LOGMTX2(ax, HTTP_HDR_RESPONSE,	Response),	\
+	LOGMTX2(ax, HTTP_HDR_STATUS,	Status),	\
+	LOGMTX2(ax, HTTP_HDR_URL,	URL),		\
+	LOGMTX2(ax, HTTP_HDR_PROTO,	Protocol),	\
+	LOGMTX2(ax, HTTP_HDR_FIRST,	Header),	\
+	}
+
+static const enum VSL_tag_e logmtx[][HTTP_HDR_FIRST + 1] = {
+	[HTTP_Rx] = LOGMTX1(Rx),
+	[HTTP_Tx] = LOGMTX1(Tx),
+	[HTTP_Obj] = LOGMTX1(Obj)
+};
+/*lint -restore */
+
+static enum VSL_tag_e
+http2shmlog(const struct http *hp, int t)
+{
+
+	CHECK_OBJ_NOTNULL(hp, HTTP_MAGIC);
+	if (t > HTTP_HDR_FIRST)
+		t = HTTP_HDR_FIRST;
+	assert(hp->logtag >= HTTP_Rx && hp->logtag <= HTTP_Obj); /*lint !e685*/
+	assert(t >= HTTP_HDR_REQ && t <= HTTP_HDR_FIRST);
+	return (logmtx[hp->logtag][t]);
+}
+
+static void
+WSLH(struct worker *w, unsigned vsl_id, const struct http *hp, unsigned hdr)
+{
+
+	AN(vsl_id & (VSL_CLIENTMARKER|VSL_BACKENDMARKER));
+	WSLR(w, http2shmlog(hp, hdr), vsl_id, hp->hd[hdr]);
+}
+
+/*--------------------------------------------------------------------*/
+/* List of canonical HTTP response code names from RFC2616 */
+
+static struct http_msg {
+	unsigned	nbr;
+	const char	*txt;
+} http_msg[] = {
+#define HTTP_RESP(n, t)	{ n, t},
+#include "tbl/http_response.h"
+	{ 0, NULL }
+};
+
+const char *
+http_StatusMessage(unsigned status)
+{
+	struct http_msg *mp;
+
+	assert(status >= 100 && status <= 999);
+	for (mp = http_msg; mp->nbr != 0 && mp->nbr <= status; mp++)
+		if (mp->nbr == status)
+			return (mp->txt);
+	return ("Unknown Error");
+}
+
+/*--------------------------------------------------------------------*/
+
+unsigned
+HTTP_estimate(unsigned nhttp)
+{
+
+	/* XXX: We trust the structs to size-aligned as necessary */
+	return (sizeof (struct http) + (sizeof (txt) + 1) * nhttp);
+}
+
+struct http *
+HTTP_create(void *p, uint16_t nhttp)
+{
+	struct http *hp;
+
+	hp = p;
+	hp->magic = HTTP_MAGIC;
+	hp->hd = (void*)(hp + 1);
+	hp->shd = nhttp;
+	hp->hdf = (void*)(hp->hd + nhttp);
+	return (hp);
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+http_Setup(struct http *hp, struct ws *ws)
+{
+	uint16_t shd;
+	txt *hd;
+	unsigned char *hdf;
+
+	/* XXX: This is not elegant, is it efficient ? */
+	shd = hp->shd;
+	hd = hp->hd;
+	hdf = hp->hdf;
+	memset(hp, 0, sizeof *hp);
+	memset(hd, 0, sizeof *hd * shd);
+	memset(hdf, 0, sizeof *hdf * shd);
+	hp->magic = HTTP_MAGIC;
+	hp->ws = ws;
+	hp->nhd = HTTP_HDR_FIRST;
+	hp->shd = shd;
+	hp->hd = hd;
+	hp->hdf = hdf;
+}
+
+/*--------------------------------------------------------------------*/
+
+static int
+http_IsHdr(const txt *hh, const char *hdr)
+{
+	unsigned l;
+
+	Tcheck(*hh);
+	AN(hdr);
+	l = hdr[0];
+	assert(l == strlen(hdr + 1));
+	assert(hdr[l] == ':');
+	hdr++;
+	return (!strncasecmp(hdr, hh->b, l));
+}
+
+/*--------------------------------------------------------------------
+ * This function collapses multiple headerlines of the same name.
+ * The lines are joined with a comma, according to [rfc2616, 4.2bot, p32]
+ */
+
+void
+http_CollectHdr(struct http *hp, const char *hdr)
+{
+	unsigned u, v, ml, f = 0, x;
+	char *b = NULL, *e = NULL;
+
+	for (u = HTTP_HDR_FIRST; u < hp->nhd; u++) {
+		while (u < hp->nhd && http_IsHdr(&hp->hd[u], hdr)) {
+			Tcheck(hp->hd[u]);
+			if (f == 0) {
+				/* Found first header, just record the fact */
+				f = u;
+				break;
+			}
+			if (b == NULL) {
+				/* Found second header, start our collection */
+				ml = WS_Reserve(hp->ws, 0);
+				b = hp->ws->f;
+				e = b + ml;
+				x = Tlen(hp->hd[f]);
+				if (b + x < e) {
+					memcpy(b, hp->hd[f].b, x);
+					b += x;
+				} else
+					b = e;
+			}
+
+			AN(b);
+			AN(e);
+
+			/* Append the Nth header we found */
+			if (b < e)
+				*b++ = ',';
+			x = Tlen(hp->hd[u]) - *hdr;
+			if (b + x < e) {
+				memcpy(b, hp->hd[u].b + *hdr, x);
+				b += x;
+			} else
+				b = e;
+
+			/* Shift remaining headers up one slot */
+			for (v = u; v < hp->nhd - 1; v++)
+				hp->hd[v] = hp->hd[v + 1];
+			hp->nhd--;
+		}
+
+	}
+	if (b == NULL)
+		return;
+	AN(e);
+	if (b >= e) {
+		WS_Release(hp->ws, 0);
+		return;
+	}
+	*b = '\0';
+	hp->hd[f].b = hp->ws->f;
+	hp->hd[f].e = b;
+	WS_ReleaseP(hp->ws, b + 1);
+}
+
+
+/*--------------------------------------------------------------------*/
+
+static unsigned
+http_findhdr(const struct http *hp, unsigned l, const char *hdr)
+{
+	unsigned u;
+
+	for (u = HTTP_HDR_FIRST; u < hp->nhd; u++) {
+		Tcheck(hp->hd[u]);
+		if (hp->hd[u].e < hp->hd[u].b + l + 1)
+			continue;
+		if (hp->hd[u].b[l] != ':')
+			continue;
+		if (strncasecmp(hdr, hp->hd[u].b, l))
+			continue;
+		return (u);
+	}
+	return (0);
+}
+
+int
+http_GetHdr(const struct http *hp, const char *hdr, char **ptr)
+{
+	unsigned u, l;
+	char *p;
+
+	l = hdr[0];
+	diagnostic(l == strlen(hdr + 1));
+	assert(hdr[l] == ':');
+	hdr++;
+	u = http_findhdr(hp, l - 1, hdr);
+	if (u == 0) {
+		if (ptr != NULL)
+			*ptr = NULL;
+		return (0);
+	}
+	if (ptr != NULL) {
+		p = hp->hd[u].b + l;
+		while (vct_issp(*p))
+			p++;
+		*ptr = p;
+	}
+	return (1);
+}
+
+
+/*--------------------------------------------------------------------
+ * Find a given data element in a header according to RFC2616's #rule
+ * (section 2.1, p15)
+ */
+
+int
+http_GetHdrData(const struct http *hp, const char *hdr,
+    const char *field, char **ptr)
+{
+	char *h, *e;
+	unsigned fl;
+
+	if (ptr != NULL)
+		*ptr = NULL;
+	if (!http_GetHdr(hp, hdr, &h))
+		return (0);
+	AN(h);
+	e = strchr(h, '\0');
+	fl = strlen(field);
+	while (h + fl <= e) {
+		/* Skip leading whitespace and commas */
+		if (vct_islws(*h) || *h == ',') {
+			h++;
+			continue;
+		}
+		/* Check for substrings before memcmp() */
+		if ((h + fl == e || vct_issepctl(h[fl])) &&
+		    !memcmp(h, field, fl)) {
+			if (ptr != NULL) {
+				h += fl;
+				while (vct_islws(*h))
+					h++;
+				*ptr = h;
+			}
+			return (1);
+		}
+		/* Skip until end of header or comma */
+		while (*h && *h != ',')
+			h++;
+	}
+	return (0);
+}
+
+/*--------------------------------------------------------------------
+ * Find a given headerfields Q value.
+ */
+
+double
+http_GetHdrQ(const struct http *hp, const char *hdr, const char *field)
+{
+	char *h;
+	int i;
+	double a, b;
+
+	h = NULL;
+	i = http_GetHdrData(hp, hdr, field, &h);
+	if (!i)
+		return (0.);
+
+	if (h == NULL)
+		return (1.);
+	/* Skip whitespace, looking for '=' */
+	while (*h && vct_issp(*h))
+		h++;
+	if (*h++ != ';')
+		return (1.);
+	while (*h && vct_issp(*h))
+		h++;
+	if (*h++ != 'q')
+		return (1.);
+	while (*h && vct_issp(*h))
+		h++;
+	if (*h++ != '=')
+		return (1.);
+	while (*h && vct_issp(*h))
+		h++;
+	a = 0.;
+	while (vct_isdigit(*h)) {
+		a *= 10.;
+		a += *h - '0';
+		h++;
+	}
+	if (*h++ != '.')
+		return (a);
+	b = .1;
+	while (vct_isdigit(*h)) {
+		a += b * (*h - '0');
+		b *= .1;
+		h++;
+	}
+	return (a);
+}
+
+/*--------------------------------------------------------------------
+ * Find a given headerfields value.
+ */
+
+int
+http_GetHdrField(const struct http *hp, const char *hdr,
+    const char *field, char **ptr)
+{
+	char *h;
+	int i;
+
+	if (ptr != NULL)
+		*ptr = NULL;
+
+	h = NULL;
+	i = http_GetHdrData(hp, hdr, field, &h);
+	if (!i)
+		return (i);
+
+	if (ptr != NULL && h != NULL) {
+		/* Skip whitespace, looking for '=' */
+		while (*h && vct_issp(*h))
+			h++;
+		if (*h == '=') {
+			h++;
+			while (*h && vct_issp(*h))
+				h++;
+			*ptr = h;
+		}
+	}
+	return (i);
+}
+
+/*--------------------------------------------------------------------
+ * XXX: redo with http_GetHdrField() ?
+ */
+
+const char *
+http_DoConnection(const struct http *hp)
+{
+	char *p, *q;
+	const char *ret;
+	unsigned u;
+
+	if (!http_GetHdr(hp, H_Connection, &p)) {
+		if (hp->protover < 11)
+			return ("not HTTP/1.1");
+		return (NULL);
+	}
+	ret = NULL;
+	AN(p);
+	for (; *p; p++) {
+		if (vct_issp(*p))
+			continue;
+		if (*p == ',')
+			continue;
+		for (q = p + 1; *q; q++)
+			if (*q == ',' || vct_issp(*q))
+				break;
+		u = pdiff(p, q);
+		if (u == 5 && !strncasecmp(p, "close", u))
+			ret = "Connection: close";
+		u = http_findhdr(hp, u, p);
+		if (u != 0)
+			hp->hdf[u] |= HDF_FILTER;
+		if (!*q)
+			break;
+		p = q;
+	}
+	return (ret);
+}
+
+/*--------------------------------------------------------------------*/
+
+int
+http_HdrIs(const struct http *hp, const char *hdr, const char *val)
+{
+	char *p;
+
+	if (!http_GetHdr(hp, hdr, &p))
+		return (0);
+	AN(p);
+	if (!strcasecmp(p, val))
+		return (1);
+	return (0);
+}
+
+/*--------------------------------------------------------------------*/
+
+uint16_t
+http_GetStatus(const struct http *hp)
+{
+
+	return (hp->status);
+}
+
+const char *
+http_GetReq(const struct http *hp)
+{
+
+	Tcheck(hp->hd[HTTP_HDR_REQ]);
+	return (hp->hd[HTTP_HDR_REQ].b);
+}
+
+/*--------------------------------------------------------------------
+ * Dissect the headers of the HTTP protocol message.
+ * Detect conditionals (headers which start with '^[Ii][Ff]-')
+ */
+
+static uint16_t
+http_dissect_hdrs(struct worker *w, struct http *hp, unsigned vsl_id, char *p,
+    const struct http_conn *htc)
+{
+	char *q, *r;
+	txt t = htc->rxbuf;
+
+	if (*p == '\r')
+		p++;
+
+	hp->nhd = HTTP_HDR_FIRST;
+	hp->conds = 0;
+	r = NULL;		/* For FlexeLint */
+	for (; p < t.e; p = r) {
+
+		/* Find end of next header */
+		q = r = p;
+		while (r < t.e) {
+			if (!vct_iscrlf(*r)) {
+				r++;
+				continue;
+			}
+			q = r;
+			assert(r < t.e);
+			r += vct_skipcrlf(r);
+			if (r >= t.e)
+				break;
+			/* If line does not continue: got it. */
+			if (!vct_issp(*r))
+				break;
+
+			/* Clear line continuation LWS to spaces */
+			while (vct_islws(*q))
+				*q++ = ' ';
+		}
+
+		if (q - p > htc->maxhdr) {
+			VSC_C_main->losthdr++;
+			WSL(w, SLT_LostHeader, vsl_id, "%.*s",
+			    q - p > 20 ? 20 : q - p, p);
+			return (413);
+		}
+
+		/* Empty header = end of headers */
+		if (p == q)
+			break;
+
+		if ((p[0] == 'i' || p[0] == 'I') &&
+		    (p[1] == 'f' || p[1] == 'F') &&
+		    p[2] == '-')
+			hp->conds = 1;
+
+		while (q > p && vct_issp(q[-1]))
+			q--;
+		*q = '\0';
+
+		if (hp->nhd < hp->shd) {
+			hp->hdf[hp->nhd] = 0;
+			hp->hd[hp->nhd].b = p;
+			hp->hd[hp->nhd].e = q;
+			WSLH(w, vsl_id, hp, hp->nhd);
+			hp->nhd++;
+		} else {
+			VSC_C_main->losthdr++;
+			WSL(w, SLT_LostHeader, vsl_id, "%.*s",
+			    q - p > 20 ? 20 : q - p, p);
+			return (413);
+		}
+	}
+	return (0);
+}
+
+/*--------------------------------------------------------------------
+ * Deal with first line of HTTP protocol message.
+ */
+
+static uint16_t
+http_splitline(struct worker *w, unsigned vsl_id, struct http *hp,
+    const struct http_conn *htc, int h1, int h2, int h3)
+{
+	char *p, *q;
+
+	CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC);
+	CHECK_OBJ_NOTNULL(hp, HTTP_MAGIC);
+
+	/* XXX: Assert a NUL at rx.e ? */
+	Tcheck(htc->rxbuf);
+
+	/* Skip leading LWS */
+	for (p = htc->rxbuf.b ; vct_islws(*p); p++)
+		continue;
+
+	/* First field cannot contain SP, CRLF or CTL */
+	q = p;
+	for (; !vct_issp(*p); p++) {
+		if (vct_isctl(*p))
+			return (400);
+	}
+	hp->hd[h1].b = q;
+	hp->hd[h1].e = p;
+
+	/* Skip SP */
+	for (; vct_issp(*p); p++) {
+		if (vct_isctl(*p))
+			return (400);
+	}
+
+	/* Second field cannot contain LWS or CTL */
+	q = p;
+	for (; !vct_islws(*p); p++) {
+		if (vct_isctl(*p))
+			return (400);
+	}
+	hp->hd[h2].b = q;
+	hp->hd[h2].e = p;
+
+	if (!Tlen(hp->hd[h2]))
+		return (413);
+
+	/* Skip SP */
+	for (; vct_issp(*p); p++) {
+		if (vct_isctl(*p))
+			return (400);
+	}
+
+	/* Third field is optional and cannot contain CTL */
+	q = p;
+	if (!vct_iscrlf(*p)) {
+		for (; !vct_iscrlf(*p); p++)
+			if (!vct_issep(*p) && vct_isctl(*p))
+				return (400);
+	}
+	hp->hd[h3].b = q;
+	hp->hd[h3].e = p;
+
+	/* Skip CRLF */
+	p += vct_skipcrlf(p);
+
+	*hp->hd[h1].e = '\0';
+	WSLH(w, vsl_id, hp, h1);
+
+	*hp->hd[h2].e = '\0';
+	WSLH(w, vsl_id, hp, h2);
+
+	if (hp->hd[h3].e != NULL) {
+		*hp->hd[h3].e = '\0';
+		WSLH(w, vsl_id, hp, h3);
+	}
+
+	return (http_dissect_hdrs(w, hp, vsl_id, p, htc));
+}
+
+/*--------------------------------------------------------------------*/
+
+static void
+http_ProtoVer(struct http *hp)
+{
+
+	if (!strcasecmp(hp->hd[HTTP_HDR_PROTO].b, "HTTP/1.0"))
+		hp->protover = 10;
+	else if (!strcasecmp(hp->hd[HTTP_HDR_PROTO].b, "HTTP/1.1"))
+		hp->protover = 11;
+	else
+		hp->protover = 9;
+}
+
+
+/*--------------------------------------------------------------------*/
+
+uint16_t
+http_DissectRequest(struct sess *sp)
+{
+	struct http_conn *htc;
+	struct http *hp;
+	uint16_t retval;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	htc = sp->htc;
+	CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC);
+	hp = sp->http;
+	CHECK_OBJ_NOTNULL(hp, HTTP_MAGIC);
+
+	hp->logtag = HTTP_Rx;
+
+	retval = http_splitline(sp->wrk, sp->vsl_id, hp, htc,
+	    HTTP_HDR_REQ, HTTP_HDR_URL, HTTP_HDR_PROTO);
+	if (retval != 0) {
+		WSPR(sp, SLT_HttpGarbage, htc->rxbuf);
+		return (retval);
+	}
+	http_ProtoVer(hp);
+	return (retval);
+}
+
+/*--------------------------------------------------------------------*/
+
+uint16_t
+http_DissectResponse(struct worker *w, const struct http_conn *htc,
+    struct http *hp)
+{
+	int j;
+	uint16_t retval = 0;
+	char *p;
+
+
+	CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC);
+	CHECK_OBJ_NOTNULL(hp, HTTP_MAGIC);
+	hp->logtag = HTTP_Rx;
+
+	if (http_splitline(w, htc->vsl_id, hp, htc,
+	    HTTP_HDR_PROTO, HTTP_HDR_STATUS, HTTP_HDR_RESPONSE))
+		retval = 503;
+
+	if (retval == 0 && memcmp(hp->hd[HTTP_HDR_PROTO].b, "HTTP/1.", 7))
+		retval = 503;
+
+	if (retval == 0 && Tlen(hp->hd[HTTP_HDR_STATUS]) != 3)
+		retval = 503;
+
+	if (retval == 0) {
+		hp->status = 0;
+		p = hp->hd[HTTP_HDR_STATUS].b;
+		for (j = 100; j != 0; j /= 10) {
+			if (!vct_isdigit(*p)) {
+				retval = 503;
+				break;
+			}
+			hp->status += (uint16_t)(j * (*p - '0'));
+			p++;
+		}
+		if (*p != '\0')
+			retval = 503;
+	}
+
+	if (retval != 0) {
+		WSLR(w, SLT_HttpGarbage, htc->vsl_id, htc->rxbuf);
+		assert(retval >= 100 && retval <= 999);
+		hp->status = retval;
+	} else {
+		http_ProtoVer(hp);
+	}
+
+	if (hp->hd[HTTP_HDR_RESPONSE].b == NULL ||
+	    !Tlen(hp->hd[HTTP_HDR_RESPONSE])) {
+		/* Backend didn't send a response string, use the standard */
+		hp->hd[HTTP_HDR_RESPONSE].b =
+		    TRUST_ME(http_StatusMessage(hp->status));
+		hp->hd[HTTP_HDR_RESPONSE].e =
+		    strchr(hp->hd[HTTP_HDR_RESPONSE].b, '\0');
+	}
+	return (retval);
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+http_SetH(const struct http *to, unsigned n, const char *fm)
+{
+
+	assert(n < to->shd);
+	AN(fm);
+	to->hd[n].b = TRUST_ME(fm);
+	to->hd[n].e = strchr(to->hd[n].b, '\0');
+	to->hdf[n] = 0;
+}
+
+static void
+http_copyh(const struct http *to, const struct http *fm, unsigned n)
+{
+
+	assert(n < HTTP_HDR_FIRST);
+	Tcheck(fm->hd[n]);
+	to->hd[n] = fm->hd[n];
+	to->hdf[n] = fm->hdf[n];
+}
+
+void
+http_ForceGet(const struct http *to)
+{
+	if (strcmp(http_GetReq(to), "GET"))
+		http_SetH(to, HTTP_HDR_REQ, "GET");
+}
+
+void
+http_CopyResp(struct http *to, const struct http *fm)
+{
+
+	CHECK_OBJ_NOTNULL(fm, HTTP_MAGIC);
+	CHECK_OBJ_NOTNULL(to, HTTP_MAGIC);
+	http_SetH(to, HTTP_HDR_PROTO, "HTTP/1.1");
+	to->status = fm->status;
+	http_copyh(to, fm, HTTP_HDR_RESPONSE);
+}
+
+void
+http_SetResp(struct http *to, const char *proto, uint16_t status,
+    const char *response)
+{
+
+	CHECK_OBJ_NOTNULL(to, HTTP_MAGIC);
+	http_SetH(to, HTTP_HDR_PROTO, proto);
+	assert(status >= 100 && status <= 999);
+	to->status = status;
+	http_SetH(to, HTTP_HDR_RESPONSE, response);
+}
+
+static void
+http_copyheader(struct worker *w, unsigned vsl_id, struct http *to,
+    const struct http *fm, unsigned n)
+{
+
+	CHECK_OBJ_NOTNULL(fm, HTTP_MAGIC);
+	CHECK_OBJ_NOTNULL(to, HTTP_MAGIC);
+	assert(n < fm->shd);
+	Tcheck(fm->hd[n]);
+	if (to->nhd < to->shd) {
+		to->hd[to->nhd] = fm->hd[n];
+		to->hdf[to->nhd] = 0;
+		to->nhd++;
+	} else  {
+		VSC_C_main->losthdr++;
+		WSLR(w, SLT_LostHeader, vsl_id, fm->hd[n]);
+	}
+}
+
+/*--------------------------------------------------------------------
+ * Estimate how much workspace we need to Filter this header according
+ * to 'how'.
+ */
+
+unsigned
+http_EstimateWS(const struct http *fm, unsigned how, uint16_t *nhd)
+{
+	unsigned u, l;
+
+	l = 0;
+	*nhd = HTTP_HDR_FIRST;
+	CHECK_OBJ_NOTNULL(fm, HTTP_MAGIC);
+	for (u = 0; u < fm->nhd; u++) {
+		if (fm->hd[u].b == NULL)
+			continue;
+		if (fm->hdf[u] & HDF_FILTER)
+			continue;
+#define HTTPH(a, b, c, d, e, f, g) \
+		if (((e) & how) && http_IsHdr(&fm->hd[u], (b))) \
+			continue;
+#include "tbl/http_headers.h"
+#undef HTTPH
+		l += PRNDUP(Tlen(fm->hd[u]) + 1);
+		(*nhd)++;
+		// fm->hdf[u] |= HDF_COPY;
+	}
+	return (l);
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+http_FilterFields(struct worker *w, unsigned vsl_id, struct http *to,
+    const struct http *fm, unsigned how)
+{
+	unsigned u;
+
+	CHECK_OBJ_NOTNULL(fm, HTTP_MAGIC);
+	CHECK_OBJ_NOTNULL(to, HTTP_MAGIC);
+	to->nhd = HTTP_HDR_FIRST;
+	to->status = fm->status;
+	for (u = HTTP_HDR_FIRST; u < fm->nhd; u++) {
+		if (fm->hd[u].b == NULL)
+			continue;
+		if (fm->hdf[u] & HDF_FILTER)
+			continue;
+#define HTTPH(a, b, c, d, e, f, g) \
+		if (((e) & how) && http_IsHdr(&fm->hd[u], (b))) \
+			continue;
+#include "tbl/http_headers.h"
+#undef HTTPH
+		http_copyheader(w, vsl_id, to, fm, u);
+	}
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+http_FilterHeader(const struct sess *sp, unsigned how)
+{
+	struct http *hp;
+
+	hp = sp->wrk->bereq;
+	CHECK_OBJ_NOTNULL(hp, HTTP_MAGIC);
+	hp->logtag = HTTP_Tx;
+
+	http_copyh(hp, sp->http, HTTP_HDR_REQ);
+	http_copyh(hp, sp->http, HTTP_HDR_URL);
+	if (how == HTTPH_R_FETCH)
+		http_SetH(hp, HTTP_HDR_PROTO, "HTTP/1.1");
+	else
+		http_copyh(hp, sp->http, HTTP_HDR_PROTO);
+	http_FilterFields(sp->wrk, sp->vsl_id, hp, sp->http, how);
+	http_PrintfHeader(sp->wrk, sp->vsl_id, hp, "X-Varnish: %u", sp->xid);
+}
+
+/*--------------------------------------------------------------------
+ * This function copies any header fields which reference foreign
+ * storage into our own WS.
+ */
+
+void
+http_CopyHome(struct worker *w, unsigned vsl_id, const struct http *hp)
+{
+	unsigned u, l;
+	char *p;
+
+	for (u = 0; u < hp->nhd; u++) {
+		if (hp->hd[u].b == NULL)
+			continue;
+		if (hp->hd[u].b >= hp->ws->s && hp->hd[u].e <= hp->ws->e) {
+			WSLH(w, vsl_id, hp, u);
+			continue;
+		}
+		l = Tlen(hp->hd[u]);
+		p = WS_Alloc(hp->ws, l + 1);
+		if (p != NULL) {
+			WSLH(w, vsl_id, hp, u);
+			memcpy(p, hp->hd[u].b, l + 1L);
+			hp->hd[u].b = p;
+			hp->hd[u].e = p + l;
+		} else {
+			/* XXX This leaves a slot empty */
+			VSC_C_main->losthdr++;
+			WSLR(w, SLT_LostHeader, vsl_id, hp->hd[u]);
+			hp->hd[u].b = NULL;
+			hp->hd[u].e = NULL;
+		}
+	}
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+http_ClrHeader(struct http *to)
+{
+
+	CHECK_OBJ_NOTNULL(to, HTTP_MAGIC);
+	to->nhd = HTTP_HDR_FIRST;
+	to->status = 0;
+	to->protover = 0;
+	to->conds = 0;
+	memset(to->hd, 0, sizeof *to->hd * to->shd);
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+http_SetHeader(struct worker *w, unsigned vsl_id, struct http *to,
+    const char *hdr)
+{
+
+	CHECK_OBJ_NOTNULL(to, HTTP_MAGIC);
+	if (to->nhd >= to->shd) {
+		VSC_C_main->losthdr++;
+		WSL(w, SLT_LostHeader, vsl_id, "%s", hdr);
+		return;
+	}
+	http_SetH(to, to->nhd++, hdr);
+}
+
+/*--------------------------------------------------------------------*/
+
+static void
+http_PutField(struct worker *w, unsigned vsl_id, const struct http *to,
+    int field, const char *string)
+{
+	char *p;
+	unsigned l;
+
+	CHECK_OBJ_NOTNULL(to, HTTP_MAGIC);
+	l = strlen(string);
+	p = WS_Alloc(to->ws, l + 1);
+	if (p == NULL) {
+		WSL(w, SLT_LostHeader, vsl_id, "%s", string);
+		to->hd[field].b = NULL;
+		to->hd[field].e = NULL;
+		to->hdf[field] = 0;
+	} else {
+		memcpy(p, string, l + 1L);
+		to->hd[field].b = p;
+		to->hd[field].e = p + l;
+		to->hdf[field] = 0;
+	}
+}
+
+void
+http_PutProtocol(struct worker *w, unsigned vsl_id, const struct http *to,
+    const char *protocol)
+{
+
+	http_PutField(w, vsl_id, to, HTTP_HDR_PROTO, protocol);
+	if (to->hd[HTTP_HDR_PROTO].b == NULL)
+		http_SetH(to, HTTP_HDR_PROTO, "HTTP/1.1");
+	Tcheck(to->hd[HTTP_HDR_PROTO]);
+}
+
+void
+http_PutStatus(struct http *to, uint16_t status)
+{
+
+	assert(status >= 100 && status <= 999);
+	to->status = status;
+}
+
+void
+http_PutResponse(struct worker *w, unsigned vsl_id, const struct http *to,
+    const char *response)
+{
+
+	http_PutField(w, vsl_id, to, HTTP_HDR_RESPONSE, response);
+	if (to->hd[HTTP_HDR_RESPONSE].b == NULL)
+		http_SetH(to, HTTP_HDR_RESPONSE, "Lost Response");
+	Tcheck(to->hd[HTTP_HDR_RESPONSE]);
+}
+
+void
+http_PrintfHeader(struct worker *w, unsigned vsl_id, struct http *to,
+    const char *fmt, ...)
+{
+	va_list ap;
+	unsigned l, n;
+
+	CHECK_OBJ_NOTNULL(to, HTTP_MAGIC);
+	l = WS_Reserve(to->ws, 0);
+	va_start(ap, fmt);
+	n = vsnprintf(to->ws->f, l, fmt, ap);
+	va_end(ap);
+	if (n + 1 >= l || to->nhd >= to->shd) {
+		VSC_C_main->losthdr++;
+		WSL(w, SLT_LostHeader, vsl_id, "%s", to->ws->f);
+		WS_Release(to->ws, 0);
+	} else {
+		to->hd[to->nhd].b = to->ws->f;
+		to->hd[to->nhd].e = to->ws->f + n;
+		to->hdf[to->nhd] = 0;
+		WS_Release(to->ws, n + 1);
+		to->nhd++;
+	}
+}
+/*--------------------------------------------------------------------*/
+
+void
+http_Unset(struct http *hp, const char *hdr)
+{
+	uint16_t u, v;
+
+	for (v = u = HTTP_HDR_FIRST; u < hp->nhd; u++) {
+		if (hp->hd[u].b == NULL)
+			continue;
+		if (http_IsHdr(&hp->hd[u], hdr))
+			continue;
+		if (v != u) {
+			memcpy(&hp->hd[v], &hp->hd[u], sizeof *hp->hd);
+			memcpy(&hp->hdf[v], &hp->hdf[u], sizeof *hp->hdf);
+		}
+		v++;
+	}
+	hp->nhd = v;
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+HTTP_Copy(struct http *to, const struct http * const fm)
+{
+
+	to->conds = fm->conds;
+	to->logtag = fm->logtag;
+	to->status = fm->status;
+	to->protover = fm->protover;
+	to->nhd = fm->nhd;
+	assert(fm->nhd <= to->shd);
+	memcpy(to->hd, fm->hd, fm->nhd * sizeof *to->hd);
+	memcpy(to->hdf, fm->hdf, fm->nhd * sizeof *to->hdf);
+}
+
+/*--------------------------------------------------------------------*/
+
+unsigned
+http_Write(struct worker *w, unsigned vsl_id, const struct http *hp, int resp)
+{
+	unsigned u, l;
+
+	if (resp) {
+		l = WRW_WriteH(w, &hp->hd[HTTP_HDR_PROTO], " ");
+		WSLH(w, vsl_id, hp, HTTP_HDR_PROTO);
+
+		hp->hd[HTTP_HDR_STATUS].b = WS_Alloc(w->ws, 4);
+		AN(hp->hd[HTTP_HDR_STATUS].b);
+
+		sprintf(hp->hd[HTTP_HDR_STATUS].b, "%3d", hp->status);
+		hp->hd[HTTP_HDR_STATUS].e = hp->hd[HTTP_HDR_STATUS].b + 3;
+
+		l += WRW_WriteH(w, &hp->hd[HTTP_HDR_STATUS], " ");
+		WSLH(w, vsl_id, hp, HTTP_HDR_STATUS);
+
+		l += WRW_WriteH(w, &hp->hd[HTTP_HDR_RESPONSE], "\r\n");
+		WSLH(w, vsl_id, hp, HTTP_HDR_RESPONSE);
+	} else {
+		AN(hp->hd[HTTP_HDR_URL].b);
+		l = WRW_WriteH(w, &hp->hd[HTTP_HDR_REQ], " ");
+		WSLH(w, vsl_id, hp, HTTP_HDR_REQ);
+		l += WRW_WriteH(w, &hp->hd[HTTP_HDR_URL], " ");
+		WSLH(w, vsl_id, hp, HTTP_HDR_URL);
+		l += WRW_WriteH(w, &hp->hd[HTTP_HDR_PROTO], "\r\n");
+		WSLH(w, vsl_id, hp, HTTP_HDR_PROTO);
+	}
+	for (u = HTTP_HDR_FIRST; u < hp->nhd; u++) {
+		if (hp->hd[u].b == NULL)
+			continue;
+		AN(hp->hd[u].b);
+		AN(hp->hd[u].e);
+		l += WRW_WriteH(w, &hp->hd[u], "\r\n");
+		WSLH(w, vsl_id, hp, u);
+	}
+	l += WRW_Write(w, "\r\n", -1);
+	return (l);
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+HTTP_Init(void)
+{
+
+#define HTTPH(a, b, c, d, e, f, g) b[0] = (char)strlen(b + 1);
+#include "tbl/http_headers.h"
+#undef HTTPH
+}
diff --git a/bin/varnishd/cache/cache_httpconn.c b/bin/varnishd/cache/cache_httpconn.c
new file mode 100644
index 0000000..9e0a052
--- /dev/null
+++ b/bin/varnishd/cache/cache_httpconn.c
@@ -0,0 +1,229 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * HTTP protocol requests
+ *
+ * The trouble with the "until magic sequence" design of HTTP protocol messages
+ * is that either you have to read a single character at a time, which is
+ * inefficient, or you risk reading too much, and pre-read some of the object,
+ * or even the next pipelined request, which follows the one you want.
+ *
+ * HTC reads a HTTP protocol header into a workspace, subject to limits,
+ * and stops when we see the magic marker (double [CR]NL), and if we overshoot,
+ * it keeps track of the "pipelined" data.
+ *
+ * We use this both for client and backend connections.
+ */
+
+#include "config.h"
+
+#include "cache.h"
+
+#include "vct.h"
+
+/*--------------------------------------------------------------------
+ * Check if we have a complete HTTP request or response yet
+ *
+ * Return values:
+ *	 0  No, keep trying
+ *	>0  Yes, it is this many bytes long.
+ */
+
+static int
+htc_header_complete(txt *t)
+{
+	const char *p;
+
+	Tcheck(*t);
+	assert(*t->e == '\0');
+	/* Skip any leading white space */
+	for (p = t->b ; vct_issp(*p); p++)
+		continue;
+	if (p == t->e) {
+		/* All white space */
+		t->e = t->b;
+		*t->e = '\0';
+		return (0);
+	}
+	while (1) {
+		p = strchr(p, '\n');
+		if (p == NULL)
+			return (0);
+		p++;
+		if (*p == '\r')
+			p++;
+		if (*p == '\n')
+			break;
+	}
+	p++;
+	return (p - t->b);
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+HTC_Init(struct http_conn *htc, struct ws *ws, int fd, unsigned vsl_id,
+    unsigned maxbytes, unsigned maxhdr)
+{
+
+	htc->magic = HTTP_CONN_MAGIC;
+	htc->ws = ws;
+	htc->fd = fd;
+	htc->vsl_id = vsl_id;
+	htc->maxbytes = maxbytes;
+	htc->maxhdr = maxhdr;
+
+	(void)WS_Reserve(htc->ws, htc->maxbytes);
+	htc->rxbuf.b = ws->f;
+	htc->rxbuf.e = ws->f;
+	*htc->rxbuf.e = '\0';
+	htc->pipeline.b = NULL;
+	htc->pipeline.e = NULL;
+}
+
+/*--------------------------------------------------------------------
+ * Start over, and recycle any pipelined input.
+ * The WS_Reset is safe, even though the pipelined input is stored in
+ * the ws somewhere, because WS_Reset only fiddles pointers.
+ */
+
+int
+HTC_Reinit(struct http_conn *htc)
+{
+	unsigned l;
+
+	CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC);
+	(void)WS_Reserve(htc->ws, htc->maxbytes);
+	htc->rxbuf.b = htc->ws->f;
+	htc->rxbuf.e = htc->ws->f;
+	if (htc->pipeline.b != NULL) {
+		l = Tlen(htc->pipeline);
+		memmove(htc->rxbuf.b, htc->pipeline.b, l);
+		htc->rxbuf.e += l;
+		htc->pipeline.b = NULL;
+		htc->pipeline.e = NULL;
+	}
+	*htc->rxbuf.e = '\0';
+	return (HTC_Complete(htc));
+}
+
+/*--------------------------------------------------------------------
+ * Return 1 if we have a complete HTTP procol header
+ */
+
+int
+HTC_Complete(struct http_conn *htc)
+{
+	int i;
+
+	CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC);
+	i = htc_header_complete(&htc->rxbuf);
+	assert(i >= 0);
+	if (i == 0)
+		return (0);
+	WS_ReleaseP(htc->ws, htc->rxbuf.e);
+	AZ(htc->pipeline.b);
+	AZ(htc->pipeline.e);
+	if (htc->rxbuf.b + i < htc->rxbuf.e) {
+		htc->pipeline.b = htc->rxbuf.b + i;
+		htc->pipeline.e = htc->rxbuf.e;
+		htc->rxbuf.e = htc->pipeline.b;
+	}
+	return (1);
+}
+
+/*--------------------------------------------------------------------
+ * Receive more HTTP protocol bytes
+ * Returns:
+ *	-2 overflow
+ *	-1 error
+ *	 0 more needed
+ *	 1 got complete HTTP header
+ */
+
+int
+HTC_Rx(struct http_conn *htc)
+{
+	int i;
+
+	CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC);
+	AN(htc->ws->r);
+	i = (htc->ws->r - htc->rxbuf.e) - 1;	/* space for NUL */
+	if (i <= 0) {
+		WS_ReleaseP(htc->ws, htc->rxbuf.b);
+		return (-2);
+	}
+	i = read(htc->fd, htc->rxbuf.e, i);
+	if (i <= 0) {
+		/*
+		 * We wouldn't come here if we had a complete HTTP header
+		 * so consequently an EOF can not be OK
+		 */
+		WS_ReleaseP(htc->ws, htc->rxbuf.b);
+		return (-1);
+	}
+	htc->rxbuf.e += i;
+	*htc->rxbuf.e = '\0';
+	return (HTC_Complete(htc));
+}
+
+/*--------------------------------------------------------------------
+ * Read up to len bytes, returning pipelined data first.
+ */
+
+ssize_t
+HTC_Read(struct worker *w, struct http_conn *htc, void *d, size_t len)
+{
+	size_t l;
+	unsigned char *p;
+	ssize_t i;
+
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC);
+	l = 0;
+	p = d;
+	if (htc->pipeline.b) {
+		l = Tlen(htc->pipeline);
+		if (l > len)
+			l = len;
+		memcpy(p, htc->pipeline.b, l);
+		p += l;
+		len -= l;
+		htc->pipeline.b += l;
+		if (htc->pipeline.b == htc->pipeline.e)
+			htc->pipeline.b = htc->pipeline.e = NULL;
+	}
+	if (len == 0)
+		return (l);
+	i = read(htc->fd, p, len);
+	if (i < 0) {
+		WSL(w, SLT_FetchError, htc->vsl_id, "%s", strerror(errno));
+		return (i);
+	}
+	return (i + l);
+}
diff --git a/bin/varnishd/cache/cache_lck.c b/bin/varnishd/cache/cache_lck.c
new file mode 100644
index 0000000..2aef6dc
--- /dev/null
+++ b/bin/varnishd/cache/cache_lck.c
@@ -0,0 +1,210 @@
+/*-
+ * Copyright (c) 2008-2010 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * The geniuses who came up with pthreads did not think operations like
+ * pthread_assert_mutex_held() were important enough to include them in
+ * the API.
+ *
+ * Build our own locks on top of pthread mutexes and hope that the next
+ * civilization is better at such crucial details than this one.
+ */
+
+#include "config.h"
+
+#include <stdlib.h>
+
+#include "cache.h"
+
+/*The constability of lck depends on platform pthreads implementation */
+
+struct ilck {
+	unsigned		magic;
+#define ILCK_MAGIC		0x7b86c8a5
+	pthread_mutex_t		mtx;
+	int			held;
+	pthread_t		owner;
+	VTAILQ_ENTRY(ilck)	list;
+	const char		*w;
+	struct VSC_C_lck	*stat;
+};
+
+static VTAILQ_HEAD(, ilck)	ilck_head =
+    VTAILQ_HEAD_INITIALIZER(ilck_head);
+
+static pthread_mutex_t		lck_mtx;
+
+void __match_proto__()
+Lck__Lock(struct lock *lck, const char *p, const char *f, int l)
+{
+	struct ilck *ilck;
+	int r;
+
+	CAST_OBJ_NOTNULL(ilck, lck->priv, ILCK_MAGIC);
+	if (!(cache_param->diag_bitmap & 0x18)) {
+		AZ(pthread_mutex_lock(&ilck->mtx));
+		AZ(ilck->held);
+		ilck->stat->locks++;
+		ilck->owner = pthread_self();
+		ilck->held = 1;
+		return;
+	}
+	r = pthread_mutex_trylock(&ilck->mtx);
+	assert(r == 0 || r == EBUSY);
+	if (r) {
+		ilck->stat->colls++;
+		if (cache_param->diag_bitmap & 0x8)
+			VSL(SLT_Debug, 0, "MTX_CONTEST(%s,%s,%d,%s)",
+			    p, f, l, ilck->w);
+		AZ(pthread_mutex_lock(&ilck->mtx));
+	} else if (cache_param->diag_bitmap & 0x8) {
+		VSL(SLT_Debug, 0, "MTX_LOCK(%s,%s,%d,%s)", p, f, l, ilck->w);
+	}
+	AZ(ilck->held);
+	ilck->stat->locks++;
+	ilck->owner = pthread_self();
+	ilck->held = 1;
+}
+
+void __match_proto__()
+Lck__Unlock(struct lock *lck, const char *p, const char *f, int l)
+{
+	struct ilck *ilck;
+
+	CAST_OBJ_NOTNULL(ilck, lck->priv, ILCK_MAGIC);
+	assert(pthread_equal(ilck->owner, pthread_self()));
+	AN(ilck->held);
+	ilck->held = 0;
+	AZ(pthread_mutex_unlock(&ilck->mtx));
+	if (cache_param->diag_bitmap & 0x8)
+		VSL(SLT_Debug, 0, "MTX_UNLOCK(%s,%s,%d,%s)", p, f, l, ilck->w);
+}
+
+int __match_proto__()
+Lck__Trylock(struct lock *lck, const char *p, const char *f, int l)
+{
+	struct ilck *ilck;
+	int r;
+
+	CAST_OBJ_NOTNULL(ilck, lck->priv, ILCK_MAGIC);
+	r = pthread_mutex_trylock(&ilck->mtx);
+	assert(r == 0 || r == EBUSY);
+	if (cache_param->diag_bitmap & 0x8)
+		VSL(SLT_Debug, 0,
+		    "MTX_TRYLOCK(%s,%s,%d,%s) = %d", p, f, l, ilck->w);
+	if (r == 0) {
+		AZ(ilck->held);
+		ilck->held = 1;
+		ilck->owner = pthread_self();
+	}
+	return (r);
+}
+
+void
+Lck__Assert(const struct lock *lck, int held)
+{
+	struct ilck *ilck;
+
+	CAST_OBJ_NOTNULL(ilck, lck->priv, ILCK_MAGIC);
+	if (held)
+		assert(ilck->held &&
+		    pthread_equal(ilck->owner, pthread_self()));
+	else
+		assert(!ilck->held ||
+		    !pthread_equal(ilck->owner, pthread_self()));
+}
+
+int __match_proto__()
+Lck_CondWait(pthread_cond_t *cond, struct lock *lck, struct timespec *ts)
+{
+	struct ilck *ilck;
+	int retval = 0;
+
+	CAST_OBJ_NOTNULL(ilck, lck->priv, ILCK_MAGIC);
+	AN(ilck->held);
+	assert(pthread_equal(ilck->owner, pthread_self()));
+	ilck->held = 0;
+	if (ts == NULL) {
+		AZ(pthread_cond_wait(cond, &ilck->mtx));
+	} else {
+		retval = pthread_cond_timedwait(cond, &ilck->mtx, ts);
+		assert(retval == 0 || retval == ETIMEDOUT);
+	}
+	AZ(ilck->held);
+	ilck->held = 1;
+	ilck->owner = pthread_self();
+	return (retval);
+}
+
+void
+Lck__New(struct lock *lck, struct VSC_C_lck *st, const char *w)
+{
+	struct ilck *ilck;
+
+	AN(st);
+	AZ(lck->priv);
+	ALLOC_OBJ(ilck, ILCK_MAGIC);
+	AN(ilck);
+	ilck->w = w;
+	ilck->stat = st;
+	ilck->stat->creat++;
+	AZ(pthread_mutex_init(&ilck->mtx, NULL));
+	AZ(pthread_mutex_lock(&lck_mtx));
+	VTAILQ_INSERT_TAIL(&ilck_head, ilck, list);
+	AZ(pthread_mutex_unlock(&lck_mtx));
+	lck->priv = ilck;
+}
+
+void
+Lck_Delete(struct lock *lck)
+{
+	struct ilck *ilck;
+
+	CAST_OBJ_NOTNULL(ilck, lck->priv, ILCK_MAGIC);
+	ilck->stat->destroy++;
+	lck->priv = NULL;
+	AZ(pthread_mutex_lock(&lck_mtx));
+	VTAILQ_REMOVE(&ilck_head, ilck, list);
+	AZ(pthread_mutex_unlock(&lck_mtx));
+	AZ(pthread_mutex_destroy(&ilck->mtx));
+	FREE_OBJ(ilck);
+}
+
+#define LOCK(nam) struct VSC_C_lck *lck_##nam;
+#include "tbl/locks.h"
+#undef LOCK
+
+void
+LCK_Init(void)
+{
+
+	AZ(pthread_mutex_init(&lck_mtx, NULL));
+#define LOCK(nam)						\
+	lck_##nam = VSM_Alloc(sizeof(struct VSC_C_lck),		\
+	   VSC_CLASS, VSC_TYPE_LCK, #nam);
+#include "tbl/locks.h"
+#undef LOCK
+}
diff --git a/bin/varnishd/cache/cache_main.c b/bin/varnishd/cache/cache_main.c
new file mode 100644
index 0000000..eb3fa1d
--- /dev/null
+++ b/bin/varnishd/cache/cache_main.c
@@ -0,0 +1,146 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2009 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include "config.h"
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#include "cache.h"
+#include "common/heritage.h"
+
+#include "waiter/cache_waiter.h"
+#include "hash/hash_slinger.h"
+
+volatile struct params	*cache_param;
+
+/*--------------------------------------------------------------------
+ * Per thread storage for the session currently being processed by
+ * the thread.  This is used for panic messages.
+ */
+
+static pthread_key_t sp_key;
+
+void
+THR_SetSession(const struct sess *sp)
+{
+
+	AZ(pthread_setspecific(sp_key, sp));
+}
+
+const struct sess *
+THR_GetSession(void)
+{
+
+	return (pthread_getspecific(sp_key));
+}
+
+/*--------------------------------------------------------------------
+ * Name threads if our pthreads implementation supports it.
+ */
+
+static pthread_key_t name_key;
+
+void
+THR_SetName(const char *name)
+{
+
+	AZ(pthread_setspecific(name_key, name));
+#ifdef HAVE_PTHREAD_SET_NAME_NP
+	pthread_set_name_np(pthread_self(), name);
+#endif
+}
+
+const char *
+THR_GetName(void)
+{
+
+	return (pthread_getspecific(name_key));
+}
+
+/*--------------------------------------------------------------------
+ * XXX: Think more about which order we start things
+ */
+
+void
+child_main(void)
+{
+
+	setbuf(stdout, NULL);
+	setbuf(stderr, NULL);
+	printf("Child starts\n");
+
+	AZ(pthread_key_create(&sp_key, NULL));
+	AZ(pthread_key_create(&name_key, NULL));
+
+	THR_SetName("cache-main");
+
+	VSL_Init();	/* First, LCK needs it. */
+
+	LCK_Init();	/* Second, locking */
+
+	WAIT_Init();
+	PAN_Init();
+	CLI_Init();
+	Fetch_Init();
+
+	CNT_Init();
+	VCL_Init();
+
+	HTTP_Init();
+
+	VBE_Init();
+	VBP_Init();
+	WRK_Init();
+	Pool_Init();
+
+	EXP_Init();
+	HSH_Init(heritage.hash);
+	BAN_Init();
+
+	VCA_Init();
+
+	SMS_Init();
+	SMP_Init();
+	STV_open();
+
+	VMOD_Init();
+
+	BAN_Compile();
+
+	/* Wait for persistent storage to load if asked to */
+	if (cache_param->diag_bitmap & 0x00020000)
+		SMP_Ready();
+
+	CLI_Run();
+
+	STV_close();
+
+	printf("Child dies\n");
+}
diff --git a/bin/varnishd/cache/cache_panic.c b/bin/varnishd/cache/cache_panic.c
new file mode 100644
index 0000000..c626ae3
--- /dev/null
+++ b/bin/varnishd/cache/cache_panic.c
@@ -0,0 +1,387 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Dag-Erling Sm?rgrav <des at des.no>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include "config.h"
+
+#ifndef HAVE_EXECINFO_H
+#include "compat/execinfo.h"
+#else
+#include <execinfo.h>
+#endif
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#include "cache.h"
+
+#include "vapi/vsm_int.h"
+
+#include "cache_backend.h"
+#include "waiter/cache_waiter.h"
+#include "vcl.h"
+
+/*
+ * The panic string is constructed in memory, then copied to the
+ * shared memory.
+ *
+ * It can be extracted post-mortem from a core dump using gdb:
+ *
+ * (gdb) printf "%s", panicstr
+ */
+
+static struct vsb vsps, *vsp;
+static pthread_mutex_t panicstr_mtx = PTHREAD_MUTEX_INITIALIZER;
+
+/*--------------------------------------------------------------------*/
+
+static void
+pan_ws(const struct ws *ws, int indent)
+{
+
+	VSB_printf(vsp, "%*sws = %p { %s\n", indent, "",
+	    ws, ws->overflow ? "overflow" : "");
+	VSB_printf(vsp, "%*sid = \"%s\",\n", indent + 2, "", ws->id);
+	VSB_printf(vsp, "%*s{s,f,r,e} = {%p", indent + 2, "", ws->s);
+	if (ws->f > ws->s)
+		VSB_printf(vsp, ",+%ld", (long) (ws->f - ws->s));
+	else
+		VSB_printf(vsp, ",%p", ws->f);
+	if (ws->r > ws->s)
+		VSB_printf(vsp, ",+%ld", (long) (ws->r - ws->s));
+	else
+		VSB_printf(vsp, ",%p", ws->r);
+	if (ws->e > ws->s)
+		VSB_printf(vsp, ",+%ld", (long) (ws->e - ws->s));
+	else
+		VSB_printf(vsp, ",%p", ws->e);
+	VSB_printf(vsp, "},\n");
+	VSB_printf(vsp, "%*s},\n", indent, "" );
+}
+
+/*--------------------------------------------------------------------*/
+
+static void
+pan_vbc(const struct vbc *vbc)
+{
+
+	struct backend *be;
+
+	be = vbc->backend;
+
+	VSB_printf(vsp, "  backend = %p fd = %d {\n", be, vbc->fd);
+	VSB_printf(vsp, "    display_name = \"%s\",\n", be->display_name);
+	VSB_printf(vsp, "  },\n");
+}
+
+/*--------------------------------------------------------------------*/
+
+static void
+pan_storage(const struct storage *st)
+{
+	int i, j;
+
+#define MAX_BYTES (4*16)
+#define show(ch) (((ch) > 31 && (ch) < 127) ? (ch) : '.')
+
+	VSB_printf(vsp, "      %u {\n", st->len);
+	for (i = 0; i < MAX_BYTES && i < st->len; i += 16) {
+		VSB_printf(vsp, "        ");
+		for (j = 0; j < 16; ++j) {
+			if (i + j < st->len)
+				VSB_printf(vsp, "%02x ", st->ptr[i + j]);
+			else
+				VSB_printf(vsp, "   ");
+		}
+		VSB_printf(vsp, "|");
+		for (j = 0; j < 16; ++j)
+			if (i + j < st->len)
+				VSB_printf(vsp, "%c", show(st->ptr[i + j]));
+		VSB_printf(vsp, "|\n");
+	}
+	if (st->len > MAX_BYTES)
+		VSB_printf(vsp, "        [%u more]\n", st->len - MAX_BYTES);
+	VSB_printf(vsp, "      },\n");
+
+#undef show
+#undef MAX_BYTES
+}
+
+/*--------------------------------------------------------------------*/
+
+static void
+pan_http(const char *id, const struct http *h, int indent)
+{
+	int i;
+
+	VSB_printf(vsp, "%*shttp[%s] = {\n", indent, "", id);
+	VSB_printf(vsp, "%*sws = %p[%s]\n", indent + 2, "",
+	    h->ws, h->ws ? h->ws->id : "");
+	for (i = 0; i < h->nhd; ++i) {
+		if (h->hd[i].b == NULL && h->hd[i].e == NULL)
+			continue;
+		VSB_printf(vsp, "%*s\"%.*s\",\n", indent + 4, "",
+		    (int)(h->hd[i].e - h->hd[i].b),
+		    h->hd[i].b);
+	}
+	VSB_printf(vsp, "%*s},\n", indent, "");
+}
+
+
+/*--------------------------------------------------------------------*/
+
+static void
+pan_object(const struct object *o)
+{
+	const struct storage *st;
+
+	VSB_printf(vsp, "  obj = %p {\n", o);
+	VSB_printf(vsp, "    xid = %u,\n", o->xid);
+	pan_ws(o->ws_o, 4);
+	pan_http("obj", o->http, 4);
+	VSB_printf(vsp, "    len = %jd,\n", (intmax_t)o->len);
+	VSB_printf(vsp, "    store = {\n");
+	VTAILQ_FOREACH(st, &o->store, list)
+		pan_storage(st);
+	VSB_printf(vsp, "    },\n");
+	VSB_printf(vsp, "  },\n");
+}
+
+/*--------------------------------------------------------------------*/
+
+static void
+pan_vcl(const struct VCL_conf *vcl)
+{
+	int i;
+
+	VSB_printf(vsp, "    vcl = {\n");
+	VSB_printf(vsp, "      srcname = {\n");
+	for (i = 0; i < vcl->nsrc; ++i)
+		VSB_printf(vsp, "        \"%s\",\n", vcl->srcname[i]);
+	VSB_printf(vsp, "      },\n");
+	VSB_printf(vsp, "    },\n");
+}
+
+
+/*--------------------------------------------------------------------*/
+
+static void
+pan_wrk(const struct worker *wrk)
+{
+
+	VSB_printf(vsp, "  worker = %p {\n", wrk);
+	pan_ws(wrk->ws, 4);
+	if (wrk->bereq->ws != NULL)
+		pan_http("bereq", wrk->bereq, 4);
+	if (wrk->beresp->ws != NULL)
+		pan_http("beresp", wrk->beresp, 4);
+	if (wrk->resp->ws != NULL)
+		pan_http("resp", wrk->resp, 4);
+	VSB_printf(vsp, "    },\n");
+}
+
+/*--------------------------------------------------------------------*/
+
+static void
+pan_sess(const struct sess *sp)
+{
+	const char *stp, *hand;
+
+	VSB_printf(vsp, "sp = %p {\n", sp);
+	VSB_printf(vsp,
+	    "  fd = %d, id = %d, xid = %u,\n",
+	    sp->fd, sp->vsl_id & VSL_IDENTMASK, sp->xid);
+	VSB_printf(vsp, "  client = %s %s,\n",
+	    sp->addr ? sp->addr : "?.?.?.?",
+	    sp->port ? sp->port : "?");
+	switch (sp->step) {
+#define STEP(l, u) case STP_##u: stp = "STP_" #u; break;
+#include "tbl/steps.h"
+#undef STEP
+		default: stp = NULL;
+	}
+	hand = VCL_Return_Name(sp->handling);
+	if (stp != NULL)
+		VSB_printf(vsp, "  step = %s,\n", stp);
+	else
+		VSB_printf(vsp, "  step = 0x%x,\n", sp->step);
+	if (hand != NULL)
+		VSB_printf(vsp, "  handling = %s,\n", hand);
+	else
+		VSB_printf(vsp, "  handling = 0x%x,\n", sp->handling);
+	if (sp->err_code)
+		VSB_printf(vsp,
+		    "  err_code = %d, err_reason = %s,\n", sp->err_code,
+		    sp->err_reason ? sp->err_reason : "(null)");
+
+	VSB_printf(vsp, "  restarts = %d, esi_level = %d\n",
+	    sp->restarts, sp->esi_level);
+
+	VSB_printf(vsp, "  flags = ");
+	if (sp->wrk->do_stream)	VSB_printf(vsp, " do_stream");
+	if (sp->wrk->do_gzip)	VSB_printf(vsp, " do_gzip");
+	if (sp->wrk->do_gunzip)	VSB_printf(vsp, " do_gunzip");
+	if (sp->wrk->do_esi)	VSB_printf(vsp, " do_esi");
+	if (sp->wrk->do_close)	VSB_printf(vsp, " do_close");
+	if (sp->wrk->is_gzip)	VSB_printf(vsp, " is_gzip");
+	if (sp->wrk->is_gunzip)	VSB_printf(vsp, " is_gunzip");
+	VSB_printf(vsp, "\n");
+	VSB_printf(vsp, "  bodystatus = %d\n", sp->wrk->body_status);
+
+	pan_ws(sp->ws, 2);
+	pan_http("req", sp->http, 2);
+
+	if (sp->wrk != NULL)
+		pan_wrk(sp->wrk);
+
+	if (VALID_OBJ(sp->vcl, VCL_CONF_MAGIC))
+		pan_vcl(sp->vcl);
+
+	if (VALID_OBJ(sp->wrk->vbc, BACKEND_MAGIC))
+		pan_vbc(sp->wrk->vbc);
+
+	if (VALID_OBJ(sp->obj, OBJECT_MAGIC))
+		pan_object(sp->obj);
+
+	VSB_printf(vsp, "},\n");
+}
+
+/*--------------------------------------------------------------------*/
+
+static void
+pan_backtrace(void)
+{
+	void *array[10];
+	size_t size;
+	size_t i;
+
+	size = backtrace (array, 10);
+	if (size == 0)
+		return;
+	VSB_printf(vsp, "Backtrace:\n");
+	for (i = 0; i < size; i++) {
+		VSB_printf (vsp, "  ");
+		if (Symbol_Lookup(vsp, array[i]) < 0) {
+			char **strings;
+			strings = backtrace_symbols(&array[i], 1);
+			if (strings != NULL && strings[0] != NULL)
+				VSB_printf(vsp, "%p: %s", array[i], strings[0]);
+			else
+				VSB_printf(vsp, "%p: (?)", array[i]);
+		}
+		VSB_printf (vsp, "\n");
+	}
+}
+
+/*--------------------------------------------------------------------*/
+
+static void
+pan_ic(const char *func, const char *file, int line, const char *cond,
+    int err, int xxx)
+{
+	const char *q;
+	const struct sess *sp;
+
+	AZ(pthread_mutex_lock(&panicstr_mtx)); /* Won't be released,
+						  we're going to die
+						  anyway */
+	switch(xxx) {
+	case 3:
+		VSB_printf(vsp,
+		    "Wrong turn at %s:%d:\n%s\n", file, line, cond);
+		break;
+	case 2:
+		VSB_printf(vsp,
+		    "Panic from VCL:\n  %s\n", cond);
+		break;
+	case 1:
+		VSB_printf(vsp,
+		    "Missing errorhandling code in %s(), %s line %d:\n"
+		    "  Condition(%s) not true.",
+		    func, file, line, cond);
+		break;
+	default:
+	case 0:
+		VSB_printf(vsp,
+		    "Assert error in %s(), %s line %d:\n"
+		    "  Condition(%s) not true.\n",
+		    func, file, line, cond);
+		break;
+	}
+	if (err)
+		VSB_printf(vsp, "errno = %d (%s)\n", err, strerror(err));
+
+	q = THR_GetName();
+	if (q != NULL)
+		VSB_printf(vsp, "thread = (%s)\n", q);
+
+	VSB_printf(vsp, "ident = %s,%s\n",
+	    VSB_data(vident) + 1, WAIT_GetName());
+
+	pan_backtrace();
+
+	if (!(cache_param->diag_bitmap & 0x2000)) {
+		sp = THR_GetSession();
+		if (sp != NULL)
+			pan_sess(sp);
+	}
+	VSB_printf(vsp, "\n");
+	VSB_bcat(vsp, "", 1);	/* NUL termination */
+
+	if (cache_param->diag_bitmap & 0x4000)
+		(void)fputs(VSM_head->panicstr, stderr);
+
+#ifdef HAVE_ABORT2
+	if (cache_param->diag_bitmap & 0x8000) {
+		void *arg[1];
+		char *p;
+
+		for (p = VSM_head->panicstr; *p; p++)
+			if (*p == '\n')
+				*p = ' ';
+		arg[0] = VSM_head->panicstr;
+		abort2(VSM_head->panicstr, 1, arg);
+	}
+#endif
+	if (cache_param->diag_bitmap & 0x1000)
+		exit(4);
+	else
+		abort();
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+PAN_Init(void)
+{
+
+	VAS_Fail = pan_ic;
+	vsp = &vsps;
+	AN(VSB_new(vsp, VSM_head->panicstr, sizeof VSM_head->panicstr,
+	    VSB_FIXEDLEN));
+}
diff --git a/bin/varnishd/cache/cache_pipe.c b/bin/varnishd/cache/cache_pipe.c
new file mode 100644
index 0000000..4180d39
--- /dev/null
+++ b/bin/varnishd/cache/cache_pipe.c
@@ -0,0 +1,133 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * XXX: charge bytes to srcaddr
+ */
+
+#include "config.h"
+
+#include <poll.h>
+#include <stdio.h>
+
+#include "cache.h"
+
+#include "cache_backend.h"
+#include "vtcp.h"
+#include "vtim.h"
+
+static int
+rdf(int fd0, int fd1)
+{
+	int i, j;
+	char buf[BUFSIZ], *p;
+
+	i = read(fd0, buf, sizeof buf);
+	if (i <= 0)
+		return (1);
+	for (p = buf; i > 0; i -= j, p += j) {
+		j = write(fd1, p, i);
+		if (j <= 0)
+			return (1);
+		if (i != j)
+			(void)usleep(100000);		/* XXX hack */
+	}
+	return (0);
+}
+
+void
+PipeSession(struct sess *sp)
+{
+	struct vbc *vc;
+	struct worker *w;
+	struct pollfd fds[2];
+	int i;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->wrk, WORKER_MAGIC);
+	w = sp->wrk;
+
+	sp->wrk->vbc = VDI_GetFd(NULL, sp);
+	if (sp->wrk->vbc == NULL)
+		return;
+	vc = sp->wrk->vbc;
+	(void)VTCP_blocking(vc->fd);
+
+	WRW_Reserve(w, &vc->fd);
+	sp->wrk->acct_tmp.hdrbytes +=
+	    http_Write(w, sp->vsl_id, sp->wrk->bereq, 0);
+
+	if (sp->htc->pipeline.b != NULL)
+		sp->wrk->acct_tmp.bodybytes +=
+		    WRW_Write(w, sp->htc->pipeline.b, Tlen(sp->htc->pipeline));
+
+	i = WRW_FlushRelease(w);
+
+	if (i) {
+		SES_Close(sp, "pipe");
+		VDI_CloseFd(sp->wrk);
+		return;
+	}
+
+	sp->t_resp = VTIM_real();
+
+	memset(fds, 0, sizeof fds);
+
+	// XXX: not yet (void)VTCP_linger(vc->fd, 0);
+	fds[0].fd = vc->fd;
+	fds[0].events = POLLIN | POLLERR;
+
+	// XXX: not yet (void)VTCP_linger(sp->fd, 0);
+	fds[1].fd = sp->fd;
+	fds[1].events = POLLIN | POLLERR;
+
+	while (fds[0].fd > -1 || fds[1].fd > -1) {
+		fds[0].revents = 0;
+		fds[1].revents = 0;
+		i = poll(fds, 2, cache_param->pipe_timeout * 1000);
+		if (i < 1)
+			break;
+		if (fds[0].revents && rdf(vc->fd, sp->fd)) {
+			if (fds[1].fd == -1)
+				break;
+			(void)shutdown(vc->fd, SHUT_RD);
+			(void)shutdown(sp->fd, SHUT_WR);
+			fds[0].events = 0;
+			fds[0].fd = -1;
+		}
+		if (fds[1].revents && rdf(sp->fd, vc->fd)) {
+			if (fds[0].fd == -1)
+				break;
+			(void)shutdown(sp->fd, SHUT_RD);
+			(void)shutdown(vc->fd, SHUT_WR);
+			fds[1].events = 0;
+			fds[1].fd = -1;
+		}
+	}
+	SES_Close(sp, "pipe");
+	VDI_CloseFd(sp->wrk);
+}
diff --git a/bin/varnishd/cache/cache_pool.c b/bin/varnishd/cache/cache_pool.c
new file mode 100644
index 0000000..79a5fcd
--- /dev/null
+++ b/bin/varnishd/cache/cache_pool.c
@@ -0,0 +1,594 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * We maintain a number of worker thread pools, to spread lock contention.
+ *
+ * Pools can be added on the fly, as a means to mitigate lock contention,
+ * but can only be removed again by a restart. (XXX: we could fix that)
+ *
+ * Two threads herd the pools, one eliminates idle threads and aggregates
+ * statistics for all the pools, the other thread creates new threads
+ * on demand, subject to various numerical constraints.
+ *
+ * The algorithm for when to create threads needs to be reactive enough
+ * to handle startup spikes, but sufficiently attenuated to not cause
+ * thread pileups.  This remains subject for improvement.
+ */
+
+#include "config.h"
+
+#include <math.h>
+#include <stdlib.h>
+
+#include "cache.h"
+#include "common/heritage.h"
+
+#include "waiter/cache_waiter.h"
+#include "vtcp.h"
+#include "vtim.h"
+
+/*--------------------------------------------------------------------
+ * MAC OS/X is incredibly moronic when it comes to time and such...
+ */
+
+#ifndef CLOCK_MONOTONIC
+#define CLOCK_MONOTONIC	0
+
+#include <sys/time.h>
+
+static int
+clock_gettime(int foo, struct timespec *ts)
+{
+	struct timeval tv;
+
+	(void)foo;
+	gettimeofday(&tv, NULL);
+	ts->tv_sec = tv.tv_sec;
+	ts->tv_nsec = tv.tv_usec * 1000;
+	return (0);
+}
+
+static int
+pthread_condattr_setclock(pthread_condattr_t *attr, int foo)
+{
+	(void)attr;
+	(void)foo;
+	return (0);
+}
+#endif /* !CLOCK_MONOTONIC */
+
+static void *waiter_priv;
+
+VTAILQ_HEAD(workerhead, worker);
+
+struct poolsock {
+	unsigned			magic;
+#define POOLSOCK_MAGIC			0x1b0a2d38
+	VTAILQ_ENTRY(poolsock)		list;
+	struct listen_sock		*lsock;
+};
+
+/* Number of work requests queued in excess of worker threads available */
+
+struct pool {
+	unsigned		magic;
+#define POOL_MAGIC		0x606658fa
+	VTAILQ_ENTRY(pool)	list;
+
+	pthread_cond_t		herder_cond;
+	struct lock		herder_mtx;
+	pthread_t		herder_thr;
+
+	struct lock		mtx;
+	struct workerhead	idle;
+	VTAILQ_HEAD(, sess)	queue;
+	VTAILQ_HEAD(, poolsock)	socks;
+	unsigned		nthr;
+	unsigned		lqueue;
+	unsigned		last_lqueue;
+	uintmax_t		ndropped;
+	uintmax_t		nqueued;
+	struct sesspool		*sesspool;
+};
+
+static struct lock		pool_mtx;
+static pthread_t		thr_pool_herder;
+
+/*--------------------------------------------------------------------
+ * Nobody is accepting on this socket, so we do.
+ *
+ * As long as we can stick the accepted connection to another thread
+ * we do so, otherwise we return and handle it ourselves.
+ *
+ * Notice calling convention:  Called locked and returns locked, but
+ * works lock in the meantime.
+ *
+ * We store data about the accept in reserved workspace, it is only used
+ * for a brief moment and it takes up around 144 bytes.
+ */
+
+static int
+pool_accept(struct pool *pp, struct worker *w, const struct poolsock *ps)
+{
+	struct worker *w2;
+	struct wrk_accept *wa, *wa2;
+
+	CHECK_OBJ_NOTNULL(pp, POOL_MAGIC);
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	CHECK_OBJ_NOTNULL(ps, POOLSOCK_MAGIC);
+
+	CHECK_OBJ_NOTNULL(ps->lsock, LISTEN_SOCK_MAGIC);
+	Lck_AssertHeld(&pp->mtx);
+	Lck_Unlock(&pp->mtx);
+	assert(sizeof *wa == WS_Reserve(w->ws, sizeof *wa));
+	wa = (void*)w->ws->f;
+	while (1) {
+		memset(wa, 0, sizeof *wa);
+		wa->magic = WRK_ACCEPT_MAGIC;
+
+		if (ps->lsock->sock < 0) {
+			/* Socket Shutdown */
+			Lck_Lock(&pp->mtx);
+			return (-1);
+		}
+		if (VCA_Accept(ps->lsock, wa) < 0) {
+			w->stats.sess_fail++;
+			/* We're going to pace in vca anyway... */
+			(void)WRK_TrySumStat(w);
+			continue;
+		}
+
+		Lck_Lock(&pp->mtx);
+		if (VTAILQ_EMPTY(&pp->idle))
+			return (0);
+		w2 = VTAILQ_FIRST(&pp->idle);
+		VTAILQ_REMOVE(&pp->idle, w2, list);
+		Lck_Unlock(&pp->mtx);
+		assert(sizeof *wa2 == WS_Reserve(w2->ws, sizeof *wa2));
+		wa2 = (void*)w2->ws->f;
+		memcpy(wa2, wa, sizeof *wa);
+		AZ(pthread_cond_signal(&w2->cond));
+	}
+}
+
+/*--------------------------------------------------------------------
+ * This is the work function for worker threads in the pool.
+ */
+
+void
+Pool_Work_Thread(void *priv, struct worker *w)
+{
+	struct pool *pp;
+	int stats_clean, i;
+	struct poolsock *ps;
+
+	CAST_OBJ_NOTNULL(pp, priv, POOL_MAGIC);
+	w->pool = pp;
+	Lck_Lock(&pp->mtx);
+	stats_clean = 1;
+	while (1) {
+
+		Lck_AssertHeld(&pp->mtx);
+
+		CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+		CHECK_OBJ_NOTNULL(w->bereq, HTTP_MAGIC);
+		CHECK_OBJ_NOTNULL(w->beresp, HTTP_MAGIC);
+		CHECK_OBJ_NOTNULL(w->resp, HTTP_MAGIC);
+
+		WS_Reset(w->ws, NULL);
+
+		w->sp = VTAILQ_FIRST(&pp->queue);
+		if (w->sp != NULL) {
+			/* Process queued requests, if any */
+			assert(pp->lqueue > 0);
+			VTAILQ_REMOVE(&pp->queue, w->sp, poollist);
+			pp->lqueue--;
+		} else if (!VTAILQ_EMPTY(&pp->socks)) {
+			/* Accept on a socket */
+			ps = VTAILQ_FIRST(&pp->socks);
+			VTAILQ_REMOVE(&pp->socks, ps, list);
+			i = pool_accept(pp, w, ps);
+			Lck_AssertHeld(&pp->mtx);
+			if (i < 0) {
+				/* Socket Shutdown */
+				FREE_OBJ(ps);
+				WS_Release(w->ws, 0);
+				continue;
+			}
+			VTAILQ_INSERT_TAIL(&pp->socks, ps, list);
+		} else if (VTAILQ_EMPTY(&pp->socks)) {
+			/* Nothing to do: To sleep, perchance to dream ... */
+			if (isnan(w->lastused))
+				w->lastused = VTIM_real();
+			VTAILQ_INSERT_HEAD(&pp->idle, w, list);
+			if (!stats_clean)
+				WRK_SumStat(w);
+			(void)Lck_CondWait(&w->cond, &pp->mtx, NULL);
+		}
+
+		/*
+		 * If we got neither session or accepted a socket, we were
+		 * woken up to die to cull the herd.
+		 */
+		if (w->sp == NULL && w->ws->r == NULL)
+			break;
+
+		Lck_Unlock(&pp->mtx);
+
+		if (w->sp == NULL) {
+			/* Turn accepted socket into a session */
+			assert(w->ws->r != NULL);
+			w->sp = SES_New(w, pp->sesspool);
+			if (w->sp == NULL)
+				VCA_FailSess(w);
+			else
+				VCA_SetupSess(w);
+			WS_Release(w->ws, 0);
+		}
+		assert(w->ws->r == NULL);
+
+		if (w->sp != NULL) {
+			CHECK_OBJ_NOTNULL(w->sp, SESS_MAGIC);
+
+			stats_clean = 0;
+			w->lastused = NAN;
+			w->storage_hint = NULL;
+
+			AZ(w->sp->wrk);
+			THR_SetSession(w->sp);
+			w->sp->wrk = w;
+			CNT_Session(w->sp);
+			THR_SetSession(NULL);
+			w->sp = NULL;
+
+			WS_Assert(w->ws);
+			AZ(w->bereq->ws);
+			AZ(w->beresp->ws);
+			AZ(w->resp->ws);
+			AZ(w->wrw.wfd);
+			AZ(w->storage_hint);
+			assert(w->wlp == w->wlb);
+			if (cache_param->diag_bitmap & 0x00040000) {
+				if (w->vcl != NULL)
+					VCL_Rel(&w->vcl);
+			}
+		}
+		stats_clean = WRK_TrySumStat(w);
+		Lck_Lock(&pp->mtx);
+	}
+	Lck_Unlock(&pp->mtx);
+	w->pool = NULL;
+}
+
+/*--------------------------------------------------------------------
+ * Queue a workrequest if possible.
+ *
+ * Return zero if the request was queued, negative if it wasn't.
+ */
+
+static int
+pool_queue(struct pool *pp, struct sess *sp)
+{
+	struct worker *w;
+
+	Lck_Lock(&pp->mtx);
+
+	/* If there are idle threads, we tickle the first one into action */
+	w = VTAILQ_FIRST(&pp->idle);
+	if (w != NULL) {
+		VTAILQ_REMOVE(&pp->idle, w, list);
+		Lck_Unlock(&pp->mtx);
+		w->sp = sp;
+		AZ(pthread_cond_signal(&w->cond));
+		return (0);
+	}
+
+	/* If we have too much in the queue already, refuse. */
+	if (pp->lqueue > (cache_param->queue_max * pp->nthr) / 100) {
+		pp->ndropped++;
+		Lck_Unlock(&pp->mtx);
+		return (-1);
+	}
+
+	VTAILQ_INSERT_TAIL(&pp->queue, sp, poollist);
+	pp->nqueued++;
+	pp->lqueue++;
+	Lck_Unlock(&pp->mtx);
+	AZ(pthread_cond_signal(&pp->herder_cond));
+	return (0);
+}
+
+/*--------------------------------------------------------------------*/
+
+int
+Pool_Schedule(struct pool *pp, struct sess *sp)
+{
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	AZ(sp->wrk);
+	if (pool_queue(pp, sp) == 0)
+		return(0);
+
+	VSC_C_main->client_drop_late++;
+
+	/*
+	 * Couldn't queue it -- kill it.
+	 *
+	 * XXX: a notice might be polite, but would potentially
+	 * XXX: sleep whichever thread got us here
+	 */
+	sp->t_end = VTIM_real();
+	if (sp->vcl != NULL) {
+		/*
+		 * A session parked on a busy object can come here
+		 * after it wakes up.  Loose the VCL reference.
+		 */
+		VCL_Rel(&sp->vcl);
+	}
+	return (1);
+}
+
+/*--------------------------------------------------------------------
+ * Wait for another request
+ */
+
+void
+Pool_Wait(struct sess *sp)
+{
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	AZ(sp->obj);
+	AZ(sp->vcl);
+	assert(sp->fd >= 0);
+	/*
+	 * Set nonblocking in the worker-thread, before passing to the
+	 * acceptor thread, to reduce syscall density of the latter.
+	 */
+	if (VTCP_nonblocking(sp->fd))
+		SES_Close(sp, "remote closed");
+	waiter->pass(waiter_priv, sp);
+}
+
+/*--------------------------------------------------------------------
+ * Create another thread, if necessary & possible
+ */
+
+static void
+pool_breed(struct pool *qp, const pthread_attr_t *tp_attr)
+{
+	pthread_t tp;
+
+	/*
+	 * If we need more threads, and have space, create
+	 * one more thread.
+	 */
+	if (qp->nthr < cache_param->wthread_min ||	/* Not enough threads yet */
+	    (qp->lqueue > cache_param->wthread_add_threshold && /* more needed */
+	    qp->lqueue > qp->last_lqueue)) {	/* not getting better since last */
+		if (qp->nthr > cache_param->wthread_max) {
+			Lck_Lock(&pool_mtx);
+			VSC_C_main->threads_limited++;
+			Lck_Unlock(&pool_mtx);
+		} else if (pthread_create(&tp, tp_attr, WRK_thread, qp)) {
+			VSL(SLT_Debug, 0, "Create worker thread failed %d %s",
+			    errno, strerror(errno));
+			Lck_Lock(&pool_mtx);
+			VSC_C_main->threads_limited++;
+			Lck_Unlock(&pool_mtx);
+			VTIM_sleep(cache_param->wthread_fail_delay * 1e-3);
+		} else {
+			AZ(pthread_detach(tp));
+			VTIM_sleep(cache_param->wthread_add_delay * 1e-3);
+			qp->nthr++;
+			Lck_Lock(&pool_mtx);
+			VSC_C_main->threads++;
+			VSC_C_main->threads_created++;
+			Lck_Unlock(&pool_mtx);
+		}
+	}
+	qp->last_lqueue = qp->lqueue;
+}
+
+/*--------------------------------------------------------------------
+ * Herd a single pool
+ *
+ * This thread wakes up whenever a pool queues.
+ *
+ * The trick here is to not be too aggressive about creating threads.
+ * We do this by only examining one pool at a time, and by sleeping
+ * a short while whenever we create a thread and a little while longer
+ * whenever we fail to, hopefully missing a lot of cond_signals in
+ * the meantime.
+ *
+ * XXX: probably need a lot more work.
+ *
+ */
+
+static void*
+pool_herder(void *priv)
+{
+	struct pool *pp;
+	pthread_attr_t tp_attr;
+	struct timespec ts;
+	double t_idle;
+	struct worker *w;
+	int i;
+
+	CAST_OBJ_NOTNULL(pp, priv, POOL_MAGIC);
+	AZ(pthread_attr_init(&tp_attr));
+
+	while (1) {
+		/* Set the stacksize for worker threads we create */
+		if (cache_param->wthread_stacksize != UINT_MAX)
+			AZ(pthread_attr_setstacksize(&tp_attr,
+			    cache_param->wthread_stacksize));
+		else {
+			AZ(pthread_attr_destroy(&tp_attr));
+			AZ(pthread_attr_init(&tp_attr));
+		}
+
+		pool_breed(pp, &tp_attr);
+
+		if (pp->nthr < cache_param->wthread_min)
+			continue;
+
+		AZ(clock_gettime(CLOCK_MONOTONIC, &ts));
+		ts.tv_sec += cache_param->wthread_purge_delay / 1000;
+		ts.tv_nsec +=
+		    (cache_param->wthread_purge_delay % 1000) * 1000000;
+		if (ts.tv_nsec >= 1000000000) {
+			ts.tv_sec++;
+			ts.tv_nsec -= 1000000000;
+		}
+
+		Lck_Lock(&pp->herder_mtx);
+		i = Lck_CondWait(&pp->herder_cond, &pp->herder_mtx, &ts);
+		Lck_Unlock(&pp->herder_mtx);
+		if (!i)
+			continue;
+
+		if (pp->nthr <= cache_param->wthread_min)
+			continue;
+
+		t_idle = VTIM_real() - cache_param->wthread_timeout;
+
+		Lck_Lock(&pp->mtx);
+		VSC_C_main->sess_queued += pp->nqueued;
+		VSC_C_main->sess_dropped += pp->ndropped;
+		pp->nqueued = pp->ndropped = 0;
+		w = VTAILQ_LAST(&pp->idle, workerhead);
+		if (w != NULL &&
+		    (w->lastused < t_idle || pp->nthr > cache_param->wthread_max)) {
+			VTAILQ_REMOVE(&pp->idle, w, list);
+		} else
+			w = NULL;
+		Lck_Unlock(&pp->mtx);
+
+		/* And give it a kiss on the cheek... */
+		if (w != NULL) {
+			pp->nthr--;
+			Lck_Lock(&pool_mtx);
+			VSC_C_main->threads--;
+			VSC_C_main->threads_destroyed++;
+			Lck_Unlock(&pool_mtx);
+			AZ(w->sp);
+			AZ(pthread_cond_signal(&w->cond));
+		}
+	}
+	NEEDLESS_RETURN(NULL);
+}
+
+/*--------------------------------------------------------------------
+ * Add a thread pool
+ */
+
+static struct pool *
+pool_mkpool(void)
+{
+	struct pool *pp;
+	struct listen_sock *ls;
+	struct poolsock *ps;
+	pthread_condattr_t cv_attr;
+
+	ALLOC_OBJ(pp, POOL_MAGIC);
+	XXXAN(pp);
+	Lck_New(&pp->mtx, lck_wq);
+
+	VTAILQ_INIT(&pp->queue);
+	VTAILQ_INIT(&pp->idle);
+	VTAILQ_INIT(&pp->socks);
+	pp->sesspool = SES_NewPool(pp);
+	AN(pp->sesspool);
+
+	VTAILQ_FOREACH(ls, &heritage.socks, list) {
+		if (ls->sock < 0)
+			continue;
+		ALLOC_OBJ(ps, POOLSOCK_MAGIC);
+		XXXAN(ps);
+		ps->lsock = ls;
+		VTAILQ_INSERT_TAIL(&pp->socks, ps, list);
+	}
+
+	AZ(pthread_condattr_init(&cv_attr));
+	AZ(pthread_condattr_setclock(&cv_attr, CLOCK_MONOTONIC));
+	AZ(pthread_cond_init(&pp->herder_cond, &cv_attr));
+	AZ(pthread_condattr_destroy(&cv_attr));
+	Lck_New(&pp->herder_mtx, lck_herder);
+	AZ(pthread_create(&pp->herder_thr, NULL, pool_herder, pp));
+
+	return (pp);
+}
+
+/*--------------------------------------------------------------------
+ * This thread adjusts the number of pools to match the parameter.
+ *
+ */
+
+static void *
+pool_poolherder(void *priv)
+{
+	unsigned nwq;
+	VTAILQ_HEAD(,pool)	pools = VTAILQ_HEAD_INITIALIZER(pools);
+	struct pool *pp;
+	uint64_t u;
+
+	THR_SetName("pool_herder");
+	(void)priv;
+
+	nwq = 0;
+	while (1) {
+		if (nwq < cache_param->wthread_pools) {
+			pp = pool_mkpool();
+			if (pp != NULL) {
+				VTAILQ_INSERT_TAIL(&pools, pp, list);
+				VSC_C_main->pools++;
+				nwq++;
+				continue;
+			}
+		}
+		/* XXX: remove pools */
+		if (0)
+			SES_DeletePool(NULL, NULL);
+		(void)sleep(1);
+		u = 0;
+		VTAILQ_FOREACH(pp, &pools, list)
+			u += pp->lqueue;
+		VSC_C_main->thread_queue_len = u;
+	}
+	NEEDLESS_RETURN(NULL);
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+Pool_Init(void)
+{
+
+	waiter_priv = waiter->init();
+	Lck_New(&pool_mtx, lck_wq);
+	AZ(pthread_create(&thr_pool_herder, NULL, pool_poolherder, NULL));
+}
diff --git a/bin/varnishd/cache/cache_response.c b/bin/varnishd/cache/cache_response.c
new file mode 100644
index 0000000..487a514
--- /dev/null
+++ b/bin/varnishd/cache/cache_response.c
@@ -0,0 +1,427 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include "config.h"
+
+#include "cache.h"
+
+#include "vct.h"
+#include "vtim.h"
+
+/*--------------------------------------------------------------------*/
+
+static void
+res_dorange(const struct sess *sp, const char *r, ssize_t *plow, ssize_t *phigh)
+{
+	ssize_t low, high, has_low;
+
+	assert(sp->obj->response == 200);
+	if (strncmp(r, "bytes=", 6))
+		return;
+	r += 6;
+
+	/* The low end of range */
+	has_low = low = 0;
+	if (!vct_isdigit(*r) && *r != '-')
+		return;
+	while (vct_isdigit(*r)) {
+		has_low = 1;
+		low *= 10;
+		low += *r - '0';
+		r++;
+	}
+
+	if (low >= sp->obj->len)
+		return;
+
+	if (*r != '-')
+		return;
+	r++;
+
+	/* The high end of range */
+	if (vct_isdigit(*r)) {
+		high = 0;
+		while (vct_isdigit(*r)) {
+			high *= 10;
+			high += *r - '0';
+			r++;
+		}
+		if (!has_low) {
+			low = sp->obj->len - high;
+			high = sp->obj->len - 1;
+		}
+	} else
+		high = sp->obj->len - 1;
+	if (*r != '\0')
+		return;
+
+	if (high >= sp->obj->len)
+		high = sp->obj->len - 1;
+
+	if (low > high)
+		return;
+
+	http_PrintfHeader(sp->wrk, sp->vsl_id, sp->wrk->resp,
+	    "Content-Range: bytes %jd-%jd/%jd",
+	    (intmax_t)low, (intmax_t)high, (intmax_t)sp->obj->len);
+	http_Unset(sp->wrk->resp, H_Content_Length);
+	assert(sp->wrk->res_mode & RES_LEN);
+	http_PrintfHeader(sp->wrk, sp->vsl_id, sp->wrk->resp,
+	    "Content-Length: %jd", (intmax_t)(1 + high - low));
+	http_SetResp(sp->wrk->resp, "HTTP/1.1", 206, "Partial Content");
+
+	*plow = low;
+	*phigh = high;
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+RES_BuildHttp(const struct sess *sp)
+{
+	char time_str[30];
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+
+
+	http_ClrHeader(sp->wrk->resp);
+	sp->wrk->resp->logtag = HTTP_Tx;
+	http_CopyResp(sp->wrk->resp, sp->obj->http);
+	http_FilterFields(sp->wrk, sp->vsl_id, sp->wrk->resp, sp->obj->http,
+	    HTTPH_A_DELIVER);
+
+	if (!(sp->wrk->res_mode & RES_LEN)) {
+		http_Unset(sp->wrk->resp, H_Content_Length);
+	} else if (cache_param->http_range_support) {
+		/* We only accept ranges if we know the length */
+		http_SetHeader(sp->wrk, sp->vsl_id, sp->wrk->resp,
+		    "Accept-Ranges: bytes");
+	}
+
+	if (sp->wrk->res_mode & RES_CHUNKED)
+		http_SetHeader(sp->wrk, sp->vsl_id, sp->wrk->resp,
+		    "Transfer-Encoding: chunked");
+
+	VTIM_format(VTIM_real(), time_str);
+	http_PrintfHeader(sp->wrk, sp->vsl_id, sp->wrk->resp, "Date: %s", time_str);
+
+	if (sp->xid != sp->obj->xid)
+		http_PrintfHeader(sp->wrk, sp->vsl_id, sp->wrk->resp,
+		    "X-Varnish: %u %u", sp->xid, sp->obj->xid);
+	else
+		http_PrintfHeader(sp->wrk, sp->vsl_id, sp->wrk->resp,
+		    "X-Varnish: %u", sp->xid);
+	http_PrintfHeader(sp->wrk, sp->vsl_id, sp->wrk->resp, "Age: %.0f",
+	    sp->obj->exp.age + sp->t_resp - sp->obj->exp.entered);
+	http_SetHeader(sp->wrk, sp->vsl_id, sp->wrk->resp, "Via: 1.1 varnish");
+	http_PrintfHeader(sp->wrk, sp->vsl_id, sp->wrk->resp, "Connection: %s",
+	    sp->doclose ? "close" : "keep-alive");
+}
+
+/*--------------------------------------------------------------------
+ * We have a gzip'ed object and need to ungzip it for a client which
+ * does not understand gzip.
+ * XXX: handle invalid gzip data better (how ?)
+ */
+
+static void
+res_WriteGunzipObj(const struct sess *sp)
+{
+	struct storage *st;
+	unsigned u = 0;
+	struct vgz *vg;
+	char obuf[cache_param->gzip_stack_buffer];
+	ssize_t obufl = 0;
+	int i;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+
+	vg = VGZ_NewUngzip(sp->wrk, "U D -");
+
+	VGZ_Obuf(vg, obuf, sizeof obuf);
+	VTAILQ_FOREACH(st, &sp->obj->store, list) {
+		CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+		CHECK_OBJ_NOTNULL(st, STORAGE_MAGIC);
+		u += st->len;
+
+		VSC_C_main->n_objwrite++;
+
+		i = VGZ_WrwGunzip(sp->wrk, vg,
+		    st->ptr, st->len,
+		    obuf, sizeof obuf, &obufl);
+		/* XXX: error check */
+		(void)i;
+	}
+	if (obufl) {
+		(void)WRW_Write(sp->wrk, obuf, obufl);
+		(void)WRW_Flush(sp->wrk);
+	}
+	(void)VGZ_Destroy(&vg, sp->vsl_id);
+	assert(u == sp->obj->len);
+}
+
+/*--------------------------------------------------------------------*/
+
+static void
+res_WriteDirObj(const struct sess *sp, ssize_t low, ssize_t high)
+{
+	ssize_t u = 0;
+	size_t ptr, off, len;
+	struct storage *st;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+
+	ptr = 0;
+	VTAILQ_FOREACH(st, &sp->obj->store, list) {
+		CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+		CHECK_OBJ_NOTNULL(st, STORAGE_MAGIC);
+		u += st->len;
+		len = st->len;
+		off = 0;
+		if (ptr + len <= low) {
+			/* This segment is too early */
+			ptr += len;
+			continue;
+		}
+		if (ptr < low) {
+			/* Chop front of segment off */
+			off += (low - ptr);
+			len -= (low - ptr);
+			ptr += (low - ptr);
+		}
+		if (ptr + len > high)
+			/* Chop tail of segment off */
+			len = 1 + high - ptr;
+
+		ptr += len;
+
+		sp->wrk->acct_tmp.bodybytes += len;
+#ifdef SENDFILE_WORKS
+		/*
+		 * XXX: the overhead of setting up sendfile is not
+		 * XXX: epsilon and maybe not even delta, so avoid
+		 * XXX: engaging sendfile for small objects.
+		 * XXX: Should use getpagesize() ?
+		 */
+		if (st->fd >= 0 &&
+		    st->len >= cache_param->sendfile_threshold) {
+			VSC_C_main->n_objsendfile++;
+			WRW_Sendfile(sp->wrk, st->fd, st->where + off, len);
+			continue;
+		}
+#endif /* SENDFILE_WORKS */
+		VSC_C_main->n_objwrite++;
+		(void)WRW_Write(sp->wrk, st->ptr + off, len);
+	}
+	assert(u == sp->obj->len);
+}
+
+/*--------------------------------------------------------------------
+ * Deliver an object.
+ * Attempt optimizations like 304 and 206 here.
+ */
+
+void
+RES_WriteObj(struct sess *sp)
+{
+	char *r;
+	ssize_t low, high;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+
+	WRW_Reserve(sp->wrk, &sp->fd);
+
+	if (sp->obj->response == 200 &&
+	    sp->http->conds &&
+	    RFC2616_Do_Cond(sp)) {
+		sp->wantbody = 0;
+		http_SetResp(sp->wrk->resp, "HTTP/1.1", 304, "Not Modified");
+		http_Unset(sp->wrk->resp, H_Content_Length);
+		http_Unset(sp->wrk->resp, H_Transfer_Encoding);
+	}
+
+	/*
+	 * If nothing special planned, we can attempt Range support
+	 */
+	low = 0;
+	high = sp->obj->len - 1;
+	if (
+	    sp->wantbody &&
+	    (sp->wrk->res_mode & RES_LEN) &&
+	    !(sp->wrk->res_mode & (RES_ESI|RES_ESI_CHILD|RES_GUNZIP)) &&
+	    cache_param->http_range_support &&
+	    sp->obj->response == 200 &&
+	    http_GetHdr(sp->http, H_Range, &r))
+		res_dorange(sp, r, &low, &high);
+
+	/*
+	 * Always remove C-E if client don't grok it
+	 */
+	if (sp->wrk->res_mode & RES_GUNZIP)
+		http_Unset(sp->wrk->resp, H_Content_Encoding);
+
+	/*
+	 * Send HTTP protocol header, unless interior ESI object
+	 */
+	if (!(sp->wrk->res_mode & RES_ESI_CHILD))
+		sp->wrk->acct_tmp.hdrbytes +=
+		    http_Write(sp->wrk, sp->vsl_id, sp->wrk->resp, 1);
+
+	if (!sp->wantbody)
+		sp->wrk->res_mode &= ~RES_CHUNKED;
+
+	if (sp->wrk->res_mode & RES_CHUNKED)
+		WRW_Chunked(sp->wrk);
+
+	if (!sp->wantbody) {
+		/* This was a HEAD or conditional request */
+	} else if (sp->obj->len == 0) {
+		/* Nothing to do here */
+	} else if (sp->wrk->res_mode & RES_ESI) {
+		ESI_Deliver(sp);
+	} else if (sp->wrk->res_mode & RES_ESI_CHILD && sp->wrk->gzip_resp) {
+		ESI_DeliverChild(sp);
+	} else if (sp->wrk->res_mode & RES_ESI_CHILD &&
+	    !sp->wrk->gzip_resp && sp->obj->gziped) {
+		res_WriteGunzipObj(sp);
+	} else if (sp->wrk->res_mode & RES_GUNZIP) {
+		res_WriteGunzipObj(sp);
+	} else {
+		res_WriteDirObj(sp, low, high);
+	}
+
+	if (sp->wrk->res_mode & RES_CHUNKED &&
+	    !(sp->wrk->res_mode & RES_ESI_CHILD))
+		WRW_EndChunk(sp->wrk);
+
+	if (WRW_FlushRelease(sp->wrk) && sp->fd >= 0)
+		SES_Close(sp, "remote closed");
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+RES_StreamStart(struct sess *sp)
+{
+	struct stream_ctx *sctx;
+
+	sctx = sp->wrk->sctx;
+	CHECK_OBJ_NOTNULL(sctx, STREAM_CTX_MAGIC);
+
+	AZ(sp->wrk->res_mode & RES_ESI_CHILD);
+	AN(sp->wantbody);
+
+	WRW_Reserve(sp->wrk, &sp->fd);
+	/*
+	 * Always remove C-E if client don't grok it
+	 */
+	if (sp->wrk->res_mode & RES_GUNZIP)
+		http_Unset(sp->wrk->resp, H_Content_Encoding);
+
+	if (!(sp->wrk->res_mode & RES_CHUNKED) &&
+	    sp->wrk->h_content_length != NULL)
+		http_PrintfHeader(sp->wrk, sp->vsl_id, sp->wrk->resp,
+		    "Content-Length: %s", sp->wrk->h_content_length);
+
+	sp->wrk->acct_tmp.hdrbytes +=
+	    http_Write(sp->wrk, sp->vsl_id, sp->wrk->resp, 1);
+
+	if (sp->wrk->res_mode & RES_CHUNKED)
+		WRW_Chunked(sp->wrk);
+}
+
+void
+RES_StreamPoll(struct worker *w)
+{
+	struct stream_ctx *sctx;
+	struct storage *st;
+	ssize_t l, l2;
+	void *ptr;
+
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	CHECK_OBJ_NOTNULL(w->fetch_obj, OBJECT_MAGIC);
+	sctx = w->sctx;
+	CHECK_OBJ_NOTNULL(sctx, STREAM_CTX_MAGIC);
+	if (w->fetch_obj->len == sctx->stream_next)
+		return;
+	assert(w->fetch_obj->len > sctx->stream_next);
+	l = sctx->stream_front;
+	VTAILQ_FOREACH(st, &w->fetch_obj->store, list) {
+		if (st->len + l <= sctx->stream_next) {
+			l += st->len;
+			continue;
+		}
+		l2 = st->len + l - sctx->stream_next;
+		ptr = st->ptr + (sctx->stream_next - l);
+		if (w->res_mode & RES_GUNZIP) {
+			(void)VGZ_WrwGunzip(w, sctx->vgz, ptr, l2,
+			    sctx->obuf, sctx->obuf_len, &sctx->obuf_ptr);
+		} else {
+			(void)WRW_Write(w, ptr, l2);
+		}
+		l += st->len;
+		sctx->stream_next += l2;
+	}
+	if (!(w->res_mode & RES_GUNZIP))
+		(void)WRW_Flush(w);
+
+	if (w->fetch_obj->objcore == NULL ||
+	    (w->fetch_obj->objcore->flags & OC_F_PASS)) {
+		/*
+		 * This is a pass object, release storage as soon as we
+		 * have delivered it.
+		 */
+		while (1) {
+			st = VTAILQ_FIRST(&w->fetch_obj->store);
+			if (st == NULL ||
+			    sctx->stream_front + st->len > sctx->stream_next)
+				break;
+			VTAILQ_REMOVE(&w->fetch_obj->store, st, list);
+			sctx->stream_front += st->len;
+			STV_free(st);
+		}
+	}
+}
+
+void
+RES_StreamEnd(struct sess *sp)
+{
+	struct stream_ctx *sctx;
+
+	sctx = sp->wrk->sctx;
+	CHECK_OBJ_NOTNULL(sctx, STREAM_CTX_MAGIC);
+
+	if (sp->wrk->res_mode & RES_GUNZIP && sctx->obuf_ptr > 0)
+		(void)WRW_Write(sp->wrk, sctx->obuf, sctx->obuf_ptr);
+	if (sp->wrk->res_mode & RES_CHUNKED &&
+	    !(sp->wrk->res_mode & RES_ESI_CHILD))
+		WRW_EndChunk(sp->wrk);
+	if (WRW_FlushRelease(sp->wrk))
+		SES_Close(sp, "remote closed");
+}
diff --git a/bin/varnishd/cache/cache_rfc2616.c b/bin/varnishd/cache/cache_rfc2616.c
new file mode 100644
index 0000000..4041f45
--- /dev/null
+++ b/bin/varnishd/cache/cache_rfc2616.c
@@ -0,0 +1,336 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include "config.h"
+
+#include <math.h>
+#include <stdlib.h>
+
+#include "cache.h"
+
+#include "vtim.h"
+
+/*--------------------------------------------------------------------
+ * TTL and Age calculation in Varnish
+ *
+ * RFC2616 has a lot to say about how caches should calculate the TTL
+ * and expiry times of objects, but it sort of misses the case that
+ * applies to Varnish:  the server-side cache.
+ *
+ * A normal cache, shared or single-client, has no symbiotic relationship
+ * with the server, and therefore must take a very defensive attitude
+ * if the Data/Expiry/Age/max-age data does not make sense.  Overall
+ * the policy described in section 13 of RFC 2616 results in no caching
+ * happening on the first little sign of trouble.
+ *
+ * Varnish on the other hand tries to offload as many transactions from
+ * the backend as possible, and therefore just passing through everything
+ * if there is a clock-skew between backend and Varnish is not a workable
+ * choice.
+ *
+ * Varnish implements a policy which is RFC2616 compliant when there
+ * is no clockskew, and falls as gracefully as possible otherwise.
+ * Our "clockless cache" model is syntehsized from the bits of RFC2616
+ * that talks about how a cache should react to a clockless origin server,
+ * and more or less uses the inverse logic for the opposite relationship.
+ *
+ */
+
+void
+RFC2616_Ttl(const struct sess *sp)
+{
+	unsigned max_age, age;
+	double h_date, h_expires;
+	char *p;
+	const struct http *hp;
+
+	hp = sp->wrk->beresp;
+
+	assert(sp->wrk->exp.entered != 0.0 && !isnan(sp->wrk->exp.entered));
+	/* If all else fails, cache using default ttl */
+	sp->wrk->exp.ttl = cache_param->default_ttl;
+
+	max_age = age = 0;
+	h_expires = 0;
+	h_date = 0;
+
+	/*
+	 * Initial cacheability determination per [RFC2616, 13.4]
+	 * We do not support ranges yet, so 206 is out.
+	 */
+
+	if (http_GetHdr(hp, H_Age, &p)) {
+		age = strtoul(p, NULL, 0);
+		sp->wrk->exp.age = age;
+	}
+	if (http_GetHdr(hp, H_Expires, &p))
+		h_expires = VTIM_parse(p);
+
+	if (http_GetHdr(hp, H_Date, &p))
+		h_date = VTIM_parse(p);
+
+	switch (sp->err_code) {
+	default:
+		sp->wrk->exp.ttl = -1.;
+		break;
+	case 200: /* OK */
+	case 203: /* Non-Authoritative Information */
+	case 300: /* Multiple Choices */
+	case 301: /* Moved Permanently */
+	case 302: /* Moved Temporarily */
+	case 307: /* Temporary Redirect */
+	case 410: /* Gone */
+	case 404: /* Not Found */
+		/*
+		 * First find any relative specification from the backend
+		 * These take precedence according to RFC2616, 13.2.4
+		 */
+
+		if ((http_GetHdrField(hp, H_Cache_Control, "s-maxage", &p) ||
+		    http_GetHdrField(hp, H_Cache_Control, "max-age", &p)) &&
+		    p != NULL) {
+
+			if (*p == '-')
+				max_age = 0;
+			else
+				max_age = strtoul(p, NULL, 0);
+
+			if (age > max_age)
+				sp->wrk->exp.ttl = 0;
+			else
+				sp->wrk->exp.ttl = max_age - age;
+			break;
+		}
+
+		/* No expire header, fall back to default */
+		if (h_expires == 0)
+			break;
+
+
+		/* If backend told us it is expired already, don't cache. */
+		if (h_expires < h_date) {
+			sp->wrk->exp.ttl = 0;
+			break;
+		}
+
+		if (h_date == 0 ||
+		    fabs(h_date - sp->wrk->exp.entered) < cache_param->clock_skew) {
+			/*
+			 * If we have no Date: header or if it is
+			 * sufficiently close to our clock we will
+			 * trust Expires: relative to our own clock.
+			 */
+			if (h_expires < sp->wrk->exp.entered)
+				sp->wrk->exp.ttl = 0;
+			else
+				sp->wrk->exp.ttl = h_expires -
+				    sp->wrk->exp.entered;
+			break;
+		} else {
+			/*
+			 * But even if the clocks are out of whack we can still
+			 * derive a relative time from the two headers.
+			 * (the negative ttl case is caught above)
+			 */
+			sp->wrk->exp.ttl = (int)(h_expires - h_date);
+		}
+
+	}
+
+	/* calculated TTL, Our time, Date, Expires, max-age, age */
+	WSP(sp, SLT_TTL,
+	    "%u RFC %.0f %.0f %.0f %.0f %.0f %.0f %.0f %u",
+	    sp->xid, sp->wrk->exp.ttl, -1., -1., sp->wrk->exp.entered,
+	    sp->wrk->exp.age, h_date, h_expires, max_age);
+}
+
+/*--------------------------------------------------------------------
+ * Body existence, fetch method and close policy.
+ */
+
+enum body_status
+RFC2616_Body(const struct sess *sp)
+{
+	struct http *hp;
+	char *b;
+
+	hp = sp->wrk->beresp;
+
+	if (hp->protover < 11 && !http_HdrIs(hp, H_Connection, "keep-alive"))
+		sp->wrk->do_close = 1;
+	else if (http_HdrIs(hp, H_Connection, "close"))
+		sp->wrk->do_close = 1;
+	else
+		sp->wrk->do_close = 0;
+
+	if (!strcasecmp(http_GetReq(sp->wrk->bereq), "head")) {
+		/*
+		 * A HEAD request can never have a body in the reply,
+		 * no matter what the headers might say.
+		 * [RFC2516 4.3 p33]
+		 */
+		sp->wrk->stats.fetch_head++;
+		return (BS_NONE);
+	}
+
+	if (hp->status <= 199) {
+		/*
+		 * 1xx responses never have a body.
+		 * [RFC2616 4.3 p33]
+		 */
+		sp->wrk->stats.fetch_1xx++;
+		return (BS_NONE);
+	}
+
+	if (hp->status == 204) {
+		/*
+		 * 204 is "No Content", obviously don't expect a body.
+		 * [RFC2616 10.2.5 p60]
+		 */
+		sp->wrk->stats.fetch_204++;
+		return (BS_NONE);
+	}
+
+	if (hp->status == 304) {
+		/*
+		 * 304 is "Not Modified" it has no body.
+		 * [RFC2616 10.3.5 p63]
+		 */
+		sp->wrk->stats.fetch_304++;
+		return (BS_NONE);
+	}
+
+	if (http_HdrIs(hp, H_Transfer_Encoding, "chunked")) {
+		 sp->wrk->stats.fetch_chunked++;
+		return (BS_CHUNKED);
+	}
+
+	if (http_GetHdr(hp, H_Transfer_Encoding, &b)) {
+		sp->wrk->stats.fetch_bad++;
+		return (BS_ERROR);
+	}
+
+	if (http_GetHdr(hp, H_Content_Length, &sp->wrk->h_content_length)) {
+		sp->wrk->stats.fetch_length++;
+		return (BS_LENGTH);
+	}
+
+	if (http_HdrIs(hp, H_Connection, "keep-alive")) {
+		/*
+		 * Keep alive with neither TE=Chunked or C-Len is impossible.
+		 * We assume a zero length body.
+		 */
+		sp->wrk->stats.fetch_zero++;
+		return (BS_ZERO);
+	}
+
+	if (http_HdrIs(hp, H_Connection, "close")) {
+		/*
+		 * In this case, it is safe to just read what comes.
+		 */
+		sp->wrk->stats.fetch_close++;
+		return (BS_EOF);
+	}
+
+	if (hp->protover < 11) {
+		/*
+		 * With no Connection header, assume EOF.
+		 */
+		sp->wrk->stats.fetch_oldhttp++;
+		return (BS_EOF);
+	}
+
+	/*
+	 * Fall back to EOF transfer.
+	 */
+	sp->wrk->stats.fetch_eof++;
+	return (BS_EOF);
+}
+
+/*--------------------------------------------------------------------
+ * Find out if the request can receive a gzip'ed response
+ */
+
+unsigned
+RFC2616_Req_Gzip(const struct sess *sp)
+{
+
+
+	/*
+	 * "x-gzip" is for http/1.0 backwards compat, final note in 14.3
+	 * p104 says to not do q values for x-gzip, so we just test
+	 * for its existence.
+	 */
+	if (http_GetHdrData(sp->http, H_Accept_Encoding, "x-gzip", NULL))
+		return (1);
+
+	/*
+	 * "gzip" is the real thing, but the 'q' value must be nonzero.
+	 * We do not care a hoot if the client prefers some other
+	 * compression more than gzip: Varnish only does gzip.
+	 */
+	if (http_GetHdrQ(sp->http, H_Accept_Encoding, "gzip") > 0.)
+		return (1);
+
+	/* Bad client, no gzip. */
+	return (0);
+}
+
+/*--------------------------------------------------------------------*/
+
+int
+RFC2616_Do_Cond(const struct sess *sp)
+{
+	char *p, *e;
+	double ims;
+	int do_cond = 0;
+
+	/* RFC 2616 13.3.4 states we need to match both ETag
+	   and If-Modified-Since if present*/
+
+	if (http_GetHdr(sp->http, H_If_Modified_Since, &p) ) {
+		if (!sp->obj->last_modified)
+			return (0);
+		ims = VTIM_parse(p);
+		if (ims > sp->t_req)	/* [RFC2616 14.25] */
+			return (0);
+		if (sp->obj->last_modified > ims)
+			return (0);
+		do_cond = 1;
+	}
+
+	if (http_GetHdr(sp->http, H_If_None_Match, &p) &&
+	    http_GetHdr(sp->obj->http, H_ETag, &e)) {
+		if (strcmp(p,e) != 0)
+			return (0);
+		do_cond = 1;
+	}
+
+	return (do_cond);
+}
diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c
new file mode 100644
index 0000000..7befbcc
--- /dev/null
+++ b/bin/varnishd/cache/cache_session.c
@@ -0,0 +1,419 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2010 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * Session management
+ *
+ * This is a little bit of a mixed back, containing both memory management
+ * and various state-change functions.
+ *
+ */
+
+#include "config.h"
+
+#include <math.h>
+#include <stdlib.h>
+
+#include "cache.h"
+
+#include "waiter/cache_waiter.h"
+
+/*--------------------------------------------------------------------*/
+
+struct sessmem {
+	unsigned		magic;
+#define SESSMEM_MAGIC		0x555859c5
+
+	struct sesspool		*pool;
+
+	unsigned		workspace;
+	uint16_t		nhttp;
+	void			*wsp;
+	struct http		*http[2];
+	VTAILQ_ENTRY(sessmem)	list;
+
+	struct sess		sess;
+};
+
+struct sesspool {
+	unsigned		magic;
+#define SESSPOOL_MAGIC		0xd916e202
+	struct pool		*pool;
+	VTAILQ_HEAD(,sessmem)	freelist;
+	struct lock		mtx;
+	unsigned		nsess;
+	unsigned		dly_free_cnt;
+};
+
+/*--------------------------------------------------------------------
+ * Charge statistics from worker to request and session.
+ */
+
+void
+SES_Charge(struct sess *sp)
+{
+	struct acct *a = &sp->wrk->acct_tmp;
+
+	sp->req_bodybytes += a->bodybytes;
+
+#define ACCT(foo)				\
+	sp->wrk->stats.s_##foo += a->foo;	\
+	sp->acct_ses.foo += a->foo;		\
+	a->foo = 0;
+#include "tbl/acct_fields.h"
+#undef ACCT
+}
+
+/*--------------------------------------------------------------------
+ * This function allocates a session + assorted peripheral data
+ * structures in one single malloc operation.
+ */
+
+static struct sessmem *
+ses_sm_alloc(void)
+{
+	struct sessmem *sm;
+	unsigned char *p, *q;
+	unsigned nws;
+	uint16_t nhttp;
+	unsigned l, hl;
+
+	/*
+	 * It is not necessary to lock these, but we need to
+	 * cache them locally, to make sure we get a consistent
+	 * view of the value.
+	 */
+	nws = cache_param->sess_workspace;
+	nhttp = (uint16_t)cache_param->http_max_hdr;
+
+	hl = HTTP_estimate(nhttp);
+	l = sizeof *sm + nws + 2 * hl;
+	VSC_C_main->sessmem_size = l;
+	p = malloc(l);
+	if (p == NULL)
+		return (NULL);
+	q = p + l;
+
+	/* Don't waste time zeroing the workspace */
+	memset(p, 0, l - nws);
+
+	sm = (void*)p;
+	p += sizeof *sm;
+
+	sm->magic = SESSMEM_MAGIC;
+	sm->workspace = nws;
+	sm->nhttp = nhttp;
+
+	sm->http[0] = HTTP_create(p, nhttp);
+	p += hl;
+
+	sm->http[1] = HTTP_create(p, nhttp);
+	p += hl;
+
+	sm->wsp = p;
+	p += nws;
+
+	assert(p == q);
+
+	return (sm);
+}
+
+/*--------------------------------------------------------------------
+ * This prepares a session for use, based on its sessmem structure.
+ */
+
+static void
+ses_setup(struct sessmem *sm)
+{
+	struct sess *sp;
+
+	CHECK_OBJ_NOTNULL(sm, SESSMEM_MAGIC);
+	sp = &sm->sess;
+	memset(sp, 0, sizeof *sp);
+
+	/* We assume that the sess has been zeroed by the time we get here */
+	AZ(sp->magic);
+
+	sp->magic = SESS_MAGIC;
+	sp->mem = sm;
+	sp->sockaddrlen = sizeof(sp->sockaddr);
+	sp->mysockaddrlen = sizeof(sp->mysockaddr);
+	sp->sockaddr.ss_family = sp->mysockaddr.ss_family = PF_UNSPEC;
+	sp->t_open = NAN;
+	sp->t_req = NAN;
+	sp->t_resp = NAN;
+	sp->t_end = NAN;
+	EXP_Clr(&sp->exp);
+
+	WS_Init(sp->ws, "sess", sm->wsp, sm->workspace);
+	sp->http = sm->http[0];
+	sp->http0 = sm->http[1];
+}
+
+/*--------------------------------------------------------------------
+ * Get a new session, preferably by recycling an already ready one
+ */
+
+struct sess *
+SES_New(struct worker *wrk, struct sesspool *pp)
+{
+	struct sessmem *sm;
+	struct sess *sp;
+	int do_alloc;
+
+	CHECK_OBJ_NOTNULL(pp, SESSPOOL_MAGIC);
+
+	do_alloc = 0;
+	Lck_Lock(&pp->mtx);
+	sm = VTAILQ_FIRST(&pp->freelist);
+	if (sm != NULL) {
+		VTAILQ_REMOVE(&pp->freelist, sm, list);
+	} else if (pp->nsess < cache_param->max_sess) {
+		pp->nsess++;
+		do_alloc = 1;
+	}
+	wrk->stats.sessmem_free += pp->dly_free_cnt;
+	pp->dly_free_cnt = 0;
+	Lck_Unlock(&pp->mtx);
+	if (do_alloc) {
+		sm = ses_sm_alloc();
+		if (sm != NULL) {
+			wrk->stats.sessmem_alloc++;
+			sm->pool = pp;
+			ses_setup(sm);
+		} else {
+			wrk->stats.sessmem_fail++;
+		}
+	} else if (sm == NULL) {
+		wrk->stats.sessmem_limit++;
+	}
+	if (sm == NULL)
+		return (NULL);
+	sp = &sm->sess;
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	return (sp);
+}
+
+/*--------------------------------------------------------------------
+ * Allocate a session for use by background threads.
+ */
+
+struct sess *
+SES_Alloc(void)
+{
+	struct sess *sp;
+	struct sessmem *sm;
+
+	sm = ses_sm_alloc();
+	AN(sm);
+	ses_setup(sm);
+	sp = &sm->sess;
+	sp->sockaddrlen = 0;
+	return (sp);
+}
+
+/*--------------------------------------------------------------------
+ * Schedule a session back on a work-thread from its pool
+ */
+
+int
+SES_Schedule(struct sess *sp)
+{
+	struct sessmem *sm;
+	struct sesspool *pp;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+
+	AZ(sp->wrk);
+
+	sm = sp->mem;
+	CHECK_OBJ_NOTNULL(sm, SESSMEM_MAGIC);
+
+	pp = sm->pool;
+	CHECK_OBJ_NOTNULL(pp, SESSPOOL_MAGIC);
+
+	AN(pp->pool);
+
+	if (Pool_Schedule(pp->pool, sp)) {
+		SES_Delete(sp, "dropped");
+		return (1);
+	}
+	return (0);
+}
+
+/*--------------------------------------------------------------------
+ * Handle a session (from waiter)
+ *
+ * Status: see HTC_Rx()
+ */
+
+void
+SES_Handle(struct sess *sp, int status)
+{
+
+	switch (status) {
+	case -2:
+		SES_Delete(sp, "blast");
+		break;
+	case -1:
+		SES_Delete(sp, "no request");
+		break;
+	case 1:
+		sp->step = STP_START;
+		(void)SES_Schedule(sp);
+		break;
+	default:
+		WRONG("Unexpected return from HTC_Rx()");
+	}
+}
+
+/*--------------------------------------------------------------------
+ * Close a sessions connection.
+ */
+
+void
+SES_Close(struct sess *sp, const char *reason)
+{
+	int i;
+
+	assert(sp->fd >= 0);
+	VSL(SLT_SessionClose, sp->vsl_id, "%s", reason);
+	i = close(sp->fd);
+	assert(i == 0 || errno != EBADF); /* XXX EINVAL seen */
+	sp->fd = -1;
+}
+
+/*--------------------------------------------------------------------
+ * (Close &) Free or Recycle a session.
+ *
+ * If the workspace has changed, deleted it, otherwise wash it, and put
+ * it up for adoption.
+ *
+ * XXX: We should also check nhttp
+ */
+
+void
+SES_Delete(struct sess *sp, const char *reason)
+{
+	struct acct *b;
+	struct sessmem *sm;
+	static char noaddr[] = "-";
+	struct worker *wrk;
+	struct sesspool *pp;
+
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	sm = sp->mem;
+	CHECK_OBJ_NOTNULL(sm, SESSMEM_MAGIC);
+	pp = sm->pool;
+	CHECK_OBJ_NOTNULL(pp, SESSPOOL_MAGIC);
+	wrk = sp->wrk;
+	CHECK_OBJ_ORNULL(wrk, WORKER_MAGIC);
+
+
+	if (reason != NULL)
+		SES_Close(sp, reason);
+	assert(sp->fd < 0);
+
+	AZ(sp->obj);
+	AZ(sp->vcl);
+	if (sp->addr == NULL)
+		sp->addr = noaddr;
+	if (sp->port == NULL)
+		sp->port = noaddr;
+
+	b = &sp->acct_ses;
+	assert(!isnan(b->first));
+	assert(!isnan(sp->t_end));
+
+	VSL(SLT_StatSess, sp->vsl_id, "%s %s %.0f %ju %ju %ju %ju %ju %ju %ju",
+	    sp->addr, sp->port, sp->t_end - b->first,
+	    b->sess, b->req, b->pipe, b->pass,
+	    b->fetch, b->hdrbytes, b->bodybytes);
+
+	if (sm->workspace != cache_param->sess_workspace ||
+	    sm->nhttp != (uint16_t)cache_param->http_max_hdr ||
+	    pp->nsess > cache_param->max_sess) {
+		free(sm);
+		Lck_Lock(&pp->mtx);
+		if (wrk != NULL)
+			wrk->stats.sessmem_free++;
+		else
+			pp->dly_free_cnt++;
+		pp->nsess--;
+		Lck_Unlock(&pp->mtx);
+	} else {
+		/* Clean and prepare for reuse */
+		ses_setup(sm);
+		Lck_Lock(&pp->mtx);
+		if (wrk != NULL) {
+			wrk->stats.sessmem_free += pp->dly_free_cnt;
+			pp->dly_free_cnt = 0;
+		}
+		VTAILQ_INSERT_HEAD(&pp->freelist, sm, list);
+		Lck_Unlock(&pp->mtx);
+	}
+}
+
+/*--------------------------------------------------------------------
+ * Create and delete pools
+ */
+
+struct sesspool *
+SES_NewPool(struct pool *pp)
+{
+	struct sesspool *sp;
+
+	ALLOC_OBJ(sp, SESSPOOL_MAGIC);
+	AN(sp);
+	sp->pool = pp;
+	VTAILQ_INIT(&sp->freelist);
+	Lck_New(&sp->mtx, lck_sessmem);
+	return (sp);
+}
+
+void
+SES_DeletePool(struct sesspool *sp, struct worker *wrk)
+{
+	struct sessmem *sm;
+
+	CHECK_OBJ_NOTNULL(sp, SESSPOOL_MAGIC);
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
+	Lck_Lock(&sp->mtx);
+	while (!VTAILQ_EMPTY(&sp->freelist)) {
+		sm = VTAILQ_FIRST(&sp->freelist);
+		CHECK_OBJ_NOTNULL(sm, SESSMEM_MAGIC);
+		VTAILQ_REMOVE(&sp->freelist, sm, list);
+		FREE_OBJ(sm);
+		wrk->stats.sessmem_free++;
+		sp->nsess--;
+	}
+	AZ(sp->nsess);
+	Lck_Unlock(&sp->mtx);
+	Lck_Delete(&sp->mtx);
+	FREE_OBJ(sp);
+}
diff --git a/bin/varnishd/cache/cache_shmlog.c b/bin/varnishd/cache/cache_shmlog.c
new file mode 100644
index 0000000..1252fa3
--- /dev/null
+++ b/bin/varnishd/cache/cache_shmlog.c
@@ -0,0 +1,346 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include "config.h"
+
+#include <stdio.h>
+
+#include "cache.h"
+
+#include "cache_backend.h"	// For w->vbc
+
+#include "vapi/vsm_int.h"
+#include "vmb.h"
+#include "vtim.h"
+
+/* These cannot be struct lock, which depends on vsm/vsl working */
+static pthread_mutex_t vsl_mtx;
+static pthread_mutex_t vsm_mtx;
+
+static uint32_t			*vsl_start;
+static const uint32_t		*vsl_end;
+static uint32_t			*vsl_ptr;
+
+static inline uint32_t
+vsl_w0(uint32_t type, uint32_t length)
+{
+
+	assert(length < 0x10000);
+        return (((type & 0xff) << 24) | length);
+}
+
+/*--------------------------------------------------------------------*/
+
+static inline void
+vsl_hdr(enum VSL_tag_e tag, uint32_t *p, unsigned len, unsigned id)
+{
+
+	assert(((uintptr_t)p & 0x3) == 0);
+
+	p[1] = id;
+	VMB();
+	p[0] = vsl_w0(tag, len);
+}
+
+/*--------------------------------------------------------------------*/
+
+static void
+vsl_wrap(void)
+{
+
+	assert(vsl_ptr >= vsl_start + 1);
+	assert(vsl_ptr < vsl_end);
+	vsl_start[1] = VSL_ENDMARKER;
+	do
+		vsl_start[0]++;
+	while (vsl_start[0] == 0);
+	VWMB();
+	if (vsl_ptr != vsl_start + 1) {
+		*vsl_ptr = VSL_WRAPMARKER;
+		vsl_ptr = vsl_start + 1;
+	}
+	VSC_C_main->shm_cycles++;
+}
+
+/*--------------------------------------------------------------------
+ * Reserve bytes for a record, wrap if necessary
+ */
+
+static uint32_t *
+vsl_get(unsigned len, unsigned records, unsigned flushes)
+{
+	uint32_t *p;
+
+	if (pthread_mutex_trylock(&vsl_mtx)) {
+		AZ(pthread_mutex_lock(&vsl_mtx));
+		VSC_C_main->shm_cont++;
+	}
+	assert(vsl_ptr < vsl_end);
+	assert(((uintptr_t)vsl_ptr & 0x3) == 0);
+
+	VSC_C_main->shm_writes++;
+	VSC_C_main->shm_flushes += flushes;
+	VSC_C_main->shm_records += records;
+
+	/* Wrap if necessary */
+	if (VSL_END(vsl_ptr, len) >= vsl_end)
+		vsl_wrap();
+
+	p = vsl_ptr;
+	vsl_ptr = VSL_END(vsl_ptr, len);
+
+	*vsl_ptr = VSL_ENDMARKER;
+
+	assert(vsl_ptr < vsl_end);
+	assert(((uintptr_t)vsl_ptr & 0x3) == 0);
+	AZ(pthread_mutex_unlock(&vsl_mtx));
+
+	return (p);
+}
+
+/*--------------------------------------------------------------------
+ * This variant copies a byte-range directly to the log, without
+ * taking the detour over sprintf()
+ */
+
+static void
+VSLR(enum VSL_tag_e tag, int id, const char *b, unsigned len)
+{
+	uint32_t *p;
+	unsigned mlen;
+
+	mlen = cache_param->shm_reclen;
+
+	/* Truncate */
+	if (len > mlen)
+		len = mlen;
+
+	p = vsl_get(len, 1, 0);
+
+	memcpy(p + 2, b, len);
+	vsl_hdr(tag, p, len, id);
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+VSL(enum VSL_tag_e tag, int id, const char *fmt, ...)
+{
+	va_list ap;
+	unsigned n, mlen = cache_param->shm_reclen;
+	char buf[mlen];
+
+	/*
+	 * XXX: consider formatting into a stack buffer then move into
+	 * XXX: shmlog with VSLR().
+	 */
+	AN(fmt);
+	va_start(ap, fmt);
+
+	if (strchr(fmt, '%') == NULL) {
+		VSLR(tag, id, fmt, strlen(fmt));
+	} else {
+		n = vsnprintf(buf, mlen, fmt, ap);
+		if (n > mlen)
+			n = mlen;
+		VSLR(tag, id, buf, n);
+	}
+	va_end(ap);
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+WSL_Flush(struct worker *w, int overflow)
+{
+	uint32_t *p;
+	unsigned l;
+
+	l = pdiff(w->wlb, w->wlp);
+	if (l == 0)
+		return;
+
+	assert(l >= 8);
+
+	p = vsl_get(l - 8, w->wlr, overflow);
+
+	memcpy(p + 1, w->wlb + 1, l - 4);
+	VWMB();
+	p[0] = w->wlb[0];
+	w->wlp = w->wlb;
+	w->wlr = 0;
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+WSLR(struct worker *w, enum VSL_tag_e tag, int id, txt t)
+{
+	unsigned l, mlen;
+
+	Tcheck(t);
+	mlen = cache_param->shm_reclen;
+
+	/* Truncate */
+	l = Tlen(t);
+	if (l > mlen) {
+		l = mlen;
+		t.e = t.b + l;
+	}
+
+	assert(w->wlp < w->wle);
+
+	/* Wrap if necessary */
+	if (VSL_END(w->wlp, l) >= w->wle)
+		WSL_Flush(w, 1);
+	assert (VSL_END(w->wlp, l) < w->wle);
+	memcpy(VSL_DATA(w->wlp), t.b, l);
+	vsl_hdr(tag, w->wlp, l, id);
+	w->wlp = VSL_END(w->wlp, l);
+	assert(w->wlp < w->wle);
+	w->wlr++;
+	if (cache_param->diag_bitmap & 0x10000)
+		WSL_Flush(w, 0);
+}
+
+/*--------------------------------------------------------------------*/
+
+static void
+wsl(struct worker *w, enum VSL_tag_e tag, int id, const char *fmt, va_list ap)
+{
+	char *p;
+	unsigned n, mlen;
+	txt t;
+
+	AN(fmt);
+	mlen = cache_param->shm_reclen;
+
+	if (strchr(fmt, '%') == NULL) {
+		t.b = TRUST_ME(fmt);
+		t.e = strchr(t.b, '\0');
+		WSLR(w, tag, id, t);
+	} else {
+		assert(w->wlp < w->wle);
+
+		/* Wrap if we cannot fit a full size record */
+		if (VSL_END(w->wlp, mlen) >= w->wle)
+			WSL_Flush(w, 1);
+
+		p = VSL_DATA(w->wlp);
+		n = vsnprintf(p, mlen, fmt, ap);
+		if (n > mlen)
+			n = mlen;	/* we truncate long fields */
+		vsl_hdr(tag, w->wlp, n, id);
+		w->wlp = VSL_END(w->wlp, n);
+		assert(w->wlp < w->wle);
+		w->wlr++;
+	}
+	if (cache_param->diag_bitmap & 0x10000)
+		WSL_Flush(w, 0);
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+WSL(struct worker *w, enum VSL_tag_e tag, int id, const char *fmt, ...)
+{
+	va_list ap;
+
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	AN(fmt);
+	va_start(ap, fmt);
+	wsl(w, tag, id, fmt, ap);
+	va_end(ap);
+}
+
+
+/*--------------------------------------------------------------------*/
+
+void
+WSLB(struct worker *w, enum VSL_tag_e tag, const char *fmt, ...)
+{
+	va_list ap;
+
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	CHECK_OBJ_NOTNULL(w->vbc, VBC_MAGIC);
+	AN(fmt);
+	va_start(ap, fmt);
+	wsl(w, tag, w->vbc->vsl_id, fmt, ap);
+	va_end(ap);
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+VSL_Init(void)
+{
+	struct VSM_chunk *vsc;
+
+	AZ(pthread_mutex_init(&vsl_mtx, NULL));
+	AZ(pthread_mutex_init(&vsm_mtx, NULL));
+
+	VSM__Clean();
+
+	VSM_ITER(vsc)
+		if (!strcmp(vsc->class, VSL_CLASS))
+			break;
+	AN(vsc);
+	vsl_start = VSM_PTR(vsc);
+	vsl_end = VSM_NEXT(vsc);
+	vsl_ptr = vsl_start + 1;
+
+	vsl_wrap();
+	VSM_head->starttime = (intmax_t)VTIM_real();
+	memset(VSM_head->panicstr, '\0', sizeof *VSM_head->panicstr);
+	memset(VSC_C_main, 0, sizeof *VSC_C_main);
+	VSM_head->child_pid = getpid();
+}
+
+/*--------------------------------------------------------------------*/
+
+void *
+VSM_Alloc(unsigned size, const char *class, const char *type,
+    const char *ident)
+{
+	void *p;
+
+	AZ(pthread_mutex_lock(&vsm_mtx));
+	p = VSM__Alloc(size, class, type, ident);
+	AZ(pthread_mutex_unlock(&vsm_mtx));
+	return (p);
+}
+
+void
+VSM_Free(const void *ptr)
+{
+
+	AZ(pthread_mutex_lock(&vsm_mtx));
+	VSM__Free(ptr);
+	AZ(pthread_mutex_unlock(&vsm_mtx));
+}
diff --git a/bin/varnishd/cache/cache_vary.c b/bin/varnishd/cache/cache_vary.c
new file mode 100644
index 0000000..026f937
--- /dev/null
+++ b/bin/varnishd/cache/cache_vary.c
@@ -0,0 +1,257 @@
+/*-
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * Do Vary processing.
+ *
+ * When we insert an object into the cache which has a Vary: header,
+ * we encode a vary matching string containing the headers mentioned
+ * and their value.
+ *
+ * When we match an object in the cache, we check the present request
+ * against the vary matching string.
+ *
+ * The only kind of header-munging we do is leading & trailing space
+ * removal.  All the potential "q=foo" gymnastics is not worth the
+ * effort.
+ *
+ * The vary matching string has the following format:
+ *
+ * Sequence of: {
+ *	<msb>			\   Length of header contents.
+ *	<lsb>			/
+ *	<length of header + 1>	\
+ *	<header>		 \  Same format as argument to http_GetHdr()
+ *	':'			 /
+ *	'\0'			/
+ *      <header>		>   Only present if length != 0xffff
+ * }
+ *      '\0'
+ */
+
+#include "config.h"
+
+#include "cache.h"
+
+#include "vct.h"
+#include "vend.h"
+
+struct vsb *
+VRY_Create(const struct sess *sp, const struct http *hp)
+{
+	char *v, *p, *q, *h, *e;
+	struct vsb *sb, *sbh;
+	int l;
+
+	/* No Vary: header, no worries */
+	if (!http_GetHdr(hp, H_Vary, &v))
+		return (NULL);
+
+	/* For vary matching string */
+	sb = VSB_new_auto();
+	AN(sb);
+
+	/* For header matching strings */
+	sbh = VSB_new_auto();
+	AN(sbh);
+
+	if (*v == ':') {
+		WSP(sp, SLT_Error, "Vary header had extra ':', fix backend");
+		v++;
+	}
+	for (p = v; *p; p++) {
+
+		/* Find next header-name */
+		if (vct_issp(*p))
+			continue;
+		for (q = p; *q && !vct_issp(*q) && *q != ','; q++)
+			continue;
+
+		/* Build a header-matching string out of it */
+		VSB_clear(sbh);
+		VSB_printf(sbh, "%c%.*s:%c",
+		    (char)(1 + (q - p)), (int)(q - p), p, 0);
+		AZ(VSB_finish(sbh));
+
+		if (http_GetHdr(sp->http, VSB_data(sbh), &h)) {
+			AZ(vct_issp(*h));
+			/* Trim trailing space */
+			e = strchr(h, '\0');
+			while (e > h && vct_issp(e[-1]))
+				e--;
+			/* Encode two byte length and contents */
+			l = e - h;
+			assert(!(l & ~0xffff));
+		} else {
+			e = h;
+			l = 0xffff;
+		}
+		VSB_printf(sb, "%c%c", (unsigned)l >> 8, l & 0xff);
+		/* Append to vary matching string */
+		VSB_bcat(sb, VSB_data(sbh), VSB_len(sbh));
+		if (e != h)
+			VSB_bcat(sb, h, e - h);
+
+		while (vct_issp(*q))
+			q++;
+		if (*q == '\0')
+			break;
+		xxxassert(*q == ',');
+		p = q;
+	}
+	/* Terminate vary matching string */
+	VSB_printf(sb, "%c%c%c", 0xff, 0xff, 0);
+
+	VSB_delete(sbh);
+	AZ(VSB_finish(sb));
+	return(sb);
+}
+
+/*
+ * Find length of a vary entry
+ */
+static unsigned
+vry_len(const uint8_t *p)
+{
+	unsigned l = vbe16dec(p);
+
+	return (2 + p[2] + 2 + (l == 0xffff ? 0 : l));
+}
+
+/*
+ * Compare two vary entries
+ */
+static int
+vry_cmp(const uint8_t * const *v1, uint8_t * const *v2)
+{
+	unsigned retval = 0;
+
+	if (!memcmp(*v1, *v2, vry_len(*v1))) {
+		/* Same same */
+		retval = 0;
+	} else if (memcmp((*v1) + 2, (*v2) + 2, (*v1)[2] + 2)) {
+		/* Different header */
+		retval = 1;
+	} else if (cache_param->http_gzip_support &&
+	    !strcasecmp(H_Accept_Encoding, (const char*)((*v1)+2))) {
+		/*
+		 * If we do gzip processing, we do not vary on Accept-Encoding,
+		 * because we want everybody to get the gzip'ed object, and
+		 * varnish will gunzip as necessary.  We implement the skip at
+		 * check time, rather than create time, so that object in
+		 * persistent storage can be used with either setting of
+		 * http_gzip_support.
+		 */
+		retval = 0;
+	} else {
+		/* Same header, different content */
+		retval = 2;
+	}
+	return (retval);
+}
+
+int
+VRY_Match(struct sess *sp, const uint8_t *vary)
+{
+	uint8_t *vsp = sp->vary_b;
+	char *h, *e;
+	unsigned lh, ln;
+	int i, retval = 1, oflo = 0;
+
+	AN(vsp);
+	while (vary[2]) {
+		i = vry_cmp(&vary, &vsp);
+		if (i == 1) {
+			/* Build a new entry */
+
+			i = http_GetHdr(sp->http, (const char*)(vary+2), &h);
+			if (i) {
+				/* Trim trailing space */
+				e = strchr(h, '\0');
+				while (e > h && vct_issp(e[-1]))
+					e--;
+				lh = e - h;
+				assert(lh < 0xffff);
+			} else {
+				e = h = NULL;
+				lh = 0xffff;
+			}
+
+			/* Length of the entire new vary entry */
+			ln = 2 + vary[2] + 2 + (lh == 0xffff ? 0 : lh);
+			if (vsp + ln >= sp->vary_e) {
+				vsp = sp->vary_b;
+				oflo = 1;
+			}
+
+			/*
+			 * We MUST have space for one entry and the end marker
+			 * after it, which prevents old junk from confusing us
+			 */
+			assert(vsp + ln + 2 < sp->vary_e);
+
+			vbe16enc(vsp, (uint16_t)lh);
+			memcpy(vsp + 2, vary + 2, vary[2] + 2);
+			if (h != NULL && e != NULL) {
+				memcpy(vsp + 2 + vsp[2] + 2, h, e - h);
+				vsp[2 + vary[2] + 2 + (e - h) + 2] = '\0';
+			} else
+				vsp[2 + vary[2] + 2 + 2] = '\0';
+
+			i = vry_cmp(&vary, &vsp);
+			assert(i != 1);	/* hdr must be the same now */
+		}
+		if (i != 0)
+			retval = 0;
+		vsp += vry_len(vsp);
+		vary += vry_len(vary);
+	}
+	if (vsp + 3 > sp->vary_e)
+		oflo = 1;
+
+	if (oflo) {
+		/* XXX: Should log this */
+		vsp = sp->vary_b;
+	}
+	vsp[0] = 0xff;
+	vsp[1] = 0xff;
+	vsp[2] = 0;
+	if (oflo)
+		sp->vary_l = NULL;
+	else
+		sp->vary_l = vsp + 3;
+	return (retval);
+}
+
+void
+VRY_Validate(const uint8_t *vary)
+{
+
+	while (vary[2] != 0) {
+		assert(strlen((const char*)vary+3) == vary[2]);
+		vary += vry_len(vary);
+	}
+}
diff --git a/bin/varnishd/cache/cache_vcl.c b/bin/varnishd/cache/cache_vcl.c
new file mode 100644
index 0000000..068b482
--- /dev/null
+++ b/bin/varnishd/cache/cache_vcl.c
@@ -0,0 +1,365 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * Interface *to* compiled VCL code:  Loading, unloading, calling into etc.
+ *
+ * The interface *from* the compiled VCL code is in cache_vrt.c.
+ */
+
+#include "config.h"
+
+#include <dlfcn.h>
+#include <stdlib.h>
+
+#include "cache.h"
+
+#include "vcl.h"
+#include "vcli.h"
+#include "vcli_priv.h"
+
+struct vcls {
+	unsigned		magic;
+#define VVCLS_MAGIC		0x214188f2
+	VTAILQ_ENTRY(vcls)	list;
+	char			*name;
+	void			*dlh;
+	struct VCL_conf		conf[1];
+};
+
+/*
+ * XXX: Presently all modifications to this list happen from the
+ * CLI event-engine, so no locking is necessary
+ */
+static VTAILQ_HEAD(, vcls)	vcl_head =
+    VTAILQ_HEAD_INITIALIZER(vcl_head);
+
+
+static struct lock		vcl_mtx;
+static struct vcls		*vcl_active; /* protected by vcl_mtx */
+
+/*--------------------------------------------------------------------*/
+
+const char *
+VCL_Return_Name(unsigned method)
+{
+
+	switch (method) {
+#define VCL_RET_MAC(l, U, B) case VCL_RET_##U: return(#l);
+#include "tbl/vcl_returns.h"
+#undef VCL_RET_MAC
+	default:
+		return (NULL);
+	}
+}
+
+/*--------------------------------------------------------------------*/
+
+static void
+VCL_Get(struct VCL_conf **vcc)
+{
+	static int once = 0;
+
+	while (!once && vcl_active == NULL) {
+		(void)sleep(1);
+	}
+	once = 1;
+
+	Lck_Lock(&vcl_mtx);
+	AN(vcl_active);
+	*vcc = vcl_active->conf;
+	AN(*vcc);
+	AZ((*vcc)->discard);
+	(*vcc)->busy++;
+	Lck_Unlock(&vcl_mtx);
+}
+
+void
+VCL_Refresh(struct VCL_conf **vcc)
+{
+	if (*vcc == vcl_active->conf)
+		return;
+	if (*vcc != NULL)
+		VCL_Rel(vcc);	/* XXX: optimize locking */
+	VCL_Get(vcc);
+}
+
+void
+VCL_Rel(struct VCL_conf **vcc)
+{
+	struct VCL_conf *vc;
+
+	AN(*vcc);
+	vc = *vcc;
+	*vcc = NULL;
+
+	Lck_Lock(&vcl_mtx);
+	assert(vc->busy > 0);
+	vc->busy--;
+	/*
+	 * We do not garbage collect discarded VCL's here, that happens
+	 * in VCL_Poll() which is called from the CLI thread.
+	 */
+	Lck_Unlock(&vcl_mtx);
+}
+
+/*--------------------------------------------------------------------*/
+
+static struct vcls *
+vcl_find(const char *name)
+{
+	struct vcls *vcl;
+
+	ASSERT_CLI();
+	VTAILQ_FOREACH(vcl, &vcl_head, list) {
+		if (vcl->conf->discard)
+			continue;
+		if (!strcmp(vcl->name, name))
+			return (vcl);
+	}
+	return (NULL);
+}
+
+static int
+VCL_Load(const char *fn, const char *name, struct cli *cli)
+{
+	struct vcls *vcl;
+	struct VCL_conf const *cnf;
+
+	ASSERT_CLI();
+	vcl = vcl_find(name);
+	if (vcl != NULL) {
+		VCLI_Out(cli, "Config '%s' already loaded", name);
+		return (1);
+	}
+
+	ALLOC_OBJ(vcl, VVCLS_MAGIC);
+	XXXAN(vcl);
+
+	vcl->dlh = dlopen(fn, RTLD_NOW | RTLD_LOCAL);
+
+	if (vcl->dlh == NULL) {
+		VCLI_Out(cli, "dlopen(%s): %s\n", fn, dlerror());
+		FREE_OBJ(vcl);
+		return (1);
+	}
+	cnf = dlsym(vcl->dlh, "VCL_conf");
+	if (cnf == NULL) {
+		VCLI_Out(cli, "Internal error: No VCL_conf symbol\n");
+		(void)dlclose(vcl->dlh);
+		FREE_OBJ(vcl);
+		return (1);
+	}
+	memcpy(vcl->conf, cnf, sizeof *cnf);
+
+	if (vcl->conf->magic != VCL_CONF_MAGIC) {
+		VCLI_Out(cli, "Wrong VCL_CONF_MAGIC\n");
+		(void)dlclose(vcl->dlh);
+		FREE_OBJ(vcl);
+		return (1);
+	}
+	if (vcl->conf->init_vcl(cli)) {
+		VCLI_Out(cli, "VCL \"%s\" Failed to initialize", name);
+		(void)dlclose(vcl->dlh);
+		FREE_OBJ(vcl);
+		return (1);
+	}
+	REPLACE(vcl->name, name);
+	VCLI_Out(cli, "Loaded \"%s\" as \"%s\"", fn , name);
+	VTAILQ_INSERT_TAIL(&vcl_head, vcl, list);
+	(void)vcl->conf->init_func(NULL);
+	Lck_Lock(&vcl_mtx);
+	if (vcl_active == NULL)
+		vcl_active = vcl;
+	Lck_Unlock(&vcl_mtx);
+	VSC_C_main->n_vcl++;
+	VSC_C_main->n_vcl_avail++;
+	return (0);
+}
+
+/*--------------------------------------------------------------------
+ * This function is polled from the CLI thread to dispose of any non-busy
+ * VCLs which have been discarded.
+ */
+
+static void
+VCL_Nuke(struct vcls *vcl)
+{
+
+	ASSERT_CLI();
+	assert(vcl != vcl_active);
+	assert(vcl->conf->discard);
+	assert(vcl->conf->busy == 0);
+	VTAILQ_REMOVE(&vcl_head, vcl, list);
+	(void)vcl->conf->fini_func(NULL);
+	vcl->conf->fini_vcl(NULL);
+	free(vcl->name);
+	(void)dlclose(vcl->dlh);
+	FREE_OBJ(vcl);
+	VSC_C_main->n_vcl--;
+	VSC_C_main->n_vcl_discard--;
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+VCL_Poll(void)
+{
+	struct vcls *vcl, *vcl2;
+
+	ASSERT_CLI();
+	VTAILQ_FOREACH_SAFE(vcl, &vcl_head, list, vcl2)
+		if (vcl->conf->discard && vcl->conf->busy == 0)
+			VCL_Nuke(vcl);
+}
+
+/*--------------------------------------------------------------------*/
+
+static void
+ccf_config_list(struct cli *cli, const char * const *av, void *priv)
+{
+	struct vcls *vcl;
+	const char *flg;
+
+	(void)av;
+	(void)priv;
+	ASSERT_CLI();
+	VTAILQ_FOREACH(vcl, &vcl_head, list) {
+		if (vcl == vcl_active) {
+			flg = "active";
+		} else if (vcl->conf->discard) {
+			flg = "discarded";
+		} else
+			flg = "available";
+		VCLI_Out(cli, "%-10s %6u %s\n",
+		    flg,
+		    vcl->conf->busy,
+		    vcl->name);
+	}
+}
+
+static void
+ccf_config_load(struct cli *cli, const char * const *av, void *priv)
+{
+
+	(void)av;
+	(void)priv;
+	ASSERT_CLI();
+	if (VCL_Load(av[3], av[2], cli))
+		VCLI_SetResult(cli, CLIS_PARAM);
+	return;
+}
+
+static void
+ccf_config_discard(struct cli *cli, const char * const *av, void *priv)
+{
+	struct vcls *vcl;
+
+	ASSERT_CLI();
+	(void)av;
+	(void)priv;
+	vcl = vcl_find(av[2]);
+	if (vcl == NULL) {
+		VCLI_SetResult(cli, CLIS_PARAM);
+		VCLI_Out(cli, "VCL '%s' unknown", av[2]);
+		return;
+	}
+	Lck_Lock(&vcl_mtx);
+	if (vcl == vcl_active) {
+		Lck_Unlock(&vcl_mtx);
+		VCLI_SetResult(cli, CLIS_PARAM);
+		VCLI_Out(cli, "VCL %s is the active VCL", av[2]);
+		return;
+	}
+	VSC_C_main->n_vcl_discard++;
+	VSC_C_main->n_vcl_avail--;
+	vcl->conf->discard = 1;
+	Lck_Unlock(&vcl_mtx);
+	if (vcl->conf->busy == 0)
+		VCL_Nuke(vcl);
+}
+
+static void
+ccf_config_use(struct cli *cli, const char * const *av, void *priv)
+{
+	struct vcls *vcl;
+	int i;
+
+	(void)av;
+	(void)priv;
+	vcl = vcl_find(av[2]);
+	if (vcl == NULL) {
+		VCLI_Out(cli, "No VCL named '%s'", av[2]);
+		VCLI_SetResult(cli, CLIS_PARAM);
+		return;
+	}
+	Lck_Lock(&vcl_mtx);
+	vcl_active = vcl;
+	Lck_Unlock(&vcl_mtx);
+
+	/* Tickle this VCL's backends to take over health polling */
+	for(i = 1; i < vcl->conf->ndirector; i++)
+		VBE_UseHealth(vcl->conf->director[i]);
+}
+
+/*--------------------------------------------------------------------*/
+
+#define VCL_MET_MAC(func, upper, bitmap)				\
+void									\
+VCL_##func##_method(struct sess *sp)					\
+{									\
+									\
+	sp->handling = 0;						\
+	sp->cur_method = VCL_MET_ ## upper;				\
+	WSP(sp, SLT_VCL_call, "%s", #func);				\
+	(void)sp->vcl->func##_func(sp);					\
+	WSP(sp, SLT_VCL_return, "%s", VCL_Return_Name(sp->handling));	\
+	sp->cur_method = 0;						\
+	assert((1U << sp->handling) & bitmap);				\
+	assert(!((1U << sp->handling) & ~bitmap));			\
+}
+
+#include "tbl/vcl_returns.h"
+#undef VCL_MET_MAC
+
+/*--------------------------------------------------------------------*/
+
+static struct cli_proto vcl_cmds[] = {
+	{ CLI_VCL_LOAD,         "i", ccf_config_load },
+	{ CLI_VCL_LIST,         "i", ccf_config_list },
+	{ CLI_VCL_DISCARD,      "i", ccf_config_discard },
+	{ CLI_VCL_USE,          "i", ccf_config_use },
+	{ NULL }
+};
+
+void
+VCL_Init()
+{
+
+	CLI_AddFuncs(vcl_cmds);
+	Lck_New(&vcl_mtx, lck_vcl);
+}
diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c
new file mode 100644
index 0000000..c20b552
--- /dev/null
+++ b/bin/varnishd/cache/cache_vrt.c
@@ -0,0 +1,535 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2010 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * Runtime support for compiled VCL programs
+ */
+
+#include "config.h"
+
+#include <netinet/in.h>
+#include <arpa/inet.h>
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#include "cache.h"
+
+#include "cache_backend.h"
+#include "hash/hash_slinger.h"
+#include "vav.h"
+#include "vcl.h"
+#include "vrt.h"
+#include "vrt_obj.h"
+#include "vtim.h"
+
+const void * const vrt_magic_string_end = &vrt_magic_string_end;
+
+/*--------------------------------------------------------------------*/
+
+void
+VRT_error(struct sess *sp, unsigned code, const char *reason)
+{
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	WSL(sp->wrk, SLT_Debug, 0, "VCL_error(%u, %s)", code, reason ?
+	    reason : "(null)");
+	if (code < 100 || code > 999)
+		code = 503;
+	sp->err_code = (uint16_t)code;
+	sp->err_reason = reason ? reason : http_StatusMessage(sp->err_code);
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+VRT_count(const struct sess *sp, unsigned u)
+{
+
+	if (sp == NULL)
+		return;
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	if (cache_param->vcl_trace)
+		WSP(sp, SLT_VCL_trace, "%u %d.%d", u,
+		    sp->vcl->ref[u].line, sp->vcl->ref[u].pos);
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+VRT_acl_log(const struct sess *sp, const char *msg)
+{
+	WSP(sp, SLT_VCL_acl, msg);
+}
+
+/*--------------------------------------------------------------------*/
+
+static struct http *
+vrt_selecthttp(const struct sess *sp, enum gethdr_e where)
+{
+	struct http *hp;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	switch (where) {
+	case HDR_REQ:
+		hp = sp->http;
+		break;
+	case HDR_BEREQ:
+		hp = sp->wrk->bereq;
+		break;
+	case HDR_BERESP:
+		hp = sp->wrk->beresp;
+		break;
+	case HDR_RESP:
+		hp = sp->wrk->resp;
+		break;
+	case HDR_OBJ:
+		CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);
+		hp = sp->obj->http;
+		break;
+	default:
+		INCOMPL();
+	}
+	CHECK_OBJ_NOTNULL(hp, HTTP_MAGIC);
+	return (hp);
+}
+
+char *
+VRT_GetHdr(const struct sess *sp, enum gethdr_e where, const char *n)
+{
+	char *p;
+	struct http *hp;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	hp = vrt_selecthttp(sp, where);
+	if (!http_GetHdr(hp, n, &p))
+		return (NULL);
+	return (p);
+}
+
+/*--------------------------------------------------------------------
+ * XXX: Optimize the single element case ?
+ */
+
+char *
+VRT_StringList(char *d, unsigned dl, const char *p, va_list ap)
+{
+	char *b, *e;
+	unsigned x;
+
+	b = d;
+	e = b + dl;
+	while (p != vrt_magic_string_end && b < e) {
+		if (p != NULL) {
+			x = strlen(p);
+			if (b + x < e)
+				memcpy(b, p, x);
+			b += x;
+		}
+		p = va_arg(ap, const char *);
+	}
+	if (b >= e)
+		return (NULL);
+	*b++ = '\0';
+	return (b);
+}
+
+/*--------------------------------------------------------------------
+ * XXX: Optimize the single element case ?
+ */
+
+char *
+VRT_String(struct ws *ws, const char *h, const char *p, va_list ap)
+{
+	char *b, *e;
+	unsigned u, x;
+
+	u = WS_Reserve(ws, 0);
+	e = b = ws->f;
+	e += u;
+	if (h != NULL) {
+		x = strlen(h);
+		if (b + x < e)
+			memcpy(b, h, x);
+		b += x;
+		if (b < e)
+			*b = ' ';
+		b++;
+	}
+	b = VRT_StringList(b, e > b ? e - b : 0, p, ap);
+	if (b == NULL || b == e) {
+		WS_Release(ws, 0);
+		return (NULL);
+	}
+	e = b;
+	b = ws->f;
+	WS_Release(ws, e - b);
+	return (b);
+}
+
+/*--------------------------------------------------------------------
+ * Build a string on the worker threads workspace
+ */
+
+const char *
+VRT_WrkString(const struct sess *sp, const char *p, ...)
+{
+	va_list ap;
+	char *b;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	va_start(ap, p);
+	b = VRT_String(sp->wrk->ws, NULL, p, ap);
+	va_end(ap);
+	return (b);
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+VRT_SetHdr(const struct sess *sp , enum gethdr_e where, const char *hdr,
+    const char *p, ...)
+{
+	struct http *hp;
+	va_list ap;
+	char *b;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	hp = vrt_selecthttp(sp, where);
+	va_start(ap, p);
+	if (p == NULL) {
+		http_Unset(hp, hdr);
+	} else {
+		b = VRT_String(hp->ws, hdr + 1, p, ap);
+		if (b == NULL) {
+			WSP(sp, SLT_LostHeader, "%s", hdr + 1);
+		} else {
+			http_Unset(hp, hdr);
+			http_SetHeader(sp->wrk, sp->vsl_id, hp, b);
+		}
+	}
+	va_end(ap);
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+VRT_handling(struct sess *sp, unsigned hand)
+{
+
+	if (sp == NULL) {
+		assert(hand == VCL_RET_OK);
+		return;
+	}
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	assert(hand < VCL_RET_MAX);
+	sp->handling = hand;
+}
+
+/*--------------------------------------------------------------------
+ * Add an element to the array/list of hash bits.
+ */
+
+void
+VRT_hashdata(const struct sess *sp, const char *str, ...)
+{
+	va_list ap;
+	const char *p;
+
+	HSH_AddString(sp, str);
+	va_start(ap, str);
+	while (1) {
+		p = va_arg(ap, const char *);
+		if (p == vrt_magic_string_end)
+			break;
+		HSH_AddString(sp, p);
+	}
+}
+
+/*--------------------------------------------------------------------*/
+
+double
+VRT_r_now(const struct sess *sp)
+{
+
+	(void)sp;
+	return (VTIM_real());
+}
+
+/*--------------------------------------------------------------------*/
+
+char *
+VRT_IP_string(const struct sess *sp, const struct sockaddr_storage *sa)
+{
+	char *p;
+	const struct sockaddr_in *si4;
+	const struct sockaddr_in6 *si6;
+	const void *addr;
+	int len;
+
+	switch (sa->ss_family) {
+	case AF_INET:
+		len = INET_ADDRSTRLEN;
+		si4 = (const void *)sa;
+		addr = &(si4->sin_addr);
+		break;
+	case AF_INET6:
+		len = INET6_ADDRSTRLEN;
+		si6 = (const void *)sa;
+		addr = &(si6->sin6_addr);
+		break;
+	default:
+		INCOMPL();
+	}
+	XXXAN(len);
+	AN(p = WS_Alloc(sp->http->ws, len));
+	AN(inet_ntop(sa->ss_family, addr, p, len));
+	return (p);
+}
+
+char *
+VRT_int_string(const struct sess *sp, int num)
+{
+	char *p;
+	int size;
+
+	size = snprintf(NULL, 0, "%d", num) + 1;
+	AN(p = WS_Alloc(sp->http->ws, size));
+	assert(snprintf(p, size, "%d", num) < size);
+	return (p);
+}
+
+char *
+VRT_double_string(const struct sess *sp, double num)
+{
+	char *p;
+	int size;
+
+	size = snprintf(NULL, 0, "%.3f", num) + 1;
+	AN(p = WS_Alloc(sp->http->ws, size));
+	assert(snprintf(p, size, "%.3f", num) < size);
+	return (p);
+}
+
+char *
+VRT_time_string(const struct sess *sp, double t)
+{
+	char *p;
+
+	AN(p = WS_Alloc(sp->http->ws, VTIM_FORMAT_SIZE));
+	VTIM_format(t, p);
+	return (p);
+}
+
+const char *
+VRT_backend_string(const struct sess *sp, const struct director *d)
+{
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	if (d == NULL)
+		d = sp->director;
+	if (d == NULL)
+		return (NULL);
+	return (d->vcl_name);
+}
+
+const char *
+VRT_bool_string(const struct sess *sp, unsigned val)
+{
+
+	(void)sp;
+	return (val ? "true" : "false");
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+VRT_Rollback(struct sess *sp)
+{
+
+	HTTP_Copy(sp->http, sp->http0);
+	WS_Reset(sp->ws, sp->ws_req);
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+VRT_panic(const struct sess *sp, const char *str, ...)
+{
+	va_list ap;
+	char *b;
+
+	va_start(ap, str);
+	b = VRT_String(sp->http->ws, "PANIC: ", str, ap);
+	va_end(ap);
+	VAS_Fail("VCL", "", 0, b, 0, 2);
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+VRT_synth_page(const struct sess *sp, unsigned flags, const char *str, ...)
+{
+	va_list ap;
+	const char *p;
+	struct vsb *vsb;
+
+	(void)flags;
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);
+	vsb = SMS_Makesynth(sp->obj);
+	AN(vsb);
+
+	VSB_cat(vsb, str);
+	va_start(ap, str);
+	p = va_arg(ap, const char *);
+	while (p != vrt_magic_string_end) {
+		if (p == NULL)
+			p = "(null)";
+		VSB_cat(vsb, p);
+		p = va_arg(ap, const char *);
+	}
+	va_end(ap);
+	SMS_Finish(sp->obj);
+	http_Unset(sp->obj->http, H_Content_Length);
+	http_PrintfHeader(sp->wrk, sp->vsl_id, sp->obj->http,
+	    "Content-Length: %d", sp->obj->len);
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+VRT_ban(struct sess *sp, char *cmds, ...)
+{
+	char *a1, *a2, *a3;
+	va_list ap;
+	struct ban *b;
+	int good;
+
+	(void)sp;
+	b = BAN_New();
+	va_start(ap, cmds);
+	a1 = cmds;
+	good = 0;
+	while (a1 != NULL) {
+		good = 0;
+		a2 = va_arg(ap, char *);
+		if (a2 == NULL)
+			break;
+		a3 = va_arg(ap, char *);
+		if (a3 == NULL)
+			break;
+		if (BAN_AddTest(NULL, b, a1, a2, a3))
+			break;
+		a1 = va_arg(ap, char *);
+		good = 1;
+	}
+	if (!good)
+		/* XXX: report error how ? */
+		BAN_Free(b);
+	else
+		BAN_Insert(b);
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+VRT_ban_string(struct sess *sp, const char *str)
+{
+	char *a1, *a2, *a3;
+	char **av;
+	struct ban *b;
+	int good;
+	int i;
+
+	(void)sp;
+	av = VAV_Parse(str, NULL, ARGV_NOESC);
+	if (av[0] != NULL) {
+		/* XXX: report error how ? */
+		VAV_Free(av);
+		return;
+	}
+	b = BAN_New();
+	good = 0;
+	for (i = 1; ;) {
+		a1 = av[i++];
+		if (a1 == NULL)
+			break;
+		good = 0;
+		a2 = av[i++];
+		if (a2 == NULL)
+			break;
+		a3 = av[i++];
+		if (a3 == NULL)
+			break;
+		if (BAN_AddTest(NULL, b, a1, a2, a3))
+			break;
+		good = 1;
+		if (av[i] == NULL)
+			break;
+		good = 0;
+		if (strcmp(av[i++], "&&"))
+			break;
+	}
+	if (!good)
+		/* XXX: report error how ? */
+		BAN_Free(b);
+	else
+		BAN_Insert(b);
+	VAV_Free(av);
+}
+
+/*--------------------------------------------------------------------
+ * "real" purges
+ */
+
+void
+VRT_purge(const struct sess *sp, double ttl, double grace)
+{
+	if (sp->cur_method == VCL_MET_HIT)
+		HSH_Purge(sp, sp->obj->objcore->objhead, ttl, grace);
+	else if (sp->cur_method == VCL_MET_MISS)
+		HSH_Purge(sp, sp->objcore->objhead, ttl, grace);
+}
+
+/*--------------------------------------------------------------------
+ * Simple stuff
+ */
+
+int
+VRT_strcmp(const char *s1, const char *s2)
+{
+	if (s1 == NULL || s2 == NULL)
+		return(1);
+	return (strcmp(s1, s2));
+}
+
+void
+VRT_memmove(void *dst, const void *src, unsigned len)
+{
+
+	(void)memmove(dst, src, len);
+}
diff --git a/bin/varnishd/cache/cache_vrt_re.c b/bin/varnishd/cache/cache_vrt_re.c
new file mode 100644
index 0000000..7759d0a
--- /dev/null
+++ b/bin/varnishd/cache/cache_vrt_re.c
@@ -0,0 +1,162 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2009 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * Runtime support for compiled VCL programs, regexps
+ */
+
+#include "config.h"
+
+#include <ctype.h>
+
+#include "cache.h"
+
+#include "vre.h"
+#include "vrt.h"
+
+void
+VRT_re_init(void **rep, const char *re)
+{
+	vre_t *t;
+	const char *error;
+	int erroroffset;
+
+	/* This was already check-compiled by the VCL compiler */
+	t = VRE_compile(re, 0, &error, &erroroffset);
+	AN(t);
+	*rep = t;
+}
+
+void
+VRT_re_fini(void *rep)
+{
+	vre_t *vv;
+
+	vv = rep;
+	if (rep != NULL)
+		VRE_free(&vv);
+}
+
+int
+VRT_re_match(const struct sess *sp, const char *s, void *re)
+{
+	vre_t *t;
+	int i;
+
+	if (s == NULL)
+		s = "";
+	AN(re);
+	t = re;
+	i = VRE_exec(t, s, strlen(s), 0, 0, NULL, 0, &cache_param->vre_limits);
+	if (i >= 0)
+		return (1);
+	if (i < VRE_ERROR_NOMATCH )
+		WSP(sp, SLT_VCL_Error, "Regexp matching returned %d", i);
+	return (0);
+}
+
+const char *
+VRT_regsub(const struct sess *sp, int all, const char *str, void *re,
+    const char *sub)
+{
+	int ovector[30];
+	vre_t *t;
+	int i, l;
+	txt res;
+	char *b0;
+	const char *s;
+	unsigned u, x;
+	int options = 0;
+	size_t len;
+
+	AN(re);
+	if (str == NULL)
+		str = "";
+	t = re;
+	memset(ovector, 0, sizeof(ovector));
+	len = strlen(str);
+	i = VRE_exec(t, str, len, 0, options, ovector, 30,
+	    &cache_param->vre_limits);
+
+	/* If it didn't match, we can return the original string */
+	if (i == VRE_ERROR_NOMATCH)
+		return(str);
+	if (i < VRE_ERROR_NOMATCH ) {
+		WSP(sp, SLT_VCL_Error, "Regexp matching returned %d", i);
+		return(str);
+	}
+
+	u = WS_Reserve(sp->http->ws, 0);
+	res.e = res.b = b0 = sp->http->ws->f;
+	res.e += u;
+
+	do {
+		/* Copy prefix to match */
+		Tadd(&res, str, ovector[0]);
+		for (s = sub ; *s != '\0'; s++ ) {
+			if (*s != '\\' || s[1] == '\0') {
+				if (res.b < res.e)
+					*res.b++ = *s;
+				continue;
+			}
+			s++;
+			if (isdigit(*s)) {
+				x = *s - '0';
+				l = ovector[2*x+1] - ovector[2*x];
+				Tadd(&res, str + ovector[2*x], l);
+				continue;
+			} else {
+				if (res.b < res.e)
+					*res.b++ = *s;
+			}
+		}
+		str += ovector[1];
+		len -= ovector[1];
+		if (!all)
+			break;
+		memset(&ovector, 0, sizeof(ovector));
+		options |= VRE_NOTEMPTY_ATSTART;
+		i = VRE_exec(t, str, len, 0, options, ovector, 30,
+		    &cache_param->vre_limits);
+		if (i < VRE_ERROR_NOMATCH ) {
+			WS_Release(sp->http->ws, 0);
+			WSP(sp, SLT_VCL_Error,
+			    "Regexp matching returned %d", i);
+			return(str);
+		}
+	} while (i != VRE_ERROR_NOMATCH);
+
+	/* Copy suffix to match */
+	Tadd(&res, str, len+1);
+	if (res.b >= res.e) {
+		WS_Release(sp->http->ws, 0);
+		return (str);
+	}
+	Tcheck(res);
+	WS_ReleaseP(sp->http->ws, res.b);
+	return (b0);
+}
diff --git a/bin/varnishd/cache/cache_vrt_var.c b/bin/varnishd/cache/cache_vrt_var.c
new file mode 100644
index 0000000..860c7aa
--- /dev/null
+++ b/bin/varnishd/cache/cache_vrt_var.c
@@ -0,0 +1,550 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * Runtime support for compiled VCL programs
+ */
+#include "config.h"
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#include "cache.h"
+#include "common/heritage.h"
+
+#include "cache_backend.h"
+#include "vrt_obj.h"
+#include "vtcp.h"
+#include "vtim.h"
+
+static char vrt_hostname[255] = "";
+
+/*--------------------------------------------------------------------*/
+
+static void
+vrt_do_string(struct worker *w, int fd, const struct http *hp, int fld,
+    const char *err, const char *p, va_list ap)
+{
+	char *b;
+
+	// AN(p);
+	AN(hp);
+	b = VRT_String(hp->ws, NULL, p, ap);
+	if (b == NULL || *b == '\0') {
+		WSL(w, SLT_LostHeader, fd, err);
+	} else {
+		http_SetH(hp, fld, b);
+	}
+	va_end(ap);
+}
+
+#define VRT_DO_HDR(obj, hdr, http, fld)				\
+void								\
+VRT_l_##obj##_##hdr(const struct sess *sp, const char *p, ...)	\
+{								\
+	va_list ap;						\
+								\
+	va_start(ap, p);					\
+	vrt_do_string(sp->wrk, sp->fd,				\
+	    http, fld, #obj "." #hdr, p, ap);			\
+	va_end(ap);						\
+}								\
+								\
+const char *							\
+VRT_r_##obj##_##hdr(const struct sess *sp)			\
+{								\
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);			\
+	CHECK_OBJ_NOTNULL(http, HTTP_MAGIC);			\
+	return (http->hd[fld].b);				\
+}
+
+VRT_DO_HDR(req,   request,	sp->http,		HTTP_HDR_REQ)
+VRT_DO_HDR(req,   url,		sp->http,		HTTP_HDR_URL)
+VRT_DO_HDR(req,   proto,	sp->http,		HTTP_HDR_PROTO)
+VRT_DO_HDR(bereq, request,	sp->wrk->bereq,		HTTP_HDR_REQ)
+VRT_DO_HDR(bereq, url,		sp->wrk->bereq,		HTTP_HDR_URL)
+VRT_DO_HDR(bereq, proto,	sp->wrk->bereq,		HTTP_HDR_PROTO)
+VRT_DO_HDR(obj,   proto,	sp->obj->http,		HTTP_HDR_PROTO)
+VRT_DO_HDR(obj,   response,	sp->obj->http,		HTTP_HDR_RESPONSE)
+VRT_DO_HDR(resp,  proto,	sp->wrk->resp,		HTTP_HDR_PROTO)
+VRT_DO_HDR(resp,  response,	sp->wrk->resp,		HTTP_HDR_RESPONSE)
+VRT_DO_HDR(beresp,  proto,	sp->wrk->beresp,	HTTP_HDR_PROTO)
+VRT_DO_HDR(beresp,  response,	sp->wrk->beresp,	HTTP_HDR_RESPONSE)
+
+/*--------------------------------------------------------------------*/
+
+#define VRT_DO_STATUS(obj, http)				\
+void								\
+VRT_l_##obj##_status(const struct sess *sp, int num)		\
+{								\
+								\
+	assert(num >= 100 && num <= 999);			\
+	http->status = (uint16_t)num;				\
+}								\
+								\
+int								\
+VRT_r_##obj##_status(const struct sess *sp)			\
+{								\
+								\
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);			\
+	return(http->status);					\
+}
+
+VRT_DO_STATUS(obj, sp->obj->http)
+VRT_DO_STATUS(beresp, sp->wrk->beresp)
+VRT_DO_STATUS(resp, sp->wrk->resp)
+
+/*--------------------------------------------------------------------*/
+
+/* XXX: review this */
+/* Add an objecthead to the saintmode list for the (hopefully) relevant
+ * backend. Some double-up asserting here to avoid assert-errors when there
+ * is no object.
+ */
+void
+VRT_l_beresp_saintmode(const struct sess *sp, double a)
+{
+	struct trouble *new;
+	struct trouble *tr;
+	struct trouble *tr2;
+	struct worker *wrk;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	wrk = sp->wrk;
+	if (!wrk->vbc)
+		return;
+	CHECK_OBJ_NOTNULL(wrk->vbc, VBC_MAGIC);
+	if (!wrk->vbc->backend)
+		return;
+	CHECK_OBJ_NOTNULL(wrk->vbc->backend, BACKEND_MAGIC);
+	if (!sp->objcore)
+		return;
+	CHECK_OBJ_NOTNULL(sp->objcore, OBJCORE_MAGIC);
+
+	/* Setting a negative holdoff period is a mistake. Detecting this
+	 * when compiling the VCL would be better.
+	 */
+	assert(a > 0);
+
+	ALLOC_OBJ(new, TROUBLE_MAGIC);
+	AN(new);
+	new->target = (uintptr_t)(sp->objcore->objhead);
+	new->timeout = sp->t_req + a;
+
+	/* Insert the new item on the list before the first item with a
+	 * timeout at a later date (ie: sort by which entry will time out
+	 * from the list
+	 */
+	Lck_Lock(&wrk->vbc->backend->mtx);
+	VTAILQ_FOREACH_SAFE(tr, &wrk->vbc->backend->troublelist, list, tr2) {
+		if (tr->timeout < new->timeout) {
+			VTAILQ_INSERT_BEFORE(tr, new, list);
+			new = NULL;
+			break;
+		}
+	}
+
+	/* Insert the item at the end if the list is empty or all other
+	 * items have a longer timeout.
+	 */
+	if (new)
+		VTAILQ_INSERT_TAIL(&wrk->vbc->backend->troublelist, new, list);
+
+	Lck_Unlock(&wrk->vbc->backend->mtx);
+}
+
+/*--------------------------------------------------------------------*/
+
+#define VBERESP(dir, type, onm, field)					\
+void									\
+VRT_l_##dir##_##onm(const struct sess *sp, type a)			\
+{									\
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);				\
+	sp->wrk->field = a;						\
+}									\
+									\
+type									\
+VRT_r_##dir##_##onm(const struct sess *sp)				\
+{									\
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);				\
+	return (sp->wrk->field);					\
+}
+
+VBERESP(beresp, unsigned, do_esi, do_esi)
+VBERESP(beresp, unsigned, do_gzip, do_gzip)
+VBERESP(beresp, unsigned, do_gunzip, do_gunzip)
+VBERESP(beresp, unsigned, do_stream, do_stream)
+
+/*--------------------------------------------------------------------*/
+
+const char * __match_proto__()
+VRT_r_client_identity(struct sess *sp)
+{
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	if (sp->client_identity != NULL)
+		return (sp->client_identity);
+	else
+		return (sp->addr);
+}
+
+void
+VRT_l_client_identity(struct sess *sp, const char *str, ...)
+{
+	va_list ap;
+	char *b;
+
+	va_start(ap, str);
+	b = VRT_String(sp->http->ws, NULL, str, ap);
+	va_end(ap);
+	sp->client_identity = b;
+}
+
+/*--------------------------------------------------------------------*/
+
+#define BEREQ_TIMEOUT(which)					\
+void __match_proto__()						\
+VRT_l_bereq_##which(struct sess *sp, double num)		\
+{								\
+								\
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);			\
+	sp->wrk->which = (num > 0.0 ? num : 0.0);		\
+}								\
+								\
+double __match_proto__()					\
+VRT_r_bereq_##which(struct sess *sp)				\
+{								\
+								\
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);			\
+	return(sp->wrk->which);					\
+}
+
+BEREQ_TIMEOUT(connect_timeout)
+BEREQ_TIMEOUT(first_byte_timeout)
+BEREQ_TIMEOUT(between_bytes_timeout)
+
+/*--------------------------------------------------------------------*/
+
+const char *
+VRT_r_beresp_backend_name(const struct sess *sp)
+{
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->wrk->vbc, VBC_MAGIC);
+	return(sp->wrk->vbc->backend->vcl_name);
+}
+
+struct sockaddr_storage *
+VRT_r_beresp_backend_ip(const struct sess *sp)
+{
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->wrk->vbc, VBC_MAGIC);
+	return(sp->wrk->vbc->addr);
+}
+
+int
+VRT_r_beresp_backend_port(const struct sess *sp)
+{
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->wrk->vbc, VBC_MAGIC);
+	return (VTCP_port(sp->wrk->vbc->addr));
+}
+
+const char * __match_proto__()
+VRT_r_beresp_storage(struct sess *sp)
+{
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	if (sp->wrk->storage_hint != NULL)
+		return (sp->wrk->storage_hint);
+	else
+		return (NULL);
+}
+
+void __match_proto__()
+VRT_l_beresp_storage(struct sess *sp, const char *str, ...)
+{
+	va_list ap;
+	char *b;
+
+	va_start(ap, str);
+	b = VRT_String(sp->wrk->ws, NULL, str, ap);
+	va_end(ap);
+	sp->wrk->storage_hint = b;
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+VRT_l_req_backend(struct sess *sp, struct director *be)
+{
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	sp->director = be;
+}
+
+struct director * __match_proto__()
+VRT_r_req_backend(struct sess *sp)
+{
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	return (sp->director);
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+VRT_l_req_esi(struct sess *sp, unsigned process_esi)
+{
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	/*
+	 * Only allow you to turn of esi in the main request
+	 * else everything gets confused
+	 */
+	if(sp->esi_level == 0)
+		sp->disable_esi = !process_esi;
+}
+
+unsigned __match_proto__()
+VRT_r_req_esi(struct sess *sp)
+{
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	return (!sp->disable_esi);
+}
+
+int
+VRT_r_req_esi_level(const struct sess *sp)
+{
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	return(sp->esi_level);
+}
+
+/*--------------------------------------------------------------------*/
+
+unsigned __match_proto__()
+VRT_r_req_can_gzip(struct sess *sp)
+{
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	return (RFC2616_Req_Gzip(sp));
+}
+
+
+/*--------------------------------------------------------------------*/
+
+int
+VRT_r_req_restarts(const struct sess *sp)
+{
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	return (sp->restarts);
+}
+
+/*--------------------------------------------------------------------
+ * NB: TTL is relative to when object was created, whereas grace and
+ * keep are relative to ttl.
+ */
+
+#define VRT_DO_EXP(which, exp, fld, offset, extra)		\
+								\
+void __match_proto__()						\
+VRT_l_##which##_##fld(struct sess *sp, double a)		\
+{								\
+								\
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);			\
+	if (a > 0.)						\
+		a += offset;					\
+	EXP_Set_##fld(&exp, a);					\
+	extra;							\
+}								\
+								\
+double __match_proto__()					\
+VRT_r_##which##_##fld(struct sess *sp)				\
+{								\
+								\
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);			\
+	return(EXP_Get_##fld(&exp) - offset);			\
+}
+
+static void
+vrt_wsp_exp(const struct sess *sp, unsigned xid, const struct exp *e)
+{
+	WSP(sp, SLT_TTL, "%u VCL %.0f %.0f %.0f %.0f %.0f",
+	    xid, e->ttl - (sp->t_req - e->entered), e->grace, e->keep,
+	    sp->t_req, e->age + (sp->t_req - e->entered));
+}
+
+VRT_DO_EXP(req, sp->exp, ttl, 0, )
+VRT_DO_EXP(req, sp->exp, grace, 0, )
+VRT_DO_EXP(req, sp->exp, keep, 0, )
+
+VRT_DO_EXP(obj, sp->obj->exp, grace, 0,
+   EXP_Rearm(sp->obj);
+   vrt_wsp_exp(sp, sp->obj->xid, &sp->obj->exp);)
+VRT_DO_EXP(obj, sp->obj->exp, ttl, (sp->t_req - sp->obj->exp.entered),
+   EXP_Rearm(sp->obj);
+   vrt_wsp_exp(sp, sp->obj->xid, &sp->obj->exp);)
+VRT_DO_EXP(obj, sp->obj->exp, keep, 0,
+   EXP_Rearm(sp->obj);
+   vrt_wsp_exp(sp, sp->obj->xid, &sp->obj->exp);)
+
+VRT_DO_EXP(beresp, sp->wrk->exp, grace, 0,
+   vrt_wsp_exp(sp, sp->xid, &sp->wrk->exp);)
+VRT_DO_EXP(beresp, sp->wrk->exp, ttl, 0,
+   vrt_wsp_exp(sp, sp->xid, &sp->wrk->exp);)
+VRT_DO_EXP(beresp, sp->wrk->exp, keep, 0,
+   vrt_wsp_exp(sp, sp->xid, &sp->wrk->exp);)
+
+/*--------------------------------------------------------------------
+ * req.xid
+ */
+
+const char * __match_proto__()
+VRT_r_req_xid(struct sess *sp)
+{
+	char *p;
+	int size;
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+
+	size = snprintf(NULL, 0, "%u", sp->xid) + 1;
+	AN(p = WS_Alloc(sp->http->ws, size));
+	assert(snprintf(p, size, "%u", sp->xid) < size);
+	return (p);
+}
+
+/*--------------------------------------------------------------------*/
+
+#define REQ_BOOL(which)						\
+void __match_proto__()						\
+VRT_l_req_##which(struct sess *sp, unsigned val)		\
+{								\
+								\
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);			\
+	sp->which = val ? 1 : 0;				\
+}								\
+								\
+unsigned __match_proto__()					\
+VRT_r_req_##which(struct sess *sp)				\
+{								\
+								\
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);			\
+	return(sp->which);					\
+}
+
+REQ_BOOL(hash_ignore_busy)
+REQ_BOOL(hash_always_miss)
+
+/*--------------------------------------------------------------------*/
+
+struct sockaddr_storage *
+VRT_r_client_ip(struct sess *sp)
+{
+
+	return (&sp->sockaddr);
+}
+
+struct sockaddr_storage *
+VRT_r_server_ip(struct sess *sp)
+{
+	int i;
+
+	if (sp->mysockaddr.ss_family == AF_UNSPEC) {
+		i = getsockname(sp->fd,
+		    (void*)&sp->mysockaddr, &sp->mysockaddrlen);
+		assert(VTCP_Check(i));
+	}
+
+	return (&sp->mysockaddr);
+}
+
+const char*
+VRT_r_server_identity(struct sess *sp)
+{
+	(void)sp;
+
+	if (heritage.identity[0] != '\0')
+		return (heritage.identity);
+	else
+		return (heritage.name);
+}
+
+
+const char*
+VRT_r_server_hostname(struct sess *sp)
+{
+	(void)sp;
+
+	if (vrt_hostname[0] == '\0')
+		AZ(gethostname(vrt_hostname, sizeof(vrt_hostname)));
+
+	return (vrt_hostname);
+}
+
+/*--------------------------------------------------------------------
+ * XXX: This is pessimistically silly
+ */
+
+int
+VRT_r_server_port(struct sess *sp)
+{
+	int i;
+
+	if (sp->mysockaddr.ss_family == AF_UNSPEC) {
+		i = getsockname(sp->fd,
+		    (void*)&sp->mysockaddr, &sp->mysockaddrlen);
+		assert(VTCP_Check(i));
+	}
+	return (VTCP_port(&sp->mysockaddr));
+}
+
+/*--------------------------------------------------------------------*/
+
+int
+VRT_r_obj_hits(const struct sess *sp)
+{
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);	/* XXX */
+	return (sp->obj->hits);
+}
+
+double
+VRT_r_obj_lastuse(const struct sess *sp)
+{
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);	/* XXX */
+	return (VTIM_real() - sp->obj->last_use);
+}
+
+unsigned
+VRT_r_req_backend_healthy(const struct sess *sp)
+{
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->director, DIRECTOR_MAGIC);
+	return (VDI_Healthy(sp->director, sp));
+}
+
diff --git a/bin/varnishd/cache/cache_vrt_vmod.c b/bin/varnishd/cache/cache_vrt_vmod.c
new file mode 100644
index 0000000..6b3b846
--- /dev/null
+++ b/bin/varnishd/cache/cache_vrt_vmod.c
@@ -0,0 +1,180 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * Runtime support for compiled VCL programs
+ */
+
+#include "config.h"
+
+#include <dlfcn.h>
+#include <stdlib.h>
+
+#include "cache.h"
+
+#include "vcli_priv.h"
+#include "vrt.h"
+
+/*--------------------------------------------------------------------
+ * Modules stuff
+ */
+
+struct vmod {
+	unsigned		magic;
+#define VMOD_MAGIC		0xb750219c
+
+	VTAILQ_ENTRY(vmod)	list;
+
+	int			ref;
+
+	char			*nm;
+	char			*path;
+	void			*hdl;
+	const void		*funcs;
+	int			funclen;
+	const void		*idptr;
+};
+
+static VTAILQ_HEAD(,vmod)	vmods = VTAILQ_HEAD_INITIALIZER(vmods);
+
+int
+VRT_Vmod_Init(void **hdl, void *ptr, int len, const char *nm,
+    const char *path, struct cli *cli)
+{
+	struct vmod *v;
+	void *x, *y, *z, *w;
+
+	ASSERT_CLI();
+
+	VTAILQ_FOREACH(v, &vmods, list)
+		if (!strcmp(v->nm, nm))	// Also path, len ?
+			break;
+	if (v == NULL) {
+		ALLOC_OBJ(v, VMOD_MAGIC);
+		AN(v);
+
+		v->hdl = dlopen(path, RTLD_NOW | RTLD_LOCAL);
+		if (v->hdl == NULL) {
+			VCLI_Out(cli, "Loading VMOD %s from %s:\n", nm, path);
+			VCLI_Out(cli, "dlopen() failed: %s\n", dlerror());
+			VCLI_Out(cli, "Check child process permissions.\n");
+			FREE_OBJ(v);
+			return (1);
+		}
+
+		x = dlsym(v->hdl, "Vmod_Name");
+		y = dlsym(v->hdl, "Vmod_Len");
+		z = dlsym(v->hdl, "Vmod_Func");
+		w = dlsym(v->hdl, "Vmod_Id");
+		if (x == NULL || y == NULL || z == NULL || w == NULL) {
+			VCLI_Out(cli, "Loading VMOD %s from %s:\n", nm, path);
+			VCLI_Out(cli, "VMOD symbols not found\n");
+			VCLI_Out(cli, "Check relative pathnames.\n");
+			(void)dlclose(v->hdl);
+			FREE_OBJ(v);
+			return (1);
+		}
+		AN(x);
+		AN(y);
+		AN(z);
+		AN(w);
+		if (strcmp(x, nm)) {
+			VCLI_Out(cli, "Loading VMOD %s from %s:\n", nm, path);
+			VCLI_Out(cli, "File contain wrong VMOD (\"%s\")\n", x);
+			VCLI_Out(cli, "Check relative pathnames ?.\n");
+			(void)dlclose(v->hdl);
+			FREE_OBJ(v);
+			return (1);
+		}
+
+		v->funclen = *(const int *)y;
+		v->funcs = z;
+
+		REPLACE(v->nm, nm);
+		REPLACE(v->path, path);
+
+		VSC_C_main->vmods++;
+		VTAILQ_INSERT_TAIL(&vmods, v, list);
+		v->idptr = w;
+	}
+
+	assert(len == v->funclen);
+	memcpy(ptr, v->funcs, v->funclen);
+	v->ref++;
+
+	*hdl = v;
+	return (0);
+}
+
+void
+VRT_Vmod_Fini(void **hdl)
+{
+	struct vmod *v;
+
+	ASSERT_CLI();
+
+	AN(*hdl);
+	CAST_OBJ_NOTNULL(v, *hdl, VMOD_MAGIC);
+	*hdl = NULL;
+	if (--v->ref != 0)
+		return;
+#ifndef DONT_DLCLOSE_VMODS
+	AZ(dlclose(v->hdl));
+#endif
+	free(v->nm);
+	free(v->path);
+	VTAILQ_REMOVE(&vmods, v, list);
+	VSC_C_main->vmods--;
+	FREE_OBJ(v);
+}
+
+/*---------------------------------------------------------------------*/
+
+static void
+ccf_debug_vmod(struct cli *cli, const char * const *av, void *priv)
+{
+	struct vmod *v;
+
+	(void)av;
+	(void)priv;
+	ASSERT_CLI();
+	VTAILQ_FOREACH(v, &vmods, list)
+		VCLI_Out(cli, "%5d %s (%s)\n", v->ref, v->nm, v->path);
+}
+
+static struct cli_proto vcl_cmds[] = {
+	{ "debug.vmod", "debug.vmod", "show loaded vmods", 0, 0,
+		"d", ccf_debug_vmod },
+	{ NULL }
+};
+
+void
+VMOD_Init(void)
+{
+
+	CLI_AddFuncs(vcl_cmds);
+}
diff --git a/bin/varnishd/cache/cache_wrk.c b/bin/varnishd/cache/cache_wrk.c
new file mode 100644
index 0000000..bfee84c
--- /dev/null
+++ b/bin/varnishd/cache/cache_wrk.c
@@ -0,0 +1,204 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * Worker thread stuff unrelated to the worker thread pools.
+ */
+
+#include "config.h"
+
+#include <math.h>
+#include <stdlib.h>
+
+#include "cache.h"
+
+#include "hash/hash_slinger.h"
+#include "vsha256.h"
+
+static struct lock		wstat_mtx;
+
+/*--------------------------------------------------------------------*/
+
+static void
+wrk_sumstat(struct worker *w)
+{
+
+	Lck_AssertHeld(&wstat_mtx);
+#define L0(n)
+#define L1(n) (VSC_C_main->n += w->stats.n)
+#define VSC_DO_MAIN
+#define VSC_F(n, t, l, f, d, e) L##l(n);
+#include "tbl/vsc_fields.h"
+#undef VSC_F
+#undef VSC_DO_MAIN
+#undef L0
+#undef L1
+	memset(&w->stats, 0, sizeof w->stats);
+}
+
+void
+WRK_SumStat(struct worker *w)
+{
+
+	Lck_Lock(&wstat_mtx);
+	wrk_sumstat(w);
+	Lck_Unlock(&wstat_mtx);
+}
+
+int
+WRK_TrySumStat(struct worker *w)
+{
+	if (Lck_Trylock(&wstat_mtx))
+		return (0);
+	wrk_sumstat(w);
+	Lck_Unlock(&wstat_mtx);
+	return (1);
+}
+
+/*--------------------------------------------------------------------
+ * Create and starte a back-ground thread which as its own worker and
+ * session data structures;
+ */
+
+struct bgthread {
+	unsigned	magic;
+#define BGTHREAD_MAGIC	0x23b5152b
+	const char	*name;
+	bgthread_t	*func;
+	void		*priv;
+};
+
+static void *
+wrk_bgthread(void *arg)
+{
+	struct bgthread *bt;
+	struct worker ww;
+	struct sess *sp;
+	uint32_t logbuf[1024];	/* XXX:  size ? */
+
+	CAST_OBJ_NOTNULL(bt, arg, BGTHREAD_MAGIC);
+	THR_SetName(bt->name);
+	sp = SES_Alloc();
+	XXXAN(sp);
+	memset(&ww, 0, sizeof ww);
+	sp->wrk = &ww;
+	ww.magic = WORKER_MAGIC;
+	ww.wlp = ww.wlb = logbuf;
+	ww.wle = logbuf + (sizeof logbuf) / 4;
+
+	(void)bt->func(sp, bt->priv);
+
+	WRONG("BgThread terminated");
+
+	NEEDLESS_RETURN(NULL);
+}
+
+void
+WRK_BgThread(pthread_t *thr, const char *name, bgthread_t *func, void *priv)
+{
+	struct bgthread *bt;
+
+	ALLOC_OBJ(bt, BGTHREAD_MAGIC);
+	AN(bt);
+
+	bt->name = name;
+	bt->func = func;
+	bt->priv = priv;
+	AZ(pthread_create(thr, NULL, wrk_bgthread, bt));
+}
+
+/*--------------------------------------------------------------------*/
+
+static void *
+wrk_thread_real(void *priv, unsigned shm_workspace, unsigned sess_workspace,
+    uint16_t nhttp, unsigned http_space, unsigned siov)
+{
+	struct worker *w, ww;
+	uint32_t wlog[shm_workspace / 4];
+	/* XXX: can we trust these to be properly aligned ? */
+	unsigned char ws[sess_workspace];
+	unsigned char http0[http_space];
+	unsigned char http1[http_space];
+	unsigned char http2[http_space];
+	struct iovec iov[siov];
+	struct SHA256Context sha256;
+
+	THR_SetName("cache-worker");
+	w = &ww;
+	memset(w, 0, sizeof *w);
+	w->magic = WORKER_MAGIC;
+	w->lastused = NAN;
+	w->wlb = w->wlp = wlog;
+	w->wle = wlog + (sizeof wlog) / 4;
+	w->sha256ctx = &sha256;
+	w->bereq = HTTP_create(http0, nhttp);
+	w->beresp = HTTP_create(http1, nhttp);
+	w->resp = HTTP_create(http2, nhttp);
+	w->wrw.iov = iov;
+	w->wrw.siov = siov;
+	w->wrw.ciov = siov;
+	AZ(pthread_cond_init(&w->cond, NULL));
+
+	WS_Init(w->ws, "wrk", ws, sess_workspace);
+
+	VSL(SLT_WorkThread, 0, "%p start", w);
+
+	Pool_Work_Thread(priv, w);
+	AZ(w->pool);
+
+	VSL(SLT_WorkThread, 0, "%p end", w);
+	if (w->vcl != NULL)
+		VCL_Rel(&w->vcl);
+	AZ(pthread_cond_destroy(&w->cond));
+	HSH_Cleanup(w);
+	WRK_SumStat(w);
+	return (NULL);
+}
+
+void *
+WRK_thread(void *priv)
+{
+	uint16_t nhttp;
+	unsigned siov;
+
+	assert(cache_param->http_max_hdr <= 65535);
+	/* We need to snapshot these two for consistency */
+	nhttp = (uint16_t)cache_param->http_max_hdr;
+	siov = nhttp * 2;
+	if (siov > IOV_MAX)
+		siov = IOV_MAX;
+	return (wrk_thread_real(priv,
+	    cache_param->shm_workspace,
+	    cache_param->wthread_workspace,
+	    nhttp, HTTP_estimate(nhttp), siov));
+}
+
+void
+WRK_Init(void)
+{
+	Lck_New(&wstat_mtx, lck_wstat);
+}
diff --git a/bin/varnishd/cache/cache_wrw.c b/bin/varnishd/cache/cache_wrw.c
new file mode 100644
index 0000000..2160f69
--- /dev/null
+++ b/bin/varnishd/cache/cache_wrw.c
@@ -0,0 +1,344 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * Write data to fd
+ * We try to use writev() if possible in order to minimize number of
+ * syscalls made and packets sent.  It also just might allow the worker
+ * thread to complete the request without holding stuff locked.
+ */
+
+#include "config.h"
+
+#include <sys/types.h>
+#ifdef SENDFILE_WORKS
+#  if defined(__FreeBSD__) || defined(__DragonFly__)
+#    include <sys/socket.h>
+#  elif defined(__linux__)
+#    include <sys/sendfile.h>
+#  elif defined(__sun)
+#    include <sys/sendfile.h>
+#  else
+#     error Unknown sendfile() implementation
+#  endif
+#endif /* SENDFILE_WORKS */
+#include <sys/uio.h>
+
+#include <stdio.h>
+
+#include "cache.h"
+#include "vtim.h"
+
+/*--------------------------------------------------------------------
+ */
+
+int
+WRW_Error(const struct worker *w)
+{
+
+	return (w->wrw.werr);
+}
+
+void
+WRW_Reserve(struct worker *w, int *fd)
+{
+	struct wrw *wrw;
+
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	wrw = &w->wrw;
+	AZ(wrw->wfd);
+	wrw->werr = 0;
+	wrw->liov = 0;
+	wrw->niov = 0;
+	wrw->ciov = wrw->siov;
+	wrw->wfd = fd;
+}
+
+static void
+WRW_Release(struct worker *w)
+{
+	struct wrw *wrw;
+
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	wrw = &w->wrw;
+	AN(wrw->wfd);
+	wrw->werr = 0;
+	wrw->liov = 0;
+	wrw->niov = 0;
+	wrw->ciov = wrw->siov;
+	wrw->wfd = NULL;
+}
+
+unsigned
+WRW_Flush(struct worker *w)
+{
+	ssize_t i;
+	struct wrw *wrw;
+	char cbuf[32];
+
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	wrw = &w->wrw;
+	AN(wrw->wfd);
+
+	/* For chunked, there must be one slot reserved for the chunked tail */
+	if (wrw->ciov < wrw->siov)
+		assert(wrw->niov < wrw->siov);
+
+	if (*wrw->wfd >= 0 && wrw->liov > 0 && wrw->werr == 0) {
+		if (wrw->ciov < wrw->siov && wrw->cliov > 0) {
+			bprintf(cbuf, "00%jx\r\n", (intmax_t)wrw->cliov);
+			i = strlen(cbuf);
+			wrw->iov[wrw->ciov].iov_base = cbuf;
+			wrw->iov[wrw->ciov].iov_len = i;
+			wrw->liov += i;
+
+			wrw->iov[wrw->niov].iov_base = cbuf + i - 2;
+			wrw->iov[wrw->niov++].iov_len = 2;
+			wrw->liov += 2;
+		} else if (wrw->ciov < wrw->siov) {
+			wrw->iov[wrw->ciov].iov_base = cbuf;
+			wrw->iov[wrw->ciov].iov_len = 0;
+		}
+		i = writev(*wrw->wfd, wrw->iov, wrw->niov);
+		while (i != wrw->liov && i > 0) {
+			/* Remove sent data from start of I/O vector,
+			 * then retry; we hit a timeout, but some data
+			 * was sent.
+
+			 XXX: Add a "minimum sent data per timeout
+			 counter to prevent slowlaris attacks
+			*/
+			size_t used = 0;
+
+			if (VTIM_real() - w->sp->t_resp > cache_param->send_timeout) {
+				WSL(w, SLT_Debug, *wrw->wfd,
+				    "Hit total send timeout, wrote = %ld/%ld; not retrying",
+				    i, wrw->liov);
+				i = -1;
+				break;
+			}
+
+			WSL(w, SLT_Debug, *wrw->wfd,
+			    "Hit send timeout, wrote = %ld/%ld; retrying",
+			    i, wrw->liov);
+
+			for (int j = 0; j < wrw->niov; j++) {
+				if (used + wrw->iov[j].iov_len > i) {
+					/* Cutoff is in this iov */
+					int used_here = i - used;
+					wrw->iov[j].iov_len -= used_here;
+					wrw->iov[j].iov_base = (char*)wrw->iov[j].iov_base + used_here;
+					memmove(wrw->iov, &wrw->iov[j],
+						(wrw->niov - j) * sizeof(struct iovec));
+					wrw->niov -= j;
+					wrw->liov -= i;
+					break;
+				}
+				used += wrw->iov[j].iov_len;
+			}
+			i = writev(*wrw->wfd, wrw->iov, wrw->niov);
+		}
+		if (i <= 0) {
+			wrw->werr++;
+			WSL(w, SLT_Debug, *wrw->wfd,
+			    "Write error, retval = %d, len = %d, errno = %s",
+			    i, wrw->liov, strerror(errno));
+		}
+	}
+	wrw->liov = 0;
+	wrw->cliov = 0;
+	wrw->niov = 0;
+	if (wrw->ciov < wrw->siov)
+		wrw->ciov = wrw->niov++;
+	return (wrw->werr);
+}
+
+unsigned
+WRW_FlushRelease(struct worker *w)
+{
+	unsigned u;
+
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	AN(w->wrw.wfd);
+	u = WRW_Flush(w);
+	WRW_Release(w);
+	return (u);
+}
+
+unsigned
+WRW_WriteH(struct worker *w, const txt *hh, const char *suf)
+{
+	unsigned u;
+
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	AN(w->wrw.wfd);
+	AN(w);
+	AN(hh);
+	AN(hh->b);
+	AN(hh->e);
+	u = WRW_Write(w, hh->b, hh->e - hh->b);
+	if (suf != NULL)
+		u += WRW_Write(w, suf, -1);
+	return (u);
+}
+
+unsigned
+WRW_Write(struct worker *w, const void *ptr, int len)
+{
+	struct wrw *wrw;
+
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	wrw = &w->wrw;
+	AN(wrw->wfd);
+	if (len == 0 || *wrw->wfd < 0)
+		return (0);
+	if (len == -1)
+		len = strlen(ptr);
+	if (wrw->niov >= wrw->siov - (wrw->ciov < wrw->siov ? 1 : 0))
+		(void)WRW_Flush(w);
+	wrw->iov[wrw->niov].iov_base = TRUST_ME(ptr);
+	wrw->iov[wrw->niov].iov_len = len;
+	wrw->liov += len;
+	wrw->niov++;
+	if (wrw->ciov < wrw->siov) {
+		assert(wrw->niov < wrw->siov);
+		wrw->cliov += len;
+	}
+	return (len);
+}
+
+void
+WRW_Chunked(struct worker *w)
+{
+	struct wrw *wrw;
+
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	wrw = &w->wrw;
+
+	assert(wrw->ciov == wrw->siov);
+	/*
+	 * If there are not space for chunked header, a chunk of data and
+	 * a chunk tail, we might as well flush right away.
+	 */
+	if (wrw->niov + 3 >= wrw->siov)
+		(void)WRW_Flush(w);
+	wrw->ciov = wrw->niov++;
+	wrw->cliov = 0;
+	assert(wrw->ciov < wrw->siov);
+	assert(wrw->niov < wrw->siov);
+}
+
+/*
+ * XXX: It is not worth the complexity to attempt to get the
+ * XXX: end of chunk into the WRW_Flush(), because most of the time
+ * XXX: if not always, that is a no-op anyway, because the calling
+ * XXX: code already called WRW_Flush() to release local storage.
+ */
+
+void
+WRW_EndChunk(struct worker *w)
+{
+	struct wrw *wrw;
+
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	wrw = &w->wrw;
+
+	assert(wrw->ciov < wrw->siov);
+	(void)WRW_Flush(w);
+	wrw->ciov = wrw->siov;
+	wrw->niov = 0;
+	wrw->cliov = 0;
+	(void)WRW_Write(w, "0\r\n\r\n", -1);
+}
+
+
+#ifdef SENDFILE_WORKS
+void
+WRW_Sendfile(struct worker *w, int fd, off_t off, unsigned len)
+{
+	struct wrw *wrw;
+
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	wrw = &w->wrw;
+	AN(wrw->wfd);
+	assert(fd >= 0);
+	assert(len > 0);
+
+#if defined(__FreeBSD__) || defined(__DragonFly__)
+	do {
+		struct sf_hdtr sfh;
+		memset(&sfh, 0, sizeof sfh);
+		if (wrw->niov > 0) {
+			sfh.headers = wrw->iov;
+			sfh.hdr_cnt = wrw->niov;
+		}
+		if (sendfile(fd, *wrw->wfd, off, len, &sfh, NULL, 0) != 0)
+			wrw->werr++;
+		wrw->liov = 0;
+		wrw->niov = 0;
+	} while (0);
+#elif defined(__linux__)
+	do {
+		if (WRW_Flush(w) == 0 &&
+		    sendfile(*wrw->wfd, fd, &off, len) != len)
+			wrw->werr++;
+	} while (0);
+#elif defined(__sun) && defined(HAVE_SENDFILEV)
+	do {
+		sendfilevec_t svvec[cache_param->http_headers * 2 + 1];
+		size_t xferred = 0, expected = 0;
+		int i;
+		for (i = 0; i < wrw->niov; i++) {
+			svvec[i].sfv_fd = SFV_FD_SELF;
+			svvec[i].sfv_flag = 0;
+			svvec[i].sfv_off = (off_t) wrw->iov[i].iov_base;
+			svvec[i].sfv_len = wrw->iov[i].iov_len;
+			expected += svvec[i].sfv_len;
+		}
+		svvec[i].sfv_fd = fd;
+		svvec[i].sfv_flag = 0;
+		svvec[i].sfv_off = off;
+		svvec[i].sfv_len = len;
+		expected += svvec[i].sfv_len;
+		if (sendfilev(*wrw->wfd, svvec, i, &xferred) == -1 ||
+		    xferred != expected)
+			wrw->werr++;
+		wrw->liov = 0;
+		wrw->niov = 0;
+	} while (0);
+#elif defined(__sun) && defined(HAVE_SENDFILE)
+	do {
+		if (WRW_Flush(w) == 0 &&
+		    sendfile(*wrw->wfd, fd, &off, len) != len)
+			wrw->werr++;
+	} while (0);
+#else
+#error Unknown sendfile() implementation
+#endif
+}
+#endif /* SENDFILE_WORKS */
+
diff --git a/bin/varnishd/cache/cache_ws.c b/bin/varnishd/cache/cache_ws.c
new file mode 100644
index 0000000..9fca215
--- /dev/null
+++ b/bin/varnishd/cache/cache_ws.c
@@ -0,0 +1,212 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2010 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ */
+
+#include "config.h"
+
+#include "cache.h"
+
+void
+WS_Assert(const struct ws *ws)
+{
+
+	CHECK_OBJ_NOTNULL(ws, WS_MAGIC);
+	DSL(0x02, SLT_Debug, 0, "WS(%p = (%s, %p %u %u %u)",
+	    ws, ws->id, ws->s, pdiff(ws->s, ws->f),
+	    ws->r == NULL ? 0 : pdiff(ws->f, ws->r),
+	    pdiff(ws->s, ws->e));
+	assert(ws->s != NULL);
+	assert(PAOK(ws->s));
+	assert(ws->e != NULL);
+	assert(PAOK(ws->e));
+	assert(ws->s < ws->e);
+	assert(ws->f >= ws->s);
+	assert(ws->f <= ws->e);
+	assert(PAOK(ws->f));
+	if (ws->r) {
+		assert(ws->r > ws->s);
+		assert(ws->r <= ws->e);
+		assert(PAOK(ws->r));
+	}
+}
+
+void
+WS_Init(struct ws *ws, const char *id, void *space, unsigned len)
+{
+
+	DSL(0x02, SLT_Debug, 0,
+	    "WS_Init(%p, \"%s\", %p, %u)", ws, id, space, len);
+	assert(space != NULL);
+	memset(ws, 0, sizeof *ws);
+	ws->magic = WS_MAGIC;
+	ws->s = space;
+	assert(PAOK(space));
+	ws->e = ws->s + len;
+	assert(PAOK(len));
+	ws->f = ws->s;
+	ws->id = id;
+	WS_Assert(ws);
+}
+
+/*
+ * Reset a WS to start or a given pointer, likely from WS_Snapshot
+ */
+
+void
+WS_Reset(struct ws *ws, char *p)
+{
+
+	WS_Assert(ws);
+	DSL(0x02, SLT_Debug, 0, "WS_Reset(%p, %p)", ws, p);
+	assert(ws->r == NULL);
+	if (p == NULL)
+		ws->f = ws->s;
+	else {
+		assert(p >= ws->s);
+		assert(p < ws->e);
+		ws->f = p;
+	}
+	WS_Assert(ws);
+}
+
+char *
+WS_Alloc(struct ws *ws, unsigned bytes)
+{
+	char *r;
+
+	WS_Assert(ws);
+	bytes = PRNDUP(bytes);
+
+	assert(ws->r == NULL);
+	if (ws->f + bytes > ws->e) {
+		ws->overflow++;
+		WS_Assert(ws);
+		return(NULL);
+	}
+	r = ws->f;
+	ws->f += bytes;
+	DSL(0x02, SLT_Debug, 0, "WS_Alloc(%p, %u) = %p", ws, bytes, r);
+	WS_Assert(ws);
+	return (r);
+}
+
+char *
+WS_Dup(struct ws *ws, const char *s)
+{
+	unsigned l;
+	char *p;
+
+	WS_Assert(ws);
+	l = strlen(s) + 1;
+	p = WS_Alloc(ws, l);
+	if (p != NULL)
+		memcpy(p, s, l);
+	DSL(0x02, SLT_Debug, 0, "WS_Dup(%p, \"%s\") = %p", ws, s, p);
+	WS_Assert(ws);
+	return (p);
+}
+
+unsigned
+WS_Free(const struct ws *ws)
+{
+
+	WS_Assert(ws);
+	return(ws->e - ws->f);
+}
+
+char *
+WS_Snapshot(struct ws *ws)
+{
+
+	WS_Assert(ws);
+	assert(ws->r == NULL);
+	DSL(0x02, SLT_Debug, 0, "WS_Snapshot(%p) = %p", ws, ws->f);
+	return (ws->f);
+}
+
+unsigned
+WS_Reserve(struct ws *ws, unsigned bytes)
+{
+	unsigned b2;
+
+	WS_Assert(ws);
+	assert(ws->r == NULL);
+	if (bytes == 0)
+		b2 = ws->e - ws->f;
+	else if (bytes > ws->e - ws->f)
+		b2 = ws->e - ws->f;
+	else
+		b2 = bytes;
+	b2 = PRNDDN(b2);
+	xxxassert(ws->f + b2 <= ws->e);
+	ws->r = ws->f + b2;
+	DSL(0x02, SLT_Debug, 0, "WS_Reserve(%p, %u/%u) = %u",
+	    ws, b2, bytes, pdiff(ws->f, ws->r));
+	WS_Assert(ws);
+	return (pdiff(ws->f, ws->r));
+}
+
+void
+WS_Release(struct ws *ws, unsigned bytes)
+{
+	WS_Assert(ws);
+	bytes = PRNDUP(bytes);
+	assert(bytes <= ws->e - ws->f);
+	DSL(0x02, SLT_Debug, 0, "WS_Release(%p, %u)", ws, bytes);
+	assert(ws->r != NULL);
+	assert(ws->f + bytes <= ws->r);
+	ws->f += bytes;
+	ws->r = NULL;
+	WS_Assert(ws);
+}
+
+void
+WS_ReleaseP(struct ws *ws, char *ptr)
+{
+	WS_Assert(ws);
+	DSL(0x02, SLT_Debug, 0, "WS_ReleaseP(%p, %p)", ws, ptr);
+	assert(ws->r != NULL);
+	assert(ptr >= ws->f);
+	assert(ptr <= ws->r);
+	ws->f += PRNDUP(ptr - ws->f);
+	ws->r = NULL;
+	WS_Assert(ws);
+}
+
+#if 0
+/* XXX: not used anywhere (yet) */
+void
+WS_Return(struct ws *ws, char *s, char *e)
+{
+
+	WS_Assert(ws);
+	if (e == ws->f)
+		ws->f = s;
+}
+#endif
diff --git a/bin/varnishd/cache_acceptor.c b/bin/varnishd/cache_acceptor.c
deleted file mode 100644
index 8c83121..0000000
--- a/bin/varnishd/cache_acceptor.c
+++ /dev/null
@@ -1,430 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- */
-
-#include "config.h"
-
-#include "cache.h"
-#include "common/heritage.h"
-
-#include "vcli.h"
-#include "vcli_priv.h"
-#include "vtcp.h"
-#include "vtim.h"
-
-static pthread_t	VCA_thread;
-static struct timeval	tv_sndtimeo;
-static struct timeval	tv_rcvtimeo;
-
-/*--------------------------------------------------------------------
- * We want to get out of any kind of trouble-hit TCP connections as fast
- * as absolutely possible, so we set them LINGER enabled with zero timeout,
- * so that even if there are outstanding write data on the socket, a close(2)
- * will return immediately.
- */
-static const struct linger linger = {
-	.l_onoff	=	0,
-};
-
-static unsigned char	need_sndtimeo, need_rcvtimeo, need_linger, need_test;
-
-static void
-sock_test(int fd)
-{
-	struct linger lin;
-	struct timeval tv;
-	socklen_t l;
-	int i;
-
-	l = sizeof lin;
-	i = getsockopt(fd, SOL_SOCKET, SO_LINGER, &lin, &l);
-	if (i) {
-		VTCP_Assert(i);
-		return;
-	}
-	assert(l == sizeof lin);
-	if (memcmp(&lin, &linger, l))
-		need_linger = 1;
-
-#ifdef SO_SNDTIMEO_WORKS
-	l = sizeof tv;
-	i = getsockopt(fd, SOL_SOCKET, SO_SNDTIMEO, &tv, &l);
-	if (i) {
-		VTCP_Assert(i);
-		return;
-	}
-	assert(l == sizeof tv);
-	if (memcmp(&tv, &tv_sndtimeo, l))
-		need_sndtimeo = 1;
-#else
-	(void)tv;
-	(void)tv_sndtimeo;
-	(void)need_sndtimeo;
-#endif
-
-#ifdef SO_RCVTIMEO_WORKS
-	l = sizeof tv;
-	i = getsockopt(fd, SOL_SOCKET, SO_RCVTIMEO, &tv, &l);
-	if (i) {
-		VTCP_Assert(i);
-		return;
-	}
-	assert(l == sizeof tv);
-	if (memcmp(&tv, &tv_rcvtimeo, l))
-		need_rcvtimeo = 1;
-#else
-	(void)tv;
-	(void)tv_rcvtimeo;
-	(void)need_rcvtimeo;
-#endif
-
-	need_test = 0;
-}
-
-/*--------------------------------------------------------------------
- * Called once the workerthread gets hold of the session, to do setup
- * setup overhead, we don't want to bother the acceptor thread with.
- */
-
-void
-VCA_Prep(struct sess *sp)
-{
-	char addr[VTCP_ADDRBUFSIZE];
-	char port[VTCP_PORTBUFSIZE];
-
-	VTCP_name(&sp->sockaddr, sp->sockaddrlen,
-	    addr, sizeof addr, port, sizeof port);
-	sp->addr = WS_Dup(sp->ws, addr);
-	sp->port = WS_Dup(sp->ws, port);
-	if (cache_param->log_local_addr) {
-		AZ(getsockname(sp->fd, (void*)&sp->mysockaddr, &sp->mysockaddrlen));
-		VTCP_name(&sp->mysockaddr, sp->mysockaddrlen,
-		    addr, sizeof addr, port, sizeof port);
-		WSP(sp, SLT_SessionOpen, "%s %s %s %s",
-		    sp->addr, sp->port, addr, port);
-	} else {
-		WSP(sp, SLT_SessionOpen, "%s %s %s",
-		    sp->addr, sp->port, sp->mylsock->name);
-	}
-	sp->acct_ses.first = sp->t_open;
-	if (need_test)
-		sock_test(sp->fd);
-	if (need_linger)
-		VTCP_Assert(setsockopt(sp->fd, SOL_SOCKET, SO_LINGER,
-		    &linger, sizeof linger));
-#ifdef SO_SNDTIMEO_WORKS
-	if (need_sndtimeo)
-		VTCP_Assert(setsockopt(sp->fd, SOL_SOCKET, SO_SNDTIMEO,
-		    &tv_sndtimeo, sizeof tv_sndtimeo));
-#endif
-#ifdef SO_RCVTIMEO_WORKS
-	if (need_rcvtimeo)
-		VTCP_Assert(setsockopt(sp->fd, SOL_SOCKET, SO_RCVTIMEO,
-		    &tv_rcvtimeo, sizeof tv_rcvtimeo));
-#endif
-}
-
-/*--------------------------------------------------------------------
- * If accept(2)'ing fails, we pace ourselves to relive any resource
- * shortage if possible.
- */
-
-static double vca_pace = 0.0;
-static struct lock pace_mtx;
-
-static void
-vca_pace_check(void)
-{
-	double p;
-
-	if (vca_pace == 0.0)
-		return;
-	Lck_Lock(&pace_mtx);
-	p = vca_pace;
-	Lck_Unlock(&pace_mtx);
-	if (p > 0.0)
-		VTIM_sleep(p);
-}
-
-static void
-vca_pace_bad(void)
-{
-
-	Lck_Lock(&pace_mtx);
-	vca_pace += cache_param->acceptor_sleep_incr;
-	if (vca_pace > cache_param->acceptor_sleep_max)
-		vca_pace = cache_param->acceptor_sleep_max;
-	Lck_Unlock(&pace_mtx);
-}
-
-static void
-vca_pace_good(void)
-{
-
-	if (vca_pace == 0.0)
-		return;
-	Lck_Lock(&pace_mtx);
-	vca_pace *= cache_param->acceptor_sleep_decay;
-	if (vca_pace < cache_param->acceptor_sleep_incr)
-		vca_pace = 0.0;
-	Lck_Unlock(&pace_mtx);
-}
-
-/*--------------------------------------------------------------------
- * Accept on a listen socket, and handle error returns.
- */
-
-static int hack_ready;
-
-int
-VCA_Accept(struct listen_sock *ls, struct wrk_accept *wa)
-{
-	int i;
-
-	CHECK_OBJ_NOTNULL(ls, LISTEN_SOCK_MAGIC);
-	vca_pace_check();
-
-	while(!hack_ready)
-		(void)usleep(100*1000);
-
-	wa->acceptaddrlen = sizeof wa->acceptaddr;
-	i = accept(ls->sock, (void*)&wa->acceptaddr, &wa->acceptaddrlen);
-
-	if (i < 0) {
-		switch (errno) {
-		case ECONNABORTED:
-			break;
-		case EMFILE:
-			VSL(SLT_Debug, ls->sock, "Too many open files");
-			vca_pace_bad();
-			break;
-		default:
-			VSL(SLT_Debug, ls->sock, "Accept failed: %s",
-			    strerror(errno));
-			vca_pace_bad();
-			break;
-		}
-	}
-	wa->acceptlsock = ls;
-	wa->acceptsock = i;
-	return (i);
-}
-
-/*--------------------------------------------------------------------
- * Fail a session
- *
- * This happens if we accept the socket, but cannot get a session
- * structure.
- *
- * We consider this a DoS situation (false positive:  Extremely popular
- * busy objects) and silently close the connection with minimum effort
- * and fuzz, rather than try to send an intelligent message back.
- */
-
-void
-VCA_FailSess(struct worker *w)
-{
-	struct wrk_accept *wa;
-
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	CAST_OBJ_NOTNULL(wa, (void*)w->ws->f, WRK_ACCEPT_MAGIC);
-	AZ(w->sp);
-	AZ(close(wa->acceptsock));
-	w->stats.sess_drop++;
-	vca_pace_bad();
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-VCA_SetupSess(struct worker *w)
-{
-	struct sess *sp;
-	struct wrk_accept *wa;
-
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	CAST_OBJ_NOTNULL(wa, (void*)w->ws->f, WRK_ACCEPT_MAGIC);
-	sp = w->sp;
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	sp->fd = wa->acceptsock;
-	sp->vsl_id = wa->acceptsock | VSL_CLIENTMARKER ;
-	wa->acceptsock = -1;
-	sp->t_open = VTIM_real();
-	sp->t_end = sp->t_open;
-	sp->mylsock = wa->acceptlsock;
-	CHECK_OBJ_NOTNULL(sp->mylsock, LISTEN_SOCK_MAGIC);
-	assert(wa->acceptaddrlen <= sp->sockaddrlen);
-	memcpy(&sp->sockaddr, &wa->acceptaddr, wa->acceptaddrlen);
-	sp->sockaddrlen = wa->acceptaddrlen;
-	sp->step = STP_FIRST;
-	vca_pace_good();
-	w->stats.sess_conn++;
-}
-
-/*--------------------------------------------------------------------*/
-
-static void *
-vca_acct(void *arg)
-{
-#ifdef SO_RCVTIMEO_WORKS
-	double sess_timeout = 0;
-#endif
-#ifdef SO_SNDTIMEO_WORKS
-	double send_timeout = 0;
-#endif
-	struct listen_sock *ls;
-	double t0, now;
-
-	THR_SetName("cache-acceptor");
-	(void)arg;
-
-	VTAILQ_FOREACH(ls, &heritage.socks, list) {
-		if (ls->sock < 0)
-			continue;
-		AZ(listen(ls->sock, cache_param->listen_depth));
-		AZ(setsockopt(ls->sock, SOL_SOCKET, SO_LINGER,
-		    &linger, sizeof linger));
-	}
-
-	hack_ready = 1;
-
-	need_test = 1;
-	t0 = VTIM_real();
-	while (1) {
-		(void)sleep(1);
-#ifdef SO_SNDTIMEO_WORKS
-		if (cache_param->idle_send_timeout != send_timeout) {
-			need_test = 1;
-			send_timeout = cache_param->idle_send_timeout;
-			tv_sndtimeo = VTIM_timeval(send_timeout);
-			VTAILQ_FOREACH(ls, &heritage.socks, list) {
-				if (ls->sock < 0)
-					continue;
-				AZ(setsockopt(ls->sock, SOL_SOCKET,
-				    SO_SNDTIMEO,
-				    &tv_sndtimeo, sizeof tv_sndtimeo));
-			}
-		}
-#endif
-#ifdef SO_RCVTIMEO_WORKS
-		if (cache_param->sess_timeout != sess_timeout) {
-			need_test = 1;
-			sess_timeout = cache_param->sess_timeout;
-			tv_rcvtimeo = VTIM_timeval(sess_timeout);
-			VTAILQ_FOREACH(ls, &heritage.socks, list) {
-				if (ls->sock < 0)
-					continue;
-				AZ(setsockopt(ls->sock, SOL_SOCKET,
-				    SO_RCVTIMEO,
-				    &tv_rcvtimeo, sizeof tv_rcvtimeo));
-			}
-		}
-#endif
-		now = VTIM_real();
-		VSC_C_main->uptime = (uint64_t)(now - t0);
-	}
-	NEEDLESS_RETURN(NULL);
-}
-
-
-/*--------------------------------------------------------------------*/
-
-static void
-ccf_start(struct cli *cli, const char * const *av, void *priv)
-{
-
-	(void)cli;
-	(void)av;
-	(void)priv;
-
-	AZ(pthread_create(&VCA_thread, NULL, vca_acct, NULL));
-}
-
-/*--------------------------------------------------------------------*/
-
-static void
-ccf_listen_address(struct cli *cli, const char * const *av, void *priv)
-{
-	struct listen_sock *ls;
-	char h[32], p[32];
-
-	(void)cli;
-	(void)av;
-	(void)priv;
-
-	/*
-	 * This CLI command is primarily used by varnishtest.  Don't
-	 * respond until liste(2) has been called, in order to avoid
-	 * a race where varnishtest::client would attempt to connect(2)
-	 * before listen(2) has been called.
-	 */
-	while(!hack_ready)
-		(void)usleep(100*1000);
-
-	VTAILQ_FOREACH(ls, &heritage.socks, list) {
-		if (ls->sock < 0)
-			continue;
-		VTCP_myname(ls->sock, h, sizeof h, p, sizeof p);
-		VCLI_Out(cli, "%s %s\n", h, p);
-	}
-}
-
-/*--------------------------------------------------------------------*/
-
-static struct cli_proto vca_cmds[] = {
-	{ CLI_SERVER_START,	"i", ccf_start },
-	{ "debug.listen_address",
-	    "debug.listen_address",
-	    "Report the actual listen address\n", 0, 0,
-	    "d", ccf_listen_address, NULL },
-	{ NULL }
-};
-
-void
-VCA_Init(void)
-{
-
-	CLI_AddFuncs(vca_cmds);
-	Lck_New(&pace_mtx, lck_vcapace);
-}
-
-void
-VCA_Shutdown(void)
-{
-	struct listen_sock *ls;
-	int i;
-
-	VTAILQ_FOREACH(ls, &heritage.socks, list) {
-		if (ls->sock < 0)
-			continue;
-		i = ls->sock;
-		ls->sock = -1;
-		(void)close(i);
-	}
-}
diff --git a/bin/varnishd/cache_backend.c b/bin/varnishd/cache_backend.c
deleted file mode 100644
index 1f290c3..0000000
--- a/bin/varnishd/cache_backend.c
+++ /dev/null
@@ -1,517 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * Handle backend connections and backend request structures.
- *
- */
-
-#include "config.h"
-
-#include <poll.h>
-#include <stdlib.h>
-
-#include "cache.h"
-
-#include "cache_backend.h"
-#include "vrt.h"
-#include "vtcp.h"
-
-/*--------------------------------------------------------------------
- * The "simple" director really isn't, since thats where all the actual
- * connections happen.  Nontheless, pretend it is simple by sequestering
- * the directoricity of it under this line.
- */
-
-struct vdi_simple {
-	unsigned		magic;
-#define VDI_SIMPLE_MAGIC	0x476d25b7
-	struct director		dir;
-	struct backend		*backend;
-	const struct vrt_backend *vrt;
-};
-
-/*--------------------------------------------------------------------
- * Create default Host: header for backend request
- */
-void
-VDI_AddHostHeader(const struct sess *sp)
-{
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->wrk->bereq, HTTP_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->wrk->vbc, VBC_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->wrk->vbc->vdis, VDI_SIMPLE_MAGIC);
-	http_PrintfHeader(sp->wrk, sp->vsl_id, sp->wrk->bereq,
-	    "Host: %s", sp->wrk->vbc->vdis->vrt->hosthdr);
-}
-
-/*--------------------------------------------------------------------*/
-
-/* Private interface from backend_cfg.c */
-void
-VBE_ReleaseConn(struct vbc *vc)
-{
-
-	CHECK_OBJ_NOTNULL(vc, VBC_MAGIC);
-	assert(vc->backend == NULL);
-	assert(vc->fd < 0);
-
-	vc->addr = NULL;
-	vc->addrlen = 0;
-	vc->recycled = 0;
-	Lck_Lock(&VBE_mtx);
-	VSC_C_main->n_vbc--;
-	Lck_Unlock(&VBE_mtx);
-	FREE_OBJ(vc);
-}
-
-#define FIND_TMO(tmx, dst, sp, be)		\
-	do {					\
-		dst = sp->wrk->tmx;		\
-		if (dst == 0.0)			\
-			dst = be->tmx;		\
-		if (dst == 0.0)			\
-			dst = cache_param->tmx;	\
-	} while (0)
-
-/*--------------------------------------------------------------------
- * Attempt to connect to a given addrinfo entry.
- *
- * Must be called with locked backend, but will release the backend
- * lock during the slow/sleeping stuff, so that other worker threads
- * can have a go, while we ponder.
- *
- */
-
-static int
-vbe_TryConnect(const struct sess *sp, int pf, const struct sockaddr_storage *sa,
-    socklen_t salen, const struct vdi_simple *vs)
-{
-	int s, i, tmo;
-	double tmod;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(vs, VDI_SIMPLE_MAGIC);
-
-	s = socket(pf, SOCK_STREAM, 0);
-	if (s < 0)
-		return (s);
-
-	FIND_TMO(connect_timeout, tmod, sp, vs->vrt);
-
-	tmo = (int)(tmod * 1000.0);
-
-	i = VTCP_connect(s, sa, salen, tmo);
-
-	if (i != 0) {
-		AZ(close(s));
-		return (-1);
-	}
-
-	return (s);
-}
-
-/*--------------------------------------------------------------------*/
-
-static void
-bes_conn_try(const struct sess *sp, struct vbc *vc, const struct vdi_simple *vs)
-{
-	int s;
-	struct backend *bp = vs->backend;
-	char abuf1[VTCP_ADDRBUFSIZE];
-	char pbuf1[VTCP_PORTBUFSIZE];
-
-	CHECK_OBJ_NOTNULL(vs, VDI_SIMPLE_MAGIC);
-
-	Lck_Lock(&bp->mtx);
-	bp->refcount++;
-	bp->n_conn++;		/* It mostly works */
-	Lck_Unlock(&bp->mtx);
-
-	s = -1;
-	assert(bp->ipv6 != NULL || bp->ipv4 != NULL);
-
-	/* release lock during stuff that can take a long time */
-
-	if (cache_param->prefer_ipv6 && bp->ipv6 != NULL) {
-		s = vbe_TryConnect(sp, PF_INET6, bp->ipv6, bp->ipv6len, vs);
-		vc->addr = bp->ipv6;
-		vc->addrlen = bp->ipv6len;
-	}
-	if (s == -1 && bp->ipv4 != NULL) {
-		s = vbe_TryConnect(sp, PF_INET, bp->ipv4, bp->ipv4len, vs);
-		vc->addr = bp->ipv4;
-		vc->addrlen = bp->ipv4len;
-	}
-	if (s == -1 && !cache_param->prefer_ipv6 && bp->ipv6 != NULL) {
-		s = vbe_TryConnect(sp, PF_INET6, bp->ipv6, bp->ipv6len, vs);
-		vc->addr = bp->ipv6;
-		vc->addrlen = bp->ipv6len;
-	}
-
-	vc->fd = s;
-	if (s < 0) {
-		Lck_Lock(&bp->mtx);
-		bp->n_conn--;
-		bp->refcount--;		/* Only keep ref on success */
-		Lck_Unlock(&bp->mtx);
-		vc->addr = NULL;
-		vc->addrlen = 0;
-	} else {
-		vc->vsl_id = s | VSL_BACKENDMARKER;
-		VTCP_myname(s, abuf1, sizeof abuf1, pbuf1, sizeof pbuf1);
-		WSL(sp->wrk, SLT_BackendOpen, vc->vsl_id, "%s %s %s ",
-		    vs->backend->display_name, abuf1, pbuf1);
-	}
-
-}
-
-/*--------------------------------------------------------------------
- * Check that there is still something at the far end of a given socket.
- * We poll the fd with instant timeout, if there are any events we can't
- * use it (backends are not allowed to pipeline).
- */
-
-static int
-vbe_CheckFd(int fd)
-{
-	struct pollfd pfd;
-
-	pfd.fd = fd;
-	pfd.events = POLLIN;
-	pfd.revents = 0;
-	return(poll(&pfd, 1, 0) == 0);
-}
-
-/*--------------------------------------------------------------------
- * Manage a pool of vbc structures.
- * XXX: as an experiment, make this caching controled by a parameter
- * XXX: so we can see if it has any effect.
- */
-
-static struct vbc *
-vbe_NewConn(void)
-{
-	struct vbc *vc;
-
-	ALLOC_OBJ(vc, VBC_MAGIC);
-	XXXAN(vc);
-	vc->fd = -1;
-	Lck_Lock(&VBE_mtx);
-	VSC_C_main->n_vbc++;
-	Lck_Unlock(&VBE_mtx);
-	return (vc);
-}
-
-/*--------------------------------------------------------------------
- * It evaluates if a backend is healthy _for_a_specific_object_.
- * That means that it relies on sp->objcore->objhead. This is mainly for
- * saint-mode, but also takes backend->healthy into account. If
- * cache_param->saintmode_threshold is 0, this is basically just a test of
- * backend->healthy.
- *
- * The threshold has to be evaluated _after_ the timeout check, otherwise
- * items would never time out once the threshold is reached.
- */
-
-static unsigned int
-vbe_Healthy(const struct vdi_simple *vs, const struct sess *sp)
-{
-	struct trouble *tr;
-	struct trouble *tr2;
-	struct trouble *old;
-	unsigned i = 0, retval;
-	unsigned int threshold;
-	struct backend *backend;
-	uintptr_t target;
-	double now;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(vs, VDI_SIMPLE_MAGIC);
-	backend = vs->backend;
-	CHECK_OBJ_NOTNULL(backend, BACKEND_MAGIC);
-
-	if (backend->admin_health == ah_probe && !backend->healthy)
-		return (0);
-
-	if (backend->admin_health == ah_sick)
-		return (0);
-
-	/* VRT/VCC sets threshold to UINT_MAX to mark that it's not
-	 * specified by VCL (thus use param).
-	 */
-	if (vs->vrt->saintmode_threshold == UINT_MAX)
-		threshold = cache_param->saintmode_threshold;
-	else
-		threshold = vs->vrt->saintmode_threshold;
-
-	if (backend->admin_health == ah_healthy)
-		threshold = UINT_MAX;
-
-	/* Saintmode is disabled */
-	if (threshold == 0)
-		return (1);
-
-	if (sp->objcore == NULL)
-		return (1);
-
-	now = sp->t_req;
-	target = (uintptr_t)(sp->objcore->objhead);
-
-	old = NULL;
-	retval = 1;
-	Lck_Lock(&backend->mtx);
-	VTAILQ_FOREACH_SAFE(tr, &backend->troublelist, list, tr2) {
-		CHECK_OBJ_NOTNULL(tr, TROUBLE_MAGIC);
-
-		if (tr->timeout < now) {
-			VTAILQ_REMOVE(&backend->troublelist, tr, list);
-			old = tr;
-			retval = 1;
-			break;
-		}
-
-		if (tr->target == target) {
-			retval = 0;
-			break;
-		}
-
-		/* If the threshold is at 1, a single entry on the list
-		 * will disable the backend. Since 0 is disable, ++i
-		 * instead of i++ to allow this behavior.
-		 */
-		if (++i >= threshold) {
-			retval = 0;
-			break;
-		}
-	}
-	Lck_Unlock(&backend->mtx);
-
-	if (old != NULL)
-		FREE_OBJ(old);
-
-	return (retval);
-}
-
-/*--------------------------------------------------------------------
- * Get a connection to a particular backend.
- */
-
-static struct vbc *
-vbe_GetVbe(const struct sess *sp, struct vdi_simple *vs)
-{
-	struct vbc *vc;
-	struct backend *bp;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(vs, VDI_SIMPLE_MAGIC);
-	bp = vs->backend;
-	CHECK_OBJ_NOTNULL(bp, BACKEND_MAGIC);
-
-	/* first look for vbc's we can recycle */
-	while (1) {
-		Lck_Lock(&bp->mtx);
-		vc = VTAILQ_FIRST(&bp->connlist);
-		if (vc != NULL) {
-			bp->refcount++;
-			assert(vc->backend == bp);
-			assert(vc->fd >= 0);
-			AN(vc->addr);
-			VTAILQ_REMOVE(&bp->connlist, vc, list);
-		}
-		Lck_Unlock(&bp->mtx);
-		if (vc == NULL)
-			break;
-		if (vbe_CheckFd(vc->fd)) {
-			/* XXX locking of stats */
-			VSC_C_main->backend_reuse += 1;
-			WSP(sp, SLT_Backend, "%d %s %s",
-			    vc->fd, sp->director->vcl_name, bp->display_name);
-			vc->vdis = vs;
-			vc->recycled = 1;
-			return (vc);
-		}
-		VSC_C_main->backend_toolate++;
-		WSL(sp->wrk, SLT_BackendClose, vc->vsl_id, "%s", bp->display_name);
-
-		/* Checkpoint log to flush all info related to this connection
-		   before the OS reuses the FD */
-		WSL_Flush(sp->wrk, 0);
-
-		VTCP_close(&vc->fd);
-		VBE_DropRefConn(bp);
-		vc->backend = NULL;
-		VBE_ReleaseConn(vc);
-	}
-
-	if (!vbe_Healthy(vs, sp)) {
-		VSC_C_main->backend_unhealthy++;
-		return (NULL);
-	}
-
-	if (vs->vrt->max_connections > 0 &&
-	    bp->n_conn >= vs->vrt->max_connections) {
-		VSC_C_main->backend_busy++;
-		return (NULL);
-	}
-
-	vc = vbe_NewConn();
-	assert(vc->fd == -1);
-	AZ(vc->backend);
-	bes_conn_try(sp, vc, vs);
-	if (vc->fd < 0) {
-		VBE_ReleaseConn(vc);
-		VSC_C_main->backend_fail++;
-		return (NULL);
-	}
-	vc->backend = bp;
-	VSC_C_main->backend_conn++;
-	WSP(sp, SLT_Backend, "%d %s %s",
-	    vc->fd, sp->director->vcl_name, bp->display_name);
-	vc->vdis = vs;
-	return (vc);
-}
-
-/*--------------------------------------------------------------------
- * Returns the backend if and only if the this is a simple director.
- * XXX: Needs a better name and possibly needs a better general approach.
- * XXX: This is mainly used by the DNS director to fetch the actual backend
- * XXX: so it can compare DNS lookups with the actual IP.
- */
-
-struct backend *
-vdi_get_backend_if_simple(const struct director *d)
-{
-	CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC);
-	struct vdi_simple *vs, *vs2;
-
-	vs2 = d->priv;
-	if (vs2->magic != VDI_SIMPLE_MAGIC)
-		return (NULL);
-	CAST_OBJ_NOTNULL(vs, d->priv, VDI_SIMPLE_MAGIC);
-	return (vs->backend);
-}
-
-/*--------------------------------------------------------------------
- *
- */
-
-void
-VBE_UseHealth(const struct director *vdi)
-{
-	struct vdi_simple *vs;
-
-	ASSERT_CLI();
-
-	if (strcmp(vdi->name, "simple"))
-		return;
-	CAST_OBJ_NOTNULL(vs, vdi->priv, VDI_SIMPLE_MAGIC);
-	if (vs->vrt->probe == NULL)
-		return;
-	VBP_Use(vs->backend, vs->vrt->probe);
-}
-
-/*--------------------------------------------------------------------
- *
- */
-
-static struct vbc * __match_proto__(vdi_getfd_f)
-vdi_simple_getfd(const struct director *d, struct sess *sp)
-{
-	struct vdi_simple *vs;
-	struct vbc *vc;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC);
-	CAST_OBJ_NOTNULL(vs, d->priv, VDI_SIMPLE_MAGIC);
-	vc = vbe_GetVbe(sp, vs);
-	if (vc != NULL) {
-		FIND_TMO(first_byte_timeout,
-		    vc->first_byte_timeout, sp, vs->vrt);
-		FIND_TMO(between_bytes_timeout,
-		    vc->between_bytes_timeout, sp, vs->vrt);
-	}
-	return (vc);
-}
-
-static unsigned
-vdi_simple_healthy(const struct director *d, const struct sess *sp)
-{
-	struct vdi_simple *vs;
-
-	CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC);
-	CAST_OBJ_NOTNULL(vs, d->priv, VDI_SIMPLE_MAGIC);
-	return (vbe_Healthy(vs, sp));
-}
-
-static void
-vdi_simple_fini(const struct director *d)
-{
-	struct vdi_simple *vs;
-
-	ASSERT_CLI();
-	CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC);
-	CAST_OBJ_NOTNULL(vs, d->priv, VDI_SIMPLE_MAGIC);
-
-	if (vs->vrt->probe != NULL)
-		VBP_Remove(vs->backend, vs->vrt->probe);
-	VBE_DropRefVcl(vs->backend);
-	free(vs->dir.vcl_name);
-	vs->dir.magic = 0;
-	FREE_OBJ(vs);
-}
-
-void
-VRT_init_dir_simple(struct cli *cli, struct director **bp, int idx,
-    const void *priv)
-{
-	const struct vrt_backend *t;
-	struct vdi_simple *vs;
-
-	ASSERT_CLI();
-	(void)cli;
-	t = priv;
-
-	ALLOC_OBJ(vs, VDI_SIMPLE_MAGIC);
-	XXXAN(vs);
-	vs->dir.magic = DIRECTOR_MAGIC;
-	vs->dir.priv = vs;
-	vs->dir.name = "simple";
-	REPLACE(vs->dir.vcl_name, t->vcl_name);
-	vs->dir.getfd = vdi_simple_getfd;
-	vs->dir.fini = vdi_simple_fini;
-	vs->dir.healthy = vdi_simple_healthy;
-
-	vs->vrt = t;
-
-	vs->backend = VBE_AddBackend(cli, t);
-	if (vs->vrt->probe != NULL)
-		VBP_Insert(vs->backend, vs->vrt->probe, vs->vrt->hosthdr);
-
-	bp[idx] = &vs->dir;
-}
diff --git a/bin/varnishd/cache_backend.h b/bin/varnishd/cache_backend.h
deleted file mode 100644
index 72a1283..0000000
--- a/bin/varnishd/cache_backend.h
+++ /dev/null
@@ -1,192 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * This is the central switch-board for backend connections and it is
- * slightly complicated by a number of optimizations.
- *
- * The data structures:
- *
- *    A vrt_backend is a definition of a backend in a VCL program.
- *
- *    A backend is a TCP destination, possibly multi-homed and it has a
- *    number of associated properties and statistics.
- *
- *    A vbc is an open TCP connection to a backend.
- *
- *    A bereq is a memory carrier for handling a HTTP transaction with
- *    a backend over a vbc.
- *
- *    A director is a piece of code that selects which backend to use,
- *    by whatever method or metric it chooses.
- *
- * The relationships:
- *
- *    Backends and directors get instantiated when VCL's are loaded,
- *    and this always happen in the CLI thread.
- *
- *    When a VCL tries to instantiate a backend, any existing backend
- *    with the same identity (== definition in VCL) will be used instead
- *    so that vbc's can be reused across VCL changes.
- *
- *    Directors disapper with the VCL that created them.
- *
- *    Backends disappear when their reference count drop to zero.
- *
- *    Backends have their host/port name looked up to addrinfo structures
- *    when they are instantiated, and we just cache that result and cycle
- *    through the entries (for multihomed backends) on failure only.
- *    XXX: add cli command to redo lookup.
- *
- *    bereq is sort of a step-child here, we just manage the pool of them.
- *
- */
-
-struct vbp_target;
-struct vbc;
-struct vrt_backend_probe;
-
-/*--------------------------------------------------------------------
- * A director is a piece of code which selects one of possibly multiple
- * backends to use.
- */
-
-typedef struct vbc *vdi_getfd_f(const struct director *, struct sess *sp);
-typedef void vdi_fini_f(const struct director *);
-typedef unsigned vdi_healthy(const struct director *, const struct sess *sp);
-
-struct director {
-	unsigned		magic;
-#define DIRECTOR_MAGIC		0x3336351d
-	const char		*name;
-	char			*vcl_name;
-	vdi_getfd_f		*getfd;
-	vdi_fini_f		*fini;
-	vdi_healthy		*healthy;
-	void			*priv;
-};
-
-/*--------------------------------------------------------------------
- * List of objectheads that have recently been rejected by VCL.
- */
-
-struct trouble {
-	unsigned		magic;
-#define TROUBLE_MAGIC		0x4211ab21
-	uintptr_t		target;
-	double			timeout;
-	VTAILQ_ENTRY(trouble)	list;
-};
-
-/*--------------------------------------------------------------------
- * An instance of a backend from a VCL program.
- */
-
-enum admin_health {
-	ah_invalid = 0,
-	ah_healthy,
-	ah_sick,
-	ah_probe
-};
-
-struct backend {
-	unsigned		magic;
-#define BACKEND_MAGIC		0x64c4c7c6
-
-	VTAILQ_ENTRY(backend)	list;
-	int			refcount;
-	struct lock		mtx;
-
-	char			*vcl_name;
-	char			*display_name;
-	char			*ipv4_addr;
-	char			*ipv6_addr;
-	char			*port;
-
-	struct sockaddr_storage	*ipv4;
-	socklen_t		ipv4len;
-	struct sockaddr_storage	*ipv6;
-	socklen_t		ipv6len;
-
-	unsigned		n_conn;
-	VTAILQ_HEAD(, vbc)	connlist;
-
-	struct vbp_target	*probe;
-	unsigned		healthy;
-	enum admin_health	admin_health;
-	VTAILQ_HEAD(, trouble)	troublelist;
-
-	struct VSC_C_vbe	*vsc;
-};
-
-/* -------------------------------------------------------------------*/
-
-/* Backend connection */
-struct vbc {
-	unsigned		magic;
-#define VBC_MAGIC		0x0c5e6592
-	VTAILQ_ENTRY(vbc)	list;
-	struct backend		*backend;
-	struct vdi_simple	*vdis;
-	unsigned		vsl_id;
-	int			fd;
-
-	struct sockaddr_storage	*addr;
-	socklen_t		addrlen;
-
-	uint8_t			recycled;
-
-	/* Timeouts */
-	double			first_byte_timeout;
-	double			between_bytes_timeout;
-};
-
-/* cache_backend.c */
-void VBE_ReleaseConn(struct vbc *vc);
-struct backend *vdi_get_backend_if_simple(const struct director *d);
-
-/* cache_backend_cfg.c */
-extern struct lock VBE_mtx;
-void VBE_DropRefConn(struct backend *);
-void VBE_DropRefVcl(struct backend *);
-void VBE_DropRefLocked(struct backend *b);
-
-/* cache_backend_poll.c */
-void VBP_Insert(struct backend *b, struct vrt_backend_probe const *p, const char *hosthdr);
-void VBP_Remove(struct backend *b, struct vrt_backend_probe const *p);
-void VBP_Use(const struct backend *b, const struct vrt_backend_probe const *p);
-void VBP_Summary(struct cli *cli, const struct vbp_target *vt);
-
-/* Init functions for directors */
-typedef void dir_init_f(struct cli *, struct director **, int , const void*);
-dir_init_f VRT_init_dir_simple;
-dir_init_f VRT_init_dir_dns;
-dir_init_f VRT_init_dir_hash;
-dir_init_f VRT_init_dir_random;
-dir_init_f VRT_init_dir_round_robin;
-dir_init_f VRT_init_dir_fallback;
-dir_init_f VRT_init_dir_client;
diff --git a/bin/varnishd/cache_backend_cfg.c b/bin/varnishd/cache_backend_cfg.c
deleted file mode 100644
index cbb8c85..0000000
--- a/bin/varnishd/cache_backend_cfg.c
+++ /dev/null
@@ -1,507 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * Handle configuration of backends from VCL programs.
- *
- */
-
-#include "config.h"
-
-#include <ctype.h>
-#include <stdio.h>
-#include <stdlib.h>
-
-#include "cache.h"
-
-#include "cache_backend.h"
-#include "vcli.h"
-#include "vcli_priv.h"
-#include "vrt.h"
-
-struct lock VBE_mtx;
-
-
-/*
- * The list of backends is not locked, it is only ever accessed from
- * the CLI thread, so there is no need.
- */
-static VTAILQ_HEAD(, backend) backends = VTAILQ_HEAD_INITIALIZER(backends);
-
-/*--------------------------------------------------------------------
- */
-
-static void
-VBE_Nuke(struct backend *b)
-{
-
-	ASSERT_CLI();
-	VTAILQ_REMOVE(&backends, b, list);
-	free(b->ipv4);
-	free(b->ipv4_addr);
-	free(b->ipv6);
-	free(b->ipv6_addr);
-	free(b->port);
-	VSM_Free(b->vsc);
-	FREE_OBJ(b);
-	VSC_C_main->n_backend--;
-}
-
-/*--------------------------------------------------------------------
- */
-
-void
-VBE_Poll(void)
-{
-	struct backend *b, *b2;
-
-	ASSERT_CLI();
-	VTAILQ_FOREACH_SAFE(b, &backends, list, b2) {
-		assert(
-			b->admin_health == ah_healthy ||
-			b->admin_health == ah_sick ||
-			b->admin_health == ah_probe
-		);
-		if (b->refcount == 0 && b->probe == NULL)
-			VBE_Nuke(b);
-	}
-}
-
-/*--------------------------------------------------------------------
- * Drop a reference to a backend.
- * The last reference must come from the watcher in the CLI thread,
- * as only that thread is allowed to clean up the backend list.
- */
-
-void
-VBE_DropRefLocked(struct backend *b)
-{
-	int i;
-	struct vbc *vbe, *vbe2;
-
-	CHECK_OBJ_NOTNULL(b, BACKEND_MAGIC);
-	assert(b->refcount > 0);
-
-	i = --b->refcount;
-	Lck_Unlock(&b->mtx);
-	if (i > 0)
-		return;
-
-	ASSERT_CLI();
-	VTAILQ_FOREACH_SAFE(vbe, &b->connlist, list, vbe2) {
-		VTAILQ_REMOVE(&b->connlist, vbe, list);
-		if (vbe->fd >= 0) {
-			AZ(close(vbe->fd));
-			vbe->fd = -1;
-		}
-		vbe->backend = NULL;
-		VBE_ReleaseConn(vbe);
-	}
-	VBE_Nuke(b);
-}
-
-void
-VBE_DropRefVcl(struct backend *b)
-{
-
-	CHECK_OBJ_NOTNULL(b, BACKEND_MAGIC);
-
-	Lck_Lock(&b->mtx);
-	b->vsc->vcls--;
-	VBE_DropRefLocked(b);
-}
-
-void
-VBE_DropRefConn(struct backend *b)
-{
-
-	CHECK_OBJ_NOTNULL(b, BACKEND_MAGIC);
-
-	Lck_Lock(&b->mtx);
-	assert(b->n_conn > 0);
-	b->n_conn--;
-	VBE_DropRefLocked(b);
-}
-
-/*--------------------------------------------------------------------
- * See lib/libvcl/vcc_backend.c::emit_sockaddr()
- */
-
-static void
-copy_sockaddr(struct sockaddr_storage **sa, socklen_t *len,
-    const unsigned char *src)
-{
-
-	assert(*src > 0);
-	*sa = calloc(sizeof **sa, 1);
-	XXXAN(*sa);
-	memcpy(*sa, src + 1, *src);
-	*len = *src;
-}
-
-/*--------------------------------------------------------------------
- * Add a backend/director instance when loading a VCL.
- * If an existing backend is matched, grab a refcount and return.
- * Else create a new backend structure with reference initialized to one.
- */
-
-struct backend *
-VBE_AddBackend(struct cli *cli, const struct vrt_backend *vb)
-{
-	struct backend *b;
-	char buf[128];
-
-	AN(vb->vcl_name);
-	assert(vb->ipv4_sockaddr != NULL || vb->ipv6_sockaddr != NULL);
-	(void)cli;
-	ASSERT_CLI();
-
-	/* Run through the list and see if we already have this backend */
-	VTAILQ_FOREACH(b, &backends, list) {
-		CHECK_OBJ_NOTNULL(b, BACKEND_MAGIC);
-		if (strcmp(b->vcl_name, vb->vcl_name))
-			continue;
-		if (vb->ipv4_sockaddr != NULL && (
-		    b->ipv4len != vb->ipv4_sockaddr[0] ||
-		    memcmp(b->ipv4, vb->ipv4_sockaddr + 1, b->ipv4len)))
-			continue;
-		if (vb->ipv6_sockaddr != NULL && (
-		    b->ipv6len != vb->ipv6_sockaddr[0] ||
-		    memcmp(b->ipv6, vb->ipv6_sockaddr + 1, b->ipv6len)))
-			continue;
-		b->refcount++;
-		b->vsc->vcls++;
-		return (b);
-	}
-
-	/* Create new backend */
-	ALLOC_OBJ(b, BACKEND_MAGIC);
-	XXXAN(b);
-	Lck_New(&b->mtx, lck_backend);
-	b->refcount = 1;
-
-	bprintf(buf, "%s(%s,%s,%s)",
-	    vb->vcl_name,
-	    vb->ipv4_addr == NULL ? "" : vb->ipv4_addr,
-	    vb->ipv6_addr == NULL ? "" : vb->ipv6_addr, vb->port);
-
-	b->vsc = VSM_Alloc(sizeof *b->vsc, VSC_CLASS, VSC_TYPE_VBE, buf);
-	b->vsc->vcls++;
-
-	VTAILQ_INIT(&b->connlist);
-
-	VTAILQ_INIT(&b->troublelist);
-
-	/*
-	 * This backend may live longer than the VCL that instantiated it
-	 * so we cannot simply reference the VCL's copy of things.
-	 */
-	REPLACE(b->vcl_name, vb->vcl_name);
-	REPLACE(b->display_name, buf);
-	REPLACE(b->ipv4_addr, vb->ipv4_addr);
-	REPLACE(b->ipv6_addr, vb->ipv6_addr);
-	REPLACE(b->port, vb->port);
-
-	/*
-	 * Copy over the sockaddrs
-	 */
-	if (vb->ipv4_sockaddr != NULL)
-		copy_sockaddr(&b->ipv4, &b->ipv4len, vb->ipv4_sockaddr);
-	if (vb->ipv6_sockaddr != NULL)
-		copy_sockaddr(&b->ipv6, &b->ipv6len, vb->ipv6_sockaddr);
-
-	assert(b->ipv4 != NULL || b->ipv6 != NULL);
-
-	b->healthy = 1;
-	b->admin_health = ah_probe;
-
-	VTAILQ_INSERT_TAIL(&backends, b, list);
-	VSC_C_main->n_backend++;
-	return (b);
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-VRT_init_dir(struct cli *cli, struct director **dir, const char *name,
-    int idx, const void *priv)
-{
-
-	ASSERT_CLI();
-	if (!strcmp(name, "simple"))
-		VRT_init_dir_simple(cli, dir, idx, priv);
-	else if (!strcmp(name, "hash"))
-		VRT_init_dir_hash(cli, dir, idx, priv);
-	else if (!strcmp(name, "random"))
-		VRT_init_dir_random(cli, dir, idx, priv);
-	else if (!strcmp(name, "dns"))
-		VRT_init_dir_dns(cli, dir, idx, priv);
-	else if (!strcmp(name, "round-robin"))
-		VRT_init_dir_round_robin(cli, dir, idx, priv);
-	else if (!strcmp(name, "fallback"))
-		VRT_init_dir_fallback(cli, dir, idx, priv);
-	else if (!strcmp(name, "client"))
-		VRT_init_dir_client(cli, dir, idx, priv);
-	else
-		INCOMPL();
-}
-
-void
-VRT_fini_dir(struct cli *cli, struct director *b)
-{
-
-	(void)cli;
-	ASSERT_CLI();
-	CHECK_OBJ_NOTNULL(b, DIRECTOR_MAGIC);
-	b->fini(b);
-	b->priv = NULL;
-}
-
-/*---------------------------------------------------------------------
- * String to admin_health
- */
-
-static enum admin_health
-vbe_str2adminhealth(const char *wstate)
-{
-
-	if (strcasecmp(wstate, "healthy") == 0)
-		return(ah_healthy);
-	if (strcasecmp(wstate, "sick") == 0)
-		return(ah_sick);
-	if (strcmp(wstate, "auto") == 0)
-		return(ah_probe);
-	return (ah_invalid);
-}
-
-/*---------------------------------------------------------------------
- * A general function for finding backends and doing things with them.
- *
- * Return -1 on match-argument parse errors.
- *
- * If the call-back function returns non-zero, the search is terminated
- * and we relay that return value.
- *
- * Otherwise we return the number of matches.
- */
-
-typedef int bf_func(struct cli *cli, struct backend *b, void *priv);
-
-static int
-backend_find(struct cli *cli, const char *matcher, bf_func *func, void *priv)
-{
-	struct backend *b;
-	const char *s;
-	const char *name_b;
-	ssize_t name_l = 0;
-	const char *ip_b = NULL;
-	ssize_t ip_l = 0;
-	const char *port_b = NULL;
-	ssize_t port_l = 0;
-	int found = 0;
-	int i;
-
-	name_b = matcher;
-	if (matcher != NULL) {
-		s = strchr(matcher,'(');
-
-		if (s != NULL)
-			name_l = s - name_b;
-		else
-			name_l = strlen(name_b);
-
-		if (s != NULL) {
-			s++;
-			while (isspace(*s))
-				s++;
-			ip_b = s;
-			while (*s != '\0' &&
-			    *s != ')' &&
-			    *s != ':' &&
-			    !isspace(*s))
-				s++;
-			ip_l = s - ip_b;
-			while (isspace(*s))
-				s++;
-			if (*s == ':') {
-				s++;
-				while (isspace(*s))
-					s++;
-				port_b = s;
-				while (*s != '\0' && *s != ')' && !isspace(*s))
-					s++;
-				port_l = s - port_b;
-			}
-			while (isspace(*s))
-				s++;
-			if (*s != ')') {
-				VCLI_Out(cli,
-				    "Match string syntax error:"
-				    " ')' not found.");
-				VCLI_SetResult(cli, CLIS_CANT);
-				return (-1);
-			}
-			s++;
-			while (isspace(*s))
-				s++;
-			if (*s != '\0') {
-				VCLI_Out(cli,
-				    "Match string syntax error:"
-				    " junk after ')'");
-				VCLI_SetResult(cli, CLIS_CANT);
-				return (-1);
-			}
-		}
-	}
-	VTAILQ_FOREACH(b, &backends, list) {
-		CHECK_OBJ_NOTNULL(b, BACKEND_MAGIC);
-		if (port_b != NULL && strncmp(b->port, port_b, port_l) != 0)
-			continue;
-		if (name_b != NULL && strncmp(b->vcl_name, name_b, name_l) != 0)
-			continue;
-		if (ip_b != NULL &&
-		    (b->ipv4_addr == NULL ||
-		      strncmp(b->ipv4_addr, ip_b, ip_l)) &&
-		    (b->ipv6_addr == NULL ||
-		      strncmp(b->ipv6_addr, ip_b, ip_l)))
-			continue;
-		found++;
-		i = func(cli, b, priv);
-		if (i)
-			return(i);
-	}
-	return (found);
-}
-
-/*---------------------------------------------------------------------*/
-
-static int __match_proto__()
-do_list(struct cli *cli, struct backend *b, void *priv)
-{
-	int *hdr;
-
-	AN(priv);
-	hdr = priv;
-	if (!*hdr) {
-		VCLI_Out(cli, "%-30s %-6s %-10s %s",
-		    "Backend name", "Refs", "Admin", "Probe");
-		*hdr = 1;
-	}
-	CHECK_OBJ_NOTNULL(b, BACKEND_MAGIC);
-
-	VCLI_Out(cli, "\n%-30s %-6d", b->display_name, b->refcount);
-
-	if (b->admin_health == ah_probe)
-		VCLI_Out(cli, " %-10s", "probe");
-	else if (b->admin_health == ah_sick)
-		VCLI_Out(cli, " %-10s", "sick");
-	else if (b->admin_health == ah_healthy)
-		VCLI_Out(cli, " %-10s", "healthy");
-	else
-		VCLI_Out(cli, " %-10s", "invalid");
-
-	if (b->probe == NULL)
-		VCLI_Out(cli, " %s", "Healthy (no probe)");
-	else {
-		if (b->healthy)
-			VCLI_Out(cli, " %s", "Healthy ");
-		else
-			VCLI_Out(cli, " %s", "Sick ");
-		VBP_Summary(cli, b->probe);
-	}
-
-	return (0);
-}
-
-static void
-cli_backend_list(struct cli *cli, const char * const *av, void *priv)
-{
-	int hdr = 0;
-
-	(void)av;
-	(void)priv;
-	ASSERT_CLI();
-	(void)backend_find(cli, av[2], do_list, &hdr);
-}
-
-/*---------------------------------------------------------------------*/
-
-static int __match_proto__()
-do_set_health(struct cli *cli, struct backend *b, void *priv)
-{
-	enum admin_health state;
-
-	(void)cli;
-	state = *(enum admin_health*)priv;
-	CHECK_OBJ_NOTNULL(b, BACKEND_MAGIC);
-	b->admin_health = state;
-	return (0);
-}
-
-static void
-cli_backend_set_health(struct cli *cli, const char * const *av, void *priv)
-{
-	enum admin_health state;
-	int n;
-
-	(void)av;
-	(void)priv;
-	ASSERT_CLI();
-	AN(av[2]);
-	AN(av[3]);
-	state = vbe_str2adminhealth(av[3]);
-	if (state == ah_invalid) {
-		VCLI_Out(cli, "Invalid state %s", av[3]);
-		VCLI_SetResult(cli, CLIS_PARAM);
-		return;
-	}
-	n = backend_find(cli, av[2], do_set_health, &state);
-	if (n == 0) {
-		VCLI_Out(cli, "No Backends matches");
-		VCLI_SetResult(cli, CLIS_PARAM);
-	}
-}
-
-/*---------------------------------------------------------------------*/
-
-static struct cli_proto backend_cmds[] = {
-	{ "backend.list", "backend.list",
-	    "\tList all backends\n", 0, 1, "d", cli_backend_list },
-	{ "backend.set_health", "backend.set_health matcher state",
-	    "\tShow a backend\n", 2, 2, "d", cli_backend_set_health },
-	{ NULL }
-};
-
-/*---------------------------------------------------------------------*/
-
-void
-VBE_Init(void)
-{
-
-	Lck_New(&VBE_mtx, lck_vbe);
-	CLI_AddFuncs(backend_cmds);
-}
diff --git a/bin/varnishd/cache_backend_poll.c b/bin/varnishd/cache_backend_poll.c
deleted file mode 100644
index efd64cb..0000000
--- a/bin/varnishd/cache_backend_poll.c
+++ /dev/null
@@ -1,597 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * Poll backends for collection of health statistics
- *
- * We co-opt threads from the worker pool for probing the backends,
- * but we want to avoid a potentially messy cleanup operation when we
- * retire the backend, so the thread owns the health information, which
- * the backend references, rather than the other way around.
- *
- */
-
-#include "config.h"
-
-#include <math.h>
-#include <poll.h>
-#include <stdio.h>
-#include <stdlib.h>
-
-#include "cache.h"
-
-#include "cache_backend.h"
-#include "vcli_priv.h"
-#include "vrt.h"
-#include "vtcp.h"
-#include "vtim.h"
-
-/* Default averaging rate, we want something pretty responsive */
-#define AVG_RATE			4
-
-struct vbp_vcl {
-	unsigned			magic;
-#define VBP_VCL_MAGIC			0x70829764
-
-	VTAILQ_ENTRY(vbp_vcl)		list;
-	const struct vrt_backend_probe	*probep;
-	struct vrt_backend_probe	probe;
-	const char			*hosthdr;
-};
-
-struct vbp_target {
-	unsigned			magic;
-#define VBP_TARGET_MAGIC		0x6b7cb656
-
-	struct backend			*backend;
-	VTAILQ_HEAD( ,vbp_vcl)		vcls;
-
-	struct vrt_backend_probe	probe;
-	int				stop;
-	struct vsb			*vsb;
-	char				*req;
-	int				req_len;
-
-	char				resp_buf[128];
-	unsigned			good;
-
-	/* Collected statistics */
-#define BITMAP(n, c, t, b)	uint64_t	n;
-#include "tbl/backend_poll.h"
-#undef BITMAP
-
-	double				last;
-	double				avg;
-	double				rate;
-
-	VTAILQ_ENTRY(vbp_target)	list;
-	pthread_t			thread;
-};
-
-static VTAILQ_HEAD(, vbp_target)	vbp_list =
-    VTAILQ_HEAD_INITIALIZER(vbp_list);
-
-static struct lock			vbp_mtx;
-
-/*--------------------------------------------------------------------
- * Poke one backend, once, but possibly at both IPv4 and IPv6 addresses.
- *
- * We do deliberately not use the stuff in cache_backend.c, because we
- * want to measure the backends response without local distractions.
- */
-
-static int
-vbp_connect(int pf, const struct sockaddr_storage *sa, socklen_t salen, int tmo)
-{
-	int s, i;
-
-	s = socket(pf, SOCK_STREAM, 0);
-	if (s < 0)
-		return (s);
-
-	i = VTCP_connect(s, sa, salen, tmo);
-	if (i == 0)
-		return (s);
-	VTCP_close(&s);
-	return (-1);
-}
-
-static void
-vbp_poke(struct vbp_target *vt)
-{
-	int s, tmo, i;
-	double t_start, t_now, t_end;
-	unsigned rlen, resp;
-	struct backend *bp;
-	char buf[8192], *p;
-	struct pollfd pfda[1], *pfd = pfda;
-
-	bp = vt->backend;
-	CHECK_OBJ_NOTNULL(bp, BACKEND_MAGIC);
-
-	t_start = t_now = VTIM_real();
-	t_end = t_start + vt->probe.timeout;
-	tmo = (int)round((t_end - t_now) * 1e3);
-
-	s = -1;
-	if (cache_param->prefer_ipv6 && bp->ipv6 != NULL) {
-		s = vbp_connect(PF_INET6, bp->ipv6, bp->ipv6len, tmo);
-		t_now = VTIM_real();
-		tmo = (int)round((t_end - t_now) * 1e3);
-		if (s >= 0)
-			vt->good_ipv6 |= 1;
-	}
-	if (tmo > 0 && s < 0 && bp->ipv4 != NULL) {
-		s = vbp_connect(PF_INET, bp->ipv4, bp->ipv4len, tmo);
-		t_now = VTIM_real();
-		tmo = (int)round((t_end - t_now) * 1e3);
-		if (s >= 0)
-			vt->good_ipv4 |= 1;
-	}
-	if (tmo > 0 && s < 0 && bp->ipv6 != NULL) {
-		s = vbp_connect(PF_INET6, bp->ipv6, bp->ipv6len, tmo);
-		t_now = VTIM_real();
-		tmo = (int)round((t_end - t_now) * 1e3);
-		if (s >= 0)
-			vt->good_ipv6 |= 1;
-	}
-	if (s < 0) {
-		/* Got no connection: failed */
-		return;
-	}
-	if (tmo <= 0) {
-		/* Spent too long time getting it */
-		VTCP_close(&s);
-		return;
-	}
-
-	/* Send the request */
-	i = write(s, vt->req, vt->req_len);
-	if (i != vt->req_len) {
-		if (i < 0)
-			vt->err_xmit |= 1;
-		VTCP_close(&s);
-		return;
-	}
-	vt->good_xmit |= 1;
-
-	pfd->fd = s;
-	rlen = 0;
-	do {
-		pfd->events = POLLIN;
-		pfd->revents = 0;
-		tmo = (int)round((t_end - t_now) * 1e3);
-		if (tmo > 0)
-			i = poll(pfd, 1, tmo);
-		if (i == 0 || tmo <= 0) {
-			VTCP_close(&s);
-			return;
-		}
-		if (rlen < sizeof vt->resp_buf)
-			i = read(s, vt->resp_buf + rlen,
-			    sizeof vt->resp_buf - rlen);
-		else
-			i = read(s, buf, sizeof buf);
-		rlen += i;
-	} while (i > 0);
-
-	VTCP_close(&s);
-
-	if (i < 0) {
-		vt->err_recv |= 1;
-		return;
-	}
-
-	if (rlen == 0)
-		return;
-
-	/* So we have a good receive ... */
-	t_now = VTIM_real();
-	vt->last = t_now - t_start;
-	vt->good_recv |= 1;
-
-	/* Now find out if we like the response */
-	vt->resp_buf[sizeof vt->resp_buf - 1] = '\0';
-	p = strchr(vt->resp_buf, '\r');
-	if (p != NULL)
-		*p = '\0';
-	p = strchr(vt->resp_buf, '\n');
-	if (p != NULL)
-		*p = '\0';
-
-	i = sscanf(vt->resp_buf, "HTTP/%*f %u %s", &resp, buf);
-
-	if (i == 2 && resp == vt->probe.exp_status)
-		vt->happy |= 1;
-}
-
-/*--------------------------------------------------------------------
- * Record pokings...
- */
-
-static void
-vbp_start_poke(struct vbp_target *vt)
-{
-	CHECK_OBJ_NOTNULL(vt, VBP_TARGET_MAGIC);
-
-#define BITMAP(n, c, t, b)	vt->n <<= 1;
-#include "tbl/backend_poll.h"
-#undef BITMAP
-
-	vt->last = 0;
-	vt->resp_buf[0] = '\0';
-}
-
-static void
-vbp_has_poked(struct vbp_target *vt)
-{
-	unsigned i, j;
-	uint64_t u;
-	const char *logmsg;
-	char bits[10];
-
-	CHECK_OBJ_NOTNULL(vt, VBP_TARGET_MAGIC);
-
-	/* Calculate exponential average */
-	if (vt->happy & 1) {
-		if (vt->rate < AVG_RATE)
-			vt->rate += 1.0;
-		vt->avg += (vt->last - vt->avg) / vt->rate;
-	}
-
-	i = 0;
-#define BITMAP(n, c, t, b)	bits[i++] = (vt->n & 1) ? c : '-';
-#include "tbl/backend_poll.h"
-#undef BITMAP
-	bits[i] = '\0';
-
-	u = vt->happy;
-	for (i = j = 0; i < vt->probe.window; i++) {
-		if (u & 1)
-			j++;
-		u >>= 1;
-	}
-	vt->good = j;
-
-	if (vt->good >= vt->probe.threshold) {
-		if (vt->backend->healthy)
-			logmsg = "Still healthy";
-		else
-			logmsg = "Back healthy";
-		vt->backend->healthy = 1;
-	} else {
-		if (vt->backend->healthy)
-			logmsg = "Went sick";
-		else
-			logmsg = "Still sick";
-		vt->backend->healthy = 0;
-	}
-	VSL(SLT_Backend_health, 0, "%s %s %s %u %u %u %.6f %.6f %s",
-	    vt->backend->vcl_name, logmsg, bits,
-	    vt->good, vt->probe.threshold, vt->probe.window,
-	    vt->last, vt->avg, vt->resp_buf);
-	vt->backend->vsc->happy = vt->happy;
-}
-
-/*--------------------------------------------------------------------
- * Build request from probe spec
- */
-
-static void
-vbp_build_req(struct vsb *vsb, const struct vbp_vcl *vcl)
-{
-
-	XXXAN(vsb);
-	XXXAN(vcl);
-	VSB_clear(vsb);
-	if(vcl->probe.request != NULL) {
-		VSB_cat(vsb, vcl->probe.request);
-	} else {
-		VSB_printf(vsb, "GET %s HTTP/1.1\r\n",
-		    vcl->probe.url != NULL ?  vcl->probe.url : "/");
-		if (vcl->hosthdr != NULL)
-			VSB_printf(vsb, "Host: %s\r\n", vcl->hosthdr);
-		VSB_printf(vsb, "Connection: close\r\n");
-		VSB_printf(vsb, "\r\n");
-	}
-	AZ(VSB_finish(vsb));
-}
-
-/*--------------------------------------------------------------------
- * One thread per backend to be poked.
- */
-
-static void *
-vbp_wrk_poll_backend(void *priv)
-{
-	struct vbp_target *vt;
-	struct vbp_vcl *vcl = NULL;
-
-	THR_SetName("backend poll");
-
-	CAST_OBJ_NOTNULL(vt, priv, VBP_TARGET_MAGIC);
-
-	while (!vt->stop) {
-		Lck_Lock(&vbp_mtx);
-		if (VTAILQ_FIRST(&vt->vcls) != vcl) {
-			vcl = VTAILQ_FIRST(&vt->vcls);
-			vbp_build_req(vt->vsb, vcl);
-			vt->probe = vcl->probe;
-		}
-		Lck_Unlock(&vbp_mtx);
-
-		vt->req = VSB_data(vt->vsb);
-		vt->req_len = VSB_len(vt->vsb);
-
-		vbp_start_poke(vt);
-		vbp_poke(vt);
-		vbp_has_poked(vt);
-
-		if (!vt->stop)
-			VTIM_sleep(vt->probe.interval);
-	}
-	return (NULL);
-}
-
-/*--------------------------------------------------------------------
- * Cli functions
- */
-
-void
-VBP_Summary(struct cli *cli, const struct vbp_target *vt)
-{
-
-	CHECK_OBJ_NOTNULL(vt, VBP_TARGET_MAGIC);
-	VCLI_Out(cli, "%d/%d", vt->good, vt->probe.window);
-}
-
-static void
-vbp_bitmap(struct cli *cli, char c, uint64_t map, const char *lbl)
-{
-	int i;
-	uint64_t u = (1ULL << 63);
-
-	for (i = 0; i < 64; i++) {
-		if (map & u)
-			VCLI_Out(cli, "%c", c);
-		else
-			VCLI_Out(cli, "-");
-		map <<= 1;
-	}
-	VCLI_Out(cli, " %s\n", lbl);
-}
-
-/*lint -e{506} constant value boolean */
-/*lint -e{774} constant value boolean */
-static void
-vbp_health_one(struct cli *cli, const struct vbp_target *vt)
-{
-
-	VCLI_Out(cli, "Backend %s is %s\n",
-	    vt->backend->vcl_name,
-	    vt->backend->healthy ? "Healthy" : "Sick");
-	VCLI_Out(cli, "Current states  good: %2u threshold: %2u window: %2u\n",
-	    vt->good, vt->probe.threshold, vt->probe.window);
-	VCLI_Out(cli, "Average responsetime of good probes: %.6f\n", vt->avg);
-	VCLI_Out(cli,
-	    "Oldest                       "
-	    "                             Newest\n");
-	VCLI_Out(cli,
-	    "============================="
-	    "===================================\n");
-
-#define BITMAP(n, c, t, b)					\
-		if ((vt->n != 0) || (b))			\
-			vbp_bitmap(cli, (c), vt->n, (t));
-#include "tbl/backend_poll.h"
-#undef BITMAP
-}
-
-static void
-vbp_health(struct cli *cli, const char * const *av, void *priv)
-{
-	struct vbp_target *vt;
-
-	ASSERT_CLI();
-	(void)av;
-	(void)priv;
-
-	VTAILQ_FOREACH(vt, &vbp_list, list)
-		vbp_health_one(cli, vt);
-}
-
-static struct cli_proto debug_cmds[] = {
-	{ "debug.health", "debug.health",
-		"\tDump backend health stuff\n",
-		0, 0, "d", vbp_health },
-	{ NULL }
-};
-
-/*--------------------------------------------------------------------
- * A new VCL wants to probe this backend,
- */
-
-static struct vbp_vcl *
-vbp_new_vcl(const struct vrt_backend_probe *p, const char *hosthdr)
-{
-	struct vbp_vcl *vcl;
-
-	ALLOC_OBJ(vcl, VBP_VCL_MAGIC);
-	XXXAN(vcl);
-	vcl->probep = p;
-	vcl->probe = *p;
-	vcl->hosthdr = hosthdr;
-
-	/*
-	 * Sanitize and insert defaults
-	 * XXX: we could make these defaults parameters
-	 */
-	if (vcl->probe.timeout == 0.0)
-		vcl->probe.timeout = 2.0;
-	if (vcl->probe.interval == 0.0)
-		vcl->probe.interval = 5.0;
-	if (vcl->probe.window == 0)
-		vcl->probe.window = 8;
-	if (vcl->probe.threshold == 0)
-		vcl->probe.threshold = 3;
-	if (vcl->probe.exp_status == 0)
-		vcl->probe.exp_status = 200;
-
-	if (vcl->probe.threshold == ~0U)
-		vcl->probe.initial = vcl->probe.threshold - 1;
-
-	if (vcl->probe.initial > vcl->probe.threshold)
-		vcl->probe.initial = vcl->probe.threshold;
-	return (vcl);
-}
-
-/*--------------------------------------------------------------------
- * Insert/Remove/Use called from cache_backend.c
- */
-
-void
-VBP_Insert(struct backend *b, const struct vrt_backend_probe *p, const char *hosthdr)
-{
-	struct vbp_target *vt;
-	struct vbp_vcl *vcl;
-	int startthread = 0;
-	unsigned u;
-
-	ASSERT_CLI();
-	AN(p);
-
-	if (b->probe == NULL) {
-		ALLOC_OBJ(vt, VBP_TARGET_MAGIC);
-		XXXAN(vt);
-		VTAILQ_INIT(&vt->vcls);
-		vt->backend = b;
-		vt->vsb = VSB_new_auto();
-		XXXAN(vt->vsb);
-		b->probe = vt;
-		startthread = 1;
-		VTAILQ_INSERT_TAIL(&vbp_list, vt, list);
-	} else {
-		vt = b->probe;
-	}
-
-	VTAILQ_FOREACH(vcl, &vt->vcls, list)
-		assert (vcl->probep != p);
-
-	vcl = vbp_new_vcl(p, hosthdr);
-	Lck_Lock(&vbp_mtx);
-	VTAILQ_INSERT_TAIL(&vt->vcls, vcl, list);
-	Lck_Unlock(&vbp_mtx);
-
-	if (startthread) {
-		for (u = 0; u < vcl->probe.initial; u++) {
-			vbp_start_poke(vt);
-			vt->happy |= 1;
-			vbp_has_poked(vt);
-		}
-		AZ(pthread_create(&vt->thread, NULL, vbp_wrk_poll_backend, vt));
-	}
-}
-
-void
-VBP_Use(const struct backend *b, const struct vrt_backend_probe *p)
-{
-	struct vbp_target *vt;
-	struct vbp_vcl *vcl;
-
-	ASSERT_CLI();
-	AN(p);
-	CHECK_OBJ_NOTNULL(b, BACKEND_MAGIC);
-	AN(b->probe);
-	vt = b->probe;
-
-	VTAILQ_FOREACH(vcl, &vt->vcls, list) {
-		if (vcl->probep != p)
-			continue;
-
-		Lck_Lock(&vbp_mtx);
-		VTAILQ_REMOVE(&vt->vcls, vcl, list);
-		VTAILQ_INSERT_HEAD(&vt->vcls, vcl, list);
-		Lck_Unlock(&vbp_mtx);
-		return;
-	}
-}
-
-void
-VBP_Remove(struct backend *b, struct vrt_backend_probe const *p)
-{
-	struct vbp_target *vt;
-	struct vbp_vcl *vcl;
-	void *ret;
-
-	ASSERT_CLI();
-	AN(p);
-	CHECK_OBJ_NOTNULL(b, BACKEND_MAGIC);
-	AN(b->probe);
-	vt = b->probe;
-
-	VTAILQ_FOREACH(vcl, &vt->vcls, list)
-		if (vcl->probep == p)
-			break;
-
-	AN(vcl);
-
-	Lck_Lock(&vbp_mtx);
-	VTAILQ_REMOVE(&vt->vcls, vcl, list);
-	Lck_Unlock(&vbp_mtx);
-
-	FREE_OBJ(vcl);
-
-	if (!VTAILQ_EMPTY(&vt->vcls))
-		return;
-
-	/* No more polling for this backend */
-
-	b->healthy = 1;
-
-	vt->stop = 1;
-	AZ(pthread_cancel(vt->thread));
-	AZ(pthread_join(vt->thread, &ret));
-
-	b->healthy = 1;
-
-	VTAILQ_REMOVE(&vbp_list, vt, list);
-	b->probe = NULL;
-	VSB_delete(vt->vsb);
-	FREE_OBJ(vt);
-}
-
-/*--------------------------------------------------------------------
- * Initialize the backend probe subsystem
- */
-
-void
-VBP_Init(void)
-{
-
-	Lck_New(&vbp_mtx, lck_vbp);
-	CLI_AddFuncs(debug_cmds);
-}
diff --git a/bin/varnishd/cache_ban.c b/bin/varnishd/cache_ban.c
deleted file mode 100644
index 768e631..0000000
--- a/bin/varnishd/cache_ban.c
+++ /dev/null
@@ -1,1108 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * Ban processing
- *
- * A ban consists of a number of conditions (or tests), all of which must be
- * satisfied.  Here are some potential bans we could support:
- *
- *	req.url == "/foo"
- *	req.url ~ ".iso" && obj.size > 10MB
- *	req.http.host ~ "web1.com" && obj.set-cookie ~ "USER=29293"
- *
- * We make the "&&" mandatory from the start, leaving the syntax space
- * for latter handling of "||" as well.
- *
- * Bans are compiled into bytestrings as follows:
- *	8 bytes	- double: timestamp		XXX: Byteorder ?
- *	4 bytes - be32: length
- *	1 byte - flags: 0x01: BAN_F_REQ
- *	N tests
- * A test have this form:
- *	1 byte - arg (see ban_vars.h col 3 "BAN_ARG_XXX")
- *	(n bytes) - http header name, canonical encoding
- *	lump - comparison arg
- *	1 byte - operation (BAN_OPER_)
- *	(lump) - compiled regexp
- * A lump is:
- *	4 bytes - be32: length
- *	n bytes - content
- *
- * In a perfect world, we should vector through VRE to get to PCRE,
- * but since we rely on PCRE's ability to encode the regexp into a
- * byte string, that would be a little bit artificial, so this is
- * the exception that confirmes the rule.
- *
- */
-
-#include "config.h"
-
-#include <pcre.h>
-#include <stdio.h>
-
-#include "cache.h"
-
-#include "hash/hash_slinger.h"
-#include "vcli.h"
-#include "vcli_priv.h"
-#include "vend.h"
-#include "vtim.h"
-
-struct ban {
-	unsigned		magic;
-#define BAN_MAGIC		0x700b08ea
-	VTAILQ_ENTRY(ban)	list;
-	unsigned		refcount;
-	unsigned		flags;
-#define BAN_F_GONE		(1 << 0)
-#define BAN_F_REQ		(1 << 2)
-#define BAN_F_LURK		(3 << 6)	/* ban-lurker-color */
-	VTAILQ_HEAD(,objcore)	objcore;
-	struct vsb		*vsb;
-	uint8_t			*spec;
-};
-
-#define LURK_SHIFT 6
-
-struct ban_test {
-	uint8_t			arg1;
-	const char		*arg1_spec;
-	uint8_t			oper;
-	const char		*arg2;
-	const void		*arg2_spec;
-};
-
-static VTAILQ_HEAD(banhead_s,ban) ban_head = VTAILQ_HEAD_INITIALIZER(ban_head);
-static struct lock ban_mtx;
-static struct ban *ban_magic;
-static pthread_t ban_thread;
-static struct ban * volatile ban_start;
-static bgthread_t ban_lurker;
-
-/*--------------------------------------------------------------------
- * BAN string magic markers
- */
-
-#define	BAN_OPER_EQ	0x10
-#define	BAN_OPER_NEQ	0x11
-#define	BAN_OPER_MATCH	0x12
-#define	BAN_OPER_NMATCH	0x13
-
-#define BAN_ARG_URL	0x18
-#define BAN_ARG_REQHTTP	0x19
-#define BAN_ARG_OBJHTTP	0x1a
-
-/*--------------------------------------------------------------------
- * Variables we can purge on
- */
-
-static const struct pvar {
-	const char		*name;
-	unsigned		flag;
-	uint8_t			tag;
-} pvars[] = {
-#define PVAR(a, b, c)	{ (a), (b), (c) },
-#include "tbl/ban_vars.h"
-#undef PVAR
-	{ 0, 0, 0}
-};
-
-/*--------------------------------------------------------------------
- * Storage handling of bans
- */
-
-struct ban *
-BAN_New(void)
-{
-	struct ban *b;
-
-	ALLOC_OBJ(b, BAN_MAGIC);
-	if (b == NULL)
-		return (b);
-	b->vsb = VSB_new_auto();
-	if (b->vsb == NULL) {
-		FREE_OBJ(b);
-		return (NULL);
-	}
-	VTAILQ_INIT(&b->objcore);
-	return (b);
-}
-
-void
-BAN_Free(struct ban *b)
-{
-
-	CHECK_OBJ_NOTNULL(b, BAN_MAGIC);
-	AZ(b->refcount);
-	assert(VTAILQ_EMPTY(&b->objcore));
-
-	if (b->vsb != NULL)
-		VSB_delete(b->vsb);
-	if (b->spec != NULL)
-		free(b->spec);
-	FREE_OBJ(b);
-}
-
-/*--------------------------------------------------------------------
- * Get & Release a tail reference, used to hold the list stable for
- * traversals etc.
- */
-
-struct ban *
-BAN_TailRef(void)
-{
-	struct ban *b;
-
-	ASSERT_CLI();
-	Lck_Lock(&ban_mtx);
-	b = VTAILQ_LAST(&ban_head, banhead_s);
-	AN(b);
-	b->refcount++;
-	Lck_Unlock(&ban_mtx);
-	return (b);
-}
-
-void
-BAN_TailDeref(struct ban **bb)
-{
-	struct ban *b;
-
-	b = *bb;
-	*bb = NULL;
-	Lck_Lock(&ban_mtx);
-	b->refcount--;
-	Lck_Unlock(&ban_mtx);
-}
-
-/*--------------------------------------------------------------------
- * Extract time and length from ban-spec
- */
-
-static double
-ban_time(const uint8_t *banspec)
-{
-	double t;
-
-	assert(sizeof t == 8);
-	memcpy(&t, banspec, sizeof t);
-	return (t);
-}
-
-static unsigned
-ban_len(const uint8_t *banspec)
-{
-	unsigned u;
-
-	u = vbe32dec(banspec + 8);
-	return (u);
-}
-
-/*--------------------------------------------------------------------
- * Access a lump of bytes in a ban test spec
- */
-
-static void
-ban_add_lump(const struct ban *b, const void *p, uint32_t len)
-{
-	uint8_t buf[sizeof len];
-
-	vbe32enc(buf, len);
-	VSB_bcat(b->vsb, buf, sizeof buf);
-	VSB_bcat(b->vsb, p, len);
-}
-
-static const void *
-ban_get_lump(const uint8_t **bs)
-{
-	const void *r;
-	unsigned ln;
-
-	ln = vbe32dec(*bs);
-	*bs += 4;
-	r = (const void*)*bs;
-	*bs += ln;
-	return (r);
-}
-
-/*--------------------------------------------------------------------
- * Pick a test apart from a spec string
- */
-
-static void
-ban_iter(const uint8_t **bs, struct ban_test *bt)
-{
-
-	memset(bt, 0, sizeof *bt);
-	bt->arg1 = *(*bs)++;
-	if (bt->arg1 == BAN_ARG_REQHTTP || bt->arg1 == BAN_ARG_OBJHTTP) {
-		bt->arg1_spec = (const char *)*bs;
-		(*bs) += (*bs)[0] + 2;
-	}
-	bt->arg2 = ban_get_lump(bs);
-	bt->oper = *(*bs)++;
-	if (bt->oper == BAN_OPER_MATCH || bt->oper == BAN_OPER_NMATCH)
-		bt->arg2_spec = ban_get_lump(bs);
-}
-
-/*--------------------------------------------------------------------
- * Parse and add a http argument specification
- * Output something which HTTP_GetHdr understands
- */
-
-static void
-ban_parse_http(const struct ban *b, const char *a1)
-{
-	int l;
-
-	l = strlen(a1) + 1;
-	assert(l <= 127);
-	VSB_putc(b->vsb, (char)l);
-	VSB_cat(b->vsb, a1);
-	VSB_putc(b->vsb, ':');
-	VSB_putc(b->vsb, '\0');
-}
-
-/*--------------------------------------------------------------------
- * Parse and add a ban test specification
- */
-
-static int
-ban_parse_regexp(struct cli *cli, const struct ban *b, const char *a3)
-{
-	const char *error;
-	int erroroffset, rc;
-	size_t sz;
-	pcre *re;
-
-	re = pcre_compile(a3, 0, &error, &erroroffset, NULL);
-	if (re == NULL) {
-		VSL(SLT_Debug, 0, "REGEX: <%s>", error);
-		VCLI_Out(cli, "%s", error);
-		VCLI_SetResult(cli, CLIS_PARAM);
-		return (-1);
-	}
-	rc = pcre_fullinfo(re, NULL, PCRE_INFO_SIZE, &sz);
-	AZ(rc);
-	ban_add_lump(b, re, sz);
-	return (0);
-}
-
-/*--------------------------------------------------------------------
- * Add a (and'ed) test-condition to a ban
- */
-
-int
-BAN_AddTest(struct cli *cli, struct ban *b, const char *a1, const char *a2,
-    const char *a3)
-{
-	const struct pvar *pv;
-	int i;
-
-	CHECK_OBJ_NOTNULL(b, BAN_MAGIC);
-
-	for (pv = pvars; pv->name != NULL; pv++)
-		if (!strncmp(a1, pv->name, strlen(pv->name)))
-			break;
-	if (pv->name == NULL) {
-		VCLI_Out(cli, "unknown or unsupported field \"%s\"", a1);
-		VCLI_SetResult(cli, CLIS_PARAM);
-		return (-1);
-	}
-
-	if (pv->flag & PVAR_REQ)
-		b->flags |= BAN_F_REQ;
-
-	VSB_putc(b->vsb, pv->tag);
-	if (pv->flag & PVAR_HTTP)
-		ban_parse_http(b, a1 + strlen(pv->name));
-
-	ban_add_lump(b, a3, strlen(a3) + 1);
-	if (!strcmp(a2, "~")) {
-		VSB_putc(b->vsb, BAN_OPER_MATCH);
-		i = ban_parse_regexp(cli, b, a3);
-		if (i)
-			return (i);
-	} else if (!strcmp(a2, "!~")) {
-		VSB_putc(b->vsb, BAN_OPER_NMATCH);
-		i = ban_parse_regexp(cli, b, a3);
-		if (i)
-			return (i);
-	} else if (!strcmp(a2, "==")) {
-		VSB_putc(b->vsb, BAN_OPER_EQ);
-	} else if (!strcmp(a2, "!=")) {
-		VSB_putc(b->vsb, BAN_OPER_NEQ);
-	} else {
-		VCLI_Out(cli,
-		    "expected conditional (~, !~, == or !=) got \"%s\"", a2);
-		VCLI_SetResult(cli, CLIS_PARAM);
-		return (-1);
-	}
-	return (0);
-}
-
-/*--------------------------------------------------------------------
- * We maintain ban_start as a pointer to the first element of the list
- * as a separate variable from the VTAILQ, to avoid depending on the
- * internals of the VTAILQ macros.  We tacitly assume that a pointer
- * write is always atomic in doing so.
- */
-
-void
-BAN_Insert(struct ban *b)
-{
-	struct ban  *bi, *be;
-	ssize_t ln;
-	double t0;
-
-	CHECK_OBJ_NOTNULL(b, BAN_MAGIC);
-
-	AZ(VSB_finish(b->vsb));
-	ln = VSB_len(b->vsb);
-	assert(ln >= 0);
-
-	b->spec = malloc(ln + 13L);	/* XXX */
-	XXXAN(b->spec);
-
-	t0 = VTIM_real();
-	memcpy(b->spec, &t0, sizeof t0);
-	b->spec[12] = (b->flags & BAN_F_REQ) ? 1 : 0;
-	memcpy(b->spec + 13, VSB_data(b->vsb), ln);
-	ln += 13;
-	vbe32enc(b->spec + 8, ln);
-
-	VSB_delete(b->vsb);
-	b->vsb = NULL;
-
-	Lck_Lock(&ban_mtx);
-	VTAILQ_INSERT_HEAD(&ban_head, b, list);
-	ban_start = b;
-	VSC_C_main->bans++;
-	VSC_C_main->bans_added++;
-	if (b->flags & BAN_F_REQ)
-		VSC_C_main->bans_req++;
-
-	be = VTAILQ_LAST(&ban_head, banhead_s);
-	if (cache_param->ban_dups && be != b)
-		be->refcount++;
-	else
-		be = NULL;
-
-	SMP_NewBan(b->spec, ln);
-	Lck_Unlock(&ban_mtx);
-
-	if (be == NULL)
-		return;
-
-	/* Hunt down duplicates, and mark them as gone */
-	bi = b;
-	Lck_Lock(&ban_mtx);
-	while(bi != be) {
-		bi = VTAILQ_NEXT(bi, list);
-		if (bi->flags & BAN_F_GONE)
-			continue;
-		/* Safe because the length is part of the fixed size hdr */
-		if (memcmp(b->spec + 8, bi->spec + 8, ln - 8))
-			continue;
-		bi->flags |= BAN_F_GONE;
-		VSC_C_main->bans_gone++;
-		VSC_C_main->bans_dups++;
-	}
-	be->refcount--;
-	Lck_Unlock(&ban_mtx);
-}
-
-/*--------------------------------------------------------------------
- * A new object is created, grab a reference to the newest ban
- */
-
-void
-BAN_NewObjCore(struct objcore *oc)
-{
-
-	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-	AZ(oc->ban);
-	Lck_Lock(&ban_mtx);
-	oc->ban = ban_start;
-	ban_start->refcount++;
-	VTAILQ_INSERT_TAIL(&ban_start->objcore, oc, ban_list);
-	Lck_Unlock(&ban_mtx);
-}
-
-/*--------------------------------------------------------------------
- * An object is destroyed, release its ban reference
- */
-
-void
-BAN_DestroyObj(struct objcore *oc)
-{
-
-	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-	if (oc->ban == NULL)
-		return;
-	CHECK_OBJ_NOTNULL(oc->ban, BAN_MAGIC);
-	Lck_Lock(&ban_mtx);
-	assert(oc->ban->refcount > 0);
-	oc->ban->refcount--;
-	VTAILQ_REMOVE(&oc->ban->objcore, oc, ban_list);
-	oc->ban = NULL;
-	Lck_Unlock(&ban_mtx);
-}
-
-/*--------------------------------------------------------------------
- * Find and/or Grab a reference to an objects ban based on timestamp
- * Assume we hold a TailRef, so list traversal is safe.
- */
-
-struct ban *
-BAN_RefBan(struct objcore *oc, double t0, const struct ban *tail)
-{
-	struct ban *b;
-	double t1 = 0;
-
-	VTAILQ_FOREACH(b, &ban_head, list) {
-		t1 = ban_time(b->spec);
-		if (t1 <= t0)
-			break;
-		if (b == tail)
-			break;
-	}
-	AN(b);
-	assert(t1 == t0);
-	Lck_Lock(&ban_mtx);
-	b->refcount++;
-	VTAILQ_INSERT_TAIL(&b->objcore, oc, ban_list);
-	Lck_Unlock(&ban_mtx);
-	return (b);
-}
-
-/*--------------------------------------------------------------------
- * Put a skeleton ban in the list, unless there is an identical,
- * time & condition, ban already in place.
- *
- * If a newer ban has same condition, mark the new ban GONE.
- * mark any older bans, with the same condition, GONE as well.
- */
-
-void
-BAN_Reload(const uint8_t *ban, unsigned len)
-{
-	struct ban *b, *b2;
-	int gone = 0;
-	double t0, t1, t2 = 9e99;
-
-	ASSERT_CLI();
-
-	t0 = ban_time(ban);
-	assert(len == ban_len(ban));
-
-	Lck_Lock(&ban_mtx);
-
-	VTAILQ_FOREACH(b, &ban_head, list) {
-		t1 = ban_time(b->spec);
-		assert (t1 < t2);
-		t2 = t1;
-		if (t1 == t0) {
-			Lck_Unlock(&ban_mtx);
-			return;
-		}
-		if (t1 < t0)
-			break;
-		if (!memcmp(b->spec + 8, ban + 8, len - 8)) {
-			gone |= BAN_F_GONE;
-			VSC_C_main->bans_dups++;
-			VSC_C_main->bans_gone++;
-		}
-	}
-
-	VSC_C_main->bans++;
-	VSC_C_main->bans_added++;
-
-	b2 = BAN_New();
-	AN(b2);
-	b2->spec = malloc(len);
-	AN(b2->spec);
-	memcpy(b2->spec, ban, len);
-	b2->flags |= gone;
-	if (ban[12])
-		b2->flags |= BAN_F_REQ;
-	if (b == NULL)
-		VTAILQ_INSERT_TAIL(&ban_head, b2, list);
-	else
-		VTAILQ_INSERT_BEFORE(b, b2, list);
-
-	/* Hunt down older duplicates */
-	for (b = VTAILQ_NEXT(b2, list); b != NULL; b = VTAILQ_NEXT(b, list)) {
-		if (b->flags & BAN_F_GONE)
-			continue;
-		if (!memcmp(b->spec + 8, ban + 8, len - 8)) {
-			b->flags |= BAN_F_GONE;
-			VSC_C_main->bans_dups++;
-			VSC_C_main->bans_gone++;
-		}
-	}
-	Lck_Unlock(&ban_mtx);
-}
-
-/*--------------------------------------------------------------------
- * Get a bans timestamp
- */
-
-double
-BAN_Time(const struct ban *b)
-{
-
-	if (b == NULL)
-		return (0.0);
-
-	CHECK_OBJ_NOTNULL(b, BAN_MAGIC);
-	return (ban_time(b->spec));
-}
-
-/*--------------------------------------------------------------------
- * All silos have read their bans, ready for action
- */
-
-void
-BAN_Compile(void)
-{
-
-	ASSERT_CLI();
-
-	SMP_NewBan(ban_magic->spec, ban_len(ban_magic->spec));
-	ban_start = VTAILQ_FIRST(&ban_head);
-	WRK_BgThread(&ban_thread, "ban-lurker", ban_lurker, NULL);
-}
-
-/*--------------------------------------------------------------------
- * Evaluate ban-spec
- */
-
-static int
-ban_evaluate(const uint8_t *bs, const struct http *objhttp,
-    const struct http *reqhttp, unsigned *tests)
-{
-	struct ban_test bt;
-	const uint8_t *be;
-	char *arg1;
-
-	be = bs + ban_len(bs);
-	bs += 13;
-	while (bs < be) {
-		(*tests)++;
-		ban_iter(&bs, &bt);
-		arg1 = NULL;
-		switch (bt.arg1) {
-		case BAN_ARG_URL:
-			arg1 = reqhttp->hd[HTTP_HDR_URL].b;
-			break;
-		case BAN_ARG_REQHTTP:
-			(void)http_GetHdr(reqhttp, bt.arg1_spec, &arg1);
-			break;
-		case BAN_ARG_OBJHTTP:
-			(void)http_GetHdr(objhttp, bt.arg1_spec, &arg1);
-			break;
-		default:
-			INCOMPL();
-		}
-
-		switch (bt.oper) {
-		case BAN_OPER_EQ:
-			if (arg1 == NULL || strcmp(arg1, bt.arg2))
-				return (0);
-			break;
-		case BAN_OPER_NEQ:
-			if (arg1 != NULL && !strcmp(arg1, bt.arg2))
-				return (0);
-			break;
-		case BAN_OPER_MATCH:
-			if (arg1 == NULL ||
-			    pcre_exec(bt.arg2_spec, NULL, arg1, strlen(arg1),
-			    0, 0, NULL, 0) < 0)
-				return (0);
-			break;
-		case BAN_OPER_NMATCH:
-			if (arg1 != NULL &&
-			    pcre_exec(bt.arg2_spec, NULL, arg1, strlen(arg1),
-			    0, 0, NULL, 0) >= 0)
-				return (0);
-			break;
-		default:
-			INCOMPL();
-		}
-	}
-	return (1);
-}
-
-/*--------------------------------------------------------------------
- * Check an object against all applicable bans
- *
- * Return:
- *	-1 not all bans checked, but none of the checked matched
- *		Only if !has_req
- *	0 No bans matched, object moved to ban_start.
- *	1 Ban matched, object removed from ban list.
- */
-
-static int
-ban_check_object(struct object *o, const struct sess *sp, int has_req)
-{
-	struct ban *b;
-	struct objcore *oc;
-	struct ban * volatile b0;
-	unsigned tests, skipped;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
-	oc = o->objcore;
-	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-	CHECK_OBJ_NOTNULL(oc->ban, BAN_MAGIC);
-
-	b0 = ban_start;
-	CHECK_OBJ_NOTNULL(b0, BAN_MAGIC);
-
-	if (b0 == oc->ban)
-		return (0);
-
-	/*
-	 * This loop is safe without locks, because we know we hold
-	 * a refcount on a ban somewhere in the list and we do not
-	 * inspect the list past that ban.
-	 */
-	tests = 0;
-	skipped = 0;
-	for (b = b0; b != oc->ban; b = VTAILQ_NEXT(b, list)) {
-		CHECK_OBJ_NOTNULL(b, BAN_MAGIC);
-		if (b->flags & BAN_F_GONE)
-			continue;
-		if ((b->flags & BAN_F_LURK) &&
-		    (b->flags & BAN_F_LURK) == (oc->flags & OC_F_LURK)) {
-			AZ(b->flags & BAN_F_REQ);
-			/* Lurker already tested this */
-			continue;
-		}
-		if (!has_req && (b->flags & BAN_F_REQ)) {
-			/*
-			 * We cannot test this one, but there might
-			 * be other bans that match, so we soldier on
-			 */
-			skipped++;
-		} else if (ban_evaluate(b->spec, o->http, sp->http, &tests))
-			break;
-	}
-
-	Lck_Lock(&ban_mtx);
-	VSC_C_main->bans_tested++;
-	VSC_C_main->bans_tests_tested += tests;
-
-	if (b == oc->ban && skipped > 0) {
-		AZ(has_req);
-		Lck_Unlock(&ban_mtx);
-		/*
-		 * Not banned, but some tests were skipped, so we cannot know
-		 * for certain that it cannot be, so we just have to give up.
-		 */
-		return (-1);
-	}
-
-	oc->ban->refcount--;
-	VTAILQ_REMOVE(&oc->ban->objcore, oc, ban_list);
-	if (b == oc->ban) {	/* not banned */
-		b->flags &= ~BAN_F_LURK;
-		VTAILQ_INSERT_TAIL(&b0->objcore, oc, ban_list);
-		b0->refcount++;
-	}
-	Lck_Unlock(&ban_mtx);
-
-	if (b == oc->ban) {	/* not banned */
-		oc->ban = b0;
-		oc_updatemeta(oc);
-		return (0);
-	} else {
-		EXP_Clr(&o->exp);
-		oc->ban = NULL;
-		oc_updatemeta(oc);
-		/* BAN also changed, but that is not important any more */
-		WSP(sp, SLT_ExpBan, "%u was banned", o->xid);
-		EXP_Rearm(o);
-		return (1);
-	}
-}
-
-int
-BAN_CheckObject(struct object *o, const struct sess *sp)
-{
-
-	return (ban_check_object(o, sp, 1) > 0);
-}
-
-static struct ban *
-ban_CheckLast(void)
-{
-	struct ban *b;
-
-	Lck_AssertHeld(&ban_mtx);
-	b = VTAILQ_LAST(&ban_head, banhead_s);
-	if (b != VTAILQ_FIRST(&ban_head) && b->refcount == 0) {
-		if (b->flags & BAN_F_GONE)
-			VSC_C_main->bans_gone--;
-		if (b->flags & BAN_F_REQ)
-			VSC_C_main->bans_req--;
-		VSC_C_main->bans--;
-		VSC_C_main->bans_deleted++;
-		VTAILQ_REMOVE(&ban_head, b, list);
-	} else {
-		b = NULL;
-	}
-	return (b);
-}
-
-/*--------------------------------------------------------------------
- * Ban lurker thread
- */
-
-static int
-ban_lurker_work(const struct sess *sp, unsigned pass)
-{
-	struct ban *b, *b0, *b2;
-	struct objhead *oh;
-	struct objcore *oc, *oc2;
-	struct object *o;
-	int i;
-
-	AN(pass & BAN_F_LURK);
-	AZ(pass & ~BAN_F_LURK);
-
-	/* First route the last ban(s) */
-	do {
-		Lck_Lock(&ban_mtx);
-		b2 = ban_CheckLast();
-		Lck_Unlock(&ban_mtx);
-		if (b2 != NULL)
-			BAN_Free(b2);
-	} while (b2 != NULL);
-
-	/*
-	 * Find out if we have any bans we can do something about
-	 * If we find any, tag them with our pass number.
-	 */
-	i = 0;
-	b0 = NULL;
-	VTAILQ_FOREACH(b, &ban_head, list) {
-		if (b->flags & BAN_F_GONE)
-			continue;
-		if (b->flags & BAN_F_REQ)
-			continue;
-		if (b == VTAILQ_LAST(&ban_head, banhead_s))
-			continue;
-		if (b0 == NULL)
-			b0 = b;
-		i++;
-		b->flags &= ~BAN_F_LURK;
-		b->flags |= pass;
-	}
-	if (cache_param->diag_bitmap & 0x80000)
-		VSL(SLT_Debug, 0, "lurker: %d actionable bans", i);
-	if (i == 0)
-		return (0);
-
-	VTAILQ_FOREACH_REVERSE(b, &ban_head, banhead_s, list) {
-		if (cache_param->diag_bitmap & 0x80000)
-			VSL(SLT_Debug, 0, "lurker doing %f %d",
-			    ban_time(b->spec), b->refcount);
-		while (1) {
-			Lck_Lock(&ban_mtx);
-			oc = VTAILQ_FIRST(&b->objcore);
-			if (oc == NULL)
-				break;
-			CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-			if (cache_param->diag_bitmap & 0x80000)
-				VSL(SLT_Debug, 0, "test: %p %d %d",
-				    oc, oc->flags & OC_F_LURK, pass);
-			if ((oc->flags & OC_F_LURK) == pass)
-				break;
-			oh = oc->objhead;
-			CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC);
-			if (Lck_Trylock(&oh->mtx)) {
-				Lck_Unlock(&ban_mtx);
-				VTIM_sleep(cache_param->ban_lurker_sleep);
-				continue;
-			}
-			/*
-			 * See if the objcore is still on the objhead since
-			 * we race against HSH_Deref() which comes in the
-			 * opposite locking order.
-			 */
-			VTAILQ_FOREACH(oc2, &oh->objcs, list)
-				if (oc == oc2)
-					break;
-			if (oc2 == NULL) {
-				Lck_Unlock(&oh->mtx);
-				Lck_Unlock(&ban_mtx);
-				VTIM_sleep(cache_param->ban_lurker_sleep);
-				continue;
-			}
-			/*
-			 * Grab a reference to the OC and we can let go of
-			 * the BAN mutex
-			 */
-			AN(oc->refcnt);
-			oc->refcnt++;
-			oc->flags &= ~OC_F_LURK;
-			Lck_Unlock(&ban_mtx);
-			/*
-			 * Get the object and check it against all relevant bans
-			 */
-			o = oc_getobj(sp->wrk, oc);
-			i = ban_check_object(o, sp, 0);
-			if (cache_param->diag_bitmap & 0x80000)
-				VSL(SLT_Debug, 0, "lurker got: %p %d",
-				    oc, i);
-			if (i == -1) {
-				/* Not banned, not moved */
-				oc->flags |= pass;
-				Lck_Lock(&ban_mtx);
-				VTAILQ_REMOVE(&b->objcore, oc, ban_list);
-				VTAILQ_INSERT_TAIL(&b->objcore, oc, ban_list);
-				Lck_Unlock(&ban_mtx);
-			}
-			Lck_Unlock(&oh->mtx);
-			if (cache_param->diag_bitmap & 0x80000)
-				VSL(SLT_Debug, 0, "lurker done: %p %d %d",
-				    oc, oc->flags & OC_F_LURK, pass);
-			(void)HSH_Deref(sp->wrk, NULL, &o);
-			VTIM_sleep(cache_param->ban_lurker_sleep);
-		}
-		Lck_AssertHeld(&ban_mtx);
-		if (!(b->flags & BAN_F_REQ)) {
-			if (!(b->flags & BAN_F_GONE)) {
-				b->flags |= BAN_F_GONE;
-				VSC_C_main->bans_gone++;
-			}
-			if (cache_param->diag_bitmap & 0x80000)
-				VSL(SLT_Debug, 0, "lurker BAN %f now gone",
-				    ban_time(b->spec));
-		}
-		Lck_Unlock(&ban_mtx);
-		VTIM_sleep(cache_param->ban_lurker_sleep);
-		if (b == b0)
-			break;
-	}
-	return (1);
-}
-
-static void * __match_proto__(bgthread_t)
-ban_lurker(struct sess *sp, void *priv)
-{
-	struct ban *bf;
-	unsigned pass = (1 << LURK_SHIFT);
-
-	int i = 0;
-	(void)priv;
-	while (1) {
-
-		while (cache_param->ban_lurker_sleep == 0.0) {
-			/*
-			 * Ban-lurker is disabled:
-			 * Clean the last ban, if possible, and sleep
-			 */
-			Lck_Lock(&ban_mtx);
-			bf = ban_CheckLast();
-			Lck_Unlock(&ban_mtx);
-			if (bf != NULL)
-				BAN_Free(bf);
-			else
-				VTIM_sleep(1.0);
-		}
-
-		i = ban_lurker_work(sp, pass);
-		WSL_Flush(sp->wrk, 0);
-		WRK_SumStat(sp->wrk);
-		if (i) {
-			pass += (1 << LURK_SHIFT);
-			pass &= BAN_F_LURK;
-			if (pass == 0)
-				pass += (1 << LURK_SHIFT);
-			VTIM_sleep(cache_param->ban_lurker_sleep);
-		} else {
-			VTIM_sleep(1.0);
-		}
-	}
-	NEEDLESS_RETURN(NULL);
-}
-
-/*--------------------------------------------------------------------
- * CLI functions to add bans
- */
-
-static void
-ccf_ban(struct cli *cli, const char * const *av, void *priv)
-{
-	int narg, i;
-	struct ban *b;
-
-	(void)priv;
-
-	/* First do some cheap checks on the arguments */
-	for (narg = 0; av[narg + 2] != NULL; narg++)
-		continue;
-	if ((narg % 4) != 3) {
-		VCLI_Out(cli, "Wrong number of arguments");
-		VCLI_SetResult(cli, CLIS_PARAM);
-		return;
-	}
-	for (i = 3; i < narg; i += 4) {
-		if (strcmp(av[i + 2], "&&")) {
-			VCLI_Out(cli, "Found \"%s\" expected &&", av[i + 2]);
-			VCLI_SetResult(cli, CLIS_PARAM);
-			return;
-		}
-	}
-
-	b = BAN_New();
-	if (b == NULL) {
-		VCLI_Out(cli, "Out of Memory");
-		VCLI_SetResult(cli, CLIS_CANT);
-		return;
-	}
-	for (i = 0; i < narg; i += 4)
-		if (BAN_AddTest(cli, b, av[i + 2], av[i + 3], av[i + 4])) {
-			BAN_Free(b);
-			return;
-		}
-	BAN_Insert(b);
-}
-
-static void
-ccf_ban_url(struct cli *cli, const char * const *av, void *priv)
-{
-	const char *aav[6];
-
-	(void)priv;
-	aav[0] = NULL;
-	aav[1] = "ban";
-	aav[2] = "req.url";
-	aav[3] = "~";
-	aav[4] = av[2];
-	aav[5] = NULL;
-	ccf_ban(cli, aav, priv);
-}
-
-static void
-ban_render(struct cli *cli, const uint8_t *bs)
-{
-	struct ban_test bt;
-	const uint8_t *be;
-
-	be = bs + ban_len(bs);
-	bs += 13;
-	while (bs < be) {
-		ban_iter(&bs, &bt);
-		switch (bt.arg1) {
-		case BAN_ARG_URL:
-			VCLI_Out(cli, "req.url");
-			break;
-		case BAN_ARG_REQHTTP:
-			VCLI_Out(cli, "req.http.%.*s",
-			    bt.arg1_spec[0] - 1, bt.arg1_spec + 1);
-			break;
-		case BAN_ARG_OBJHTTP:
-			VCLI_Out(cli, "obj.http.%.*s",
-			    bt.arg1_spec[0] - 1, bt.arg1_spec + 1);
-			break;
-		default:
-			INCOMPL();
-		}
-		switch (bt.oper) {
-		case BAN_OPER_EQ:	VCLI_Out(cli, " == "); break;
-		case BAN_OPER_NEQ:	VCLI_Out(cli, " != "); break;
-		case BAN_OPER_MATCH:	VCLI_Out(cli, " ~ "); break;
-		case BAN_OPER_NMATCH:	VCLI_Out(cli, " !~ "); break;
-		default:
-			INCOMPL();
-		}
-		VCLI_Out(cli, "%s", bt.arg2);
-		if (bs < be)
-			VCLI_Out(cli, " && ");
-	}
-}
-
-static void
-ccf_ban_list(struct cli *cli, const char * const *av, void *priv)
-{
-	struct ban *b, *bl;
-
-	(void)av;
-	(void)priv;
-
-	/* Get a reference so we are safe to traverse the list */
-	bl = BAN_TailRef();
-
-	VCLI_Out(cli, "Present bans:\n");
-	VTAILQ_FOREACH(b, &ban_head, list) {
-		if (b == bl && !(cache_param->diag_bitmap & 0x80000))
-			break;
-		VCLI_Out(cli, "%10.6f %5u%s\t", ban_time(b->spec),
-		    bl == b ? b->refcount - 1 : b->refcount,
-		    b->flags & BAN_F_GONE ? "G" : " ");
-		ban_render(cli, b->spec);
-		VCLI_Out(cli, "\n");
-		if (cache_param->diag_bitmap & 0x80000) {
-			Lck_Lock(&ban_mtx);
-			struct objcore *oc;
-			VTAILQ_FOREACH(oc, &b->objcore, ban_list)
-				VCLI_Out(cli, "     %p\n", oc);
-			Lck_Unlock(&ban_mtx);
-		}
-	}
-
-	BAN_TailDeref(&bl);
-}
-
-static struct cli_proto ban_cmds[] = {
-	{ CLI_BAN_URL,				"", ccf_ban_url },
-	{ CLI_BAN,				"", ccf_ban },
-	{ CLI_BAN_LIST,				"", ccf_ban_list },
-	{ NULL }
-};
-
-void
-BAN_Init(void)
-{
-
-	Lck_New(&ban_mtx, lck_ban);
-	CLI_AddFuncs(ban_cmds);
-	assert(BAN_F_LURK == OC_F_LURK);
-	AN((1 << LURK_SHIFT) & BAN_F_LURK);
-	AN((2 << LURK_SHIFT) & BAN_F_LURK);
-
-	ban_magic = BAN_New();
-	AN(ban_magic);
-	ban_magic->flags |= BAN_F_GONE;
-	VSC_C_main->bans_gone++;
-	BAN_Insert(ban_magic);
-}
diff --git a/bin/varnishd/cache_center.c b/bin/varnishd/cache_center.c
deleted file mode 100644
index e42fac8..0000000
--- a/bin/varnishd/cache_center.c
+++ /dev/null
@@ -1,1691 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * This file contains the central state machine for pushing requests.
- *
- * We cannot just use direct calls because it is possible to kick a
- * request back to the lookup stage (usually after a rewrite).  The
- * state engine also allows us to break the processing up into some
- * logical chunks which improves readability a little bit.
- *
- * Since the states are rather nasty in detail, I have decided to embedd
- * a dot(1) graph in the source code comments.  So to see the big picture,
- * extract the DOT lines and run though dot(1), for instance with the
- * command:
- *	sed -n '/^DOT/s///p' cache_center.c | dot -Tps > /tmp/_.ps
- */
-
-/*
-DOT digraph vcl_center {
-xDOT	page="8.2,11.5"
-DOT	size="7.2,10.5"
-DOT	margin="0.5"
-DOT	center="1"
-DOT acceptor [
-DOT	shape=hexagon
-DOT	label="Request received"
-DOT ]
-DOT ERROR [shape=plaintext]
-DOT RESTART [shape=plaintext]
-DOT acceptor -> start [style=bold,color=green]
- */
-
-#include "config.h"
-
-#include <math.h>
-#include <poll.h>
-#include <stdio.h>
-#include <stdlib.h>
-
-#include "cache.h"
-
-#include "hash/hash_slinger.h"
-#include "vcl.h"
-#include "vcli_priv.h"
-#include "vsha256.h"
-#include "vtcp.h"
-#include "vtim.h"
-
-#ifndef HAVE_SRANDOMDEV
-#include "compat/srandomdev.h"
-#endif
-
-static unsigned xids;
-
-/*--------------------------------------------------------------------
- * WAIT
- * Wait (briefly) until we have a full request in our htc.
- */
-
-static int
-cnt_wait(struct sess *sp)
-{
-	int i;
-	struct pollfd pfd[1];
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	AZ(sp->vcl);
-	AZ(sp->obj);
-	assert(sp->xid == 0);
-
-	i = HTC_Complete(sp->htc);
-	if (i == 0 && cache_param->session_linger > 0) {
-		pfd[0].fd = sp->fd;
-		pfd[0].events = POLLIN;
-		pfd[0].revents = 0;
-		i = poll(pfd, 1, cache_param->session_linger);
-		if (i)
-			i = HTC_Rx(sp->htc);
-	}
-	if (i == 0) {
-		WSP(sp, SLT_Debug, "herding");
-		sp->wrk->stats.sess_herd++;
-		SES_Charge(sp);
-		sp->wrk = NULL;
-		Pool_Wait(sp);
-		return (1);
-	}
-	if (i == 1) {
-		sp->step = STP_START;
-		return (0);
-	}
-	if (i == -2) {
-		SES_Close(sp, "overflow");
-		return (0);
-	}
-	if (i == -1 && Tlen(sp->htc->rxbuf) == 0 &&
-	    (errno == 0 || errno == ECONNRESET))
-		SES_Close(sp, "EOF");
-	else
-		SES_Close(sp, "error");
-	sp->step = STP_DONE;
-	return (0);
-}
-
-/*--------------------------------------------------------------------
- * We have a refcounted object on the session, now deliver it.
- *
-DOT subgraph xcluster_prepresp {
-DOT	prepresp [
-DOT		shape=ellipse
-DOT		label="Filter obj.->resp."
-DOT	]
-DOT	vcl_deliver [
-DOT		shape=record
-DOT		label="vcl_deliver()|resp."
-DOT	]
-DOT	prepresp -> vcl_deliver [style=bold,color=green]
-DOT	prepresp -> vcl_deliver [style=bold,color=cyan]
-DOT	prepresp -> vcl_deliver [style=bold,color=red]
-DOT	prepresp -> vcl_deliver [style=bold,color=blue,]
-DOT	vcl_deliver -> deliver [style=bold,color=green,label=deliver]
-DOT	vcl_deliver -> deliver [style=bold,color=red]
-DOT	vcl_deliver -> deliver [style=bold,color=blue]
-DOT     vcl_deliver -> errdeliver [label="error"]
-DOT     errdeliver [label="ERROR",shape=plaintext]
-DOT     vcl_deliver -> rstdeliver [label="restart",color=purple]
-DOT     rstdeliver [label="RESTART",shape=plaintext]
-DOT     vcl_deliver -> streambody [style=bold,color=cyan,label="deliver"]
-DOT }
- *
- */
-
-static int
-cnt_prepresp(struct sess *sp)
-{
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
-
-	if (sp->wrk->do_stream)
-		AssertObjCorePassOrBusy(sp->obj->objcore);
-
-	sp->wrk->res_mode = 0;
-
-	if ((sp->wrk->h_content_length != NULL || !sp->wrk->do_stream) &&
-	    !sp->wrk->do_gzip && !sp->wrk->do_gunzip)
-		sp->wrk->res_mode |= RES_LEN;
-
-	if (!sp->disable_esi && sp->obj->esidata != NULL) {
-		/* In ESI mode, we don't know the aggregate length */
-		sp->wrk->res_mode &= ~RES_LEN;
-		sp->wrk->res_mode |= RES_ESI;
-	}
-
-	if (sp->esi_level > 0) {
-		sp->wrk->res_mode &= ~RES_LEN;
-		sp->wrk->res_mode |= RES_ESI_CHILD;
-	}
-
-	if (cache_param->http_gzip_support && sp->obj->gziped &&
-	    !RFC2616_Req_Gzip(sp)) {
-		/*
-		 * We don't know what it uncompresses to
-		 * XXX: we could cache that
-		 */
-		sp->wrk->res_mode &= ~RES_LEN;
-		sp->wrk->res_mode |= RES_GUNZIP;
-	}
-
-	if (!(sp->wrk->res_mode & (RES_LEN|RES_CHUNKED|RES_EOF))) {
-		if (sp->obj->len == 0 && !sp->wrk->do_stream)
-			/*
-			 * If the object is empty, neither ESI nor GUNZIP
-			 * can make it any different size
-			 */
-			sp->wrk->res_mode |= RES_LEN;
-		else if (!sp->wantbody) {
-			/* Nothing */
-		} else if (sp->http->protover >= 11) {
-			sp->wrk->res_mode |= RES_CHUNKED;
-		} else {
-			sp->wrk->res_mode |= RES_EOF;
-			sp->doclose = "EOF mode";
-		}
-	}
-
-	sp->t_resp = VTIM_real();
-	if (sp->obj->objcore != NULL) {
-		if ((sp->t_resp - sp->obj->last_lru) > cache_param->lru_timeout &&
-		    EXP_Touch(sp->obj->objcore))
-			sp->obj->last_lru = sp->t_resp;
-		sp->obj->last_use = sp->t_resp;	/* XXX: locking ? */
-	}
-	http_Setup(sp->wrk->resp, sp->wrk->ws);
-	RES_BuildHttp(sp);
-	VCL_deliver_method(sp);
-	switch (sp->handling) {
-	case VCL_RET_DELIVER:
-		break;
-	case VCL_RET_RESTART:
-		if (sp->restarts >= cache_param->max_restarts)
-			break;
-		if (sp->wrk->do_stream) {
-			VDI_CloseFd(sp->wrk);
-			HSH_Drop(sp);
-		} else {
-			(void)HSH_Deref(sp->wrk, NULL, &sp->obj);
-		}
-		AZ(sp->obj);
-		sp->restarts++;
-		sp->director = NULL;
-		sp->wrk->h_content_length = NULL;
-		http_Setup(sp->wrk->bereq, NULL);
-		http_Setup(sp->wrk->beresp, NULL);
-		http_Setup(sp->wrk->resp, NULL);
-		sp->step = STP_RECV;
-		return (0);
-	default:
-		WRONG("Illegal action in vcl_deliver{}");
-	}
-	if (sp->wrk->do_stream) {
-		AssertObjCorePassOrBusy(sp->obj->objcore);
-		sp->step = STP_STREAMBODY;
-	} else {
-		sp->step = STP_DELIVER;
-	}
-	return (0);
-}
-
-/*--------------------------------------------------------------------
- * Deliver an already stored object
- *
-DOT subgraph xcluster_deliver {
-DOT	deliver [
-DOT		shape=ellipse
-DOT		label="Send body"
-DOT	]
-DOT }
-DOT deliver -> DONE [style=bold,color=green]
-DOT deliver -> DONE [style=bold,color=red]
-DOT deliver -> DONE [style=bold,color=blue]
- *
- */
-
-static int
-cnt_deliver(struct sess *sp)
-{
-
-	sp->director = NULL;
-	sp->restarts = 0;
-
-	RES_WriteObj(sp);
-
-	assert(WRW_IsReleased(sp->wrk));
-	assert(sp->wrk->wrw.ciov == sp->wrk->wrw.siov);
-	(void)HSH_Deref(sp->wrk, NULL, &sp->obj);
-	http_Setup(sp->wrk->resp, NULL);
-	sp->step = STP_DONE;
-	return (0);
-}
-
-/*--------------------------------------------------------------------
- * This is the final state, figure out if we should close or recycle
- * the client connection
- *
-DOT	DONE [
-DOT		shape=hexagon
-DOT		label="Request completed"
-DOT	]
- */
-
-static int
-cnt_done(struct sess *sp)
-{
-	double dh, dp, da;
-	int i;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_ORNULL(sp->vcl, VCL_CONF_MAGIC);
-
-	AZ(sp->obj);
-	AZ(sp->wrk->vbc);
-	sp->director = NULL;
-	sp->restarts = 0;
-
-	sp->wrk->do_esi = 0;
-	sp->wrk->do_gunzip = 0;
-	sp->wrk->do_gzip = 0;
-	sp->wrk->do_stream = 0;
-	sp->wrk->is_gunzip = 0;
-	sp->wrk->is_gzip = 0;
-
-	if (sp->vcl != NULL && sp->esi_level == 0) {
-		if (sp->wrk->vcl != NULL)
-			VCL_Rel(&sp->wrk->vcl);
-		sp->wrk->vcl = sp->vcl;
-		sp->vcl = NULL;
-	}
-
-	SES_Charge(sp);
-
-	sp->t_end = VTIM_real();
-	sp->wrk->lastused = sp->t_end;
-	if (sp->xid == 0) {
-		sp->t_req = sp->t_end;
-		sp->t_resp = sp->t_end;
-	} else if (sp->esi_level == 0) {
-		dp = sp->t_resp - sp->t_req;
-		da = sp->t_end - sp->t_resp;
-		dh = sp->t_req - sp->t_open;
-		/* XXX: Add StatReq == StatSess */
-		/* XXX: Workaround for pipe */
-		if (sp->fd >= 0) {
-			WSP(sp, SLT_Length, "%ju",
-			    (uintmax_t)sp->req_bodybytes);
-		}
-		WSP(sp, SLT_ReqEnd, "%u %.9f %.9f %.9f %.9f %.9f",
-		    sp->xid, sp->t_req, sp->t_end, dh, dp, da);
-	}
-	sp->xid = 0;
-	sp->t_open = sp->t_end;
-	sp->t_resp = NAN;
-	WSL_Flush(sp->wrk, 0);
-
-	/* If we did an ESI include, don't mess up our state */
-	if (sp->esi_level > 0)
-		return (1);
-
-	sp->req_bodybytes = 0;
-
-	sp->t_req = NAN;
-	sp->hash_always_miss = 0;
-	sp->hash_ignore_busy = 0;
-
-	if (sp->fd >= 0 && sp->doclose != NULL) {
-		/*
-		 * This is an orderly close of the connection; ditch nolinger
-		 * before we close, to get queued data transmitted.
-		 */
-		// XXX: not yet (void)VTCP_linger(sp->fd, 0);
-		SES_Close(sp, sp->doclose);
-	}
-
-	if (sp->fd < 0) {
-		sp->wrk->stats.sess_closed++;
-		SES_Delete(sp, NULL);
-		return (1);
-	}
-
-	if (sp->wrk->stats.client_req >= cache_param->wthread_stats_rate)
-		WRK_SumStat(sp->wrk);
-	/* Reset the workspace to the session-watermark */
-	WS_Reset(sp->ws, sp->ws_ses);
-	WS_Reset(sp->wrk->ws, NULL);
-
-	i = HTC_Reinit(sp->htc);
-	if (i == 1) {
-		sp->wrk->stats.sess_pipeline++;
-		sp->step = STP_START;
-		return (0);
-	}
-	if (Tlen(sp->htc->rxbuf)) {
-		sp->wrk->stats.sess_readahead++;
-		sp->step = STP_WAIT;
-		return (0);
-	}
-	if (cache_param->session_linger > 0) {
-		sp->wrk->stats.sess_linger++;
-		sp->step = STP_WAIT;
-		return (0);
-	}
-	sp->wrk->stats.sess_herd++;
-	sp->wrk = NULL;
-	Pool_Wait(sp);
-	return (1);
-}
-
-/*--------------------------------------------------------------------
- * Emit an error
- *
-DOT subgraph xcluster_error {
-DOT	vcl_error [
-DOT		shape=record
-DOT		label="vcl_error()|resp."
-DOT	]
-DOT	ERROR -> vcl_error
-DOT	vcl_error-> prepresp [label=deliver]
-DOT }
-DOT vcl_error-> rsterr [label="restart",color=purple]
-DOT rsterr [label="RESTART",shape=plaintext]
- */
-
-static int
-cnt_error(struct sess *sp)
-{
-	struct worker *w;
-	struct http *h;
-	char date[40];
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-
-	sp->wrk->do_esi = 0;
-	sp->wrk->is_gzip = 0;
-	sp->wrk->is_gunzip = 0;
-	sp->wrk->do_gzip = 0;
-	sp->wrk->do_gunzip = 0;
-	sp->wrk->do_stream = 0;
-
-	w = sp->wrk;
-	if (sp->obj == NULL) {
-		HSH_Prealloc(sp);
-		EXP_Clr(&w->exp);
-		sp->obj = STV_NewObject(sp, NULL, cache_param->http_resp_size,
-		     &w->exp, (uint16_t)cache_param->http_max_hdr);
-		if (sp->obj == NULL)
-			sp->obj = STV_NewObject(sp, TRANSIENT_STORAGE,
-			    cache_param->http_resp_size, &w->exp,
-			    (uint16_t)cache_param->http_max_hdr);
-		if (sp->obj == NULL) {
-			sp->doclose = "Out of objects";
-			sp->director = NULL;
-			sp->wrk->h_content_length = NULL;
-			http_Setup(sp->wrk->beresp, NULL);
-			http_Setup(sp->wrk->bereq, NULL);
-			sp->step = STP_DONE;
-			return(0);
-		}
-		AN(sp->obj);
-		sp->obj->xid = sp->xid;
-		sp->obj->exp.entered = sp->t_req;
-	} else {
-		/* XXX: Null the headers ? */
-	}
-	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);
-	h = sp->obj->http;
-
-	if (sp->err_code < 100 || sp->err_code > 999)
-		sp->err_code = 501;
-
-	http_PutProtocol(w, sp->vsl_id, h, "HTTP/1.1");
-	http_PutStatus(h, sp->err_code);
-	VTIM_format(VTIM_real(), date);
-	http_PrintfHeader(w, sp->vsl_id, h, "Date: %s", date);
-	http_SetHeader(w, sp->vsl_id, h, "Server: Varnish");
-
-	if (sp->err_reason != NULL)
-		http_PutResponse(w, sp->vsl_id, h, sp->err_reason);
-	else
-		http_PutResponse(w, sp->vsl_id, h,
-		    http_StatusMessage(sp->err_code));
-	VCL_error_method(sp);
-
-	if (sp->handling == VCL_RET_RESTART &&
-	    sp->restarts <  cache_param->max_restarts) {
-		HSH_Drop(sp);
-		sp->director = NULL;
-		sp->restarts++;
-		sp->step = STP_RECV;
-		return (0);
-	} else if (sp->handling == VCL_RET_RESTART)
-		sp->handling = VCL_RET_DELIVER;
-
-
-	/* We always close when we take this path */
-	sp->doclose = "error";
-	sp->wantbody = 1;
-
-	assert(sp->handling == VCL_RET_DELIVER);
-	sp->err_code = 0;
-	sp->err_reason = NULL;
-	http_Setup(sp->wrk->bereq, NULL);
-	sp->step = STP_PREPRESP;
-	return (0);
-}
-
-/*--------------------------------------------------------------------
- * Fetch response headers from the backend
- *
-DOT subgraph xcluster_fetch {
-DOT	fetch [
-DOT		shape=ellipse
-DOT		label="fetch hdr\nfrom backend\n(find obj.ttl)"
-DOT	]
-DOT	vcl_fetch [
-DOT		shape=record
-DOT		label="vcl_fetch()|req.\nbereq.\nberesp."
-DOT	]
-DOT	fetch -> vcl_fetch [style=bold,color=blue]
-DOT	fetch -> vcl_fetch [style=bold,color=red]
-DOT	fetch_pass [
-DOT		shape=ellipse
-DOT		label="obj.f.pass=true"
-DOT	]
-DOT	vcl_fetch -> fetch_pass [label="hit_for_pass",style=bold,color=red]
-DOT }
-DOT fetch_pass -> fetchbody [style=bold,color=red]
-DOT vcl_fetch -> fetchbody [label="deliver",style=bold,color=blue]
-DOT vcl_fetch -> rstfetch [label="restart",color=purple]
-DOT rstfetch [label="RESTART",shape=plaintext]
-DOT fetch -> errfetch
-DOT vcl_fetch -> errfetch [label="error"]
-DOT errfetch [label="ERROR",shape=plaintext]
- */
-
-static int
-cnt_fetch(struct sess *sp)
-{
-	int i;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
-
-	AN(sp->director);
-	AZ(sp->wrk->vbc);
-	AZ(sp->wrk->h_content_length);
-	AZ(sp->wrk->do_close);
-	AZ(sp->wrk->storage_hint);
-
-	http_Setup(sp->wrk->beresp, sp->wrk->ws);
-
-	i = FetchHdr(sp);
-	/*
-	 * If we recycle a backend connection, there is a finite chance
-	 * that the backend closed it before we get a request to it.
-	 * Do a single retry in that case.
-	 */
-	if (i == 1) {
-		VSC_C_main->backend_retry++;
-		i = FetchHdr(sp);
-	}
-
-	if (i) {
-		sp->handling = VCL_RET_ERROR;
-		sp->err_code = 503;
-	} else {
-		/*
-		 * These two headers can be spread over multiple actual headers
-		 * and we rely on their content outside of VCL, so collect them
-		 * into one line here.
-		 */
-		http_CollectHdr(sp->wrk->beresp, H_Cache_Control);
-		http_CollectHdr(sp->wrk->beresp, H_Vary);
-
-		/*
-		 * Figure out how the fetch is supposed to happen, before the
-		 * headers are adultered by VCL
-		 * NB: Also sets other sp->wrk variables
-		 */
-		sp->wrk->body_status = RFC2616_Body(sp);
-
-		sp->err_code = http_GetStatus(sp->wrk->beresp);
-
-		/*
-		 * What does RFC2616 think about TTL ?
-		 */
-		EXP_Clr(&sp->wrk->exp);
-		sp->wrk->exp.entered = VTIM_real();
-		RFC2616_Ttl(sp);
-
-		/* pass from vclrecv{} has negative TTL */
-		if (sp->objcore == NULL)
-			sp->wrk->exp.ttl = -1.;
-
-		AZ(sp->wrk->do_esi);
-
-		VCL_fetch_method(sp);
-
-		switch (sp->handling) {
-		case VCL_RET_HIT_FOR_PASS:
-			if (sp->objcore != NULL)
-				sp->objcore->flags |= OC_F_PASS;
-			sp->step = STP_FETCHBODY;
-			return (0);
-		case VCL_RET_DELIVER:
-			AssertObjCorePassOrBusy(sp->objcore);
-			sp->step = STP_FETCHBODY;
-			return (0);
-		default:
-			break;
-		}
-
-		/* We are not going to fetch the body, Close the connection */
-		VDI_CloseFd(sp->wrk);
-	}
-
-	/* Clean up partial fetch */
-	AZ(sp->wrk->vbc);
-
-	if (sp->objcore != NULL) {
-		CHECK_OBJ_NOTNULL(sp->objcore, OBJCORE_MAGIC);
-		AZ(HSH_Deref(sp->wrk, sp->objcore, NULL));
-		sp->objcore = NULL;
-	}
-	http_Setup(sp->wrk->bereq, NULL);
-	http_Setup(sp->wrk->beresp, NULL);
-	sp->wrk->h_content_length = NULL;
-	sp->director = NULL;
-	sp->wrk->storage_hint = NULL;
-
-	switch (sp->handling) {
-	case VCL_RET_RESTART:
-		sp->restarts++;
-		sp->step = STP_RECV;
-		return (0);
-	case VCL_RET_ERROR:
-		sp->step = STP_ERROR;
-		return (0);
-	default:
-		WRONG("Illegal action in vcl_fetch{}");
-	}
-}
-
-/*--------------------------------------------------------------------
- * Fetch response body from the backend
- *
-DOT subgraph xcluster_body {
-DOT	fetchbody [
-DOT		shape=diamond
-DOT		label="stream ?"
-DOT	]
-DOT	fetchbody2 [
-DOT		shape=ellipse
-DOT		label="fetch body\nfrom backend\n"
-DOT	]
-DOT }
-DOT fetchbody -> fetchbody2 [label=no,style=bold,color=red]
-DOT fetchbody -> fetchbody2 [style=bold,color=blue]
-DOT fetchbody -> prepresp [label=yes,style=bold,color=cyan]
-DOT fetchbody2 -> prepresp [style=bold,color=red]
-DOT fetchbody2 -> prepresp [style=bold,color=blue]
- */
-
-
-static int
-cnt_fetchbody(struct sess *sp)
-{
-	int i;
-	struct http *hp, *hp2;
-	char *b;
-	uint16_t nhttp;
-	unsigned l;
-	struct vsb *vary = NULL;
-	int varyl = 0, pass;
-
-	assert(sp->handling == VCL_RET_HIT_FOR_PASS ||
-	    sp->handling == VCL_RET_DELIVER);
-
-	if (sp->objcore == NULL) {
-		/* This is a pass from vcl_recv */
-		pass = 1;
-		/* VCL may have fiddled this, but that doesn't help */
-		sp->wrk->exp.ttl = -1.;
-	} else if (sp->handling == VCL_RET_HIT_FOR_PASS) {
-		/* pass from vcl_fetch{} -> hit-for-pass */
-		/* XXX: the bereq was not filtered pass... */
-		pass = 1;
-	} else {
-		/* regular object */
-		pass = 0;
-	}
-
-	/*
-	 * The VCL variables beresp.do_g[un]zip tells us how we want the
-	 * object processed before it is stored.
-	 *
-	 * The backend Content-Encoding header tells us what we are going
-	 * to receive, which we classify in the following three classes:
-	 *
-	 *	"Content-Encoding: gzip"	--> object is gzip'ed.
-	 *	no Content-Encoding		--> object is not gzip'ed.
-	 *	anything else			--> do nothing wrt gzip
-	 *
-	 */
-
-	AZ(sp->wrk->vfp);
-
-	/* We do nothing unless the param is set */
-	if (!cache_param->http_gzip_support)
-		sp->wrk->do_gzip = sp->wrk->do_gunzip = 0;
-
-	sp->wrk->is_gzip =
-	    http_HdrIs(sp->wrk->beresp, H_Content_Encoding, "gzip");
-
-	sp->wrk->is_gunzip =
-	    !http_GetHdr(sp->wrk->beresp, H_Content_Encoding, NULL);
-
-	/* It can't be both */
-	assert(sp->wrk->is_gzip == 0 || sp->wrk->is_gunzip == 0);
-
-	/* We won't gunzip unless it is gzip'ed */
-	if (sp->wrk->do_gunzip && !sp->wrk->is_gzip)
-		sp->wrk->do_gunzip = 0;
-
-	/* If we do gunzip, remove the C-E header */
-	if (sp->wrk->do_gunzip)
-		http_Unset(sp->wrk->beresp, H_Content_Encoding);
-
-	/* We wont gzip unless it is ungziped */
-	if (sp->wrk->do_gzip && !sp->wrk->is_gunzip)
-		sp->wrk->do_gzip = 0;
-
-	/* If we do gzip, add the C-E header */
-	if (sp->wrk->do_gzip)
-		http_SetHeader(sp->wrk, sp->vsl_id, sp->wrk->beresp,
-		    "Content-Encoding: gzip");
-
-	/* But we can't do both at the same time */
-	assert(sp->wrk->do_gzip == 0 || sp->wrk->do_gunzip == 0);
-
-	/* ESI takes precedence and handles gzip/gunzip itself */
-	if (sp->wrk->do_esi)
-		sp->wrk->vfp = &vfp_esi;
-	else if (sp->wrk->do_gunzip)
-		sp->wrk->vfp = &vfp_gunzip;
-	else if (sp->wrk->do_gzip)
-		sp->wrk->vfp = &vfp_gzip;
-	else if (sp->wrk->is_gzip)
-		sp->wrk->vfp = &vfp_testgzip;
-
-	if (sp->wrk->do_esi || sp->esi_level > 0)
-		sp->wrk->do_stream = 0;
-	if (!sp->wantbody)
-		sp->wrk->do_stream = 0;
-
-	l = http_EstimateWS(sp->wrk->beresp,
-	    pass ? HTTPH_R_PASS : HTTPH_A_INS, &nhttp);
-
-	/* Create Vary instructions */
-	if (sp->objcore != NULL) {
-		CHECK_OBJ_NOTNULL(sp->objcore, OBJCORE_MAGIC);
-		vary = VRY_Create(sp, sp->wrk->beresp);
-		if (vary != NULL) {
-			varyl = VSB_len(vary);
-			assert(varyl > 0);
-			l += varyl;
-		}
-	}
-
-	/*
-	 * Space for producing a Content-Length: header including padding
-	 * A billion gigabytes is enough for anybody.
-	 */
-	l += strlen("Content-Length: XxxXxxXxxXxxXxxXxx") + sizeof(void *);
-
-	if (sp->wrk->exp.ttl < cache_param->shortlived || sp->objcore == NULL)
-		sp->wrk->storage_hint = TRANSIENT_STORAGE;
-
-	sp->obj = STV_NewObject(sp, sp->wrk->storage_hint, l,
-	    &sp->wrk->exp, nhttp);
-	if (sp->obj == NULL) {
-		/*
-		 * Try to salvage the transaction by allocating a
-		 * shortlived object on Transient storage.
-		 */
-		sp->obj = STV_NewObject(sp, TRANSIENT_STORAGE, l,
-		    &sp->wrk->exp, nhttp);
-		if (sp->wrk->exp.ttl > cache_param->shortlived)
-			sp->wrk->exp.ttl = cache_param->shortlived;
-		sp->wrk->exp.grace = 0.0;
-		sp->wrk->exp.keep = 0.0;
-	}
-	if (sp->obj == NULL) {
-		sp->err_code = 503;
-		sp->step = STP_ERROR;
-		VDI_CloseFd(sp->wrk);
-		return (0);
-	}
-	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);
-
-	sp->wrk->storage_hint = NULL;
-
-	if (sp->wrk->do_gzip || (sp->wrk->is_gzip && !sp->wrk->do_gunzip))
-		sp->obj->gziped = 1;
-
-	if (vary != NULL) {
-		sp->obj->vary =
-		    (void *)WS_Alloc(sp->obj->http->ws, varyl);
-		AN(sp->obj->vary);
-		memcpy(sp->obj->vary, VSB_data(vary), varyl);
-		VRY_Validate(sp->obj->vary);
-		VSB_delete(vary);
-	}
-
-	sp->obj->xid = sp->xid;
-	sp->obj->response = sp->err_code;
-	WS_Assert(sp->obj->ws_o);
-
-	/* Filter into object */
-	hp = sp->wrk->beresp;
-	hp2 = sp->obj->http;
-
-	hp2->logtag = HTTP_Obj;
-	http_CopyResp(hp2, hp);
-	http_FilterFields(sp->wrk, sp->vsl_id, hp2, hp,
-	    pass ? HTTPH_R_PASS : HTTPH_A_INS);
-	http_CopyHome(sp->wrk, sp->vsl_id, hp2);
-
-	if (http_GetHdr(hp, H_Last_Modified, &b))
-		sp->obj->last_modified = VTIM_parse(b);
-	else
-		sp->obj->last_modified = floor(sp->wrk->exp.entered);
-
-	assert(WRW_IsReleased(sp->wrk));
-
-	/*
-	 * If we can deliver a 304 reply, we don't bother streaming.
-	 * Notice that vcl_deliver{} could still nuke the headers
-	 * that allow the 304, in which case we return 200 non-stream.
-	 */
-	if (sp->obj->response == 200 &&
-	    sp->http->conds &&
-	    RFC2616_Do_Cond(sp))
-		sp->wrk->do_stream = 0;
-
-	AssertObjCorePassOrBusy(sp->obj->objcore);
-
-	if (sp->wrk->do_stream) {
-		sp->step = STP_PREPRESP;
-		return (0);
-	}
-
-	/* Use unmodified headers*/
-	i = FetchBody(sp->wrk, sp->obj);
-
-	sp->wrk->h_content_length = NULL;
-
-	http_Setup(sp->wrk->bereq, NULL);
-	http_Setup(sp->wrk->beresp, NULL);
-	sp->wrk->vfp = NULL;
-	assert(WRW_IsReleased(sp->wrk));
-	AZ(sp->wrk->vbc);
-	AN(sp->director);
-
-	if (i) {
-		HSH_Drop(sp);
-		AZ(sp->obj);
-		sp->err_code = 503;
-		sp->step = STP_ERROR;
-		return (0);
-	}
-
-	if (sp->obj->objcore != NULL) {
-		EXP_Insert(sp->obj);
-		AN(sp->obj->objcore);
-		AN(sp->obj->objcore->ban);
-		HSH_Unbusy(sp);
-	}
-	sp->wrk->acct_tmp.fetch++;
-	sp->step = STP_PREPRESP;
-	return (0);
-}
-
-/*--------------------------------------------------------------------
- * Stream the body as we fetch it
-DOT subgraph xstreambody {
-DOT	streambody [
-DOT		shape=ellipse
-DOT		label="streaming\nfetch/deliver"
-DOT	]
-DOT }
-DOT streambody -> DONE [style=bold,color=cyan]
- */
-
-static int
-cnt_streambody(struct sess *sp)
-{
-	int i;
-	struct stream_ctx sctx;
-	uint8_t obuf[sp->wrk->res_mode & RES_GUNZIP ?
-	    cache_param->gzip_stack_buffer : 1];
-
-	memset(&sctx, 0, sizeof sctx);
-	sctx.magic = STREAM_CTX_MAGIC;
-	AZ(sp->wrk->sctx);
-	sp->wrk->sctx = &sctx;
-
-	if (sp->wrk->res_mode & RES_GUNZIP) {
-		sctx.vgz = VGZ_NewUngzip(sp->wrk, "U S -");
-		sctx.obuf = obuf;
-		sctx.obuf_len = sizeof (obuf);
-	}
-
-	RES_StreamStart(sp);
-
-	AssertObjCorePassOrBusy(sp->obj->objcore);
-
-	i = FetchBody(sp->wrk, sp->obj);
-
-	sp->wrk->h_content_length = NULL;
-
-	http_Setup(sp->wrk->bereq, NULL);
-	http_Setup(sp->wrk->beresp, NULL);
-	sp->wrk->vfp = NULL;
-	AZ(sp->wrk->vbc);
-	AN(sp->director);
-
-	if (!i && sp->obj->objcore != NULL) {
-		EXP_Insert(sp->obj);
-		AN(sp->obj->objcore);
-		AN(sp->obj->objcore->ban);
-		HSH_Unbusy(sp);
-	} else {
-		sp->doclose = "Stream error";
-	}
-	sp->wrk->acct_tmp.fetch++;
-	sp->director = NULL;
-	sp->restarts = 0;
-
-	RES_StreamEnd(sp);
-	if (sp->wrk->res_mode & RES_GUNZIP)
-		(void)VGZ_Destroy(&sctx.vgz, sp->vsl_id);
-
-	sp->wrk->sctx = NULL;
-	assert(WRW_IsReleased(sp->wrk));
-	assert(sp->wrk->wrw.ciov == sp->wrk->wrw.siov);
-	(void)HSH_Deref(sp->wrk, NULL, &sp->obj);
-	http_Setup(sp->wrk->resp, NULL);
-	sp->step = STP_DONE;
-	return (0);
-}
-
-/*--------------------------------------------------------------------
- * The very first request
- */
-static int
-cnt_first(struct sess *sp)
-{
-
-	/*
-	 * XXX: If we don't have acceptfilters we are somewhat subject
-	 * XXX: to DoS'ing here.  One remedy would be to set a shorter
-	 * XXX: SO_RCVTIMEO and once we have received something here
-	 * XXX: increase it to the normal value.
-	 */
-
-	assert(sp->xid == 0);
-	assert(sp->restarts == 0);
-	VCA_Prep(sp);
-
-	/* Record the session watermark */
-	sp->ws_ses = WS_Snapshot(sp->ws);
-
-	/* Receive a HTTP protocol request */
-	HTC_Init(sp->htc, sp->ws, sp->fd, sp->vsl_id, cache_param->http_req_size,
-	    cache_param->http_req_hdr_len);
-	sp->wrk->lastused = sp->t_open;
-	sp->wrk->acct_tmp.sess++;
-
-	sp->step = STP_WAIT;
-	return (0);
-}
-
-/*--------------------------------------------------------------------
- * HIT
- * We had a cache hit.  Ask VCL, then march off as instructed.
- *
-DOT subgraph xcluster_hit {
-DOT	hit [
-DOT		shape=record
-DOT		label="vcl_hit()|req.\nobj."
-DOT	]
-DOT }
-DOT hit -> err_hit [label="error"]
-DOT err_hit [label="ERROR",shape=plaintext]
-DOT hit -> rst_hit [label="restart",color=purple]
-DOT rst_hit [label="RESTART",shape=plaintext]
-DOT hit -> pass [label=pass,style=bold,color=red]
-DOT hit -> prepresp [label="deliver",style=bold,color=green]
- */
-
-static int
-cnt_hit(struct sess *sp)
-{
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
-
-	assert(!(sp->obj->objcore->flags & OC_F_PASS));
-
-	AZ(sp->wrk->do_stream);
-
-	VCL_hit_method(sp);
-
-	if (sp->handling == VCL_RET_DELIVER) {
-		/* Dispose of any body part of the request */
-		(void)FetchReqBody(sp);
-		AZ(sp->wrk->bereq->ws);
-		AZ(sp->wrk->beresp->ws);
-		sp->step = STP_PREPRESP;
-		return (0);
-	}
-
-	/* Drop our object, we won't need it */
-	(void)HSH_Deref(sp->wrk, NULL, &sp->obj);
-	sp->objcore = NULL;
-
-	switch(sp->handling) {
-	case VCL_RET_PASS:
-		sp->step = STP_PASS;
-		return (0);
-	case VCL_RET_ERROR:
-		sp->step = STP_ERROR;
-		return (0);
-	case VCL_RET_RESTART:
-		sp->director = NULL;
-		sp->restarts++;
-		sp->step = STP_RECV;
-		return (0);
-	default:
-		WRONG("Illegal action in vcl_hit{}");
-	}
-}
-
-/*--------------------------------------------------------------------
- * LOOKUP
- * Hash things together and look object up in hash-table.
- *
- * LOOKUP consists of two substates so that we can reenter if we
- * encounter a busy object.
- *
-DOT subgraph xcluster_lookup {
-DOT	hash [
-DOT		shape=record
-DOT		label="vcl_hash()|req."
-DOT	]
-DOT	lookup [
-DOT		shape=diamond
-DOT		label="obj in cache ?\ncreate if not"
-DOT	]
-DOT	lookup2 [
-DOT		shape=diamond
-DOT		label="obj.f.pass ?"
-DOT	]
-DOT	hash -> lookup [label="hash",style=bold,color=green]
-DOT	lookup -> lookup2 [label="yes",style=bold,color=green]
-DOT }
-DOT lookup2 -> hit [label="no", style=bold,color=green]
-DOT lookup2 -> pass [label="yes",style=bold,color=red]
-DOT lookup -> miss [label="no",style=bold,color=blue]
- */
-
-static int
-cnt_lookup(struct sess *sp)
-{
-	struct objcore *oc;
-	struct object *o;
-	struct objhead *oh;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
-
-	if (sp->hash_objhead == NULL) {
-		/* Not a waiting list return */
-		AZ(sp->vary_b);
-		AZ(sp->vary_l);
-		AZ(sp->vary_e);
-		(void)WS_Reserve(sp->ws, 0);
-	} else {
-		AN(sp->ws->r);
-	}
-	sp->vary_b = (void*)sp->ws->f;
-	sp->vary_e = (void*)sp->ws->r;
-	sp->vary_b[2] = '\0';
-
-	oc = HSH_Lookup(sp, &oh);
-
-	if (oc == NULL) {
-		/*
-		 * We lost the session to a busy object, disembark the
-		 * worker thread.   The hash code to restart the session,
-		 * still in STP_LOOKUP, later when the busy object isn't.
-		 * NB:  Do not access sp any more !
-		 */
-		return (1);
-	}
-
-
-	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-	CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC);
-
-	/* If we inserted a new object it's a miss */
-	if (oc->flags & OC_F_BUSY) {
-		sp->wrk->stats.cache_miss++;
-
-		if (sp->vary_l != NULL) {
-			assert(oc->busyobj->vary == sp->vary_b);
-			VRY_Validate(oc->busyobj->vary);
-			WS_ReleaseP(sp->ws, (void*)sp->vary_l);
-		} else {
-			AZ(oc->busyobj->vary);
-			WS_Release(sp->ws, 0);
-		}
-		sp->vary_b = NULL;
-		sp->vary_l = NULL;
-		sp->vary_e = NULL;
-
-		sp->objcore = oc;
-		sp->step = STP_MISS;
-		return (0);
-	}
-
-	o = oc_getobj(sp->wrk, oc);
-	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
-	sp->obj = o;
-
-	WS_Release(sp->ws, 0);
-	sp->vary_b = NULL;
-	sp->vary_l = NULL;
-	sp->vary_e = NULL;
-
-	if (oc->flags & OC_F_PASS) {
-		sp->wrk->stats.cache_hitpass++;
-		WSP(sp, SLT_HitPass, "%u", sp->obj->xid);
-		(void)HSH_Deref(sp->wrk, NULL, &sp->obj);
-		sp->objcore = NULL;
-		sp->step = STP_PASS;
-		return (0);
-	}
-
-	sp->wrk->stats.cache_hit++;
-	WSP(sp, SLT_Hit, "%u", sp->obj->xid);
-	sp->step = STP_HIT;
-	return (0);
-}
-
-/*--------------------------------------------------------------------
- * We had a miss, ask VCL, proceed as instructed
- *
-DOT subgraph xcluster_miss {
-DOT	miss [
-DOT		shape=ellipse
-DOT		label="filter req.->bereq."
-DOT	]
-DOT	vcl_miss [
-DOT		shape=record
-DOT		label="vcl_miss()|req.\nbereq."
-DOT	]
-DOT	miss -> vcl_miss [style=bold,color=blue]
-DOT }
-DOT vcl_miss -> rst_miss [label="restart",color=purple]
-DOT rst_miss [label="RESTART",shape=plaintext]
-DOT vcl_miss -> err_miss [label="error"]
-DOT err_miss [label="ERROR",shape=plaintext]
-DOT vcl_miss -> fetch [label="fetch",style=bold,color=blue]
-DOT vcl_miss -> pass [label="pass",style=bold,color=red]
-DOT
- */
-
-static int
-cnt_miss(struct sess *sp)
-{
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
-
-	AZ(sp->obj);
-	AN(sp->objcore);
-	WS_Reset(sp->wrk->ws, NULL);
-	http_Setup(sp->wrk->bereq, sp->wrk->ws);
-	http_FilterHeader(sp, HTTPH_R_FETCH);
-	http_ForceGet(sp->wrk->bereq);
-	if (cache_param->http_gzip_support) {
-		/*
-		 * We always ask the backend for gzip, even if the
-		 * client doesn't grok it.  We will uncompress for
-		 * the minority of clients which don't.
-		 */
-		http_Unset(sp->wrk->bereq, H_Accept_Encoding);
-		http_SetHeader(sp->wrk, sp->vsl_id, sp->wrk->bereq,
-		    "Accept-Encoding: gzip");
-	}
-	sp->wrk->connect_timeout = 0;
-	sp->wrk->first_byte_timeout = 0;
-	sp->wrk->between_bytes_timeout = 0;
-	VCL_miss_method(sp);
-	switch(sp->handling) {
-	case VCL_RET_ERROR:
-		AZ(HSH_Deref(sp->wrk, sp->objcore, NULL));
-		sp->objcore = NULL;
-		http_Setup(sp->wrk->bereq, NULL);
-		sp->step = STP_ERROR;
-		return (0);
-	case VCL_RET_PASS:
-		AZ(HSH_Deref(sp->wrk, sp->objcore, NULL));
-		sp->objcore = NULL;
-		sp->step = STP_PASS;
-		return (0);
-	case VCL_RET_FETCH:
-		sp->step = STP_FETCH;
-		return (0);
-	case VCL_RET_RESTART:
-		AZ(HSH_Deref(sp->wrk, sp->objcore, NULL));
-		sp->objcore = NULL;
-		INCOMPL();
-	default:
-		WRONG("Illegal action in vcl_miss{}");
-	}
-}
-
-/*--------------------------------------------------------------------
- * Start pass processing by getting headers from backend, then
- * continue in passbody.
- *
-DOT subgraph xcluster_pass {
-DOT	pass [
-DOT		shape=ellipse
-DOT		label="deref obj."
-DOT	]
-DOT	pass2 [
-DOT		shape=ellipse
-DOT		label="filter req.->bereq."
-DOT	]
-DOT	vcl_pass [
-DOT		shape=record
-DOT		label="vcl_pass()|req.\nbereq."
-DOT	]
-DOT	pass_do [
-DOT		shape=ellipse
-DOT		label="create anon object\n"
-DOT	]
-DOT	pass -> pass2 [style=bold, color=red]
-DOT	pass2 -> vcl_pass [style=bold, color=red]
-DOT	vcl_pass -> pass_do [label="pass"] [style=bold, color=red]
-DOT }
-DOT pass_do -> fetch [style=bold, color=red]
-DOT vcl_pass -> rst_pass [label="restart",color=purple]
-DOT rst_pass [label="RESTART",shape=plaintext]
-DOT vcl_pass -> err_pass [label="error"]
-DOT err_pass [label="ERROR",shape=plaintext]
- */
-
-static int
-cnt_pass(struct sess *sp)
-{
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
-	AZ(sp->obj);
-
-	WS_Reset(sp->wrk->ws, NULL);
-	http_Setup(sp->wrk->bereq, sp->wrk->ws);
-	http_FilterHeader(sp, HTTPH_R_PASS);
-
-	sp->wrk->connect_timeout = 0;
-	sp->wrk->first_byte_timeout = 0;
-	sp->wrk->between_bytes_timeout = 0;
-	VCL_pass_method(sp);
-	if (sp->handling == VCL_RET_ERROR) {
-		http_Setup(sp->wrk->bereq, NULL);
-		sp->step = STP_ERROR;
-		return (0);
-	}
-	assert(sp->handling == VCL_RET_PASS);
-	sp->wrk->acct_tmp.pass++;
-	sp->sendbody = 1;
-	sp->step = STP_FETCH;
-	return (0);
-}
-
-/*--------------------------------------------------------------------
- * Ship the request header to the backend unchanged, then pipe
- * until one of the ends close the connection.
- *
-DOT subgraph xcluster_pipe {
-DOT	pipe [
-DOT		shape=ellipse
-DOT		label="Filter req.->bereq."
-DOT	]
-DOT	vcl_pipe [
-DOT		shape=record
-DOT		label="vcl_pipe()|req.\nbereq\."
-DOT	]
-DOT	pipe_do [
-DOT		shape=ellipse
-DOT		label="send bereq.\npipe until close"
-DOT	]
-DOT	vcl_pipe -> pipe_do [label="pipe",style=bold,color=orange]
-DOT	pipe -> vcl_pipe [style=bold,color=orange]
-DOT }
-DOT pipe_do -> DONE [style=bold,color=orange]
-DOT vcl_pipe -> err_pipe [label="error"]
-DOT err_pipe [label="ERROR",shape=plaintext]
- */
-
-static int
-cnt_pipe(struct sess *sp)
-{
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
-
-	sp->wrk->acct_tmp.pipe++;
-	WS_Reset(sp->wrk->ws, NULL);
-	http_Setup(sp->wrk->bereq, sp->wrk->ws);
-	http_FilterHeader(sp, HTTPH_R_PIPE);
-
-	VCL_pipe_method(sp);
-
-	if (sp->handling == VCL_RET_ERROR)
-		INCOMPL();
-	assert(sp->handling == VCL_RET_PIPE);
-
-	PipeSession(sp);
-	assert(WRW_IsReleased(sp->wrk));
-	http_Setup(sp->wrk->bereq, NULL);
-	sp->step = STP_DONE;
-	return (0);
-}
-
-/*--------------------------------------------------------------------
- * RECV
- * We have a complete request, set everything up and start it.
- *
-DOT subgraph xcluster_recv {
-DOT	recv [
-DOT		shape=record
-DOT		label="vcl_recv()|req."
-DOT	]
-DOT }
-DOT RESTART -> recv
-DOT recv -> pipe [label="pipe",style=bold,color=orange]
-DOT recv -> pass2 [label="pass",style=bold,color=red]
-DOT recv -> err_recv [label="error"]
-DOT err_recv [label="ERROR",shape=plaintext]
-DOT recv -> hash [label="lookup",style=bold,color=green]
- */
-
-static int
-cnt_recv(struct sess *sp)
-{
-	unsigned recv_handling;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
-	AZ(sp->obj);
-	assert(sp->wrk->wrw.ciov == sp->wrk->wrw.siov);
-
-	/* By default we use the first backend */
-	AZ(sp->director);
-	sp->director = sp->vcl->director[0];
-	AN(sp->director);
-
-	sp->disable_esi = 0;
-	sp->hash_always_miss = 0;
-	sp->hash_ignore_busy = 0;
-	sp->client_identity = NULL;
-
-	http_CollectHdr(sp->http, H_Cache_Control);
-
-	VCL_recv_method(sp);
-	recv_handling = sp->handling;
-
-	if (sp->restarts >= cache_param->max_restarts) {
-		if (sp->err_code == 0)
-			sp->err_code = 503;
-		sp->step = STP_ERROR;
-		return (0);
-	}
-
-	/* Zap these, in case we came here through restart */
-	sp->wrk->do_esi = 0;
-	sp->wrk->is_gzip = 0;
-	sp->wrk->is_gunzip = 0;
-	sp->wrk->do_gzip = 0;
-	sp->wrk->do_gunzip = 0;
-	sp->wrk->do_stream = 0;
-
-	if (cache_param->http_gzip_support &&
-	     (recv_handling != VCL_RET_PIPE) &&
-	     (recv_handling != VCL_RET_PASS)) {
-		if (RFC2616_Req_Gzip(sp)) {
-			http_Unset(sp->http, H_Accept_Encoding);
-			http_SetHeader(sp->wrk, sp->vsl_id, sp->http,
-			    "Accept-Encoding: gzip");
-		} else {
-			http_Unset(sp->http, H_Accept_Encoding);
-		}
-	}
-
-	SHA256_Init(sp->wrk->sha256ctx);
-	VCL_hash_method(sp);
-	assert(sp->handling == VCL_RET_HASH);
-	SHA256_Final(sp->digest, sp->wrk->sha256ctx);
-
-	if (!strcmp(sp->http->hd[HTTP_HDR_REQ].b, "HEAD"))
-		sp->wantbody = 0;
-	else
-		sp->wantbody = 1;
-
-	sp->sendbody = 0;
-	switch(recv_handling) {
-	case VCL_RET_LOOKUP:
-		/* XXX: discard req body, if any */
-		sp->step = STP_LOOKUP;
-		return (0);
-	case VCL_RET_PIPE:
-		if (sp->esi_level > 0) {
-			/* XXX: VSL something */
-			INCOMPL();
-			/* sp->step = STP_DONE; */
-			return (1);
-		}
-		sp->step = STP_PIPE;
-		return (0);
-	case VCL_RET_PASS:
-		sp->step = STP_PASS;
-		return (0);
-	case VCL_RET_ERROR:
-		/* XXX: discard req body, if any */
-		sp->step = STP_ERROR;
-		return (0);
-	default:
-		WRONG("Illegal action in vcl_recv{}");
-	}
-}
-
-/*--------------------------------------------------------------------
- * START
- * Handle a request, wherever it came from recv/restart.
- *
-DOT start [shape=box,label="Dissect request"]
-DOT start -> recv [style=bold,color=green]
- */
-
-static int
-cnt_start(struct sess *sp)
-{
-	uint16_t done;
-	char *p;
-	const char *r = "HTTP/1.1 100 Continue\r\n\r\n";
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	AZ(sp->restarts);
-	AZ(sp->obj);
-	AZ(sp->vcl);
-
-	/* Update stats of various sorts */
-	sp->wrk->stats.client_req++;
-	sp->t_req = VTIM_real();
-	sp->wrk->lastused = sp->t_req;
-	sp->wrk->acct_tmp.req++;
-
-	/* Assign XID and log */
-	sp->xid = ++xids;				/* XXX not locked */
-	WSP(sp, SLT_ReqStart, "%s %s %u", sp->addr, sp->port,  sp->xid);
-
-	/* Borrow VCL reference from worker thread */
-	VCL_Refresh(&sp->wrk->vcl);
-	sp->vcl = sp->wrk->vcl;
-	sp->wrk->vcl = NULL;
-
-	http_Setup(sp->http, sp->ws);
-	done = http_DissectRequest(sp);
-
-	/* If we could not even parse the request, just close */
-	if (done == 400) {
-		sp->step = STP_DONE;
-		SES_Close(sp, "junk");
-		return (0);
-	}
-
-	/* Catch request snapshot */
-	sp->ws_req = WS_Snapshot(sp->ws);
-
-	/* Catch original request, before modification */
-	HTTP_Copy(sp->http0, sp->http);
-
-	if (done != 0) {
-		sp->err_code = done;
-		sp->step = STP_ERROR;
-		return (0);
-	}
-
-	sp->doclose = http_DoConnection(sp->http);
-
-	/* XXX: Handle TRACE & OPTIONS of Max-Forwards = 0 */
-
-	/*
-	 * Handle Expect headers
-	 */
-	if (http_GetHdr(sp->http, H_Expect, &p)) {
-		if (strcasecmp(p, "100-continue")) {
-			sp->err_code = 417;
-			sp->step = STP_ERROR;
-			return (0);
-		}
-
-		/* XXX: Don't bother with write failures for now */
-		(void)write(sp->fd, r, strlen(r));
-		/* XXX: When we do ESI includes, this is not removed
-		 * XXX: because we use http0 as our basis.  Believed
-		 * XXX: safe, but potentially confusing.
-		 */
-		http_Unset(sp->http, H_Expect);
-	}
-
-	sp->step = STP_RECV;
-	return (0);
-}
-
-/*--------------------------------------------------------------------
- * Central state engine dispatcher.
- *
- * Kick the session around until it has had enough.
- *
- */
-
-static void
-cnt_diag(struct sess *sp, const char *state)
-{
-	if (sp->wrk != NULL) {
-		WSP(sp, SLT_Debug, "thr %p STP_%s sp %p obj %p vcl %p",
-		    pthread_self(), state, sp, sp->obj, sp->vcl);
-		WSL_Flush(sp->wrk, 0);
-	} else {
-		VSL(SLT_Debug, sp->vsl_id,
-		    "thr %p STP_%s sp %p obj %p vcl %p",
-		    pthread_self(), state, sp, sp->obj, sp->vcl);
-	}
-}
-
-void
-CNT_Session(struct sess *sp)
-{
-	int done;
-	struct worker *w;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	w = sp->wrk;
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-
-	/*
-	 * Possible entrance states
-	 */
-	assert(
-	    sp->step == STP_FIRST ||
-	    sp->step == STP_START ||
-	    sp->step == STP_LOOKUP ||
-	    sp->step == STP_RECV);
-
-	AZ(w->do_stream);
-	AZ(w->is_gzip);
-	AZ(w->do_gzip);
-	AZ(w->is_gunzip);
-	AZ(w->do_gunzip);
-	AZ(w->do_esi);
-
-	/*
-	 * Whenever we come in from the acceptor or waiter, we need to set
-	 * blocking mode, but there is no point in setting it when we come from
-	 * ESI or when a parked sessions returns.
-	 * It would be simpler to do this in the acceptor or waiter, but we'd
-	 * rather do the syscall in the worker thread.
-	 * On systems which return errors for ioctl, we close early
-	 */
-	if ((sp->step == STP_FIRST || sp->step == STP_START) &&
-	    VTCP_blocking(sp->fd)) {
-		if (errno == ECONNRESET)
-			SES_Close(sp, "remote closed");
-		else
-			SES_Close(sp, "error");
-		sp->step = STP_DONE;
-	}
-
-	/*
-	 * NB: Once done is set, we can no longer touch sp!
-	 */
-	for (done = 0; !done; ) {
-		assert(sp->wrk == w);
-		/*
-		 * This is a good place to be paranoid about the various
-		 * pointers still pointing to the things we expect.
-		 */
-		CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-		CHECK_OBJ_ORNULL(sp->obj, OBJECT_MAGIC);
-		CHECK_OBJ_NOTNULL(sp->wrk, WORKER_MAGIC);
-		CHECK_OBJ_ORNULL(w->nobjhead, OBJHEAD_MAGIC);
-		WS_Assert(w->ws);
-
-		switch (sp->step) {
-#define STEP(l,u) \
-		    case STP_##u: \
-			if (cache_param->diag_bitmap & 0x01) \
-				cnt_diag(sp, #u); \
-			done = cnt_##l(sp); \
-		        break;
-#include "tbl/steps.h"
-#undef STEP
-		default:
-			WRONG("State engine misfire");
-		}
-		WS_Assert(w->ws);
-		CHECK_OBJ_ORNULL(w->nobjhead, OBJHEAD_MAGIC);
-	}
-	WSL_Flush(w, 0);
-	AZ(w->do_stream);
-	AZ(w->is_gzip);
-	AZ(w->do_gzip);
-	AZ(w->is_gunzip);
-	AZ(w->do_gunzip);
-	AZ(w->do_esi);
-#define ACCT(foo)	AZ(w->acct_tmp.foo);
-#include "tbl/acct_fields.h"
-#undef ACCT
-	assert(WRW_IsReleased(w));
-}
-
-/*
-DOT }
-*/
-
-/*--------------------------------------------------------------------
- * Debugging aids
- */
-
-static void
-cli_debug_xid(struct cli *cli, const char * const *av, void *priv)
-{
-	(void)priv;
-	if (av[2] != NULL)
-		xids = strtoul(av[2], NULL, 0);
-	VCLI_Out(cli, "XID is %u", xids);
-}
-
-/*
- * Default to seed=1, this is the only seed value POSIXl guarantees will
- * result in a reproducible random number sequence.
- */
-static void
-cli_debug_srandom(struct cli *cli, const char * const *av, void *priv)
-{
-	(void)priv;
-	unsigned seed = 1;
-
-	if (av[2] != NULL)
-		seed = strtoul(av[2], NULL, 0);
-	srandom(seed);
-	srand48(random());
-	VCLI_Out(cli, "Random(3) seeded with %lu", seed);
-}
-
-static struct cli_proto debug_cmds[] = {
-	{ "debug.xid", "debug.xid",
-		"\tExamine or set XID\n", 0, 1, "d", cli_debug_xid },
-	{ "debug.srandom", "debug.srandom",
-		"\tSeed the random(3) function\n", 0, 1, "d", cli_debug_srandom },
-	{ NULL }
-};
-
-/*--------------------------------------------------------------------
- *
- */
-
-void
-CNT_Init(void)
-{
-
-	srandomdev();
-	srand48(random());
-	xids = random();
-	CLI_AddFuncs(debug_cmds);
-}
-
-
diff --git a/bin/varnishd/cache_cli.c b/bin/varnishd/cache_cli.c
deleted file mode 100644
index f29f86a..0000000
--- a/bin/varnishd/cache_cli.c
+++ /dev/null
@@ -1,243 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * Caching process CLI handling.
- *
- * We only have one CLI source, the stdin/stdout pipes from the manager
- * process, but we complicate things by having undocumented commands that
- * we do not want to show in a plain help, and by having commands that the
- * manager has already shown in help before asking us.
- */
-
-#include "config.h"
-
-#include <stddef.h>			// offsetof
-
-#include "cache.h"
-#include "common/heritage.h"
-
-#include "cache_backend.h"		// struct vbc
-#include "hash/hash_slinger.h"		// struct objhead
-#include "vcli.h"
-#include "vcli_common.h"
-#include "vcli_priv.h"
-#include "vcli_serve.h"
-
-pthread_t		cli_thread;
-static struct lock	cli_mtx;
-static int		add_check;
-static struct VCLS	*cls;
-
-/*
- * The CLI commandlist is split in three:
- *  - Commands we get from/share with the manager, we don't show these
- *	in help, as the manager already did that.
- *  - Cache process commands, show in help
- *  - Undocumented debug commands, show in undocumented "help -d"
- */
-
-/*--------------------------------------------------------------------
- * Add CLI functions to the appropriate command set
- */
-
-void
-CLI_AddFuncs(struct cli_proto *p)
-{
-
-	AZ(add_check);
-	Lck_Lock(&cli_mtx);
-	AZ(VCLS_AddFunc(cls, 0, p));
-	Lck_Unlock(&cli_mtx);
-}
-
-static void
-cli_cb_before(const struct cli *cli)
-{
-
-	ASSERT_CLI();
-	VSL(SLT_CLI, 0, "Rd %s", cli->cmd);
-	VCL_Poll();
-	VBE_Poll();
-	Lck_Lock(&cli_mtx);
-}
-
-static void
-cli_cb_after(const struct cli *cli)
-{
-
-	ASSERT_CLI();
-	Lck_Unlock(&cli_mtx);
-	VSL(SLT_CLI, 0, "Wr %03u %u %s",
-	    cli->result, VSB_len(cli->sb), VSB_data(cli->sb));
-}
-
-void
-CLI_Run(void)
-{
-	int i;
-
-	add_check = 1;
-
-	AN(VCLS_AddFd(cls, heritage.cli_in, heritage.cli_out, NULL, NULL));
-
-	do {
-		i = VCLS_Poll(cls, -1);
-	} while(i > 0);
-	VSL(SLT_CLI, 0, "EOF on CLI connection, worker stops");
-	VCA_Shutdown();
-}
-
-/*--------------------------------------------------------------------*/
-
-static void
-cli_debug_sizeof(struct cli *cli, const char * const *av, void *priv)
-{
-	(void)av;
-	(void)priv;
-
-#define SZOF(foo)       VCLI_Out(cli, \
-    "sizeof(%s) = %zd = 0x%zx\n", #foo, sizeof(foo), sizeof(foo))
-	SZOF(struct ws);
-	SZOF(struct http);
-	SZOF(struct http_conn);
-	SZOF(struct acct);
-	SZOF(struct worker);
-	SZOF(struct storage);
-	SZOF(struct object);
-	SZOF(struct objcore);
-	SZOF(struct objhead);
-	SZOF(struct sess);
-	SZOF(struct vbc);
-	SZOF(struct VSC_C_main);
-	SZOF(struct lock);
-#if 0
-#define OFOF(foo, bar)	{ foo __foo; VCLI_Out(cli, \
-    "%-30s = 0x%4zx @ 0x%4zx\n", \
-	#foo "." #bar, sizeof(__foo.bar), offsetof(foo, bar)); }
-#if 0
-	OFOF(struct objhead, magic);
-	OFOF(struct objhead, refcnt);
-	OFOF(struct objhead, mtx);
-	OFOF(struct objhead, objcs);
-	OFOF(struct objhead, digest);
-	OFOF(struct objhead, waitinglist);
-	OFOF(struct objhead, _u);
-#endif
-#if 0
-	OFOF(struct http, magic);
-	OFOF(struct http, logtag);
-	OFOF(struct http, ws);
-	OFOF(struct http, hd);
-	OFOF(struct http, hdf);
-	OFOF(struct http, shd);
-	OFOF(struct http, nhd);
-	OFOF(struct http, status);
-	OFOF(struct http, protover);
-	OFOF(struct http, conds);
-#endif
-#if 0
-	OFOF(struct storage, magic);
-	OFOF(struct storage, fd);
-	OFOF(struct storage, where);
-	OFOF(struct storage, list);
-	OFOF(struct storage, stevedore);
-	OFOF(struct storage, priv);
-	OFOF(struct storage, ptr);
-	OFOF(struct storage, len);
-	OFOF(struct storage, space);
-#endif
-#if 0
-	OFOF(struct object, magic);
-	OFOF(struct object, xid);
-	OFOF(struct object, objstore);
-	OFOF(struct object, objcore);
-	OFOF(struct object, ws_o);
-	OFOF(struct object, vary);
-	OFOF(struct object, hits);
-	OFOF(struct object, response);
-	OFOF(struct object, gziped);
-	OFOF(struct object, gzip_start);
-	OFOF(struct object, gzip_last);
-	OFOF(struct object, gzip_stop);
-	OFOF(struct object, len);
-	OFOF(struct object, age);
-	OFOF(struct object, entered);
-	OFOF(struct object, exp);
-	OFOF(struct object, last_modified);
-	OFOF(struct object, last_lru);
-	OFOF(struct object, http);
-	OFOF(struct object, store);
-	OFOF(struct object, esidata);
-	OFOF(struct object, last_use);
-#endif
-#undef OFOF
-#endif
-}
-
-/*--------------------------------------------------------------------*/
-
-static void
-ccf_panic(struct cli *cli, const char * const *av, void *priv)
-{
-
-	(void)cli;
-	(void)av;
-	(void)priv;
-	assert(!strcmp("", "You asked for it"));
-}
-
-/*--------------------------------------------------------------------*/
-
-static struct cli_proto master_cmds[] = {
-	{ CLI_PING,		"i", VCLS_func_ping },
-	{ CLI_HELP,             "i", VCLS_func_help },
-	{ "debug.sizeof", "debug.sizeof",
-		"\tDump sizeof various data structures\n",
-		0, 0, "d", cli_debug_sizeof },
-	{ "debug.panic.worker", "debug.panic.worker",
-		"\tPanic the worker process.\n",
-		0, 0, "d", ccf_panic },
-	{ NULL }
-};
-
-/*--------------------------------------------------------------------
- * Initialize the CLI subsystem
- */
-
-void
-CLI_Init(void)
-{
-
-	Lck_New(&cli_mtx, lck_cli);
-	cli_thread = pthread_self();
-
-	cls = VCLS_New(cli_cb_before, cli_cb_after, cache_param->cli_buffer);
-	AN(cls);
-
-	CLI_AddFuncs(master_cmds);
-}
diff --git a/bin/varnishd/cache_dir.c b/bin/varnishd/cache_dir.c
deleted file mode 100644
index d4794f9..0000000
--- a/bin/varnishd/cache_dir.c
+++ /dev/null
@@ -1,121 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * Handle backend connections and backend request structures.
- *
- */
-
-#include "config.h"
-
-#include "cache.h"
-
-#include "cache_backend.h"
-#include "vtcp.h"
-
-/* Close a connection ------------------------------------------------*/
-
-void
-VDI_CloseFd(struct worker *wrk)
-{
-	struct backend *bp;
-
-	CHECK_OBJ_NOTNULL(wrk->vbc, VBC_MAGIC);
-	CHECK_OBJ_NOTNULL(wrk->vbc->backend, BACKEND_MAGIC);
-	assert(wrk->vbc->fd >= 0);
-
-	bp = wrk->vbc->backend;
-
-	WSL(wrk, SLT_BackendClose, wrk->vbc->vsl_id, "%s", bp->display_name);
-
-	/* Checkpoint log to flush all info related to this connection
-	   before the OS reuses the FD */
-	WSL_Flush(wrk, 0);
-
-	VTCP_close(&wrk->vbc->fd);
-	VBE_DropRefConn(bp);
-	wrk->vbc->backend = NULL;
-	VBE_ReleaseConn(wrk->vbc);
-	wrk->vbc = NULL;
-	wrk->do_close = 0;
-}
-
-/* Recycle a connection ----------------------------------------------*/
-
-void
-VDI_RecycleFd(struct worker *wrk)
-{
-	struct backend *bp;
-
-	CHECK_OBJ_NOTNULL(wrk->vbc, VBC_MAGIC);
-	CHECK_OBJ_NOTNULL(wrk->vbc->backend, BACKEND_MAGIC);
-	assert(wrk->vbc->fd >= 0);
-	AZ(wrk->do_close);
-
-	bp = wrk->vbc->backend;
-
-	WSL(wrk, SLT_BackendReuse, wrk->vbc->vsl_id, "%s", bp->display_name);
-	/*
-	 * Flush the shmlog, so that another session reusing this backend
-	 * will log chronologically later than our use of it.
-	 */
-	WSL_Flush(wrk, 0);
-	Lck_Lock(&bp->mtx);
-	VSC_C_main->backend_recycle++;
-	VTAILQ_INSERT_HEAD(&bp->connlist, wrk->vbc, list);
-	wrk->vbc = NULL;
-	VBE_DropRefLocked(bp);
-}
-
-/* Get a connection --------------------------------------------------*/
-
-struct vbc *
-VDI_GetFd(const struct director *d, struct sess *sp)
-{
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	if (d == NULL)
-		d = sp->director;
-	CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC);
-	return (d->getfd(d, sp));
-}
-
-/* Check health ------------------------------------------------------
- *
- * The target is really an objhead pointer, but since it can not be
- * dereferenced during health-checks, we pass it as uintptr_t, which
- * hopefully will make people investigate, before mucking about with it.
- */
-
-int
-VDI_Healthy(const struct director *d, const struct sess *sp)
-{
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC);
-	return (d->healthy(d, sp));
-}
diff --git a/bin/varnishd/cache_dir_dns.c b/bin/varnishd/cache_dir_dns.c
deleted file mode 100644
index ae96dbb..0000000
--- a/bin/varnishd/cache_dir_dns.c
+++ /dev/null
@@ -1,469 +0,0 @@
-/*-
- * Copyright (c) 2009-2010 Varnish Software AS
- * All rights reserved.
- *
- * Author: Kristian Lyngstol <kristian at redpill-linpro.com>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- */
-
-#include "config.h"
-
-#include <netinet/in.h>
-
-#include <netdb.h>
-#include <stdio.h>
-#include <stdlib.h>
-
-#include "cache.h"
-
-#include "cache_backend.h"
-#include "vrt.h"
-
-/*--------------------------------------------------------------------*/
-
-/* FIXME: Should eventually be a configurable variable. */
-#define VDI_DNS_MAX_CACHE		1024
-#define VDI_DNS_GROUP_MAX_BACKENDS	1024
-
-/* DNS Cache entry
- */
-struct vdi_dns_hostgroup {
-	unsigned			magic;
-#define VDI_DNSDIR_MAGIC		0x1bacab21
-	char				*hostname;
-	struct director			*hosts[VDI_DNS_GROUP_MAX_BACKENDS];
-	unsigned			nhosts;
-	unsigned			next_host; /* Next to use...*/
-	double				ttl;
-	VTAILQ_ENTRY(vdi_dns_hostgroup)	list;
-};
-
-struct vdi_dns {
-	unsigned			magic;
-#define VDI_DNS_MAGIC			0x1337a178
-	struct director			dir;
-	struct director			**hosts;
-	unsigned			nhosts;
-	VTAILQ_HEAD(_cachelist,vdi_dns_hostgroup)	cachelist;
-	unsigned			ncachelist;
-	pthread_rwlock_t		rwlock;
-	const char			*suffix;
-	double			ttl;
-};
-
-/* Compare an IPv4 backend to a IPv4 addr/len */
-static int
-vdi_dns_comp_addrinfo4(const struct backend *bp,
-		       const struct sockaddr_storage *addr,
-		       const socklen_t len)
-{
-	uint32_t u, p;
-	const struct sockaddr_in *bps = (const void *)bp->ipv4;
-	const struct sockaddr_in *bpd = (const void *)addr;
-
-	if (bp->ipv4len != len || len <= 0)
-		return (0);
-
-	u = bpd->sin_addr.s_addr;
-	p = bps->sin_addr.s_addr;
-
-	return (u == p);
-}
-
-/* Compare an IPv6 backend to a IPv6 addr/len */
-static int
-vdi_dns_comp_addrinfo6(const struct backend *bp,
-		       const struct sockaddr_storage *addr,
-		       const socklen_t len)
-{
-	const uint8_t *u, *p;
-	const struct sockaddr_in6 *bps = (const void *)bp->ipv6;
-	const struct sockaddr_in6 *bpd = (const void *)addr;
-
-	if (bp->ipv6len != len || len <= 0)
-		return (0);
-
-	u = bpd->sin6_addr.s6_addr;
-	p = bps->sin6_addr.s6_addr;
-
-	return (!memcmp(u, p, 16));
-}
-
-/* Check if a backends socket is the same as addr */
-static int
-vdi_dns_comp_addrinfo(const struct director *dir,
-		      const struct sockaddr_storage *addr,
-		      const socklen_t len)
-{
-	struct backend *bp;
-
-	bp = vdi_get_backend_if_simple(dir);
-	AN(bp);
-	if (addr->ss_family == PF_INET && bp->ipv4) {
-		return (vdi_dns_comp_addrinfo4(bp, addr, len));
-	} else if (addr->ss_family == PF_INET6 && bp->ipv6) {
-		return (vdi_dns_comp_addrinfo6(bp, addr, len));
-	}
-	return (0);
-}
-
-/* Pick a host from an existing hostgroup.
- * Balance on round-robin if multiple backends are available and only pick
- * healthy ones.
- */
-static struct director *
-vdi_dns_pick_host(const struct sess *sp, struct vdi_dns_hostgroup *group) {
-	int initial, i, nhosts, current;
-	if (group->nhosts == 0)
-		return (NULL); // In case of error.
-	if (group->next_host >= group->nhosts)
-		group->next_host = 0;
-
-	/* Pick a healthy backend */
-	initial = group->next_host;
-	nhosts = group->nhosts;
-	for (i=0; i < nhosts; i++) {
-		if (i + initial >= nhosts)
-			current = i + initial - nhosts;
-		else
-			current = i + initial;
-		if (VDI_Healthy(group->hosts[current], sp)) {
-			group->next_host = current+1;
-			return (group->hosts[current]);
-		}
-	}
-
-	return (NULL);
-}
-
-/* Remove an item from the dns cache.
- * If *group is NULL, the head is popped.
- * Remember locking.
- */
-static void
-vdi_dns_pop_cache(struct vdi_dns *vs,
-		  struct vdi_dns_hostgroup *group)
-{
-	if (group == NULL)
-		group = VTAILQ_LAST( &vs->cachelist, _cachelist );
-	assert(group != NULL);
-	free(group->hostname);
-	VTAILQ_REMOVE(&vs->cachelist, group, list);
-	FREE_OBJ(group);
-	vs->ncachelist--;
-}
-
-/* Dummy in case someone feels like optimizing it? meh...
- */
-static inline int
-vdi_dns_groupmatch(const struct vdi_dns_hostgroup *group, const char *hostname)
-{
-	return (!strcmp(group->hostname, hostname));
-}
-
-/* Search the cache for 'hostname' and put a backend-pointer as necessary,
- * return true for cache hit. This could still be a NULL backend if we did
- * a lookup earlier and didn't find a host (ie: cache failed too)
- *
- * if rwlock is true, the first timed out object found (if any) is popped
- * and freed.
- */
-static int
-vdi_dns_cache_has(const struct sess *sp,
-		  struct vdi_dns *vs,
-		  const char *hostname,
-		  struct director **backend,
-		  int rwlock)
-{
-	struct director *ret;
-	struct vdi_dns_hostgroup *hostgr;
-	struct vdi_dns_hostgroup *hostgr2;
-	VTAILQ_FOREACH_SAFE(hostgr, &vs->cachelist, list, hostgr2) {
-		CHECK_OBJ_NOTNULL(hostgr, VDI_DNSDIR_MAGIC);
-		if (hostgr->ttl <= sp->t_req) {
-			if (rwlock)
-				vdi_dns_pop_cache(vs, hostgr);
-			return (0);
-		}
-		if (vdi_dns_groupmatch(hostgr, hostname)) {
-			ret = (vdi_dns_pick_host(sp, hostgr));
-			*backend = ret;
-			if (*backend != NULL)
-				CHECK_OBJ_NOTNULL(*backend, DIRECTOR_MAGIC);
-			return (1);
-		}
-	}
-	return (0);
-}
-
-/* Add a newly cached item to the dns cache list.
- * (Sorry for the list_add/_add confusion...)
- */
-static void
-vdi_dns_cache_list_add(const struct sess *sp,
-		       struct vdi_dns *vs,
-		       struct vdi_dns_hostgroup *new)
-{
-	if (vs->ncachelist >= VDI_DNS_MAX_CACHE) {
-		VSC_C_main->dir_dns_cache_full++;
-		vdi_dns_pop_cache(vs, NULL);
-	}
-	CHECK_OBJ_NOTNULL(new, VDI_DNSDIR_MAGIC);
-	assert(new->hostname != 0);
-	new->ttl = sp->t_req + vs->ttl;
-	VTAILQ_INSERT_HEAD(&vs->cachelist, new, list);
-	vs->ncachelist++;
-}
-
-/* Add an item to the dns cache.
- * XXX: Might want to factor the getaddrinfo() out of the lock and do the
- * cache_has() afterwards to do multiple dns lookups in parallel...
- */
-static int
-vdi_dns_cache_add(const struct sess *sp,
-		  struct vdi_dns *vs,
-		  const char *hostname,
-		  struct director **backend)
-{
-	int error, i, host = 0;
-	struct addrinfo *res0, *res, hint;
-	struct vdi_dns_hostgroup *new;
-
-	/* Due to possible race while upgrading the lock, we have to
-	 * recheck if the result is already looked up. The overhead for
-	 * this is insignificant unless dns isn't cached properly (all
-	 * unique names or something equally troublesome).
-	 */
-
-	if (vdi_dns_cache_has(sp, vs, hostname, backend, 1))
-		return (1);
-
-	memset(&hint, 0, sizeof hint);
-	hint.ai_family = PF_UNSPEC;
-	hint.ai_socktype = SOCK_STREAM;
-
-	ALLOC_OBJ(new, VDI_DNSDIR_MAGIC);
-	XXXAN(new);
-
-	REPLACE(new->hostname, hostname);
-
-	error = getaddrinfo(hostname, "80", &hint, &res0);
-	VSC_C_main->dir_dns_lookups++;
-	if (error) {
-		vdi_dns_cache_list_add(sp, vs, new);
-		VSC_C_main->dir_dns_failed++;
-		return (0);
-	}
-
-	for (res = res0; res; res = res->ai_next) {
-		if (res->ai_family != PF_INET && res->ai_family != PF_INET6)
-			continue;
-
-		for (i = 0; i < vs->nhosts; i++) {
-			struct sockaddr_storage ss_hack;
-			memcpy(&ss_hack, res->ai_addr, res->ai_addrlen);
-			if (vdi_dns_comp_addrinfo(vs->hosts[i],
-			    &ss_hack, res->ai_addrlen)) {
-				new->hosts[host] = vs->hosts[i];
-				CHECK_OBJ_NOTNULL(new->hosts[host],
-				    DIRECTOR_MAGIC);
-				host++;
-			}
-		}
-	}
-	freeaddrinfo(res0);
-
-	new->nhosts = host;
-	vdi_dns_cache_list_add(sp, vs, new);
-	*backend = vdi_dns_pick_host(sp, new);
-	return (1);
-}
-
-/* Walk through the cached lookups looking for the relevant host, add one
- * if it isn't already cached.
- *
- * Returns a backend or NULL.
- */
-static struct director *
-vdi_dns_walk_cache(const struct sess *sp,
-		   struct vdi_dns *vs,
-		   const char *hostname)
-{
-	struct director *backend = NULL;
-	int ret;
-
-	AZ(pthread_rwlock_rdlock(&vs->rwlock));
-	ret = vdi_dns_cache_has(sp, vs, hostname, &backend, 0);
-	AZ(pthread_rwlock_unlock(&vs->rwlock));
-	if (!ret) {
-		/*
-		 * XXX: Isn't there a race here where another thread
-		 * XXX: could grab the lock and add it before we do ?
-		 * XXX: Should 'ret' be checked for that ?
-		 */
-		AZ(pthread_rwlock_wrlock(&vs->rwlock));
-		ret = vdi_dns_cache_add(sp, vs, hostname, &backend);
-		AZ(pthread_rwlock_unlock(&vs->rwlock));
-	} else
-		VSC_C_main->dir_dns_hit++;
-
-	/* Bank backend == cached a failure, so to speak */
-	if (backend != NULL)
-		CHECK_OBJ_NOTNULL(backend, DIRECTOR_MAGIC);
-	return (backend);
-}
-
-/* Parses the Host:-header and heads out to find a backend.
- */
-static struct director *
-vdi_dns_find_backend(const struct sess *sp, struct vdi_dns *vs)
-{
-	struct director *ret;
-	struct http *hp;
-	char *p, *q;
-	char hostname[NI_MAXHOST];
-
-	/* bereq is only present after recv et. al, otherwise use req (ie:
-	 * use req for health checks in vcl_recv and such).
-	 */
-	if (sp->wrk->bereq)
-		hp = sp->wrk->bereq;
-	else
-		hp = sp->http;
-
-
-	CHECK_OBJ_NOTNULL(hp, HTTP_MAGIC);
-	if (http_GetHdr(hp, H_Host, &p) == 0)
-		return (NULL);
-
-	q = strchr(p, ':');
-	if (q == NULL)
-		q = strchr(p, '\0');
-	AN(q);
-
-	bprintf(hostname, "%.*s%s", (int)(q - p), p,
-	    vs->suffix ? vs->suffix : "");
-
-	ret = vdi_dns_walk_cache(sp, vs, hostname);
-	return (ret);
-}
-
-static struct vbc *
-vdi_dns_getfd(const struct director *director, struct sess *sp)
-{
-	struct vdi_dns *vs;
-	struct director *dir;
-	struct vbc *vbe;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(director, DIRECTOR_MAGIC);
-	CAST_OBJ_NOTNULL(vs, director->priv, VDI_DNS_MAGIC);
-
-	dir = vdi_dns_find_backend(sp, vs);
-	if (!dir || !VDI_Healthy(dir, sp))
-		return (NULL);
-
-	vbe = VDI_GetFd(dir, sp);
-	return (vbe);
-}
-
-static unsigned
-vdi_dns_healthy(const struct director *dir, const struct sess *sp)
-{
-	/* XXX: Fooling -Werror for a bit until it's actually implemented.
-	 */
-	(void)dir;
-	(void)sp;
-	return (1);
-
-	/*
-	struct vdi_dns *vs;
-	struct director *dir;
-	int i;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->director, DIRECTOR_MAGIC);
-	CAST_OBJ_NOTNULL(vs, sp->director->priv, VDI_DNS_MAGIC);
-
-	dir = vdi_dns_find_backend(sp, vs);
-
-	if (dir)
-		return (1);
-	return (0);
-	*/
-}
-
-static void
-vdi_dns_fini(const struct director *d)
-{
-	struct vdi_dns *vs;
-
-	CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC);
-	CAST_OBJ_NOTNULL(vs, d->priv, VDI_DNS_MAGIC);
-
-	free(vs->hosts);
-	free(vs->dir.vcl_name);
-	vs->dir.magic = 0;
-	/* FIXME: Free the cache */
-	AZ(pthread_rwlock_destroy(&vs->rwlock));
-	FREE_OBJ(vs);
-}
-
-void
-VRT_init_dir_dns(struct cli *cli, struct director **bp, int idx,
-    const void *priv)
-{
-	const struct vrt_dir_dns *t;
-	struct vdi_dns *vs;
-	const struct vrt_dir_dns_entry *te;
-	int i;
-
-	ASSERT_CLI();
-	(void)cli;
-	t = priv;
-	ALLOC_OBJ(vs, VDI_DNS_MAGIC);
-	XXXAN(vs);
-	vs->hosts = calloc(sizeof(struct director *), t->nmember);
-	XXXAN(vs->hosts);
-
-	vs->dir.magic = DIRECTOR_MAGIC;
-	vs->dir.priv = vs;
-	vs->dir.name = "dns";
-	REPLACE(vs->dir.vcl_name, t->name);
-	vs->dir.getfd = vdi_dns_getfd;
-	vs->dir.fini = vdi_dns_fini;
-	vs->dir.healthy = vdi_dns_healthy;
-
-	vs->suffix = t->suffix;
-	vs->ttl = t->ttl;
-
-	te = t->members;
-	for (i = 0; i < t->nmember; i++, te++)
-		vs->hosts[i] = bp[te->host];
-	vs->nhosts = t->nmember;
-	vs->ttl = t->ttl;
-	VTAILQ_INIT(&vs->cachelist);
-	AZ(pthread_rwlock_init(&vs->rwlock, NULL));
-	bp[idx] = &vs->dir;
-}
diff --git a/bin/varnishd/cache_dir_random.c b/bin/varnishd/cache_dir_random.c
deleted file mode 100644
index d6570ed..0000000
--- a/bin/varnishd/cache_dir_random.c
+++ /dev/null
@@ -1,285 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * This code is shared between the random, client and hash directors, because
- * they share the same properties and most of the same selection logic.
- *
- * The random director picks a backend on random.
- *
- * The hash director picks based on the hash from vcl_hash{}
- *
- * The client director picks based on client identity or IP-address
- *
- * In all cases, the choice is by weight of the healthy subset of
- * configured backends.
- *
- * Failures to get a connection are retried, here all three policies
- * fall back to a deterministically random choice, by weight in the
- * healthy subset.
- *
- */
-
-#include "config.h"
-
-#include <math.h>
-#include <stdlib.h>
-
-#include "cache.h"
-
-#include "cache_backend.h"
-#include "vend.h"
-#include "vrt.h"
-#include "vsha256.h"
-
-/*--------------------------------------------------------------------*/
-
-struct vdi_random_host {
-	struct director		*backend;
-	double			weight;
-};
-
-enum crit_e {c_random, c_hash, c_client};
-
-struct vdi_random {
-	unsigned		magic;
-#define VDI_RANDOM_MAGIC	0x3771ae23
-	struct director		dir;
-
-	enum crit_e		criteria;
-	unsigned		retries;
-	double			tot_weight;
-	struct vdi_random_host	*hosts;
-	unsigned		nhosts;
-};
-
-/*
- * Applies sha256 using the given context and input/length, and returns
- * a double in the range [0...1[ based on the hash.
- */
-static double
-vdi_random_sha(const char *input, ssize_t len)
-{
-	struct SHA256Context ctx;
-	uint8_t sign[SHA256_LEN];
-
-	AN(input);
-	SHA256_Init(&ctx);
-	SHA256_Update(&ctx, input, len);
-	SHA256_Final(sign, &ctx);
-	return (scalbn(vle32dec(sign), -32));
-}
-
-/*
- * Sets up the initial seed for picking a backend according to policy.
- */
-static double
-vdi_random_init_seed(const struct vdi_random *vs, const struct sess *sp)
-{
-	const char *p;
-	double retval;
-
-	switch (vs->criteria) {
-	case c_client:
-		if (sp->client_identity != NULL)
-			p = sp->client_identity;
-		else
-			p = sp->addr;
-		retval = vdi_random_sha(p, strlen(p));
-		break;
-	case c_hash:
-		AN(sp->digest);
-		retval = scalbn(vle32dec(sp->digest), -32);
-		break;
-	case c_random:
-	default:
-		retval = scalbn(random(), -31);
-		break;
-	}
-	return (retval);
-}
-
-/*
- * Find the healthy backend corresponding to the weight r [0...1[
- */
-static struct vbc *
-vdi_random_pick_one(struct sess *sp, const struct vdi_random *vs, double r)
-{
-	double w[vs->nhosts];
-	int i;
-	double s1;
-
-	assert(r >= 0.0 && r < 1.0);
-
-	memset(w, 0, sizeof w);
-	/* Sum up the weights of healty backends */
-	s1 = 0.0;
-	for (i = 0; i < vs->nhosts; i++) {
-		if (VDI_Healthy(vs->hosts[i].backend, sp))
-			w[i] = vs->hosts[i].weight;
-		s1 += w[i];
-	}
-
-	if (s1 == 0.0)
-		return (NULL);
-
-	r *= s1;
-	s1 = 0.0;
-	for (i = 0; i < vs->nhosts; i++)  {
-		s1 += w[i];
-		if (r < s1)
-			return(VDI_GetFd(vs->hosts[i].backend, sp));
-	}
-	return (NULL);
-}
-
-/*
- * Try the specified number of times to get a backend.
- * First one according to policy, after that, deterministically
- * random by rehashing the key.
- */
-static struct vbc *
-vdi_random_getfd(const struct director *d, struct sess *sp)
-{
-	int k;
-	struct vdi_random *vs;
-	double r;
-	struct vbc *vbe;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC);
-	CAST_OBJ_NOTNULL(vs, d->priv, VDI_RANDOM_MAGIC);
-
-	r = vdi_random_init_seed(vs, sp);
-
-	for (k = 0; k < vs->retries; k++) {
-		vbe = vdi_random_pick_one(sp, vs, r);
-		if (vbe != NULL)
-			return (vbe);
-		r = vdi_random_sha((void *)&r, sizeof(r));
-	}
-	return (NULL);
-}
-
-/*
- * Healthy if just a single backend is...
- */
-static unsigned
-vdi_random_healthy(const struct director *d, const struct sess *sp)
-{
-	struct vdi_random *vs;
-	int i;
-
-	CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC);
-	CAST_OBJ_NOTNULL(vs, d->priv, VDI_RANDOM_MAGIC);
-
-	for (i = 0; i < vs->nhosts; i++) {
-		if (VDI_Healthy(vs->hosts[i].backend, sp))
-			return (1);
-	}
-	return (0);
-}
-
-static void
-vdi_random_fini(const struct director *d)
-{
-	struct vdi_random *vs;
-
-	CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC);
-	CAST_OBJ_NOTNULL(vs, d->priv, VDI_RANDOM_MAGIC);
-
-	free(vs->hosts);
-	free(vs->dir.vcl_name);
-	vs->dir.magic = 0;
-	FREE_OBJ(vs);
-}
-
-static void
-vrt_init(struct cli *cli, struct director **bp, int idx,
-    const void *priv, enum crit_e criteria)
-{
-	const struct vrt_dir_random *t;
-	struct vdi_random *vs;
-	const struct vrt_dir_random_entry *te;
-	struct vdi_random_host *vh;
-	int i;
-
-	ASSERT_CLI();
-	(void)cli;
-	t = priv;
-
-	ALLOC_OBJ(vs, VDI_RANDOM_MAGIC);
-	XXXAN(vs);
-	vs->hosts = calloc(sizeof *vh, t->nmember);
-	XXXAN(vs->hosts);
-
-	vs->dir.magic = DIRECTOR_MAGIC;
-	vs->dir.priv = vs;
-	vs->dir.name = "random";
-	REPLACE(vs->dir.vcl_name, t->name);
-	vs->dir.getfd = vdi_random_getfd;
-	vs->dir.fini = vdi_random_fini;
-	vs->dir.healthy = vdi_random_healthy;
-
-	vs->criteria = criteria;
-	vs->retries = t->retries;
-	if (vs->retries == 0)
-		vs->retries = t->nmember;
-	vh = vs->hosts;
-	te = t->members;
-	vs->tot_weight = 0.;
-	for (i = 0; i < t->nmember; i++, vh++, te++) {
-		assert(te->weight > 0.0);
-		vh->weight = te->weight;
-		vs->tot_weight += vh->weight;
-		vh->backend = bp[te->host];
-		AN(vh->backend);
-	}
-	vs->nhosts = t->nmember;
-	bp[idx] = &vs->dir;
-}
-
-void
-VRT_init_dir_random(struct cli *cli, struct director **bp, int idx,
-    const void *priv)
-{
-	vrt_init(cli, bp, idx, priv, c_random);
-}
-
-void
-VRT_init_dir_hash(struct cli *cli, struct director **bp, int idx,
-    const void *priv)
-{
-	vrt_init(cli, bp, idx, priv, c_hash);
-}
-
-void
-VRT_init_dir_client(struct cli *cli, struct director **bp, int idx,
-    const void *priv)
-{
-	vrt_init(cli, bp, idx, priv, c_client);
-}
diff --git a/bin/varnishd/cache_dir_round_robin.c b/bin/varnishd/cache_dir_round_robin.c
deleted file mode 100644
index 7d75473..0000000
--- a/bin/varnishd/cache_dir_round_robin.c
+++ /dev/null
@@ -1,176 +0,0 @@
-/*-
- * Copyright (c) 2008-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Petter Knudsen <petter at linpro.no>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- */
-
-#include "config.h"
-
-#include <stdlib.h>
-
-#include "cache.h"
-
-#include "cache_backend.h"
-#include "vrt.h"
-
-/*--------------------------------------------------------------------*/
-
-struct vdi_round_robin_host {
-	struct director			*backend;
-};
-
-enum mode_e { m_round_robin, m_fallback };
-
-struct vdi_round_robin {
-	unsigned			magic;
-#define VDI_ROUND_ROBIN_MAGIC		0x2114a178
-	struct director			dir;
-	enum mode_e			mode;
-	struct vdi_round_robin_host	*hosts;
-	unsigned			nhosts;
-	unsigned			next_host;
-};
-
-static struct vbc *
-vdi_round_robin_getfd(const struct director *d, struct sess *sp)
-{
-	int i;
-	struct vdi_round_robin *vs;
-	struct director *backend;
-	struct vbc *vbe;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC);
-	CAST_OBJ_NOTNULL(vs, d->priv, VDI_ROUND_ROBIN_MAGIC);
-
-	/*
-	 * In fallback mode we ignore the next_host and always grab the
-	 * first healthy backend we can find.
-	 */
-	for (i = 0; i < vs->nhosts; i++) {
-		if (vs->mode == m_round_robin) {
-			backend = vs->hosts[vs->next_host].backend;
-			vs->next_host = (vs->next_host + 1) % vs->nhosts;
-		} else /* m_fallback */ {
-			backend = vs->hosts[i].backend;
-		}
-		if (!VDI_Healthy(backend, sp))
-			continue;
-		vbe = VDI_GetFd(backend, sp);
-		if (vbe != NULL)
-			return (vbe);
-	}
-
-	return (NULL);
-}
-
-static unsigned
-vdi_round_robin_healthy(const struct director *d, const struct sess *sp)
-{
-	struct vdi_round_robin *vs;
-	struct director *backend;
-	int i;
-
-	CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC);
-	CAST_OBJ_NOTNULL(vs, d->priv, VDI_ROUND_ROBIN_MAGIC);
-
-	for (i = 0; i < vs->nhosts; i++) {
-		backend = vs->hosts[i].backend;
-		if (VDI_Healthy(backend, sp))
-			return (1);
-	}
-	return (0);
-}
-
-static void
-vdi_round_robin_fini(const struct director *d)
-{
-	struct vdi_round_robin *vs;
-
-	CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC);
-	CAST_OBJ_NOTNULL(vs, d->priv, VDI_ROUND_ROBIN_MAGIC);
-
-	free(vs->hosts);
-	free(vs->dir.vcl_name);
-	vs->dir.magic = 0;
-	vs->next_host = 0;
-	FREE_OBJ(vs);
-}
-
-static void
-vrt_init_dir(struct cli *cli, struct director **bp, int idx,
-    const void *priv, enum mode_e mode)
-{
-	const struct vrt_dir_round_robin *t;
-	struct vdi_round_robin *vs;
-	const struct vrt_dir_round_robin_entry *te;
-	struct vdi_round_robin_host *vh;
-	int i;
-
-	ASSERT_CLI();
-	(void)cli;
-	t = priv;
-
-	ALLOC_OBJ(vs, VDI_ROUND_ROBIN_MAGIC);
-	XXXAN(vs);
-	vs->hosts = calloc(sizeof *vh, t->nmember);
-	XXXAN(vs->hosts);
-
-	vs->dir.magic = DIRECTOR_MAGIC;
-	vs->dir.priv = vs;
-	vs->dir.name = "round_robin";
-	REPLACE(vs->dir.vcl_name, t->name);
-	vs->dir.getfd = vdi_round_robin_getfd;
-	vs->dir.fini = vdi_round_robin_fini;
-	vs->dir.healthy = vdi_round_robin_healthy;
-
-	vs->mode = mode;
-	vh = vs->hosts;
-	te = t->members;
-	for (i = 0; i < t->nmember; i++, vh++, te++) {
-		vh->backend = bp[te->host];
-		AN (vh->backend);
-	}
-	vs->nhosts = t->nmember;
-	vs->next_host = 0;
-
-	bp[idx] = &vs->dir;
-}
-
-void
-VRT_init_dir_round_robin(struct cli *cli, struct director **bp, int idx,
-    const void *priv)
-{
-	vrt_init_dir(cli, bp, idx, priv, m_round_robin);
-}
-
-void
-VRT_init_dir_fallback(struct cli *cli, struct director **bp, int idx,
-    const void *priv)
-{
-	vrt_init_dir(cli, bp, idx, priv, m_fallback);
-}
-
diff --git a/bin/varnishd/cache_esi.h b/bin/varnishd/cache_esi.h
deleted file mode 100644
index ff84d41..0000000
--- a/bin/varnishd/cache_esi.h
+++ /dev/null
@@ -1,48 +0,0 @@
-/*-
- * Copyright (c) 2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- */
-
-#define	VEC_GZ	(0x21)
-#define	VEC_V1	(0x40 + 1)
-#define	VEC_V2	(0x40 + 2)
-#define	VEC_V8	(0x40 + 8)
-#define	VEC_C1	(0x50 + 1)
-#define	VEC_C2	(0x50 + 2)
-#define	VEC_C8	(0x50 + 8)
-#define	VEC_S1	(0x60 + 1)
-#define	VEC_S2	(0x60 + 2)
-#define	VEC_S8	(0x60 + 8)
-#define	VEC_INCL	'I'
-
-typedef ssize_t vep_callback_t(struct worker *w, ssize_t l, enum vgz_flag flg);
-
-void VEP_Init(struct worker *w, vep_callback_t *cb);
-void VEP_Parse(const struct worker *w, const char *p, size_t l);
-struct vsb *VEP_Finish(struct worker *w);
-
-
diff --git a/bin/varnishd/cache_esi_deliver.c b/bin/varnishd/cache_esi_deliver.c
deleted file mode 100644
index 4051027..0000000
--- a/bin/varnishd/cache_esi_deliver.c
+++ /dev/null
@@ -1,570 +0,0 @@
-/*-
- * Copyright (c) 2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * VED - Varnish Esi Delivery
- */
-
-#include "config.h"
-
-#include <stdio.h>
-#include <stdlib.h>
-
-#include "cache.h"
-
-#include "cache_esi.h"
-#include "vend.h"
-#include "vgz.h"
-
-/*--------------------------------------------------------------------*/
-
-static void
-ved_include(struct sess *sp, const char *src, const char *host)
-{
-	struct object *obj;
-	struct worker *w;
-	char *sp_ws_wm;
-	char *wrk_ws_wm;
-	unsigned sxid, res_mode;
-
-	w = sp->wrk;
-
-	if (sp->esi_level >= cache_param->max_esi_depth)
-		return;
-	sp->esi_level++;
-
-	(void)WRW_FlushRelease(w);
-
-	obj = sp->obj;
-	sp->obj = NULL;
-	res_mode = sp->wrk->res_mode;
-
-	/* Reset request to status before we started messing with it */
-	HTTP_Copy(sp->http, sp->http0);
-
-	/* Take a workspace snapshot */
-	sp_ws_wm = WS_Snapshot(sp->ws);
-	wrk_ws_wm = WS_Snapshot(w->ws);
-
-	http_SetH(sp->http, HTTP_HDR_URL, src);
-	if (host != NULL && *host != '\0')  {
-		http_Unset(sp->http, H_Host);
-		http_Unset(sp->http, H_If_Modified_Since);
-		http_SetHeader(w, sp->vsl_id, sp->http, host);
-	}
-	/*
-	 * XXX: We should decide if we should cache the director
-	 * XXX: or not (for session/backend coupling).  Until then
-	 * XXX: make sure we don't trip up the check in vcl_recv.
-	 */
-	sp->director = NULL;
-	sp->step = STP_RECV;
-	http_ForceGet(sp->http);
-
-	/* Don't do conditionals */
-	sp->http->conds = 0;
-	http_Unset(sp->http, H_If_Modified_Since);
-
-	/* Client content already taken care of */
-	http_Unset(sp->http, H_Content_Length);
-
-	sp->wrk->do_esi = 0;
-	sp->wrk->is_gzip = 0;
-	sp->wrk->is_gunzip = 0;
-	sp->wrk->do_gzip = 0;
-	sp->wrk->do_gunzip = 0;
-	sp->wrk->do_stream = 0;
-
-	sxid = sp->xid;
-	while (1) {
-		sp->wrk = w;
-		CNT_Session(sp);
-		if (sp->step == STP_DONE)
-			break;
-		AZ(sp->wrk);
-		WSL_Flush(w, 0);
-		DSL(0x20, SLT_Debug, sp->vsl_id, "loop waiting for ESI");
-		(void)usleep(10000);
-	}
-	sp->xid = sxid;
-	AN(sp->wrk);
-	assert(sp->step == STP_DONE);
-	sp->esi_level--;
-	sp->obj = obj;
-	sp->wrk->res_mode = res_mode;
-
-	/* Reset the workspace */
-	WS_Reset(sp->ws, sp_ws_wm);
-	WS_Reset(w->ws, wrk_ws_wm);
-
-	WRW_Reserve(sp->wrk, &sp->fd);
-	if (sp->wrk->res_mode & RES_CHUNKED)
-		WRW_Chunked(sp->wrk);
-}
-
-/*--------------------------------------------------------------------*/
-
-
-//#define Debug(fmt, ...) printf(fmt, __VA_ARGS__)
-#define Debug(fmt, ...) /**/
-
-static ssize_t
-ved_decode_len(uint8_t **pp)
-{
-	uint8_t *p;
-	ssize_t l;
-
-	p = *pp;
-	switch (*p & 15) {
-	case 1:
-		l = p[1];
-		p += 2;
-		break;
-	case 2:
-		l = vbe16dec(p + 1);
-		p += 3;
-		break;
-	case 8:
-		l = vbe64dec(p + 1);
-		p += 9;
-		break;
-	default:
-		printf("Illegal Length %d %d\n", *p, (*p & 15));
-		INCOMPL();
-	}
-	*pp = p;
-	assert(l > 0);
-	return (l);
-}
-
-/*---------------------------------------------------------------------
- * If a gzip'ed ESI object includes a ungzip'ed object, we need to make
- * it looked like a gzip'ed data stream.  The official way to do so would
- * be to fire up libvgz and gzip it, but we don't, we fake it.
- *
- * First, we cannot know if it is ungzip'ed on purpose, the admin may
- * know something we don't.
- *
- * What do you mean "BS ?"
- *
- * All right then...
- *
- * The matter of the fact is that we simply will not fire up a gzip in
- * the output path because it costs too much memory and CPU, so we simply
- * wrap the data in very convenient "gzip copy-blocks" and send it down
- * the stream with a bit more overhead.
- */
-
-static void
-ved_pretend_gzip(const struct sess *sp, const uint8_t *p, ssize_t l)
-{
-	uint8_t buf1[5], buf2[5];
-	uint16_t lx;
-
-	lx = 65535;
-	buf1[0] = 0;
-	vle16enc(buf1 + 1, lx);
-	vle16enc(buf1 + 3, ~lx);
-
-	while (l > 0) {
-		if (l >= 65535) {
-			lx = 65535;
-			(void)WRW_Write(sp->wrk, buf1, sizeof buf1);
-		} else {
-			lx = (uint16_t)l;
-			buf2[0] = 0;
-			vle16enc(buf2 + 1, lx);
-			vle16enc(buf2 + 3, ~lx);
-			(void)WRW_Write(sp->wrk, buf2, sizeof buf2);
-		}
-		(void)WRW_Write(sp->wrk, p, lx);
-		sp->wrk->crc = crc32(sp->wrk->crc, p, lx);
-		sp->wrk->l_crc += lx;
-		l -= lx;
-		p += lx;
-	}
-	/* buf2 is local, have to flush */
-	(void)WRW_Flush(sp->wrk);
-}
-
-/*---------------------------------------------------------------------
- */
-
-static const uint8_t gzip_hdr[] = {
-	0x1f, 0x8b, 0x08,
-	0x00, 0x00, 0x00, 0x00,
-	0x00,
-	0x02, 0x03
-};
-
-void
-ESI_Deliver(struct sess *sp)
-{
-	struct storage *st;
-	uint8_t *p, *e, *q, *r;
-	unsigned off;
-	ssize_t l, l2, l_icrc = 0;
-	uint32_t icrc = 0;
-	uint8_t tailbuf[8 + 5];
-	int isgzip;
-	struct vgz *vgz = NULL;
-	char obuf[cache_param->gzip_stack_buffer];
-	ssize_t obufl = 0;
-	size_t dl;
-	const void *dp;
-	int i;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	st = sp->obj->esidata;
-	AN(st);
-	assert(sizeof obuf >= 1024);
-
-	obuf[0] = 0;	/* For flexelint */
-
-	p = st->ptr;
-	e = st->ptr + st->len;
-
-	if (*p == VEC_GZ) {
-		isgzip = 1;
-		p++;
-	} else {
-		isgzip = 0;
-	}
-
-	if (sp->esi_level == 0) {
-		/*
-		 * Only the top level document gets to decide this.
-		 */
-		sp->wrk->gzip_resp = 0;
-		if (isgzip && !(sp->wrk->res_mode & RES_GUNZIP)) {
-			assert(sizeof gzip_hdr == 10);
-			/* Send out the gzip header */
-			(void)WRW_Write(sp->wrk, gzip_hdr, 10);
-			sp->wrk->l_crc = 0;
-			sp->wrk->gzip_resp = 1;
-			sp->wrk->crc = crc32(0L, Z_NULL, 0);
-		}
-	}
-
-	if (isgzip && !sp->wrk->gzip_resp) {
-		vgz = VGZ_NewUngzip(sp->wrk, "U D E");
-
-		/* Feed a gzip header to gunzip to make it happy */
-		VGZ_Ibuf(vgz, gzip_hdr, sizeof gzip_hdr);
-		VGZ_Obuf(vgz, obuf, sizeof obuf);
-		i = VGZ_Gunzip(vgz, &dp, &dl);
-		assert(i == VGZ_OK);
-		assert(VGZ_IbufEmpty(vgz));
-		assert(dl == 0);
-
-		obufl = 0;
-	}
-
-	st = VTAILQ_FIRST(&sp->obj->store);
-	off = 0;
-
-	while (p < e) {
-		switch (*p) {
-		case VEC_V1:
-		case VEC_V2:
-		case VEC_V8:
-			l = ved_decode_len(&p);
-			if (isgzip) {
-				assert(*p == VEC_C1 || *p == VEC_C2 ||
-				    *p == VEC_C8);
-				l_icrc = ved_decode_len(&p);
-				icrc = vbe32dec(p);
-				p += 4;
-				if (sp->wrk->gzip_resp) {
-					sp->wrk->crc = crc32_combine(
-					    sp->wrk->crc, icrc, l_icrc);
-					sp->wrk->l_crc += l_icrc;
-				}
-			}
-			/*
-			 * There is no guarantee that the 'l' bytes are all
-			 * in the same storage segment, so loop over storage
-			 * until we have processed them all.
-			 */
-			while (l > 0) {
-				l2 = l;
-				if (l2 > st->len - off)
-					l2 = st->len - off;
-				l -= l2;
-
-				if (sp->wrk->gzip_resp && isgzip) {
-					/*
-					 * We have a gzip'ed VEC and delivers
-					 * a gzip'ed ESI response.
-					 */
-					(void)WRW_Write(sp->wrk, st->ptr + off, l2);
-				} else if (sp->wrk->gzip_resp) {
-					/*
-					 * A gzip'ed ESI response, but the VEC
-					 * was not gzip'ed.
-					 */
-					ved_pretend_gzip(sp, st->ptr + off, l2);
-				} else if (isgzip) {
-					/*
-					 * A gzip'ed VEC, but ungzip'ed ESI
-					 * response
-					 */
-					AN(vgz);
-					i = VGZ_WrwGunzip(sp->wrk, vgz,
-						st->ptr + off, l2,
-						obuf, sizeof obuf, &obufl);
-					if (WRW_Error(sp->wrk)) {
-						SES_Close(sp, "remote closed");
-						p = e;
-						break;
-					}
-					assert (i == VGZ_OK || i == VGZ_END);
-				} else {
-					/*
-					 * Ungzip'ed VEC, ungzip'ed ESI response
-					 */
-					(void)WRW_Write(sp->wrk, st->ptr + off, l2);
-				}
-				off += l2;
-				if (off == st->len) {
-					st = VTAILQ_NEXT(st, list);
-					off = 0;
-				}
-			}
-			break;
-		case VEC_S1:
-		case VEC_S2:
-		case VEC_S8:
-			l = ved_decode_len(&p);
-			Debug("SKIP1(%d)\n", (int)l);
-			/*
-			 * There is no guarantee that the 'l' bytes are all
-			 * in the same storage segment, so loop over storage
-			 * until we have processed them all.
-			 */
-			while (l > 0) {
-				l2 = l;
-				if (l2 > st->len - off)
-					l2 = st->len - off;
-				l -= l2;
-				off += l2;
-				if (off == st->len) {
-					st = VTAILQ_NEXT(st, list);
-					off = 0;
-				}
-			}
-			break;
-		case VEC_INCL:
-			p++;
-			q = (void*)strchr((const char*)p, '\0');
-			AN(q);
-			q++;
-			r = (void*)strchr((const char*)q, '\0');
-			AN(r);
-			if (obufl > 0) {
-				(void)WRW_Write(sp->wrk, obuf, obufl);
-				obufl = 0;
-			}
-			if (WRW_Flush(sp->wrk)) {
-				SES_Close(sp, "remote closed");
-				p = e;
-				break;
-			}
-			Debug("INCL [%s][%s] BEGIN\n", q, p);
-			ved_include(sp, (const char*)q, (const char*)p);
-			Debug("INCL [%s][%s] END\n", q, p);
-			p = r + 1;
-			break;
-		default:
-			printf("XXXX 0x%02x [%s]\n", *p, p);
-			INCOMPL();
-		}
-	}
-	if (vgz != NULL) {
-		if (obufl > 0)
-			(void)WRW_Write(sp->wrk, obuf, obufl);
-		(void)VGZ_Destroy(&vgz, sp->vsl_id);
-	}
-	if (sp->wrk->gzip_resp && sp->esi_level == 0) {
-		/* Emit a gzip literal block with finish bit set */
-		tailbuf[0] = 0x01;
-		tailbuf[1] = 0x00;
-		tailbuf[2] = 0x00;
-		tailbuf[3] = 0xff;
-		tailbuf[4] = 0xff;
-
-		/* Emit CRC32 */
-		vle32enc(tailbuf + 5, sp->wrk->crc);
-
-		/* MOD(2^32) length */
-		vle32enc(tailbuf + 9, sp->wrk->l_crc);
-
-		(void)WRW_Write(sp->wrk, tailbuf, 13);
-	}
-	(void)WRW_Flush(sp->wrk);
-}
-
-/*---------------------------------------------------------------------
- * Include an object in a gzip'ed ESI object delivery
- */
-
-static uint8_t
-ved_deliver_byterange(const struct sess *sp, ssize_t low, ssize_t high)
-{
-	struct storage *st;
-	ssize_t l, lx;
-	u_char *p;
-
-//printf("BR %jd %jd\n", low, high);
-	lx = 0;
-	VTAILQ_FOREACH(st, &sp->obj->store, list) {
-		p = st->ptr;
-		l = st->len;
-//printf("[0-] %jd %jd\n", lx, lx + l);
-		if (lx + l < low) {
-			lx += l;
-			continue;
-		}
-		if (lx == high)
-			return (p[0]);
-		assert(lx < high);
-		if (lx < low) {
-			p += (low - lx);
-			l -= (low - lx);
-			lx = low;
-		}
-//printf("[1-] %jd %jd\n", lx, lx + l);
-		if (lx + l >= high)
-			l = high - lx;
-//printf("[2-] %jd %jd\n", lx, lx + l);
-		assert(lx >= low && lx + l <= high);
-		if (l != 0)
-			(void)WRW_Write(sp->wrk, p, l);
-		if (lx + st->len > high)
-			return(p[l]);
-		lx += st->len;
-	}
-	INCOMPL();
-}
-
-void
-ESI_DeliverChild(const struct sess *sp)
-{
-	struct storage *st;
-	struct object *obj;
-	ssize_t start, last, stop, lpad;
-	u_char *p, cc;
-	uint32_t icrc;
-	uint32_t ilen;
-	uint8_t *dbits;
-
-	if (!sp->obj->gziped) {
-		VTAILQ_FOREACH(st, &sp->obj->store, list)
-			ved_pretend_gzip(sp, st->ptr, st->len);
-		return;
-	}
-	/*
-	 * This is the interesting case: Deliver all the deflate
-	 * blocks, stripping the "LAST" bit of the last one and
-	 * padding it, as necessary, to a byte boundary.
-	 */
-
-	dbits = (void*)WS_Alloc(sp->wrk->ws, 8);
-	AN(dbits);
-	obj = sp->obj;
-	CHECK_OBJ_NOTNULL(obj, OBJECT_MAGIC);
-	start = obj->gzip_start;
-	last = obj->gzip_last;
-	stop = obj->gzip_stop;
-	assert(start > 0 && start < obj->len * 8);
-	assert(last > 0 && last < obj->len * 8);
-	assert(stop > 0 && stop < obj->len * 8);
-	assert(last >= start);
-	assert(last < stop);
-
-	/* The start bit must be byte aligned. */
-	AZ(start & 7);
-
-	/*
-	 * XXX: optimize for the case where the 'last'
-	 * XXX: bit is in a empty copy block
-	 */
-	*dbits = ved_deliver_byterange(sp, start/8, last/8);
-	*dbits &= ~(1U << (last & 7));
-	(void)WRW_Write(sp->wrk, dbits, 1);
-	cc = ved_deliver_byterange(sp, 1 + last/8, stop/8);
-	switch((int)(stop & 7)) {
-	case 0: /* xxxxxxxx */
-		/* I think we have an off by one here, but that's OK */
-		lpad = 0;
-		break;
-	case 1: /* x000.... 00000000 00000000 11111111 11111111 */
-	case 3: /* xxx000.. 00000000 00000000 11111111 11111111 */
-	case 5: /* xxxxx000 00000000 00000000 11111111 11111111 */
-		dbits[1] = cc | 0x00;
-		dbits[2] = 0x00; dbits[3] = 0x00;
-	        dbits[4] = 0xff; dbits[5] = 0xff;
-		lpad = 5;
-		break;
-	case 2: /* xx010000 00000100 00000001 00000000 */
-		dbits[1] = cc | 0x08;
-		dbits[2] = 0x20;
-		dbits[3] = 0x80;
-		dbits[4] = 0x00;
-		lpad = 4;
-		break;
-	case 4: /* xxxx0100 00000001 00000000 */
-		dbits[1] = cc | 0x20;
-		dbits[2] = 0x80;
-		dbits[3] = 0x00;
-		lpad = 3;
-		break;
-	case 6: /* xxxxxx01 00000000 */
-		dbits[1] = cc | 0x80;
-		dbits[2] = 0x00;
-		lpad = 2;
-		break;
-	case 7:	/* xxxxxxx0 00...... 00000000 00000000 11111111 11111111 */
-		dbits[1] = cc | 0x00;
-		dbits[2] = 0x00;
-		dbits[3] = 0x00; dbits[4] = 0x00;
-	        dbits[5] = 0xff; dbits[6] = 0xff;
-		lpad = 6;
-		break;
-	default:
-		INCOMPL();
-	}
-	if (lpad > 0)
-		(void)WRW_Write(sp->wrk, dbits + 1, lpad);
-	st = VTAILQ_LAST(&sp->obj->store, storagehead);
-	assert(st->len > 8);
-
-	p = st->ptr + st->len - 8;
-	icrc = vle32dec(p);
-	ilen = vle32dec(p + 4);
-	sp->wrk->crc = crc32_combine(sp->wrk->crc, icrc, ilen);
-	sp->wrk->l_crc += ilen;
-}
diff --git a/bin/varnishd/cache_esi_fetch.c b/bin/varnishd/cache_esi_fetch.c
deleted file mode 100644
index 5ec8f6b..0000000
--- a/bin/varnishd/cache_esi_fetch.c
+++ /dev/null
@@ -1,405 +0,0 @@
-/*-
- * Copyright (c) 2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * VEF Varnish Esi Fetching
- */
-
-#include "config.h"
-
-#include <stdlib.h>
-
-#include "cache.h"
-
-#include "cache_esi.h"
-
-/*---------------------------------------------------------------------
- * Read some bytes.
- *
- * If the esi_syntax&8 bit is set, we read only a couple of bytes at
- * a time, in order to stress the parse/pending/callback code.
- */
-
-static ssize_t
-vef_read(struct worker *w, struct http_conn *htc, void *buf, ssize_t buflen,
-    ssize_t bytes)
-{
-	ssize_t d;
-
-	if (buflen < bytes)
-		bytes = buflen;
-	if (cache_param->esi_syntax & 0x8) {
-		d = (random() & 3) + 1;
-		if (d < bytes)
-			bytes = d;
-	}
-	return (HTC_Read(w, htc, buf, bytes));
-}
-
-/*---------------------------------------------------------------------
- * We receive a ungzip'ed object, and want to store it ungzip'ed.
- */
-
-static int
-vfp_esi_bytes_uu(struct worker *w, struct http_conn *htc, ssize_t bytes)
-{
-	ssize_t wl;
-	struct storage *st;
-
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-
-	while (bytes > 0) {
-		st = FetchStorage(w, 0);
-		if (st == NULL)
-			return (-1);
-		wl = vef_read(w, htc,
-		    st->ptr + st->len, st->space - st->len, bytes);
-		if (wl <= 0)
-			return (wl);
-		VEP_Parse(w, (const char *)st->ptr + st->len, wl);
-		st->len += wl;
-		w->fetch_obj->len += wl;
-		bytes -= wl;
-	}
-	return (1);
-}
-
-/*---------------------------------------------------------------------
- * We receive a gzip'ed object, and want to store it ungzip'ed.
- */
-
-static int
-vfp_esi_bytes_gu(struct worker *w, struct http_conn *htc, ssize_t bytes)
-{
-	struct vgz *vg;
-	ssize_t wl;
-	uint8_t	ibuf[cache_param->gzip_stack_buffer];
-	int i;
-	size_t dl;
-	const void *dp;
-
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	vg = w->vgz_rx;
-
-	while (bytes > 0) {
-		if (VGZ_IbufEmpty(vg) && bytes > 0) {
-			wl = vef_read(w, htc, ibuf, sizeof ibuf, bytes);
-			if (wl <= 0)
-				return (wl);
-			VGZ_Ibuf(vg, ibuf, wl);
-			bytes -= wl;
-		}
-		if (VGZ_ObufStorage(w, vg))
-			return(-1);
-		i = VGZ_Gunzip(vg, &dp, &dl);
-		xxxassert(i == VGZ_OK || i == VGZ_END);
-		VEP_Parse(w, dp, dl);
-		w->fetch_obj->len += dl;
-	}
-	return (1);
-}
-
-/*---------------------------------------------------------------------
- */
-
-struct vef_priv {
-	unsigned		magic;
-#define VEF_MAGIC		0xf104b51f
-	struct vgz		*vgz;
-
-	char			*bufp;
-	ssize_t			tot;
-	int			error;
-	char			pending[20];
-	ssize_t			npend;
-};
-
-/*---------------------------------------------------------------------
- * We receive a [un]gzip'ed object, and want to store it gzip'ed.
- */
-
-static ssize_t
-vfp_vep_callback(struct worker *w, ssize_t l, enum vgz_flag flg)
-{
-	struct vef_priv *vef;
-	size_t dl, px;
-	const void *dp;
-	int i;
-
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	vef = w->vef_priv;
-	CHECK_OBJ_NOTNULL(vef, VEF_MAGIC);
-	assert(l >= 0);
-
-	if (vef->error) {
-		vef->tot += l;
-		return (vef->tot);
-	}
-
-	/*
-	 * l == 0 is valid when 'flg' calls for action, but in the
-	 * normal case we can just ignore a l==0 request.
-	 * (It would cause Z_BUF_ERROR anyway)
-	 */
-	if (l == 0 && flg == VGZ_NORMAL)
-		return (vef->tot);
-
-	do {
-		px = vef->npend;
-		if (l < px)
-			px = l;
-		if (px != 0) {
-			VGZ_Ibuf(vef->vgz, vef->pending, px);
-			l -= px;
-		} else {
-			VGZ_Ibuf(vef->vgz, vef->bufp, l);
-			vef->bufp += l;
-			l = 0;
-		}
-		do {
-			if (VGZ_ObufStorage(w, vef->vgz)) {
-				vef->error = ENOMEM;
-				vef->tot += l;
-				return (vef->tot);
-			}
-			i = VGZ_Gzip(vef->vgz, &dp, &dl, flg);
-			vef->tot += dl;
-			w->fetch_obj->len += dl;
-		} while (!VGZ_IbufEmpty(vef->vgz) ||
-		    (flg != VGZ_NORMAL && VGZ_ObufFull(vef->vgz)));
-		if (px != 0) {
-			memmove(vef->pending, vef->pending + px,
-			    vef->npend - px);
-			vef->npend -= px;
-		}
-	} while (l > 0);
-	if (flg == VGZ_FINISH)
-		assert(i == 1);			/* XXX */
-	else
-		assert(i == 0);			/* XXX */
-	return (vef->tot);
-}
-
-static int
-vfp_esi_bytes_ug(struct worker *w, struct http_conn *htc, ssize_t bytes)
-{
-	ssize_t wl;
-	char ibuf[cache_param->gzip_stack_buffer];
-	struct vef_priv *vef;
-
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	vef = w->vef_priv;
-	CHECK_OBJ_NOTNULL(vef, VEF_MAGIC);
-
-	while (bytes > 0) {
-		wl = vef_read(w, htc, ibuf, sizeof ibuf, bytes);
-		if (wl <= 0)
-			return (wl);
-		bytes -= wl;
-		vef->bufp = ibuf;
-		VEP_Parse(w, ibuf, wl);
-		assert(vef->bufp >= ibuf && vef->bufp <= ibuf + wl);
-		if (vef->error) {
-			errno = vef->error;
-			return (-1);
-		}
-		if (vef->bufp < ibuf + wl) {
-			wl = (ibuf + wl) - vef->bufp;
-			assert(wl + vef->npend < sizeof vef->pending);
-			memmove(vef->pending + vef->npend, vef->bufp, wl);
-			vef->npend += wl;
-		}
-	}
-	return (1);
-}
-
-/*---------------------------------------------------------------------
- * We receive a gzip'ed object, and want to store it gzip'ed.
- */
-
-static int
-vfp_esi_bytes_gg(struct worker *w, struct http_conn *htc, size_t bytes)
-{
-	ssize_t wl;
-	char ibuf[cache_param->gzip_stack_buffer];
-	char ibuf2[cache_param->gzip_stack_buffer];
-	struct vef_priv *vef;
-	size_t dl;
-	const void *dp;
-	int i;
-
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	vef = w->vef_priv;
-	CHECK_OBJ_NOTNULL(vef, VEF_MAGIC);
-	assert(sizeof ibuf >= 1024);
-	ibuf2[0] = 0; /* For Flexelint */
-
-	while (bytes > 0) {
-		wl = vef_read(w, htc, ibuf, sizeof ibuf, bytes);
-		if (wl <= 0)
-			return (wl);
-		bytes -= wl;
-
-		vef->bufp = ibuf;
-		VGZ_Ibuf(w->vgz_rx, ibuf, wl);
-		do {
-			VGZ_Obuf(w->vgz_rx, ibuf2, sizeof ibuf2);
-			i = VGZ_Gunzip(w->vgz_rx, &dp, &dl);
-			/* XXX: check i */
-			assert(i >= VGZ_OK);
-			vef->bufp = ibuf2;
-			if (dl > 0)
-				VEP_Parse(w, ibuf2, dl);
-			if (vef->error) {
-				errno = vef->error;
-				return (-1);
-			}
-			if (vef->bufp < ibuf2 + dl) {
-				dl = (ibuf2 + dl) - vef->bufp;
-				assert(dl + vef->npend < sizeof vef->pending);
-				memmove(vef->pending + vef->npend,
-				    vef->bufp, dl);
-				vef->npend += dl;
-			}
-		} while (!VGZ_IbufEmpty(w->vgz_rx));
-	}
-	return (1);
-}
-
-
-/*---------------------------------------------------------------------*/
-
-static void __match_proto__()
-vfp_esi_begin(struct worker *w, size_t estimate)
-{
-	struct vef_priv *vef;
-
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-
-	AZ(w->vgz_rx);
-	if (w->is_gzip && w->do_gunzip) {
-		w->vgz_rx = VGZ_NewUngzip(w, "U F E");
-		VEP_Init(w, NULL);
-	} else if (w->is_gunzip && w->do_gzip) {
-		ALLOC_OBJ(vef, VEF_MAGIC);
-		AN(vef);
-		vef->vgz = VGZ_NewGzip(w, "G F E");
-		AZ(w->vef_priv);
-		w->vef_priv = vef;
-		VEP_Init(w, vfp_vep_callback);
-	} else if (w->is_gzip) {
-		w->vgz_rx = VGZ_NewUngzip(w, "U F E");
-		ALLOC_OBJ(vef, VEF_MAGIC);
-		AN(vef);
-		vef->vgz = VGZ_NewGzip(w, "G F E");
-		AZ(w->vef_priv);
-		w->vef_priv = vef;
-		VEP_Init(w, vfp_vep_callback);
-	} else {
-		AZ(w->vef_priv);
-		VEP_Init(w, NULL);
-	}
-
-	(void)estimate;
-	AN(w->vep);
-}
-
-static int __match_proto__()
-vfp_esi_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
-{
-	int i;
-
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	AZ(w->fetch_failed);
-	AN(w->vep);
-	assert(w->htc == htc);
-	if (w->is_gzip && w->do_gunzip)
-		i = vfp_esi_bytes_gu(w, htc, bytes);
-	else if (w->is_gunzip && w->do_gzip)
-		i = vfp_esi_bytes_ug(w, htc, bytes);
-	else if (w->is_gzip)
-		i = vfp_esi_bytes_gg(w, htc, bytes);
-	else
-		i = vfp_esi_bytes_uu(w, htc, bytes);
-	AN(w->vep);
-	return (i);
-}
-
-static int __match_proto__()
-vfp_esi_end(struct worker *w)
-{
-	struct vsb *vsb;
-	struct vef_priv *vef;
-	ssize_t l;
-	int retval;
-
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	AN(w->vep);
-
-	retval = w->fetch_failed;
-
-	if (w->vgz_rx != NULL && VGZ_Destroy(&w->vgz_rx, -1) != VGZ_END)
-		retval = FetchError(w,
-		    "Gunzip+ESI Failed at the very end");
-
-	vsb = VEP_Finish(w);
-
-	if (vsb != NULL) {
-		if (!retval) {
-			l = VSB_len(vsb);
-			assert(l > 0);
-			/* XXX: This is a huge waste of storage... */
-			w->fetch_obj->esidata = STV_alloc(w, l);
-			if (w->fetch_obj->esidata != NULL) {
-				memcpy(w->fetch_obj->esidata->ptr,
-				    VSB_data(vsb), l);
-				w->fetch_obj->esidata->len = l;
-			} else {
-				retval = FetchError(w,
-				    "Could not allocate storage for esidata");
-			}
-		}
-		VSB_delete(vsb);
-	}
-
-	if (w->vef_priv != NULL) {
-		vef = w->vef_priv;
-		CHECK_OBJ_NOTNULL(vef, VEF_MAGIC);
-		w->vef_priv = NULL;
-		VGZ_UpdateObj(vef->vgz, w->fetch_obj);
-		if (VGZ_Destroy(&vef->vgz,  -1) != VGZ_END)
-			retval = FetchError(w,
-			    "ESI+Gzip Failed at the very end");
-		FREE_OBJ(vef);
-	}
-	return (retval);
-}
-
-struct vfp vfp_esi = {
-        .begin  =       vfp_esi_begin,
-        .bytes  =       vfp_esi_bytes,
-        .end    =       vfp_esi_end,
-};
diff --git a/bin/varnishd/cache_esi_parse.c b/bin/varnishd/cache_esi_parse.c
deleted file mode 100644
index 9e2b4f6..0000000
--- a/bin/varnishd/cache_esi_parse.c
+++ /dev/null
@@ -1,1189 +0,0 @@
-/*-
- * Copyright (c) 2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * VEP Varnish Esi Parsing
- */
-
-#include "config.h"
-
-#include <stdio.h>
-#include <stdlib.h>
-
-#include "cache.h"
-
-#include "cache_esi.h"
-#include "vct.h"
-#include "vend.h"
-#include "vgz.h"
-
-//#define Debug(fmt, ...) printf(fmt, __VA_ARGS__)
-#define Debug(fmt, ...) /**/
-
-struct vep_state;
-
-enum dowhat {DO_ATTR, DO_TAG};
-typedef void dostuff_f(struct vep_state *, enum dowhat);
-
-struct vep_match {
-	const char	*match;
-	const char	* const *state;
-};
-
-enum vep_mark { VERBATIM = 0, SKIP };
-
-struct vep_state {
-	unsigned		magic;
-#define VEP_MAGIC		0x55cb9b82
-	struct vsb		*vsb;
-
-	struct worker		*wrk;
-	int			dogzip;
-	vep_callback_t		*cb;
-
-	/* Internal Counter for default call-back function */
-	ssize_t			cb_x;
-
-	/* parser state */
-	const char		*state;
-	unsigned		startup;
-	unsigned		esi_found;
-
-	unsigned		endtag;
-	unsigned		emptytag;
-	unsigned		canattr;
-
-	unsigned		remove;
-
-	ssize_t			o_wait;
-	ssize_t			o_pending;
-	ssize_t			o_total;
-	uint32_t		crc;
-	ssize_t			o_crc;
-	uint32_t		crcp;
-	ssize_t			o_last;
-
-const char		*hack_p;
-	const char		*ver_p;
-
-	const char		*until;
-	const char		*until_p;
-	const char		*until_s;
-
-	int			in_esi_tag;
-
-	const char		*esicmt;
-	const char		*esicmt_p;
-
-	struct vep_match	*attr;
-	struct vsb		*attr_vsb;
-	int			attr_delim;
-
-	struct vep_match	*match;
-	struct vep_match	*match_hit;
-
-	char			tag[10];
-	int			tag_i;
-
-	dostuff_f		*dostuff;
-
-	struct vsb		*include_src;
-
-	unsigned		nm_skip;
-	unsigned		nm_verbatim;
-	unsigned		nm_pending;
-	enum vep_mark		last_mark;
-};
-
-/*---------------------------------------------------------------------*/
-
-static const char * const VEP_START =		"[Start]";
-static const char * const VEP_TESTXML =		"[TestXml]";
-static const char * const VEP_NOTXML =		"[NotXml]";
-
-static const char * const VEP_NEXTTAG =		"[NxtTag]";
-static const char * const VEP_NOTMYTAG =	"[NotMyTag]";
-
-static const char * const VEP_STARTTAG =	"[StartTag]";
-static const char * const VEP_COMMENT =		"[Comment]";
-static const char * const VEP_CDATA =		"[CDATA]";
-static const char * const VEP_ESITAG =		"[ESITag]";
-
-static const char * const VEP_ESIREMOVE =	"[ESI:Remove]";
-static const char * const VEP_ESIINCLUDE =	"[ESI:Include]";
-static const char * const VEP_ESICOMMENT =	"[ESI:Comment]";
-static const char * const VEP_ESIBOGON =	"[ESI:Bogon]";
-
-static const char * const VEP_INTAG =		"[InTag]";
-static const char * const VEP_TAGERROR =	"[TagError]";
-
-static const char * const VEP_ATTR =		"[Attribute]";
-static const char * const VEP_SKIPATTR =	"[SkipAttribute]";
-static const char * const VEP_ATTRDELIM =	"[AttrDelim]";
-static const char * const VEP_ATTRGETVAL =	"[AttrGetValue]";
-static const char * const VEP_ATTRVAL =		"[AttrValue]";
-
-static const char * const VEP_UNTIL =		"[Until]";
-static const char * const VEP_MATCHBUF =	"[MatchBuf]";
-static const char * const VEP_MATCH =		"[Match]";
-
-/*---------------------------------------------------------------------*/
-
-static struct vep_match vep_match_starttag[] = {
-	{ "!--",	&VEP_COMMENT },
-	{ "esi:",	&VEP_ESITAG },
-	{ "![CDATA[",	&VEP_CDATA },
-	{ NULL,		&VEP_NOTMYTAG }
-};
-
-/*---------------------------------------------------------------------*/
-
-static struct vep_match vep_match_esi[] = {
-	{ "include",	&VEP_ESIINCLUDE },
-	{ "remove",	&VEP_ESIREMOVE },
-	{ "comment",	&VEP_ESICOMMENT },
-	{ NULL,		&VEP_ESIBOGON }
-};
-
-/*---------------------------------------------------------------------*/
-
-static struct vep_match vep_match_attr_include[] = {
-	{ "src=",	&VEP_ATTRGETVAL },
-	{ NULL,		&VEP_SKIPATTR }
-};
-
-/*--------------------------------------------------------------------
- * Report a parsing error
- */
-
-static void
-vep_error(const struct vep_state *vep, const char *p)
-{
-	intmax_t l;
-
-	VSC_C_main->esi_errors++;
-	l = (intmax_t)(vep->ver_p - vep->hack_p);
-	WSLB(vep->wrk, SLT_ESI_xmlerror, "ERR at %jd %s", l, p);
-
-}
-
-/*--------------------------------------------------------------------
- * Report a parsing warning
- */
-
-static void
-vep_warn(const struct vep_state *vep, const char *p)
-{
-	intmax_t l;
-
-	VSC_C_main->esi_warnings++;
-	l = (intmax_t)(vep->ver_p - vep->hack_p);
-	printf("WARNING at %jd %s\n", l, p);
-	WSLB(vep->wrk, SLT_ESI_xmlerror, "WARN at %jd %s", l, p);
-
-}
-
-/*---------------------------------------------------------------------
- * return match or NULL if more input needed.
- */
-
-static struct vep_match *
-vep_match(struct vep_state *vep, const char *b, const char *e)
-{
-	struct vep_match *vm;
-	const char *q, *r;
-	ssize_t l;
-
-	for (vm = vep->match; vm->match; vm++) {
-		r = b;
-		for (q = vm->match; *q && r < e; q++, r++)
-			if (*q != *r)
-				break;
-		if (*q != '\0' && r == e) {
-			if (b != vep->tag) {
-				l = e - b;
-				assert(l < sizeof vep->tag);
-				memmove(vep->tag, b, l);
-				vep->tag_i = l;
-			}
-			return (NULL);
-		}
-		if (*q == '\0')
-			return (vm);
-	}
-	return (vm);
-}
-
-/*---------------------------------------------------------------------
- *
- */
-
-static void
-vep_emit_len(const struct vep_state *vep, ssize_t l, int m8, int m16, int m64)
-{
-	uint8_t buf[9];
-
-	assert(l > 0);
-	if (l < 256) {
-		buf[0] = (uint8_t)m8;
-		buf[1] = (uint8_t)l;
-		assert((ssize_t)buf[1] == l);
-		VSB_bcat(vep->vsb, buf, 2);
-	} else if (l < 65536) {
-		buf[0] = (uint8_t)m16;
-		vbe16enc(buf + 1, (uint16_t)l);
-		assert((ssize_t)vbe16dec(buf + 1) == l);
-		VSB_bcat(vep->vsb, buf, 3);
-	} else {
-		buf[0] = (uint8_t)m64;
-		vbe64enc(buf + 1, l);
-		assert((ssize_t)vbe64dec(buf + 1) == l);
-		VSB_bcat(vep->vsb, buf, 9);
-	}
-}
-
-static void
-vep_emit_skip(const struct vep_state *vep, ssize_t l)
-{
-
-	if (cache_param->esi_syntax & 0x20) {
-		Debug("---> SKIP(%jd)\n", (intmax_t)l);
-	}
-	vep_emit_len(vep, l, VEC_S1, VEC_S2, VEC_S8);
-}
-
-static void
-vep_emit_verbatim(const struct vep_state *vep, ssize_t l, ssize_t l_crc)
-{
-	uint8_t buf[4];
-
-	if (cache_param->esi_syntax & 0x20) {
-		Debug("---> VERBATIM(%jd)\n", (intmax_t)l);
-	}
-	vep_emit_len(vep, l, VEC_V1, VEC_V2, VEC_V8);
-	if (vep->dogzip) {
-		vep_emit_len(vep, l_crc, VEC_C1, VEC_C2, VEC_C8);
-		vbe32enc(buf, vep->crc);
-		VSB_bcat(vep->vsb, buf, sizeof buf);
-	}
-}
-
-static void
-vep_emit_common(struct vep_state *vep, ssize_t l, enum vep_mark mark)
-{
-
-	assert(l > 0);
-	assert(mark == SKIP || mark == VERBATIM);
-	if (mark == SKIP)
-		vep_emit_skip(vep, l);
-	else
-		vep_emit_verbatim(vep, l, vep->o_crc);
-
-	vep->crc = crc32(0L, Z_NULL, 0);
-	vep->o_crc = 0;
-	vep->o_total += l;
-}
-
-/*---------------------------------------------------------------------
- *
- */
-
-static void
-vep_mark_common(struct vep_state *vep, const char *p, enum vep_mark mark)
-{
-	ssize_t l, lcb;
-
-	assert(mark == SKIP || mark == VERBATIM);
-
-	/* The NO-OP case, no data, no pending data & no change of mode */
-	if (vep->last_mark == mark && p == vep->ver_p && vep->o_pending == 0)
-		return;
-
-	/*
-	 * If we changed mode, emit whatever the opposite mode
-	 * assembled before the pending bytes.
-	 */
-
-	if (vep->last_mark != mark && (vep->o_wait > 0 || vep->startup)) {
-		lcb = vep->cb(vep->wrk, 0,
-		    mark == VERBATIM ? VGZ_RESET : VGZ_ALIGN);
-		if (lcb - vep->o_last > 0)
-			vep_emit_common(vep, lcb - vep->o_last, vep->last_mark);
-		vep->o_last = lcb;
-		vep->o_wait = 0;
-	}
-
-	/* Transfer pending bytes CRC into active mode CRC */
-	if (vep->o_pending) {
-		(void)vep->cb(vep->wrk, vep->o_pending, VGZ_NORMAL);
-		if (vep->o_crc == 0) {
-			vep->crc = vep->crcp;
-			vep->o_crc = vep->o_pending;
-		} else {
-			vep->crc = crc32_combine(vep->crc,
-			    vep->crcp, vep->o_pending);
-			vep->o_crc += vep->o_pending;
-		}
-		vep->crcp = crc32(0L, Z_NULL, 0);
-		vep->o_wait += vep->o_pending;
-		vep->o_pending = 0;
-	}
-
-	/* * Process this bit of input */
-	AN(vep->ver_p);
-	l = p - vep->ver_p;
-	assert(l >= 0);
-	vep->crc = crc32(vep->crc, (const void*)vep->ver_p, l);
-	vep->o_crc += l;
-	vep->ver_p = p;
-
-	vep->o_wait += l;
-	vep->last_mark = mark;
-	(void)vep->cb(vep->wrk, l, VGZ_NORMAL);
-}
-
-static void
-vep_mark_verbatim(struct vep_state *vep, const char *p)
-{
-
-	vep_mark_common(vep, p, VERBATIM);
-	vep->nm_verbatim++;
-}
-
-static void
-vep_mark_skip(struct vep_state *vep, const char *p)
-{
-
-	vep_mark_common(vep, p, SKIP);
-	vep->nm_skip++;
-}
-
-static void
-vep_mark_pending(struct vep_state *vep, const char *p)
-{
-	ssize_t l;
-
-	AN(vep->ver_p);
-	l = p - vep->ver_p;
-	assert(l > 0);
-	assert(l >= 0);
-	vep->crcp = crc32(vep->crcp, (const void *)vep->ver_p, l);
-	vep->ver_p = p;
-
-	vep->o_pending += l;
-	vep->nm_pending++;
-}
-
-/*---------------------------------------------------------------------
- */
-
-static void __match_proto__()
-vep_do_comment(struct vep_state *vep, enum dowhat what)
-{
-	Debug("DO_COMMENT(%d)\n", what);
-	assert(what == DO_TAG);
-	if (!vep->emptytag)
-		vep_error(vep, "ESI 1.0 <esi:comment> needs final '/'");
-}
-
-/*---------------------------------------------------------------------
- */
-
-static void __match_proto__()
-vep_do_remove(struct vep_state *vep, enum dowhat what)
-{
-	Debug("DO_REMOVE(%d, end %d empty %d remove %d)\n",
-	    what, vep->endtag, vep->emptytag, vep->remove);
-	assert(what == DO_TAG);
-	if (vep->emptytag) {
-		vep_error(vep,
-		    "ESI 1.0 <esi:remove/> not legal");
-	} else {
-		if (vep->remove && !vep->endtag)
-			vep_error(vep,
-			    "ESI 1.0 <esi:remove> already open");
-		else if (!vep->remove && vep->endtag)
-			vep_error(vep,
-			    "ESI 1.0 <esi:remove> not open");
-		else
-			vep->remove = !vep->endtag;
-	}
-}
-
-/*---------------------------------------------------------------------
- */
-
-static void __match_proto__()
-vep_do_include(struct vep_state *vep, enum dowhat what)
-{
-	char *p, *q, *h;
-	ssize_t l;
-	txt url;
-
-	Debug("DO_INCLUDE(%d)\n", what);
-	if (what == DO_ATTR) {
-		Debug("ATTR (%s) (%s)\n", vep->match_hit->match,
-			VSB_data(vep->attr_vsb));
-		if (vep->include_src != NULL) {
-			vep_error(vep,
-			    "ESI 1.0 <esi:include> "
-			    "has multiple src= attributes");
-			vep->state = VEP_TAGERROR;
-			VSB_delete(vep->attr_vsb);
-			VSB_delete(vep->include_src);
-			vep->attr_vsb = NULL;
-			vep->include_src = NULL;
-			return;
-		}
-		XXXAZ(vep->include_src);	/* multiple src= */
-		vep->include_src = vep->attr_vsb;
-		return;
-	}
-	assert(what == DO_TAG);
-	if (!vep->emptytag)
-		vep_warn(vep,
-		    "ESI 1.0 <esi:include> lacks final '/'");
-	if (vep->include_src == NULL) {
-		vep_error(vep,
-		    "ESI 1.0 <esi:include> lacks src attr");
-		return;
-	}
-
-	/*
-	 * Strictly speaking, we ought to spit out any piled up skip before
-	 * emitting the VEC for the include, but objectively that makes no
-	 * difference and robs us of a chance to collapse another skip into
-	 * this on so we don't do that.
-	 * However, we cannot tolerate any verbatim stuff piling up.
-	 * The mark_skip() before calling dostuff should have taken
-	 * care of that.  Make sure.
-	 */
-	assert(vep->o_wait == 0 || vep->last_mark == SKIP);
-	/* XXX: what if it contains NUL bytes ?? */
-	p = VSB_data(vep->include_src);
-	l = VSB_len(vep->include_src);
-	h = 0;
-
-	VSB_printf(vep->vsb, "%c", VEC_INCL);
-	if (l > 7 && !memcmp(p, "http://", 7)) {
-		h = p + 7;
-		p = strchr(h, '/');
-		AN(p);
-		Debug("HOST <%.*s> PATH <%s>\n", (int)(p-h),h, p);
-		VSB_printf(vep->vsb, "Host: %.*s%c",
-		    (int)(p-h), h, 0);
-	} else if (*p == '/') {
-		VSB_printf(vep->vsb, "%c", 0);
-	} else {
-		VSB_printf(vep->vsb, "%c", 0);
-		url = vep->wrk->bereq->hd[HTTP_HDR_URL];
-		/* Look for the last / before a '?' */
-		h = NULL;
-		for (q = url.b; q < url.e && *q != '?'; q++)
-			if (*q == '/')
-				h = q;
-		if (h == NULL)
-			h = q + 1;
-
-		Debug("INCL:: [%.*s]/[%s]\n",
-		    (int)(h - url.b), url.b, p);
-		VSB_printf(vep->vsb, "%.*s/", (int)(h - url.b), url.b);
-	}
-	l -= (p - VSB_data(vep->include_src));
-	for (q = p; *q != '\0'; ) {
-		if (*q == '&') {
-#define R(w,f,r)							\
-			if (q + w <= p + l && !memcmp(q, f, w)) { \
-				VSB_printf(vep->vsb, "%c", r);	\
-				q += w;				\
-				continue;			\
-			}
-			R(6, "&apos;", '\'');
-			R(6, "&quot;", '"');
-			R(4, "&lt;", '<');
-			R(4, "&gt;", '>');
-			R(5, "&amp;", '&');
-		}
-		VSB_printf(vep->vsb, "%c", *q++);
-	}
-#undef R
-	VSB_printf(vep->vsb, "%c", 0);
-
-	VSB_delete(vep->include_src);
-	vep->include_src = NULL;
-}
-
-/*---------------------------------------------------------------------
- * Lex/Parse object for ESI instructions
- *
- * This function is called with the input object piecemal so do not
- * assume that we have more than one char available at at time, but
- * optimize for getting huge chunks.
- *
- * NB: At the bottom of this source-file, there is a dot-diagram matching
- * NB: the state-machine.  Please maintain it along with the code.
- */
-
-void
-VEP_Parse(const struct worker *w, const char *p, size_t l)
-{
-	struct vep_state *vep;
-	const char *e;
-	struct vep_match *vm;
-	int i;
-
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	vep = w->vep;
-	CHECK_OBJ_NOTNULL(vep, VEP_MAGIC);
-	assert(l > 0);
-
-	/* XXX: Really need to fix this */
-	if (vep->hack_p == NULL)
-		vep->hack_p = p;
-
-	vep->ver_p = p;
-
-	e = p + l;
-
-	while (p < e) {
-		AN(vep->state);
-		i = e - p;
-		if (i > 10)
-			i = 10;
-		Debug("EP %s %d (%.*s) [%.*s]\n",
-		    vep->state,
-		    vep->remove,
-		    vep->tag_i, vep->tag,
-		    i, p);
-		assert(p >= vep->ver_p);
-
-		/******************************************************
-		 * SECTION A
-		 */
-
-		if (vep->state == VEP_START) {
-			if (cache_param->esi_syntax & 0x1)
-				vep->state = VEP_NEXTTAG;
-			else
-				vep->state = VEP_TESTXML;
-		} else if (vep->state == VEP_TESTXML) {
-			/*
-			 * If the first non-whitespace char is different
-			 * from '<' we assume this is not XML.
-			 */
-			while (p < e && vct_islws(*p))
-				p++;
-			vep_mark_verbatim(vep, p);
-			if (p < e && *p == '<') {
-				p++;
-				vep->state = VEP_STARTTAG;
-			} else if (p < e) {
-				WSLB(vep->wrk, SLT_ESI_xmlerror,
-				    "No ESI processing, first char not '<'");
-				vep->state = VEP_NOTXML;
-			}
-		} else if (vep->state == VEP_NOTXML) {
-			/*
-			 * This is not recognized as XML, just skip thru
-			 * vfp_esi_end() will handle the rest
-			 */
-			p = e;
-			vep_mark_verbatim(vep, p);
-
-		/******************************************************
-		 * SECTION B
-		 */
-
-		} else if (vep->state == VEP_NOTMYTAG) {
-			if (cache_param->esi_syntax & 0x2) {
-				p++;
-				vep->state = VEP_NEXTTAG;
-			} else {
-				vep->tag_i = 0;
-				while (p < e) {
-					if (*p++ == '>') {
-						vep->state = VEP_NEXTTAG;
-						break;
-					}
-				}
-			}
-			if (p == e && !vep->remove)
-				vep_mark_verbatim(vep, p);
-		} else if (vep->state == VEP_NEXTTAG) {
-			/*
-			 * Hunt for start of next tag and keep an eye
-			 * out for end of EsiCmt if armed.
-			 */
-			vep->emptytag = 0;
-			vep->endtag = 0;
-			vep->attr = NULL;
-			vep->dostuff = NULL;
-			while (p < e && *p != '<') {
-				if (vep->esicmt_p == NULL) {
-					p++;
-					continue;
-				}
-				if (*p != *vep->esicmt_p) {
-					p++;
-					vep->esicmt_p = vep->esicmt;
-					continue;
-				}
-				if (!vep->remove &&
-				    vep->esicmt_p == vep->esicmt)
-					vep_mark_verbatim(vep, p);
-				p++;
-				if (*++vep->esicmt_p == '\0') {
-					vep->esi_found = 1;
-					vep->esicmt = NULL;
-					vep->esicmt_p = NULL;
-					/*
-					 * The end of the esicmt
-					 * should not be emitted.
-					 * But the stuff before should
-					 */
-					vep_mark_skip(vep, p);
-				}
-			}
-			if (p < e) {
-				if (!vep->remove)
-					vep_mark_verbatim(vep, p);
-				assert(*p == '<');
-				p++;
-				vep->state = VEP_STARTTAG;
-			} else if (vep->esicmt_p == vep->esicmt && !vep->remove)
-				vep_mark_verbatim(vep, p);
-
-		/******************************************************
-		 * SECTION C
-		 */
-
-		} else if (vep->state == VEP_STARTTAG) {
-			/*
-			 * Start of tag, set up match table
-			 */
-			if (p < e) {
-				if (*p == '/') {
-					vep->endtag = 1;
-					p++;
-				}
-				vep->match = vep_match_starttag;
-				vep->state = VEP_MATCH;
-			}
-		} else if (vep->state == VEP_COMMENT) {
-			/*
-			 * We are in a comment, find out if it is an
-			 * ESI comment or a regular comment
-			 */
-			if (vep->esicmt == NULL)
-				vep->esicmt_p = vep->esicmt = "esi";
-			while (p < e) {
-				if (*p != *vep->esicmt_p) {
-					vep->esicmt_p = vep->esicmt = NULL;
-					vep->until_p = vep->until = "-->";
-					vep->until_s = VEP_NEXTTAG;
-					vep->state = VEP_UNTIL;
-					vep_mark_verbatim(vep, p);
-					break;
-				}
-				p++;
-				if (*++vep->esicmt_p != '\0')
-					continue;
-				if (vep->remove)
-					vep_error(vep,
-					    "ESI 1.0 Nested <!--esi"
-					    " element in <esi:remove>");
-				vep->esicmt_p = vep->esicmt = "-->";
-				vep->state = VEP_NEXTTAG;
-				vep_mark_skip(vep, p);
-				break;
-			}
-		} else if (vep->state == VEP_CDATA) {
-			/*
-			 * Easy: just look for the end of CDATA
-			 */
-			vep->until_p = vep->until = "]]>";
-			vep->until_s = VEP_NEXTTAG;
-			vep->state = VEP_UNTIL;
-		} else if (vep->state == VEP_ESITAG) {
-			vep->in_esi_tag = 1;
-			vep->esi_found = 1;
-			vep_mark_skip(vep, p);
-			vep->match = vep_match_esi;
-			vep->state = VEP_MATCH;
-		} else if (vep->state == VEP_ESIINCLUDE) {
-			if (vep->remove) {
-				vep_error(vep,
-				    "ESI 1.0 <esi:include> element"
-				    " nested in <esi:remove>");
-				vep->state = VEP_TAGERROR;
-			} else if (vep->endtag) {
-				vep_error(vep,
-				    "ESI 1.0 </esi:include> illegal end-tag");
-				vep->state = VEP_TAGERROR;
-			} else {
-				vep->dostuff = vep_do_include;
-				vep->state = VEP_INTAG;
-				vep->attr = vep_match_attr_include;
-			}
-		} else if (vep->state == VEP_ESIREMOVE) {
-			vep->dostuff = vep_do_remove;
-			vep->state = VEP_INTAG;
-		} else if (vep->state == VEP_ESICOMMENT) {
-			if (vep->remove) {
-				vep_error(vep,
-				    "ESI 1.0 <esi:comment> element"
-				    " nested in <esi:remove>");
-				vep->state = VEP_TAGERROR;
-			} else if (vep->endtag) {
-				vep_error(vep,
-				    "ESI 1.0 </esi:comment> illegal end-tag");
-				vep->state = VEP_TAGERROR;
-			} else {
-				vep->dostuff = vep_do_comment;
-				vep->state = VEP_INTAG;
-			}
-		} else if (vep->state == VEP_ESIBOGON) {
-			vep_error(vep,
-			    "ESI 1.0 <esi:bogus> element");
-			vep->state = VEP_TAGERROR;
-
-		/******************************************************
-		 * SECTION D
-		 */
-
-		} else if (vep->state == VEP_INTAG) {
-			vep->tag_i = 0;
-			while (p < e && vct_islws(*p) && !vep->emptytag) {
-				p++;
-				vep->canattr = 1;
-			}
-			if (p < e && *p == '/' && !vep->emptytag) {
-				p++;
-				vep->emptytag = 1;
-				vep->canattr = 0;
-			}
-			if (p < e && *p == '>') {
-				p++;
-				AN(vep->dostuff);
-				vep_mark_skip(vep, p);
-				vep->dostuff(vep, DO_TAG);
-				vep->in_esi_tag = 0;
-				vep->state = VEP_NEXTTAG;
-			} else if (p < e && vep->emptytag) {
-				vep_error(vep,
-				    "XML 1.0 '>' does not follow '/' in tag");
-				vep->state = VEP_TAGERROR;
-			} else if (p < e && vep->canattr &&
-			    vct_isxmlnamestart(*p)) {
-				vep->state = VEP_ATTR;
-			} else if (p < e) {
-				vep_error(vep,
-				    "XML 1.0 Illegal attribute start char");
-				vep->state = VEP_TAGERROR;
-			}
-		} else if (vep->state == VEP_TAGERROR) {
-			while (p < e && *p != '>')
-				p++;
-			if (p < e) {
-				p++;
-				vep_mark_skip(vep, p);
-				vep->in_esi_tag = 0;
-				vep->state = VEP_NEXTTAG;
-			}
-
-		/******************************************************
-		 * SECTION E
-		 */
-
-		} else if (vep->state == VEP_ATTR) {
-			AZ(vep->attr_delim);
-			if (vep->attr == NULL) {
-				p++;
-				AZ(vep->attr_vsb);
-				vep->state = VEP_SKIPATTR;
-			} else {
-				vep->match = vep->attr;
-				vep->state = VEP_MATCH;
-			}
-		} else if (vep->state == VEP_SKIPATTR) {
-			while (p < e && vct_isxmlname(*p))
-				p++;
-			if (p < e && *p == '=') {
-				p++;
-				vep->state = VEP_ATTRDELIM;
-			} else if (p < e && *p == '>') {
-				vep->state = VEP_INTAG;
-			} else if (p < e && *p == '/') {
-				vep->state = VEP_INTAG;
-			} else if (p < e && vct_issp(*p)) {
-				vep->state = VEP_INTAG;
-			} else if (p < e) {
-				vep_error(vep,
-				    "XML 1.0 Illegal attr char");
-				vep->state = VEP_TAGERROR;
-			}
-		} else if (vep->state == VEP_ATTRGETVAL) {
-			vep->attr_vsb = VSB_new_auto();
-			vep->state = VEP_ATTRDELIM;
-		} else if (vep->state == VEP_ATTRDELIM) {
-			AZ(vep->attr_delim);
-			if (*p == '"' || *p == '\'') {
-				vep->attr_delim = *p++;
-				vep->state = VEP_ATTRVAL;
-			} else if (!vct_issp(*p)) {
-				vep->attr_delim = ' ';
-				vep->state = VEP_ATTRVAL;
-			} else {
-				vep_error(vep,
-				    "XML 1.0 Illegal attribute delimiter");
-				vep->state = VEP_TAGERROR;
-			}
-
-		} else if (vep->state == VEP_ATTRVAL) {
-			while (p < e && *p != '>' && *p != vep->attr_delim &&
-			   (vep->attr_delim != ' ' || !vct_issp(*p))) {
-				if (vep->attr_vsb != NULL)
-					VSB_bcat(vep->attr_vsb, p, 1);
-				p++;
-			}
-			if (p < e && *p == '>') {
-				vep_error(vep,
-				    "XML 1.0 Missing end attribute delimiter");
-				vep->state = VEP_TAGERROR;
-				vep->attr_delim = 0;
-				if (vep->attr_vsb != NULL) {
-					AZ(VSB_finish(vep->attr_vsb));
-					VSB_delete(vep->attr_vsb);
-					vep->attr_vsb = NULL;
-				}
-			} else if (p < e) {
-				vep->attr_delim = 0;
-				p++;
-				vep->state = VEP_INTAG;
-				if (vep->attr_vsb != NULL) {
-					AZ(VSB_finish(vep->attr_vsb));
-					AN(vep->dostuff);
-					vep->dostuff(vep, DO_ATTR);
-					vep->attr_vsb = NULL;
-				}
-			}
-
-		/******************************************************
-		 * Utility Section
-		 */
-
-		} else if (vep->state == VEP_MATCH) {
-			/*
-			 * Match against a table
-			 */
-			vm = vep_match(vep, p, e);
-			vep->match_hit = vm;
-			if (vm != NULL) {
-				if (vm->match != NULL)
-					p += strlen(vm->match);
-				vep->state = *vm->state;
-				vep->match = NULL;
-				vep->tag_i = 0;
-			} else {
-				memcpy(vep->tag, p, e - p);
-				vep->tag_i = e - p;
-				vep->state = VEP_MATCHBUF;
-				p = e;
-			}
-		} else if (vep->state == VEP_MATCHBUF) {
-			/*
-			 * Match against a table while split over input
-			 * sections.
-			 */
-			do {
-				if (*p == '>') {
-					for (vm = vep->match;
-					    vm->match != NULL; vm++)
-						continue;
-					AZ(vm->match);
-				} else {
-					vep->tag[vep->tag_i++] = *p++;
-					vm = vep_match(vep,
-					    vep->tag, vep->tag + vep->tag_i);
-					if (vm && vm->match == NULL) {
-						vep->tag_i--;
-						p--;
-					}
-				}
-			} while (vm == NULL && p < e);
-			vep->match_hit = vm;
-			if (vm == NULL) {
-				assert(p == e);
-			} else {
-				vep->state = *vm->state;
-				vep->match = NULL;
-			}
-		} else if (vep->state == VEP_UNTIL) {
-			/*
-			 * Skip until we see magic string
-			 */
-			while (p < e) {
-				if (*p++ != *vep->until_p++) {
-					vep->until_p = vep->until;
-				} else if (*vep->until_p == '\0') {
-					vep->state = vep->until_s;
-					break;
-				}
-			}
-			if (p == e && !vep->remove)
-				vep_mark_verbatim(vep, p);
-		} else {
-			Debug("*** Unknown state %s\n", vep->state);
-			INCOMPL();
-		}
-	}
-	/*
-	 * We must always mark up the storage we got, try to do so
-	 * in the most efficient way, in particular with respect to
-	 * minimizing and limiting use of pending.
-	 */
-	if (p == vep->ver_p)
-		;
-	else if (vep->in_esi_tag)
-		vep_mark_skip(vep, p);
-	else if (vep->remove)
-		vep_mark_skip(vep, p);
-	else
-		vep_mark_pending(vep, p);
-}
-
-/*---------------------------------------------------------------------
- */
-
-static ssize_t __match_proto__()
-vep_default_cb(struct worker *w, ssize_t l, enum vgz_flag flg)
-{
-
-	(void)flg;
-	AN(w->vep);
-	w->vep->cb_x += l;
-Debug("CB(%jd,%d) = %jd\n", (intmax_t)l, flg, (intmax_t)w->vep->cb_x);
-	return (w->vep->cb_x);
-}
-
-/*---------------------------------------------------------------------
- */
-
-void
-VEP_Init(struct worker *w, vep_callback_t *cb)
-{
-	struct vep_state *vep;
-
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	AZ(w->vep);
-	w->vep = (void*)WS_Alloc(w->ws, sizeof *vep);
-	AN(w->vep);
-
-	vep = w->vep;
-	memset(vep, 0, sizeof *vep);
-	vep->magic = VEP_MAGIC;
-	vep->wrk = w;
-	vep->vsb = VSB_new_auto();
-	AN(vep->vsb);
-
-	if (cb != NULL) {
-		vep->dogzip = 1;
-		/* XXX */
-		VSB_printf(vep->vsb, "%c", VEC_GZ);
-		vep->cb = cb;
-	} else {
-		vep->cb = vep_default_cb;
-	}
-
-	vep->state = VEP_START;
-	vep->crc = crc32(0L, Z_NULL, 0);
-	vep->crcp = crc32(0L, Z_NULL, 0);
-
-	/*
-	 * We must force the GZIP header out as a SKIP string, otherwise
-	 * an object starting with <esi:include would have its GZIP header
-	 * appear after the included object (e000026.vtc)
-	 */
-	vep->startup = 1;
-	vep->ver_p = "";
-	vep->last_mark = SKIP;
-	vep_mark_common(vep, vep->ver_p, VERBATIM);
-	vep->startup = 0;
-}
-
-/*---------------------------------------------------------------------
- */
-
-struct vsb *
-VEP_Finish(struct worker *w)
-{
-	struct vep_state *vep;
-	ssize_t l, lcb;
-
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	vep = w->vep;
-	CHECK_OBJ_NOTNULL(vep, VEP_MAGIC);
-
-	if (vep->o_pending)
-		vep_mark_common(vep, vep->ver_p, vep->last_mark);
-	if (vep->o_wait > 0) {
-		lcb = vep->cb(vep->wrk, 0, VGZ_ALIGN);
-		vep_emit_common(vep, lcb - vep->o_last, vep->last_mark);
-	}
-	(void)vep->cb(vep->wrk, 0, VGZ_FINISH);
-
-	w->vep = NULL;
-
-	AZ(VSB_finish(vep->vsb));
-	l = VSB_len(vep->vsb);
-	if (vep->esi_found && l > 0)
-		return (vep->vsb);
-	VSB_delete(vep->vsb);
-	return (NULL);
-}
-
-#if 0
-
-digraph xml {
-	rankdir="LR"
-	size="7,10"
-#################################################################
-# SECTION A
-#
-
-START		[shape=ellipse]
-TESTXML		[shape=ellipse]
-NOTXML		[shape=ellipse]
-NEXTTAGa	[shape=hexagon, label="NEXTTAG"]
-STARTTAGa	[shape=hexagon, label="STARTTAG"]
-START		-> TESTXML
-START		-> NEXTTAGa	[style=dotted, label="syntax:1"]
-TESTXML		-> TESTXML	[label="lws"]
-TESTXML		-> NOTXML
-TESTXML		-> STARTTAGa	[label="'<'"]
-
-#################################################################
-# SECTION B
-
-NOTMYTAG	[shape=ellipse]
-NEXTTAG		[shape=ellipse]
-NOTMYTAG	-> NEXTTAG	[style=dotted, label="syntax:2"]
-STARTTAGb	[shape=hexagon, label="STARTTAG"]
-NOTMYTAG	-> NEXTTAG	[label="'>'"]
-NOTMYTAG	-> NOTMYTAG	[label="*"]
-NEXTTAG		-> NEXTTAG	[label="'-->'"]
-NEXTTAG		-> NEXTTAG	[label="*"]
-NEXTTAG		-> STARTTAGb	[label="'<'"]
-
-#################################################################
-# SECTION C
-
-STARTTAG	[shape=ellipse]
-COMMENT		[shape=ellipse]
-CDATA		[shape=ellipse]
-ESITAG		[shape=ellipse]
-ESIETAG		[shape=ellipse]
-ESIINCLUDE	[shape=ellipse]
-ESIREMOVE	[shape=ellipse]
-ESICOMMENT	[shape=ellipse]
-ESIBOGON	[shape=ellipse]
-INTAGc		[shape=hexagon, label="INTAG"]
-NOTMYTAGc	[shape=hexagon, label="NOTMYTAG"]
-NEXTTAGc	[shape=hexagon, label="NEXTTAG"]
-TAGERRORc	[shape=hexagon, label="TAGERROR"]
-C1		[shape=circle,label=""]
-STARTTAG	-> COMMENT	[label="'<!--'"]
-STARTTAG	-> ESITAG	[label="'<esi'"]
-STARTTAG	-> CDATA	[label="'<![CDATA['"]
-STARTTAG	-> NOTMYTAGc	[label="'*'"]
-COMMENT		-> NEXTTAGc	[label="'esi'"]
-COMMENT		-> C1		[label="*"]
-C1		-> C1		[label="*"]
-C1		-> NEXTTAGc	[label="-->"]
-CDATA		-> CDATA	[label="*"]
-CDATA		-> NEXTTAGc	[label="]]>"]
-ESITAG		-> ESIINCLUDE	[label="'include'"]
-ESITAG		-> ESIREMOVE	[label="'remove'"]
-ESITAG		-> ESICOMMENT	[label="'comment'"]
-ESITAG		-> ESIBOGON	[label="*"]
-ESICOMMENT	-> INTAGc
-ESICOMMENT	-> TAGERRORc
-ESICOMMENT	-> TAGERRORc	[style=dotted, label="nested\nin\nremove"]
-ESIREMOVE	-> INTAGc
-ESIREMOVE	-> TAGERRORc
-ESIINCLUDE	-> INTAGc
-ESIINCLUDE	-> TAGERRORc
-ESIINCLUDE	-> TAGERRORc	[style=dotted, label="nested\nin\nremove"]
-ESIBOGON	-> TAGERRORc
-
-#################################################################
-# SECTION D
-
-INTAG		[shape=ellipse]
-TAGERROR	[shape=ellipse]
-NEXTTAGd	[shape=hexagon, label="NEXTTAG"]
-ATTRd		[shape=hexagon, label="ATTR"]
-D1		[shape=circle, label=""]
-D2		[shape=circle, label=""]
-INTAG		-> D1		[label="lws"]
-D1		-> D2		[label="/"]
-INTAG		-> D2		[label="/"]
-INTAG		-> NEXTTAGd	[label=">"]
-D1		-> NEXTTAGd	[label=">"]
-D2		-> NEXTTAGd	[label=">"]
-D1		-> ATTRd	[label="XMLstartchar"]
-D1		-> TAGERROR	[label="*"]
-D2		-> TAGERROR	[label="*"]
-TAGERROR	-> TAGERROR	[label="*"]
-TAGERROR	-> NEXTTAGd	[label="'>'"]
-
-#################################################################
-# SECTION E
-
-ATTR		[shape=ellipse]
-SKIPATTR	[shape=ellipse]
-ATTRGETVAL	[shape=ellipse]
-ATTRDELIM	[shape=ellipse]
-ATTRVAL		[shape=ellipse]
-TAGERRORe	[shape=hexagon, label="TAGERROR"]
-INTAGe		[shape=hexagon, label="INTAG"]
-ATTR		-> SKIPATTR	[label="*"]
-ATTR		-> ATTRGETVAL	[label="wanted attr"]
-SKIPATTR	-> SKIPATTR	[label="XMLname"]
-SKIPATTR	-> ATTRDELIM	[label="'='"]
-SKIPATTR	-> TAGERRORe	[label="*"]
-ATTRGETVAL	-> ATTRDELIM
-ATTRDELIM	-> ATTRVAL	[label="\""]
-ATTRDELIM	-> ATTRVAL	[label="\'"]
-ATTRDELIM	-> ATTRVAL	[label="*"]
-ATTRDELIM	-> TAGERRORe	[label="lws"]
-ATTRVAL		-> TAGERRORe	[label="'>'"]
-ATTRVAL		-> INTAGe	[label="delim"]
-ATTRVAL		-> ATTRVAL	[label="*"]
-
-}
-
-#endif
diff --git a/bin/varnishd/cache_expire.c b/bin/varnishd/cache_expire.c
deleted file mode 100644
index 23e3fc6..0000000
--- a/bin/varnishd/cache_expire.c
+++ /dev/null
@@ -1,490 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * LRU and object timer handling.
- *
- * We have two data structures, a LRU-list and a binary heap for the timers
- * and two ways to kill objects: TTL-timeouts and LRU cleanups.
- *
- * Any object on the LRU is also on the binheap and vice versa.
- *
- * We hold a single object reference for both data structures.
- *
- * An attempted overview:
- *
- *	                        EXP_Ttl()      EXP_Grace()   EXP_Keep()
- *				   |                |            |
- *      entered                    v                v            |
- *         |                       +--------------->+            |
- *         v                       |      grace                  |
- *         +---------------------->+                             |
- *                  ttl            |                             v
- *                                 +---------------------------->+
- *                                     keep
- *
- */
-
-#include "config.h"
-
-#include <math.h>
-
-#include "cache.h"
-
-#include "binary_heap.h"
-#include "hash/hash_slinger.h"
-#include "vtim.h"
-
-static pthread_t exp_thread;
-static struct binheap *exp_heap;
-static struct lock exp_mtx;
-
-/*--------------------------------------------------------------------
- * struct exp manipulations
- *
- * The Get/Set functions encapsulate the mutual magic between the
- * fields in one single place.
- */
-
-void
-EXP_Clr(struct exp *e)
-{
-
-	e->ttl = -1;
-	e->grace = -1;
-	e->keep = -1;
-	e->age = 0;
-	e->entered = 0;
-}
-
-#define EXP_ACCESS(fld, low_val, extra)				\
-	double							\
-	EXP_Get_##fld(const struct exp *e)			\
-	{							\
-		return (e->fld > 0. ? e->fld : low_val);	\
-	}							\
-								\
-	void							\
-	EXP_Set_##fld(struct exp *e, double v)			\
-	{							\
-		if (v > 0.)					\
-			e->fld = v;				\
-		else {						\
-			e->fld = -1.;				\
-			extra;					\
-		}						\
-	}							\
-
-EXP_ACCESS(ttl, -1., (e->grace = e->keep = -1.))
-EXP_ACCESS(grace, 0., )
-EXP_ACCESS(keep, 0.,)
-
-/*--------------------------------------------------------------------
- * Calculate an objects effective keep, grace or ttl time, suitably
- * adjusted for defaults and by per-session limits.
- */
-
-static double
-EXP_Keep(const struct sess *sp, const struct object *o)
-{
-	double r;
-
-	r = (double)cache_param->default_keep;
-	if (o->exp.keep > 0.)
-		r = o->exp.keep;
-	if (sp != NULL && sp->exp.keep > 0. && sp->exp.keep < r)
-		r = sp->exp.keep;
-	return (EXP_Ttl(sp, o) + r);
-}
-
-double
-EXP_Grace(const struct sess *sp, const struct object *o)
-{
-	double r;
-
-	r = (double)cache_param->default_grace;
-	if (o->exp.grace >= 0.)
-		r = o->exp.grace;
-	if (sp != NULL && sp->exp.grace > 0. && sp->exp.grace < r)
-		r = sp->exp.grace;
-	return (EXP_Ttl(sp, o) + r);
-}
-
-double
-EXP_Ttl(const struct sess *sp, const struct object *o)
-{
-	double r;
-
-	r = o->exp.ttl;
-	if (sp != NULL && sp->exp.ttl > 0. && sp->exp.ttl < r)
-		r = sp->exp.ttl;
-	return (o->exp.entered + r);
-}
-
-/*--------------------------------------------------------------------
- * When & why does the timer fire for this object ?
- */
-
-static int
-update_object_when(const struct object *o)
-{
-	struct objcore *oc;
-	double when, w2;
-
-	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
-	oc = o->objcore;
-	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-	Lck_AssertHeld(&exp_mtx);
-
-	when = EXP_Keep(NULL, o);
-	w2 = EXP_Grace(NULL, o);
-	if (w2 > when)
-		when = w2;
-	assert(!isnan(when));
-	if (when == oc->timer_when)
-		return (0);
-	oc->timer_when = when;
-	return (1);
-}
-
-/*--------------------------------------------------------------------*/
-
-static void
-exp_insert(struct objcore *oc, struct lru *lru)
-{
-	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
-
-	Lck_AssertHeld(&lru->mtx);
-	Lck_AssertHeld(&exp_mtx);
-	assert(oc->timer_idx == BINHEAP_NOIDX);
-	binheap_insert(exp_heap, oc);
-	assert(oc->timer_idx != BINHEAP_NOIDX);
-	VTAILQ_INSERT_TAIL(&lru->lru_head, oc, lru_list);
-}
-
-/*--------------------------------------------------------------------
- * Object has been added to cache, record in lru & binheap.
- *
- * The objcore comes with a reference, which we inherit.
- */
-
-void
-EXP_Inject(struct objcore *oc, struct lru *lru, double when)
-{
-
-	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
-
-	Lck_Lock(&lru->mtx);
-	Lck_Lock(&exp_mtx);
-	oc->timer_when = when;
-	exp_insert(oc, lru);
-	Lck_Unlock(&exp_mtx);
-	Lck_Unlock(&lru->mtx);
-}
-
-/*--------------------------------------------------------------------
- * Object has been added to cache, record in lru & binheap.
- *
- * We grab a reference to the object, which will keep it around until
- * we decide its time to let it go.
- */
-
-void
-EXP_Insert(struct object *o)
-{
-	struct objcore *oc;
-	struct lru *lru;
-
-	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
-	oc = o->objcore;
-	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-	AssertObjBusy(o);
-	HSH_Ref(oc);
-
-	assert(o->exp.entered != 0 && !isnan(o->exp.entered));
-	o->last_lru = o->exp.entered;
-
-	lru = oc_getlru(oc);
-	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
-	Lck_Lock(&lru->mtx);
-	Lck_Lock(&exp_mtx);
-	(void)update_object_when(o);
-	exp_insert(oc, lru);
-	Lck_Unlock(&exp_mtx);
-	Lck_Unlock(&lru->mtx);
-	oc_updatemeta(oc);
-}
-
-/*--------------------------------------------------------------------
- * Object was used, move to tail of LRU list.
- *
- * To avoid the exp_mtx becoming a hotspot, we only attempt to move
- * objects if they have not been moved recently and if the lock is available.
- * This optimization obviously leaves the LRU list imperfectly sorted.
- */
-
-int
-EXP_Touch(struct objcore *oc)
-{
-	struct lru *lru;
-
-	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-
-	/*
-	 * For -spersistent we don't move objects on the lru list.  Each
-	 * segment has its own LRU list, and the order on it is not material
-	 * for anything.  The code below would move the objects to the
-	 * LRU list of the currently open segment, which would prevent
-	 * the cleaner from doing its job.
-	 */
-	if (oc->flags & OC_F_LRUDONTMOVE)
-		return (0);
-
-	lru = oc_getlru(oc);
-	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
-
-	/*
-	 * We only need the LRU lock here.  The locking order is LRU->EXP
-	 * so we can trust the content of the oc->timer_idx without the
-	 * EXP lock.   Since each lru list has its own lock, this should
-	 * reduce contention a fair bit
-	 */
-	if (Lck_Trylock(&lru->mtx))
-		return (0);
-
-	if (oc->timer_idx != BINHEAP_NOIDX) {
-		VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
-		VTAILQ_INSERT_TAIL(&lru->lru_head, oc, lru_list);
-		VSC_C_main->n_lru_moved++;
-	}
-	Lck_Unlock(&lru->mtx);
-	return (1);
-}
-
-/*--------------------------------------------------------------------
- * We have changed one or more of the object timers, shuffle it
- * accordingly in the binheap
- *
- * The VCL code can send us here on a non-cached object, just return.
- *
- * XXX: special case check for ttl = 0 ?
- */
-
-void
-EXP_Rearm(const struct object *o)
-{
-	struct objcore *oc;
-	struct lru *lru;
-
-	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
-	oc = o->objcore;
-	if (oc == NULL)
-		return;
-	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-	lru = oc_getlru(oc);
-	Lck_Lock(&lru->mtx);
-	Lck_Lock(&exp_mtx);
-	/*
-	 * The hang-man might have this object of the binheap while
-	 * tending to a timer.  If so, we do not muck with it here.
-	 */
-	if (oc->timer_idx != BINHEAP_NOIDX && update_object_when(o)) {
-		assert(oc->timer_idx != BINHEAP_NOIDX);
-		binheap_reorder(exp_heap, oc->timer_idx);
-		assert(oc->timer_idx != BINHEAP_NOIDX);
-	}
-	Lck_Unlock(&exp_mtx);
-	Lck_Unlock(&lru->mtx);
-	oc_updatemeta(oc);
-}
-
-/*--------------------------------------------------------------------
- * This thread monitors the root of the binary heap and whenever an
- * object expires, accounting also for graceability, it is killed.
- */
-
-static void * __match_proto__(void *start_routine(void *))
-exp_timer(struct sess *sp, void *priv)
-{
-	struct objcore *oc;
-	struct lru *lru;
-	double t;
-	struct object *o;
-
-	(void)priv;
-	t = VTIM_real();
-	oc = NULL;
-	while (1) {
-		if (oc == NULL) {
-			WSL_Flush(sp->wrk, 0);
-			WRK_SumStat(sp->wrk);
-			VTIM_sleep(cache_param->expiry_sleep);
-			t = VTIM_real();
-		}
-
-		Lck_Lock(&exp_mtx);
-		oc = binheap_root(exp_heap);
-		if (oc == NULL) {
-			Lck_Unlock(&exp_mtx);
-			continue;
-		}
-		CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-
-		/*
-		 * We may have expired so many objects that our timestamp
-		 * got out of date, refresh it and check again.
-		 */
-		if (oc->timer_when > t)
-			t = VTIM_real();
-		if (oc->timer_when > t) {
-			Lck_Unlock(&exp_mtx);
-			oc = NULL;
-			continue;
-		}
-
-		/*
-		 * It's time...
-		 * Technically we should drop the exp_mtx, get the lru->mtx
-		 * get the exp_mtx again and then check that the oc is still
-		 * on the binheap.  We take the shorter route and try to
-		 * get the lru->mtx and punt if we fail.
-		 */
-
-		lru = oc_getlru(oc);
-		CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
-		if (Lck_Trylock(&lru->mtx)) {
-			Lck_Unlock(&exp_mtx);
-			oc = NULL;
-			continue;
-		}
-
-		/* Remove from binheap */
-		assert(oc->timer_idx != BINHEAP_NOIDX);
-		binheap_delete(exp_heap, oc->timer_idx);
-		assert(oc->timer_idx == BINHEAP_NOIDX);
-
-		/* And from LRU */
-		lru = oc_getlru(oc);
-		VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
-
-		Lck_Unlock(&exp_mtx);
-		Lck_Unlock(&lru->mtx);
-
-		VSC_C_main->n_expired++;
-
-		CHECK_OBJ_NOTNULL(oc->objhead, OBJHEAD_MAGIC);
-		o = oc_getobj(sp->wrk, oc);
-		WSL(sp->wrk, SLT_ExpKill, 0, "%u %.0f",
-		    o->xid, EXP_Ttl(NULL, o) - t);
-		(void)HSH_Deref(sp->wrk, oc, NULL);
-	}
-	NEEDLESS_RETURN(NULL);
-}
-
-/*--------------------------------------------------------------------
- * Attempt to make space by nuking the oldest object on the LRU list
- * which isn't in use.
- * Returns: 1: did, 0: didn't, -1: can't
- */
-
-int
-EXP_NukeOne(struct worker *w, struct lru *lru)
-{
-	struct objcore *oc;
-	struct object *o;
-
-	/* Find the first currently unused object on the LRU.  */
-	Lck_Lock(&lru->mtx);
-	Lck_Lock(&exp_mtx);
-	VTAILQ_FOREACH(oc, &lru->lru_head, lru_list) {
-		CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-		assert (oc->timer_idx != BINHEAP_NOIDX);
-		/*
-		 * It wont release any space if we cannot release the last
-		 * reference, besides, if somebody else has a reference,
-		 * it's a bad idea to nuke this object anyway.
-		 */
-		if (oc->refcnt == 1)
-			break;
-	}
-	if (oc != NULL) {
-		VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
-		binheap_delete(exp_heap, oc->timer_idx);
-		assert(oc->timer_idx == BINHEAP_NOIDX);
-		VSC_C_main->n_lru_nuked++;
-	}
-	Lck_Unlock(&exp_mtx);
-	Lck_Unlock(&lru->mtx);
-
-	if (oc == NULL)
-		return (-1);
-
-	/* XXX: bad idea for -spersistent */
-	o = oc_getobj(w, oc);
-	WSL(w, SLT_ExpKill, 0, "%u LRU", o->xid);
-	(void)HSH_Deref(w, NULL, &o);
-	return (1);
-}
-
-/*--------------------------------------------------------------------
- * BinHeap helper functions for objcore.
- */
-
-static int
-object_cmp(void *priv, void *a, void *b)
-{
-	struct objcore *aa, *bb;
-
-	(void)priv;
-	CAST_OBJ_NOTNULL(aa, a, OBJCORE_MAGIC);
-	CAST_OBJ_NOTNULL(bb, b, OBJCORE_MAGIC);
-	return (aa->timer_when < bb->timer_when);
-}
-
-static void
-object_update(void *priv, void *p, unsigned u)
-{
-	struct objcore *oc;
-
-	(void)priv;
-	CAST_OBJ_NOTNULL(oc, p, OBJCORE_MAGIC);
-	oc->timer_idx = u;
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-EXP_Init(void)
-{
-
-	Lck_New(&exp_mtx, lck_exp);
-	exp_heap = binheap_new(NULL, object_cmp, object_update);
-	XXXAN(exp_heap);
-	WRK_BgThread(&exp_thread, "cache-timeout", exp_timer, NULL);
-}
diff --git a/bin/varnishd/cache_fetch.c b/bin/varnishd/cache_fetch.c
deleted file mode 100644
index a678dcc..0000000
--- a/bin/varnishd/cache_fetch.c
+++ /dev/null
@@ -1,645 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- */
-
-#include "config.h"
-
-#include <inttypes.h>
-#include <stdio.h>
-#include <stdlib.h>
-
-#include "cache.h"
-
-#include "cache_backend.h"
-#include "vcli_priv.h"
-#include "vct.h"
-#include "vtcp.h"
-
-static unsigned fetchfrag;
-
-/*--------------------------------------------------------------------
- * We want to issue the first error we encounter on fetching and
- * supress the rest.  This function does that.
- *
- * Other code is allowed to look at w->fetch_failed to bail out
- *
- * For convenience, always return -1
- */
-
-int
-FetchError2(struct worker *w, const char *error, const char *more)
-{
-
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	if (!w->fetch_failed) {
-		if (more == NULL)
-			WSLB(w, SLT_FetchError, "%s", error);
-		else
-			WSLB(w, SLT_FetchError, "%s: %s", error, more);
-	}
-	w->fetch_failed = 1;
-	return (-1);
-}
-
-int
-FetchError(struct worker *w, const char *error)
-{
-	return(FetchError2(w, error, NULL));
-}
-
-/*--------------------------------------------------------------------
- * VFP_NOP
- *
- * This fetch-processor does nothing but store the object.
- * It also documents the API
- */
-
-/*--------------------------------------------------------------------
- * VFP_BEGIN
- *
- * Called to set up stuff.
- *
- * 'estimate' is the estimate of the number of bytes we expect to receive,
- * as seen on the socket, or zero if unknown.
- */
-static void __match_proto__()
-vfp_nop_begin(struct worker *w, size_t estimate)
-{
-
-	if (estimate > 0)
-		(void)FetchStorage(w, estimate);
-}
-
-/*--------------------------------------------------------------------
- * VFP_BYTES
- *
- * Process (up to) 'bytes' from the socket.
- *
- * Return -1 on error, issue FetchError()
- *	will not be called again, once error happens.
- * Return 0 on EOF on socket even if bytes not reached.
- * Return 1 when 'bytes' have been processed.
- */
-
-static int __match_proto__()
-vfp_nop_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
-{
-	ssize_t l, wl;
-	struct storage *st;
-
-	AZ(w->fetch_failed);
-	while (bytes > 0) {
-		st = FetchStorage(w, 0);
-		if (st == NULL)
-			return(-1);
-		l = st->space - st->len;
-		if (l > bytes)
-			l = bytes;
-		wl = HTC_Read(w, htc, st->ptr + st->len, l);
-		if (wl <= 0)
-			return (wl);
-		st->len += wl;
-		w->fetch_obj->len += wl;
-		bytes -= wl;
-		if (w->do_stream)
-			RES_StreamPoll(w);
-	}
-	return (1);
-}
-
-/*--------------------------------------------------------------------
- * VFP_END
- *
- * Finish & cleanup
- *
- * Return -1 for error
- * Return 0 for OK
- */
-
-static int __match_proto__()
-vfp_nop_end(struct worker *w)
-{
-	struct storage *st;
-
-	st = VTAILQ_LAST(&w->fetch_obj->store, storagehead);
-	if (st == NULL)
-		return (0);
-
-	if (st->len == 0) {
-		VTAILQ_REMOVE(&w->fetch_obj->store, st, list);
-		STV_free(st);
-		return (0);
-	}
-	if (st->len < st->space)
-		STV_trim(st, st->len);
-	return (0);
-}
-
-static struct vfp vfp_nop = {
-	.begin	=	vfp_nop_begin,
-	.bytes	=	vfp_nop_bytes,
-	.end	=	vfp_nop_end,
-};
-
-/*--------------------------------------------------------------------
- * Fetch Storage to put object into.
- *
- */
-
-struct storage *
-FetchStorage(struct worker *w, ssize_t sz)
-{
-	ssize_t l;
-	struct storage *st;
-	struct object *obj;
-
-	obj = w->fetch_obj;
-	CHECK_OBJ_NOTNULL(obj, OBJECT_MAGIC);
-	st = VTAILQ_LAST(&obj->store, storagehead);
-	if (st != NULL && st->len < st->space)
-		return (st);
-
-	l = fetchfrag;
-	if (l == 0)
-		l = sz;
-	if (l == 0)
-		l = cache_param->fetch_chunksize * 1024LL;
-	st = STV_alloc(w, l);
-	if (st == NULL) {
-		(void)FetchError(w, "Could not get storage");
-		return (NULL);
-	}
-	AZ(st->len);
-	VTAILQ_INSERT_TAIL(&obj->store, st, list);
-	return (st);
-}
-
-/*--------------------------------------------------------------------
- * Convert a string to a size_t safely
- */
-
-static ssize_t
-fetch_number(const char *nbr, int radix)
-{
-	uintmax_t cll;
-	ssize_t cl;
-	char *q;
-
-	if (*nbr == '\0')
-		return (-1);
-	cll = strtoumax(nbr, &q, radix);
-	if (q == NULL || *q != '\0')
-		return (-1);
-
-	cl = (ssize_t)cll;
-	if((uintmax_t)cl != cll) /* Protect against bogusly large values */
-		return (-1);
-	return (cl);
-}
-
-/*--------------------------------------------------------------------*/
-
-static int
-fetch_straight(struct worker *w, struct http_conn *htc, ssize_t cl)
-{
-	int i;
-
-	assert(w->body_status == BS_LENGTH);
-
-	if (cl < 0) {
-		return (FetchError(w, "straight length field bogus"));
-	} else if (cl == 0)
-		return (0);
-
-	i = w->vfp->bytes(w, htc, cl);
-	if (i <= 0)
-		return (FetchError(w, "straight insufficient bytes"));
-	return (0);
-}
-
-/*--------------------------------------------------------------------
- * Read a chunked HTTP object.
- *
- * XXX: Reading one byte at a time is pretty pessimal.
- */
-
-static int
-fetch_chunked(struct worker *w, struct http_conn *htc)
-{
-	int i;
-	char buf[20];		/* XXX: 20 is arbitrary */
-	unsigned u;
-	ssize_t cl;
-
-	assert(w->body_status == BS_CHUNKED);
-	do {
-		/* Skip leading whitespace */
-		do {
-			if (HTC_Read(w, htc, buf, 1) <= 0)
-				return (-1);
-		} while (vct_islws(buf[0]));
-
-		if (!vct_ishex(buf[0]))
-			return (FetchError(w,"chunked header non-hex"));
-
-		/* Collect hex digits, skipping leading zeros */
-		for (u = 1; u < sizeof buf; u++) {
-			do {
-				if (HTC_Read(w, htc, buf + u, 1) <= 0)
-					return (-1);
-			} while (u == 1 && buf[0] == '0' && buf[u] == '0');
-			if (!vct_ishex(buf[u]))
-				break;
-		}
-
-		if (u >= sizeof buf)
-			return (FetchError(w,"chunked header too long"));
-
-		/* Skip trailing white space */
-		while(vct_islws(buf[u]) && buf[u] != '\n')
-			if (HTC_Read(w, htc, buf + u, 1) <= 0)
-				return (-1);
-
-		if (buf[u] != '\n')
-			return (FetchError(w,"chunked header no NL"));
-
-		buf[u] = '\0';
-		cl = fetch_number(buf, 16);
-		if (cl < 0)
-			return (FetchError(w,"chunked header number syntax"));
-
-		if (cl > 0 && w->vfp->bytes(w, htc, cl) <= 0)
-			return (-1);
-
-		i = HTC_Read(w, htc, buf, 1);
-		if (i <= 0)
-			return (-1);
-		if (buf[0] == '\r' && HTC_Read(w, htc, buf, 1) <= 0)
-			return (-1);
-		if (buf[0] != '\n')
-			return (FetchError(w,"chunked tail no NL"));
-	} while (cl > 0);
-	return (0);
-}
-
-/*--------------------------------------------------------------------*/
-
-static int
-fetch_eof(struct worker *w, struct http_conn *htc)
-{
-	int i;
-
-	assert(w->body_status == BS_EOF);
-	i = w->vfp->bytes(w, htc, SSIZE_MAX);
-	if (i < 0)
-		return (-1);
-	return (0);
-}
-
-/*--------------------------------------------------------------------
- * Fetch any body attached to the incoming request, and either write it
- * to the backend (if we pass) or discard it (anything else).
- * This is mainly a separate function to isolate the stack buffer and
- * to contain the complexity when we start handling chunked encoding.
- */
-
-int
-FetchReqBody(struct sess *sp)
-{
-	unsigned long content_length;
-	char buf[8192];
-	char *ptr, *endp;
-	int rdcnt;
-
-	if (http_GetHdr(sp->http, H_Content_Length, &ptr)) {
-
-		content_length = strtoul(ptr, &endp, 10);
-		/* XXX should check result of conversion */
-		while (content_length) {
-			if (content_length > sizeof buf)
-				rdcnt = sizeof buf;
-			else
-				rdcnt = content_length;
-			rdcnt = HTC_Read(sp->wrk, sp->htc, buf, rdcnt);
-			if (rdcnt <= 0)
-				return (1);
-			content_length -= rdcnt;
-			if (!sp->sendbody)
-				continue;
-			(void)WRW_Write(sp->wrk, buf, rdcnt); /* XXX: stats ? */
-			if (WRW_Flush(sp->wrk))
-				return (2);
-		}
-	}
-	if (http_GetHdr(sp->http, H_Transfer_Encoding, NULL)) {
-		/* XXX: Handle chunked encoding. */
-		WSP(sp, SLT_Debug, "Transfer-Encoding in request");
-		return (1);
-	}
-	return (0);
-}
-
-/*--------------------------------------------------------------------
- * Send request, and receive the HTTP protocol response, but not the
- * response body.
- *
- * Return value:
- *	-1 failure, not retryable
- *	 0 success
- *	 1 failure which can be retried.
- */
-
-int
-FetchHdr(struct sess *sp)
-{
-	struct vbc *vc;
-	struct worker *w;
-	char *b;
-	struct http *hp;
-	int retry = -1;
-	int i;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->wrk, WORKER_MAGIC);
-	w = sp->wrk;
-
-	AN(sp->director);
-	AZ(sp->obj);
-
-	if (sp->objcore != NULL) {		/* pass has no objcore */
-		CHECK_OBJ_NOTNULL(sp->objcore, OBJCORE_MAGIC);
-		AN(sp->objcore->flags & OC_F_BUSY);
-	}
-
-	hp = w->bereq;
-
-	sp->wrk->vbc = VDI_GetFd(NULL, sp);
-	if (sp->wrk->vbc == NULL) {
-		WSP(sp, SLT_FetchError, "no backend connection");
-		return (-1);
-	}
-	vc = sp->wrk->vbc;
-	if (vc->recycled)
-		retry = 1;
-
-	/*
-	 * Now that we know our backend, we can set a default Host:
-	 * header if one is necessary.  This cannot be done in the VCL
-	 * because the backend may be chosen by a director.
-	 */
-	if (!http_GetHdr(hp, H_Host, &b))
-		VDI_AddHostHeader(sp);
-
-	(void)VTCP_blocking(vc->fd);	/* XXX: we should timeout instead */
-	WRW_Reserve(w, &vc->fd);
-	(void)http_Write(w, vc->vsl_id, hp, 0);	/* XXX: stats ? */
-
-	/* Deal with any message-body the request might have */
-	i = FetchReqBody(sp);
-	if (WRW_FlushRelease(w) || i > 0) {
-		WSP(sp, SLT_FetchError, "backend write error: %d (%s)",
-		    errno, strerror(errno));
-		VDI_CloseFd(sp->wrk);
-		/* XXX: other cleanup ? */
-		return (retry);
-	}
-
-	/* Checkpoint the vsl.here */
-	WSL_Flush(w, 0);
-
-	/* XXX is this the right place? */
-	VSC_C_main->backend_req++;
-
-	/* Receive response */
-
-	HTC_Init(w->htc, w->ws, vc->fd, vc->vsl_id, cache_param->http_resp_size,
-	    cache_param->http_resp_hdr_len);
-
-	VTCP_set_read_timeout(vc->fd, vc->first_byte_timeout);
-
-	i = HTC_Rx(w->htc);
-
-	if (i < 0) {
-		WSP(sp, SLT_FetchError, "http first read error: %d %d (%s)",
-		    i, errno, strerror(errno));
-		VDI_CloseFd(sp->wrk);
-		/* XXX: other cleanup ? */
-		/* Retryable if we never received anything */
-		return (i == -1 ? retry : -1);
-	}
-
-	VTCP_set_read_timeout(vc->fd, vc->between_bytes_timeout);
-
-	while (i == 0) {
-		i = HTC_Rx(w->htc);
-		if (i < 0) {
-			WSP(sp, SLT_FetchError,
-			    "http first read error: %d %d (%s)",
-			    i, errno, strerror(errno));
-			VDI_CloseFd(sp->wrk);
-			/* XXX: other cleanup ? */
-			return (-1);
-		}
-	}
-
-	hp = w->beresp;
-
-	if (http_DissectResponse(w, w->htc, hp)) {
-		WSP(sp, SLT_FetchError, "http format error");
-		VDI_CloseFd(sp->wrk);
-		/* XXX: other cleanup ? */
-		return (-1);
-	}
-	return (0);
-}
-
-/*--------------------------------------------------------------------*/
-
-int
-FetchBody(struct worker *w, struct object *obj)
-{
-	int cls;
-	struct storage *st;
-	int mklen;
-	ssize_t cl;
-
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	AZ(w->fetch_obj);
-	CHECK_OBJ_NOTNULL(w->vbc, VBC_MAGIC);
-	CHECK_OBJ_NOTNULL(obj, OBJECT_MAGIC);
-	CHECK_OBJ_NOTNULL(obj->http, HTTP_MAGIC);
-
-	if (w->vfp == NULL)
-		w->vfp = &vfp_nop;
-
-	AssertObjCorePassOrBusy(obj->objcore);
-
-	AZ(w->vgz_rx);
-	AZ(VTAILQ_FIRST(&obj->store));
-
-	w->fetch_obj = obj;
-	w->fetch_failed = 0;
-
-	/* XXX: pick up estimate from objdr ? */
-	cl = 0;
-	switch (w->body_status) {
-	case BS_NONE:
-		cls = 0;
-		mklen = 0;
-		break;
-	case BS_ZERO:
-		cls = 0;
-		mklen = 1;
-		break;
-	case BS_LENGTH:
-		cl = fetch_number( w->h_content_length, 10);
-		w->vfp->begin(w, cl > 0 ? cl : 0);
-		cls = fetch_straight(w, w->htc, cl);
-		mklen = 1;
-		if (w->vfp->end(w))
-			cls = -1;
-		break;
-	case BS_CHUNKED:
-		w->vfp->begin(w, cl);
-		cls = fetch_chunked(w, w->htc);
-		mklen = 1;
-		if (w->vfp->end(w))
-			cls = -1;
-		break;
-	case BS_EOF:
-		w->vfp->begin(w, cl);
-		cls = fetch_eof(w, w->htc);
-		mklen = 1;
-		if (w->vfp->end(w))
-			cls = -1;
-		break;
-	case BS_ERROR:
-		cls = 1;
-		mklen = 0;
-		break;
-	default:
-		cls = 0;
-		mklen = 0;
-		INCOMPL();
-	}
-	AZ(w->vgz_rx);
-
-	/*
-	 * It is OK for ->end to just leave the last storage segment
-	 * sitting on w->storage, we will always call vfp_nop_end()
-	 * to get it trimmed or thrown out if empty.
-	 */
-	AZ(vfp_nop_end(w));
-
-	w->fetch_obj = NULL;
-
-	WSLB(w, SLT_Fetch_Body, "%u(%s) cls %d mklen %u",
-	    w->body_status, body_status(w->body_status),
-	    cls, mklen);
-
-	if (w->body_status == BS_ERROR) {
-		VDI_CloseFd(w);
-		return (__LINE__);
-	}
-
-	if (cls < 0) {
-		w->stats.fetch_failed++;
-		/* XXX: Wouldn't this store automatically be released ? */
-		while (!VTAILQ_EMPTY(&obj->store)) {
-			st = VTAILQ_FIRST(&obj->store);
-			VTAILQ_REMOVE(&obj->store, st, list);
-			STV_free(st);
-		}
-		VDI_CloseFd(w);
-		obj->len = 0;
-		return (__LINE__);
-	}
-	AZ(w->fetch_failed);
-
-	if (cls == 0 && w->do_close)
-		cls = 1;
-
-	WSLB(w, SLT_Length, "%u", obj->len);
-
-	{
-	/* Sanity check fetch methods accounting */
-		ssize_t uu;
-
-		uu = 0;
-		VTAILQ_FOREACH(st, &obj->store, list)
-			uu += st->len;
-		if (w->do_stream)
-			/* Streaming might have started freeing stuff */
-			assert (uu <= obj->len);
-
-		else
-			assert(uu == obj->len);
-	}
-
-	if (mklen > 0) {
-		http_Unset(obj->http, H_Content_Length);
-		http_PrintfHeader(w, w->vbc->vsl_id, obj->http,
-		    "Content-Length: %jd", (intmax_t)obj->len);
-	}
-
-	if (cls)
-		VDI_CloseFd(w);
-	else
-		VDI_RecycleFd(w);
-
-	return (0);
-}
-
-/*--------------------------------------------------------------------
- * Debugging aids
- */
-
-static void
-debug_fragfetch(struct cli *cli, const char * const *av, void *priv)
-{
-	(void)priv;
-	(void)cli;
-	fetchfrag = strtoul(av[2], NULL, 0);
-}
-
-static struct cli_proto debug_cmds[] = {
-	{ "debug.fragfetch", "debug.fragfetch",
-		"\tEnable fetch fragmentation\n", 1, 1, "d", debug_fragfetch },
-	{ NULL }
-};
-
-/*--------------------------------------------------------------------
- *
- */
-
-void
-Fetch_Init(void)
-{
-
-	CLI_AddFuncs(debug_cmds);
-}
diff --git a/bin/varnishd/cache_gzip.c b/bin/varnishd/cache_gzip.c
deleted file mode 100644
index 32a7413..0000000
--- a/bin/varnishd/cache_gzip.c
+++ /dev/null
@@ -1,697 +0,0 @@
-/*-
- * Copyright (c) 2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * Interaction with the linvgz (zlib) library.
- *
- * The zlib library pollutes namespace a LOT when you include the "vgz.h"
- * (aka (zlib.h") file so we contain the damage by vectoring all access
- * to libz through this source file.
- *
- * The API defined by this file, will also insulate the rest of the code,
- * should we find a better gzip library at a later date.
- *
- * The absolutely worst case gzip processing path, once we have pipe-lining,
- * will be the following, so we need to be a bit careful with the scratch
- * space we use:
- *
- *	Backend		Tmp	Input	Output
- *         |		----------------------
- *	   v
- *	 gunzip		wrk	stack	?
- *         |
- *	   v
- *	  esi
- *         |
- *	   v
- *	  gzip		wrk	?	storage
- *         |
- *	   v
- *	  cache
- *         |
- *	   v
- *	 gunzip		wrk	storage	stack
- *         |
- *	   v
- *	 client
- *
- * XXXX: The two '?' are obviously the same memory, but I have yet to decide
- * where it goes.   As usual we try to avoid the session->ws if we can but
- * I may have to use that.
- *
- */
-
-#include "config.h"
-
-#include <stdio.h>
-#include <stdlib.h>
-
-#include "cache.h"
-
-#include "vgz.h"
-
-struct vgz {
-	unsigned		magic;
-#define VGZ_MAGIC		0x162df0cb
-	enum {VGZ_GZ,VGZ_UN}	dir;
-	struct worker		*wrk;
-	const char		*id;
-	struct ws		*tmp;
-	char			*tmp_snapshot;
-	int			last_i;
-
-	struct storage		*obuf;
-
-	z_stream		vz;
-};
-
-/*--------------------------------------------------------------------*/
-
-static voidpf
-vgz_alloc(voidpf opaque, uInt items, uInt size)
-{
-	struct vgz *vg;
-
-	CAST_OBJ_NOTNULL(vg, opaque, VGZ_MAGIC);
-
-	return (WS_Alloc(vg->tmp, items * size));
-}
-
-static void
-vgz_free(voidpf opaque, voidpf address)
-{
-	struct vgz *vg;
-
-	CAST_OBJ_NOTNULL(vg, opaque, VGZ_MAGIC);
-	(void)address;
-}
-
-/*--------------------------------------------------------------------
- * Set up a gunzip instance
- */
-
-static struct vgz *
-vgz_alloc_vgz(struct worker *wrk, const char *id)
-{
-	struct vgz *vg;
-	struct ws *ws;
-
-	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
-	ws = wrk->ws;
-	WS_Assert(ws);
-	// XXX: we restore workspace in esi:include
-	// vg = (void*)WS_Alloc(ws, sizeof *vg);
-	ALLOC_OBJ(vg, VGZ_MAGIC);
-	AN(vg);
-	memset(vg, 0, sizeof *vg);
-	vg->magic = VGZ_MAGIC;
-	vg->wrk = wrk;
-	vg->id = id;
-
-	switch (cache_param->gzip_tmp_space) {
-	case 0:
-	case 1:
-		/* malloc, the default */
-		break;
-	case 2:
-		vg->tmp = wrk->ws;
-		vg->tmp_snapshot = WS_Snapshot(vg->tmp);
-		vg->vz.zalloc = vgz_alloc;
-		vg->vz.zfree = vgz_free;
-		vg->vz.opaque = vg;
-		break;
-	default:
-		assert(0 == __LINE__);
-	}
-	return (vg);
-}
-
-struct vgz *
-VGZ_NewUngzip(struct worker *wrk, const char *id)
-{
-	struct vgz *vg;
-
-	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
-	vg = vgz_alloc_vgz(wrk, id);
-	vg->dir = VGZ_UN;
-	VSC_C_main->n_gunzip++;
-
-	/*
-	 * Max memory usage according to zonf.h:
-	 *	mem_needed = "a few kb" + (1 << (windowBits))
-	 * Since we don't control windowBits, we have to assume
-	 * it is 15, so 34-35KB or so.
-	 */
-	assert(Z_OK == inflateInit2(&vg->vz, 31));
-	return (vg);
-}
-
-struct vgz *
-VGZ_NewGzip(struct worker *wrk, const char *id)
-{
-	struct vgz *vg;
-	int i;
-
-	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
-	vg = vgz_alloc_vgz(wrk, id);
-	vg->dir = VGZ_GZ;
-	VSC_C_main->n_gzip++;
-
-	/*
-	 * From zconf.h:
-	 *
-	 *	mem_needed = "a few kb"
-	 *		+ (1 << (windowBits+2))
-	 *		+  (1 << (memLevel+9))
-	 *
-	 * windowBits [8..15] (-> 1K..128K)
-	 * memLevel [1..9] (-> 1K->256K)
-	 *
-	 * XXX: They probably needs to be params...
-	 *
-	 * XXX: It may be more efficent to malloc them, rather than have
-	 * XXX: too many worker threads grow the stacks.
-	 */
-	i = deflateInit2(&vg->vz,
-	    cache_param->gzip_level,		/* Level */
-	    Z_DEFLATED,			/* Method */
-	    16 + cache_param->gzip_window,	/* Window bits (16=gzip + 15) */
-	    cache_param->gzip_memlevel,	/* memLevel */
-	    Z_DEFAULT_STRATEGY);
-	assert(Z_OK == i);
-	return (vg);
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-VGZ_Ibuf(struct vgz *vg, const void *ptr, ssize_t len)
-{
-
-	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
-
-	AZ(vg->vz.avail_in);
-	vg->vz.next_in = TRUST_ME(ptr);
-	vg->vz.avail_in = len;
-}
-
-int
-VGZ_IbufEmpty(const struct vgz *vg)
-{
-
-	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
-	return (vg->vz.avail_in == 0);
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-VGZ_Obuf(struct vgz *vg, void *ptr, ssize_t len)
-{
-
-	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
-
-	vg->vz.next_out = TRUST_ME(ptr);
-	vg->vz.avail_out = len;
-}
-
-int
-VGZ_ObufFull(const struct vgz *vg)
-{
-
-	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
-	return (vg->vz.avail_out == 0);
-}
-
-/*--------------------------------------------------------------------
- * Keep the outbuffer supplied with storage and file it under the
- * sp->obj as it fills.
- */
-
-int
-VGZ_ObufStorage(struct worker *w, struct vgz *vg)
-{
-	struct storage *st;
-
-	st = FetchStorage(w, 0);
-	if (st == NULL)
-		return (-1);
-
-	vg->obuf = st;
-	VGZ_Obuf(vg, st->ptr + st->len, st->space - st->len);
-
-	return (0);
-}
-
-/*--------------------------------------------------------------------*/
-
-int
-VGZ_Gunzip(struct vgz *vg, const void **pptr, size_t *plen)
-{
-	int i;
-	ssize_t l;
-	const uint8_t *before;
-
-	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
-
-	*pptr = NULL;
-	*plen = 0;
-	AN(vg->vz.next_out);
-	AN(vg->vz.avail_out);
-	before = vg->vz.next_out;
-	i = inflate(&vg->vz, 0);
-	if (i == Z_OK || i == Z_STREAM_END) {
-		*pptr = before;
-		l = (const uint8_t *)vg->vz.next_out - before;
-		*plen = l;
-		if (vg->obuf != NULL)
-			vg->obuf->len += l;
-	}
-	vg->last_i = i;
-	if (i == Z_OK)
-		return (VGZ_OK);
-	if (i == Z_STREAM_END)
-		return (VGZ_END);
-	if (i == Z_BUF_ERROR)
-		return (VGZ_STUCK);
-	VSL(SLT_Debug, 0, "Unknown INFLATE=%d (%s)\n", i, vg->vz.msg);
-	return (VGZ_ERROR);
-}
-
-/*--------------------------------------------------------------------*/
-
-int
-VGZ_Gzip(struct vgz *vg, const void **pptr, size_t *plen, enum vgz_flag flags)
-{
-	int i;
-	int zflg;
-	ssize_t l;
-	const uint8_t *before;
-
-	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
-
-	*pptr = NULL;
-	*plen = 0;
-	AN(vg->vz.next_out);
-	AN(vg->vz.avail_out);
-	before = vg->vz.next_out;
-	switch(flags) {
-	case VGZ_NORMAL:	zflg = Z_NO_FLUSH; break;
-	case VGZ_ALIGN:		zflg = Z_SYNC_FLUSH; break;
-	case VGZ_RESET:		zflg = Z_FULL_FLUSH; break;
-	case VGZ_FINISH:	zflg = Z_FINISH; break;
-	default:		INCOMPL();
-	}
-	i = deflate(&vg->vz, zflg);
-	if (i == Z_OK || i == Z_STREAM_END) {
-		*pptr = before;
-		l = (const uint8_t *)vg->vz.next_out - before;
-		*plen = l;
-		if (vg->obuf != NULL)
-			vg->obuf->len += l;
-	}
-	vg->last_i = i;
-	if (i == Z_OK)
-		return (0);
-	if (i == Z_STREAM_END)
-		return (1);
-	if (i == Z_BUF_ERROR)
-		return (2);
-	return (-1);
-}
-
-/*--------------------------------------------------------------------
- * Gunzip ibuf into outb, if it runs full, emit it with WRW.
- * Leave flushing to caller, more data may be coming.
- */
-
-int
-VGZ_WrwGunzip(struct worker *w, struct vgz *vg, const void *ibuf,
-    ssize_t ibufl, char *obuf, ssize_t obufl, ssize_t *obufp)
-{
-	int i;
-	size_t dl;
-	const void *dp;
-
-	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
-	assert(obufl > 16);
-	VGZ_Ibuf(vg, ibuf, ibufl);
-	if (ibufl == 0)
-		return (VGZ_OK);
-	VGZ_Obuf(vg, obuf + *obufp, obufl - *obufp);
-	do {
-		if (obufl == *obufp)
-			i = VGZ_STUCK;
-		else {
-			i = VGZ_Gunzip(vg, &dp, &dl);
-			*obufp += dl;
-		}
-		if (i < VGZ_OK) {
-			/* XXX: VSL ? */
-			return (-1);
-		}
-		if (obufl == *obufp || i == VGZ_STUCK) {
-			w->acct_tmp.bodybytes += *obufp;
-			(void)WRW_Write(w, obuf, *obufp);
-			(void)WRW_Flush(w);
-			*obufp = 0;
-			VGZ_Obuf(vg, obuf + *obufp, obufl - *obufp);
-		}
-	} while (!VGZ_IbufEmpty(vg));
-	if (i == VGZ_STUCK)
-		i = VGZ_OK;
-	return (i);
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-VGZ_UpdateObj(const struct vgz *vg, struct object *obj)
-{
-
-	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
-	CHECK_OBJ_NOTNULL(obj, OBJECT_MAGIC);
-	obj->gzip_start	= vg->vz.start_bit;
-	obj->gzip_last	= vg->vz.last_bit;
-	obj->gzip_stop	= vg->vz.stop_bit;
-}
-
-/*--------------------------------------------------------------------
- * Passing a vsl_id of -1 means "use w->vbc->vsl_id"
- */
-
-int
-VGZ_Destroy(struct vgz **vgp, int vsl_id)
-{
-	struct vgz *vg;
-	int i;
-
-	vg = *vgp;
-	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
-	*vgp = NULL;
-
-	if (vsl_id < 0)
-		WSLB(vg->wrk, SLT_Gzip, "%s %jd %jd %jd %jd %jd",
-		    vg->id,
-		    (intmax_t)vg->vz.total_in,
-		    (intmax_t)vg->vz.total_out,
-		    (intmax_t)vg->vz.start_bit,
-		    (intmax_t)vg->vz.last_bit,
-		    (intmax_t)vg->vz.stop_bit);
-	else
-		WSL(vg->wrk, SLT_Gzip, vsl_id, "%s %jd %jd %jd %jd %jd",
-		    vg->id,
-		    (intmax_t)vg->vz.total_in,
-		    (intmax_t)vg->vz.total_out,
-		    (intmax_t)vg->vz.start_bit,
-		    (intmax_t)vg->vz.last_bit,
-		    (intmax_t)vg->vz.stop_bit);
-	if (vg->tmp != NULL)
-		WS_Reset(vg->tmp, vg->tmp_snapshot);
-	if (vg->dir == VGZ_GZ)
-		i = deflateEnd(&vg->vz);
-	else
-		i = inflateEnd(&vg->vz);
-	if (vg->last_i == Z_STREAM_END && i == Z_OK)
-		i = Z_STREAM_END;
-	FREE_OBJ(vg);
-	if (i == Z_OK)
-		return (VGZ_OK);
-	if (i == Z_STREAM_END)
-		return (VGZ_END);
-	if (i == Z_BUF_ERROR)
-		return (VGZ_STUCK);
-	return (VGZ_ERROR);
-}
-
-/*--------------------------------------------------------------------
- * VFP_GUNZIP
- *
- * A VFP for gunzip'ing an object as we receive it from the backend
- */
-
-static void __match_proto__()
-vfp_gunzip_begin(struct worker *w, size_t estimate)
-{
-	(void)estimate;
-	AZ(w->vgz_rx);
-	w->vgz_rx = VGZ_NewUngzip(w, "U F -");
-}
-
-static int __match_proto__()
-vfp_gunzip_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
-{
-	struct vgz *vg;
-	ssize_t l, wl;
-	int i = -100;
-	uint8_t	ibuf[cache_param->gzip_stack_buffer];
-	size_t dl;
-	const void *dp;
-
-	AZ(w->fetch_failed);
-	vg = w->vgz_rx;
-	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
-	AZ(vg->vz.avail_in);
-	while (bytes > 0 || vg->vz.avail_in > 0) {
-		if (vg->vz.avail_in == 0 && bytes > 0) {
-			l = sizeof ibuf;
-			if (l > bytes)
-				l = bytes;
-			wl = HTC_Read(w, htc, ibuf, l);
-			if (wl <= 0)
-				return (wl);
-			VGZ_Ibuf(vg, ibuf, wl);
-			bytes -= wl;
-		}
-
-		if (VGZ_ObufStorage(w, vg))
-			return(-1);
-		i = VGZ_Gunzip(vg, &dp, &dl);
-		if (i != VGZ_OK && i != VGZ_END)
-			return(FetchError(w, "Gunzip data error"));
-		w->fetch_obj->len += dl;
-		if (w->do_stream)
-			RES_StreamPoll(w);
-	}
-	assert(i == Z_OK || i == Z_STREAM_END);
-	return (1);
-}
-
-static int __match_proto__()
-vfp_gunzip_end(struct worker *w)
-{
-	struct vgz *vg;
-
-	vg = w->vgz_rx;
-	w->vgz_rx = NULL;
-	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
-	if (w->fetch_failed) {
-		(void)VGZ_Destroy(&vg, -1);
-		return(0);
-	}
-	if (VGZ_Destroy(&vg, -1) != VGZ_END)
-		return(FetchError(w, "Gunzip error at the very end"));
-	return (0);
-}
-
-struct vfp vfp_gunzip = {
-        .begin  =       vfp_gunzip_begin,
-        .bytes  =       vfp_gunzip_bytes,
-        .end    =       vfp_gunzip_end,
-};
-
-
-/*--------------------------------------------------------------------
- * VFP_GZIP
- *
- * A VFP for gzip'ing an object as we receive it from the backend
- */
-
-static void __match_proto__()
-vfp_gzip_begin(struct worker *w, size_t estimate)
-{
-	(void)estimate;
-
-	AZ(w->vgz_rx);
-	w->vgz_rx = VGZ_NewGzip(w, "G F -");
-}
-
-static int __match_proto__()
-vfp_gzip_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
-{
-	struct vgz *vg;
-	ssize_t l, wl;
-	int i = -100;
-	uint8_t ibuf[cache_param->gzip_stack_buffer];
-	size_t dl;
-	const void *dp;
-
-	AZ(w->fetch_failed);
-	vg = w->vgz_rx;
-	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
-	AZ(vg->vz.avail_in);
-	while (bytes > 0 || !VGZ_IbufEmpty(vg)) {
-		if (VGZ_IbufEmpty(vg) && bytes > 0) {
-			l = sizeof ibuf;
-			if (l > bytes)
-				l = bytes;
-			wl = HTC_Read(w, htc, ibuf, l);
-			if (wl <= 0)
-				return (wl);
-			VGZ_Ibuf(vg, ibuf, wl);
-			bytes -= wl;
-		}
-		if (VGZ_ObufStorage(w, vg))
-			return(-1);
-		i = VGZ_Gzip(vg, &dp, &dl, VGZ_NORMAL);
-		assert(i == Z_OK);
-		w->fetch_obj->len += dl;
-		if (w->do_stream)
-			RES_StreamPoll(w);
-	}
-	return (1);
-}
-
-static int __match_proto__()
-vfp_gzip_end(struct worker *w)
-{
-	struct vgz *vg;
-	size_t dl;
-	const void *dp;
-	int i;
-
-	vg = w->vgz_rx;
-	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
-	w->vgz_rx = NULL;
-	if (w->fetch_failed) {
-		(void)VGZ_Destroy(&vg, -1);
-		return(0);
-	}
-	do {
-		VGZ_Ibuf(vg, "", 0);
-		if (VGZ_ObufStorage(w, vg))
-			return(-1);
-		i = VGZ_Gzip(vg, &dp, &dl, VGZ_FINISH);
-		w->fetch_obj->len += dl;
-	} while (i != Z_STREAM_END);
-	if (w->do_stream)
-		RES_StreamPoll(w);
-	VGZ_UpdateObj(vg, w->fetch_obj);
-	if (VGZ_Destroy(&vg, -1) != VGZ_END)
-		return(FetchError(w, "Gzip error at the very end"));
-	return (0);
-}
-
-struct vfp vfp_gzip = {
-        .begin  =       vfp_gzip_begin,
-        .bytes  =       vfp_gzip_bytes,
-        .end    =       vfp_gzip_end,
-};
-
-/*--------------------------------------------------------------------
- * VFP_TESTGZIP
- *
- * A VFP for testing that received gzip data is valid, and for
- * collecting the magic bits while we're at it.
- */
-
-static void __match_proto__()
-vfp_testgzip_begin(struct worker *w, size_t estimate)
-{
-	(void)estimate;
-	w->vgz_rx = VGZ_NewUngzip(w, "u F -");
-	CHECK_OBJ_NOTNULL(w->vgz_rx, VGZ_MAGIC);
-}
-
-static int __match_proto__()
-vfp_testgzip_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
-{
-	struct vgz *vg;
-	ssize_t l, wl;
-	int i = -100;
-	uint8_t	obuf[cache_param->gzip_stack_buffer];
-	size_t dl;
-	const void *dp;
-	struct storage *st;
-
-	AZ(w->fetch_failed);
-	vg = w->vgz_rx;
-	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
-	AZ(vg->vz.avail_in);
-	while (bytes > 0) {
-		st = FetchStorage(w, 0);
-		if (st == NULL)
-			return(-1);
-		l = st->space - st->len;
-		if (l > bytes)
-			l = bytes;
-		wl = HTC_Read(w, htc, st->ptr + st->len, l);
-		if (wl <= 0)
-			return (wl);
-		bytes -= wl;
-		VGZ_Ibuf(vg, st->ptr + st->len, wl);
-		st->len += wl;
-		w->fetch_obj->len += wl;
-		if (w->do_stream)
-			RES_StreamPoll(w);
-
-		while (!VGZ_IbufEmpty(vg)) {
-			VGZ_Obuf(vg, obuf, sizeof obuf);
-			i = VGZ_Gunzip(vg, &dp, &dl);
-			if (i == VGZ_END && !VGZ_IbufEmpty(vg))
-				return(FetchError(w, "Junk after gzip data"));
-			if (i != VGZ_OK && i != VGZ_END)
-				return(FetchError2(w,
-				    "Invalid Gzip data", vg->vz.msg));
-		}
-	}
-	assert(i == VGZ_OK || i == VGZ_END);
-	return (1);
-}
-
-static int __match_proto__()
-vfp_testgzip_end(struct worker *w)
-{
-	struct vgz *vg;
-
-	vg = w->vgz_rx;
-	w->vgz_rx = NULL;
-	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
-	if (w->fetch_failed) {
-		(void)VGZ_Destroy(&vg, -1);
-		return(0);
-	}
-	VGZ_UpdateObj(vg, w->fetch_obj);
-	if (VGZ_Destroy(&vg, -1) != VGZ_END)
-		return(FetchError(w, "TestGunzip error at the very end"));
-	return (0);
-}
-
-struct vfp vfp_testgzip = {
-        .begin  =       vfp_testgzip_begin,
-        .bytes  =       vfp_testgzip_bytes,
-        .end    =       vfp_testgzip_end,
-};
diff --git a/bin/varnishd/cache_hash.c b/bin/varnishd/cache_hash.c
deleted file mode 100644
index db865de..0000000
--- a/bin/varnishd/cache_hash.c
+++ /dev/null
@@ -1,752 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * This is the central hash-table code, it relies on a chosen hash
- * implementation only for the actual hashing, all the housekeeping
- * happens here.
- *
- * We have two kinds of structures, objecthead and object.  An objecthead
- * corresponds to a given (Host:, URL) tupple, and the objects hung from
- * the objecthead may represent various variations (ie: Vary: header,
- * different TTL etc) instances of that web-entity.
- *
- * Each objecthead has a mutex which locks both its own fields, the
- * list of objects and fields in the objects.
- *
- * The hash implementation must supply a reference count facility on
- * the objecthead, and return with a reference held after a lookup.
- *
- * Lookups in the hash implementation returns with a ref held and each
- * object hung from the objhead holds a ref as well.
- *
- * Objects have refcounts which are locked by the objecthead mutex.
- *
- * New objects are always marked busy, and they can go from busy to
- * not busy only once.
- */
-
-#include "config.h"
-
-#include <math.h>
-#include <stdio.h>
-#include <stdlib.h>
-
-#include "cache.h"
-
-#include "hash/hash_slinger.h"
-#include "vsha256.h"
-
-static const struct hash_slinger *hash;
-
-/*---------------------------------------------------------------------*/
-/* Precreate an objhead and object for later use */
-void
-HSH_Prealloc(const struct sess *sp)
-{
-	struct worker *w;
-	struct objhead *oh;
-	struct objcore *oc;
-	struct waitinglist *wl;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->wrk, WORKER_MAGIC);
-	w = sp->wrk;
-
-	if (w->nobjcore == NULL) {
-		ALLOC_OBJ(oc, OBJCORE_MAGIC);
-		XXXAN(oc);
-		w->nobjcore = oc;
-		w->stats.n_objectcore++;
-		oc->flags |= OC_F_BUSY;
-	}
-	CHECK_OBJ_NOTNULL(w->nobjcore, OBJCORE_MAGIC);
-
-	if (w->nobjhead == NULL) {
-		ALLOC_OBJ(oh, OBJHEAD_MAGIC);
-		XXXAN(oh);
-		oh->refcnt = 1;
-		VTAILQ_INIT(&oh->objcs);
-		Lck_New(&oh->mtx, lck_objhdr);
-		w->nobjhead = oh;
-		w->stats.n_objecthead++;
-	}
-	CHECK_OBJ_NOTNULL(w->nobjhead, OBJHEAD_MAGIC);
-
-	if (w->nwaitinglist == NULL) {
-		ALLOC_OBJ(wl, WAITINGLIST_MAGIC);
-		XXXAN(wl);
-		VTAILQ_INIT(&wl->list);
-		w->nwaitinglist = wl;
-		w->stats.n_waitinglist++;
-	}
-	CHECK_OBJ_NOTNULL(w->nwaitinglist, WAITINGLIST_MAGIC);
-
-	if (w->nbusyobj == NULL) {
-		ALLOC_OBJ(w->nbusyobj, BUSYOBJ_MAGIC);
-		XXXAN(w->nbusyobj);
-	}
-
-	if (hash->prep != NULL)
-		hash->prep(sp);
-}
-
-void
-HSH_Cleanup(struct worker *w)
-{
-
-	if (w->nobjcore != NULL) {
-		FREE_OBJ(w->nobjcore);
-		w->stats.n_objectcore--;
-		w->nobjcore = NULL;
-	}
-	if (w->nobjhead != NULL) {
-		Lck_Delete(&w->nobjhead->mtx);
-		FREE_OBJ(w->nobjhead);
-		w->nobjhead = NULL;
-		w->stats.n_objecthead--;
-	}
-	if (w->nwaitinglist != NULL) {
-		FREE_OBJ(w->nwaitinglist);
-		w->nwaitinglist = NULL;
-	}
-	if (w->nhashpriv != NULL) {
-		/* XXX: If needed, add slinger method for this */
-		free(w->nhashpriv);
-		w->nhashpriv = NULL;
-	}
-	if (w->nbusyobj != NULL) {
-		FREE_OBJ(w->nbusyobj);
-		w->nbusyobj = NULL;
-	}
-}
-
-void
-HSH_DeleteObjHead(struct worker *w, struct objhead *oh)
-{
-
-	AZ(oh->refcnt);
-	assert(VTAILQ_EMPTY(&oh->objcs));
-	Lck_Delete(&oh->mtx);
-	w->stats.n_objecthead--;
-	FREE_OBJ(oh);
-}
-
-void
-HSH_AddString(const struct sess *sp, const char *str)
-{
-	int l;
-
-	if (str == NULL)
-		str = "";
-	l = strlen(str);
-
-	SHA256_Update(sp->wrk->sha256ctx, str, l);
-	SHA256_Update(sp->wrk->sha256ctx, "#", 1);
-
-	if (cache_param->log_hash)
-		WSP(sp, SLT_Hash, "%s", str);
-}
-
-/*---------------------------------------------------------------------
- * This is a debugging hack to enable testing of boundary conditions
- * in the hash algorithm.
- * We trap the first 9 different digests and translate them to different
- * digests with edge bit conditions
- */
-
-static struct hsh_magiclist {
-	unsigned char was[SHA256_LEN];
-	unsigned char now[SHA256_LEN];
-} hsh_magiclist[] = {
-	{ .now = {	0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 } },
-	{ .now = {	0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01 } },
-	{ .now = {	0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02 } },
-	{ .now = {	0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x40 } },
-	{ .now = {	0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x80 } },
-	{ .now = {	0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 } },
-	{ .now = {	0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 } },
-	{ .now = {	0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 } },
-	{ .now = {	0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-			0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 } },
-};
-
-#define HSH_NMAGIC (sizeof hsh_magiclist / sizeof hsh_magiclist[0])
-
-static void
-hsh_testmagic(void *result)
-{
-	int i, j;
-	static int nused = 0;
-
-	for (i = 0; i < nused; i++)
-		if (!memcmp(hsh_magiclist[i].was, result, SHA256_LEN))
-			break;
-	if (i == nused && i < HSH_NMAGIC)
-		memcpy(hsh_magiclist[nused++].was, result, SHA256_LEN);
-	if (i == nused)
-		return;
-	assert(i < HSH_NMAGIC);
-	fprintf(stderr, "HASHMAGIC: <");
-	for (j = 0; j < SHA256_LEN; j++)
-		fprintf(stderr, "%02x", ((unsigned char*)result)[j]);
-	fprintf(stderr, "> -> <");
-	memcpy(result, hsh_magiclist[i].now, SHA256_LEN);
-	for (j = 0; j < SHA256_LEN; j++)
-		fprintf(stderr, "%02x", ((unsigned char*)result)[j]);
-	fprintf(stderr, ">\n");
-}
-
-/*---------------------------------------------------------------------
- * Insert an object which magically appears out of nowhere or, more likely,
- * comes off some persistent storage device.
- * Return it with a reference held.
- */
-
-struct objcore *
-HSH_Insert(const struct sess *sp)
-{
-	struct worker *w;
-	struct objhead *oh;
-	struct objcore *oc;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->wrk, WORKER_MAGIC);
-	AN(hash);
-	w = sp->wrk;
-
-	HSH_Prealloc(sp);
-	if (cache_param->diag_bitmap & 0x80000000)
-		hsh_testmagic(sp->wrk->nobjhead->digest);
-
-	AZ(sp->hash_objhead);
-	AN(w->nobjhead);
-	oh = hash->lookup(sp, w->nobjhead);
-	CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC);
-	if (oh == w->nobjhead)
-		w->nobjhead = NULL;
-	Lck_Lock(&oh->mtx);
-	assert(oh->refcnt > 0);
-
-	/* Insert (precreated) objcore in objecthead */
-	oc = w->nobjcore;
-	w->nobjcore = NULL;
-	oc->refcnt = 1;
-	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-	AZ(oc->flags & OC_F_BUSY);
-
-	VTAILQ_INSERT_HEAD(&oh->objcs, oc, list);
-	/* NB: do not deref objhead the new object inherits our reference */
-	oc->objhead = oh;
-	Lck_Unlock(&oh->mtx);
-	sp->wrk->stats.n_vampireobject++;
-	return (oc);
-}
-
-/*---------------------------------------------------------------------
- */
-
-struct objcore *
-HSH_Lookup(struct sess *sp, struct objhead **poh)
-{
-	struct worker *w;
-	struct objhead *oh;
-	struct objcore *oc;
-	struct objcore *busy_oc, *grace_oc;
-	struct object *o;
-	double grace_ttl;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->wrk, WORKER_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->http, HTTP_MAGIC);
-	AN(sp->director);
-	AN(hash);
-	w = sp->wrk;
-
-	HSH_Prealloc(sp);
-	memcpy(sp->wrk->nobjhead->digest, sp->digest, sizeof sp->digest);
-	if (cache_param->diag_bitmap & 0x80000000)
-		hsh_testmagic(sp->wrk->nobjhead->digest);
-
-	if (sp->hash_objhead != NULL) {
-		/*
-		 * This sess came off the waiting list, and brings a
-		 * oh refcnt with it.
-		 */
-		CHECK_OBJ_NOTNULL(sp->hash_objhead, OBJHEAD_MAGIC);
-		oh = sp->hash_objhead;
-		sp->hash_objhead = NULL;
-	} else {
-		AN(w->nobjhead);
-		oh = hash->lookup(sp, w->nobjhead);
-		if (oh == w->nobjhead)
-			w->nobjhead = NULL;
-	}
-
-	CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC);
-	Lck_Lock(&oh->mtx);
-	assert(oh->refcnt > 0);
-	busy_oc = NULL;
-	grace_oc = NULL;
-	grace_ttl = NAN;
-	VTAILQ_FOREACH(oc, &oh->objcs, list) {
-		/* Must be at least our own ref + the objcore we examine */
-		assert(oh->refcnt > 1);
-		CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-		assert(oc->objhead == oh);
-
-		if (oc->flags & OC_F_BUSY) {
-			CHECK_OBJ_NOTNULL(oc->busyobj, BUSYOBJ_MAGIC);
-			if (sp->hash_ignore_busy)
-				continue;
-
-			if (oc->busyobj->vary != NULL &&
-			    !VRY_Match(sp, oc->busyobj->vary))
-				continue;
-
-			busy_oc = oc;
-			continue;
-		}
-
-		o = oc_getobj(sp->wrk, oc);
-		CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
-
-		if (o->exp.ttl <= 0.)
-			continue;
-		if (BAN_CheckObject(o, sp))
-			continue;
-		if (o->vary != NULL && !VRY_Match(sp, o->vary))
-			continue;
-
-		/* If still valid, use it */
-		if (EXP_Ttl(sp, o) >= sp->t_req)
-			break;
-
-		/*
-		 * Remember any matching objects inside their grace period
-		 * and if there are several, use the least expired one.
-		 */
-		if (EXP_Grace(sp, o) >= sp->t_req) {
-			if (grace_oc == NULL ||
-			    grace_ttl < o->exp.entered + o->exp.ttl) {
-				grace_oc = oc;
-				grace_ttl = o->exp.entered + o->exp.ttl;
-			}
-		}
-	}
-
-	/*
-	 * If we have seen a busy object or the backend is unhealthy, and
-	 * we have an object in grace, use it, if req.grace is also
-	 * satisified.
-	 * XXX: Interesting footnote:  The busy object might be for a
-	 * XXX: different "Vary:" than we sought.  We have no way of knowing
-	 * XXX: this until the object is unbusy'ed, so in practice we
-	 * XXX: serialize fetch of all Vary's if grace is possible.
-	 */
-
-	AZ(sp->objcore);
-	sp->objcore = grace_oc;		/* XXX: Hack-ish */
-	if (oc == NULL			/* We found no live object */
-	    && grace_oc != NULL		/* There is a grace candidate */
-	    && (busy_oc != NULL		/* Somebody else is already busy */
-	    || !VDI_Healthy(sp->director, sp))) {
-					/* Or it is impossible to fetch */
-		o = oc_getobj(sp->wrk, grace_oc);
-		CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
-		oc = grace_oc;
-	}
-	sp->objcore = NULL;
-
-	if (oc != NULL && !sp->hash_always_miss) {
-		o = oc_getobj(sp->wrk, oc);
-		CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
-		assert(oc->objhead == oh);
-
-		/* We found an object we like */
-		oc->refcnt++;
-		if (o->hits < INT_MAX)
-			o->hits++;
-		assert(oh->refcnt > 1);
-		Lck_Unlock(&oh->mtx);
-		assert(hash->deref(oh));
-		*poh = oh;
-		return (oc);
-	}
-
-	if (busy_oc != NULL) {
-		/* There are one or more busy objects, wait for them */
-		if (sp->esi_level == 0) {
-			CHECK_OBJ_NOTNULL(sp->wrk->nwaitinglist,
-			    WAITINGLIST_MAGIC);
-			if (oh->waitinglist == NULL) {
-				oh->waitinglist = sp->wrk->nwaitinglist;
-				sp->wrk->nwaitinglist = NULL;
-			}
-			VTAILQ_INSERT_TAIL(&oh->waitinglist->list, sp, list);
-		}
-		if (cache_param->diag_bitmap & 0x20)
-			WSP(sp, SLT_Debug,
-				"on waiting list <%p>", oh);
-		SES_Charge(sp);
-		/*
-		 * The objhead reference transfers to the sess, we get it
-		 * back when the sess comes off the waiting list and
-		 * calls us again
-		 */
-		sp->hash_objhead = oh;
-		sp->wrk = NULL;
-		Lck_Unlock(&oh->mtx);
-		return (NULL);
-	}
-
-	/* Insert (precreated) objcore in objecthead */
-	oc = w->nobjcore;
-	w->nobjcore = NULL;
-	AN(oc->flags & OC_F_BUSY);
-	oc->refcnt = 1;
-
-	/* XXX: clear w->nbusyobj before use */
-	VRY_Validate(sp->vary_b);
-	if (sp->vary_l != NULL)
-		w->nbusyobj->vary = sp->vary_b;
-	else
-		w->nbusyobj->vary = NULL;
-	oc->busyobj = w->nbusyobj;
-	w->nbusyobj = NULL;
-
-	/*
-	 * Busy objects go on the tail, so they will not trip up searches.
-	 * HSH_Unbusy() will move them to the front.
-	 */
-	VTAILQ_INSERT_TAIL(&oh->objcs, oc, list);
-	oc->objhead = oh;
-	/* NB: do not deref objhead the new object inherits our reference */
-	Lck_Unlock(&oh->mtx);
-	*poh = oh;
-	return (oc);
-}
-
-/*---------------------------------------------------------------------
- */
-
-static void
-hsh_rush(struct objhead *oh)
-{
-	unsigned u;
-	struct sess *sp;
-	struct waitinglist *wl;
-
-	CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC);
-	Lck_AssertHeld(&oh->mtx);
-	wl = oh->waitinglist;
-	CHECK_OBJ_NOTNULL(wl, WAITINGLIST_MAGIC);
-	for (u = 0; u < cache_param->rush_exponent; u++) {
-		sp = VTAILQ_FIRST(&wl->list);
-		if (sp == NULL)
-			break;
-		CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-		AZ(sp->wrk);
-		VTAILQ_REMOVE(&wl->list, sp, list);
-		DSL(0x20, SLT_Debug, sp->vsl_id, "off waiting list");
-		if (SES_Schedule(sp)) {
-			/*
-			 * We could not schedule the session, leave the
-			 * rest on the busy list.
-			 */
-			break;
-		}
-	}
-	if (VTAILQ_EMPTY(&wl->list)) {
-		oh->waitinglist = NULL;
-		FREE_OBJ(wl);
-	}
-}
-
-/*---------------------------------------------------------------------
- * Purge an entire objhead
- */
-
-void
-HSH_Purge(const struct sess *sp, struct objhead *oh, double ttl, double grace)
-{
-	struct objcore *oc, **ocp;
-	unsigned spc, nobj, n;
-	struct object *o;
-
-	CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC);
-	spc = WS_Reserve(sp->wrk->ws, 0);
-	ocp = (void*)sp->wrk->ws->f;
-	Lck_Lock(&oh->mtx);
-	assert(oh->refcnt > 0);
-	nobj = 0;
-	VTAILQ_FOREACH(oc, &oh->objcs, list) {
-		CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-		assert(oc->objhead == oh);
-		if (oc->flags & OC_F_BUSY) {
-			/*
-			 * We cannot purge busy objects here, because their
-			 * owners have special rights to them, and may nuke
-			 * them without concern for the refcount, which by
-			 * definition always must be one, so they don't check.
-			 */
-			continue;
-		}
-
-		(void)oc_getobj(sp->wrk, oc); /* XXX: still needed ? */
-
-		xxxassert(spc >= sizeof *ocp);
-		oc->refcnt++;
-		spc -= sizeof *ocp;
-		ocp[nobj++] = oc;
-	}
-	Lck_Unlock(&oh->mtx);
-
-	/* NB: inverse test to catch NAN also */
-	if (!(ttl > 0.))
-		ttl = -1.;
-	if (!(grace > 0.))
-		grace = -1.;
-	for (n = 0; n < nobj; n++) {
-		oc = ocp[n];
-		CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-		o = oc_getobj(sp->wrk, oc);
-		if (o == NULL)
-			continue;
-		CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
-		o->exp.ttl = ttl;
-		o->exp.grace = grace;
-		EXP_Rearm(o);
-		(void)HSH_Deref(sp->wrk, NULL, &o);
-	}
-	WS_Release(sp->wrk->ws, 0);
-}
-
-
-/*---------------------------------------------------------------------
- * Kill a busy object we don't need anyway.
- * There may be sessions on the waiting list, so we cannot just blow
- * it out of the water.
- */
-
-void
-HSH_Drop(struct sess *sp)
-{
-	struct object *o;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	o = sp->obj;
-	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
-	AssertObjCorePassOrBusy(o->objcore);
-	o->exp.ttl = -1.;
-	if (o->objcore != NULL)		/* Pass has no objcore */
-		HSH_Unbusy(sp);
-	(void)HSH_Deref(sp->wrk, NULL, &sp->obj);
-}
-
-void
-HSH_Unbusy(const struct sess *sp)
-{
-	struct object *o;
-	struct objhead *oh;
-	struct objcore *oc;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	o = sp->obj;
-	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
-	oc = o->objcore;
-	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-	oh = oc->objhead;
-	CHECK_OBJ(oh, OBJHEAD_MAGIC);
-
-	AssertObjBusy(o);
-	AN(oc->ban);
-	assert(oc->refcnt > 0);
-	assert(oh->refcnt > 0);
-	if (o->ws_o->overflow)
-		sp->wrk->stats.n_objoverflow++;
-	if (cache_param->diag_bitmap & 0x40)
-		WSP(sp, SLT_Debug,
-		    "Object %u workspace free %u", o->xid, WS_Free(o->ws_o));
-
-	/* XXX: pretouch neighbors on oh->objcs to prevent page-on under mtx */
-	Lck_Lock(&oh->mtx);
-	assert(oh->refcnt > 0);
-	/* XXX: strictly speaking, we should sort in Date: order. */
-	VTAILQ_REMOVE(&oh->objcs, oc, list);
-	VTAILQ_INSERT_HEAD(&oh->objcs, oc, list);
-	oc->flags &= ~OC_F_BUSY;
-	AZ(sp->wrk->nbusyobj);
-	sp->wrk->nbusyobj = oc->busyobj;
-	oc->busyobj = NULL;
-	if (oh->waitinglist != NULL)
-		hsh_rush(oh);
-	AN(oc->ban);
-	Lck_Unlock(&oh->mtx);
-	assert(oc_getobj(sp->wrk, oc) == o);
-}
-
-void
-HSH_Ref(struct objcore *oc)
-{
-	struct objhead *oh;
-
-	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-	oh = oc->objhead;
-	CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC);
-	Lck_Lock(&oh->mtx);
-	assert(oc->refcnt > 0);
-	oc->refcnt++;
-	Lck_Unlock(&oh->mtx);
-}
-
-/*--------------------------------------------------------------------
- * Dereference objcore and or object
- *
- * Can deal with:
- *	bare objcore (incomplete fetch)
- *	bare object (pass)
- *	object with objcore
- *	XXX later:  objcore with object (?)
- *
- * But you can only supply one of the two arguments at a time.
- *
- * Returns zero if target was destroyed.
- */
-
-int
-HSH_Deref(struct worker *w, struct objcore *oc, struct object **oo)
-{
-	struct object *o = NULL;
-	struct objhead *oh;
-	unsigned r;
-
-	/* Only one arg at a time */
-	assert(oc == NULL || oo == NULL);
-
-	if (oo != NULL) {
-		o = *oo;
-		*oo = NULL;
-		CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
-		oc = o->objcore;
-	}
-
-	if (o != NULL && oc == NULL) {
-		/*
-		 * A pass object with neither objcore nor objhdr reference.
-		 * -> simply free the (Transient) storage
-		 */
-		STV_Freestore(o);
-		STV_free(o->objstore);
-		w->stats.n_object--;
-		return (0);
-	}
-
-	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-
-	oh = oc->objhead;
-	CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC);
-
-	Lck_Lock(&oh->mtx);
-	assert(oh->refcnt > 0);
-	assert(oc->refcnt > 0);
-	r = --oc->refcnt;
-	if (!r)
-		VTAILQ_REMOVE(&oh->objcs, oc, list);
-	else {
-		/* Must have an object */
-		AN(oc->methods);
-	}
-	if (oh->waitinglist != NULL)
-		hsh_rush(oh);
-	Lck_Unlock(&oh->mtx);
-	if (r != 0)
-		return (r);
-
-	BAN_DestroyObj(oc);
-	AZ(oc->ban);
-
-	if (oc->flags & OC_F_BUSY) {
-		CHECK_OBJ_NOTNULL(oc->busyobj, BUSYOBJ_MAGIC);
-		if (w->nbusyobj == NULL)
-			w->nbusyobj = oc->busyobj;
-		else
-			FREE_OBJ(oc->busyobj);
-		oc->busyobj = NULL;
-	}
-	AZ(oc->busyobj);
-
-	if (oc->methods != NULL) {
-		oc_freeobj(oc);
-		w->stats.n_object--;
-	}
-	FREE_OBJ(oc);
-
-	w->stats.n_objectcore--;
-	/* Drop our ref on the objhead */
-	assert(oh->refcnt > 0);
-	if (hash->deref(oh))
-		return (0);
-	HSH_DeleteObjHead(w, oh);
-	return (0);
-}
-
-void
-HSH_Init(const struct hash_slinger *slinger)
-{
-
-	assert(DIGEST_LEN == SHA256_LEN);	/* avoid #include pollution */
-	hash = slinger;
-	if (hash->start != NULL)
-		hash->start();
-}
diff --git a/bin/varnishd/cache_http.c b/bin/varnishd/cache_http.c
deleted file mode 100644
index 784eb28..0000000
--- a/bin/varnishd/cache_http.c
+++ /dev/null
@@ -1,1119 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * HTTP request storage and manipulation
- */
-
-#include "config.h"
-
-#include <stdio.h>
-
-#include "cache.h"
-
-#include "vct.h"
-
-#define HTTPH(a, b, c, d, e, f, g) char b[] = "*" a ":";
-#include "tbl/http_headers.h"
-#undef HTTPH
-
-/*lint -save -e773 not () */
-#define LOGMTX2(ax, bx, cx)	[bx] = SLT_##ax##cx
-
-#define LOGMTX1(ax) {					\
-	LOGMTX2(ax, HTTP_HDR_REQ,	Request),	\
-	LOGMTX2(ax, HTTP_HDR_RESPONSE,	Response),	\
-	LOGMTX2(ax, HTTP_HDR_STATUS,	Status),	\
-	LOGMTX2(ax, HTTP_HDR_URL,	URL),		\
-	LOGMTX2(ax, HTTP_HDR_PROTO,	Protocol),	\
-	LOGMTX2(ax, HTTP_HDR_FIRST,	Header),	\
-	}
-
-static const enum VSL_tag_e logmtx[][HTTP_HDR_FIRST + 1] = {
-	[HTTP_Rx] = LOGMTX1(Rx),
-	[HTTP_Tx] = LOGMTX1(Tx),
-	[HTTP_Obj] = LOGMTX1(Obj)
-};
-/*lint -restore */
-
-static enum VSL_tag_e
-http2shmlog(const struct http *hp, int t)
-{
-
-	CHECK_OBJ_NOTNULL(hp, HTTP_MAGIC);
-	if (t > HTTP_HDR_FIRST)
-		t = HTTP_HDR_FIRST;
-	assert(hp->logtag >= HTTP_Rx && hp->logtag <= HTTP_Obj); /*lint !e685*/
-	assert(t >= HTTP_HDR_REQ && t <= HTTP_HDR_FIRST);
-	return (logmtx[hp->logtag][t]);
-}
-
-static void
-WSLH(struct worker *w, unsigned vsl_id, const struct http *hp, unsigned hdr)
-{
-
-	AN(vsl_id & (VSL_CLIENTMARKER|VSL_BACKENDMARKER));
-	WSLR(w, http2shmlog(hp, hdr), vsl_id, hp->hd[hdr]);
-}
-
-/*--------------------------------------------------------------------*/
-/* List of canonical HTTP response code names from RFC2616 */
-
-static struct http_msg {
-	unsigned	nbr;
-	const char	*txt;
-} http_msg[] = {
-#define HTTP_RESP(n, t)	{ n, t},
-#include "tbl/http_response.h"
-	{ 0, NULL }
-};
-
-const char *
-http_StatusMessage(unsigned status)
-{
-	struct http_msg *mp;
-
-	assert(status >= 100 && status <= 999);
-	for (mp = http_msg; mp->nbr != 0 && mp->nbr <= status; mp++)
-		if (mp->nbr == status)
-			return (mp->txt);
-	return ("Unknown Error");
-}
-
-/*--------------------------------------------------------------------*/
-
-unsigned
-HTTP_estimate(unsigned nhttp)
-{
-
-	/* XXX: We trust the structs to size-aligned as necessary */
-	return (sizeof (struct http) + (sizeof (txt) + 1) * nhttp);
-}
-
-struct http *
-HTTP_create(void *p, uint16_t nhttp)
-{
-	struct http *hp;
-
-	hp = p;
-	hp->magic = HTTP_MAGIC;
-	hp->hd = (void*)(hp + 1);
-	hp->shd = nhttp;
-	hp->hdf = (void*)(hp->hd + nhttp);
-	return (hp);
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-http_Setup(struct http *hp, struct ws *ws)
-{
-	uint16_t shd;
-	txt *hd;
-	unsigned char *hdf;
-
-	/* XXX: This is not elegant, is it efficient ? */
-	shd = hp->shd;
-	hd = hp->hd;
-	hdf = hp->hdf;
-	memset(hp, 0, sizeof *hp);
-	memset(hd, 0, sizeof *hd * shd);
-	memset(hdf, 0, sizeof *hdf * shd);
-	hp->magic = HTTP_MAGIC;
-	hp->ws = ws;
-	hp->nhd = HTTP_HDR_FIRST;
-	hp->shd = shd;
-	hp->hd = hd;
-	hp->hdf = hdf;
-}
-
-/*--------------------------------------------------------------------*/
-
-static int
-http_IsHdr(const txt *hh, const char *hdr)
-{
-	unsigned l;
-
-	Tcheck(*hh);
-	AN(hdr);
-	l = hdr[0];
-	assert(l == strlen(hdr + 1));
-	assert(hdr[l] == ':');
-	hdr++;
-	return (!strncasecmp(hdr, hh->b, l));
-}
-
-/*--------------------------------------------------------------------
- * This function collapses multiple headerlines of the same name.
- * The lines are joined with a comma, according to [rfc2616, 4.2bot, p32]
- */
-
-void
-http_CollectHdr(struct http *hp, const char *hdr)
-{
-	unsigned u, v, ml, f = 0, x;
-	char *b = NULL, *e = NULL;
-
-	for (u = HTTP_HDR_FIRST; u < hp->nhd; u++) {
-		while (u < hp->nhd && http_IsHdr(&hp->hd[u], hdr)) {
-			Tcheck(hp->hd[u]);
-			if (f == 0) {
-				/* Found first header, just record the fact */
-				f = u;
-				break;
-			}
-			if (b == NULL) {
-				/* Found second header, start our collection */
-				ml = WS_Reserve(hp->ws, 0);
-				b = hp->ws->f;
-				e = b + ml;
-				x = Tlen(hp->hd[f]);
-				if (b + x < e) {
-					memcpy(b, hp->hd[f].b, x);
-					b += x;
-				} else
-					b = e;
-			}
-
-			AN(b);
-			AN(e);
-
-			/* Append the Nth header we found */
-			if (b < e)
-				*b++ = ',';
-			x = Tlen(hp->hd[u]) - *hdr;
-			if (b + x < e) {
-				memcpy(b, hp->hd[u].b + *hdr, x);
-				b += x;
-			} else
-				b = e;
-
-			/* Shift remaining headers up one slot */
-			for (v = u; v < hp->nhd - 1; v++)
-				hp->hd[v] = hp->hd[v + 1];
-			hp->nhd--;
-		}
-
-	}
-	if (b == NULL)
-		return;
-	AN(e);
-	if (b >= e) {
-		WS_Release(hp->ws, 0);
-		return;
-	}
-	*b = '\0';
-	hp->hd[f].b = hp->ws->f;
-	hp->hd[f].e = b;
-	WS_ReleaseP(hp->ws, b + 1);
-}
-
-
-/*--------------------------------------------------------------------*/
-
-static unsigned
-http_findhdr(const struct http *hp, unsigned l, const char *hdr)
-{
-	unsigned u;
-
-	for (u = HTTP_HDR_FIRST; u < hp->nhd; u++) {
-		Tcheck(hp->hd[u]);
-		if (hp->hd[u].e < hp->hd[u].b + l + 1)
-			continue;
-		if (hp->hd[u].b[l] != ':')
-			continue;
-		if (strncasecmp(hdr, hp->hd[u].b, l))
-			continue;
-		return (u);
-	}
-	return (0);
-}
-
-int
-http_GetHdr(const struct http *hp, const char *hdr, char **ptr)
-{
-	unsigned u, l;
-	char *p;
-
-	l = hdr[0];
-	diagnostic(l == strlen(hdr + 1));
-	assert(hdr[l] == ':');
-	hdr++;
-	u = http_findhdr(hp, l - 1, hdr);
-	if (u == 0) {
-		if (ptr != NULL)
-			*ptr = NULL;
-		return (0);
-	}
-	if (ptr != NULL) {
-		p = hp->hd[u].b + l;
-		while (vct_issp(*p))
-			p++;
-		*ptr = p;
-	}
-	return (1);
-}
-
-
-/*--------------------------------------------------------------------
- * Find a given data element in a header according to RFC2616's #rule
- * (section 2.1, p15)
- */
-
-int
-http_GetHdrData(const struct http *hp, const char *hdr,
-    const char *field, char **ptr)
-{
-	char *h, *e;
-	unsigned fl;
-
-	if (ptr != NULL)
-		*ptr = NULL;
-	if (!http_GetHdr(hp, hdr, &h))
-		return (0);
-	AN(h);
-	e = strchr(h, '\0');
-	fl = strlen(field);
-	while (h + fl <= e) {
-		/* Skip leading whitespace and commas */
-		if (vct_islws(*h) || *h == ',') {
-			h++;
-			continue;
-		}
-		/* Check for substrings before memcmp() */
-		if ((h + fl == e || vct_issepctl(h[fl])) &&
-		    !memcmp(h, field, fl)) {
-			if (ptr != NULL) {
-				h += fl;
-				while (vct_islws(*h))
-					h++;
-				*ptr = h;
-			}
-			return (1);
-		}
-		/* Skip until end of header or comma */
-		while (*h && *h != ',')
-			h++;
-	}
-	return (0);
-}
-
-/*--------------------------------------------------------------------
- * Find a given headerfields Q value.
- */
-
-double
-http_GetHdrQ(const struct http *hp, const char *hdr, const char *field)
-{
-	char *h;
-	int i;
-	double a, b;
-
-	h = NULL;
-	i = http_GetHdrData(hp, hdr, field, &h);
-	if (!i)
-		return (0.);
-
-	if (h == NULL)
-		return (1.);
-	/* Skip whitespace, looking for '=' */
-	while (*h && vct_issp(*h))
-		h++;
-	if (*h++ != ';')
-		return (1.);
-	while (*h && vct_issp(*h))
-		h++;
-	if (*h++ != 'q')
-		return (1.);
-	while (*h && vct_issp(*h))
-		h++;
-	if (*h++ != '=')
-		return (1.);
-	while (*h && vct_issp(*h))
-		h++;
-	a = 0.;
-	while (vct_isdigit(*h)) {
-		a *= 10.;
-		a += *h - '0';
-		h++;
-	}
-	if (*h++ != '.')
-		return (a);
-	b = .1;
-	while (vct_isdigit(*h)) {
-		a += b * (*h - '0');
-		b *= .1;
-		h++;
-	}
-	return (a);
-}
-
-/*--------------------------------------------------------------------
- * Find a given headerfields value.
- */
-
-int
-http_GetHdrField(const struct http *hp, const char *hdr,
-    const char *field, char **ptr)
-{
-	char *h;
-	int i;
-
-	if (ptr != NULL)
-		*ptr = NULL;
-
-	h = NULL;
-	i = http_GetHdrData(hp, hdr, field, &h);
-	if (!i)
-		return (i);
-
-	if (ptr != NULL && h != NULL) {
-		/* Skip whitespace, looking for '=' */
-		while (*h && vct_issp(*h))
-			h++;
-		if (*h == '=') {
-			h++;
-			while (*h && vct_issp(*h))
-				h++;
-			*ptr = h;
-		}
-	}
-	return (i);
-}
-
-/*--------------------------------------------------------------------
- * XXX: redo with http_GetHdrField() ?
- */
-
-const char *
-http_DoConnection(const struct http *hp)
-{
-	char *p, *q;
-	const char *ret;
-	unsigned u;
-
-	if (!http_GetHdr(hp, H_Connection, &p)) {
-		if (hp->protover < 11)
-			return ("not HTTP/1.1");
-		return (NULL);
-	}
-	ret = NULL;
-	AN(p);
-	for (; *p; p++) {
-		if (vct_issp(*p))
-			continue;
-		if (*p == ',')
-			continue;
-		for (q = p + 1; *q; q++)
-			if (*q == ',' || vct_issp(*q))
-				break;
-		u = pdiff(p, q);
-		if (u == 5 && !strncasecmp(p, "close", u))
-			ret = "Connection: close";
-		u = http_findhdr(hp, u, p);
-		if (u != 0)
-			hp->hdf[u] |= HDF_FILTER;
-		if (!*q)
-			break;
-		p = q;
-	}
-	return (ret);
-}
-
-/*--------------------------------------------------------------------*/
-
-int
-http_HdrIs(const struct http *hp, const char *hdr, const char *val)
-{
-	char *p;
-
-	if (!http_GetHdr(hp, hdr, &p))
-		return (0);
-	AN(p);
-	if (!strcasecmp(p, val))
-		return (1);
-	return (0);
-}
-
-/*--------------------------------------------------------------------*/
-
-uint16_t
-http_GetStatus(const struct http *hp)
-{
-
-	return (hp->status);
-}
-
-const char *
-http_GetReq(const struct http *hp)
-{
-
-	Tcheck(hp->hd[HTTP_HDR_REQ]);
-	return (hp->hd[HTTP_HDR_REQ].b);
-}
-
-/*--------------------------------------------------------------------
- * Dissect the headers of the HTTP protocol message.
- * Detect conditionals (headers which start with '^[Ii][Ff]-')
- */
-
-static uint16_t
-http_dissect_hdrs(struct worker *w, struct http *hp, unsigned vsl_id, char *p,
-    const struct http_conn *htc)
-{
-	char *q, *r;
-	txt t = htc->rxbuf;
-
-	if (*p == '\r')
-		p++;
-
-	hp->nhd = HTTP_HDR_FIRST;
-	hp->conds = 0;
-	r = NULL;		/* For FlexeLint */
-	for (; p < t.e; p = r) {
-
-		/* Find end of next header */
-		q = r = p;
-		while (r < t.e) {
-			if (!vct_iscrlf(*r)) {
-				r++;
-				continue;
-			}
-			q = r;
-			assert(r < t.e);
-			r += vct_skipcrlf(r);
-			if (r >= t.e)
-				break;
-			/* If line does not continue: got it. */
-			if (!vct_issp(*r))
-				break;
-
-			/* Clear line continuation LWS to spaces */
-			while (vct_islws(*q))
-				*q++ = ' ';
-		}
-
-		if (q - p > htc->maxhdr) {
-			VSC_C_main->losthdr++;
-			WSL(w, SLT_LostHeader, vsl_id, "%.*s",
-			    q - p > 20 ? 20 : q - p, p);
-			return (413);
-		}
-
-		/* Empty header = end of headers */
-		if (p == q)
-			break;
-
-		if ((p[0] == 'i' || p[0] == 'I') &&
-		    (p[1] == 'f' || p[1] == 'F') &&
-		    p[2] == '-')
-			hp->conds = 1;
-
-		while (q > p && vct_issp(q[-1]))
-			q--;
-		*q = '\0';
-
-		if (hp->nhd < hp->shd) {
-			hp->hdf[hp->nhd] = 0;
-			hp->hd[hp->nhd].b = p;
-			hp->hd[hp->nhd].e = q;
-			WSLH(w, vsl_id, hp, hp->nhd);
-			hp->nhd++;
-		} else {
-			VSC_C_main->losthdr++;
-			WSL(w, SLT_LostHeader, vsl_id, "%.*s",
-			    q - p > 20 ? 20 : q - p, p);
-			return (413);
-		}
-	}
-	return (0);
-}
-
-/*--------------------------------------------------------------------
- * Deal with first line of HTTP protocol message.
- */
-
-static uint16_t
-http_splitline(struct worker *w, unsigned vsl_id, struct http *hp,
-    const struct http_conn *htc, int h1, int h2, int h3)
-{
-	char *p, *q;
-
-	CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC);
-	CHECK_OBJ_NOTNULL(hp, HTTP_MAGIC);
-
-	/* XXX: Assert a NUL at rx.e ? */
-	Tcheck(htc->rxbuf);
-
-	/* Skip leading LWS */
-	for (p = htc->rxbuf.b ; vct_islws(*p); p++)
-		continue;
-
-	/* First field cannot contain SP, CRLF or CTL */
-	q = p;
-	for (; !vct_issp(*p); p++) {
-		if (vct_isctl(*p))
-			return (400);
-	}
-	hp->hd[h1].b = q;
-	hp->hd[h1].e = p;
-
-	/* Skip SP */
-	for (; vct_issp(*p); p++) {
-		if (vct_isctl(*p))
-			return (400);
-	}
-
-	/* Second field cannot contain LWS or CTL */
-	q = p;
-	for (; !vct_islws(*p); p++) {
-		if (vct_isctl(*p))
-			return (400);
-	}
-	hp->hd[h2].b = q;
-	hp->hd[h2].e = p;
-
-	if (!Tlen(hp->hd[h2]))
-		return (413);
-
-	/* Skip SP */
-	for (; vct_issp(*p); p++) {
-		if (vct_isctl(*p))
-			return (400);
-	}
-
-	/* Third field is optional and cannot contain CTL */
-	q = p;
-	if (!vct_iscrlf(*p)) {
-		for (; !vct_iscrlf(*p); p++)
-			if (!vct_issep(*p) && vct_isctl(*p))
-				return (400);
-	}
-	hp->hd[h3].b = q;
-	hp->hd[h3].e = p;
-
-	/* Skip CRLF */
-	p += vct_skipcrlf(p);
-
-	*hp->hd[h1].e = '\0';
-	WSLH(w, vsl_id, hp, h1);
-
-	*hp->hd[h2].e = '\0';
-	WSLH(w, vsl_id, hp, h2);
-
-	if (hp->hd[h3].e != NULL) {
-		*hp->hd[h3].e = '\0';
-		WSLH(w, vsl_id, hp, h3);
-	}
-
-	return (http_dissect_hdrs(w, hp, vsl_id, p, htc));
-}
-
-/*--------------------------------------------------------------------*/
-
-static void
-http_ProtoVer(struct http *hp)
-{
-
-	if (!strcasecmp(hp->hd[HTTP_HDR_PROTO].b, "HTTP/1.0"))
-		hp->protover = 10;
-	else if (!strcasecmp(hp->hd[HTTP_HDR_PROTO].b, "HTTP/1.1"))
-		hp->protover = 11;
-	else
-		hp->protover = 9;
-}
-
-
-/*--------------------------------------------------------------------*/
-
-uint16_t
-http_DissectRequest(struct sess *sp)
-{
-	struct http_conn *htc;
-	struct http *hp;
-	uint16_t retval;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	htc = sp->htc;
-	CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC);
-	hp = sp->http;
-	CHECK_OBJ_NOTNULL(hp, HTTP_MAGIC);
-
-	hp->logtag = HTTP_Rx;
-
-	retval = http_splitline(sp->wrk, sp->vsl_id, hp, htc,
-	    HTTP_HDR_REQ, HTTP_HDR_URL, HTTP_HDR_PROTO);
-	if (retval != 0) {
-		WSPR(sp, SLT_HttpGarbage, htc->rxbuf);
-		return (retval);
-	}
-	http_ProtoVer(hp);
-	return (retval);
-}
-
-/*--------------------------------------------------------------------*/
-
-uint16_t
-http_DissectResponse(struct worker *w, const struct http_conn *htc,
-    struct http *hp)
-{
-	int j;
-	uint16_t retval = 0;
-	char *p;
-
-
-	CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC);
-	CHECK_OBJ_NOTNULL(hp, HTTP_MAGIC);
-	hp->logtag = HTTP_Rx;
-
-	if (http_splitline(w, htc->vsl_id, hp, htc,
-	    HTTP_HDR_PROTO, HTTP_HDR_STATUS, HTTP_HDR_RESPONSE))
-		retval = 503;
-
-	if (retval == 0 && memcmp(hp->hd[HTTP_HDR_PROTO].b, "HTTP/1.", 7))
-		retval = 503;
-
-	if (retval == 0 && Tlen(hp->hd[HTTP_HDR_STATUS]) != 3)
-		retval = 503;
-
-	if (retval == 0) {
-		hp->status = 0;
-		p = hp->hd[HTTP_HDR_STATUS].b;
-		for (j = 100; j != 0; j /= 10) {
-			if (!vct_isdigit(*p)) {
-				retval = 503;
-				break;
-			}
-			hp->status += (uint16_t)(j * (*p - '0'));
-			p++;
-		}
-		if (*p != '\0')
-			retval = 503;
-	}
-
-	if (retval != 0) {
-		WSLR(w, SLT_HttpGarbage, htc->vsl_id, htc->rxbuf);
-		assert(retval >= 100 && retval <= 999);
-		hp->status = retval;
-	} else {
-		http_ProtoVer(hp);
-	}
-
-	if (hp->hd[HTTP_HDR_RESPONSE].b == NULL ||
-	    !Tlen(hp->hd[HTTP_HDR_RESPONSE])) {
-		/* Backend didn't send a response string, use the standard */
-		hp->hd[HTTP_HDR_RESPONSE].b =
-		    TRUST_ME(http_StatusMessage(hp->status));
-		hp->hd[HTTP_HDR_RESPONSE].e =
-		    strchr(hp->hd[HTTP_HDR_RESPONSE].b, '\0');
-	}
-	return (retval);
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-http_SetH(const struct http *to, unsigned n, const char *fm)
-{
-
-	assert(n < to->shd);
-	AN(fm);
-	to->hd[n].b = TRUST_ME(fm);
-	to->hd[n].e = strchr(to->hd[n].b, '\0');
-	to->hdf[n] = 0;
-}
-
-static void
-http_copyh(const struct http *to, const struct http *fm, unsigned n)
-{
-
-	assert(n < HTTP_HDR_FIRST);
-	Tcheck(fm->hd[n]);
-	to->hd[n] = fm->hd[n];
-	to->hdf[n] = fm->hdf[n];
-}
-
-void
-http_ForceGet(const struct http *to)
-{
-	if (strcmp(http_GetReq(to), "GET"))
-		http_SetH(to, HTTP_HDR_REQ, "GET");
-}
-
-void
-http_CopyResp(struct http *to, const struct http *fm)
-{
-
-	CHECK_OBJ_NOTNULL(fm, HTTP_MAGIC);
-	CHECK_OBJ_NOTNULL(to, HTTP_MAGIC);
-	http_SetH(to, HTTP_HDR_PROTO, "HTTP/1.1");
-	to->status = fm->status;
-	http_copyh(to, fm, HTTP_HDR_RESPONSE);
-}
-
-void
-http_SetResp(struct http *to, const char *proto, uint16_t status,
-    const char *response)
-{
-
-	CHECK_OBJ_NOTNULL(to, HTTP_MAGIC);
-	http_SetH(to, HTTP_HDR_PROTO, proto);
-	assert(status >= 100 && status <= 999);
-	to->status = status;
-	http_SetH(to, HTTP_HDR_RESPONSE, response);
-}
-
-static void
-http_copyheader(struct worker *w, unsigned vsl_id, struct http *to,
-    const struct http *fm, unsigned n)
-{
-
-	CHECK_OBJ_NOTNULL(fm, HTTP_MAGIC);
-	CHECK_OBJ_NOTNULL(to, HTTP_MAGIC);
-	assert(n < fm->shd);
-	Tcheck(fm->hd[n]);
-	if (to->nhd < to->shd) {
-		to->hd[to->nhd] = fm->hd[n];
-		to->hdf[to->nhd] = 0;
-		to->nhd++;
-	} else  {
-		VSC_C_main->losthdr++;
-		WSLR(w, SLT_LostHeader, vsl_id, fm->hd[n]);
-	}
-}
-
-/*--------------------------------------------------------------------
- * Estimate how much workspace we need to Filter this header according
- * to 'how'.
- */
-
-unsigned
-http_EstimateWS(const struct http *fm, unsigned how, uint16_t *nhd)
-{
-	unsigned u, l;
-
-	l = 0;
-	*nhd = HTTP_HDR_FIRST;
-	CHECK_OBJ_NOTNULL(fm, HTTP_MAGIC);
-	for (u = 0; u < fm->nhd; u++) {
-		if (fm->hd[u].b == NULL)
-			continue;
-		if (fm->hdf[u] & HDF_FILTER)
-			continue;
-#define HTTPH(a, b, c, d, e, f, g) \
-		if (((e) & how) && http_IsHdr(&fm->hd[u], (b))) \
-			continue;
-#include "tbl/http_headers.h"
-#undef HTTPH
-		l += PRNDUP(Tlen(fm->hd[u]) + 1);
-		(*nhd)++;
-		// fm->hdf[u] |= HDF_COPY;
-	}
-	return (l);
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-http_FilterFields(struct worker *w, unsigned vsl_id, struct http *to,
-    const struct http *fm, unsigned how)
-{
-	unsigned u;
-
-	CHECK_OBJ_NOTNULL(fm, HTTP_MAGIC);
-	CHECK_OBJ_NOTNULL(to, HTTP_MAGIC);
-	to->nhd = HTTP_HDR_FIRST;
-	to->status = fm->status;
-	for (u = HTTP_HDR_FIRST; u < fm->nhd; u++) {
-		if (fm->hd[u].b == NULL)
-			continue;
-		if (fm->hdf[u] & HDF_FILTER)
-			continue;
-#define HTTPH(a, b, c, d, e, f, g) \
-		if (((e) & how) && http_IsHdr(&fm->hd[u], (b))) \
-			continue;
-#include "tbl/http_headers.h"
-#undef HTTPH
-		http_copyheader(w, vsl_id, to, fm, u);
-	}
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-http_FilterHeader(const struct sess *sp, unsigned how)
-{
-	struct http *hp;
-
-	hp = sp->wrk->bereq;
-	CHECK_OBJ_NOTNULL(hp, HTTP_MAGIC);
-	hp->logtag = HTTP_Tx;
-
-	http_copyh(hp, sp->http, HTTP_HDR_REQ);
-	http_copyh(hp, sp->http, HTTP_HDR_URL);
-	if (how == HTTPH_R_FETCH)
-		http_SetH(hp, HTTP_HDR_PROTO, "HTTP/1.1");
-	else
-		http_copyh(hp, sp->http, HTTP_HDR_PROTO);
-	http_FilterFields(sp->wrk, sp->vsl_id, hp, sp->http, how);
-	http_PrintfHeader(sp->wrk, sp->vsl_id, hp, "X-Varnish: %u", sp->xid);
-}
-
-/*--------------------------------------------------------------------
- * This function copies any header fields which reference foreign
- * storage into our own WS.
- */
-
-void
-http_CopyHome(struct worker *w, unsigned vsl_id, const struct http *hp)
-{
-	unsigned u, l;
-	char *p;
-
-	for (u = 0; u < hp->nhd; u++) {
-		if (hp->hd[u].b == NULL)
-			continue;
-		if (hp->hd[u].b >= hp->ws->s && hp->hd[u].e <= hp->ws->e) {
-			WSLH(w, vsl_id, hp, u);
-			continue;
-		}
-		l = Tlen(hp->hd[u]);
-		p = WS_Alloc(hp->ws, l + 1);
-		if (p != NULL) {
-			WSLH(w, vsl_id, hp, u);
-			memcpy(p, hp->hd[u].b, l + 1L);
-			hp->hd[u].b = p;
-			hp->hd[u].e = p + l;
-		} else {
-			/* XXX This leaves a slot empty */
-			VSC_C_main->losthdr++;
-			WSLR(w, SLT_LostHeader, vsl_id, hp->hd[u]);
-			hp->hd[u].b = NULL;
-			hp->hd[u].e = NULL;
-		}
-	}
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-http_ClrHeader(struct http *to)
-{
-
-	CHECK_OBJ_NOTNULL(to, HTTP_MAGIC);
-	to->nhd = HTTP_HDR_FIRST;
-	to->status = 0;
-	to->protover = 0;
-	to->conds = 0;
-	memset(to->hd, 0, sizeof *to->hd * to->shd);
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-http_SetHeader(struct worker *w, unsigned vsl_id, struct http *to,
-    const char *hdr)
-{
-
-	CHECK_OBJ_NOTNULL(to, HTTP_MAGIC);
-	if (to->nhd >= to->shd) {
-		VSC_C_main->losthdr++;
-		WSL(w, SLT_LostHeader, vsl_id, "%s", hdr);
-		return;
-	}
-	http_SetH(to, to->nhd++, hdr);
-}
-
-/*--------------------------------------------------------------------*/
-
-static void
-http_PutField(struct worker *w, unsigned vsl_id, const struct http *to,
-    int field, const char *string)
-{
-	char *p;
-	unsigned l;
-
-	CHECK_OBJ_NOTNULL(to, HTTP_MAGIC);
-	l = strlen(string);
-	p = WS_Alloc(to->ws, l + 1);
-	if (p == NULL) {
-		WSL(w, SLT_LostHeader, vsl_id, "%s", string);
-		to->hd[field].b = NULL;
-		to->hd[field].e = NULL;
-		to->hdf[field] = 0;
-	} else {
-		memcpy(p, string, l + 1L);
-		to->hd[field].b = p;
-		to->hd[field].e = p + l;
-		to->hdf[field] = 0;
-	}
-}
-
-void
-http_PutProtocol(struct worker *w, unsigned vsl_id, const struct http *to,
-    const char *protocol)
-{
-
-	http_PutField(w, vsl_id, to, HTTP_HDR_PROTO, protocol);
-	if (to->hd[HTTP_HDR_PROTO].b == NULL)
-		http_SetH(to, HTTP_HDR_PROTO, "HTTP/1.1");
-	Tcheck(to->hd[HTTP_HDR_PROTO]);
-}
-
-void
-http_PutStatus(struct http *to, uint16_t status)
-{
-
-	assert(status >= 100 && status <= 999);
-	to->status = status;
-}
-
-void
-http_PutResponse(struct worker *w, unsigned vsl_id, const struct http *to,
-    const char *response)
-{
-
-	http_PutField(w, vsl_id, to, HTTP_HDR_RESPONSE, response);
-	if (to->hd[HTTP_HDR_RESPONSE].b == NULL)
-		http_SetH(to, HTTP_HDR_RESPONSE, "Lost Response");
-	Tcheck(to->hd[HTTP_HDR_RESPONSE]);
-}
-
-void
-http_PrintfHeader(struct worker *w, unsigned vsl_id, struct http *to,
-    const char *fmt, ...)
-{
-	va_list ap;
-	unsigned l, n;
-
-	CHECK_OBJ_NOTNULL(to, HTTP_MAGIC);
-	l = WS_Reserve(to->ws, 0);
-	va_start(ap, fmt);
-	n = vsnprintf(to->ws->f, l, fmt, ap);
-	va_end(ap);
-	if (n + 1 >= l || to->nhd >= to->shd) {
-		VSC_C_main->losthdr++;
-		WSL(w, SLT_LostHeader, vsl_id, "%s", to->ws->f);
-		WS_Release(to->ws, 0);
-	} else {
-		to->hd[to->nhd].b = to->ws->f;
-		to->hd[to->nhd].e = to->ws->f + n;
-		to->hdf[to->nhd] = 0;
-		WS_Release(to->ws, n + 1);
-		to->nhd++;
-	}
-}
-/*--------------------------------------------------------------------*/
-
-void
-http_Unset(struct http *hp, const char *hdr)
-{
-	uint16_t u, v;
-
-	for (v = u = HTTP_HDR_FIRST; u < hp->nhd; u++) {
-		if (hp->hd[u].b == NULL)
-			continue;
-		if (http_IsHdr(&hp->hd[u], hdr))
-			continue;
-		if (v != u) {
-			memcpy(&hp->hd[v], &hp->hd[u], sizeof *hp->hd);
-			memcpy(&hp->hdf[v], &hp->hdf[u], sizeof *hp->hdf);
-		}
-		v++;
-	}
-	hp->nhd = v;
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-HTTP_Copy(struct http *to, const struct http * const fm)
-{
-
-	to->conds = fm->conds;
-	to->logtag = fm->logtag;
-	to->status = fm->status;
-	to->protover = fm->protover;
-	to->nhd = fm->nhd;
-	assert(fm->nhd <= to->shd);
-	memcpy(to->hd, fm->hd, fm->nhd * sizeof *to->hd);
-	memcpy(to->hdf, fm->hdf, fm->nhd * sizeof *to->hdf);
-}
-
-/*--------------------------------------------------------------------*/
-
-unsigned
-http_Write(struct worker *w, unsigned vsl_id, const struct http *hp, int resp)
-{
-	unsigned u, l;
-
-	if (resp) {
-		l = WRW_WriteH(w, &hp->hd[HTTP_HDR_PROTO], " ");
-		WSLH(w, vsl_id, hp, HTTP_HDR_PROTO);
-
-		hp->hd[HTTP_HDR_STATUS].b = WS_Alloc(w->ws, 4);
-		AN(hp->hd[HTTP_HDR_STATUS].b);
-
-		sprintf(hp->hd[HTTP_HDR_STATUS].b, "%3d", hp->status);
-		hp->hd[HTTP_HDR_STATUS].e = hp->hd[HTTP_HDR_STATUS].b + 3;
-
-		l += WRW_WriteH(w, &hp->hd[HTTP_HDR_STATUS], " ");
-		WSLH(w, vsl_id, hp, HTTP_HDR_STATUS);
-
-		l += WRW_WriteH(w, &hp->hd[HTTP_HDR_RESPONSE], "\r\n");
-		WSLH(w, vsl_id, hp, HTTP_HDR_RESPONSE);
-	} else {
-		AN(hp->hd[HTTP_HDR_URL].b);
-		l = WRW_WriteH(w, &hp->hd[HTTP_HDR_REQ], " ");
-		WSLH(w, vsl_id, hp, HTTP_HDR_REQ);
-		l += WRW_WriteH(w, &hp->hd[HTTP_HDR_URL], " ");
-		WSLH(w, vsl_id, hp, HTTP_HDR_URL);
-		l += WRW_WriteH(w, &hp->hd[HTTP_HDR_PROTO], "\r\n");
-		WSLH(w, vsl_id, hp, HTTP_HDR_PROTO);
-	}
-	for (u = HTTP_HDR_FIRST; u < hp->nhd; u++) {
-		if (hp->hd[u].b == NULL)
-			continue;
-		AN(hp->hd[u].b);
-		AN(hp->hd[u].e);
-		l += WRW_WriteH(w, &hp->hd[u], "\r\n");
-		WSLH(w, vsl_id, hp, u);
-	}
-	l += WRW_Write(w, "\r\n", -1);
-	return (l);
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-HTTP_Init(void)
-{
-
-#define HTTPH(a, b, c, d, e, f, g) b[0] = (char)strlen(b + 1);
-#include "tbl/http_headers.h"
-#undef HTTPH
-}
diff --git a/bin/varnishd/cache_httpconn.c b/bin/varnishd/cache_httpconn.c
deleted file mode 100644
index 9e0a052..0000000
--- a/bin/varnishd/cache_httpconn.c
+++ /dev/null
@@ -1,229 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * HTTP protocol requests
- *
- * The trouble with the "until magic sequence" design of HTTP protocol messages
- * is that either you have to read a single character at a time, which is
- * inefficient, or you risk reading too much, and pre-read some of the object,
- * or even the next pipelined request, which follows the one you want.
- *
- * HTC reads a HTTP protocol header into a workspace, subject to limits,
- * and stops when we see the magic marker (double [CR]NL), and if we overshoot,
- * it keeps track of the "pipelined" data.
- *
- * We use this both for client and backend connections.
- */
-
-#include "config.h"
-
-#include "cache.h"
-
-#include "vct.h"
-
-/*--------------------------------------------------------------------
- * Check if we have a complete HTTP request or response yet
- *
- * Return values:
- *	 0  No, keep trying
- *	>0  Yes, it is this many bytes long.
- */
-
-static int
-htc_header_complete(txt *t)
-{
-	const char *p;
-
-	Tcheck(*t);
-	assert(*t->e == '\0');
-	/* Skip any leading white space */
-	for (p = t->b ; vct_issp(*p); p++)
-		continue;
-	if (p == t->e) {
-		/* All white space */
-		t->e = t->b;
-		*t->e = '\0';
-		return (0);
-	}
-	while (1) {
-		p = strchr(p, '\n');
-		if (p == NULL)
-			return (0);
-		p++;
-		if (*p == '\r')
-			p++;
-		if (*p == '\n')
-			break;
-	}
-	p++;
-	return (p - t->b);
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-HTC_Init(struct http_conn *htc, struct ws *ws, int fd, unsigned vsl_id,
-    unsigned maxbytes, unsigned maxhdr)
-{
-
-	htc->magic = HTTP_CONN_MAGIC;
-	htc->ws = ws;
-	htc->fd = fd;
-	htc->vsl_id = vsl_id;
-	htc->maxbytes = maxbytes;
-	htc->maxhdr = maxhdr;
-
-	(void)WS_Reserve(htc->ws, htc->maxbytes);
-	htc->rxbuf.b = ws->f;
-	htc->rxbuf.e = ws->f;
-	*htc->rxbuf.e = '\0';
-	htc->pipeline.b = NULL;
-	htc->pipeline.e = NULL;
-}
-
-/*--------------------------------------------------------------------
- * Start over, and recycle any pipelined input.
- * The WS_Reset is safe, even though the pipelined input is stored in
- * the ws somewhere, because WS_Reset only fiddles pointers.
- */
-
-int
-HTC_Reinit(struct http_conn *htc)
-{
-	unsigned l;
-
-	CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC);
-	(void)WS_Reserve(htc->ws, htc->maxbytes);
-	htc->rxbuf.b = htc->ws->f;
-	htc->rxbuf.e = htc->ws->f;
-	if (htc->pipeline.b != NULL) {
-		l = Tlen(htc->pipeline);
-		memmove(htc->rxbuf.b, htc->pipeline.b, l);
-		htc->rxbuf.e += l;
-		htc->pipeline.b = NULL;
-		htc->pipeline.e = NULL;
-	}
-	*htc->rxbuf.e = '\0';
-	return (HTC_Complete(htc));
-}
-
-/*--------------------------------------------------------------------
- * Return 1 if we have a complete HTTP procol header
- */
-
-int
-HTC_Complete(struct http_conn *htc)
-{
-	int i;
-
-	CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC);
-	i = htc_header_complete(&htc->rxbuf);
-	assert(i >= 0);
-	if (i == 0)
-		return (0);
-	WS_ReleaseP(htc->ws, htc->rxbuf.e);
-	AZ(htc->pipeline.b);
-	AZ(htc->pipeline.e);
-	if (htc->rxbuf.b + i < htc->rxbuf.e) {
-		htc->pipeline.b = htc->rxbuf.b + i;
-		htc->pipeline.e = htc->rxbuf.e;
-		htc->rxbuf.e = htc->pipeline.b;
-	}
-	return (1);
-}
-
-/*--------------------------------------------------------------------
- * Receive more HTTP protocol bytes
- * Returns:
- *	-2 overflow
- *	-1 error
- *	 0 more needed
- *	 1 got complete HTTP header
- */
-
-int
-HTC_Rx(struct http_conn *htc)
-{
-	int i;
-
-	CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC);
-	AN(htc->ws->r);
-	i = (htc->ws->r - htc->rxbuf.e) - 1;	/* space for NUL */
-	if (i <= 0) {
-		WS_ReleaseP(htc->ws, htc->rxbuf.b);
-		return (-2);
-	}
-	i = read(htc->fd, htc->rxbuf.e, i);
-	if (i <= 0) {
-		/*
-		 * We wouldn't come here if we had a complete HTTP header
-		 * so consequently an EOF can not be OK
-		 */
-		WS_ReleaseP(htc->ws, htc->rxbuf.b);
-		return (-1);
-	}
-	htc->rxbuf.e += i;
-	*htc->rxbuf.e = '\0';
-	return (HTC_Complete(htc));
-}
-
-/*--------------------------------------------------------------------
- * Read up to len bytes, returning pipelined data first.
- */
-
-ssize_t
-HTC_Read(struct worker *w, struct http_conn *htc, void *d, size_t len)
-{
-	size_t l;
-	unsigned char *p;
-	ssize_t i;
-
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC);
-	l = 0;
-	p = d;
-	if (htc->pipeline.b) {
-		l = Tlen(htc->pipeline);
-		if (l > len)
-			l = len;
-		memcpy(p, htc->pipeline.b, l);
-		p += l;
-		len -= l;
-		htc->pipeline.b += l;
-		if (htc->pipeline.b == htc->pipeline.e)
-			htc->pipeline.b = htc->pipeline.e = NULL;
-	}
-	if (len == 0)
-		return (l);
-	i = read(htc->fd, p, len);
-	if (i < 0) {
-		WSL(w, SLT_FetchError, htc->vsl_id, "%s", strerror(errno));
-		return (i);
-	}
-	return (i + l);
-}
diff --git a/bin/varnishd/cache_lck.c b/bin/varnishd/cache_lck.c
deleted file mode 100644
index 2aef6dc..0000000
--- a/bin/varnishd/cache_lck.c
+++ /dev/null
@@ -1,210 +0,0 @@
-/*-
- * Copyright (c) 2008-2010 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * The geniuses who came up with pthreads did not think operations like
- * pthread_assert_mutex_held() were important enough to include them in
- * the API.
- *
- * Build our own locks on top of pthread mutexes and hope that the next
- * civilization is better at such crucial details than this one.
- */
-
-#include "config.h"
-
-#include <stdlib.h>
-
-#include "cache.h"
-
-/*The constability of lck depends on platform pthreads implementation */
-
-struct ilck {
-	unsigned		magic;
-#define ILCK_MAGIC		0x7b86c8a5
-	pthread_mutex_t		mtx;
-	int			held;
-	pthread_t		owner;
-	VTAILQ_ENTRY(ilck)	list;
-	const char		*w;
-	struct VSC_C_lck	*stat;
-};
-
-static VTAILQ_HEAD(, ilck)	ilck_head =
-    VTAILQ_HEAD_INITIALIZER(ilck_head);
-
-static pthread_mutex_t		lck_mtx;
-
-void __match_proto__()
-Lck__Lock(struct lock *lck, const char *p, const char *f, int l)
-{
-	struct ilck *ilck;
-	int r;
-
-	CAST_OBJ_NOTNULL(ilck, lck->priv, ILCK_MAGIC);
-	if (!(cache_param->diag_bitmap & 0x18)) {
-		AZ(pthread_mutex_lock(&ilck->mtx));
-		AZ(ilck->held);
-		ilck->stat->locks++;
-		ilck->owner = pthread_self();
-		ilck->held = 1;
-		return;
-	}
-	r = pthread_mutex_trylock(&ilck->mtx);
-	assert(r == 0 || r == EBUSY);
-	if (r) {
-		ilck->stat->colls++;
-		if (cache_param->diag_bitmap & 0x8)
-			VSL(SLT_Debug, 0, "MTX_CONTEST(%s,%s,%d,%s)",
-			    p, f, l, ilck->w);
-		AZ(pthread_mutex_lock(&ilck->mtx));
-	} else if (cache_param->diag_bitmap & 0x8) {
-		VSL(SLT_Debug, 0, "MTX_LOCK(%s,%s,%d,%s)", p, f, l, ilck->w);
-	}
-	AZ(ilck->held);
-	ilck->stat->locks++;
-	ilck->owner = pthread_self();
-	ilck->held = 1;
-}
-
-void __match_proto__()
-Lck__Unlock(struct lock *lck, const char *p, const char *f, int l)
-{
-	struct ilck *ilck;
-
-	CAST_OBJ_NOTNULL(ilck, lck->priv, ILCK_MAGIC);
-	assert(pthread_equal(ilck->owner, pthread_self()));
-	AN(ilck->held);
-	ilck->held = 0;
-	AZ(pthread_mutex_unlock(&ilck->mtx));
-	if (cache_param->diag_bitmap & 0x8)
-		VSL(SLT_Debug, 0, "MTX_UNLOCK(%s,%s,%d,%s)", p, f, l, ilck->w);
-}
-
-int __match_proto__()
-Lck__Trylock(struct lock *lck, const char *p, const char *f, int l)
-{
-	struct ilck *ilck;
-	int r;
-
-	CAST_OBJ_NOTNULL(ilck, lck->priv, ILCK_MAGIC);
-	r = pthread_mutex_trylock(&ilck->mtx);
-	assert(r == 0 || r == EBUSY);
-	if (cache_param->diag_bitmap & 0x8)
-		VSL(SLT_Debug, 0,
-		    "MTX_TRYLOCK(%s,%s,%d,%s) = %d", p, f, l, ilck->w);
-	if (r == 0) {
-		AZ(ilck->held);
-		ilck->held = 1;
-		ilck->owner = pthread_self();
-	}
-	return (r);
-}
-
-void
-Lck__Assert(const struct lock *lck, int held)
-{
-	struct ilck *ilck;
-
-	CAST_OBJ_NOTNULL(ilck, lck->priv, ILCK_MAGIC);
-	if (held)
-		assert(ilck->held &&
-		    pthread_equal(ilck->owner, pthread_self()));
-	else
-		assert(!ilck->held ||
-		    !pthread_equal(ilck->owner, pthread_self()));
-}
-
-int __match_proto__()
-Lck_CondWait(pthread_cond_t *cond, struct lock *lck, struct timespec *ts)
-{
-	struct ilck *ilck;
-	int retval = 0;
-
-	CAST_OBJ_NOTNULL(ilck, lck->priv, ILCK_MAGIC);
-	AN(ilck->held);
-	assert(pthread_equal(ilck->owner, pthread_self()));
-	ilck->held = 0;
-	if (ts == NULL) {
-		AZ(pthread_cond_wait(cond, &ilck->mtx));
-	} else {
-		retval = pthread_cond_timedwait(cond, &ilck->mtx, ts);
-		assert(retval == 0 || retval == ETIMEDOUT);
-	}
-	AZ(ilck->held);
-	ilck->held = 1;
-	ilck->owner = pthread_self();
-	return (retval);
-}
-
-void
-Lck__New(struct lock *lck, struct VSC_C_lck *st, const char *w)
-{
-	struct ilck *ilck;
-
-	AN(st);
-	AZ(lck->priv);
-	ALLOC_OBJ(ilck, ILCK_MAGIC);
-	AN(ilck);
-	ilck->w = w;
-	ilck->stat = st;
-	ilck->stat->creat++;
-	AZ(pthread_mutex_init(&ilck->mtx, NULL));
-	AZ(pthread_mutex_lock(&lck_mtx));
-	VTAILQ_INSERT_TAIL(&ilck_head, ilck, list);
-	AZ(pthread_mutex_unlock(&lck_mtx));
-	lck->priv = ilck;
-}
-
-void
-Lck_Delete(struct lock *lck)
-{
-	struct ilck *ilck;
-
-	CAST_OBJ_NOTNULL(ilck, lck->priv, ILCK_MAGIC);
-	ilck->stat->destroy++;
-	lck->priv = NULL;
-	AZ(pthread_mutex_lock(&lck_mtx));
-	VTAILQ_REMOVE(&ilck_head, ilck, list);
-	AZ(pthread_mutex_unlock(&lck_mtx));
-	AZ(pthread_mutex_destroy(&ilck->mtx));
-	FREE_OBJ(ilck);
-}
-
-#define LOCK(nam) struct VSC_C_lck *lck_##nam;
-#include "tbl/locks.h"
-#undef LOCK
-
-void
-LCK_Init(void)
-{
-
-	AZ(pthread_mutex_init(&lck_mtx, NULL));
-#define LOCK(nam)						\
-	lck_##nam = VSM_Alloc(sizeof(struct VSC_C_lck),		\
-	   VSC_CLASS, VSC_TYPE_LCK, #nam);
-#include "tbl/locks.h"
-#undef LOCK
-}
diff --git a/bin/varnishd/cache_main.c b/bin/varnishd/cache_main.c
deleted file mode 100644
index eb3fa1d..0000000
--- a/bin/varnishd/cache_main.c
+++ /dev/null
@@ -1,146 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2009 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- */
-
-#include "config.h"
-
-#include <stdio.h>
-#include <stdlib.h>
-
-#include "cache.h"
-#include "common/heritage.h"
-
-#include "waiter/cache_waiter.h"
-#include "hash/hash_slinger.h"
-
-volatile struct params	*cache_param;
-
-/*--------------------------------------------------------------------
- * Per thread storage for the session currently being processed by
- * the thread.  This is used for panic messages.
- */
-
-static pthread_key_t sp_key;
-
-void
-THR_SetSession(const struct sess *sp)
-{
-
-	AZ(pthread_setspecific(sp_key, sp));
-}
-
-const struct sess *
-THR_GetSession(void)
-{
-
-	return (pthread_getspecific(sp_key));
-}
-
-/*--------------------------------------------------------------------
- * Name threads if our pthreads implementation supports it.
- */
-
-static pthread_key_t name_key;
-
-void
-THR_SetName(const char *name)
-{
-
-	AZ(pthread_setspecific(name_key, name));
-#ifdef HAVE_PTHREAD_SET_NAME_NP
-	pthread_set_name_np(pthread_self(), name);
-#endif
-}
-
-const char *
-THR_GetName(void)
-{
-
-	return (pthread_getspecific(name_key));
-}
-
-/*--------------------------------------------------------------------
- * XXX: Think more about which order we start things
- */
-
-void
-child_main(void)
-{
-
-	setbuf(stdout, NULL);
-	setbuf(stderr, NULL);
-	printf("Child starts\n");
-
-	AZ(pthread_key_create(&sp_key, NULL));
-	AZ(pthread_key_create(&name_key, NULL));
-
-	THR_SetName("cache-main");
-
-	VSL_Init();	/* First, LCK needs it. */
-
-	LCK_Init();	/* Second, locking */
-
-	WAIT_Init();
-	PAN_Init();
-	CLI_Init();
-	Fetch_Init();
-
-	CNT_Init();
-	VCL_Init();
-
-	HTTP_Init();
-
-	VBE_Init();
-	VBP_Init();
-	WRK_Init();
-	Pool_Init();
-
-	EXP_Init();
-	HSH_Init(heritage.hash);
-	BAN_Init();
-
-	VCA_Init();
-
-	SMS_Init();
-	SMP_Init();
-	STV_open();
-
-	VMOD_Init();
-
-	BAN_Compile();
-
-	/* Wait for persistent storage to load if asked to */
-	if (cache_param->diag_bitmap & 0x00020000)
-		SMP_Ready();
-
-	CLI_Run();
-
-	STV_close();
-
-	printf("Child dies\n");
-}
diff --git a/bin/varnishd/cache_panic.c b/bin/varnishd/cache_panic.c
deleted file mode 100644
index c626ae3..0000000
--- a/bin/varnishd/cache_panic.c
+++ /dev/null
@@ -1,387 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Dag-Erling Sm?rgrav <des at des.no>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- */
-
-#include "config.h"
-
-#ifndef HAVE_EXECINFO_H
-#include "compat/execinfo.h"
-#else
-#include <execinfo.h>
-#endif
-
-#include <stdio.h>
-#include <stdlib.h>
-
-#include "cache.h"
-
-#include "vapi/vsm_int.h"
-
-#include "cache_backend.h"
-#include "waiter/cache_waiter.h"
-#include "vcl.h"
-
-/*
- * The panic string is constructed in memory, then copied to the
- * shared memory.
- *
- * It can be extracted post-mortem from a core dump using gdb:
- *
- * (gdb) printf "%s", panicstr
- */
-
-static struct vsb vsps, *vsp;
-static pthread_mutex_t panicstr_mtx = PTHREAD_MUTEX_INITIALIZER;
-
-/*--------------------------------------------------------------------*/
-
-static void
-pan_ws(const struct ws *ws, int indent)
-{
-
-	VSB_printf(vsp, "%*sws = %p { %s\n", indent, "",
-	    ws, ws->overflow ? "overflow" : "");
-	VSB_printf(vsp, "%*sid = \"%s\",\n", indent + 2, "", ws->id);
-	VSB_printf(vsp, "%*s{s,f,r,e} = {%p", indent + 2, "", ws->s);
-	if (ws->f > ws->s)
-		VSB_printf(vsp, ",+%ld", (long) (ws->f - ws->s));
-	else
-		VSB_printf(vsp, ",%p", ws->f);
-	if (ws->r > ws->s)
-		VSB_printf(vsp, ",+%ld", (long) (ws->r - ws->s));
-	else
-		VSB_printf(vsp, ",%p", ws->r);
-	if (ws->e > ws->s)
-		VSB_printf(vsp, ",+%ld", (long) (ws->e - ws->s));
-	else
-		VSB_printf(vsp, ",%p", ws->e);
-	VSB_printf(vsp, "},\n");
-	VSB_printf(vsp, "%*s},\n", indent, "" );
-}
-
-/*--------------------------------------------------------------------*/
-
-static void
-pan_vbc(const struct vbc *vbc)
-{
-
-	struct backend *be;
-
-	be = vbc->backend;
-
-	VSB_printf(vsp, "  backend = %p fd = %d {\n", be, vbc->fd);
-	VSB_printf(vsp, "    display_name = \"%s\",\n", be->display_name);
-	VSB_printf(vsp, "  },\n");
-}
-
-/*--------------------------------------------------------------------*/
-
-static void
-pan_storage(const struct storage *st)
-{
-	int i, j;
-
-#define MAX_BYTES (4*16)
-#define show(ch) (((ch) > 31 && (ch) < 127) ? (ch) : '.')
-
-	VSB_printf(vsp, "      %u {\n", st->len);
-	for (i = 0; i < MAX_BYTES && i < st->len; i += 16) {
-		VSB_printf(vsp, "        ");
-		for (j = 0; j < 16; ++j) {
-			if (i + j < st->len)
-				VSB_printf(vsp, "%02x ", st->ptr[i + j]);
-			else
-				VSB_printf(vsp, "   ");
-		}
-		VSB_printf(vsp, "|");
-		for (j = 0; j < 16; ++j)
-			if (i + j < st->len)
-				VSB_printf(vsp, "%c", show(st->ptr[i + j]));
-		VSB_printf(vsp, "|\n");
-	}
-	if (st->len > MAX_BYTES)
-		VSB_printf(vsp, "        [%u more]\n", st->len - MAX_BYTES);
-	VSB_printf(vsp, "      },\n");
-
-#undef show
-#undef MAX_BYTES
-}
-
-/*--------------------------------------------------------------------*/
-
-static void
-pan_http(const char *id, const struct http *h, int indent)
-{
-	int i;
-
-	VSB_printf(vsp, "%*shttp[%s] = {\n", indent, "", id);
-	VSB_printf(vsp, "%*sws = %p[%s]\n", indent + 2, "",
-	    h->ws, h->ws ? h->ws->id : "");
-	for (i = 0; i < h->nhd; ++i) {
-		if (h->hd[i].b == NULL && h->hd[i].e == NULL)
-			continue;
-		VSB_printf(vsp, "%*s\"%.*s\",\n", indent + 4, "",
-		    (int)(h->hd[i].e - h->hd[i].b),
-		    h->hd[i].b);
-	}
-	VSB_printf(vsp, "%*s},\n", indent, "");
-}
-
-
-/*--------------------------------------------------------------------*/
-
-static void
-pan_object(const struct object *o)
-{
-	const struct storage *st;
-
-	VSB_printf(vsp, "  obj = %p {\n", o);
-	VSB_printf(vsp, "    xid = %u,\n", o->xid);
-	pan_ws(o->ws_o, 4);
-	pan_http("obj", o->http, 4);
-	VSB_printf(vsp, "    len = %jd,\n", (intmax_t)o->len);
-	VSB_printf(vsp, "    store = {\n");
-	VTAILQ_FOREACH(st, &o->store, list)
-		pan_storage(st);
-	VSB_printf(vsp, "    },\n");
-	VSB_printf(vsp, "  },\n");
-}
-
-/*--------------------------------------------------------------------*/
-
-static void
-pan_vcl(const struct VCL_conf *vcl)
-{
-	int i;
-
-	VSB_printf(vsp, "    vcl = {\n");
-	VSB_printf(vsp, "      srcname = {\n");
-	for (i = 0; i < vcl->nsrc; ++i)
-		VSB_printf(vsp, "        \"%s\",\n", vcl->srcname[i]);
-	VSB_printf(vsp, "      },\n");
-	VSB_printf(vsp, "    },\n");
-}
-
-
-/*--------------------------------------------------------------------*/
-
-static void
-pan_wrk(const struct worker *wrk)
-{
-
-	VSB_printf(vsp, "  worker = %p {\n", wrk);
-	pan_ws(wrk->ws, 4);
-	if (wrk->bereq->ws != NULL)
-		pan_http("bereq", wrk->bereq, 4);
-	if (wrk->beresp->ws != NULL)
-		pan_http("beresp", wrk->beresp, 4);
-	if (wrk->resp->ws != NULL)
-		pan_http("resp", wrk->resp, 4);
-	VSB_printf(vsp, "    },\n");
-}
-
-/*--------------------------------------------------------------------*/
-
-static void
-pan_sess(const struct sess *sp)
-{
-	const char *stp, *hand;
-
-	VSB_printf(vsp, "sp = %p {\n", sp);
-	VSB_printf(vsp,
-	    "  fd = %d, id = %d, xid = %u,\n",
-	    sp->fd, sp->vsl_id & VSL_IDENTMASK, sp->xid);
-	VSB_printf(vsp, "  client = %s %s,\n",
-	    sp->addr ? sp->addr : "?.?.?.?",
-	    sp->port ? sp->port : "?");
-	switch (sp->step) {
-#define STEP(l, u) case STP_##u: stp = "STP_" #u; break;
-#include "tbl/steps.h"
-#undef STEP
-		default: stp = NULL;
-	}
-	hand = VCL_Return_Name(sp->handling);
-	if (stp != NULL)
-		VSB_printf(vsp, "  step = %s,\n", stp);
-	else
-		VSB_printf(vsp, "  step = 0x%x,\n", sp->step);
-	if (hand != NULL)
-		VSB_printf(vsp, "  handling = %s,\n", hand);
-	else
-		VSB_printf(vsp, "  handling = 0x%x,\n", sp->handling);
-	if (sp->err_code)
-		VSB_printf(vsp,
-		    "  err_code = %d, err_reason = %s,\n", sp->err_code,
-		    sp->err_reason ? sp->err_reason : "(null)");
-
-	VSB_printf(vsp, "  restarts = %d, esi_level = %d\n",
-	    sp->restarts, sp->esi_level);
-
-	VSB_printf(vsp, "  flags = ");
-	if (sp->wrk->do_stream)	VSB_printf(vsp, " do_stream");
-	if (sp->wrk->do_gzip)	VSB_printf(vsp, " do_gzip");
-	if (sp->wrk->do_gunzip)	VSB_printf(vsp, " do_gunzip");
-	if (sp->wrk->do_esi)	VSB_printf(vsp, " do_esi");
-	if (sp->wrk->do_close)	VSB_printf(vsp, " do_close");
-	if (sp->wrk->is_gzip)	VSB_printf(vsp, " is_gzip");
-	if (sp->wrk->is_gunzip)	VSB_printf(vsp, " is_gunzip");
-	VSB_printf(vsp, "\n");
-	VSB_printf(vsp, "  bodystatus = %d\n", sp->wrk->body_status);
-
-	pan_ws(sp->ws, 2);
-	pan_http("req", sp->http, 2);
-
-	if (sp->wrk != NULL)
-		pan_wrk(sp->wrk);
-
-	if (VALID_OBJ(sp->vcl, VCL_CONF_MAGIC))
-		pan_vcl(sp->vcl);
-
-	if (VALID_OBJ(sp->wrk->vbc, BACKEND_MAGIC))
-		pan_vbc(sp->wrk->vbc);
-
-	if (VALID_OBJ(sp->obj, OBJECT_MAGIC))
-		pan_object(sp->obj);
-
-	VSB_printf(vsp, "},\n");
-}
-
-/*--------------------------------------------------------------------*/
-
-static void
-pan_backtrace(void)
-{
-	void *array[10];
-	size_t size;
-	size_t i;
-
-	size = backtrace (array, 10);
-	if (size == 0)
-		return;
-	VSB_printf(vsp, "Backtrace:\n");
-	for (i = 0; i < size; i++) {
-		VSB_printf (vsp, "  ");
-		if (Symbol_Lookup(vsp, array[i]) < 0) {
-			char **strings;
-			strings = backtrace_symbols(&array[i], 1);
-			if (strings != NULL && strings[0] != NULL)
-				VSB_printf(vsp, "%p: %s", array[i], strings[0]);
-			else
-				VSB_printf(vsp, "%p: (?)", array[i]);
-		}
-		VSB_printf (vsp, "\n");
-	}
-}
-
-/*--------------------------------------------------------------------*/
-
-static void
-pan_ic(const char *func, const char *file, int line, const char *cond,
-    int err, int xxx)
-{
-	const char *q;
-	const struct sess *sp;
-
-	AZ(pthread_mutex_lock(&panicstr_mtx)); /* Won't be released,
-						  we're going to die
-						  anyway */
-	switch(xxx) {
-	case 3:
-		VSB_printf(vsp,
-		    "Wrong turn at %s:%d:\n%s\n", file, line, cond);
-		break;
-	case 2:
-		VSB_printf(vsp,
-		    "Panic from VCL:\n  %s\n", cond);
-		break;
-	case 1:
-		VSB_printf(vsp,
-		    "Missing errorhandling code in %s(), %s line %d:\n"
-		    "  Condition(%s) not true.",
-		    func, file, line, cond);
-		break;
-	default:
-	case 0:
-		VSB_printf(vsp,
-		    "Assert error in %s(), %s line %d:\n"
-		    "  Condition(%s) not true.\n",
-		    func, file, line, cond);
-		break;
-	}
-	if (err)
-		VSB_printf(vsp, "errno = %d (%s)\n", err, strerror(err));
-
-	q = THR_GetName();
-	if (q != NULL)
-		VSB_printf(vsp, "thread = (%s)\n", q);
-
-	VSB_printf(vsp, "ident = %s,%s\n",
-	    VSB_data(vident) + 1, WAIT_GetName());
-
-	pan_backtrace();
-
-	if (!(cache_param->diag_bitmap & 0x2000)) {
-		sp = THR_GetSession();
-		if (sp != NULL)
-			pan_sess(sp);
-	}
-	VSB_printf(vsp, "\n");
-	VSB_bcat(vsp, "", 1);	/* NUL termination */
-
-	if (cache_param->diag_bitmap & 0x4000)
-		(void)fputs(VSM_head->panicstr, stderr);
-
-#ifdef HAVE_ABORT2
-	if (cache_param->diag_bitmap & 0x8000) {
-		void *arg[1];
-		char *p;
-
-		for (p = VSM_head->panicstr; *p; p++)
-			if (*p == '\n')
-				*p = ' ';
-		arg[0] = VSM_head->panicstr;
-		abort2(VSM_head->panicstr, 1, arg);
-	}
-#endif
-	if (cache_param->diag_bitmap & 0x1000)
-		exit(4);
-	else
-		abort();
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-PAN_Init(void)
-{
-
-	VAS_Fail = pan_ic;
-	vsp = &vsps;
-	AN(VSB_new(vsp, VSM_head->panicstr, sizeof VSM_head->panicstr,
-	    VSB_FIXEDLEN));
-}
diff --git a/bin/varnishd/cache_pipe.c b/bin/varnishd/cache_pipe.c
deleted file mode 100644
index 4180d39..0000000
--- a/bin/varnishd/cache_pipe.c
+++ /dev/null
@@ -1,133 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * XXX: charge bytes to srcaddr
- */
-
-#include "config.h"
-
-#include <poll.h>
-#include <stdio.h>
-
-#include "cache.h"
-
-#include "cache_backend.h"
-#include "vtcp.h"
-#include "vtim.h"
-
-static int
-rdf(int fd0, int fd1)
-{
-	int i, j;
-	char buf[BUFSIZ], *p;
-
-	i = read(fd0, buf, sizeof buf);
-	if (i <= 0)
-		return (1);
-	for (p = buf; i > 0; i -= j, p += j) {
-		j = write(fd1, p, i);
-		if (j <= 0)
-			return (1);
-		if (i != j)
-			(void)usleep(100000);		/* XXX hack */
-	}
-	return (0);
-}
-
-void
-PipeSession(struct sess *sp)
-{
-	struct vbc *vc;
-	struct worker *w;
-	struct pollfd fds[2];
-	int i;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->wrk, WORKER_MAGIC);
-	w = sp->wrk;
-
-	sp->wrk->vbc = VDI_GetFd(NULL, sp);
-	if (sp->wrk->vbc == NULL)
-		return;
-	vc = sp->wrk->vbc;
-	(void)VTCP_blocking(vc->fd);
-
-	WRW_Reserve(w, &vc->fd);
-	sp->wrk->acct_tmp.hdrbytes +=
-	    http_Write(w, sp->vsl_id, sp->wrk->bereq, 0);
-
-	if (sp->htc->pipeline.b != NULL)
-		sp->wrk->acct_tmp.bodybytes +=
-		    WRW_Write(w, sp->htc->pipeline.b, Tlen(sp->htc->pipeline));
-
-	i = WRW_FlushRelease(w);
-
-	if (i) {
-		SES_Close(sp, "pipe");
-		VDI_CloseFd(sp->wrk);
-		return;
-	}
-
-	sp->t_resp = VTIM_real();
-
-	memset(fds, 0, sizeof fds);
-
-	// XXX: not yet (void)VTCP_linger(vc->fd, 0);
-	fds[0].fd = vc->fd;
-	fds[0].events = POLLIN | POLLERR;
-
-	// XXX: not yet (void)VTCP_linger(sp->fd, 0);
-	fds[1].fd = sp->fd;
-	fds[1].events = POLLIN | POLLERR;
-
-	while (fds[0].fd > -1 || fds[1].fd > -1) {
-		fds[0].revents = 0;
-		fds[1].revents = 0;
-		i = poll(fds, 2, cache_param->pipe_timeout * 1000);
-		if (i < 1)
-			break;
-		if (fds[0].revents && rdf(vc->fd, sp->fd)) {
-			if (fds[1].fd == -1)
-				break;
-			(void)shutdown(vc->fd, SHUT_RD);
-			(void)shutdown(sp->fd, SHUT_WR);
-			fds[0].events = 0;
-			fds[0].fd = -1;
-		}
-		if (fds[1].revents && rdf(sp->fd, vc->fd)) {
-			if (fds[0].fd == -1)
-				break;
-			(void)shutdown(sp->fd, SHUT_RD);
-			(void)shutdown(vc->fd, SHUT_WR);
-			fds[1].events = 0;
-			fds[1].fd = -1;
-		}
-	}
-	SES_Close(sp, "pipe");
-	VDI_CloseFd(sp->wrk);
-}
diff --git a/bin/varnishd/cache_pool.c b/bin/varnishd/cache_pool.c
deleted file mode 100644
index 79a5fcd..0000000
--- a/bin/varnishd/cache_pool.c
+++ /dev/null
@@ -1,594 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * We maintain a number of worker thread pools, to spread lock contention.
- *
- * Pools can be added on the fly, as a means to mitigate lock contention,
- * but can only be removed again by a restart. (XXX: we could fix that)
- *
- * Two threads herd the pools, one eliminates idle threads and aggregates
- * statistics for all the pools, the other thread creates new threads
- * on demand, subject to various numerical constraints.
- *
- * The algorithm for when to create threads needs to be reactive enough
- * to handle startup spikes, but sufficiently attenuated to not cause
- * thread pileups.  This remains subject for improvement.
- */
-
-#include "config.h"
-
-#include <math.h>
-#include <stdlib.h>
-
-#include "cache.h"
-#include "common/heritage.h"
-
-#include "waiter/cache_waiter.h"
-#include "vtcp.h"
-#include "vtim.h"
-
-/*--------------------------------------------------------------------
- * MAC OS/X is incredibly moronic when it comes to time and such...
- */
-
-#ifndef CLOCK_MONOTONIC
-#define CLOCK_MONOTONIC	0
-
-#include <sys/time.h>
-
-static int
-clock_gettime(int foo, struct timespec *ts)
-{
-	struct timeval tv;
-
-	(void)foo;
-	gettimeofday(&tv, NULL);
-	ts->tv_sec = tv.tv_sec;
-	ts->tv_nsec = tv.tv_usec * 1000;
-	return (0);
-}
-
-static int
-pthread_condattr_setclock(pthread_condattr_t *attr, int foo)
-{
-	(void)attr;
-	(void)foo;
-	return (0);
-}
-#endif /* !CLOCK_MONOTONIC */
-
-static void *waiter_priv;
-
-VTAILQ_HEAD(workerhead, worker);
-
-struct poolsock {
-	unsigned			magic;
-#define POOLSOCK_MAGIC			0x1b0a2d38
-	VTAILQ_ENTRY(poolsock)		list;
-	struct listen_sock		*lsock;
-};
-
-/* Number of work requests queued in excess of worker threads available */
-
-struct pool {
-	unsigned		magic;
-#define POOL_MAGIC		0x606658fa
-	VTAILQ_ENTRY(pool)	list;
-
-	pthread_cond_t		herder_cond;
-	struct lock		herder_mtx;
-	pthread_t		herder_thr;
-
-	struct lock		mtx;
-	struct workerhead	idle;
-	VTAILQ_HEAD(, sess)	queue;
-	VTAILQ_HEAD(, poolsock)	socks;
-	unsigned		nthr;
-	unsigned		lqueue;
-	unsigned		last_lqueue;
-	uintmax_t		ndropped;
-	uintmax_t		nqueued;
-	struct sesspool		*sesspool;
-};
-
-static struct lock		pool_mtx;
-static pthread_t		thr_pool_herder;
-
-/*--------------------------------------------------------------------
- * Nobody is accepting on this socket, so we do.
- *
- * As long as we can stick the accepted connection to another thread
- * we do so, otherwise we return and handle it ourselves.
- *
- * Notice calling convention:  Called locked and returns locked, but
- * works lock in the meantime.
- *
- * We store data about the accept in reserved workspace, it is only used
- * for a brief moment and it takes up around 144 bytes.
- */
-
-static int
-pool_accept(struct pool *pp, struct worker *w, const struct poolsock *ps)
-{
-	struct worker *w2;
-	struct wrk_accept *wa, *wa2;
-
-	CHECK_OBJ_NOTNULL(pp, POOL_MAGIC);
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	CHECK_OBJ_NOTNULL(ps, POOLSOCK_MAGIC);
-
-	CHECK_OBJ_NOTNULL(ps->lsock, LISTEN_SOCK_MAGIC);
-	Lck_AssertHeld(&pp->mtx);
-	Lck_Unlock(&pp->mtx);
-	assert(sizeof *wa == WS_Reserve(w->ws, sizeof *wa));
-	wa = (void*)w->ws->f;
-	while (1) {
-		memset(wa, 0, sizeof *wa);
-		wa->magic = WRK_ACCEPT_MAGIC;
-
-		if (ps->lsock->sock < 0) {
-			/* Socket Shutdown */
-			Lck_Lock(&pp->mtx);
-			return (-1);
-		}
-		if (VCA_Accept(ps->lsock, wa) < 0) {
-			w->stats.sess_fail++;
-			/* We're going to pace in vca anyway... */
-			(void)WRK_TrySumStat(w);
-			continue;
-		}
-
-		Lck_Lock(&pp->mtx);
-		if (VTAILQ_EMPTY(&pp->idle))
-			return (0);
-		w2 = VTAILQ_FIRST(&pp->idle);
-		VTAILQ_REMOVE(&pp->idle, w2, list);
-		Lck_Unlock(&pp->mtx);
-		assert(sizeof *wa2 == WS_Reserve(w2->ws, sizeof *wa2));
-		wa2 = (void*)w2->ws->f;
-		memcpy(wa2, wa, sizeof *wa);
-		AZ(pthread_cond_signal(&w2->cond));
-	}
-}
-
-/*--------------------------------------------------------------------
- * This is the work function for worker threads in the pool.
- */
-
-void
-Pool_Work_Thread(void *priv, struct worker *w)
-{
-	struct pool *pp;
-	int stats_clean, i;
-	struct poolsock *ps;
-
-	CAST_OBJ_NOTNULL(pp, priv, POOL_MAGIC);
-	w->pool = pp;
-	Lck_Lock(&pp->mtx);
-	stats_clean = 1;
-	while (1) {
-
-		Lck_AssertHeld(&pp->mtx);
-
-		CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-		CHECK_OBJ_NOTNULL(w->bereq, HTTP_MAGIC);
-		CHECK_OBJ_NOTNULL(w->beresp, HTTP_MAGIC);
-		CHECK_OBJ_NOTNULL(w->resp, HTTP_MAGIC);
-
-		WS_Reset(w->ws, NULL);
-
-		w->sp = VTAILQ_FIRST(&pp->queue);
-		if (w->sp != NULL) {
-			/* Process queued requests, if any */
-			assert(pp->lqueue > 0);
-			VTAILQ_REMOVE(&pp->queue, w->sp, poollist);
-			pp->lqueue--;
-		} else if (!VTAILQ_EMPTY(&pp->socks)) {
-			/* Accept on a socket */
-			ps = VTAILQ_FIRST(&pp->socks);
-			VTAILQ_REMOVE(&pp->socks, ps, list);
-			i = pool_accept(pp, w, ps);
-			Lck_AssertHeld(&pp->mtx);
-			if (i < 0) {
-				/* Socket Shutdown */
-				FREE_OBJ(ps);
-				WS_Release(w->ws, 0);
-				continue;
-			}
-			VTAILQ_INSERT_TAIL(&pp->socks, ps, list);
-		} else if (VTAILQ_EMPTY(&pp->socks)) {
-			/* Nothing to do: To sleep, perchance to dream ... */
-			if (isnan(w->lastused))
-				w->lastused = VTIM_real();
-			VTAILQ_INSERT_HEAD(&pp->idle, w, list);
-			if (!stats_clean)
-				WRK_SumStat(w);
-			(void)Lck_CondWait(&w->cond, &pp->mtx, NULL);
-		}
-
-		/*
-		 * If we got neither session or accepted a socket, we were
-		 * woken up to die to cull the herd.
-		 */
-		if (w->sp == NULL && w->ws->r == NULL)
-			break;
-
-		Lck_Unlock(&pp->mtx);
-
-		if (w->sp == NULL) {
-			/* Turn accepted socket into a session */
-			assert(w->ws->r != NULL);
-			w->sp = SES_New(w, pp->sesspool);
-			if (w->sp == NULL)
-				VCA_FailSess(w);
-			else
-				VCA_SetupSess(w);
-			WS_Release(w->ws, 0);
-		}
-		assert(w->ws->r == NULL);
-
-		if (w->sp != NULL) {
-			CHECK_OBJ_NOTNULL(w->sp, SESS_MAGIC);
-
-			stats_clean = 0;
-			w->lastused = NAN;
-			w->storage_hint = NULL;
-
-			AZ(w->sp->wrk);
-			THR_SetSession(w->sp);
-			w->sp->wrk = w;
-			CNT_Session(w->sp);
-			THR_SetSession(NULL);
-			w->sp = NULL;
-
-			WS_Assert(w->ws);
-			AZ(w->bereq->ws);
-			AZ(w->beresp->ws);
-			AZ(w->resp->ws);
-			AZ(w->wrw.wfd);
-			AZ(w->storage_hint);
-			assert(w->wlp == w->wlb);
-			if (cache_param->diag_bitmap & 0x00040000) {
-				if (w->vcl != NULL)
-					VCL_Rel(&w->vcl);
-			}
-		}
-		stats_clean = WRK_TrySumStat(w);
-		Lck_Lock(&pp->mtx);
-	}
-	Lck_Unlock(&pp->mtx);
-	w->pool = NULL;
-}
-
-/*--------------------------------------------------------------------
- * Queue a workrequest if possible.
- *
- * Return zero if the request was queued, negative if it wasn't.
- */
-
-static int
-pool_queue(struct pool *pp, struct sess *sp)
-{
-	struct worker *w;
-
-	Lck_Lock(&pp->mtx);
-
-	/* If there are idle threads, we tickle the first one into action */
-	w = VTAILQ_FIRST(&pp->idle);
-	if (w != NULL) {
-		VTAILQ_REMOVE(&pp->idle, w, list);
-		Lck_Unlock(&pp->mtx);
-		w->sp = sp;
-		AZ(pthread_cond_signal(&w->cond));
-		return (0);
-	}
-
-	/* If we have too much in the queue already, refuse. */
-	if (pp->lqueue > (cache_param->queue_max * pp->nthr) / 100) {
-		pp->ndropped++;
-		Lck_Unlock(&pp->mtx);
-		return (-1);
-	}
-
-	VTAILQ_INSERT_TAIL(&pp->queue, sp, poollist);
-	pp->nqueued++;
-	pp->lqueue++;
-	Lck_Unlock(&pp->mtx);
-	AZ(pthread_cond_signal(&pp->herder_cond));
-	return (0);
-}
-
-/*--------------------------------------------------------------------*/
-
-int
-Pool_Schedule(struct pool *pp, struct sess *sp)
-{
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	AZ(sp->wrk);
-	if (pool_queue(pp, sp) == 0)
-		return(0);
-
-	VSC_C_main->client_drop_late++;
-
-	/*
-	 * Couldn't queue it -- kill it.
-	 *
-	 * XXX: a notice might be polite, but would potentially
-	 * XXX: sleep whichever thread got us here
-	 */
-	sp->t_end = VTIM_real();
-	if (sp->vcl != NULL) {
-		/*
-		 * A session parked on a busy object can come here
-		 * after it wakes up.  Loose the VCL reference.
-		 */
-		VCL_Rel(&sp->vcl);
-	}
-	return (1);
-}
-
-/*--------------------------------------------------------------------
- * Wait for another request
- */
-
-void
-Pool_Wait(struct sess *sp)
-{
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	AZ(sp->obj);
-	AZ(sp->vcl);
-	assert(sp->fd >= 0);
-	/*
-	 * Set nonblocking in the worker-thread, before passing to the
-	 * acceptor thread, to reduce syscall density of the latter.
-	 */
-	if (VTCP_nonblocking(sp->fd))
-		SES_Close(sp, "remote closed");
-	waiter->pass(waiter_priv, sp);
-}
-
-/*--------------------------------------------------------------------
- * Create another thread, if necessary & possible
- */
-
-static void
-pool_breed(struct pool *qp, const pthread_attr_t *tp_attr)
-{
-	pthread_t tp;
-
-	/*
-	 * If we need more threads, and have space, create
-	 * one more thread.
-	 */
-	if (qp->nthr < cache_param->wthread_min ||	/* Not enough threads yet */
-	    (qp->lqueue > cache_param->wthread_add_threshold && /* more needed */
-	    qp->lqueue > qp->last_lqueue)) {	/* not getting better since last */
-		if (qp->nthr > cache_param->wthread_max) {
-			Lck_Lock(&pool_mtx);
-			VSC_C_main->threads_limited++;
-			Lck_Unlock(&pool_mtx);
-		} else if (pthread_create(&tp, tp_attr, WRK_thread, qp)) {
-			VSL(SLT_Debug, 0, "Create worker thread failed %d %s",
-			    errno, strerror(errno));
-			Lck_Lock(&pool_mtx);
-			VSC_C_main->threads_limited++;
-			Lck_Unlock(&pool_mtx);
-			VTIM_sleep(cache_param->wthread_fail_delay * 1e-3);
-		} else {
-			AZ(pthread_detach(tp));
-			VTIM_sleep(cache_param->wthread_add_delay * 1e-3);
-			qp->nthr++;
-			Lck_Lock(&pool_mtx);
-			VSC_C_main->threads++;
-			VSC_C_main->threads_created++;
-			Lck_Unlock(&pool_mtx);
-		}
-	}
-	qp->last_lqueue = qp->lqueue;
-}
-
-/*--------------------------------------------------------------------
- * Herd a single pool
- *
- * This thread wakes up whenever a pool queues.
- *
- * The trick here is to not be too aggressive about creating threads.
- * We do this by only examining one pool at a time, and by sleeping
- * a short while whenever we create a thread and a little while longer
- * whenever we fail to, hopefully missing a lot of cond_signals in
- * the meantime.
- *
- * XXX: probably need a lot more work.
- *
- */
-
-static void*
-pool_herder(void *priv)
-{
-	struct pool *pp;
-	pthread_attr_t tp_attr;
-	struct timespec ts;
-	double t_idle;
-	struct worker *w;
-	int i;
-
-	CAST_OBJ_NOTNULL(pp, priv, POOL_MAGIC);
-	AZ(pthread_attr_init(&tp_attr));
-
-	while (1) {
-		/* Set the stacksize for worker threads we create */
-		if (cache_param->wthread_stacksize != UINT_MAX)
-			AZ(pthread_attr_setstacksize(&tp_attr,
-			    cache_param->wthread_stacksize));
-		else {
-			AZ(pthread_attr_destroy(&tp_attr));
-			AZ(pthread_attr_init(&tp_attr));
-		}
-
-		pool_breed(pp, &tp_attr);
-
-		if (pp->nthr < cache_param->wthread_min)
-			continue;
-
-		AZ(clock_gettime(CLOCK_MONOTONIC, &ts));
-		ts.tv_sec += cache_param->wthread_purge_delay / 1000;
-		ts.tv_nsec +=
-		    (cache_param->wthread_purge_delay % 1000) * 1000000;
-		if (ts.tv_nsec >= 1000000000) {
-			ts.tv_sec++;
-			ts.tv_nsec -= 1000000000;
-		}
-
-		Lck_Lock(&pp->herder_mtx);
-		i = Lck_CondWait(&pp->herder_cond, &pp->herder_mtx, &ts);
-		Lck_Unlock(&pp->herder_mtx);
-		if (!i)
-			continue;
-
-		if (pp->nthr <= cache_param->wthread_min)
-			continue;
-
-		t_idle = VTIM_real() - cache_param->wthread_timeout;
-
-		Lck_Lock(&pp->mtx);
-		VSC_C_main->sess_queued += pp->nqueued;
-		VSC_C_main->sess_dropped += pp->ndropped;
-		pp->nqueued = pp->ndropped = 0;
-		w = VTAILQ_LAST(&pp->idle, workerhead);
-		if (w != NULL &&
-		    (w->lastused < t_idle || pp->nthr > cache_param->wthread_max)) {
-			VTAILQ_REMOVE(&pp->idle, w, list);
-		} else
-			w = NULL;
-		Lck_Unlock(&pp->mtx);
-
-		/* And give it a kiss on the cheek... */
-		if (w != NULL) {
-			pp->nthr--;
-			Lck_Lock(&pool_mtx);
-			VSC_C_main->threads--;
-			VSC_C_main->threads_destroyed++;
-			Lck_Unlock(&pool_mtx);
-			AZ(w->sp);
-			AZ(pthread_cond_signal(&w->cond));
-		}
-	}
-	NEEDLESS_RETURN(NULL);
-}
-
-/*--------------------------------------------------------------------
- * Add a thread pool
- */
-
-static struct pool *
-pool_mkpool(void)
-{
-	struct pool *pp;
-	struct listen_sock *ls;
-	struct poolsock *ps;
-	pthread_condattr_t cv_attr;
-
-	ALLOC_OBJ(pp, POOL_MAGIC);
-	XXXAN(pp);
-	Lck_New(&pp->mtx, lck_wq);
-
-	VTAILQ_INIT(&pp->queue);
-	VTAILQ_INIT(&pp->idle);
-	VTAILQ_INIT(&pp->socks);
-	pp->sesspool = SES_NewPool(pp);
-	AN(pp->sesspool);
-
-	VTAILQ_FOREACH(ls, &heritage.socks, list) {
-		if (ls->sock < 0)
-			continue;
-		ALLOC_OBJ(ps, POOLSOCK_MAGIC);
-		XXXAN(ps);
-		ps->lsock = ls;
-		VTAILQ_INSERT_TAIL(&pp->socks, ps, list);
-	}
-
-	AZ(pthread_condattr_init(&cv_attr));
-	AZ(pthread_condattr_setclock(&cv_attr, CLOCK_MONOTONIC));
-	AZ(pthread_cond_init(&pp->herder_cond, &cv_attr));
-	AZ(pthread_condattr_destroy(&cv_attr));
-	Lck_New(&pp->herder_mtx, lck_herder);
-	AZ(pthread_create(&pp->herder_thr, NULL, pool_herder, pp));
-
-	return (pp);
-}
-
-/*--------------------------------------------------------------------
- * This thread adjusts the number of pools to match the parameter.
- *
- */
-
-static void *
-pool_poolherder(void *priv)
-{
-	unsigned nwq;
-	VTAILQ_HEAD(,pool)	pools = VTAILQ_HEAD_INITIALIZER(pools);
-	struct pool *pp;
-	uint64_t u;
-
-	THR_SetName("pool_herder");
-	(void)priv;
-
-	nwq = 0;
-	while (1) {
-		if (nwq < cache_param->wthread_pools) {
-			pp = pool_mkpool();
-			if (pp != NULL) {
-				VTAILQ_INSERT_TAIL(&pools, pp, list);
-				VSC_C_main->pools++;
-				nwq++;
-				continue;
-			}
-		}
-		/* XXX: remove pools */
-		if (0)
-			SES_DeletePool(NULL, NULL);
-		(void)sleep(1);
-		u = 0;
-		VTAILQ_FOREACH(pp, &pools, list)
-			u += pp->lqueue;
-		VSC_C_main->thread_queue_len = u;
-	}
-	NEEDLESS_RETURN(NULL);
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-Pool_Init(void)
-{
-
-	waiter_priv = waiter->init();
-	Lck_New(&pool_mtx, lck_wq);
-	AZ(pthread_create(&thr_pool_herder, NULL, pool_poolherder, NULL));
-}
diff --git a/bin/varnishd/cache_response.c b/bin/varnishd/cache_response.c
deleted file mode 100644
index 487a514..0000000
--- a/bin/varnishd/cache_response.c
+++ /dev/null
@@ -1,427 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- */
-
-#include "config.h"
-
-#include "cache.h"
-
-#include "vct.h"
-#include "vtim.h"
-
-/*--------------------------------------------------------------------*/
-
-static void
-res_dorange(const struct sess *sp, const char *r, ssize_t *plow, ssize_t *phigh)
-{
-	ssize_t low, high, has_low;
-
-	assert(sp->obj->response == 200);
-	if (strncmp(r, "bytes=", 6))
-		return;
-	r += 6;
-
-	/* The low end of range */
-	has_low = low = 0;
-	if (!vct_isdigit(*r) && *r != '-')
-		return;
-	while (vct_isdigit(*r)) {
-		has_low = 1;
-		low *= 10;
-		low += *r - '0';
-		r++;
-	}
-
-	if (low >= sp->obj->len)
-		return;
-
-	if (*r != '-')
-		return;
-	r++;
-
-	/* The high end of range */
-	if (vct_isdigit(*r)) {
-		high = 0;
-		while (vct_isdigit(*r)) {
-			high *= 10;
-			high += *r - '0';
-			r++;
-		}
-		if (!has_low) {
-			low = sp->obj->len - high;
-			high = sp->obj->len - 1;
-		}
-	} else
-		high = sp->obj->len - 1;
-	if (*r != '\0')
-		return;
-
-	if (high >= sp->obj->len)
-		high = sp->obj->len - 1;
-
-	if (low > high)
-		return;
-
-	http_PrintfHeader(sp->wrk, sp->vsl_id, sp->wrk->resp,
-	    "Content-Range: bytes %jd-%jd/%jd",
-	    (intmax_t)low, (intmax_t)high, (intmax_t)sp->obj->len);
-	http_Unset(sp->wrk->resp, H_Content_Length);
-	assert(sp->wrk->res_mode & RES_LEN);
-	http_PrintfHeader(sp->wrk, sp->vsl_id, sp->wrk->resp,
-	    "Content-Length: %jd", (intmax_t)(1 + high - low));
-	http_SetResp(sp->wrk->resp, "HTTP/1.1", 206, "Partial Content");
-
-	*plow = low;
-	*phigh = high;
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-RES_BuildHttp(const struct sess *sp)
-{
-	char time_str[30];
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-
-
-	http_ClrHeader(sp->wrk->resp);
-	sp->wrk->resp->logtag = HTTP_Tx;
-	http_CopyResp(sp->wrk->resp, sp->obj->http);
-	http_FilterFields(sp->wrk, sp->vsl_id, sp->wrk->resp, sp->obj->http,
-	    HTTPH_A_DELIVER);
-
-	if (!(sp->wrk->res_mode & RES_LEN)) {
-		http_Unset(sp->wrk->resp, H_Content_Length);
-	} else if (cache_param->http_range_support) {
-		/* We only accept ranges if we know the length */
-		http_SetHeader(sp->wrk, sp->vsl_id, sp->wrk->resp,
-		    "Accept-Ranges: bytes");
-	}
-
-	if (sp->wrk->res_mode & RES_CHUNKED)
-		http_SetHeader(sp->wrk, sp->vsl_id, sp->wrk->resp,
-		    "Transfer-Encoding: chunked");
-
-	VTIM_format(VTIM_real(), time_str);
-	http_PrintfHeader(sp->wrk, sp->vsl_id, sp->wrk->resp, "Date: %s", time_str);
-
-	if (sp->xid != sp->obj->xid)
-		http_PrintfHeader(sp->wrk, sp->vsl_id, sp->wrk->resp,
-		    "X-Varnish: %u %u", sp->xid, sp->obj->xid);
-	else
-		http_PrintfHeader(sp->wrk, sp->vsl_id, sp->wrk->resp,
-		    "X-Varnish: %u", sp->xid);
-	http_PrintfHeader(sp->wrk, sp->vsl_id, sp->wrk->resp, "Age: %.0f",
-	    sp->obj->exp.age + sp->t_resp - sp->obj->exp.entered);
-	http_SetHeader(sp->wrk, sp->vsl_id, sp->wrk->resp, "Via: 1.1 varnish");
-	http_PrintfHeader(sp->wrk, sp->vsl_id, sp->wrk->resp, "Connection: %s",
-	    sp->doclose ? "close" : "keep-alive");
-}
-
-/*--------------------------------------------------------------------
- * We have a gzip'ed object and need to ungzip it for a client which
- * does not understand gzip.
- * XXX: handle invalid gzip data better (how ?)
- */
-
-static void
-res_WriteGunzipObj(const struct sess *sp)
-{
-	struct storage *st;
-	unsigned u = 0;
-	struct vgz *vg;
-	char obuf[cache_param->gzip_stack_buffer];
-	ssize_t obufl = 0;
-	int i;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-
-	vg = VGZ_NewUngzip(sp->wrk, "U D -");
-
-	VGZ_Obuf(vg, obuf, sizeof obuf);
-	VTAILQ_FOREACH(st, &sp->obj->store, list) {
-		CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-		CHECK_OBJ_NOTNULL(st, STORAGE_MAGIC);
-		u += st->len;
-
-		VSC_C_main->n_objwrite++;
-
-		i = VGZ_WrwGunzip(sp->wrk, vg,
-		    st->ptr, st->len,
-		    obuf, sizeof obuf, &obufl);
-		/* XXX: error check */
-		(void)i;
-	}
-	if (obufl) {
-		(void)WRW_Write(sp->wrk, obuf, obufl);
-		(void)WRW_Flush(sp->wrk);
-	}
-	(void)VGZ_Destroy(&vg, sp->vsl_id);
-	assert(u == sp->obj->len);
-}
-
-/*--------------------------------------------------------------------*/
-
-static void
-res_WriteDirObj(const struct sess *sp, ssize_t low, ssize_t high)
-{
-	ssize_t u = 0;
-	size_t ptr, off, len;
-	struct storage *st;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-
-	ptr = 0;
-	VTAILQ_FOREACH(st, &sp->obj->store, list) {
-		CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-		CHECK_OBJ_NOTNULL(st, STORAGE_MAGIC);
-		u += st->len;
-		len = st->len;
-		off = 0;
-		if (ptr + len <= low) {
-			/* This segment is too early */
-			ptr += len;
-			continue;
-		}
-		if (ptr < low) {
-			/* Chop front of segment off */
-			off += (low - ptr);
-			len -= (low - ptr);
-			ptr += (low - ptr);
-		}
-		if (ptr + len > high)
-			/* Chop tail of segment off */
-			len = 1 + high - ptr;
-
-		ptr += len;
-
-		sp->wrk->acct_tmp.bodybytes += len;
-#ifdef SENDFILE_WORKS
-		/*
-		 * XXX: the overhead of setting up sendfile is not
-		 * XXX: epsilon and maybe not even delta, so avoid
-		 * XXX: engaging sendfile for small objects.
-		 * XXX: Should use getpagesize() ?
-		 */
-		if (st->fd >= 0 &&
-		    st->len >= cache_param->sendfile_threshold) {
-			VSC_C_main->n_objsendfile++;
-			WRW_Sendfile(sp->wrk, st->fd, st->where + off, len);
-			continue;
-		}
-#endif /* SENDFILE_WORKS */
-		VSC_C_main->n_objwrite++;
-		(void)WRW_Write(sp->wrk, st->ptr + off, len);
-	}
-	assert(u == sp->obj->len);
-}
-
-/*--------------------------------------------------------------------
- * Deliver an object.
- * Attempt optimizations like 304 and 206 here.
- */
-
-void
-RES_WriteObj(struct sess *sp)
-{
-	char *r;
-	ssize_t low, high;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-
-	WRW_Reserve(sp->wrk, &sp->fd);
-
-	if (sp->obj->response == 200 &&
-	    sp->http->conds &&
-	    RFC2616_Do_Cond(sp)) {
-		sp->wantbody = 0;
-		http_SetResp(sp->wrk->resp, "HTTP/1.1", 304, "Not Modified");
-		http_Unset(sp->wrk->resp, H_Content_Length);
-		http_Unset(sp->wrk->resp, H_Transfer_Encoding);
-	}
-
-	/*
-	 * If nothing special planned, we can attempt Range support
-	 */
-	low = 0;
-	high = sp->obj->len - 1;
-	if (
-	    sp->wantbody &&
-	    (sp->wrk->res_mode & RES_LEN) &&
-	    !(sp->wrk->res_mode & (RES_ESI|RES_ESI_CHILD|RES_GUNZIP)) &&
-	    cache_param->http_range_support &&
-	    sp->obj->response == 200 &&
-	    http_GetHdr(sp->http, H_Range, &r))
-		res_dorange(sp, r, &low, &high);
-
-	/*
-	 * Always remove C-E if client don't grok it
-	 */
-	if (sp->wrk->res_mode & RES_GUNZIP)
-		http_Unset(sp->wrk->resp, H_Content_Encoding);
-
-	/*
-	 * Send HTTP protocol header, unless interior ESI object
-	 */
-	if (!(sp->wrk->res_mode & RES_ESI_CHILD))
-		sp->wrk->acct_tmp.hdrbytes +=
-		    http_Write(sp->wrk, sp->vsl_id, sp->wrk->resp, 1);
-
-	if (!sp->wantbody)
-		sp->wrk->res_mode &= ~RES_CHUNKED;
-
-	if (sp->wrk->res_mode & RES_CHUNKED)
-		WRW_Chunked(sp->wrk);
-
-	if (!sp->wantbody) {
-		/* This was a HEAD or conditional request */
-	} else if (sp->obj->len == 0) {
-		/* Nothing to do here */
-	} else if (sp->wrk->res_mode & RES_ESI) {
-		ESI_Deliver(sp);
-	} else if (sp->wrk->res_mode & RES_ESI_CHILD && sp->wrk->gzip_resp) {
-		ESI_DeliverChild(sp);
-	} else if (sp->wrk->res_mode & RES_ESI_CHILD &&
-	    !sp->wrk->gzip_resp && sp->obj->gziped) {
-		res_WriteGunzipObj(sp);
-	} else if (sp->wrk->res_mode & RES_GUNZIP) {
-		res_WriteGunzipObj(sp);
-	} else {
-		res_WriteDirObj(sp, low, high);
-	}
-
-	if (sp->wrk->res_mode & RES_CHUNKED &&
-	    !(sp->wrk->res_mode & RES_ESI_CHILD))
-		WRW_EndChunk(sp->wrk);
-
-	if (WRW_FlushRelease(sp->wrk) && sp->fd >= 0)
-		SES_Close(sp, "remote closed");
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-RES_StreamStart(struct sess *sp)
-{
-	struct stream_ctx *sctx;
-
-	sctx = sp->wrk->sctx;
-	CHECK_OBJ_NOTNULL(sctx, STREAM_CTX_MAGIC);
-
-	AZ(sp->wrk->res_mode & RES_ESI_CHILD);
-	AN(sp->wantbody);
-
-	WRW_Reserve(sp->wrk, &sp->fd);
-	/*
-	 * Always remove C-E if client don't grok it
-	 */
-	if (sp->wrk->res_mode & RES_GUNZIP)
-		http_Unset(sp->wrk->resp, H_Content_Encoding);
-
-	if (!(sp->wrk->res_mode & RES_CHUNKED) &&
-	    sp->wrk->h_content_length != NULL)
-		http_PrintfHeader(sp->wrk, sp->vsl_id, sp->wrk->resp,
-		    "Content-Length: %s", sp->wrk->h_content_length);
-
-	sp->wrk->acct_tmp.hdrbytes +=
-	    http_Write(sp->wrk, sp->vsl_id, sp->wrk->resp, 1);
-
-	if (sp->wrk->res_mode & RES_CHUNKED)
-		WRW_Chunked(sp->wrk);
-}
-
-void
-RES_StreamPoll(struct worker *w)
-{
-	struct stream_ctx *sctx;
-	struct storage *st;
-	ssize_t l, l2;
-	void *ptr;
-
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	CHECK_OBJ_NOTNULL(w->fetch_obj, OBJECT_MAGIC);
-	sctx = w->sctx;
-	CHECK_OBJ_NOTNULL(sctx, STREAM_CTX_MAGIC);
-	if (w->fetch_obj->len == sctx->stream_next)
-		return;
-	assert(w->fetch_obj->len > sctx->stream_next);
-	l = sctx->stream_front;
-	VTAILQ_FOREACH(st, &w->fetch_obj->store, list) {
-		if (st->len + l <= sctx->stream_next) {
-			l += st->len;
-			continue;
-		}
-		l2 = st->len + l - sctx->stream_next;
-		ptr = st->ptr + (sctx->stream_next - l);
-		if (w->res_mode & RES_GUNZIP) {
-			(void)VGZ_WrwGunzip(w, sctx->vgz, ptr, l2,
-			    sctx->obuf, sctx->obuf_len, &sctx->obuf_ptr);
-		} else {
-			(void)WRW_Write(w, ptr, l2);
-		}
-		l += st->len;
-		sctx->stream_next += l2;
-	}
-	if (!(w->res_mode & RES_GUNZIP))
-		(void)WRW_Flush(w);
-
-	if (w->fetch_obj->objcore == NULL ||
-	    (w->fetch_obj->objcore->flags & OC_F_PASS)) {
-		/*
-		 * This is a pass object, release storage as soon as we
-		 * have delivered it.
-		 */
-		while (1) {
-			st = VTAILQ_FIRST(&w->fetch_obj->store);
-			if (st == NULL ||
-			    sctx->stream_front + st->len > sctx->stream_next)
-				break;
-			VTAILQ_REMOVE(&w->fetch_obj->store, st, list);
-			sctx->stream_front += st->len;
-			STV_free(st);
-		}
-	}
-}
-
-void
-RES_StreamEnd(struct sess *sp)
-{
-	struct stream_ctx *sctx;
-
-	sctx = sp->wrk->sctx;
-	CHECK_OBJ_NOTNULL(sctx, STREAM_CTX_MAGIC);
-
-	if (sp->wrk->res_mode & RES_GUNZIP && sctx->obuf_ptr > 0)
-		(void)WRW_Write(sp->wrk, sctx->obuf, sctx->obuf_ptr);
-	if (sp->wrk->res_mode & RES_CHUNKED &&
-	    !(sp->wrk->res_mode & RES_ESI_CHILD))
-		WRW_EndChunk(sp->wrk);
-	if (WRW_FlushRelease(sp->wrk))
-		SES_Close(sp, "remote closed");
-}
diff --git a/bin/varnishd/cache_rfc2616.c b/bin/varnishd/cache_rfc2616.c
deleted file mode 100644
index 4041f45..0000000
--- a/bin/varnishd/cache_rfc2616.c
+++ /dev/null
@@ -1,336 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- */
-
-#include "config.h"
-
-#include <math.h>
-#include <stdlib.h>
-
-#include "cache.h"
-
-#include "vtim.h"
-
-/*--------------------------------------------------------------------
- * TTL and Age calculation in Varnish
- *
- * RFC2616 has a lot to say about how caches should calculate the TTL
- * and expiry times of objects, but it sort of misses the case that
- * applies to Varnish:  the server-side cache.
- *
- * A normal cache, shared or single-client, has no symbiotic relationship
- * with the server, and therefore must take a very defensive attitude
- * if the Data/Expiry/Age/max-age data does not make sense.  Overall
- * the policy described in section 13 of RFC 2616 results in no caching
- * happening on the first little sign of trouble.
- *
- * Varnish on the other hand tries to offload as many transactions from
- * the backend as possible, and therefore just passing through everything
- * if there is a clock-skew between backend and Varnish is not a workable
- * choice.
- *
- * Varnish implements a policy which is RFC2616 compliant when there
- * is no clockskew, and falls as gracefully as possible otherwise.
- * Our "clockless cache" model is syntehsized from the bits of RFC2616
- * that talks about how a cache should react to a clockless origin server,
- * and more or less uses the inverse logic for the opposite relationship.
- *
- */
-
-void
-RFC2616_Ttl(const struct sess *sp)
-{
-	unsigned max_age, age;
-	double h_date, h_expires;
-	char *p;
-	const struct http *hp;
-
-	hp = sp->wrk->beresp;
-
-	assert(sp->wrk->exp.entered != 0.0 && !isnan(sp->wrk->exp.entered));
-	/* If all else fails, cache using default ttl */
-	sp->wrk->exp.ttl = cache_param->default_ttl;
-
-	max_age = age = 0;
-	h_expires = 0;
-	h_date = 0;
-
-	/*
-	 * Initial cacheability determination per [RFC2616, 13.4]
-	 * We do not support ranges yet, so 206 is out.
-	 */
-
-	if (http_GetHdr(hp, H_Age, &p)) {
-		age = strtoul(p, NULL, 0);
-		sp->wrk->exp.age = age;
-	}
-	if (http_GetHdr(hp, H_Expires, &p))
-		h_expires = VTIM_parse(p);
-
-	if (http_GetHdr(hp, H_Date, &p))
-		h_date = VTIM_parse(p);
-
-	switch (sp->err_code) {
-	default:
-		sp->wrk->exp.ttl = -1.;
-		break;
-	case 200: /* OK */
-	case 203: /* Non-Authoritative Information */
-	case 300: /* Multiple Choices */
-	case 301: /* Moved Permanently */
-	case 302: /* Moved Temporarily */
-	case 307: /* Temporary Redirect */
-	case 410: /* Gone */
-	case 404: /* Not Found */
-		/*
-		 * First find any relative specification from the backend
-		 * These take precedence according to RFC2616, 13.2.4
-		 */
-
-		if ((http_GetHdrField(hp, H_Cache_Control, "s-maxage", &p) ||
-		    http_GetHdrField(hp, H_Cache_Control, "max-age", &p)) &&
-		    p != NULL) {
-
-			if (*p == '-')
-				max_age = 0;
-			else
-				max_age = strtoul(p, NULL, 0);
-
-			if (age > max_age)
-				sp->wrk->exp.ttl = 0;
-			else
-				sp->wrk->exp.ttl = max_age - age;
-			break;
-		}
-
-		/* No expire header, fall back to default */
-		if (h_expires == 0)
-			break;
-
-
-		/* If backend told us it is expired already, don't cache. */
-		if (h_expires < h_date) {
-			sp->wrk->exp.ttl = 0;
-			break;
-		}
-
-		if (h_date == 0 ||
-		    fabs(h_date - sp->wrk->exp.entered) < cache_param->clock_skew) {
-			/*
-			 * If we have no Date: header or if it is
-			 * sufficiently close to our clock we will
-			 * trust Expires: relative to our own clock.
-			 */
-			if (h_expires < sp->wrk->exp.entered)
-				sp->wrk->exp.ttl = 0;
-			else
-				sp->wrk->exp.ttl = h_expires -
-				    sp->wrk->exp.entered;
-			break;
-		} else {
-			/*
-			 * But even if the clocks are out of whack we can still
-			 * derive a relative time from the two headers.
-			 * (the negative ttl case is caught above)
-			 */
-			sp->wrk->exp.ttl = (int)(h_expires - h_date);
-		}
-
-	}
-
-	/* calculated TTL, Our time, Date, Expires, max-age, age */
-	WSP(sp, SLT_TTL,
-	    "%u RFC %.0f %.0f %.0f %.0f %.0f %.0f %.0f %u",
-	    sp->xid, sp->wrk->exp.ttl, -1., -1., sp->wrk->exp.entered,
-	    sp->wrk->exp.age, h_date, h_expires, max_age);
-}
-
-/*--------------------------------------------------------------------
- * Body existence, fetch method and close policy.
- */
-
-enum body_status
-RFC2616_Body(const struct sess *sp)
-{
-	struct http *hp;
-	char *b;
-
-	hp = sp->wrk->beresp;
-
-	if (hp->protover < 11 && !http_HdrIs(hp, H_Connection, "keep-alive"))
-		sp->wrk->do_close = 1;
-	else if (http_HdrIs(hp, H_Connection, "close"))
-		sp->wrk->do_close = 1;
-	else
-		sp->wrk->do_close = 0;
-
-	if (!strcasecmp(http_GetReq(sp->wrk->bereq), "head")) {
-		/*
-		 * A HEAD request can never have a body in the reply,
-		 * no matter what the headers might say.
-		 * [RFC2516 4.3 p33]
-		 */
-		sp->wrk->stats.fetch_head++;
-		return (BS_NONE);
-	}
-
-	if (hp->status <= 199) {
-		/*
-		 * 1xx responses never have a body.
-		 * [RFC2616 4.3 p33]
-		 */
-		sp->wrk->stats.fetch_1xx++;
-		return (BS_NONE);
-	}
-
-	if (hp->status == 204) {
-		/*
-		 * 204 is "No Content", obviously don't expect a body.
-		 * [RFC2616 10.2.5 p60]
-		 */
-		sp->wrk->stats.fetch_204++;
-		return (BS_NONE);
-	}
-
-	if (hp->status == 304) {
-		/*
-		 * 304 is "Not Modified" it has no body.
-		 * [RFC2616 10.3.5 p63]
-		 */
-		sp->wrk->stats.fetch_304++;
-		return (BS_NONE);
-	}
-
-	if (http_HdrIs(hp, H_Transfer_Encoding, "chunked")) {
-		 sp->wrk->stats.fetch_chunked++;
-		return (BS_CHUNKED);
-	}
-
-	if (http_GetHdr(hp, H_Transfer_Encoding, &b)) {
-		sp->wrk->stats.fetch_bad++;
-		return (BS_ERROR);
-	}
-
-	if (http_GetHdr(hp, H_Content_Length, &sp->wrk->h_content_length)) {
-		sp->wrk->stats.fetch_length++;
-		return (BS_LENGTH);
-	}
-
-	if (http_HdrIs(hp, H_Connection, "keep-alive")) {
-		/*
-		 * Keep alive with neither TE=Chunked or C-Len is impossible.
-		 * We assume a zero length body.
-		 */
-		sp->wrk->stats.fetch_zero++;
-		return (BS_ZERO);
-	}
-
-	if (http_HdrIs(hp, H_Connection, "close")) {
-		/*
-		 * In this case, it is safe to just read what comes.
-		 */
-		sp->wrk->stats.fetch_close++;
-		return (BS_EOF);
-	}
-
-	if (hp->protover < 11) {
-		/*
-		 * With no Connection header, assume EOF.
-		 */
-		sp->wrk->stats.fetch_oldhttp++;
-		return (BS_EOF);
-	}
-
-	/*
-	 * Fall back to EOF transfer.
-	 */
-	sp->wrk->stats.fetch_eof++;
-	return (BS_EOF);
-}
-
-/*--------------------------------------------------------------------
- * Find out if the request can receive a gzip'ed response
- */
-
-unsigned
-RFC2616_Req_Gzip(const struct sess *sp)
-{
-
-
-	/*
-	 * "x-gzip" is for http/1.0 backwards compat, final note in 14.3
-	 * p104 says to not do q values for x-gzip, so we just test
-	 * for its existence.
-	 */
-	if (http_GetHdrData(sp->http, H_Accept_Encoding, "x-gzip", NULL))
-		return (1);
-
-	/*
-	 * "gzip" is the real thing, but the 'q' value must be nonzero.
-	 * We do not care a hoot if the client prefers some other
-	 * compression more than gzip: Varnish only does gzip.
-	 */
-	if (http_GetHdrQ(sp->http, H_Accept_Encoding, "gzip") > 0.)
-		return (1);
-
-	/* Bad client, no gzip. */
-	return (0);
-}
-
-/*--------------------------------------------------------------------*/
-
-int
-RFC2616_Do_Cond(const struct sess *sp)
-{
-	char *p, *e;
-	double ims;
-	int do_cond = 0;
-
-	/* RFC 2616 13.3.4 states we need to match both ETag
-	   and If-Modified-Since if present*/
-
-	if (http_GetHdr(sp->http, H_If_Modified_Since, &p) ) {
-		if (!sp->obj->last_modified)
-			return (0);
-		ims = VTIM_parse(p);
-		if (ims > sp->t_req)	/* [RFC2616 14.25] */
-			return (0);
-		if (sp->obj->last_modified > ims)
-			return (0);
-		do_cond = 1;
-	}
-
-	if (http_GetHdr(sp->http, H_If_None_Match, &p) &&
-	    http_GetHdr(sp->obj->http, H_ETag, &e)) {
-		if (strcmp(p,e) != 0)
-			return (0);
-		do_cond = 1;
-	}
-
-	return (do_cond);
-}
diff --git a/bin/varnishd/cache_session.c b/bin/varnishd/cache_session.c
deleted file mode 100644
index 7befbcc..0000000
--- a/bin/varnishd/cache_session.c
+++ /dev/null
@@ -1,419 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2010 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * Session management
- *
- * This is a little bit of a mixed back, containing both memory management
- * and various state-change functions.
- *
- */
-
-#include "config.h"
-
-#include <math.h>
-#include <stdlib.h>
-
-#include "cache.h"
-
-#include "waiter/cache_waiter.h"
-
-/*--------------------------------------------------------------------*/
-
-struct sessmem {
-	unsigned		magic;
-#define SESSMEM_MAGIC		0x555859c5
-
-	struct sesspool		*pool;
-
-	unsigned		workspace;
-	uint16_t		nhttp;
-	void			*wsp;
-	struct http		*http[2];
-	VTAILQ_ENTRY(sessmem)	list;
-
-	struct sess		sess;
-};
-
-struct sesspool {
-	unsigned		magic;
-#define SESSPOOL_MAGIC		0xd916e202
-	struct pool		*pool;
-	VTAILQ_HEAD(,sessmem)	freelist;
-	struct lock		mtx;
-	unsigned		nsess;
-	unsigned		dly_free_cnt;
-};
-
-/*--------------------------------------------------------------------
- * Charge statistics from worker to request and session.
- */
-
-void
-SES_Charge(struct sess *sp)
-{
-	struct acct *a = &sp->wrk->acct_tmp;
-
-	sp->req_bodybytes += a->bodybytes;
-
-#define ACCT(foo)				\
-	sp->wrk->stats.s_##foo += a->foo;	\
-	sp->acct_ses.foo += a->foo;		\
-	a->foo = 0;
-#include "tbl/acct_fields.h"
-#undef ACCT
-}
-
-/*--------------------------------------------------------------------
- * This function allocates a session + assorted peripheral data
- * structures in one single malloc operation.
- */
-
-static struct sessmem *
-ses_sm_alloc(void)
-{
-	struct sessmem *sm;
-	unsigned char *p, *q;
-	unsigned nws;
-	uint16_t nhttp;
-	unsigned l, hl;
-
-	/*
-	 * It is not necessary to lock these, but we need to
-	 * cache them locally, to make sure we get a consistent
-	 * view of the value.
-	 */
-	nws = cache_param->sess_workspace;
-	nhttp = (uint16_t)cache_param->http_max_hdr;
-
-	hl = HTTP_estimate(nhttp);
-	l = sizeof *sm + nws + 2 * hl;
-	VSC_C_main->sessmem_size = l;
-	p = malloc(l);
-	if (p == NULL)
-		return (NULL);
-	q = p + l;
-
-	/* Don't waste time zeroing the workspace */
-	memset(p, 0, l - nws);
-
-	sm = (void*)p;
-	p += sizeof *sm;
-
-	sm->magic = SESSMEM_MAGIC;
-	sm->workspace = nws;
-	sm->nhttp = nhttp;
-
-	sm->http[0] = HTTP_create(p, nhttp);
-	p += hl;
-
-	sm->http[1] = HTTP_create(p, nhttp);
-	p += hl;
-
-	sm->wsp = p;
-	p += nws;
-
-	assert(p == q);
-
-	return (sm);
-}
-
-/*--------------------------------------------------------------------
- * This prepares a session for use, based on its sessmem structure.
- */
-
-static void
-ses_setup(struct sessmem *sm)
-{
-	struct sess *sp;
-
-	CHECK_OBJ_NOTNULL(sm, SESSMEM_MAGIC);
-	sp = &sm->sess;
-	memset(sp, 0, sizeof *sp);
-
-	/* We assume that the sess has been zeroed by the time we get here */
-	AZ(sp->magic);
-
-	sp->magic = SESS_MAGIC;
-	sp->mem = sm;
-	sp->sockaddrlen = sizeof(sp->sockaddr);
-	sp->mysockaddrlen = sizeof(sp->mysockaddr);
-	sp->sockaddr.ss_family = sp->mysockaddr.ss_family = PF_UNSPEC;
-	sp->t_open = NAN;
-	sp->t_req = NAN;
-	sp->t_resp = NAN;
-	sp->t_end = NAN;
-	EXP_Clr(&sp->exp);
-
-	WS_Init(sp->ws, "sess", sm->wsp, sm->workspace);
-	sp->http = sm->http[0];
-	sp->http0 = sm->http[1];
-}
-
-/*--------------------------------------------------------------------
- * Get a new session, preferably by recycling an already ready one
- */
-
-struct sess *
-SES_New(struct worker *wrk, struct sesspool *pp)
-{
-	struct sessmem *sm;
-	struct sess *sp;
-	int do_alloc;
-
-	CHECK_OBJ_NOTNULL(pp, SESSPOOL_MAGIC);
-
-	do_alloc = 0;
-	Lck_Lock(&pp->mtx);
-	sm = VTAILQ_FIRST(&pp->freelist);
-	if (sm != NULL) {
-		VTAILQ_REMOVE(&pp->freelist, sm, list);
-	} else if (pp->nsess < cache_param->max_sess) {
-		pp->nsess++;
-		do_alloc = 1;
-	}
-	wrk->stats.sessmem_free += pp->dly_free_cnt;
-	pp->dly_free_cnt = 0;
-	Lck_Unlock(&pp->mtx);
-	if (do_alloc) {
-		sm = ses_sm_alloc();
-		if (sm != NULL) {
-			wrk->stats.sessmem_alloc++;
-			sm->pool = pp;
-			ses_setup(sm);
-		} else {
-			wrk->stats.sessmem_fail++;
-		}
-	} else if (sm == NULL) {
-		wrk->stats.sessmem_limit++;
-	}
-	if (sm == NULL)
-		return (NULL);
-	sp = &sm->sess;
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	return (sp);
-}
-
-/*--------------------------------------------------------------------
- * Allocate a session for use by background threads.
- */
-
-struct sess *
-SES_Alloc(void)
-{
-	struct sess *sp;
-	struct sessmem *sm;
-
-	sm = ses_sm_alloc();
-	AN(sm);
-	ses_setup(sm);
-	sp = &sm->sess;
-	sp->sockaddrlen = 0;
-	return (sp);
-}
-
-/*--------------------------------------------------------------------
- * Schedule a session back on a work-thread from its pool
- */
-
-int
-SES_Schedule(struct sess *sp)
-{
-	struct sessmem *sm;
-	struct sesspool *pp;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-
-	AZ(sp->wrk);
-
-	sm = sp->mem;
-	CHECK_OBJ_NOTNULL(sm, SESSMEM_MAGIC);
-
-	pp = sm->pool;
-	CHECK_OBJ_NOTNULL(pp, SESSPOOL_MAGIC);
-
-	AN(pp->pool);
-
-	if (Pool_Schedule(pp->pool, sp)) {
-		SES_Delete(sp, "dropped");
-		return (1);
-	}
-	return (0);
-}
-
-/*--------------------------------------------------------------------
- * Handle a session (from waiter)
- *
- * Status: see HTC_Rx()
- */
-
-void
-SES_Handle(struct sess *sp, int status)
-{
-
-	switch (status) {
-	case -2:
-		SES_Delete(sp, "blast");
-		break;
-	case -1:
-		SES_Delete(sp, "no request");
-		break;
-	case 1:
-		sp->step = STP_START;
-		(void)SES_Schedule(sp);
-		break;
-	default:
-		WRONG("Unexpected return from HTC_Rx()");
-	}
-}
-
-/*--------------------------------------------------------------------
- * Close a sessions connection.
- */
-
-void
-SES_Close(struct sess *sp, const char *reason)
-{
-	int i;
-
-	assert(sp->fd >= 0);
-	VSL(SLT_SessionClose, sp->vsl_id, "%s", reason);
-	i = close(sp->fd);
-	assert(i == 0 || errno != EBADF); /* XXX EINVAL seen */
-	sp->fd = -1;
-}
-
-/*--------------------------------------------------------------------
- * (Close &) Free or Recycle a session.
- *
- * If the workspace has changed, deleted it, otherwise wash it, and put
- * it up for adoption.
- *
- * XXX: We should also check nhttp
- */
-
-void
-SES_Delete(struct sess *sp, const char *reason)
-{
-	struct acct *b;
-	struct sessmem *sm;
-	static char noaddr[] = "-";
-	struct worker *wrk;
-	struct sesspool *pp;
-
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	sm = sp->mem;
-	CHECK_OBJ_NOTNULL(sm, SESSMEM_MAGIC);
-	pp = sm->pool;
-	CHECK_OBJ_NOTNULL(pp, SESSPOOL_MAGIC);
-	wrk = sp->wrk;
-	CHECK_OBJ_ORNULL(wrk, WORKER_MAGIC);
-
-
-	if (reason != NULL)
-		SES_Close(sp, reason);
-	assert(sp->fd < 0);
-
-	AZ(sp->obj);
-	AZ(sp->vcl);
-	if (sp->addr == NULL)
-		sp->addr = noaddr;
-	if (sp->port == NULL)
-		sp->port = noaddr;
-
-	b = &sp->acct_ses;
-	assert(!isnan(b->first));
-	assert(!isnan(sp->t_end));
-
-	VSL(SLT_StatSess, sp->vsl_id, "%s %s %.0f %ju %ju %ju %ju %ju %ju %ju",
-	    sp->addr, sp->port, sp->t_end - b->first,
-	    b->sess, b->req, b->pipe, b->pass,
-	    b->fetch, b->hdrbytes, b->bodybytes);
-
-	if (sm->workspace != cache_param->sess_workspace ||
-	    sm->nhttp != (uint16_t)cache_param->http_max_hdr ||
-	    pp->nsess > cache_param->max_sess) {
-		free(sm);
-		Lck_Lock(&pp->mtx);
-		if (wrk != NULL)
-			wrk->stats.sessmem_free++;
-		else
-			pp->dly_free_cnt++;
-		pp->nsess--;
-		Lck_Unlock(&pp->mtx);
-	} else {
-		/* Clean and prepare for reuse */
-		ses_setup(sm);
-		Lck_Lock(&pp->mtx);
-		if (wrk != NULL) {
-			wrk->stats.sessmem_free += pp->dly_free_cnt;
-			pp->dly_free_cnt = 0;
-		}
-		VTAILQ_INSERT_HEAD(&pp->freelist, sm, list);
-		Lck_Unlock(&pp->mtx);
-	}
-}
-
-/*--------------------------------------------------------------------
- * Create and delete pools
- */
-
-struct sesspool *
-SES_NewPool(struct pool *pp)
-{
-	struct sesspool *sp;
-
-	ALLOC_OBJ(sp, SESSPOOL_MAGIC);
-	AN(sp);
-	sp->pool = pp;
-	VTAILQ_INIT(&sp->freelist);
-	Lck_New(&sp->mtx, lck_sessmem);
-	return (sp);
-}
-
-void
-SES_DeletePool(struct sesspool *sp, struct worker *wrk)
-{
-	struct sessmem *sm;
-
-	CHECK_OBJ_NOTNULL(sp, SESSPOOL_MAGIC);
-	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
-	Lck_Lock(&sp->mtx);
-	while (!VTAILQ_EMPTY(&sp->freelist)) {
-		sm = VTAILQ_FIRST(&sp->freelist);
-		CHECK_OBJ_NOTNULL(sm, SESSMEM_MAGIC);
-		VTAILQ_REMOVE(&sp->freelist, sm, list);
-		FREE_OBJ(sm);
-		wrk->stats.sessmem_free++;
-		sp->nsess--;
-	}
-	AZ(sp->nsess);
-	Lck_Unlock(&sp->mtx);
-	Lck_Delete(&sp->mtx);
-	FREE_OBJ(sp);
-}
diff --git a/bin/varnishd/cache_shmlog.c b/bin/varnishd/cache_shmlog.c
deleted file mode 100644
index 1252fa3..0000000
--- a/bin/varnishd/cache_shmlog.c
+++ /dev/null
@@ -1,346 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- */
-
-#include "config.h"
-
-#include <stdio.h>
-
-#include "cache.h"
-
-#include "cache_backend.h"	// For w->vbc
-
-#include "vapi/vsm_int.h"
-#include "vmb.h"
-#include "vtim.h"
-
-/* These cannot be struct lock, which depends on vsm/vsl working */
-static pthread_mutex_t vsl_mtx;
-static pthread_mutex_t vsm_mtx;
-
-static uint32_t			*vsl_start;
-static const uint32_t		*vsl_end;
-static uint32_t			*vsl_ptr;
-
-static inline uint32_t
-vsl_w0(uint32_t type, uint32_t length)
-{
-
-	assert(length < 0x10000);
-        return (((type & 0xff) << 24) | length);
-}
-
-/*--------------------------------------------------------------------*/
-
-static inline void
-vsl_hdr(enum VSL_tag_e tag, uint32_t *p, unsigned len, unsigned id)
-{
-
-	assert(((uintptr_t)p & 0x3) == 0);
-
-	p[1] = id;
-	VMB();
-	p[0] = vsl_w0(tag, len);
-}
-
-/*--------------------------------------------------------------------*/
-
-static void
-vsl_wrap(void)
-{
-
-	assert(vsl_ptr >= vsl_start + 1);
-	assert(vsl_ptr < vsl_end);
-	vsl_start[1] = VSL_ENDMARKER;
-	do
-		vsl_start[0]++;
-	while (vsl_start[0] == 0);
-	VWMB();
-	if (vsl_ptr != vsl_start + 1) {
-		*vsl_ptr = VSL_WRAPMARKER;
-		vsl_ptr = vsl_start + 1;
-	}
-	VSC_C_main->shm_cycles++;
-}
-
-/*--------------------------------------------------------------------
- * Reserve bytes for a record, wrap if necessary
- */
-
-static uint32_t *
-vsl_get(unsigned len, unsigned records, unsigned flushes)
-{
-	uint32_t *p;
-
-	if (pthread_mutex_trylock(&vsl_mtx)) {
-		AZ(pthread_mutex_lock(&vsl_mtx));
-		VSC_C_main->shm_cont++;
-	}
-	assert(vsl_ptr < vsl_end);
-	assert(((uintptr_t)vsl_ptr & 0x3) == 0);
-
-	VSC_C_main->shm_writes++;
-	VSC_C_main->shm_flushes += flushes;
-	VSC_C_main->shm_records += records;
-
-	/* Wrap if necessary */
-	if (VSL_END(vsl_ptr, len) >= vsl_end)
-		vsl_wrap();
-
-	p = vsl_ptr;
-	vsl_ptr = VSL_END(vsl_ptr, len);
-
-	*vsl_ptr = VSL_ENDMARKER;
-
-	assert(vsl_ptr < vsl_end);
-	assert(((uintptr_t)vsl_ptr & 0x3) == 0);
-	AZ(pthread_mutex_unlock(&vsl_mtx));
-
-	return (p);
-}
-
-/*--------------------------------------------------------------------
- * This variant copies a byte-range directly to the log, without
- * taking the detour over sprintf()
- */
-
-static void
-VSLR(enum VSL_tag_e tag, int id, const char *b, unsigned len)
-{
-	uint32_t *p;
-	unsigned mlen;
-
-	mlen = cache_param->shm_reclen;
-
-	/* Truncate */
-	if (len > mlen)
-		len = mlen;
-
-	p = vsl_get(len, 1, 0);
-
-	memcpy(p + 2, b, len);
-	vsl_hdr(tag, p, len, id);
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-VSL(enum VSL_tag_e tag, int id, const char *fmt, ...)
-{
-	va_list ap;
-	unsigned n, mlen = cache_param->shm_reclen;
-	char buf[mlen];
-
-	/*
-	 * XXX: consider formatting into a stack buffer then move into
-	 * XXX: shmlog with VSLR().
-	 */
-	AN(fmt);
-	va_start(ap, fmt);
-
-	if (strchr(fmt, '%') == NULL) {
-		VSLR(tag, id, fmt, strlen(fmt));
-	} else {
-		n = vsnprintf(buf, mlen, fmt, ap);
-		if (n > mlen)
-			n = mlen;
-		VSLR(tag, id, buf, n);
-	}
-	va_end(ap);
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-WSL_Flush(struct worker *w, int overflow)
-{
-	uint32_t *p;
-	unsigned l;
-
-	l = pdiff(w->wlb, w->wlp);
-	if (l == 0)
-		return;
-
-	assert(l >= 8);
-
-	p = vsl_get(l - 8, w->wlr, overflow);
-
-	memcpy(p + 1, w->wlb + 1, l - 4);
-	VWMB();
-	p[0] = w->wlb[0];
-	w->wlp = w->wlb;
-	w->wlr = 0;
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-WSLR(struct worker *w, enum VSL_tag_e tag, int id, txt t)
-{
-	unsigned l, mlen;
-
-	Tcheck(t);
-	mlen = cache_param->shm_reclen;
-
-	/* Truncate */
-	l = Tlen(t);
-	if (l > mlen) {
-		l = mlen;
-		t.e = t.b + l;
-	}
-
-	assert(w->wlp < w->wle);
-
-	/* Wrap if necessary */
-	if (VSL_END(w->wlp, l) >= w->wle)
-		WSL_Flush(w, 1);
-	assert (VSL_END(w->wlp, l) < w->wle);
-	memcpy(VSL_DATA(w->wlp), t.b, l);
-	vsl_hdr(tag, w->wlp, l, id);
-	w->wlp = VSL_END(w->wlp, l);
-	assert(w->wlp < w->wle);
-	w->wlr++;
-	if (cache_param->diag_bitmap & 0x10000)
-		WSL_Flush(w, 0);
-}
-
-/*--------------------------------------------------------------------*/
-
-static void
-wsl(struct worker *w, enum VSL_tag_e tag, int id, const char *fmt, va_list ap)
-{
-	char *p;
-	unsigned n, mlen;
-	txt t;
-
-	AN(fmt);
-	mlen = cache_param->shm_reclen;
-
-	if (strchr(fmt, '%') == NULL) {
-		t.b = TRUST_ME(fmt);
-		t.e = strchr(t.b, '\0');
-		WSLR(w, tag, id, t);
-	} else {
-		assert(w->wlp < w->wle);
-
-		/* Wrap if we cannot fit a full size record */
-		if (VSL_END(w->wlp, mlen) >= w->wle)
-			WSL_Flush(w, 1);
-
-		p = VSL_DATA(w->wlp);
-		n = vsnprintf(p, mlen, fmt, ap);
-		if (n > mlen)
-			n = mlen;	/* we truncate long fields */
-		vsl_hdr(tag, w->wlp, n, id);
-		w->wlp = VSL_END(w->wlp, n);
-		assert(w->wlp < w->wle);
-		w->wlr++;
-	}
-	if (cache_param->diag_bitmap & 0x10000)
-		WSL_Flush(w, 0);
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-WSL(struct worker *w, enum VSL_tag_e tag, int id, const char *fmt, ...)
-{
-	va_list ap;
-
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	AN(fmt);
-	va_start(ap, fmt);
-	wsl(w, tag, id, fmt, ap);
-	va_end(ap);
-}
-
-
-/*--------------------------------------------------------------------*/
-
-void
-WSLB(struct worker *w, enum VSL_tag_e tag, const char *fmt, ...)
-{
-	va_list ap;
-
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	CHECK_OBJ_NOTNULL(w->vbc, VBC_MAGIC);
-	AN(fmt);
-	va_start(ap, fmt);
-	wsl(w, tag, w->vbc->vsl_id, fmt, ap);
-	va_end(ap);
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-VSL_Init(void)
-{
-	struct VSM_chunk *vsc;
-
-	AZ(pthread_mutex_init(&vsl_mtx, NULL));
-	AZ(pthread_mutex_init(&vsm_mtx, NULL));
-
-	VSM__Clean();
-
-	VSM_ITER(vsc)
-		if (!strcmp(vsc->class, VSL_CLASS))
-			break;
-	AN(vsc);
-	vsl_start = VSM_PTR(vsc);
-	vsl_end = VSM_NEXT(vsc);
-	vsl_ptr = vsl_start + 1;
-
-	vsl_wrap();
-	VSM_head->starttime = (intmax_t)VTIM_real();
-	memset(VSM_head->panicstr, '\0', sizeof *VSM_head->panicstr);
-	memset(VSC_C_main, 0, sizeof *VSC_C_main);
-	VSM_head->child_pid = getpid();
-}
-
-/*--------------------------------------------------------------------*/
-
-void *
-VSM_Alloc(unsigned size, const char *class, const char *type,
-    const char *ident)
-{
-	void *p;
-
-	AZ(pthread_mutex_lock(&vsm_mtx));
-	p = VSM__Alloc(size, class, type, ident);
-	AZ(pthread_mutex_unlock(&vsm_mtx));
-	return (p);
-}
-
-void
-VSM_Free(const void *ptr)
-{
-
-	AZ(pthread_mutex_lock(&vsm_mtx));
-	VSM__Free(ptr);
-	AZ(pthread_mutex_unlock(&vsm_mtx));
-}
diff --git a/bin/varnishd/cache_vary.c b/bin/varnishd/cache_vary.c
deleted file mode 100644
index 026f937..0000000
--- a/bin/varnishd/cache_vary.c
+++ /dev/null
@@ -1,257 +0,0 @@
-/*-
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * Do Vary processing.
- *
- * When we insert an object into the cache which has a Vary: header,
- * we encode a vary matching string containing the headers mentioned
- * and their value.
- *
- * When we match an object in the cache, we check the present request
- * against the vary matching string.
- *
- * The only kind of header-munging we do is leading & trailing space
- * removal.  All the potential "q=foo" gymnastics is not worth the
- * effort.
- *
- * The vary matching string has the following format:
- *
- * Sequence of: {
- *	<msb>			\   Length of header contents.
- *	<lsb>			/
- *	<length of header + 1>	\
- *	<header>		 \  Same format as argument to http_GetHdr()
- *	':'			 /
- *	'\0'			/
- *      <header>		>   Only present if length != 0xffff
- * }
- *      '\0'
- */
-
-#include "config.h"
-
-#include "cache.h"
-
-#include "vct.h"
-#include "vend.h"
-
-struct vsb *
-VRY_Create(const struct sess *sp, const struct http *hp)
-{
-	char *v, *p, *q, *h, *e;
-	struct vsb *sb, *sbh;
-	int l;
-
-	/* No Vary: header, no worries */
-	if (!http_GetHdr(hp, H_Vary, &v))
-		return (NULL);
-
-	/* For vary matching string */
-	sb = VSB_new_auto();
-	AN(sb);
-
-	/* For header matching strings */
-	sbh = VSB_new_auto();
-	AN(sbh);
-
-	if (*v == ':') {
-		WSP(sp, SLT_Error, "Vary header had extra ':', fix backend");
-		v++;
-	}
-	for (p = v; *p; p++) {
-
-		/* Find next header-name */
-		if (vct_issp(*p))
-			continue;
-		for (q = p; *q && !vct_issp(*q) && *q != ','; q++)
-			continue;
-
-		/* Build a header-matching string out of it */
-		VSB_clear(sbh);
-		VSB_printf(sbh, "%c%.*s:%c",
-		    (char)(1 + (q - p)), (int)(q - p), p, 0);
-		AZ(VSB_finish(sbh));
-
-		if (http_GetHdr(sp->http, VSB_data(sbh), &h)) {
-			AZ(vct_issp(*h));
-			/* Trim trailing space */
-			e = strchr(h, '\0');
-			while (e > h && vct_issp(e[-1]))
-				e--;
-			/* Encode two byte length and contents */
-			l = e - h;
-			assert(!(l & ~0xffff));
-		} else {
-			e = h;
-			l = 0xffff;
-		}
-		VSB_printf(sb, "%c%c", (unsigned)l >> 8, l & 0xff);
-		/* Append to vary matching string */
-		VSB_bcat(sb, VSB_data(sbh), VSB_len(sbh));
-		if (e != h)
-			VSB_bcat(sb, h, e - h);
-
-		while (vct_issp(*q))
-			q++;
-		if (*q == '\0')
-			break;
-		xxxassert(*q == ',');
-		p = q;
-	}
-	/* Terminate vary matching string */
-	VSB_printf(sb, "%c%c%c", 0xff, 0xff, 0);
-
-	VSB_delete(sbh);
-	AZ(VSB_finish(sb));
-	return(sb);
-}
-
-/*
- * Find length of a vary entry
- */
-static unsigned
-vry_len(const uint8_t *p)
-{
-	unsigned l = vbe16dec(p);
-
-	return (2 + p[2] + 2 + (l == 0xffff ? 0 : l));
-}
-
-/*
- * Compare two vary entries
- */
-static int
-vry_cmp(const uint8_t * const *v1, uint8_t * const *v2)
-{
-	unsigned retval = 0;
-
-	if (!memcmp(*v1, *v2, vry_len(*v1))) {
-		/* Same same */
-		retval = 0;
-	} else if (memcmp((*v1) + 2, (*v2) + 2, (*v1)[2] + 2)) {
-		/* Different header */
-		retval = 1;
-	} else if (cache_param->http_gzip_support &&
-	    !strcasecmp(H_Accept_Encoding, (const char*)((*v1)+2))) {
-		/*
-		 * If we do gzip processing, we do not vary on Accept-Encoding,
-		 * because we want everybody to get the gzip'ed object, and
-		 * varnish will gunzip as necessary.  We implement the skip at
-		 * check time, rather than create time, so that object in
-		 * persistent storage can be used with either setting of
-		 * http_gzip_support.
-		 */
-		retval = 0;
-	} else {
-		/* Same header, different content */
-		retval = 2;
-	}
-	return (retval);
-}
-
-int
-VRY_Match(struct sess *sp, const uint8_t *vary)
-{
-	uint8_t *vsp = sp->vary_b;
-	char *h, *e;
-	unsigned lh, ln;
-	int i, retval = 1, oflo = 0;
-
-	AN(vsp);
-	while (vary[2]) {
-		i = vry_cmp(&vary, &vsp);
-		if (i == 1) {
-			/* Build a new entry */
-
-			i = http_GetHdr(sp->http, (const char*)(vary+2), &h);
-			if (i) {
-				/* Trim trailing space */
-				e = strchr(h, '\0');
-				while (e > h && vct_issp(e[-1]))
-					e--;
-				lh = e - h;
-				assert(lh < 0xffff);
-			} else {
-				e = h = NULL;
-				lh = 0xffff;
-			}
-
-			/* Length of the entire new vary entry */
-			ln = 2 + vary[2] + 2 + (lh == 0xffff ? 0 : lh);
-			if (vsp + ln >= sp->vary_e) {
-				vsp = sp->vary_b;
-				oflo = 1;
-			}
-
-			/*
-			 * We MUST have space for one entry and the end marker
-			 * after it, which prevents old junk from confusing us
-			 */
-			assert(vsp + ln + 2 < sp->vary_e);
-
-			vbe16enc(vsp, (uint16_t)lh);
-			memcpy(vsp + 2, vary + 2, vary[2] + 2);
-			if (h != NULL && e != NULL) {
-				memcpy(vsp + 2 + vsp[2] + 2, h, e - h);
-				vsp[2 + vary[2] + 2 + (e - h) + 2] = '\0';
-			} else
-				vsp[2 + vary[2] + 2 + 2] = '\0';
-
-			i = vry_cmp(&vary, &vsp);
-			assert(i != 1);	/* hdr must be the same now */
-		}
-		if (i != 0)
-			retval = 0;
-		vsp += vry_len(vsp);
-		vary += vry_len(vary);
-	}
-	if (vsp + 3 > sp->vary_e)
-		oflo = 1;
-
-	if (oflo) {
-		/* XXX: Should log this */
-		vsp = sp->vary_b;
-	}
-	vsp[0] = 0xff;
-	vsp[1] = 0xff;
-	vsp[2] = 0;
-	if (oflo)
-		sp->vary_l = NULL;
-	else
-		sp->vary_l = vsp + 3;
-	return (retval);
-}
-
-void
-VRY_Validate(const uint8_t *vary)
-{
-
-	while (vary[2] != 0) {
-		assert(strlen((const char*)vary+3) == vary[2]);
-		vary += vry_len(vary);
-	}
-}
diff --git a/bin/varnishd/cache_vcl.c b/bin/varnishd/cache_vcl.c
deleted file mode 100644
index 068b482..0000000
--- a/bin/varnishd/cache_vcl.c
+++ /dev/null
@@ -1,365 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * Interface *to* compiled VCL code:  Loading, unloading, calling into etc.
- *
- * The interface *from* the compiled VCL code is in cache_vrt.c.
- */
-
-#include "config.h"
-
-#include <dlfcn.h>
-#include <stdlib.h>
-
-#include "cache.h"
-
-#include "vcl.h"
-#include "vcli.h"
-#include "vcli_priv.h"
-
-struct vcls {
-	unsigned		magic;
-#define VVCLS_MAGIC		0x214188f2
-	VTAILQ_ENTRY(vcls)	list;
-	char			*name;
-	void			*dlh;
-	struct VCL_conf		conf[1];
-};
-
-/*
- * XXX: Presently all modifications to this list happen from the
- * CLI event-engine, so no locking is necessary
- */
-static VTAILQ_HEAD(, vcls)	vcl_head =
-    VTAILQ_HEAD_INITIALIZER(vcl_head);
-
-
-static struct lock		vcl_mtx;
-static struct vcls		*vcl_active; /* protected by vcl_mtx */
-
-/*--------------------------------------------------------------------*/
-
-const char *
-VCL_Return_Name(unsigned method)
-{
-
-	switch (method) {
-#define VCL_RET_MAC(l, U, B) case VCL_RET_##U: return(#l);
-#include "tbl/vcl_returns.h"
-#undef VCL_RET_MAC
-	default:
-		return (NULL);
-	}
-}
-
-/*--------------------------------------------------------------------*/
-
-static void
-VCL_Get(struct VCL_conf **vcc)
-{
-	static int once = 0;
-
-	while (!once && vcl_active == NULL) {
-		(void)sleep(1);
-	}
-	once = 1;
-
-	Lck_Lock(&vcl_mtx);
-	AN(vcl_active);
-	*vcc = vcl_active->conf;
-	AN(*vcc);
-	AZ((*vcc)->discard);
-	(*vcc)->busy++;
-	Lck_Unlock(&vcl_mtx);
-}
-
-void
-VCL_Refresh(struct VCL_conf **vcc)
-{
-	if (*vcc == vcl_active->conf)
-		return;
-	if (*vcc != NULL)
-		VCL_Rel(vcc);	/* XXX: optimize locking */
-	VCL_Get(vcc);
-}
-
-void
-VCL_Rel(struct VCL_conf **vcc)
-{
-	struct VCL_conf *vc;
-
-	AN(*vcc);
-	vc = *vcc;
-	*vcc = NULL;
-
-	Lck_Lock(&vcl_mtx);
-	assert(vc->busy > 0);
-	vc->busy--;
-	/*
-	 * We do not garbage collect discarded VCL's here, that happens
-	 * in VCL_Poll() which is called from the CLI thread.
-	 */
-	Lck_Unlock(&vcl_mtx);
-}
-
-/*--------------------------------------------------------------------*/
-
-static struct vcls *
-vcl_find(const char *name)
-{
-	struct vcls *vcl;
-
-	ASSERT_CLI();
-	VTAILQ_FOREACH(vcl, &vcl_head, list) {
-		if (vcl->conf->discard)
-			continue;
-		if (!strcmp(vcl->name, name))
-			return (vcl);
-	}
-	return (NULL);
-}
-
-static int
-VCL_Load(const char *fn, const char *name, struct cli *cli)
-{
-	struct vcls *vcl;
-	struct VCL_conf const *cnf;
-
-	ASSERT_CLI();
-	vcl = vcl_find(name);
-	if (vcl != NULL) {
-		VCLI_Out(cli, "Config '%s' already loaded", name);
-		return (1);
-	}
-
-	ALLOC_OBJ(vcl, VVCLS_MAGIC);
-	XXXAN(vcl);
-
-	vcl->dlh = dlopen(fn, RTLD_NOW | RTLD_LOCAL);
-
-	if (vcl->dlh == NULL) {
-		VCLI_Out(cli, "dlopen(%s): %s\n", fn, dlerror());
-		FREE_OBJ(vcl);
-		return (1);
-	}
-	cnf = dlsym(vcl->dlh, "VCL_conf");
-	if (cnf == NULL) {
-		VCLI_Out(cli, "Internal error: No VCL_conf symbol\n");
-		(void)dlclose(vcl->dlh);
-		FREE_OBJ(vcl);
-		return (1);
-	}
-	memcpy(vcl->conf, cnf, sizeof *cnf);
-
-	if (vcl->conf->magic != VCL_CONF_MAGIC) {
-		VCLI_Out(cli, "Wrong VCL_CONF_MAGIC\n");
-		(void)dlclose(vcl->dlh);
-		FREE_OBJ(vcl);
-		return (1);
-	}
-	if (vcl->conf->init_vcl(cli)) {
-		VCLI_Out(cli, "VCL \"%s\" Failed to initialize", name);
-		(void)dlclose(vcl->dlh);
-		FREE_OBJ(vcl);
-		return (1);
-	}
-	REPLACE(vcl->name, name);
-	VCLI_Out(cli, "Loaded \"%s\" as \"%s\"", fn , name);
-	VTAILQ_INSERT_TAIL(&vcl_head, vcl, list);
-	(void)vcl->conf->init_func(NULL);
-	Lck_Lock(&vcl_mtx);
-	if (vcl_active == NULL)
-		vcl_active = vcl;
-	Lck_Unlock(&vcl_mtx);
-	VSC_C_main->n_vcl++;
-	VSC_C_main->n_vcl_avail++;
-	return (0);
-}
-
-/*--------------------------------------------------------------------
- * This function is polled from the CLI thread to dispose of any non-busy
- * VCLs which have been discarded.
- */
-
-static void
-VCL_Nuke(struct vcls *vcl)
-{
-
-	ASSERT_CLI();
-	assert(vcl != vcl_active);
-	assert(vcl->conf->discard);
-	assert(vcl->conf->busy == 0);
-	VTAILQ_REMOVE(&vcl_head, vcl, list);
-	(void)vcl->conf->fini_func(NULL);
-	vcl->conf->fini_vcl(NULL);
-	free(vcl->name);
-	(void)dlclose(vcl->dlh);
-	FREE_OBJ(vcl);
-	VSC_C_main->n_vcl--;
-	VSC_C_main->n_vcl_discard--;
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-VCL_Poll(void)
-{
-	struct vcls *vcl, *vcl2;
-
-	ASSERT_CLI();
-	VTAILQ_FOREACH_SAFE(vcl, &vcl_head, list, vcl2)
-		if (vcl->conf->discard && vcl->conf->busy == 0)
-			VCL_Nuke(vcl);
-}
-
-/*--------------------------------------------------------------------*/
-
-static void
-ccf_config_list(struct cli *cli, const char * const *av, void *priv)
-{
-	struct vcls *vcl;
-	const char *flg;
-
-	(void)av;
-	(void)priv;
-	ASSERT_CLI();
-	VTAILQ_FOREACH(vcl, &vcl_head, list) {
-		if (vcl == vcl_active) {
-			flg = "active";
-		} else if (vcl->conf->discard) {
-			flg = "discarded";
-		} else
-			flg = "available";
-		VCLI_Out(cli, "%-10s %6u %s\n",
-		    flg,
-		    vcl->conf->busy,
-		    vcl->name);
-	}
-}
-
-static void
-ccf_config_load(struct cli *cli, const char * const *av, void *priv)
-{
-
-	(void)av;
-	(void)priv;
-	ASSERT_CLI();
-	if (VCL_Load(av[3], av[2], cli))
-		VCLI_SetResult(cli, CLIS_PARAM);
-	return;
-}
-
-static void
-ccf_config_discard(struct cli *cli, const char * const *av, void *priv)
-{
-	struct vcls *vcl;
-
-	ASSERT_CLI();
-	(void)av;
-	(void)priv;
-	vcl = vcl_find(av[2]);
-	if (vcl == NULL) {
-		VCLI_SetResult(cli, CLIS_PARAM);
-		VCLI_Out(cli, "VCL '%s' unknown", av[2]);
-		return;
-	}
-	Lck_Lock(&vcl_mtx);
-	if (vcl == vcl_active) {
-		Lck_Unlock(&vcl_mtx);
-		VCLI_SetResult(cli, CLIS_PARAM);
-		VCLI_Out(cli, "VCL %s is the active VCL", av[2]);
-		return;
-	}
-	VSC_C_main->n_vcl_discard++;
-	VSC_C_main->n_vcl_avail--;
-	vcl->conf->discard = 1;
-	Lck_Unlock(&vcl_mtx);
-	if (vcl->conf->busy == 0)
-		VCL_Nuke(vcl);
-}
-
-static void
-ccf_config_use(struct cli *cli, const char * const *av, void *priv)
-{
-	struct vcls *vcl;
-	int i;
-
-	(void)av;
-	(void)priv;
-	vcl = vcl_find(av[2]);
-	if (vcl == NULL) {
-		VCLI_Out(cli, "No VCL named '%s'", av[2]);
-		VCLI_SetResult(cli, CLIS_PARAM);
-		return;
-	}
-	Lck_Lock(&vcl_mtx);
-	vcl_active = vcl;
-	Lck_Unlock(&vcl_mtx);
-
-	/* Tickle this VCL's backends to take over health polling */
-	for(i = 1; i < vcl->conf->ndirector; i++)
-		VBE_UseHealth(vcl->conf->director[i]);
-}
-
-/*--------------------------------------------------------------------*/
-
-#define VCL_MET_MAC(func, upper, bitmap)				\
-void									\
-VCL_##func##_method(struct sess *sp)					\
-{									\
-									\
-	sp->handling = 0;						\
-	sp->cur_method = VCL_MET_ ## upper;				\
-	WSP(sp, SLT_VCL_call, "%s", #func);				\
-	(void)sp->vcl->func##_func(sp);					\
-	WSP(sp, SLT_VCL_return, "%s", VCL_Return_Name(sp->handling));	\
-	sp->cur_method = 0;						\
-	assert((1U << sp->handling) & bitmap);				\
-	assert(!((1U << sp->handling) & ~bitmap));			\
-}
-
-#include "tbl/vcl_returns.h"
-#undef VCL_MET_MAC
-
-/*--------------------------------------------------------------------*/
-
-static struct cli_proto vcl_cmds[] = {
-	{ CLI_VCL_LOAD,         "i", ccf_config_load },
-	{ CLI_VCL_LIST,         "i", ccf_config_list },
-	{ CLI_VCL_DISCARD,      "i", ccf_config_discard },
-	{ CLI_VCL_USE,          "i", ccf_config_use },
-	{ NULL }
-};
-
-void
-VCL_Init()
-{
-
-	CLI_AddFuncs(vcl_cmds);
-	Lck_New(&vcl_mtx, lck_vcl);
-}
diff --git a/bin/varnishd/cache_vrt.c b/bin/varnishd/cache_vrt.c
deleted file mode 100644
index c20b552..0000000
--- a/bin/varnishd/cache_vrt.c
+++ /dev/null
@@ -1,535 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2010 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * Runtime support for compiled VCL programs
- */
-
-#include "config.h"
-
-#include <netinet/in.h>
-#include <arpa/inet.h>
-
-#include <stdio.h>
-#include <stdlib.h>
-
-#include "cache.h"
-
-#include "cache_backend.h"
-#include "hash/hash_slinger.h"
-#include "vav.h"
-#include "vcl.h"
-#include "vrt.h"
-#include "vrt_obj.h"
-#include "vtim.h"
-
-const void * const vrt_magic_string_end = &vrt_magic_string_end;
-
-/*--------------------------------------------------------------------*/
-
-void
-VRT_error(struct sess *sp, unsigned code, const char *reason)
-{
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	WSL(sp->wrk, SLT_Debug, 0, "VCL_error(%u, %s)", code, reason ?
-	    reason : "(null)");
-	if (code < 100 || code > 999)
-		code = 503;
-	sp->err_code = (uint16_t)code;
-	sp->err_reason = reason ? reason : http_StatusMessage(sp->err_code);
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-VRT_count(const struct sess *sp, unsigned u)
-{
-
-	if (sp == NULL)
-		return;
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	if (cache_param->vcl_trace)
-		WSP(sp, SLT_VCL_trace, "%u %d.%d", u,
-		    sp->vcl->ref[u].line, sp->vcl->ref[u].pos);
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-VRT_acl_log(const struct sess *sp, const char *msg)
-{
-	WSP(sp, SLT_VCL_acl, msg);
-}
-
-/*--------------------------------------------------------------------*/
-
-static struct http *
-vrt_selecthttp(const struct sess *sp, enum gethdr_e where)
-{
-	struct http *hp;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	switch (where) {
-	case HDR_REQ:
-		hp = sp->http;
-		break;
-	case HDR_BEREQ:
-		hp = sp->wrk->bereq;
-		break;
-	case HDR_BERESP:
-		hp = sp->wrk->beresp;
-		break;
-	case HDR_RESP:
-		hp = sp->wrk->resp;
-		break;
-	case HDR_OBJ:
-		CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);
-		hp = sp->obj->http;
-		break;
-	default:
-		INCOMPL();
-	}
-	CHECK_OBJ_NOTNULL(hp, HTTP_MAGIC);
-	return (hp);
-}
-
-char *
-VRT_GetHdr(const struct sess *sp, enum gethdr_e where, const char *n)
-{
-	char *p;
-	struct http *hp;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	hp = vrt_selecthttp(sp, where);
-	if (!http_GetHdr(hp, n, &p))
-		return (NULL);
-	return (p);
-}
-
-/*--------------------------------------------------------------------
- * XXX: Optimize the single element case ?
- */
-
-char *
-VRT_StringList(char *d, unsigned dl, const char *p, va_list ap)
-{
-	char *b, *e;
-	unsigned x;
-
-	b = d;
-	e = b + dl;
-	while (p != vrt_magic_string_end && b < e) {
-		if (p != NULL) {
-			x = strlen(p);
-			if (b + x < e)
-				memcpy(b, p, x);
-			b += x;
-		}
-		p = va_arg(ap, const char *);
-	}
-	if (b >= e)
-		return (NULL);
-	*b++ = '\0';
-	return (b);
-}
-
-/*--------------------------------------------------------------------
- * XXX: Optimize the single element case ?
- */
-
-char *
-VRT_String(struct ws *ws, const char *h, const char *p, va_list ap)
-{
-	char *b, *e;
-	unsigned u, x;
-
-	u = WS_Reserve(ws, 0);
-	e = b = ws->f;
-	e += u;
-	if (h != NULL) {
-		x = strlen(h);
-		if (b + x < e)
-			memcpy(b, h, x);
-		b += x;
-		if (b < e)
-			*b = ' ';
-		b++;
-	}
-	b = VRT_StringList(b, e > b ? e - b : 0, p, ap);
-	if (b == NULL || b == e) {
-		WS_Release(ws, 0);
-		return (NULL);
-	}
-	e = b;
-	b = ws->f;
-	WS_Release(ws, e - b);
-	return (b);
-}
-
-/*--------------------------------------------------------------------
- * Build a string on the worker threads workspace
- */
-
-const char *
-VRT_WrkString(const struct sess *sp, const char *p, ...)
-{
-	va_list ap;
-	char *b;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	va_start(ap, p);
-	b = VRT_String(sp->wrk->ws, NULL, p, ap);
-	va_end(ap);
-	return (b);
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-VRT_SetHdr(const struct sess *sp , enum gethdr_e where, const char *hdr,
-    const char *p, ...)
-{
-	struct http *hp;
-	va_list ap;
-	char *b;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	hp = vrt_selecthttp(sp, where);
-	va_start(ap, p);
-	if (p == NULL) {
-		http_Unset(hp, hdr);
-	} else {
-		b = VRT_String(hp->ws, hdr + 1, p, ap);
-		if (b == NULL) {
-			WSP(sp, SLT_LostHeader, "%s", hdr + 1);
-		} else {
-			http_Unset(hp, hdr);
-			http_SetHeader(sp->wrk, sp->vsl_id, hp, b);
-		}
-	}
-	va_end(ap);
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-VRT_handling(struct sess *sp, unsigned hand)
-{
-
-	if (sp == NULL) {
-		assert(hand == VCL_RET_OK);
-		return;
-	}
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	assert(hand < VCL_RET_MAX);
-	sp->handling = hand;
-}
-
-/*--------------------------------------------------------------------
- * Add an element to the array/list of hash bits.
- */
-
-void
-VRT_hashdata(const struct sess *sp, const char *str, ...)
-{
-	va_list ap;
-	const char *p;
-
-	HSH_AddString(sp, str);
-	va_start(ap, str);
-	while (1) {
-		p = va_arg(ap, const char *);
-		if (p == vrt_magic_string_end)
-			break;
-		HSH_AddString(sp, p);
-	}
-}
-
-/*--------------------------------------------------------------------*/
-
-double
-VRT_r_now(const struct sess *sp)
-{
-
-	(void)sp;
-	return (VTIM_real());
-}
-
-/*--------------------------------------------------------------------*/
-
-char *
-VRT_IP_string(const struct sess *sp, const struct sockaddr_storage *sa)
-{
-	char *p;
-	const struct sockaddr_in *si4;
-	const struct sockaddr_in6 *si6;
-	const void *addr;
-	int len;
-
-	switch (sa->ss_family) {
-	case AF_INET:
-		len = INET_ADDRSTRLEN;
-		si4 = (const void *)sa;
-		addr = &(si4->sin_addr);
-		break;
-	case AF_INET6:
-		len = INET6_ADDRSTRLEN;
-		si6 = (const void *)sa;
-		addr = &(si6->sin6_addr);
-		break;
-	default:
-		INCOMPL();
-	}
-	XXXAN(len);
-	AN(p = WS_Alloc(sp->http->ws, len));
-	AN(inet_ntop(sa->ss_family, addr, p, len));
-	return (p);
-}
-
-char *
-VRT_int_string(const struct sess *sp, int num)
-{
-	char *p;
-	int size;
-
-	size = snprintf(NULL, 0, "%d", num) + 1;
-	AN(p = WS_Alloc(sp->http->ws, size));
-	assert(snprintf(p, size, "%d", num) < size);
-	return (p);
-}
-
-char *
-VRT_double_string(const struct sess *sp, double num)
-{
-	char *p;
-	int size;
-
-	size = snprintf(NULL, 0, "%.3f", num) + 1;
-	AN(p = WS_Alloc(sp->http->ws, size));
-	assert(snprintf(p, size, "%.3f", num) < size);
-	return (p);
-}
-
-char *
-VRT_time_string(const struct sess *sp, double t)
-{
-	char *p;
-
-	AN(p = WS_Alloc(sp->http->ws, VTIM_FORMAT_SIZE));
-	VTIM_format(t, p);
-	return (p);
-}
-
-const char *
-VRT_backend_string(const struct sess *sp, const struct director *d)
-{
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	if (d == NULL)
-		d = sp->director;
-	if (d == NULL)
-		return (NULL);
-	return (d->vcl_name);
-}
-
-const char *
-VRT_bool_string(const struct sess *sp, unsigned val)
-{
-
-	(void)sp;
-	return (val ? "true" : "false");
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-VRT_Rollback(struct sess *sp)
-{
-
-	HTTP_Copy(sp->http, sp->http0);
-	WS_Reset(sp->ws, sp->ws_req);
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-VRT_panic(const struct sess *sp, const char *str, ...)
-{
-	va_list ap;
-	char *b;
-
-	va_start(ap, str);
-	b = VRT_String(sp->http->ws, "PANIC: ", str, ap);
-	va_end(ap);
-	VAS_Fail("VCL", "", 0, b, 0, 2);
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-VRT_synth_page(const struct sess *sp, unsigned flags, const char *str, ...)
-{
-	va_list ap;
-	const char *p;
-	struct vsb *vsb;
-
-	(void)flags;
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);
-	vsb = SMS_Makesynth(sp->obj);
-	AN(vsb);
-
-	VSB_cat(vsb, str);
-	va_start(ap, str);
-	p = va_arg(ap, const char *);
-	while (p != vrt_magic_string_end) {
-		if (p == NULL)
-			p = "(null)";
-		VSB_cat(vsb, p);
-		p = va_arg(ap, const char *);
-	}
-	va_end(ap);
-	SMS_Finish(sp->obj);
-	http_Unset(sp->obj->http, H_Content_Length);
-	http_PrintfHeader(sp->wrk, sp->vsl_id, sp->obj->http,
-	    "Content-Length: %d", sp->obj->len);
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-VRT_ban(struct sess *sp, char *cmds, ...)
-{
-	char *a1, *a2, *a3;
-	va_list ap;
-	struct ban *b;
-	int good;
-
-	(void)sp;
-	b = BAN_New();
-	va_start(ap, cmds);
-	a1 = cmds;
-	good = 0;
-	while (a1 != NULL) {
-		good = 0;
-		a2 = va_arg(ap, char *);
-		if (a2 == NULL)
-			break;
-		a3 = va_arg(ap, char *);
-		if (a3 == NULL)
-			break;
-		if (BAN_AddTest(NULL, b, a1, a2, a3))
-			break;
-		a1 = va_arg(ap, char *);
-		good = 1;
-	}
-	if (!good)
-		/* XXX: report error how ? */
-		BAN_Free(b);
-	else
-		BAN_Insert(b);
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-VRT_ban_string(struct sess *sp, const char *str)
-{
-	char *a1, *a2, *a3;
-	char **av;
-	struct ban *b;
-	int good;
-	int i;
-
-	(void)sp;
-	av = VAV_Parse(str, NULL, ARGV_NOESC);
-	if (av[0] != NULL) {
-		/* XXX: report error how ? */
-		VAV_Free(av);
-		return;
-	}
-	b = BAN_New();
-	good = 0;
-	for (i = 1; ;) {
-		a1 = av[i++];
-		if (a1 == NULL)
-			break;
-		good = 0;
-		a2 = av[i++];
-		if (a2 == NULL)
-			break;
-		a3 = av[i++];
-		if (a3 == NULL)
-			break;
-		if (BAN_AddTest(NULL, b, a1, a2, a3))
-			break;
-		good = 1;
-		if (av[i] == NULL)
-			break;
-		good = 0;
-		if (strcmp(av[i++], "&&"))
-			break;
-	}
-	if (!good)
-		/* XXX: report error how ? */
-		BAN_Free(b);
-	else
-		BAN_Insert(b);
-	VAV_Free(av);
-}
-
-/*--------------------------------------------------------------------
- * "real" purges
- */
-
-void
-VRT_purge(const struct sess *sp, double ttl, double grace)
-{
-	if (sp->cur_method == VCL_MET_HIT)
-		HSH_Purge(sp, sp->obj->objcore->objhead, ttl, grace);
-	else if (sp->cur_method == VCL_MET_MISS)
-		HSH_Purge(sp, sp->objcore->objhead, ttl, grace);
-}
-
-/*--------------------------------------------------------------------
- * Simple stuff
- */
-
-int
-VRT_strcmp(const char *s1, const char *s2)
-{
-	if (s1 == NULL || s2 == NULL)
-		return(1);
-	return (strcmp(s1, s2));
-}
-
-void
-VRT_memmove(void *dst, const void *src, unsigned len)
-{
-
-	(void)memmove(dst, src, len);
-}
diff --git a/bin/varnishd/cache_vrt_re.c b/bin/varnishd/cache_vrt_re.c
deleted file mode 100644
index 7759d0a..0000000
--- a/bin/varnishd/cache_vrt_re.c
+++ /dev/null
@@ -1,162 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2009 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * Runtime support for compiled VCL programs, regexps
- */
-
-#include "config.h"
-
-#include <ctype.h>
-
-#include "cache.h"
-
-#include "vre.h"
-#include "vrt.h"
-
-void
-VRT_re_init(void **rep, const char *re)
-{
-	vre_t *t;
-	const char *error;
-	int erroroffset;
-
-	/* This was already check-compiled by the VCL compiler */
-	t = VRE_compile(re, 0, &error, &erroroffset);
-	AN(t);
-	*rep = t;
-}
-
-void
-VRT_re_fini(void *rep)
-{
-	vre_t *vv;
-
-	vv = rep;
-	if (rep != NULL)
-		VRE_free(&vv);
-}
-
-int
-VRT_re_match(const struct sess *sp, const char *s, void *re)
-{
-	vre_t *t;
-	int i;
-
-	if (s == NULL)
-		s = "";
-	AN(re);
-	t = re;
-	i = VRE_exec(t, s, strlen(s), 0, 0, NULL, 0, &cache_param->vre_limits);
-	if (i >= 0)
-		return (1);
-	if (i < VRE_ERROR_NOMATCH )
-		WSP(sp, SLT_VCL_Error, "Regexp matching returned %d", i);
-	return (0);
-}
-
-const char *
-VRT_regsub(const struct sess *sp, int all, const char *str, void *re,
-    const char *sub)
-{
-	int ovector[30];
-	vre_t *t;
-	int i, l;
-	txt res;
-	char *b0;
-	const char *s;
-	unsigned u, x;
-	int options = 0;
-	size_t len;
-
-	AN(re);
-	if (str == NULL)
-		str = "";
-	t = re;
-	memset(ovector, 0, sizeof(ovector));
-	len = strlen(str);
-	i = VRE_exec(t, str, len, 0, options, ovector, 30,
-	    &cache_param->vre_limits);
-
-	/* If it didn't match, we can return the original string */
-	if (i == VRE_ERROR_NOMATCH)
-		return(str);
-	if (i < VRE_ERROR_NOMATCH ) {
-		WSP(sp, SLT_VCL_Error, "Regexp matching returned %d", i);
-		return(str);
-	}
-
-	u = WS_Reserve(sp->http->ws, 0);
-	res.e = res.b = b0 = sp->http->ws->f;
-	res.e += u;
-
-	do {
-		/* Copy prefix to match */
-		Tadd(&res, str, ovector[0]);
-		for (s = sub ; *s != '\0'; s++ ) {
-			if (*s != '\\' || s[1] == '\0') {
-				if (res.b < res.e)
-					*res.b++ = *s;
-				continue;
-			}
-			s++;
-			if (isdigit(*s)) {
-				x = *s - '0';
-				l = ovector[2*x+1] - ovector[2*x];
-				Tadd(&res, str + ovector[2*x], l);
-				continue;
-			} else {
-				if (res.b < res.e)
-					*res.b++ = *s;
-			}
-		}
-		str += ovector[1];
-		len -= ovector[1];
-		if (!all)
-			break;
-		memset(&ovector, 0, sizeof(ovector));
-		options |= VRE_NOTEMPTY_ATSTART;
-		i = VRE_exec(t, str, len, 0, options, ovector, 30,
-		    &cache_param->vre_limits);
-		if (i < VRE_ERROR_NOMATCH ) {
-			WS_Release(sp->http->ws, 0);
-			WSP(sp, SLT_VCL_Error,
-			    "Regexp matching returned %d", i);
-			return(str);
-		}
-	} while (i != VRE_ERROR_NOMATCH);
-
-	/* Copy suffix to match */
-	Tadd(&res, str, len+1);
-	if (res.b >= res.e) {
-		WS_Release(sp->http->ws, 0);
-		return (str);
-	}
-	Tcheck(res);
-	WS_ReleaseP(sp->http->ws, res.b);
-	return (b0);
-}
diff --git a/bin/varnishd/cache_vrt_var.c b/bin/varnishd/cache_vrt_var.c
deleted file mode 100644
index 860c7aa..0000000
--- a/bin/varnishd/cache_vrt_var.c
+++ /dev/null
@@ -1,550 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * Runtime support for compiled VCL programs
- */
-#include "config.h"
-
-#include <stdio.h>
-#include <stdlib.h>
-
-#include "cache.h"
-#include "common/heritage.h"
-
-#include "cache_backend.h"
-#include "vrt_obj.h"
-#include "vtcp.h"
-#include "vtim.h"
-
-static char vrt_hostname[255] = "";
-
-/*--------------------------------------------------------------------*/
-
-static void
-vrt_do_string(struct worker *w, int fd, const struct http *hp, int fld,
-    const char *err, const char *p, va_list ap)
-{
-	char *b;
-
-	// AN(p);
-	AN(hp);
-	b = VRT_String(hp->ws, NULL, p, ap);
-	if (b == NULL || *b == '\0') {
-		WSL(w, SLT_LostHeader, fd, err);
-	} else {
-		http_SetH(hp, fld, b);
-	}
-	va_end(ap);
-}
-
-#define VRT_DO_HDR(obj, hdr, http, fld)				\
-void								\
-VRT_l_##obj##_##hdr(const struct sess *sp, const char *p, ...)	\
-{								\
-	va_list ap;						\
-								\
-	va_start(ap, p);					\
-	vrt_do_string(sp->wrk, sp->fd,				\
-	    http, fld, #obj "." #hdr, p, ap);			\
-	va_end(ap);						\
-}								\
-								\
-const char *							\
-VRT_r_##obj##_##hdr(const struct sess *sp)			\
-{								\
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);			\
-	CHECK_OBJ_NOTNULL(http, HTTP_MAGIC);			\
-	return (http->hd[fld].b);				\
-}
-
-VRT_DO_HDR(req,   request,	sp->http,		HTTP_HDR_REQ)
-VRT_DO_HDR(req,   url,		sp->http,		HTTP_HDR_URL)
-VRT_DO_HDR(req,   proto,	sp->http,		HTTP_HDR_PROTO)
-VRT_DO_HDR(bereq, request,	sp->wrk->bereq,		HTTP_HDR_REQ)
-VRT_DO_HDR(bereq, url,		sp->wrk->bereq,		HTTP_HDR_URL)
-VRT_DO_HDR(bereq, proto,	sp->wrk->bereq,		HTTP_HDR_PROTO)
-VRT_DO_HDR(obj,   proto,	sp->obj->http,		HTTP_HDR_PROTO)
-VRT_DO_HDR(obj,   response,	sp->obj->http,		HTTP_HDR_RESPONSE)
-VRT_DO_HDR(resp,  proto,	sp->wrk->resp,		HTTP_HDR_PROTO)
-VRT_DO_HDR(resp,  response,	sp->wrk->resp,		HTTP_HDR_RESPONSE)
-VRT_DO_HDR(beresp,  proto,	sp->wrk->beresp,	HTTP_HDR_PROTO)
-VRT_DO_HDR(beresp,  response,	sp->wrk->beresp,	HTTP_HDR_RESPONSE)
-
-/*--------------------------------------------------------------------*/
-
-#define VRT_DO_STATUS(obj, http)				\
-void								\
-VRT_l_##obj##_status(const struct sess *sp, int num)		\
-{								\
-								\
-	assert(num >= 100 && num <= 999);			\
-	http->status = (uint16_t)num;				\
-}								\
-								\
-int								\
-VRT_r_##obj##_status(const struct sess *sp)			\
-{								\
-								\
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);			\
-	return(http->status);					\
-}
-
-VRT_DO_STATUS(obj, sp->obj->http)
-VRT_DO_STATUS(beresp, sp->wrk->beresp)
-VRT_DO_STATUS(resp, sp->wrk->resp)
-
-/*--------------------------------------------------------------------*/
-
-/* XXX: review this */
-/* Add an objecthead to the saintmode list for the (hopefully) relevant
- * backend. Some double-up asserting here to avoid assert-errors when there
- * is no object.
- */
-void
-VRT_l_beresp_saintmode(const struct sess *sp, double a)
-{
-	struct trouble *new;
-	struct trouble *tr;
-	struct trouble *tr2;
-	struct worker *wrk;
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	wrk = sp->wrk;
-	if (!wrk->vbc)
-		return;
-	CHECK_OBJ_NOTNULL(wrk->vbc, VBC_MAGIC);
-	if (!wrk->vbc->backend)
-		return;
-	CHECK_OBJ_NOTNULL(wrk->vbc->backend, BACKEND_MAGIC);
-	if (!sp->objcore)
-		return;
-	CHECK_OBJ_NOTNULL(sp->objcore, OBJCORE_MAGIC);
-
-	/* Setting a negative holdoff period is a mistake. Detecting this
-	 * when compiling the VCL would be better.
-	 */
-	assert(a > 0);
-
-	ALLOC_OBJ(new, TROUBLE_MAGIC);
-	AN(new);
-	new->target = (uintptr_t)(sp->objcore->objhead);
-	new->timeout = sp->t_req + a;
-
-	/* Insert the new item on the list before the first item with a
-	 * timeout at a later date (ie: sort by which entry will time out
-	 * from the list
-	 */
-	Lck_Lock(&wrk->vbc->backend->mtx);
-	VTAILQ_FOREACH_SAFE(tr, &wrk->vbc->backend->troublelist, list, tr2) {
-		if (tr->timeout < new->timeout) {
-			VTAILQ_INSERT_BEFORE(tr, new, list);
-			new = NULL;
-			break;
-		}
-	}
-
-	/* Insert the item at the end if the list is empty or all other
-	 * items have a longer timeout.
-	 */
-	if (new)
-		VTAILQ_INSERT_TAIL(&wrk->vbc->backend->troublelist, new, list);
-
-	Lck_Unlock(&wrk->vbc->backend->mtx);
-}
-
-/*--------------------------------------------------------------------*/
-
-#define VBERESP(dir, type, onm, field)					\
-void									\
-VRT_l_##dir##_##onm(const struct sess *sp, type a)			\
-{									\
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);				\
-	sp->wrk->field = a;						\
-}									\
-									\
-type									\
-VRT_r_##dir##_##onm(const struct sess *sp)				\
-{									\
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);				\
-	return (sp->wrk->field);					\
-}
-
-VBERESP(beresp, unsigned, do_esi, do_esi)
-VBERESP(beresp, unsigned, do_gzip, do_gzip)
-VBERESP(beresp, unsigned, do_gunzip, do_gunzip)
-VBERESP(beresp, unsigned, do_stream, do_stream)
-
-/*--------------------------------------------------------------------*/
-
-const char * __match_proto__()
-VRT_r_client_identity(struct sess *sp)
-{
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	if (sp->client_identity != NULL)
-		return (sp->client_identity);
-	else
-		return (sp->addr);
-}
-
-void
-VRT_l_client_identity(struct sess *sp, const char *str, ...)
-{
-	va_list ap;
-	char *b;
-
-	va_start(ap, str);
-	b = VRT_String(sp->http->ws, NULL, str, ap);
-	va_end(ap);
-	sp->client_identity = b;
-}
-
-/*--------------------------------------------------------------------*/
-
-#define BEREQ_TIMEOUT(which)					\
-void __match_proto__()						\
-VRT_l_bereq_##which(struct sess *sp, double num)		\
-{								\
-								\
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);			\
-	sp->wrk->which = (num > 0.0 ? num : 0.0);		\
-}								\
-								\
-double __match_proto__()					\
-VRT_r_bereq_##which(struct sess *sp)				\
-{								\
-								\
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);			\
-	return(sp->wrk->which);					\
-}
-
-BEREQ_TIMEOUT(connect_timeout)
-BEREQ_TIMEOUT(first_byte_timeout)
-BEREQ_TIMEOUT(between_bytes_timeout)
-
-/*--------------------------------------------------------------------*/
-
-const char *
-VRT_r_beresp_backend_name(const struct sess *sp)
-{
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->wrk->vbc, VBC_MAGIC);
-	return(sp->wrk->vbc->backend->vcl_name);
-}
-
-struct sockaddr_storage *
-VRT_r_beresp_backend_ip(const struct sess *sp)
-{
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->wrk->vbc, VBC_MAGIC);
-	return(sp->wrk->vbc->addr);
-}
-
-int
-VRT_r_beresp_backend_port(const struct sess *sp)
-{
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->wrk->vbc, VBC_MAGIC);
-	return (VTCP_port(sp->wrk->vbc->addr));
-}
-
-const char * __match_proto__()
-VRT_r_beresp_storage(struct sess *sp)
-{
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	if (sp->wrk->storage_hint != NULL)
-		return (sp->wrk->storage_hint);
-	else
-		return (NULL);
-}
-
-void __match_proto__()
-VRT_l_beresp_storage(struct sess *sp, const char *str, ...)
-{
-	va_list ap;
-	char *b;
-
-	va_start(ap, str);
-	b = VRT_String(sp->wrk->ws, NULL, str, ap);
-	va_end(ap);
-	sp->wrk->storage_hint = b;
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-VRT_l_req_backend(struct sess *sp, struct director *be)
-{
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	sp->director = be;
-}
-
-struct director * __match_proto__()
-VRT_r_req_backend(struct sess *sp)
-{
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	return (sp->director);
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-VRT_l_req_esi(struct sess *sp, unsigned process_esi)
-{
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	/*
-	 * Only allow you to turn of esi in the main request
-	 * else everything gets confused
-	 */
-	if(sp->esi_level == 0)
-		sp->disable_esi = !process_esi;
-}
-
-unsigned __match_proto__()
-VRT_r_req_esi(struct sess *sp)
-{
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	return (!sp->disable_esi);
-}
-
-int
-VRT_r_req_esi_level(const struct sess *sp)
-{
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	return(sp->esi_level);
-}
-
-/*--------------------------------------------------------------------*/
-
-unsigned __match_proto__()
-VRT_r_req_can_gzip(struct sess *sp)
-{
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	return (RFC2616_Req_Gzip(sp));
-}
-
-
-/*--------------------------------------------------------------------*/
-
-int
-VRT_r_req_restarts(const struct sess *sp)
-{
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	return (sp->restarts);
-}
-
-/*--------------------------------------------------------------------
- * NB: TTL is relative to when object was created, whereas grace and
- * keep are relative to ttl.
- */
-
-#define VRT_DO_EXP(which, exp, fld, offset, extra)		\
-								\
-void __match_proto__()						\
-VRT_l_##which##_##fld(struct sess *sp, double a)		\
-{								\
-								\
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);			\
-	if (a > 0.)						\
-		a += offset;					\
-	EXP_Set_##fld(&exp, a);					\
-	extra;							\
-}								\
-								\
-double __match_proto__()					\
-VRT_r_##which##_##fld(struct sess *sp)				\
-{								\
-								\
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);			\
-	return(EXP_Get_##fld(&exp) - offset);			\
-}
-
-static void
-vrt_wsp_exp(const struct sess *sp, unsigned xid, const struct exp *e)
-{
-	WSP(sp, SLT_TTL, "%u VCL %.0f %.0f %.0f %.0f %.0f",
-	    xid, e->ttl - (sp->t_req - e->entered), e->grace, e->keep,
-	    sp->t_req, e->age + (sp->t_req - e->entered));
-}
-
-VRT_DO_EXP(req, sp->exp, ttl, 0, )
-VRT_DO_EXP(req, sp->exp, grace, 0, )
-VRT_DO_EXP(req, sp->exp, keep, 0, )
-
-VRT_DO_EXP(obj, sp->obj->exp, grace, 0,
-   EXP_Rearm(sp->obj);
-   vrt_wsp_exp(sp, sp->obj->xid, &sp->obj->exp);)
-VRT_DO_EXP(obj, sp->obj->exp, ttl, (sp->t_req - sp->obj->exp.entered),
-   EXP_Rearm(sp->obj);
-   vrt_wsp_exp(sp, sp->obj->xid, &sp->obj->exp);)
-VRT_DO_EXP(obj, sp->obj->exp, keep, 0,
-   EXP_Rearm(sp->obj);
-   vrt_wsp_exp(sp, sp->obj->xid, &sp->obj->exp);)
-
-VRT_DO_EXP(beresp, sp->wrk->exp, grace, 0,
-   vrt_wsp_exp(sp, sp->xid, &sp->wrk->exp);)
-VRT_DO_EXP(beresp, sp->wrk->exp, ttl, 0,
-   vrt_wsp_exp(sp, sp->xid, &sp->wrk->exp);)
-VRT_DO_EXP(beresp, sp->wrk->exp, keep, 0,
-   vrt_wsp_exp(sp, sp->xid, &sp->wrk->exp);)
-
-/*--------------------------------------------------------------------
- * req.xid
- */
-
-const char * __match_proto__()
-VRT_r_req_xid(struct sess *sp)
-{
-	char *p;
-	int size;
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-
-	size = snprintf(NULL, 0, "%u", sp->xid) + 1;
-	AN(p = WS_Alloc(sp->http->ws, size));
-	assert(snprintf(p, size, "%u", sp->xid) < size);
-	return (p);
-}
-
-/*--------------------------------------------------------------------*/
-
-#define REQ_BOOL(which)						\
-void __match_proto__()						\
-VRT_l_req_##which(struct sess *sp, unsigned val)		\
-{								\
-								\
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);			\
-	sp->which = val ? 1 : 0;				\
-}								\
-								\
-unsigned __match_proto__()					\
-VRT_r_req_##which(struct sess *sp)				\
-{								\
-								\
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);			\
-	return(sp->which);					\
-}
-
-REQ_BOOL(hash_ignore_busy)
-REQ_BOOL(hash_always_miss)
-
-/*--------------------------------------------------------------------*/
-
-struct sockaddr_storage *
-VRT_r_client_ip(struct sess *sp)
-{
-
-	return (&sp->sockaddr);
-}
-
-struct sockaddr_storage *
-VRT_r_server_ip(struct sess *sp)
-{
-	int i;
-
-	if (sp->mysockaddr.ss_family == AF_UNSPEC) {
-		i = getsockname(sp->fd,
-		    (void*)&sp->mysockaddr, &sp->mysockaddrlen);
-		assert(VTCP_Check(i));
-	}
-
-	return (&sp->mysockaddr);
-}
-
-const char*
-VRT_r_server_identity(struct sess *sp)
-{
-	(void)sp;
-
-	if (heritage.identity[0] != '\0')
-		return (heritage.identity);
-	else
-		return (heritage.name);
-}
-
-
-const char*
-VRT_r_server_hostname(struct sess *sp)
-{
-	(void)sp;
-
-	if (vrt_hostname[0] == '\0')
-		AZ(gethostname(vrt_hostname, sizeof(vrt_hostname)));
-
-	return (vrt_hostname);
-}
-
-/*--------------------------------------------------------------------
- * XXX: This is pessimistically silly
- */
-
-int
-VRT_r_server_port(struct sess *sp)
-{
-	int i;
-
-	if (sp->mysockaddr.ss_family == AF_UNSPEC) {
-		i = getsockname(sp->fd,
-		    (void*)&sp->mysockaddr, &sp->mysockaddrlen);
-		assert(VTCP_Check(i));
-	}
-	return (VTCP_port(&sp->mysockaddr));
-}
-
-/*--------------------------------------------------------------------*/
-
-int
-VRT_r_obj_hits(const struct sess *sp)
-{
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);	/* XXX */
-	return (sp->obj->hits);
-}
-
-double
-VRT_r_obj_lastuse(const struct sess *sp)
-{
-
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);	/* XXX */
-	return (VTIM_real() - sp->obj->last_use);
-}
-
-unsigned
-VRT_r_req_backend_healthy(const struct sess *sp)
-{
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->director, DIRECTOR_MAGIC);
-	return (VDI_Healthy(sp->director, sp));
-}
-
diff --git a/bin/varnishd/cache_vrt_vmod.c b/bin/varnishd/cache_vrt_vmod.c
deleted file mode 100644
index 6b3b846..0000000
--- a/bin/varnishd/cache_vrt_vmod.c
+++ /dev/null
@@ -1,180 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * Runtime support for compiled VCL programs
- */
-
-#include "config.h"
-
-#include <dlfcn.h>
-#include <stdlib.h>
-
-#include "cache.h"
-
-#include "vcli_priv.h"
-#include "vrt.h"
-
-/*--------------------------------------------------------------------
- * Modules stuff
- */
-
-struct vmod {
-	unsigned		magic;
-#define VMOD_MAGIC		0xb750219c
-
-	VTAILQ_ENTRY(vmod)	list;
-
-	int			ref;
-
-	char			*nm;
-	char			*path;
-	void			*hdl;
-	const void		*funcs;
-	int			funclen;
-	const void		*idptr;
-};
-
-static VTAILQ_HEAD(,vmod)	vmods = VTAILQ_HEAD_INITIALIZER(vmods);
-
-int
-VRT_Vmod_Init(void **hdl, void *ptr, int len, const char *nm,
-    const char *path, struct cli *cli)
-{
-	struct vmod *v;
-	void *x, *y, *z, *w;
-
-	ASSERT_CLI();
-
-	VTAILQ_FOREACH(v, &vmods, list)
-		if (!strcmp(v->nm, nm))	// Also path, len ?
-			break;
-	if (v == NULL) {
-		ALLOC_OBJ(v, VMOD_MAGIC);
-		AN(v);
-
-		v->hdl = dlopen(path, RTLD_NOW | RTLD_LOCAL);
-		if (v->hdl == NULL) {
-			VCLI_Out(cli, "Loading VMOD %s from %s:\n", nm, path);
-			VCLI_Out(cli, "dlopen() failed: %s\n", dlerror());
-			VCLI_Out(cli, "Check child process permissions.\n");
-			FREE_OBJ(v);
-			return (1);
-		}
-
-		x = dlsym(v->hdl, "Vmod_Name");
-		y = dlsym(v->hdl, "Vmod_Len");
-		z = dlsym(v->hdl, "Vmod_Func");
-		w = dlsym(v->hdl, "Vmod_Id");
-		if (x == NULL || y == NULL || z == NULL || w == NULL) {
-			VCLI_Out(cli, "Loading VMOD %s from %s:\n", nm, path);
-			VCLI_Out(cli, "VMOD symbols not found\n");
-			VCLI_Out(cli, "Check relative pathnames.\n");
-			(void)dlclose(v->hdl);
-			FREE_OBJ(v);
-			return (1);
-		}
-		AN(x);
-		AN(y);
-		AN(z);
-		AN(w);
-		if (strcmp(x, nm)) {
-			VCLI_Out(cli, "Loading VMOD %s from %s:\n", nm, path);
-			VCLI_Out(cli, "File contain wrong VMOD (\"%s\")\n", x);
-			VCLI_Out(cli, "Check relative pathnames ?.\n");
-			(void)dlclose(v->hdl);
-			FREE_OBJ(v);
-			return (1);
-		}
-
-		v->funclen = *(const int *)y;
-		v->funcs = z;
-
-		REPLACE(v->nm, nm);
-		REPLACE(v->path, path);
-
-		VSC_C_main->vmods++;
-		VTAILQ_INSERT_TAIL(&vmods, v, list);
-		v->idptr = w;
-	}
-
-	assert(len == v->funclen);
-	memcpy(ptr, v->funcs, v->funclen);
-	v->ref++;
-
-	*hdl = v;
-	return (0);
-}
-
-void
-VRT_Vmod_Fini(void **hdl)
-{
-	struct vmod *v;
-
-	ASSERT_CLI();
-
-	AN(*hdl);
-	CAST_OBJ_NOTNULL(v, *hdl, VMOD_MAGIC);
-	*hdl = NULL;
-	if (--v->ref != 0)
-		return;
-#ifndef DONT_DLCLOSE_VMODS
-	AZ(dlclose(v->hdl));
-#endif
-	free(v->nm);
-	free(v->path);
-	VTAILQ_REMOVE(&vmods, v, list);
-	VSC_C_main->vmods--;
-	FREE_OBJ(v);
-}
-
-/*---------------------------------------------------------------------*/
-
-static void
-ccf_debug_vmod(struct cli *cli, const char * const *av, void *priv)
-{
-	struct vmod *v;
-
-	(void)av;
-	(void)priv;
-	ASSERT_CLI();
-	VTAILQ_FOREACH(v, &vmods, list)
-		VCLI_Out(cli, "%5d %s (%s)\n", v->ref, v->nm, v->path);
-}
-
-static struct cli_proto vcl_cmds[] = {
-	{ "debug.vmod", "debug.vmod", "show loaded vmods", 0, 0,
-		"d", ccf_debug_vmod },
-	{ NULL }
-};
-
-void
-VMOD_Init(void)
-{
-
-	CLI_AddFuncs(vcl_cmds);
-}
diff --git a/bin/varnishd/cache_wrk.c b/bin/varnishd/cache_wrk.c
deleted file mode 100644
index bfee84c..0000000
--- a/bin/varnishd/cache_wrk.c
+++ /dev/null
@@ -1,204 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * Worker thread stuff unrelated to the worker thread pools.
- */
-
-#include "config.h"
-
-#include <math.h>
-#include <stdlib.h>
-
-#include "cache.h"
-
-#include "hash/hash_slinger.h"
-#include "vsha256.h"
-
-static struct lock		wstat_mtx;
-
-/*--------------------------------------------------------------------*/
-
-static void
-wrk_sumstat(struct worker *w)
-{
-
-	Lck_AssertHeld(&wstat_mtx);
-#define L0(n)
-#define L1(n) (VSC_C_main->n += w->stats.n)
-#define VSC_DO_MAIN
-#define VSC_F(n, t, l, f, d, e) L##l(n);
-#include "tbl/vsc_fields.h"
-#undef VSC_F
-#undef VSC_DO_MAIN
-#undef L0
-#undef L1
-	memset(&w->stats, 0, sizeof w->stats);
-}
-
-void
-WRK_SumStat(struct worker *w)
-{
-
-	Lck_Lock(&wstat_mtx);
-	wrk_sumstat(w);
-	Lck_Unlock(&wstat_mtx);
-}
-
-int
-WRK_TrySumStat(struct worker *w)
-{
-	if (Lck_Trylock(&wstat_mtx))
-		return (0);
-	wrk_sumstat(w);
-	Lck_Unlock(&wstat_mtx);
-	return (1);
-}
-
-/*--------------------------------------------------------------------
- * Create and starte a back-ground thread which as its own worker and
- * session data structures;
- */
-
-struct bgthread {
-	unsigned	magic;
-#define BGTHREAD_MAGIC	0x23b5152b
-	const char	*name;
-	bgthread_t	*func;
-	void		*priv;
-};
-
-static void *
-wrk_bgthread(void *arg)
-{
-	struct bgthread *bt;
-	struct worker ww;
-	struct sess *sp;
-	uint32_t logbuf[1024];	/* XXX:  size ? */
-
-	CAST_OBJ_NOTNULL(bt, arg, BGTHREAD_MAGIC);
-	THR_SetName(bt->name);
-	sp = SES_Alloc();
-	XXXAN(sp);
-	memset(&ww, 0, sizeof ww);
-	sp->wrk = &ww;
-	ww.magic = WORKER_MAGIC;
-	ww.wlp = ww.wlb = logbuf;
-	ww.wle = logbuf + (sizeof logbuf) / 4;
-
-	(void)bt->func(sp, bt->priv);
-
-	WRONG("BgThread terminated");
-
-	NEEDLESS_RETURN(NULL);
-}
-
-void
-WRK_BgThread(pthread_t *thr, const char *name, bgthread_t *func, void *priv)
-{
-	struct bgthread *bt;
-
-	ALLOC_OBJ(bt, BGTHREAD_MAGIC);
-	AN(bt);
-
-	bt->name = name;
-	bt->func = func;
-	bt->priv = priv;
-	AZ(pthread_create(thr, NULL, wrk_bgthread, bt));
-}
-
-/*--------------------------------------------------------------------*/
-
-static void *
-wrk_thread_real(void *priv, unsigned shm_workspace, unsigned sess_workspace,
-    uint16_t nhttp, unsigned http_space, unsigned siov)
-{
-	struct worker *w, ww;
-	uint32_t wlog[shm_workspace / 4];
-	/* XXX: can we trust these to be properly aligned ? */
-	unsigned char ws[sess_workspace];
-	unsigned char http0[http_space];
-	unsigned char http1[http_space];
-	unsigned char http2[http_space];
-	struct iovec iov[siov];
-	struct SHA256Context sha256;
-
-	THR_SetName("cache-worker");
-	w = &ww;
-	memset(w, 0, sizeof *w);
-	w->magic = WORKER_MAGIC;
-	w->lastused = NAN;
-	w->wlb = w->wlp = wlog;
-	w->wle = wlog + (sizeof wlog) / 4;
-	w->sha256ctx = &sha256;
-	w->bereq = HTTP_create(http0, nhttp);
-	w->beresp = HTTP_create(http1, nhttp);
-	w->resp = HTTP_create(http2, nhttp);
-	w->wrw.iov = iov;
-	w->wrw.siov = siov;
-	w->wrw.ciov = siov;
-	AZ(pthread_cond_init(&w->cond, NULL));
-
-	WS_Init(w->ws, "wrk", ws, sess_workspace);
-
-	VSL(SLT_WorkThread, 0, "%p start", w);
-
-	Pool_Work_Thread(priv, w);
-	AZ(w->pool);
-
-	VSL(SLT_WorkThread, 0, "%p end", w);
-	if (w->vcl != NULL)
-		VCL_Rel(&w->vcl);
-	AZ(pthread_cond_destroy(&w->cond));
-	HSH_Cleanup(w);
-	WRK_SumStat(w);
-	return (NULL);
-}
-
-void *
-WRK_thread(void *priv)
-{
-	uint16_t nhttp;
-	unsigned siov;
-
-	assert(cache_param->http_max_hdr <= 65535);
-	/* We need to snapshot these two for consistency */
-	nhttp = (uint16_t)cache_param->http_max_hdr;
-	siov = nhttp * 2;
-	if (siov > IOV_MAX)
-		siov = IOV_MAX;
-	return (wrk_thread_real(priv,
-	    cache_param->shm_workspace,
-	    cache_param->wthread_workspace,
-	    nhttp, HTTP_estimate(nhttp), siov));
-}
-
-void
-WRK_Init(void)
-{
-	Lck_New(&wstat_mtx, lck_wstat);
-}
diff --git a/bin/varnishd/cache_wrw.c b/bin/varnishd/cache_wrw.c
deleted file mode 100644
index 2160f69..0000000
--- a/bin/varnishd/cache_wrw.c
+++ /dev/null
@@ -1,344 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * Write data to fd
- * We try to use writev() if possible in order to minimize number of
- * syscalls made and packets sent.  It also just might allow the worker
- * thread to complete the request without holding stuff locked.
- */
-
-#include "config.h"
-
-#include <sys/types.h>
-#ifdef SENDFILE_WORKS
-#  if defined(__FreeBSD__) || defined(__DragonFly__)
-#    include <sys/socket.h>
-#  elif defined(__linux__)
-#    include <sys/sendfile.h>
-#  elif defined(__sun)
-#    include <sys/sendfile.h>
-#  else
-#     error Unknown sendfile() implementation
-#  endif
-#endif /* SENDFILE_WORKS */
-#include <sys/uio.h>
-
-#include <stdio.h>
-
-#include "cache.h"
-#include "vtim.h"
-
-/*--------------------------------------------------------------------
- */
-
-int
-WRW_Error(const struct worker *w)
-{
-
-	return (w->wrw.werr);
-}
-
-void
-WRW_Reserve(struct worker *w, int *fd)
-{
-	struct wrw *wrw;
-
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	wrw = &w->wrw;
-	AZ(wrw->wfd);
-	wrw->werr = 0;
-	wrw->liov = 0;
-	wrw->niov = 0;
-	wrw->ciov = wrw->siov;
-	wrw->wfd = fd;
-}
-
-static void
-WRW_Release(struct worker *w)
-{
-	struct wrw *wrw;
-
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	wrw = &w->wrw;
-	AN(wrw->wfd);
-	wrw->werr = 0;
-	wrw->liov = 0;
-	wrw->niov = 0;
-	wrw->ciov = wrw->siov;
-	wrw->wfd = NULL;
-}
-
-unsigned
-WRW_Flush(struct worker *w)
-{
-	ssize_t i;
-	struct wrw *wrw;
-	char cbuf[32];
-
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	wrw = &w->wrw;
-	AN(wrw->wfd);
-
-	/* For chunked, there must be one slot reserved for the chunked tail */
-	if (wrw->ciov < wrw->siov)
-		assert(wrw->niov < wrw->siov);
-
-	if (*wrw->wfd >= 0 && wrw->liov > 0 && wrw->werr == 0) {
-		if (wrw->ciov < wrw->siov && wrw->cliov > 0) {
-			bprintf(cbuf, "00%jx\r\n", (intmax_t)wrw->cliov);
-			i = strlen(cbuf);
-			wrw->iov[wrw->ciov].iov_base = cbuf;
-			wrw->iov[wrw->ciov].iov_len = i;
-			wrw->liov += i;
-
-			wrw->iov[wrw->niov].iov_base = cbuf + i - 2;
-			wrw->iov[wrw->niov++].iov_len = 2;
-			wrw->liov += 2;
-		} else if (wrw->ciov < wrw->siov) {
-			wrw->iov[wrw->ciov].iov_base = cbuf;
-			wrw->iov[wrw->ciov].iov_len = 0;
-		}
-		i = writev(*wrw->wfd, wrw->iov, wrw->niov);
-		while (i != wrw->liov && i > 0) {
-			/* Remove sent data from start of I/O vector,
-			 * then retry; we hit a timeout, but some data
-			 * was sent.
-
-			 XXX: Add a "minimum sent data per timeout
-			 counter to prevent slowlaris attacks
-			*/
-			size_t used = 0;
-
-			if (VTIM_real() - w->sp->t_resp > cache_param->send_timeout) {
-				WSL(w, SLT_Debug, *wrw->wfd,
-				    "Hit total send timeout, wrote = %ld/%ld; not retrying",
-				    i, wrw->liov);
-				i = -1;
-				break;
-			}
-
-			WSL(w, SLT_Debug, *wrw->wfd,
-			    "Hit send timeout, wrote = %ld/%ld; retrying",
-			    i, wrw->liov);
-
-			for (int j = 0; j < wrw->niov; j++) {
-				if (used + wrw->iov[j].iov_len > i) {
-					/* Cutoff is in this iov */
-					int used_here = i - used;
-					wrw->iov[j].iov_len -= used_here;
-					wrw->iov[j].iov_base = (char*)wrw->iov[j].iov_base + used_here;
-					memmove(wrw->iov, &wrw->iov[j],
-						(wrw->niov - j) * sizeof(struct iovec));
-					wrw->niov -= j;
-					wrw->liov -= i;
-					break;
-				}
-				used += wrw->iov[j].iov_len;
-			}
-			i = writev(*wrw->wfd, wrw->iov, wrw->niov);
-		}
-		if (i <= 0) {
-			wrw->werr++;
-			WSL(w, SLT_Debug, *wrw->wfd,
-			    "Write error, retval = %d, len = %d, errno = %s",
-			    i, wrw->liov, strerror(errno));
-		}
-	}
-	wrw->liov = 0;
-	wrw->cliov = 0;
-	wrw->niov = 0;
-	if (wrw->ciov < wrw->siov)
-		wrw->ciov = wrw->niov++;
-	return (wrw->werr);
-}
-
-unsigned
-WRW_FlushRelease(struct worker *w)
-{
-	unsigned u;
-
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	AN(w->wrw.wfd);
-	u = WRW_Flush(w);
-	WRW_Release(w);
-	return (u);
-}
-
-unsigned
-WRW_WriteH(struct worker *w, const txt *hh, const char *suf)
-{
-	unsigned u;
-
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	AN(w->wrw.wfd);
-	AN(w);
-	AN(hh);
-	AN(hh->b);
-	AN(hh->e);
-	u = WRW_Write(w, hh->b, hh->e - hh->b);
-	if (suf != NULL)
-		u += WRW_Write(w, suf, -1);
-	return (u);
-}
-
-unsigned
-WRW_Write(struct worker *w, const void *ptr, int len)
-{
-	struct wrw *wrw;
-
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	wrw = &w->wrw;
-	AN(wrw->wfd);
-	if (len == 0 || *wrw->wfd < 0)
-		return (0);
-	if (len == -1)
-		len = strlen(ptr);
-	if (wrw->niov >= wrw->siov - (wrw->ciov < wrw->siov ? 1 : 0))
-		(void)WRW_Flush(w);
-	wrw->iov[wrw->niov].iov_base = TRUST_ME(ptr);
-	wrw->iov[wrw->niov].iov_len = len;
-	wrw->liov += len;
-	wrw->niov++;
-	if (wrw->ciov < wrw->siov) {
-		assert(wrw->niov < wrw->siov);
-		wrw->cliov += len;
-	}
-	return (len);
-}
-
-void
-WRW_Chunked(struct worker *w)
-{
-	struct wrw *wrw;
-
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	wrw = &w->wrw;
-
-	assert(wrw->ciov == wrw->siov);
-	/*
-	 * If there are not space for chunked header, a chunk of data and
-	 * a chunk tail, we might as well flush right away.
-	 */
-	if (wrw->niov + 3 >= wrw->siov)
-		(void)WRW_Flush(w);
-	wrw->ciov = wrw->niov++;
-	wrw->cliov = 0;
-	assert(wrw->ciov < wrw->siov);
-	assert(wrw->niov < wrw->siov);
-}
-
-/*
- * XXX: It is not worth the complexity to attempt to get the
- * XXX: end of chunk into the WRW_Flush(), because most of the time
- * XXX: if not always, that is a no-op anyway, because the calling
- * XXX: code already called WRW_Flush() to release local storage.
- */
-
-void
-WRW_EndChunk(struct worker *w)
-{
-	struct wrw *wrw;
-
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	wrw = &w->wrw;
-
-	assert(wrw->ciov < wrw->siov);
-	(void)WRW_Flush(w);
-	wrw->ciov = wrw->siov;
-	wrw->niov = 0;
-	wrw->cliov = 0;
-	(void)WRW_Write(w, "0\r\n\r\n", -1);
-}
-
-
-#ifdef SENDFILE_WORKS
-void
-WRW_Sendfile(struct worker *w, int fd, off_t off, unsigned len)
-{
-	struct wrw *wrw;
-
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	wrw = &w->wrw;
-	AN(wrw->wfd);
-	assert(fd >= 0);
-	assert(len > 0);
-
-#if defined(__FreeBSD__) || defined(__DragonFly__)
-	do {
-		struct sf_hdtr sfh;
-		memset(&sfh, 0, sizeof sfh);
-		if (wrw->niov > 0) {
-			sfh.headers = wrw->iov;
-			sfh.hdr_cnt = wrw->niov;
-		}
-		if (sendfile(fd, *wrw->wfd, off, len, &sfh, NULL, 0) != 0)
-			wrw->werr++;
-		wrw->liov = 0;
-		wrw->niov = 0;
-	} while (0);
-#elif defined(__linux__)
-	do {
-		if (WRW_Flush(w) == 0 &&
-		    sendfile(*wrw->wfd, fd, &off, len) != len)
-			wrw->werr++;
-	} while (0);
-#elif defined(__sun) && defined(HAVE_SENDFILEV)
-	do {
-		sendfilevec_t svvec[cache_param->http_headers * 2 + 1];
-		size_t xferred = 0, expected = 0;
-		int i;
-		for (i = 0; i < wrw->niov; i++) {
-			svvec[i].sfv_fd = SFV_FD_SELF;
-			svvec[i].sfv_flag = 0;
-			svvec[i].sfv_off = (off_t) wrw->iov[i].iov_base;
-			svvec[i].sfv_len = wrw->iov[i].iov_len;
-			expected += svvec[i].sfv_len;
-		}
-		svvec[i].sfv_fd = fd;
-		svvec[i].sfv_flag = 0;
-		svvec[i].sfv_off = off;
-		svvec[i].sfv_len = len;
-		expected += svvec[i].sfv_len;
-		if (sendfilev(*wrw->wfd, svvec, i, &xferred) == -1 ||
-		    xferred != expected)
-			wrw->werr++;
-		wrw->liov = 0;
-		wrw->niov = 0;
-	} while (0);
-#elif defined(__sun) && defined(HAVE_SENDFILE)
-	do {
-		if (WRW_Flush(w) == 0 &&
-		    sendfile(*wrw->wfd, fd, &off, len) != len)
-			wrw->werr++;
-	} while (0);
-#else
-#error Unknown sendfile() implementation
-#endif
-}
-#endif /* SENDFILE_WORKS */
-
diff --git a/bin/varnishd/cache_ws.c b/bin/varnishd/cache_ws.c
deleted file mode 100644
index 9fca215..0000000
--- a/bin/varnishd/cache_ws.c
+++ /dev/null
@@ -1,212 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2010 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- */
-
-#include "config.h"
-
-#include "cache.h"
-
-void
-WS_Assert(const struct ws *ws)
-{
-
-	CHECK_OBJ_NOTNULL(ws, WS_MAGIC);
-	DSL(0x02, SLT_Debug, 0, "WS(%p = (%s, %p %u %u %u)",
-	    ws, ws->id, ws->s, pdiff(ws->s, ws->f),
-	    ws->r == NULL ? 0 : pdiff(ws->f, ws->r),
-	    pdiff(ws->s, ws->e));
-	assert(ws->s != NULL);
-	assert(PAOK(ws->s));
-	assert(ws->e != NULL);
-	assert(PAOK(ws->e));
-	assert(ws->s < ws->e);
-	assert(ws->f >= ws->s);
-	assert(ws->f <= ws->e);
-	assert(PAOK(ws->f));
-	if (ws->r) {
-		assert(ws->r > ws->s);
-		assert(ws->r <= ws->e);
-		assert(PAOK(ws->r));
-	}
-}
-
-void
-WS_Init(struct ws *ws, const char *id, void *space, unsigned len)
-{
-
-	DSL(0x02, SLT_Debug, 0,
-	    "WS_Init(%p, \"%s\", %p, %u)", ws, id, space, len);
-	assert(space != NULL);
-	memset(ws, 0, sizeof *ws);
-	ws->magic = WS_MAGIC;
-	ws->s = space;
-	assert(PAOK(space));
-	ws->e = ws->s + len;
-	assert(PAOK(len));
-	ws->f = ws->s;
-	ws->id = id;
-	WS_Assert(ws);
-}
-
-/*
- * Reset a WS to start or a given pointer, likely from WS_Snapshot
- */
-
-void
-WS_Reset(struct ws *ws, char *p)
-{
-
-	WS_Assert(ws);
-	DSL(0x02, SLT_Debug, 0, "WS_Reset(%p, %p)", ws, p);
-	assert(ws->r == NULL);
-	if (p == NULL)
-		ws->f = ws->s;
-	else {
-		assert(p >= ws->s);
-		assert(p < ws->e);
-		ws->f = p;
-	}
-	WS_Assert(ws);
-}
-
-char *
-WS_Alloc(struct ws *ws, unsigned bytes)
-{
-	char *r;
-
-	WS_Assert(ws);
-	bytes = PRNDUP(bytes);
-
-	assert(ws->r == NULL);
-	if (ws->f + bytes > ws->e) {
-		ws->overflow++;
-		WS_Assert(ws);
-		return(NULL);
-	}
-	r = ws->f;
-	ws->f += bytes;
-	DSL(0x02, SLT_Debug, 0, "WS_Alloc(%p, %u) = %p", ws, bytes, r);
-	WS_Assert(ws);
-	return (r);
-}
-
-char *
-WS_Dup(struct ws *ws, const char *s)
-{
-	unsigned l;
-	char *p;
-
-	WS_Assert(ws);
-	l = strlen(s) + 1;
-	p = WS_Alloc(ws, l);
-	if (p != NULL)
-		memcpy(p, s, l);
-	DSL(0x02, SLT_Debug, 0, "WS_Dup(%p, \"%s\") = %p", ws, s, p);
-	WS_Assert(ws);
-	return (p);
-}
-
-unsigned
-WS_Free(const struct ws *ws)
-{
-
-	WS_Assert(ws);
-	return(ws->e - ws->f);
-}
-
-char *
-WS_Snapshot(struct ws *ws)
-{
-
-	WS_Assert(ws);
-	assert(ws->r == NULL);
-	DSL(0x02, SLT_Debug, 0, "WS_Snapshot(%p) = %p", ws, ws->f);
-	return (ws->f);
-}
-
-unsigned
-WS_Reserve(struct ws *ws, unsigned bytes)
-{
-	unsigned b2;
-
-	WS_Assert(ws);
-	assert(ws->r == NULL);
-	if (bytes == 0)
-		b2 = ws->e - ws->f;
-	else if (bytes > ws->e - ws->f)
-		b2 = ws->e - ws->f;
-	else
-		b2 = bytes;
-	b2 = PRNDDN(b2);
-	xxxassert(ws->f + b2 <= ws->e);
-	ws->r = ws->f + b2;
-	DSL(0x02, SLT_Debug, 0, "WS_Reserve(%p, %u/%u) = %u",
-	    ws, b2, bytes, pdiff(ws->f, ws->r));
-	WS_Assert(ws);
-	return (pdiff(ws->f, ws->r));
-}
-
-void
-WS_Release(struct ws *ws, unsigned bytes)
-{
-	WS_Assert(ws);
-	bytes = PRNDUP(bytes);
-	assert(bytes <= ws->e - ws->f);
-	DSL(0x02, SLT_Debug, 0, "WS_Release(%p, %u)", ws, bytes);
-	assert(ws->r != NULL);
-	assert(ws->f + bytes <= ws->r);
-	ws->f += bytes;
-	ws->r = NULL;
-	WS_Assert(ws);
-}
-
-void
-WS_ReleaseP(struct ws *ws, char *ptr)
-{
-	WS_Assert(ws);
-	DSL(0x02, SLT_Debug, 0, "WS_ReleaseP(%p, %p)", ws, ptr);
-	assert(ws->r != NULL);
-	assert(ptr >= ws->f);
-	assert(ptr <= ws->r);
-	ws->f += PRNDUP(ptr - ws->f);
-	ws->r = NULL;
-	WS_Assert(ws);
-}
-
-#if 0
-/* XXX: not used anywhere (yet) */
-void
-WS_Return(struct ws *ws, char *s, char *e)
-{
-
-	WS_Assert(ws);
-	if (e == ws->f)
-		ws->f = s;
-}
-#endif
diff --git a/bin/varnishd/hash/hash_classic.c b/bin/varnishd/hash/hash_classic.c
index 6b1d589..9ea27de 100644
--- a/bin/varnishd/hash/hash_classic.c
+++ b/bin/varnishd/hash/hash_classic.c
@@ -34,7 +34,7 @@
 #include <stdio.h>
 #include <stdlib.h>
 
-#include "cache.h"
+#include "cache/cache.h"
 
 #include "hash/hash_slinger.h"
 
diff --git a/bin/varnishd/hash/hash_critbit.c b/bin/varnishd/hash/hash_critbit.c
index fd50b26..ecb7b62 100644
--- a/bin/varnishd/hash/hash_critbit.c
+++ b/bin/varnishd/hash/hash_critbit.c
@@ -34,7 +34,7 @@
 
 #include <stdlib.h>
 
-#include "cache.h"
+#include "cache/cache.h"
 
 #include "hash/hash_slinger.h"
 #include "vcli_priv.h"
diff --git a/bin/varnishd/hash/hash_simple_list.c b/bin/varnishd/hash/hash_simple_list.c
index 1a0cae5..4bdae02 100644
--- a/bin/varnishd/hash/hash_simple_list.c
+++ b/bin/varnishd/hash/hash_simple_list.c
@@ -31,7 +31,7 @@
 
 #include "config.h"
 
-#include "cache.h"
+#include "cache/cache.h"
 
 #include "hash/hash_slinger.h"
 
diff --git a/bin/varnishd/storage/stevedore.c b/bin/varnishd/storage/stevedore.c
index b119c06..3978f64 100644
--- a/bin/varnishd/storage/stevedore.c
+++ b/bin/varnishd/storage/stevedore.c
@@ -36,7 +36,7 @@
 #include <stdio.h>
 #include <stdlib.h>
 
-#include "cache.h"
+#include "cache/cache.h"
 
 #include "storage/storage.h"
 #include "vav.h"
diff --git a/bin/varnishd/storage/storage_file.c b/bin/varnishd/storage/storage_file.c
index 981d43e..2d9ccf4 100644
--- a/bin/varnishd/storage/storage_file.c
+++ b/bin/varnishd/storage/storage_file.c
@@ -36,7 +36,7 @@
 #include <stdio.h>
 #include <stdlib.h>
 
-#include "cache.h"
+#include "cache/cache.h"
 #include "storage/storage.h"
 
 #include "vnum.h"
diff --git a/bin/varnishd/storage/storage_malloc.c b/bin/varnishd/storage/storage_malloc.c
index 03d6a55..156c832 100644
--- a/bin/varnishd/storage/storage_malloc.c
+++ b/bin/varnishd/storage/storage_malloc.c
@@ -34,7 +34,7 @@
 #include <stdio.h>
 #include <stdlib.h>
 
-#include "cache.h"
+#include "cache/cache.h"
 #include "storage/storage.h"
 
 #include "vnum.h"
diff --git a/bin/varnishd/storage/storage_persistent.c b/bin/varnishd/storage/storage_persistent.c
index 02d244e..ded638b 100644
--- a/bin/varnishd/storage/storage_persistent.c
+++ b/bin/varnishd/storage/storage_persistent.c
@@ -43,7 +43,7 @@
 #include <stdlib.h>
 #include <string.h>
 
-#include "cache.h"
+#include "cache/cache.h"
 #include "storage/storage.h"
 
 #include "hash/hash_slinger.h"
diff --git a/bin/varnishd/storage/storage_persistent_mgt.c b/bin/varnishd/storage/storage_persistent_mgt.c
index 1631ea6..bb0eb38 100644
--- a/bin/varnishd/storage/storage_persistent_mgt.c
+++ b/bin/varnishd/storage/storage_persistent_mgt.c
@@ -41,7 +41,7 @@
 #include <stdio.h>
 #include <stdlib.h>
 
-#include "cache.h"
+#include "cache/cache.h"
 #include "storage/storage.h"
 
 #include "vsha256.h"
diff --git a/bin/varnishd/storage/storage_persistent_silo.c b/bin/varnishd/storage/storage_persistent_silo.c
index 27f0c98..b7ef952 100644
--- a/bin/varnishd/storage/storage_persistent_silo.c
+++ b/bin/varnishd/storage/storage_persistent_silo.c
@@ -37,7 +37,7 @@
 #include <stdio.h>
 #include <stdlib.h>
 
-#include "cache.h"
+#include "cache/cache.h"
 #include "storage/storage.h"
 
 #include "hash/hash_slinger.h"
diff --git a/bin/varnishd/storage/storage_persistent_subr.c b/bin/varnishd/storage/storage_persistent_subr.c
index 2066e72..004bba3 100644
--- a/bin/varnishd/storage/storage_persistent_subr.c
+++ b/bin/varnishd/storage/storage_persistent_subr.c
@@ -41,7 +41,7 @@
 #include <stdio.h>
 #include <stdlib.h>
 
-#include "cache.h"
+#include "cache/cache.h"
 #include "storage/storage.h"
 
 #include "vsha256.h"
diff --git a/bin/varnishd/storage/storage_synth.c b/bin/varnishd/storage/storage_synth.c
index 0519738..e9e9b2f 100644
--- a/bin/varnishd/storage/storage_synth.c
+++ b/bin/varnishd/storage/storage_synth.c
@@ -32,7 +32,7 @@
 
 #include <stdlib.h>
 
-#include "cache.h"
+#include "cache/cache.h"
 #include "storage/storage.h"
 
 
diff --git a/bin/varnishd/storage/storage_umem.c b/bin/varnishd/storage/storage_umem.c
index 78110d4..52d238d 100644
--- a/bin/varnishd/storage/storage_umem.c
+++ b/bin/varnishd/storage/storage_umem.c
@@ -39,7 +39,7 @@
 #include <stdlib.h>
 #include <umem.h>
 
-#include "cache.h"
+#include "cache/cache.h"
 #include "storage/storage.h"
 
 static size_t			smu_max = SIZE_MAX;
diff --git a/bin/varnishd/waiter/cache_waiter.c b/bin/varnishd/waiter/cache_waiter.c
index e6654fd..c428294 100644
--- a/bin/varnishd/waiter/cache_waiter.c
+++ b/bin/varnishd/waiter/cache_waiter.c
@@ -30,7 +30,7 @@
 
 #include "config.h"
 
-#include "cache.h"
+#include "cache/cache.h"
 
 #include "waiter/cache_waiter.h"
 #include "vcli.h"
diff --git a/bin/varnishd/waiter/cache_waiter_epoll.c b/bin/varnishd/waiter/cache_waiter_epoll.c
index 46cb288..b4b40f5 100644
--- a/bin/varnishd/waiter/cache_waiter_epoll.c
+++ b/bin/varnishd/waiter/cache_waiter_epoll.c
@@ -40,7 +40,7 @@
 #include <fcntl.h>
 #include <stdlib.h>
 
-#include "cache.h"
+#include "cache/cache.h"
 
 #include "waiter/cache_waiter.h"
 #include "vtim.h"
diff --git a/bin/varnishd/waiter/cache_waiter_kqueue.c b/bin/varnishd/waiter/cache_waiter_kqueue.c
index a631606..c6a03de 100644
--- a/bin/varnishd/waiter/cache_waiter_kqueue.c
+++ b/bin/varnishd/waiter/cache_waiter_kqueue.c
@@ -42,7 +42,7 @@
 #include <stdlib.h>
 #include <unistd.h>
 
-#include "cache.h"
+#include "cache/cache.h"
 
 #include "waiter/cache_waiter.h"
 #include "vtim.h"
diff --git a/bin/varnishd/waiter/cache_waiter_poll.c b/bin/varnishd/waiter/cache_waiter_poll.c
index 5f8dbd7..3f5cea3 100644
--- a/bin/varnishd/waiter/cache_waiter_poll.c
+++ b/bin/varnishd/waiter/cache_waiter_poll.c
@@ -33,7 +33,7 @@
 #include <poll.h>
 #include <stdlib.h>
 
-#include "cache.h"
+#include "cache/cache.h"
 
 #include "waiter/cache_waiter.h"
 #include "vtim.h"
diff --git a/bin/varnishd/waiter/cache_waiter_ports.c b/bin/varnishd/waiter/cache_waiter_ports.c
index aa9f86d..a1699d0 100644
--- a/bin/varnishd/waiter/cache_waiter_ports.c
+++ b/bin/varnishd/waiter/cache_waiter_ports.c
@@ -40,7 +40,7 @@
 #include <stdio.h>
 #include <stdlib.h>
 
-#include "cache.h"
+#include "cache/cache.h"
 
 #include "waiter/cache_waiter.h"
 #include "vtim.h"
diff --git a/lib/libvmod_std/Makefile.am b/lib/libvmod_std/Makefile.am
index c9979fa..065948a 100644
--- a/lib/libvmod_std/Makefile.am
+++ b/lib/libvmod_std/Makefile.am
@@ -1,6 +1,9 @@
 #
 
-INCLUDES = -I$(top_srcdir)/include -I$(top_builddir)/include
+INCLUDES = \
+	-I$(top_srcdir)/include \
+	-I$(top_srcdir)/bin/varnishd \
+	-I$(top_builddir)/include
 
 dist_man_MANS = vmod_std.3
 
diff --git a/lib/libvmod_std/vmod_std.c b/lib/libvmod_std/vmod_std.c
index f54958f..14d8503 100644
--- a/lib/libvmod_std/vmod_std.c
+++ b/lib/libvmod_std/vmod_std.c
@@ -39,7 +39,7 @@
 #include "vrt.h"
 #include "vtcp.h"
 
-#include "../../bin/varnishd/cache.h"
+#include "cache/cache.h"
 
 #include "vcc_if.h"
 
diff --git a/lib/libvmod_std/vmod_std_conversions.c b/lib/libvmod_std/vmod_std_conversions.c
index fe8ae96..149be81 100644
--- a/lib/libvmod_std/vmod_std_conversions.c
+++ b/lib/libvmod_std/vmod_std_conversions.c
@@ -33,7 +33,7 @@
 #include <math.h>
 #include <stdlib.h>
 
-#include "../../bin/varnishd/cache.h"
+#include "cache/cache.h"
 
 #include "vrt.h"
 #include "vcc_if.h"
diff --git a/lib/libvmod_std/vmod_std_fileread.c b/lib/libvmod_std/vmod_std_fileread.c
index 8476bd6..89e4f50 100644
--- a/lib/libvmod_std/vmod_std_fileread.c
+++ b/lib/libvmod_std/vmod_std_fileread.c
@@ -43,7 +43,7 @@
 #include <stdint.h>
 #include <stdlib.h>
 
-#include "../../bin/varnishd/cache.h"
+#include "cache/cache.h"
 
 #include "vrt.h"
 #include "vfil.h"



From phk at varnish-cache.org  Sun Nov 13 21:59:20 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Sun, 13 Nov 2011 22:59:20 +0100
Subject: [master] 6298611 Update to match new layout
Message-ID: <E1RPi5A-0008Bh-0n@project.varnish-software.com>

commit 629861111eeb9260428573c3ef9eceff52a67cf0
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Nov 13 11:05:45 2011 +0000

    Update to match new layout

diff --git a/bin/varnishd/flint.sh b/bin/varnishd/flint.sh
index 756f65a..9067263 100755
--- a/bin/varnishd/flint.sh
+++ b/bin/varnishd/flint.sh
@@ -15,7 +15,7 @@ flexelint \
 	-I../.. \
 	-I/usr/local/include \
 	-DVARNISH_STATE_DIR=\"foo\" \
-	*.c \
+	cache/*.c \
 	common/*.c \
 	storage/*.c \
 	waiter/*.c \



From phk at varnish-cache.org  Sun Nov 13 21:59:20 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Sun, 13 Nov 2011 22:59:20 +0100
Subject: [master] 10bfefc Rewrite VCLI_ReadResult() to be much more robust.
Message-ID: <E1RPi5A-0008Bm-63@project.varnish-software.com>

commit 10bfefcaf40f4fd3302bc29237e263b99f590d6e
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Nov 13 21:58:52 2011 +0000

    Rewrite VCLI_ReadResult() to be much more robust.

diff --git a/lib/libvarnish/cli_common.c b/lib/libvarnish/cli_common.c
index 43b91c3..3a44e9f 100644
--- a/lib/libvarnish/cli_common.c
+++ b/lib/libvarnish/cli_common.c
@@ -121,7 +121,7 @@ read_tmo(int fd, char *ptr, unsigned len, double tmo)
 	struct pollfd pfd;
 
 	if (tmo > 0)
-		to = tmo * 1e3;
+		to = (int)(tmo * 1e3);
 	else
 		to = -1;
 	pfd.fd = fd;
@@ -150,44 +150,56 @@ VCLI_ReadResult(int fd, unsigned *status, char **ptr, double tmo)
 	char res[CLI_LINE0_LEN];	/* For NUL */
 	int i, j;
 	unsigned u, v, s;
-	char *p;
+	char *p = NULL;
+	const char *err = "CLI communication error (hdr)";
 
 	if (status == NULL)
 		status = &s;
 	if (ptr != NULL)
 		*ptr = NULL;
-	i = read_tmo(fd, res, CLI_LINE0_LEN, tmo);
-	if (i != CLI_LINE0_LEN) {
-		*status = CLIS_COMMS;
-		if (ptr != NULL)
-			*ptr = strdup("CLI communication error (hdr)");
-		if (i != 0)
-			return (i);
-		return (*status);
-	}
-	assert(i == CLI_LINE0_LEN);
-	assert(res[3] == ' ');
-	assert(res[CLI_LINE0_LEN - 1] == '\n');
-	res[CLI_LINE0_LEN - 1] = '\0';
-	j = sscanf(res, "%u %u\n", &u, &v);
-	assert(j == 2);
-	*status = u;
-	p = malloc(v + 1L);
-	assert(p != NULL);
-	i = read_tmo(fd, p, v + 1, tmo);
-	if (i < 0) {
-		*status = CLIS_COMMS;
-		free(p);
-		if (ptr != NULL)
-			*ptr = strdup("CLI communication error (body)");
-		return (i);
-	}
-	assert(i == v + 1);
-	assert(p[v] == '\n');
-	p[v] = '\0';
-	if (ptr == NULL)
+	do {
+		i = read_tmo(fd, res, CLI_LINE0_LEN, tmo);
+		if (i != CLI_LINE0_LEN)
+			break;
+
+		if (res[3] != ' ')
+			break;
+
+		if (res[CLI_LINE0_LEN - 1] != '\n')
+			break;
+
+		res[CLI_LINE0_LEN - 1] = '\0';
+		j = sscanf(res, "%u %u\n", &u, &v);
+		if (j != 2)
+			break;
+
+		err = "CLI communication error (body)";
+
+		*status = u;
+		p = malloc(v + 1L);
+		if (p == NULL)
+			break;
+
+		i = read_tmo(fd, p, v + 1, tmo);
+		if (i < 0)
+			break;
+		if (i != v + 1)
+			break;
+		if (p[v] != '\n')
+			break;
+		
+		p[v] = '\0';
+		if (ptr == NULL)
+			free(p);
+		else
+			*ptr = p;
+		return (0);
+	} while(0);
+
+	if (p != NULL)
 		free(p);
-	else
-		*ptr = p;
-	return (0);
+	*status = CLIS_COMMS;
+	if (ptr != NULL)
+		*ptr = strdup(err);
+	return (*status);
 }



From phk at varnish-cache.org  Sun Nov 13 22:03:11 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Sun, 13 Nov 2011 23:03:11 +0100
Subject: [master] 2144dc7 Use better criteria for determining if child CLI
	connection is hosed.
Message-ID: <E1RPi8t-0000Ac-U9@project.varnish-software.com>

commit 2144dc7885a425926be4a74ecb89c5b38d062ac8
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Nov 13 22:02:44 2011 +0000

    Use better criteria for determining if child CLI connection is hosed.

diff --git a/bin/varnishd/mgt/mgt_cli.c b/bin/varnishd/mgt/mgt_cli.c
index 3a6f365..70e8557 100644
--- a/bin/varnishd/mgt/mgt_cli.c
+++ b/bin/varnishd/mgt/mgt_cli.c
@@ -171,7 +171,8 @@ mcf_askchild(struct cli *cli, const char * const *av, void *priv)
 		return;
 	}
 	VSB_delete(vsb);
-	(void)VCLI_ReadResult(cli_i, &u, &q, mgt_param.cli_timeout);
+	if (VCLI_ReadResult(cli_i, &u, &q, mgt_param.cli_timeout))
+		MGT_Child_Cli_Fail();
 	VCLI_SetResult(cli, u);
 	VCLI_Out(cli, "%s", q);
 	free(q);
@@ -220,11 +221,10 @@ mgt_cli_askchild(unsigned *status, char **resp, const char *fmt, ...) {
 		return (CLIS_COMMS);
 	}
 
-	(void)VCLI_ReadResult(cli_i, &u, resp, mgt_param.cli_timeout);
+	if (VCLI_ReadResult(cli_i, &u, resp, mgt_param.cli_timeout))
+		MGT_Child_Cli_Fail();
 	if (status != NULL)
 		*status = u;
-	if (u == CLIS_COMMS)
-		MGT_Child_Cli_Fail();
 	return (u == CLIS_OK ? 0 : u);
 }
 



From phk at varnish-cache.org  Sun Nov 13 22:08:51 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Sun, 13 Nov 2011 23:08:51 +0100
Subject: [master] 0af2c24 Make sure the CLI headers we produce tell the truth
	about the length
Message-ID: <E1RPiEN-0000fD-Jv@project.varnish-software.com>

commit 0af2c245047f904d91d800b659f62c3af2908413
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Nov 13 22:08:33 2011 +0000

    Make sure the CLI headers we produce tell the truth about the length

diff --git a/lib/libvarnish/cli_common.c b/lib/libvarnish/cli_common.c
index 3a44e9f..4a17de5 100644
--- a/lib/libvarnish/cli_common.c
+++ b/lib/libvarnish/cli_common.c
@@ -87,6 +87,7 @@ VCLI_WriteResult(int fd, unsigned status, const char *result)
 	int i, l;
 	struct iovec iov[3];
 	char nl[2] = "\n";
+	size_t len;
 	char res[CLI_LINE0_LEN + 2];	/*
 					 * NUL + one more so we can catch
 					 * any misformats by snprintf
@@ -95,15 +96,18 @@ VCLI_WriteResult(int fd, unsigned status, const char *result)
 	assert(status >= 100);
 	assert(status <= 999);		/*lint !e650 const out of range */
 
+	len = strlen(result);
+
 	i = snprintf(res, sizeof res,
-	    "%-3d %-8jd\n", status, (intmax_t)strlen(result));
+	    "%-3d %-8zd\n", status, len);
 	assert(i == CLI_LINE0_LEN);
+	assert(strtoul(res + 3, NULL, 10) == len);
 
 	iov[0].iov_base = res;
 	iov[0].iov_len = CLI_LINE0_LEN;
 
 	iov[1].iov_base = (void*)(uintptr_t)result;	/* TRUST ME */
-	iov[1].iov_len = strlen(result);
+	iov[1].iov_len = len;
 
 	iov[2].iov_base = nl;
 	iov[2].iov_len = 1;



From phk at varnish-cache.org  Mon Nov 14 10:21:31 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 14 Nov 2011 11:21:31 +0100
Subject: [master] ccf9f85 The CLI protocol and its implementation is not
	designed for transferring huge amounts of data,
	and it should not be used that way.
Message-ID: <E1RPtfP-0003dt-T4@project.varnish-software.com>

commit ccf9f85a217bb240550e9c2fac69c25ab23d8187
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Nov 14 10:18:18 2011 +0000

    The CLI protocol and its implementation is not designed for
    transferring huge amounts of data, and it should not be used that
    way.
    
    But we have commands like ban.list which potentially can return at
    metric shitload of data and this is a problem.
    
    This commit implements a cli_limit paramter which (with its upper
    limit) protects from responses that cannot be moved by the CLI
    protocol no matter what.
    
    We set the default to a low 4096 bytes, in order to make people
    specifically shoot their feet off if they really want to see 200k
    bans.

diff --git a/bin/varnishd/cache/cache_cli.c b/bin/varnishd/cache/cache_cli.c
index f29f86a..f299bef 100644
--- a/bin/varnishd/cache/cache_cli.c
+++ b/bin/varnishd/cache/cache_cli.c
@@ -236,7 +236,8 @@ CLI_Init(void)
 	Lck_New(&cli_mtx, lck_cli);
 	cli_thread = pthread_self();
 
-	cls = VCLS_New(cli_cb_before, cli_cb_after, cache_param->cli_buffer);
+	cls = VCLS_New(cli_cb_before, cli_cb_after,
+	    &cache_param->cli_buffer, &cache_param->cli_limit);
 	AN(cls);
 
 	CLI_AddFuncs(master_cmds);
diff --git a/bin/varnishd/common/params.h b/bin/varnishd/common/params.h
index fb71e33..ee8bad2 100644
--- a/bin/varnishd/common/params.h
+++ b/bin/varnishd/common/params.h
@@ -107,6 +107,7 @@ struct params {
 
 	/* CLI related */
 	unsigned		cli_timeout;
+	unsigned		cli_limit;
 	unsigned		ping_interval;
 
 	/* LRU list ordering interval */
diff --git a/bin/varnishd/mgt/mgt_cli.c b/bin/varnishd/mgt/mgt_cli.c
index 70e8557..1d46d82 100644
--- a/bin/varnishd/mgt/mgt_cli.c
+++ b/bin/varnishd/mgt/mgt_cli.c
@@ -336,7 +336,8 @@ static void
 mgt_cli_init_cls(void)
 {
 
-	cls = VCLS_New(mgt_cli_cb_before, mgt_cli_cb_after, mgt_param.cli_buffer);
+	cls = VCLS_New(mgt_cli_cb_before, mgt_cli_cb_after,
+	    &mgt_param.cli_buffer, &mgt_param.cli_limit);
 	AN(cls);
 	AZ(VCLS_AddFunc(cls, MCF_NOAUTH, cli_auth));
 	AZ(VCLS_AddFunc(cls, MCF_AUTH, cli_proto));
diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c
index 981841b..8a9e1fc 100644
--- a/bin/varnishd/mgt/mgt_main.c
+++ b/bin/varnishd/mgt/mgt_main.c
@@ -349,6 +349,7 @@ main(int argc, char * const *argv)
 	struct cli cli[1];
 	struct vpf_fh *pfh = NULL;
 	char *dirname;
+	unsigned clilim;
 
 	/*
 	 * Start out by closing all unwanted file descriptors we might
@@ -389,9 +390,12 @@ main(int argc, char * const *argv)
 	SHA256_Test();
 
 	memset(cli, 0, sizeof cli);
+	cli[0].magic = CLI_MAGIC;
 	cli[0].sb = VSB_new_auto();
 	XXXAN(cli[0].sb);
 	cli[0].result = CLIS_OK;
+	clilim = 32768;
+	cli[0].limit = &clilim;
 
 	VTAILQ_INIT(&heritage.socks);
 
diff --git a/bin/varnishd/mgt/mgt_param.c b/bin/varnishd/mgt/mgt_param.c
index 8050d08..afaab84 100644
--- a/bin/varnishd/mgt/mgt_param.c
+++ b/bin/varnishd/mgt/mgt_param.c
@@ -673,6 +673,19 @@ static const struct parspec input_parspec[] = {
 		"Listen queue depth.",
 		MUST_RESTART,
 		"1024", "connections" },
+	{ "cli_buffer", tweak_uint, &mgt_param.cli_buffer, 4096, UINT_MAX,
+		"Size of buffer for CLI command input."
+		"\nYou may need to increase this if you have big VCL files "
+		"and use the vcl.inline CLI command.\n"
+		"NB: Must be specified with -p to have effect.\n",
+		0,
+		"8192", "bytes" },
+	{ "cli_limit", tweak_uint, &mgt_param.cli_limit, 128, 99999999,
+		"Maximum size of CLI response.  If the response exceeds"
+		" this limit, the reponse code will be 201 instead of"
+		" 200 and the last line will indicate the truncation.",
+		0,
+		"4096", "bytes" },
 	{ "cli_timeout", tweak_timeout, &mgt_param.cli_timeout, 0, 0,
 		"Timeout for the childs replies to CLI requests from "
 		"the mgt_param.",
@@ -805,13 +818,6 @@ static const struct parspec input_parspec[] = {
 		"more sessions take a detour around the waiter.",
 		EXPERIMENTAL,
 		"50", "ms" },
-	{ "cli_buffer", tweak_uint, &mgt_param.cli_buffer, 4096, UINT_MAX,
-		"Size of buffer for CLI input."
-		"\nYou may need to increase this if you have big VCL files "
-		"and use the vcl.inline CLI command.\n"
-		"NB: Must be specified with -p to have effect.\n",
-		0,
-		"8192", "bytes" },
 	{ "log_hashstring", tweak_bool, &mgt_param.log_hash, 0, 0,
 		"Log the hash string components to shared memory log.\n",
 		0,
diff --git a/bin/varnishtest/tests/b00008.vtc b/bin/varnishtest/tests/b00008.vtc
index 1e9aa9d..cee8d4c 100644
--- a/bin/varnishtest/tests/b00008.vtc
+++ b/bin/varnishtest/tests/b00008.vtc
@@ -33,3 +33,7 @@ varnish v1 -cliok "help"
 varnish v1 -cliok "param.set waiter default"
 
 varnish v1 -clierr 106 "param.set waiter HASH(0x8839c4c)"
+
+varnish v1 -cliok "param.set cli_limit 128"
+
+varnish v1 -clierr 201 "param.show"
diff --git a/include/vcli.h b/include/vcli.h
index cd98f58..04ac11c 100644
--- a/include/vcli.h
+++ b/include/vcli.h
@@ -199,6 +199,7 @@ enum VCLI_status_e {
 	CLIS_PARAM	= 106,
 	CLIS_AUTH	= 107,
 	CLIS_OK		= 200,
+	CLIS_TRUNCATED	= 201,
 	CLIS_CANT	= 300,
 	CLIS_COMMS	= 400,
 	CLIS_CLOSE	= 500
diff --git a/include/vcli_common.h b/include/vcli_common.h
index 80d680f..8d0a623 100644
--- a/include/vcli_common.h
+++ b/include/vcli_common.h
@@ -42,4 +42,5 @@ struct cli {
 	char			*ident;
 	struct vlu		*vlu;
 	struct VCLS		*cls;
+	volatile unsigned	*limit;
 };
diff --git a/include/vcli_serve.h b/include/vcli_serve.h
index 0ac29f8..7e4e6e4 100644
--- a/include/vcli_serve.h
+++ b/include/vcli_serve.h
@@ -30,7 +30,8 @@
 struct VCLS;
 typedef void cls_cb_f(void *priv);
 typedef void cls_cbc_f(const struct cli*);
-struct VCLS *VCLS_New(cls_cbc_f *before, cls_cbc_f *after, unsigned maxlen);
+struct VCLS *VCLS_New(cls_cbc_f *before, cls_cbc_f *after,
+    volatile unsigned *maxlen, volatile unsigned *limit);
 struct cli *VCLS_AddFd(struct VCLS *cs, int fdi, int fdo, cls_cb_f *closefunc,
     void *priv);
 int VCLS_AddFunc(struct VCLS *cs, unsigned auth, struct cli_proto *clp);
diff --git a/lib/libvarnish/cli_common.c b/lib/libvarnish/cli_common.c
index 4a17de5..7b09bdb 100644
--- a/lib/libvarnish/cli_common.c
+++ b/lib/libvarnish/cli_common.c
@@ -43,6 +43,7 @@
 #include <time.h>
 #include <unistd.h>
 
+#include "miniobj.h"
 #include "vas.h"
 #include "vcli.h"
 #include "vcli_common.h"
@@ -56,10 +57,15 @@ VCLI_Out(struct cli *cli, const char *fmt, ...)
 	va_list ap;
 
 	va_start(ap, fmt);
-	if (cli != NULL)
-		(void)VSB_vprintf(cli->sb, fmt, ap);
-	else
+	if (cli != NULL) {
+		CHECK_OBJ_NOTNULL(cli, CLI_MAGIC);
+		if (VSB_len(cli->sb) < *cli->limit)
+			(void)VSB_vprintf(cli->sb, fmt, ap);
+		else if (cli->result == CLIS_OK)
+			cli->result = CLIS_TRUNCATED;
+	} else {
 		(void)vfprintf(stdout, fmt, ap);
+	}
 	va_end(ap);
 }
 
@@ -68,6 +74,7 @@ void
 VCLI_Quote(struct cli *cli, const char *s)
 {
 
+	CHECK_OBJ_NOTNULL(cli, CLI_MAGIC);
 	VSB_quote(cli->sb, s, -1, 0);
 }
 
@@ -75,10 +82,13 @@ void
 VCLI_SetResult(struct cli *cli, unsigned res)
 {
 
-	if (cli != NULL)
-		cli->result = res;	/*lint !e64 type mismatch */
-	else
+	if (cli != NULL) {
+		CHECK_OBJ_NOTNULL(cli, CLI_MAGIC);
+		if (cli->result != CLIS_TRUNCATED || res != CLIS_OK)
+			cli->result = res;	/*lint !e64 type mismatch */
+	} else {
 		printf("CLI result = %u\n", res);
+	}
 }
 
 int
diff --git a/lib/libvarnish/cli_serve.c b/lib/libvarnish/cli_serve.c
index f3def96..547061c 100644
--- a/lib/libvarnish/cli_serve.c
+++ b/lib/libvarnish/cli_serve.c
@@ -81,7 +81,8 @@ struct VCLS {
 	unsigned			nfd;
 	VTAILQ_HEAD(,VCLS_func)		funcs;
 	cls_cbc_f			*before, *after;
-	unsigned			maxlen;
+	volatile unsigned		*maxlen;
+	volatile unsigned		*limit;
 };
 
 /*--------------------------------------------------------------------*/
@@ -245,6 +246,10 @@ cls_vlu2(void *priv, char * const *av)
 	struct VCLS_func *cfn;
 	struct cli *cli;
 	unsigned na;
+	ssize_t len;
+	char *s;
+	unsigned lim;
+	const char *trunc = "!\n[response was truncated]\n";
 
 	CAST_OBJ_NOTNULL(cfd, priv, VCLS_FD_MAGIC);
 	cs = cfd->cls;
@@ -297,7 +302,16 @@ cls_vlu2(void *priv, char * const *av)
 
 	cli->cls = NULL;
 
-	if (VCLI_WriteResult(cfd->fdo, cli->result, VSB_data(cli->sb)) ||
+	s = VSB_data(cli->sb);
+	len = VSB_len(cli->sb);
+	lim = *cs->limit;
+	if (len > lim) {
+		if (cli->result == CLIS_OK)
+			cli->result = CLIS_TRUNCATED;
+		strcpy(s + (lim - strlen(trunc)), trunc);
+		assert(strlen(s) <= lim);
+	}
+	if (VCLI_WriteResult(cfd->fdo, cli->result, s) ||
 	    cli->result == CLIS_CLOSE)
 		return (1);
 
@@ -380,7 +394,8 @@ cls_vlu(void *priv, const char *p)
 }
 
 struct VCLS *
-VCLS_New(cls_cbc_f *before, cls_cbc_f *after, unsigned maxlen)
+VCLS_New(cls_cbc_f *before, cls_cbc_f *after, volatile unsigned *maxlen,
+    volatile unsigned *limit)
 {
 	struct VCLS *cs;
 
@@ -391,6 +406,7 @@ VCLS_New(cls_cbc_f *before, cls_cbc_f *after, unsigned maxlen)
 	cs->before = before;
 	cs->after = after;
 	cs->maxlen = maxlen;
+	cs->limit = limit;
 	return (cs);
 }
 
@@ -409,8 +425,9 @@ VCLS_AddFd(struct VCLS *cs, int fdi, int fdo, cls_cb_f *closefunc, void *priv)
 	cfd->fdo = fdo;
 	cfd->cli = &cfd->clis;
 	cfd->cli->magic = CLI_MAGIC;
-	cfd->cli->vlu = VLU_New(cfd, cls_vlu, cs->maxlen);
+	cfd->cli->vlu = VLU_New(cfd, cls_vlu, *cs->maxlen);
 	cfd->cli->sb = VSB_new_auto();
+	cfd->cli->limit = cs->limit;
 	cfd->closefunc = closefunc;
 	cfd->priv = priv;
 	AN(cfd->cli->sb);



From tfheen at varnish-cache.org  Mon Nov 14 11:08:36 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Mon, 14 Nov 2011 12:08:36 +0100
Subject: [master] ce8a772 Use PCRE_NOTEMPTY rather than NOTEMPTY_ATSTART,
	it suffices for us
Message-ID: <E1RPuOy-0003ss-MJ@project.varnish-software.com>

commit ce8a772629dc143d49427f14e7e1b431be344633
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Thu Nov 10 12:24:22 2011 +0100

    Use PCRE_NOTEMPTY rather than NOTEMPTY_ATSTART, it suffices for us

diff --git a/bin/varnishd/cache/cache_vrt_re.c b/bin/varnishd/cache/cache_vrt_re.c
index 7759d0a..9b4a2e6 100644
--- a/bin/varnishd/cache/cache_vrt_re.c
+++ b/bin/varnishd/cache/cache_vrt_re.c
@@ -139,7 +139,7 @@ VRT_regsub(const struct sess *sp, int all, const char *str, void *re,
 		if (!all)
 			break;
 		memset(&ovector, 0, sizeof(ovector));
-		options |= VRE_NOTEMPTY_ATSTART;
+		options |= VRE_NOTEMPTY;
 		i = VRE_exec(t, str, len, 0, options, ovector, 30,
 		    &cache_param->vre_limits);
 		if (i < VRE_ERROR_NOMATCH ) {
diff --git a/include/vre.h b/include/vre.h
index 59ffeb0..a59e8d7 100644
--- a/include/vre.h
+++ b/include/vre.h
@@ -49,7 +49,7 @@ typedef struct vre vre_t;
 
 /* And those to PCRE options */
 extern const unsigned VRE_CASELESS;
-extern const unsigned VRE_NOTEMPTY_ATSTART;
+extern const unsigned VRE_NOTEMPTY;
 
 vre_t *VRE_compile(const char *, int, const char **, int *);
 int VRE_exec(const vre_t *code, const char *subject, int length,
diff --git a/lib/libvarnish/vre.c b/lib/libvarnish/vre.c
index eac8121..fb2dcda 100644
--- a/lib/libvarnish/vre.c
+++ b/lib/libvarnish/vre.c
@@ -49,12 +49,7 @@ struct vre {
  * here.
  */
 const unsigned VRE_CASELESS = PCRE_CASELESS;
-const unsigned VRE_NOTEMPTY_ATSTART =
-#ifdef PCRE_NOTEMPTY_ATSTART
-	PCRE_NOTEMPTY_ATSTART;
-#else
-	0;
-#endif
+const unsigned VRE_NOTEMPTY = PCRE_NOTEMPTY;
 
 vre_t *
 VRE_compile(const char *pattern, int options,



From tfheen at varnish-cache.org  Mon Nov 14 11:08:36 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Mon, 14 Nov 2011 12:08:36 +0100
Subject: [master] 60ddf62 fix a block of text rendering as part of source code
Message-ID: <E1RPuOy-0003sx-Rd@project.varnish-software.com>

commit 60ddf626604a625a151c3d1dc011d3a9eba38bd2
Author: Kolia Morev <kolyuchiy at gmail.com>
Date:   Sun Nov 13 13:40:47 2011 +0400

    fix a block of text rendering as part of source code

diff --git a/doc/sphinx/reference/vcl.rst b/doc/sphinx/reference/vcl.rst
index 016e84f..6873fcf 100644
--- a/doc/sphinx/reference/vcl.rst
+++ b/doc/sphinx/reference/vcl.rst
@@ -976,10 +976,10 @@ based on the request URL:::
     }
   }
 
-  The following snippet demonstrates how to force a minimum TTL for
-  all documents.  Note that this is not the same as setting the
-  default_ttl run-time parameter, as that only affects document for
-  which the backend did not specify a TTL:::
+The following snippet demonstrates how to force a minimum TTL for
+all documents.  Note that this is not the same as setting the
+default_ttl run-time parameter, as that only affects document for
+which the backend did not specify a TTL:::
   
   import std; # needed for std.log
 



From tfheen at varnish-cache.org  Mon Nov 14 11:08:37 2011
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Mon, 14 Nov 2011 12:08:37 +0100
Subject: [master] 4e9bf8c Revert "Teach varnishtest about VRE_NOTEMPTY_ATSTART
	and skip c00047"
Message-ID: <E1RPuOz-0003t6-0D@project.varnish-software.com>

commit 4e9bf8c846308e3ead7b9f1051fe9865e0acf0c5
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Mon Nov 14 11:59:17 2011 +0100

    Revert "Teach varnishtest about VRE_NOTEMPTY_ATSTART and skip c00047"
    
    This isn't needed since we no longer use VRE_NOTEMPTY_ATSTART
    
    This reverts commit 43b009a8454f18c7aaaa211b19d2e54004dce24e.

diff --git a/bin/varnishtest/tests/c00047.vtc b/bin/varnishtest/tests/c00047.vtc
index dd1d256..4b23194 100644
--- a/bin/varnishtest/tests/c00047.vtc
+++ b/bin/varnishtest/tests/c00047.vtc
@@ -1,7 +1,5 @@
 varnishtest "Test VCL regsuball()"
 
-feature VRE_NOTEMPTY_ATSTART
-
 server s1 {
 	rxreq 
 	txresp \
diff --git a/bin/varnishtest/vtc.c b/bin/varnishtest/vtc.c
index d6c3b35..9581b3c 100644
--- a/bin/varnishtest/vtc.c
+++ b/bin/varnishtest/vtc.c
@@ -43,7 +43,6 @@
 #include "vtc.h"
 
 #include "vav.h"
-#include "vre.h"
 #include "vtim.h"
 
 
@@ -495,9 +494,6 @@ cmd_feature(CMD_ARGS)
 			continue;
 #endif
 		}
-		if (!strcmp(av[i], "VRE_NOTEMPTY_ATSTART") &&
-		    VRE_NOTEMPTY_ATSTART)
-			continue;
 
 		vtc_log(vl, 1, "SKIPPING test, missing feature: %s", av[i]);
 		vtc_stop = 1;



From phk at varnish-cache.org  Tue Nov 15 19:52:48 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 15 Nov 2011 20:52:48 +0100
Subject: [master] 9857918 Add VCLI_Overflow() so we can stop rendering bans
	when the CLI output buffer is full.
Message-ID: <E1RQP3o-0004FT-51@project.varnish-software.com>

commit 9857918cf7916d860fe3f4358960467359c6fd5c
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Nov 14 11:26:44 2011 +0000

    Add VCLI_Overflow() so we can stop rendering bans when the CLI
    output buffer is full.

diff --git a/bin/varnishd/cache/cache_ban.c b/bin/varnishd/cache/cache_ban.c
index 768e631..ca2b616 100644
--- a/bin/varnishd/cache/cache_ban.c
+++ b/bin/varnishd/cache/cache_ban.c
@@ -1071,6 +1071,8 @@ ccf_ban_list(struct cli *cli, const char * const *av, void *priv)
 		    b->flags & BAN_F_GONE ? "G" : " ");
 		ban_render(cli, b->spec);
 		VCLI_Out(cli, "\n");
+		if (VCLI_Overflow(cli))
+			break;
 		if (cache_param->diag_bitmap & 0x80000) {
 			Lck_Lock(&ban_mtx);
 			struct objcore *oc;
diff --git a/include/vcli_priv.h b/include/vcli_priv.h
index 60d6d69..a265d2e 100644
--- a/include/vcli_priv.h
+++ b/include/vcli_priv.h
@@ -52,6 +52,7 @@ struct cli_proto {
 };
 
 /* The implementation must provide these functions */
+int VCLI_Overflow(struct cli *cli);
 void VCLI_Out(struct cli *cli, const char *fmt, ...);
 void VCLI_Quote(struct cli *cli, const char *str);
 void VCLI_SetResult(struct cli *cli, unsigned r);
diff --git a/lib/libvarnish/cli_common.c b/lib/libvarnish/cli_common.c
index 7b09bdb..aa5fd63 100644
--- a/lib/libvarnish/cli_common.c
+++ b/lib/libvarnish/cli_common.c
@@ -70,6 +70,17 @@ VCLI_Out(struct cli *cli, const char *fmt, ...)
 }
 
 /*lint -e{818} cli could be const */
+int
+VCLI_Overflow(struct cli *cli)
+{
+	CHECK_OBJ_NOTNULL(cli, CLI_MAGIC);
+	if (cli->result == CLIS_TRUNCATED ||
+	    VSB_len(cli->sb) >= *cli->limit)
+		return (1);
+	return (0);
+}
+
+/*lint -e{818} cli could be const */
 void
 VCLI_Quote(struct cli *cli, const char *s)
 {



From phk at varnish-cache.org  Tue Nov 15 19:52:48 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 15 Nov 2011 20:52:48 +0100
Subject: [master] e7694ea Merge branch 'master' of
	ssh://git.varnish-cache.org/git/varnish-cache
Message-ID: <E1RQP3o-0004FY-Dq@project.varnish-software.com>

commit e7694ea92d0c9e1953b70ea9059fb3cbfe3eb257
Merge: 9857918 4e9bf8c
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Nov 14 16:32:18 2011 +0000

    Merge branch 'master' of ssh://git.varnish-cache.org/git/varnish-cache



From phk at varnish-cache.org  Tue Nov 15 19:52:48 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 15 Nov 2011 20:52:48 +0100
Subject: [master] 5cfe903 Combine two small functions
Message-ID: <E1RQP3o-0004Fh-KO@project.varnish-software.com>

commit 5cfe903c15a76c3b98fb1c42e8c862a5553286c2
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 15 19:52:36 2011 +0000

    Combine two small functions

diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c
index a2c0b58..7e60c1a 100644
--- a/bin/varnishd/mgt/mgt_child.c
+++ b/bin/varnishd/mgt/mgt_child.c
@@ -424,21 +424,14 @@ mgt_stop_child(void)
 /*--------------------------------------------------------------------*/
 
 static void
-mgt_report_panic(pid_t r)
+mgt_handle_panicstr(pid_t r)
 {
+	char time_str[30];
 
 	if (VSM_head->panicstr[0] == '\0')
 		return;
 	REPORT(LOG_ERR, "Child (%jd) Panic message: %s",
 	    (intmax_t)r, VSM_head->panicstr);
-}
-
-static void
-mgt_save_panic(void)
-{
-	char time_str[30];
-	if (VSM_head->panicstr[0] == '\0')
-		return;
 
 	if (child_panic)
 		VSB_delete(child_panic);
@@ -500,8 +493,7 @@ mgt_sigchld(const struct vev *e, int what)
 	REPORT(LOG_INFO, "%s", VSB_data(vsb));
 	VSB_delete(vsb);
 
-	mgt_report_panic(r);
-	mgt_save_panic();
+	mgt_handle_panicstr(r);
 
 	child_pid = -1;
 



From phk at varnish-cache.org  Tue Nov 15 21:08:03 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 15 Nov 2011 22:08:03 +0100
Subject: [master] 0f13e7e Split stevedore.c in a cache and a worker source
	file.
Message-ID: <E1RQQEd-00025l-DB@project.varnish-software.com>

commit 0f13e7ebf1bd4f2e6f9e3b91353c407ba9b8295e
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 15 21:07:39 2011 +0000

    Split stevedore.c in a cache and a worker source file.

diff --git a/bin/varnishd/Makefile.am b/bin/varnishd/Makefile.am
index 850a18a..359373d 100644
--- a/bin/varnishd/Makefile.am
+++ b/bin/varnishd/Makefile.am
@@ -64,6 +64,7 @@ varnishd_SOURCES = \
 	mgt/mgt_shmem.c \
 	mgt/mgt_vcc.c \
 	storage/stevedore.c \
+	storage/stevedore_mgt.c \
 	storage/stevedore_utils.c \
 	storage/storage_file.c \
 	storage/storage_malloc.c \
diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 653fe77..129dbf0 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -114,9 +114,6 @@ struct worker;
 
 #define DIGEST_LEN		32
 
-/* Name of transient storage */
-#define TRANSIENT_STORAGE	"Transient"
-
 /*--------------------------------------------------------------------
  * Pointer aligment magic
  */
diff --git a/bin/varnishd/common/common.h b/bin/varnishd/common/common.h
index 4e56a40..ff06c64 100644
--- a/bin/varnishd/common/common.h
+++ b/bin/varnishd/common/common.h
@@ -42,6 +42,9 @@
 
 struct cli;
 
+/* Name of transient storage */
+#define TRANSIENT_STORAGE	"Transient"
+
 extern pid_t mgt_pid;
 #define ASSERT_MGT() do { assert(getpid() == mgt_pid);} while (0)
 
@@ -68,13 +71,6 @@ void mgt_child_inherit(int fd, const char *what);
 		exit(2);					\
 	} while (0);
 
-/* A tiny helper for choosing hash/storage modules */
-struct choice {
-	const char      *name;
-	const void	*ptr;
-};
-const void *pick(const struct choice *cp, const char *which, const char *kind);
-
 #define NEEDLESS_RETURN(foo)	return (foo)
 
 /* stevedore.c */
diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h
index 54352a0..4aec482 100644
--- a/bin/varnishd/mgt/mgt.h
+++ b/bin/varnishd/mgt/mgt.h
@@ -57,13 +57,19 @@ void mgt_cli_master(const char *M_arg);
 void mgt_cli_secret(const char *S_arg);
 void mgt_cli_close_all(void);
 
+/* mgt_main.c */
+struct choice {
+	const char      *name;
+	const void	*ptr;
+};
+const void *pick(const struct choice *cp, const char *which, const char *kind);
+
 /* mgt_param.c */
 void MCF_ParamInit(struct cli *);
 void MCF_ParamSet(struct cli *, const char *param, const char *val);
 void MCF_DumpRst(void);
 extern struct params mgt_param;
 
-
 /* mgt_sandbox.c */
 void mgt_sandbox(void);
 
@@ -88,7 +94,6 @@ extern const char *mgt_vcl_dir;
 extern const char *mgt_vmod_dir;
 extern unsigned mgt_vcc_err_unref;
 
-
 #define REPORT0(pri, fmt)				\
 	do {						\
 		fprintf(stderr, fmt "\n");		\
@@ -105,7 +110,6 @@ extern unsigned mgt_vcc_err_unref;
 #define VSM_Free(a)		VSM__Free(a)
 #define VSM_Clean()		VSM__Clean()
 
-
 #if defined(PTHREAD_CANCELED) || defined(PTHREAD_MUTEX_DEFAULT)
 #error "Keep pthreads out of in manager process"
 #endif
diff --git a/bin/varnishd/storage/stevedore.c b/bin/varnishd/storage/stevedore.c
index 3978f64..9b22cd3 100644
--- a/bin/varnishd/storage/stevedore.c
+++ b/bin/varnishd/storage/stevedore.c
@@ -39,18 +39,12 @@
 #include "cache/cache.h"
 
 #include "storage/storage.h"
-#include "vav.h"
 #include "vcli_priv.h"
 #include "vrt.h"
 #include "vrt_obj.h"
 
-static VTAILQ_HEAD(, stevedore)	stevedores =
-    VTAILQ_HEAD_INITIALIZER(stevedores);
-
 static const struct stevedore * volatile stv_next;
 
-static struct stevedore *stv_transient;
-
 /*---------------------------------------------------------------------
  * Default objcore methods
  */
@@ -130,7 +124,7 @@ stv_pick_stevedore(const struct sess *sp, const char **hint)
 
 	AN(hint);
 	if (*hint != NULL && **hint != '\0') {
-		VTAILQ_FOREACH(stv, &stevedores, list) {
+		VTAILQ_FOREACH(stv, &stv_stevedores, list) {
 			if (!strcmp(stv->ident, *hint))
 				return (stv);
 		}
@@ -144,7 +138,7 @@ stv_pick_stevedore(const struct sess *sp, const char **hint)
 	/* pick a stevedore and bump the head along */
 	stv = VTAILQ_NEXT(stv_next, list);
 	if (stv == NULL)
-		stv = VTAILQ_FIRST(&stevedores);
+		stv = VTAILQ_FIRST(&stv_stevedores);
 	AN(stv);
 	AN(stv->name);
 	stv_next = stv;
@@ -271,7 +265,7 @@ STV_MkObject(struct sess *sp, void *ptr, unsigned ltot,
  * implement persistent storage can rely on.
  */
 
-static struct object *
+struct object *
 stv_default_allocobj(struct stevedore *stv, struct sess *sp, unsigned ltot,
     const struct stv_objsecrets *soc)
 {
@@ -402,7 +396,7 @@ STV_open(void)
 {
 	struct stevedore *stv;
 
-	VTAILQ_FOREACH(stv, &stevedores, list) {
+	VTAILQ_FOREACH(stv, &stv_stevedores, list) {
 		stv->lru = LRU_Alloc();
 		if (stv->open != NULL)
 			stv->open(stv);
@@ -412,6 +406,7 @@ STV_open(void)
 		stv->lru = LRU_Alloc();
 		stv->open(stv);
 	}
+	stv_next = VTAILQ_FIRST(&stv_stevedores);
 }
 
 void
@@ -419,7 +414,7 @@ STV_close(void)
 {
 	struct stevedore *stv;
 
-	VTAILQ_FOREACH(stv, &stevedores, list)
+	VTAILQ_FOREACH(stv, &stv_stevedores, list)
 		if (stv->close != NULL)
 			stv->close(stv);
 	stv = stv_transient;
@@ -427,113 +422,6 @@ STV_close(void)
 		stv->close(stv);
 }
 
-/*--------------------------------------------------------------------
- * Parse a stevedore argument on the form:
- *	[ name '=' ] strategy [ ',' arg ] *
- */
-
-static const struct choice STV_choice[] = {
-	{ "file",	&smf_stevedore },
-	{ "malloc",	&sma_stevedore },
-	{ "persistent",	&smp_stevedore },
-#ifdef HAVE_LIBUMEM
-	{ "umem",	&smu_stevedore },
-#endif
-	{ NULL,		NULL }
-};
-
-void
-STV_Config(const char *spec)
-{
-	char **av;
-	const char *p, *q;
-	struct stevedore *stv;
-	const struct stevedore *stv2;
-	int ac, l;
-	static unsigned seq = 0;
-
-	ASSERT_MGT();
-	p = strchr(spec, '=');
-	q = strchr(spec, ',');
-	if (p != NULL && (q == NULL || q > p)) {
-		av = VAV_Parse(p + 1, NULL, ARGV_COMMA);
-	} else {
-		av = VAV_Parse(spec, NULL, ARGV_COMMA);
-		p = NULL;
-	}
-	AN(av);
-
-	if (av[0] != NULL)
-		ARGV_ERR("%s\n", av[0]);
-
-	if (av[1] == NULL)
-		ARGV_ERR("-s argument lacks strategy {malloc, file, ...}\n");
-
-	for (ac = 0; av[ac + 2] != NULL; ac++)
-		continue;
-
-	stv2 = pick(STV_choice, av[1], "storage");
-	AN(stv2);
-
-	/* Append strategy to ident string */
-	VSB_printf(vident, ",-s%s", av[1]);
-
-	av += 2;
-
-	CHECK_OBJ_NOTNULL(stv2, STEVEDORE_MAGIC);
-	ALLOC_OBJ(stv, STEVEDORE_MAGIC);
-	AN(stv);
-
-	*stv = *stv2;
-	AN(stv->name);
-	AN(stv->alloc);
-	if (stv->allocobj == NULL)
-		stv->allocobj = stv_default_allocobj;
-
-	if (p == NULL)
-		bprintf(stv->ident, "s%u", seq++);
-	else {
-		l = p - spec;
-		if (l > sizeof stv->ident - 1)
-			l = sizeof stv->ident - 1;
-		bprintf(stv->ident, "%.*s", l, spec);
-	}
-
-	VTAILQ_FOREACH(stv2, &stevedores, list) {
-		if (strcmp(stv2->ident, stv->ident))
-			continue;
-		ARGV_ERR("(-s%s=%s) already defined once\n",
-		    stv->ident, stv->name);
-	}
-
-	if (stv->init != NULL)
-		stv->init(stv, ac, av);
-	else if (ac != 0)
-		ARGV_ERR("(-s%s) too many arguments\n", stv->name);
-
-	if (!strcmp(stv->ident, TRANSIENT_STORAGE)) {
-		stv->transient = 1;
-		AZ(stv_transient);
-		stv_transient = stv;
-	} else {
-		VTAILQ_INSERT_TAIL(&stevedores, stv, list);
-		if (!stv_next)
-			stv_next = VTAILQ_FIRST(&stevedores);
-	}
-}
-
-/*--------------------------------------------------------------------*/
-
-void
-STV_Config_Transient(void)
-{
-
-	ASSERT_MGT();
-
-	if (stv_transient == NULL)
-		STV_Config(TRANSIENT_STORAGE "=malloc");
-}
-
 /*--------------------------------------------------------------------*/
 
 static void
@@ -547,7 +435,7 @@ stv_cli_list(struct cli *cli, const char * const *av, void *priv)
 	VCLI_Out(cli, "Storage devices:\n");
 	stv = stv_transient;
 		VCLI_Out(cli, "\tstorage.%s = %s\n", stv->ident, stv->name);
-	VTAILQ_FOREACH(stv, &stevedores, list)
+	VTAILQ_FOREACH(stv, &stv_stevedores, list)
 		VCLI_Out(cli, "\tstorage.%s = %s\n", stv->ident, stv->name);
 }
 
@@ -568,7 +456,7 @@ stv_find(const char *nm)
 {
 	const struct stevedore *stv;
 
-	VTAILQ_FOREACH(stv, &stevedores, list)
+	VTAILQ_FOREACH(stv, &stv_stevedores, list)
 		if (!strcmp(stv->ident, nm))
 			return (stv);
 	if (!strcmp(TRANSIENT_STORAGE, nm))
diff --git a/bin/varnishd/storage/stevedore_mgt.c b/bin/varnishd/storage/stevedore_mgt.c
new file mode 100644
index 0000000..fc0d6a6
--- /dev/null
+++ b/bin/varnishd/storage/stevedore_mgt.c
@@ -0,0 +1,156 @@
+/*-
+ * Copyright (c) 2007-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Dag-Erling Sm?rgav <des at des.no>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * STEVEDORE: one who works at or is responsible for loading and
+ * unloading ships in port.  Example: "on the wharves, stevedores were
+ * unloading cargo from the far corners of the world." Origin: Spanish
+ * estibador, from estibar to pack.  First Known Use: 1788
+ */
+
+#include "config.h"
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+
+#include "mgt/mgt.h"
+
+#include "storage/storage.h"
+#include "vav.h"
+
+struct stevedore_head stv_stevedores =
+    VTAILQ_HEAD_INITIALIZER(stv_stevedores);
+
+struct stevedore *stv_transient;
+
+/*--------------------------------------------------------------------
+ * Parse a stevedore argument on the form:
+ *	[ name '=' ] strategy [ ',' arg ] *
+ */
+
+static const struct choice STV_choice[] = {
+	{ "file",	&smf_stevedore },
+	{ "malloc",	&sma_stevedore },
+	{ "persistent",	&smp_stevedore },
+#ifdef HAVE_LIBUMEM
+	{ "umem",	&smu_stevedore },
+#endif
+	{ NULL,		NULL }
+};
+
+void
+STV_Config(const char *spec)
+{
+	char **av;
+	const char *p, *q;
+	struct stevedore *stv;
+	const struct stevedore *stv2;
+	int ac, l;
+	static unsigned seq = 0;
+
+	ASSERT_MGT();
+	p = strchr(spec, '=');
+	q = strchr(spec, ',');
+	if (p != NULL && (q == NULL || q > p)) {
+		av = VAV_Parse(p + 1, NULL, ARGV_COMMA);
+	} else {
+		av = VAV_Parse(spec, NULL, ARGV_COMMA);
+		p = NULL;
+	}
+	AN(av);
+
+	if (av[0] != NULL)
+		ARGV_ERR("%s\n", av[0]);
+
+	if (av[1] == NULL)
+		ARGV_ERR("-s argument lacks strategy {malloc, file, ...}\n");
+
+	for (ac = 0; av[ac + 2] != NULL; ac++)
+		continue;
+
+	stv2 = pick(STV_choice, av[1], "storage");
+	AN(stv2);
+
+	/* Append strategy to ident string */
+	VSB_printf(vident, ",-s%s", av[1]);
+
+	av += 2;
+
+	CHECK_OBJ_NOTNULL(stv2, STEVEDORE_MAGIC);
+	ALLOC_OBJ(stv, STEVEDORE_MAGIC);
+	AN(stv);
+
+	*stv = *stv2;
+	AN(stv->name);
+	AN(stv->alloc);
+	if (stv->allocobj == NULL)
+		stv->allocobj = stv_default_allocobj;
+
+	if (p == NULL)
+		bprintf(stv->ident, "s%u", seq++);
+	else {
+		l = p - spec;
+		if (l > sizeof stv->ident - 1)
+			l = sizeof stv->ident - 1;
+		bprintf(stv->ident, "%.*s", l, spec);
+	}
+
+	VTAILQ_FOREACH(stv2, &stv_stevedores, list) {
+		if (strcmp(stv2->ident, stv->ident))
+			continue;
+		ARGV_ERR("(-s%s=%s) already defined once\n",
+		    stv->ident, stv->name);
+	}
+
+	if (stv->init != NULL)
+		stv->init(stv, ac, av);
+	else if (ac != 0)
+		ARGV_ERR("(-s%s) too many arguments\n", stv->name);
+
+	if (!strcmp(stv->ident, TRANSIENT_STORAGE)) {
+		stv->transient = 1;
+		AZ(stv_transient);
+		stv_transient = stv;
+	} else {
+		VTAILQ_INSERT_TAIL(&stv_stevedores, stv, list);
+	}
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+STV_Config_Transient(void)
+{
+
+	ASSERT_MGT();
+
+	if (stv_transient == NULL)
+		STV_Config(TRANSIENT_STORAGE "=malloc");
+}
+
+/*--------------------------------------------------------------------*/
diff --git a/bin/varnishd/storage/storage.h b/bin/varnishd/storage/storage.h
index 80cad13..a813a36 100644
--- a/bin/varnishd/storage/storage.h
+++ b/bin/varnishd/storage/storage.h
@@ -50,6 +50,8 @@ typedef void storage_close_f(const struct stevedore *);
 #include "tbl/vrt_stv_var.h"
 #undef VRTSTVTYPE
 
+extern storage_allocobj_f stv_default_allocobj;
+
 /*--------------------------------------------------------------------*/
 
 struct stevedore {
@@ -78,6 +80,11 @@ struct stevedore {
 	char			ident[16];	/* XXX: match VSM_chunk.ident */
 };
 
+VTAILQ_HEAD(stevedore_head, stevedore);
+
+extern struct stevedore_head stv_stevedores;
+extern struct stevedore *stv_transient;
+
 /*--------------------------------------------------------------------*/
 int STV_GetFile(const char *fn, int *fdp, const char **fnp, const char *ctx);
 uintmax_t STV_FileSize(int fd, const char *size, unsigned *granularity,



From phk at varnish-cache.org  Wed Nov 16 06:43:23 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 16 Nov 2011 07:43:23 +0100
Subject: [master] 54e7eb5 Move prototypes to smaller scope
Message-ID: <E1RQZDP-0006gt-6f@project.varnish-software.com>

commit 54e7eb52c299a09f2adada5c9fc9d4173a5886ff
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Nov 16 05:59:49 2011 +0000

    Move prototypes to smaller scope

diff --git a/bin/varnishd/common/common.h b/bin/varnishd/common/common.h
index ff06c64..f6d01db 100644
--- a/bin/varnishd/common/common.h
+++ b/bin/varnishd/common/common.h
@@ -73,9 +73,6 @@ void mgt_child_inherit(int fd, const char *what);
 
 #define NEEDLESS_RETURN(foo)	return (foo)
 
-/* stevedore.c */
-void STV_Config(const char *spec);
-void STV_Config_Transient(void);
 
 /* vsm.c */
 // extern struct VSM_head		*VSM_head;
diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h
index 4aec482..1eabe10 100644
--- a/bin/varnishd/mgt/mgt.h
+++ b/bin/varnishd/mgt/mgt.h
@@ -84,6 +84,10 @@ void mgt_sandbox_solaris_privsep(void);
 void mgt_SHM_Init(const char *arg);
 void mgt_SHM_Pid(void);
 
+/* stevedore_mgt.c */
+void STV_Config(const char *spec);
+void STV_Config_Transient(void);
+
 /* mgt_vcc.c */
 void mgt_vcc_init(void);
 int mgt_vcc_default(const char *bflag, const char *f_arg, char *vcl, int Cflag);



From phk at varnish-cache.org  Wed Nov 16 06:43:23 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 16 Nov 2011 07:43:23 +0100
Subject: [master] ceaf73d Make the panicstr a VSM allocation
Message-ID: <E1RQZDP-0006h7-Dk@project.varnish-software.com>

commit ceaf73d70ae92b81440615709789cf6101f18a51
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Nov 16 06:22:49 2011 +0000

    Make the panicstr a VSM allocation

diff --git a/bin/varnishd/cache/cache_panic.c b/bin/varnishd/cache/cache_panic.c
index c626ae3..d112eb8 100644
--- a/bin/varnishd/cache/cache_panic.c
+++ b/bin/varnishd/cache/cache_panic.c
@@ -40,8 +40,6 @@
 
 #include "cache.h"
 
-#include "vapi/vsm_int.h"
-
 #include "cache_backend.h"
 #include "waiter/cache_waiter.h"
 #include "vcl.h"
@@ -58,6 +56,10 @@
 static struct vsb vsps, *vsp;
 static pthread_mutex_t panicstr_mtx = PTHREAD_MUTEX_INITIALIZER;
 
+/* Initialized in mgt_shmem.c, points into VSM */
+char *PAN_panicstr;
+unsigned PAN_panicstr_len;
+
 /*--------------------------------------------------------------------*/
 
 static void
@@ -354,20 +356,8 @@ pan_ic(const char *func, const char *file, int line, const char *cond,
 	VSB_bcat(vsp, "", 1);	/* NUL termination */
 
 	if (cache_param->diag_bitmap & 0x4000)
-		(void)fputs(VSM_head->panicstr, stderr);
-
-#ifdef HAVE_ABORT2
-	if (cache_param->diag_bitmap & 0x8000) {
-		void *arg[1];
-		char *p;
-
-		for (p = VSM_head->panicstr; *p; p++)
-			if (*p == '\n')
-				*p = ' ';
-		arg[0] = VSM_head->panicstr;
-		abort2(VSM_head->panicstr, 1, arg);
-	}
-#endif
+		(void)fputs(PAN_panicstr, stderr);
+
 	if (cache_param->diag_bitmap & 0x1000)
 		exit(4);
 	else
@@ -382,6 +372,8 @@ PAN_Init(void)
 
 	VAS_Fail = pan_ic;
 	vsp = &vsps;
-	AN(VSB_new(vsp, VSM_head->panicstr, sizeof VSM_head->panicstr,
+	AN(PAN_panicstr);
+	AN(PAN_panicstr_len);
+	AN(VSB_new(vsp, PAN_panicstr, PAN_panicstr_len,
 	    VSB_FIXEDLEN));
 }
diff --git a/bin/varnishd/cache/cache_shmlog.c b/bin/varnishd/cache/cache_shmlog.c
index 1252fa3..e004896 100644
--- a/bin/varnishd/cache/cache_shmlog.c
+++ b/bin/varnishd/cache/cache_shmlog.c
@@ -317,7 +317,6 @@ VSL_Init(void)
 
 	vsl_wrap();
 	VSM_head->starttime = (intmax_t)VTIM_real();
-	memset(VSM_head->panicstr, '\0', sizeof *VSM_head->panicstr);
 	memset(VSC_C_main, 0, sizeof *VSC_C_main);
 	VSM_head->child_pid = getpid();
 }
diff --git a/bin/varnishd/common/common.h b/bin/varnishd/common/common.h
index f6d01db..f59cfc9 100644
--- a/bin/varnishd/common/common.h
+++ b/bin/varnishd/common/common.h
@@ -50,6 +50,9 @@ extern pid_t mgt_pid;
 
 /* mgt_shmem.c */
 extern struct VSC_C_main *VSC_C_main;
+#define PAN_CLASS "Panic"
+extern char *PAN_panicstr;
+extern unsigned PAN_panicstr_len;
 
 /* varnishd.c */
 struct vsb;
diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c
index 7e60c1a..b619b66 100644
--- a/bin/varnishd/mgt/mgt_child.c
+++ b/bin/varnishd/mgt/mgt_child.c
@@ -47,7 +47,6 @@
 #include "common/heritage.h"
 #include "common/params.h"
 
-#include "vapi/vsm_int.h"
 #include "vbm.h"
 #include "vcli.h"
 #include "vcli_priv.h"
@@ -428,10 +427,10 @@ mgt_handle_panicstr(pid_t r)
 {
 	char time_str[30];
 
-	if (VSM_head->panicstr[0] == '\0')
+	if (PAN_panicstr[0] == '\0')
 		return;
 	REPORT(LOG_ERR, "Child (%jd) Panic message: %s",
-	    (intmax_t)r, VSM_head->panicstr);
+	    (intmax_t)r, PAN_panicstr);
 
 	if (child_panic)
 		VSB_delete(child_panic);
@@ -439,7 +438,7 @@ mgt_handle_panicstr(pid_t r)
 	XXXAN(child_panic);
 	VTIM_format(VTIM_real(), time_str);
 	VSB_printf(child_panic, "Last panic at: %s\n", time_str);
-	VSB_cat(child_panic, VSM_head->panicstr);
+	VSB_cat(child_panic, PAN_panicstr);
 	AZ(VSB_finish(child_panic));
 }
 
diff --git a/bin/varnishd/mgt/mgt_param.c b/bin/varnishd/mgt/mgt_param.c
index afaab84..b7c18b2 100644
--- a/bin/varnishd/mgt/mgt_param.c
+++ b/bin/varnishd/mgt/mgt_param.c
@@ -843,9 +843,6 @@ static const struct parspec input_parspec[] = {
 		"  0x00001000 - do not core-dump child process.\n"
 		"  0x00002000 - only short panic message.\n"
 		"  0x00004000 - panic to stderr.\n"
-#ifdef HAVE_ABORT2
-		"  0x00008000 - panic to abort2().\n"
-#endif
 		"  0x00010000 - synchronize shmlog.\n"
 		"  0x00020000 - synchronous start of persistence.\n"
 		"  0x00040000 - release VCL early.\n"
diff --git a/bin/varnishd/mgt/mgt_shmem.c b/bin/varnishd/mgt/mgt_shmem.c
index d4346df..24bc396 100644
--- a/bin/varnishd/mgt/mgt_shmem.c
+++ b/bin/varnishd/mgt/mgt_shmem.c
@@ -331,6 +331,12 @@ mgt_SHM_Init(const char *l_arg)
 	vsl_log_start[1] = VSL_ENDMARKER;
 	VWMB();
 
+	PAN_panicstr_len = 64 * 1024;
+	PAN_panicstr = VSM_Alloc(PAN_panicstr_len, PAN_CLASS, "", "");
+	AN(PAN_panicstr);
+	/* XXX: shouldn't VSM_Alloc zero ? */
+	memset(PAN_panicstr, '\0', PAN_panicstr_len);
+
 	do
 		*vsl_log_start = random() & 0xffff;
 	while (*vsl_log_start == 0);
diff --git a/configure.ac b/configure.ac
index bbe44b5..8404de4 100644
--- a/configure.ac
+++ b/configure.ac
@@ -187,7 +187,6 @@ AC_CHECK_FUNCS([socket])
 AC_CHECK_FUNCS([strptime])
 AC_CHECK_FUNCS([fmtcheck])
 AC_CHECK_FUNCS([getdtablesize])
-AC_CHECK_FUNCS([abort2])
 AC_CHECK_FUNCS([timegm])
 AC_CHECK_FUNCS([nanosleep])
 AC_CHECK_FUNCS([setppriv])
diff --git a/include/vapi/vsm_int.h b/include/vapi/vsm_int.h
index 3554982..521f89d 100644
--- a/include/vapi/vsm_int.h
+++ b/include/vapi/vsm_int.h
@@ -65,9 +65,6 @@ struct VSM_head {
 
 	unsigned		shm_size;
 
-	/* Panic message buffer */
-	char			panicstr[64 * 1024];
-
 	unsigned		alloc_seq;
 	/* Must be last element */
 	struct VSM_chunk	head;



From phk at varnish-cache.org  Wed Nov 16 06:43:23 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 16 Nov 2011 07:43:23 +0100
Subject: [master] 919696b More include file cleanup
Message-ID: <E1RQZDP-0006hI-I9@project.varnish-software.com>

commit 919696b0d969c7e2400434344c3c0658942160c8
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Nov 16 06:30:16 2011 +0000

    More include file cleanup

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 129dbf0..bae2572 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -114,15 +114,6 @@ struct worker;
 
 #define DIGEST_LEN		32
 
-/*--------------------------------------------------------------------
- * Pointer aligment magic
- */
-
-#define PALGN		(sizeof(void *) - 1)
-#define PAOK(p)		(((uintptr_t)(p) & PALGN) == 0)
-#define PRNDDN(p)	((uintptr_t)(p) & ~PALGN)
-#define PRNDUP(p)	(((uintptr_t)(p) + PALGN) & ~PALGN)
-
 /*--------------------------------------------------------------------*/
 
 typedef struct {
@@ -138,6 +129,9 @@ enum step {
 #undef STEP
 };
 
+/*--------------------------------------------------------------------*/
+struct lock { void *priv; };	// Opaque
+
 /*--------------------------------------------------------------------
  * Workspace structure for quick memory allocation.
  */
diff --git a/bin/varnishd/common/common.h b/bin/varnishd/common/common.h
index f59cfc9..8e32ec9 100644
--- a/bin/varnishd/common/common.h
+++ b/bin/varnishd/common/common.h
@@ -55,13 +55,9 @@ extern char *PAN_panicstr;
 extern unsigned PAN_panicstr_len;
 
 /* varnishd.c */
-struct vsb;
-extern struct vsb *vident;
+extern struct vsb *vident;		// XXX: -> heritage ?
 int Symbol_Lookup(struct vsb *vsb, void *ptr);
 
-#define TRUST_ME(ptr)	((void*)(uintptr_t)(ptr))
-
-
 /* Help shut up FlexeLint */
 #define __match_proto__(xxx) /*lint -e{818} */
 
@@ -97,9 +93,6 @@ void VSM__Clean(void);
 #define VSM_CLASS_MARK	"MgrCld"
 #define VSM_COOL_TIME	5
 
-/* cache_lck.c */
-struct lock { void *priv; };		// Opaque
-
 /*---------------------------------------------------------------------
  * Generic power-2 rounding macros
  */
@@ -107,3 +100,17 @@ struct lock { void *priv; };		// Opaque
 #define PWR2(x)     ((((x)-1)&(x))==0)		/* Is a power of two */
 #define RDN2(x, y)  ((x)&(~((y)-1)))		/* if y is powers of two */
 #define RUP2(x, y)  (((x)+((y)-1))&(~((y)-1)))	/* if y is powers of two */
+
+/*--------------------------------------------------------------------
+ * Pointer aligment magic
+ */
+
+#define PALGN	    (sizeof(void *) - 1)	/* size of alignment */
+#define PAOK(p)	    (((uintptr_t)(p) & PALGN) == 0)	/* is aligned */
+#define PRNDDN(p)   ((uintptr_t)(p) & ~PALGN)		/* Round down */
+#define PRNDUP(p)   (((uintptr_t)(p) + PALGN) & ~PALGN)	/* Round up */
+
+/*--------------------------------------------------------------------
+ * To be used as little as possible to wash off const/volatile etc.
+ */
+#define TRUST_ME(ptr)	((void*)(uintptr_t)(ptr))



From phk at varnish-cache.org  Wed Nov 16 06:43:24 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 16 Nov 2011 07:43:24 +0100
Subject: [master] 46b0d2d Move mgt cli stuff to _mgt.c file
Message-ID: <E1RQZDQ-0006hP-5V@project.varnish-software.com>

commit 46b0d2d22700ec66782591cbfcb4245baa978913
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Nov 16 06:42:09 2011 +0000

    Move mgt cli stuff to _mgt.c file

diff --git a/bin/varnishd/storage/stevedore.c b/bin/varnishd/storage/stevedore.c
index 9b22cd3..79b1292 100644
--- a/bin/varnishd/storage/stevedore.c
+++ b/bin/varnishd/storage/stevedore.c
@@ -39,7 +39,6 @@
 #include "cache/cache.h"
 
 #include "storage/storage.h"
-#include "vcli_priv.h"
 #include "vrt.h"
 #include "vrt_obj.h"
 
@@ -422,30 +421,6 @@ STV_close(void)
 		stv->close(stv);
 }
 
-/*--------------------------------------------------------------------*/
-
-static void
-stv_cli_list(struct cli *cli, const char * const *av, void *priv)
-{
-	struct stevedore *stv;
-
-	ASSERT_MGT();
-	(void)av;
-	(void)priv;
-	VCLI_Out(cli, "Storage devices:\n");
-	stv = stv_transient;
-		VCLI_Out(cli, "\tstorage.%s = %s\n", stv->ident, stv->name);
-	VTAILQ_FOREACH(stv, &stv_stevedores, list)
-		VCLI_Out(cli, "\tstorage.%s = %s\n", stv->ident, stv->name);
-}
-
-/*--------------------------------------------------------------------*/
-
-struct cli_proto cli_stv[] = {
-	{ "storage.list", "storage.list", "List storage devices\n",
-	    0, 0, "", stv_cli_list },
-	{ NULL}
-};
 
 /*--------------------------------------------------------------------
  * VRT functions for stevedores
diff --git a/bin/varnishd/storage/stevedore_mgt.c b/bin/varnishd/storage/stevedore_mgt.c
index fc0d6a6..4251839 100644
--- a/bin/varnishd/storage/stevedore_mgt.c
+++ b/bin/varnishd/storage/stevedore_mgt.c
@@ -42,12 +42,37 @@
 
 #include "storage/storage.h"
 #include "vav.h"
+#include "vcli_priv.h"
 
 struct stevedore_head stv_stevedores =
     VTAILQ_HEAD_INITIALIZER(stv_stevedores);
 
 struct stevedore *stv_transient;
 
+/*--------------------------------------------------------------------*/
+
+static void
+stv_cli_list(struct cli *cli, const char * const *av, void *priv)
+{
+	struct stevedore *stv;
+
+	ASSERT_MGT();
+	(void)av;
+	(void)priv;
+	VCLI_Out(cli, "Storage devices:\n");
+	stv = stv_transient;
+		VCLI_Out(cli, "\tstorage.%s = %s\n", stv->ident, stv->name);
+	VTAILQ_FOREACH(stv, &stv_stevedores, list)
+		VCLI_Out(cli, "\tstorage.%s = %s\n", stv->ident, stv->name);
+}
+
+/*--------------------------------------------------------------------*/
+
+struct cli_proto cli_stv[] = {
+	{ "storage.list", "storage.list", "List storage devices\n",
+	    0, 0, "", stv_cli_list },
+	{ NULL}
+};
 /*--------------------------------------------------------------------
  * Parse a stevedore argument on the form:
  *	[ name '=' ] strategy [ ',' arg ] *



From phk at varnish-cache.org  Thu Nov 17 09:18:32 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Thu, 17 Nov 2011 10:18:32 +0100
Subject: [master] 86aa540 Reduce #include scope to common.h
Message-ID: <E1RQy76-0002Ro-JZ@project.varnish-software.com>

commit 86aa540f2bb65ddc0b6bb7adf5559d08ab3f8a6b
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Nov 17 09:18:11 2011 +0000

    Reduce #include scope to common.h

diff --git a/bin/varnishd/waiter/cache_waiter.c b/bin/varnishd/waiter/cache_waiter.c
index c428294..e09a296 100644
--- a/bin/varnishd/waiter/cache_waiter.c
+++ b/bin/varnishd/waiter/cache_waiter.c
@@ -29,8 +29,11 @@
  */
 
 #include "config.h"
+#include <stdio.h>
+#include <unistd.h>
+#include <string.h>
 
-#include "cache/cache.h"
+#include "common/common.h"
 
 #include "waiter/cache_waiter.h"
 #include "vcli.h"



From drwilco at varnish-cache.org  Fri Nov 18 11:38:26 2011
From: drwilco at varnish-cache.org (Rogier Mulhuijzen)
Date: Fri, 18 Nov 2011 12:38:26 +0100
Subject: [master] bdbb1d5 Use the right counter for directors index
Message-ID: <E1RRMm2-0007Pm-1R@project.varnish-software.com>

commit bdbb1d59513cba8b268ed1dbe2d948619ef4ae07
Author: Rogier 'DocWilco' Mulhuijzen <github at bsdchicks.com>
Date:   Fri Nov 18 12:29:04 2011 +0100

    Use the right counter for directors index
    
    Fixes	#1060

diff --git a/bin/varnishtest/tests/r01060.vtc b/bin/varnishtest/tests/r01060.vtc
new file mode 100644
index 0000000..25a3988
--- /dev/null
+++ b/bin/varnishtest/tests/r01060.vtc
@@ -0,0 +1,48 @@
+varnishtest "DNS director tramples on other backends"
+
+server s1 {
+	rxreq
+	txresp
+} -start
+
+server s2 {
+	rxreq
+	txresp
+} -start
+
+varnish v1 -vcl {
+	backend b1 {
+		.host = "${s1_addr}";
+		.port = "${s1_port}";
+	}
+
+	director d1 dns {
+		.list = {
+			.port = "80";
+			"${bad_ip}"/32;
+		}
+		.ttl = 1m;
+	}
+
+	backend b2 {
+		.host = "${s2_addr}";
+		.port = "${s2_port}";
+	}
+
+	sub vcl_recv {
+		set req.backend = d1;
+		if (req.url == "/1") {
+			set req.backend = b1;
+		} else {
+			set req.backend = b2;
+		}
+	}
+} -start
+
+
+client c1 {
+	txreq -url "/1"
+	rxresp
+	txreq -url "/2"
+	rxresp
+} -run
diff --git a/lib/libvcl/vcc_dir_dns.c b/lib/libvcl/vcc_dir_dns.c
index a91c021..bdc0a21 100644
--- a/lib/libvcl/vcc_dir_dns.c
+++ b/lib/libvcl/vcc_dir_dns.c
@@ -79,7 +79,7 @@ print_backend(struct vcc *tl,
 	AN(vsb);
 	tl->fb = vsb;
 	Fc(tl, 0, "\t{ .host = VGC_backend_%s },\n",vgcname);
-	Fh(tl, 1, "\n#define VGC_backend_%s %d\n", vgcname, serial);
+	Fh(tl, 1, "\n#define VGC_backend_%s %d\n", vgcname, tl->ndirector);
 
 	Fb(tl, 0, "\nstatic const struct vrt_backend vgc_dir_priv_%s = {\n",
 	    vgcname);



From kristian at varnish-cache.org  Fri Nov 18 14:10:26 2011
From: kristian at varnish-cache.org (=?UTF-8?Q?Kristian_Lyngst=C3=B8l?=)
Date: Fri, 18 Nov 2011 15:10:26 +0100
Subject: [master] 4895d9f Update varnishtest(1) documentation somewhat
Message-ID: <E1RRP98-0005OE-Vb@project.varnish-software.com>

commit 4895d9f94730e5b5c4b73758c03708d21aa30f79
Author: Kristian Lyngstol <kristian at bohemians.org>
Date:   Fri Nov 18 15:09:51 2011 +0100

    Update varnishtest(1) documentation somewhat

diff --git a/doc/sphinx/reference/varnishtest.rst b/doc/sphinx/reference/varnishtest.rst
index c2aaae9..ff68337 100644
--- a/doc/sphinx/reference/varnishtest.rst
+++ b/doc/sphinx/reference/varnishtest.rst
@@ -7,14 +7,16 @@ Test program for Varnish
 ------------------------
 
 :Author: Stig Sandbeck Mathisen
-:Date:   2010-05-31
-:Version: 1.0
+:Author: Kristian Lyngst?l
+:Date:   2011-11-15
+:Version: 1.1
 :Manual section: 1
 
 
 SYNOPSIS
 ========
-     varnishtest [-n iter] [-q] [-v] file [file ...]
+
+     varnishtest [-iklLqv] [-n iter] [-D name=val] [-j jobs] [-t duration] file [file ...]
 
 DESCRIPTION
 ===========
@@ -24,160 +26,95 @@ Varnish Cache.
 
 The varnishtest program, when started and given one or more script
 files, can create a number of threads representing backends, some
-threads representing clients, and a varnishd process.
+threads representing clients, and a varnishd process. This is then used to
+simulate a transaction to provoke a specific behavior.
 
 The following options are available:
 
--n iter     Run iter number of iterations.
+-D name=val      Define macro for use in scripts
+
+-i               Find varnishd in build tree
+
+-j jobs          Run this many tests in parallel
+
+-k               Continue on test failure
+
+-l               Leave /tmp/vtc.* if test fails
+
+-L               Always leave /tmp/vtc.*
+
+-n iterations    Run tests this many times
+
+-q               Quiet mode: report only failures
+
+-t duration      Time tests out after this long
+
+-v               Verbose mode: always report test log
+
+-h               Show help
 
--q          Be quiet.
+file             File to use as a script
 
--v          Be verbose.
 
--t          Dunno.
+Macro definitions that can be overridden.
 
-file        File to use as a script
+varnishd         Path to varnishd to use [varnishd]
 
 
 SCRIPTS
 =======
 
-Example script
-~~~~~~~~~~~~~~
-::
-
-    # Start a varnish instance called "v1"
-    varnish v1 -arg "-b localhost:9080" -start
-    
-    # Create a server thread called "s1"
-    server s1 {
-        # Receive a request
-        rxreq
-        # Send a standard response
-        txresp -hdr "Connection: close" -body "012345\n"
-    }
-    
-    # Start the server thread
-    server s1 -start
-    
-    # Create a client thread called "c1"
-    client c1 {
-        # Send a request
-        txreq -url "/"
-        # Wait for a response
-        rxresp
-    # Insist that it be a success
-    expect resp.status == 200
-    }
-    
-    # Run the client
-    client c1 -run
-    
-    # Wait for the server to die
-    server s1 -wait
-
-    # (Forcefully) Stop the varnish instance.
-    varnish v1 -stop
-
-Example script output
-~~~~~~~~~~~~~~~~~~~~~
-
-The output, running this script looks as follows. The "bargraph" at
-the beginning of the line is an indication of the level of detail in
-the line. The second field where the message comes from. The rest of
-the line is anyones guess :-)
-::
-
-    #  TEST tests/b00000.vtc starting
-    ### v1  CMD: cd ../varnishd && ./varnishd -d -d -n v1 -a :9081 -T :9001 -b localhost:9080
-    ### v1  opening CLI connection
-    #### v1  debug| NB: Storage size limited to 2GB on 32 bit architecture,\n
-    #### v1  debug| NB: otherwise we could run out of address space.\n
-    #### v1  debug| storage_file: filename: ./varnish.Shkoq5 (unlinked) size 2047 MB.\n
-    ### v1  CLI connection fd = 3
-    #### v1  CLI TX| start
-    #### v1  debug| Using old SHMFILE\n
-    #### v1  debug| Notice: locking SHMFILE in core failed: Operation not permitted\n
-    #### v1  debug| bind(): Address already in use\n
-    #### v1  debug| rolling(1)...
-    #### v1  debug| \n
-    #### v1  debug| rolling(2)...\n
-    #### v1  debug| Debugging mode, enter "start" to start child\n
-    ### v1  CLI 200 <start>
-    ##  s1  Starting server
-    ### s1  listen on :9080 (fd 6)
-    ##  c1  Starting client
-    ##  c1  Waiting for client
-    ##  s1  started on :9080
-    ##  c1  started
-    ### c1  connect to :9081
-    ### c1  connected to :9081 fd is 8
-    #### c1  | GET / HTTP/1.1\r\n
-    #### c1  | \r\n
-    ### c1  rxresp
-    #### s1  Accepted socket 7
-    ### s1  rxreq
-    #### s1  | GET / HTTP/1.1\r\n
-    #### s1  | X-Varnish: 422080121\r\n
-    #### s1  | X-Forwarded-For: 127.0.0.1\r\n
-    #### s1  | Host: localhost\r\n
-    #### s1  | \r\n
-    #### s1  http[ 0] | GET
-    #### s1  http[ 1] | /
-    #### s1  http[ 2] | HTTP/1.1
-    #### s1  http[ 3] | X-Varnish: 422080121
-    #### s1  http[ 4] | X-Forwarded-For: 127.0.0.1
-    #### s1  http[ 5] | Host: localhost
-    #### s1  | HTTP/1.1 200 Ok\r\n
-    #### s1  | Connection: close\r\n
-    #### s1  | \r\n
-    #### s1  | 012345\n
-    #### s1  | \r\n
-    ##  s1  ending
-    #### c1  | HTTP/1.1 200 Ok\r\n
-    #### c1  | Content-Length: 9\r\n
-    #### c1  | Date: Mon, 16 Jun 2008 22:16:55 GMT\r\n
-    #### c1  | X-Varnish: 422080121\r\n
-    #### c1  | Age: 0\r\n
-    #### c1  | Via: 1.1 varnish\r\n
-    #### c1  | Connection: keep-alive\r\n
-    #### c1  | \r\n
-    #### c1  http[ 0] | HTTP/1.1
-    #### c1  http[ 1] | 200
-    #### c1  http[ 2] | Ok
-    #### c1  http[ 3] | Content-Length: 9
-    #### c1  http[ 4] | Date: Mon, 16 Jun 2008 22:16:55 GMT
-    #### c1  http[ 5] | X-Varnish: 422080121
-    #### c1  http[ 6] | Age: 0
-    #### c1  http[ 7] | Via: 1.1 varnish
-    #### c1  http[ 8] | Connection: keep-alive
-    #### c1  EXPECT resp.status (200) == 200 (200) match
-    ##  c1  ending
-    ##  s1  Waiting for server
-    #### v1  CLI TX| stop
-    ### v1  CLI 200 <stop>
-    #  TEST tests/b00000.vtc completed
-
-If instead of 200 we had expected 201 with the line:::
-
-  expect resp.status == 201
-
-The output would have ended with:::
-
-  #### c1  http[ 0] | HTTP/1.1
-  #### c1  http[ 1] | 200
-  #### c1  http[ 2] | Ok
-  #### c1  http[ 3] | Content-Length: 9
-  #### c1  http[ 4] | Date: Mon, 16 Jun 2008 22:26:35 GMT
-  #### c1  http[ 5] | X-Varnish: 648043653 648043652
-  #### c1  http[ 6] | Age: 6
-  #### c1  http[ 7] | Via: 1.1 varnish
-  #### c1  http[ 8] | Connection: keep-alive
-  ---- c1  EXPECT resp.status (200) == 201 (201) failed
+The script language used for Varnishtest is not a strictly defined
+language. The best reference for writing scripts is the varnishtest program
+itself. In the Varnish source code repository, under
+`bin/varnishtest/tests/`, all the regression tests for Varnish are kept.
+
+An example::
+
+        varnishtest "#1029"
+
+        server s1 {
+                rxreq
+                expect req.url == "/bar"
+                txresp -gzipbody {[bar]}
+
+                rxreq
+                expect req.url == "/foo"
+                txresp -body {<h1>FOO<esi:include src="/bar"/>BARF</h1>}
+
+        } -start
+
+        varnish v1 -vcl+backend {
+                sub vcl_fetch {
+                        set beresp.do_esi = true;
+                        if (req.url == "/foo") {
+                                set beresp.ttl = 0s;
+                        } else {
+                                set beresp.ttl = 10m;
+                        }
+                }
+        } -start
+
+        client c1 {
+                txreq -url "/bar" -hdr "Accept-Encoding: gzip"
+                rxresp
+                gunzip
+                expect resp.bodylen == 5
+
+                txreq -url "/foo" -hdr "Accept-Encoding: gzip"
+                rxresp
+                expect resp.bodylen == 21
+        } -run
+
+When run, the above script will simulate a server (s1) that expects two
+different requests. It will start a varnish server (v1) and add the backend
+definition to the VCL specified (-vcl+backend). Finally it starts the
+c1-client, which is a single client sending two requests.
 
 SEE ALSO
 ========
 
+* varnishtest source code repository with tests
 * varnishhist(1)
 * varnishlog(1)
 * varnishncsa(1)
@@ -190,9 +127,9 @@ HISTORY
 
 The varnishtest program was developed by Poul-Henning Kamp
 ?phk at phk.freebsd.dk? in cooperation with Varnish Software AS.
-This manual page
-was written by Stig Sandbeck Mathisen ?ssm at linpro.no? using examples
-by Poul-Henning Kamp ?phk at phk.freebsd.dk?.
+This manual page was originally written by Stig Sandbeck Mathisen
+?ssm at linpro.no? and updated by Kristian Lyngst?l
+(kristian at varnish-cache.org).
 
 COPYRIGHT
 =========
@@ -200,4 +137,4 @@ COPYRIGHT
 This document is licensed under the same licence as Varnish
 itself. See LICENCE for details.
 
-* Copyright (c) 2007-2008 Varnish Software AS
+* Copyright (c) 2007-2011 Varnish Software AS



From kristian at varnish-cache.org  Fri Nov 18 14:10:27 2011
From: kristian at varnish-cache.org (=?UTF-8?Q?Kristian_Lyngst=C3=B8l?=)
Date: Fri, 18 Nov 2011 15:10:27 +0100
Subject: [master] ab9cec7 Merge branch 'master' of
	git+ssh://git.varnish-cache.org/git/varnish-cache
Message-ID: <E1RRP99-0005OK-4o@project.varnish-software.com>

commit ab9cec7c2c81bee78a388ade260f1b18beef8ba7
Merge: 4895d9f bdbb1d5
Author: Kristian Lyngstol <kristian at bohemians.org>
Date:   Fri Nov 18 15:10:05 2011 +0100

    Merge branch 'master' of git+ssh://git.varnish-cache.org/git/varnish-cache



From phk at varnish-cache.org  Fri Nov 18 19:03:51 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Fri, 18 Nov 2011 20:03:51 +0100
Subject: [master] aaad3cf Rename to clean up namespace a bit
Message-ID: <E1RRTj5-0003wt-Sw@project.varnish-software.com>

commit aaad3cf561fc2f938155fad4f66c4ee46a69dc64
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Nov 17 09:33:11 2011 +0000

    Rename to clean up namespace a bit

diff --git a/bin/varnishd/Makefile.am b/bin/varnishd/Makefile.am
index 359373d..2116701 100644
--- a/bin/varnishd/Makefile.am
+++ b/bin/varnishd/Makefile.am
@@ -74,7 +74,7 @@ varnishd_SOURCES = \
 	storage/storage_persistent_subr.c \
 	storage/storage_synth.c \
 	storage/storage_umem.c \
-	waiter/cache_waiter.c \
+	waiter/waiter_common.c \
 	waiter/cache_waiter_epoll.c \
 	waiter/cache_waiter_kqueue.c \
 	waiter/cache_waiter_poll.c \
diff --git a/bin/varnishd/waiter/cache_waiter.c b/bin/varnishd/waiter/cache_waiter.c
deleted file mode 100644
index e09a296..0000000
--- a/bin/varnishd/waiter/cache_waiter.c
+++ /dev/null
@@ -1,113 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2011 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- */
-
-#include "config.h"
-#include <stdio.h>
-#include <unistd.h>
-#include <string.h>
-
-#include "common/common.h"
-
-#include "waiter/cache_waiter.h"
-#include "vcli.h"
-#include "vcli_priv.h"
-
-static const struct waiter * const vca_waiters[] = {
-    #if defined(HAVE_KQUEUE)
-	&waiter_kqueue,
-    #endif
-    #if defined(HAVE_EPOLL_CTL)
-	&waiter_epoll,
-    #endif
-    #if defined(HAVE_PORT_CREATE)
-	&waiter_ports,
-    #endif
-	&waiter_poll,
-	NULL,
-};
-
-struct waiter const * waiter;
-
-const char *
-WAIT_GetName(void)
-{
-
-	if (waiter != NULL)
-		return (waiter->name);
-	else
-		return ("no_waiter");
-}
-
-void
-WAIT_tweak_waiter(struct cli *cli, const char *arg)
-{
-	int i;
-
-	ASSERT_MGT();
-
-	if (arg == NULL) {
-		if (waiter == NULL)
-			VCLI_Out(cli, "default");
-		else
-			VCLI_Out(cli, "%s", waiter->name);
-
-		VCLI_Out(cli, " (");
-		for (i = 0; vca_waiters[i] != NULL; i++)
-			VCLI_Out(cli, "%s%s", i == 0 ? "" : ", ",
-			    vca_waiters[i]->name);
-		VCLI_Out(cli, ")");
-		return;
-	}
-	if (!strcmp(arg, "default")) {
-		waiter = NULL;
-		return;
-	}
-	for (i = 0; vca_waiters[i]; i++) {
-		if (!strcmp(arg, vca_waiters[i]->name)) {
-			waiter = vca_waiters[i];
-			return;
-		}
-	}
-	VCLI_Out(cli, "Unknown waiter");
-	VCLI_SetResult(cli, CLIS_PARAM);
-}
-
-void
-WAIT_Init(void)
-{
-
-	if (waiter == NULL)
-		waiter = vca_waiters[0];
-
-	AN(waiter);
-	AN(waiter->name);
-	AN(waiter->init);
-	AN(waiter->pass);
-}
diff --git a/bin/varnishd/waiter/waiter_common.c b/bin/varnishd/waiter/waiter_common.c
new file mode 100644
index 0000000..e09a296
--- /dev/null
+++ b/bin/varnishd/waiter/waiter_common.c
@@ -0,0 +1,113 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ */
+
+#include "config.h"
+#include <stdio.h>
+#include <unistd.h>
+#include <string.h>
+
+#include "common/common.h"
+
+#include "waiter/cache_waiter.h"
+#include "vcli.h"
+#include "vcli_priv.h"
+
+static const struct waiter * const vca_waiters[] = {
+    #if defined(HAVE_KQUEUE)
+	&waiter_kqueue,
+    #endif
+    #if defined(HAVE_EPOLL_CTL)
+	&waiter_epoll,
+    #endif
+    #if defined(HAVE_PORT_CREATE)
+	&waiter_ports,
+    #endif
+	&waiter_poll,
+	NULL,
+};
+
+struct waiter const * waiter;
+
+const char *
+WAIT_GetName(void)
+{
+
+	if (waiter != NULL)
+		return (waiter->name);
+	else
+		return ("no_waiter");
+}
+
+void
+WAIT_tweak_waiter(struct cli *cli, const char *arg)
+{
+	int i;
+
+	ASSERT_MGT();
+
+	if (arg == NULL) {
+		if (waiter == NULL)
+			VCLI_Out(cli, "default");
+		else
+			VCLI_Out(cli, "%s", waiter->name);
+
+		VCLI_Out(cli, " (");
+		for (i = 0; vca_waiters[i] != NULL; i++)
+			VCLI_Out(cli, "%s%s", i == 0 ? "" : ", ",
+			    vca_waiters[i]->name);
+		VCLI_Out(cli, ")");
+		return;
+	}
+	if (!strcmp(arg, "default")) {
+		waiter = NULL;
+		return;
+	}
+	for (i = 0; vca_waiters[i]; i++) {
+		if (!strcmp(arg, vca_waiters[i]->name)) {
+			waiter = vca_waiters[i];
+			return;
+		}
+	}
+	VCLI_Out(cli, "Unknown waiter");
+	VCLI_SetResult(cli, CLIS_PARAM);
+}
+
+void
+WAIT_Init(void)
+{
+
+	if (waiter == NULL)
+		waiter = vca_waiters[0];
+
+	AN(waiter);
+	AN(waiter->name);
+	AN(waiter->init);
+	AN(waiter->pass);
+}



From phk at varnish-cache.org  Fri Nov 18 19:03:52 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Fri, 18 Nov 2011 20:03:52 +0100
Subject: [master] 8921e99 This started out as something entirely different,
	but I got tired of the fact that we have so many params in units of
	bytes which you cannot tell "64k" so now you can.
Message-ID: <E1RRTj6-0003x0-2b@project.varnish-software.com>

commit 8921e9908fb557b2128e99235bdddb4b23a48d64
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Nov 18 19:02:16 2011 +0000

    This started out as something entirely different, but I got tired
    of the fact that we have so many params in units of bytes which
    you cannot tell "64k" so now you can.
    
    Notice that a couple of specialist params (fetch_chunksize and
    fetch_maxchunksize) have changed units from kilobytes to bytes,
    but the default values are the same.

diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index a678dcc..a5c0323 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -188,7 +188,7 @@ FetchStorage(struct worker *w, ssize_t sz)
 	if (l == 0)
 		l = sz;
 	if (l == 0)
-		l = cache_param->fetch_chunksize * 1024LL;
+		l = cache_param->fetch_chunksize;
 	st = STV_alloc(w, l);
 	if (st == NULL) {
 		(void)FetchError(w, "Could not get storage");
diff --git a/bin/varnishd/common/params.h b/bin/varnishd/common/params.h
index ee8bad2..51721b1 100644
--- a/bin/varnishd/common/params.h
+++ b/bin/varnishd/common/params.h
@@ -87,13 +87,13 @@ struct params {
 	unsigned		auto_restart;
 
 	/* Fetcher hints */
-	unsigned		fetch_chunksize;
-	unsigned		fetch_maxchunksize;
+	ssize_t			fetch_chunksize;
+	ssize_t			fetch_maxchunksize;
 	unsigned		nuke_limit;
 
 #ifdef SENDFILE_WORKS
 	/* Sendfile object minimum size */
-	unsigned		sendfile_threshold;
+	ssize_t			sendfile_threshold;
 #endif
 
 	/* VCL traces */
diff --git a/bin/varnishd/mgt/mgt_param.c b/bin/varnishd/mgt/mgt_param.c
index b7c18b2..10e64b5 100644
--- a/bin/varnishd/mgt/mgt_param.c
+++ b/bin/varnishd/mgt/mgt_param.c
@@ -31,6 +31,7 @@
 
 #include <grp.h>
 #include <limits.h>
+#include <math.h>
 #include <pwd.h>
 #include <stdio.h>
 #include <stdlib.h>
@@ -47,6 +48,7 @@
 #include "vcli.h"
 #include "vcli_common.h"
 #include "vcli_priv.h"
+#include "vnum.h"
 #include "vss.h"
 
 #include "mgt_cli.h"
@@ -244,6 +246,97 @@ tweak_uint(struct cli *cli, const struct parspec *par, const char *arg)
 	tweak_generic_uint(cli, dest, arg, (uint)par->min, (uint)par->max);
 }
 
+/*--------------------------------------------------------------------*/
+
+static void
+fmt_bytes(struct cli *cli, ssize_t t)
+{
+	const char *p;
+
+	if (t & 0xff) {
+		VCLI_Out(cli, "%zub", t);
+		return;
+	}
+	for (p = "kMGTPEZY"; *p; p++) {
+		if (t & 0x300) {
+			VCLI_Out(cli, "%.2f%c", t / 1024.0, *p);
+			return;
+		}
+		t /= 1024;
+		if (t & 0x0ff) {
+			VCLI_Out(cli, "%zu%c", t, *p);
+			return;
+		}
+	}
+	VCLI_Out(cli, "(bogus number)");
+}
+
+static void
+tweak_generic_bytes(struct cli *cli, volatile ssize_t *dest, const char *arg,
+    double min, double max)
+{
+	uintmax_t r;
+	const char *p;
+
+	if (arg != NULL) {
+		p = VNUM_2bytes(arg, &r, 0);
+		if (p != NULL) {
+			VCLI_Out(cli, "Could not convert to bytes.\n");
+			VCLI_Out(cli, "%s\n", p);
+			VCLI_Out(cli,
+			    "  Try something like '80k' or '120M'\n");
+			VCLI_SetResult(cli, CLIS_PARAM);
+			return;
+		}
+		if ((uintmax_t)((ssize_t)r) != r || r > max) {
+			VCLI_Out(cli, "Must be no more than ");
+			fmt_bytes(cli, (ssize_t)max);
+			VCLI_Out(cli, "\n");
+			VCLI_SetResult(cli, CLIS_PARAM);
+			return;
+		}
+		if (r < min) {
+			VCLI_Out(cli, "Must be at least ");
+			fmt_bytes(cli, (ssize_t)min);
+			VCLI_Out(cli, "\n");
+			VCLI_SetResult(cli, CLIS_PARAM);
+			return;
+		}
+		*dest = r;
+	} else {
+		fmt_bytes(cli, *dest);
+	}
+}
+
+/*--------------------------------------------------------------------*/
+
+static void
+tweak_bytes(struct cli *cli, const struct parspec *par, const char *arg)
+{
+	volatile ssize_t *dest;
+
+	assert(par->min >= 0);
+	dest = par->priv;
+	tweak_generic_bytes(cli, dest, arg, par->min, par->max);
+}
+
+
+/*--------------------------------------------------------------------*/
+
+static void
+tweak_bytes_u(struct cli *cli, const struct parspec *par, const char *arg)
+{
+	volatile unsigned *d1;
+	volatile ssize_t dest;
+
+	assert(par->max <= UINT_MAX);
+	assert(par->min >= 0);
+	d1 = par->priv;
+	dest = *d1;
+	tweak_generic_bytes(cli, &dest, arg, par->min, par->max);
+	*d1 = dest;
+}
+
 /*--------------------------------------------------------------------
  * XXX: slightly magic.  We want to initialize to "nobody" (XXX: shouldn't
  * XXX: that be something autocrap found for us ?) but we don't want to
@@ -511,21 +604,23 @@ static const struct parspec input_parspec[] = {
 		"flush of the cache use \"ban.url .\"",
 		0,
 		"120", "seconds" },
-	{ "sess_workspace", tweak_uint, &mgt_param.sess_workspace, 1024, UINT_MAX,
+	{ "sess_workspace",
+		tweak_bytes_u, &mgt_param.sess_workspace, 1024, UINT_MAX,
 		"Bytes of HTTP protocol workspace allocated for sessions. "
 		"This space must be big enough for the entire HTTP protocol "
 		"header and any edits done to it in the VCL code.\n"
 		"Minimum is 1024 bytes.",
 		DELAYED_EFFECT,
-		"65536",
-		"bytes" },
-	{ "http_req_hdr_len", tweak_uint, &mgt_param.http_req_hdr_len,
+		"64k", "bytes" },
+	{ "http_req_hdr_len",
+		tweak_bytes_u, &mgt_param.http_req_hdr_len,
 		40, UINT_MAX,
 		"Maximum length of any HTTP client request header we will "
 		"allow.  The limit is inclusive its continuation lines.\n",
 		0,
-		"8192", "bytes" },
-	{ "http_req_size", tweak_uint, &mgt_param.http_req_size,
+		"8k", "bytes" },
+	{ "http_req_size",
+		tweak_bytes_u, &mgt_param.http_req_size,
 		256, UINT_MAX,
 		"Maximum number of bytes of HTTP client request we will deal "
 		"with.  This is a limit on all bytes up to the double blank "
@@ -534,14 +629,16 @@ static const struct parspec input_parspec[] = {
 		"workspace (param: sess_workspace) and this parameter limits "
 		"how much of that the request is allowed to take up.",
 		0,
-		"32768", "bytes" },
-	{ "http_resp_hdr_len", tweak_uint, &mgt_param.http_resp_hdr_len,
+		"32k", "bytes" },
+	{ "http_resp_hdr_len",
+		tweak_bytes_u, &mgt_param.http_resp_hdr_len,
 		40, UINT_MAX,
 		"Maximum length of any HTTP backend response header we will "
 		"allow.  The limit is inclusive its continuation lines.\n",
 		0,
-		"8192", "bytes" },
-	{ "http_resp_size", tweak_uint, &mgt_param.http_resp_size,
+		"8k", "bytes" },
+	{ "http_resp_size",
+		tweak_bytes_u, &mgt_param.http_resp_size,
 		256, UINT_MAX,
 		"Maximum number of bytes of HTTP backend resonse we will deal "
 		"with.  This is a limit on all bytes up to the double blank "
@@ -550,7 +647,7 @@ static const struct parspec input_parspec[] = {
 		"workspace (param: sess_workspace) and this parameter limits "
 		"how much of that the request is allowed to take up.",
 		0,
-		"32768", "bytes" },
+		"32k", "bytes" },
 	{ "http_max_hdr", tweak_uint, &mgt_param.http_max_hdr, 32, 65535,
 		"Maximum number of HTTP headers we will deal with in "
 		"client request or backend reponses.  "
@@ -559,7 +656,8 @@ static const struct parspec input_parspec[] = {
 		"objects allocate exact space for the headers they store.\n",
 		0,
 		"64", "header lines" },
-	{ "shm_workspace", tweak_uint, &mgt_param.shm_workspace, 4096, UINT_MAX,
+	{ "shm_workspace",
+		tweak_bytes_u, &mgt_param.shm_workspace, 4096, UINT_MAX,
 		"Bytes of shmlog workspace allocated for worker threads. "
 		"If too big, it wastes some ram, if too small it causes "
 		"needless flushes of the SHM workspace.\n"
@@ -567,8 +665,9 @@ static const struct parspec input_parspec[] = {
 		"\"SHM flushes due to overflow\".\n"
 		"Minimum is 4096 bytes.",
 		DELAYED_EFFECT,
-		"8192", "bytes" },
-	{ "shm_reclen", tweak_uint, &mgt_param.shm_reclen, 16, 65535,
+		"8k", "bytes" },
+	{ "shm_reclen",
+		tweak_bytes_u, &mgt_param.shm_reclen, 16, 65535,
 		"Maximum number of bytes in SHM log record.\n"
 		"Maximum is 65535 bytes.",
 		0,
@@ -633,27 +732,29 @@ static const struct parspec input_parspec[] = {
 		EXPERIMENTAL,
 		"50", "allocations" },
 	{ "fetch_chunksize",
-		tweak_uint, &mgt_param.fetch_chunksize, 4, UINT_MAX / 1024.,
+		tweak_bytes_u,
+		    &mgt_param.fetch_chunksize, 4 * 1024, UINT_MAX,
 		"The default chunksize used by fetcher. "
 		"This should be bigger than the majority of objects with "
 		"short TTLs.\n"
 		"Internal limits in the storage_file module makes increases "
 		"above 128kb a dubious idea.",
 		EXPERIMENTAL,
-		"128", "kilobytes" },
+		"128k", "bytes" },
 	{ "fetch_maxchunksize",
-		tweak_uint, &mgt_param.fetch_maxchunksize, 64, UINT_MAX / 1024.,
+		tweak_bytes_u,
+		    &mgt_param.fetch_maxchunksize, 64 * 1024, UINT_MAX,
 		"The maximum chunksize we attempt to allocate from storage. "
 		"Making this too large may cause delays and storage "
 		"fragmentation.\n",
 		EXPERIMENTAL,
-		"262144", "kilobytes" },
+		"256m", "bytes" },
 #ifdef SENDFILE_WORKS
 	{ "sendfile_threshold",
-		tweak_uint, &mgt_param.sendfile_threshold, 0, UINT_MAX,
+		tweak_bytes, &mgt_param.sendfile_threshold, 0, HUGE_VAL,
 		"The minimum size of objects transmitted with sendfile.",
 		EXPERIMENTAL,
-		"-1", "bytes" },
+		"1E", "bytes" },
 #endif /* SENDFILE_WORKS */
 	{ "vcl_trace", tweak_bool,  &mgt_param.vcl_trace, 0, 0,
 		"Trace VCL execution in the shmlog.\n"
@@ -673,19 +774,21 @@ static const struct parspec input_parspec[] = {
 		"Listen queue depth.",
 		MUST_RESTART,
 		"1024", "connections" },
-	{ "cli_buffer", tweak_uint, &mgt_param.cli_buffer, 4096, UINT_MAX,
+	{ "cli_buffer",
+		tweak_bytes_u, &mgt_param.cli_buffer, 4096, UINT_MAX,
 		"Size of buffer for CLI command input."
 		"\nYou may need to increase this if you have big VCL files "
 		"and use the vcl.inline CLI command.\n"
 		"NB: Must be specified with -p to have effect.\n",
 		0,
-		"8192", "bytes" },
-	{ "cli_limit", tweak_uint, &mgt_param.cli_limit, 128, 99999999,
+		"8k", "bytes" },
+	{ "cli_limit",
+		tweak_bytes_u, &mgt_param.cli_limit, 128, 99999999,
 		"Maximum size of CLI response.  If the response exceeds"
 		" this limit, the reponse code will be 201 instead of"
 		" 200 and the last line will indicate the truncation.",
 		0,
-		"4096", "bytes" },
+		"4k", "bytes" },
 	{ "cli_timeout", tweak_timeout, &mgt_param.cli_timeout, 0, 0,
 		"Timeout for the childs replies to CLI requests from "
 		"the mgt_param.",
@@ -918,7 +1021,8 @@ static const struct parspec input_parspec[] = {
 		"Memory impact is 1=1k, 2=2k, ... 9=256k.",
 		0,
 		"8", ""},
-	{ "gzip_stack_buffer", tweak_uint, &mgt_param.gzip_stack_buffer,
+	{ "gzip_stack_buffer",
+		tweak_bytes_u, &mgt_param.gzip_stack_buffer,
 	        2048, UINT_MAX,
 		"Size of stack buffer used for gzip processing.\n"
 		"The stack buffers are used for in-transit data,"
@@ -927,7 +1031,7 @@ static const struct parspec input_parspec[] = {
 		" writes to sockets etc, making it too big is probably"
 		" just a waste of memory.",
 		EXPERIMENTAL,
-		"32768", "Bytes" },
+		"32k", "bytes" },
 	{ "shortlived", tweak_timeout_double,
 		&mgt_param.shortlived, 0, UINT_MAX,
 		"Objects created with TTL shorter than this are always "
diff --git a/bin/varnishd/storage/stevedore.c b/bin/varnishd/storage/stevedore.c
index 79b1292..860604e 100644
--- a/bin/varnishd/storage/stevedore.c
+++ b/bin/varnishd/storage/stevedore.c
@@ -162,8 +162,8 @@ stv_alloc(struct worker *w, const struct object *obj, size_t size)
 	stv = obj->objstore->stevedore;
 	CHECK_OBJ_NOTNULL(stv, STEVEDORE_MAGIC);
 
-	if (size > (size_t)(cache_param->fetch_maxchunksize) << 10)
-		size = (size_t)(cache_param->fetch_maxchunksize) << 10;
+	if (size > cache_param->fetch_maxchunksize)
+		size = cache_param->fetch_maxchunksize;
 
 	for (;;) {
 		/* try to allocate from it */
@@ -172,7 +172,7 @@ stv_alloc(struct worker *w, const struct object *obj, size_t size)
 		if (st != NULL)
 			break;
 
-		if (size > cache_param->fetch_chunksize * 1024LL) {
+		if (size > cache_param->fetch_chunksize) {
 			size >>= 1;
 			continue;
 		}
diff --git a/bin/varnishtest/tests/g00002.vtc b/bin/varnishtest/tests/g00002.vtc
index 1a0ab35..1bf3384 100644
--- a/bin/varnishtest/tests/g00002.vtc
+++ b/bin/varnishtest/tests/g00002.vtc
@@ -22,7 +22,7 @@ varnish v1 \
 	}
 } -start
 
-varnish v1 -cliok "param.set fetch_chunksize 4"
+varnish v1 -cliok "param.set fetch_chunksize 4k"
 
 client c1 {
 	txreq -url /foo -hdr "Accept-Encoding: gzip"
diff --git a/bin/varnishtest/tests/r00776.vtc b/bin/varnishtest/tests/r00776.vtc
index 776810e..53cc37e 100644
--- a/bin/varnishtest/tests/r00776.vtc
+++ b/bin/varnishtest/tests/r00776.vtc
@@ -8,7 +8,7 @@ server s1 {
 } -start
 
 varnish v1 \
-	-arg "-p fetch_chunksize=4" \
+	-arg "-p fetch_chunksize=4k" \
 	-arg "-s malloc,1m" -vcl+backend { } -start
 
 client c1 {



From phk at varnish-cache.org  Fri Nov 18 19:03:52 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Fri, 18 Nov 2011 20:03:52 +0100
Subject: [master] 085f6d2 Merge branch 'master' of
	ssh://git.varnish-cache.org/git/varnish-cache
Message-ID: <E1RRTj6-0003x9-8x@project.varnish-software.com>

commit 085f6d2dec9c05259f94a8282eac8a118b0e8baa
Merge: 8921e99 ab9cec7
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Nov 18 19:03:48 2011 +0000

    Merge branch 'master' of ssh://git.varnish-cache.org/git/varnish-cache



From phk at varnish-cache.org  Fri Nov 18 19:33:30 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Fri, 18 Nov 2011 20:33:30 +0100
Subject: [master] b40bdeb Convert the -l argument into two params. For now
	setting them any other way than -l does you no good.
Message-ID: <E1RRUBm-0000nx-KU@project.varnish-software.com>

commit b40bdeb066636492b43eb863a002d191aeac557f
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Nov 18 19:33:02 2011 +0000

    Convert the -l argument into two params.  For now setting them any
    other way than -l does you no good.

diff --git a/bin/varnishd/common/params.h b/bin/varnishd/common/params.h
index 51721b1..95258fb 100644
--- a/bin/varnishd/common/params.h
+++ b/bin/varnishd/common/params.h
@@ -186,6 +186,10 @@ struct params {
 	double			shortlived;
 
 	struct vre_limits	vre_limits;
+
+	/* VSM dimensions */
+	ssize_t			vsm_space;
+	ssize_t			vsl_space;
 };
 
 /*
diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h
index 1eabe10..0650413 100644
--- a/bin/varnishd/mgt/mgt.h
+++ b/bin/varnishd/mgt/mgt.h
@@ -81,7 +81,7 @@ void mgt_sandbox_solaris_privsep(void);
 #endif
 
 /* mgt_shmem.c */
-void mgt_SHM_Init(const char *arg);
+void mgt_SHM_Init(void);
 void mgt_SHM_Pid(void);
 
 /* stevedore_mgt.c */
diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c
index 8a9e1fc..ff8992e 100644
--- a/bin/varnishd/mgt/mgt_main.c
+++ b/bin/varnishd/mgt/mgt_main.c
@@ -336,7 +336,6 @@ main(int argc, char * const *argv)
 	const char *b_arg = NULL;
 	const char *f_arg = NULL;
 	const char *i_arg = NULL;
-	const char *l_arg = NULL;	/* default in mgt_shmem.c */
 	const char *h_arg = "critbit";
 	const char *M_arg = NULL;
 	const char *n_arg = NULL;
@@ -349,6 +348,7 @@ main(int argc, char * const *argv)
 	struct cli cli[1];
 	struct vpf_fh *pfh = NULL;
 	char *dirname;
+	char **av;
 	unsigned clilim;
 
 	/*
@@ -459,7 +459,19 @@ main(int argc, char * const *argv)
 			i_arg = optarg;
 			break;
 		case 'l':
-			l_arg = optarg;
+			av = VAV_Parse(optarg, NULL, ARGV_COMMA);
+			AN(av);
+			if (av[0] != NULL)
+				ARGV_ERR("\t-l ...: %s", av[0]);
+			if (av[1] != NULL) {
+				MCF_ParamSet(cli, "vsl_space", av[1]);
+				cli_check(cli);
+			}
+			if (av[1] != NULL && av[2] != NULL) {
+				MCF_ParamSet(cli, "vsm_space", av[2]);
+				cli_check(cli);
+			}
+			VAV_Free(av);
 			break;
 		case 'M':
 			M_arg = optarg;
@@ -611,7 +623,7 @@ main(int argc, char * const *argv)
 
 	HSH_config(h_arg);
 
-	mgt_SHM_Init(l_arg);
+	mgt_SHM_Init();
 
 	AZ(VSB_finish(vident));
 
diff --git a/bin/varnishd/mgt/mgt_param.c b/bin/varnishd/mgt/mgt_param.c
index 10e64b5..f6de562 100644
--- a/bin/varnishd/mgt/mgt_param.c
+++ b/bin/varnishd/mgt/mgt_param.c
@@ -1085,6 +1085,26 @@ static const struct parspec input_parspec[] = {
 		0,
 		"10000", ""},
 
+	{ "vsl_space", tweak_bytes,
+		&mgt_param.vsl_space, 1024*1024, HUGE_VAL,
+		"The amount of space to allocate for the VSL fifo buffer"
+		" in the VSM memory segment."
+		"  If you make this too small, varnish{ncsa|log} etc will"
+		" not be able to keep up."
+		"  Making it too large just costs memory resources.",
+		MUST_RESTART,
+		"80M", "bytes"},
+
+	{ "vsm_space", tweak_bytes,
+		&mgt_param.vsm_space, 1024*1024, HUGE_VAL,
+		"The amount of space to allocate for stats counters"
+		" in the VSM memory segment."
+		"  If you make this too small, some counters will be"
+		" invisible."
+		"  Making it too large just costs memory resources.",
+		MUST_RESTART,
+		"1M", "bytes"},
+
 	{ NULL, NULL, NULL }
 };
 
diff --git a/bin/varnishd/mgt/mgt_shmem.c b/bin/varnishd/mgt/mgt_shmem.c
index 24bc396..d1bf18e 100644
--- a/bin/varnishd/mgt/mgt_shmem.c
+++ b/bin/varnishd/mgt/mgt_shmem.c
@@ -105,9 +105,7 @@
 #include "vapi/vsc_int.h"
 #include "vapi/vsl_int.h"
 #include "vapi/vsm_int.h"
-#include "vav.h"
 #include "vmb.h"
-#include "vnum.h"
 
 #ifndef MAP_HASSEMAPHORE
 #define MAP_HASSEMAPHORE 0 /* XXX Linux */
@@ -227,67 +225,15 @@ mgt_shm_atexit(void)
 }
 
 void
-mgt_SHM_Init(const char *l_arg)
+mgt_SHM_Init(void)
 {
 	int i, fill;
-	const char *q;
-	uintmax_t size, s1, s2, ps;
-	char **av, **ap;
+	uintmax_t size, ps;
 	uint32_t *vsl_log_start;
 
-	if (l_arg == NULL)
-		l_arg = "";
+	fill = 1;
 
-	av = VAV_Parse(l_arg, NULL, ARGV_COMMA);
-	AN(av);
-	if (av[0] != NULL)
-		ARGV_ERR("\t-l ...: %s", av[0]);
-
-	ap = av + 1;
-
-	/* Size of SHMLOG */
-	if (*ap != NULL && **ap != '\0') {
-		q = VNUM_2bytes(*ap, &s1, 0);
-		if (q != NULL)
-			ARGV_ERR("\t-l[1] ...:  %s\n", q);
-	} else {
-		s1 = 80 * 1024 * 1024;
-	}
-	if (*ap != NULL)
-		ap++;
-
-	/* Size of space for other stuff */
-	if (*ap != NULL && **ap != '\0') {
-		q = VNUM_2bytes(*ap, &s2, 0);
-		if (q != NULL)
-			ARGV_ERR("\t-l[2] ...:  %s\n", q);
-	} else {
-		s2 = 1024 * 1024;
-	}
-	if (*ap != NULL)
-		ap++;
-
-	/* Fill or not ? */
-	if (*ap != NULL) {
-		if (**ap == '\0')
-			fill = 1;
-		else if (!strcmp(*ap, "-"))
-			fill = 0;
-		else if (!strcmp(*ap, "+"))
-			fill = 1;
-		else
-			ARGV_ERR("\t-l[3] ...:  Must be \"-\" or \"+\"\n");
-		ap++;
-	} else {
-		fill = 1;
-	}
-
-	if (*ap != NULL)
-		ARGV_ERR("\t-l ...:  Too many sub-args\n");
-
-	VAV_Free(av);
-
-	size = s1 + s2;
+	size = mgt_param.vsl_space + mgt_param.vsm_space;
 	ps = getpagesize();
 	size += ps - 1;
 	size &= ~(ps - 1);
@@ -326,7 +272,7 @@ mgt_SHM_Init(const char *l_arg)
 	AN(cache_param);
 	*cache_param = mgt_param;
 
-	vsl_log_start = VSM_Alloc(s1, VSL_CLASS, "", "");
+	vsl_log_start = VSM_Alloc(mgt_param.vsl_space, VSL_CLASS, "", "");
 	AN(vsl_log_start);
 	vsl_log_start[1] = VSL_ENDMARKER;
 	VWMB();



From phk at varnish-cache.org  Sun Nov 20 23:39:22 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 21 Nov 2011 00:39:22 +0100
Subject: [master] 356d46a First part of VSM overhaul,
	this compiles and varnishd runs,
	but varnishapi and users do not work yet.
Message-ID: <E1RSGyo-000874-8x@project.varnish-software.com>

commit 356d46a8fdb03f7179c757aa4d2160c238321f9d
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Nov 18 22:18:06 2011 +0000

    First part of VSM overhaul, this compiles and varnishd runs,
    but varnishapi and users do not work yet.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index bae2572..432e22b 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -853,7 +853,7 @@ int SES_Schedule(struct sess *sp);
 void VSL_Init(void);
 void *VSM_Alloc(unsigned size, const char *class, const char *type,
     const char *ident);
-void VSM_Free(const void *ptr);
+void VSM_Free(void *ptr);
 #ifdef VSL_ENDMARKER
 void VSL(enum VSL_tag_e tag, int id, const char *fmt, ...);
 void WSLR(struct worker *w, enum VSL_tag_e tag, int id, txt t);
diff --git a/bin/varnishd/cache/cache_shmlog.c b/bin/varnishd/cache/cache_shmlog.c
index e004896..63157c7 100644
--- a/bin/varnishd/cache/cache_shmlog.c
+++ b/bin/varnishd/cache/cache_shmlog.c
@@ -30,8 +30,10 @@
 #include "config.h"
 
 #include <stdio.h>
+#include <stdlib.h>
 
 #include "cache.h"
+#include "common/heritage.h"
 
 #include "cache_backend.h"	// For w->vbc
 
@@ -300,25 +302,32 @@ WSLB(struct worker *w, enum VSL_tag_e tag, const char *fmt, ...)
 void
 VSL_Init(void)
 {
-	struct VSM_chunk *vsc;
+	uint32_t *vsl_log_start;
 
 	AZ(pthread_mutex_init(&vsl_mtx, NULL));
 	AZ(pthread_mutex_init(&vsm_mtx, NULL));
 
-	VSM__Clean();
+	vsl_log_start = VSM_Alloc(cache_param->vsl_space, VSL_CLASS, "", "");
+	AN(vsl_log_start);
+	vsl_log_start[1] = VSL_ENDMARKER;
+	VWMB();
+	do
+		*vsl_log_start = random() & 0xffff;
+	while (*vsl_log_start == 0);
+	VWMB();
 
-	VSM_ITER(vsc)
-		if (!strcmp(vsc->class, VSL_CLASS))
-			break;
-	AN(vsc);
-	vsl_start = VSM_PTR(vsc);
-	vsl_end = VSM_NEXT(vsc);
+	vsl_start = vsl_log_start;
+	vsl_end = vsl_start + cache_param->vsl_space;
 	vsl_ptr = vsl_start + 1;
 
+	VSC_C_main = VSM_Alloc(sizeof *VSC_C_main,
+	    VSC_CLASS, VSC_TYPE_MAIN, "");
+	AN(VSC_C_main);
+
 	vsl_wrap();
-	VSM_head->starttime = (intmax_t)VTIM_real();
+	// VSM_head->starttime = (intmax_t)VTIM_real();
 	memset(VSC_C_main, 0, sizeof *VSC_C_main);
-	VSM_head->child_pid = getpid();
+	// VSM_head->child_pid = getpid();
 }
 
 /*--------------------------------------------------------------------*/
@@ -327,19 +336,19 @@ void *
 VSM_Alloc(unsigned size, const char *class, const char *type,
     const char *ident)
 {
-	void *p;
+	volatile void *p;
 
 	AZ(pthread_mutex_lock(&vsm_mtx));
-	p = VSM__Alloc(size, class, type, ident);
+	p = VSM_common_alloc(heritage.vsm, size, class, type, ident);
 	AZ(pthread_mutex_unlock(&vsm_mtx));
-	return (p);
+	return (TRUST_ME(p));
 }
 
 void
-VSM_Free(const void *ptr)
+VSM_Free(void *ptr)
 {
 
 	AZ(pthread_mutex_lock(&vsm_mtx));
-	VSM__Free(ptr);
+	VSM_common_free(heritage.vsm, ptr);
 	AZ(pthread_mutex_unlock(&vsm_mtx));
 }
diff --git a/bin/varnishd/common/common.h b/bin/varnishd/common/common.h
index 8e32ec9..a35e649 100644
--- a/bin/varnishd/common/common.h
+++ b/bin/varnishd/common/common.h
@@ -74,6 +74,13 @@ void mgt_child_inherit(int fd, const char *what);
 
 
 /* vsm.c */
+struct vsm_sc;
+struct vsm_sc *VSM_common_new(void *ptr, unsigned len);
+void *VSM_common_alloc(struct vsm_sc *sc, unsigned size,
+    const char *class, const char *type, const char *ident);
+void VSM_common_free(struct vsm_sc *sc, void *ptr);
+void VSM_common_delete(struct vsm_sc *sc);
+
 // extern struct VSM_head		*VSM_head;
 // extern const struct VSM_chunk	*vsm_end;
 
diff --git a/bin/varnishd/common/common_vsm.c b/bin/varnishd/common/common_vsm.c
index 5298381..682f542 100644
--- a/bin/varnishd/common/common_vsm.c
+++ b/bin/varnishd/common/common_vsm.c
@@ -27,28 +27,35 @@
  *
  * VSM stuff common to manager and child.
  *
- * We have three potential conflicts we need to lock against here:
+ * We have three potential conflicts we need to deal with:
  *
  * VSM-studying programs (varnishstat...) vs. everybody else
  *	The VSM studying programs only have read-only access to the VSM
  *	so everybody else must use memory barriers, stable storage and
  *	similar tricks to keep the VSM image in sync (long enough) for
  *	the studying programs.
+ *	It can not be prevented, and may indeed in some cases be
+ *	desirable for such programs to write to VSM, for instance to
+ *	zero counters.
+ *	Varnishd should never trust the integrity of VSM content.
  *
  * Manager process vs child process.
- *	Will only muck about in VSM when child process is not running
- *	Responsible for cleaning up any mess left behind by dying child.
+ *	The manager will create a fresh VSM for each child process launch
+ *	and not muck about with VSM while the child runs.  If the child
+ *	crashes, the panicstring will be evacuated and the VSM possibly
+ *	saved for debugging, and a new VSM created before the child is
+ *	started again.
  *
  * Child process threads
  *	Pthread locking necessary.
  *
- * XXX: not all of this is in place yet.
  */
 
 #include "config.h"
 
 #include <stdint.h>
 #include <stdio.h>
+#include <stdlib.h>
 #include <string.h>
 #include <unistd.h>
 
@@ -58,181 +65,267 @@
 #include "vmb.h"
 #include "vtim.h"
 
-/* These two come from beyond (mgt_shmem.c actually) */
-struct VSM_head		*VSM_head;
-const struct VSM_chunk	*vsm_end;
-
-static unsigned
-vsm_mark(void)
-{
-	unsigned seq;
-
-	seq = VSM_head->alloc_seq;
-	VSM_head->alloc_seq = 0;
-	VWMB();
-	return (seq);
-}
-
-static void
-vsm_release(unsigned seq)
-{
+/*--------------------------------------------------------------------*/
 
-	if (seq == 0)
-		return;
-	VWMB();
-	do
-		VSM_head->alloc_seq = ++seq;
-	while (VSM_head->alloc_seq == 0);
-}
+struct vsm_range {
+	unsigned 			magic;
+#define VSM_RANGE_MAGIC			0x8d30f14
+	VTAILQ_ENTRY(vsm_range)		list;
+	unsigned			off; 
+	unsigned			len;
+	double				cool;
+	struct VSM_chunk		*chunk;
+	void				*ptr;
+};
+
+struct vsm_sc {
+	unsigned 			magic;
+#define VSM_SC_MAGIC			0x8b83270d
+	char				*b;
+	unsigned			len;
+	struct VSM_head			*head;
+	VTAILQ_HEAD(,vsm_range)		r_used;
+	VTAILQ_HEAD(,vsm_range)		r_cooling;
+	VTAILQ_HEAD(,vsm_range)		r_free;
+	VTAILQ_HEAD(,vsm_range)		r_bogus;
+};
 
-/*--------------------------------------------------------------------*/
+/*--------------------------------------------------------------------
+ * The free list is sorted by size, which means that collapsing ranges
+ * on free becomes a multi-pass operation.
+ */
 
 static void
-vsm_cleanup(void)
+vsm_common_insert_free(struct vsm_sc *sc, struct vsm_range *vr)
 {
-	unsigned now = (unsigned)VTIM_mono();
-	struct VSM_chunk *sha, *sha2;
-	unsigned seq;
-
-	CHECK_OBJ_NOTNULL(VSM_head, VSM_HEAD_MAGIC);
-	VSM_ITER(sha) {
-		if (strcmp(sha->class, VSM_CLASS_COOL))
-			continue;
-		if (sha->state + VSM_COOL_TIME < now)
-			break;
-	}
-	if (sha == NULL)
-		return;
-	seq = vsm_mark();
-	/* First pass, free, and collapse with next if applicable */
-	VSM_ITER(sha) {
-		if (strcmp(sha->class, VSM_CLASS_COOL))
-			continue;
-		if (sha->state + VSM_COOL_TIME >= now)
-			continue;
-
-		bprintf(sha->class, "%s", VSM_CLASS_FREE);
-		bprintf(sha->type, "%s", "");
-		bprintf(sha->ident, "%s", "");
-		sha2 = VSM_NEXT(sha);
-		assert(sha2 <= vsm_end);
-		if (sha2 == vsm_end)
-			break;
-		CHECK_OBJ_NOTNULL(sha2, VSM_CHUNK_MAGIC);
-		if (!strcmp(sha2->class, VSM_CLASS_FREE)) {
-			sha->len += sha2->len;
-			memset(sha2, 0, sizeof *sha2);
+	struct vsm_range *vr2;
+
+	CHECK_OBJ_NOTNULL(sc, VSM_SC_MAGIC);
+	CHECK_OBJ_NOTNULL(vr, VSM_RANGE_MAGIC);
+
+	/* First try to see if we can collapse anything */
+	VTAILQ_FOREACH(vr2, &sc->r_free, list) {
+		if (vr2->off == vr->off + vr->len) {
+			vr2->off = vr->off;
+			vr2->len += vr->len;
+			FREE_OBJ(vr);
+			VTAILQ_REMOVE(&sc->r_free, vr2, list);
+			vsm_common_insert_free(sc, vr2);
+			return;
+		}
+		if (vr->off == vr2->off + vr2->len) {
+			vr2->len += vr->len;
+			FREE_OBJ(vr);
+			VTAILQ_REMOVE(&sc->r_free, vr2, list);
+			vsm_common_insert_free(sc, vr2);
+			return;
 		}
-		sha->state = 0;
 	}
-	/* Second pass, collaps with prev if applicable */
-	VSM_ITER(sha) {
-		if (strcmp(sha->class, VSM_CLASS_FREE))
-			continue;
-		sha2 = VSM_NEXT(sha);
-		assert(sha2 <= vsm_end);
-		if (sha2 == vsm_end)
-			break;
-		CHECK_OBJ_NOTNULL(sha2, VSM_CHUNK_MAGIC);
-		if (!strcmp(sha2->class, VSM_CLASS_FREE)) {
-			sha->len += sha2->len;
-			memset(sha2, 0, sizeof *sha2);
+	/* Insert in size order */
+	VTAILQ_FOREACH(vr2, &sc->r_free, list) {
+		if (vr2->len > vr->len) {
+			VTAILQ_INSERT_BEFORE(vr2, vr, list);
+			return;
 		}
 	}
-	vsm_release(seq);
+	/* At tail, if everything in the list is smaller */
+	VTAILQ_INSERT_TAIL(&sc->r_free, vr, list);
 }
 
-/*--------------------------------------------------------------------*/
+/*--------------------------------------------------------------------
+ * Initialize a new VSM segment
+ */
 
-void *
-VSM__Alloc(unsigned size, const char *class, const char *type, const char *ident)
+struct vsm_sc *
+VSM_common_new(void *p, unsigned l)
 {
-	struct VSM_chunk *sha, *sha2;
-	unsigned seq;
-
-	CHECK_OBJ_NOTNULL(VSM_head, VSM_HEAD_MAGIC);
-
-	vsm_cleanup();
+	struct vsm_sc *sc;
+	struct vsm_range *vr;
+
+	assert(PAOK(sizeof(struct VSM_chunk)));
+	assert(PAOK(p));
+	ALLOC_OBJ(sc, VSM_SC_MAGIC);
+	AN(sc);
+	VTAILQ_INIT(&sc->r_used);
+	VTAILQ_INIT(&sc->r_cooling);
+	VTAILQ_INIT(&sc->r_free);
+	VTAILQ_INIT(&sc->r_bogus);
+	sc->b = p;
+	sc->len = l;
+
+	sc->head = (void *)sc->b;
+	memset(TRUST_ME(sc->head), 0, sizeof *sc->head);
+	sc->head->magic = VSM_HEAD_MAGIC;
+	sc->head->hdrsize = sizeof *sc->head;
+	sc->head->shm_size = l;
+
+	ALLOC_OBJ(vr, VSM_RANGE_MAGIC);
+	AN(vr);
+	vr->off = PRNDUP(sizeof(*sc->head));
+	vr->len = l - vr->off;
+	VTAILQ_INSERT_TAIL(&sc->r_free, vr, list);
+	return (sc);
+}
 
-	/* Round up to pointersize */
-	size = RUP2(size, sizeof(void*));
+/*--------------------------------------------------------------------
+ * Allocate a chunk from VSM
+ */
 
-	size += sizeof *sha;		/* Make space for the header */
+void *
+VSM_common_alloc(struct vsm_sc *sc, unsigned size,
+    const char *class, const char *type, const char *ident)
+{
+	struct vsm_range *vr, *vr2, *vr3;
+	double now = VTIM_real();
+	unsigned l1, l2;
+
+	CHECK_OBJ_NOTNULL(sc, VSM_SC_MAGIC);
+	AN(size);
+
+	/* XXX: silent truncation instead of assert ? */
+	AN(class);
+	assert(strlen(class) < sizeof(vr->chunk->class));
+	AN(type);
+	assert(strlen(type) < sizeof(vr->chunk->type));
+	AN(ident);
+	assert(strlen(ident) < sizeof(vr->chunk->ident));
+
+	/* Move cooled off stuff to free list */
+	VTAILQ_FOREACH_SAFE(vr, &sc->r_cooling, list, vr2) {
+		if (vr->cool > now)
+			break;
+		VTAILQ_REMOVE(&sc->r_cooling, vr, list);
+		vsm_common_insert_free(sc, vr);
+	}
 
-	VSM_ITER(sha) {
-		CHECK_OBJ_NOTNULL(sha, VSM_CHUNK_MAGIC);
+	size = PRNDUP(size);
+	l1 = size + sizeof(struct VSM_chunk);
+	l2 = size + 2 * sizeof(struct VSM_chunk);
 
-		if (strcmp(sha->class, VSM_CLASS_FREE))
+	/* Find space in free-list */
+	VTAILQ_FOREACH_SAFE(vr, &sc->r_free, list, vr2) {
+		if (vr->len < l1)
 			continue;
+		if (vr->len <= l2) {
+			VTAILQ_REMOVE(&sc->r_free, vr, list);
+		} else {
+			ALLOC_OBJ(vr3, VSM_RANGE_MAGIC);
+			AN(vr3);
+			vr3->off = vr->off;
+			vr3->len = l1;
+			vr->off += l1;
+			vr->len -= l1;
+			VTAILQ_REMOVE(&sc->r_free, vr, list);
+			vsm_common_insert_free(sc, vr);
+			vr = vr3;
+		}
+		break;
+	}
 
-		if (size > sha->len)
-			continue;
+	if (vr == NULL) {
+		/*
+		 * No space in VSM, return malloc'd space
+		 */
+		ALLOC_OBJ(vr, VSM_RANGE_MAGIC);
+		AN(vr);
+		vr->ptr = malloc(size);
+		AN(vr->ptr);
+		VTAILQ_INSERT_TAIL(&sc->r_bogus, vr, list);
+		/* XXX: log + stats */
+		return (vr->ptr);
+	}
 
-		/* Mark as inconsistent while we write string fields */
-		seq = vsm_mark();
+	/* XXX: stats ? */
 
-		if (size + sizeof (*sha) < sha->len) {
-			sha2 = (void*)((uintptr_t)sha + size);
+	/* Zero the entire allocation, to avoid garbage confusing readers */
+	memset(TRUST_ME(sc->b + vr->off), 0, vr->len);
 
-			memset(sha2, 0, sizeof *sha2);
-			sha2->magic = VSM_CHUNK_MAGIC;
-			sha2->len = sha->len - size;
-			bprintf(sha2->class, "%s", VSM_CLASS_FREE);
-			sha->len = size;
-		}
+	vr->chunk = (void *)(sc->b + vr->off);
+	vr->ptr = (vr->chunk + 1);
+
+	vr->chunk->magic = VSM_CHUNK_MAGIC;
+	strcpy(TRUST_ME(vr->chunk->class), class);
+	strcpy(TRUST_ME(vr->chunk->type), type);
+	strcpy(TRUST_ME(vr->chunk->ident), ident);
+	VWMB();
 
-		bprintf(sha->class, "%s", class);
-		bprintf(sha->type, "%s", type);
-		bprintf(sha->ident, "%s", ident);
+	vr3 = VTAILQ_FIRST(&sc->r_used);
+	VTAILQ_INSERT_HEAD(&sc->r_used, vr, list);
 
-		vsm_release(seq);
-		return (VSM_PTR(sha));
+	if (vr3 != NULL) {	
+		AZ(vr3->chunk->next);
+		vr3->chunk->next = vr->off;
+	} else {
+		sc->head->first = vr->off;
 	}
-	return (NULL);
+	VWMB();
+	return (vr->ptr);
 }
 
-/*--------------------------------------------------------------------*/
+/*--------------------------------------------------------------------
+ * Free a chunk
+ */
 
 void
-VSM__Free(const void *ptr)
+VSM_common_free(struct vsm_sc *sc, void *ptr)
 {
-	struct VSM_chunk *sha;
-	unsigned seq;
+	struct vsm_range *vr, *vr2;
 
-	CHECK_OBJ_NOTNULL(VSM_head, VSM_HEAD_MAGIC);
-	VSM_ITER(sha)
-		if (VSM_PTR(sha) == ptr)
-			break;
-	AN(sha);
-	seq = vsm_mark();
-	bprintf(sha->class, "%s", VSM_CLASS_COOL);
-	sha->state = (unsigned)VTIM_mono();
-	vsm_release(seq);
+	CHECK_OBJ_NOTNULL(sc, VSM_SC_MAGIC);
+	AN(ptr);
+
+	/* Look in used list, move to cooling list */
+	VTAILQ_FOREACH(vr, &sc->r_used, list) {
+		if (vr->ptr != ptr)
+			continue;
+		/* XXX: stats ? */
+		vr2 = VTAILQ_NEXT(vr, list);
+		VTAILQ_REMOVE(&sc->r_used, vr, list);
+		VTAILQ_INSERT_TAIL(&sc->r_cooling, vr, list);
+		vr->cool = VTIM_real() + 60;	/* XXX: param ? */
+		if (vr2 != NULL)
+			vr2->chunk->next = vr->chunk->next;
+		else
+			sc->head->first = vr->chunk->next;
+		VWMB();
+		vr->chunk->len = 0;
+		VWMB();
+		return;
+	}
+	/* Look in bogus list, free */
+	VTAILQ_FOREACH(vr, &sc->r_bogus, list) {
+		if (vr->ptr == ptr) {
+			VTAILQ_REMOVE(&sc->r_bogus, vr, list);
+			FREE_OBJ(vr);
+			/* XXX: stats ? */
+			free(TRUST_ME(ptr));
+			return;
+		}
+	}
+	/* Panic */
+	assert(ptr == "Bogus pointer freed");
 }
 
 /*--------------------------------------------------------------------
- * Free all allocations after the mark (ie: allocated by child).
+ * Delete a VSM segment
  */
 
 void
-VSM__Clean(void)
+VSM_common_delete(struct vsm_sc *sc)
 {
-	struct VSM_chunk *sha;
-	unsigned f, seq;
-
-	CHECK_OBJ_NOTNULL(VSM_head, VSM_HEAD_MAGIC);
-	f = 0;
-	seq = vsm_mark();
-	VSM_ITER(sha) {
-		if (f == 0 && !strcmp(sha->class, VSM_CLASS_MARK)) {
-			f = 1;
-			continue;
-		}
-		if (f == 0)
-			continue;
-		if (strcmp(sha->class, VSM_CLASS_FREE) &&
-		    strcmp(sha->class, VSM_CLASS_COOL))
-			VSM__Free(VSM_PTR(sha));
+	struct vsm_range *vr, *vr2;
+
+	CHECK_OBJ_NOTNULL(sc, VSM_SC_MAGIC);
+	VTAILQ_FOREACH_SAFE(vr, &sc->r_free, list, vr2)
+		FREE_OBJ(vr);
+	VTAILQ_FOREACH_SAFE(vr, &sc->r_used, list, vr2)
+		FREE_OBJ(vr);
+	VTAILQ_FOREACH_SAFE(vr, &sc->r_cooling, list, vr2)
+		FREE_OBJ(vr);
+	VTAILQ_FOREACH_SAFE(vr, &sc->r_bogus, list, vr2) {
+		free(TRUST_ME(vr->ptr));
+		FREE_OBJ(vr);
 	}
-	vsm_release(seq);
+	sc->head->magic = 0;
+	FREE_OBJ(sc);
 }
diff --git a/bin/varnishd/common/heritage.h b/bin/varnishd/common/heritage.h
index 36433bb..db8b1f0 100644
--- a/bin/varnishd/common/heritage.h
+++ b/bin/varnishd/common/heritage.h
@@ -29,6 +29,8 @@
  * This file contains the heritage passed when mgt forks cache
  */
 
+struct vsm_sc;
+
 struct listen_sock {
 	unsigned			magic;
 #define LISTEN_SOCK_MAGIC		0x999e4b57
@@ -56,6 +58,8 @@ struct heritage {
 	/* Hash method */
 	const struct hash_slinger	*hash;
 
+	struct vsm_sc			*vsm;
+
 	char				*name;
 	char                            identity[1024];
 };
diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h
index 0650413..571aa23 100644
--- a/bin/varnishd/mgt/mgt.h
+++ b/bin/varnishd/mgt/mgt.h
@@ -110,9 +110,8 @@ extern unsigned mgt_vcc_err_unref;
 		syslog(pri, fmt, __VA_ARGS__);		\
 	} while (0)
 
-#define VSM_Alloc(a, b, c, d)	VSM__Alloc(a,b,c,d)
-#define VSM_Free(a)		VSM__Free(a)
-#define VSM_Clean()		VSM__Clean()
+#define VSM_Alloc(a, b, c, d)	VSM_common_alloc(heritage.vsm, a,b,c,d)
+#define VSM_Free(a)		VSM_common_free(heritage.vsm, a)
 
 #if defined(PTHREAD_CANCELED) || defined(PTHREAD_MUTEX_DEFAULT)
 #error "Keep pthreads out of in manager process"
diff --git a/bin/varnishd/mgt/mgt_cli.c b/bin/varnishd/mgt/mgt_cli.c
index 1d46d82..aa4fffe 100644
--- a/bin/varnishd/mgt/mgt_cli.c
+++ b/bin/varnishd/mgt/mgt_cli.c
@@ -493,15 +493,15 @@ mgt_cli_secret(const char *S_arg)
 {
 	int i, fd;
 	char buf[BUFSIZ];
-	char *p;
+	volatile char *p;
 
 	/* Save in shmem */
 	i = strlen(S_arg);
 	p = VSM_Alloc(i + 1, "Arg", "-S", "");
 	AN(p);
-	strcpy(p, S_arg);
+	memcpy(TRUST_ME(p), S_arg, i + 1);
 
-	srandomdev();
+	srandomdev();			/* XXX: why here ??? */
 	fd = open(S_arg, O_RDONLY);
 	if (fd < 0) {
 		fprintf(stderr, "Can not open secret-file \"%s\"\n", S_arg);
@@ -527,7 +527,7 @@ mgt_cli_telnet(const char *T_arg)
 	struct vss_addr **ta;
 	int i, n, sock, good;
 	struct telnet *tn;
-	char *p;
+	volatile char *p;
 	struct vsb *vsb;
 	char abuf[VTCP_ADDRBUFSIZE];
 	char pbuf[VTCP_PORTBUFSIZE];
@@ -566,7 +566,7 @@ mgt_cli_telnet(const char *T_arg)
 	/* Save in shmem */
 	p = VSM_Alloc(VSB_len(vsb) + 1, "Arg", "-T", "");
 	AN(p);
-	strcpy(p, VSB_data(vsb));
+	memcpy(TRUST_ME(p), VSB_data(vsb), VSB_len(vsb) + 1);
 	VSB_delete(vsb);
 }
 
diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c
index ff8992e..4e63c16 100644
--- a/bin/varnishd/mgt/mgt_main.c
+++ b/bin/varnishd/mgt/mgt_main.c
@@ -654,8 +654,6 @@ main(int argc, char * const *argv)
 	if (T_arg != NULL)
 		mgt_cli_telnet(T_arg);
 
-	AN(VSM_Alloc(0, VSM_CLASS_MARK, "", ""));
-
 	MGT_Run();
 
 	if (pfh != NULL)
diff --git a/bin/varnishd/mgt/mgt_shmem.c b/bin/varnishd/mgt/mgt_shmem.c
index d1bf18e..6095d9f 100644
--- a/bin/varnishd/mgt/mgt_shmem.c
+++ b/bin/varnishd/mgt/mgt_shmem.c
@@ -169,9 +169,8 @@ vsl_n_check(int fd)
  */
 
 static void
-vsl_buildnew(const char *fn, unsigned size, int fill)
+vsl_buildnew(const char *fn, ssize_t size)
 {
-	struct VSM_head slh;
 	int i;
 	unsigned u;
 	char buf[64*1024];
@@ -189,26 +188,16 @@ vsl_buildnew(const char *fn, unsigned size, int fill)
 	flags &= ~O_NONBLOCK;
 	AZ(fcntl(vsl_fd, F_SETFL, flags));
 
-	memset(&slh, 0, sizeof slh);
-	slh.magic = VSM_HEAD_MAGIC;
-	slh.hdrsize = sizeof slh;
-	slh.shm_size = size;
-	i = write(vsl_fd, &slh, sizeof slh);
-	xxxassert(i == sizeof slh);
-
-	if (fill) {
-		memset(buf, 0, sizeof buf);
-		for (u = sizeof slh; u < size; ) {
-			i = write(vsl_fd, buf, sizeof buf);
-			if (i <= 0) {
-				fprintf(stderr, "Write error %s: %s\n",
-				    fn, strerror(errno));
-				exit (1);
-			}
-			u += i;
+	memset(buf, 0, sizeof buf);
+	for (u = 0; u < size; ) {
+		i = write(vsl_fd, buf, sizeof buf);
+		if (i <= 0) {
+			fprintf(stderr, "Write error %s: %s\n",
+			    fn, strerror(errno));
+			exit (1);
 		}
+		u += i;
 	}
-
 	AZ(ftruncate(vsl_fd, (off_t)size));
 }
 
@@ -220,8 +209,10 @@ static
 void
 mgt_shm_atexit(void)
 {
+#if 0
 	if (getpid() == VSM_head->master_pid)
 		VSM_head->master_pid = 0;
+#endif
 }
 
 void
@@ -229,7 +220,10 @@ mgt_SHM_Init(void)
 {
 	int i, fill;
 	uintmax_t size, ps;
+	void *p;
+#if 0
 	uint32_t *vsl_log_start;
+#endif
 
 	fill = 1;
 
@@ -243,55 +237,49 @@ mgt_SHM_Init(void)
 		vsl_n_check(i);
 		(void)close(i);
 	}
-	vsl_buildnew(VSM_FILENAME, size, fill);
+	vsl_buildnew(VSM_FILENAME, size);
 
-	VSM_head = (void *)mmap(NULL, size,
+	p = (void *)mmap(NULL, size,
 	    PROT_READ|PROT_WRITE,
 	    MAP_HASSEMAPHORE | MAP_NOSYNC | MAP_SHARED,
 	    vsl_fd, 0);
-	VSM_head->master_pid = getpid();
-	AZ(atexit(mgt_shm_atexit));
-	xxxassert(VSM_head != MAP_FAILED);
-	(void)mlock((void*)VSM_head, size);
-
-	memset(&VSM_head->head, 0, sizeof VSM_head->head);
-	VSM_head->head.magic = VSM_CHUNK_MAGIC;
-	VSM_head->head.len =
-	    (uint8_t*)(VSM_head) + size - (uint8_t*)&VSM_head->head;
-	bprintf(VSM_head->head.class, "%s", VSM_CLASS_FREE);
-	VWMB();
+	xxxassert(p != MAP_FAILED);
 
-	vsm_end = (void*)((uint8_t*)VSM_head + size);
+	heritage.vsm = VSM_common_new(p, size);
 
-	VSC_C_main = VSM_Alloc(sizeof *VSC_C_main,
-	    VSC_CLASS, VSC_TYPE_MAIN, "");
-	AN(VSC_C_main);
+	(void)mlock(p, size);
+	AZ(atexit(mgt_shm_atexit));
 
 	/* XXX: We need to zero params if we dealloc/clean/wash */
 	cache_param = VSM_Alloc(sizeof *cache_param, VSM_CLASS_PARAM, "", "");
 	AN(cache_param);
 	*cache_param = mgt_param;
 
-	vsl_log_start = VSM_Alloc(mgt_param.vsl_space, VSL_CLASS, "", "");
-	AN(vsl_log_start);
-	vsl_log_start[1] = VSL_ENDMARKER;
-	VWMB();
-
 	PAN_panicstr_len = 64 * 1024;
 	PAN_panicstr = VSM_Alloc(PAN_panicstr_len, PAN_CLASS, "", "");
 	AN(PAN_panicstr);
-	/* XXX: shouldn't VSM_Alloc zero ? */
-	memset(PAN_panicstr, '\0', PAN_panicstr_len);
 
-	do
-		*vsl_log_start = random() & 0xffff;
-	while (*vsl_log_start == 0);
+#if 0
+
+	VSM_head->master_pid = getpid();
 
+	memset(&VSM_head->head, 0, sizeof VSM_head->head);
+	VSM_head->head.magic = VSM_CHUNK_MAGIC;
+	VSM_head->head.len =
+	    (uint8_t*)(VSM_head) + size - (uint8_t*)&VSM_head->head;
+	bprintf(VSM_head->head.class, "%s", VSM_CLASS_FREE);
 	VWMB();
 
+	vsm_end = (void*)((uint8_t*)VSM_head + size);
+
+	VSC_C_main = VSM_Alloc(sizeof *VSC_C_main,
+	    VSC_CLASS, VSC_TYPE_MAIN, "");
+	AN(VSC_C_main);
+
 	do
 		VSM_head->alloc_seq = random();
 	while (VSM_head->alloc_seq == 0);
+#endif
 
 }
 
@@ -299,5 +287,7 @@ void
 mgt_SHM_Pid(void)
 {
 
+#if 0
 	VSM_head->master_pid = getpid();
+#endif
 }
diff --git a/include/vapi/vsm_int.h b/include/vapi/vsm_int.h
index 521f89d..cad1e36 100644
--- a/include/vapi/vsm_int.h
+++ b/include/vapi/vsm_int.h
@@ -41,10 +41,11 @@
  */
 
 struct VSM_chunk {
-#define VSM_CHUNK_MAGIC		0x43907b6e	/* From /dev/random */
+#define VSM_CHUNK_MAGIC		0xa15712e5	/* From /dev/random */
 	unsigned		magic;
-	unsigned		len;
-	unsigned		state;
+	unsigned		len;		/* Incl VSM_chunk */
+	unsigned		next;		/* Offset in shmem */
+	unsigned		state;		/* XXX remove */
 	char			class[8];
 	char			type[8];
 	char			ident[64];
@@ -54,10 +55,11 @@ struct VSM_chunk {
 #define VSM_PTR(sha)		((void*)((uintptr_t)((sha) + 1)))
 
 struct VSM_head {
-#define VSM_HEAD_MAGIC		4185512502U	/* From /dev/random */
+#define VSM_HEAD_MAGIC		0xe75f7e91	/* From /dev/random */
 	unsigned		magic;
 
 	unsigned		hdrsize;
+	unsigned		first;		/* Offset, first chunk */
 
 	uint64_t		starttime;
 	int64_t			master_pid;



From phk at varnish-cache.org  Sun Nov 20 23:39:22 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 21 Nov 2011 00:39:22 +0100
Subject: [master] 401ddc4 Make the VSM layout more hexdump friendly
Message-ID: <E1RSGyo-00087L-QR@project.varnish-software.com>

commit 401ddc4e9462b2c8476e945c6e148982ea7325e7
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Nov 20 13:29:14 2011 +0000

    Make the VSM layout more hexdump friendly

diff --git a/bin/varnishd/common/common_vsm.c b/bin/varnishd/common/common_vsm.c
index f525a1e..781b01c 100644
--- a/bin/varnishd/common/common_vsm.c
+++ b/bin/varnishd/common/common_vsm.c
@@ -137,7 +137,7 @@ VSM_common_new(void *p, ssize_t l)
 	sc->head = (void *)sc->b;
 	/* This should not be necessary, but just in case...*/
 	memset(sc->head, 0, sizeof *sc->head);
-	sc->head->magic = VSM_HEAD_MAGIC;
+	memcpy(sc->head->marker, VSM_HEAD_MARKER, sizeof sc->head->marker);
 	sc->head->hdrsize = sizeof *sc->head;
 	sc->head->shm_size = l;
 	sc->head->alloc_seq = random() | 1;
@@ -145,8 +145,8 @@ VSM_common_new(void *p, ssize_t l)
 
 	ALLOC_OBJ(vr, VSM_RANGE_MAGIC);
 	AN(vr);
-	vr->off = PRNDUP(sizeof(*sc->head));
-	vr->len = l - vr->off;
+	vr->off = RUP2(sizeof(*sc->head), 16);
+	vr->len = RDN2(l - vr->off, 16);
 	VTAILQ_INSERT_TAIL(&sc->r_free, vr, list);
 	return (sc);
 }
@@ -182,9 +182,8 @@ VSM_common_alloc(struct vsm_sc *sc, ssize_t size,
 		vsm_common_insert_free(sc, vr);
 	}
 
-	size = PRNDUP(size);
-	l1 = size + sizeof(struct VSM_chunk);
-	l2 = size + 2 * sizeof(struct VSM_chunk);
+	l1 = RUP2(size + sizeof(struct VSM_chunk), 16);
+	l2 = RUP2(size + 2 * sizeof(struct VSM_chunk), 16);
 
 	/* Find space in free-list */
 	VTAILQ_FOREACH_SAFE(vr, &sc->r_free, list, vr2) {
@@ -227,7 +226,8 @@ VSM_common_alloc(struct vsm_sc *sc, ssize_t size,
 	vr->chunk = (void *)(sc->b + vr->off);
 	vr->ptr = (vr->chunk + 1);
 
-	vr->chunk->magic = VSM_CHUNK_MAGIC;
+	memcpy(vr->chunk->marker, VSM_CHUNK_MARKER, sizeof vr->chunk->marker);
+	vr->chunk->len = vr->len;
 	strcpy(vr->chunk->class, class);
 	strcpy(vr->chunk->type, type);
 	strcpy(vr->chunk->ident, ident);
diff --git a/include/vapi/vsm_int.h b/include/vapi/vsm_int.h
index c33ba35..c0a2df9 100644
--- a/include/vapi/vsm_int.h
+++ b/include/vapi/vsm_int.h
@@ -98,8 +98,8 @@
 #define VSM_FILENAME		"_.vsm"
 
 struct VSM_chunk {
-#define VSM_CHUNK_MAGIC		0xa15712e5	/* From /dev/random */
-	unsigned		magic;
+#define VSM_CHUNK_MARKER	"VSMCHUNK"
+	char			marker[8];
 	ssize_t			len;		/* Incl VSM_chunk */
 	ssize_t			next;		/* Offset in shmem */
 	char			class[8];
@@ -108,8 +108,8 @@ struct VSM_chunk {
 };
 
 struct VSM_head {
-#define VSM_HEAD_MAGIC		0xe75f7e91	/* From /dev/random */
-	unsigned		magic;
+#define VSM_HEAD_MARKER		"VSMHEAD0"	/* Incr. as version# *
+	char			marker[8];
 	ssize_t			hdrsize;
 	ssize_t			shm_size;
 	ssize_t			first;		/* Offset, first chunk */



From phk at varnish-cache.org  Sun Nov 20 23:39:22 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 21 Nov 2011 00:39:22 +0100
Subject: [master] e4d1d36 Polish round over varnishd part of VSM
Message-ID: <E1RSGyo-00087B-HY@project.varnish-software.com>

commit e4d1d360beca6917fe2ee1f866ca1ed71302b639
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Nov 20 12:44:19 2011 +0000

    Polish round over varnishd part of VSM

diff --git a/bin/varnishd/cache/cache_shmlog.c b/bin/varnishd/cache/cache_shmlog.c
index 63157c7..950f57f 100644
--- a/bin/varnishd/cache/cache_shmlog.c
+++ b/bin/varnishd/cache/cache_shmlog.c
@@ -37,9 +37,7 @@
 
 #include "cache_backend.h"	// For w->vbc
 
-#include "vapi/vsm_int.h"
 #include "vmb.h"
-#include "vtim.h"
 
 /* These cannot be struct lock, which depends on vsm/vsl working */
 static pthread_mutex_t vsl_mtx;
diff --git a/bin/varnishd/common/common.h b/bin/varnishd/common/common.h
index a35e649..a5d7633 100644
--- a/bin/varnishd/common/common.h
+++ b/bin/varnishd/common/common.h
@@ -75,30 +75,11 @@ void mgt_child_inherit(int fd, const char *what);
 
 /* vsm.c */
 struct vsm_sc;
-struct vsm_sc *VSM_common_new(void *ptr, unsigned len);
-void *VSM_common_alloc(struct vsm_sc *sc, unsigned size,
+struct vsm_sc *VSM_common_new(void *ptr, ssize_t len);
+void *VSM_common_alloc(struct vsm_sc *sc, ssize_t size,
     const char *class, const char *type, const char *ident);
 void VSM_common_free(struct vsm_sc *sc, void *ptr);
-void VSM_common_delete(struct vsm_sc *sc);
-
-// extern struct VSM_head		*VSM_head;
-// extern const struct VSM_chunk	*vsm_end;
-
-/*
- * These three should not be called directly, but only through
- * proper vectors in mgt.h/cache.h, hence the __
- */
-void *VSM__Alloc(unsigned size, const char *class, const char *type,
-    const char *ident);
-void VSM__Free(const void *ptr);
-void VSM__Clean(void);
-
-/* These classes are opaque to other programs, so we define the here */
-#define VSM_CLASS_FREE	"Free"
-#define VSM_CLASS_COOL	"Cool"
-#define VSM_CLASS_PARAM	"Params"
-#define VSM_CLASS_MARK	"MgrCld"
-#define VSM_COOL_TIME	5
+void VSM_common_delete(struct vsm_sc **sc);
 
 /*---------------------------------------------------------------------
  * Generic power-2 rounding macros
diff --git a/bin/varnishd/common/common_vsm.c b/bin/varnishd/common/common_vsm.c
index 682f542..f525a1e 100644
--- a/bin/varnishd/common/common_vsm.c
+++ b/bin/varnishd/common/common_vsm.c
@@ -27,27 +27,8 @@
  *
  * VSM stuff common to manager and child.
  *
- * We have three potential conflicts we need to deal with:
- *
- * VSM-studying programs (varnishstat...) vs. everybody else
- *	The VSM studying programs only have read-only access to the VSM
- *	so everybody else must use memory barriers, stable storage and
- *	similar tricks to keep the VSM image in sync (long enough) for
- *	the studying programs.
- *	It can not be prevented, and may indeed in some cases be
- *	desirable for such programs to write to VSM, for instance to
- *	zero counters.
- *	Varnishd should never trust the integrity of VSM content.
- *
- * Manager process vs child process.
- *	The manager will create a fresh VSM for each child process launch
- *	and not muck about with VSM while the child runs.  If the child
- *	crashes, the panicstring will be evacuated and the VSM possibly
- *	saved for debugging, and a new VSM created before the child is
- *	started again.
- *
- * Child process threads
- *	Pthread locking necessary.
+ * Please see comments in <vapi/vsm_int.h> for details of protocols and
+ * data consistency.
  *
  */
 
@@ -71,8 +52,8 @@ struct vsm_range {
 	unsigned 			magic;
 #define VSM_RANGE_MAGIC			0x8d30f14
 	VTAILQ_ENTRY(vsm_range)		list;
-	unsigned			off; 
-	unsigned			len;
+	ssize_t				off; 
+	ssize_t				len;
 	double				cool;
 	struct VSM_chunk		*chunk;
 	void				*ptr;
@@ -82,7 +63,7 @@ struct vsm_sc {
 	unsigned 			magic;
 #define VSM_SC_MAGIC			0x8b83270d
 	char				*b;
-	unsigned			len;
+	ssize_t				len;
 	struct VSM_head			*head;
 	VTAILQ_HEAD(,vsm_range)		r_used;
 	VTAILQ_HEAD(,vsm_range)		r_cooling;
@@ -137,7 +118,7 @@ vsm_common_insert_free(struct vsm_sc *sc, struct vsm_range *vr)
  */
 
 struct vsm_sc *
-VSM_common_new(void *p, unsigned l)
+VSM_common_new(void *p, ssize_t l)
 {
 	struct vsm_sc *sc;
 	struct vsm_range *vr;
@@ -154,10 +135,13 @@ VSM_common_new(void *p, unsigned l)
 	sc->len = l;
 
 	sc->head = (void *)sc->b;
-	memset(TRUST_ME(sc->head), 0, sizeof *sc->head);
+	/* This should not be necessary, but just in case...*/
+	memset(sc->head, 0, sizeof *sc->head);
 	sc->head->magic = VSM_HEAD_MAGIC;
 	sc->head->hdrsize = sizeof *sc->head;
 	sc->head->shm_size = l;
+	sc->head->alloc_seq = random() | 1;
+	VWMB();
 
 	ALLOC_OBJ(vr, VSM_RANGE_MAGIC);
 	AN(vr);
@@ -172,7 +156,7 @@ VSM_common_new(void *p, unsigned l)
  */
 
 void *
-VSM_common_alloc(struct vsm_sc *sc, unsigned size,
+VSM_common_alloc(struct vsm_sc *sc, ssize_t size,
     const char *class, const char *type, const char *ident)
 {
 	struct vsm_range *vr, *vr2, *vr3;
@@ -238,15 +222,15 @@ VSM_common_alloc(struct vsm_sc *sc, unsigned size,
 	/* XXX: stats ? */
 
 	/* Zero the entire allocation, to avoid garbage confusing readers */
-	memset(TRUST_ME(sc->b + vr->off), 0, vr->len);
+	memset(sc->b + vr->off, 0, vr->len);
 
 	vr->chunk = (void *)(sc->b + vr->off);
 	vr->ptr = (vr->chunk + 1);
 
 	vr->chunk->magic = VSM_CHUNK_MAGIC;
-	strcpy(TRUST_ME(vr->chunk->class), class);
-	strcpy(TRUST_ME(vr->chunk->type), type);
-	strcpy(TRUST_ME(vr->chunk->ident), ident);
+	strcpy(vr->chunk->class, class);
+	strcpy(vr->chunk->type, type);
+	strcpy(vr->chunk->ident, ident);
 	VWMB();
 
 	vr3 = VTAILQ_FIRST(&sc->r_used);
@@ -258,6 +242,7 @@ VSM_common_alloc(struct vsm_sc *sc, unsigned size,
 	} else {
 		sc->head->first = vr->off;
 	}
+	sc->head->alloc_seq += 2;
 	VWMB();
 	return (vr->ptr);
 }
@@ -289,6 +274,7 @@ VSM_common_free(struct vsm_sc *sc, void *ptr)
 			sc->head->first = vr->chunk->next;
 		VWMB();
 		vr->chunk->len = 0;
+		sc->head->alloc_seq += 2;
 		VWMB();
 		return;
 	}
@@ -298,7 +284,7 @@ VSM_common_free(struct vsm_sc *sc, void *ptr)
 			VTAILQ_REMOVE(&sc->r_bogus, vr, list);
 			FREE_OBJ(vr);
 			/* XXX: stats ? */
-			free(TRUST_ME(ptr));
+			free(ptr);
 			return;
 		}
 	}
@@ -311,9 +297,14 @@ VSM_common_free(struct vsm_sc *sc, void *ptr)
  */
 
 void
-VSM_common_delete(struct vsm_sc *sc)
+VSM_common_delete(struct vsm_sc **scp)
 {
 	struct vsm_range *vr, *vr2;
+	struct vsm_sc *sc;
+
+	AN(scp);
+	sc =*scp;
+	*scp = NULL;
 
 	CHECK_OBJ_NOTNULL(sc, VSM_SC_MAGIC);
 	VTAILQ_FOREACH_SAFE(vr, &sc->r_free, list, vr2)
@@ -323,9 +314,10 @@ VSM_common_delete(struct vsm_sc *sc)
 	VTAILQ_FOREACH_SAFE(vr, &sc->r_cooling, list, vr2)
 		FREE_OBJ(vr);
 	VTAILQ_FOREACH_SAFE(vr, &sc->r_bogus, list, vr2) {
-		free(TRUST_ME(vr->ptr));
+		free(vr->ptr);
 		FREE_OBJ(vr);
 	}
-	sc->head->magic = 0;
+	sc->head->alloc_seq = 0;
+	VWMB();
 	FREE_OBJ(sc);
 }
diff --git a/bin/varnishd/common/params.h b/bin/varnishd/common/params.h
index 95258fb..cab2f49 100644
--- a/bin/varnishd/common/params.h
+++ b/bin/varnishd/common/params.h
@@ -31,6 +31,8 @@
 
 #include "vre.h"
 
+#define VSM_CLASS_PARAM		"Params"
+
 struct params {
 
 	/* Unprivileged user / group */
diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h
index 571aa23..775fe2c 100644
--- a/bin/varnishd/mgt/mgt.h
+++ b/bin/varnishd/mgt/mgt.h
@@ -82,7 +82,6 @@ void mgt_sandbox_solaris_privsep(void);
 
 /* mgt_shmem.c */
 void mgt_SHM_Init(void);
-void mgt_SHM_Pid(void);
 
 /* stevedore_mgt.c */
 void STV_Config(const char *spec);
diff --git a/bin/varnishd/mgt/mgt_cli.c b/bin/varnishd/mgt/mgt_cli.c
index aa4fffe..d3b7396 100644
--- a/bin/varnishd/mgt/mgt_cli.c
+++ b/bin/varnishd/mgt/mgt_cli.c
@@ -493,13 +493,13 @@ mgt_cli_secret(const char *S_arg)
 {
 	int i, fd;
 	char buf[BUFSIZ];
-	volatile char *p;
+	char *p;
 
 	/* Save in shmem */
 	i = strlen(S_arg);
-	p = VSM_Alloc(i + 1, "Arg", "-S", "");
+	p = VSM_Alloc(i + 1L, "Arg", "-S", "");
 	AN(p);
-	memcpy(TRUST_ME(p), S_arg, i + 1);
+	memcpy(p, S_arg, i + 1L);
 
 	srandomdev();			/* XXX: why here ??? */
 	fd = open(S_arg, O_RDONLY);
@@ -527,7 +527,7 @@ mgt_cli_telnet(const char *T_arg)
 	struct vss_addr **ta;
 	int i, n, sock, good;
 	struct telnet *tn;
-	volatile char *p;
+	char *p;
 	struct vsb *vsb;
 	char abuf[VTCP_ADDRBUFSIZE];
 	char pbuf[VTCP_PORTBUFSIZE];
@@ -566,7 +566,7 @@ mgt_cli_telnet(const char *T_arg)
 	/* Save in shmem */
 	p = VSM_Alloc(VSB_len(vsb) + 1, "Arg", "-T", "");
 	AN(p);
-	memcpy(TRUST_ME(p), VSB_data(vsb), VSB_len(vsb) + 1);
+	memcpy(p, VSB_data(vsb), VSB_len(vsb) + 1);
 	VSB_delete(vsb);
 }
 
diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c
index 4e63c16..f782d03 100644
--- a/bin/varnishd/mgt/mgt_main.c
+++ b/bin/varnishd/mgt/mgt_main.c
@@ -630,8 +630,6 @@ main(int argc, char * const *argv)
 	if (!d_flag && !F_flag)
 		AZ(varnish_daemon(1, 0));
 
-	mgt_SHM_Pid();
-
 	if (pfh != NULL && VPF_Write(pfh))
 		fprintf(stderr, "NOTE: Could not write PID file\n");
 
diff --git a/bin/varnishd/mgt/mgt_pool.c b/bin/varnishd/mgt/mgt_pool.c
index e8c4f27..badb3f4 100644
--- a/bin/varnishd/mgt/mgt_pool.c
+++ b/bin/varnishd/mgt/mgt_pool.c
@@ -48,7 +48,6 @@
 #include <unistd.h>
 
 #include "mgt/mgt.h"
-#include "common/heritage.h"
 #include "common/params.h"
 
 #include "mgt/mgt_param.h"
diff --git a/bin/varnishd/mgt/mgt_sandbox.c b/bin/varnishd/mgt/mgt_sandbox.c
index cdba825..3cd1d98 100644
--- a/bin/varnishd/mgt/mgt_sandbox.c
+++ b/bin/varnishd/mgt/mgt_sandbox.c
@@ -53,7 +53,6 @@
 #include <unistd.h>
 
 #include "mgt/mgt.h"
-#include "common/heritage.h"
 #include "common/params.h"
 
 /*--------------------------------------------------------------------*/
diff --git a/bin/varnishd/mgt/mgt_shmem.c b/bin/varnishd/mgt/mgt_shmem.c
index 6095d9f..6bb8e13 100644
--- a/bin/varnishd/mgt/mgt_shmem.c
+++ b/bin/varnishd/mgt/mgt_shmem.c
@@ -90,7 +90,6 @@
 #include <sys/stat.h>
 
 #include <fcntl.h>
-#include <signal.h>
 #include <stdint.h>
 #include <stdio.h>
 #include <stdlib.h>
@@ -103,7 +102,6 @@
 
 #include "flopen.h"
 #include "vapi/vsc_int.h"
-#include "vapi/vsl_int.h"
 #include "vapi/vsm_int.h"
 #include "vmb.h"
 
@@ -117,88 +115,96 @@
 
 struct VSC_C_main	*VSC_C_main;
 
-static int vsl_fd = -1;
+static int vsm_fd = -1;
 
 /*--------------------------------------------------------------------
  * Check that we are not started with the same -n argument as an already
- * running varnishd
+ * running varnishd.
+ *
+ * Non-zero return means we should exit and not trample the file.
+ *
  */
 
-static void
-vsl_n_check(int fd)
+static int
+vsm_n_check(void)
 {
-	struct VSM_head slh;
-	int i;
+	int fd, i;
 	struct stat st;
 	pid_t pid;
+	struct VSM_head vsmh;
+	int retval = 2;
 
-	AZ(fstat(fd, &st));
-	if (!S_ISREG(st.st_mode))
-		ARGV_ERR("\tshmlog: Not a file\n");
-
-	/* Test if the SHMFILE is locked by other Varnish */
-	if (fltest(fd, &pid) > 0) {
-		fprintf(stderr,
-			"SHMFILE locked by running varnishd master (pid=%jd)\n",
-			(intmax_t)pid);
-		fprintf(stderr,
-			"(Use unique -n arguments if you want multiple "
-			"instances)\n");
-		exit(2);
-	}
+	fd = open(VSM_FILENAME, O_RDWR, 0644);
+	if (fd < 0)
+		return (0);
 
-	/* Read owning pid from SHMFILE */
-	memset(&slh, 0, sizeof slh);	/* XXX: for flexelint */
-	i = read(fd, &slh, sizeof slh);
-	if (i != sizeof slh)
-		return;
-	if (slh.magic != VSM_HEAD_MAGIC)
-		return;
-	if (slh.hdrsize != sizeof slh)
-		return;
-	if (slh.master_pid != 0 && !kill(slh.master_pid, 0)) {
+	AZ(fstat(fd, &st));
+	if (!S_ISREG(st.st_mode)) {
 		fprintf(stderr,
-			"WARNING: Taking over SHMFILE marked as owned by "
-			"running process (pid=%jd)\n",
-			(intmax_t)slh.master_pid);
+		    "VSM (%s) not a regular file.\n", VSM_FILENAME);
+	} else {
+		i = fltest(fd, &pid);
+		if (i < 0) {
+			fprintf(stderr,
+			    "Cannot determine locking status of VSM (%s)\n.",
+			    VSM_FILENAME);
+		} else if (i == 0) {
+			/*
+			 * File is unlocked, mark it as dead, to help any
+			 * consumers still stuck on it.
+			 */
+			if (pread(fd, &vsmh, sizeof vsmh, 0) == sizeof vsmh) {
+				vsmh.alloc_seq = 0;
+				(void)pwrite(fd, &vsmh, sizeof vsmh, 0);
+			}
+			retval = 0;
+		} else {
+			/* The VSM is locked, we won't touch it. */
+			fprintf(stderr,
+			    "VSM locked by running varnishd master (pid=%jd)\n"
+			    "(Use unique -n arguments if you want"
+			    "  multiple instances)\n", (intmax_t)pid);
+		}
 	}
+	(void)close(fd);
+	return (retval);
 }
 
 /*--------------------------------------------------------------------
- * Build a new shmlog file
+ * Build a zeroed file
  */
 
-static void
-vsl_buildnew(const char *fn, ssize_t size)
+static int
+vsm_zerofile(const char *fn, ssize_t size)
 {
-	int i;
-	unsigned u;
+	int fd;
+	ssize_t i, u;
 	char buf[64*1024];
 	int flags;
 
-	(void)unlink(fn);
-	vsl_fd = flopen(fn, O_RDWR | O_CREAT | O_EXCL | O_NONBLOCK, 0644);
-	if (vsl_fd < 0) {
+	fd = flopen(fn, O_RDWR | O_CREAT | O_EXCL | O_NONBLOCK, 0644);
+	if (fd < 0) {
 		fprintf(stderr, "Could not create %s: %s\n",
 		    fn, strerror(errno));
-		exit (1);
+		return (-1);
 	}
-	flags = fcntl(vsl_fd, F_GETFL);
+	flags = fcntl(fd, F_GETFL);
 	assert(flags != -1);
 	flags &= ~O_NONBLOCK;
-	AZ(fcntl(vsl_fd, F_SETFL, flags));
+	AZ(fcntl(fd, F_SETFL, flags));
 
 	memset(buf, 0, sizeof buf);
 	for (u = 0; u < size; ) {
-		i = write(vsl_fd, buf, sizeof buf);
+		i = write(fd, buf, sizeof buf);
 		if (i <= 0) {
 			fprintf(stderr, "Write error %s: %s\n",
 			    fn, strerror(errno));
-			exit (1);
+			return (-1);
 		}
 		u += i;
 	}
-	AZ(ftruncate(vsl_fd, (off_t)size));
+	AZ(ftruncate(fd, (off_t)size));
+	return (fd);
 }
 
 /*--------------------------------------------------------------------
@@ -209,48 +215,59 @@ static
 void
 mgt_shm_atexit(void)
 {
-#if 0
-	if (getpid() == VSM_head->master_pid)
-		VSM_head->master_pid = 0;
-#endif
+
+	if (heritage.vsm != NULL)
+		VSM_common_delete(&heritage.vsm);
 }
 
 void
 mgt_SHM_Init(void)
 {
-	int i, fill;
+	int i;
 	uintmax_t size, ps;
 	void *p;
-#if 0
-	uint32_t *vsl_log_start;
-#endif
-
-	fill = 1;
+	char fnbuf[64];
 
 	size = mgt_param.vsl_space + mgt_param.vsm_space;
 	ps = getpagesize();
 	size += ps - 1;
 	size &= ~(ps - 1);
 
-	i = open(VSM_FILENAME, O_RDWR, 0644);
-	if (i >= 0) {
-		vsl_n_check(i);
-		(void)close(i);
-	}
-	vsl_buildnew(VSM_FILENAME, size);
+	/* Collision check with already running varnishd */
+	i = vsm_n_check();
+	if (i)
+		exit(i);
+
+	bprintf(fnbuf, "%s.%jd", VSM_FILENAME, (intmax_t)getpid());
+
+	vsm_fd = vsm_zerofile(fnbuf, size);
+	if (vsm_fd < 0) 
+		exit(1);
 
 	p = (void *)mmap(NULL, size,
 	    PROT_READ|PROT_WRITE,
 	    MAP_HASSEMAPHORE | MAP_NOSYNC | MAP_SHARED,
-	    vsl_fd, 0);
-	xxxassert(p != MAP_FAILED);
+	    vsm_fd, 0);
 
-	heritage.vsm = VSM_common_new(p, size);
+	if (p == MAP_FAILED) {
+		fprintf(stderr, "Mmap error %s: %s\n", fnbuf, strerror(errno));
+		exit (-1);
+	}
 
+	/* This may or may not work */
 	(void)mlock(p, size);
+
+	heritage.vsm = VSM_common_new(p, size);
+
+	if (rename(fnbuf, VSM_FILENAME)) {
+		fprintf(stderr, "Rename failed %s -> %s: %s\n",
+		    fnbuf, VSM_FILENAME, strerror(errno));
+		(void)unlink(fnbuf);
+		exit (-1);
+	}
+
 	AZ(atexit(mgt_shm_atexit));
 
-	/* XXX: We need to zero params if we dealloc/clean/wash */
 	cache_param = VSM_Alloc(sizeof *cache_param, VSM_CLASS_PARAM, "", "");
 	AN(cache_param);
 	*cache_param = mgt_param;
@@ -258,36 +275,4 @@ mgt_SHM_Init(void)
 	PAN_panicstr_len = 64 * 1024;
 	PAN_panicstr = VSM_Alloc(PAN_panicstr_len, PAN_CLASS, "", "");
 	AN(PAN_panicstr);
-
-#if 0
-
-	VSM_head->master_pid = getpid();
-
-	memset(&VSM_head->head, 0, sizeof VSM_head->head);
-	VSM_head->head.magic = VSM_CHUNK_MAGIC;
-	VSM_head->head.len =
-	    (uint8_t*)(VSM_head) + size - (uint8_t*)&VSM_head->head;
-	bprintf(VSM_head->head.class, "%s", VSM_CLASS_FREE);
-	VWMB();
-
-	vsm_end = (void*)((uint8_t*)VSM_head + size);
-
-	VSC_C_main = VSM_Alloc(sizeof *VSC_C_main,
-	    VSC_CLASS, VSC_TYPE_MAIN, "");
-	AN(VSC_C_main);
-
-	do
-		VSM_head->alloc_seq = random();
-	while (VSM_head->alloc_seq == 0);
-#endif
-
-}
-
-void
-mgt_SHM_Pid(void)
-{
-
-#if 0
-	VSM_head->master_pid = getpid();
-#endif
 }
diff --git a/include/vapi/vsm.h b/include/vapi/vsm.h
index fd7fbee..275f6d9 100644
--- a/include/vapi/vsm.h
+++ b/include/vapi/vsm.h
@@ -32,8 +32,16 @@
 #define VAPI_VSM_H_INCLUDED
 
 struct VSM_head;
+struct VSM_chunk;
 struct VSM_data;
 
+struct VSM_fantom {
+	struct VSM_chunk	*chunk;
+	void			*b;
+	void			*e;
+	uintptr_t		priv;
+};
+
 /*---------------------------------------------------------------------
  * VSM level access functions
  */
@@ -46,6 +54,7 @@ struct VSM_data *VSM_New(void);
 	 * referencing the same or different shared memory files.
 	 * Returns:
 	 *	Pointer to usable VSL_data handle.
+	 *	NULL: malloc failed.
 	 */
 
 typedef void VSM_diag_f(void *priv, const char *fmt, ...);
@@ -66,7 +75,7 @@ int VSM_n_Arg(struct VSM_data *vd, const char *n_arg);
 	 * and VSC_Arg() functions.
 	 * Returns:
 	 *	 1 on success
-	 *	 -1 on failure, with diagnostic on stderr.
+	 *	 -1 on failure, with diagnostic.
 	 */
 
 const char *VSM_Name(const struct VSM_data *vd);
diff --git a/include/vapi/vsm_int.h b/include/vapi/vsm_int.h
index cad1e36..c33ba35 100644
--- a/include/vapi/vsm_int.h
+++ b/include/vapi/vsm_int.h
@@ -29,6 +29,67 @@
  * Define the layout of the shared memory log segment.
  *
  * NB: THIS IS NOT A PUBLIC API TO VARNISH!
+ *
+ * There is a lot of diplomacy and protocol involved with the VSM segment
+ * since there is no way to (and no desire to!) lock between the readers
+ * and the writer.
+ *
+ * In particular we want the readers to seamlessly jump from one VSM instance
+ * to another when the child restarts.
+ *
+ * The VSM life-cycle there is:
+ *
+ *	Manager creates VSM file under temp name
+ *
+ *	Temp VSM file is initialized such that VSM_head is consistent
+ *	with a non-zero alloc_seq
+ *
+ *	Manager renames Temp VSM file to correct filename as atomic
+ *	operation.
+ *
+ *	When manager abandons VSM file, alloc_seq is set to zero, which
+ *	never happens in any other circumstances.
+ *
+ *	If a manager is started and finds and old abandonned VSM segment
+ *	it will zero the alloc_seq in it, before replacing the file.
+ *
+ * Subscribers will have to monitor two things to make sure they have
+ * the current VSM instance:  The alloc_seq field and the inode number
+ * of the path-name.  The former check is by far the cheaper and the
+ * latter check should only be employed when lack of activity in the
+ * VSM segment raises suspicion that something has happened.
+ *
+ * The allocations ("chunks") in the VSM forms a linked list, starting with
+ * VSM_head->first, with the first/next fields being byte offsets relative
+ * to the start of the VSM segment.
+ *
+ * The last chunk on the list, has next == 0.
+ *
+ * New chunks are appended to the list, no matter where in the VSM
+ * they happen to be allocated.
+ *
+ * Chunk allocation sequence is:
+ *	Find free space
+ *	Zero payload
+ *	Init Chunk header
+ *	Write memory barrier
+ *	update hdr->first or $last->next pointer
+ *	hdr->alloc_seq changes
+ *	Write memory barrier
+ *
+ * Chunk contents should be designed so that zero bytes are not mistaken
+ * for valid contents.
+ *
+ * Chunk deallocation sequence is:
+ *	update hdr->first or $prev->next pointer
+ *	Write memory barrier
+ *	this->len = 0
+ *	hdr->alloc_seq changes
+ *	Write memory barrier
+ *
+ * The space occupied by the chunk is put on a cooling list and is not
+ * recycled for at least a minute.
+ *
  */
 
 #ifndef VSM_INT_H_INCLUDED
@@ -36,80 +97,23 @@
 
 #define VSM_FILENAME		"_.vsm"
 
-/*
- * This structure describes each allocation from the shmlog
- */
-
 struct VSM_chunk {
 #define VSM_CHUNK_MAGIC		0xa15712e5	/* From /dev/random */
 	unsigned		magic;
-	unsigned		len;		/* Incl VSM_chunk */
-	unsigned		next;		/* Offset in shmem */
-	unsigned		state;		/* XXX remove */
+	ssize_t			len;		/* Incl VSM_chunk */
+	ssize_t			next;		/* Offset in shmem */
 	char			class[8];
 	char			type[8];
 	char			ident[64];
 };
 
-#define VSM_NEXT(sha)		((void*)((uintptr_t)(sha) + (sha)->len))
-#define VSM_PTR(sha)		((void*)((uintptr_t)((sha) + 1)))
-
 struct VSM_head {
 #define VSM_HEAD_MAGIC		0xe75f7e91	/* From /dev/random */
 	unsigned		magic;
-
-	unsigned		hdrsize;
-	unsigned		first;		/* Offset, first chunk */
-
-	uint64_t		starttime;
-	int64_t			master_pid;
-	int64_t			child_pid;
-
-	unsigned		shm_size;
-
+	ssize_t			hdrsize;
+	ssize_t			shm_size;
+	ssize_t			first;		/* Offset, first chunk */
 	unsigned		alloc_seq;
-	/* Must be last element */
-	struct VSM_chunk	head;
 };
 
-/*
- * You must include "miniobj.h" and have an assert function to be
- * able to use the VSM_ITER() macro.
- */
-#ifdef CHECK_OBJ_NOTNULL
-
-extern struct VSM_head *VSM_head;
-extern const struct VSM_chunk *vsm_end;
-
-static inline struct VSM_chunk *
-vsm_iter_0(void)
-{
-
-	CHECK_OBJ_NOTNULL(VSM_head, VSM_HEAD_MAGIC);
-	CHECK_OBJ_NOTNULL(&VSM_head->head, VSM_CHUNK_MAGIC);
-	return (&VSM_head->head);
-}
-
-static inline void
-vsm_iter_n(struct VSM_chunk **pp)
-{
-
-	CHECK_OBJ_NOTNULL(VSM_head, VSM_HEAD_MAGIC);
-	CHECK_OBJ_NOTNULL(*pp, VSM_CHUNK_MAGIC);
-	*pp = VSM_NEXT(*pp);
-	if (*pp >= vsm_end) {
-		*pp = NULL;
-		return;
-	}
-	CHECK_OBJ_NOTNULL(*pp, VSM_CHUNK_MAGIC);
-}
-
-#define VSM_ITER(vd) for ((vd) = vsm_iter_0(); (vd) != NULL; vsm_iter_n(&vd))
-
-#else
-
-#define VSM_ITER(vd) while (YOU_NEED_MINIOBJ_TO_USE_VSM_ITER)
-
-#endif /* CHECK_OBJ_NOTNULL */
-
 #endif /* VSM_INT_H_INCLUDED */
diff --git a/lib/libvarnishapi/vsl.c b/lib/libvarnishapi/vsl.c
index 88bd9d6..b17cd36 100644
--- a/lib/libvarnishapi/vsl.c
+++ b/lib/libvarnishapi/vsl.c
@@ -390,7 +390,8 @@ VSL_Open(struct VSM_data *vd, int diag)
 
 /*--------------------------------------------------------------------*/
 
-int VSL_Matched(const struct VSM_data *vd, uint64_t bitmap)
+int
+VSL_Matched(const struct VSM_data *vd, uint64_t bitmap)
 {
 	if (vd->vsl->num_matchers > 0) {
 		uint64_t t;
diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c
index 8e8b27b..97e3847 100644
--- a/lib/libvarnishapi/vsm.c
+++ b/lib/libvarnishapi/vsm.c
@@ -62,7 +62,8 @@ VSM_New(void)
 	struct VSM_data *vd;
 
 	ALLOC_OBJ(vd, VSM_MAGIC);
-	AN(vd);
+	if (vd == NULL)
+		return (vd);
 
 	vd->diag = (VSM_diag_f*)fprintf;
 	vd->priv = stderr;
@@ -132,15 +133,22 @@ VSM_Delete(struct VSM_data *vd)
 	if (vd->vsl != NULL)
 		VSL_Delete(vd);
 
-	free(vd);
+	FREE_OBJ(vd);
 }
 
-/*--------------------------------------------------------------------*/
+/*--------------------------------------------------------------------
+ * The internal VSM open function
+ *
+ * Return:
+ *	0 = sucess
+ * 	1 = failure
+ *
+ */
 
 static int
 vsm_open(struct VSM_data *vd, int diag)
 {
-	int i, j;
+	int i;
 	struct VSM_head slh;
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
@@ -155,7 +163,7 @@ vsm_open(struct VSM_data *vd, int diag)
 		return (1);
 	}
 
-	assert(fstat(vd->vsm_fd, &vd->fstat) == 0);
+	AZ(fstat(vd->vsm_fd, &vd->fstat));
 	if (!S_ISREG(vd->fstat.st_mode)) {
 		if (diag)
 			vd->diag(vd->priv, "%s is not a regular file\n",
@@ -174,16 +182,16 @@ vsm_open(struct VSM_data *vd, int diag)
 		vd->vsm_fd = -1;
 		return (1);
 	}
-	if (slh.magic != VSM_HEAD_MAGIC) {
+	if (slh.magic != VSM_HEAD_MAGIC || slh.alloc_seq == 0) {
 		if (diag)
-			vd->diag(vd->priv, "Wrong magic number in file %s\n",
+			vd->diag(vd->priv, "Not a ready VSM file %s\n",
 			    vd->fname);
 		AZ(close(vd->vsm_fd));
 		vd->vsm_fd = -1;
 		return (1);
 	}
 
-	vd->VSM_head = (void *)mmap(NULL, slh.shm_size,
+	vd->VSM_head = mmap(NULL, slh.shm_size,
 	    PROT_READ, MAP_SHARED|MAP_HASSEMAPHORE, vd->vsm_fd, 0);
 	if (vd->VSM_head == MAP_FAILED) {
 		if (diag)
@@ -195,20 +203,7 @@ vsm_open(struct VSM_data *vd, int diag)
 		return (1);
 	}
 	vd->vsm_end = (uint8_t *)vd->VSM_head + slh.shm_size;
-
-	for (j = 0; j < 20 && vd->VSM_head->alloc_seq == 0; j++)
-		(void)usleep(50000);
-	if (vd->VSM_head->alloc_seq == 0) {
-		if (diag)
-			vd->diag(vd->priv, "File not initialized %s\n",
-			    vd->fname);
-		assert(0 == munmap((void*)vd->VSM_head, slh.shm_size));
-		AZ(close(vd->vsm_fd));
-		vd->vsm_fd = -1;
-		vd->VSM_head = NULL;
-		return (1);
-	}
-	vd->alloc_seq = vd->VSM_head->alloc_seq;
+	vd->my_alloc_seq = vd->VSM_head->alloc_seq;
 
 	if (vd->vsl != NULL)
 		VSL_Open_CallBack(vd);
@@ -238,10 +233,10 @@ VSM_Close(struct VSM_data *vd)
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
 	if (vd->VSM_head == NULL)
 		return;
-	assert(0 == munmap((void*)vd->VSM_head, vd->VSM_head->shm_size));
+	AZ(munmap((void*)vd->VSM_head, vd->VSM_head->shm_size));
 	vd->VSM_head = NULL;
 	assert(vd->vsm_fd >= 0);
-	assert(0 == close(vd->vsm_fd));
+	AZ(close(vd->vsm_fd));
 	vd->vsm_fd = -1;
 }
 
@@ -330,10 +325,10 @@ VSM_iter0(struct VSM_data *vd)
 {
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	vd->alloc_seq = vd->VSM_head->alloc_seq;
-	while (vd->alloc_seq == 0) {
+	vd->my_alloc_seq = vd->VSM_head->alloc_seq;
+	while (vd->my_alloc_seq == 0) {
 		(void)usleep(50000);
-		vd->alloc_seq = vd->VSM_head->alloc_seq;
+		vd->my_alloc_seq = vd->VSM_head->alloc_seq;
 	}
 	CHECK_OBJ_NOTNULL(&vd->VSM_head->head, VSM_CHUNK_MAGIC);
 	return (&vd->VSM_head->head);
@@ -344,7 +339,7 @@ VSM_itern(const struct VSM_data *vd, struct VSM_chunk **pp)
 {
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	if (vd->alloc_seq != vd->VSM_head->alloc_seq) {
+	if (vd->my_alloc_seq != vd->VSM_head->alloc_seq) {
 		*pp = NULL;
 		return;
 	}
diff --git a/lib/libvarnishapi/vsm_api.h b/lib/libvarnishapi/vsm_api.h
index 519e46d..da5222e 100644
--- a/lib/libvarnishapi/vsm_api.h
+++ b/lib/libvarnishapi/vsm_api.h
@@ -44,13 +44,12 @@ struct VSM_data {
 	char			*n_opt;
 	char			*fname;
 
-
 	struct stat		fstat;
 
 	int			vsm_fd;
 	struct VSM_head		*VSM_head;
 	void			*vsm_end;
-	unsigned		alloc_seq;
+	unsigned		my_alloc_seq;
 
 	/* Stuff relating the stats fields start here */
 



From phk at varnish-cache.org  Sun Nov 20 23:39:23 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 21 Nov 2011 00:39:23 +0100
Subject: [master] 9aaaf23 Typo
Message-ID: <E1RSGyp-00087W-0u@project.varnish-software.com>

commit 9aaaf23a86853bf98494e0b69aafa311d98fbb0a
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Nov 20 14:21:17 2011 +0000

    Typo

diff --git a/include/vapi/vsm_int.h b/include/vapi/vsm_int.h
index c0a2df9..b45fdf7 100644
--- a/include/vapi/vsm_int.h
+++ b/include/vapi/vsm_int.h
@@ -108,7 +108,7 @@ struct VSM_chunk {
 };
 
 struct VSM_head {
-#define VSM_HEAD_MARKER		"VSMHEAD0"	/* Incr. as version# *
+#define VSM_HEAD_MARKER		"VSMHEAD0"	/* Incr. as version# */
 	char			marker[8];
 	ssize_t			hdrsize;
 	ssize_t			shm_size;



From phk at varnish-cache.org  Sun Nov 20 23:39:23 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 21 Nov 2011 00:39:23 +0100
Subject: [master] 5cbd8ca Make Varnish API aware of the new structure and make
	everything compile again. Much still outstanding.
Message-ID: <E1RSGyp-00087f-4j@project.varnish-software.com>

commit 5cbd8caab0650ab4178a5b387667fa8fc0672744
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Nov 20 16:11:50 2011 +0000

    Make Varnish API aware of the new structure and make everything compile
    again.  Much still outstanding.

diff --git a/bin/varnishstat/varnishstat.c b/bin/varnishstat/varnishstat.c
index d0ef409..2fe7ffb 100644
--- a/bin/varnishstat/varnishstat.c
+++ b/bin/varnishstat/varnishstat.c
@@ -289,6 +289,7 @@ main(int argc, char * const *argv)
 		exit(1);
 
 	VSC_C_main = VSC_Main(vd);
+	AN(VSC_C_main);
 
 	if (!(xml || json || once)) {
 		do_curses(vd, VSC_C_main, delay);
diff --git a/include/vapi/vsc.h b/include/vapi/vsc.h
index bc52d65..ac711ea 100644
--- a/include/vapi/vsc.h
+++ b/include/vapi/vsc.h
@@ -42,11 +42,14 @@ struct VSM_data;
 void VSC_Setup(struct VSM_data *vd);
 	/*
 	 * Setup vd for use with VSC functions.
+	 * Must be called once before any other VSC function is called
 	 */
 
 #define VSC_ARGS	"f:n:"
 #define VSC_n_USAGE	VSM_n_USAGE
-#define VSC_USAGE	VSC_N_USAGE
+#define VSC_f_USAGE	"[-f field_name,...]"
+#define VSC_USAGE	VSC_n_USAGE \
+			VSC_f_USAGE
 
 int VSC_Arg(struct VSM_data *vd, int arg, const char *opt);
 	/*
@@ -66,6 +69,7 @@ int VSC_Open(struct VSM_data *vd, int diag);
 struct VSC_C_main *VSC_Main(struct VSM_data *vd);
 	/*
 	 * return Main stats structure
+	 * returns NULL until child has been started.
 	 */
 
 struct VSC_point {
diff --git a/include/vapi/vsl.h b/include/vapi/vsl.h
index 9c9a4c9..fb02051 100644
--- a/include/vapi/vsl.h
+++ b/include/vapi/vsl.h
@@ -42,11 +42,12 @@ struct VSM_data;
 void VSL_Setup(struct VSM_data *vd);
 	/*
 	 * Setup vd for use with VSL functions.
+	 * Must be called once before any other VSL function is called.
 	 */
 
 int VSL_Open(struct VSM_data *vd, int diag);
 	/*
-	 * Attempt to open and map the shared memory file.
+	 * Attempt to open the VSM (unless -r given)
 	 * If diag is non-zero, diagnostics are emitted.
 	 * Returns:
 	 *	0 on success
diff --git a/include/vapi/vsm.h b/include/vapi/vsm.h
index 275f6d9..50a7dde 100644
--- a/include/vapi/vsm.h
+++ b/include/vapi/vsm.h
@@ -26,6 +26,8 @@
  * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
  * SUCH DAMAGE.
  *
+ * This is the public API for the VSM/VSC/VSL access.
+ *
  */
 
 #ifndef VAPI_VSM_H_INCLUDED
@@ -35,10 +37,14 @@ struct VSM_head;
 struct VSM_chunk;
 struct VSM_data;
 
+/*
+ * This structure is used to reference a VSM chunk
+ */
+
 struct VSM_fantom {
 	struct VSM_chunk	*chunk;
-	void			*b;
-	void			*e;
+	void			*b;		/* first byte of payload */
+	void			*e;		/* first byte past payload */
 	uintptr_t		priv;
 };
 
@@ -120,12 +126,31 @@ struct VSM_head *VSM_Head(const struct VSM_data *vd);
 	 * Return the head of the VSM.
 	 */
 
-void *VSM_Find_Chunk(struct VSM_data *vd, const char *class,
-    const char *type, const char *ident, unsigned *lenp);
+void VSM__iter0(struct VSM_data *vd, struct VSM_fantom *vf);
+int VSM__itern(struct VSM_data *vd, struct VSM_fantom *vf);
+
+#define VSM_FOREACH_SAFE(vf, vd) \
+    for(VSM__iter0((vd), (vf)); VSM__itern((vd), (vf));)
 	/*
-	 * Find a given chunk in the shared memory.
-	 * Returns pointer or NULL.
-	 * Lenp, if non-NULL, is set to length of chunk.
+	 * Iterate over all chunks in shared memory
+	 * vf = "struct VSM_fantom *"
+	 * vd = "struct VSM_data *"
+	 */
+
+int VSM_StillValid(struct VSM_data *vd, struct VSM_fantom *vf);
+	/*
+	 * Return:
+	 *	0: fantom is invalid now.
+	 *	1: fantom is still the same.
+	 *	2: a fantom with same dimensions exist, check class/type/ident
+	 */
+
+int VSM_Get(struct VSM_data *vd, struct VSM_fantom *vf, const char *class,
+    const char *type, const char *ident);
+	/*
+	 * Find a chunk, produce fantom for it.
+	 * Returns zero on failure.
+	 * class is mandatory, type and ident optional.
 	 */
 
 void VSM_Close(struct VSM_data *vd);
@@ -134,9 +159,28 @@ void VSM_Close(struct VSM_data *vd);
 	 * Deallocate all storage (including VSC and VSL allocations)
 	 */
 
+/**********************************************************************
+ * These are old API functions which are less safe because there is
+ * fantom protecting the chunks worked on.
+ * They will g
+ */
+
+/* OBSOLETE: Will disappear from Varnish 4.x */
+void *VSM_Find_Chunk(struct VSM_data *vd, const char *class,
+    const char *type, const char *ident, unsigned *lenp);
+	/*
+	 * Find a given chunk in the shared memory.
+	 * Returns pointer or NULL.
+	 * Lenp, if non-NULL, is set to length of chunk.
+	 */
+
+/* OBSOLETE: Will disappear from Varnish 4.x */
 struct VSM_chunk *VSM_iter0(struct VSM_data *vd);
-void VSM_itern(const struct VSM_data *vd, struct VSM_chunk **pp);
 
+/* OBSOLETE: Will disappear from Varnish 4.x */
+void VSM_itern(struct VSM_data *vd, struct VSM_chunk **pp);
+
+/* OBSOLETE: Will disappear from Varnish 4.x */
 #define VSM_FOREACH(var, vd) \
     for((var) = VSM_iter0((vd)); (var) != NULL; VSM_itern((vd), &(var)))
 
diff --git a/lib/libvarnishapi/vsc.c b/lib/libvarnishapi/vsc.c
index eea7f5c..1e2206b 100644
--- a/lib/libvarnishapi/vsc.c
+++ b/lib/libvarnishapi/vsc.c
@@ -79,7 +79,6 @@ VSC_Setup(struct VSM_data *vd)
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
 	AZ(vd->vsc);
-	AZ(vd->vsl);
 	ALLOC_OBJ(vd->vsc, VSC_MAGIC);
 	AN(vd->vsc);
 	VTAILQ_INIT(&vd->vsc->sf_list);
@@ -225,14 +224,14 @@ VSC_Open(struct VSM_data *vd, int diag)
 struct VSC_C_main *
 VSC_Main(struct VSM_data *vd)
 {
-	struct VSM_chunk *sha;
+	struct VSM_fantom vf;
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
 	CHECK_OBJ_NOTNULL(vd->vsc, VSC_MAGIC);
 
-	sha = VSM_find_alloc(vd, VSC_CLASS, "", "");
-	assert(sha != NULL);
-	return (VSM_PTR(sha));
+	if (!VSM_Get(vd, &vf, VSC_CLASS, "", ""))
+		return (NULL);
+	return ((void*)vf.b);
 }
 
 /*--------------------------------------------------------------------
@@ -285,7 +284,7 @@ iter_call(const struct vsc *vsc, VSC_iter_f *func, void *priv,
 
 #define VSC_DO(U,l,t)							\
 	static int							\
-	iter_##l(const struct vsc *vsc, struct VSM_chunk *sha,		\
+	iter_##l(const struct vsc *vsc, struct VSM_fantom *vf,		\
 	    VSC_iter_f *func, void *priv)				\
 	{								\
 		struct VSC_C_##l *st;					\
@@ -293,10 +292,9 @@ iter_call(const struct vsc *vsc, VSC_iter_f *func, void *priv,
 		int i;							\
 									\
 		CHECK_OBJ_NOTNULL(vsc, VSC_MAGIC);			\
-		CHECK_OBJ_NOTNULL(sha, VSM_CHUNK_MAGIC);		\
-		st = VSM_PTR(sha);					\
+		st = vf->b; 						\
 		sp.class = t;						\
-		sp.ident = sha->ident;
+		sp.ident = vf->chunk->ident;
 
 #define VSC_F(nn,tt,ll,ff,dd,ee)					\
 		sp.name = #nn;						\
@@ -321,24 +319,22 @@ int
 VSC_Iter(struct VSM_data *vd, VSC_iter_f *func, void *priv)
 {
 	struct vsc *vsc;
-	struct VSM_chunk *sha;
+	struct VSM_fantom vf;
 	int i;
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
 	vsc = vd->vsc;
 	CHECK_OBJ_NOTNULL(vsc, VSC_MAGIC);
-
 	i = 0;
-	VSM_FOREACH(sha, vd) {
-		CHECK_OBJ_NOTNULL(sha, VSM_CHUNK_MAGIC);
-		if (strcmp(sha->class, VSC_CLASS))
+	VSM_FOREACH_SAFE(&vf, vd) {
+		if (strcmp(vf.chunk->class, VSC_CLASS))
 			continue;
 		/*lint -save -e525 -e539 */
 #define VSC_F(n,t,l,f,d,e)
 #define VSC_DONE(a,b,c)
 #define VSC_DO(U,l,t)						\
-		if (!strcmp(sha->type, t)) {			\
-			i = iter_##l(vsc, sha, func, priv);	\
+		if (!strcmp(vf.chunk->type, t)) {		\
+			i = iter_##l(vsc, &vf, func, priv);	\
 			if (!i)					\
 				continue;			\
 		}
diff --git a/lib/libvarnishapi/vsl.c b/lib/libvarnishapi/vsl.c
index b17cd36..d896301 100644
--- a/lib/libvarnishapi/vsl.c
+++ b/lib/libvarnishapi/vsl.c
@@ -67,25 +67,14 @@ VSL_Setup(struct VSM_data *vd)
 	struct vsl *vsl;
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	AZ(vd->vsc);
 	AZ(vd->vsl);
-	ALLOC_OBJ(vsl, VSL_MAGIC);
-	AN(vsl);
-
-	vd->vsl = vsl;
-
+	ALLOC_OBJ(vd->vsl, VSL_MAGIC);
+	AN(vd->vsl);
+	vsl = vd->vsl;
 	vsl->regflags = 0;
-
-	/* XXX: Allocate only if log access */
 	vsl->vbm_supress = vbit_init(256);
 	vsl->vbm_select = vbit_init(256);
-
 	vsl->r_fd = -1;
-	/* XXX: Allocate only if -r option given ? */
-	vsl->rbuflen = 256;      /* XXX ?? */
-	vsl->rbuf = malloc(vsl->rbuflen * 4L);
-	assert(vsl->rbuf != NULL);
-
 	vsl->num_matchers = 0;
 	VTAILQ_INIT(&vsl->matchers);
 }
@@ -343,28 +332,6 @@ VSL_H_Print(void *priv, enum VSL_tag_e tag, unsigned fd, unsigned len,
 
 /*--------------------------------------------------------------------*/
 
-void
-VSL_Open_CallBack(struct VSM_data *vd)
-{
-	struct vsl *vsl;
-	struct VSM_chunk *sha;
-
-	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	vsl = vd->vsl;
-	CHECK_OBJ_NOTNULL(vsl, VSL_MAGIC);
-	sha = VSM_find_alloc(vd, VSL_CLASS, "", "");
-	assert(sha != NULL);
-
-	vsl->log_start = VSM_PTR(sha);
-	vsl->log_end = VSM_NEXT(sha);
-	vsl->log_ptr = vsl->log_start + 1;
-
-	vsl->last_seq = vsl->log_start[0];
-	VRMB();
-}
-
-/*--------------------------------------------------------------------*/
-
 int
 VSL_Open(struct VSM_data *vd, int diag)
 {
@@ -379,11 +346,21 @@ VSL_Open(struct VSM_data *vd, int diag)
 		i = VSM_Open(vd, diag);
 		if (i)
 			return (i);
-	}
-
-	if (!vsl->d_opt && vsl->r_fd == -1) {
-		while (*vsl->log_ptr != VSL_ENDMARKER)
-			vsl->log_ptr = VSL_NEXT(vsl->log_ptr);
+		if (!VSM_Get(vd, &vsl->vf, VSL_CLASS, NULL, NULL)) {
+			VSM_Close(vd);
+			if (diag)
+				vd->diag(vd->priv,
+				    "No VSL chunk found "
+				    " (child not started ?)\n");
+			return (1);
+		}
+		vsl->log_start = vsl->vf.b;
+		vsl->log_end = vsl->vf.e;
+		vsl->log_ptr = vsl->log_start + 1;
+		if (!vsl->d_opt) {
+			while (*vsl->log_ptr != VSL_ENDMARKER)
+				vsl->log_ptr = VSL_NEXT(vsl->log_ptr);
+		}
 	}
 	return (0);
 }
diff --git a/lib/libvarnishapi/vsl_api.h b/lib/libvarnishapi/vsl_api.h
index aa3d0f4..013d1ae 100644
--- a/lib/libvarnishapi/vsl_api.h
+++ b/lib/libvarnishapi/vsl_api.h
@@ -30,6 +30,10 @@
 
 #include "vqueue.h"
 
+#define	SLEEP_USEC			(50*1000)
+#define	TIMEOUT_USEC			(5*1000*1000)
+
+
 struct vsl_re_match {
 	unsigned			magic;
 #define VSL_RE_MATCH_MAGIC		0x4013151e
@@ -42,6 +46,8 @@ struct vsl {
 	unsigned		magic;
 #define VSL_MAGIC		0x7a31db38
 
+	struct VSM_fantom	vf;
+
 	/* Stuff relating the log records below here */
 
 	volatile uint32_t	*log_start;
diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c
index 97e3847..1669278 100644
--- a/lib/libvarnishapi/vsm.c
+++ b/lib/libvarnishapi/vsm.c
@@ -141,7 +141,7 @@ VSM_Delete(struct VSM_data *vd)
  *
  * Return:
  *	0 = sucess
- * 	1 = failure
+ * 	<0 = failure
  *
  */
 
@@ -150,9 +150,10 @@ vsm_open(struct VSM_data *vd, int diag)
 {
 	int i;
 	struct VSM_head slh;
+	void *v;
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	AZ(vd->VSM_head);
+	AZ(vd->head);
 	AN(vd->fname);
 
 	vd->vsm_fd = open(vd->fname, O_RDONLY);
@@ -160,7 +161,7 @@ vsm_open(struct VSM_data *vd, int diag)
 		if (diag)
 			vd->diag(vd->priv, "Cannot open %s: %s\n",
 			    vd->fname, strerror(errno));
-		return (1);
+		return (-1);
 	}
 
 	AZ(fstat(vd->vsm_fd, &vd->fstat));
@@ -170,7 +171,7 @@ vsm_open(struct VSM_data *vd, int diag)
 			    vd->fname);
 		AZ(close(vd->vsm_fd));
 		vd->vsm_fd = -1;
-		return (1);
+		return (-1);
 	}
 
 	i = read(vd->vsm_fd, &slh, sizeof slh);
@@ -180,33 +181,33 @@ vsm_open(struct VSM_data *vd, int diag)
 			    vd->fname, strerror(errno));
 		AZ(close(vd->vsm_fd));
 		vd->vsm_fd = -1;
-		return (1);
+		return (-1);
 	}
-	if (slh.magic != VSM_HEAD_MAGIC || slh.alloc_seq == 0) {
+
+	if (memcmp(slh.marker, VSM_HEAD_MARKER, sizeof slh.marker) ||
+	    slh.alloc_seq == 0) {
 		if (diag)
-			vd->diag(vd->priv, "Not a ready VSM file %s\n",
+			vd->diag(vd->priv, "Not a VSM file %s\n",
 			    vd->fname);
 		AZ(close(vd->vsm_fd));
 		vd->vsm_fd = -1;
-		return (1);
+		return (-1);
 	}
 
-	vd->VSM_head = mmap(NULL, slh.shm_size,
+	v = mmap(NULL, slh.shm_size,
 	    PROT_READ, MAP_SHARED|MAP_HASSEMAPHORE, vd->vsm_fd, 0);
-	if (vd->VSM_head == MAP_FAILED) {
+	if (v == MAP_FAILED) {
 		if (diag)
 			vd->diag(vd->priv, "Cannot mmap %s: %s\n",
 			    vd->fname, strerror(errno));
 		AZ(close(vd->vsm_fd));
 		vd->vsm_fd = -1;
-		vd->VSM_head = NULL;
-		return (1);
+		return (-1);
 	}
-	vd->vsm_end = (uint8_t *)vd->VSM_head + slh.shm_size;
-	vd->my_alloc_seq = vd->VSM_head->alloc_seq;
+	vd->head = v;
+	vd->b = v;
+	vd->e = vd->b + slh.shm_size;
 
-	if (vd->vsl != NULL)
-		VSL_Open_CallBack(vd);
 	return (0);
 }
 
@@ -218,7 +219,7 @@ VSM_Open(struct VSM_data *vd, int diag)
 {
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	AZ(vd->VSM_head);
+	AZ(vd->head);
 	if (!vd->n_opt)
 		(void)VSM_n_Arg(vd, "");
 	return (vsm_open(vd, diag));
@@ -231,11 +232,14 @@ VSM_Close(struct VSM_data *vd)
 {
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	if (vd->VSM_head == NULL)
+	if (vd->head == NULL)
 		return;
-	AZ(munmap((void*)vd->VSM_head, vd->VSM_head->shm_size));
-	vd->VSM_head = NULL;
+
 	assert(vd->vsm_fd >= 0);
+	AZ(munmap((void*)vd->b, vd->e - vd->b));
+	vd->b = NULL;
+	vd->e = NULL;
+	vd->head = NULL;
 	AZ(close(vd->vsm_fd));
 	vd->vsm_fd = -1;
 }
@@ -246,25 +250,28 @@ int
 VSM_ReOpen(struct VSM_data *vd, int diag)
 {
 	struct stat st;
-	int i;
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	AN(vd->VSM_head);
-
-	if (stat(vd->fname, &st))
-		return (0);
+	AN(vd->head);
 
-	if (st.st_dev == vd->fstat.st_dev && st.st_ino == vd->fstat.st_ino)
+	if (vd->head->alloc_seq &&
+	    !stat(vd->fname, &st) &&
+	    st.st_dev == vd->fstat.st_dev &&
+	    st.st_ino == vd->fstat.st_ino)
 		return (0);
 
 	VSM_Close(vd);
-	for (i = 0; i < 5; i++) {		/* XXX param */
-		if (!vsm_open(vd, 0))
-			return (1);
-	}
-	if (vsm_open(vd, diag))
-		return (-1);
-	return (1);
+	return (vsm_open(vd, diag));
+}
+
+/*--------------------------------------------------------------------*/
+
+unsigned
+VSM_Seq(const struct VSM_data *vd)
+{
+
+	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
+	return (vd->head->alloc_seq);
 }
 
 /*--------------------------------------------------------------------*/
@@ -274,31 +281,87 @@ VSM_Head(const struct VSM_data *vd)
 {
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	AN(vd->VSM_head);
-	return(vd->VSM_head);
+	AN(vd->head);
+	return(vd->head);
+}
+
+/*--------------------------------------------------------------------*/
+
+void
+VSM__iter0(struct VSM_data *vd, struct VSM_fantom *vf)
+{
+
+	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
+	memset(vf, 0, sizeof *vf);
 }
 
+int
+VSM__itern(struct VSM_data *vd, struct VSM_fantom *vf)
+{
+
+	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
+	if (vf->priv != 0) {
+		if (vf->priv != vd->head->alloc_seq)
+			return (0);
+		if (vf->chunk->len == 0)
+			return (0);
+		if (vf->chunk->next == 0)
+			return (0);
+		vf->chunk = (void*)(vd->b + vf->chunk->next);
+	} else if (vd->head->first == 0) {
+		return (0);
+	} else {
+		vf->chunk = (void*)(vd->b + vd->head->first);
+	}
+	if (memcmp(vf->chunk->marker, VSM_CHUNK_MARKER,
+	    sizeof vf->chunk->marker))
+		return (0);
+	vf->priv = vd->head->alloc_seq;
+	vf->b = (void*)(vf->chunk + 1);
+	vf->e = (char*)vf->b + vf->chunk->len;
+	if (vf->b == vf->e)
+		return (0);
+	return (1);
+}
 
 /*--------------------------------------------------------------------*/
 
-struct VSM_chunk *
-VSM_find_alloc(struct VSM_data *vd, const char *class, const char *type, const char *ident)
+int
+VSM_StillValid(struct VSM_data *vd, struct VSM_fantom *vf)
+{
+	struct VSM_fantom f2;
+
+	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
+	if (vf->priv == vd->head->alloc_seq)
+		return (1);
+	VSM_FOREACH_SAFE(&f2, vd) {
+		if (f2.chunk == vf->chunk &&
+		   f2.b == vf->b &&
+		   f2.e == vf->e) {
+			vf->priv = vd->head->alloc_seq;
+			return (2);
+		}
+	}
+	return (0);
+}
+
+int
+VSM_Get(struct VSM_data *vd, struct VSM_fantom *vf, const char *class,
+    const char *type, const char *ident)
 {
-	struct VSM_chunk *sha;
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	AN(vd->VSM_head);
-	VSM_FOREACH(sha, vd) {
-		CHECK_OBJ_NOTNULL(sha, VSM_CHUNK_MAGIC);
-		if (strcmp(sha->class, class))
+	VSM_FOREACH_SAFE(vf, vd) {
+		if (strcmp(vf->chunk->class, class))
 			continue;
-		if (type != NULL && strcmp(sha->type, type))
+		if (type != NULL && strcmp(vf->chunk->type, type))
 			continue;
-		if (ident != NULL && strcmp(sha->ident, ident))
+		if (ident != NULL && strcmp(vf->chunk->ident, ident))
 			continue;
-		return (sha);
+		return (1);
 	}
-	return (NULL);
+	memset(vf, 0, sizeof *vf);
+	return (0);
 }
 
 /*--------------------------------------------------------------------*/
@@ -307,15 +370,16 @@ void *
 VSM_Find_Chunk(struct VSM_data *vd, const char *class, const char *type,
     const char *ident, unsigned *lenp)
 {
-	struct VSM_chunk *sha;
+	struct VSM_fantom vf;
 
-	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	sha = VSM_find_alloc(vd, class, type, ident);
-	if (sha == NULL)
-		return (NULL);
+	if (VSM_Get(vd, &vf, class, type, ident)) {
+		if (lenp != NULL)
+			*lenp = (char*)vf.e - (char*)vf.b;
+		return (vf.chunk);
+	}
 	if (lenp != NULL)
-		*lenp = sha->len - sizeof *sha;
-	return (VSM_PTR(sha));
+		*lenp = 0;
+	return (NULL);
 }
 
 /*--------------------------------------------------------------------*/
@@ -325,38 +389,19 @@ VSM_iter0(struct VSM_data *vd)
 {
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	vd->my_alloc_seq = vd->VSM_head->alloc_seq;
-	while (vd->my_alloc_seq == 0) {
-		(void)usleep(50000);
-		vd->my_alloc_seq = vd->VSM_head->alloc_seq;
-	}
-	CHECK_OBJ_NOTNULL(&vd->VSM_head->head, VSM_CHUNK_MAGIC);
-	return (&vd->VSM_head->head);
+	VSM__iter0(vd, &vd->compat_vf);
+	if (VSM__itern(vd, &vd->compat_vf))
+		return(vd->compat_vf.chunk);
+	return (NULL);
 }
 
 void
-VSM_itern(const struct VSM_data *vd, struct VSM_chunk **pp)
+VSM_itern(struct VSM_data *vd, struct VSM_chunk **pp)
 {
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	if (vd->my_alloc_seq != vd->VSM_head->alloc_seq) {
-		*pp = NULL;
-		return;
-	}
-	CHECK_OBJ_NOTNULL(*pp, VSM_CHUNK_MAGIC);
-	*pp = VSM_NEXT(*pp);
-	if ((void*)(*pp) >= vd->vsm_end) {
+	if (VSM__itern(vd, &vd->compat_vf))
+		*pp = vd->compat_vf.chunk;
+	else
 		*pp = NULL;
-		return;
-	}
-	CHECK_OBJ_NOTNULL(*pp, VSM_CHUNK_MAGIC);
-}
-
-/*--------------------------------------------------------------------*/
-unsigned
-VSM_Seq(const struct VSM_data *vd)
-{
-
-	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	return (vd->VSM_head->alloc_seq);
 }
diff --git a/lib/libvarnishapi/vsm_api.h b/lib/libvarnishapi/vsm_api.h
index da5222e..84ea523 100644
--- a/lib/libvarnishapi/vsm_api.h
+++ b/lib/libvarnishapi/vsm_api.h
@@ -28,10 +28,6 @@
  *
  */
 
-/* Parameters */
-#define			SLEEP_USEC	(50*1000)
-#define			TIMEOUT_USEC	(5*1000*1000)
-
 struct vsc;
 
 struct VSM_data {
@@ -47,9 +43,12 @@ struct VSM_data {
 	struct stat		fstat;
 
 	int			vsm_fd;
-	struct VSM_head		*VSM_head;
-	void			*vsm_end;
-	unsigned		my_alloc_seq;
+	struct VSM_head		*head;
+	char			*b;
+	char			*e;
+
+	/* Stuff for backwards compat */
+	struct VSM_fantom	compat_vf;
 
 	/* Stuff relating the stats fields start here */
 
@@ -62,4 +61,3 @@ struct VSM_chunk *VSM_find_alloc(struct VSM_data *vd, const char *class,
 
 void VSC_Delete(struct VSM_data *vd);
 void VSL_Delete(struct VSM_data *vd);
-void VSL_Open_CallBack(struct VSM_data *vd);



From phk at varnish-cache.org  Sun Nov 20 23:39:23 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 21 Nov 2011 00:39:23 +0100
Subject: [master] 889a8e4 White-space & comment cleanup
Message-ID: <E1RSGyp-00087u-Ex@project.varnish-software.com>

commit 889a8e4c05d8fb9dd5e17dfbd63f825e4822798e
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Nov 20 19:39:46 2011 +0000

    White-space & comment cleanup

diff --git a/bin/varnishd/common/common_vsm.c b/bin/varnishd/common/common_vsm.c
index 781b01c..a2a4fe4 100644
--- a/bin/varnishd/common/common_vsm.c
+++ b/bin/varnishd/common/common_vsm.c
@@ -49,10 +49,10 @@
 /*--------------------------------------------------------------------*/
 
 struct vsm_range {
-	unsigned 			magic;
+	unsigned			magic;
 #define VSM_RANGE_MAGIC			0x8d30f14
 	VTAILQ_ENTRY(vsm_range)		list;
-	ssize_t				off; 
+	ssize_t				off;
 	ssize_t				len;
 	double				cool;
 	struct VSM_chunk		*chunk;
@@ -60,7 +60,7 @@ struct vsm_range {
 };
 
 struct vsm_sc {
-	unsigned 			magic;
+	unsigned			magic;
 #define VSM_SC_MAGIC			0x8b83270d
 	char				*b;
 	ssize_t				len;
@@ -236,7 +236,7 @@ VSM_common_alloc(struct vsm_sc *sc, ssize_t size,
 	vr3 = VTAILQ_FIRST(&sc->r_used);
 	VTAILQ_INSERT_HEAD(&sc->r_used, vr, list);
 
-	if (vr3 != NULL) {	
+	if (vr3 != NULL) {
 		AZ(vr3->chunk->next);
 		vr3->chunk->next = vr->off;
 	} else {
diff --git a/bin/varnishd/mgt/mgt_shmem.c b/bin/varnishd/mgt/mgt_shmem.c
index 6bb8e13..40f0c24 100644
--- a/bin/varnishd/mgt/mgt_shmem.c
+++ b/bin/varnishd/mgt/mgt_shmem.c
@@ -26,62 +26,6 @@
  * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
  * SUCH DAMAGE.
  *
- *
- * TODO:
- *
- * There is a risk that the child process might corrupt the VSM segment
- * and we should capture that event and recover gracefully.
- *
- * A possible state diagram could be:
- *
- *	[manager start]
- *		|
- *		v
- *      Open old VSM,
- *	check pid	--------> exit/fail (-n message)
- *		|
- *		+<----------------------+
- *		|			^
- *		v			|
- *	Create new VSM			|
- *		|			|
- *		v			|
- *	Init header			|
- *	Alloc VSL			|
- *	Alloc VSC:Main			|
- *	Alloc Args etc.			|
- *		|			|
- *		+<--------------+	|
- *		|		^	|
- *		v		|	|
- *	start worker		|	|
- *		|		|	|
- *		|		|	+<---- worker crash
- *		v		|	^
- *	Reset VSL ptr.		|	|
- *	Reset VSC counters	|	|
- *		|		|	|
- *		+<------+	|	|
- *		|	^	|	|
- *		v	|	|	|
- *	alloc dynamics	|	|	|
- *	free dynamics	|	|	|
- *		|	|	|	|
- *		v	|	|	|
- *		+------>+	|	|
- *		|		|	|
- *		v		|	|
- *	stop worker		|	|
- *		|		|	|
- *		v		|	|
- *	Check consist---------- | ----->+
- *		|		|
- *		v		|
- *	Free dynamics		|
- *		|		|
- *		v		|
- *		+-------------->+
- *
  */
 
 #include "config.h"
@@ -241,7 +185,7 @@ mgt_SHM_Init(void)
 	bprintf(fnbuf, "%s.%jd", VSM_FILENAME, (intmax_t)getpid());
 
 	vsm_fd = vsm_zerofile(fnbuf, size);
-	if (vsm_fd < 0) 
+	if (vsm_fd < 0)
 		exit(1);
 
 	p = (void *)mmap(NULL, size,
diff --git a/lib/libvarnish/cli_common.c b/lib/libvarnish/cli_common.c
index aa5fd63..74c966f 100644
--- a/lib/libvarnish/cli_common.c
+++ b/lib/libvarnish/cli_common.c
@@ -212,7 +212,7 @@ VCLI_ReadResult(int fd, unsigned *status, char **ptr, double tmo)
 			break;
 		if (p[v] != '\n')
 			break;
-		
+
 		p[v] = '\0';
 		if (ptr == NULL)
 			free(p);
diff --git a/lib/libvarnishapi/vsc.c b/lib/libvarnishapi/vsc.c
index 1e2206b..4fc2233 100644
--- a/lib/libvarnishapi/vsc.c
+++ b/lib/libvarnishapi/vsc.c
@@ -292,7 +292,7 @@ iter_call(const struct vsc *vsc, VSC_iter_f *func, void *priv,
 		int i;							\
 									\
 		CHECK_OBJ_NOTNULL(vsc, VSC_MAGIC);			\
-		st = vf->b; 						\
+		st = vf->b;						\
 		sp.class = t;						\
 		sp.ident = vf->chunk->ident;
 
diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c
index 1669278..81007d8 100644
--- a/lib/libvarnishapi/vsm.c
+++ b/lib/libvarnishapi/vsm.c
@@ -141,7 +141,7 @@ VSM_Delete(struct VSM_data *vd)
  *
  * Return:
  *	0 = sucess
- * 	<0 = failure
+ *	<0 = failure
  *
  */
 



From phk at varnish-cache.org  Sun Nov 20 23:39:23 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 21 Nov 2011 00:39:23 +0100
Subject: [master] eb18d9b Move VSC_C_main to cache namespace only
Message-ID: <E1RSGyp-000883-Kw@project.varnish-software.com>

commit eb18d9be13c042ee95b4c102641edb7693003c9f
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Nov 20 19:42:14 2011 +0000

    Move VSC_C_main to cache namespace only

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 432e22b..e08cc32 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -850,6 +850,7 @@ int SES_Schedule(struct sess *sp);
 
 
 /* cache_shmlog.c */
+extern struct VSC_C_main *VSC_C_main;
 void VSL_Init(void);
 void *VSM_Alloc(unsigned size, const char *class, const char *type,
     const char *ident);
diff --git a/bin/varnishd/cache/cache_shmlog.c b/bin/varnishd/cache/cache_shmlog.c
index 950f57f..01f6bce 100644
--- a/bin/varnishd/cache/cache_shmlog.c
+++ b/bin/varnishd/cache/cache_shmlog.c
@@ -47,6 +47,8 @@ static uint32_t			*vsl_start;
 static const uint32_t		*vsl_end;
 static uint32_t			*vsl_ptr;
 
+struct VSC_C_main       *VSC_C_main;
+
 static inline uint32_t
 vsl_w0(uint32_t type, uint32_t length)
 {
diff --git a/bin/varnishd/common/common.h b/bin/varnishd/common/common.h
index a5d7633..157e5f7 100644
--- a/bin/varnishd/common/common.h
+++ b/bin/varnishd/common/common.h
@@ -49,7 +49,6 @@ extern pid_t mgt_pid;
 #define ASSERT_MGT() do { assert(getpid() == mgt_pid);} while (0)
 
 /* mgt_shmem.c */
-extern struct VSC_C_main *VSC_C_main;
 #define PAN_CLASS "Panic"
 extern char *PAN_panicstr;
 extern unsigned PAN_panicstr_len;
diff --git a/bin/varnishd/mgt/mgt_shmem.c b/bin/varnishd/mgt/mgt_shmem.c
index 40f0c24..5915b19 100644
--- a/bin/varnishd/mgt/mgt_shmem.c
+++ b/bin/varnishd/mgt/mgt_shmem.c
@@ -57,8 +57,6 @@
 #define MAP_NOSYNC 0 /* XXX Linux */
 #endif
 
-struct VSC_C_main	*VSC_C_main;
-
 static int vsm_fd = -1;
 
 /*--------------------------------------------------------------------



From phk at varnish-cache.org  Sun Nov 20 23:39:24 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 21 Nov 2011 00:39:24 +0100
Subject: [master] 9584dc0 Move cache_param entirely into the cache * namespace
	and communicate it via heritage.
Message-ID: <E1RSGyq-00088J-08@project.varnish-software.com>

commit 9584dc0365f172d2d59569d15836f1c58c803771
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Nov 20 19:59:58 2011 +0000

    Move cache_param entirely into the cache * namespace and communicate
    it via heritage.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index e08cc32..4b66309 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -784,6 +784,7 @@ int HTC_Complete(struct http_conn *htc);
 #undef HTTPH
 
 /* cache_main.c */
+extern volatile struct params * cache_param;
 void THR_SetName(const char *name);
 const char* THR_GetName(void);
 void THR_SetSession(const struct sess *sp);
diff --git a/bin/varnishd/cache/cache_main.c b/bin/varnishd/cache/cache_main.c
index eb3fa1d..568e5e5 100644
--- a/bin/varnishd/cache/cache_main.c
+++ b/bin/varnishd/cache/cache_main.c
@@ -96,6 +96,8 @@ child_main(void)
 	setbuf(stderr, NULL);
 	printf("Child starts\n");
 
+	cache_param = heritage.param;
+
 	AZ(pthread_key_create(&sp_key, NULL));
 	AZ(pthread_key_create(&name_key, NULL));
 
diff --git a/bin/varnishd/common/heritage.h b/bin/varnishd/common/heritage.h
index db8b1f0..4ad9deb 100644
--- a/bin/varnishd/common/heritage.h
+++ b/bin/varnishd/common/heritage.h
@@ -60,6 +60,8 @@ struct heritage {
 
 	struct vsm_sc			*vsm;
 
+	struct params			*param;
+
 	char				*name;
 	char                            identity[1024];
 };
diff --git a/bin/varnishd/common/params.h b/bin/varnishd/common/params.h
index cab2f49..b9b758f 100644
--- a/bin/varnishd/common/params.h
+++ b/bin/varnishd/common/params.h
@@ -193,9 +193,3 @@ struct params {
 	ssize_t			vsm_space;
 	ssize_t			vsl_space;
 };
-
-/*
- * We declare this a volatile pointer, so that reads of parameters
- * become atomic, leaving the CLI thread lattitude to change the values
- */
-extern volatile struct params * cache_param;
diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c
index b619b66..4afcf81 100644
--- a/bin/varnishd/mgt/mgt_child.c
+++ b/bin/varnishd/mgt/mgt_child.c
@@ -315,7 +315,7 @@ start_child(struct cli *cli)
 	heritage.std_fd = cp[1];
 	child_output = cp[0];
 
-	AN(cache_param);
+	AN(heritage.param);
 	if ((pid = fork()) < 0) {
 		perror("Could not fork child");
 		exit(1);
diff --git a/bin/varnishd/mgt/mgt_param.c b/bin/varnishd/mgt/mgt_param.c
index f6de562..e4be5d8 100644
--- a/bin/varnishd/mgt/mgt_param.c
+++ b/bin/varnishd/mgt/mgt_param.c
@@ -1205,8 +1205,8 @@ MCF_ParamSet(struct cli *cli, const char *param, const char *val)
 	}
 	pp->func(cli, pp, val);
 
-	if (cli->result == CLIS_OK && cache_param != NULL)
-		*cache_param = mgt_param;
+	if (cli->result == CLIS_OK && heritage.param != NULL)
+		*heritage.param = mgt_param;
 
 	if (cli->result != CLIS_OK) {
 		VCLI_Out(cli, "(attempting to set param %s to %s)\n",
diff --git a/bin/varnishd/mgt/mgt_shmem.c b/bin/varnishd/mgt/mgt_shmem.c
index 5915b19..66eb035 100644
--- a/bin/varnishd/mgt/mgt_shmem.c
+++ b/bin/varnishd/mgt/mgt_shmem.c
@@ -210,9 +210,10 @@ mgt_SHM_Init(void)
 
 	AZ(atexit(mgt_shm_atexit));
 
-	cache_param = VSM_Alloc(sizeof *cache_param, VSM_CLASS_PARAM, "", "");
-	AN(cache_param);
-	*cache_param = mgt_param;
+	heritage.param =
+	    VSM_Alloc(sizeof *heritage.param, VSM_CLASS_PARAM, "", "");
+	AN(heritage.param);
+	*heritage.param = mgt_param;
 
 	PAN_panicstr_len = 64 * 1024;
 	PAN_panicstr = VSM_Alloc(PAN_panicstr_len, PAN_CLASS, "", "");



From phk at varnish-cache.org  Sun Nov 20 23:39:24 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 21 Nov 2011 00:39:24 +0100
Subject: [master] 0ae79f5 Communicate the panic string through heritage too.
Message-ID: <E1RSGyq-00088Y-BL@project.varnish-software.com>

commit 0ae79f5e6603f0b7ff7abf4280c9d16da78f1f63
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Nov 20 20:06:52 2011 +0000

    Communicate the panic string through heritage too.

diff --git a/bin/varnishd/cache/cache_panic.c b/bin/varnishd/cache/cache_panic.c
index d112eb8..9d69c1b 100644
--- a/bin/varnishd/cache/cache_panic.c
+++ b/bin/varnishd/cache/cache_panic.c
@@ -39,6 +39,7 @@
 #include <stdlib.h>
 
 #include "cache.h"
+#include "common/heritage.h"
 
 #include "cache_backend.h"
 #include "waiter/cache_waiter.h"
@@ -53,37 +54,33 @@
  * (gdb) printf "%s", panicstr
  */
 
-static struct vsb vsps, *vsp;
+static struct vsb pan_vsp_storage, *pan_vsp;
 static pthread_mutex_t panicstr_mtx = PTHREAD_MUTEX_INITIALIZER;
 
-/* Initialized in mgt_shmem.c, points into VSM */
-char *PAN_panicstr;
-unsigned PAN_panicstr_len;
-
 /*--------------------------------------------------------------------*/
 
 static void
 pan_ws(const struct ws *ws, int indent)
 {
 
-	VSB_printf(vsp, "%*sws = %p { %s\n", indent, "",
+	VSB_printf(pan_vsp, "%*sws = %p { %s\n", indent, "",
 	    ws, ws->overflow ? "overflow" : "");
-	VSB_printf(vsp, "%*sid = \"%s\",\n", indent + 2, "", ws->id);
-	VSB_printf(vsp, "%*s{s,f,r,e} = {%p", indent + 2, "", ws->s);
+	VSB_printf(pan_vsp, "%*sid = \"%s\",\n", indent + 2, "", ws->id);
+	VSB_printf(pan_vsp, "%*s{s,f,r,e} = {%p", indent + 2, "", ws->s);
 	if (ws->f > ws->s)
-		VSB_printf(vsp, ",+%ld", (long) (ws->f - ws->s));
+		VSB_printf(pan_vsp, ",+%ld", (long) (ws->f - ws->s));
 	else
-		VSB_printf(vsp, ",%p", ws->f);
+		VSB_printf(pan_vsp, ",%p", ws->f);
 	if (ws->r > ws->s)
-		VSB_printf(vsp, ",+%ld", (long) (ws->r - ws->s));
+		VSB_printf(pan_vsp, ",+%ld", (long) (ws->r - ws->s));
 	else
-		VSB_printf(vsp, ",%p", ws->r);
+		VSB_printf(pan_vsp, ",%p", ws->r);
 	if (ws->e > ws->s)
-		VSB_printf(vsp, ",+%ld", (long) (ws->e - ws->s));
+		VSB_printf(pan_vsp, ",+%ld", (long) (ws->e - ws->s));
 	else
-		VSB_printf(vsp, ",%p", ws->e);
-	VSB_printf(vsp, "},\n");
-	VSB_printf(vsp, "%*s},\n", indent, "" );
+		VSB_printf(pan_vsp, ",%p", ws->e);
+	VSB_printf(pan_vsp, "},\n");
+	VSB_printf(pan_vsp, "%*s},\n", indent, "" );
 }
 
 /*--------------------------------------------------------------------*/
@@ -96,9 +93,9 @@ pan_vbc(const struct vbc *vbc)
 
 	be = vbc->backend;
 
-	VSB_printf(vsp, "  backend = %p fd = %d {\n", be, vbc->fd);
-	VSB_printf(vsp, "    display_name = \"%s\",\n", be->display_name);
-	VSB_printf(vsp, "  },\n");
+	VSB_printf(pan_vsp, "  backend = %p fd = %d {\n", be, vbc->fd);
+	VSB_printf(pan_vsp, "    display_name = \"%s\",\n", be->display_name);
+	VSB_printf(pan_vsp, "  },\n");
 }
 
 /*--------------------------------------------------------------------*/
@@ -111,24 +108,26 @@ pan_storage(const struct storage *st)
 #define MAX_BYTES (4*16)
 #define show(ch) (((ch) > 31 && (ch) < 127) ? (ch) : '.')
 
-	VSB_printf(vsp, "      %u {\n", st->len);
+	VSB_printf(pan_vsp, "      %u {\n", st->len);
 	for (i = 0; i < MAX_BYTES && i < st->len; i += 16) {
-		VSB_printf(vsp, "        ");
+		VSB_printf(pan_vsp, "        ");
 		for (j = 0; j < 16; ++j) {
 			if (i + j < st->len)
-				VSB_printf(vsp, "%02x ", st->ptr[i + j]);
+				VSB_printf(pan_vsp, "%02x ", st->ptr[i + j]);
 			else
-				VSB_printf(vsp, "   ");
+				VSB_printf(pan_vsp, "   ");
 		}
-		VSB_printf(vsp, "|");
+		VSB_printf(pan_vsp, "|");
 		for (j = 0; j < 16; ++j)
 			if (i + j < st->len)
-				VSB_printf(vsp, "%c", show(st->ptr[i + j]));
-		VSB_printf(vsp, "|\n");
+				VSB_printf(pan_vsp,
+				    "%c", show(st->ptr[i + j]));
+		VSB_printf(pan_vsp, "|\n");
 	}
 	if (st->len > MAX_BYTES)
-		VSB_printf(vsp, "        [%u more]\n", st->len - MAX_BYTES);
-	VSB_printf(vsp, "      },\n");
+		VSB_printf(pan_vsp,
+		    "        [%u more]\n", st->len - MAX_BYTES);
+	VSB_printf(pan_vsp, "      },\n");
 
 #undef show
 #undef MAX_BYTES
@@ -141,17 +140,17 @@ pan_http(const char *id, const struct http *h, int indent)
 {
 	int i;
 
-	VSB_printf(vsp, "%*shttp[%s] = {\n", indent, "", id);
-	VSB_printf(vsp, "%*sws = %p[%s]\n", indent + 2, "",
+	VSB_printf(pan_vsp, "%*shttp[%s] = {\n", indent, "", id);
+	VSB_printf(pan_vsp, "%*sws = %p[%s]\n", indent + 2, "",
 	    h->ws, h->ws ? h->ws->id : "");
 	for (i = 0; i < h->nhd; ++i) {
 		if (h->hd[i].b == NULL && h->hd[i].e == NULL)
 			continue;
-		VSB_printf(vsp, "%*s\"%.*s\",\n", indent + 4, "",
+		VSB_printf(pan_vsp, "%*s\"%.*s\",\n", indent + 4, "",
 		    (int)(h->hd[i].e - h->hd[i].b),
 		    h->hd[i].b);
 	}
-	VSB_printf(vsp, "%*s},\n", indent, "");
+	VSB_printf(pan_vsp, "%*s},\n", indent, "");
 }
 
 
@@ -162,16 +161,16 @@ pan_object(const struct object *o)
 {
 	const struct storage *st;
 
-	VSB_printf(vsp, "  obj = %p {\n", o);
-	VSB_printf(vsp, "    xid = %u,\n", o->xid);
+	VSB_printf(pan_vsp, "  obj = %p {\n", o);
+	VSB_printf(pan_vsp, "    xid = %u,\n", o->xid);
 	pan_ws(o->ws_o, 4);
 	pan_http("obj", o->http, 4);
-	VSB_printf(vsp, "    len = %jd,\n", (intmax_t)o->len);
-	VSB_printf(vsp, "    store = {\n");
+	VSB_printf(pan_vsp, "    len = %jd,\n", (intmax_t)o->len);
+	VSB_printf(pan_vsp, "    store = {\n");
 	VTAILQ_FOREACH(st, &o->store, list)
 		pan_storage(st);
-	VSB_printf(vsp, "    },\n");
-	VSB_printf(vsp, "  },\n");
+	VSB_printf(pan_vsp, "    },\n");
+	VSB_printf(pan_vsp, "  },\n");
 }
 
 /*--------------------------------------------------------------------*/
@@ -181,12 +180,12 @@ pan_vcl(const struct VCL_conf *vcl)
 {
 	int i;
 
-	VSB_printf(vsp, "    vcl = {\n");
-	VSB_printf(vsp, "      srcname = {\n");
+	VSB_printf(pan_vsp, "    vcl = {\n");
+	VSB_printf(pan_vsp, "      srcname = {\n");
 	for (i = 0; i < vcl->nsrc; ++i)
-		VSB_printf(vsp, "        \"%s\",\n", vcl->srcname[i]);
-	VSB_printf(vsp, "      },\n");
-	VSB_printf(vsp, "    },\n");
+		VSB_printf(pan_vsp, "        \"%s\",\n", vcl->srcname[i]);
+	VSB_printf(pan_vsp, "      },\n");
+	VSB_printf(pan_vsp, "    },\n");
 }
 
 
@@ -196,7 +195,7 @@ static void
 pan_wrk(const struct worker *wrk)
 {
 
-	VSB_printf(vsp, "  worker = %p {\n", wrk);
+	VSB_printf(pan_vsp, "  worker = %p {\n", wrk);
 	pan_ws(wrk->ws, 4);
 	if (wrk->bereq->ws != NULL)
 		pan_http("bereq", wrk->bereq, 4);
@@ -204,7 +203,7 @@ pan_wrk(const struct worker *wrk)
 		pan_http("beresp", wrk->beresp, 4);
 	if (wrk->resp->ws != NULL)
 		pan_http("resp", wrk->resp, 4);
-	VSB_printf(vsp, "    },\n");
+	VSB_printf(pan_vsp, "    },\n");
 }
 
 /*--------------------------------------------------------------------*/
@@ -214,11 +213,11 @@ pan_sess(const struct sess *sp)
 {
 	const char *stp, *hand;
 
-	VSB_printf(vsp, "sp = %p {\n", sp);
-	VSB_printf(vsp,
+	VSB_printf(pan_vsp, "sp = %p {\n", sp);
+	VSB_printf(pan_vsp,
 	    "  fd = %d, id = %d, xid = %u,\n",
 	    sp->fd, sp->vsl_id & VSL_IDENTMASK, sp->xid);
-	VSB_printf(vsp, "  client = %s %s,\n",
+	VSB_printf(pan_vsp, "  client = %s %s,\n",
 	    sp->addr ? sp->addr : "?.?.?.?",
 	    sp->port ? sp->port : "?");
 	switch (sp->step) {
@@ -229,31 +228,31 @@ pan_sess(const struct sess *sp)
 	}
 	hand = VCL_Return_Name(sp->handling);
 	if (stp != NULL)
-		VSB_printf(vsp, "  step = %s,\n", stp);
+		VSB_printf(pan_vsp, "  step = %s,\n", stp);
 	else
-		VSB_printf(vsp, "  step = 0x%x,\n", sp->step);
+		VSB_printf(pan_vsp, "  step = 0x%x,\n", sp->step);
 	if (hand != NULL)
-		VSB_printf(vsp, "  handling = %s,\n", hand);
+		VSB_printf(pan_vsp, "  handling = %s,\n", hand);
 	else
-		VSB_printf(vsp, "  handling = 0x%x,\n", sp->handling);
+		VSB_printf(pan_vsp, "  handling = 0x%x,\n", sp->handling);
 	if (sp->err_code)
-		VSB_printf(vsp,
+		VSB_printf(pan_vsp,
 		    "  err_code = %d, err_reason = %s,\n", sp->err_code,
 		    sp->err_reason ? sp->err_reason : "(null)");
 
-	VSB_printf(vsp, "  restarts = %d, esi_level = %d\n",
+	VSB_printf(pan_vsp, "  restarts = %d, esi_level = %d\n",
 	    sp->restarts, sp->esi_level);
 
-	VSB_printf(vsp, "  flags = ");
-	if (sp->wrk->do_stream)	VSB_printf(vsp, " do_stream");
-	if (sp->wrk->do_gzip)	VSB_printf(vsp, " do_gzip");
-	if (sp->wrk->do_gunzip)	VSB_printf(vsp, " do_gunzip");
-	if (sp->wrk->do_esi)	VSB_printf(vsp, " do_esi");
-	if (sp->wrk->do_close)	VSB_printf(vsp, " do_close");
-	if (sp->wrk->is_gzip)	VSB_printf(vsp, " is_gzip");
-	if (sp->wrk->is_gunzip)	VSB_printf(vsp, " is_gunzip");
-	VSB_printf(vsp, "\n");
-	VSB_printf(vsp, "  bodystatus = %d\n", sp->wrk->body_status);
+	VSB_printf(pan_vsp, "  flags = ");
+	if (sp->wrk->do_stream)	VSB_printf(pan_vsp, " do_stream");
+	if (sp->wrk->do_gzip)	VSB_printf(pan_vsp, " do_gzip");
+	if (sp->wrk->do_gunzip)	VSB_printf(pan_vsp, " do_gunzip");
+	if (sp->wrk->do_esi)	VSB_printf(pan_vsp, " do_esi");
+	if (sp->wrk->do_close)	VSB_printf(pan_vsp, " do_close");
+	if (sp->wrk->is_gzip)	VSB_printf(pan_vsp, " is_gzip");
+	if (sp->wrk->is_gunzip)	VSB_printf(pan_vsp, " is_gunzip");
+	VSB_printf(pan_vsp, "\n");
+	VSB_printf(pan_vsp, "  bodystatus = %d\n", sp->wrk->body_status);
 
 	pan_ws(sp->ws, 2);
 	pan_http("req", sp->http, 2);
@@ -270,7 +269,7 @@ pan_sess(const struct sess *sp)
 	if (VALID_OBJ(sp->obj, OBJECT_MAGIC))
 		pan_object(sp->obj);
 
-	VSB_printf(vsp, "},\n");
+	VSB_printf(pan_vsp, "},\n");
 }
 
 /*--------------------------------------------------------------------*/
@@ -285,18 +284,19 @@ pan_backtrace(void)
 	size = backtrace (array, 10);
 	if (size == 0)
 		return;
-	VSB_printf(vsp, "Backtrace:\n");
+	VSB_printf(pan_vsp, "Backtrace:\n");
 	for (i = 0; i < size; i++) {
-		VSB_printf (vsp, "  ");
-		if (Symbol_Lookup(vsp, array[i]) < 0) {
+		VSB_printf (pan_vsp, "  ");
+		if (Symbol_Lookup(pan_vsp, array[i]) < 0) {
 			char **strings;
 			strings = backtrace_symbols(&array[i], 1);
 			if (strings != NULL && strings[0] != NULL)
-				VSB_printf(vsp, "%p: %s", array[i], strings[0]);
+				VSB_printf(pan_vsp,
+				     "%p: %s", array[i], strings[0]);
 			else
-				VSB_printf(vsp, "%p: (?)", array[i]);
+				VSB_printf(pan_vsp, "%p: (?)", array[i]);
 		}
-		VSB_printf (vsp, "\n");
+		VSB_printf (pan_vsp, "\n");
 	}
 }
 
@@ -314,35 +314,35 @@ pan_ic(const char *func, const char *file, int line, const char *cond,
 						  anyway */
 	switch(xxx) {
 	case 3:
-		VSB_printf(vsp,
+		VSB_printf(pan_vsp,
 		    "Wrong turn at %s:%d:\n%s\n", file, line, cond);
 		break;
 	case 2:
-		VSB_printf(vsp,
+		VSB_printf(pan_vsp,
 		    "Panic from VCL:\n  %s\n", cond);
 		break;
 	case 1:
-		VSB_printf(vsp,
+		VSB_printf(pan_vsp,
 		    "Missing errorhandling code in %s(), %s line %d:\n"
 		    "  Condition(%s) not true.",
 		    func, file, line, cond);
 		break;
 	default:
 	case 0:
-		VSB_printf(vsp,
+		VSB_printf(pan_vsp,
 		    "Assert error in %s(), %s line %d:\n"
 		    "  Condition(%s) not true.\n",
 		    func, file, line, cond);
 		break;
 	}
 	if (err)
-		VSB_printf(vsp, "errno = %d (%s)\n", err, strerror(err));
+		VSB_printf(pan_vsp, "errno = %d (%s)\n", err, strerror(err));
 
 	q = THR_GetName();
 	if (q != NULL)
-		VSB_printf(vsp, "thread = (%s)\n", q);
+		VSB_printf(pan_vsp, "thread = (%s)\n", q);
 
-	VSB_printf(vsp, "ident = %s,%s\n",
+	VSB_printf(pan_vsp, "ident = %s,%s\n",
 	    VSB_data(vident) + 1, WAIT_GetName());
 
 	pan_backtrace();
@@ -352,11 +352,11 @@ pan_ic(const char *func, const char *file, int line, const char *cond,
 		if (sp != NULL)
 			pan_sess(sp);
 	}
-	VSB_printf(vsp, "\n");
-	VSB_bcat(vsp, "", 1);	/* NUL termination */
+	VSB_printf(pan_vsp, "\n");
+	VSB_bcat(pan_vsp, "", 1);	/* NUL termination */
 
 	if (cache_param->diag_bitmap & 0x4000)
-		(void)fputs(PAN_panicstr, stderr);
+		(void)fputs(heritage.panic_str, stderr);
 
 	if (cache_param->diag_bitmap & 0x1000)
 		exit(4);
@@ -371,9 +371,9 @@ PAN_Init(void)
 {
 
 	VAS_Fail = pan_ic;
-	vsp = &vsps;
-	AN(PAN_panicstr);
-	AN(PAN_panicstr_len);
-	AN(VSB_new(vsp, PAN_panicstr, PAN_panicstr_len,
+	pan_vsp = &pan_vsp_storage;
+	AN(heritage.panic_str);
+	AN(heritage.panic_str_len);
+	AN(VSB_new(pan_vsp, heritage.panic_str, heritage.panic_str_len,
 	    VSB_FIXEDLEN));
 }
diff --git a/bin/varnishd/common/common.h b/bin/varnishd/common/common.h
index 157e5f7..d6de47d 100644
--- a/bin/varnishd/common/common.h
+++ b/bin/varnishd/common/common.h
@@ -50,8 +50,6 @@ extern pid_t mgt_pid;
 
 /* mgt_shmem.c */
 #define PAN_CLASS "Panic"
-extern char *PAN_panicstr;
-extern unsigned PAN_panicstr_len;
 
 /* varnishd.c */
 extern struct vsb *vident;		// XXX: -> heritage ?
diff --git a/bin/varnishd/common/heritage.h b/bin/varnishd/common/heritage.h
index 4ad9deb..9169bd5 100644
--- a/bin/varnishd/common/heritage.h
+++ b/bin/varnishd/common/heritage.h
@@ -64,6 +64,10 @@ struct heritage {
 
 	char				*name;
 	char                            identity[1024];
+
+	char				*panic_str;
+	ssize_t				panic_str_len;
+	
 };
 
 extern struct heritage heritage;
diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c
index 4afcf81..6d42260 100644
--- a/bin/varnishd/mgt/mgt_child.c
+++ b/bin/varnishd/mgt/mgt_child.c
@@ -427,10 +427,10 @@ mgt_handle_panicstr(pid_t r)
 {
 	char time_str[30];
 
-	if (PAN_panicstr[0] == '\0')
+	if (heritage.panic_str[0] == '\0')
 		return;
 	REPORT(LOG_ERR, "Child (%jd) Panic message: %s",
-	    (intmax_t)r, PAN_panicstr);
+	    (intmax_t)r, heritage.panic_str);
 
 	if (child_panic)
 		VSB_delete(child_panic);
@@ -438,7 +438,7 @@ mgt_handle_panicstr(pid_t r)
 	XXXAN(child_panic);
 	VTIM_format(VTIM_real(), time_str);
 	VSB_printf(child_panic, "Last panic at: %s\n", time_str);
-	VSB_cat(child_panic, PAN_panicstr);
+	VSB_cat(child_panic, heritage.panic_str);
 	AZ(VSB_finish(child_panic));
 }
 
diff --git a/bin/varnishd/mgt/mgt_shmem.c b/bin/varnishd/mgt/mgt_shmem.c
index 66eb035..a9a2a7e 100644
--- a/bin/varnishd/mgt/mgt_shmem.c
+++ b/bin/varnishd/mgt/mgt_shmem.c
@@ -215,7 +215,8 @@ mgt_SHM_Init(void)
 	AN(heritage.param);
 	*heritage.param = mgt_param;
 
-	PAN_panicstr_len = 64 * 1024;
-	PAN_panicstr = VSM_Alloc(PAN_panicstr_len, PAN_CLASS, "", "");
-	AN(PAN_panicstr);
+	heritage.panic_str_len = 64 * 1024;
+	heritage.panic_str =
+	    VSM_Alloc(heritage.panic_str_len, PAN_CLASS, "", "");
+	AN(heritage.panic_str);
 }



From phk at varnish-cache.org  Sun Nov 20 23:39:24 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 21 Nov 2011 00:39:24 +0100
Subject: [master] 2d1ca36 Add a "vsm" instance for the managers static VSM
	allocations (-S,
	-T args) and copy them to the "real vsm" when we create it.
Message-ID: <E1RSGyq-00088h-J6@project.varnish-software.com>

commit 2d1ca36652afc1196189f60e7aa11b3c9cb948b7
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Nov 20 20:58:29 2011 +0000

    Add a "vsm" instance for the managers static VSM allocations (-S, -T args)
    and copy them to the "real vsm" when we create it.

diff --git a/bin/varnishd/common/common.h b/bin/varnishd/common/common.h
index d6de47d..0f25d31 100644
--- a/bin/varnishd/common/common.h
+++ b/bin/varnishd/common/common.h
@@ -48,9 +48,6 @@ struct cli;
 extern pid_t mgt_pid;
 #define ASSERT_MGT() do { assert(getpid() == mgt_pid);} while (0)
 
-/* mgt_shmem.c */
-#define PAN_CLASS "Panic"
-
 /* varnishd.c */
 extern struct vsb *vident;		// XXX: -> heritage ?
 int Symbol_Lookup(struct vsb *vsb, void *ptr);
@@ -77,6 +74,7 @@ void *VSM_common_alloc(struct vsm_sc *sc, ssize_t size,
     const char *class, const char *type, const char *ident);
 void VSM_common_free(struct vsm_sc *sc, void *ptr);
 void VSM_common_delete(struct vsm_sc **sc);
+void VSM_common_copy(struct vsm_sc *to, const struct vsm_sc *from);
 
 /*---------------------------------------------------------------------
  * Generic power-2 rounding macros
diff --git a/bin/varnishd/common/common_vsm.c b/bin/varnishd/common/common_vsm.c
index a2a4fe4..b67e721 100644
--- a/bin/varnishd/common/common_vsm.c
+++ b/bin/varnishd/common/common_vsm.c
@@ -321,3 +321,23 @@ VSM_common_delete(struct vsm_sc **scp)
 	VWMB();
 	FREE_OBJ(sc);
 }
+
+/*--------------------------------------------------------------------
+ * Copy one VSM to another
+ */
+
+void
+VSM_common_copy(struct vsm_sc *to, const struct vsm_sc *from)
+{
+	struct vsm_range *vr;
+	void *p;
+
+	CHECK_OBJ_NOTNULL(to, VSM_SC_MAGIC);
+	CHECK_OBJ_NOTNULL(from, VSM_SC_MAGIC);
+	VTAILQ_FOREACH(vr, &from->r_used, list) {
+		p = VSM_common_alloc(to, vr->chunk->len,
+		    vr->chunk->class, vr->chunk->type, vr->chunk->ident);
+		AN(p);
+		memcpy(p, vr->chunk + 1, vr->chunk->len);
+	}
+}
diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h
index 775fe2c..1b5a872 100644
--- a/bin/varnishd/mgt/mgt.h
+++ b/bin/varnishd/mgt/mgt.h
@@ -82,6 +82,8 @@ void mgt_sandbox_solaris_privsep(void);
 
 /* mgt_shmem.c */
 void mgt_SHM_Init(void);
+void mgt_SHM_static_alloc(const void *, ssize_t size,
+    const char *class, const char *type, const char *ident);
 
 /* stevedore_mgt.c */
 void STV_Config(const char *spec);
@@ -109,9 +111,6 @@ extern unsigned mgt_vcc_err_unref;
 		syslog(pri, fmt, __VA_ARGS__);		\
 	} while (0)
 
-#define VSM_Alloc(a, b, c, d)	VSM_common_alloc(heritage.vsm, a,b,c,d)
-#define VSM_Free(a)		VSM_common_free(heritage.vsm, a)
-
 #if defined(PTHREAD_CANCELED) || defined(PTHREAD_MUTEX_DEFAULT)
 #error "Keep pthreads out of in manager process"
 #endif
diff --git a/bin/varnishd/mgt/mgt_cli.c b/bin/varnishd/mgt/mgt_cli.c
index d3b7396..b141f5b 100644
--- a/bin/varnishd/mgt/mgt_cli.c
+++ b/bin/varnishd/mgt/mgt_cli.c
@@ -43,7 +43,6 @@
 #include <unistd.h>
 
 #include "mgt/mgt.h"
-#include "common/heritage.h"
 #include "common/params.h"
 
 #include "vcli.h"
@@ -493,13 +492,9 @@ mgt_cli_secret(const char *S_arg)
 {
 	int i, fd;
 	char buf[BUFSIZ];
-	char *p;
 
 	/* Save in shmem */
-	i = strlen(S_arg);
-	p = VSM_Alloc(i + 1L, "Arg", "-S", "");
-	AN(p);
-	memcpy(p, S_arg, i + 1L);
+	mgt_SHM_static_alloc(S_arg, strlen(S_arg) + 1L, "Arg", "-S", "");
 
 	srandomdev();			/* XXX: why here ??? */
 	fd = open(S_arg, O_RDONLY);
@@ -527,7 +522,6 @@ mgt_cli_telnet(const char *T_arg)
 	struct vss_addr **ta;
 	int i, n, sock, good;
 	struct telnet *tn;
-	char *p;
 	struct vsb *vsb;
 	char abuf[VTCP_ADDRBUFSIZE];
 	char pbuf[VTCP_PORTBUFSIZE];
@@ -564,9 +558,7 @@ mgt_cli_telnet(const char *T_arg)
 	}
 	AZ(VSB_finish(vsb));
 	/* Save in shmem */
-	p = VSM_Alloc(VSB_len(vsb) + 1, "Arg", "-T", "");
-	AN(p);
-	memcpy(p, VSB_data(vsb), VSB_len(vsb) + 1);
+	mgt_SHM_static_alloc(VSB_data(vsb), VSB_len(vsb) + 1, "Arg", "-T", "");
 	VSB_delete(vsb);
 }
 
diff --git a/bin/varnishd/mgt/mgt_shmem.c b/bin/varnishd/mgt/mgt_shmem.c
index a9a2a7e..075b838 100644
--- a/bin/varnishd/mgt/mgt_shmem.c
+++ b/bin/varnishd/mgt/mgt_shmem.c
@@ -57,7 +57,31 @@
 #define MAP_NOSYNC 0 /* XXX Linux */
 #endif
 
-static int vsm_fd = -1;
+#define PAN_CLASS "Panic"
+
+/*--------------------------------------------------------------------
+ * Use a bogo-VSM to hold master-copies of the VSM chunks the master
+ * publishes, such as -S & -T arguments.
+ */
+
+static struct vsm_sc *static_vsm;
+static char static_vsm_buf[1024];
+
+void
+mgt_SHM_static_alloc(const void *ptr, ssize_t size,
+    const char *class, const char *type, const char *ident)
+{
+	void *p;
+
+	p = VSM_common_alloc(static_vsm, size, class, type, ident);
+	AN(p);
+	memcpy(p, ptr, size);
+	if (heritage.vsm != NULL) {
+		p = VSM_common_alloc(heritage.vsm, size, class, type, ident);
+		AN(p);
+		memcpy(p, ptr, size);
+	}
+}
 
 /*--------------------------------------------------------------------
  * Check that we are not started with the same -n argument as an already
@@ -150,36 +174,21 @@ vsm_zerofile(const char *fn, ssize_t size)
 }
 
 /*--------------------------------------------------------------------
- * Exit handler that clears the owning pid from the SHMLOG
  */
 
-static
-void
-mgt_shm_atexit(void)
+static void
+mgt_SHM_Setup(void)
 {
-
-	if (heritage.vsm != NULL)
-		VSM_common_delete(&heritage.vsm);
-}
-
-void
-mgt_SHM_Init(void)
-{
-	int i;
 	uintmax_t size, ps;
 	void *p;
 	char fnbuf[64];
+	int vsm_fd;
 
 	size = mgt_param.vsl_space + mgt_param.vsm_space;
 	ps = getpagesize();
 	size += ps - 1;
 	size &= ~(ps - 1);
 
-	/* Collision check with already running varnishd */
-	i = vsm_n_check();
-	if (i)
-		exit(i);
-
 	bprintf(fnbuf, "%s.%jd", VSM_FILENAME, (intmax_t)getpid());
 
 	vsm_fd = vsm_zerofile(fnbuf, size);
@@ -191,6 +200,8 @@ mgt_SHM_Init(void)
 	    MAP_HASSEMAPHORE | MAP_NOSYNC | MAP_SHARED,
 	    vsm_fd, 0);
 
+	AZ(close(vsm_fd));
+
 	if (p == MAP_FAILED) {
 		fprintf(stderr, "Mmap error %s: %s\n", fnbuf, strerror(errno));
 		exit (-1);
@@ -207,16 +218,46 @@ mgt_SHM_Init(void)
 		(void)unlink(fnbuf);
 		exit (-1);
 	}
+}
+
+/*--------------------------------------------------------------------
+ * Exit handler that clears the owning pid from the SHMLOG
+ */
+
+static
+void
+mgt_shm_atexit(void)
+{
+
+	if (heritage.vsm != NULL)
+		VSM_common_delete(&heritage.vsm);
+}
+
+void
+mgt_SHM_Init(void)
+{
+	int i;
+
+	/* Collision check with already running varnishd */
+	i = vsm_n_check();
+	if (i)
+		exit(i);
+
+	static_vsm = VSM_common_new(static_vsm_buf, sizeof static_vsm_buf);
+
+	mgt_SHM_Setup();
 
 	AZ(atexit(mgt_shm_atexit));
 
-	heritage.param =
-	    VSM_Alloc(sizeof *heritage.param, VSM_CLASS_PARAM, "", "");
+	VSM_common_copy(heritage.vsm, static_vsm);
+
+	heritage.param = VSM_common_alloc(heritage.vsm,
+	    sizeof *heritage.param, VSM_CLASS_PARAM, "", "");
 	AN(heritage.param);
 	*heritage.param = mgt_param;
 
 	heritage.panic_str_len = 64 * 1024;
-	heritage.panic_str =
-	    VSM_Alloc(heritage.panic_str_len, PAN_CLASS, "", "");
+	heritage.panic_str = VSM_common_alloc(heritage.vsm,
+	    heritage.panic_str_len, PAN_CLASS, "", "");
 	AN(heritage.panic_str);
 }



From phk at varnish-cache.org  Sun Nov 20 23:39:24 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 21 Nov 2011 00:39:24 +0100
Subject: [master] d68a721 Introduce the new VSM "per-child" lifetime.
Message-ID: <E1RSGyq-00088v-RZ@project.varnish-software.com>

commit d68a7210c63e3caefb2f010d48dd4c5d2ce37b90
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Nov 20 21:26:25 2011 +0000

    Introduce the new VSM "per-child" lifetime.

diff --git a/bin/varnishd/common/common_vsm.c b/bin/varnishd/common/common_vsm.c
index b67e721..4eb4195 100644
--- a/bin/varnishd/common/common_vsm.c
+++ b/bin/varnishd/common/common_vsm.c
@@ -317,7 +317,10 @@ VSM_common_delete(struct vsm_sc **scp)
 		free(vr->ptr);
 		FREE_OBJ(vr);
 	}
+
+	/* Mark VSM as abandoned */
 	sc->head->alloc_seq = 0;
+
 	VWMB();
 	FREE_OBJ(sc);
 }
diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h
index 1b5a872..52463e6 100644
--- a/bin/varnishd/mgt/mgt.h
+++ b/bin/varnishd/mgt/mgt.h
@@ -84,6 +84,10 @@ void mgt_sandbox_solaris_privsep(void);
 void mgt_SHM_Init(void);
 void mgt_SHM_static_alloc(const void *, ssize_t size,
     const char *class, const char *type, const char *ident);
+void mgt_SHM_Create(void);
+void mgt_SHM_Destroy(int keep);
+void mgt_SHM_Size_Adjust(void);
+
 
 /* stevedore_mgt.c */
 void STV_Config(const char *spec);
diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c
index 6d42260..0f91d05 100644
--- a/bin/varnishd/mgt/mgt_child.c
+++ b/bin/varnishd/mgt/mgt_child.c
@@ -315,6 +315,9 @@ start_child(struct cli *cli)
 	heritage.std_fd = cp[1];
 	child_output = cp[0];
 
+	AN(heritage.vsm);
+	mgt_SHM_Size_Adjust();
+	AN(heritage.vsm);
 	AN(heritage.param);
 	if ((pid = fork()) < 0) {
 		perror("Could not fork child");
@@ -427,8 +430,7 @@ mgt_handle_panicstr(pid_t r)
 {
 	char time_str[30];
 
-	if (heritage.panic_str[0] == '\0')
-		return;
+	AN(heritage.panic_str[0]);
 	REPORT(LOG_ERR, "Child (%jd) Panic message: %s",
 	    (intmax_t)r, heritage.panic_str);
 
@@ -492,7 +494,13 @@ mgt_sigchld(const struct vev *e, int what)
 	REPORT(LOG_INFO, "%s", VSB_data(vsb));
 	VSB_delete(vsb);
 
-	mgt_handle_panicstr(r);
+	if (heritage.panic_str[0] != '\0') {
+		mgt_handle_panicstr(r);
+		mgt_SHM_Destroy(1);
+	} else {
+		mgt_SHM_Destroy(0);
+	}
+	mgt_SHM_Create();
 
 	child_pid = -1;
 
diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c
index f782d03..cc918c1 100644
--- a/bin/varnishd/mgt/mgt_main.c
+++ b/bin/varnishd/mgt/mgt_main.c
@@ -652,6 +652,9 @@ main(int argc, char * const *argv)
 	if (T_arg != NULL)
 		mgt_cli_telnet(T_arg);
 
+	/* Instantiate VSM */
+	mgt_SHM_Create();
+
 	MGT_Run();
 
 	if (pfh != NULL)
diff --git a/bin/varnishd/mgt/mgt_shmem.c b/bin/varnishd/mgt/mgt_shmem.c
index 075b838..9f88f58 100644
--- a/bin/varnishd/mgt/mgt_shmem.c
+++ b/bin/varnishd/mgt/mgt_shmem.c
@@ -59,6 +59,9 @@
 
 #define PAN_CLASS "Panic"
 
+static void *mgt_vsm_p;
+static ssize_t mgt_vsm_l;
+
 /*--------------------------------------------------------------------
  * Use a bogo-VSM to hold master-copies of the VSM chunks the master
  * publishes, such as -S & -T arguments.
@@ -174,20 +177,31 @@ vsm_zerofile(const char *fn, ssize_t size)
 }
 
 /*--------------------------------------------------------------------
+ * Create a VSM instance
  */
 
-static void
-mgt_SHM_Setup(void)
+static size_t
+mgt_shm_size(void)
 {
-	uintmax_t size, ps;
-	void *p;
-	char fnbuf[64];
-	int vsm_fd;
+	size_t size, ps;
 
 	size = mgt_param.vsl_space + mgt_param.vsm_space;
 	ps = getpagesize();
 	size += ps - 1;
-	size &= ~(ps - 1);
+	size &= ~(ps - 1U);
+	return (size);
+}
+
+void
+mgt_SHM_Create(void)
+{
+	size_t size;
+	void *p;
+	char fnbuf[64];
+	int vsm_fd;
+
+	AZ(heritage.vsm);
+	size = mgt_shm_size();
 
 	bprintf(fnbuf, "%s.%jd", VSM_FILENAME, (intmax_t)getpid());
 
@@ -207,11 +221,26 @@ mgt_SHM_Setup(void)
 		exit (-1);
 	}
 
+	mgt_vsm_p = p;
+	mgt_vsm_l = size;
+
 	/* This may or may not work */
 	(void)mlock(p, size);
 
 	heritage.vsm = VSM_common_new(p, size);
 
+	VSM_common_copy(heritage.vsm, static_vsm);
+
+	heritage.param = VSM_common_alloc(heritage.vsm,
+	    sizeof *heritage.param, VSM_CLASS_PARAM, "", "");
+	AN(heritage.param);
+	*heritage.param = mgt_param;
+
+	heritage.panic_str_len = 64 * 1024;
+	heritage.panic_str = VSM_common_alloc(heritage.vsm,
+	    heritage.panic_str_len, PAN_CLASS, "", "");
+	AN(heritage.panic_str);
+
 	if (rename(fnbuf, VSM_FILENAME)) {
 		fprintf(stderr, "Rename failed %s -> %s: %s\n",
 		    fnbuf, VSM_FILENAME, strerror(errno));
@@ -221,6 +250,41 @@ mgt_SHM_Setup(void)
 }
 
 /*--------------------------------------------------------------------
+ * Destroy a VSM instance
+ */
+
+void
+mgt_SHM_Destroy(int keep)
+{
+
+	AN(heritage.vsm);
+	if (keep)
+		(void)rename(VSM_FILENAME, VSM_FILENAME ".keep");
+	heritage.panic_str = NULL;
+	heritage.panic_str_len = 0;
+	heritage.param = NULL;
+	VSM_common_delete(&heritage.vsm);
+	AZ(munmap(mgt_vsm_p, mgt_vsm_l));
+	mgt_vsm_p = NULL;
+	mgt_vsm_l = 0;
+}
+
+/*--------------------------------------------------------------------
+ * Destroy and recreate VSM if its size should change
+ */
+
+void
+mgt_SHM_Size_Adjust(void)
+{
+
+	AN(heritage.vsm);
+	if (mgt_vsm_l == mgt_shm_size())
+		return;
+	mgt_SHM_Destroy(0);
+	mgt_SHM_Create();
+}
+
+/*--------------------------------------------------------------------
  * Exit handler that clears the owning pid from the SHMLOG
  */
 
@@ -233,6 +297,10 @@ mgt_shm_atexit(void)
 		VSM_common_delete(&heritage.vsm);
 }
 
+/*--------------------------------------------------------------------
+ * Initialize VSM subsystem
+ */
+
 void
 mgt_SHM_Init(void)
 {
@@ -243,21 +311,9 @@ mgt_SHM_Init(void)
 	if (i)
 		exit(i);
 
+	/* Create our static VSM instance */
 	static_vsm = VSM_common_new(static_vsm_buf, sizeof static_vsm_buf);
 
-	mgt_SHM_Setup();
-
+	/* Setup atexit handler */
 	AZ(atexit(mgt_shm_atexit));
-
-	VSM_common_copy(heritage.vsm, static_vsm);
-
-	heritage.param = VSM_common_alloc(heritage.vsm,
-	    sizeof *heritage.param, VSM_CLASS_PARAM, "", "");
-	AN(heritage.param);
-	*heritage.param = mgt_param;
-
-	heritage.panic_str_len = 64 * 1024;
-	heritage.panic_str = VSM_common_alloc(heritage.vsm,
-	    heritage.panic_str_len, PAN_CLASS, "", "");
-	AN(heritage.panic_str);
 }



From phk at varnish-cache.org  Sun Nov 20 23:39:25 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 21 Nov 2011 00:39:25 +0100
Subject: [master] 635bc5f Give libvarnishapi a good flexelinting
Message-ID: <E1RSGyr-000897-2Y@project.varnish-software.com>

commit 635bc5f3a291bf5717bad4c82499dda0e4fb1bb2
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Nov 20 21:40:50 2011 +0000

    Give libvarnishapi a good flexelinting

diff --git a/bin/varnishtest/flint.lnt b/bin/varnishtest/flint.lnt
index 8f3d880..53ad3dc 100644
--- a/bin/varnishtest/flint.lnt
+++ b/bin/varnishtest/flint.lnt
@@ -1,5 +1,7 @@
 
 -esym(850, av)
+-esym(785, VSL_tags)
+-esym(528, iter_call)		// Flexelint bug
 
 -e712	// 14      Info    712     Loss of precision (___) (___ to ___)
 -e747	// 16      Info    747     Significant prototype coercion (___) ___ to ___
diff --git a/bin/varnishtest/flint.sh b/bin/varnishtest/flint.sh
index eed1686..3a67bd7 100755
--- a/bin/varnishtest/flint.sh
+++ b/bin/varnishtest/flint.sh
@@ -9,4 +9,5 @@ flexelint \
 	-I../.. \
 	../flint.lnt \
 	flint.lnt \
-	*.c 
+	*.c \
+	../../lib/libvarnishapi/*.c
diff --git a/include/vapi/vsc.h b/include/vapi/vsc.h
index ac711ea..3cf16c6 100644
--- a/include/vapi/vsc.h
+++ b/include/vapi/vsc.h
@@ -66,7 +66,7 @@ int VSC_Open(struct VSM_data *vd, int diag);
 	 * args and returns as VSM_Open()
 	 */
 
-struct VSC_C_main *VSC_Main(struct VSM_data *vd);
+struct VSC_C_main *VSC_Main(const struct VSM_data *vd);
 	/*
 	 * return Main stats structure
 	 * returns NULL until child has been started.
@@ -84,7 +84,7 @@ struct VSC_point {
 
 typedef int VSC_iter_f(void *priv, const struct VSC_point *const pt);
 
-int VSC_Iter(struct VSM_data *vd, VSC_iter_f *func, void *priv);
+int VSC_Iter(const struct VSM_data *vd, VSC_iter_f *func, void *priv);
 	/*
 	 * Iterate over all statistics counters, calling "func" for
 	 * each counter not suppressed by any "-f" arguments.
diff --git a/include/vapi/vsm.h b/include/vapi/vsm.h
index 50a7dde..48cbe70 100644
--- a/include/vapi/vsm.h
+++ b/include/vapi/vsm.h
@@ -126,8 +126,8 @@ struct VSM_head *VSM_Head(const struct VSM_data *vd);
 	 * Return the head of the VSM.
 	 */
 
-void VSM__iter0(struct VSM_data *vd, struct VSM_fantom *vf);
-int VSM__itern(struct VSM_data *vd, struct VSM_fantom *vf);
+void VSM__iter0(const struct VSM_data *vd, struct VSM_fantom *vf);
+int VSM__itern(const struct VSM_data *vd, struct VSM_fantom *vf);
 
 #define VSM_FOREACH_SAFE(vf, vd) \
     for(VSM__iter0((vd), (vf)); VSM__itern((vd), (vf));)
@@ -137,7 +137,7 @@ int VSM__itern(struct VSM_data *vd, struct VSM_fantom *vf);
 	 * vd = "struct VSM_data *"
 	 */
 
-int VSM_StillValid(struct VSM_data *vd, struct VSM_fantom *vf);
+int VSM_StillValid(const struct VSM_data *vd, struct VSM_fantom *vf);
 	/*
 	 * Return:
 	 *	0: fantom is invalid now.
@@ -145,8 +145,8 @@ int VSM_StillValid(struct VSM_data *vd, struct VSM_fantom *vf);
 	 *	2: a fantom with same dimensions exist, check class/type/ident
 	 */
 
-int VSM_Get(struct VSM_data *vd, struct VSM_fantom *vf, const char *class,
-    const char *type, const char *ident);
+int VSM_Get(const struct VSM_data *vd, struct VSM_fantom *vf,
+    const char *class, const char *type, const char *ident);
 	/*
 	 * Find a chunk, produce fantom for it.
 	 * Returns zero on failure.
@@ -166,7 +166,7 @@ void VSM_Close(struct VSM_data *vd);
  */
 
 /* OBSOLETE: Will disappear from Varnish 4.x */
-void *VSM_Find_Chunk(struct VSM_data *vd, const char *class,
+void *VSM_Find_Chunk(const struct VSM_data *vd, const char *class,
     const char *type, const char *ident, unsigned *lenp);
 	/*
 	 * Find a given chunk in the shared memory.
diff --git a/lib/libvarnishapi/vsc.c b/lib/libvarnishapi/vsc.c
index 4fc2233..5791e85 100644
--- a/lib/libvarnishapi/vsc.c
+++ b/lib/libvarnishapi/vsc.c
@@ -222,7 +222,7 @@ VSC_Open(struct VSM_data *vd, int diag)
 /*--------------------------------------------------------------------*/
 
 struct VSC_C_main *
-VSC_Main(struct VSM_data *vd)
+VSC_Main(const struct VSM_data *vd)
 {
 	struct VSM_fantom vf;
 
@@ -316,7 +316,7 @@ iter_call(const struct vsc *vsc, VSC_iter_f *func, void *priv,
 #undef VSC_DONE
 
 int
-VSC_Iter(struct VSM_data *vd, VSC_iter_f *func, void *priv)
+VSC_Iter(const struct VSM_data *vd, VSC_iter_f *func, void *priv)
 {
 	struct vsc *vsc;
 	struct VSM_fantom vf;
diff --git a/lib/libvarnishapi/vsl.c b/lib/libvarnishapi/vsl.c
index d896301..3c90381 100644
--- a/lib/libvarnishapi/vsl.c
+++ b/lib/libvarnishapi/vsl.c
@@ -294,7 +294,7 @@ VSL_Dispatch(struct VSM_data *vd, VSL_handler_f *func, void *priv)
 			s |= VSL_S_CLIENT;
 		if (VSL_BACKEND(p))
 			s |= VSL_S_BACKEND;
-		if (func(priv, VSL_TAG(p), u, l, s, VSL_DATA(p), bitmap))
+		if (func(priv, (enum VSL_tag_e)VSL_TAG(p), u, l, s, VSL_DATA(p), bitmap))
 			return (1);
 	}
 }
@@ -326,7 +326,8 @@ VSL_H_Print(void *priv, enum VSL_tag_e tag, unsigned fd, unsigned len,
 		fprintf(fo, "\"\n");
 		return (0);
 	}
-	fprintf(fo, "%5u %-12s %c %.*s\n", fd, VSL_tags[tag], type, len, ptr);
+	fprintf(fo, "%5u %-12s %c %.*s\n",
+	    fd, VSL_tags[tag], type, (int)len, ptr);
 	return (0);
 }
 
diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c
index 81007d8..c410fde 100644
--- a/lib/libvarnishapi/vsm.c
+++ b/lib/libvarnishapi/vsm.c
@@ -46,7 +46,6 @@
 
 #include "vapi/vsm.h"
 #include "vapi/vsm_int.h"
-#include "vbm.h"
 #include "vin.h"
 #include "vsm_api.h"
 
@@ -288,7 +287,7 @@ VSM_Head(const struct VSM_data *vd)
 /*--------------------------------------------------------------------*/
 
 void
-VSM__iter0(struct VSM_data *vd, struct VSM_fantom *vf)
+VSM__iter0(const struct VSM_data *vd, struct VSM_fantom *vf)
 {
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
@@ -296,7 +295,7 @@ VSM__iter0(struct VSM_data *vd, struct VSM_fantom *vf)
 }
 
 int
-VSM__itern(struct VSM_data *vd, struct VSM_fantom *vf)
+VSM__itern(const struct VSM_data *vd, struct VSM_fantom *vf)
 {
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
@@ -327,7 +326,7 @@ VSM__itern(struct VSM_data *vd, struct VSM_fantom *vf)
 /*--------------------------------------------------------------------*/
 
 int
-VSM_StillValid(struct VSM_data *vd, struct VSM_fantom *vf)
+VSM_StillValid(const struct VSM_data *vd, struct VSM_fantom *vf)
 {
 	struct VSM_fantom f2;
 
@@ -346,8 +345,8 @@ VSM_StillValid(struct VSM_data *vd, struct VSM_fantom *vf)
 }
 
 int
-VSM_Get(struct VSM_data *vd, struct VSM_fantom *vf, const char *class,
-    const char *type, const char *ident)
+VSM_Get(const struct VSM_data *vd, struct VSM_fantom *vf,
+    const char *class, const char *type, const char *ident)
 {
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
@@ -367,8 +366,8 @@ VSM_Get(struct VSM_data *vd, struct VSM_fantom *vf, const char *class,
 /*--------------------------------------------------------------------*/
 
 void *
-VSM_Find_Chunk(struct VSM_data *vd, const char *class, const char *type,
-    const char *ident, unsigned *lenp)
+VSM_Find_Chunk(const struct VSM_data *vd, const char *class,
+    const char *type, const char *ident, unsigned *lenp)
 {
 	struct VSM_fantom vf;
 
diff --git a/lib/libvarnishapi/vsm_api.h b/lib/libvarnishapi/vsm_api.h
index 84ea523..3355ee4 100644
--- a/lib/libvarnishapi/vsm_api.h
+++ b/lib/libvarnishapi/vsm_api.h
@@ -56,8 +56,5 @@ struct VSM_data {
 	struct vsl		*vsl;
 };
 
-struct VSM_chunk *VSM_find_alloc(struct VSM_data *vd, const char *class,
-    const char *type, const char *ident);
-
 void VSC_Delete(struct VSM_data *vd);
 void VSL_Delete(struct VSM_data *vd);



From phk at varnish-cache.org  Sun Nov 20 23:39:25 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 21 Nov 2011 00:39:25 +0100
Subject: [master] 07c4179 ...and make it all work again.
Message-ID: <E1RSGyr-00089Q-EB@project.varnish-software.com>

commit 07c41790c25e31914a5f6e6297f3591537efd9fd
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Nov 20 23:38:24 2011 +0000

    ...and make it all work again.
    
    At least "work" in the sense that varnishtest does not obviously
    break, but since it doesn't test the utils at all (but does use
    the varnishapi) more work & testing will be needed.

diff --git a/bin/varnishd/common/heritage.h b/bin/varnishd/common/heritage.h
index 9169bd5..b62a6ca 100644
--- a/bin/varnishd/common/heritage.h
+++ b/bin/varnishd/common/heritage.h
@@ -67,7 +67,6 @@ struct heritage {
 
 	char				*panic_str;
 	ssize_t				panic_str_len;
-	
 };
 
 extern struct heritage heritage;
diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c
index cf1742d..080f48a 100644
--- a/bin/varnishtest/vtc_varnish.c
+++ b/bin/varnishtest/vtc_varnish.c
@@ -200,17 +200,12 @@ varnishlog_thread(void *priv)
 	vsl = VSM_New();
 	VSL_Setup(vsl);
 	(void)VSL_Arg(vsl, 'n', v->workdir);
-	VSL_NonBlocking(vsl, 1);
 	while (v->pid  && VSL_Open(vsl, 0) != 0) {
 		assert(usleep(VSL_SLEEP_USEC) == 0 || errno == EINTR);
 	}
 	while (v->pid) {
-		if (VSL_Dispatch(vsl, h_addlog, v) < 0) {
-			assert(usleep(v->vsl_sleep) == 0 || errno == EINTR);
-			v->vsl_sleep += v->vsl_sleep;
-			if (v->vsl_sleep > VSL_SLEEP_USEC)
-				v->vsl_sleep = VSL_SLEEP_USEC;
-		}
+		if (VSL_Dispatch(vsl, h_addlog, v) <= 0)
+			break;
 	}
 	VSM_Delete(vsl);
 	return (NULL);
@@ -732,7 +727,7 @@ varnish_expect(const struct varnish *v, char * const *av) {
 	uint64_t ref;
 	int good;
 	char *p;
-	int i;
+	int i, j;
 	struct stat_priv sp;
 
 	good = -1;
@@ -742,9 +737,20 @@ varnish_expect(const struct varnish *v, char * const *av) {
 	ref = 0;
 	for (i = 0; i < 10; i++, (void)usleep(100000)) {
 
-		good = -1;
-		if (!VSC_Iter(v->vd, do_stat_cb, &sp))
-			continue;
+		good = VSC_Iter(v->vd, do_stat_cb, &sp);
+		if (good < 0) {
+			VSM_Close(v->vd);
+			j = VSM_Open(v->vd, 0);
+			if (j == 0)
+				continue;
+			do {
+				(void)usleep(100000);
+				j = VSM_Open(v->vd, 0);
+				i++;
+			} while(i < 10 && j < 0);
+			if (j < 0)
+				break;
+		}
 		good = 0;
 
 		ref = strtoumax(av[2], &p, 0);
diff --git a/lib/libvarnishapi/vsc.c b/lib/libvarnishapi/vsc.c
index 5791e85..c9facfd 100644
--- a/lib/libvarnishapi/vsc.c
+++ b/lib/libvarnishapi/vsc.c
@@ -326,6 +326,8 @@ VSC_Iter(const struct VSM_data *vd, VSC_iter_f *func, void *priv)
 	vsc = vd->vsc;
 	CHECK_OBJ_NOTNULL(vsc, VSC_MAGIC);
 	i = 0;
+	if (!VSM_StillValid(vd, NULL))
+		return (-1);
 	VSM_FOREACH_SAFE(&vf, vd) {
 		if (strcmp(vf.chunk->class, VSC_CLASS))
 			continue;
diff --git a/lib/libvarnishapi/vsl.c b/lib/libvarnishapi/vsl.c
index 3c90381..3e7eb40 100644
--- a/lib/libvarnishapi/vsl.c
+++ b/lib/libvarnishapi/vsl.c
@@ -51,6 +51,8 @@
 #include "vsl_api.h"
 #include "vsm_api.h"
 
+static void VSL_Close(struct VSM_data *vd);
+
 /*--------------------------------------------------------------------*/
 
 const char *VSL_tags[256] = {
@@ -128,20 +130,30 @@ VSL_NonBlocking(const struct VSM_data *vd, int nb)
 		vsl->flags &= ~F_NON_BLOCKING;
 }
 
-/*--------------------------------------------------------------------*/
+/*--------------------------------------------------------------------
+ * Return the next log record, if there is one
+ *
+ * Return:
+ *	<0: error
+ *	0: no record
+ *	>0: record available at pp
+ */
 
 static int
 vsl_nextlog(struct vsl *vsl, uint32_t **pp)
 {
-	unsigned w, l;
+	unsigned l;
 	uint32_t t;
 	int i;
 
 	CHECK_OBJ_NOTNULL(vsl, VSL_MAGIC);
 
+	*pp = NULL;
 	if (vsl->r_fd != -1) {
 		assert(vsl->rbuflen >= 8);
 		i = read(vsl->r_fd, vsl->rbuf, 8);
+		if (i == 0)
+			return (0);
 		if (i != 8)
 			return (-1);
 		l = 2 + VSL_WORDS(VSL_LEN(vsl->rbuf));
@@ -157,38 +169,36 @@ vsl_nextlog(struct vsl *vsl, uint32_t **pp)
 		*pp = vsl->rbuf;
 		return (1);
 	}
-	for (w = 0; w < TIMEOUT_USEC;) {
+	while (1) {
+		if (vsl->log_ptr == NULL)
+			return (0);
+		assert(vsl->log_ptr >= vsl->log_start + 1);
+		assert(vsl->log_ptr < vsl->log_end);
 		t = *vsl->log_ptr;
 
 		if (t == VSL_WRAPMARKER) {
 			/* Wrap around not possible at front */
-			assert(vsl->log_ptr != vsl->log_start + 1);
+			if (vsl->log_ptr == vsl->log_start + 1)
+				return (-1);
 			vsl->log_ptr = vsl->log_start + 1;
-			VRMB();
 			continue;
 		}
+
 		if (t == VSL_ENDMARKER) {
 			if (vsl->log_ptr != vsl->log_start + 1 &&
 			    vsl->last_seq != vsl->log_start[0]) {
 				/* ENDMARKER not at front and seq wrapped */
 				vsl->log_ptr = vsl->log_start + 1;
-				VRMB();
 				continue;
 			}
-			if (vsl->flags & F_NON_BLOCKING)
-				return (-1);
-			w += SLEEP_USEC;
-			assert(usleep(SLEEP_USEC) == 0 || errno == EINTR);
-			VRMB();
-			continue;
+			return (0);
 		}
+
 		if (t == 0) {
-			/* Zero-initialized VSL */
-			w += SLEEP_USEC;
-			assert(usleep(SLEEP_USEC) == 0 || errno == EINTR);
-			VRMB();
-			continue;
+			/* Uninitialized VSL */
+			return (0);
 		}
+
 		if (vsl->log_ptr == vsl->log_start + 1)
 			vsl->last_seq = vsl->log_start[0];
 
@@ -196,8 +206,6 @@ vsl_nextlog(struct vsl *vsl, uint32_t **pp)
 		vsl->log_ptr = VSL_NEXT(vsl->log_ptr);
 		return (1);
 	}
-	*pp = NULL;
-	return (0);
 }
 
 int
@@ -214,7 +222,7 @@ VSL_NextLog(const struct VSM_data *vd, uint32_t **pp, uint64_t *bits)
 
 	while (1) {
 		i = vsl_nextlog(vsl, &p);
-		if (i != 1)
+		if (i <= 0)
 			return (i);
 		t = VSL_TAG(p);
 		if (vsl->skip) {
@@ -275,17 +283,37 @@ VSL_Dispatch(struct VSM_data *vd, VSL_handler_f *func, void *priv)
 	unsigned u, l, s;
 	uint32_t *p;
 	uint64_t bitmap;
+	int tmo;
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
 	vsl = vd->vsl;
 	CHECK_OBJ_NOTNULL(vsl, VSL_MAGIC);
 
+	tmo = 0;
 	while (1) {
 		bitmap = 0;
 		i = VSL_NextLog(vd, &p, &bitmap);
-		if (i == 0 && VSM_ReOpen(vd, 0) == 1)
+		if (i == 0) {
+			if (vsl->r_fd != -1)
+				return(0);
+			if (vsl->flags & F_NON_BLOCKING)
+				return (0);
+			if (VSM_StillValid(vd, &vsl->vf) != 1) {
+				VSL_Close(vd);
+				if (VSL_Open(vd, 0))
+					return (-1);
+				AN(vsl->log_ptr);
+				assert(vsl->log_ptr >= vsl->log_start + 1);
+				assert(vsl->log_ptr < vsl->log_end);
+				continue;
+			}
+			tmo += SLEEP_USEC;
+			if (tmo > TIMEOUT_USEC)
+				return (0);
+			(void)usleep(SLEEP_USEC);
 			continue;
-		if (i != 1)
+		}
+		if (i <= 0)
 			return (i);
 		u = VSL_ID(p);
 		l = VSL_LEN(p);
@@ -294,7 +322,8 @@ VSL_Dispatch(struct VSM_data *vd, VSL_handler_f *func, void *priv)
 			s |= VSL_S_CLIENT;
 		if (VSL_BACKEND(p))
 			s |= VSL_S_BACKEND;
-		if (func(priv, (enum VSL_tag_e)VSL_TAG(p), u, l, s, VSL_DATA(p), bitmap))
+		if (func(priv, (enum VSL_tag_e)VSL_TAG(p),
+		    u, l, s, VSL_DATA(p), bitmap))
 			return (1);
 	}
 }
@@ -333,6 +362,23 @@ VSL_H_Print(void *priv, enum VSL_tag_e tag, unsigned fd, unsigned len,
 
 /*--------------------------------------------------------------------*/
 
+static void
+VSL_Close(struct VSM_data *vd)
+{
+	struct vsl *vsl;
+
+	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
+	vsl = vd->vsl;
+	CHECK_OBJ_NOTNULL(vsl, VSL_MAGIC);
+
+	VSM_Close(vd);
+	vsl->log_start = NULL;
+	vsl->log_end = NULL;
+	vsl->log_ptr = NULL;
+}
+
+/*--------------------------------------------------------------------*/
+
 int
 VSL_Open(struct VSM_data *vd, int diag)
 {
diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c
index c410fde..6c78baa 100644
--- a/lib/libvarnishapi/vsm.c
+++ b/lib/libvarnishapi/vsm.c
@@ -297,19 +297,25 @@ VSM__iter0(const struct VSM_data *vd, struct VSM_fantom *vf)
 int
 VSM__itern(const struct VSM_data *vd, struct VSM_fantom *vf)
 {
+	void *p;
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	if (vf->priv != 0) {
+	if (vd->head->alloc_seq == 0)
+		return (0);	/* abandonned VSM */
+	else if (vf->priv != 0) {
 		if (vf->priv != vd->head->alloc_seq)
 			return (0);
 		if (vf->chunk->len == 0)
 			return (0);
 		if (vf->chunk->next == 0)
 			return (0);
-		vf->chunk = (void*)(vd->b + vf->chunk->next);
+		p = (void*)(vd->b + vf->chunk->next);
+		assert(p != vf->chunk);
+		vf->chunk = p;
 	} else if (vd->head->first == 0) {
 		return (0);
 	} else {
+		AZ(vf->chunk);
 		vf->chunk = (void*)(vd->b + vd->head->first);
 	}
 	if (memcmp(vf->chunk->marker, VSM_CHUNK_MARKER,
@@ -318,8 +324,13 @@ VSM__itern(const struct VSM_data *vd, struct VSM_fantom *vf)
 	vf->priv = vd->head->alloc_seq;
 	vf->b = (void*)(vf->chunk + 1);
 	vf->e = (char*)vf->b + vf->chunk->len;
+
+	if (vd->priv == 0)
+		return (0);	/* abandonned VSM */
 	if (vf->b == vf->e)
-		return (0);
+		return (0);	/* freed chunk */
+	AN(vf->priv);
+	AN(vf->chunk);
 	return (1);
 }
 
@@ -331,6 +342,8 @@ VSM_StillValid(const struct VSM_data *vd, struct VSM_fantom *vf)
 	struct VSM_fantom f2;
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
+	if (vf == NULL)
+		return (vd->head->alloc_seq ? 1 : 0);
 	if (vf->priv == vd->head->alloc_seq)
 		return (1);
 	VSM_FOREACH_SAFE(&f2, vd) {



From phk at varnish-cache.org  Sun Nov 20 23:39:25 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 21 Nov 2011 00:39:25 +0100
Subject: [master] a77b576 Don't let VCC compilers or other subprocesses kill
	our VSM.
Message-ID: <E1RSGyr-00089I-83@project.varnish-software.com>

commit a77b5767b42ddd8ee89c9b4ffd4c75522df4442d
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Nov 20 23:27:03 2011 +0000

    Don't let VCC compilers or other subprocesses kill our VSM.

diff --git a/bin/varnishd/mgt/mgt_shmem.c b/bin/varnishd/mgt/mgt_shmem.c
index 9f88f58..79f8c41 100644
--- a/bin/varnishd/mgt/mgt_shmem.c
+++ b/bin/varnishd/mgt/mgt_shmem.c
@@ -293,6 +293,9 @@ void
 mgt_shm_atexit(void)
 {
 
+	/* Do not let VCC kill our VSM */
+	if (getpid() != mgt_pid)
+		return;
 	if (heritage.vsm != NULL)
 		VSM_common_delete(&heritage.vsm);
 }



From phk at varnish-cache.org  Mon Nov 21 08:15:36 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 21 Nov 2011 09:15:36 +0100
Subject: [master] 30a37c0 Compilers these days have no sense of fun!
Message-ID: <E1RSP2O-000229-5Q@project.varnish-software.com>

commit 30a37c06d190338089b4e0e819b063298e34d37d
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Nov 21 08:15:21 2011 +0000

    Compilers these days have no sense of fun!

diff --git a/bin/varnishd/common/common_vsm.c b/bin/varnishd/common/common_vsm.c
index 4eb4195..3853ed8 100644
--- a/bin/varnishd/common/common_vsm.c
+++ b/bin/varnishd/common/common_vsm.c
@@ -289,7 +289,7 @@ VSM_common_free(struct vsm_sc *sc, void *ptr)
 		}
 	}
 	/* Panic */
-	assert(ptr == "Bogus pointer freed");
+	assert(ptr == NULL);
 }
 
 /*--------------------------------------------------------------------



From phk at varnish-cache.org  Mon Nov 21 08:44:19 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 21 Nov 2011 09:44:19 +0100
Subject: [master] 3262173 Copyright year update
Message-ID: <E1RSPUB-0004rQ-4l@project.varnish-software.com>

commit 3262173468003a8c9ee7576620251da59a480cf4
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Nov 21 08:44:06 2011 +0000

    Copyright year update

diff --git a/bin/varnishd/cache/cache_dir_dns.c b/bin/varnishd/cache/cache_dir_dns.c
index ae96dbb..f735ebf 100644
--- a/bin/varnishd/cache/cache_dir_dns.c
+++ b/bin/varnishd/cache/cache_dir_dns.c
@@ -1,5 +1,5 @@
 /*-
- * Copyright (c) 2009-2010 Varnish Software AS
+ * Copyright (c) 2009-2011 Varnish Software AS
  * All rights reserved.
  *
  * Author: Kristian Lyngstol <kristian at redpill-linpro.com>
diff --git a/bin/varnishd/cache/cache_lck.c b/bin/varnishd/cache/cache_lck.c
index 2aef6dc..d74e31a 100644
--- a/bin/varnishd/cache/cache_lck.c
+++ b/bin/varnishd/cache/cache_lck.c
@@ -1,5 +1,5 @@
 /*-
- * Copyright (c) 2008-2010 Varnish Software AS
+ * Copyright (c) 2008-2011 Varnish Software AS
  * All rights reserved.
  *
  * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
diff --git a/bin/varnishd/cache/cache_main.c b/bin/varnishd/cache/cache_main.c
index 568e5e5..1f876cd 100644
--- a/bin/varnishd/cache/cache_main.c
+++ b/bin/varnishd/cache/cache_main.c
@@ -1,6 +1,6 @@
 /*-
  * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2009 Varnish Software AS
+ * Copyright (c) 2006-2011 Varnish Software AS
  * All rights reserved.
  *
  * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c
index 7befbcc..eee8e9a 100644
--- a/bin/varnishd/cache/cache_session.c
+++ b/bin/varnishd/cache/cache_session.c
@@ -1,6 +1,6 @@
 /*-
  * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2010 Varnish Software AS
+ * Copyright (c) 2006-2011 Varnish Software AS
  * All rights reserved.
  *
  * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c
index c20b552..5e19ccc 100644
--- a/bin/varnishd/cache/cache_vrt.c
+++ b/bin/varnishd/cache/cache_vrt.c
@@ -1,6 +1,6 @@
 /*-
  * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2010 Varnish Software AS
+ * Copyright (c) 2006-2011 Varnish Software AS
  * All rights reserved.
  *
  * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
diff --git a/bin/varnishd/cache/cache_vrt_re.c b/bin/varnishd/cache/cache_vrt_re.c
index 9b4a2e6..e9ac9de 100644
--- a/bin/varnishd/cache/cache_vrt_re.c
+++ b/bin/varnishd/cache/cache_vrt_re.c
@@ -1,6 +1,6 @@
 /*-
  * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2009 Varnish Software AS
+ * Copyright (c) 2006-2011 Varnish Software AS
  * All rights reserved.
  *
  * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
diff --git a/bin/varnishd/cache/cache_ws.c b/bin/varnishd/cache/cache_ws.c
index 9fca215..01937b3 100644
--- a/bin/varnishd/cache/cache_ws.c
+++ b/bin/varnishd/cache/cache_ws.c
@@ -1,6 +1,6 @@
 /*-
  * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2010 Varnish Software AS
+ * Copyright (c) 2006-2011 Varnish Software AS
  * All rights reserved.
  *
  * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
diff --git a/bin/varnishd/hash/hash_classic.c b/bin/varnishd/hash/hash_classic.c
index 9ea27de..b530f25 100644
--- a/bin/varnishd/hash/hash_classic.c
+++ b/bin/varnishd/hash/hash_classic.c
@@ -1,6 +1,6 @@
 /*-
  * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2009 Varnish Software AS
+ * Copyright (c) 2006-2011 Varnish Software AS
  * All rights reserved.
  *
  * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
diff --git a/bin/varnishd/hash/hash_critbit.c b/bin/varnishd/hash/hash_critbit.c
index ecb7b62..0b3a7b1 100644
--- a/bin/varnishd/hash/hash_critbit.c
+++ b/bin/varnishd/hash/hash_critbit.c
@@ -1,5 +1,5 @@
 /*-
- * Copyright (c) 2008-2010 Varnish Software AS
+ * Copyright (c) 2008-2011 Varnish Software AS
  * All rights reserved.
  *
  * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
diff --git a/bin/varnishd/hash/hash_simple_list.c b/bin/varnishd/hash/hash_simple_list.c
index 4bdae02..c1cc9c8 100644
--- a/bin/varnishd/hash/hash_simple_list.c
+++ b/bin/varnishd/hash/hash_simple_list.c
@@ -1,6 +1,6 @@
 /*-
  * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2009 Varnish Software AS
+ * Copyright (c) 2006-2011 Varnish Software AS
  * All rights reserved.
  *
  * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
diff --git a/bin/varnishd/hash/hash_slinger.h b/bin/varnishd/hash/hash_slinger.h
index 23b892d..9e3b13f 100644
--- a/bin/varnishd/hash/hash_slinger.h
+++ b/bin/varnishd/hash/hash_slinger.h
@@ -1,6 +1,6 @@
 /*-
  * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2010 Varnish Software AS
+ * Copyright (c) 2006-2011 Varnish Software AS
  * All rights reserved.
  *
  * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
diff --git a/bin/varnishd/mgt/mgt_cli.h b/bin/varnishd/mgt/mgt_cli.h
index 8c49abb..5d3ce8b 100644
--- a/bin/varnishd/mgt/mgt_cli.h
+++ b/bin/varnishd/mgt/mgt_cli.h
@@ -1,6 +1,6 @@
 /*-
  * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2009 Varnish Software AS
+ * Copyright (c) 2006-2011 Varnish Software AS
  * All rights reserved.
  *
  * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
diff --git a/bin/varnishd/storage/stevedore_utils.c b/bin/varnishd/storage/stevedore_utils.c
index 7e9c7d5..89c6027 100644
--- a/bin/varnishd/storage/stevedore_utils.c
+++ b/bin/varnishd/storage/stevedore_utils.c
@@ -1,6 +1,6 @@
 /*-
  * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2010 Varnish Software AS
+ * Copyright (c) 2006-2011 Varnish Software AS
  * All rights reserved.
  *
  * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
diff --git a/bin/varnishd/storage/storage_file.c b/bin/varnishd/storage/storage_file.c
index 2d9ccf4..9eb44d9 100644
--- a/bin/varnishd/storage/storage_file.c
+++ b/bin/varnishd/storage/storage_file.c
@@ -1,6 +1,6 @@
 /*-
  * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2010 Varnish Software AS
+ * Copyright (c) 2006-2011 Varnish Software AS
  * All rights reserved.
  *
  * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
diff --git a/bin/varnishd/waiter/cache_waiter.h b/bin/varnishd/waiter/cache_waiter.h
index 28bc39d..1d6e679 100644
--- a/bin/varnishd/waiter/cache_waiter.h
+++ b/bin/varnishd/waiter/cache_waiter.h
@@ -1,6 +1,6 @@
 /*-
  * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2009 Varnish Software AS
+ * Copyright (c) 2006-2011 Varnish Software AS
  * All rights reserved.
  *
  * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
diff --git a/bin/varnishd/waiter/cache_waiter_epoll.c b/bin/varnishd/waiter/cache_waiter_epoll.c
index b4b40f5..4d45909 100644
--- a/bin/varnishd/waiter/cache_waiter_epoll.c
+++ b/bin/varnishd/waiter/cache_waiter_epoll.c
@@ -1,6 +1,6 @@
 /*-
  * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2010 Varnish Software AS
+ * Copyright (c) 2006-2011 Varnish Software AS
  * All rights reserved.
  *
  * Author: Rogerio Carvalho Schneider <stockrt at gmail.com>
diff --git a/bin/varnishd/waiter/cache_waiter_kqueue.c b/bin/varnishd/waiter/cache_waiter_kqueue.c
index c6a03de..b331a31 100644
--- a/bin/varnishd/waiter/cache_waiter_kqueue.c
+++ b/bin/varnishd/waiter/cache_waiter_kqueue.c
@@ -1,6 +1,6 @@
 /*-
  * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2009 Varnish Software AS
+ * Copyright (c) 2006-2011 Varnish Software AS
  * All rights reserved.
  *
  * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
diff --git a/bin/varnishd/waiter/cache_waiter_poll.c b/bin/varnishd/waiter/cache_waiter_poll.c
index 3f5cea3..b4863a8 100644
--- a/bin/varnishd/waiter/cache_waiter_poll.c
+++ b/bin/varnishd/waiter/cache_waiter_poll.c
@@ -1,6 +1,6 @@
 /*-
  * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2010 Varnish Software AS
+ * Copyright (c) 2006-2011 Varnish Software AS
  * All rights reserved.
  *
  * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>



From phk at varnish-cache.org  Mon Nov 21 09:11:25 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 21 Nov 2011 10:11:25 +0100
Subject: [master] 7090d80 Allocate rbuf if -r is specified
Message-ID: <E1RSPuP-0007xr-7G@project.varnish-software.com>

commit 7090d8029742c592014dba0f34fb9db1ec4e999a
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Nov 21 09:11:13 2011 +0000

    Allocate rbuf if -r is specified

diff --git a/lib/libvarnishapi/vsl_arg.c b/lib/libvarnishapi/vsl_arg.c
index 4bcd44a..b855628 100644
--- a/lib/libvarnishapi/vsl_arg.c
+++ b/lib/libvarnishapi/vsl_arg.c
@@ -93,6 +93,10 @@ vsl_r_arg(const struct VSM_data *vd, const char *opt)
 	if (vd->vsl->r_fd < 0) {
 		perror(opt);
 		return (-1);
+	} else if (vd->vsl->rbuflen == 0) {
+		vd->vsl->rbuf = malloc(1024);
+		AN(vd->vsl->rbuf);
+		vd->vsl->rbuflen = 1024;
 	}
 	return (1);
 }



From phk at varnish-cache.org  Mon Nov 21 09:13:19 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 21 Nov 2011 10:13:19 +0100
Subject: [master] 900920c Terminate when -r reached EOF
Message-ID: <E1RSPwF-00088o-De@project.varnish-software.com>

commit 900920c301449ac631518f0da3e1830401603c56
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Nov 21 09:13:08 2011 +0000

    Terminate when -r reached EOF

diff --git a/lib/libvarnishapi/vsl.c b/lib/libvarnishapi/vsl.c
index 3e7eb40..e1ace33 100644
--- a/lib/libvarnishapi/vsl.c
+++ b/lib/libvarnishapi/vsl.c
@@ -153,7 +153,7 @@ vsl_nextlog(struct vsl *vsl, uint32_t **pp)
 		assert(vsl->rbuflen >= 8);
 		i = read(vsl->r_fd, vsl->rbuf, 8);
 		if (i == 0)
-			return (0);
+			return (-1);
 		if (i != 8)
 			return (-1);
 		l = 2 + VSL_WORDS(VSL_LEN(vsl->rbuf));



From drwilco at varnish-cache.org  Mon Nov 21 20:54:10 2011
From: drwilco at varnish-cache.org (Rogier Mulhuijzen)
Date: Mon, 21 Nov 2011 21:54:10 +0100
Subject: [master] aed74d6 Enable regression test for bug 971,
	since 1060 was a dup of it
Message-ID: <E1RSasU-0000VB-Ii@project.varnish-software.com>

commit aed74d6c93f932abcd111c2496c543d16f2745d5
Author: Rogier 'DocWilco' Mulhuijzen <github at bsdchicks.com>
Date:   Mon Nov 21 21:35:22 2011 +0100

    Enable regression test for bug 971, since 1060 was a dup of it
    
    1060 was fixed with bdbb1d59513cba8b268ed1dbe2d948619ef4ae07
    
    Fixes	#971

diff --git a/bin/varnishtest/tests.disabled/r00971.vtc b/bin/varnishtest/tests.disabled/r00971.vtc
deleted file mode 100644
index f8b57f0..0000000
--- a/bin/varnishtest/tests.disabled/r00971.vtc
+++ /dev/null
@@ -1,25 +0,0 @@
-
-varnishtest "Test DNS director order"
-
-varnish v1 -vcl+backend {
-	
-	backend test {
-		.host = "192.168.0.1";
-	}
-
-	director foo dns {
-		{ .backend = { .host = "127.0.0.1";} }
-		.list = {
-			"192.168.0.0"/24;
-		}
-	}
-
-	sub vcl_recv {
-		set req.backend = foo;
-		if (req.http.x-aa) {
-			set req.backend = test;
-		}
-	}
-
-} -start
-
diff --git a/bin/varnishtest/tests/r00971.vtc b/bin/varnishtest/tests/r00971.vtc
new file mode 100644
index 0000000..f8b57f0
--- /dev/null
+++ b/bin/varnishtest/tests/r00971.vtc
@@ -0,0 +1,25 @@
+
+varnishtest "Test DNS director order"
+
+varnish v1 -vcl+backend {
+	
+	backend test {
+		.host = "192.168.0.1";
+	}
+
+	director foo dns {
+		{ .backend = { .host = "127.0.0.1";} }
+		.list = {
+			"192.168.0.0"/24;
+		}
+	}
+
+	sub vcl_recv {
+		set req.backend = foo;
+		if (req.http.x-aa) {
+			set req.backend = test;
+		}
+	}
+
+} -start
+



From phk at varnish-cache.org  Wed Nov 23 21:11:08 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 23 Nov 2011 22:11:08 +0100
Subject: [master] f394f0a The new VSM api:
Message-ID: <E1RTK60-0007eP-18@project.varnish-software.com>

commit f394f0adba53e7a60a516c85968a0f4c95095423
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 22 08:51:07 2011 +0000

    The new VSM api:
    
    Instead of the diag() callback, use a vsb to collect diagnostics
    and return it with VSM_Error().
    
    Simpler logic around open/close.
    
    More useful Abandonned() and StillValid() functions.
    
    Still not happy with __itern(), will revisit later.

diff --git a/include/vapi/vsm.h b/include/vapi/vsm.h
index 48cbe70..277a670 100644
--- a/include/vapi/vsm.h
+++ b/include/vapi/vsm.h
@@ -26,14 +26,15 @@
  * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
  * SUCH DAMAGE.
  *
- * This is the public API for the VSM/VSC/VSL access.
+ * This is the public API for the VSM access.
+ *
+ * The VSM "class" acts as parent class for the VSL and VSC subclasses.
  *
  */
 
 #ifndef VAPI_VSM_H_INCLUDED
 #define VAPI_VSM_H_INCLUDED
 
-struct VSM_head;
 struct VSM_chunk;
 struct VSM_data;
 
@@ -45,7 +46,7 @@ struct VSM_fantom {
 	struct VSM_chunk	*chunk;
 	void			*b;		/* first byte of payload */
 	void			*e;		/* first byte past payload */
-	uintptr_t		priv;
+	uintptr_t		priv;		/* VSM private */
 };
 
 /*---------------------------------------------------------------------
@@ -56,20 +57,22 @@ struct VSM_data *VSM_New(void);
 	/*
 	 * Allocate and initialize a VSL_data handle structure.
 	 * This is the first thing you will have to do, always.
-	 * You can have multiple active VSL_data handles at the same time
+	 * You can have multiple active VSM_data handles at the same time
 	 * referencing the same or different shared memory files.
 	 * Returns:
 	 *	Pointer to usable VSL_data handle.
 	 *	NULL: malloc failed.
 	 */
 
-typedef void VSM_diag_f(void *priv, const char *fmt, ...);
+void VSM_Delete(struct VSM_data *vd);
+	/*
+	 * Close and deallocate all storage and mappings.
+	 * (including any VSC and VSL "sub-classes")
+	 */
 
-void VSM_Diag(struct VSM_data *vd, VSM_diag_f *func, void *priv);
+const char *VSM_Error(const struct VSM_data *vd);
 	/*
-	 * Set the diagnostics reporting function.
-	 * Default is fprintf(stderr, ...)
-	 * If func is NULL, diagnostics are disabled.
+	 * Return the latest error message.
 	 */
 
 #define VSM_n_USAGE	"[-n varnish_name]"
@@ -77,11 +80,10 @@ void VSM_Diag(struct VSM_data *vd, VSM_diag_f *func, void *priv);
 int VSM_n_Arg(struct VSM_data *vd, const char *n_arg);
 	/*
 	 * Configure which varnishd instance to access.
-	 * Can also be, and normally is done through the VSL_Log_arg()
-	 * and VSC_Arg() functions.
+	 * Can also be, and normally is done through VSC_Arg()/VSL_Arg().
 	 * Returns:
 	 *	 1 on success
-	 *	 -1 on failure, with diagnostic.
+	 *	 <0 on failure, use VSM_Error() to get diagnostics.
 	 */
 
 const char *VSM_Name(const struct VSM_data *vd);
@@ -89,42 +91,32 @@ const char *VSM_Name(const struct VSM_data *vd);
 	 * Return the instance name.
 	 */
 
-void VSM_Delete(struct VSM_data *vd);
+int VSM_Open(struct VSM_data *vd);
 	/*
-	 * Close and deallocate all storage and mappings.
-	 */
-
-/* XXX: extension:  Patience argument for sleeps */
-
-int VSM_Open(struct VSM_data *vd, int diag);
-	/*
-	 * Attempt to open and map the shared memory file.
+	 * Attempt to open and map the VSM file.
 	 * If diag is non-zero, diagnostics are emitted.
 	 * Returns:
 	 *	0 on success
-	 *	!= 0 on failure
+	 *	<0 on failure, use VSM_Error() to get diagnostics.
 	 */
 
-int VSM_ReOpen(struct VSM_data *vd, int diag);
+int VSM_Abandonned(const struct VSM_data *vd);
 	/*
-	 * Check if shared memory segment needs to be reopened/remapped
-	 * typically when the varnishd master process restarts.
-	 * diag is passed to VSM_Open()
+	 * Find out if the VSM file has been abandonned or closed and should
+	 * be reopened.  This function calls stat(2) and should only be
+	 * used when lack of activity or invalidation of fantoms indicate
+	 * abandonment.
+	 *
 	 * Returns:
 	 *	0  No reopen needed.
-	 *	1  shared memory reopened/remapped.
-	 *	-1 failure to reopen.
+	 *	1  VSM abandonned.
 	 */
 
-unsigned VSM_Seq(const struct VSM_data *vd);
+void VSM_Close(struct VSM_data *vd);
 	/*
-	 * Return the allocation sequence number
+	 * Close and unmap shared memory, if open.
 	 */
 
-struct VSM_head *VSM_Head(const struct VSM_data *vd);
-	/*
-	 * Return the head of the VSM.
-	 */
 
 void VSM__iter0(const struct VSM_data *vd, struct VSM_fantom *vf);
 int VSM__itern(const struct VSM_data *vd, struct VSM_fantom *vf);
@@ -139,8 +131,12 @@ int VSM__itern(const struct VSM_data *vd, struct VSM_fantom *vf);
 
 int VSM_StillValid(const struct VSM_data *vd, struct VSM_fantom *vf);
 	/*
+	 * This is a cheap syscall-less check to see if the fantom is still
+	 * valid.  Further checking with VSM_Abandonned() may be a good
+	 * idea.
+	 *
 	 * Return:
-	 *	0: fantom is invalid now.
+	 *	0: fantom is not valid any more.
 	 *	1: fantom is still the same.
 	 *	2: a fantom with same dimensions exist, check class/type/ident
 	 */
@@ -153,41 +149,4 @@ int VSM_Get(const struct VSM_data *vd, struct VSM_fantom *vf,
 	 * class is mandatory, type and ident optional.
 	 */
 
-void VSM_Close(struct VSM_data *vd);
-	/*
-	 * Unmap shared memory
-	 * Deallocate all storage (including VSC and VSL allocations)
-	 */
-
-/**********************************************************************
- * These are old API functions which are less safe because there is
- * fantom protecting the chunks worked on.
- * They will g
- */
-
-/* OBSOLETE: Will disappear from Varnish 4.x */
-void *VSM_Find_Chunk(const struct VSM_data *vd, const char *class,
-    const char *type, const char *ident, unsigned *lenp);
-	/*
-	 * Find a given chunk in the shared memory.
-	 * Returns pointer or NULL.
-	 * Lenp, if non-NULL, is set to length of chunk.
-	 */
-
-/* OBSOLETE: Will disappear from Varnish 4.x */
-struct VSM_chunk *VSM_iter0(struct VSM_data *vd);
-
-/* OBSOLETE: Will disappear from Varnish 4.x */
-void VSM_itern(struct VSM_data *vd, struct VSM_chunk **pp);
-
-/* OBSOLETE: Will disappear from Varnish 4.x */
-#define VSM_FOREACH(var, vd) \
-    for((var) = VSM_iter0((vd)); (var) != NULL; VSM_itern((vd), &(var)))
-
-	/*
-	 * Iterate over all chunks in shared memory
-	 * var = "struct VSM_chunk *"
-	 * vd = "struct VSM_data"
-	 */
-
 #endif /* VAPI_VSM_H_INCLUDED */
diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c
index 6c78baa..d136c98 100644
--- a/lib/libvarnishapi/vsm.c
+++ b/lib/libvarnishapi/vsm.c
@@ -35,6 +35,7 @@
 
 #include <errno.h>
 #include <fcntl.h>
+#include <stdarg.h>
 #include <stdint.h>
 #include <stdio.h>
 #include <stdlib.h>
@@ -47,12 +48,14 @@
 #include "vapi/vsm.h"
 #include "vapi/vsm_int.h"
 #include "vin.h"
+#include "vsb.h"
 #include "vsm_api.h"
 
 #ifndef MAP_HASSEMAPHORE
 #define MAP_HASSEMAPHORE 0 /* XXX Linux */
 #endif
 
+
 /*--------------------------------------------------------------------*/
 
 struct VSM_data *
@@ -64,9 +67,6 @@ VSM_New(void)
 	if (vd == NULL)
 		return (vd);
 
-	vd->diag = (VSM_diag_f*)fprintf;
-	vd->priv = stderr;
-
 	vd->vsm_fd = -1;
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
@@ -75,16 +75,35 @@ VSM_New(void)
 
 /*--------------------------------------------------------------------*/
 
-void
-VSM_Diag(struct VSM_data *vd, VSM_diag_f *func, void *priv)
+static int
+vsm_diag(struct VSM_data *vd, const char *fmt, ...)
+{
+	va_list ap;
+
+	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
+	AN(fmt);
+
+	if (vd->diag == NULL)
+		vd->diag = VSB_new_auto();
+	AN(vd->diag);
+	va_start(ap, fmt);
+	VSB_vprintf(vd->diag, fmt, ap);
+	va_end(ap);
+	AZ(VSB_finish(vd->diag));
+	return (-1);
+}
+/*--------------------------------------------------------------------*/
+
+const char *
+VSM_Error(const struct VSM_data *vd)
 {
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	if (func == NULL)
-		vd->diag = (VSM_diag_f*)getpid;
+
+	if (vd->diag == NULL)
+		return (NULL);
 	else
-		vd->diag = func;
-	vd->priv = priv;
+		return (VSB_data(vd->diag));
 }
 
 /*--------------------------------------------------------------------*/
@@ -94,13 +113,12 @@ VSM_n_Arg(struct VSM_data *vd, const char *opt)
 {
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	REPLACE(vd->n_opt, opt);
 	AN(vd->n_opt);
-	if (VIN_N_Arg(vd->n_opt, NULL, NULL, &vd->fname)) {
-		vd->diag(vd->priv, "Invalid instance name: %s\n",
-		    strerror(errno));
-		return (-1);
-	}
+
+	REPLACE(vd->n_opt, opt);
+	if (VIN_N_Arg(vd->n_opt, NULL, NULL, &vd->fname))
+		return (vsm_diag(vd, "Invalid instance name: %s\n",
+		    strerror(errno)));
 	return (1);
 }
 
@@ -111,6 +129,7 @@ VSM_Name(const struct VSM_data *vd)
 {
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
+
 	return (vd->n_opt);
 }
 
@@ -123,15 +142,12 @@ VSM_Delete(struct VSM_data *vd)
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
 
 	VSM_Close(vd);
-
 	free(vd->n_opt);
 	free(vd->fname);
-
 	if (vd->vsc != NULL)
 		VSC_Delete(vd);
 	if (vd->vsl != NULL)
 		VSL_Delete(vd);
-
 	FREE_OBJ(vd);
 }
 
@@ -144,64 +160,59 @@ VSM_Delete(struct VSM_data *vd)
  *
  */
 
-static int
-vsm_open(struct VSM_data *vd, int diag)
+/*--------------------------------------------------------------------*/
+
+int
+VSM_Open(struct VSM_data *vd)
 {
 	int i;
 	struct VSM_head slh;
 	void *v;
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
+
+	AZ(vd->head);
+	if (!vd->n_opt)
+		(void)VSM_n_Arg(vd, "");
+
 	AZ(vd->head);
 	AN(vd->fname);
 
 	vd->vsm_fd = open(vd->fname, O_RDONLY);
-	if (vd->vsm_fd < 0) {
-		if (diag)
-			vd->diag(vd->priv, "Cannot open %s: %s\n",
-			    vd->fname, strerror(errno));
-		return (-1);
-	}
+	if (vd->vsm_fd < 0)
+		return (vsm_diag(vd, "Cannot open %s: %s\n",
+		    vd->fname, strerror(errno)));
 
 	AZ(fstat(vd->vsm_fd, &vd->fstat));
 	if (!S_ISREG(vd->fstat.st_mode)) {
-		if (diag)
-			vd->diag(vd->priv, "%s is not a regular file\n",
-			    vd->fname);
 		AZ(close(vd->vsm_fd));
 		vd->vsm_fd = -1;
-		return (-1);
+		return (vsm_diag(vd, "%s is not a regular file\n",
+		    vd->fname));
 	}
 
 	i = read(vd->vsm_fd, &slh, sizeof slh);
 	if (i != sizeof slh) {
-		if (diag)
-			vd->diag(vd->priv, "Cannot read %s: %s\n",
-			    vd->fname, strerror(errno));
 		AZ(close(vd->vsm_fd));
 		vd->vsm_fd = -1;
-		return (-1);
+		return(vsm_diag(vd, "Cannot read %s: %s\n",
+		    vd->fname, strerror(errno)));
 	}
 
 	if (memcmp(slh.marker, VSM_HEAD_MARKER, sizeof slh.marker) ||
 	    slh.alloc_seq == 0) {
-		if (diag)
-			vd->diag(vd->priv, "Not a VSM file %s\n",
-			    vd->fname);
 		AZ(close(vd->vsm_fd));
 		vd->vsm_fd = -1;
-		return (-1);
+		return (vsm_diag(vd, "Not a VSM file %s\n", vd->fname));
 	}
 
 	v = mmap(NULL, slh.shm_size,
 	    PROT_READ, MAP_SHARED|MAP_HASSEMAPHORE, vd->vsm_fd, 0);
 	if (v == MAP_FAILED) {
-		if (diag)
-			vd->diag(vd->priv, "Cannot mmap %s: %s\n",
-			    vd->fname, strerror(errno));
 		AZ(close(vd->vsm_fd));
 		vd->vsm_fd = -1;
-		return (-1);
+		return (vsm_diag(vd, "Cannot mmap %s: %s\n",
+		    vd->fname, strerror(errno)));
 	}
 	vd->head = v;
 	vd->b = v;
@@ -212,25 +223,12 @@ vsm_open(struct VSM_data *vd, int diag)
 
 /*--------------------------------------------------------------------*/
 
-int
-VSM_Open(struct VSM_data *vd, int diag)
-
-{
-
-	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	AZ(vd->head);
-	if (!vd->n_opt)
-		(void)VSM_n_Arg(vd, "");
-	return (vsm_open(vd, diag));
-}
-
-/*--------------------------------------------------------------------*/
-
 void
 VSM_Close(struct VSM_data *vd)
 {
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
+
 	if (vd->head == NULL)
 		return;
 
@@ -246,42 +244,24 @@ VSM_Close(struct VSM_data *vd)
 /*--------------------------------------------------------------------*/
 
 int
-VSM_ReOpen(struct VSM_data *vd, int diag)
+VSM_Abandonned(const struct VSM_data *vd)
 {
 	struct stat st;
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	AN(vd->head);
-
-	if (vd->head->alloc_seq &&
-	    !stat(vd->fname, &st) &&
-	    st.st_dev == vd->fstat.st_dev &&
-	    st.st_ino == vd->fstat.st_ino)
-		return (0);
-
-	VSM_Close(vd);
-	return (vsm_open(vd, diag));
-}
-
-/*--------------------------------------------------------------------*/
-
-unsigned
-VSM_Seq(const struct VSM_data *vd)
-{
-
-	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	return (vd->head->alloc_seq);
-}
-
-/*--------------------------------------------------------------------*/
 
-struct VSM_head *
-VSM_Head(const struct VSM_data *vd)
-{
+	if (vd->head == NULL)
+		return (1);
 
-	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	AN(vd->head);
-	return(vd->head);
+	if (!vd->head->alloc_seq)
+		return (1);
+	if (!stat(vd->fname, &st))
+		return (1);
+	if (st.st_dev != vd->fstat.st_dev)
+		return (1);
+	if (st.st_ino != vd->fstat.st_ino)
+		return (1);
+	return (0);
 }
 
 /*--------------------------------------------------------------------*/
@@ -291,15 +271,20 @@ VSM__iter0(const struct VSM_data *vd, struct VSM_fantom *vf)
 {
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
+	AN(vf);
+
 	memset(vf, 0, sizeof *vf);
 }
 
+/* XXX: revisit, logic is unclear */
 int
 VSM__itern(const struct VSM_data *vd, struct VSM_fantom *vf)
 {
 	void *p;
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
+	AN(vf);
+
 	if (vd->head->alloc_seq == 0)
 		return (0);	/* abandonned VSM */
 	else if (vf->priv != 0) {
@@ -325,7 +310,7 @@ VSM__itern(const struct VSM_data *vd, struct VSM_fantom *vf)
 	vf->b = (void*)(vf->chunk + 1);
 	vf->e = (char*)vf->b + vf->chunk->len;
 
-	if (vd->priv == 0)
+	if (vf->priv == 0)
 		return (0);	/* abandonned VSM */
 	if (vf->b == vf->e)
 		return (0);	/* freed chunk */
@@ -342,14 +327,13 @@ VSM_StillValid(const struct VSM_data *vd, struct VSM_fantom *vf)
 	struct VSM_fantom f2;
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	if (vf == NULL)
-		return (vd->head->alloc_seq ? 1 : 0);
+	AN(vf);
+	if (!vd->head->alloc_seq)
+		return (0);
 	if (vf->priv == vd->head->alloc_seq)
 		return (1);
 	VSM_FOREACH_SAFE(&f2, vd) {
-		if (f2.chunk == vf->chunk &&
-		   f2.b == vf->b &&
-		   f2.e == vf->e) {
+		if (f2.chunk == vf->chunk && f2.b == vf->b && f2.e == vf->e) {
 			vf->priv = vd->head->alloc_seq;
 			return (2);
 		}
@@ -375,45 +359,3 @@ VSM_Get(const struct VSM_data *vd, struct VSM_fantom *vf,
 	memset(vf, 0, sizeof *vf);
 	return (0);
 }
-
-/*--------------------------------------------------------------------*/
-
-void *
-VSM_Find_Chunk(const struct VSM_data *vd, const char *class,
-    const char *type, const char *ident, unsigned *lenp)
-{
-	struct VSM_fantom vf;
-
-	if (VSM_Get(vd, &vf, class, type, ident)) {
-		if (lenp != NULL)
-			*lenp = (char*)vf.e - (char*)vf.b;
-		return (vf.chunk);
-	}
-	if (lenp != NULL)
-		*lenp = 0;
-	return (NULL);
-}
-
-/*--------------------------------------------------------------------*/
-
-struct VSM_chunk *
-VSM_iter0(struct VSM_data *vd)
-{
-
-	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	VSM__iter0(vd, &vd->compat_vf);
-	if (VSM__itern(vd, &vd->compat_vf))
-		return(vd->compat_vf.chunk);
-	return (NULL);
-}
-
-void
-VSM_itern(struct VSM_data *vd, struct VSM_chunk **pp)
-{
-
-	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	if (VSM__itern(vd, &vd->compat_vf))
-		*pp = vd->compat_vf.chunk;
-	else
-		*pp = NULL;
-}
diff --git a/lib/libvarnishapi/vsm_api.h b/lib/libvarnishapi/vsm_api.h
index 3355ee4..6690ba6 100644
--- a/lib/libvarnishapi/vsm_api.h
+++ b/lib/libvarnishapi/vsm_api.h
@@ -29,13 +29,13 @@
  */
 
 struct vsc;
+struct vsb;
 
 struct VSM_data {
 	unsigned		magic;
 #define VSM_MAGIC		0x6e3bd69b
 
-	VSM_diag_f		*diag;
-	void			*priv;
+	struct vsb		*diag;
 
 	char			*n_opt;
 	char			*fname;



From phk at varnish-cache.org  Wed Nov 23 21:11:09 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 23 Nov 2011 22:11:09 +0100
Subject: [master] 5a7a850 New VSL api
Message-ID: <E1RTK61-0007ed-3q@project.varnish-software.com>

commit 5a7a850f2d42aea69a782cff92d50aad88cfce43
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 22 10:35:34 2011 +0000

    New VSL api

diff --git a/include/vapi/vsl.h b/include/vapi/vsl.h
index fb02051..35e09dc 100644
--- a/include/vapi/vsl.h
+++ b/include/vapi/vsl.h
@@ -26,6 +26,17 @@
  * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
  * SUCH DAMAGE.
  *
+ * This is the public API for the VSL access.
+ *
+ * VSL is a "subclass" of VSM.
+ *
+ * VSL can either read from VSM or from a file.
+ *
+ * When reading from a file, the filename is passed in with:
+ *	VSL_Arg(vd, "r", "/some/file");
+ * and once VSL_Dispatch()/VSL_NextSLT() will indicate EOF by returning -2.
+ * Another file can then be opened with VSL_Arg() and processed.
+ * 
  */
 
 #ifndef VAPI_VSL_H_INCLUDED
@@ -39,21 +50,6 @@ struct VSM_data;
  * VSL level access functions
  */
 
-void VSL_Setup(struct VSM_data *vd);
-	/*
-	 * Setup vd for use with VSL functions.
-	 * Must be called once before any other VSL function is called.
-	 */
-
-int VSL_Open(struct VSM_data *vd, int diag);
-	/*
-	 * Attempt to open the VSM (unless -r given)
-	 * If diag is non-zero, diagnostics are emitted.
-	 * Returns:
-	 *	0 on success
-	 *	!= 0 on failure
-	 */
-
 #define VSL_ARGS	"bCcdI:i:k:n:r:s:X:x:m:"
 #define VSL_b_USAGE	"[-b]"
 #define VSL_c_USAGE	"[-c]"
@@ -68,6 +64,7 @@ int VSL_Open(struct VSM_data *vd, int diag);
 #define VSL_s_USAGE	"[-s skip]"
 #define VSL_x_USAGE	"[-x tag]"
 #define VSL_X_USAGE	"[-X regexp]"
+
 #define VSL_USAGE	"[-bCcd] "		\
 			VSL_i_USAGE " "		\
 			VSL_I_USAGE " "		\
@@ -83,24 +80,79 @@ int VSL_Arg(struct VSM_data *vd, int arg, const char *opt);
 	/*
 	 * Handle standard log-presenter arguments
 	 * Return:
-	 *	-1 error
+	 *	-1 error, VSM_Error() returns diagnostic string
 	 *	 0 not handled
 	 *	 1 Handled.
 	 */
 
 typedef int VSL_handler_f(void *priv, enum VSL_tag_e tag, unsigned fd,
     unsigned len, unsigned spec, const char *ptr, uint64_t bitmap);
+	/*
+	 * This is the call-back function you must provide.
+	 *	priv is whatever you asked for it to be.
+	 *	tag is the SLT_mumble tag
+	 *	fd is the filedescriptor associated with this record
+	 *	len is the length of the data at ptr
+	 *	spec are the VSL_S_* flags
+	 *	ptr points to the data, beware of non-printables.
+	 *	bitmap is XXX ???
+	 */
 
 #define VSL_S_CLIENT	(1 << 0)
 #define VSL_S_BACKEND	(1 << 1)
+
 VSL_handler_f VSL_H_Print;
-struct VSM_data;
-void VSL_Select(const struct VSM_data *vd, unsigned tag);
-void VSL_NonBlocking(const struct VSM_data *vd, int nb);
+	/*
+	 * This call-back function will printf() the record to the FILE *
+	 * specified in priv.
+	 */
+
+void VSL_Select(struct VSM_data *vd, enum VSL_tag_e tag);
+	/*
+	 * This adds tags which shall always be selected, similar to using
+	 * the '-i' option.
+	 * VSL_Select()/-i takes precedence over all other filtering.
+	 */
+
 int VSL_Dispatch(struct VSM_data *vd, VSL_handler_f *func, void *priv);
-int VSL_NextLog(const struct VSM_data *lh, uint32_t **pp, uint64_t *bitmap);
-int VSL_Matched(const struct VSM_data *vd, uint64_t bitmap);
+	/*
+	 * Call func(priv, ...) for all filtered VSL records.
+	 *
+	 * Return values:
+	 *	!=0:	Non-zero return value from func()
+	 *	0:	no VSL records.
+	 *	-1:	VSL chunk was abandonned.
+	 *	-2:	End of file (-r) / -k arg exhausted / "done"
+	 */
+
+int VSL_NextSLT(struct VSM_data *lh, uint32_t **pp, uint64_t *bitmap);
+	/*
+	 * Return raw pointer to next filtered VSL record.
+	 *
+	 * Return values:
+	 *	1:	Valid VSL record at *pp
+	 *	0:	no VSL records
+	 *	-1:	VSL cunkwas abandonned
+	 *	-2:	End of file (-r) / -k arg exhausted / "done"
+	 */
+
+int VSL_Matched(struct VSM_data *vd, uint64_t bitmap);
+	/*
+	 */
+
 int VSL_Name2Tag(const char *name, int l);
+	/*
+	 * Convert string to tag number (= enum VSL_tag_e)
+	 *
+	 * Return values:
+	 *	>=0:	Tag number
+	 *	-1:	No tag matches
+	 *	-2:	Multiple tags match substring
+	 */
+
 extern const char *VSL_tags[256];
+	/*
+	 * Tag to string array.  Contains NULL for invalid tags.
+	 */
 
 #endif /* VAPI_VSL_H_INCLUDED */
diff --git a/include/vapi/vsm.h b/include/vapi/vsm.h
index 277a670..9635e8f 100644
--- a/include/vapi/vsm.h
+++ b/include/vapi/vsm.h
@@ -83,7 +83,7 @@ int VSM_n_Arg(struct VSM_data *vd, const char *n_arg);
 	 * Can also be, and normally is done through VSC_Arg()/VSL_Arg().
 	 * Returns:
 	 *	 1 on success
-	 *	 <0 on failure, use VSM_Error() to get diagnostics.
+	 *	 <0 on failure, VSM_Error() returns diagnostic string
 	 */
 
 const char *VSM_Name(const struct VSM_data *vd);
@@ -97,7 +97,7 @@ int VSM_Open(struct VSM_data *vd);
 	 * If diag is non-zero, diagnostics are emitted.
 	 * Returns:
 	 *	0 on success
-	 *	<0 on failure, use VSM_Error() to get diagnostics.
+	 *	<0 on failure, VSM_Error() returns diagnostic string
 	 */
 
 int VSM_Abandonned(const struct VSM_data *vd);
diff --git a/lib/libvarnishapi/vsl.c b/lib/libvarnishapi/vsl.c
index e1ace33..d45e1c3 100644
--- a/lib/libvarnishapi/vsl.c
+++ b/lib/libvarnishapi/vsl.c
@@ -51,37 +51,40 @@
 #include "vsl_api.h"
 #include "vsm_api.h"
 
-static void VSL_Close(struct VSM_data *vd);
-
 /*--------------------------------------------------------------------*/
 
 const char *VSL_tags[256] = {
-#define SLTM(foo)       [SLT_##foo] = #foo,
-#include "tbl/vsl_tags.h"
-#undef SLTM
+#  define SLTM(foo)       [SLT_##foo] = #foo,
+#  include "tbl/vsl_tags.h"
+#  undef SLTM
 };
 
 /*--------------------------------------------------------------------*/
 
-void
-VSL_Setup(struct VSM_data *vd)
+struct vsl *
+vsl_Setup(struct VSM_data *vd)
 {
 	struct vsl *vsl;
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	AZ(vd->vsl);
-	ALLOC_OBJ(vd->vsl, VSL_MAGIC);
-	AN(vd->vsl);
-	vsl = vd->vsl;
-	vsl->regflags = 0;
-	vsl->vbm_supress = vbit_init(256);
-	vsl->vbm_select = vbit_init(256);
-	vsl->r_fd = -1;
-	vsl->num_matchers = 0;
-	VTAILQ_INIT(&vsl->matchers);
+	if (vd->vsl == NULL) {
+		ALLOC_OBJ(vd->vsl, VSL_MAGIC);
+		AN(vd->vsl);
+		vsl = vd->vsl;
+		vsl->regflags = 0;
+		vsl->vbm_supress = vbit_init(256);
+		vsl->vbm_select = vbit_init(256);
+		vsl->r_fd = -1;
+		vsl->num_matchers = 0;
+		VTAILQ_INIT(&vsl->matchers);
+	}
+	CHECK_OBJ_NOTNULL(vd->vsl, VSL_MAGIC);
+	return (vd->vsl);
 }
 
-/*--------------------------------------------------------------------*/
+/*--------------------------------------------------------------------
+ * Called from VSM_Delete()
+ */
 
 void
 VSL_Delete(struct VSM_data *vd)
@@ -93,41 +96,66 @@ VSL_Delete(struct VSM_data *vd)
 	vd->vsl = NULL;
 	CHECK_OBJ_NOTNULL(vsl, VSL_MAGIC);
 
+	if (vsl->r_fd > STDIN_FILENO)
+		(void)close(vsl->r_fd);
 	vbit_destroy(vsl->vbm_supress);
 	vbit_destroy(vsl->vbm_select);
 	free(vsl->rbuf);
-
 	FREE_OBJ(vsl);
 }
 
 /*--------------------------------------------------------------------*/
 
 void
-VSL_Select(const struct VSM_data *vd, unsigned tag)
+VSL_Select(struct VSM_data *vd, enum VSL_tag_e tag)
 {
-	struct vsl *vsl;
+	struct vsl *vsl = vsl_Setup(vd);
 
-	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	vsl = vd->vsl;
-	CHECK_OBJ_NOTNULL(vsl, VSL_MAGIC);
-	vbit_set(vsl->vbm_select, tag);
+	vbit_set(vsl->vbm_select, (int)tag);
 }
 
+/*--------------------------------------------------------------------
+ */
 
-/*--------------------------------------------------------------------*/
+static int
+vsl_open(struct VSM_data *vd)
+{
+	struct vsl *vsl = vsl_Setup(vd);
+	int i;
 
-void
-VSL_NonBlocking(const struct VSM_data *vd, int nb)
+	assert(vsl->r_fd < 0);
+	i = VSM_Open(vd);
+	if (i)
+		return (i);
+	if (!VSM_Get(vd, &vsl->vf, VSL_CLASS, NULL, NULL)) {
+		VSM_Close(vd);
+		return (vsm_diag(vd, "No VSL chunk found "
+		    " (child not started ?)\n"));
+	}
+	vsl->log_start = vsl->vf.b;
+	vsl->log_end = vsl->vf.e;
+	vsl->log_ptr = vsl->log_start + 1;
+	if (!vsl->d_opt) {
+		while (*vsl->log_ptr != VSL_ENDMARKER)
+			vsl->log_ptr = VSL_NEXT(vsl->log_ptr);
+	}
+	return (0);
+}
+
+/*--------------------------------------------------------------------
+ */
+
+static void
+vsl_close(struct VSM_data *vd)
 {
-	struct vsl *vsl;
+	struct vsl *vsl = vsl_Setup(vd);
 
-	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	vsl = vd->vsl;
-	CHECK_OBJ_NOTNULL(vsl, VSL_MAGIC);
-	if (nb)
-		vsl->flags |= F_NON_BLOCKING;
-	else
-		vsl->flags &= ~F_NON_BLOCKING;
+	assert(vsl->r_fd < 0);
+	VSM_Close(vd);
+	memset(&vsl->vf, 0, sizeof vsl->vf);
+	vsl->log_start = NULL;
+	vsl->log_end = NULL;
+	vsl->log_ptr = NULL;
 }
 
 /*--------------------------------------------------------------------
@@ -140,20 +168,19 @@ VSL_NonBlocking(const struct VSM_data *vd, int nb)
  */
 
 static int
-vsl_nextlog(struct vsl *vsl, uint32_t **pp)
+vsl_nextslt(struct VSM_data *vd, uint32_t **pp)
 {
+	struct vsl *vsl = vsl_Setup(vd);
 	unsigned l;
 	uint32_t t;
 	int i;
 
-	CHECK_OBJ_NOTNULL(vsl, VSL_MAGIC);
-
 	*pp = NULL;
 	if (vsl->r_fd != -1) {
 		assert(vsl->rbuflen >= 8);
 		i = read(vsl->r_fd, vsl->rbuf, 8);
 		if (i == 0)
-			return (-1);
+			return (-2);
 		if (i != 8)
 			return (-1);
 		l = 2 + VSL_WORDS(VSL_LEN(vsl->rbuf));
@@ -169,9 +196,11 @@ vsl_nextlog(struct vsl *vsl, uint32_t **pp)
 		*pp = vsl->rbuf;
 		return (1);
 	}
+
+	if (vsl->log_ptr == NULL && vsl_open(vd))
+		return (0);
+
 	while (1) {
-		if (vsl->log_ptr == NULL)
-			return (0);
 		assert(vsl->log_ptr >= vsl->log_start + 1);
 		assert(vsl->log_ptr < vsl->log_end);
 		t = *vsl->log_ptr;
@@ -209,60 +238,65 @@ vsl_nextlog(struct vsl *vsl, uint32_t **pp)
 }
 
 int
-VSL_NextLog(const struct VSM_data *vd, uint32_t **pp, uint64_t *bits)
+VSL_NextSLT(struct VSM_data *vd, uint32_t **pp, uint64_t *bits)
 {
-	struct vsl *vsl;
+	struct vsl *vsl = vsl_Setup(vd);
 	uint32_t *p;
 	unsigned char t;
 	int i;
+	struct vsl_re_match *vrm;
+	int j;
 
-	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	vsl = vd->vsl;
-	CHECK_OBJ_NOTNULL(vsl, VSL_MAGIC);
+	if (bits != NULL)
+		*bits = 0;
 
 	while (1) {
-		i = vsl_nextlog(vsl, &p);
-		if (i <= 0)
+		i = vsl_nextslt(vd, &p);
+		if (i < 0)
+			return (i);
+		if (i == 0 && (vsl->d_opt || vsl->r_fd >= 0))
+			return (i);
+		if (i == 0 && !VSM_StillValid(vd, &vsl->vf)) {
+			vsl_close(vd);
 			return (i);
-		t = VSL_TAG(p);
-		if (vsl->skip) {
-			--vsl->skip;
-			continue;
-		} else if (vsl->keep) {
-			if (--vsl->keep == 0)
-				return (-1);
 		}
 
+		t = VSL_TAG(p);
 		if (vbit_test(vsl->vbm_select, t)) {
-			*pp = p;
-			return (1);
-		}
-		if (vbit_test(vsl->vbm_supress, t))
+			/* nothing */
+		} else if (vbit_test(vsl->vbm_supress, t)) {
 			continue;
-		if (vsl->b_opt && !VSL_BACKEND(p))
+		} else if (vsl->b_opt && !VSL_BACKEND(p)) {
 			continue;
-		if (vsl->c_opt && !VSL_CLIENT(p))
+		} else if (vsl->c_opt && !VSL_CLIENT(p)) {
 			continue;
-		if (vsl->regincl != NULL) {
+		} else if (vsl->regincl != NULL) {
 			i = VRE_exec(vsl->regincl, VSL_DATA(p), VSL_LEN(p),
 			    0, 0, NULL, 0, NULL);
 			if (i == VRE_ERROR_NOMATCH)
 				continue;
-		}
-		if (vsl->regexcl != NULL) {
+		} else if (vsl->regexcl != NULL) {
 			i = VRE_exec(vsl->regexcl, VSL_DATA(p), VSL_LEN(p),
 			    0, 0, NULL, 0, NULL);
 			if (i != VRE_ERROR_NOMATCH)
 				continue;
 		}
+
+		if (vsl->skip) {
+			--vsl->skip;
+			continue;
+		} else if (vsl->keep) {
+			if (--vsl->keep == 0)
+				return (-2);
+		}
+
 		if (bits != NULL) {
-			struct vsl_re_match *vrm;
-			int j = 0;
+			j = 0;
 			VTAILQ_FOREACH(vrm, &vsl->matchers, next) {
 				if (vrm->tag == t) {
 					i = VRE_exec(vrm->re, VSL_DATA(p),
 					    VSL_LEN(p), 0, 0, NULL, 0, NULL);
-					if (i >= 0)
+					if (i >= 0) 	/* XXX ?? */
 						*bits |= (uintmax_t)1 << j;
 				}
 				j++;
@@ -278,41 +312,13 @@ VSL_NextLog(const struct VSM_data *vd, uint32_t **pp, uint64_t *bits)
 int
 VSL_Dispatch(struct VSM_data *vd, VSL_handler_f *func, void *priv)
 {
-	struct vsl *vsl;
 	int i;
 	unsigned u, l, s;
 	uint32_t *p;
 	uint64_t bitmap;
-	int tmo;
-
-	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	vsl = vd->vsl;
-	CHECK_OBJ_NOTNULL(vsl, VSL_MAGIC);
 
-	tmo = 0;
 	while (1) {
-		bitmap = 0;
-		i = VSL_NextLog(vd, &p, &bitmap);
-		if (i == 0) {
-			if (vsl->r_fd != -1)
-				return(0);
-			if (vsl->flags & F_NON_BLOCKING)
-				return (0);
-			if (VSM_StillValid(vd, &vsl->vf) != 1) {
-				VSL_Close(vd);
-				if (VSL_Open(vd, 0))
-					return (-1);
-				AN(vsl->log_ptr);
-				assert(vsl->log_ptr >= vsl->log_start + 1);
-				assert(vsl->log_ptr < vsl->log_end);
-				continue;
-			}
-			tmo += SLEEP_USEC;
-			if (tmo > TIMEOUT_USEC)
-				return (0);
-			(void)usleep(SLEEP_USEC);
-			continue;
-		}
+		i = VSL_NextSLT(vd, &p, &bitmap);
 		if (i <= 0)
 			return (i);
 		u = VSL_ID(p);
@@ -322,9 +328,10 @@ VSL_Dispatch(struct VSM_data *vd, VSL_handler_f *func, void *priv)
 			s |= VSL_S_CLIENT;
 		if (VSL_BACKEND(p))
 			s |= VSL_S_BACKEND;
-		if (func(priv, (enum VSL_tag_e)VSL_TAG(p),
-		    u, l, s, VSL_DATA(p), bitmap))
-			return (1);
+		i = func(priv, (enum VSL_tag_e)VSL_TAG(p),
+		    u, l, s, VSL_DATA(p), bitmap);
+		if (i)
+			return (i);
 	}
 }
 
@@ -362,64 +369,14 @@ VSL_H_Print(void *priv, enum VSL_tag_e tag, unsigned fd, unsigned len,
 
 /*--------------------------------------------------------------------*/
 
-static void
-VSL_Close(struct VSM_data *vd)
-{
-	struct vsl *vsl;
-
-	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	vsl = vd->vsl;
-	CHECK_OBJ_NOTNULL(vsl, VSL_MAGIC);
-
-	VSM_Close(vd);
-	vsl->log_start = NULL;
-	vsl->log_end = NULL;
-	vsl->log_ptr = NULL;
-}
-
-/*--------------------------------------------------------------------*/
-
 int
-VSL_Open(struct VSM_data *vd, int diag)
+VSL_Matched(struct VSM_data *vd, uint64_t bitmap)
 {
-	struct vsl *vsl;
-	int i;
-
-	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	vsl = vd->vsl;
-	CHECK_OBJ_NOTNULL(vsl, VSL_MAGIC);
-
-	if (vsl->r_fd == -1) {
-		i = VSM_Open(vd, diag);
-		if (i)
-			return (i);
-		if (!VSM_Get(vd, &vsl->vf, VSL_CLASS, NULL, NULL)) {
-			VSM_Close(vd);
-			if (diag)
-				vd->diag(vd->priv,
-				    "No VSL chunk found "
-				    " (child not started ?)\n");
-			return (1);
-		}
-		vsl->log_start = vsl->vf.b;
-		vsl->log_end = vsl->vf.e;
-		vsl->log_ptr = vsl->log_start + 1;
-		if (!vsl->d_opt) {
-			while (*vsl->log_ptr != VSL_ENDMARKER)
-				vsl->log_ptr = VSL_NEXT(vsl->log_ptr);
-		}
-	}
-	return (0);
-}
+	struct vsl *vsl = vsl_Setup(vd);
 
-/*--------------------------------------------------------------------*/
-
-int
-VSL_Matched(const struct VSM_data *vd, uint64_t bitmap)
-{
-	if (vd->vsl->num_matchers > 0) {
+	if (vsl->num_matchers > 0) {
 		uint64_t t;
-		t = vd->vsl->num_matchers | (vd->vsl->num_matchers - 1);
+		t = vsl->num_matchers | (vsl->num_matchers - 1);
 		return (bitmap == t);
 	}
 	return (1);
diff --git a/lib/libvarnishapi/vsl_api.h b/lib/libvarnishapi/vsl_api.h
index 013d1ae..0b38539 100644
--- a/lib/libvarnishapi/vsl_api.h
+++ b/lib/libvarnishapi/vsl_api.h
@@ -67,7 +67,6 @@ struct vsl {
 
 	unsigned		flags;
 #define F_SEEN_IX		(1 << 0)
-#define F_NON_BLOCKING		(1 << 1)
 
 	/*
 	 * Bit map of programatically selected tags, that cannot be suppressed.
@@ -89,3 +88,5 @@ struct vsl {
 	unsigned long		keep;
 };
 
+struct vsl *vsl_Setup(struct VSM_data *vd);
+
diff --git a/lib/libvarnishapi/vsl_arg.c b/lib/libvarnishapi/vsl_arg.c
index b855628..100d75a 100644
--- a/lib/libvarnishapi/vsl_arg.c
+++ b/lib/libvarnishapi/vsl_arg.c
@@ -82,21 +82,23 @@ VSL_Name2Tag(const char *name, int l)
 /*--------------------------------------------------------------------*/
 
 static int
-vsl_r_arg(const struct VSM_data *vd, const char *opt)
+vsl_r_arg(struct VSM_data *vd, const char *opt)
 {
+	struct vsl *vsl = vsl_Setup(vd);
 
-	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
+	if (vsl->r_fd > STDIN_FILENO)
+		(void)close(vsl->r_fd);
 	if (!strcmp(opt, "-"))
-		vd->vsl->r_fd = STDIN_FILENO;
+		vsl->r_fd = STDIN_FILENO;
 	else
-		vd->vsl->r_fd = open(opt, O_RDONLY);
-	if (vd->vsl->r_fd < 0) {
-		perror(opt);
-		return (-1);
-	} else if (vd->vsl->rbuflen == 0) {
-		vd->vsl->rbuf = malloc(1024);
-		AN(vd->vsl->rbuf);
-		vd->vsl->rbuflen = 1024;
+		vsl->r_fd = open(opt, O_RDONLY);
+	if (vsl->r_fd < 0)
+		return (vsm_diag(vd,
+		    "Could not open %s: %s", opt, strerror(errno)));
+	if (vsl->rbuflen == 0) {
+		vsl->rbuflen = BUFSIZ;
+		vsl->rbuf = malloc(vsl->rbuflen);
+		AN(vsl->rbuf);
 	}
 	return (1);
 }
@@ -104,43 +106,41 @@ vsl_r_arg(const struct VSM_data *vd, const char *opt)
 /*--------------------------------------------------------------------*/
 
 static int
-vsl_IX_arg(const struct VSM_data *vd, const char *opt, int arg)
+vsl_IX_arg(struct VSM_data *vd, const char *opt, int arg)
 {
+	struct vsl *vsl = vsl_Setup(vd);
 	vre_t **rp;
 	const char *error;
 	int erroroffset;
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
 	if (arg == 'I')
-		rp = &vd->vsl->regincl;
+		rp = &vsl->regincl;
 	else
-		rp = &vd->vsl->regexcl;
-	if (*rp != NULL) {
-		fprintf(stderr, "Option %c can only be given once", arg);
-		return (-1);
-	}
-	*rp = VRE_compile(opt, vd->vsl->regflags, &error, &erroroffset);
-	if (*rp == NULL) {
-		fprintf(stderr, "Illegal regex: %s\n", error);
-		return (-1);
-	}
+		rp = &vsl->regexcl;
+	if (*rp != NULL)
+		return (vsm_diag(vd, "Option %c can only be given once", arg));
+	*rp = VRE_compile(opt, vsl->regflags, &error, &erroroffset);
+	if (*rp == NULL)
+		return (vsm_diag(vd, "Illegal regex: %s\n", error));
 	return (1);
 }
 
 /*--------------------------------------------------------------------*/
 
 static int
-vsl_ix_arg(const struct VSM_data *vd, const char *opt, int arg)
+vsl_ix_arg(struct VSM_data *vd, const char *opt, int arg)
 {
+	struct vsl *vsl = vsl_Setup(vd);
 	int i, l;
 	const char *b, *e;
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
 	/* If first option is 'i', set all bits for supression */
-	if (arg == 'i' && !(vd->vsl->flags & F_SEEN_IX))
+	if (arg == 'i' && !(vsl->flags & F_SEEN_IX))
 		for (i = 0; i < 256; i++)
-			vbit_set(vd->vsl->vbm_supress, i);
-	vd->vsl->flags |= F_SEEN_IX;
+			vbit_set(vsl->vbm_supress, i);
+	vsl->flags |= F_SEEN_IX;
 
 	for (b = opt; *b; b = e) {
 		while (isspace(*b))
@@ -156,17 +156,15 @@ vsl_ix_arg(const struct VSM_data *vd, const char *opt, int arg)
 		i = VSL_Name2Tag(b, l);
 		if (i >= 0) {
 			if (arg == 'x')
-				vbit_set(vd->vsl->vbm_supress, i);
+				vbit_set(vsl->vbm_supress, i);
 			else
-				vbit_clr(vd->vsl->vbm_supress, i);
+				vbit_clr(vsl->vbm_supress, i);
 		} else if (i == -2) {
-			fprintf(stderr,
-			    "\"%*.*s\" matches multiple tags\n", l, l, b);
-			return (-1);
+			return (vsm_diag(vd, 
+			    "\"%*.*s\" matches multiple tags\n", l, l, b));
 		} else {
-			fprintf(stderr,
-			    "Could not match \"%*.*s\" to any tag\n", l, l, b);
-			return (-1);
+			return (vsm_diag(vd, 
+			    "Could not match \"%*.*s\" to any tag\n", l, l, b));
 		}
 	}
 	return (1);
@@ -176,8 +174,9 @@ vsl_ix_arg(const struct VSM_data *vd, const char *opt, int arg)
 
 
 static int
-vsl_m_arg(const struct VSM_data *vd, const char *opt)
+vsl_m_arg(struct VSM_data *vd, const char *opt)
 {
+	struct vsl *vsl = vsl_Setup(vd);
 	struct vsl_re_match *m;
 	const char *error;
 	char *o, *regex;
@@ -185,10 +184,9 @@ vsl_m_arg(const struct VSM_data *vd, const char *opt)
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
 
-	if (!strchr(opt, ':')) {
-		fprintf(stderr, "No : found in -m option %s\n", opt);
-		return (-1);
-	}
+	if (!strchr(opt, ':'))
+		return (vsm_diag(vd,
+		    "No : found in -m option %s\n", opt));
 
 	o = strdup(opt);
 	AN(o);
@@ -200,21 +198,21 @@ vsl_m_arg(const struct VSM_data *vd, const char *opt)
 	AN(m);
 	m->tag = VSL_Name2Tag(o, -1);
 	if (m->tag < 0) {
-		fprintf(stderr, "Illegal tag %s specified\n", o);
+		(void)vsm_diag(vd, "Illegal tag %s specified\n", o);
 		free(o);
 		FREE_OBJ(m);
 		return (-1);
 	}
 	/* Get tag, regex */
-	m->re = VRE_compile(regex, vd->vsl->regflags, &error, &erroroffset);
+	m->re = VRE_compile(regex, vsl->regflags, &error, &erroroffset);
 	if (m->re == NULL) {
-		fprintf(stderr, "Illegal regex: %s\n", error);
+		(void)vsm_diag(vd, "Illegal regex: %s\n", error);
 		free(o);
 		FREE_OBJ(m);
 		return (-1);
 	}
-	vd->vsl->num_matchers++;
-	VTAILQ_INSERT_TAIL(&vd->vsl->matchers, m, next);
+	vsl->num_matchers++;
+	VTAILQ_INSERT_TAIL(&vsl->matchers, m, next);
 	free(o);
 	return (1);
 }
@@ -222,40 +220,34 @@ vsl_m_arg(const struct VSM_data *vd, const char *opt)
 /*--------------------------------------------------------------------*/
 
 static int
-vsl_s_arg(const struct VSM_data *vd, const char *opt)
+vsl_s_arg(struct VSM_data *vd, const char *opt)
 {
+	struct vsl *vsl = vsl_Setup(vd);
 	char *end;
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	if (*opt == '\0') {
-		fprintf(stderr, "number required for -s\n");
-		return (-1);
-	}
-	vd->vsl->skip = strtoul(opt, &end, 10);
-	if (*end != '\0') {
-		fprintf(stderr, "invalid number for -s\n");
-		return (-1);
-	}
+	if (*opt == '\0')
+		return (vsm_diag(vd, "number required for -s\n"));
+	vsl->skip = strtoul(opt, &end, 10);
+	if (*end != '\0')
+		return (vsm_diag(vd, "invalid number for -k\n"));
 	return (1);
 }
 
 /*--------------------------------------------------------------------*/
 
 static int
-vsl_k_arg(const struct VSM_data *vd, const char *opt)
+vsl_k_arg(struct VSM_data *vd, const char *opt)
 {
+	struct vsl *vsl = vsl_Setup(vd);
 	char *end;
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	if (*opt == '\0') {
-		fprintf(stderr, "number required for -k\n");
-		return (-1);
-	}
-	vd->vsl->keep = strtoul(opt, &end, 10);
-	if (*end != '\0') {
-		fprintf(stderr, "invalid number for -k\n");
-		return (-1);
-	}
+	if (*opt == '\0') 
+		return (vsm_diag(vd, "number required for -k\n"));
+	vsl->keep = strtoul(opt, &end, 10);
+	if (*end != '\0') 
+		return (vsm_diag(vd, "invalid number for -k\n"));
 	return (1);
 }
 
@@ -264,14 +256,13 @@ vsl_k_arg(const struct VSM_data *vd, const char *opt)
 int
 VSL_Arg(struct VSM_data *vd, int arg, const char *opt)
 {
+	struct vsl *vsl = vsl_Setup(vd);
 
-	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
 	switch (arg) {
-	case 'b': vd->vsl->b_opt = !vd->vsl->b_opt; return (1);
-	case 'c': vd->vsl->c_opt = !vd->vsl->c_opt; return (1);
+	case 'b': vsl->b_opt = !vsl->b_opt; return (1);
+	case 'c': vsl->c_opt = !vsl->c_opt; return (1);
 	case 'd':
-		vd->vsl->d_opt = !vd->vsl->d_opt;
-		vd->vsl->flags |= F_NON_BLOCKING;
+		vsl->d_opt = !vsl->d_opt;
 		return (1);
 	case 'i': case 'x': return (vsl_ix_arg(vd, opt, arg));
 	case 'k': return (vsl_k_arg(vd, opt));
@@ -280,7 +271,7 @@ VSL_Arg(struct VSM_data *vd, int arg, const char *opt)
 	case 's': return (vsl_s_arg(vd, opt));
 	case 'I': case 'X': return (vsl_IX_arg(vd, opt, arg));
 	case 'm': return (vsl_m_arg(vd, opt));
-	case 'C': vd->vsl->regflags = VRE_CASELESS; return (1);
+	case 'C': vsl->regflags = VRE_CASELESS; return (1);
 	default:
 		return (0);
 	}
diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c
index d136c98..1d762e5 100644
--- a/lib/libvarnishapi/vsm.c
+++ b/lib/libvarnishapi/vsm.c
@@ -75,7 +75,7 @@ VSM_New(void)
 
 /*--------------------------------------------------------------------*/
 
-static int
+int
 vsm_diag(struct VSM_data *vd, const char *fmt, ...)
 {
 	va_list ap;
diff --git a/lib/libvarnishapi/vsm_api.h b/lib/libvarnishapi/vsm_api.h
index 6690ba6..79be982 100644
--- a/lib/libvarnishapi/vsm_api.h
+++ b/lib/libvarnishapi/vsm_api.h
@@ -47,14 +47,10 @@ struct VSM_data {
 	char			*b;
 	char			*e;
 
-	/* Stuff for backwards compat */
-	struct VSM_fantom	compat_vf;
-
-	/* Stuff relating the stats fields start here */
-
 	struct vsc		*vsc;
 	struct vsl		*vsl;
 };
 
+int vsm_diag(struct VSM_data *vd, const char *fmt, ...);
 void VSC_Delete(struct VSM_data *vd);
 void VSL_Delete(struct VSM_data *vd);



From phk at varnish-cache.org  Wed Nov 23 21:11:11 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 23 Nov 2011 22:11:11 +0100
Subject: [master] 6cc6f23 Merge branch 'master' of
	ssh://git.varnish-cache.org/git/varnish-cache
Message-ID: <E1RTK63-0007eq-Ms@project.varnish-software.com>

commit 6cc6f23fe4294dd0e33841a1dafbbe92cecf9874
Merge: 5a7a850 aed74d6
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Nov 23 08:27:05 2011 +0000

    Merge branch 'master' of ssh://git.varnish-cache.org/git/varnish-cache



From phk at varnish-cache.org  Wed Nov 23 21:11:13 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 23 Nov 2011 22:11:13 +0100
Subject: [master] 1f3fbd8 Make VSC/varnishstat sorta-work again.
Message-ID: <E1RTK65-0007fb-VP@project.varnish-software.com>

commit 1f3fbd8eb67062fe211e731cccaf7251b74d2840
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Nov 23 08:57:48 2011 +0000

    Make VSC/varnishstat sorta-work again.

diff --git a/bin/varnishstat/varnishstat.c b/bin/varnishstat/varnishstat.c
index 2fe7ffb..5d9226f 100644
--- a/bin/varnishstat/varnishstat.c
+++ b/bin/varnishstat/varnishstat.c
@@ -52,7 +52,9 @@ do_xml_cb(void *priv, const struct VSC_point * const pt)
 	uint64_t val;
 
 	(void)priv;
-	assert(!strcmp(pt->fmt, "uint64_t"));
+	if (pt == NULL)
+		return (0);
+	assert(!strcmp(pt->desc->fmt, "uint64_t"));
 	val = *(const volatile uint64_t*)pt->ptr;
 
 	printf("\t<stat>\n");
@@ -60,10 +62,10 @@ do_xml_cb(void *priv, const struct VSC_point * const pt)
 		printf("\t\t<type>%s</type>\n", pt->class);
 	if (strcmp(pt->ident, ""))
 		printf("\t\t<ident>%s</ident>\n", pt->ident);
-	printf("\t\t<name>%s</name>\n", pt->name);
+	printf("\t\t<name>%s</name>\n", pt->desc->name);
 	printf("\t\t<value>%ju</value>\n", val);
-	printf("\t\t<flag>%c</flag>\n", pt->flag);
-	printf("\t\t<description>%s</description>\n", pt->desc);
+	printf("\t\t<flag>%c</flag>\n", pt->desc->flag);
+	printf("\t\t<description>%s</description>\n", pt->desc->sdesc);
 	printf("\t</stat>\n");
 	return (0);
 }
@@ -91,9 +93,11 @@ do_json_cb(void *priv, const struct VSC_point * const pt)
 	uint64_t val;
 	int *jp;
 
-	jp = priv;
+	if (pt == NULL)
+		return (0);
 
-	assert(!strcmp(pt->fmt, "uint64_t"));
+	jp = priv;
+	assert(!strcmp(pt->desc->fmt, "uint64_t"));
 	val = *(const volatile uint64_t*)pt->ptr;
 
 	if (*jp) *jp = 0; else printf(",\n");
@@ -104,15 +108,15 @@ do_json_cb(void *priv, const struct VSC_point * const pt)
 		printf("%s.", pt->class);
 	if (pt->ident[0])
 		printf("%s.", pt->ident);
-	printf("%s\": {", pt->name);
+	printf("%s\": {", pt->desc->name);
 
 	if (strcmp(pt->class, "")) printf("\"type\": \"%s\", ",  pt->class);
 	if (strcmp(pt->ident, "")) printf("\"ident\": \"%s\", ", pt->ident);
 
 	printf("\"value\": %ju, ", val);
 
-	printf("\"flag\": \"%c\", ", pt->flag);
-	printf("\"description\": \"%s\"", pt->desc);
+	printf("\"flag\": \"%c\", ", pt->desc->flag);
+	printf("\"description\": \"%s\"", pt->desc->sdesc);
 	printf("}");
 
 	if (*jp) printf("\n");
@@ -153,22 +157,24 @@ do_once_cb(void *priv, const struct VSC_point * const pt)
 	uint64_t val;
 	int i;
 
+	if (pt == NULL)
+		return (0);
 	op = priv;
-	assert(!strcmp(pt->fmt, "uint64_t"));
+	assert(!strcmp(pt->desc->fmt, "uint64_t"));
 	val = *(const volatile uint64_t*)pt->ptr;
 	i = 0;
 	if (strcmp(pt->class, ""))
 		i += printf("%s.", pt->class);
 	if (strcmp(pt->ident, ""))
 		i += printf("%s.", pt->ident);
-	i += printf("%s", pt->name);
+	i += printf("%s", pt->desc->name);
 	if (i > op->pad)
 		op->pad = i + 1;
 	printf("%*.*s", op->pad - i, op->pad - i, "");
-	if (pt->flag == 'a' || pt->flag == 'c')
-		printf("%12ju %12.2f %s\n", val, val / op->up, pt->desc);
+	if (pt->desc->flag == 'a' || pt->desc->flag == 'c')
+		printf("%12ju %12.2f %s\n", val, val / op->up, pt->desc->sdesc);
 	else
-		printf("%12ju %12s %s\n", val, ".  ", pt->desc);
+		printf("%12ju %12s %s\n", val, ".  ", pt->desc->sdesc);
 	return (0);
 }
 
@@ -197,10 +203,10 @@ do_list_cb(void *priv, const struct VSC_point * const pt)
 		i += fprintf(stderr, "%s.", pt->class);
 	if (strcmp(pt->ident, ""))
 		i += fprintf(stderr, "%s.", pt->ident);
-	i += fprintf(stderr, "%s", pt->name);
+	i += fprintf(stderr, "%s", pt->desc->name);
 	if (i < 30)
 		fprintf(stderr, "%*s", i - 30, "");
-	fprintf(stderr, " %s\n", pt->desc);
+	fprintf(stderr, " %s\n", pt->desc->sdesc);
 	return (0);
 }
 
@@ -253,7 +259,6 @@ main(int argc, char * const *argv)
 	int delay = 1, once = 0, xml = 0, json = 0, do_repeat = 0;
 
 	vd = VSM_New();
-	VSC_Setup(vd);
 
 	while ((c = getopt(argc, argv, VSC_ARGS "1f:lVw:xjt:")) != -1) {
 		switch (c) {
@@ -261,8 +266,6 @@ main(int argc, char * const *argv)
 			once = 1;
 			break;
 		case 'l':
-			if (VSC_Open(vd, 1))
-				exit(1);
 			list_fields(vd);
 			exit(0);
 		case 'V':
@@ -285,9 +288,6 @@ main(int argc, char * const *argv)
 		}
 	}
 
-	if (VSC_Open(vd, 1))
-		exit(1);
-
 	VSC_C_main = VSC_Main(vd);
 	AN(VSC_C_main);
 
diff --git a/bin/varnishstat/varnishstat_curses.c b/bin/varnishstat/varnishstat_curses.c
index 9fa6b82..d4e714c 100644
--- a/bin/varnishstat/varnishstat_curses.c
+++ b/bin/varnishstat/varnishstat_curses.c
@@ -81,7 +81,9 @@ do_curses_cb(void *priv, const struct VSC_point * const sp)
 	char buf[128];
 
 	(void)priv;
-	assert(!strcmp(sp->fmt, "uint64_t"));
+	if (sp == NULL)
+		return (0);
+	assert(!strcmp(sp->desc->fmt, "uint64_t"));
 
 	pt = calloc(sizeof *pt, 1);
 	AN(pt);
@@ -89,7 +91,7 @@ do_curses_cb(void *priv, const struct VSC_point * const sp)
 
 	pt->ptr = sp->ptr;
 	pt->ref = *pt->ptr;
-	pt->flag = sp->flag;
+	pt->flag = sp->desc->flag;
 
 	*buf = '\0';
 	if (strcmp(sp->class, "")) {
@@ -100,9 +102,9 @@ do_curses_cb(void *priv, const struct VSC_point * const sp)
 		strcat(buf, sp->ident);
 		strcat(buf, ".");
 	}
-	strcat(buf, sp->name);
+	strcat(buf, sp->desc->name);
 	strcat(buf, " - ");
-	strcat(buf, sp->desc);
+	strcat(buf, sp->desc->sdesc);
 	pt->name = strdup(buf);
 	AN(pt->name);
 	return (0);
@@ -144,7 +146,6 @@ do_curses(struct VSM_data *vd, const struct VSC_C_main *VSC_C_main,
 	int ch, line;
 	struct pt *pt;
 	double act, lact;
-	unsigned seq;
 
 	(void)initscr();
 	AC(raw());
@@ -157,7 +158,6 @@ do_curses(struct VSM_data *vd, const struct VSC_C_main *VSC_C_main,
 		/*
 		 * Initialization goes in outher loop
 		 */
-		seq = VSM_Seq(vd);
 		prep_pts(vd);
 		AC(erase());
 		AC(refresh());
@@ -170,15 +170,11 @@ do_curses(struct VSM_data *vd, const struct VSC_C_main *VSC_C_main,
 		lact = 0;
 
 		while (1) {
-			if (seq != VSM_Seq(vd))
-				break;
 			/*
 			 * Break to outher loop if we need to re-read file.
 			 * Only check if it looks like nothing is happening.
 			 */
 			act = VSC_C_main->cache_hit + VSC_C_main->cache_miss + 1;
-			if (act == lact && VSM_ReOpen(vd, 1))
-				break;
 			lact = act;
 
 			AZ(gettimeofday(&tv, NULL));
diff --git a/include/vapi/vsc.h b/include/vapi/vsc.h
index 3cf16c6..6ebab31 100644
--- a/include/vapi/vsc.h
+++ b/include/vapi/vsc.h
@@ -26,6 +26,10 @@
  * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
  * SUCH DAMAGE.
  *
+ * This is the public API for the VSC access.
+ *
+ * VSC is a "subclass" of VSM.
+ *
  */
 
 #ifndef VAPI_VSC_H_INCLUDED
@@ -34,17 +38,12 @@
 #include "vapi/vsc_int.h"
 
 struct VSM_data;
+struct VSM_fantom;
 
 /*---------------------------------------------------------------------
  * VSC level access functions
  */
 
-void VSC_Setup(struct VSM_data *vd);
-	/*
-	 * Setup vd for use with VSC functions.
-	 * Must be called once before any other VSC function is called
-	 */
-
 #define VSC_ARGS	"f:n:"
 #define VSC_n_USAGE	VSM_n_USAGE
 #define VSC_f_USAGE	"[-f field_name,...]"
@@ -55,39 +54,58 @@ int VSC_Arg(struct VSM_data *vd, int arg, const char *opt);
 	/*
 	 * Handle standard stat-presenter arguments
 	 * Return:
-	 *	-1 error
+	 *	-1 error, VSM_Error() returns diagnostic string
 	 *	 0 not handled
 	 *	 1 Handled.
 	 */
 
-int VSC_Open(struct VSM_data *vd, int diag);
-	/*
-	 * Open shared memory for VSC processing.
-	 * args and returns as VSM_Open()
-	 */
-
-struct VSC_C_main *VSC_Main(const struct VSM_data *vd);
+struct VSC_C_main *VSC_Main(struct VSM_data *vd);
 	/*
 	 * return Main stats structure
 	 * returns NULL until child has been started.
 	 */
 
+struct VSC_desc {
+	const char *name;		/* field name			*/
+	const char *fmt;		/* field format ("uint64_t")	*/
+	int flag;			/* 'c' = counter, 'g' = gauge	*/
+	const char *sdesc;		/* short description		*/
+	const char *ldesc;		/* long description		*/
+};
+
 struct VSC_point {
 	const char *class;		/* stat struct type		*/
 	const char *ident;		/* stat struct ident		*/
-	const char *name;		/* field name			*/
-	const char *fmt;		/* field format ("uint64_t")	*/
-	int flag;			/* 'a' = counter, 'i' = gauge	*/
-	const char *desc;		/* description			*/
+	const struct VSC_desc *desc;	/* point description		*/
 	const volatile void *ptr;	/* field value			*/
+	struct VSM_fantom *vf;
 };
 
 typedef int VSC_iter_f(void *priv, const struct VSC_point *const pt);
 
-int VSC_Iter(const struct VSM_data *vd, VSC_iter_f *func, void *priv);
+int VSC_Iter(struct VSM_data *vd, VSC_iter_f *func, void *priv);
 	/*
 	 * Iterate over all statistics counters, calling "func" for
 	 * each counter not suppressed by any "-f" arguments.
+	 *
+ 	 * Func is called with pt == NULL, whenever VSM allocations
+	 * change (child restart, allocations/deallocations)
+	 *
+	 * Returns:
+	 * 	!=0:	func returned non-zero
+	 *	-1:	No VSC's available
+	 *	0: 	Done
 	 */
 
+/**********************************************************************
+ * Precompiled VSC_desc's for all know VSCs.
+ */
+#define VSC_F(n,t,l,f,d,e)
+#define VSC_DO(U,l,t) extern const struct VSC_desc VSC_desc_##l[];
+#define VSC_DONE(U,l,t)
+#include "tbl/vsc_all.h"
+#undef VSC_F
+#undef VSC_DO
+#undef VSC_DONE
+
 #endif /* VAPI_VSC_H_INCLUDED */
diff --git a/lib/libvarnishapi/vsc.c b/lib/libvarnishapi/vsc.c
index c9facfd..54bf20f 100644
--- a/lib/libvarnishapi/vsc.c
+++ b/lib/libvarnishapi/vsc.c
@@ -47,15 +47,22 @@
 #include "vqueue.h"
 #include "vsm_api.h"
 
+struct vsc_pt {
+	unsigned		magic;
+#define VSC_PT_MAGIC		0xa4ff159a
+	struct VSC_point	point;
+	VTAILQ_ENTRY(vsc_pt)	list;
+};
+
 struct vsc_sf {
 	unsigned		magic;
-#define VSL_SF_MAGIC		0x558478dd
-	VTAILQ_ENTRY(vsc_sf)	next;
+#define VSC_SF_MAGIC		0x558478dd
+	VTAILQ_ENTRY(vsc_sf)	list;
 	int			flags;
-#define VSL_SF_EXCL		(1 << 0)
-#define VSL_SF_CL_WC		(1 << 1)
-#define VSL_SF_ID_WC		(1 << 2)
-#define VSL_SF_NM_WC		(1 << 3)
+#define VSC_SF_EXCL		(1 << 0)
+#define VSC_SF_CL_WC		(1 << 1)
+#define VSC_SF_ID_WC		(1 << 2)
+#define VSC_SF_NM_WC		(1 << 3)
 	char			*class;
 	char			*ident;
 	char			*name;
@@ -65,40 +72,56 @@ struct vsc {
 	unsigned		magic;
 #define VSC_MAGIC		0x3373554a
 
-	int			sf_init;
+	VTAILQ_HEAD(, vsc_pt)	pt_list;
 	VTAILQ_HEAD(, vsc_sf)	sf_list;
-
+	struct VSM_fantom	main_fantom;
+	struct VSM_fantom	iter_fantom;
 };
 
 
 /*--------------------------------------------------------------------*/
 
-void
-VSC_Setup(struct VSM_data *vd)
+static struct vsc *
+vsc_setup(struct VSM_data *vd)
 {
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	AZ(vd->vsc);
-	ALLOC_OBJ(vd->vsc, VSC_MAGIC);
-	AN(vd->vsc);
-	VTAILQ_INIT(&vd->vsc->sf_list);
+	if (vd->vsc == NULL) {
+		ALLOC_OBJ(vd->vsc, VSC_MAGIC);
+		VTAILQ_INIT(&vd->vsc->sf_list);
+		VTAILQ_INIT(&vd->vsc->pt_list);
+	}
+	CHECK_OBJ_NOTNULL(vd->vsc, VSC_MAGIC);
+	return (vd->vsc);
 }
 
 /*--------------------------------------------------------------------*/
 
-void
-VSC_Delete(struct VSM_data *vd)
+static void
+vsc_delete_pts(struct vsc *vsc)
+{
+	struct vsc_pt *pt;
+	struct VSM_fantom *vf = NULL;
+
+	while(!VTAILQ_EMPTY(&vsc->pt_list)) {
+		pt = VTAILQ_FIRST(&vsc->pt_list);
+		VTAILQ_REMOVE(&vsc->pt_list, pt, list);
+		if (pt->point.vf != vf) {
+			vf = pt->point.vf;
+			free(vf);
+		}
+		FREE_OBJ(pt);
+	}
+}
+
+static void
+vsc_delete_sfs(struct vsc *vsc)
 {
 	struct vsc_sf *sf;
-	struct vsc *vsc;
 
-	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	vsc = vd->vsc;
-	vd->vsc = NULL;
-	CHECK_OBJ_NOTNULL(vsc, VSC_MAGIC);
 	while(!VTAILQ_EMPTY(&vsc->sf_list)) {
 		sf = VTAILQ_FIRST(&vsc->sf_list);
-		VTAILQ_REMOVE(&vsc->sf_list, sf, next);
+		VTAILQ_REMOVE(&vsc->sf_list, sf, list);
 		free(sf->class);
 		free(sf->ident);
 		free(sf->name);
@@ -106,39 +129,42 @@ VSC_Delete(struct VSM_data *vd)
 	}
 }
 
-/*--------------------------------------------------------------------*/
-
-static int
-vsc_sf_arg(const struct VSM_data *vd, const char *opt)
+void
+VSC_Delete(struct VSM_data *vd)
 {
 	struct vsc *vsc;
-	struct vsc_sf *sf;
-	char **av, *q, *p;
-	int i;
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
 	vsc = vd->vsc;
+	vd->vsc = NULL;
 	CHECK_OBJ_NOTNULL(vsc, VSC_MAGIC);
+	vsc_delete_sfs(vsc);
+	vsc_delete_pts(vsc);
+	FREE_OBJ(vsc);
+}
 
-	if (VTAILQ_EMPTY(&vsc->sf_list)) {
-		if (*opt == '^')
-			vsc->sf_init = 1;
-	}
+/*--------------------------------------------------------------------*/
+
+static int
+vsc_f_arg(struct VSM_data *vd, const char *opt)
+{
+	struct vsc *vsc = vsc_setup(vd);
+	struct vsc_sf *sf;
+	char **av, *q, *p;
+	int i;
 
 	av = VAV_Parse(opt, NULL, ARGV_COMMA);
 	AN(av);
-	if (av[0] != NULL) {
-		vd->diag(vd->priv, "Parse error: %s", av[0]);
-		return (-1);
-	}
+	if (av[0] != NULL)
+		return (vsm_diag(vd, "Parse error: %s", av[0]));
 	for (i = 1; av[i] != NULL; i++) {
-		ALLOC_OBJ(sf, VSL_SF_MAGIC);
+		ALLOC_OBJ(sf, VSC_SF_MAGIC);
 		AN(sf);
-		VTAILQ_INSERT_TAIL(&vsc->sf_list, sf, next);
+		VTAILQ_INSERT_TAIL(&vsc->sf_list, sf, list);
 
 		p = av[i];
 		if (*p == '^') {
-			sf->flags |= VSL_SF_EXCL;
+			sf->flags |= VSC_SF_EXCL;
 			p++;
 		}
 
@@ -167,21 +193,21 @@ vsc_sf_arg(const struct VSM_data *vd, const char *opt)
 			q = strchr(sf->class, '*');
 			if (q != NULL && q[1] == '\0') {
 				*q = '\0';
-				sf->flags |= VSL_SF_CL_WC;
+				sf->flags |= VSC_SF_CL_WC;
 			}
 		}
 		if (sf->ident != NULL) {
 			q = strchr(sf->ident, '*');
 			if (q != NULL && q[1] == '\0') {
 				*q = '\0';
-				sf->flags |= VSL_SF_ID_WC;
+				sf->flags |= VSC_SF_ID_WC;
 			}
 		}
 		if (sf->name != NULL) {
 			q = strchr(sf->name, '*');
 			if (q != NULL && q[1] == '\0') {
 				*q = '\0';
-				sf->flags |= VSL_SF_NM_WC;
+				sf->flags |= VSC_SF_NM_WC;
 			}
 		}
 	}
@@ -195,10 +221,8 @@ int
 VSC_Arg(struct VSM_data *vd, int arg, const char *opt)
 {
 
-	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	AN(vd->vsc);
 	switch (arg) {
-	case 'f': return (vsc_sf_arg(vd, opt));
+	case 'f': return (vsc_f_arg(vd, opt));
 	case 'n': return (VSM_n_Arg(vd, opt));
 	default:
 		return (0);
@@ -207,33 +231,19 @@ VSC_Arg(struct VSM_data *vd, int arg, const char *opt)
 
 /*--------------------------------------------------------------------*/
 
-int
-VSC_Open(struct VSM_data *vd, int diag)
-{
-	int i;
-
-	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	AN(vd->vsc);
-
-	i = VSM_Open(vd, diag);
-	return (i);
-}
-
-/*--------------------------------------------------------------------*/
-
 struct VSC_C_main *
-VSC_Main(const struct VSM_data *vd)
+VSC_Main(struct VSM_data *vd)
 {
-	struct VSM_fantom vf;
+	struct vsc *vsc = vsc_setup(vd);
 
-	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	CHECK_OBJ_NOTNULL(vd->vsc, VSC_MAGIC);
-
-	if (!VSM_Get(vd, &vf, VSC_CLASS, "", ""))
+	if (!vd->head && VSM_Open(vd))
 		return (NULL);
-	return ((void*)vf.b);
+	if (!VSM_Get(vd, &vsc->main_fantom, VSC_CLASS, "", ""))
+		return (NULL);
+	return ((void*)vsc->main_fantom.b);
 }
 
+#if 0
 /*--------------------------------------------------------------------
  * -1 -> unknown stats encountered.
  */
@@ -256,23 +266,28 @@ iter_call(const struct vsc *vsc, VSC_iter_f *func, void *priv,
     const struct VSC_point *const sp)
 {
 	struct vsc_sf *sf;
+	struct vsc_pt *pt;
 	int good;
 
 	CHECK_OBJ_NOTNULL(vsc, VSC_MAGIC);
 
-	if (VTAILQ_EMPTY(&vsc->sf_list))
-		return (func(priv, sp));
+	ALLOC_OBJ(pt, VSC_PT_MAGIC);
+	AN(pt);
 
-	good = vsc->sf_init;
+	if (VTAILQ_EMPTY(&vsc->sf_list)) {
+		VTAILQ_INSERT_TAIL(&vsc->pt_list, pt, list);
+		return (func(priv, sp));
+	}
 
-	VTAILQ_FOREACH(sf, &vsc->sf_list, next) {
-		if (iter_test(sf->class, sp->class, sf->flags & VSL_SF_CL_WC))
+	good = 0;
+	VTAILQ_FOREACH(sf, &vsc->sf_list, list) {
+		if (iter_test(sf->class, sp->class, sf->flags & VSC_SF_CL_WC))
 			continue;
-		if (iter_test(sf->ident, sp->ident, sf->flags & VSL_SF_ID_WC))
+		if (iter_test(sf->ident, sp->ident, sf->flags & VSC_SF_ID_WC))
 			continue;
-		if (iter_test(sf->name, sp->name, sf->flags & VSL_SF_NM_WC))
+		if (iter_test(sf->name, sp->desc->name, sf->flags & VSC_SF_NM_WC))
 			continue;
-		if (sf->flags & VSL_SF_EXCL)
+		if (sf->flags & VSC_SF_EXCL)
 			good = 0;
 		else
 			good = 1;
@@ -283,26 +298,27 @@ iter_call(const struct vsc *vsc, VSC_iter_f *func, void *priv,
 }
 
 #define VSC_DO(U,l,t)							\
-	static int							\
+	static void							\
 	iter_##l(const struct vsc *vsc, struct VSM_fantom *vf,		\
-	    VSC_iter_f *func, void *priv)				\
+	    const struct VSC_desc *descs)				\
 	{								\
 		struct VSC_C_##l *st;					\
-		struct VSC_point sp;					\
+		struct VSM_fantom *vf2;					\
 		int i;							\
 									\
 		CHECK_OBJ_NOTNULL(vsc, VSC_MAGIC);			\
 		st = vf->b;						\
 		sp.class = t;						\
-		sp.ident = vf->chunk->ident;
+		sp.ident = vf->chunk->ident;				\
+		sp.desc = descs++;					\
+		vf2 = malloc(sizeof *vf2);				\
+		AN(vf2);						\
+		memcpy(vf2, vf, sizeof *vf2);
 
 #define VSC_F(nn,tt,ll,ff,dd,ee)					\
-		sp.name = #nn;						\
-		sp.fmt = #tt;						\
-		sp.flag = ff;						\
-		sp.desc = dd;						\
 		sp.ptr = &st->nn;					\
-		i = iter_call(vsc, func, priv, &sp);			\
+		sp.vf = vf2;						\
+		i = iter_call(vsc, &sp);				\
 		if (i)							\
 			return(i);
 
@@ -315,19 +331,14 @@ iter_call(const struct vsc *vsc, VSC_iter_f *func, void *priv,
 #undef VSC_F
 #undef VSC_DONE
 
-int
-VSC_Iter(const struct VSM_data *vd, VSC_iter_f *func, void *priv)
+static void
+vsc_build_pt_list(struct VSM_data *vd)
 {
-	struct vsc *vsc;
+	struct vsc *vsc = vsc_setup(vd);
 	struct VSM_fantom vf;
-	int i;
 
-	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	vsc = vd->vsc;
-	CHECK_OBJ_NOTNULL(vsc, VSC_MAGIC);
-	i = 0;
-	if (!VSM_StillValid(vd, NULL))
-		return (-1);
+	vsc_delete_pts(vsc *vsc);
+
 	VSM_FOREACH_SAFE(&vf, vd) {
 		if (strcmp(vf.chunk->class, VSC_CLASS))
 			continue;
@@ -335,11 +346,8 @@ VSC_Iter(const struct VSM_data *vd, VSC_iter_f *func, void *priv)
 #define VSC_F(n,t,l,f,d,e)
 #define VSC_DONE(a,b,c)
 #define VSC_DO(U,l,t)						\
-		if (!strcmp(vf.chunk->type, t)) {		\
-			i = iter_##l(vsc, &vf, func, priv);	\
-			if (!i)					\
-				continue;			\
-		}
+		if (!strcmp(vf.chunk->type, t))			\
+			iter_##l(vsc, &vf, VSC_desc_##l);
 #include "tbl/vsc_all.h"
 #undef VSC_F
 #undef VSC_DO
@@ -349,3 +357,127 @@ VSC_Iter(const struct VSM_data *vd, VSC_iter_f *func, void *priv)
 	}
 	return (i);
 }
+#endif
+
+/*--------------------------------------------------------------------
+ */
+
+static void
+vsc_add_pt(struct vsc *vsc, const char *class, const char *ident,
+    const struct VSC_desc *desc, const volatile void *ptr,
+    struct VSM_fantom *vf)
+{
+	struct vsc_pt *pt;
+
+	ALLOC_OBJ(pt, VSC_PT_MAGIC);
+	AN(pt);
+	pt->point.class = class;
+	pt->point.ident = ident;
+	pt->point.desc = desc;
+	pt->point.ptr = ptr;
+	pt->point.vf = vf;
+	VTAILQ_INSERT_TAIL(&vsc->pt_list, pt, list);
+}
+
+#define VSC_DO(U,l,t)							\
+	static void							\
+	iter_##l(struct vsc *vsc, struct VSM_fantom *vf,		\
+	    const struct VSC_desc *descs)				\
+	{								\
+		struct VSC_C_##l *st;					\
+		struct VSM_fantom *vf2;					\
+		const char *class = t;					\
+									\
+		CHECK_OBJ_NOTNULL(vsc, VSC_MAGIC);			\
+		st = vf->b;						\
+		vf2 = malloc(sizeof *vf2);				\
+		AN(vf2);						\
+		memcpy(vf2, vf, sizeof *vf2);
+
+#define VSC_F(nn,tt,ll,ff,dd,ee)					\
+		vsc_add_pt(vsc, class, vf->chunk->ident, descs++,	\
+		    &st->nn, vf2);
+
+#define VSC_DONE(U,l,t)							\
+	}
+
+#include "tbl/vsc_all.h"
+#undef VSC_DO
+#undef VSC_F
+#undef VSC_DONE
+
+/*--------------------------------------------------------------------
+ */
+
+static void
+vsc_build_pt_list(struct VSM_data *vd)
+{
+	struct vsc *vsc = vsc_setup(vd);
+	struct VSM_fantom vf;
+
+	vsc_delete_pts(vsc);
+
+	VSM_FOREACH_SAFE(&vf, vd) {
+		if (strcmp(vf.chunk->class, VSC_CLASS))
+			continue;
+		/*lint -save -e525 -e539 */
+#define VSC_F(n,t,l,f,d,e)
+#define VSC_DONE(a,b,c)
+#define VSC_DO(U,l,t)						\
+		if (!strcmp(vf.chunk->type, t))			\
+			iter_##l(vsc, &vf, VSC_desc_##l);
+#include "tbl/vsc_all.h"
+#undef VSC_F
+#undef VSC_DO
+#undef VSC_DONE
+		/*lint -restore */
+	}
+
+	/* XXX: filter pt list */
+}
+
+/*--------------------------------------------------------------------
+ */
+
+int
+VSC_Iter(struct VSM_data *vd, VSC_iter_f *func, void *priv)
+{
+	struct vsc *vsc = vsc_setup(vd);
+	struct vsc_pt *pt;
+	int i;
+
+	if (1 != VSM_StillValid(vd, &vsc->iter_fantom)) {
+		if (!VSM_Get(vd, &vsc->iter_fantom, VSC_CLASS, "", "")) {
+			VSM_Close(vd);
+			if (!vd->head && VSM_Open(vd))
+				return (-1);
+			if (!VSM_Get(vd, &vsc->iter_fantom, VSC_CLASS, "", "")) {
+				return (-1);
+			}
+		}
+		AN(vd->head);
+		func(priv, NULL);
+		vsc_build_pt_list(vd);
+	}
+	AN(vd->head);
+	VTAILQ_FOREACH(pt, &vsc->pt_list, list) {
+		i = func(priv, &pt->point);
+		if (i)
+			return (i);
+	}
+	return (0);
+}
+
+/*--------------------------------------------------------------------
+ * Build the static point descriptions
+ */
+
+#define VSC_F(n,t,l,f,d,e)	{#n,#t,f,d,e},
+#define VSC_DO(U,l,t) const struct VSC_desc VSC_desc_##l[] = {
+#define VSC_DONE(U,l,t) };
+#include "tbl/vsc_all.h"
+#undef VSC_F
+#undef VSC_DO
+#undef VSC_DONE
+
+
diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c
index 1d762e5..a9f176c 100644
--- a/lib/libvarnishapi/vsm.c
+++ b/lib/libvarnishapi/vsm.c
@@ -113,7 +113,7 @@ VSM_n_Arg(struct VSM_data *vd, const char *opt)
 {
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	AN(vd->n_opt);
+	AN(opt);
 
 	REPLACE(vd->n_opt, opt);
 	if (VIN_N_Arg(vd->n_opt, NULL, NULL, &vd->fname))



From phk at varnish-cache.org  Wed Nov 23 21:11:17 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 23 Nov 2011 22:11:17 +0100
Subject: [master] 852d2a4 Add missing #include
Message-ID: <E1RTK69-0007gu-BA@project.varnish-software.com>

commit 852d2a42a35e47c770d03e95c03e2254ff405597
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Nov 23 20:15:40 2011 +0000

    Add missing #include

diff --git a/bin/varnishd/mgt/mgt_sandbox_solaris.c b/bin/varnishd/mgt/mgt_sandbox_solaris.c
index 79f6650..114d6a4 100644
--- a/bin/varnishd/mgt/mgt_sandbox_solaris.c
+++ b/bin/varnishd/mgt/mgt_sandbox_solaris.c
@@ -45,6 +45,7 @@
 #include "mgt/mgt.h"
 
 #include "common/heritage.h"
+#include "common/params.h"
 
 /*--------------------------------------------------------------------
  * SOLARIS PRIVILEGES: Note on use of symbolic PRIV_* constants



From phk at varnish-cache.org  Wed Nov 23 21:11:20 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 23 Nov 2011 22:11:20 +0100
Subject: [master] a8aa0ba Complete the VSC filtering,
	and make everything compile and pass tests.
Message-ID: <E1RTK6C-0007hP-GX@project.varnish-software.com>

commit a8aa0bafdcf6ffbabb4bd5a21fa2fd5d1c612f39
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Nov 23 21:09:18 2011 +0000

    Complete the VSC filtering, and make everything compile and pass tests.
    
    Still not done, in particular: Do not roll any releases until
    libvarnishapi symbo/version stuff has been polished.
    
    Fixes	#829

diff --git a/bin/varnishadm/varnishadm.c b/bin/varnishadm/varnishadm.c
index f5d6054..4570e3c 100644
--- a/bin/varnishadm/varnishadm.c
+++ b/bin/varnishadm/varnishadm.c
@@ -282,25 +282,27 @@ n_arg_sock(const char *n_arg)
 	struct VSM_data *vsd;
 	char *p;
 	int sock;
+	struct VSM_fantom vt;
 
 	vsd = VSM_New();
 	assert(VSL_Arg(vsd, 'n', n_arg));
-	if (VSM_Open(vsd, 1)) {
-		fprintf(stderr, "Could not open shared memory\n");
+	if (VSM_Open(vsd)) {
+		fprintf(stderr, "%s\n", VSM_Error(vsd));
 		return (-1);
 	}
 	if (T_arg == NULL) {
-		p = VSM_Find_Chunk(vsd, "Arg", "-T", "", NULL);
-		if (p == NULL)  {
+		if (VSM_Get(vsd, &vt, "Arg", "-T", "")) {
 			fprintf(stderr, "No -T arg in shared memory\n");
 			return (-1);
 		}
-		T_start = T_arg = strdup(p);
+		T_start = T_arg = strdup(vt.b);
 	}
 	if (S_arg == NULL) {
-		p = VSM_Find_Chunk(vsd, "Arg", "-S", "", NULL);
-		if (p != NULL)
-			S_arg = strdup(p);
+		if (VSM_Get(vsd, &vt, "Arg", "-S", "")) {
+			fprintf(stderr, "No -S arg in shared memory\n");
+			return (-1);
+		}
+		S_arg = strdup(vt.b);
 	}
 	sock = -1;
 	while (*T_arg) {
diff --git a/bin/varnishhist/varnishhist.c b/bin/varnishhist/varnishhist.c
index a4bc6b7..852ff41 100644
--- a/bin/varnishhist/varnishhist.c
+++ b/bin/varnishhist/varnishhist.c
@@ -351,7 +351,6 @@ main(int argc, char **argv)
 	match_tag = -1;
 
 	vd = VSM_New();
-	VSL_Setup(vd);
 
 	while ((o = getopt(argc, argv, VSL_ARGS "Vw:r:R:f:p:")) != -1) {
 		switch (o) {
@@ -424,8 +423,10 @@ main(int argc, char **argv)
 	}
 	strcpy(format + 4*(fnum-1), "%lf");
 
-	if (VSL_Open(vd, 1))
+	if (VSM_Open(vd)) {
+		fprintf(stderr, "%s\n", VSM_Error(vd));
 		exit(1);
+	}
 
 	log_ten = log(10.0);
 
diff --git a/bin/varnishlog/varnishlog.c b/bin/varnishlog/varnishlog.c
index e15d102..45b4327 100644
--- a/bin/varnishlog/varnishlog.c
+++ b/bin/varnishlog/varnishlog.c
@@ -60,7 +60,7 @@ static uint64_t       bitmap[65536];
 #define F_INVCL		(1 << 0)
 
 static void
-h_order_finish(int fd, const struct VSM_data *vd)
+h_order_finish(int fd, struct VSM_data *vd)
 {
 
 	AZ(VSB_finish(ob[fd]));
@@ -72,7 +72,7 @@ h_order_finish(int fd, const struct VSM_data *vd)
 }
 
 static void
-clean_order(const struct VSM_data *vd)
+clean_order(struct VSM_data *vd)
 {
 	unsigned u;
 
@@ -234,7 +234,7 @@ open_log(const char *w_arg, int a_flag)
 }
 
 static void
-do_write(const struct VSM_data *vd, const char *w_arg, int a_flag)
+do_write(struct VSM_data *vd, const char *w_arg, int a_flag)
 {
 	int fd, i, l;
 	uint32_t *p;
@@ -243,7 +243,7 @@ do_write(const struct VSM_data *vd, const char *w_arg, int a_flag)
 	XXXAN(fd >= 0);
 	(void)signal(SIGHUP, sighup);
 	while (1) {
-		i = VSL_NextLog(vd, &p, NULL);
+		i = VSL_NextSLT(vd, &p, NULL);
 		if (i < 0)
 			break;
 		if (i > 0) {
@@ -285,7 +285,6 @@ main(int argc, char * const *argv)
 	struct VSM_data *vd;
 
 	vd = VSM_New();
-	VSL_Setup(vd);
 
 	while ((c = getopt(argc, argv, VSL_ARGS "aDP:uVw:oO")) != -1) {
 		switch (c) {
@@ -336,8 +335,10 @@ main(int argc, char * const *argv)
 	if ((argc - optind) > 0)
 		usage();
 
-	if (VSL_Open(vd, 1))
+	if (VSM_Open(vd)) {
+		fprintf(stderr, "%s\n", VSM_Error(vd));
 		exit(1);
+	}
 
 	if (P_arg && (pfh = VPF_Open(P_arg, 0644, NULL)) == NULL) {
 		perror(P_arg);
diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c
index 9765796..5561adc 100644
--- a/bin/varnishncsa/varnishncsa.c
+++ b/bin/varnishncsa/varnishncsa.c
@@ -846,7 +846,6 @@ main(int argc, char *argv[])
 	format = "%h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\"";
 
 	vd = VSM_New();
-	VSL_Setup(vd);
 
 	while ((c = getopt(argc, argv, VSL_ARGS "aDP:Vw:fF:")) != -1) {
 		switch (c) {
@@ -907,8 +906,10 @@ main(int argc, char *argv[])
 
 	VSL_Arg(vd, 'c', optarg);
 
-	if (VSL_Open(vd, 1))
-		exit(1);
+	if (VSM_Open(vd)) {
+		fprintf(stderr, "%s\n", VSM_Error(vd));
+		return (-1);
+	}
 
 	if (P_arg && (pfh = VPF_Open(P_arg, 0644, NULL)) == NULL) {
 		perror(P_arg);
diff --git a/bin/varnishreplay/varnishreplay.c b/bin/varnishreplay/varnishreplay.c
index 12b6c41..8e694da 100644
--- a/bin/varnishreplay/varnishreplay.c
+++ b/bin/varnishreplay/varnishreplay.c
@@ -707,7 +707,6 @@ main(int argc, char *argv[])
 	const char *address = NULL;
 
 	vd = VSM_New();
-	VSL_Setup(vd);
 	debug = 0;
 
 	VSL_Arg(vd, 'c', NULL);
@@ -730,8 +729,10 @@ main(int argc, char *argv[])
 		usage();
 	}
 
-	if (VSL_Open(vd, 1))
+	if (VSM_Open(vd)) {
+		fprintf(stderr, "%s\n", VSM_Error(vd));
 		exit(1);
+	}
 
 	addr_info = init_connection(address);
 
diff --git a/bin/varnishtest/flint.lnt b/bin/varnishtest/flint.lnt
index 53ad3dc..9ff3c68 100644
--- a/bin/varnishtest/flint.lnt
+++ b/bin/varnishtest/flint.lnt
@@ -25,3 +25,5 @@
 -e713	// Loss of precision (assignment) (unsigned long long to long long)
 -e574	// Signed-unsigned mix with relational
 -e835	//  A zero has been given as ___ argument to operator '___'  (<<)
+-e786	// String concatenation within initializer
+
diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c
index 080f48a..2f7514a 100644
--- a/bin/varnishtest/vtc_varnish.c
+++ b/bin/varnishtest/vtc_varnish.c
@@ -198,11 +198,7 @@ varnishlog_thread(void *priv)
 
 	CAST_OBJ_NOTNULL(v, priv, VARNISH_MAGIC);
 	vsl = VSM_New();
-	VSL_Setup(vsl);
 	(void)VSL_Arg(vsl, 'n', v->workdir);
-	while (v->pid  && VSL_Open(vsl, 0) != 0) {
-		assert(usleep(VSL_SLEEP_USEC) == 0 || errno == EINTR);
-	}
 	while (v->pid) {
 		if (VSL_Dispatch(vsl, h_addlog, v) <= 0)
 			break;
@@ -332,7 +328,6 @@ varnish_launch(struct varnish *v)
 	char *r;
 
 	v->vd = VSM_New();
-	VSC_Setup(v->vd);
 
 	/* Create listener socket */
 	nap = VSS_resolve("127.0.0.1", "0", &ap);
@@ -457,7 +452,7 @@ varnish_launch(struct varnish *v)
 	free(r);
 
 	(void)VSL_Arg(v->vd, 'n', v->workdir);
-	AZ(VSC_Open(v->vd, 1));
+	AZ(VSM_Open(v->vd));
 }
 
 /**********************************************************************
@@ -696,6 +691,8 @@ do_stat_cb(void *priv, const struct VSC_point * const pt)
 	const char *p = sp->target;
 	int i;
 
+	if (pt == NULL)
+		return(0);
 	if (strcmp(pt->class, "")) {
 		i = strlen(pt->class);
 		if (memcmp(pt->class, p, i))
@@ -714,10 +711,10 @@ do_stat_cb(void *priv, const struct VSC_point * const pt)
 			return (0);
 		p++;
 	}
-	if (strcmp(pt->name, p))
+	if (strcmp(pt->desc->name, p))
 		return (0);
 
-	assert(!strcmp(pt->fmt, "uint64_t"));
+	assert(!strcmp(pt->desc->fmt, "uint64_t"));
 	sp->val = *(const volatile uint64_t*)pt->ptr;
 	return (1);
 }
@@ -740,12 +737,12 @@ varnish_expect(const struct varnish *v, char * const *av) {
 		good = VSC_Iter(v->vd, do_stat_cb, &sp);
 		if (good < 0) {
 			VSM_Close(v->vd);
-			j = VSM_Open(v->vd, 0);
+			j = VSM_Open(v->vd);
 			if (j == 0)
 				continue;
 			do {
 				(void)usleep(100000);
-				j = VSM_Open(v->vd, 0);
+				j = VSM_Open(v->vd);
 				i++;
 			} while(i < 10 && j < 0);
 			if (j < 0)
diff --git a/bin/varnishtop/varnishtop.c b/bin/varnishtop/varnishtop.c
index efe312e..62efb15 100644
--- a/bin/varnishtop/varnishtop.c
+++ b/bin/varnishtop/varnishtop.c
@@ -195,7 +195,7 @@ accumulate_thread(void *arg)
 
 	for (;;) {
 
-		i = VSL_NextLog(vd, &p, NULL);
+		i = VSL_NextSLT(vd, &p, NULL);
 		if (i < 0)
 			break;
 		if (i == 0) {
@@ -292,7 +292,7 @@ do_once(struct VSM_data *vd)
 {
 	uint32_t *p;
 
-	while (VSL_NextLog(vd, &p, NULL) > 0)
+	while (VSL_NextSLT(vd, &p, NULL) > 0)
 		accumulate(p);
 	dump();
 }
@@ -313,7 +313,6 @@ main(int argc, char **argv)
 	float period = 60; /* seconds */
 
 	vd = VSM_New();
-	VSL_Setup(vd);
 
 	while ((o = getopt(argc, argv, VSL_ARGS "1fVp:")) != -1) {
 		switch (o) {
@@ -345,11 +344,12 @@ main(int argc, char **argv)
 		}
 	}
 
-	if (VSL_Open(vd, 1))
+	if (VSM_Open(vd)) {
+		fprintf(stderr, "%s\n", VSM_Error(vd));
 		exit (1);
+	}
 
 	if (once) {
-		VSL_NonBlocking(vd, 1);
 		do_once(vd);
 	} else {
 		do_curses(vd, period);
diff --git a/include/vapi/vsc.h b/include/vapi/vsc.h
index 6ebab31..16a1583 100644
--- a/include/vapi/vsc.h
+++ b/include/vapi/vsc.h
@@ -88,13 +88,13 @@ int VSC_Iter(struct VSM_data *vd, VSC_iter_f *func, void *priv);
 	 * Iterate over all statistics counters, calling "func" for
 	 * each counter not suppressed by any "-f" arguments.
 	 *
- 	 * Func is called with pt == NULL, whenever VSM allocations
+	 * Func is called with pt == NULL, whenever VSM allocations
 	 * change (child restart, allocations/deallocations)
 	 *
 	 * Returns:
-	 * 	!=0:	func returned non-zero
+	 *	!=0:	func returned non-zero
 	 *	-1:	No VSC's available
-	 *	0: 	Done
+	 *	0:	Done
 	 */
 
 /**********************************************************************
diff --git a/include/vapi/vsl.h b/include/vapi/vsl.h
index 35e09dc..dfe5a83 100644
--- a/include/vapi/vsl.h
+++ b/include/vapi/vsl.h
@@ -36,7 +36,7 @@
  *	VSL_Arg(vd, "r", "/some/file");
  * and once VSL_Dispatch()/VSL_NextSLT() will indicate EOF by returning -2.
  * Another file can then be opened with VSL_Arg() and processed.
- * 
+ *
  */
 
 #ifndef VAPI_VSL_H_INCLUDED
diff --git a/lib/libvarnishapi/libvarnishapi.map b/lib/libvarnishapi/libvarnishapi.map
index 83f5169..5640868 100644
--- a/lib/libvarnishapi/libvarnishapi.map
+++ b/lib/libvarnishapi/libvarnishapi.map
@@ -75,3 +75,12 @@ LIBVARNISHAPI_1.1 {
 	VSL_Name2Tag;
 	# Variables:
 } LIBVARNISHAPI_1.0;
+
+LIBVARNISHAPI_1.2 {
+  global:
+	# Functions:
+	VSL_NextSLT;
+	VSM_Error;
+	VSM_Get;
+	# Variables:
+} LIBVARNISHAPI_1.0;
diff --git a/lib/libvarnishapi/vsc.c b/lib/libvarnishapi/vsc.c
index 54bf20f..b399c98 100644
--- a/lib/libvarnishapi/vsc.c
+++ b/lib/libvarnishapi/vsc.c
@@ -50,8 +50,9 @@
 struct vsc_pt {
 	unsigned		magic;
 #define VSC_PT_MAGIC		0xa4ff159a
-	struct VSC_point	point;
 	VTAILQ_ENTRY(vsc_pt)	list;
+	struct VSM_fantom	vf;
+	struct VSC_point	point;
 };
 
 struct vsc_sf {
@@ -101,15 +102,10 @@ static void
 vsc_delete_pts(struct vsc *vsc)
 {
 	struct vsc_pt *pt;
-	struct VSM_fantom *vf = NULL;
 
 	while(!VTAILQ_EMPTY(&vsc->pt_list)) {
 		pt = VTAILQ_FIRST(&vsc->pt_list);
 		VTAILQ_REMOVE(&vsc->pt_list, pt, list);
-		if (pt->point.vf != vf) {
-			vf = pt->point.vf;
-			free(vf);
-		}
 		FREE_OBJ(pt);
 	}
 }
@@ -243,129 +239,13 @@ VSC_Main(struct VSM_data *vd)
 	return ((void*)vsc->main_fantom.b);
 }
 
-#if 0
-/*--------------------------------------------------------------------
- * -1 -> unknown stats encountered.
- */
-
-static inline int
-iter_test(const char *s1, const char *s2, int wc)
-{
-
-	if (s1 == NULL)
-		return (0);
-	if (!wc)
-		return (strcmp(s1, s2));
-	for (; *s1 != '\0' && *s1 == *s2; s1++, s2++)
-		continue;
-	return (*s1 != '\0');
-}
-
-static int
-iter_call(const struct vsc *vsc, VSC_iter_f *func, void *priv,
-    const struct VSC_point *const sp)
-{
-	struct vsc_sf *sf;
-	struct vsc_pt *pt;
-	int good;
-
-	CHECK_OBJ_NOTNULL(vsc, VSC_MAGIC);
-
-	ALLOC_OBJ(pt, VSC_PT_MAGIC);
-	AN(pt);
-
-	if (VTAILQ_EMPTY(&vsc->sf_list)) {
-		VTAILQ_INSERT_TAIL(&vsc->pt_list, pt, list);
-		return (func(priv, sp));
-	}
-
-	good = 0;
-	VTAILQ_FOREACH(sf, &vsc->sf_list, list) {
-		if (iter_test(sf->class, sp->class, sf->flags & VSC_SF_CL_WC))
-			continue;
-		if (iter_test(sf->ident, sp->ident, sf->flags & VSC_SF_ID_WC))
-			continue;
-		if (iter_test(sf->name, sp->desc->name, sf->flags & VSC_SF_NM_WC))
-			continue;
-		if (sf->flags & VSC_SF_EXCL)
-			good = 0;
-		else
-			good = 1;
-	}
-	if (!good)
-		return (0);
-	return (func(priv, sp));
-}
-
-#define VSC_DO(U,l,t)							\
-	static void							\
-	iter_##l(const struct vsc *vsc, struct VSM_fantom *vf,		\
-	    const struct VSC_desc *descs)				\
-	{								\
-		struct VSC_C_##l *st;					\
-		struct VSM_fantom *vf2;					\
-		int i;							\
-									\
-		CHECK_OBJ_NOTNULL(vsc, VSC_MAGIC);			\
-		st = vf->b;						\
-		sp.class = t;						\
-		sp.ident = vf->chunk->ident;				\
-		sp.desc = descs++;					\
-		vf2 = malloc(sizeof *vf2);				\
-		AN(vf2);						\
-		memcpy(vf2, vf, sizeof *vf2);
-
-#define VSC_F(nn,tt,ll,ff,dd,ee)					\
-		sp.ptr = &st->nn;					\
-		sp.vf = vf2;						\
-		i = iter_call(vsc, &sp);				\
-		if (i)							\
-			return(i);
-
-#define VSC_DONE(U,l,t)							\
-		return (0);						\
-	}
-
-#include "tbl/vsc_all.h"
-#undef VSC_DO
-#undef VSC_F
-#undef VSC_DONE
-
-static void
-vsc_build_pt_list(struct VSM_data *vd)
-{
-	struct vsc *vsc = vsc_setup(vd);
-	struct VSM_fantom vf;
-
-	vsc_delete_pts(vsc *vsc);
-
-	VSM_FOREACH_SAFE(&vf, vd) {
-		if (strcmp(vf.chunk->class, VSC_CLASS))
-			continue;
-		/*lint -save -e525 -e539 */
-#define VSC_F(n,t,l,f,d,e)
-#define VSC_DONE(a,b,c)
-#define VSC_DO(U,l,t)						\
-		if (!strcmp(vf.chunk->type, t))			\
-			iter_##l(vsc, &vf, VSC_desc_##l);
-#include "tbl/vsc_all.h"
-#undef VSC_F
-#undef VSC_DO
-#undef VSC_DONE
-		/*lint -restore */
-		break;
-	}
-	return (i);
-}
-#endif
-
 /*--------------------------------------------------------------------
  */
 
 static void
 vsc_add_pt(struct vsc *vsc, const char *class, const char *ident,
     const struct VSC_desc *desc, const volatile void *ptr,
-    struct VSM_fantom *vf)
+    const struct VSM_fantom *vf)
 {
 	struct vsc_pt *pt;
 
@@ -375,7 +255,8 @@ vsc_add_pt(struct vsc *vsc, const char *class, const char *ident,
 	pt->point.ident = ident;
 	pt->point.desc = desc;
 	pt->point.ptr = ptr;
-	pt->point.vf = vf;
+	pt->point.vf = &pt->vf;
+	pt->vf = *vf;
 	VTAILQ_INSERT_TAIL(&vsc->pt_list, pt, list);
 }
 
@@ -385,18 +266,14 @@ vsc_add_pt(struct vsc *vsc, const char *class, const char *ident,
 	    const struct VSC_desc *descs)				\
 	{								\
 		struct VSC_C_##l *st;					\
-		struct VSM_fantom *vf2;					\
 		const char *class = t;					\
 									\
 		CHECK_OBJ_NOTNULL(vsc, VSC_MAGIC);			\
 		st = vf->b;						\
-		vf2 = malloc(sizeof *vf2);				\
-		AN(vf2);						\
-		memcpy(vf2, vf, sizeof *vf2);
 
 #define VSC_F(nn,tt,ll,ff,dd,ee)					\
 		vsc_add_pt(vsc, class, vf->chunk->ident, descs++,	\
-		    &st->nn, vf2);
+		    &st->nn, vf);
 
 #define VSC_DONE(U,l,t)							\
 	}
@@ -409,6 +286,8 @@ vsc_add_pt(struct vsc *vsc, const char *class, const char *ident,
 /*--------------------------------------------------------------------
  */
 
+#include <stdio.h>
+
 static void
 vsc_build_pt_list(struct VSM_data *vd)
 {
@@ -432,8 +311,57 @@ vsc_build_pt_list(struct VSM_data *vd)
 #undef VSC_DONE
 		/*lint -restore */
 	}
+}
 
-	/* XXX: filter pt list */
+/*--------------------------------------------------------------------
+ */
+
+static inline int
+iter_test(const char *s1, const char *s2, int wc)
+{
+
+	if (s1 == NULL)
+		return (0);
+	if (!wc)
+		return (strcmp(s1, s2));
+	for (; *s1 != '\0' && *s1 == *s2; s1++, s2++)
+		continue;
+	return (*s1 != '\0');
+}
+
+static void
+vsc_filter_pt_list(struct VSM_data *vd)
+{
+	struct vsc *vsc = vsc_setup(vd);
+	struct vsc_sf *sf;
+	struct vsc_pt *pt, *pt2;
+	VTAILQ_HEAD(, vsc_pt)	pt_list;
+
+	if (VTAILQ_EMPTY(&vsc->sf_list))
+		return;
+
+	VTAILQ_INIT(&pt_list);
+	VTAILQ_FOREACH(sf, &vsc->sf_list, list) {
+		VTAILQ_FOREACH_SAFE(pt, &vsc->pt_list, list, pt2) {
+			if (iter_test(sf->class, pt->point.class,
+			    sf->flags & VSC_SF_CL_WC))
+				continue;
+			if (iter_test(sf->ident, pt->point.ident,
+			    sf->flags & VSC_SF_ID_WC))
+				continue;
+			if (iter_test(sf->name, pt->point.desc->name,
+			    sf->flags & VSC_SF_NM_WC))
+				continue;
+			VTAILQ_REMOVE(&vsc->pt_list, pt, list);
+			if (sf->flags & VSC_SF_EXCL) {
+				FREE_OBJ(pt);
+			} else {
+				VTAILQ_INSERT_TAIL(&pt_list, pt, list);
+			}
+		}
+	}
+	vsc_delete_pts(vsc);
+	VTAILQ_CONCAT(&vsc->pt_list, &pt_list, list);
 }
 
 /*--------------------------------------------------------------------
@@ -456,8 +384,10 @@ VSC_Iter(struct VSM_data *vd, VSC_iter_f *func, void *priv)
 			}
 		}
 		AN(vd->head);
-		func(priv, NULL);
+		/* Tell app that list will be nuked */
+		(void)func(priv, NULL);
 		vsc_build_pt_list(vd);
+		vsc_filter_pt_list(vd);
 	}
 	AN(vd->head);
 	VTAILQ_FOREACH(pt, &vsc->pt_list, list) {
diff --git a/lib/libvarnishapi/vsl.c b/lib/libvarnishapi/vsl.c
index d45e1c3..fdd93d8 100644
--- a/lib/libvarnishapi/vsl.c
+++ b/lib/libvarnishapi/vsl.c
@@ -296,7 +296,7 @@ VSL_NextSLT(struct VSM_data *vd, uint32_t **pp, uint64_t *bits)
 				if (vrm->tag == t) {
 					i = VRE_exec(vrm->re, VSL_DATA(p),
 					    VSL_LEN(p), 0, 0, NULL, 0, NULL);
-					if (i >= 0) 	/* XXX ?? */
+					if (i >= 0)	/* XXX ?? */
 						*bits |= (uintmax_t)1 << j;
 				}
 				j++;
diff --git a/lib/libvarnishapi/vsl_arg.c b/lib/libvarnishapi/vsl_arg.c
index 100d75a..068d539 100644
--- a/lib/libvarnishapi/vsl_arg.c
+++ b/lib/libvarnishapi/vsl_arg.c
@@ -160,10 +160,10 @@ vsl_ix_arg(struct VSM_data *vd, const char *opt, int arg)
 			else
 				vbit_clr(vsl->vbm_supress, i);
 		} else if (i == -2) {
-			return (vsm_diag(vd, 
+			return (vsm_diag(vd,
 			    "\"%*.*s\" matches multiple tags\n", l, l, b));
 		} else {
-			return (vsm_diag(vd, 
+			return (vsm_diag(vd,
 			    "Could not match \"%*.*s\" to any tag\n", l, l, b));
 		}
 	}
@@ -243,10 +243,10 @@ vsl_k_arg(struct VSM_data *vd, const char *opt)
 	char *end;
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	if (*opt == '\0') 
+	if (*opt == '\0')
 		return (vsm_diag(vd, "number required for -k\n"));
 	vsl->keep = strtoul(opt, &end, 10);
-	if (*end != '\0') 
+	if (*end != '\0')
 		return (vsm_diag(vd, "invalid number for -k\n"));
 	return (1);
 }
diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c
index a9f176c..18de315 100644
--- a/lib/libvarnishapi/vsm.c
+++ b/lib/libvarnishapi/vsm.c
@@ -86,6 +86,7 @@ vsm_diag(struct VSM_data *vd, const char *fmt, ...)
 	if (vd->diag == NULL)
 		vd->diag = VSB_new_auto();
 	AN(vd->diag);
+	VSB_clear(vd->diag);
 	va_start(ap, fmt);
 	VSB_vprintf(vd->diag, fmt, ap);
 	va_end(ap);
@@ -328,6 +329,8 @@ VSM_StillValid(const struct VSM_data *vd, struct VSM_fantom *vf)
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
 	AN(vf);
+	if (!vd->head)
+		return (0);
 	if (!vd->head->alloc_seq)
 		return (0);
 	if (vf->priv == vd->head->alloc_seq)



From apj at varnish-cache.org  Wed Nov 23 22:48:58 2011
From: apj at varnish-cache.org (Andreas Plesner Jacobsen)
Date: Wed, 23 Nov 2011 23:48:58 +0100
Subject: [master] 6b84197 Spelling cleanup
Message-ID: <E1RTLcg-00071O-CG@project.varnish-software.com>

commit 6b84197ee8b0763b43fddea284166298af7e0a73
Author: Andreas Plesner Jacobsen <apj at mutt.dk>
Date:   Wed Nov 23 23:48:45 2011 +0100

    Spelling cleanup

diff --git a/include/vapi/vsl.h b/include/vapi/vsl.h
index dfe5a83..db8b953 100644
--- a/include/vapi/vsl.h
+++ b/include/vapi/vsl.h
@@ -121,7 +121,7 @@ int VSL_Dispatch(struct VSM_data *vd, VSL_handler_f *func, void *priv);
 	 * Return values:
 	 *	!=0:	Non-zero return value from func()
 	 *	0:	no VSL records.
-	 *	-1:	VSL chunk was abandonned.
+	 *	-1:	VSL chunk was abandoned.
 	 *	-2:	End of file (-r) / -k arg exhausted / "done"
 	 */
 
@@ -132,7 +132,7 @@ int VSL_NextSLT(struct VSM_data *lh, uint32_t **pp, uint64_t *bitmap);
 	 * Return values:
 	 *	1:	Valid VSL record at *pp
 	 *	0:	no VSL records
-	 *	-1:	VSL cunkwas abandonned
+	 *	-1:	VSL chunk was abandoned
 	 *	-2:	End of file (-r) / -k arg exhausted / "done"
 	 */
 
diff --git a/include/vapi/vsm.h b/include/vapi/vsm.h
index 9635e8f..800cb76 100644
--- a/include/vapi/vsm.h
+++ b/include/vapi/vsm.h
@@ -100,16 +100,16 @@ int VSM_Open(struct VSM_data *vd);
 	 *	<0 on failure, VSM_Error() returns diagnostic string
 	 */
 
-int VSM_Abandonned(const struct VSM_data *vd);
+int VSM_Abandoned(const struct VSM_data *vd);
 	/*
-	 * Find out if the VSM file has been abandonned or closed and should
+	 * Find out if the VSM file has been abandoned or closed and should
 	 * be reopened.  This function calls stat(2) and should only be
 	 * used when lack of activity or invalidation of fantoms indicate
 	 * abandonment.
 	 *
 	 * Returns:
 	 *	0  No reopen needed.
-	 *	1  VSM abandonned.
+	 *	1  VSM abandoned.
 	 */
 
 void VSM_Close(struct VSM_data *vd);
@@ -132,7 +132,7 @@ int VSM__itern(const struct VSM_data *vd, struct VSM_fantom *vf);
 int VSM_StillValid(const struct VSM_data *vd, struct VSM_fantom *vf);
 	/*
 	 * This is a cheap syscall-less check to see if the fantom is still
-	 * valid.  Further checking with VSM_Abandonned() may be a good
+	 * valid.  Further checking with VSM_Abandoned() may be a good
 	 * idea.
 	 *
 	 * Return:
diff --git a/include/vapi/vsm_int.h b/include/vapi/vsm_int.h
index b45fdf7..237fabd 100644
--- a/include/vapi/vsm_int.h
+++ b/include/vapi/vsm_int.h
@@ -50,7 +50,7 @@
  *	When manager abandons VSM file, alloc_seq is set to zero, which
  *	never happens in any other circumstances.
  *
- *	If a manager is started and finds and old abandonned VSM segment
+ *	If a manager is started and finds and old abandoned VSM segment
  *	it will zero the alloc_seq in it, before replacing the file.
  *
  * Subscribers will have to monitor two things to make sure they have
diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c
index 18de315..e11fe26 100644
--- a/lib/libvarnishapi/vsm.c
+++ b/lib/libvarnishapi/vsm.c
@@ -245,7 +245,7 @@ VSM_Close(struct VSM_data *vd)
 /*--------------------------------------------------------------------*/
 
 int
-VSM_Abandonned(const struct VSM_data *vd)
+VSM_Abandoned(const struct VSM_data *vd)
 {
 	struct stat st;
 
@@ -287,7 +287,7 @@ VSM__itern(const struct VSM_data *vd, struct VSM_fantom *vf)
 	AN(vf);
 
 	if (vd->head->alloc_seq == 0)
-		return (0);	/* abandonned VSM */
+		return (0);	/* abandoned VSM */
 	else if (vf->priv != 0) {
 		if (vf->priv != vd->head->alloc_seq)
 			return (0);
@@ -312,7 +312,7 @@ VSM__itern(const struct VSM_data *vd, struct VSM_fantom *vf)
 	vf->e = (char*)vf->b + vf->chunk->len;
 
 	if (vf->priv == 0)
-		return (0);	/* abandonned VSM */
+		return (0);	/* abandoned VSM */
 	if (vf->b == vf->e)
 		return (0);	/* freed chunk */
 	AN(vf->priv);



From phk at varnish-cache.org  Thu Nov 24 09:47:31 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Thu, 24 Nov 2011 10:47:31 +0100
Subject: [master] 40c0fef Split the main stats counters into their own file.
Message-ID: <E1RTVtz-0000oE-CW@project.varnish-software.com>

commit 40c0fef63a2e90bab9ea3a5418a3670815fb00ca
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Nov 24 09:46:47 2011 +0000

    Split the main stats counters into their own file.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 4b66309..190e945 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -206,12 +206,10 @@ struct acct {
 #define L0(t, n)
 #define L1(t, n)		t n;
 #define VSC_F(n, t, l, f, e,d)	L##l(t, n)
-#define VSC_DO_MAIN
 struct dstat {
-#include "tbl/vsc_fields.h"
+#include "tbl/vsc_f_main.h"
 };
 #undef VSC_F
-#undef VSC_DO_MAIN
 #undef L0
 #undef L1
 
diff --git a/bin/varnishd/cache/cache_wrk.c b/bin/varnishd/cache/cache_wrk.c
index bfee84c..1d80805 100644
--- a/bin/varnishd/cache/cache_wrk.c
+++ b/bin/varnishd/cache/cache_wrk.c
@@ -50,11 +50,9 @@ wrk_sumstat(struct worker *w)
 	Lck_AssertHeld(&wstat_mtx);
 #define L0(n)
 #define L1(n) (VSC_C_main->n += w->stats.n)
-#define VSC_DO_MAIN
 #define VSC_F(n, t, l, f, d, e) L##l(n);
-#include "tbl/vsc_fields.h"
+#include "tbl/vsc_f_main.h"
 #undef VSC_F
-#undef VSC_DO_MAIN
 #undef L0
 #undef L1
 	memset(&w->stats, 0, sizeof w->stats);
diff --git a/include/Makefile.am b/include/Makefile.am
index 62292fc..bb00ce6 100644
--- a/include/Makefile.am
+++ b/include/Makefile.am
@@ -15,6 +15,7 @@ pkginclude_HEADERS = \
 	tbl/vrt_stv_var.h \
 	tbl/vsc_all.h \
 	tbl/vsc_fields.h \
+	tbl/vsc_f_main.h \
 	tbl/vsl_tags.h \
 	vapi/vsm.h \
 	vapi/vsm_int.h \
diff --git a/include/tbl/vsc_all.h b/include/tbl/vsc_all.h
index e12bd2f..4d7b927 100644
--- a/include/tbl/vsc_all.h
+++ b/include/tbl/vsc_all.h
@@ -36,9 +36,7 @@ VSC_DO(LCK, lck, VSC_TYPE_LCK)
 VSC_DONE(LCK, lck, VSC_TYPE_LCK)
 
 VSC_DO(MAIN, main, VSC_TYPE_MAIN)
-#define VSC_DO_MAIN
-#include "tbl/vsc_fields.h"
-#undef VSC_DO_MAIN
+#include "tbl/vsc_f_main.h"
 VSC_DONE(MAIN, main, VSC_TYPE_MAIN)
 
 VSC_DO(SMA, sma, VSC_TYPE_SMA)
diff --git a/include/tbl/vsc_f_main.h b/include/tbl/vsc_f_main.h
new file mode 100644
index 0000000..72d8842
--- /dev/null
+++ b/include/tbl/vsc_f_main.h
@@ -0,0 +1,361 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2011 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION, "")
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * Definition of all shared memory statistics below.
+ *
+ * Fields (n, t, l, f, e, d):
+ *    n - Name:		Field name, in C-source and stats programs
+ *    t - Type:		C-type, uint64_t, unless marked in 'f'
+ *    l - Local:	Local counter in worker thread.
+ *    f - Format:	Semantics of the value in this field
+ *				'a' - Accumulator (deprecated, use 'c')
+ *				'b' - Bitmap
+ *				'c' - Counter, never decreases.
+ *				'g' - Gauge, goes up and down
+ *				'i' - Integer (deprecated, use 'g')
+ *    e - Explantion:	Short explanation of field (for screen use)
+ *    d - Description:	Long explanation of field (for doc use)
+ *
+ * Please describe Gauge variables as "Number of..." to indicate that
+ * this is a snapshot, and Counter variables as "Count of" to indicate
+ * accumulative count.
+ *
+ * -----------------------
+ * NB: Cleanup in progress
+ * -----------------------
+ *
+ * Insufficient attention has caused this to become a swamp of conflicting
+ * conventions, shorthands and general mumbo-jumbo.  I'm trying to clean
+ * it up as I go over the code in other business.
+ *
+ * Please see the sessmem section for how it should look.
+ *
+ */
+
+/*---------------------------------------------------------------------
+ * Sessions
+ *    see: cache_acceptor.c and cache_pool.c
+ */
+
+VSC_F(sess_conn,		uint64_t, 1, 'c',
+    "Sessions accepted",
+	"Count of sessions succesfully accepted"
+)
+VSC_F(sess_drop,		uint64_t, 1, 'c',
+    "Sessions dropped",
+	"Count of sessions silently dropped due to lack of session memory."
+	"  See parameter 'max_sess'."
+)
+
+VSC_F(sess_fail,		uint64_t, 1, 'c',
+    "Session accept failures",
+	"Count of failures to accept TCP connection."
+	"  Either the client changed its mind, or the kernel ran out of"
+	" some resource like filedescriptors."
+)
+
+/*---------------------------------------------------------------------*/
+
+VSC_F(client_req,		uint64_t, 1, 'a',
+      "Client requests received",
+      "")
+
+VSC_F(cache_hit,		uint64_t, 1, 'a',
+      "Cache hits",
+      "Count of cache hits. "
+      "  A cache hit indicates that an object has been delivered to a"
+      "  client without fetching it from a backend server."
+)
+
+VSC_F(cache_hitpass,	uint64_t, 1, 'a',
+      "Cache hits for pass",
+      "Count of hits for pass"
+      "  A cache hit for pass indicates that Varnish is going to"
+      "  pass the request to the backend and this decision has been "
+      "  cached in it self. This counts how many times the cached "
+      "  decision is being used."
+)
+VSC_F(cache_miss,		uint64_t, 1, 'a',
+      "Cache misses",
+      "Count of misses"
+      "  A cache miss indicates the object was fetched from the"
+      "  backend before delivering it to the backend.")
+
+VSC_F(backend_conn,	uint64_t, 0, 'a',
+      "Backend conn. success",
+      "")
+
+VSC_F(backend_unhealthy,	uint64_t, 0, 'a',
+      "Backend conn. not attempted",
+      ""
+)
+VSC_F(backend_busy,	uint64_t, 0, 'a', "Backend conn. too many", "")
+VSC_F(backend_fail,	uint64_t, 0, 'a', "Backend conn. failures", "")
+VSC_F(backend_reuse,	uint64_t, 0, 'a',
+      "Backend conn. reuses",
+      "Count of backend connection reuses"
+      "  This counter is increased whenever we reuse a recycled connection.")
+VSC_F(backend_toolate,	uint64_t, 0, 'a', "Backend conn. was closed", "")
+VSC_F(backend_recycle,	uint64_t, 0, 'a',
+      "Backend conn. recycles",
+      "Count of backend connection recycles"
+      "  This counter is increased whenever we have a keep-alive"
+      "  connection that is put back into the pool of connections."
+      "  It has not yet been used, but it might be, unless the backend"
+      "  closes it.")
+VSC_F(backend_retry,	uint64_t, 0, 'a', "Backend conn. retry", "")
+
+VSC_F(fetch_head,		uint64_t, 1, 'a', "Fetch head", "")
+VSC_F(fetch_length,		uint64_t, 1, 'a', "Fetch with Length", "")
+VSC_F(fetch_chunked,		uint64_t, 1, 'a', "Fetch chunked", "")
+VSC_F(fetch_eof,		uint64_t, 1, 'a', "Fetch EOF", "")
+VSC_F(fetch_bad,		uint64_t, 1, 'a', "Fetch had bad headers", "")
+VSC_F(fetch_close,		uint64_t, 1, 'a', "Fetch wanted close", "")
+VSC_F(fetch_oldhttp,		uint64_t, 1, 'a', "Fetch pre HTTP/1.1 closed", "")
+VSC_F(fetch_zero,		uint64_t, 1, 'a', "Fetch zero len", "")
+VSC_F(fetch_failed,		uint64_t, 1, 'a', "Fetch failed", "")
+VSC_F(fetch_1xx,		uint64_t, 1, 'a', "Fetch no body (1xx)", "")
+VSC_F(fetch_204,		uint64_t, 1, 'a', "Fetch no body (204)", "")
+VSC_F(fetch_304,		uint64_t, 1, 'a', "Fetch no body (304)", "")
+
+/*---------------------------------------------------------------------
+ * Session Memory
+ *    see: cache_session.c
+ */
+
+VSC_F(sessmem_size,		uint64_t, 1, 'g',
+    "Session mem size",
+	"Bytes of memory allocated for last allocated session."
+)
+
+VSC_F(sessmem_alloc,		uint64_t, 1, 'c',
+    "Session mem allocated",
+	"Count of all allocations of session memory."
+)
+
+VSC_F(sessmem_free,		uint64_t, 1, 'c',
+    "Session mem freed",
+	"Count of all frees of session memory."
+)
+
+VSC_F(sessmem_fail,		uint64_t, 1, 'c',
+    "Session mem alloc failed",
+	"Count of session memory allocation failures."
+)
+
+VSC_F(sessmem_limit,		uint64_t, 1, 'c',
+    "Session mem alloc limited",
+	"Count of session memory allocations blocked by limit (max_sess)."
+)
+
+/*---------------------------------------------------------------------
+ * Pools, threads, and sessions
+ *    see: cache_pool.c
+ *
+ */
+
+VSC_F(pools,			uint64_t, 1, 'g',
+    "Number of thread pools",
+	"Number of thread pools.  See also param wthread_pools."
+	"  NB: Presently pools cannot be removed once created."
+)
+
+VSC_F(threads,			uint64_t, 1, 'g',
+    "Total number of threads",
+	"Number of threads in all pools."
+	"  See also params thread_pools, thread_pool_min & thread_pool_max."
+)
+
+VSC_F(threads_limited,		uint64_t, 1, 'c',
+    "Threads hit max",
+	"Number of times more threads were needed, but limit was reached"
+	" in a thread pool."
+	"  See also param thread_pool_max."
+)
+
+VSC_F(threads_created,		uint64_t, 1, 'c',
+    "Threads created",
+	"Total number of threads created in all pools."
+)
+
+VSC_F(threads_destroyed,	uint64_t, 1, 'c',
+    "Threads destoryed",
+	"Total number of threads destroyed in all pools."
+)
+
+VSC_F(threads_failed,		uint64_t, 1, 'c',
+    "Thread creation failed",
+	"Number of times creating a thread failed."
+	"  See VSL::Debug for diagnostics."
+	"  See also param thread_fail_delay."
+)
+
+VSC_F(thread_queue_len,		uint64_t, 1, 'g',
+    "Length of session queue",
+	"Length of session queue waiting for threads."
+	"  NB: Only updates once per second."
+	"  See also param queue_max."
+)
+
+VSC_F(sess_queued,		uint64_t, 1, 'c',
+    "Sessions queued for thread",
+	"Number of times session was queued waiting for a thread."
+	"  See also param queue_max."
+)
+
+VSC_F(sess_dropped,		uint64_t, 1, 'c',
+    "Sessions dropped for thread",
+	"Number of times session was dropped because the queue were too"
+	" long already."
+	"  See also param queue_max."
+)
+
+/*---------------------------------------------------------------------*/
+
+VSC_F(n_sess_mem,		uint64_t, 0, 'i', "N struct sess_mem", "")
+VSC_F(n_sess,			uint64_t, 0, 'i', "N struct sess", "")
+VSC_F(n_object,			uint64_t, 1, 'i', "N struct object", "")
+VSC_F(n_vampireobject,		uint64_t, 1, 'i', "N unresurrected objects", "")
+VSC_F(n_objectcore,		uint64_t, 1, 'i', "N struct objectcore", "")
+VSC_F(n_objecthead,		uint64_t, 1, 'i', "N struct objecthead", "")
+VSC_F(n_waitinglist,		uint64_t, 1, 'i', "N struct waitinglist", "")
+
+VSC_F(n_vbc,		uint64_t, 0, 'i', "N struct vbc", "")
+
+VSC_F(n_backend,		uint64_t, 0, 'i', "N backends", "")
+
+VSC_F(n_expired,		uint64_t, 0, 'i', "N expired objects", "")
+VSC_F(n_lru_nuked,		uint64_t, 0, 'i', "N LRU nuked objects", "")
+VSC_F(n_lru_moved,		uint64_t, 0, 'i', "N LRU moved objects", "")
+
+VSC_F(losthdr,		uint64_t, 0, 'a', "HTTP header overflows", "")
+
+VSC_F(n_objsendfile,	uint64_t, 0, 'a', "Objects sent with sendfile",
+      "The number of objects sent with the sendfile system call. If enabled "
+      "sendfile will be used on object larger than a certain size.")
+VSC_F(n_objwrite,		uint64_t, 0, 'a', "Objects sent with write",
+      "The number of objects sent with regular write calls."
+      "Writes are used when the objects are too small for sendfile "
+      "or if the sendfile call has been disabled")
+VSC_F(n_objoverflow,	uint64_t, 1, 'a',
+					"Objects overflowing workspace", "")
+
+VSC_F(s_sess,		uint64_t, 1, 'a', "Total Sessions", "")
+VSC_F(s_req,		uint64_t, 1, 'a', "Total Requests", "")
+VSC_F(s_pipe,		uint64_t, 1, 'a', "Total pipe", "")
+VSC_F(s_pass,		uint64_t, 1, 'a', "Total pass", "")
+VSC_F(s_fetch,		uint64_t, 1, 'a', "Total fetch", "")
+VSC_F(s_hdrbytes,		uint64_t, 1, 'a', "Total header bytes", "")
+VSC_F(s_bodybytes,		uint64_t, 1, 'a', "Total body bytes", "")
+
+VSC_F(sess_closed,		uint64_t, 1, 'a', "Session Closed", "")
+VSC_F(sess_pipeline,	uint64_t, 1, 'a', "Session Pipeline", "")
+VSC_F(sess_readahead,	uint64_t, 1, 'a', "Session Read Ahead", "")
+VSC_F(sess_linger,		uint64_t, 1, 'a', "Session Linger", "")
+VSC_F(sess_herd,		uint64_t, 1, 'a', "Session herd", "")
+
+VSC_F(shm_records,		uint64_t, 0, 'a', "SHM records", "")
+VSC_F(shm_writes,		uint64_t, 0, 'a', "SHM writes", "")
+VSC_F(shm_flushes,		uint64_t, 0, 'a', "SHM flushes due to overflow", "")
+VSC_F(shm_cont,		uint64_t, 0, 'a', "SHM MTX contention", "")
+VSC_F(shm_cycles,		uint64_t, 0, 'a', "SHM cycles through buffer", "")
+
+VSC_F(sms_nreq,		uint64_t, 0, 'a', "SMS allocator requests", "")
+VSC_F(sms_nobj,		uint64_t, 0, 'i', "SMS outstanding allocations", "")
+VSC_F(sms_nbytes,		uint64_t, 0, 'i', "SMS outstanding bytes", "")
+VSC_F(sms_balloc,		uint64_t, 0, 'i', "SMS bytes allocated", "")
+VSC_F(sms_bfree,		uint64_t, 0, 'i', "SMS bytes freed", "")
+
+VSC_F(backend_req,		uint64_t, 0, 'a', "Backend requests made", "")
+
+VSC_F(n_vcl,		uint64_t, 0, 'a', "N vcl total", "")
+VSC_F(n_vcl_avail,		uint64_t, 0, 'a', "N vcl available", "")
+VSC_F(n_vcl_discard,	uint64_t, 0, 'a', "N vcl discarded", "")
+
+/**********************************************************************/
+
+VSC_F(bans,			uint64_t, 0, 'g',
+   "Count of bans",
+	"Number of all bans in system, including bans superseded"
+	" by newer bans and bans already checked by the ban-lurker."
+)
+VSC_F(bans_gone,		uint64_t, 0, 'g',
+    "Number of bans marked 'gone'",
+	"Number of bans which are no longer active, either because they"
+	" got checked by the ban-lurker or superseded by newer identical bans."
+)
+VSC_F(bans_req,			uint64_t, 0, 'g',
+    "Number of bans using req.*",
+	"Number of bans which use req.* variables.  These bans can not"
+	" be washed by the ban-lurker."
+)
+VSC_F(bans_added,		uint64_t, 0, 'c',
+    "Bans added",
+	"Counter of bans added to ban list."
+)
+VSC_F(bans_deleted,		uint64_t, 0, 'c',
+    "Bans deleted",
+	"Counter of bans deleted from ban list."
+)
+
+VSC_F(bans_tested,		uint64_t, 0, 'c',
+    "Bans tested against objects",
+	"Count of how many bans and objects have been tested against"
+	" each other."
+)
+VSC_F(bans_tests_tested,	uint64_t, 0, 'c',
+    "Ban tests tested against objects",
+	"Count of how many tests and objects have been tested against"
+	" each other.  'ban req.url == foo && req.http.host == bar'"
+	" counts as one in 'bans_tested' and as two in 'bans_tests_tested'"
+)
+VSC_F(bans_dups,		uint64_t, 0, 'c',
+    "Bans superseded by other bans",
+	"Count of bans replaced by later identical bans."
+)
+
+/**********************************************************************/
+
+VSC_F(hcb_nolock,		uint64_t, 0, 'a', "HCB Lookups without lock", "")
+VSC_F(hcb_lock,		uint64_t, 0, 'a', "HCB Lookups with lock", "")
+VSC_F(hcb_insert,		uint64_t, 0, 'a', "HCB Inserts", "")
+
+VSC_F(esi_errors,		uint64_t, 0, 'a', "ESI parse errors (unlock)", "")
+VSC_F(esi_warnings,		uint64_t, 0, 'a', "ESI parse warnings (unlock)", "")
+VSC_F(client_drop_late,	uint64_t, 0, 'a', "Connection dropped late", "")
+VSC_F(uptime,		uint64_t, 0, 'a', "Client uptime", "")
+
+VSC_F(dir_dns_lookups,	uint64_t, 0, 'a', "DNS director lookups", "")
+VSC_F(dir_dns_failed,	uint64_t, 0, 'a', "DNS director failed lookups", "")
+VSC_F(dir_dns_hit,		uint64_t, 0, 'a', "DNS director cached lookups hit", "")
+VSC_F(dir_dns_cache_full,	uint64_t, 0, 'a', "DNS director full dnscache", "")
+
+VSC_F(vmods,		uint64_t, 0, 'i', "Loaded VMODs", "")
+
+VSC_F(n_gzip,			uint64_t, 0, 'a', "Gzip operations", "")
+VSC_F(n_gunzip,			uint64_t, 0, 'a', "Gunzip operations", "")
diff --git a/include/tbl/vsc_fields.h b/include/tbl/vsc_fields.h
index 738703c..60cde5d 100644
--- a/include/tbl/vsc_fields.h
+++ b/include/tbl/vsc_fields.h
@@ -59,315 +59,6 @@
 
 /**********************************************************************/
 
-#ifdef VSC_DO_MAIN
-
-/*---------------------------------------------------------------------
- * Sessions
- *    see: cache_acceptor.c and cache_pool.c
- */
-
-VSC_F(sess_conn,		uint64_t, 1, 'c',
-    "Sessions accepted",
-	"Count of sessions succesfully accepted"
-)
-VSC_F(sess_drop,		uint64_t, 1, 'c',
-    "Sessions dropped",
-	"Count of sessions silently dropped due to lack of session memory."
-	"  See parameter 'max_sess'."
-)
-
-VSC_F(sess_fail,		uint64_t, 1, 'c',
-    "Session accept failures",
-	"Count of failures to accept TCP connection."
-	"  Either the client changed its mind, or the kernel ran out of"
-	" some resource like filedescriptors."
-)
-
-/*---------------------------------------------------------------------*/
-
-VSC_F(client_req,		uint64_t, 1, 'a',
-      "Client requests received",
-      "")
-
-VSC_F(cache_hit,		uint64_t, 1, 'a',
-      "Cache hits",
-      "Count of cache hits. "
-      "  A cache hit indicates that an object has been delivered to a"
-      "  client without fetching it from a backend server."
-)
-
-VSC_F(cache_hitpass,	uint64_t, 1, 'a',
-      "Cache hits for pass",
-      "Count of hits for pass"
-      "  A cache hit for pass indicates that Varnish is going to"
-      "  pass the request to the backend and this decision has been "
-      "  cached in it self. This counts how many times the cached "
-      "  decision is being used."
-)
-VSC_F(cache_miss,		uint64_t, 1, 'a',
-      "Cache misses",
-      "Count of misses"
-      "  A cache miss indicates the object was fetched from the"
-      "  backend before delivering it to the backend.")
-
-VSC_F(backend_conn,	uint64_t, 0, 'a',
-      "Backend conn. success",
-      "")
-
-VSC_F(backend_unhealthy,	uint64_t, 0, 'a',
-      "Backend conn. not attempted",
-      ""
-)
-VSC_F(backend_busy,	uint64_t, 0, 'a', "Backend conn. too many", "")
-VSC_F(backend_fail,	uint64_t, 0, 'a', "Backend conn. failures", "")
-VSC_F(backend_reuse,	uint64_t, 0, 'a',
-      "Backend conn. reuses",
-      "Count of backend connection reuses"
-      "  This counter is increased whenever we reuse a recycled connection.")
-VSC_F(backend_toolate,	uint64_t, 0, 'a', "Backend conn. was closed", "")
-VSC_F(backend_recycle,	uint64_t, 0, 'a',
-      "Backend conn. recycles",
-      "Count of backend connection recycles"
-      "  This counter is increased whenever we have a keep-alive"
-      "  connection that is put back into the pool of connections."
-      "  It has not yet been used, but it might be, unless the backend"
-      "  closes it.")
-VSC_F(backend_retry,	uint64_t, 0, 'a', "Backend conn. retry", "")
-
-VSC_F(fetch_head,		uint64_t, 1, 'a', "Fetch head", "")
-VSC_F(fetch_length,		uint64_t, 1, 'a', "Fetch with Length", "")
-VSC_F(fetch_chunked,		uint64_t, 1, 'a', "Fetch chunked", "")
-VSC_F(fetch_eof,		uint64_t, 1, 'a', "Fetch EOF", "")
-VSC_F(fetch_bad,		uint64_t, 1, 'a', "Fetch had bad headers", "")
-VSC_F(fetch_close,		uint64_t, 1, 'a', "Fetch wanted close", "")
-VSC_F(fetch_oldhttp,		uint64_t, 1, 'a', "Fetch pre HTTP/1.1 closed", "")
-VSC_F(fetch_zero,		uint64_t, 1, 'a', "Fetch zero len", "")
-VSC_F(fetch_failed,		uint64_t, 1, 'a', "Fetch failed", "")
-VSC_F(fetch_1xx,		uint64_t, 1, 'a', "Fetch no body (1xx)", "")
-VSC_F(fetch_204,		uint64_t, 1, 'a', "Fetch no body (204)", "")
-VSC_F(fetch_304,		uint64_t, 1, 'a', "Fetch no body (304)", "")
-
-/*---------------------------------------------------------------------
- * Session Memory
- *    see: cache_session.c
- */
-
-VSC_F(sessmem_size,		uint64_t, 1, 'g',
-    "Session mem size",
-	"Bytes of memory allocated for last allocated session."
-)
-
-VSC_F(sessmem_alloc,		uint64_t, 1, 'c',
-    "Session mem allocated",
-	"Count of all allocations of session memory."
-)
-
-VSC_F(sessmem_free,		uint64_t, 1, 'c',
-    "Session mem freed",
-	"Count of all frees of session memory."
-)
-
-VSC_F(sessmem_fail,		uint64_t, 1, 'c',
-    "Session mem alloc failed",
-	"Count of session memory allocation failures."
-)
-
-VSC_F(sessmem_limit,		uint64_t, 1, 'c',
-    "Session mem alloc limited",
-	"Count of session memory allocations blocked by limit (max_sess)."
-)
-
-/*---------------------------------------------------------------------
- * Pools, threads, and sessions
- *    see: cache_pool.c
- *
- */
-
-VSC_F(pools,			uint64_t, 1, 'g',
-    "Number of thread pools",
-	"Number of thread pools.  See also param wthread_pools."
-	"  NB: Presently pools cannot be removed once created."
-)
-
-VSC_F(threads,			uint64_t, 1, 'g',
-    "Total number of threads",
-	"Number of threads in all pools."
-	"  See also params thread_pools, thread_pool_min & thread_pool_max."
-)
-
-VSC_F(threads_limited,		uint64_t, 1, 'c',
-    "Threads hit max",
-	"Number of times more threads were needed, but limit was reached"
-	" in a thread pool."
-	"  See also param thread_pool_max."
-)
-
-VSC_F(threads_created,		uint64_t, 1, 'c',
-    "Threads created",
-	"Total number of threads created in all pools."
-)
-
-VSC_F(threads_destroyed,	uint64_t, 1, 'c',
-    "Threads destoryed",
-	"Total number of threads destroyed in all pools."
-)
-
-VSC_F(threads_failed,		uint64_t, 1, 'c',
-    "Thread creation failed",
-	"Number of times creating a thread failed."
-	"  See VSL::Debug for diagnostics."
-	"  See also param thread_fail_delay."
-)
-
-VSC_F(thread_queue_len,		uint64_t, 1, 'g',
-    "Length of session queue",
-	"Length of session queue waiting for threads."
-	"  NB: Only updates once per second."
-	"  See also param queue_max."
-)
-
-VSC_F(sess_queued,		uint64_t, 1, 'c',
-    "Sessions queued for thread",
-	"Number of times session was queued waiting for a thread."
-	"  See also param queue_max."
-)
-
-VSC_F(sess_dropped,		uint64_t, 1, 'c',
-    "Sessions dropped for thread",
-	"Number of times session was dropped because the queue were too"
-	" long already."
-	"  See also param queue_max."
-)
-
-/*---------------------------------------------------------------------*/
-
-VSC_F(n_sess_mem,		uint64_t, 0, 'i', "N struct sess_mem", "")
-VSC_F(n_sess,			uint64_t, 0, 'i', "N struct sess", "")
-VSC_F(n_object,			uint64_t, 1, 'i', "N struct object", "")
-VSC_F(n_vampireobject,		uint64_t, 1, 'i', "N unresurrected objects", "")
-VSC_F(n_objectcore,		uint64_t, 1, 'i', "N struct objectcore", "")
-VSC_F(n_objecthead,		uint64_t, 1, 'i', "N struct objecthead", "")
-VSC_F(n_waitinglist,		uint64_t, 1, 'i', "N struct waitinglist", "")
-
-VSC_F(n_vbc,		uint64_t, 0, 'i', "N struct vbc", "")
-
-VSC_F(n_backend,		uint64_t, 0, 'i', "N backends", "")
-
-VSC_F(n_expired,		uint64_t, 0, 'i', "N expired objects", "")
-VSC_F(n_lru_nuked,		uint64_t, 0, 'i', "N LRU nuked objects", "")
-VSC_F(n_lru_moved,		uint64_t, 0, 'i', "N LRU moved objects", "")
-
-VSC_F(losthdr,		uint64_t, 0, 'a', "HTTP header overflows", "")
-
-VSC_F(n_objsendfile,	uint64_t, 0, 'a', "Objects sent with sendfile",
-      "The number of objects sent with the sendfile system call. If enabled "
-      "sendfile will be used on object larger than a certain size.")
-VSC_F(n_objwrite,		uint64_t, 0, 'a', "Objects sent with write",
-      "The number of objects sent with regular write calls."
-      "Writes are used when the objects are too small for sendfile "
-      "or if the sendfile call has been disabled")
-VSC_F(n_objoverflow,	uint64_t, 1, 'a',
-					"Objects overflowing workspace", "")
-
-VSC_F(s_sess,		uint64_t, 1, 'a', "Total Sessions", "")
-VSC_F(s_req,		uint64_t, 1, 'a', "Total Requests", "")
-VSC_F(s_pipe,		uint64_t, 1, 'a', "Total pipe", "")
-VSC_F(s_pass,		uint64_t, 1, 'a', "Total pass", "")
-VSC_F(s_fetch,		uint64_t, 1, 'a', "Total fetch", "")
-VSC_F(s_hdrbytes,		uint64_t, 1, 'a', "Total header bytes", "")
-VSC_F(s_bodybytes,		uint64_t, 1, 'a', "Total body bytes", "")
-
-VSC_F(sess_closed,		uint64_t, 1, 'a', "Session Closed", "")
-VSC_F(sess_pipeline,	uint64_t, 1, 'a', "Session Pipeline", "")
-VSC_F(sess_readahead,	uint64_t, 1, 'a', "Session Read Ahead", "")
-VSC_F(sess_linger,		uint64_t, 1, 'a', "Session Linger", "")
-VSC_F(sess_herd,		uint64_t, 1, 'a', "Session herd", "")
-
-VSC_F(shm_records,		uint64_t, 0, 'a', "SHM records", "")
-VSC_F(shm_writes,		uint64_t, 0, 'a', "SHM writes", "")
-VSC_F(shm_flushes,		uint64_t, 0, 'a', "SHM flushes due to overflow", "")
-VSC_F(shm_cont,		uint64_t, 0, 'a', "SHM MTX contention", "")
-VSC_F(shm_cycles,		uint64_t, 0, 'a', "SHM cycles through buffer", "")
-
-VSC_F(sms_nreq,		uint64_t, 0, 'a', "SMS allocator requests", "")
-VSC_F(sms_nobj,		uint64_t, 0, 'i', "SMS outstanding allocations", "")
-VSC_F(sms_nbytes,		uint64_t, 0, 'i', "SMS outstanding bytes", "")
-VSC_F(sms_balloc,		uint64_t, 0, 'i', "SMS bytes allocated", "")
-VSC_F(sms_bfree,		uint64_t, 0, 'i', "SMS bytes freed", "")
-
-VSC_F(backend_req,		uint64_t, 0, 'a', "Backend requests made", "")
-
-VSC_F(n_vcl,		uint64_t, 0, 'a', "N vcl total", "")
-VSC_F(n_vcl_avail,		uint64_t, 0, 'a', "N vcl available", "")
-VSC_F(n_vcl_discard,	uint64_t, 0, 'a', "N vcl discarded", "")
-
-/**********************************************************************/
-
-VSC_F(bans,			uint64_t, 0, 'g',
-   "Count of bans",
-	"Number of all bans in system, including bans superseded"
-	" by newer bans and bans already checked by the ban-lurker."
-)
-VSC_F(bans_gone,		uint64_t, 0, 'g',
-    "Number of bans marked 'gone'",
-	"Number of bans which are no longer active, either because they"
-	" got checked by the ban-lurker or superseded by newer identical bans."
-)
-VSC_F(bans_req,			uint64_t, 0, 'g',
-    "Number of bans using req.*",
-	"Number of bans which use req.* variables.  These bans can not"
-	" be washed by the ban-lurker."
-)
-VSC_F(bans_added,		uint64_t, 0, 'c',
-    "Bans added",
-	"Counter of bans added to ban list."
-)
-VSC_F(bans_deleted,		uint64_t, 0, 'c',
-    "Bans deleted",
-	"Counter of bans deleted from ban list."
-)
-
-VSC_F(bans_tested,		uint64_t, 0, 'c',
-    "Bans tested against objects",
-	"Count of how many bans and objects have been tested against"
-	" each other."
-)
-VSC_F(bans_tests_tested,	uint64_t, 0, 'c',
-    "Ban tests tested against objects",
-	"Count of how many tests and objects have been tested against"
-	" each other.  'ban req.url == foo && req.http.host == bar'"
-	" counts as one in 'bans_tested' and as two in 'bans_tests_tested'"
-)
-VSC_F(bans_dups,		uint64_t, 0, 'c',
-    "Bans superseded by other bans",
-	"Count of bans replaced by later identical bans."
-)
-
-/**********************************************************************/
-
-VSC_F(hcb_nolock,		uint64_t, 0, 'a', "HCB Lookups without lock", "")
-VSC_F(hcb_lock,		uint64_t, 0, 'a', "HCB Lookups with lock", "")
-VSC_F(hcb_insert,		uint64_t, 0, 'a', "HCB Inserts", "")
-
-VSC_F(esi_errors,		uint64_t, 0, 'a', "ESI parse errors (unlock)", "")
-VSC_F(esi_warnings,		uint64_t, 0, 'a', "ESI parse warnings (unlock)", "")
-VSC_F(client_drop_late,	uint64_t, 0, 'a', "Connection dropped late", "")
-VSC_F(uptime,		uint64_t, 0, 'a', "Client uptime", "")
-
-VSC_F(dir_dns_lookups,	uint64_t, 0, 'a', "DNS director lookups", "")
-VSC_F(dir_dns_failed,	uint64_t, 0, 'a', "DNS director failed lookups", "")
-VSC_F(dir_dns_hit,		uint64_t, 0, 'a', "DNS director cached lookups hit", "")
-VSC_F(dir_dns_cache_full,	uint64_t, 0, 'a', "DNS director full dnscache", "")
-
-VSC_F(vmods,		uint64_t, 0, 'i', "Loaded VMODs", "")
-
-VSC_F(n_gzip,			uint64_t, 0, 'a', "Gzip operations", "")
-VSC_F(n_gunzip,			uint64_t, 0, 'a', "Gunzip operations", "")
-
-#endif
-
-/**********************************************************************/
-
 #ifdef VSC_DO_LCK
 
 VSC_F(creat,		uint64_t, 0, 'a', "Created locks", "")
diff --git a/man/vsc2rst.c b/man/vsc2rst.c
index 18250d8..4ea94be 100644
--- a/man/vsc2rst.c
+++ b/man/vsc2rst.c
@@ -28,9 +28,7 @@ int main(int argc, char **argv)
 	P("MAIN COUNTERS");
 	P("=============");
 	P("");
-#define VSC_DO_MAIN
-#include "tbl/vsc_fields.h"
-#undef VSC_DO_MAIN
+#include "tbl/vsc_f_main.h"
 
 	P("");
 	P("LOCK COUNTERS");



From phk at varnish-cache.org  Thu Nov 24 10:02:16 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Thu, 24 Nov 2011 11:02:16 +0100
Subject: [master] d13c674 Add stats counters for VSM usage/overflows.
Message-ID: <E1RTW8G-0002ZM-6C@project.varnish-software.com>

commit d13c674d347663ec812c846f637b31ec48867ab5
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Nov 24 10:01:23 2011 +0000

    Add stats counters for VSM usage/overflows.
    
    Various minor polish to VSM area.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 190e945..5e813ca 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -850,7 +850,7 @@ int SES_Schedule(struct sess *sp);
 
 /* cache_shmlog.c */
 extern struct VSC_C_main *VSC_C_main;
-void VSL_Init(void);
+void VSM_Init(void);
 void *VSM_Alloc(unsigned size, const char *class, const char *type,
     const char *ident);
 void VSM_Free(void *ptr);
diff --git a/bin/varnishd/cache/cache_main.c b/bin/varnishd/cache/cache_main.c
index 1f876cd..fd31de4 100644
--- a/bin/varnishd/cache/cache_main.c
+++ b/bin/varnishd/cache/cache_main.c
@@ -103,7 +103,7 @@ child_main(void)
 
 	THR_SetName("cache-main");
 
-	VSL_Init();	/* First, LCK needs it. */
+	VSM_Init();	/* First, LCK needs it. */
 
 	LCK_Init();	/* Second, locking */
 
diff --git a/bin/varnishd/cache/cache_shmlog.c b/bin/varnishd/cache/cache_shmlog.c
index 01f6bce..893525b 100644
--- a/bin/varnishd/cache/cache_shmlog.c
+++ b/bin/varnishd/cache/cache_shmlog.c
@@ -38,6 +38,7 @@
 #include "cache_backend.h"	// For w->vbc
 
 #include "vmb.h"
+#include "vtim.h"
 
 /* These cannot be struct lock, which depends on vsm/vsl working */
 static pthread_mutex_t vsl_mtx;
@@ -299,10 +300,26 @@ WSLB(struct worker *w, enum VSL_tag_e tag, const char *fmt, ...)
 
 /*--------------------------------------------------------------------*/
 
+static void *
+vsm_cleaner(void *priv)
+{
+	(void)priv;
+	THR_SetName("vsm_cleaner");
+	while (1) {
+		AZ(pthread_mutex_lock(&vsm_mtx));
+		VSM_common_cleaner(heritage.vsm, VSC_C_main);
+		AZ(pthread_mutex_unlock(&vsm_mtx));
+		VTIM_sleep(1.1);
+	}
+}
+
+/*--------------------------------------------------------------------*/
+
 void
-VSL_Init(void)
+VSM_Init(void)
 {
 	uint32_t *vsl_log_start;
+	pthread_t tp;
 
 	AZ(pthread_mutex_init(&vsl_mtx, NULL));
 	AZ(pthread_mutex_init(&vsm_mtx, NULL));
@@ -328,6 +345,8 @@ VSL_Init(void)
 	// VSM_head->starttime = (intmax_t)VTIM_real();
 	memset(VSC_C_main, 0, sizeof *VSC_C_main);
 	// VSM_head->child_pid = getpid();
+
+	AZ(pthread_create(&tp, NULL, vsm_cleaner, NULL));
 }
 
 /*--------------------------------------------------------------------*/
diff --git a/bin/varnishd/common/common.h b/bin/varnishd/common/common.h
index 0f25d31..2ff5623 100644
--- a/bin/varnishd/common/common.h
+++ b/bin/varnishd/common/common.h
@@ -69,12 +69,14 @@ void mgt_child_inherit(int fd, const char *what);
 
 /* vsm.c */
 struct vsm_sc;
+struct VSC_C_main;
 struct vsm_sc *VSM_common_new(void *ptr, ssize_t len);
 void *VSM_common_alloc(struct vsm_sc *sc, ssize_t size,
     const char *class, const char *type, const char *ident);
 void VSM_common_free(struct vsm_sc *sc, void *ptr);
 void VSM_common_delete(struct vsm_sc **sc);
 void VSM_common_copy(struct vsm_sc *to, const struct vsm_sc *from);
+void VSM_common_cleaner(struct vsm_sc *sc, struct VSC_C_main *stats);
 
 /*---------------------------------------------------------------------
  * Generic power-2 rounding macros
diff --git a/bin/varnishd/common/common_vsm.c b/bin/varnishd/common/common_vsm.c
index 3853ed8..e53972c 100644
--- a/bin/varnishd/common/common_vsm.c
+++ b/bin/varnishd/common/common_vsm.c
@@ -43,6 +43,7 @@
 #include "common.h"
 
 #include "vapi/vsm_int.h"
+#include "vapi/vsc_int.h"
 #include "vmb.h"
 #include "vtim.h"
 
@@ -69,6 +70,11 @@ struct vsm_sc {
 	VTAILQ_HEAD(,vsm_range)		r_cooling;
 	VTAILQ_HEAD(,vsm_range)		r_free;
 	VTAILQ_HEAD(,vsm_range)		r_bogus;
+	uint64_t			g_free;
+	uint64_t			g_used;
+	uint64_t			g_cooling;
+	uint64_t			g_overflow;
+	uint64_t			c_overflow;
 };
 
 /*--------------------------------------------------------------------
@@ -148,10 +154,37 @@ VSM_common_new(void *p, ssize_t l)
 	vr->off = RUP2(sizeof(*sc->head), 16);
 	vr->len = RDN2(l - vr->off, 16);
 	VTAILQ_INSERT_TAIL(&sc->r_free, vr, list);
+	sc->g_free = vr->len;
 	return (sc);
 }
 
 /*--------------------------------------------------------------------
+ * Move from cooling list to free list
+ */
+
+void
+VSM_common_cleaner(struct vsm_sc *sc, struct VSC_C_main *stats)
+{
+	double now = VTIM_real();
+	struct vsm_range *vr, *vr2;
+
+	CHECK_OBJ_NOTNULL(sc, VSM_SC_MAGIC);
+
+	/* Move cooled off stuff to free list */
+	VTAILQ_FOREACH_SAFE(vr, &sc->r_cooling, list, vr2) {
+		if (vr->cool > now)
+			break;
+		VTAILQ_REMOVE(&sc->r_cooling, vr, list);
+		vsm_common_insert_free(sc, vr);
+	}
+	stats->vsm_free = sc->g_free;
+	stats->vsm_used = sc->g_used;
+	stats->vsm_cooling = sc->g_cooling;
+	stats->vsm_overflow = sc->g_overflow;
+	stats->vsm_overflowed = sc->c_overflow;
+}
+
+/*--------------------------------------------------------------------
  * Allocate a chunk from VSM
  */
 
@@ -160,7 +193,6 @@ VSM_common_alloc(struct vsm_sc *sc, ssize_t size,
     const char *class, const char *type, const char *ident)
 {
 	struct vsm_range *vr, *vr2, *vr3;
-	double now = VTIM_real();
 	unsigned l1, l2;
 
 	CHECK_OBJ_NOTNULL(sc, VSM_SC_MAGIC);
@@ -174,14 +206,6 @@ VSM_common_alloc(struct vsm_sc *sc, ssize_t size,
 	AN(ident);
 	assert(strlen(ident) < sizeof(vr->chunk->ident));
 
-	/* Move cooled off stuff to free list */
-	VTAILQ_FOREACH_SAFE(vr, &sc->r_cooling, list, vr2) {
-		if (vr->cool > now)
-			break;
-		VTAILQ_REMOVE(&sc->r_cooling, vr, list);
-		vsm_common_insert_free(sc, vr);
-	}
-
 	l1 = RUP2(size + sizeof(struct VSM_chunk), 16);
 	l2 = RUP2(size + 2 * sizeof(struct VSM_chunk), 16);
 
@@ -213,12 +237,15 @@ VSM_common_alloc(struct vsm_sc *sc, ssize_t size,
 		AN(vr);
 		vr->ptr = malloc(size);
 		AN(vr->ptr);
+		vr->len = size;
 		VTAILQ_INSERT_TAIL(&sc->r_bogus, vr, list);
-		/* XXX: log + stats */
+		sc->g_overflow += vr->len;
+		sc->c_overflow += vr->len;
 		return (vr->ptr);
 	}
 
-	/* XXX: stats ? */
+	sc->g_free -= vr->len;
+	sc->g_used += vr->len;
 
 	/* Zero the entire allocation, to avoid garbage confusing readers */
 	memset(sc->b + vr->off, 0, vr->len);
@@ -263,7 +290,10 @@ VSM_common_free(struct vsm_sc *sc, void *ptr)
 	VTAILQ_FOREACH(vr, &sc->r_used, list) {
 		if (vr->ptr != ptr)
 			continue;
-		/* XXX: stats ? */
+
+		sc->g_used -= vr->len;
+		sc->g_cooling += vr->len;
+
 		vr2 = VTAILQ_NEXT(vr, list);
 		VTAILQ_REMOVE(&sc->r_used, vr, list);
 		VTAILQ_INSERT_TAIL(&sc->r_cooling, vr, list);
@@ -278,15 +308,18 @@ VSM_common_free(struct vsm_sc *sc, void *ptr)
 		VWMB();
 		return;
 	}
+
 	/* Look in bogus list, free */
 	VTAILQ_FOREACH(vr, &sc->r_bogus, list) {
-		if (vr->ptr == ptr) {
-			VTAILQ_REMOVE(&sc->r_bogus, vr, list);
-			FREE_OBJ(vr);
-			/* XXX: stats ? */
-			free(ptr);
-			return;
-		}
+		if (vr->ptr != ptr)
+			continue;
+
+		sc->g_overflow -= vr->len;
+
+		VTAILQ_REMOVE(&sc->r_bogus, vr, list);
+		FREE_OBJ(vr);
+		free(ptr);
+		return;
 	}
 	/* Panic */
 	assert(ptr == NULL);
@@ -326,7 +359,7 @@ VSM_common_delete(struct vsm_sc **scp)
 }
 
 /*--------------------------------------------------------------------
- * Copy one VSM to another
+ * Copy all chunks in one VSM segment to another VSM segment
  */
 
 void
diff --git a/include/tbl/vsc_f_main.h b/include/tbl/vsc_f_main.h
index 72d8842..a04767d 100644
--- a/include/tbl/vsc_f_main.h
+++ b/include/tbl/vsc_f_main.h
@@ -359,3 +359,38 @@ VSC_F(vmods,		uint64_t, 0, 'i', "Loaded VMODs", "")
 
 VSC_F(n_gzip,			uint64_t, 0, 'a', "Gzip operations", "")
 VSC_F(n_gunzip,			uint64_t, 0, 'a', "Gunzip operations", "")
+
+/**********************************************************************/
+
+VSC_F(vsm_free,			uint64_t, 0, 'g',
+    "Free VSM space",
+	"Number of bytes free in the shared memory used to communicate"
+	" with tools like varnishstat, varnishlog etc."
+)
+
+VSC_F(vsm_used,			uint64_t, 0, 'g',
+    "Used VSM space",
+	"Number of bytes used in the shared memory used to communicate"
+	" with tools like varnishstat, varnishlog etc."
+)
+
+VSC_F(vsm_cooling,		uint64_t, 0, 'g',
+    "Cooling VSM space",
+	"Number of bytes which will soon (max 1 minute) be freed"
+	" in the shared memory used to communicate"
+	" with tools like varnishstat, varnishlog etc."
+)
+
+VSC_F(vsm_overflow,		uint64_t, 0, 'g',
+    "Overflow VSM space",
+	"Number of bytes which does not fit"
+	" in the shared memory used to communicate"
+	" with tools like varnishstat, varnishlog etc."
+)
+
+VSC_F(vsm_overflowed,		uint64_t, 0, 'c',
+    "Overflowed VSM space",
+	"Total number of bytes which did not fit"
+	" in the shared memory used to communicate"
+	" with tools like varnishstat, varnishlog etc."
+)
diff --git a/include/vapi/vsm_int.h b/include/vapi/vsm_int.h
index 237fabd..4272aef 100644
--- a/include/vapi/vsm_int.h
+++ b/include/vapi/vsm_int.h
@@ -37,7 +37,7 @@
  * In particular we want the readers to seamlessly jump from one VSM instance
  * to another when the child restarts.
  *
- * The VSM life-cycle there is:
+ * The VSM segment life-cycle is:
  *
  *	Manager creates VSM file under temp name
  *
@@ -54,7 +54,7 @@
  *	it will zero the alloc_seq in it, before replacing the file.
  *
  * Subscribers will have to monitor two things to make sure they have
- * the current VSM instance:  The alloc_seq field and the inode number
+ * the current VSM instance:  The alloc_seq field and the dev+inode
  * of the path-name.  The former check is by far the cheaper and the
  * latter check should only be employed when lack of activity in the
  * VSM segment raises suspicion that something has happened.



From phk at varnish-cache.org  Thu Nov 24 10:50:55 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Thu, 24 Nov 2011 11:50:55 +0100
Subject: [master] e67da13 Rename shmem.rst to vsm.rst
Message-ID: <E1RTWtL-0005Rb-Jz@project.varnish-software.com>

commit e67da13c4d8d35543ce5b6bfc7092a1efbc6c6de
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Nov 24 10:49:45 2011 +0000

    Rename shmem.rst to vsm.rst

diff --git a/doc/sphinx/Makefile.am b/doc/sphinx/Makefile.am
index dc20abd..d2e5258 100644
--- a/doc/sphinx/Makefile.am
+++ b/doc/sphinx/Makefile.am
@@ -116,7 +116,6 @@ EXTRA_DIST = \
 	phk/three-zero.rst \
 	phk/vcl_expr.rst \
 	reference/index.rst \
-	reference/shmem.rst \
 	reference/params.rst \
 	reference/varnishadm.rst \
 	reference/varnish-cli.rst \
@@ -132,6 +131,7 @@ EXTRA_DIST = \
 	reference/vcl.rst \
 	reference/vmod.rst \
 	reference/vmod_std.rst \
+	reference/vsm.rst \
 	tutorial/advanced_backend_servers.rst \
 	tutorial/advanced_topics.rst \
 	tutorial/backend_servers.rst \
diff --git a/doc/sphinx/reference/index.rst b/doc/sphinx/reference/index.rst
index 84449da..3af59c9 100644
--- a/doc/sphinx/reference/index.rst
+++ b/doc/sphinx/reference/index.rst
@@ -18,7 +18,7 @@ The Varnish Reference Manual
 	varnishstat.rst
 	varnishtest.rst
 	varnishtop.rst
-	shmem.rst
+	vsm.rst
 	vmod.rst
 	vmod_std.rst
 	vsl.rst
diff --git a/doc/sphinx/reference/shmem.rst b/doc/sphinx/reference/shmem.rst
deleted file mode 100644
index 26c5525..0000000
--- a/doc/sphinx/reference/shmem.rst
+++ /dev/null
@@ -1,65 +0,0 @@
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-Shared Memory Logging and Statistics
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-
-Varnish uses shared memory for logging and statistics, because it
-is faster and much more efficient.  But it is also tricky in ways
-a regular logfile is not.
-
-When you open a file in "append" mode, the operating system guarantees
-that whatever you write will not overwrite existing data in the file.
-The neat result of this is that multiple procesess or threads writing
-to the same file does not even need to know about each other, it all
-works just as you would expect.
-
-With a shared memory log, we get no help from the kernel, the writers
-need to make sure they do not stomp on each other, and they need to
-make it possible and safe for the readers to access the data.
-
-The "CS101" way, is to introduce locks, and much time is spent examining
-the relative merits of the many kinds of locks available.
-
-Inside the varnishd (worker) process, we use mutexes to guarantee
-consistency, both with respect to allocations, log entries and stats
-counters.
-
-We do not want a varnishncsa trying to push data through a stalled
-ssh connection to stall the delivery of content, so readers like
-that are purely read-only, they do not get to affect the varnishd
-process and that means no locks for them.
-
-Instead we use "stable storage" concepts, to make sure the view
-seen by the readers is consistent at all times.
-
-As long as you only add stuff, that is trivial, but taking away
-stuff, such as when a backend is taken out of the configuration,
-we need to give the readers a chance to discover this, a "cooling
-off" period.
-
-When Varnishd starts, if it finds an existing shared memory file,
-and it can safely read the master_pid field, it will check if that
-process is running, and if so, fail with an error message, indicating
-that -n arguments collide.
-
-In all other cases, it will delete and create a new shmlog file,
-in order to provide running readers a cooling off period, where
-they can discover that there is a new shmlog file, by doing a
-stat(2) call and checking the st_dev & st_inode fields.
-
-Allocations
------------
-
-Sections inside the shared memory file are allocated dynamically,
-for instance when a new backend is added.
-
-While changes happen to the linked list of allocations, the "alloc_seq"
-header field is zero, and after the change, it gets a value different
-from what it had before.
-
-Deallocations
--------------
-
-When a section is freed, its class will change to "Cool" for at
-least 10 seconds, giving programs using it time to detect the 
-change in alloc_seq header field and/or the change of class.
-
diff --git a/doc/sphinx/reference/vsm.rst b/doc/sphinx/reference/vsm.rst
new file mode 100644
index 0000000..26c5525
--- /dev/null
+++ b/doc/sphinx/reference/vsm.rst
@@ -0,0 +1,65 @@
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+Shared Memory Logging and Statistics
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+Varnish uses shared memory for logging and statistics, because it
+is faster and much more efficient.  But it is also tricky in ways
+a regular logfile is not.
+
+When you open a file in "append" mode, the operating system guarantees
+that whatever you write will not overwrite existing data in the file.
+The neat result of this is that multiple procesess or threads writing
+to the same file does not even need to know about each other, it all
+works just as you would expect.
+
+With a shared memory log, we get no help from the kernel, the writers
+need to make sure they do not stomp on each other, and they need to
+make it possible and safe for the readers to access the data.
+
+The "CS101" way, is to introduce locks, and much time is spent examining
+the relative merits of the many kinds of locks available.
+
+Inside the varnishd (worker) process, we use mutexes to guarantee
+consistency, both with respect to allocations, log entries and stats
+counters.
+
+We do not want a varnishncsa trying to push data through a stalled
+ssh connection to stall the delivery of content, so readers like
+that are purely read-only, they do not get to affect the varnishd
+process and that means no locks for them.
+
+Instead we use "stable storage" concepts, to make sure the view
+seen by the readers is consistent at all times.
+
+As long as you only add stuff, that is trivial, but taking away
+stuff, such as when a backend is taken out of the configuration,
+we need to give the readers a chance to discover this, a "cooling
+off" period.
+
+When Varnishd starts, if it finds an existing shared memory file,
+and it can safely read the master_pid field, it will check if that
+process is running, and if so, fail with an error message, indicating
+that -n arguments collide.
+
+In all other cases, it will delete and create a new shmlog file,
+in order to provide running readers a cooling off period, where
+they can discover that there is a new shmlog file, by doing a
+stat(2) call and checking the st_dev & st_inode fields.
+
+Allocations
+-----------
+
+Sections inside the shared memory file are allocated dynamically,
+for instance when a new backend is added.
+
+While changes happen to the linked list of allocations, the "alloc_seq"
+header field is zero, and after the change, it gets a value different
+from what it had before.
+
+Deallocations
+-------------
+
+When a section is freed, its class will change to "Cool" for at
+least 10 seconds, giving programs using it time to detect the 
+change in alloc_seq header field and/or the change of class.
+



From phk at varnish-cache.org  Thu Nov 24 10:50:59 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Thu, 24 Nov 2011 11:50:59 +0100
Subject: [master] 699607a Explain more details about VSM
Message-ID: <E1RTWtP-0005Rh-6V@project.varnish-software.com>

commit 699607a0bcc7b7aed43cf83ef8746771c8a7d754
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Nov 24 10:50:36 2011 +0000

    Explain more details about VSM

diff --git a/doc/sphinx/reference/vsm.rst b/doc/sphinx/reference/vsm.rst
index 26c5525..cdae8e7 100644
--- a/doc/sphinx/reference/vsm.rst
+++ b/doc/sphinx/reference/vsm.rst
@@ -1,10 +1,36 @@
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-Shared Memory Logging and Statistics
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+VSM: Shared Memory Logging and Statistics
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-Varnish uses shared memory for logging and statistics, because it
-is faster and much more efficient.  But it is also tricky in ways
-a regular logfile is not.
+Varnish uses shared memory to export parameters, logging and
+statistics, because it is faster and much more efficient than
+regular files.
+
+"Varnish Shared Memory" or VSM, is the overall mechanism, which
+manages a number of allocated "chunks" inside the same shared
+memory file.
+
+Each Chunk is just a slap of memory, which has
+a three-part name (class, type, ident) and a length.
+
+The Class indicates what type of data is stored in the chunk,
+for instance "Arg" for command line arguments useful for
+establishing an CLI connection to the varnishd, "Stat" for
+statistics counters (VSC) and "Log" for log records (VSL).
+
+The type and ident name parts are mostly used with stats
+counters, where they identify dynamic counters, such as:
+
+	SMA.Transient.c_bytes
+
+The size of the VSM is a parameter, but changes only take
+effect when the child process is restarted.
+
+Shared memory trickery
+----------------------
+
+Shared memory is faster than regular files, but it is also slightly
+tricky in ways a regular logfile is not.
 
 When you open a file in "append" mode, the operating system guarantees
 that whatever you write will not overwrite existing data in the file.
@@ -12,12 +38,14 @@ The neat result of this is that multiple procesess or threads writing
 to the same file does not even need to know about each other, it all
 works just as you would expect.
 
-With a shared memory log, we get no help from the kernel, the writers
-need to make sure they do not stomp on each other, and they need to
-make it possible and safe for the readers to access the data.
+With a shared memory log, we get no such help from the kernel, the
+writers need to make sure they do not stomp on each other, and they
+need to make it possible and safe for the readers to access the
+data.
 
-The "CS101" way, is to introduce locks, and much time is spent examining
-the relative merits of the many kinds of locks available.
+The "CS101" way to deal with that, is to introduce locks, and much
+time is spent examining the relative merits of the many kinds of
+locks available.
 
 Inside the varnishd (worker) process, we use mutexes to guarantee
 consistency, both with respect to allocations, log entries and stats
@@ -36,30 +64,41 @@ stuff, such as when a backend is taken out of the configuration,
 we need to give the readers a chance to discover this, a "cooling
 off" period.
 
-When Varnishd starts, if it finds an existing shared memory file,
-and it can safely read the master_pid field, it will check if that
-process is running, and if so, fail with an error message, indicating
-that -n arguments collide.
+The Varnish way:
+----------------
+
+If Varnishd starts, and finds a locked shared memory file, it will
+exit with a message about using different -n arguments if you want
+multiple instances of varnishd.
+
+Otherwise, it will create a new shared memory file each time it
+starts a child process, since that marks a clean break in operation
+anyway.
 
-In all other cases, it will delete and create a new shmlog file,
-in order to provide running readers a cooling off period, where
-they can discover that there is a new shmlog file, by doing a
-stat(2) call and checking the st_dev & st_inode fields.
+To the extent possible, old shared memory files are marked as
+abandoned by setting the alloc_seq field to zero, which should be
+monitored by all readers of the VSM.
 
-Allocations
------------
+Processes subscribing to VSM files for a long time, should notice
+if the VSM file goes "silent" and check that the file has not been
+renamed due to a child restart.
 
-Sections inside the shared memory file are allocated dynamically,
-for instance when a new backend is added.
+Chunks inside the shared memory file form a linked list, and whenever
+that list changes, the alloc_seq field changes.
 
-While changes happen to the linked list of allocations, the "alloc_seq"
-header field is zero, and after the change, it gets a value different
-from what it had before.
+The linked list and other metadata in the VSM file, works with
+offsets relative to the start address of where the VSM file is
+memory mapped, so it need not be mapped at any particular address.
 
-Deallocations
--------------
+When new chunks are allocated, for instance when a new backend is
+added, they are appended to the list, no matter where they are
+located in the VSM file.
 
-When a section is freed, its class will change to "Cool" for at
-least 10 seconds, giving programs using it time to detect the 
-change in alloc_seq header field and/or the change of class.
+When a chunk is freed, it will be taken out of the linked list of
+allocations, its length will be set to zero and alloc_seq will be
+changed to indicate a change of layout.  For the next 60 seconds
+the chunk will not be touched or reused, giving other subscribers
+a chance to discover the deallocation.
 
+The include file <vapi/vsm.h> provides the supported API for accessing
+VSM files.



From phk at varnish-cache.org  Fri Nov 25 11:20:55 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Fri, 25 Nov 2011 12:20:55 +0100
Subject: [master] 84b1ca0 Gcc isn't always very smart...
Message-ID: <E1RTtpv-0007gD-8S@project.varnish-software.com>

commit 84b1ca029b635f47555495b46ef9153d3ac34567
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Nov 25 11:20:40 2011 +0000

    Gcc isn't always very smart...

diff --git a/bin/varnishd/cache/cache_shmlog.c b/bin/varnishd/cache/cache_shmlog.c
index 893525b..a18a08e 100644
--- a/bin/varnishd/cache/cache_shmlog.c
+++ b/bin/varnishd/cache/cache_shmlog.c
@@ -311,6 +311,7 @@ vsm_cleaner(void *priv)
 		AZ(pthread_mutex_unlock(&vsm_mtx));
 		VTIM_sleep(1.1);
 	}
+	NEEDLESS_RETURN(NULL);
 }
 
 /*--------------------------------------------------------------------*/



From phk at varnish-cache.org  Fri Nov 25 11:41:43 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Fri, 25 Nov 2011 12:41:43 +0100
Subject: [master] 42d9a31 Always remember what your pointers point to,
	when doing pointer addition.
Message-ID: <E1RTuA3-0001db-If@project.varnish-software.com>

commit 42d9a31ac0dc63f0ae971974b4a4c36c20f421ef
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Nov 25 11:40:54 2011 +0000

    Always remember what your pointers point to, when doing pointer addition.
    
    Fixes	#1065

diff --git a/bin/varnishd/cache/cache_shmlog.c b/bin/varnishd/cache/cache_shmlog.c
index a18a08e..1e414b8 100644
--- a/bin/varnishd/cache/cache_shmlog.c
+++ b/bin/varnishd/cache/cache_shmlog.c
@@ -335,7 +335,7 @@ VSM_Init(void)
 	VWMB();
 
 	vsl_start = vsl_log_start;
-	vsl_end = vsl_start + cache_param->vsl_space;
+	vsl_end = vsl_start + cache_param->vsl_space / sizeof *vsl_end;
 	vsl_ptr = vsl_start + 1;
 
 	VSC_C_main = VSM_Alloc(sizeof *VSC_C_main,



From phk at varnish-cache.org  Fri Nov 25 11:43:18 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Fri, 25 Nov 2011 12:43:18 +0100
Subject: [master] f4ba8ee Make varnishlog not explode
Message-ID: <E1RTuBa-0001mE-0k@project.varnish-software.com>

commit f4ba8eeaf67cbd8f993ecf14ea5ade0862cf3ba2
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Nov 25 11:43:08 2011 +0000

    Make varnishlog not explode

diff --git a/lib/libvarnishapi/vsl.c b/lib/libvarnishapi/vsl.c
index fdd93d8..6950c1b 100644
--- a/lib/libvarnishapi/vsl.c
+++ b/lib/libvarnishapi/vsl.c
@@ -124,9 +124,11 @@ vsl_open(struct VSM_data *vd)
 	int i;
 
 	assert(vsl->r_fd < 0);
-	i = VSM_Open(vd);
-	if (i)
-		return (i);
+	if (vd->head == NULL) {
+		i = VSM_Open(vd);
+		if (i)
+			return (i);
+	}
 	if (!VSM_Get(vd, &vsl->vf, VSL_CLASS, NULL, NULL)) {
 		VSM_Close(vd);
 		return (vsm_diag(vd, "No VSL chunk found "
@@ -254,10 +256,11 @@ VSL_NextSLT(struct VSM_data *vd, uint32_t **pp, uint64_t *bits)
 		i = vsl_nextslt(vd, &p);
 		if (i < 0)
 			return (i);
-		if (i == 0 && (vsl->d_opt || vsl->r_fd >= 0))
-			return (i);
-		if (i == 0 && !VSM_StillValid(vd, &vsl->vf)) {
-			vsl_close(vd);
+		if (i == 0) {
+			if (vsl->d_opt || vsl->r_fd >= 0)
+				return (i);
+			if (!VSM_StillValid(vd, &vsl->vf))
+				vsl_close(vd);
 			return (i);
 		}
 



From phk at varnish-cache.org  Fri Nov 25 11:51:13 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Fri, 25 Nov 2011 12:51:13 +0100
Subject: [master] cd6f804 Remove a duplicate assert
Message-ID: <E1RTuJF-0002FV-B9@project.varnish-software.com>

commit cd6f804c1567fc9e40a87231135a361d5d092cc8
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Nov 25 11:50:46 2011 +0000

    Remove a duplicate assert
    
    Fixes	#1064

diff --git a/lib/libvcl/vcc_compile.c b/lib/libvcl/vcc_compile.c
index 5719cbf..fddd019 100644
--- a/lib/libvcl/vcc_compile.c
+++ b/lib/libvcl/vcc_compile.c
@@ -444,13 +444,13 @@ vcc_resolve_includes(struct vcc *tl)
 		}
 		t2 = VTAILQ_NEXT(t1, list);
 		assert(t2 != NULL);	/* There's always an EOI */
+
 		if (t2->tok != ';') {
 			VSB_printf(tl->sb,
 			    "include <string> not followed by semicolon.\n");
 			vcc_ErrWhere(tl, t1);
 			return;
 		}
-		assert(t2 != NULL);
 
 		sp = vcc_file_source(tl, tl->sb, t1->dec);
 		if (sp == NULL) {



From phk at varnish-cache.org  Mon Nov 28 09:11:52 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 28 Nov 2011 10:11:52 +0100
Subject: [master] c4629a0 Make it possible to test for the non-definition of a
	http header.
Message-ID: <E1RUxFg-0000aM-JE@project.varnish-software.com>

commit c4629a0fb7731416f5f130d8ccefa55dc58e5191
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Nov 28 09:11:28 2011 +0000

    Make it possible to test for the non-definition of a http header.
    
    Fixes #1062

diff --git a/bin/varnishtest/tests/a00012.vtc b/bin/varnishtest/tests/a00012.vtc
new file mode 100644
index 0000000..07e51c1
--- /dev/null
+++ b/bin/varnishtest/tests/a00012.vtc
@@ -0,0 +1,13 @@
+varnishtest "Ensure that we can test non-existence of headers (#1062)"
+
+server s1 {
+        rxreq
+        txresp
+} -start
+
+client c1 -connect ${s1_sock} {
+        txreq
+        rxresp
+        expect resp.http.X-Test == <undef>
+} -run
+
diff --git a/bin/varnishtest/tests/c00016.vtc b/bin/varnishtest/tests/c00016.vtc
index d028a64..99e9c53 100644
--- a/bin/varnishtest/tests/c00016.vtc
+++ b/bin/varnishtest/tests/c00016.vtc
@@ -8,7 +8,7 @@ server s1 {
 
 	rxreq
 	expect req.url == "/bar"
-	expect req.http.Foo == "req.http.Foo"
+	expect req.http.Foo == <undef>
 	txresp -hdr "Bar: fnry,glyf, FOO ,brok" -hdr "Connection: bar" -body "foobar" 
 } -start
 
@@ -21,5 +21,5 @@ client c1 {
 
 	txreq -url "/bar" -hdr "Foo: bar2" -hdr "Connection: foo, close"
 	rxresp
-	expect req.http.Bar == "req.http.Bar"
+	expect req.http.Bar == <undef>
 } -run
diff --git a/bin/varnishtest/tests/e00024.vtc b/bin/varnishtest/tests/e00024.vtc
index 2aa8e3a..f728b3c 100644
--- a/bin/varnishtest/tests/e00024.vtc
+++ b/bin/varnishtest/tests/e00024.vtc
@@ -81,7 +81,7 @@ client c1 {
 
 	txreq 
 	rxresp
-	expect resp.http.content-encoding == resp.http.content-encoding
+	expect resp.http.content-encoding == <undef>
 	expect resp.status == 200
 	expect resp.bodylen == 252
 } -run
diff --git a/bin/varnishtest/tests/e00025.vtc b/bin/varnishtest/tests/e00025.vtc
index 5549f5c..5e8a12b 100644
--- a/bin/varnishtest/tests/e00025.vtc
+++ b/bin/varnishtest/tests/e00025.vtc
@@ -18,7 +18,7 @@ client c1 {
 
 	txreq 
 	rxresp
-	expect resp.http.content-encoding == resp.http.content-encoding
+	expect resp.http.content-encoding == <undef>
 	expect resp.status == 200
 	expect resp.bodylen == 3
 } -run
diff --git a/bin/varnishtest/tests/g00001.vtc b/bin/varnishtest/tests/g00001.vtc
index 34ad00f..e10b743 100644
--- a/bin/varnishtest/tests/g00001.vtc
+++ b/bin/varnishtest/tests/g00001.vtc
@@ -13,7 +13,7 @@ client c1 {
 	txreq 
 	rxresp
 	expect resp.bodylen == "3"
-	expect resp.http.content-encoding == "resp.http.content-encoding"
+	expect resp.http.content-encoding == <undef>
 
 	txreq -hdr "Accept-encoding: gzip;q=0.1"
 	rxresp
@@ -26,13 +26,13 @@ client c1 {
 	txreq -proto HTTP/1.0
 	rxresp
 	expect resp.bodylen == "3"
-	expect resp.http.content-encoding == "resp.http.content-encoding"
+	expect resp.http.content-encoding == <undef>
 } -run
 
 client c1 {
 	txreq -req HEAD
 	rxresp -no_obj
-	expect resp.http.content-encoding == "resp.http.content-encoding"
+	expect resp.http.content-encoding == <undef>
 
 	txreq -req HEAD -hdr "Accept-encoding: gzip;q=0.1"
 	rxresp -no_obj
diff --git a/bin/varnishtest/tests/g00002.vtc b/bin/varnishtest/tests/g00002.vtc
index 1bf3384..eb10ba0 100644
--- a/bin/varnishtest/tests/g00002.vtc
+++ b/bin/varnishtest/tests/g00002.vtc
@@ -39,12 +39,12 @@ client c1 {
 	# See varnish can gunzip it.
 	txreq -url /foo -hdr "Accept-Encoding: null"
 	rxresp
-	expect resp.http.content-encoding == "resp.http.content-encoding"
+	expect resp.http.content-encoding == <undef>
 	expect resp.bodylen == 4100
 
 	# See varnish can gunzip it, inside ESI
 	txreq -url /bar -hdr "Accept-Encoding: null"
 	rxresp
-	expect resp.http.content-encoding == "resp.http.content-encoding"
+	expect resp.http.content-encoding == <undef>
 	expect resp.bodylen == 4109
 } -run
diff --git a/bin/varnishtest/tests/g00003.vtc b/bin/varnishtest/tests/g00003.vtc
index dcd8936..05edb5d 100644
--- a/bin/varnishtest/tests/g00003.vtc
+++ b/bin/varnishtest/tests/g00003.vtc
@@ -30,12 +30,12 @@ varnish v1 -cliok "param.set http_gzip_support true" -vcl+backend {
 client c1 {
 	txreq -url /foo -hdr "Accept-Encoding: gzip"
 	rxresp
-	expect resp.http.content-encoding == "resp.http.content-encoding"
+	expect resp.http.content-encoding == <undef>
 	expect resp.bodylen == 41
 
 	txreq -url /bar -hdr "Accept-Encoding: gzip"
 	rxresp
-	expect resp.http.content-encoding == "resp.http.content-encoding"
+	expect resp.http.content-encoding == <undef>
 	expect resp.bodylen == 42
 
 	txreq -url /foobar -hdr "Accept-Encoding: gzip"
@@ -46,6 +46,6 @@ client c1 {
 
 	txreq -url /foobar 
 	rxresp
-	expect resp.http.content-encoding == "resp.http.content-encoding"
+	expect resp.http.content-encoding == <undef>
 	expect resp.bodylen == 43
 } -run
diff --git a/bin/varnishtest/tests/r00292.vtc b/bin/varnishtest/tests/r00292.vtc
index aaa914c..7739e85 100644
--- a/bin/varnishtest/tests/r00292.vtc
+++ b/bin/varnishtest/tests/r00292.vtc
@@ -6,11 +6,11 @@ varnishtest "Header deletion test"
 server s1 {
 	rxreq
 	expect req.url == "/foo"
-	expect req.http.hdr1 == "req.http.hdr1"
+	expect req.http.hdr1 == <undef>
 	expect req.http.hdr2 == "2"
-	expect req.http.hdr3 == "req.http.hdr3"
+	expect req.http.hdr3 == <undef>
 	expect req.http.hdr4 == "4"
-	expect req.http.hdr5 == "req.http.hdr5"
+	expect req.http.hdr5 == <undef>
 	expect req.http.hdr6 == "6"
 	txresp -body "foobar"
 } -start
diff --git a/bin/varnishtest/tests/r00466.vtc b/bin/varnishtest/tests/r00466.vtc
index acff68a..8d753ba 100644
--- a/bin/varnishtest/tests/r00466.vtc
+++ b/bin/varnishtest/tests/r00466.vtc
@@ -3,7 +3,7 @@ varnishtest "Check Range forwarding to backend"
 server s1 {
 	rxreq 
 	expect req.url == "/foo"
-	expect req.http.range == "req.http.range"
+	expect req.http.range == <undef>
 	txresp \
 		-hdr "Foobar: _barf_" \
 		-body "012345\n"
diff --git a/bin/varnishtest/tests/r00494.vtc b/bin/varnishtest/tests/r00494.vtc
index d1dedd1..1a7fec3 100644
--- a/bin/varnishtest/tests/r00494.vtc
+++ b/bin/varnishtest/tests/r00494.vtc
@@ -19,6 +19,6 @@ client c1 {
 	txreq
 	rxresp
 	expect resp.http.bar == "bar,  barf: fail"
-	expect resp.http.barf == resp.http.barf
-	expect resp.http.foo == resp.http.foo
+	expect resp.http.barf == <undef>
+	expect resp.http.foo == <undef>
 } -run
diff --git a/bin/varnishtest/tests/r00693.vtc b/bin/varnishtest/tests/r00693.vtc
index 6cf7b30..37b0a98 100644
--- a/bin/varnishtest/tests/r00693.vtc
+++ b/bin/varnishtest/tests/r00693.vtc
@@ -4,11 +4,11 @@ feature 64bit
 
 server s1 {
 	rxreq
-	expect req.http.baz == "req.http.baz"
+	expect req.http.baz == <undef>
 	txresp -status 201
 
 	rxreq
-	expect req.http.baz == "req.http.baz"
+	expect req.http.baz == <undef>
 	txresp -status 202
 
 	rxreq
@@ -52,7 +52,7 @@ varnish v1 -arg "-p sess_workspace=1024" -vcl+backend {
 			"0123456789abcdef" + "0123456789abcdef" +
 			"0123456789abcdef" + "0123456789abcdef" +
 			"0123456789abcdef" +
-			"01234567";
+			"0123456";
 		set req.http.baz = "BAZ";
 		return (pass);
 	}
diff --git a/bin/varnishtest/tests/r00861.vtc b/bin/varnishtest/tests/r00861.vtc
index 165795b..8a3e8a0 100644
--- a/bin/varnishtest/tests/r00861.vtc
+++ b/bin/varnishtest/tests/r00861.vtc
@@ -34,12 +34,12 @@ varnish v1 \
 client c1 {
 	txreq -url "/1"
 	rxresp
-	expect resp.http.Content-Encoding == resp.http.Content-Encoding
+	expect resp.http.Content-Encoding == <undef>
 	expect resp.bodylen == 22
 
 	txreq -url "/barf" -hdr "Accept-Encoding: gzip"
 	rxresp
-	expect resp.http.Content-Encoding == resp.http.Content-Encoding
+	expect resp.http.Content-Encoding == <undef>
 	expect resp.bodylen == 909
 
 	txreq -url "/2" -hdr "Accept-Encoding: gzip"
diff --git a/bin/varnishtest/tests/r00980.vtc b/bin/varnishtest/tests/r00980.vtc
index 9591786..b7d307e 100644
--- a/bin/varnishtest/tests/r00980.vtc
+++ b/bin/varnishtest/tests/r00980.vtc
@@ -24,6 +24,6 @@ client c1 {
 
 	txreq -url /foobar 
 	rxresp
-	expect resp.http.content-encoding == "resp.http.content-encoding"
+	expect resp.http.content-encoding == <undef>
 	expect resp.bodylen == 43
 } -run
diff --git a/bin/varnishtest/vtc_http.c b/bin/varnishtest/vtc_http.c
index 69485cc..d3753a5 100644
--- a/bin/varnishtest/vtc_http.c
+++ b/bin/varnishtest/vtc_http.c
@@ -169,7 +169,7 @@ http_find_header(char * const *hh, const char *hdr)
  * Expect
  */
 
-static char *
+static const char *
 cmd_var_resolve(struct http *hp, char *spec)
 {
 	char **hh, *hdr;
@@ -201,16 +201,16 @@ cmd_var_resolve(struct http *hp, char *spec)
 	hdr = http_find_header(hh, hdr);
 	if (hdr != NULL)
 		return (hdr);
-	return (spec);
+	return ("<undef>");
 }
 
 static void
 cmd_http_expect(CMD_ARGS)
 {
 	struct http *hp;
-	char *lhs;
+	const char *lhs;
 	char *cmp;
-	char *rhs;
+	const char *rhs;
 
 	(void)cmd;
 	(void)vl;



From phk at varnish-cache.org  Mon Nov 28 10:56:31 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 28 Nov 2011 11:56:31 +0100
Subject: [master] 0034032 Fix VSL no longer working
Message-ID: <E1RUysx-0007ia-IZ@project.varnish-software.com>

commit 0034032ba77f9e58895fbe0a9a8bfed0f3b5df8c
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Nov 28 10:56:22 2011 +0000

    Fix VSL no longer working

diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c
index 2f7514a..880f966 100644
--- a/bin/varnishtest/vtc_varnish.c
+++ b/bin/varnishtest/vtc_varnish.c
@@ -201,7 +201,7 @@ varnishlog_thread(void *priv)
 	(void)VSL_Arg(vsl, 'n', v->workdir);
 	while (v->pid) {
 		if (VSL_Dispatch(vsl, h_addlog, v) <= 0)
-			break;
+			usleep(100000);
 	}
 	VSM_Delete(vsl);
 	return (NULL);



From phk at varnish-cache.org  Tue Nov 29 10:55:24 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 29 Nov 2011 11:55:24 +0100
Subject: [master] a68844e Clarify various details while trying to figure out
	req-timing reporting
Message-ID: <E1RVLLQ-0007jC-0X@project.varnish-software.com>

commit a68844e29db79e880390331f189e3a07f1b83af8
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 29 10:54:47 2011 +0000

    Clarify various details while trying to figure out req-timing reporting

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 5e813ca..1475d6d 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -1028,3 +1028,9 @@ AssertObjCorePassOrBusy(const struct objcore *oc)
 	if (oc != NULL)
 		AN (oc->flags & OC_F_BUSY);
 }
+
+/*
+ * We want to cache the most recent timestamp in wrk->lastused to avoid
+ * extra timestamps in cache_pool.c.  Hide this detail with a macro
+ */
+#define W_TIM_real(w) ((w)->lastused = VTIM_real())
diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c
index e42fac8..89886c2 100644
--- a/bin/varnishd/cache/cache_center.c
+++ b/bin/varnishd/cache/cache_center.c
@@ -52,7 +52,7 @@ DOT	label="Request received"
 DOT ]
 DOT ERROR [shape=plaintext]
 DOT RESTART [shape=plaintext]
-DOT acceptor -> start [style=bold,color=green]
+DOT acceptor -> first [style=bold,color=green]
  */
 
 #include "config.h"
@@ -80,6 +80,17 @@ static unsigned xids;
 /*--------------------------------------------------------------------
  * WAIT
  * Wait (briefly) until we have a full request in our htc.
+ *
+DOT subgraph xcluster_wait {
+DOT	wait [
+DOT		shape=box
+DOT		label="wait for\nrequest"
+DOT	]
+DOT	herding [shape=hexagon]
+DOT	wait -> start [label="got req"]
+DOT	wait -> DONE [label="errors"]
+DOT	wait -> herding [label="timeout"]
+DOT }
  */
 
 static int
@@ -91,6 +102,7 @@ cnt_wait(struct sess *sp)
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
 	AZ(sp->vcl);
 	AZ(sp->obj);
+	AZ(sp->esi_level);
 	assert(sp->xid == 0);
 
 	i = HTC_Complete(sp->htc);
@@ -116,9 +128,7 @@ cnt_wait(struct sess *sp)
 	}
 	if (i == -2) {
 		SES_Close(sp, "overflow");
-		return (0);
-	}
-	if (i == -1 && Tlen(sp->htc->rxbuf) == 0 &&
+	} else if (i == -1 && Tlen(sp->htc->rxbuf) == 0 &&
 	    (errno == 0 || errno == ECONNRESET))
 		SES_Close(sp, "EOF");
 	else
@@ -210,7 +220,7 @@ cnt_prepresp(struct sess *sp)
 		}
 	}
 
-	sp->t_resp = VTIM_real();
+	sp->t_resp = W_TIM_real(sp->wrk);
 	if (sp->obj->objcore != NULL) {
 		if ((sp->t_resp - sp->obj->last_lru) > cache_param->lru_timeout &&
 		    EXP_Touch(sp->obj->objcore))
@@ -293,6 +303,10 @@ DOT	DONE [
 DOT		shape=hexagon
 DOT		label="Request completed"
 DOT	]
+DOT	ESI_RESP [ shape=hexagon ]
+DOT	DONE -> start
+DOT	DONE -> wait
+DOT	DONE -> ESI_RESP
  */
 
 static int
@@ -325,10 +339,11 @@ cnt_done(struct sess *sp)
 
 	SES_Charge(sp);
 
-	sp->t_end = VTIM_real();
-	sp->wrk->lastused = sp->t_end;
+	sp->t_end = W_TIM_real(sp->wrk);
+WSP(sp, SLT_Debug, "PHK req %.9f resp %.9f end %.9f open %.9f",
+    sp->t_req, sp->t_resp, sp->t_end,  sp->t_open);
 	if (sp->xid == 0) {
-		sp->t_req = sp->t_end;
+		// sp->t_req = sp->t_end;
 		sp->t_resp = sp->t_end;
 	} else if (sp->esi_level == 0) {
 		dp = sp->t_resp - sp->t_req;
@@ -344,14 +359,15 @@ cnt_done(struct sess *sp)
 		    sp->xid, sp->t_req, sp->t_end, dh, dp, da);
 	}
 	sp->xid = 0;
-	sp->t_open = sp->t_end;
-	sp->t_resp = NAN;
 	WSL_Flush(sp->wrk, 0);
 
 	/* If we did an ESI include, don't mess up our state */
 	if (sp->esi_level > 0)
 		return (1);
 
+	sp->t_open = sp->t_end;
+	sp->t_resp = NAN;
+
 	sp->req_bodybytes = 0;
 
 	sp->t_req = NAN;
@@ -465,7 +481,7 @@ cnt_error(struct sess *sp)
 
 	http_PutProtocol(w, sp->vsl_id, h, "HTTP/1.1");
 	http_PutStatus(h, sp->err_code);
-	VTIM_format(VTIM_real(), date);
+	VTIM_format(W_TIM_real(sp->wrk), date);
 	http_PrintfHeader(w, sp->vsl_id, h, "Date: %s", date);
 	http_SetHeader(w, sp->vsl_id, h, "Server: Varnish");
 
@@ -580,7 +596,7 @@ cnt_fetch(struct sess *sp)
 		 * What does RFC2616 think about TTL ?
 		 */
 		EXP_Clr(&sp->wrk->exp);
-		sp->wrk->exp.entered = VTIM_real();
+		sp->wrk->exp.entered = W_TIM_real(sp->wrk);
 		RFC2616_Ttl(sp);
 
 		/* pass from vclrecv{} has negative TTL */
@@ -947,7 +963,15 @@ cnt_streambody(struct sess *sp)
 
 /*--------------------------------------------------------------------
  * The very first request
+DOT subgraph xcluster_first {
+DOT	first [
+DOT		shape=box
+DOT		label="first\nConfigure data structures"
+DOT	]
+DOT }
+DOT first -> wait 
  */
+
 static int
 cnt_first(struct sess *sp)
 {
@@ -961,6 +985,7 @@ cnt_first(struct sess *sp)
 
 	assert(sp->xid == 0);
 	assert(sp->restarts == 0);
+	AZ(sp->esi_level);
 	VCA_Prep(sp);
 
 	/* Record the session watermark */
@@ -969,7 +994,6 @@ cnt_first(struct sess *sp)
 	/* Receive a HTTP protocol request */
 	HTC_Init(sp->htc, sp->ws, sp->fd, sp->vsl_id, cache_param->http_req_size,
 	    cache_param->http_req_hdr_len);
-	sp->wrk->lastused = sp->t_open;
 	sp->wrk->acct_tmp.sess++;
 
 	sp->step = STP_WAIT;
@@ -1344,7 +1368,9 @@ DOT		shape=record
 DOT		label="vcl_recv()|req."
 DOT	]
 DOT }
+DOT ESI_REQ [ shape=hexagon ]
 DOT RESTART -> recv
+DOT ESI_REQ -> recv
 DOT recv -> pipe [label="pipe",style=bold,color=orange]
 DOT recv -> pass2 [label="pass",style=bold,color=red]
 DOT recv -> err_recv [label="error"]
@@ -1445,8 +1471,12 @@ cnt_recv(struct sess *sp)
  * START
  * Handle a request, wherever it came from recv/restart.
  *
-DOT start [shape=box,label="Dissect request"]
+DOT start [
+DOT	shape=box
+DOT	label="Dissect request\nHandle expect"
+DOT ]
 DOT start -> recv [style=bold,color=green]
+DOT start -> DONE [label=errors]
  */
 
 static int
@@ -1460,11 +1490,11 @@ cnt_start(struct sess *sp)
 	AZ(sp->restarts);
 	AZ(sp->obj);
 	AZ(sp->vcl);
+	AZ(sp->esi_level);
 
 	/* Update stats of various sorts */
 	sp->wrk->stats.client_req++;
-	sp->t_req = VTIM_real();
-	sp->wrk->lastused = sp->t_req;
+	sp->t_req = W_TIM_real(sp->wrk);
 	sp->wrk->acct_tmp.req++;
 
 	/* Assign XID and log */



From phk at varnish-cache.org  Tue Nov 29 11:23:33 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 29 Nov 2011 12:23:33 +0100
Subject: [master] 1bf8b02 Reorder code in cnt_done() for improved clarity
Message-ID: <E1RVLmf-0004EF-CE@project.varnish-software.com>

commit 1bf8b02f961c2d0a2f567bce5bdd2374db017e67
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 29 11:23:08 2011 +0000

    Reorder code in cnt_done() for improved clarity

diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c
index 89886c2..b79aa31 100644
--- a/bin/varnishd/cache/cache_center.c
+++ b/bin/varnishd/cache/cache_center.c
@@ -330,14 +330,19 @@ cnt_done(struct sess *sp)
 	sp->wrk->is_gunzip = 0;
 	sp->wrk->is_gzip = 0;
 
-	if (sp->vcl != NULL && sp->esi_level == 0) {
+	SES_Charge(sp);
+
+	/* If we did an ESI include, don't mess up our state */
+	if (sp->esi_level > 0)
+		return (1);
+
+	if (sp->vcl != NULL) {
 		if (sp->wrk->vcl != NULL)
 			VCL_Rel(&sp->wrk->vcl);
 		sp->wrk->vcl = sp->vcl;
 		sp->vcl = NULL;
 	}
 
-	SES_Charge(sp);
 
 	sp->t_end = W_TIM_real(sp->wrk);
 WSP(sp, SLT_Debug, "PHK req %.9f resp %.9f end %.9f open %.9f",
@@ -345,7 +350,7 @@ WSP(sp, SLT_Debug, "PHK req %.9f resp %.9f end %.9f open %.9f",
 	if (sp->xid == 0) {
 		// sp->t_req = sp->t_end;
 		sp->t_resp = sp->t_end;
-	} else if (sp->esi_level == 0) {
+	} else {
 		dp = sp->t_resp - sp->t_req;
 		da = sp->t_end - sp->t_resp;
 		dh = sp->t_req - sp->t_open;
@@ -361,10 +366,6 @@ WSP(sp, SLT_Debug, "PHK req %.9f resp %.9f end %.9f open %.9f",
 	sp->xid = 0;
 	WSL_Flush(sp->wrk, 0);
 
-	/* If we did an ESI include, don't mess up our state */
-	if (sp->esi_level > 0)
-		return (1);
-
 	sp->t_open = sp->t_end;
 	sp->t_resp = NAN;
 



From phk at varnish-cache.org  Tue Nov 29 12:14:23 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 29 Nov 2011 13:14:23 +0100
Subject: [master] 3150388 The obj and objcore are not really properties of the
	session,
	and we need to be able to reach them during fetching without worrying
	about a particular session.
Message-ID: <E1RVMZr-0000A1-Qz@project.varnish-software.com>

commit 3150388c776ce53d94451cf7364020000710e55e
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 29 12:12:00 2011 +0000

    The obj and objcore are not really properties of the session, and we
    need to be able to reach them during fetching without worrying about
    a particular session.
    
    Move them to the worker thread, which corresponds much closer to their
    lifetime.
    
    This is mostly a brute force s/sp->obj/sp->wrk->obj/, with a few
    necessary edits.  Consequential fallouts and sp-scope-elimination to
    follow in subsequent commits.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 1475d6d..db001ba 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -321,6 +321,9 @@ struct worker {
 	struct http		*beresp;
 	struct http		*resp;
 
+	struct object		*obj;
+	struct objcore		*objcore;
+
 	struct exp		exp;
 
 	/* This is only here so VRT can find it */
@@ -599,8 +602,6 @@ struct sess {
 	VTAILQ_ENTRY(sess)	list;
 
 	struct director		*director;
-	struct object		*obj;
-	struct objcore		*objcore;
 	struct VCL_conf		*vcl;
 
 	/* The busy objhead we sleep on */
diff --git a/bin/varnishd/cache/cache_backend.c b/bin/varnishd/cache/cache_backend.c
index 1f290c3..de4318f 100644
--- a/bin/varnishd/cache/cache_backend.c
+++ b/bin/varnishd/cache/cache_backend.c
@@ -230,7 +230,7 @@ vbe_NewConn(void)
 
 /*--------------------------------------------------------------------
  * It evaluates if a backend is healthy _for_a_specific_object_.
- * That means that it relies on sp->objcore->objhead. This is mainly for
+ * That means that it relies on sp->wrk->objcore->objhead. This is mainly for
  * saint-mode, but also takes backend->healthy into account. If
  * cache_param->saintmode_threshold is 0, this is basically just a test of
  * backend->healthy.
@@ -277,11 +277,11 @@ vbe_Healthy(const struct vdi_simple *vs, const struct sess *sp)
 	if (threshold == 0)
 		return (1);
 
-	if (sp->objcore == NULL)
+	if (sp->wrk->objcore == NULL)
 		return (1);
 
 	now = sp->t_req;
-	target = (uintptr_t)(sp->objcore->objhead);
+	target = (uintptr_t)(sp->wrk->objcore->objhead);
 
 	old = NULL;
 	retval = 1;
diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c
index b79aa31..28c4760 100644
--- a/bin/varnishd/cache/cache_center.c
+++ b/bin/varnishd/cache/cache_center.c
@@ -101,7 +101,7 @@ cnt_wait(struct sess *sp)
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
 	AZ(sp->vcl);
-	AZ(sp->obj);
+	AZ(sp->wrk->obj);
 	AZ(sp->esi_level);
 	assert(sp->xid == 0);
 
@@ -170,11 +170,11 @@ cnt_prepresp(struct sess *sp)
 {
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->wrk->obj, OBJECT_MAGIC);
 	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
 
 	if (sp->wrk->do_stream)
-		AssertObjCorePassOrBusy(sp->obj->objcore);
+		AssertObjCorePassOrBusy(sp->wrk->obj->objcore);
 
 	sp->wrk->res_mode = 0;
 
@@ -182,7 +182,7 @@ cnt_prepresp(struct sess *sp)
 	    !sp->wrk->do_gzip && !sp->wrk->do_gunzip)
 		sp->wrk->res_mode |= RES_LEN;
 
-	if (!sp->disable_esi && sp->obj->esidata != NULL) {
+	if (!sp->disable_esi && sp->wrk->obj->esidata != NULL) {
 		/* In ESI mode, we don't know the aggregate length */
 		sp->wrk->res_mode &= ~RES_LEN;
 		sp->wrk->res_mode |= RES_ESI;
@@ -193,7 +193,7 @@ cnt_prepresp(struct sess *sp)
 		sp->wrk->res_mode |= RES_ESI_CHILD;
 	}
 
-	if (cache_param->http_gzip_support && sp->obj->gziped &&
+	if (cache_param->http_gzip_support && sp->wrk->obj->gziped &&
 	    !RFC2616_Req_Gzip(sp)) {
 		/*
 		 * We don't know what it uncompresses to
@@ -204,7 +204,7 @@ cnt_prepresp(struct sess *sp)
 	}
 
 	if (!(sp->wrk->res_mode & (RES_LEN|RES_CHUNKED|RES_EOF))) {
-		if (sp->obj->len == 0 && !sp->wrk->do_stream)
+		if (sp->wrk->obj->len == 0 && !sp->wrk->do_stream)
 			/*
 			 * If the object is empty, neither ESI nor GUNZIP
 			 * can make it any different size
@@ -221,11 +221,11 @@ cnt_prepresp(struct sess *sp)
 	}
 
 	sp->t_resp = W_TIM_real(sp->wrk);
-	if (sp->obj->objcore != NULL) {
-		if ((sp->t_resp - sp->obj->last_lru) > cache_param->lru_timeout &&
-		    EXP_Touch(sp->obj->objcore))
-			sp->obj->last_lru = sp->t_resp;
-		sp->obj->last_use = sp->t_resp;	/* XXX: locking ? */
+	if (sp->wrk->obj->objcore != NULL) {
+		if ((sp->t_resp - sp->wrk->obj->last_lru) > cache_param->lru_timeout &&
+		    EXP_Touch(sp->wrk->obj->objcore))
+			sp->wrk->obj->last_lru = sp->t_resp;
+		sp->wrk->obj->last_use = sp->t_resp;	/* XXX: locking ? */
 	}
 	http_Setup(sp->wrk->resp, sp->wrk->ws);
 	RES_BuildHttp(sp);
@@ -240,9 +240,9 @@ cnt_prepresp(struct sess *sp)
 			VDI_CloseFd(sp->wrk);
 			HSH_Drop(sp);
 		} else {
-			(void)HSH_Deref(sp->wrk, NULL, &sp->obj);
+			(void)HSH_Deref(sp->wrk, NULL, &sp->wrk->obj);
 		}
-		AZ(sp->obj);
+		AZ(sp->wrk->obj);
 		sp->restarts++;
 		sp->director = NULL;
 		sp->wrk->h_content_length = NULL;
@@ -255,7 +255,7 @@ cnt_prepresp(struct sess *sp)
 		WRONG("Illegal action in vcl_deliver{}");
 	}
 	if (sp->wrk->do_stream) {
-		AssertObjCorePassOrBusy(sp->obj->objcore);
+		AssertObjCorePassOrBusy(sp->wrk->obj->objcore);
 		sp->step = STP_STREAMBODY;
 	} else {
 		sp->step = STP_DELIVER;
@@ -289,7 +289,7 @@ cnt_deliver(struct sess *sp)
 
 	assert(WRW_IsReleased(sp->wrk));
 	assert(sp->wrk->wrw.ciov == sp->wrk->wrw.siov);
-	(void)HSH_Deref(sp->wrk, NULL, &sp->obj);
+	(void)HSH_Deref(sp->wrk, NULL, &sp->wrk->obj);
 	http_Setup(sp->wrk->resp, NULL);
 	sp->step = STP_DONE;
 	return (0);
@@ -318,7 +318,7 @@ cnt_done(struct sess *sp)
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
 	CHECK_OBJ_ORNULL(sp->vcl, VCL_CONF_MAGIC);
 
-	AZ(sp->obj);
+	AZ(sp->wrk->obj);
 	AZ(sp->wrk->vbc);
 	sp->director = NULL;
 	sp->restarts = 0;
@@ -450,16 +450,16 @@ cnt_error(struct sess *sp)
 	sp->wrk->do_stream = 0;
 
 	w = sp->wrk;
-	if (sp->obj == NULL) {
+	if (sp->wrk->obj == NULL) {
 		HSH_Prealloc(sp);
 		EXP_Clr(&w->exp);
-		sp->obj = STV_NewObject(sp, NULL, cache_param->http_resp_size,
+		sp->wrk->obj = STV_NewObject(sp, NULL, cache_param->http_resp_size,
 		     &w->exp, (uint16_t)cache_param->http_max_hdr);
-		if (sp->obj == NULL)
-			sp->obj = STV_NewObject(sp, TRANSIENT_STORAGE,
+		if (sp->wrk->obj == NULL)
+			sp->wrk->obj = STV_NewObject(sp, TRANSIENT_STORAGE,
 			    cache_param->http_resp_size, &w->exp,
 			    (uint16_t)cache_param->http_max_hdr);
-		if (sp->obj == NULL) {
+		if (sp->wrk->obj == NULL) {
 			sp->doclose = "Out of objects";
 			sp->director = NULL;
 			sp->wrk->h_content_length = NULL;
@@ -468,14 +468,14 @@ cnt_error(struct sess *sp)
 			sp->step = STP_DONE;
 			return(0);
 		}
-		AN(sp->obj);
-		sp->obj->xid = sp->xid;
-		sp->obj->exp.entered = sp->t_req;
+		AN(sp->wrk->obj);
+		sp->wrk->obj->xid = sp->xid;
+		sp->wrk->obj->exp.entered = sp->t_req;
 	} else {
 		/* XXX: Null the headers ? */
 	}
-	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);
-	h = sp->obj->http;
+	CHECK_OBJ_NOTNULL(sp->wrk->obj, OBJECT_MAGIC);
+	h = sp->wrk->obj->http;
 
 	if (sp->err_code < 100 || sp->err_code > 999)
 		sp->err_code = 501;
@@ -601,7 +601,7 @@ cnt_fetch(struct sess *sp)
 		RFC2616_Ttl(sp);
 
 		/* pass from vclrecv{} has negative TTL */
-		if (sp->objcore == NULL)
+		if (sp->wrk->objcore == NULL)
 			sp->wrk->exp.ttl = -1.;
 
 		AZ(sp->wrk->do_esi);
@@ -610,12 +610,12 @@ cnt_fetch(struct sess *sp)
 
 		switch (sp->handling) {
 		case VCL_RET_HIT_FOR_PASS:
-			if (sp->objcore != NULL)
-				sp->objcore->flags |= OC_F_PASS;
+			if (sp->wrk->objcore != NULL)
+				sp->wrk->objcore->flags |= OC_F_PASS;
 			sp->step = STP_FETCHBODY;
 			return (0);
 		case VCL_RET_DELIVER:
-			AssertObjCorePassOrBusy(sp->objcore);
+			AssertObjCorePassOrBusy(sp->wrk->objcore);
 			sp->step = STP_FETCHBODY;
 			return (0);
 		default:
@@ -629,10 +629,10 @@ cnt_fetch(struct sess *sp)
 	/* Clean up partial fetch */
 	AZ(sp->wrk->vbc);
 
-	if (sp->objcore != NULL) {
-		CHECK_OBJ_NOTNULL(sp->objcore, OBJCORE_MAGIC);
-		AZ(HSH_Deref(sp->wrk, sp->objcore, NULL));
-		sp->objcore = NULL;
+	if (sp->wrk->objcore != NULL) {
+		CHECK_OBJ_NOTNULL(sp->wrk->objcore, OBJCORE_MAGIC);
+		AZ(HSH_Deref(sp->wrk, sp->wrk->objcore, NULL));
+		sp->wrk->objcore = NULL;
 	}
 	http_Setup(sp->wrk->bereq, NULL);
 	http_Setup(sp->wrk->beresp, NULL);
@@ -688,7 +688,7 @@ cnt_fetchbody(struct sess *sp)
 	assert(sp->handling == VCL_RET_HIT_FOR_PASS ||
 	    sp->handling == VCL_RET_DELIVER);
 
-	if (sp->objcore == NULL) {
+	if (sp->wrk->objcore == NULL) {
 		/* This is a pass from vcl_recv */
 		pass = 1;
 		/* VCL may have fiddled this, but that doesn't help */
@@ -769,8 +769,8 @@ cnt_fetchbody(struct sess *sp)
 	    pass ? HTTPH_R_PASS : HTTPH_A_INS, &nhttp);
 
 	/* Create Vary instructions */
-	if (sp->objcore != NULL) {
-		CHECK_OBJ_NOTNULL(sp->objcore, OBJCORE_MAGIC);
+	if (sp->wrk->objcore != NULL) {
+		CHECK_OBJ_NOTNULL(sp->wrk->objcore, OBJCORE_MAGIC);
 		vary = VRY_Create(sp, sp->wrk->beresp);
 		if (vary != NULL) {
 			varyl = VSB_len(vary);
@@ -785,52 +785,52 @@ cnt_fetchbody(struct sess *sp)
 	 */
 	l += strlen("Content-Length: XxxXxxXxxXxxXxxXxx") + sizeof(void *);
 
-	if (sp->wrk->exp.ttl < cache_param->shortlived || sp->objcore == NULL)
+	if (sp->wrk->exp.ttl < cache_param->shortlived || sp->wrk->objcore == NULL)
 		sp->wrk->storage_hint = TRANSIENT_STORAGE;
 
-	sp->obj = STV_NewObject(sp, sp->wrk->storage_hint, l,
+	sp->wrk->obj = STV_NewObject(sp, sp->wrk->storage_hint, l,
 	    &sp->wrk->exp, nhttp);
-	if (sp->obj == NULL) {
+	if (sp->wrk->obj == NULL) {
 		/*
 		 * Try to salvage the transaction by allocating a
 		 * shortlived object on Transient storage.
 		 */
-		sp->obj = STV_NewObject(sp, TRANSIENT_STORAGE, l,
+		sp->wrk->obj = STV_NewObject(sp, TRANSIENT_STORAGE, l,
 		    &sp->wrk->exp, nhttp);
 		if (sp->wrk->exp.ttl > cache_param->shortlived)
 			sp->wrk->exp.ttl = cache_param->shortlived;
 		sp->wrk->exp.grace = 0.0;
 		sp->wrk->exp.keep = 0.0;
 	}
-	if (sp->obj == NULL) {
+	if (sp->wrk->obj == NULL) {
 		sp->err_code = 503;
 		sp->step = STP_ERROR;
 		VDI_CloseFd(sp->wrk);
 		return (0);
 	}
-	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->wrk->obj, OBJECT_MAGIC);
 
 	sp->wrk->storage_hint = NULL;
 
 	if (sp->wrk->do_gzip || (sp->wrk->is_gzip && !sp->wrk->do_gunzip))
-		sp->obj->gziped = 1;
+		sp->wrk->obj->gziped = 1;
 
 	if (vary != NULL) {
-		sp->obj->vary =
-		    (void *)WS_Alloc(sp->obj->http->ws, varyl);
-		AN(sp->obj->vary);
-		memcpy(sp->obj->vary, VSB_data(vary), varyl);
-		VRY_Validate(sp->obj->vary);
+		sp->wrk->obj->vary =
+		    (void *)WS_Alloc(sp->wrk->obj->http->ws, varyl);
+		AN(sp->wrk->obj->vary);
+		memcpy(sp->wrk->obj->vary, VSB_data(vary), varyl);
+		VRY_Validate(sp->wrk->obj->vary);
 		VSB_delete(vary);
 	}
 
-	sp->obj->xid = sp->xid;
-	sp->obj->response = sp->err_code;
-	WS_Assert(sp->obj->ws_o);
+	sp->wrk->obj->xid = sp->xid;
+	sp->wrk->obj->response = sp->err_code;
+	WS_Assert(sp->wrk->obj->ws_o);
 
 	/* Filter into object */
 	hp = sp->wrk->beresp;
-	hp2 = sp->obj->http;
+	hp2 = sp->wrk->obj->http;
 
 	hp2->logtag = HTTP_Obj;
 	http_CopyResp(hp2, hp);
@@ -839,9 +839,9 @@ cnt_fetchbody(struct sess *sp)
 	http_CopyHome(sp->wrk, sp->vsl_id, hp2);
 
 	if (http_GetHdr(hp, H_Last_Modified, &b))
-		sp->obj->last_modified = VTIM_parse(b);
+		sp->wrk->obj->last_modified = VTIM_parse(b);
 	else
-		sp->obj->last_modified = floor(sp->wrk->exp.entered);
+		sp->wrk->obj->last_modified = floor(sp->wrk->exp.entered);
 
 	assert(WRW_IsReleased(sp->wrk));
 
@@ -850,12 +850,12 @@ cnt_fetchbody(struct sess *sp)
 	 * Notice that vcl_deliver{} could still nuke the headers
 	 * that allow the 304, in which case we return 200 non-stream.
 	 */
-	if (sp->obj->response == 200 &&
+	if (sp->wrk->obj->response == 200 &&
 	    sp->http->conds &&
 	    RFC2616_Do_Cond(sp))
 		sp->wrk->do_stream = 0;
 
-	AssertObjCorePassOrBusy(sp->obj->objcore);
+	AssertObjCorePassOrBusy(sp->wrk->obj->objcore);
 
 	if (sp->wrk->do_stream) {
 		sp->step = STP_PREPRESP;
@@ -863,7 +863,7 @@ cnt_fetchbody(struct sess *sp)
 	}
 
 	/* Use unmodified headers*/
-	i = FetchBody(sp->wrk, sp->obj);
+	i = FetchBody(sp->wrk, sp->wrk->obj);
 
 	sp->wrk->h_content_length = NULL;
 
@@ -876,16 +876,16 @@ cnt_fetchbody(struct sess *sp)
 
 	if (i) {
 		HSH_Drop(sp);
-		AZ(sp->obj);
+		AZ(sp->wrk->obj);
 		sp->err_code = 503;
 		sp->step = STP_ERROR;
 		return (0);
 	}
 
-	if (sp->obj->objcore != NULL) {
-		EXP_Insert(sp->obj);
-		AN(sp->obj->objcore);
-		AN(sp->obj->objcore->ban);
+	if (sp->wrk->obj->objcore != NULL) {
+		EXP_Insert(sp->wrk->obj);
+		AN(sp->wrk->obj->objcore);
+		AN(sp->wrk->obj->objcore->ban);
 		HSH_Unbusy(sp);
 	}
 	sp->wrk->acct_tmp.fetch++;
@@ -925,9 +925,9 @@ cnt_streambody(struct sess *sp)
 
 	RES_StreamStart(sp);
 
-	AssertObjCorePassOrBusy(sp->obj->objcore);
+	AssertObjCorePassOrBusy(sp->wrk->obj->objcore);
 
-	i = FetchBody(sp->wrk, sp->obj);
+	i = FetchBody(sp->wrk, sp->wrk->obj);
 
 	sp->wrk->h_content_length = NULL;
 
@@ -937,10 +937,10 @@ cnt_streambody(struct sess *sp)
 	AZ(sp->wrk->vbc);
 	AN(sp->director);
 
-	if (!i && sp->obj->objcore != NULL) {
-		EXP_Insert(sp->obj);
-		AN(sp->obj->objcore);
-		AN(sp->obj->objcore->ban);
+	if (!i && sp->wrk->obj->objcore != NULL) {
+		EXP_Insert(sp->wrk->obj);
+		AN(sp->wrk->obj->objcore);
+		AN(sp->wrk->obj->objcore->ban);
 		HSH_Unbusy(sp);
 	} else {
 		sp->doclose = "Stream error";
@@ -956,7 +956,7 @@ cnt_streambody(struct sess *sp)
 	sp->wrk->sctx = NULL;
 	assert(WRW_IsReleased(sp->wrk));
 	assert(sp->wrk->wrw.ciov == sp->wrk->wrw.siov);
-	(void)HSH_Deref(sp->wrk, NULL, &sp->obj);
+	(void)HSH_Deref(sp->wrk, NULL, &sp->wrk->obj);
 	http_Setup(sp->wrk->resp, NULL);
 	sp->step = STP_DONE;
 	return (0);
@@ -1024,10 +1024,10 @@ cnt_hit(struct sess *sp)
 {
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->wrk->obj, OBJECT_MAGIC);
 	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
 
-	assert(!(sp->obj->objcore->flags & OC_F_PASS));
+	assert(!(sp->wrk->obj->objcore->flags & OC_F_PASS));
 
 	AZ(sp->wrk->do_stream);
 
@@ -1043,8 +1043,8 @@ cnt_hit(struct sess *sp)
 	}
 
 	/* Drop our object, we won't need it */
-	(void)HSH_Deref(sp->wrk, NULL, &sp->obj);
-	sp->objcore = NULL;
+	(void)HSH_Deref(sp->wrk, NULL, &sp->wrk->obj);
+	sp->wrk->objcore = NULL;
 
 	switch(sp->handling) {
 	case VCL_RET_PASS:
@@ -1146,14 +1146,14 @@ cnt_lookup(struct sess *sp)
 		sp->vary_l = NULL;
 		sp->vary_e = NULL;
 
-		sp->objcore = oc;
+		sp->wrk->objcore = oc;
 		sp->step = STP_MISS;
 		return (0);
 	}
 
 	o = oc_getobj(sp->wrk, oc);
 	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
-	sp->obj = o;
+	sp->wrk->obj = o;
 
 	WS_Release(sp->ws, 0);
 	sp->vary_b = NULL;
@@ -1162,15 +1162,15 @@ cnt_lookup(struct sess *sp)
 
 	if (oc->flags & OC_F_PASS) {
 		sp->wrk->stats.cache_hitpass++;
-		WSP(sp, SLT_HitPass, "%u", sp->obj->xid);
-		(void)HSH_Deref(sp->wrk, NULL, &sp->obj);
-		sp->objcore = NULL;
+		WSP(sp, SLT_HitPass, "%u", sp->wrk->obj->xid);
+		(void)HSH_Deref(sp->wrk, NULL, &sp->wrk->obj);
+		sp->wrk->objcore = NULL;
 		sp->step = STP_PASS;
 		return (0);
 	}
 
 	sp->wrk->stats.cache_hit++;
-	WSP(sp, SLT_Hit, "%u", sp->obj->xid);
+	WSP(sp, SLT_Hit, "%u", sp->wrk->obj->xid);
 	sp->step = STP_HIT;
 	return (0);
 }
@@ -1205,8 +1205,8 @@ cnt_miss(struct sess *sp)
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
 	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
 
-	AZ(sp->obj);
-	AN(sp->objcore);
+	AZ(sp->wrk->obj);
+	AN(sp->wrk->objcore);
 	WS_Reset(sp->wrk->ws, NULL);
 	http_Setup(sp->wrk->bereq, sp->wrk->ws);
 	http_FilterHeader(sp, HTTPH_R_FETCH);
@@ -1227,22 +1227,22 @@ cnt_miss(struct sess *sp)
 	VCL_miss_method(sp);
 	switch(sp->handling) {
 	case VCL_RET_ERROR:
-		AZ(HSH_Deref(sp->wrk, sp->objcore, NULL));
-		sp->objcore = NULL;
+		AZ(HSH_Deref(sp->wrk, sp->wrk->objcore, NULL));
+		sp->wrk->objcore = NULL;
 		http_Setup(sp->wrk->bereq, NULL);
 		sp->step = STP_ERROR;
 		return (0);
 	case VCL_RET_PASS:
-		AZ(HSH_Deref(sp->wrk, sp->objcore, NULL));
-		sp->objcore = NULL;
+		AZ(HSH_Deref(sp->wrk, sp->wrk->objcore, NULL));
+		sp->wrk->objcore = NULL;
 		sp->step = STP_PASS;
 		return (0);
 	case VCL_RET_FETCH:
 		sp->step = STP_FETCH;
 		return (0);
 	case VCL_RET_RESTART:
-		AZ(HSH_Deref(sp->wrk, sp->objcore, NULL));
-		sp->objcore = NULL;
+		AZ(HSH_Deref(sp->wrk, sp->wrk->objcore, NULL));
+		sp->wrk->objcore = NULL;
 		INCOMPL();
 	default:
 		WRONG("Illegal action in vcl_miss{}");
@@ -1287,7 +1287,7 @@ cnt_pass(struct sess *sp)
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
 	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
-	AZ(sp->obj);
+	AZ(sp->wrk->obj);
 
 	WS_Reset(sp->wrk->ws, NULL);
 	http_Setup(sp->wrk->bereq, sp->wrk->ws);
@@ -1386,7 +1386,7 @@ cnt_recv(struct sess *sp)
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
 	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
-	AZ(sp->obj);
+	AZ(sp->wrk->obj);
 	assert(sp->wrk->wrw.ciov == sp->wrk->wrw.siov);
 
 	/* By default we use the first backend */
@@ -1489,7 +1489,7 @@ cnt_start(struct sess *sp)
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
 	AZ(sp->restarts);
-	AZ(sp->obj);
+	AZ(sp->wrk->obj);
 	AZ(sp->vcl);
 	AZ(sp->esi_level);
 
@@ -1568,12 +1568,12 @@ cnt_diag(struct sess *sp, const char *state)
 {
 	if (sp->wrk != NULL) {
 		WSP(sp, SLT_Debug, "thr %p STP_%s sp %p obj %p vcl %p",
-		    pthread_self(), state, sp, sp->obj, sp->vcl);
+		    pthread_self(), state, sp, sp->wrk->obj, sp->vcl);
 		WSL_Flush(sp->wrk, 0);
 	} else {
 		VSL(SLT_Debug, sp->vsl_id,
 		    "thr %p STP_%s sp %p obj %p vcl %p",
-		    pthread_self(), state, sp, sp->obj, sp->vcl);
+		    pthread_self(), state, sp, sp->wrk->obj, sp->vcl);
 	}
 }
 
@@ -1602,6 +1602,8 @@ CNT_Session(struct sess *sp)
 	AZ(w->is_gunzip);
 	AZ(w->do_gunzip);
 	AZ(w->do_esi);
+	AZ(w->obj);
+	AZ(w->objcore);
 
 	/*
 	 * Whenever we come in from the acceptor or waiter, we need to set
@@ -1630,7 +1632,7 @@ CNT_Session(struct sess *sp)
 		 * pointers still pointing to the things we expect.
 		 */
 		CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-		CHECK_OBJ_ORNULL(sp->obj, OBJECT_MAGIC);
+		CHECK_OBJ_ORNULL(sp->wrk->obj, OBJECT_MAGIC);
 		CHECK_OBJ_NOTNULL(sp->wrk, WORKER_MAGIC);
 		CHECK_OBJ_ORNULL(w->nobjhead, OBJHEAD_MAGIC);
 		WS_Assert(w->ws);
@@ -1651,6 +1653,8 @@ CNT_Session(struct sess *sp)
 		CHECK_OBJ_ORNULL(w->nobjhead, OBJHEAD_MAGIC);
 	}
 	WSL_Flush(w, 0);
+	AZ(w->obj);
+	AZ(w->objcore);
 	AZ(w->do_stream);
 	AZ(w->is_gzip);
 	AZ(w->do_gzip);
diff --git a/bin/varnishd/cache/cache_esi_deliver.c b/bin/varnishd/cache/cache_esi_deliver.c
index 4051027..5ac8e71 100644
--- a/bin/varnishd/cache/cache_esi_deliver.c
+++ b/bin/varnishd/cache/cache_esi_deliver.c
@@ -58,8 +58,8 @@ ved_include(struct sess *sp, const char *src, const char *host)
 
 	(void)WRW_FlushRelease(w);
 
-	obj = sp->obj;
-	sp->obj = NULL;
+	obj = sp->wrk->obj;
+	sp->wrk->obj = NULL;
 	res_mode = sp->wrk->res_mode;
 
 	/* Reset request to status before we started messing with it */
@@ -113,7 +113,7 @@ ved_include(struct sess *sp, const char *src, const char *host)
 	AN(sp->wrk);
 	assert(sp->step == STP_DONE);
 	sp->esi_level--;
-	sp->obj = obj;
+	sp->wrk->obj = obj;
 	sp->wrk->res_mode = res_mode;
 
 	/* Reset the workspace */
@@ -238,7 +238,7 @@ ESI_Deliver(struct sess *sp)
 	int i;
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	st = sp->obj->esidata;
+	st = sp->wrk->obj->esidata;
 	AN(st);
 	assert(sizeof obuf >= 1024);
 
@@ -283,7 +283,7 @@ ESI_Deliver(struct sess *sp)
 		obufl = 0;
 	}
 
-	st = VTAILQ_FIRST(&sp->obj->store);
+	st = VTAILQ_FIRST(&sp->wrk->obj->store);
 	off = 0;
 
 	while (p < e) {
@@ -440,7 +440,7 @@ ved_deliver_byterange(const struct sess *sp, ssize_t low, ssize_t high)
 
 //printf("BR %jd %jd\n", low, high);
 	lx = 0;
-	VTAILQ_FOREACH(st, &sp->obj->store, list) {
+	VTAILQ_FOREACH(st, &sp->wrk->obj->store, list) {
 		p = st->ptr;
 		l = st->len;
 //printf("[0-] %jd %jd\n", lx, lx + l);
@@ -481,8 +481,8 @@ ESI_DeliverChild(const struct sess *sp)
 	uint32_t ilen;
 	uint8_t *dbits;
 
-	if (!sp->obj->gziped) {
-		VTAILQ_FOREACH(st, &sp->obj->store, list)
+	if (!sp->wrk->obj->gziped) {
+		VTAILQ_FOREACH(st, &sp->wrk->obj->store, list)
 			ved_pretend_gzip(sp, st->ptr, st->len);
 		return;
 	}
@@ -494,7 +494,7 @@ ESI_DeliverChild(const struct sess *sp)
 
 	dbits = (void*)WS_Alloc(sp->wrk->ws, 8);
 	AN(dbits);
-	obj = sp->obj;
+	obj = sp->wrk->obj;
 	CHECK_OBJ_NOTNULL(obj, OBJECT_MAGIC);
 	start = obj->gzip_start;
 	last = obj->gzip_last;
@@ -559,7 +559,7 @@ ESI_DeliverChild(const struct sess *sp)
 	}
 	if (lpad > 0)
 		(void)WRW_Write(sp->wrk, dbits + 1, lpad);
-	st = VTAILQ_LAST(&sp->obj->store, storagehead);
+	st = VTAILQ_LAST(&sp->wrk->obj->store, storagehead);
 	assert(st->len > 8);
 
 	p = st->ptr + st->len - 8;
diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index a5c0323..66241bc 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -389,11 +389,11 @@ FetchHdr(struct sess *sp)
 	w = sp->wrk;
 
 	AN(sp->director);
-	AZ(sp->obj);
+	AZ(sp->wrk->obj);
 
-	if (sp->objcore != NULL) {		/* pass has no objcore */
-		CHECK_OBJ_NOTNULL(sp->objcore, OBJCORE_MAGIC);
-		AN(sp->objcore->flags & OC_F_BUSY);
+	if (sp->wrk->objcore != NULL) {		/* pass has no objcore */
+		CHECK_OBJ_NOTNULL(sp->wrk->objcore, OBJCORE_MAGIC);
+		AN(sp->wrk->objcore->flags & OC_F_BUSY);
 	}
 
 	hp = w->bereq;
diff --git a/bin/varnishd/cache/cache_gzip.c b/bin/varnishd/cache/cache_gzip.c
index 32a7413..57b25a5 100644
--- a/bin/varnishd/cache/cache_gzip.c
+++ b/bin/varnishd/cache/cache_gzip.c
@@ -248,7 +248,7 @@ VGZ_ObufFull(const struct vgz *vg)
 
 /*--------------------------------------------------------------------
  * Keep the outbuffer supplied with storage and file it under the
- * sp->obj as it fills.
+ * sp->wrk->obj as it fills.
  */
 
 int
diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index db865de..2820c45 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -395,8 +395,8 @@ HSH_Lookup(struct sess *sp, struct objhead **poh)
 	 * XXX: serialize fetch of all Vary's if grace is possible.
 	 */
 
-	AZ(sp->objcore);
-	sp->objcore = grace_oc;		/* XXX: Hack-ish */
+	AZ(sp->wrk->objcore);
+	sp->wrk->objcore = grace_oc;		/* XXX: Hack-ish */
 	if (oc == NULL			/* We found no live object */
 	    && grace_oc != NULL		/* There is a grace candidate */
 	    && (busy_oc != NULL		/* Somebody else is already busy */
@@ -406,7 +406,7 @@ HSH_Lookup(struct sess *sp, struct objhead **poh)
 		CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
 		oc = grace_oc;
 	}
-	sp->objcore = NULL;
+	sp->wrk->objcore = NULL;
 
 	if (oc != NULL && !sp->hash_always_miss) {
 		o = oc_getobj(sp->wrk, oc);
@@ -585,13 +585,13 @@ HSH_Drop(struct sess *sp)
 	struct object *o;
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	o = sp->obj;
+	o = sp->wrk->obj;
 	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
 	AssertObjCorePassOrBusy(o->objcore);
 	o->exp.ttl = -1.;
 	if (o->objcore != NULL)		/* Pass has no objcore */
 		HSH_Unbusy(sp);
-	(void)HSH_Deref(sp->wrk, NULL, &sp->obj);
+	(void)HSH_Deref(sp->wrk, NULL, &sp->wrk->obj);
 }
 
 void
@@ -602,7 +602,7 @@ HSH_Unbusy(const struct sess *sp)
 	struct objcore *oc;
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	o = sp->obj;
+	o = sp->wrk->obj;
 	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
 	oc = o->objcore;
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
diff --git a/bin/varnishd/cache/cache_panic.c b/bin/varnishd/cache/cache_panic.c
index 9d69c1b..f6f1ff5 100644
--- a/bin/varnishd/cache/cache_panic.c
+++ b/bin/varnishd/cache/cache_panic.c
@@ -266,8 +266,8 @@ pan_sess(const struct sess *sp)
 	if (VALID_OBJ(sp->wrk->vbc, BACKEND_MAGIC))
 		pan_vbc(sp->wrk->vbc);
 
-	if (VALID_OBJ(sp->obj, OBJECT_MAGIC))
-		pan_object(sp->obj);
+	if (VALID_OBJ(sp->wrk->obj, OBJECT_MAGIC))
+		pan_object(sp->wrk->obj);
 
 	VSB_printf(pan_vsp, "},\n");
 }
diff --git a/bin/varnishd/cache/cache_pool.c b/bin/varnishd/cache/cache_pool.c
index 79a5fcd..730e391 100644
--- a/bin/varnishd/cache/cache_pool.c
+++ b/bin/varnishd/cache/cache_pool.c
@@ -361,7 +361,6 @@ Pool_Wait(struct sess *sp)
 {
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	AZ(sp->obj);
 	AZ(sp->vcl);
 	assert(sp->fd >= 0);
 	/*
diff --git a/bin/varnishd/cache/cache_response.c b/bin/varnishd/cache/cache_response.c
index 487a514..bdc0fca 100644
--- a/bin/varnishd/cache/cache_response.c
+++ b/bin/varnishd/cache/cache_response.c
@@ -41,7 +41,7 @@ res_dorange(const struct sess *sp, const char *r, ssize_t *plow, ssize_t *phigh)
 {
 	ssize_t low, high, has_low;
 
-	assert(sp->obj->response == 200);
+	assert(sp->wrk->obj->response == 200);
 	if (strncmp(r, "bytes=", 6))
 		return;
 	r += 6;
@@ -57,7 +57,7 @@ res_dorange(const struct sess *sp, const char *r, ssize_t *plow, ssize_t *phigh)
 		r++;
 	}
 
-	if (low >= sp->obj->len)
+	if (low >= sp->wrk->obj->len)
 		return;
 
 	if (*r != '-')
@@ -73,23 +73,23 @@ res_dorange(const struct sess *sp, const char *r, ssize_t *plow, ssize_t *phigh)
 			r++;
 		}
 		if (!has_low) {
-			low = sp->obj->len - high;
-			high = sp->obj->len - 1;
+			low = sp->wrk->obj->len - high;
+			high = sp->wrk->obj->len - 1;
 		}
 	} else
-		high = sp->obj->len - 1;
+		high = sp->wrk->obj->len - 1;
 	if (*r != '\0')
 		return;
 
-	if (high >= sp->obj->len)
-		high = sp->obj->len - 1;
+	if (high >= sp->wrk->obj->len)
+		high = sp->wrk->obj->len - 1;
 
 	if (low > high)
 		return;
 
 	http_PrintfHeader(sp->wrk, sp->vsl_id, sp->wrk->resp,
 	    "Content-Range: bytes %jd-%jd/%jd",
-	    (intmax_t)low, (intmax_t)high, (intmax_t)sp->obj->len);
+	    (intmax_t)low, (intmax_t)high, (intmax_t)sp->wrk->obj->len);
 	http_Unset(sp->wrk->resp, H_Content_Length);
 	assert(sp->wrk->res_mode & RES_LEN);
 	http_PrintfHeader(sp->wrk, sp->vsl_id, sp->wrk->resp,
@@ -112,8 +112,8 @@ RES_BuildHttp(const struct sess *sp)
 
 	http_ClrHeader(sp->wrk->resp);
 	sp->wrk->resp->logtag = HTTP_Tx;
-	http_CopyResp(sp->wrk->resp, sp->obj->http);
-	http_FilterFields(sp->wrk, sp->vsl_id, sp->wrk->resp, sp->obj->http,
+	http_CopyResp(sp->wrk->resp, sp->wrk->obj->http);
+	http_FilterFields(sp->wrk, sp->vsl_id, sp->wrk->resp, sp->wrk->obj->http,
 	    HTTPH_A_DELIVER);
 
 	if (!(sp->wrk->res_mode & RES_LEN)) {
@@ -131,14 +131,14 @@ RES_BuildHttp(const struct sess *sp)
 	VTIM_format(VTIM_real(), time_str);
 	http_PrintfHeader(sp->wrk, sp->vsl_id, sp->wrk->resp, "Date: %s", time_str);
 
-	if (sp->xid != sp->obj->xid)
+	if (sp->xid != sp->wrk->obj->xid)
 		http_PrintfHeader(sp->wrk, sp->vsl_id, sp->wrk->resp,
-		    "X-Varnish: %u %u", sp->xid, sp->obj->xid);
+		    "X-Varnish: %u %u", sp->xid, sp->wrk->obj->xid);
 	else
 		http_PrintfHeader(sp->wrk, sp->vsl_id, sp->wrk->resp,
 		    "X-Varnish: %u", sp->xid);
 	http_PrintfHeader(sp->wrk, sp->vsl_id, sp->wrk->resp, "Age: %.0f",
-	    sp->obj->exp.age + sp->t_resp - sp->obj->exp.entered);
+	    sp->wrk->obj->exp.age + sp->t_resp - sp->wrk->obj->exp.entered);
 	http_SetHeader(sp->wrk, sp->vsl_id, sp->wrk->resp, "Via: 1.1 varnish");
 	http_PrintfHeader(sp->wrk, sp->vsl_id, sp->wrk->resp, "Connection: %s",
 	    sp->doclose ? "close" : "keep-alive");
@@ -165,7 +165,7 @@ res_WriteGunzipObj(const struct sess *sp)
 	vg = VGZ_NewUngzip(sp->wrk, "U D -");
 
 	VGZ_Obuf(vg, obuf, sizeof obuf);
-	VTAILQ_FOREACH(st, &sp->obj->store, list) {
+	VTAILQ_FOREACH(st, &sp->wrk->obj->store, list) {
 		CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
 		CHECK_OBJ_NOTNULL(st, STORAGE_MAGIC);
 		u += st->len;
@@ -183,7 +183,7 @@ res_WriteGunzipObj(const struct sess *sp)
 		(void)WRW_Flush(sp->wrk);
 	}
 	(void)VGZ_Destroy(&vg, sp->vsl_id);
-	assert(u == sp->obj->len);
+	assert(u == sp->wrk->obj->len);
 }
 
 /*--------------------------------------------------------------------*/
@@ -198,7 +198,7 @@ res_WriteDirObj(const struct sess *sp, ssize_t low, ssize_t high)
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
 
 	ptr = 0;
-	VTAILQ_FOREACH(st, &sp->obj->store, list) {
+	VTAILQ_FOREACH(st, &sp->wrk->obj->store, list) {
 		CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
 		CHECK_OBJ_NOTNULL(st, STORAGE_MAGIC);
 		u += st->len;
@@ -239,7 +239,7 @@ res_WriteDirObj(const struct sess *sp, ssize_t low, ssize_t high)
 		VSC_C_main->n_objwrite++;
 		(void)WRW_Write(sp->wrk, st->ptr + off, len);
 	}
-	assert(u == sp->obj->len);
+	assert(u == sp->wrk->obj->len);
 }
 
 /*--------------------------------------------------------------------
@@ -257,7 +257,7 @@ RES_WriteObj(struct sess *sp)
 
 	WRW_Reserve(sp->wrk, &sp->fd);
 
-	if (sp->obj->response == 200 &&
+	if (sp->wrk->obj->response == 200 &&
 	    sp->http->conds &&
 	    RFC2616_Do_Cond(sp)) {
 		sp->wantbody = 0;
@@ -270,13 +270,13 @@ RES_WriteObj(struct sess *sp)
 	 * If nothing special planned, we can attempt Range support
 	 */
 	low = 0;
-	high = sp->obj->len - 1;
+	high = sp->wrk->obj->len - 1;
 	if (
 	    sp->wantbody &&
 	    (sp->wrk->res_mode & RES_LEN) &&
 	    !(sp->wrk->res_mode & (RES_ESI|RES_ESI_CHILD|RES_GUNZIP)) &&
 	    cache_param->http_range_support &&
-	    sp->obj->response == 200 &&
+	    sp->wrk->obj->response == 200 &&
 	    http_GetHdr(sp->http, H_Range, &r))
 		res_dorange(sp, r, &low, &high);
 
@@ -301,14 +301,14 @@ RES_WriteObj(struct sess *sp)
 
 	if (!sp->wantbody) {
 		/* This was a HEAD or conditional request */
-	} else if (sp->obj->len == 0) {
+	} else if (sp->wrk->obj->len == 0) {
 		/* Nothing to do here */
 	} else if (sp->wrk->res_mode & RES_ESI) {
 		ESI_Deliver(sp);
 	} else if (sp->wrk->res_mode & RES_ESI_CHILD && sp->wrk->gzip_resp) {
 		ESI_DeliverChild(sp);
 	} else if (sp->wrk->res_mode & RES_ESI_CHILD &&
-	    !sp->wrk->gzip_resp && sp->obj->gziped) {
+	    !sp->wrk->gzip_resp && sp->wrk->obj->gziped) {
 		res_WriteGunzipObj(sp);
 	} else if (sp->wrk->res_mode & RES_GUNZIP) {
 		res_WriteGunzipObj(sp);
diff --git a/bin/varnishd/cache/cache_rfc2616.c b/bin/varnishd/cache/cache_rfc2616.c
index 4041f45..e22da50 100644
--- a/bin/varnishd/cache/cache_rfc2616.c
+++ b/bin/varnishd/cache/cache_rfc2616.c
@@ -315,18 +315,18 @@ RFC2616_Do_Cond(const struct sess *sp)
 	   and If-Modified-Since if present*/
 
 	if (http_GetHdr(sp->http, H_If_Modified_Since, &p) ) {
-		if (!sp->obj->last_modified)
+		if (!sp->wrk->obj->last_modified)
 			return (0);
 		ims = VTIM_parse(p);
 		if (ims > sp->t_req)	/* [RFC2616 14.25] */
 			return (0);
-		if (sp->obj->last_modified > ims)
+		if (sp->wrk->obj->last_modified > ims)
 			return (0);
 		do_cond = 1;
 	}
 
 	if (http_GetHdr(sp->http, H_If_None_Match, &p) &&
-	    http_GetHdr(sp->obj->http, H_ETag, &e)) {
+	    http_GetHdr(sp->wrk->obj->http, H_ETag, &e)) {
 		if (strcmp(p,e) != 0)
 			return (0);
 		do_cond = 1;
diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c
index eee8e9a..21d7f51 100644
--- a/bin/varnishd/cache/cache_session.c
+++ b/bin/varnishd/cache/cache_session.c
@@ -339,7 +339,6 @@ SES_Delete(struct sess *sp, const char *reason)
 		SES_Close(sp, reason);
 	assert(sp->fd < 0);
 
-	AZ(sp->obj);
 	AZ(sp->vcl);
 	if (sp->addr == NULL)
 		sp->addr = noaddr;
diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c
index 5e19ccc..a0eb4c8 100644
--- a/bin/varnishd/cache/cache_vrt.c
+++ b/bin/varnishd/cache/cache_vrt.c
@@ -108,8 +108,8 @@ vrt_selecthttp(const struct sess *sp, enum gethdr_e where)
 		hp = sp->wrk->resp;
 		break;
 	case HDR_OBJ:
-		CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);
-		hp = sp->obj->http;
+		CHECK_OBJ_NOTNULL(sp->wrk->obj, OBJECT_MAGIC);
+		hp = sp->wrk->obj->http;
 		break;
 	default:
 		INCOMPL();
@@ -399,8 +399,8 @@ VRT_synth_page(const struct sess *sp, unsigned flags, const char *str, ...)
 
 	(void)flags;
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);
-	vsb = SMS_Makesynth(sp->obj);
+	CHECK_OBJ_NOTNULL(sp->wrk->obj, OBJECT_MAGIC);
+	vsb = SMS_Makesynth(sp->wrk->obj);
 	AN(vsb);
 
 	VSB_cat(vsb, str);
@@ -413,10 +413,10 @@ VRT_synth_page(const struct sess *sp, unsigned flags, const char *str, ...)
 		p = va_arg(ap, const char *);
 	}
 	va_end(ap);
-	SMS_Finish(sp->obj);
-	http_Unset(sp->obj->http, H_Content_Length);
-	http_PrintfHeader(sp->wrk, sp->vsl_id, sp->obj->http,
-	    "Content-Length: %d", sp->obj->len);
+	SMS_Finish(sp->wrk->obj);
+	http_Unset(sp->wrk->obj->http, H_Content_Length);
+	http_PrintfHeader(sp->wrk, sp->vsl_id, sp->wrk->obj->http,
+	    "Content-Length: %d", sp->wrk->obj->len);
 }
 
 /*--------------------------------------------------------------------*/
@@ -510,9 +510,9 @@ void
 VRT_purge(const struct sess *sp, double ttl, double grace)
 {
 	if (sp->cur_method == VCL_MET_HIT)
-		HSH_Purge(sp, sp->obj->objcore->objhead, ttl, grace);
+		HSH_Purge(sp, sp->wrk->obj->objcore->objhead, ttl, grace);
 	else if (sp->cur_method == VCL_MET_MISS)
-		HSH_Purge(sp, sp->objcore->objhead, ttl, grace);
+		HSH_Purge(sp, sp->wrk->objcore->objhead, ttl, grace);
 }
 
 /*--------------------------------------------------------------------
diff --git a/bin/varnishd/cache/cache_vrt_var.c b/bin/varnishd/cache/cache_vrt_var.c
index 860c7aa..5951b9c 100644
--- a/bin/varnishd/cache/cache_vrt_var.c
+++ b/bin/varnishd/cache/cache_vrt_var.c
@@ -88,8 +88,8 @@ VRT_DO_HDR(req,   proto,	sp->http,		HTTP_HDR_PROTO)
 VRT_DO_HDR(bereq, request,	sp->wrk->bereq,		HTTP_HDR_REQ)
 VRT_DO_HDR(bereq, url,		sp->wrk->bereq,		HTTP_HDR_URL)
 VRT_DO_HDR(bereq, proto,	sp->wrk->bereq,		HTTP_HDR_PROTO)
-VRT_DO_HDR(obj,   proto,	sp->obj->http,		HTTP_HDR_PROTO)
-VRT_DO_HDR(obj,   response,	sp->obj->http,		HTTP_HDR_RESPONSE)
+VRT_DO_HDR(obj,   proto,	sp->wrk->obj->http,		HTTP_HDR_PROTO)
+VRT_DO_HDR(obj,   response,	sp->wrk->obj->http,		HTTP_HDR_RESPONSE)
 VRT_DO_HDR(resp,  proto,	sp->wrk->resp,		HTTP_HDR_PROTO)
 VRT_DO_HDR(resp,  response,	sp->wrk->resp,		HTTP_HDR_RESPONSE)
 VRT_DO_HDR(beresp,  proto,	sp->wrk->beresp,	HTTP_HDR_PROTO)
@@ -114,7 +114,7 @@ VRT_r_##obj##_status(const struct sess *sp)			\
 	return(http->status);					\
 }
 
-VRT_DO_STATUS(obj, sp->obj->http)
+VRT_DO_STATUS(obj, sp->wrk->obj->http)
 VRT_DO_STATUS(beresp, sp->wrk->beresp)
 VRT_DO_STATUS(resp, sp->wrk->resp)
 
@@ -141,9 +141,9 @@ VRT_l_beresp_saintmode(const struct sess *sp, double a)
 	if (!wrk->vbc->backend)
 		return;
 	CHECK_OBJ_NOTNULL(wrk->vbc->backend, BACKEND_MAGIC);
-	if (!sp->objcore)
+	if (!sp->wrk->objcore)
 		return;
-	CHECK_OBJ_NOTNULL(sp->objcore, OBJCORE_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->wrk->objcore, OBJCORE_MAGIC);
 
 	/* Setting a negative holdoff period is a mistake. Detecting this
 	 * when compiling the VCL would be better.
@@ -152,7 +152,7 @@ VRT_l_beresp_saintmode(const struct sess *sp, double a)
 
 	ALLOC_OBJ(new, TROUBLE_MAGIC);
 	AN(new);
-	new->target = (uintptr_t)(sp->objcore->objhead);
+	new->target = (uintptr_t)(sp->wrk->objcore->objhead);
 	new->timeout = sp->t_req + a;
 
 	/* Insert the new item on the list before the first item with a
@@ -401,15 +401,15 @@ VRT_DO_EXP(req, sp->exp, ttl, 0, )
 VRT_DO_EXP(req, sp->exp, grace, 0, )
 VRT_DO_EXP(req, sp->exp, keep, 0, )
 
-VRT_DO_EXP(obj, sp->obj->exp, grace, 0,
-   EXP_Rearm(sp->obj);
-   vrt_wsp_exp(sp, sp->obj->xid, &sp->obj->exp);)
-VRT_DO_EXP(obj, sp->obj->exp, ttl, (sp->t_req - sp->obj->exp.entered),
-   EXP_Rearm(sp->obj);
-   vrt_wsp_exp(sp, sp->obj->xid, &sp->obj->exp);)
-VRT_DO_EXP(obj, sp->obj->exp, keep, 0,
-   EXP_Rearm(sp->obj);
-   vrt_wsp_exp(sp, sp->obj->xid, &sp->obj->exp);)
+VRT_DO_EXP(obj, sp->wrk->obj->exp, grace, 0,
+   EXP_Rearm(sp->wrk->obj);
+   vrt_wsp_exp(sp, sp->wrk->obj->xid, &sp->wrk->obj->exp);)
+VRT_DO_EXP(obj, sp->wrk->obj->exp, ttl, (sp->t_req - sp->wrk->obj->exp.entered),
+   EXP_Rearm(sp->wrk->obj);
+   vrt_wsp_exp(sp, sp->wrk->obj->xid, &sp->wrk->obj->exp);)
+VRT_DO_EXP(obj, sp->wrk->obj->exp, keep, 0,
+   EXP_Rearm(sp->wrk->obj);
+   vrt_wsp_exp(sp, sp->wrk->obj->xid, &sp->wrk->obj->exp);)
 
 VRT_DO_EXP(beresp, sp->wrk->exp, grace, 0,
    vrt_wsp_exp(sp, sp->xid, &sp->wrk->exp);)
@@ -527,8 +527,8 @@ VRT_r_obj_hits(const struct sess *sp)
 {
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);	/* XXX */
-	return (sp->obj->hits);
+	CHECK_OBJ_NOTNULL(sp->wrk->obj, OBJECT_MAGIC);	/* XXX */
+	return (sp->wrk->obj->hits);
 }
 
 double
@@ -536,8 +536,8 @@ VRT_r_obj_lastuse(const struct sess *sp)
 {
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);	/* XXX */
-	return (VTIM_real() - sp->obj->last_use);
+	CHECK_OBJ_NOTNULL(sp->wrk->obj, OBJECT_MAGIC);	/* XXX */
+	return (VTIM_real() - sp->wrk->obj->last_use);
 }
 
 unsigned
diff --git a/bin/varnishd/storage/stevedore.c b/bin/varnishd/storage/stevedore.c
index 860604e..42479c3 100644
--- a/bin/varnishd/storage/stevedore.c
+++ b/bin/varnishd/storage/stevedore.c
@@ -246,11 +246,11 @@ STV_MkObject(struct sess *sp, void *ptr, unsigned ltot,
 	VTAILQ_INIT(&o->store);
 	sp->wrk->stats.n_object++;
 
-	if (sp->objcore != NULL) {
-		CHECK_OBJ_NOTNULL(sp->objcore, OBJCORE_MAGIC);
+	if (sp->wrk->objcore != NULL) {
+		CHECK_OBJ_NOTNULL(sp->wrk->objcore, OBJCORE_MAGIC);
 
-		o->objcore = sp->objcore;
-		sp->objcore = NULL;     /* refcnt follows pointer. */
+		o->objcore = sp->wrk->objcore;
+		sp->wrk->objcore = NULL;     /* refcnt follows pointer. */
 		BAN_NewObjCore(o->objcore);
 
 		o->objcore->methods = &default_oc_methods;
diff --git a/bin/varnishd/storage/storage_persistent.c b/bin/varnishd/storage/storage_persistent.c
index ded638b..838dc8c 100644
--- a/bin/varnishd/storage/storage_persistent.c
+++ b/bin/varnishd/storage/storage_persistent.c
@@ -474,10 +474,10 @@ smp_allocobj(struct stevedore *stv, struct sess *sp, unsigned ltot,
 	struct objcore *oc;
 	unsigned objidx;
 
-	if (sp->objcore == NULL)
+	if (sp->wrk->objcore == NULL)
 		return (NULL);		/* from cnt_error */
 	CAST_OBJ_NOTNULL(sc, stv->priv, SMP_SC_MAGIC);
-	AN(sp->objcore);
+	AN(sp->wrk->objcore);
 	AN(sp->wrk->exp.ttl > 0.);
 
 	ltot = IRNUP(sc, ltot);
diff --git a/bin/varnishd/waiter/cache_waiter_kqueue.c b/bin/varnishd/waiter/cache_waiter_kqueue.c
index b331a31..23df3de 100644
--- a/bin/varnishd/waiter/cache_waiter_kqueue.c
+++ b/bin/varnishd/waiter/cache_waiter_kqueue.c
@@ -102,7 +102,6 @@ vwk_kev(struct vwk *vwk, const struct kevent *kp)
 		while (i >= sizeof ss[0]) {
 			CHECK_OBJ_NOTNULL(ss[j], SESS_MAGIC);
 			assert(ss[j]->fd >= 0);
-			AZ(ss[j]->obj);
 			VTAILQ_INSERT_TAIL(&vwk->sesshead, ss[j], list);
 			vwk_kq_sess(vwk, ss[j], EV_ADD | EV_ONESHOT);
 			j++;
diff --git a/bin/varnishd/waiter/cache_waiter_ports.c b/bin/varnishd/waiter/cache_waiter_ports.c
index a1699d0..17d7ce0 100644
--- a/bin/varnishd/waiter/cache_waiter_ports.c
+++ b/bin/varnishd/waiter/cache_waiter_ports.c
@@ -77,7 +77,6 @@ vws_port_ev(struct vws *vws, port_event_t *ev) {
 	if(ev->portev_source == PORT_SOURCE_USER) {
 		CAST_OBJ_NOTNULL(sp, ev->portev_user, SESS_MAGIC);
 		assert(sp->fd >= 0);
-		AZ(sp->obj);
 		VTAILQ_INSERT_TAIL(&vws->sesshead, sp, list);
 		vws_add(vws, sp->fd, sp);
 	} else {



From phk at varnish-cache.org  Tue Nov 29 12:39:16 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 29 Nov 2011 13:39:16 +0100
Subject: [master] efd6a98 Consequence change of obj/objcore move from sp to
	wrk:
Message-ID: <E1RVMxw-0001RN-46@project.varnish-software.com>

commit efd6a9801ac6664d6d166880d73c75356dc47aeb
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 29 12:38:36 2011 +0000

    Consequence change of obj/objcore move from sp to wrk:
    
    Take sess out of STV_NewObject
    eliminate a lot of sp->wrk with a local wrk variable

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index db001ba..8371278 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -948,8 +948,8 @@ unsigned RFC2616_Req_Gzip(const struct sess *sp);
 int RFC2616_Do_Cond(const struct sess *sp);
 
 /* stevedore.c */
-struct object *STV_NewObject(struct sess *sp, const char *hint, unsigned len,
-    struct exp *, uint16_t nhttp);
+struct object *STV_NewObject(struct worker *wrk, const char *hint, unsigned len,
+    uint16_t nhttp);
 struct storage *STV_alloc(struct worker *w, size_t size);
 void STV_trim(struct storage *st, size_t size);
 void STV_free(struct storage *st);
diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c
index 28c4760..13db2e3 100644
--- a/bin/varnishd/cache/cache_center.c
+++ b/bin/varnishd/cache/cache_center.c
@@ -442,22 +442,22 @@ cnt_error(struct sess *sp)
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
 
-	sp->wrk->do_esi = 0;
-	sp->wrk->is_gzip = 0;
-	sp->wrk->is_gunzip = 0;
-	sp->wrk->do_gzip = 0;
-	sp->wrk->do_gunzip = 0;
-	sp->wrk->do_stream = 0;
-
 	w = sp->wrk;
-	if (sp->wrk->obj == NULL) {
+	w->do_esi = 0;
+	w->is_gzip = 0;
+	w->is_gunzip = 0;
+	w->do_gzip = 0;
+	w->do_gunzip = 0;
+	w->do_stream = 0;
+
+	if (w->obj == NULL) {
 		HSH_Prealloc(sp);
 		EXP_Clr(&w->exp);
-		sp->wrk->obj = STV_NewObject(sp, NULL, cache_param->http_resp_size,
-		     &w->exp, (uint16_t)cache_param->http_max_hdr);
+		w->obj = STV_NewObject(w, NULL, cache_param->http_resp_size,
+		     (uint16_t)cache_param->http_max_hdr);
 		if (sp->wrk->obj == NULL)
-			sp->wrk->obj = STV_NewObject(sp, TRANSIENT_STORAGE,
-			    cache_param->http_resp_size, &w->exp,
+			w->obj = STV_NewObject(w, TRANSIENT_STORAGE,
+			    cache_param->http_resp_size,
 			    (uint16_t)cache_param->http_max_hdr);
 		if (sp->wrk->obj == NULL) {
 			sp->doclose = "Out of objects";
@@ -684,15 +684,18 @@ cnt_fetchbody(struct sess *sp)
 	unsigned l;
 	struct vsb *vary = NULL;
 	int varyl = 0, pass;
+	struct worker *wrk;
+
+	wrk = sp->wrk;
 
 	assert(sp->handling == VCL_RET_HIT_FOR_PASS ||
 	    sp->handling == VCL_RET_DELIVER);
 
-	if (sp->wrk->objcore == NULL) {
+	if (wrk->objcore == NULL) {
 		/* This is a pass from vcl_recv */
 		pass = 1;
 		/* VCL may have fiddled this, but that doesn't help */
-		sp->wrk->exp.ttl = -1.;
+		wrk->exp.ttl = -1.;
 	} else if (sp->handling == VCL_RET_HIT_FOR_PASS) {
 		/* pass from vcl_fetch{} -> hit-for-pass */
 		/* XXX: the bereq was not filtered pass... */
@@ -715,63 +718,61 @@ cnt_fetchbody(struct sess *sp)
 	 *
 	 */
 
-	AZ(sp->wrk->vfp);
+	AZ(wrk->vfp);
 
 	/* We do nothing unless the param is set */
 	if (!cache_param->http_gzip_support)
-		sp->wrk->do_gzip = sp->wrk->do_gunzip = 0;
+		wrk->do_gzip = wrk->do_gunzip = 0;
 
-	sp->wrk->is_gzip =
-	    http_HdrIs(sp->wrk->beresp, H_Content_Encoding, "gzip");
+	wrk->is_gzip = http_HdrIs(wrk->beresp, H_Content_Encoding, "gzip");
 
-	sp->wrk->is_gunzip =
-	    !http_GetHdr(sp->wrk->beresp, H_Content_Encoding, NULL);
+	wrk->is_gunzip = !http_GetHdr(wrk->beresp, H_Content_Encoding, NULL);
 
 	/* It can't be both */
-	assert(sp->wrk->is_gzip == 0 || sp->wrk->is_gunzip == 0);
+	assert(wrk->is_gzip == 0 || wrk->is_gunzip == 0);
 
 	/* We won't gunzip unless it is gzip'ed */
-	if (sp->wrk->do_gunzip && !sp->wrk->is_gzip)
-		sp->wrk->do_gunzip = 0;
+	if (wrk->do_gunzip && !wrk->is_gzip)
+		wrk->do_gunzip = 0;
 
 	/* If we do gunzip, remove the C-E header */
-	if (sp->wrk->do_gunzip)
-		http_Unset(sp->wrk->beresp, H_Content_Encoding);
+	if (wrk->do_gunzip)
+		http_Unset(wrk->beresp, H_Content_Encoding);
 
 	/* We wont gzip unless it is ungziped */
-	if (sp->wrk->do_gzip && !sp->wrk->is_gunzip)
-		sp->wrk->do_gzip = 0;
+	if (wrk->do_gzip && !wrk->is_gunzip)
+		wrk->do_gzip = 0;
 
 	/* If we do gzip, add the C-E header */
-	if (sp->wrk->do_gzip)
-		http_SetHeader(sp->wrk, sp->vsl_id, sp->wrk->beresp,
+	if (wrk->do_gzip)
+		http_SetHeader(wrk, sp->vsl_id, wrk->beresp,
 		    "Content-Encoding: gzip");
 
 	/* But we can't do both at the same time */
-	assert(sp->wrk->do_gzip == 0 || sp->wrk->do_gunzip == 0);
+	assert(wrk->do_gzip == 0 || wrk->do_gunzip == 0);
 
 	/* ESI takes precedence and handles gzip/gunzip itself */
-	if (sp->wrk->do_esi)
-		sp->wrk->vfp = &vfp_esi;
-	else if (sp->wrk->do_gunzip)
-		sp->wrk->vfp = &vfp_gunzip;
-	else if (sp->wrk->do_gzip)
-		sp->wrk->vfp = &vfp_gzip;
-	else if (sp->wrk->is_gzip)
-		sp->wrk->vfp = &vfp_testgzip;
-
-	if (sp->wrk->do_esi || sp->esi_level > 0)
-		sp->wrk->do_stream = 0;
+	if (wrk->do_esi)
+		wrk->vfp = &vfp_esi;
+	else if (wrk->do_gunzip)
+		wrk->vfp = &vfp_gunzip;
+	else if (wrk->do_gzip)
+		wrk->vfp = &vfp_gzip;
+	else if (wrk->is_gzip)
+		wrk->vfp = &vfp_testgzip;
+
+	if (wrk->do_esi || sp->esi_level > 0)
+		wrk->do_stream = 0;
 	if (!sp->wantbody)
-		sp->wrk->do_stream = 0;
+		wrk->do_stream = 0;
 
-	l = http_EstimateWS(sp->wrk->beresp,
+	l = http_EstimateWS(wrk->beresp,
 	    pass ? HTTPH_R_PASS : HTTPH_A_INS, &nhttp);
 
 	/* Create Vary instructions */
-	if (sp->wrk->objcore != NULL) {
-		CHECK_OBJ_NOTNULL(sp->wrk->objcore, OBJCORE_MAGIC);
-		vary = VRY_Create(sp, sp->wrk->beresp);
+	if (wrk->objcore != NULL) {
+		CHECK_OBJ_NOTNULL(wrk->objcore, OBJCORE_MAGIC);
+		vary = VRY_Create(sp, wrk->beresp);
 		if (vary != NULL) {
 			varyl = VSB_len(vary);
 			assert(varyl > 0);
@@ -785,110 +786,107 @@ cnt_fetchbody(struct sess *sp)
 	 */
 	l += strlen("Content-Length: XxxXxxXxxXxxXxxXxx") + sizeof(void *);
 
-	if (sp->wrk->exp.ttl < cache_param->shortlived || sp->wrk->objcore == NULL)
-		sp->wrk->storage_hint = TRANSIENT_STORAGE;
+	if (wrk->exp.ttl < cache_param->shortlived || wrk->objcore == NULL)
+		wrk->storage_hint = TRANSIENT_STORAGE;
 
-	sp->wrk->obj = STV_NewObject(sp, sp->wrk->storage_hint, l,
-	    &sp->wrk->exp, nhttp);
-	if (sp->wrk->obj == NULL) {
+	wrk->obj = STV_NewObject(wrk, wrk->storage_hint, l, nhttp);
+	if (wrk->obj == NULL) {
 		/*
 		 * Try to salvage the transaction by allocating a
 		 * shortlived object on Transient storage.
 		 */
-		sp->wrk->obj = STV_NewObject(sp, TRANSIENT_STORAGE, l,
-		    &sp->wrk->exp, nhttp);
-		if (sp->wrk->exp.ttl > cache_param->shortlived)
-			sp->wrk->exp.ttl = cache_param->shortlived;
-		sp->wrk->exp.grace = 0.0;
-		sp->wrk->exp.keep = 0.0;
+		wrk->obj = STV_NewObject(wrk, TRANSIENT_STORAGE, l, nhttp);
+		if (wrk->exp.ttl > cache_param->shortlived)
+			wrk->exp.ttl = cache_param->shortlived;
+		wrk->exp.grace = 0.0;
+		wrk->exp.keep = 0.0;
 	}
-	if (sp->wrk->obj == NULL) {
+	if (wrk->obj == NULL) {
 		sp->err_code = 503;
 		sp->step = STP_ERROR;
-		VDI_CloseFd(sp->wrk);
+		VDI_CloseFd(wrk);
 		return (0);
 	}
-	CHECK_OBJ_NOTNULL(sp->wrk->obj, OBJECT_MAGIC);
+	CHECK_OBJ_NOTNULL(wrk->obj, OBJECT_MAGIC);
 
-	sp->wrk->storage_hint = NULL;
+	wrk->storage_hint = NULL;
 
-	if (sp->wrk->do_gzip || (sp->wrk->is_gzip && !sp->wrk->do_gunzip))
-		sp->wrk->obj->gziped = 1;
+	if (wrk->do_gzip || (wrk->is_gzip && !wrk->do_gunzip))
+		wrk->obj->gziped = 1;
 
 	if (vary != NULL) {
-		sp->wrk->obj->vary =
-		    (void *)WS_Alloc(sp->wrk->obj->http->ws, varyl);
-		AN(sp->wrk->obj->vary);
-		memcpy(sp->wrk->obj->vary, VSB_data(vary), varyl);
-		VRY_Validate(sp->wrk->obj->vary);
+		wrk->obj->vary = (void *)WS_Alloc(wrk->obj->http->ws, varyl);
+		AN(wrk->obj->vary);
+		memcpy(wrk->obj->vary, VSB_data(vary), varyl);
+		VRY_Validate(wrk->obj->vary);
 		VSB_delete(vary);
 	}
 
-	sp->wrk->obj->xid = sp->xid;
-	sp->wrk->obj->response = sp->err_code;
-	WS_Assert(sp->wrk->obj->ws_o);
+	wrk->obj->xid = sp->xid;
+	wrk->obj->response = sp->err_code;
+	WS_Assert(wrk->obj->ws_o);
 
 	/* Filter into object */
-	hp = sp->wrk->beresp;
-	hp2 = sp->wrk->obj->http;
+	hp = wrk->beresp;
+	hp2 = wrk->obj->http;
 
 	hp2->logtag = HTTP_Obj;
 	http_CopyResp(hp2, hp);
-	http_FilterFields(sp->wrk, sp->vsl_id, hp2, hp,
+	http_FilterFields(wrk, sp->vsl_id, hp2, hp,
 	    pass ? HTTPH_R_PASS : HTTPH_A_INS);
-	http_CopyHome(sp->wrk, sp->vsl_id, hp2);
+	http_CopyHome(wrk, sp->vsl_id, hp2);
 
 	if (http_GetHdr(hp, H_Last_Modified, &b))
-		sp->wrk->obj->last_modified = VTIM_parse(b);
+		wrk->obj->last_modified = VTIM_parse(b);
 	else
-		sp->wrk->obj->last_modified = floor(sp->wrk->exp.entered);
+		wrk->obj->last_modified = floor(wrk->exp.entered);
 
-	assert(WRW_IsReleased(sp->wrk));
+	assert(WRW_IsReleased(wrk));
 
 	/*
 	 * If we can deliver a 304 reply, we don't bother streaming.
 	 * Notice that vcl_deliver{} could still nuke the headers
 	 * that allow the 304, in which case we return 200 non-stream.
 	 */
-	if (sp->wrk->obj->response == 200 &&
+	if (wrk->obj->response == 200 &&
 	    sp->http->conds &&
 	    RFC2616_Do_Cond(sp))
-		sp->wrk->do_stream = 0;
+		wrk->do_stream = 0;
 
-	AssertObjCorePassOrBusy(sp->wrk->obj->objcore);
+	AssertObjCorePassOrBusy(wrk->obj->objcore);
 
-	if (sp->wrk->do_stream) {
+	if (wrk->do_stream) {
 		sp->step = STP_PREPRESP;
 		return (0);
 	}
 
 	/* Use unmodified headers*/
-	i = FetchBody(sp->wrk, sp->wrk->obj);
+	i = FetchBody(wrk, wrk->obj);
 
-	sp->wrk->h_content_length = NULL;
+	wrk->h_content_length = NULL;
 
-	http_Setup(sp->wrk->bereq, NULL);
-	http_Setup(sp->wrk->beresp, NULL);
-	sp->wrk->vfp = NULL;
-	assert(WRW_IsReleased(sp->wrk));
-	AZ(sp->wrk->vbc);
+	http_Setup(wrk->bereq, NULL);
+	http_Setup(wrk->beresp, NULL);
+	wrk->vfp = NULL;
+	assert(WRW_IsReleased(wrk));
+	AZ(wrk->vbc);
 	AN(sp->director);
 
 	if (i) {
 		HSH_Drop(sp);
-		AZ(sp->wrk->obj);
+		AZ(wrk->obj);
 		sp->err_code = 503;
 		sp->step = STP_ERROR;
 		return (0);
 	}
 
-	if (sp->wrk->obj->objcore != NULL) {
-		EXP_Insert(sp->wrk->obj);
-		AN(sp->wrk->obj->objcore);
-		AN(sp->wrk->obj->objcore->ban);
+	if (wrk->obj->objcore != NULL) {
+		EXP_Insert(wrk->obj);
+		AN(wrk->obj->objcore);
+		AN(wrk->obj->objcore->ban);
 		HSH_Unbusy(sp);
 	}
-	sp->wrk->acct_tmp.fetch++;
+	wrk->acct_tmp.fetch++;
 	sp->step = STP_PREPRESP;
 	return (0);
 }
@@ -970,7 +968,7 @@ DOT		shape=box
 DOT		label="first\nConfigure data structures"
 DOT	]
 DOT }
-DOT first -> wait 
+DOT first -> wait
  */
 
 static int
diff --git a/bin/varnishd/storage/stevedore.c b/bin/varnishd/storage/stevedore.c
index 42479c3..1a4b8ac 100644
--- a/bin/varnishd/storage/stevedore.c
+++ b/bin/varnishd/storage/stevedore.c
@@ -117,7 +117,7 @@ LRU_Free(struct lru *lru)
  */
 
 static struct stevedore *
-stv_pick_stevedore(const struct sess *sp, const char **hint)
+stv_pick_stevedore(struct worker *wrk, const char **hint)
 {
 	struct stevedore *stv;
 
@@ -131,7 +131,8 @@ stv_pick_stevedore(const struct sess *sp, const char **hint)
 			return (stv_transient);
 
 		/* Hint was not valid, nuke it */
-		WSP(sp, SLT_Debug, "Storage hint not usable");
+		WSL(wrk, SLT_Debug, wrk->htc->vsl_id,
+		    "Storage hint not usable");
 		*hint = NULL;
 	}
 	/* pick a stevedore and bump the head along */
@@ -202,7 +203,6 @@ struct stv_objsecrets {
 	uint16_t	nhttp;
 	unsigned	lhttp;
 	unsigned	wsl;
-	struct exp	*exp;
 };
 
 /*--------------------------------------------------------------------
@@ -214,7 +214,7 @@ struct stv_objsecrets {
  */
 
 struct object *
-STV_MkObject(struct sess *sp, void *ptr, unsigned ltot,
+STV_MkObject(struct worker *wrk, void *ptr, unsigned ltot,
     const struct stv_objsecrets *soc)
 {
 	struct object *o;
@@ -242,15 +242,15 @@ STV_MkObject(struct sess *sp, void *ptr, unsigned ltot,
 
 	http_Setup(o->http, o->ws_o);
 	o->http->magic = HTTP_MAGIC;
-	o->exp = *soc->exp;
+	o->exp = wrk->exp;
 	VTAILQ_INIT(&o->store);
-	sp->wrk->stats.n_object++;
+	wrk->stats.n_object++;
 
-	if (sp->wrk->objcore != NULL) {
-		CHECK_OBJ_NOTNULL(sp->wrk->objcore, OBJCORE_MAGIC);
+	if (wrk->objcore != NULL) {
+		CHECK_OBJ_NOTNULL(wrk->objcore, OBJCORE_MAGIC);
 
-		o->objcore = sp->wrk->objcore;
-		sp->wrk->objcore = NULL;     /* refcnt follows pointer. */
+		o->objcore = wrk->objcore;
+		wrk->objcore = NULL;     /* refcnt follows pointer. */
 		BAN_NewObjCore(o->objcore);
 
 		o->objcore->methods = &default_oc_methods;
@@ -265,7 +265,7 @@ STV_MkObject(struct sess *sp, void *ptr, unsigned ltot,
  */
 
 struct object *
-stv_default_allocobj(struct stevedore *stv, struct sess *sp, unsigned ltot,
+stv_default_allocobj(struct stevedore *stv, struct worker *wrk, unsigned ltot,
     const struct stv_objsecrets *soc)
 {
 	struct object *o;
@@ -280,7 +280,7 @@ stv_default_allocobj(struct stevedore *stv, struct sess *sp, unsigned ltot,
 		return (NULL);
 	}
 	ltot = st->len = st->space;
-	o = STV_MkObject(sp, st->ptr, ltot, soc);
+	o = STV_MkObject(wrk, st->ptr, ltot, soc);
 	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
 	o->objstore = st;
 	return (o);
@@ -293,8 +293,8 @@ stv_default_allocobj(struct stevedore *stv, struct sess *sp, unsigned ltot,
  */
 
 struct object *
-STV_NewObject(struct sess *sp, const char *hint, unsigned wsl, struct exp *ep,
-    uint16_t nhttp)
+STV_NewObject(struct worker *wrk, const char *hint, unsigned wsl,
+     uint16_t nhttp)
 {
 	struct object *o;
 	struct stevedore *stv, *stv0;
@@ -302,6 +302,7 @@ STV_NewObject(struct sess *sp, const char *hint, unsigned wsl, struct exp *ep,
 	struct stv_objsecrets soc;
 	int i;
 
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	assert(wsl > 0);
 	wsl = PRNDUP(wsl);
 
@@ -313,26 +314,25 @@ STV_NewObject(struct sess *sp, const char *hint, unsigned wsl, struct exp *ep,
 	soc.nhttp = nhttp;
 	soc.lhttp = lhttp;
 	soc.wsl = wsl;
-	soc.exp = ep;
 
 	ltot = sizeof *o + wsl + lhttp;
 
-	stv = stv0 = stv_pick_stevedore(sp, &hint);
+	stv = stv0 = stv_pick_stevedore(wrk, &hint);
 	AN(stv->allocobj);
-	o = stv->allocobj(stv, sp, ltot, &soc);
+	o = stv->allocobj(stv, wrk, ltot, &soc);
 	if (o == NULL && hint == NULL) {
 		do {
-			stv = stv_pick_stevedore(sp, &hint);
+			stv = stv_pick_stevedore(wrk, &hint);
 			AN(stv->allocobj);
-			o = stv->allocobj(stv, sp, ltot, &soc);
+			o = stv->allocobj(stv, wrk, ltot, &soc);
 		} while (o == NULL && stv != stv0);
 	}
 	if (o == NULL) {
 		/* no luck; try to free some space and keep trying */
 		for (i = 0; o == NULL && i < cache_param->nuke_limit; i++) {
-			if (EXP_NukeOne(sp->wrk, stv->lru) == -1)
+			if (EXP_NukeOne(wrk, stv->lru) == -1)
 				break;
-			o = stv->allocobj(stv, sp, ltot, &soc);
+			o = stv->allocobj(stv, wrk, ltot, &soc);
 		}
 	}
 
diff --git a/bin/varnishd/storage/storage.h b/bin/varnishd/storage/storage.h
index a813a36..c8c3689 100644
--- a/bin/varnishd/storage/storage.h
+++ b/bin/varnishd/storage/storage.h
@@ -34,6 +34,7 @@
 struct stv_objsecrets;
 struct stevedore;
 struct sess;
+struct worker;
 struct lru;
 
 typedef void storage_init_f(struct stevedore *, int ac, char * const *av);
@@ -41,8 +42,8 @@ typedef void storage_open_f(const struct stevedore *);
 typedef struct storage *storage_alloc_f(struct stevedore *, size_t size);
 typedef void storage_trim_f(struct storage *, size_t size);
 typedef void storage_free_f(struct storage *);
-typedef struct object *storage_allocobj_f(struct stevedore *, struct sess *sp,
-    unsigned ltot, const struct stv_objsecrets *);
+typedef struct object *storage_allocobj_f(struct stevedore *,
+    struct worker *wrk, unsigned ltot, const struct stv_objsecrets *);
 typedef void storage_close_f(const struct stevedore *);
 
 /* Prototypes for VCL variable responders */
@@ -89,7 +90,7 @@ extern struct stevedore *stv_transient;
 int STV_GetFile(const char *fn, int *fdp, const char **fnp, const char *ctx);
 uintmax_t STV_FileSize(int fd, const char *size, unsigned *granularity,
     const char *ctx);
-struct object *STV_MkObject(struct sess *sp, void *ptr, unsigned ltot,
+struct object *STV_MkObject(struct worker *wrk, void *ptr, unsigned ltot,
     const struct stv_objsecrets *soc);
 
 struct lru *LRU_Alloc(void);
diff --git a/bin/varnishd/storage/storage_persistent.c b/bin/varnishd/storage/storage_persistent.c
index 838dc8c..15087b3 100644
--- a/bin/varnishd/storage/storage_persistent.c
+++ b/bin/varnishd/storage/storage_persistent.c
@@ -463,7 +463,7 @@ smp_allocx(struct stevedore *st, size_t min_size, size_t max_size,
  */
 
 static struct object *
-smp_allocobj(struct stevedore *stv, struct sess *sp, unsigned ltot,
+smp_allocobj(struct stevedore *stv, struct worker *wrk, unsigned ltot,
     const struct stv_objsecrets *soc)
 {
 	struct object *o;
@@ -474,11 +474,11 @@ smp_allocobj(struct stevedore *stv, struct sess *sp, unsigned ltot,
 	struct objcore *oc;
 	unsigned objidx;
 
-	if (sp->wrk->objcore == NULL)
+	if (wrk->objcore == NULL)
 		return (NULL);		/* from cnt_error */
 	CAST_OBJ_NOTNULL(sc, stv->priv, SMP_SC_MAGIC);
-	AN(sp->wrk->objcore);
-	AN(sp->wrk->exp.ttl > 0.);
+	AN(wrk->objcore);
+	AN(wrk->exp.ttl > 0.);
 
 	ltot = IRNUP(sc, ltot);
 
@@ -489,7 +489,7 @@ smp_allocobj(struct stevedore *stv, struct sess *sp, unsigned ltot,
 	assert(st->space >= ltot);
 	ltot = st->len = st->space;
 
-	o = STV_MkObject(sp, st->ptr, ltot, soc);
+	o = STV_MkObject(wrk, st->ptr, ltot, soc);
 	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
 	o->objstore = st;
 



From phk at varnish-cache.org  Tue Nov 29 12:49:33 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 29 Nov 2011 13:49:33 +0100
Subject: [master] 5b77d34 Push sess out of HSH_Drop() and HSH_Unbusy()
Message-ID: <E1RVN7t-0002CQ-A3@project.varnish-software.com>

commit 5b77d3496a174b30f1dadb21a8ce14293422f276
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 29 12:49:08 2011 +0000

    Push sess out of HSH_Drop() and HSH_Unbusy()

diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c
index 13db2e3..3832116 100644
--- a/bin/varnishd/cache/cache_center.c
+++ b/bin/varnishd/cache/cache_center.c
@@ -238,7 +238,7 @@ cnt_prepresp(struct sess *sp)
 			break;
 		if (sp->wrk->do_stream) {
 			VDI_CloseFd(sp->wrk);
-			HSH_Drop(sp);
+			HSH_Drop(sp->wrk);
 		} else {
 			(void)HSH_Deref(sp->wrk, NULL, &sp->wrk->obj);
 		}
@@ -495,7 +495,7 @@ cnt_error(struct sess *sp)
 
 	if (sp->handling == VCL_RET_RESTART &&
 	    sp->restarts <  cache_param->max_restarts) {
-		HSH_Drop(sp);
+		HSH_Drop(sp->wrk);
 		sp->director = NULL;
 		sp->restarts++;
 		sp->step = STP_RECV;
@@ -873,7 +873,7 @@ cnt_fetchbody(struct sess *sp)
 	AN(sp->director);
 
 	if (i) {
-		HSH_Drop(sp);
+		HSH_Drop(wrk);
 		AZ(wrk->obj);
 		sp->err_code = 503;
 		sp->step = STP_ERROR;
@@ -884,7 +884,7 @@ cnt_fetchbody(struct sess *sp)
 		EXP_Insert(wrk->obj);
 		AN(wrk->obj->objcore);
 		AN(wrk->obj->objcore->ban);
-		HSH_Unbusy(sp);
+		HSH_Unbusy(wrk);
 	}
 	wrk->acct_tmp.fetch++;
 	sp->step = STP_PREPRESP;
@@ -939,7 +939,7 @@ cnt_streambody(struct sess *sp)
 		EXP_Insert(sp->wrk->obj);
 		AN(sp->wrk->obj->objcore);
 		AN(sp->wrk->obj->objcore->ban);
-		HSH_Unbusy(sp);
+		HSH_Unbusy(sp->wrk);
 	} else {
 		sp->doclose = "Stream error";
 	}
diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index 2820c45..a2fc985 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -580,29 +580,29 @@ HSH_Purge(const struct sess *sp, struct objhead *oh, double ttl, double grace)
  */
 
 void
-HSH_Drop(struct sess *sp)
+HSH_Drop(struct worker *wrk)
 {
 	struct object *o;
 
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	o = sp->wrk->obj;
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
+	o = wrk->obj;
 	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
 	AssertObjCorePassOrBusy(o->objcore);
 	o->exp.ttl = -1.;
 	if (o->objcore != NULL)		/* Pass has no objcore */
-		HSH_Unbusy(sp);
-	(void)HSH_Deref(sp->wrk, NULL, &sp->wrk->obj);
+		HSH_Unbusy(wrk);
+	(void)HSH_Deref(wrk, NULL, &wrk->obj);
 }
 
 void
-HSH_Unbusy(const struct sess *sp)
+HSH_Unbusy(struct worker *wrk)
 {
 	struct object *o;
 	struct objhead *oh;
 	struct objcore *oc;
 
-	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	o = sp->wrk->obj;
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
+	o = wrk->obj;
 	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
 	oc = o->objcore;
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
@@ -614,9 +614,9 @@ HSH_Unbusy(const struct sess *sp)
 	assert(oc->refcnt > 0);
 	assert(oh->refcnt > 0);
 	if (o->ws_o->overflow)
-		sp->wrk->stats.n_objoverflow++;
+		wrk->stats.n_objoverflow++;
 	if (cache_param->diag_bitmap & 0x40)
-		WSP(sp, SLT_Debug,
+		WSL(wrk, SLT_Debug, 0,
 		    "Object %u workspace free %u", o->xid, WS_Free(o->ws_o));
 
 	/* XXX: pretouch neighbors on oh->objcs to prevent page-on under mtx */
@@ -626,14 +626,14 @@ HSH_Unbusy(const struct sess *sp)
 	VTAILQ_REMOVE(&oh->objcs, oc, list);
 	VTAILQ_INSERT_HEAD(&oh->objcs, oc, list);
 	oc->flags &= ~OC_F_BUSY;
-	AZ(sp->wrk->nbusyobj);
-	sp->wrk->nbusyobj = oc->busyobj;
+	AZ(wrk->nbusyobj);
+	wrk->nbusyobj = oc->busyobj;
 	oc->busyobj = NULL;
 	if (oh->waitinglist != NULL)
 		hsh_rush(oh);
 	AN(oc->ban);
 	Lck_Unlock(&oh->mtx);
-	assert(oc_getobj(sp->wrk, oc) == o);
+	assert(oc_getobj(wrk, oc) == o);
 }
 
 void
diff --git a/bin/varnishd/hash/hash_slinger.h b/bin/varnishd/hash/hash_slinger.h
index 9e3b13f..b45e604 100644
--- a/bin/varnishd/hash/hash_slinger.h
+++ b/bin/varnishd/hash/hash_slinger.h
@@ -54,9 +54,9 @@ struct hash_slinger {
 void HSH_Prealloc(const struct sess *sp);
 void HSH_Cleanup(struct worker *w);
 struct objcore *HSH_Lookup(struct sess *sp, struct objhead **poh);
-void HSH_Unbusy(const struct sess *sp);
+void HSH_Unbusy(struct worker *wrk);
 void HSH_Ref(struct objcore *o);
-void HSH_Drop(struct sess *sp);
+void HSH_Drop(struct worker *wrk);
 void HSH_Init(const struct hash_slinger *slinger);
 void HSH_AddString(const struct sess *sp, const char *str);
 struct objcore *HSH_Insert(const struct sess *sp);



From phk at varnish-cache.org  Tue Nov 29 15:56:56 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 29 Nov 2011 16:56:56 +0100
Subject: [master] ef234ec Start descoping the fetch-from-backend stuff into
	struct busyobj.
Message-ID: <E1RVQ3E-0002gy-Pg@project.varnish-software.com>

commit ef234ec9c32a7c0d209df5501858c100d37235b2
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 29 15:55:42 2011 +0000

    Start descoping the fetch-from-backend stuff into struct busyobj.
    
    The fetching worker has wrk->busyobj set
    
    Push is_g[un]zip flags into busyobj as proof of concept.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 8371278..7ad7fbd 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -323,6 +323,7 @@ struct worker {
 
 	struct object		*obj;
 	struct objcore		*objcore;
+	struct busyobj		*busyobj;
 
 	struct exp		exp;
 
@@ -340,9 +341,7 @@ struct worker {
 	unsigned		do_stream;
 	unsigned		do_esi;
 	unsigned		do_gzip;
-	unsigned		is_gzip;
 	unsigned		do_gunzip;
-	unsigned		is_gunzip;
 	unsigned		do_close;
 	char			*h_content_length;
 
@@ -492,6 +491,8 @@ struct busyobj {
 	unsigned		magic;
 #define BUSYOBJ_MAGIC		0x23b95567
 	uint8_t			*vary;
+	unsigned		is_gzip;
+	unsigned		is_gunzip;
 };
 
 /* Object structure --------------------------------------------------*/
@@ -967,6 +968,21 @@ void SMP_Init(void);
 void SMP_Ready(void);
 void SMP_NewBan(const uint8_t *ban, unsigned len);
 
+#define New_BusyObj(wrk)						\
+	do {								\
+		if (wrk->nbusyobj != NULL) {				\
+			CHECK_OBJ_NOTNULL(wrk->nbusyobj, BUSYOBJ_MAGIC);\
+			wrk->busyobj = wrk->nbusyobj;			\
+			wrk->nbusyobj = NULL;				\
+			memset(wrk->busyobj, 0, sizeof *wrk->busyobj);	\
+			wrk->busyobj->magic = BUSYOBJ_MAGIC;		\
+		} else {						\
+			ALLOC_OBJ(wrk->busyobj, BUSYOBJ_MAGIC);		\
+		}							\
+		CHECK_OBJ_NOTNULL(wrk->busyobj, BUSYOBJ_MAGIC);		\
+		AZ(wrk->nbusyobj);					\
+	} while (0)
+
 /*
  * A normal pointer difference is signed, but we never want a negative value
  * so this little tool will make sure we don't get that.
diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c
index 3832116..8f9e527 100644
--- a/bin/varnishd/cache/cache_center.c
+++ b/bin/varnishd/cache/cache_center.c
@@ -323,12 +323,12 @@ cnt_done(struct sess *sp)
 	sp->director = NULL;
 	sp->restarts = 0;
 
+	sp->wrk->busyobj = NULL;
+
 	sp->wrk->do_esi = 0;
 	sp->wrk->do_gunzip = 0;
 	sp->wrk->do_gzip = 0;
 	sp->wrk->do_stream = 0;
-	sp->wrk->is_gunzip = 0;
-	sp->wrk->is_gzip = 0;
 
 	SES_Charge(sp);
 
@@ -444,8 +444,6 @@ cnt_error(struct sess *sp)
 
 	w = sp->wrk;
 	w->do_esi = 0;
-	w->is_gzip = 0;
-	w->is_gunzip = 0;
 	w->do_gzip = 0;
 	w->do_gunzip = 0;
 	w->do_stream = 0;
@@ -552,6 +550,8 @@ cnt_fetch(struct sess *sp)
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
 	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->wrk, WORKER_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->wrk->busyobj, BUSYOBJ_MAGIC);
 
 	AN(sp->director);
 	AZ(sp->wrk->vbc);
@@ -688,6 +688,8 @@ cnt_fetchbody(struct sess *sp)
 
 	wrk = sp->wrk;
 
+	CHECK_OBJ_NOTNULL(wrk->busyobj, BUSYOBJ_MAGIC);
+
 	assert(sp->handling == VCL_RET_HIT_FOR_PASS ||
 	    sp->handling == VCL_RET_DELIVER);
 
@@ -724,15 +726,17 @@ cnt_fetchbody(struct sess *sp)
 	if (!cache_param->http_gzip_support)
 		wrk->do_gzip = wrk->do_gunzip = 0;
 
-	wrk->is_gzip = http_HdrIs(wrk->beresp, H_Content_Encoding, "gzip");
+	wrk->busyobj->is_gzip =
+	    http_HdrIs(wrk->beresp, H_Content_Encoding, "gzip");
 
-	wrk->is_gunzip = !http_GetHdr(wrk->beresp, H_Content_Encoding, NULL);
+	wrk->busyobj->is_gunzip =
+	    !http_GetHdr(wrk->beresp, H_Content_Encoding, NULL);
 
 	/* It can't be both */
-	assert(wrk->is_gzip == 0 || wrk->is_gunzip == 0);
+	assert(wrk->busyobj->is_gzip == 0 || wrk->busyobj->is_gunzip == 0);
 
 	/* We won't gunzip unless it is gzip'ed */
-	if (wrk->do_gunzip && !wrk->is_gzip)
+	if (wrk->do_gunzip && !wrk->busyobj->is_gzip)
 		wrk->do_gunzip = 0;
 
 	/* If we do gunzip, remove the C-E header */
@@ -740,7 +744,7 @@ cnt_fetchbody(struct sess *sp)
 		http_Unset(wrk->beresp, H_Content_Encoding);
 
 	/* We wont gzip unless it is ungziped */
-	if (wrk->do_gzip && !wrk->is_gunzip)
+	if (wrk->do_gzip && !wrk->busyobj->is_gunzip)
 		wrk->do_gzip = 0;
 
 	/* If we do gzip, add the C-E header */
@@ -758,7 +762,7 @@ cnt_fetchbody(struct sess *sp)
 		wrk->vfp = &vfp_gunzip;
 	else if (wrk->do_gzip)
 		wrk->vfp = &vfp_gzip;
-	else if (wrk->is_gzip)
+	else if (wrk->busyobj->is_gzip)
 		wrk->vfp = &vfp_testgzip;
 
 	if (wrk->do_esi || sp->esi_level > 0)
@@ -811,7 +815,7 @@ cnt_fetchbody(struct sess *sp)
 
 	wrk->storage_hint = NULL;
 
-	if (wrk->do_gzip || (wrk->is_gzip && !wrk->do_gunzip))
+	if (wrk->do_gzip || (wrk->busyobj->is_gzip && !wrk->do_gunzip))
 		wrk->obj->gziped = 1;
 
 	if (vary != NULL) {
@@ -910,6 +914,7 @@ cnt_streambody(struct sess *sp)
 	uint8_t obuf[sp->wrk->res_mode & RES_GUNZIP ?
 	    cache_param->gzip_stack_buffer : 1];
 
+	CHECK_OBJ_NOTNULL(sp->wrk->busyobj, BUSYOBJ_MAGIC);
 	memset(&sctx, 0, sizeof sctx);
 	sctx.magic = STREAM_CTX_MAGIC;
 	AZ(sp->wrk->sctx);
@@ -1020,29 +1025,34 @@ DOT hit -> prepresp [label="deliver",style=bold,color=green]
 static int
 cnt_hit(struct sess *sp)
 {
+	struct worker *wrk;
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->wrk->obj, OBJECT_MAGIC);
+	wrk = sp->wrk;
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
+
+	CHECK_OBJ_NOTNULL(wrk->obj, OBJECT_MAGIC);
 	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
 
-	assert(!(sp->wrk->obj->objcore->flags & OC_F_PASS));
+	assert(!(wrk->obj->objcore->flags & OC_F_PASS));
 
-	AZ(sp->wrk->do_stream);
+	AZ(wrk->do_stream);
 
 	VCL_hit_method(sp);
 
 	if (sp->handling == VCL_RET_DELIVER) {
 		/* Dispose of any body part of the request */
 		(void)FetchReqBody(sp);
-		AZ(sp->wrk->bereq->ws);
-		AZ(sp->wrk->beresp->ws);
+		AZ(wrk->bereq->ws);
+		AZ(wrk->beresp->ws);
 		sp->step = STP_PREPRESP;
 		return (0);
 	}
 
 	/* Drop our object, we won't need it */
-	(void)HSH_Deref(sp->wrk, NULL, &sp->wrk->obj);
-	sp->wrk->objcore = NULL;
+	(void)HSH_Deref(wrk, NULL, &wrk->obj);
+	wrk->objcore = NULL;
+	wrk->busyobj = NULL;
 
 	switch(sp->handling) {
 	case VCL_RET_PASS:
@@ -1145,6 +1155,7 @@ cnt_lookup(struct sess *sp)
 		sp->vary_e = NULL;
 
 		sp->wrk->objcore = oc;
+		CHECK_OBJ_NOTNULL(sp->wrk->busyobj, BUSYOBJ_MAGIC);
 		sp->step = STP_MISS;
 		return (0);
 	}
@@ -1205,6 +1216,7 @@ cnt_miss(struct sess *sp)
 
 	AZ(sp->wrk->obj);
 	AN(sp->wrk->objcore);
+	CHECK_OBJ_NOTNULL(sp->wrk->busyobj, BUSYOBJ_MAGIC);
 	WS_Reset(sp->wrk->ws, NULL);
 	http_Setup(sp->wrk->bereq, sp->wrk->ws);
 	http_FilterHeader(sp, HTTPH_R_FETCH);
@@ -1222,7 +1234,10 @@ cnt_miss(struct sess *sp)
 	sp->wrk->connect_timeout = 0;
 	sp->wrk->first_byte_timeout = 0;
 	sp->wrk->between_bytes_timeout = 0;
+	CHECK_OBJ_NOTNULL(sp->wrk->busyobj, BUSYOBJ_MAGIC);
+
 	VCL_miss_method(sp);
+	CHECK_OBJ_NOTNULL(sp->wrk->busyobj, BUSYOBJ_MAGIC);
 	switch(sp->handling) {
 	case VCL_RET_ERROR:
 		AZ(HSH_Deref(sp->wrk, sp->wrk->objcore, NULL));
@@ -1236,6 +1251,7 @@ cnt_miss(struct sess *sp)
 		sp->step = STP_PASS;
 		return (0);
 	case VCL_RET_FETCH:
+		CHECK_OBJ_NOTNULL(sp->wrk->busyobj, BUSYOBJ_MAGIC);
 		sp->step = STP_FETCH;
 		return (0);
 	case VCL_RET_RESTART:
@@ -1304,6 +1320,7 @@ cnt_pass(struct sess *sp)
 	sp->wrk->acct_tmp.pass++;
 	sp->sendbody = 1;
 	sp->step = STP_FETCH;
+	New_BusyObj(sp->wrk);
 	return (0);
 }
 
@@ -1411,8 +1428,6 @@ cnt_recv(struct sess *sp)
 
 	/* Zap these, in case we came here through restart */
 	sp->wrk->do_esi = 0;
-	sp->wrk->is_gzip = 0;
-	sp->wrk->is_gunzip = 0;
 	sp->wrk->do_gzip = 0;
 	sp->wrk->do_gunzip = 0;
 	sp->wrk->do_stream = 0;
@@ -1595,9 +1610,7 @@ CNT_Session(struct sess *sp)
 	    sp->step == STP_RECV);
 
 	AZ(w->do_stream);
-	AZ(w->is_gzip);
 	AZ(w->do_gzip);
-	AZ(w->is_gunzip);
 	AZ(w->do_gunzip);
 	AZ(w->do_esi);
 	AZ(w->obj);
@@ -1654,9 +1667,7 @@ CNT_Session(struct sess *sp)
 	AZ(w->obj);
 	AZ(w->objcore);
 	AZ(w->do_stream);
-	AZ(w->is_gzip);
 	AZ(w->do_gzip);
-	AZ(w->is_gunzip);
 	AZ(w->do_gunzip);
 	AZ(w->do_esi);
 #define ACCT(foo)	AZ(w->acct_tmp.foo);
diff --git a/bin/varnishd/cache/cache_esi_deliver.c b/bin/varnishd/cache/cache_esi_deliver.c
index 5ac8e71..6c8a16f 100644
--- a/bin/varnishd/cache/cache_esi_deliver.c
+++ b/bin/varnishd/cache/cache_esi_deliver.c
@@ -92,8 +92,6 @@ ved_include(struct sess *sp, const char *src, const char *host)
 	http_Unset(sp->http, H_Content_Length);
 
 	sp->wrk->do_esi = 0;
-	sp->wrk->is_gzip = 0;
-	sp->wrk->is_gunzip = 0;
 	sp->wrk->do_gzip = 0;
 	sp->wrk->do_gunzip = 0;
 	sp->wrk->do_stream = 0;
diff --git a/bin/varnishd/cache/cache_esi_fetch.c b/bin/varnishd/cache/cache_esi_fetch.c
index 5ec8f6b..35166e7 100644
--- a/bin/varnishd/cache/cache_esi_fetch.c
+++ b/bin/varnishd/cache/cache_esi_fetch.c
@@ -298,19 +298,20 @@ vfp_esi_begin(struct worker *w, size_t estimate)
 	struct vef_priv *vef;
 
 	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	CHECK_OBJ_NOTNULL(w->busyobj, BUSYOBJ_MAGIC);
 
 	AZ(w->vgz_rx);
-	if (w->is_gzip && w->do_gunzip) {
+	if (w->busyobj->is_gzip && w->do_gunzip) {
 		w->vgz_rx = VGZ_NewUngzip(w, "U F E");
 		VEP_Init(w, NULL);
-	} else if (w->is_gunzip && w->do_gzip) {
+	} else if (w->busyobj->is_gunzip && w->do_gzip) {
 		ALLOC_OBJ(vef, VEF_MAGIC);
 		AN(vef);
 		vef->vgz = VGZ_NewGzip(w, "G F E");
 		AZ(w->vef_priv);
 		w->vef_priv = vef;
 		VEP_Init(w, vfp_vep_callback);
-	} else if (w->is_gzip) {
+	} else if (w->busyobj->is_gzip) {
 		w->vgz_rx = VGZ_NewUngzip(w, "U F E");
 		ALLOC_OBJ(vef, VEF_MAGIC);
 		AN(vef);
@@ -336,11 +337,11 @@ vfp_esi_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
 	AZ(w->fetch_failed);
 	AN(w->vep);
 	assert(w->htc == htc);
-	if (w->is_gzip && w->do_gunzip)
+	if (w->busyobj->is_gzip && w->do_gunzip)
 		i = vfp_esi_bytes_gu(w, htc, bytes);
-	else if (w->is_gunzip && w->do_gzip)
+	else if (w->busyobj->is_gunzip && w->do_gzip)
 		i = vfp_esi_bytes_ug(w, htc, bytes);
-	else if (w->is_gzip)
+	else if (w->busyobj->is_gzip)
 		i = vfp_esi_bytes_gg(w, htc, bytes);
 	else
 		i = vfp_esi_bytes_uu(w, htc, bytes);
diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index a2fc985..1f1795c 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -456,14 +456,14 @@ HSH_Lookup(struct sess *sp, struct objhead **poh)
 	AN(oc->flags & OC_F_BUSY);
 	oc->refcnt = 1;
 
-	/* XXX: clear w->nbusyobj before use */
+	New_BusyObj(w);
+
 	VRY_Validate(sp->vary_b);
 	if (sp->vary_l != NULL)
-		w->nbusyobj->vary = sp->vary_b;
+		w->busyobj->vary = sp->vary_b;
 	else
-		w->nbusyobj->vary = NULL;
-	oc->busyobj = w->nbusyobj;
-	w->nbusyobj = NULL;
+		w->busyobj->vary = NULL;
+	oc->busyobj = w->busyobj;
 
 	/*
 	 * Busy objects go on the tail, so they will not trip up searches.
diff --git a/bin/varnishd/cache/cache_panic.c b/bin/varnishd/cache/cache_panic.c
index f6f1ff5..641fe99 100644
--- a/bin/varnishd/cache/cache_panic.c
+++ b/bin/varnishd/cache/cache_panic.c
@@ -206,6 +206,16 @@ pan_wrk(const struct worker *wrk)
 	VSB_printf(pan_vsp, "    },\n");
 }
 
+static void
+pan_busyobj(const struct busyobj *bo)
+{
+
+	VSB_printf(pan_vsp, "  busyobj = %p {\n", bo);
+	if (bo->is_gzip)	VSB_printf(pan_vsp, "    is_gzip\n");
+	if (bo->is_gunzip)	VSB_printf(pan_vsp, "    is_gunzip\n");
+	VSB_printf(pan_vsp, "    },\n");
+}
+
 /*--------------------------------------------------------------------*/
 
 static void
@@ -249,10 +259,9 @@ pan_sess(const struct sess *sp)
 	if (sp->wrk->do_gunzip)	VSB_printf(pan_vsp, " do_gunzip");
 	if (sp->wrk->do_esi)	VSB_printf(pan_vsp, " do_esi");
 	if (sp->wrk->do_close)	VSB_printf(pan_vsp, " do_close");
-	if (sp->wrk->is_gzip)	VSB_printf(pan_vsp, " is_gzip");
-	if (sp->wrk->is_gunzip)	VSB_printf(pan_vsp, " is_gunzip");
 	VSB_printf(pan_vsp, "\n");
 	VSB_printf(pan_vsp, "  bodystatus = %d\n", sp->wrk->body_status);
+	pan_busyobj(sp->wrk->busyobj);
 
 	pan_ws(sp->ws, 2);
 	pan_http("req", sp->http, 2);



From phk at varnish-cache.org  Tue Nov 29 17:01:19 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 29 Nov 2011 18:01:19 +0100
Subject: [master] 58dc8e5 Introduce a consistent wrk local variable with in
	all cnt_*() steps.
Message-ID: <E1RVR3X-0007kH-0N@project.varnish-software.com>

commit 58dc8e5326adeab4af6b1172dbf8bd2dabb327ef
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 29 17:00:41 2011 +0000

    Introduce a consistent wrk local variable with in all cnt_*()
    steps.
    
    A couple of minor polishishes.

diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c
index 8f9e527..85224b6 100644
--- a/bin/varnishd/cache/cache_center.c
+++ b/bin/varnishd/cache/cache_center.c
@@ -98,10 +98,13 @@ cnt_wait(struct sess *sp)
 {
 	int i;
 	struct pollfd pfd[1];
+	struct worker *wrk;
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	wrk = sp->wrk;
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	AZ(sp->vcl);
-	AZ(sp->wrk->obj);
+	AZ(wrk->obj);
 	AZ(sp->esi_level);
 	assert(sp->xid == 0);
 
@@ -116,9 +119,8 @@ cnt_wait(struct sess *sp)
 	}
 	if (i == 0) {
 		WSP(sp, SLT_Debug, "herding");
-		sp->wrk->stats.sess_herd++;
+		wrk->stats.sess_herd++;
 		SES_Charge(sp);
-		sp->wrk = NULL;
 		Pool_Wait(sp);
 		return (1);
 	}
@@ -168,66 +170,70 @@ DOT }
 static int
 cnt_prepresp(struct sess *sp)
 {
+	struct worker *wrk;
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->wrk->obj, OBJECT_MAGIC);
+	wrk = sp->wrk;
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
+
+	CHECK_OBJ_NOTNULL(wrk->obj, OBJECT_MAGIC);
 	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
 
-	if (sp->wrk->do_stream)
-		AssertObjCorePassOrBusy(sp->wrk->obj->objcore);
+	if (wrk->do_stream)
+		AssertObjCorePassOrBusy(wrk->obj->objcore);
 
-	sp->wrk->res_mode = 0;
+	wrk->res_mode = 0;
 
-	if ((sp->wrk->h_content_length != NULL || !sp->wrk->do_stream) &&
-	    !sp->wrk->do_gzip && !sp->wrk->do_gunzip)
-		sp->wrk->res_mode |= RES_LEN;
+	if ((wrk->h_content_length != NULL || !wrk->do_stream) &&
+	    !wrk->do_gzip && !wrk->do_gunzip)
+		wrk->res_mode |= RES_LEN;
 
-	if (!sp->disable_esi && sp->wrk->obj->esidata != NULL) {
+	if (!sp->disable_esi && wrk->obj->esidata != NULL) {
 		/* In ESI mode, we don't know the aggregate length */
-		sp->wrk->res_mode &= ~RES_LEN;
-		sp->wrk->res_mode |= RES_ESI;
+		wrk->res_mode &= ~RES_LEN;
+		wrk->res_mode |= RES_ESI;
 	}
 
 	if (sp->esi_level > 0) {
-		sp->wrk->res_mode &= ~RES_LEN;
-		sp->wrk->res_mode |= RES_ESI_CHILD;
+		wrk->res_mode &= ~RES_LEN;
+		wrk->res_mode |= RES_ESI_CHILD;
 	}
 
-	if (cache_param->http_gzip_support && sp->wrk->obj->gziped &&
+	if (cache_param->http_gzip_support && wrk->obj->gziped &&
 	    !RFC2616_Req_Gzip(sp)) {
 		/*
 		 * We don't know what it uncompresses to
 		 * XXX: we could cache that
 		 */
-		sp->wrk->res_mode &= ~RES_LEN;
-		sp->wrk->res_mode |= RES_GUNZIP;
+		wrk->res_mode &= ~RES_LEN;
+		wrk->res_mode |= RES_GUNZIP;
 	}
 
-	if (!(sp->wrk->res_mode & (RES_LEN|RES_CHUNKED|RES_EOF))) {
-		if (sp->wrk->obj->len == 0 && !sp->wrk->do_stream)
+	if (!(wrk->res_mode & (RES_LEN|RES_CHUNKED|RES_EOF))) {
+		if (wrk->obj->len == 0 && !wrk->do_stream)
 			/*
 			 * If the object is empty, neither ESI nor GUNZIP
 			 * can make it any different size
 			 */
-			sp->wrk->res_mode |= RES_LEN;
+			wrk->res_mode |= RES_LEN;
 		else if (!sp->wantbody) {
 			/* Nothing */
 		} else if (sp->http->protover >= 11) {
-			sp->wrk->res_mode |= RES_CHUNKED;
+			wrk->res_mode |= RES_CHUNKED;
 		} else {
-			sp->wrk->res_mode |= RES_EOF;
+			wrk->res_mode |= RES_EOF;
 			sp->doclose = "EOF mode";
 		}
 	}
 
-	sp->t_resp = W_TIM_real(sp->wrk);
-	if (sp->wrk->obj->objcore != NULL) {
-		if ((sp->t_resp - sp->wrk->obj->last_lru) > cache_param->lru_timeout &&
-		    EXP_Touch(sp->wrk->obj->objcore))
-			sp->wrk->obj->last_lru = sp->t_resp;
-		sp->wrk->obj->last_use = sp->t_resp;	/* XXX: locking ? */
+	sp->t_resp = W_TIM_real(wrk);
+	if (wrk->obj->objcore != NULL) {
+		if ((sp->t_resp - wrk->obj->last_lru) > cache_param->lru_timeout &&
+		    EXP_Touch(wrk->obj->objcore))
+			wrk->obj->last_lru = sp->t_resp;
+		wrk->obj->last_use = sp->t_resp;	/* XXX: locking ? */
 	}
-	http_Setup(sp->wrk->resp, sp->wrk->ws);
+	http_Setup(wrk->resp, wrk->ws);
 	RES_BuildHttp(sp);
 	VCL_deliver_method(sp);
 	switch (sp->handling) {
@@ -236,26 +242,26 @@ cnt_prepresp(struct sess *sp)
 	case VCL_RET_RESTART:
 		if (sp->restarts >= cache_param->max_restarts)
 			break;
-		if (sp->wrk->do_stream) {
-			VDI_CloseFd(sp->wrk);
-			HSH_Drop(sp->wrk);
+		if (wrk->do_stream) {
+			VDI_CloseFd(wrk);
+			HSH_Drop(wrk);
 		} else {
-			(void)HSH_Deref(sp->wrk, NULL, &sp->wrk->obj);
+			(void)HSH_Deref(wrk, NULL, &wrk->obj);
 		}
-		AZ(sp->wrk->obj);
+		AZ(wrk->obj);
 		sp->restarts++;
 		sp->director = NULL;
-		sp->wrk->h_content_length = NULL;
-		http_Setup(sp->wrk->bereq, NULL);
-		http_Setup(sp->wrk->beresp, NULL);
-		http_Setup(sp->wrk->resp, NULL);
+		wrk->h_content_length = NULL;
+		http_Setup(wrk->bereq, NULL);
+		http_Setup(wrk->beresp, NULL);
+		http_Setup(wrk->resp, NULL);
 		sp->step = STP_RECV;
 		return (0);
 	default:
 		WRONG("Illegal action in vcl_deliver{}");
 	}
-	if (sp->wrk->do_stream) {
-		AssertObjCorePassOrBusy(sp->wrk->obj->objcore);
+	if (wrk->do_stream) {
+		AssertObjCorePassOrBusy(wrk->obj->objcore);
 		sp->step = STP_STREAMBODY;
 	} else {
 		sp->step = STP_DELIVER;
@@ -281,16 +287,21 @@ DOT deliver -> DONE [style=bold,color=blue]
 static int
 cnt_deliver(struct sess *sp)
 {
+	struct worker *wrk;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	wrk = sp->wrk;
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 
 	sp->director = NULL;
 	sp->restarts = 0;
 
 	RES_WriteObj(sp);
 
-	assert(WRW_IsReleased(sp->wrk));
-	assert(sp->wrk->wrw.ciov == sp->wrk->wrw.siov);
-	(void)HSH_Deref(sp->wrk, NULL, &sp->wrk->obj);
-	http_Setup(sp->wrk->resp, NULL);
+	assert(WRW_IsReleased(wrk));
+	assert(wrk->wrw.ciov == wrk->wrw.siov);
+	(void)HSH_Deref(wrk, NULL, &wrk->obj);
+	http_Setup(wrk->resp, NULL);
 	sp->step = STP_DONE;
 	return (0);
 }
@@ -314,21 +325,24 @@ cnt_done(struct sess *sp)
 {
 	double dh, dp, da;
 	int i;
+	struct worker *wrk;
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	wrk = sp->wrk;
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_ORNULL(sp->vcl, VCL_CONF_MAGIC);
 
-	AZ(sp->wrk->obj);
-	AZ(sp->wrk->vbc);
+	AZ(wrk->obj);
+	AZ(wrk->vbc);
 	sp->director = NULL;
 	sp->restarts = 0;
 
-	sp->wrk->busyobj = NULL;
+	wrk->busyobj = NULL;
 
-	sp->wrk->do_esi = 0;
-	sp->wrk->do_gunzip = 0;
-	sp->wrk->do_gzip = 0;
-	sp->wrk->do_stream = 0;
+	wrk->do_esi = 0;
+	wrk->do_gunzip = 0;
+	wrk->do_gzip = 0;
+	wrk->do_stream = 0;
 
 	SES_Charge(sp);
 
@@ -337,14 +351,14 @@ cnt_done(struct sess *sp)
 		return (1);
 
 	if (sp->vcl != NULL) {
-		if (sp->wrk->vcl != NULL)
-			VCL_Rel(&sp->wrk->vcl);
-		sp->wrk->vcl = sp->vcl;
+		if (wrk->vcl != NULL)
+			VCL_Rel(&wrk->vcl);
+		wrk->vcl = sp->vcl;
 		sp->vcl = NULL;
 	}
 
 
-	sp->t_end = W_TIM_real(sp->wrk);
+	sp->t_end = W_TIM_real(wrk);
 WSP(sp, SLT_Debug, "PHK req %.9f resp %.9f end %.9f open %.9f",
     sp->t_req, sp->t_resp, sp->t_end,  sp->t_open);
 	if (sp->xid == 0) {
@@ -364,7 +378,7 @@ WSP(sp, SLT_Debug, "PHK req %.9f resp %.9f end %.9f open %.9f",
 		    sp->xid, sp->t_req, sp->t_end, dh, dp, da);
 	}
 	sp->xid = 0;
-	WSL_Flush(sp->wrk, 0);
+	WSL_Flush(wrk, 0);
 
 	sp->t_open = sp->t_end;
 	sp->t_resp = NAN;
@@ -385,35 +399,34 @@ WSP(sp, SLT_Debug, "PHK req %.9f resp %.9f end %.9f open %.9f",
 	}
 
 	if (sp->fd < 0) {
-		sp->wrk->stats.sess_closed++;
+		wrk->stats.sess_closed++;
 		SES_Delete(sp, NULL);
 		return (1);
 	}
 
-	if (sp->wrk->stats.client_req >= cache_param->wthread_stats_rate)
-		WRK_SumStat(sp->wrk);
+	if (wrk->stats.client_req >= cache_param->wthread_stats_rate)
+		WRK_SumStat(wrk);
 	/* Reset the workspace to the session-watermark */
 	WS_Reset(sp->ws, sp->ws_ses);
-	WS_Reset(sp->wrk->ws, NULL);
+	WS_Reset(wrk->ws, NULL);
 
 	i = HTC_Reinit(sp->htc);
 	if (i == 1) {
-		sp->wrk->stats.sess_pipeline++;
+		wrk->stats.sess_pipeline++;
 		sp->step = STP_START;
 		return (0);
 	}
 	if (Tlen(sp->htc->rxbuf)) {
-		sp->wrk->stats.sess_readahead++;
+		wrk->stats.sess_readahead++;
 		sp->step = STP_WAIT;
 		return (0);
 	}
 	if (cache_param->session_linger > 0) {
-		sp->wrk->stats.sess_linger++;
+		wrk->stats.sess_linger++;
 		sp->step = STP_WAIT;
 		return (0);
 	}
-	sp->wrk->stats.sess_herd++;
-	sp->wrk = NULL;
+	wrk->stats.sess_herd++;
 	Pool_Wait(sp);
 	return (1);
 }
@@ -436,64 +449,65 @@ DOT rsterr [label="RESTART",shape=plaintext]
 static int
 cnt_error(struct sess *sp)
 {
-	struct worker *w;
 	struct http *h;
 	char date[40];
+	struct worker *wrk;
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	wrk = sp->wrk;
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 
-	w = sp->wrk;
-	w->do_esi = 0;
-	w->do_gzip = 0;
-	w->do_gunzip = 0;
-	w->do_stream = 0;
+	wrk->do_esi = 0;
+	wrk->do_gzip = 0;
+	wrk->do_gunzip = 0;
+	wrk->do_stream = 0;
 
-	if (w->obj == NULL) {
+	if (wrk->obj == NULL) {
 		HSH_Prealloc(sp);
-		EXP_Clr(&w->exp);
-		w->obj = STV_NewObject(w, NULL, cache_param->http_resp_size,
+		EXP_Clr(&wrk->exp);
+		wrk->obj = STV_NewObject(wrk, NULL, cache_param->http_resp_size,
 		     (uint16_t)cache_param->http_max_hdr);
-		if (sp->wrk->obj == NULL)
-			w->obj = STV_NewObject(w, TRANSIENT_STORAGE,
+		if (wrk->obj == NULL)
+			wrk->obj = STV_NewObject(wrk, TRANSIENT_STORAGE,
 			    cache_param->http_resp_size,
 			    (uint16_t)cache_param->http_max_hdr);
-		if (sp->wrk->obj == NULL) {
+		if (wrk->obj == NULL) {
 			sp->doclose = "Out of objects";
 			sp->director = NULL;
-			sp->wrk->h_content_length = NULL;
-			http_Setup(sp->wrk->beresp, NULL);
-			http_Setup(sp->wrk->bereq, NULL);
+			wrk->h_content_length = NULL;
+			http_Setup(wrk->beresp, NULL);
+			http_Setup(wrk->bereq, NULL);
 			sp->step = STP_DONE;
 			return(0);
 		}
-		AN(sp->wrk->obj);
-		sp->wrk->obj->xid = sp->xid;
-		sp->wrk->obj->exp.entered = sp->t_req;
+		AN(wrk->obj);
+		wrk->obj->xid = sp->xid;
+		wrk->obj->exp.entered = sp->t_req;
 	} else {
 		/* XXX: Null the headers ? */
 	}
-	CHECK_OBJ_NOTNULL(sp->wrk->obj, OBJECT_MAGIC);
-	h = sp->wrk->obj->http;
+	CHECK_OBJ_NOTNULL(wrk->obj, OBJECT_MAGIC);
+	h = wrk->obj->http;
 
 	if (sp->err_code < 100 || sp->err_code > 999)
 		sp->err_code = 501;
 
-	http_PutProtocol(w, sp->vsl_id, h, "HTTP/1.1");
+	http_PutProtocol(wrk, sp->vsl_id, h, "HTTP/1.1");
 	http_PutStatus(h, sp->err_code);
-	VTIM_format(W_TIM_real(sp->wrk), date);
-	http_PrintfHeader(w, sp->vsl_id, h, "Date: %s", date);
-	http_SetHeader(w, sp->vsl_id, h, "Server: Varnish");
+	VTIM_format(W_TIM_real(wrk), date);
+	http_PrintfHeader(wrk, sp->vsl_id, h, "Date: %s", date);
+	http_SetHeader(wrk, sp->vsl_id, h, "Server: Varnish");
 
 	if (sp->err_reason != NULL)
-		http_PutResponse(w, sp->vsl_id, h, sp->err_reason);
+		http_PutResponse(wrk, sp->vsl_id, h, sp->err_reason);
 	else
-		http_PutResponse(w, sp->vsl_id, h,
+		http_PutResponse(wrk, sp->vsl_id, h,
 		    http_StatusMessage(sp->err_code));
 	VCL_error_method(sp);
 
 	if (sp->handling == VCL_RET_RESTART &&
 	    sp->restarts <  cache_param->max_restarts) {
-		HSH_Drop(sp->wrk);
+		HSH_Drop(wrk);
 		sp->director = NULL;
 		sp->restarts++;
 		sp->step = STP_RECV;
@@ -509,7 +523,7 @@ cnt_error(struct sess *sp)
 	assert(sp->handling == VCL_RET_DELIVER);
 	sp->err_code = 0;
 	sp->err_reason = NULL;
-	http_Setup(sp->wrk->bereq, NULL);
+	http_Setup(wrk->bereq, NULL);
 	sp->step = STP_PREPRESP;
 	return (0);
 }
@@ -547,19 +561,22 @@ static int
 cnt_fetch(struct sess *sp)
 {
 	int i;
+	struct worker *wrk;
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	wrk = sp->wrk;
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
+
 	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->wrk, WORKER_MAGIC);
-	CHECK_OBJ_NOTNULL(sp->wrk->busyobj, BUSYOBJ_MAGIC);
+	CHECK_OBJ_NOTNULL(wrk->busyobj, BUSYOBJ_MAGIC);
 
 	AN(sp->director);
-	AZ(sp->wrk->vbc);
-	AZ(sp->wrk->h_content_length);
-	AZ(sp->wrk->do_close);
-	AZ(sp->wrk->storage_hint);
+	AZ(wrk->vbc);
+	AZ(wrk->h_content_length);
+	AZ(wrk->do_close);
+	AZ(wrk->storage_hint);
 
-	http_Setup(sp->wrk->beresp, sp->wrk->ws);
+	http_Setup(wrk->beresp, wrk->ws);
 
 	i = FetchHdr(sp);
 	/*
@@ -581,41 +598,41 @@ cnt_fetch(struct sess *sp)
 		 * and we rely on their content outside of VCL, so collect them
 		 * into one line here.
 		 */
-		http_CollectHdr(sp->wrk->beresp, H_Cache_Control);
-		http_CollectHdr(sp->wrk->beresp, H_Vary);
+		http_CollectHdr(wrk->beresp, H_Cache_Control);
+		http_CollectHdr(wrk->beresp, H_Vary);
 
 		/*
 		 * Figure out how the fetch is supposed to happen, before the
 		 * headers are adultered by VCL
-		 * NB: Also sets other sp->wrk variables
+		 * NB: Also sets other wrk variables
 		 */
-		sp->wrk->body_status = RFC2616_Body(sp);
+		wrk->body_status = RFC2616_Body(sp);
 
-		sp->err_code = http_GetStatus(sp->wrk->beresp);
+		sp->err_code = http_GetStatus(wrk->beresp);
 
 		/*
 		 * What does RFC2616 think about TTL ?
 		 */
-		EXP_Clr(&sp->wrk->exp);
-		sp->wrk->exp.entered = W_TIM_real(sp->wrk);
+		EXP_Clr(&wrk->exp);
+		wrk->exp.entered = W_TIM_real(wrk);
 		RFC2616_Ttl(sp);
 
 		/* pass from vclrecv{} has negative TTL */
-		if (sp->wrk->objcore == NULL)
-			sp->wrk->exp.ttl = -1.;
+		if (wrk->objcore == NULL)
+			wrk->exp.ttl = -1.;
 
-		AZ(sp->wrk->do_esi);
+		AZ(wrk->do_esi);
 
 		VCL_fetch_method(sp);
 
 		switch (sp->handling) {
 		case VCL_RET_HIT_FOR_PASS:
-			if (sp->wrk->objcore != NULL)
-				sp->wrk->objcore->flags |= OC_F_PASS;
+			if (wrk->objcore != NULL)
+				wrk->objcore->flags |= OC_F_PASS;
 			sp->step = STP_FETCHBODY;
 			return (0);
 		case VCL_RET_DELIVER:
-			AssertObjCorePassOrBusy(sp->wrk->objcore);
+			AssertObjCorePassOrBusy(wrk->objcore);
 			sp->step = STP_FETCHBODY;
 			return (0);
 		default:
@@ -623,22 +640,22 @@ cnt_fetch(struct sess *sp)
 		}
 
 		/* We are not going to fetch the body, Close the connection */
-		VDI_CloseFd(sp->wrk);
+		VDI_CloseFd(wrk);
 	}
 
 	/* Clean up partial fetch */
-	AZ(sp->wrk->vbc);
+	AZ(wrk->vbc);
 
-	if (sp->wrk->objcore != NULL) {
-		CHECK_OBJ_NOTNULL(sp->wrk->objcore, OBJCORE_MAGIC);
-		AZ(HSH_Deref(sp->wrk, sp->wrk->objcore, NULL));
-		sp->wrk->objcore = NULL;
+	if (wrk->objcore != NULL) {
+		CHECK_OBJ_NOTNULL(wrk->objcore, OBJCORE_MAGIC);
+		AZ(HSH_Deref(wrk, wrk->objcore, NULL));
+		wrk->objcore = NULL;
 	}
-	http_Setup(sp->wrk->bereq, NULL);
-	http_Setup(sp->wrk->beresp, NULL);
-	sp->wrk->h_content_length = NULL;
+	http_Setup(wrk->bereq, NULL);
+	http_Setup(wrk->beresp, NULL);
+	wrk->h_content_length = NULL;
 	sp->director = NULL;
-	sp->wrk->storage_hint = NULL;
+	wrk->storage_hint = NULL;
 
 	switch (sp->handling) {
 	case VCL_RET_RESTART:
@@ -686,7 +703,9 @@ cnt_fetchbody(struct sess *sp)
 	int varyl = 0, pass;
 	struct worker *wrk;
 
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
 	wrk = sp->wrk;
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 
 	CHECK_OBJ_NOTNULL(wrk->busyobj, BUSYOBJ_MAGIC);
 
@@ -913,54 +932,59 @@ cnt_streambody(struct sess *sp)
 	struct stream_ctx sctx;
 	uint8_t obuf[sp->wrk->res_mode & RES_GUNZIP ?
 	    cache_param->gzip_stack_buffer : 1];
+	struct worker *wrk;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	wrk = sp->wrk;
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 
-	CHECK_OBJ_NOTNULL(sp->wrk->busyobj, BUSYOBJ_MAGIC);
+	CHECK_OBJ_NOTNULL(wrk->busyobj, BUSYOBJ_MAGIC);
 	memset(&sctx, 0, sizeof sctx);
 	sctx.magic = STREAM_CTX_MAGIC;
-	AZ(sp->wrk->sctx);
-	sp->wrk->sctx = &sctx;
+	AZ(wrk->sctx);
+	wrk->sctx = &sctx;
 
-	if (sp->wrk->res_mode & RES_GUNZIP) {
-		sctx.vgz = VGZ_NewUngzip(sp->wrk, "U S -");
+	if (wrk->res_mode & RES_GUNZIP) {
+		sctx.vgz = VGZ_NewUngzip(wrk, "U S -");
 		sctx.obuf = obuf;
 		sctx.obuf_len = sizeof (obuf);
 	}
 
 	RES_StreamStart(sp);
 
-	AssertObjCorePassOrBusy(sp->wrk->obj->objcore);
+	AssertObjCorePassOrBusy(wrk->obj->objcore);
 
-	i = FetchBody(sp->wrk, sp->wrk->obj);
+	i = FetchBody(wrk, wrk->obj);
 
-	sp->wrk->h_content_length = NULL;
+	wrk->h_content_length = NULL;
 
-	http_Setup(sp->wrk->bereq, NULL);
-	http_Setup(sp->wrk->beresp, NULL);
-	sp->wrk->vfp = NULL;
-	AZ(sp->wrk->vbc);
+	http_Setup(wrk->bereq, NULL);
+	http_Setup(wrk->beresp, NULL);
+	wrk->vfp = NULL;
+	AZ(wrk->vbc);
 	AN(sp->director);
 
-	if (!i && sp->wrk->obj->objcore != NULL) {
-		EXP_Insert(sp->wrk->obj);
-		AN(sp->wrk->obj->objcore);
-		AN(sp->wrk->obj->objcore->ban);
-		HSH_Unbusy(sp->wrk);
+	if (!i && wrk->obj->objcore != NULL) {
+		EXP_Insert(wrk->obj);
+		AN(wrk->obj->objcore);
+		AN(wrk->obj->objcore->ban);
+		HSH_Unbusy(wrk);
 	} else {
 		sp->doclose = "Stream error";
 	}
-	sp->wrk->acct_tmp.fetch++;
+	wrk->acct_tmp.fetch++;
 	sp->director = NULL;
 	sp->restarts = 0;
 
 	RES_StreamEnd(sp);
-	if (sp->wrk->res_mode & RES_GUNZIP)
+	if (wrk->res_mode & RES_GUNZIP)
 		(void)VGZ_Destroy(&sctx.vgz, sp->vsl_id);
 
-	sp->wrk->sctx = NULL;
-	assert(WRW_IsReleased(sp->wrk));
-	assert(sp->wrk->wrw.ciov == sp->wrk->wrw.siov);
-	(void)HSH_Deref(sp->wrk, NULL, &sp->wrk->obj);
-	http_Setup(sp->wrk->resp, NULL);
+	wrk->sctx = NULL;
+	assert(WRW_IsReleased(wrk));
+	assert(wrk->wrw.ciov == wrk->wrw.siov);
+	(void)HSH_Deref(wrk, NULL, &wrk->obj);
+	http_Setup(wrk->resp, NULL);
 	sp->step = STP_DONE;
 	return (0);
 }
@@ -979,6 +1003,11 @@ DOT first -> wait
 static int
 cnt_first(struct sess *sp)
 {
+	struct worker *wrk;
+
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	wrk = sp->wrk;
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 
 	/*
 	 * XXX: If we don't have acceptfilters we are somewhat subject
@@ -998,7 +1027,7 @@ cnt_first(struct sess *sp)
 	/* Receive a HTTP protocol request */
 	HTC_Init(sp->htc, sp->ws, sp->fd, sp->vsl_id, cache_param->http_req_size,
 	    cache_param->http_req_hdr_len);
-	sp->wrk->acct_tmp.sess++;
+	wrk->acct_tmp.sess++;
 
 	sp->step = STP_WAIT;
 	return (0);
@@ -1105,8 +1134,12 @@ cnt_lookup(struct sess *sp)
 	struct objcore *oc;
 	struct object *o;
 	struct objhead *oh;
+	struct worker *wrk;
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	wrk = sp->wrk;
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
+
 	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
 
 	if (sp->hash_objhead == NULL) {
@@ -1140,7 +1173,7 @@ cnt_lookup(struct sess *sp)
 
 	/* If we inserted a new object it's a miss */
 	if (oc->flags & OC_F_BUSY) {
-		sp->wrk->stats.cache_miss++;
+		wrk->stats.cache_miss++;
 
 		if (sp->vary_l != NULL) {
 			assert(oc->busyobj->vary == sp->vary_b);
@@ -1154,15 +1187,15 @@ cnt_lookup(struct sess *sp)
 		sp->vary_l = NULL;
 		sp->vary_e = NULL;
 
-		sp->wrk->objcore = oc;
-		CHECK_OBJ_NOTNULL(sp->wrk->busyobj, BUSYOBJ_MAGIC);
+		wrk->objcore = oc;
+		CHECK_OBJ_NOTNULL(wrk->busyobj, BUSYOBJ_MAGIC);
 		sp->step = STP_MISS;
 		return (0);
 	}
 
-	o = oc_getobj(sp->wrk, oc);
+	o = oc_getobj(wrk, oc);
 	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
-	sp->wrk->obj = o;
+	wrk->obj = o;
 
 	WS_Release(sp->ws, 0);
 	sp->vary_b = NULL;
@@ -1170,16 +1203,16 @@ cnt_lookup(struct sess *sp)
 	sp->vary_e = NULL;
 
 	if (oc->flags & OC_F_PASS) {
-		sp->wrk->stats.cache_hitpass++;
-		WSP(sp, SLT_HitPass, "%u", sp->wrk->obj->xid);
-		(void)HSH_Deref(sp->wrk, NULL, &sp->wrk->obj);
-		sp->wrk->objcore = NULL;
+		wrk->stats.cache_hitpass++;
+		WSP(sp, SLT_HitPass, "%u", wrk->obj->xid);
+		(void)HSH_Deref(wrk, NULL, &wrk->obj);
+		wrk->objcore = NULL;
 		sp->step = STP_PASS;
 		return (0);
 	}
 
-	sp->wrk->stats.cache_hit++;
-	WSP(sp, SLT_Hit, "%u", sp->wrk->obj->xid);
+	wrk->stats.cache_hit++;
+	WSP(sp, SLT_Hit, "%u", wrk->obj->xid);
 	sp->step = STP_HIT;
 	return (0);
 }
@@ -1210,53 +1243,56 @@ DOT
 static int
 cnt_miss(struct sess *sp)
 {
+	struct worker *wrk;
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	wrk = sp->wrk;
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
 
-	AZ(sp->wrk->obj);
-	AN(sp->wrk->objcore);
-	CHECK_OBJ_NOTNULL(sp->wrk->busyobj, BUSYOBJ_MAGIC);
-	WS_Reset(sp->wrk->ws, NULL);
-	http_Setup(sp->wrk->bereq, sp->wrk->ws);
+	AZ(wrk->obj);
+	AN(wrk->objcore);
+	CHECK_OBJ_NOTNULL(wrk->busyobj, BUSYOBJ_MAGIC);
+	WS_Reset(wrk->ws, NULL);
+	http_Setup(wrk->bereq, wrk->ws);
 	http_FilterHeader(sp, HTTPH_R_FETCH);
-	http_ForceGet(sp->wrk->bereq);
+	http_ForceGet(wrk->bereq);
 	if (cache_param->http_gzip_support) {
 		/*
 		 * We always ask the backend for gzip, even if the
 		 * client doesn't grok it.  We will uncompress for
 		 * the minority of clients which don't.
 		 */
-		http_Unset(sp->wrk->bereq, H_Accept_Encoding);
-		http_SetHeader(sp->wrk, sp->vsl_id, sp->wrk->bereq,
+		http_Unset(wrk->bereq, H_Accept_Encoding);
+		http_SetHeader(wrk, sp->vsl_id, wrk->bereq,
 		    "Accept-Encoding: gzip");
 	}
-	sp->wrk->connect_timeout = 0;
-	sp->wrk->first_byte_timeout = 0;
-	sp->wrk->between_bytes_timeout = 0;
-	CHECK_OBJ_NOTNULL(sp->wrk->busyobj, BUSYOBJ_MAGIC);
+	wrk->connect_timeout = 0;
+	wrk->first_byte_timeout = 0;
+	wrk->between_bytes_timeout = 0;
+	CHECK_OBJ_NOTNULL(wrk->busyobj, BUSYOBJ_MAGIC);
 
 	VCL_miss_method(sp);
-	CHECK_OBJ_NOTNULL(sp->wrk->busyobj, BUSYOBJ_MAGIC);
+	CHECK_OBJ_NOTNULL(wrk->busyobj, BUSYOBJ_MAGIC);
 	switch(sp->handling) {
 	case VCL_RET_ERROR:
-		AZ(HSH_Deref(sp->wrk, sp->wrk->objcore, NULL));
-		sp->wrk->objcore = NULL;
-		http_Setup(sp->wrk->bereq, NULL);
+		AZ(HSH_Deref(wrk, wrk->objcore, NULL));
+		wrk->objcore = NULL;
+		http_Setup(wrk->bereq, NULL);
 		sp->step = STP_ERROR;
 		return (0);
 	case VCL_RET_PASS:
-		AZ(HSH_Deref(sp->wrk, sp->wrk->objcore, NULL));
-		sp->wrk->objcore = NULL;
+		AZ(HSH_Deref(wrk, wrk->objcore, NULL));
+		wrk->objcore = NULL;
 		sp->step = STP_PASS;
 		return (0);
 	case VCL_RET_FETCH:
-		CHECK_OBJ_NOTNULL(sp->wrk->busyobj, BUSYOBJ_MAGIC);
+		CHECK_OBJ_NOTNULL(wrk->busyobj, BUSYOBJ_MAGIC);
 		sp->step = STP_FETCH;
 		return (0);
 	case VCL_RET_RESTART:
-		AZ(HSH_Deref(sp->wrk, sp->wrk->objcore, NULL));
-		sp->wrk->objcore = NULL;
+		AZ(HSH_Deref(wrk, wrk->objcore, NULL));
+		wrk->objcore = NULL;
 		INCOMPL();
 	default:
 		WRONG("Illegal action in vcl_miss{}");
@@ -1298,29 +1334,32 @@ DOT err_pass [label="ERROR",shape=plaintext]
 static int
 cnt_pass(struct sess *sp)
 {
+	struct worker *wrk;
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	wrk = sp->wrk;
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
-	AZ(sp->wrk->obj);
+	AZ(wrk->obj);
 
-	WS_Reset(sp->wrk->ws, NULL);
-	http_Setup(sp->wrk->bereq, sp->wrk->ws);
+	WS_Reset(wrk->ws, NULL);
+	http_Setup(wrk->bereq, wrk->ws);
 	http_FilterHeader(sp, HTTPH_R_PASS);
 
-	sp->wrk->connect_timeout = 0;
-	sp->wrk->first_byte_timeout = 0;
-	sp->wrk->between_bytes_timeout = 0;
+	wrk->connect_timeout = 0;
+	wrk->first_byte_timeout = 0;
+	wrk->between_bytes_timeout = 0;
 	VCL_pass_method(sp);
 	if (sp->handling == VCL_RET_ERROR) {
-		http_Setup(sp->wrk->bereq, NULL);
+		http_Setup(wrk->bereq, NULL);
 		sp->step = STP_ERROR;
 		return (0);
 	}
 	assert(sp->handling == VCL_RET_PASS);
-	sp->wrk->acct_tmp.pass++;
+	wrk->acct_tmp.pass++;
 	sp->sendbody = 1;
 	sp->step = STP_FETCH;
-	New_BusyObj(sp->wrk);
+	New_BusyObj(wrk);
 	return (0);
 }
 
@@ -1352,13 +1391,16 @@ DOT err_pipe [label="ERROR",shape=plaintext]
 static int
 cnt_pipe(struct sess *sp)
 {
+	struct worker *wrk;
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	wrk = sp->wrk;
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
 
-	sp->wrk->acct_tmp.pipe++;
-	WS_Reset(sp->wrk->ws, NULL);
-	http_Setup(sp->wrk->bereq, sp->wrk->ws);
+	wrk->acct_tmp.pipe++;
+	WS_Reset(wrk->ws, NULL);
+	http_Setup(wrk->bereq, wrk->ws);
 	http_FilterHeader(sp, HTTPH_R_PIPE);
 
 	VCL_pipe_method(sp);
@@ -1368,8 +1410,8 @@ cnt_pipe(struct sess *sp)
 	assert(sp->handling == VCL_RET_PIPE);
 
 	PipeSession(sp);
-	assert(WRW_IsReleased(sp->wrk));
-	http_Setup(sp->wrk->bereq, NULL);
+	assert(WRW_IsReleased(wrk));
+	http_Setup(wrk->bereq, NULL);
 	sp->step = STP_DONE;
 	return (0);
 }
@@ -1397,12 +1439,15 @@ DOT recv -> hash [label="lookup",style=bold,color=green]
 static int
 cnt_recv(struct sess *sp)
 {
+	struct worker *wrk;
 	unsigned recv_handling;
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	wrk = sp->wrk;
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
-	AZ(sp->wrk->obj);
-	assert(sp->wrk->wrw.ciov == sp->wrk->wrw.siov);
+	AZ(wrk->obj);
+	assert(wrk->wrw.ciov == wrk->wrw.siov);
 
 	/* By default we use the first backend */
 	AZ(sp->director);
@@ -1427,27 +1472,27 @@ cnt_recv(struct sess *sp)
 	}
 
 	/* Zap these, in case we came here through restart */
-	sp->wrk->do_esi = 0;
-	sp->wrk->do_gzip = 0;
-	sp->wrk->do_gunzip = 0;
-	sp->wrk->do_stream = 0;
+	wrk->do_esi = 0;
+	wrk->do_gzip = 0;
+	wrk->do_gunzip = 0;
+	wrk->do_stream = 0;
 
 	if (cache_param->http_gzip_support &&
 	     (recv_handling != VCL_RET_PIPE) &&
 	     (recv_handling != VCL_RET_PASS)) {
 		if (RFC2616_Req_Gzip(sp)) {
 			http_Unset(sp->http, H_Accept_Encoding);
-			http_SetHeader(sp->wrk, sp->vsl_id, sp->http,
+			http_SetHeader(wrk, sp->vsl_id, sp->http,
 			    "Accept-Encoding: gzip");
 		} else {
 			http_Unset(sp->http, H_Accept_Encoding);
 		}
 	}
 
-	SHA256_Init(sp->wrk->sha256ctx);
+	SHA256_Init(wrk->sha256ctx);
 	VCL_hash_method(sp);
 	assert(sp->handling == VCL_RET_HASH);
-	SHA256_Final(sp->digest, sp->wrk->sha256ctx);
+	SHA256_Final(sp->digest, wrk->sha256ctx);
 
 	if (!strcmp(sp->http->hd[HTTP_HDR_REQ].b, "HEAD"))
 		sp->wantbody = 0;
@@ -1499,26 +1544,29 @@ cnt_start(struct sess *sp)
 	uint16_t done;
 	char *p;
 	const char *r = "HTTP/1.1 100 Continue\r\n\r\n";
+	struct worker *wrk;
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	wrk = sp->wrk;
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	AZ(sp->restarts);
-	AZ(sp->wrk->obj);
+	AZ(wrk->obj);
 	AZ(sp->vcl);
 	AZ(sp->esi_level);
 
 	/* Update stats of various sorts */
-	sp->wrk->stats.client_req++;
-	sp->t_req = W_TIM_real(sp->wrk);
-	sp->wrk->acct_tmp.req++;
+	wrk->stats.client_req++;
+	sp->t_req = W_TIM_real(wrk);
+	wrk->acct_tmp.req++;
 
 	/* Assign XID and log */
 	sp->xid = ++xids;				/* XXX not locked */
 	WSP(sp, SLT_ReqStart, "%s %s %u", sp->addr, sp->port,  sp->xid);
 
 	/* Borrow VCL reference from worker thread */
-	VCL_Refresh(&sp->wrk->vcl);
-	sp->vcl = sp->wrk->vcl;
-	sp->wrk->vcl = NULL;
+	VCL_Refresh(&wrk->vcl);
+	sp->vcl = wrk->vcl;
+	wrk->vcl = NULL;
 
 	http_Setup(sp->http, sp->ws);
 	done = http_DissectRequest(sp);
@@ -1594,11 +1642,11 @@ void
 CNT_Session(struct sess *sp)
 {
 	int done;
-	struct worker *w;
+	struct worker *wrk;
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-	w = sp->wrk;
-	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	wrk = sp->wrk;
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 
 	/*
 	 * Possible entrance states
@@ -1609,12 +1657,12 @@ CNT_Session(struct sess *sp)
 	    sp->step == STP_LOOKUP ||
 	    sp->step == STP_RECV);
 
-	AZ(w->do_stream);
-	AZ(w->do_gzip);
-	AZ(w->do_gunzip);
-	AZ(w->do_esi);
-	AZ(w->obj);
-	AZ(w->objcore);
+	AZ(wrk->do_stream);
+	AZ(wrk->do_gzip);
+	AZ(wrk->do_gunzip);
+	AZ(wrk->do_esi);
+	AZ(wrk->obj);
+	AZ(wrk->objcore);
 
 	/*
 	 * Whenever we come in from the acceptor or waiter, we need to set
@@ -1637,16 +1685,16 @@ CNT_Session(struct sess *sp)
 	 * NB: Once done is set, we can no longer touch sp!
 	 */
 	for (done = 0; !done; ) {
-		assert(sp->wrk == w);
+		assert(sp->wrk == wrk);
 		/*
 		 * This is a good place to be paranoid about the various
 		 * pointers still pointing to the things we expect.
 		 */
 		CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
-		CHECK_OBJ_ORNULL(sp->wrk->obj, OBJECT_MAGIC);
-		CHECK_OBJ_NOTNULL(sp->wrk, WORKER_MAGIC);
-		CHECK_OBJ_ORNULL(w->nobjhead, OBJHEAD_MAGIC);
-		WS_Assert(w->ws);
+		CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
+		CHECK_OBJ_ORNULL(wrk->obj, OBJECT_MAGIC);
+		CHECK_OBJ_ORNULL(wrk->nobjhead, OBJHEAD_MAGIC);
+		WS_Assert(wrk->ws);
 
 		switch (sp->step) {
 #define STEP(l,u) \
@@ -1660,20 +1708,20 @@ CNT_Session(struct sess *sp)
 		default:
 			WRONG("State engine misfire");
 		}
-		WS_Assert(w->ws);
-		CHECK_OBJ_ORNULL(w->nobjhead, OBJHEAD_MAGIC);
+		WS_Assert(wrk->ws);
+		CHECK_OBJ_ORNULL(wrk->nobjhead, OBJHEAD_MAGIC);
 	}
-	WSL_Flush(w, 0);
-	AZ(w->obj);
-	AZ(w->objcore);
-	AZ(w->do_stream);
-	AZ(w->do_gzip);
-	AZ(w->do_gunzip);
-	AZ(w->do_esi);
-#define ACCT(foo)	AZ(w->acct_tmp.foo);
+	WSL_Flush(wrk, 0);
+	AZ(wrk->obj);
+	AZ(wrk->objcore);
+	AZ(wrk->do_stream);
+	AZ(wrk->do_gzip);
+	AZ(wrk->do_gunzip);
+	AZ(wrk->do_esi);
+#define ACCT(foo)	AZ(wrk->acct_tmp.foo);
 #include "tbl/acct_fields.h"
 #undef ACCT
-	assert(WRW_IsReleased(w));
+	assert(WRW_IsReleased(wrk));
 }
 
 /*
diff --git a/bin/varnishd/cache/cache_pool.c b/bin/varnishd/cache/cache_pool.c
index 730e391..582bb70 100644
--- a/bin/varnishd/cache/cache_pool.c
+++ b/bin/varnishd/cache/cache_pool.c
@@ -361,8 +361,11 @@ Pool_Wait(struct sess *sp)
 {
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(sp->wrk, WORKER_MAGIC);
 	AZ(sp->vcl);
 	assert(sp->fd >= 0);
+	sp->wrk = NULL;
+
 	/*
 	 * Set nonblocking in the worker-thread, before passing to the
 	 * acceptor thread, to reduce syscall density of the latter.
diff --git a/bin/varnishd/cache/cache_shmlog.c b/bin/varnishd/cache/cache_shmlog.c
index 1e414b8..3cfd451 100644
--- a/bin/varnishd/cache/cache_shmlog.c
+++ b/bin/varnishd/cache/cache_shmlog.c
@@ -335,7 +335,8 @@ VSM_Init(void)
 	VWMB();
 
 	vsl_start = vsl_log_start;
-	vsl_end = vsl_start + cache_param->vsl_space / sizeof *vsl_end;
+	vsl_end = vsl_start +
+	    cache_param->vsl_space / (unsigned)sizeof *vsl_end;
 	vsl_ptr = vsl_start + 1;
 
 	VSC_C_main = VSM_Alloc(sizeof *VSC_C_main,



From phk at varnish-cache.org  Tue Nov 29 17:30:54 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 29 Nov 2011 18:30:54 +0100
Subject: [master] eb6fa2f Move vfp from worker to busyobj
Message-ID: <E1RVRWA-0001LY-CK@project.varnish-software.com>

commit eb6fa2f150979d12ede60d2075f76102f45eb1ee
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 29 17:30:40 2011 +0000

    Move vfp from worker to busyobj

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 7ad7fbd..21382d7 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -334,7 +334,6 @@ struct worker {
 	struct vbc		*vbc;
 	struct object		*fetch_obj;
 	enum body_status	body_status;
-	struct vfp		*vfp;
 	struct vgz		*vgz_rx;
 	struct vef_priv		*vef_priv;
 	unsigned		fetch_failed;
@@ -485,7 +484,16 @@ oc_getlru(const struct objcore *oc)
 	return (oc->methods->getlru(oc));
 }
 
-/* Busy Object structure ---------------------------------------------*/
+/* Busy Object structure ---------------------------------------------
+ *
+ * The busyobj structure captures the aspects of an object related to,
+ * and while it is being fetched from the backend.
+ *
+ * One of these aspects will be how much has been fetched, which
+ * streaming delivery will make use of.
+ *
+ * XXX: many fields from worker needs to move here.
+ */
 
 struct busyobj {
 	unsigned		magic;
@@ -493,6 +501,8 @@ struct busyobj {
 	uint8_t			*vary;
 	unsigned		is_gzip;
 	unsigned		is_gunzip;
+
+	struct vfp		*vfp;
 };
 
 /* Object structure --------------------------------------------------*/
diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c
index 85224b6..27209de 100644
--- a/bin/varnishd/cache/cache_center.c
+++ b/bin/varnishd/cache/cache_center.c
@@ -706,7 +706,6 @@ cnt_fetchbody(struct sess *sp)
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
 	wrk = sp->wrk;
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
-
 	CHECK_OBJ_NOTNULL(wrk->busyobj, BUSYOBJ_MAGIC);
 
 	assert(sp->handling == VCL_RET_HIT_FOR_PASS ||
@@ -739,8 +738,6 @@ cnt_fetchbody(struct sess *sp)
 	 *
 	 */
 
-	AZ(wrk->vfp);
-
 	/* We do nothing unless the param is set */
 	if (!cache_param->http_gzip_support)
 		wrk->do_gzip = wrk->do_gunzip = 0;
@@ -776,13 +773,13 @@ cnt_fetchbody(struct sess *sp)
 
 	/* ESI takes precedence and handles gzip/gunzip itself */
 	if (wrk->do_esi)
-		wrk->vfp = &vfp_esi;
+		wrk->busyobj->vfp = &vfp_esi;
 	else if (wrk->do_gunzip)
-		wrk->vfp = &vfp_gunzip;
+		wrk->busyobj->vfp = &vfp_gunzip;
 	else if (wrk->do_gzip)
-		wrk->vfp = &vfp_gzip;
+		wrk->busyobj->vfp = &vfp_gzip;
 	else if (wrk->busyobj->is_gzip)
-		wrk->vfp = &vfp_testgzip;
+		wrk->busyobj->vfp = &vfp_testgzip;
 
 	if (wrk->do_esi || sp->esi_level > 0)
 		wrk->do_stream = 0;
@@ -890,7 +887,7 @@ cnt_fetchbody(struct sess *sp)
 
 	http_Setup(wrk->bereq, NULL);
 	http_Setup(wrk->beresp, NULL);
-	wrk->vfp = NULL;
+	wrk->busyobj->vfp = NULL;
 	assert(WRW_IsReleased(wrk));
 	AZ(wrk->vbc);
 	AN(sp->director);
@@ -960,7 +957,7 @@ cnt_streambody(struct sess *sp)
 
 	http_Setup(wrk->bereq, NULL);
 	http_Setup(wrk->beresp, NULL);
-	wrk->vfp = NULL;
+	wrk->busyobj->vfp = NULL;
 	AZ(wrk->vbc);
 	AN(sp->director);
 
diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 66241bc..9413300 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -236,7 +236,7 @@ fetch_straight(struct worker *w, struct http_conn *htc, ssize_t cl)
 	} else if (cl == 0)
 		return (0);
 
-	i = w->vfp->bytes(w, htc, cl);
+	i = w->busyobj->vfp->bytes(w, htc, cl);
 	if (i <= 0)
 		return (FetchError(w, "straight insufficient bytes"));
 	return (0);
@@ -293,7 +293,7 @@ fetch_chunked(struct worker *w, struct http_conn *htc)
 		if (cl < 0)
 			return (FetchError(w,"chunked header number syntax"));
 
-		if (cl > 0 && w->vfp->bytes(w, htc, cl) <= 0)
+		if (cl > 0 && w->busyobj->vfp->bytes(w, htc, cl) <= 0)
 			return (-1);
 
 		i = HTC_Read(w, htc, buf, 1);
@@ -315,7 +315,7 @@ fetch_eof(struct worker *w, struct http_conn *htc)
 	int i;
 
 	assert(w->body_status == BS_EOF);
-	i = w->vfp->bytes(w, htc, SSIZE_MAX);
+	i = w->busyobj->vfp->bytes(w, htc, SSIZE_MAX);
 	if (i < 0)
 		return (-1);
 	return (0);
@@ -494,8 +494,8 @@ FetchBody(struct worker *w, struct object *obj)
 	CHECK_OBJ_NOTNULL(obj, OBJECT_MAGIC);
 	CHECK_OBJ_NOTNULL(obj->http, HTTP_MAGIC);
 
-	if (w->vfp == NULL)
-		w->vfp = &vfp_nop;
+	if (w->busyobj->vfp == NULL)
+		w->busyobj->vfp = &vfp_nop;
 
 	AssertObjCorePassOrBusy(obj->objcore);
 
@@ -518,24 +518,24 @@ FetchBody(struct worker *w, struct object *obj)
 		break;
 	case BS_LENGTH:
 		cl = fetch_number( w->h_content_length, 10);
-		w->vfp->begin(w, cl > 0 ? cl : 0);
+		w->busyobj->vfp->begin(w, cl > 0 ? cl : 0);
 		cls = fetch_straight(w, w->htc, cl);
 		mklen = 1;
-		if (w->vfp->end(w))
+		if (w->busyobj->vfp->end(w))
 			cls = -1;
 		break;
 	case BS_CHUNKED:
-		w->vfp->begin(w, cl);
+		w->busyobj->vfp->begin(w, cl);
 		cls = fetch_chunked(w, w->htc);
 		mklen = 1;
-		if (w->vfp->end(w))
+		if (w->busyobj->vfp->end(w))
 			cls = -1;
 		break;
 	case BS_EOF:
-		w->vfp->begin(w, cl);
+		w->busyobj->vfp->begin(w, cl);
 		cls = fetch_eof(w, w->htc);
 		mklen = 1;
-		if (w->vfp->end(w))
+		if (w->busyobj->vfp->end(w))
 			cls = -1;
 		break;
 	case BS_ERROR:



From phk at varnish-cache.org  Tue Nov 29 17:51:15 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 29 Nov 2011 18:51:15 +0100
Subject: [master] 64e06ad Move vep from worker to busyobj
Message-ID: <E1RVRpr-000308-Ns@project.varnish-software.com>

commit 64e06add5bacfb6b4a9ba855617f704e0691fdf6
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 29 17:44:41 2011 +0000

    Move vep from worker to busyobj

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 21382d7..634d9aa 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -348,7 +348,6 @@ struct worker {
 	struct stream_ctx	*sctx;
 
 	/* ESI stuff */
-	struct vep_state	*vep;
 	int			gzip_resp;
 	ssize_t			l_crc;
 	uint32_t		crc;
@@ -503,6 +502,7 @@ struct busyobj {
 	unsigned		is_gunzip;
 
 	struct vfp		*vfp;
+	struct vep_state	*vep;
 };
 
 /* Object structure --------------------------------------------------*/
diff --git a/bin/varnishd/cache/cache_esi.h b/bin/varnishd/cache/cache_esi.h
index ff84d41..4867614 100644
--- a/bin/varnishd/cache/cache_esi.h
+++ b/bin/varnishd/cache/cache_esi.h
@@ -43,6 +43,6 @@ typedef ssize_t vep_callback_t(struct worker *w, ssize_t l, enum vgz_flag flg);
 
 void VEP_Init(struct worker *w, vep_callback_t *cb);
 void VEP_Parse(const struct worker *w, const char *p, size_t l);
-struct vsb *VEP_Finish(struct worker *w);
+struct vsb *VEP_Finish(const struct worker *w);
 
 
diff --git a/bin/varnishd/cache/cache_esi_fetch.c b/bin/varnishd/cache/cache_esi_fetch.c
index 35166e7..f5748a3 100644
--- a/bin/varnishd/cache/cache_esi_fetch.c
+++ b/bin/varnishd/cache/cache_esi_fetch.c
@@ -325,7 +325,7 @@ vfp_esi_begin(struct worker *w, size_t estimate)
 	}
 
 	(void)estimate;
-	AN(w->vep);
+	AN(w->busyobj->vep);
 }
 
 static int __match_proto__()
@@ -335,7 +335,7 @@ vfp_esi_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
 
 	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
 	AZ(w->fetch_failed);
-	AN(w->vep);
+	AN(w->busyobj->vep);
 	assert(w->htc == htc);
 	if (w->busyobj->is_gzip && w->do_gunzip)
 		i = vfp_esi_bytes_gu(w, htc, bytes);
@@ -345,7 +345,7 @@ vfp_esi_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
 		i = vfp_esi_bytes_gg(w, htc, bytes);
 	else
 		i = vfp_esi_bytes_uu(w, htc, bytes);
-	AN(w->vep);
+	AN(w->busyobj->vep);
 	return (i);
 }
 
@@ -358,7 +358,7 @@ vfp_esi_end(struct worker *w)
 	int retval;
 
 	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	AN(w->vep);
+	AN(w->busyobj->vep);
 
 	retval = w->fetch_failed;
 
diff --git a/bin/varnishd/cache/cache_esi_parse.c b/bin/varnishd/cache/cache_esi_parse.c
index 9e2b4f6..71ec16d 100644
--- a/bin/varnishd/cache/cache_esi_parse.c
+++ b/bin/varnishd/cache/cache_esi_parse.c
@@ -556,7 +556,8 @@ VEP_Parse(const struct worker *w, const char *p, size_t l)
 	int i;
 
 	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	vep = w->vep;
+	CHECK_OBJ_NOTNULL(w->busyobj, BUSYOBJ_MAGIC);
+	vep = w->busyobj->vep;
 	CHECK_OBJ_NOTNULL(vep, VEP_MAGIC);
 	assert(l > 0);
 
@@ -981,12 +982,15 @@ VEP_Parse(const struct worker *w, const char *p, size_t l)
 static ssize_t __match_proto__()
 vep_default_cb(struct worker *w, ssize_t l, enum vgz_flag flg)
 {
+	struct vep_state *vep;
 
+	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	CHECK_OBJ_NOTNULL(w->busyobj, BUSYOBJ_MAGIC);
+	vep = w->busyobj->vep;
+	CHECK_OBJ_NOTNULL(vep, VEP_MAGIC);
 	(void)flg;
-	AN(w->vep);
-	w->vep->cb_x += l;
-Debug("CB(%jd,%d) = %jd\n", (intmax_t)l, flg, (intmax_t)w->vep->cb_x);
-	return (w->vep->cb_x);
+	vep->cb_x += l;
+	return (vep->cb_x);
 }
 
 /*---------------------------------------------------------------------
@@ -998,16 +1002,17 @@ VEP_Init(struct worker *w, vep_callback_t *cb)
 	struct vep_state *vep;
 
 	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	AZ(w->vep);
-	w->vep = (void*)WS_Alloc(w->ws, sizeof *vep);
-	AN(w->vep);
+	CHECK_OBJ_NOTNULL(w->busyobj, BUSYOBJ_MAGIC);
+	AZ(w->busyobj->vep);
+	vep = (void*)WS_Alloc(w->ws, sizeof *vep);
+	AN(vep);
 
-	vep = w->vep;
 	memset(vep, 0, sizeof *vep);
 	vep->magic = VEP_MAGIC;
 	vep->wrk = w;
 	vep->vsb = VSB_new_auto();
 	AN(vep->vsb);
+	w->busyobj->vep = vep;
 
 	if (cb != NULL) {
 		vep->dogzip = 1;
@@ -1038,13 +1043,14 @@ VEP_Init(struct worker *w, vep_callback_t *cb)
  */
 
 struct vsb *
-VEP_Finish(struct worker *w)
+VEP_Finish(const struct worker *w)
 {
 	struct vep_state *vep;
 	ssize_t l, lcb;
 
 	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	vep = w->vep;
+	CHECK_OBJ_NOTNULL(w->busyobj, BUSYOBJ_MAGIC);
+	vep = w->busyobj->vep;
 	CHECK_OBJ_NOTNULL(vep, VEP_MAGIC);
 
 	if (vep->o_pending)
@@ -1055,7 +1061,7 @@ VEP_Finish(struct worker *w)
 	}
 	(void)vep->cb(vep->wrk, 0, VGZ_FINISH);
 
-	w->vep = NULL;
+	w->busyobj->vep = NULL;
 
 	AZ(VSB_finish(vep->vsb));
 	l = VSB_len(vep->vsb);



From phk at varnish-cache.org  Tue Nov 29 17:51:18 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 29 Nov 2011 18:51:18 +0100
Subject: [master] 0f0eae3 Move fetch_failed from worker to busyobj
Message-ID: <E1RVRpu-00030I-3E@project.varnish-software.com>

commit 0f0eae3726097ec9e77ef6dd15471a7b8b077856
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 29 17:50:53 2011 +0000

    Move fetch_failed from worker to busyobj

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 634d9aa..8cffdef 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -336,7 +336,6 @@ struct worker {
 	enum body_status	body_status;
 	struct vgz		*vgz_rx;
 	struct vef_priv		*vef_priv;
-	unsigned		fetch_failed;
 	unsigned		do_stream;
 	unsigned		do_esi;
 	unsigned		do_gzip;
@@ -347,7 +346,7 @@ struct worker {
 	/* Stream state */
 	struct stream_ctx	*sctx;
 
-	/* ESI stuff */
+	/* ESI delivery stuff */
 	int			gzip_resp;
 	ssize_t			l_crc;
 	uint32_t		crc;
@@ -503,6 +502,7 @@ struct busyobj {
 
 	struct vfp		*vfp;
 	struct vep_state	*vep;
+	unsigned		fetch_failed;
 };
 
 /* Object structure --------------------------------------------------*/
diff --git a/bin/varnishd/cache/cache_esi_fetch.c b/bin/varnishd/cache/cache_esi_fetch.c
index f5748a3..b23538c 100644
--- a/bin/varnishd/cache/cache_esi_fetch.c
+++ b/bin/varnishd/cache/cache_esi_fetch.c
@@ -334,7 +334,7 @@ vfp_esi_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
 	int i;
 
 	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	AZ(w->fetch_failed);
+	AZ(w->busyobj->fetch_failed);
 	AN(w->busyobj->vep);
 	assert(w->htc == htc);
 	if (w->busyobj->is_gzip && w->do_gunzip)
@@ -360,7 +360,7 @@ vfp_esi_end(struct worker *w)
 	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
 	AN(w->busyobj->vep);
 
-	retval = w->fetch_failed;
+	retval = w->busyobj->fetch_failed;
 
 	if (w->vgz_rx != NULL && VGZ_Destroy(&w->vgz_rx, -1) != VGZ_END)
 		retval = FetchError(w,
diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 9413300..895a5cb 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -46,7 +46,7 @@ static unsigned fetchfrag;
  * We want to issue the first error we encounter on fetching and
  * supress the rest.  This function does that.
  *
- * Other code is allowed to look at w->fetch_failed to bail out
+ * Other code is allowed to look at w->busyobj->fetch_failed to bail out
  *
  * For convenience, always return -1
  */
@@ -56,13 +56,13 @@ FetchError2(struct worker *w, const char *error, const char *more)
 {
 
 	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	if (!w->fetch_failed) {
+	if (!w->busyobj->fetch_failed) {
 		if (more == NULL)
 			WSLB(w, SLT_FetchError, "%s", error);
 		else
 			WSLB(w, SLT_FetchError, "%s: %s", error, more);
 	}
-	w->fetch_failed = 1;
+	w->busyobj->fetch_failed = 1;
 	return (-1);
 }
 
@@ -112,7 +112,7 @@ vfp_nop_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
 	ssize_t l, wl;
 	struct storage *st;
 
-	AZ(w->fetch_failed);
+	AZ(w->busyobj->fetch_failed);
 	while (bytes > 0) {
 		st = FetchStorage(w, 0);
 		if (st == NULL)
@@ -503,7 +503,7 @@ FetchBody(struct worker *w, struct object *obj)
 	AZ(VTAILQ_FIRST(&obj->store));
 
 	w->fetch_obj = obj;
-	w->fetch_failed = 0;
+	w->busyobj->fetch_failed = 0;
 
 	/* XXX: pick up estimate from objdr ? */
 	cl = 0;
@@ -579,7 +579,7 @@ FetchBody(struct worker *w, struct object *obj)
 		obj->len = 0;
 		return (__LINE__);
 	}
-	AZ(w->fetch_failed);
+	AZ(w->busyobj->fetch_failed);
 
 	if (cls == 0 && w->do_close)
 		cls = 1;
diff --git a/bin/varnishd/cache/cache_gzip.c b/bin/varnishd/cache/cache_gzip.c
index 57b25a5..a175fdf 100644
--- a/bin/varnishd/cache/cache_gzip.c
+++ b/bin/varnishd/cache/cache_gzip.c
@@ -471,7 +471,7 @@ vfp_gunzip_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
 	size_t dl;
 	const void *dp;
 
-	AZ(w->fetch_failed);
+	AZ(w->busyobj->fetch_failed);
 	vg = w->vgz_rx;
 	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
 	AZ(vg->vz.avail_in);
@@ -508,7 +508,7 @@ vfp_gunzip_end(struct worker *w)
 	vg = w->vgz_rx;
 	w->vgz_rx = NULL;
 	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
-	if (w->fetch_failed) {
+	if (w->busyobj->fetch_failed) {
 		(void)VGZ_Destroy(&vg, -1);
 		return(0);
 	}
@@ -549,7 +549,7 @@ vfp_gzip_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
 	size_t dl;
 	const void *dp;
 
-	AZ(w->fetch_failed);
+	AZ(w->busyobj->fetch_failed);
 	vg = w->vgz_rx;
 	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
 	AZ(vg->vz.avail_in);
@@ -586,7 +586,7 @@ vfp_gzip_end(struct worker *w)
 	vg = w->vgz_rx;
 	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
 	w->vgz_rx = NULL;
-	if (w->fetch_failed) {
+	if (w->busyobj->fetch_failed) {
 		(void)VGZ_Destroy(&vg, -1);
 		return(0);
 	}
@@ -637,7 +637,7 @@ vfp_testgzip_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
 	const void *dp;
 	struct storage *st;
 
-	AZ(w->fetch_failed);
+	AZ(w->busyobj->fetch_failed);
 	vg = w->vgz_rx;
 	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
 	AZ(vg->vz.avail_in);
@@ -680,7 +680,7 @@ vfp_testgzip_end(struct worker *w)
 	vg = w->vgz_rx;
 	w->vgz_rx = NULL;
 	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
-	if (w->fetch_failed) {
+	if (w->busyobj->fetch_failed) {
 		(void)VGZ_Destroy(&vg, -1);
 		return(0);
 	}



From phk at varnish-cache.org  Tue Nov 29 17:55:47 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 29 Nov 2011 18:55:47 +0100
Subject: [master] 672fb60 Move vgz_rx from worker to busyobj
Message-ID: <E1RVRuF-0003Vn-TE@project.varnish-software.com>

commit 672fb604cd38bb81a8d65c5b1c24d7bfc019fdf9
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 29 17:55:25 2011 +0000

    Move vgz_rx from worker to busyobj

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 8cffdef..7a0167c 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -334,7 +334,6 @@ struct worker {
 	struct vbc		*vbc;
 	struct object		*fetch_obj;
 	enum body_status	body_status;
-	struct vgz		*vgz_rx;
 	struct vef_priv		*vef_priv;
 	unsigned		do_stream;
 	unsigned		do_esi;
@@ -503,6 +502,7 @@ struct busyobj {
 	struct vfp		*vfp;
 	struct vep_state	*vep;
 	unsigned		fetch_failed;
+	struct vgz		*vgz_rx;
 };
 
 /* Object structure --------------------------------------------------*/
diff --git a/bin/varnishd/cache/cache_esi_fetch.c b/bin/varnishd/cache/cache_esi_fetch.c
index b23538c..3dbc5ce 100644
--- a/bin/varnishd/cache/cache_esi_fetch.c
+++ b/bin/varnishd/cache/cache_esi_fetch.c
@@ -102,7 +102,7 @@ vfp_esi_bytes_gu(struct worker *w, struct http_conn *htc, ssize_t bytes)
 	const void *dp;
 
 	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	vg = w->vgz_rx;
+	vg = w->busyobj->vgz_rx;
 
 	while (bytes > 0) {
 		if (VGZ_IbufEmpty(vg) && bytes > 0) {
@@ -264,10 +264,10 @@ vfp_esi_bytes_gg(struct worker *w, struct http_conn *htc, size_t bytes)
 		bytes -= wl;
 
 		vef->bufp = ibuf;
-		VGZ_Ibuf(w->vgz_rx, ibuf, wl);
+		VGZ_Ibuf(w->busyobj->vgz_rx, ibuf, wl);
 		do {
-			VGZ_Obuf(w->vgz_rx, ibuf2, sizeof ibuf2);
-			i = VGZ_Gunzip(w->vgz_rx, &dp, &dl);
+			VGZ_Obuf(w->busyobj->vgz_rx, ibuf2, sizeof ibuf2);
+			i = VGZ_Gunzip(w->busyobj->vgz_rx, &dp, &dl);
 			/* XXX: check i */
 			assert(i >= VGZ_OK);
 			vef->bufp = ibuf2;
@@ -284,7 +284,7 @@ vfp_esi_bytes_gg(struct worker *w, struct http_conn *htc, size_t bytes)
 				    vef->bufp, dl);
 				vef->npend += dl;
 			}
-		} while (!VGZ_IbufEmpty(w->vgz_rx));
+		} while (!VGZ_IbufEmpty(w->busyobj->vgz_rx));
 	}
 	return (1);
 }
@@ -300,9 +300,9 @@ vfp_esi_begin(struct worker *w, size_t estimate)
 	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(w->busyobj, BUSYOBJ_MAGIC);
 
-	AZ(w->vgz_rx);
+	AZ(w->busyobj->vgz_rx);
 	if (w->busyobj->is_gzip && w->do_gunzip) {
-		w->vgz_rx = VGZ_NewUngzip(w, "U F E");
+		w->busyobj->vgz_rx = VGZ_NewUngzip(w, "U F E");
 		VEP_Init(w, NULL);
 	} else if (w->busyobj->is_gunzip && w->do_gzip) {
 		ALLOC_OBJ(vef, VEF_MAGIC);
@@ -312,7 +312,7 @@ vfp_esi_begin(struct worker *w, size_t estimate)
 		w->vef_priv = vef;
 		VEP_Init(w, vfp_vep_callback);
 	} else if (w->busyobj->is_gzip) {
-		w->vgz_rx = VGZ_NewUngzip(w, "U F E");
+		w->busyobj->vgz_rx = VGZ_NewUngzip(w, "U F E");
 		ALLOC_OBJ(vef, VEF_MAGIC);
 		AN(vef);
 		vef->vgz = VGZ_NewGzip(w, "G F E");
@@ -362,7 +362,8 @@ vfp_esi_end(struct worker *w)
 
 	retval = w->busyobj->fetch_failed;
 
-	if (w->vgz_rx != NULL && VGZ_Destroy(&w->vgz_rx, -1) != VGZ_END)
+	if (w->busyobj->vgz_rx != NULL &&
+	     VGZ_Destroy(&w->busyobj->vgz_rx, -1) != VGZ_END)
 		retval = FetchError(w,
 		    "Gunzip+ESI Failed at the very end");
 
diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 895a5cb..2a2de5b 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -499,7 +499,7 @@ FetchBody(struct worker *w, struct object *obj)
 
 	AssertObjCorePassOrBusy(obj->objcore);
 
-	AZ(w->vgz_rx);
+	AZ(w->busyobj->vgz_rx);
 	AZ(VTAILQ_FIRST(&obj->store));
 
 	w->fetch_obj = obj;
@@ -547,7 +547,7 @@ FetchBody(struct worker *w, struct object *obj)
 		mklen = 0;
 		INCOMPL();
 	}
-	AZ(w->vgz_rx);
+	AZ(w->busyobj->vgz_rx);
 
 	/*
 	 * It is OK for ->end to just leave the last storage segment
diff --git a/bin/varnishd/cache/cache_gzip.c b/bin/varnishd/cache/cache_gzip.c
index a175fdf..f325292 100644
--- a/bin/varnishd/cache/cache_gzip.c
+++ b/bin/varnishd/cache/cache_gzip.c
@@ -457,8 +457,8 @@ static void __match_proto__()
 vfp_gunzip_begin(struct worker *w, size_t estimate)
 {
 	(void)estimate;
-	AZ(w->vgz_rx);
-	w->vgz_rx = VGZ_NewUngzip(w, "U F -");
+	AZ(w->busyobj->vgz_rx);
+	w->busyobj->vgz_rx = VGZ_NewUngzip(w, "U F -");
 }
 
 static int __match_proto__()
@@ -472,7 +472,7 @@ vfp_gunzip_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
 	const void *dp;
 
 	AZ(w->busyobj->fetch_failed);
-	vg = w->vgz_rx;
+	vg = w->busyobj->vgz_rx;
 	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
 	AZ(vg->vz.avail_in);
 	while (bytes > 0 || vg->vz.avail_in > 0) {
@@ -505,8 +505,8 @@ vfp_gunzip_end(struct worker *w)
 {
 	struct vgz *vg;
 
-	vg = w->vgz_rx;
-	w->vgz_rx = NULL;
+	vg = w->busyobj->vgz_rx;
+	w->busyobj->vgz_rx = NULL;
 	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
 	if (w->busyobj->fetch_failed) {
 		(void)VGZ_Destroy(&vg, -1);
@@ -535,8 +535,8 @@ vfp_gzip_begin(struct worker *w, size_t estimate)
 {
 	(void)estimate;
 
-	AZ(w->vgz_rx);
-	w->vgz_rx = VGZ_NewGzip(w, "G F -");
+	AZ(w->busyobj->vgz_rx);
+	w->busyobj->vgz_rx = VGZ_NewGzip(w, "G F -");
 }
 
 static int __match_proto__()
@@ -550,7 +550,7 @@ vfp_gzip_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
 	const void *dp;
 
 	AZ(w->busyobj->fetch_failed);
-	vg = w->vgz_rx;
+	vg = w->busyobj->vgz_rx;
 	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
 	AZ(vg->vz.avail_in);
 	while (bytes > 0 || !VGZ_IbufEmpty(vg)) {
@@ -583,9 +583,9 @@ vfp_gzip_end(struct worker *w)
 	const void *dp;
 	int i;
 
-	vg = w->vgz_rx;
+	vg = w->busyobj->vgz_rx;
 	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
-	w->vgz_rx = NULL;
+	w->busyobj->vgz_rx = NULL;
 	if (w->busyobj->fetch_failed) {
 		(void)VGZ_Destroy(&vg, -1);
 		return(0);
@@ -622,8 +622,8 @@ static void __match_proto__()
 vfp_testgzip_begin(struct worker *w, size_t estimate)
 {
 	(void)estimate;
-	w->vgz_rx = VGZ_NewUngzip(w, "u F -");
-	CHECK_OBJ_NOTNULL(w->vgz_rx, VGZ_MAGIC);
+	w->busyobj->vgz_rx = VGZ_NewUngzip(w, "u F -");
+	CHECK_OBJ_NOTNULL(w->busyobj->vgz_rx, VGZ_MAGIC);
 }
 
 static int __match_proto__()
@@ -638,7 +638,7 @@ vfp_testgzip_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
 	struct storage *st;
 
 	AZ(w->busyobj->fetch_failed);
-	vg = w->vgz_rx;
+	vg = w->busyobj->vgz_rx;
 	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
 	AZ(vg->vz.avail_in);
 	while (bytes > 0) {
@@ -677,8 +677,8 @@ vfp_testgzip_end(struct worker *w)
 {
 	struct vgz *vg;
 
-	vg = w->vgz_rx;
-	w->vgz_rx = NULL;
+	vg = w->busyobj->vgz_rx;
+	w->busyobj->vgz_rx = NULL;
 	CHECK_OBJ_NOTNULL(vg, VGZ_MAGIC);
 	if (w->busyobj->fetch_failed) {
 		(void)VGZ_Destroy(&vg, -1);



From phk at varnish-cache.org  Tue Nov 29 18:10:39 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 29 Nov 2011 19:10:39 +0100
Subject: [master] 9d67fce Move (beresp.)exp from worker to busyobj
Message-ID: <E1RVS8d-0004j1-IC@project.varnish-software.com>

commit 9d67fcef723e583e56dbb43bf202259fe45f4b07
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 29 18:10:22 2011 +0000

    Move (beresp.)exp from worker to busyobj

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 7a0167c..5196575 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -325,8 +325,6 @@ struct worker {
 	struct objcore		*objcore;
 	struct busyobj		*busyobj;
 
-	struct exp		exp;
-
 	/* This is only here so VRT can find it */
 	const char		*storage_hint;
 
@@ -503,6 +501,8 @@ struct busyobj {
 	struct vep_state	*vep;
 	unsigned		fetch_failed;
 	struct vgz		*vgz_rx;
+
+	struct exp		exp;
 };
 
 /* Object structure --------------------------------------------------*/
diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c
index 27209de..86b54d0 100644
--- a/bin/varnishd/cache/cache_center.c
+++ b/bin/varnishd/cache/cache_center.c
@@ -464,7 +464,7 @@ cnt_error(struct sess *sp)
 
 	if (wrk->obj == NULL) {
 		HSH_Prealloc(sp);
-		EXP_Clr(&wrk->exp);
+		New_BusyObj(wrk);
 		wrk->obj = STV_NewObject(wrk, NULL, cache_param->http_resp_size,
 		     (uint16_t)cache_param->http_max_hdr);
 		if (wrk->obj == NULL)
@@ -613,13 +613,13 @@ cnt_fetch(struct sess *sp)
 		/*
 		 * What does RFC2616 think about TTL ?
 		 */
-		EXP_Clr(&wrk->exp);
-		wrk->exp.entered = W_TIM_real(wrk);
+		EXP_Clr(&wrk->busyobj->exp);
+		wrk->busyobj->exp.entered = W_TIM_real(wrk);
 		RFC2616_Ttl(sp);
 
 		/* pass from vclrecv{} has negative TTL */
 		if (wrk->objcore == NULL)
-			wrk->exp.ttl = -1.;
+			wrk->busyobj->exp.ttl = -1.;
 
 		AZ(wrk->do_esi);
 
@@ -715,7 +715,7 @@ cnt_fetchbody(struct sess *sp)
 		/* This is a pass from vcl_recv */
 		pass = 1;
 		/* VCL may have fiddled this, but that doesn't help */
-		wrk->exp.ttl = -1.;
+		wrk->busyobj->exp.ttl = -1.;
 	} else if (sp->handling == VCL_RET_HIT_FOR_PASS) {
 		/* pass from vcl_fetch{} -> hit-for-pass */
 		/* XXX: the bereq was not filtered pass... */
@@ -806,7 +806,8 @@ cnt_fetchbody(struct sess *sp)
 	 */
 	l += strlen("Content-Length: XxxXxxXxxXxxXxxXxx") + sizeof(void *);
 
-	if (wrk->exp.ttl < cache_param->shortlived || wrk->objcore == NULL)
+	if (wrk->busyobj->exp.ttl < cache_param->shortlived ||
+	    wrk->objcore == NULL)
 		wrk->storage_hint = TRANSIENT_STORAGE;
 
 	wrk->obj = STV_NewObject(wrk, wrk->storage_hint, l, nhttp);
@@ -816,10 +817,10 @@ cnt_fetchbody(struct sess *sp)
 		 * shortlived object on Transient storage.
 		 */
 		wrk->obj = STV_NewObject(wrk, TRANSIENT_STORAGE, l, nhttp);
-		if (wrk->exp.ttl > cache_param->shortlived)
-			wrk->exp.ttl = cache_param->shortlived;
-		wrk->exp.grace = 0.0;
-		wrk->exp.keep = 0.0;
+		if (wrk->busyobj->exp.ttl > cache_param->shortlived)
+			wrk->busyobj->exp.ttl = cache_param->shortlived;
+		wrk->busyobj->exp.grace = 0.0;
+		wrk->busyobj->exp.keep = 0.0;
 	}
 	if (wrk->obj == NULL) {
 		sp->err_code = 503;
@@ -859,7 +860,7 @@ cnt_fetchbody(struct sess *sp)
 	if (http_GetHdr(hp, H_Last_Modified, &b))
 		wrk->obj->last_modified = VTIM_parse(b);
 	else
-		wrk->obj->last_modified = floor(wrk->exp.entered);
+		wrk->obj->last_modified = floor(wrk->busyobj->exp.entered);
 
 	assert(WRW_IsReleased(wrk));
 
diff --git a/bin/varnishd/cache/cache_rfc2616.c b/bin/varnishd/cache/cache_rfc2616.c
index e22da50..ae9614d 100644
--- a/bin/varnishd/cache/cache_rfc2616.c
+++ b/bin/varnishd/cache/cache_rfc2616.c
@@ -69,12 +69,15 @@ RFC2616_Ttl(const struct sess *sp)
 	double h_date, h_expires;
 	char *p;
 	const struct http *hp;
+	struct exp *expp;
+
+	expp = &sp->wrk->busyobj->exp;
 
 	hp = sp->wrk->beresp;
 
-	assert(sp->wrk->exp.entered != 0.0 && !isnan(sp->wrk->exp.entered));
+	assert(expp->entered != 0.0 && !isnan(expp->entered));
 	/* If all else fails, cache using default ttl */
-	sp->wrk->exp.ttl = cache_param->default_ttl;
+	expp->ttl = cache_param->default_ttl;
 
 	max_age = age = 0;
 	h_expires = 0;
@@ -87,7 +90,7 @@ RFC2616_Ttl(const struct sess *sp)
 
 	if (http_GetHdr(hp, H_Age, &p)) {
 		age = strtoul(p, NULL, 0);
-		sp->wrk->exp.age = age;
+		expp->age = age;
 	}
 	if (http_GetHdr(hp, H_Expires, &p))
 		h_expires = VTIM_parse(p);
@@ -97,7 +100,7 @@ RFC2616_Ttl(const struct sess *sp)
 
 	switch (sp->err_code) {
 	default:
-		sp->wrk->exp.ttl = -1.;
+		expp->ttl = -1.;
 		break;
 	case 200: /* OK */
 	case 203: /* Non-Authoritative Information */
@@ -122,9 +125,9 @@ RFC2616_Ttl(const struct sess *sp)
 				max_age = strtoul(p, NULL, 0);
 
 			if (age > max_age)
-				sp->wrk->exp.ttl = 0;
+				expp->ttl = 0;
 			else
-				sp->wrk->exp.ttl = max_age - age;
+				expp->ttl = max_age - age;
 			break;
 		}
 
@@ -135,22 +138,22 @@ RFC2616_Ttl(const struct sess *sp)
 
 		/* If backend told us it is expired already, don't cache. */
 		if (h_expires < h_date) {
-			sp->wrk->exp.ttl = 0;
+			expp->ttl = 0;
 			break;
 		}
 
 		if (h_date == 0 ||
-		    fabs(h_date - sp->wrk->exp.entered) < cache_param->clock_skew) {
+		    fabs(h_date - expp->entered) < cache_param->clock_skew) {
 			/*
 			 * If we have no Date: header or if it is
 			 * sufficiently close to our clock we will
 			 * trust Expires: relative to our own clock.
 			 */
-			if (h_expires < sp->wrk->exp.entered)
-				sp->wrk->exp.ttl = 0;
+			if (h_expires < expp->entered)
+				expp->ttl = 0;
 			else
-				sp->wrk->exp.ttl = h_expires -
-				    sp->wrk->exp.entered;
+				expp->ttl = h_expires -
+				    expp->entered;
 			break;
 		} else {
 			/*
@@ -158,7 +161,7 @@ RFC2616_Ttl(const struct sess *sp)
 			 * derive a relative time from the two headers.
 			 * (the negative ttl case is caught above)
 			 */
-			sp->wrk->exp.ttl = (int)(h_expires - h_date);
+			expp->ttl = (int)(h_expires - h_date);
 		}
 
 	}
@@ -166,8 +169,8 @@ RFC2616_Ttl(const struct sess *sp)
 	/* calculated TTL, Our time, Date, Expires, max-age, age */
 	WSP(sp, SLT_TTL,
 	    "%u RFC %.0f %.0f %.0f %.0f %.0f %.0f %.0f %u",
-	    sp->xid, sp->wrk->exp.ttl, -1., -1., sp->wrk->exp.entered,
-	    sp->wrk->exp.age, h_date, h_expires, max_age);
+	    sp->xid, expp->ttl, -1., -1., expp->entered,
+	    expp->age, h_date, h_expires, max_age);
 }
 
 /*--------------------------------------------------------------------
diff --git a/bin/varnishd/cache/cache_vrt_var.c b/bin/varnishd/cache/cache_vrt_var.c
index 5951b9c..b889d69 100644
--- a/bin/varnishd/cache/cache_vrt_var.c
+++ b/bin/varnishd/cache/cache_vrt_var.c
@@ -411,12 +411,12 @@ VRT_DO_EXP(obj, sp->wrk->obj->exp, keep, 0,
    EXP_Rearm(sp->wrk->obj);
    vrt_wsp_exp(sp, sp->wrk->obj->xid, &sp->wrk->obj->exp);)
 
-VRT_DO_EXP(beresp, sp->wrk->exp, grace, 0,
-   vrt_wsp_exp(sp, sp->xid, &sp->wrk->exp);)
-VRT_DO_EXP(beresp, sp->wrk->exp, ttl, 0,
-   vrt_wsp_exp(sp, sp->xid, &sp->wrk->exp);)
-VRT_DO_EXP(beresp, sp->wrk->exp, keep, 0,
-   vrt_wsp_exp(sp, sp->xid, &sp->wrk->exp);)
+VRT_DO_EXP(beresp, sp->wrk->busyobj->exp, grace, 0,
+   vrt_wsp_exp(sp, sp->xid, &sp->wrk->busyobj->exp);)
+VRT_DO_EXP(beresp, sp->wrk->busyobj->exp, ttl, 0,
+   vrt_wsp_exp(sp, sp->xid, &sp->wrk->busyobj->exp);)
+VRT_DO_EXP(beresp, sp->wrk->busyobj->exp, keep, 0,
+   vrt_wsp_exp(sp, sp->xid, &sp->wrk->busyobj->exp);)
 
 /*--------------------------------------------------------------------
  * req.xid
diff --git a/bin/varnishd/storage/stevedore.c b/bin/varnishd/storage/stevedore.c
index 1a4b8ac..7c364c0 100644
--- a/bin/varnishd/storage/stevedore.c
+++ b/bin/varnishd/storage/stevedore.c
@@ -242,7 +242,7 @@ STV_MkObject(struct worker *wrk, void *ptr, unsigned ltot,
 
 	http_Setup(o->http, o->ws_o);
 	o->http->magic = HTTP_MAGIC;
-	o->exp = wrk->exp;
+	o->exp = wrk->busyobj->exp;
 	VTAILQ_INIT(&o->store);
 	wrk->stats.n_object++;
 
diff --git a/bin/varnishd/storage/storage_persistent.c b/bin/varnishd/storage/storage_persistent.c
index 15087b3..0791941 100644
--- a/bin/varnishd/storage/storage_persistent.c
+++ b/bin/varnishd/storage/storage_persistent.c
@@ -478,7 +478,7 @@ smp_allocobj(struct stevedore *stv, struct worker *wrk, unsigned ltot,
 		return (NULL);		/* from cnt_error */
 	CAST_OBJ_NOTNULL(sc, stv->priv, SMP_SC_MAGIC);
 	AN(wrk->objcore);
-	AN(wrk->exp.ttl > 0.);
+	AN(wrk->busyobj->exp.ttl > 0.);
 
 	ltot = IRNUP(sc, ltot);
 



From phk at varnish-cache.org  Tue Nov 29 18:22:45 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 29 Nov 2011 19:22:45 +0100
Subject: [master] 49c619b Move htc from worker to busyobj
Message-ID: <E1RVSKL-0005T6-9b@project.varnish-software.com>

commit 49c619b275799f619ea69a300e5c6b9c430da80d
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 29 18:22:33 2011 +0000

    Move htc from worker to busyobj

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 5196575..125dd53 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -315,7 +315,6 @@ struct worker {
 	/* Lookup stuff */
 	struct SHA256Context	*sha256ctx;
 
-	struct http_conn	htc[1];
 	struct ws		ws[1];
 	struct http		*bereq;
 	struct http		*beresp;
@@ -503,6 +502,7 @@ struct busyobj {
 	struct vgz		*vgz_rx;
 
 	struct exp		exp;
+	struct http_conn	htc;
 };
 
 /* Object structure --------------------------------------------------*/
diff --git a/bin/varnishd/cache/cache_esi_fetch.c b/bin/varnishd/cache/cache_esi_fetch.c
index 3dbc5ce..71dd815 100644
--- a/bin/varnishd/cache/cache_esi_fetch.c
+++ b/bin/varnishd/cache/cache_esi_fetch.c
@@ -336,7 +336,7 @@ vfp_esi_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
 	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
 	AZ(w->busyobj->fetch_failed);
 	AN(w->busyobj->vep);
-	assert(w->htc == htc);
+	assert(&w->busyobj->htc == htc);
 	if (w->busyobj->is_gzip && w->do_gunzip)
 		i = vfp_esi_bytes_gu(w, htc, bytes);
 	else if (w->busyobj->is_gunzip && w->do_gzip)
diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 2a2de5b..0b86821 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -383,10 +383,13 @@ FetchHdr(struct sess *sp)
 	struct http *hp;
 	int retry = -1;
 	int i;
+	struct http_conn *htc;
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
 	CHECK_OBJ_NOTNULL(sp->wrk, WORKER_MAGIC);
 	w = sp->wrk;
+	CHECK_OBJ_NOTNULL(w->busyobj, BUSYOBJ_MAGIC);
+	htc = &w->busyobj->htc;
 
 	AN(sp->director);
 	AZ(sp->wrk->obj);
@@ -437,12 +440,13 @@ FetchHdr(struct sess *sp)
 
 	/* Receive response */
 
-	HTC_Init(w->htc, w->ws, vc->fd, vc->vsl_id, cache_param->http_resp_size,
+	HTC_Init(htc, w->ws, vc->fd, vc->vsl_id,
+	    cache_param->http_resp_size,
 	    cache_param->http_resp_hdr_len);
 
 	VTCP_set_read_timeout(vc->fd, vc->first_byte_timeout);
 
-	i = HTC_Rx(w->htc);
+	i = HTC_Rx(htc);
 
 	if (i < 0) {
 		WSP(sp, SLT_FetchError, "http first read error: %d %d (%s)",
@@ -456,7 +460,7 @@ FetchHdr(struct sess *sp)
 	VTCP_set_read_timeout(vc->fd, vc->between_bytes_timeout);
 
 	while (i == 0) {
-		i = HTC_Rx(w->htc);
+		i = HTC_Rx(htc);
 		if (i < 0) {
 			WSP(sp, SLT_FetchError,
 			    "http first read error: %d %d (%s)",
@@ -469,7 +473,7 @@ FetchHdr(struct sess *sp)
 
 	hp = w->beresp;
 
-	if (http_DissectResponse(w, w->htc, hp)) {
+	if (http_DissectResponse(w, htc, hp)) {
 		WSP(sp, SLT_FetchError, "http format error");
 		VDI_CloseFd(sp->wrk);
 		/* XXX: other cleanup ? */
@@ -487,13 +491,17 @@ FetchBody(struct worker *w, struct object *obj)
 	struct storage *st;
 	int mklen;
 	ssize_t cl;
+	struct http_conn *htc;
 
 	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	CHECK_OBJ_NOTNULL(w->busyobj, BUSYOBJ_MAGIC);
 	AZ(w->fetch_obj);
 	CHECK_OBJ_NOTNULL(w->vbc, VBC_MAGIC);
 	CHECK_OBJ_NOTNULL(obj, OBJECT_MAGIC);
 	CHECK_OBJ_NOTNULL(obj->http, HTTP_MAGIC);
 
+	htc = &w->busyobj->htc;
+
 	if (w->busyobj->vfp == NULL)
 		w->busyobj->vfp = &vfp_nop;
 
@@ -519,21 +527,21 @@ FetchBody(struct worker *w, struct object *obj)
 	case BS_LENGTH:
 		cl = fetch_number( w->h_content_length, 10);
 		w->busyobj->vfp->begin(w, cl > 0 ? cl : 0);
-		cls = fetch_straight(w, w->htc, cl);
+		cls = fetch_straight(w, htc, cl);
 		mklen = 1;
 		if (w->busyobj->vfp->end(w))
 			cls = -1;
 		break;
 	case BS_CHUNKED:
 		w->busyobj->vfp->begin(w, cl);
-		cls = fetch_chunked(w, w->htc);
+		cls = fetch_chunked(w, htc);
 		mklen = 1;
 		if (w->busyobj->vfp->end(w))
 			cls = -1;
 		break;
 	case BS_EOF:
 		w->busyobj->vfp->begin(w, cl);
-		cls = fetch_eof(w, w->htc);
+		cls = fetch_eof(w, htc);
 		mklen = 1;
 		if (w->busyobj->vfp->end(w))
 			cls = -1;
diff --git a/bin/varnishd/storage/stevedore.c b/bin/varnishd/storage/stevedore.c
index 7c364c0..d0b4226 100644
--- a/bin/varnishd/storage/stevedore.c
+++ b/bin/varnishd/storage/stevedore.c
@@ -131,7 +131,7 @@ stv_pick_stevedore(struct worker *wrk, const char **hint)
 			return (stv_transient);
 
 		/* Hint was not valid, nuke it */
-		WSL(wrk, SLT_Debug, wrk->htc->vsl_id,
+		WSL(wrk, SLT_Debug, 0, 			/* XXX VSL_id ?? */
 		    "Storage hint not usable");
 		*hint = NULL;
 	}



From phk at varnish-cache.org  Tue Nov 29 18:30:17 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 29 Nov 2011 19:30:17 +0100
Subject: [master] 2c88943 Move fetch_object to busyobj
Message-ID: <E1RVSRd-0007VA-NT@project.varnish-software.com>

commit 2c88943245cc48241823b3848b40cfe56f8b7dcf
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 29 18:30:02 2011 +0000

    Move fetch_object to busyobj

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 125dd53..cb2a68e 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -329,7 +329,6 @@ struct worker {
 
 	/* Fetch stuff */
 	struct vbc		*vbc;
-	struct object		*fetch_obj;
 	enum body_status	body_status;
 	struct vef_priv		*vef_priv;
 	unsigned		do_stream;
@@ -501,6 +500,7 @@ struct busyobj {
 	unsigned		fetch_failed;
 	struct vgz		*vgz_rx;
 
+	struct object		*fetch_obj;
 	struct exp		exp;
 	struct http_conn	htc;
 };
diff --git a/bin/varnishd/cache/cache_esi_fetch.c b/bin/varnishd/cache/cache_esi_fetch.c
index 71dd815..5795849 100644
--- a/bin/varnishd/cache/cache_esi_fetch.c
+++ b/bin/varnishd/cache/cache_esi_fetch.c
@@ -81,7 +81,7 @@ vfp_esi_bytes_uu(struct worker *w, struct http_conn *htc, ssize_t bytes)
 			return (wl);
 		VEP_Parse(w, (const char *)st->ptr + st->len, wl);
 		st->len += wl;
-		w->fetch_obj->len += wl;
+		w->busyobj->fetch_obj->len += wl;
 		bytes -= wl;
 	}
 	return (1);
@@ -117,7 +117,7 @@ vfp_esi_bytes_gu(struct worker *w, struct http_conn *htc, ssize_t bytes)
 		i = VGZ_Gunzip(vg, &dp, &dl);
 		xxxassert(i == VGZ_OK || i == VGZ_END);
 		VEP_Parse(w, dp, dl);
-		w->fetch_obj->len += dl;
+		w->busyobj->fetch_obj->len += dl;
 	}
 	return (1);
 }
@@ -187,7 +187,7 @@ vfp_vep_callback(struct worker *w, ssize_t l, enum vgz_flag flg)
 			}
 			i = VGZ_Gzip(vef->vgz, &dp, &dl, flg);
 			vef->tot += dl;
-			w->fetch_obj->len += dl;
+			w->busyobj->fetch_obj->len += dl;
 		} while (!VGZ_IbufEmpty(vef->vgz) ||
 		    (flg != VGZ_NORMAL && VGZ_ObufFull(vef->vgz)));
 		if (px != 0) {
@@ -374,11 +374,11 @@ vfp_esi_end(struct worker *w)
 			l = VSB_len(vsb);
 			assert(l > 0);
 			/* XXX: This is a huge waste of storage... */
-			w->fetch_obj->esidata = STV_alloc(w, l);
-			if (w->fetch_obj->esidata != NULL) {
-				memcpy(w->fetch_obj->esidata->ptr,
+			w->busyobj->fetch_obj->esidata = STV_alloc(w, l);
+			if (w->busyobj->fetch_obj->esidata != NULL) {
+				memcpy(w->busyobj->fetch_obj->esidata->ptr,
 				    VSB_data(vsb), l);
-				w->fetch_obj->esidata->len = l;
+				w->busyobj->fetch_obj->esidata->len = l;
 			} else {
 				retval = FetchError(w,
 				    "Could not allocate storage for esidata");
@@ -391,7 +391,7 @@ vfp_esi_end(struct worker *w)
 		vef = w->vef_priv;
 		CHECK_OBJ_NOTNULL(vef, VEF_MAGIC);
 		w->vef_priv = NULL;
-		VGZ_UpdateObj(vef->vgz, w->fetch_obj);
+		VGZ_UpdateObj(vef->vgz, w->busyobj->fetch_obj);
 		if (VGZ_Destroy(&vef->vgz,  -1) != VGZ_END)
 			retval = FetchError(w,
 			    "ESI+Gzip Failed at the very end");
diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 0b86821..7180a5b 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -124,7 +124,7 @@ vfp_nop_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
 		if (wl <= 0)
 			return (wl);
 		st->len += wl;
-		w->fetch_obj->len += wl;
+		w->busyobj->fetch_obj->len += wl;
 		bytes -= wl;
 		if (w->do_stream)
 			RES_StreamPoll(w);
@@ -146,12 +146,12 @@ vfp_nop_end(struct worker *w)
 {
 	struct storage *st;
 
-	st = VTAILQ_LAST(&w->fetch_obj->store, storagehead);
+	st = VTAILQ_LAST(&w->busyobj->fetch_obj->store, storagehead);
 	if (st == NULL)
 		return (0);
 
 	if (st->len == 0) {
-		VTAILQ_REMOVE(&w->fetch_obj->store, st, list);
+		VTAILQ_REMOVE(&w->busyobj->fetch_obj->store, st, list);
 		STV_free(st);
 		return (0);
 	}
@@ -178,7 +178,7 @@ FetchStorage(struct worker *w, ssize_t sz)
 	struct storage *st;
 	struct object *obj;
 
-	obj = w->fetch_obj;
+	obj = w->busyobj->fetch_obj;
 	CHECK_OBJ_NOTNULL(obj, OBJECT_MAGIC);
 	st = VTAILQ_LAST(&obj->store, storagehead);
 	if (st != NULL && st->len < st->space)
@@ -495,7 +495,7 @@ FetchBody(struct worker *w, struct object *obj)
 
 	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(w->busyobj, BUSYOBJ_MAGIC);
-	AZ(w->fetch_obj);
+	AZ(w->busyobj->fetch_obj);
 	CHECK_OBJ_NOTNULL(w->vbc, VBC_MAGIC);
 	CHECK_OBJ_NOTNULL(obj, OBJECT_MAGIC);
 	CHECK_OBJ_NOTNULL(obj->http, HTTP_MAGIC);
@@ -510,7 +510,7 @@ FetchBody(struct worker *w, struct object *obj)
 	AZ(w->busyobj->vgz_rx);
 	AZ(VTAILQ_FIRST(&obj->store));
 
-	w->fetch_obj = obj;
+	w->busyobj->fetch_obj = obj;
 	w->busyobj->fetch_failed = 0;
 
 	/* XXX: pick up estimate from objdr ? */
@@ -564,7 +564,7 @@ FetchBody(struct worker *w, struct object *obj)
 	 */
 	AZ(vfp_nop_end(w));
 
-	w->fetch_obj = NULL;
+	w->busyobj->fetch_obj = NULL;
 
 	WSLB(w, SLT_Fetch_Body, "%u(%s) cls %d mklen %u",
 	    w->body_status, body_status(w->body_status),
diff --git a/bin/varnishd/cache/cache_gzip.c b/bin/varnishd/cache/cache_gzip.c
index f325292..4076081 100644
--- a/bin/varnishd/cache/cache_gzip.c
+++ b/bin/varnishd/cache/cache_gzip.c
@@ -492,7 +492,7 @@ vfp_gunzip_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
 		i = VGZ_Gunzip(vg, &dp, &dl);
 		if (i != VGZ_OK && i != VGZ_END)
 			return(FetchError(w, "Gunzip data error"));
-		w->fetch_obj->len += dl;
+		w->busyobj->fetch_obj->len += dl;
 		if (w->do_stream)
 			RES_StreamPoll(w);
 	}
@@ -568,7 +568,7 @@ vfp_gzip_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
 			return(-1);
 		i = VGZ_Gzip(vg, &dp, &dl, VGZ_NORMAL);
 		assert(i == Z_OK);
-		w->fetch_obj->len += dl;
+		w->busyobj->fetch_obj->len += dl;
 		if (w->do_stream)
 			RES_StreamPoll(w);
 	}
@@ -595,11 +595,11 @@ vfp_gzip_end(struct worker *w)
 		if (VGZ_ObufStorage(w, vg))
 			return(-1);
 		i = VGZ_Gzip(vg, &dp, &dl, VGZ_FINISH);
-		w->fetch_obj->len += dl;
+		w->busyobj->fetch_obj->len += dl;
 	} while (i != Z_STREAM_END);
 	if (w->do_stream)
 		RES_StreamPoll(w);
-	VGZ_UpdateObj(vg, w->fetch_obj);
+	VGZ_UpdateObj(vg, w->busyobj->fetch_obj);
 	if (VGZ_Destroy(&vg, -1) != VGZ_END)
 		return(FetchError(w, "Gzip error at the very end"));
 	return (0);
@@ -654,7 +654,7 @@ vfp_testgzip_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
 		bytes -= wl;
 		VGZ_Ibuf(vg, st->ptr + st->len, wl);
 		st->len += wl;
-		w->fetch_obj->len += wl;
+		w->busyobj->fetch_obj->len += wl;
 		if (w->do_stream)
 			RES_StreamPoll(w);
 
@@ -684,7 +684,7 @@ vfp_testgzip_end(struct worker *w)
 		(void)VGZ_Destroy(&vg, -1);
 		return(0);
 	}
-	VGZ_UpdateObj(vg, w->fetch_obj);
+	VGZ_UpdateObj(vg, w->busyobj->fetch_obj);
 	if (VGZ_Destroy(&vg, -1) != VGZ_END)
 		return(FetchError(w, "TestGunzip error at the very end"));
 	return (0);
diff --git a/bin/varnishd/cache/cache_response.c b/bin/varnishd/cache/cache_response.c
index bdc0fca..39c0ee6 100644
--- a/bin/varnishd/cache/cache_response.c
+++ b/bin/varnishd/cache/cache_response.c
@@ -365,14 +365,14 @@ RES_StreamPoll(struct worker *w)
 	void *ptr;
 
 	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	CHECK_OBJ_NOTNULL(w->fetch_obj, OBJECT_MAGIC);
+	CHECK_OBJ_NOTNULL(w->busyobj->fetch_obj, OBJECT_MAGIC);
 	sctx = w->sctx;
 	CHECK_OBJ_NOTNULL(sctx, STREAM_CTX_MAGIC);
-	if (w->fetch_obj->len == sctx->stream_next)
+	if (w->busyobj->fetch_obj->len == sctx->stream_next)
 		return;
-	assert(w->fetch_obj->len > sctx->stream_next);
+	assert(w->busyobj->fetch_obj->len > sctx->stream_next);
 	l = sctx->stream_front;
-	VTAILQ_FOREACH(st, &w->fetch_obj->store, list) {
+	VTAILQ_FOREACH(st, &w->busyobj->fetch_obj->store, list) {
 		if (st->len + l <= sctx->stream_next) {
 			l += st->len;
 			continue;
@@ -391,18 +391,18 @@ RES_StreamPoll(struct worker *w)
 	if (!(w->res_mode & RES_GUNZIP))
 		(void)WRW_Flush(w);
 
-	if (w->fetch_obj->objcore == NULL ||
-	    (w->fetch_obj->objcore->flags & OC_F_PASS)) {
+	if (w->busyobj->fetch_obj->objcore == NULL ||
+	    (w->busyobj->fetch_obj->objcore->flags & OC_F_PASS)) {
 		/*
 		 * This is a pass object, release storage as soon as we
 		 * have delivered it.
 		 */
 		while (1) {
-			st = VTAILQ_FIRST(&w->fetch_obj->store);
+			st = VTAILQ_FIRST(&w->busyobj->fetch_obj->store);
 			if (st == NULL ||
 			    sctx->stream_front + st->len > sctx->stream_next)
 				break;
-			VTAILQ_REMOVE(&w->fetch_obj->store, st, list);
+			VTAILQ_REMOVE(&w->busyobj->fetch_obj->store, st, list);
 			sctx->stream_front += st->len;
 			STV_free(st);
 		}
diff --git a/bin/varnishd/storage/stevedore.c b/bin/varnishd/storage/stevedore.c
index d0b4226..2748999 100644
--- a/bin/varnishd/storage/stevedore.c
+++ b/bin/varnishd/storage/stevedore.c
@@ -367,7 +367,7 @@ struct storage *
 STV_alloc(struct worker *w, size_t size)
 {
 
-	return (stv_alloc(w, w->fetch_obj, size));
+	return (stv_alloc(w, w->busyobj->fetch_obj, size));
 }
 
 void



From phk at varnish-cache.org  Tue Nov 29 19:00:22 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 29 Nov 2011 20:00:22 +0100
Subject: [master] 4faa178 Move body_status from worker to busyobj
Message-ID: <E1RVSuk-0002in-9C@project.varnish-software.com>

commit 4faa178440e6b73cc7e7924a845e1f26bf1f386e
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 29 19:00:10 2011 +0000

    Move body_status from worker to busyobj

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index cb2a68e..c4d6f08 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -316,8 +316,8 @@ struct worker {
 	struct SHA256Context	*sha256ctx;
 
 	struct ws		ws[1];
-	struct http		*bereq;
-	struct http		*beresp;
+
+
 	struct http		*resp;
 
 	struct object		*obj;
@@ -327,9 +327,11 @@ struct worker {
 	/* This is only here so VRT can find it */
 	const char		*storage_hint;
 
-	/* Fetch stuff */
+	/* Fetch stuff.  Here because pipe has no busyobj */
+	struct http		*bereq;
+	struct http		*beresp;
 	struct vbc		*vbc;
-	enum body_status	body_status;
+
 	struct vef_priv		*vef_priv;
 	unsigned		do_stream;
 	unsigned		do_esi;
@@ -503,6 +505,8 @@ struct busyobj {
 	struct object		*fetch_obj;
 	struct exp		exp;
 	struct http_conn	htc;
+
+	enum body_status	body_status;
 };
 
 /* Object structure --------------------------------------------------*/
diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c
index 86b54d0..7d4a280 100644
--- a/bin/varnishd/cache/cache_center.c
+++ b/bin/varnishd/cache/cache_center.c
@@ -606,7 +606,7 @@ cnt_fetch(struct sess *sp)
 		 * headers are adultered by VCL
 		 * NB: Also sets other wrk variables
 		 */
-		wrk->body_status = RFC2616_Body(sp);
+		wrk->busyobj->body_status = RFC2616_Body(sp);
 
 		sp->err_code = http_GetStatus(wrk->beresp);
 
diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 7180a5b..15eeb9b 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -229,7 +229,7 @@ fetch_straight(struct worker *w, struct http_conn *htc, ssize_t cl)
 {
 	int i;
 
-	assert(w->body_status == BS_LENGTH);
+	assert(w->busyobj->body_status == BS_LENGTH);
 
 	if (cl < 0) {
 		return (FetchError(w, "straight length field bogus"));
@@ -256,7 +256,7 @@ fetch_chunked(struct worker *w, struct http_conn *htc)
 	unsigned u;
 	ssize_t cl;
 
-	assert(w->body_status == BS_CHUNKED);
+	assert(w->busyobj->body_status == BS_CHUNKED);
 	do {
 		/* Skip leading whitespace */
 		do {
@@ -314,7 +314,7 @@ fetch_eof(struct worker *w, struct http_conn *htc)
 {
 	int i;
 
-	assert(w->body_status == BS_EOF);
+	assert(w->busyobj->body_status == BS_EOF);
 	i = w->busyobj->vfp->bytes(w, htc, SSIZE_MAX);
 	if (i < 0)
 		return (-1);
@@ -515,7 +515,7 @@ FetchBody(struct worker *w, struct object *obj)
 
 	/* XXX: pick up estimate from objdr ? */
 	cl = 0;
-	switch (w->body_status) {
+	switch (w->busyobj->body_status) {
 	case BS_NONE:
 		cls = 0;
 		mklen = 0;
@@ -567,10 +567,10 @@ FetchBody(struct worker *w, struct object *obj)
 	w->busyobj->fetch_obj = NULL;
 
 	WSLB(w, SLT_Fetch_Body, "%u(%s) cls %d mklen %u",
-	    w->body_status, body_status(w->body_status),
+	    w->busyobj->body_status, body_status(w->busyobj->body_status),
 	    cls, mklen);
 
-	if (w->body_status == BS_ERROR) {
+	if (w->busyobj->body_status == BS_ERROR) {
 		VDI_CloseFd(w);
 		return (__LINE__);
 	}
diff --git a/bin/varnishd/cache/cache_panic.c b/bin/varnishd/cache/cache_panic.c
index 641fe99..85fcae0 100644
--- a/bin/varnishd/cache/cache_panic.c
+++ b/bin/varnishd/cache/cache_panic.c
@@ -213,6 +213,7 @@ pan_busyobj(const struct busyobj *bo)
 	VSB_printf(pan_vsp, "  busyobj = %p {\n", bo);
 	if (bo->is_gzip)	VSB_printf(pan_vsp, "    is_gzip\n");
 	if (bo->is_gunzip)	VSB_printf(pan_vsp, "    is_gunzip\n");
+	VSB_printf(pan_vsp, "    bodystatus = %d,\n", bo->body_status);
 	VSB_printf(pan_vsp, "    },\n");
 }
 
@@ -260,7 +261,6 @@ pan_sess(const struct sess *sp)
 	if (sp->wrk->do_esi)	VSB_printf(pan_vsp, " do_esi");
 	if (sp->wrk->do_close)	VSB_printf(pan_vsp, " do_close");
 	VSB_printf(pan_vsp, "\n");
-	VSB_printf(pan_vsp, "  bodystatus = %d\n", sp->wrk->body_status);
 	pan_busyobj(sp->wrk->busyobj);
 
 	pan_ws(sp->ws, 2);



From phk at varnish-cache.org  Tue Nov 29 19:09:50 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 29 Nov 2011 20:09:50 +0100
Subject: [master] c1dac64 Move vef_priv from worker to busyobj
Message-ID: <E1RVT3u-0005Dg-Ds@project.varnish-software.com>

commit c1dac64284b5f9298d2fb42ab4f822f16f4dc1a4
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 29 19:09:39 2011 +0000

    Move vef_priv from worker to busyobj

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index c4d6f08..a0e2821 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -332,7 +332,6 @@ struct worker {
 	struct http		*beresp;
 	struct vbc		*vbc;
 
-	struct vef_priv		*vef_priv;
 	unsigned		do_stream;
 	unsigned		do_esi;
 	unsigned		do_gzip;
@@ -507,6 +506,7 @@ struct busyobj {
 	struct http_conn	htc;
 
 	enum body_status	body_status;
+	struct vef_priv		*vef_priv;
 };
 
 /* Object structure --------------------------------------------------*/
diff --git a/bin/varnishd/cache/cache_esi_fetch.c b/bin/varnishd/cache/cache_esi_fetch.c
index 5795849..e0765b6 100644
--- a/bin/varnishd/cache/cache_esi_fetch.c
+++ b/bin/varnishd/cache/cache_esi_fetch.c
@@ -150,7 +150,8 @@ vfp_vep_callback(struct worker *w, ssize_t l, enum vgz_flag flg)
 	int i;
 
 	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	vef = w->vef_priv;
+	CHECK_OBJ_NOTNULL(w->busyobj, BUSYOBJ_MAGIC);
+	vef = w->busyobj->vef_priv;
 	CHECK_OBJ_NOTNULL(vef, VEF_MAGIC);
 	assert(l >= 0);
 
@@ -211,7 +212,8 @@ vfp_esi_bytes_ug(struct worker *w, struct http_conn *htc, ssize_t bytes)
 	struct vef_priv *vef;
 
 	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	vef = w->vef_priv;
+	CHECK_OBJ_NOTNULL(w->busyobj, BUSYOBJ_MAGIC);
+	vef = w->busyobj->vef_priv;
 	CHECK_OBJ_NOTNULL(vef, VEF_MAGIC);
 
 	while (bytes > 0) {
@@ -252,7 +254,8 @@ vfp_esi_bytes_gg(struct worker *w, struct http_conn *htc, size_t bytes)
 	int i;
 
 	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
-	vef = w->vef_priv;
+	CHECK_OBJ_NOTNULL(w->busyobj, BUSYOBJ_MAGIC);
+	vef = w->busyobj->vef_priv;
 	CHECK_OBJ_NOTNULL(vef, VEF_MAGIC);
 	assert(sizeof ibuf >= 1024);
 	ibuf2[0] = 0; /* For Flexelint */
@@ -308,19 +311,19 @@ vfp_esi_begin(struct worker *w, size_t estimate)
 		ALLOC_OBJ(vef, VEF_MAGIC);
 		AN(vef);
 		vef->vgz = VGZ_NewGzip(w, "G F E");
-		AZ(w->vef_priv);
-		w->vef_priv = vef;
+		AZ(w->busyobj->vef_priv);
+		w->busyobj->vef_priv = vef;
 		VEP_Init(w, vfp_vep_callback);
 	} else if (w->busyobj->is_gzip) {
 		w->busyobj->vgz_rx = VGZ_NewUngzip(w, "U F E");
 		ALLOC_OBJ(vef, VEF_MAGIC);
 		AN(vef);
 		vef->vgz = VGZ_NewGzip(w, "G F E");
-		AZ(w->vef_priv);
-		w->vef_priv = vef;
+		AZ(w->busyobj->vef_priv);
+		w->busyobj->vef_priv = vef;
 		VEP_Init(w, vfp_vep_callback);
 	} else {
-		AZ(w->vef_priv);
+		AZ(w->busyobj->vef_priv);
 		VEP_Init(w, NULL);
 	}
 
@@ -358,6 +361,7 @@ vfp_esi_end(struct worker *w)
 	int retval;
 
 	CHECK_OBJ_NOTNULL(w, WORKER_MAGIC);
+	CHECK_OBJ_NOTNULL(w->busyobj, BUSYOBJ_MAGIC);
 	AN(w->busyobj->vep);
 
 	retval = w->busyobj->fetch_failed;
@@ -387,10 +391,10 @@ vfp_esi_end(struct worker *w)
 		VSB_delete(vsb);
 	}
 
-	if (w->vef_priv != NULL) {
-		vef = w->vef_priv;
+	vef = w->busyobj->vef_priv;
+	if (vef != NULL) {
 		CHECK_OBJ_NOTNULL(vef, VEF_MAGIC);
-		w->vef_priv = NULL;
+		w->busyobj->vef_priv = NULL;
 		VGZ_UpdateObj(vef->vgz, w->busyobj->fetch_obj);
 		if (VGZ_Destroy(&vef->vgz,  -1) != VGZ_END)
 			retval = FetchError(w,



From apj at varnish-cache.org  Tue Nov 29 19:17:16 2011
From: apj at varnish-cache.org (Andreas Plesner Jacobsen)
Date: Tue, 29 Nov 2011 20:17:16 +0100
Subject: [master] 84628ca Be more precise about default behaviour in the
	presence of cookie headers
Message-ID: <E1RVTB6-0000AP-3u@project.varnish-software.com>

commit 84628cad04d8cc2ae131e42b6973fda4140ba2fa
Author: Andreas Plesner Jacobsen <apj at mutt.dk>
Date:   Tue Nov 29 20:16:03 2011 +0100

    Be more precise about default behaviour in the presence of cookie headers

diff --git a/doc/sphinx/tutorial/cookies.rst b/doc/sphinx/tutorial/cookies.rst
index 1d6f291..2ee14cd 100644
--- a/doc/sphinx/tutorial/cookies.rst
+++ b/doc/sphinx/tutorial/cookies.rst
@@ -3,9 +3,10 @@
 Cookies
 -------
 
-Varnish will not cache a object coming from the backend with a
-Set-Cookie header present. Also, if the client sends a Cookie header,
-Varnish will bypass the cache and go directly to the backend.
+Varnish will, in the default configuration, not cache a object coming
+from the backend with a Set-Cookie header present. Also, if the client
+sends a Cookie header, Varnish will bypass the cache and go directly to
+the backend.
 
 This can be overly conservative. A lot of sites use Google Analytics
 (GA) to analyze their traffic. GA sets a cookie to track you. This



From phk at varnish-cache.org  Tue Nov 29 21:11:13 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 29 Nov 2011 22:11:13 +0100
Subject: [master] 7813b33 Compile fix.
Message-ID: <E1RVUxN-00010R-GS@project.varnish-software.com>

commit 7813b33f99ee139e0ef93766e928af19da2a2cac
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 29 21:05:23 2011 +0000

    Compile fix.

diff --git a/bin/varnishd/waiter/cache_waiter_epoll.c b/bin/varnishd/waiter/cache_waiter_epoll.c
index 4d45909..06cd8bb 100644
--- a/bin/varnishd/waiter/cache_waiter_epoll.c
+++ b/bin/varnishd/waiter/cache_waiter_epoll.c
@@ -119,7 +119,6 @@ vwe_eev(struct vwe *vwe, const struct epoll_event *ep)
 			while (i >= sizeof ss[0]) {
 				CHECK_OBJ_NOTNULL(ss[j], SESS_MAGIC);
 				assert(ss[j]->fd >= 0);
-				AZ(ss[j]->obj);
 				VTAILQ_INSERT_TAIL(&vwe->sesshead, ss[j], list);
 				vwe_cond_modadd(vwe, ss[j]->fd, ss[j]);
 				j++;



From phk at varnish-cache.org  Tue Nov 29 21:11:15 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 29 Nov 2011 22:11:15 +0100
Subject: [master] bd4d340 Merge branch 'master' of
	ssh://git.varnish-cache.org/git/varnish-cache
Message-ID: <E1RVUxP-00010X-DI@project.varnish-software.com>

commit bd4d3401adefe9f37968b955fec23cad0a697107
Merge: 7813b33 84628ca
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Nov 29 21:11:05 2011 +0000

    Merge branch 'master' of ssh://git.varnish-cache.org/git/varnish-cache



From phk at varnish-cache.org  Wed Nov 30 06:50:13 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 30 Nov 2011 07:50:13 +0100
Subject: [master] 22c90f1 Move do_gzip,
	do_gunzip and do_esi from worker to busyobj
Message-ID: <E1RVdzh-0006Nb-AO@project.varnish-software.com>

commit 22c90f14b4c95c6691a00815d76096056465e6af
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Nov 30 06:49:49 2011 +0000

    Move do_gzip, do_gunzip and do_esi from worker to busyobj

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index a0e2821..9a4e5f3 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -333,9 +333,6 @@ struct worker {
 	struct vbc		*vbc;
 
 	unsigned		do_stream;
-	unsigned		do_esi;
-	unsigned		do_gzip;
-	unsigned		do_gunzip;
 	unsigned		do_close;
 	char			*h_content_length;
 
@@ -507,6 +504,10 @@ struct busyobj {
 
 	enum body_status	body_status;
 	struct vef_priv		*vef_priv;
+
+	unsigned		do_esi;
+	unsigned		do_gzip;
+	unsigned		do_gunzip;
 };
 
 /* Object structure --------------------------------------------------*/
diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c
index 7d4a280..6acd218 100644
--- a/bin/varnishd/cache/cache_center.c
+++ b/bin/varnishd/cache/cache_center.c
@@ -184,8 +184,12 @@ cnt_prepresp(struct sess *sp)
 
 	wrk->res_mode = 0;
 
-	if ((wrk->h_content_length != NULL || !wrk->do_stream) &&
-	    !wrk->do_gzip && !wrk->do_gunzip)
+	if (wrk->busyobj == NULL)
+		wrk->res_mode |= RES_LEN;
+
+	if (wrk->busyobj != NULL &&
+	    (wrk->h_content_length != NULL || !wrk->do_stream) &&
+	    !wrk->busyobj->do_gzip && !wrk->busyobj->do_gunzip)
 		wrk->res_mode |= RES_LEN;
 
 	if (!sp->disable_esi && wrk->obj->esidata != NULL) {
@@ -339,9 +343,6 @@ cnt_done(struct sess *sp)
 
 	wrk->busyobj = NULL;
 
-	wrk->do_esi = 0;
-	wrk->do_gunzip = 0;
-	wrk->do_gzip = 0;
 	wrk->do_stream = 0;
 
 	SES_Charge(sp);
@@ -457,9 +458,6 @@ cnt_error(struct sess *sp)
 	wrk = sp->wrk;
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 
-	wrk->do_esi = 0;
-	wrk->do_gzip = 0;
-	wrk->do_gunzip = 0;
 	wrk->do_stream = 0;
 
 	if (wrk->obj == NULL) {
@@ -621,7 +619,7 @@ cnt_fetch(struct sess *sp)
 		if (wrk->objcore == NULL)
 			wrk->busyobj->exp.ttl = -1.;
 
-		AZ(wrk->do_esi);
+		AZ(wrk->busyobj->do_esi);
 
 		VCL_fetch_method(sp);
 
@@ -740,7 +738,7 @@ cnt_fetchbody(struct sess *sp)
 
 	/* We do nothing unless the param is set */
 	if (!cache_param->http_gzip_support)
-		wrk->do_gzip = wrk->do_gunzip = 0;
+		wrk->busyobj->do_gzip = wrk->busyobj->do_gunzip = 0;
 
 	wrk->busyobj->is_gzip =
 	    http_HdrIs(wrk->beresp, H_Content_Encoding, "gzip");
@@ -752,36 +750,36 @@ cnt_fetchbody(struct sess *sp)
 	assert(wrk->busyobj->is_gzip == 0 || wrk->busyobj->is_gunzip == 0);
 
 	/* We won't gunzip unless it is gzip'ed */
-	if (wrk->do_gunzip && !wrk->busyobj->is_gzip)
-		wrk->do_gunzip = 0;
+	if (wrk->busyobj->do_gunzip && !wrk->busyobj->is_gzip)
+		wrk->busyobj->do_gunzip = 0;
 
 	/* If we do gunzip, remove the C-E header */
-	if (wrk->do_gunzip)
+	if (wrk->busyobj->do_gunzip)
 		http_Unset(wrk->beresp, H_Content_Encoding);
 
 	/* We wont gzip unless it is ungziped */
-	if (wrk->do_gzip && !wrk->busyobj->is_gunzip)
-		wrk->do_gzip = 0;
+	if (wrk->busyobj->do_gzip && !wrk->busyobj->is_gunzip)
+		wrk->busyobj->do_gzip = 0;
 
 	/* If we do gzip, add the C-E header */
-	if (wrk->do_gzip)
+	if (wrk->busyobj->do_gzip)
 		http_SetHeader(wrk, sp->vsl_id, wrk->beresp,
 		    "Content-Encoding: gzip");
 
 	/* But we can't do both at the same time */
-	assert(wrk->do_gzip == 0 || wrk->do_gunzip == 0);
+	assert(wrk->busyobj->do_gzip == 0 || wrk->busyobj->do_gunzip == 0);
 
 	/* ESI takes precedence and handles gzip/gunzip itself */
-	if (wrk->do_esi)
+	if (wrk->busyobj->do_esi)
 		wrk->busyobj->vfp = &vfp_esi;
-	else if (wrk->do_gunzip)
+	else if (wrk->busyobj->do_gunzip)
 		wrk->busyobj->vfp = &vfp_gunzip;
-	else if (wrk->do_gzip)
+	else if (wrk->busyobj->do_gzip)
 		wrk->busyobj->vfp = &vfp_gzip;
 	else if (wrk->busyobj->is_gzip)
 		wrk->busyobj->vfp = &vfp_testgzip;
 
-	if (wrk->do_esi || sp->esi_level > 0)
+	if (wrk->busyobj->do_esi || sp->esi_level > 0)
 		wrk->do_stream = 0;
 	if (!sp->wantbody)
 		wrk->do_stream = 0;
@@ -832,7 +830,8 @@ cnt_fetchbody(struct sess *sp)
 
 	wrk->storage_hint = NULL;
 
-	if (wrk->do_gzip || (wrk->busyobj->is_gzip && !wrk->do_gunzip))
+	if (wrk->busyobj->do_gzip ||
+	    (wrk->busyobj->is_gzip && !wrk->busyobj->do_gunzip))
 		wrk->obj->gziped = 1;
 
 	if (vary != NULL) {
@@ -1470,9 +1469,6 @@ cnt_recv(struct sess *sp)
 	}
 
 	/* Zap these, in case we came here through restart */
-	wrk->do_esi = 0;
-	wrk->do_gzip = 0;
-	wrk->do_gunzip = 0;
 	wrk->do_stream = 0;
 
 	if (cache_param->http_gzip_support &&
@@ -1656,9 +1652,6 @@ CNT_Session(struct sess *sp)
 	    sp->step == STP_RECV);
 
 	AZ(wrk->do_stream);
-	AZ(wrk->do_gzip);
-	AZ(wrk->do_gunzip);
-	AZ(wrk->do_esi);
 	AZ(wrk->obj);
 	AZ(wrk->objcore);
 
@@ -1713,9 +1706,6 @@ CNT_Session(struct sess *sp)
 	AZ(wrk->obj);
 	AZ(wrk->objcore);
 	AZ(wrk->do_stream);
-	AZ(wrk->do_gzip);
-	AZ(wrk->do_gunzip);
-	AZ(wrk->do_esi);
 #define ACCT(foo)	AZ(wrk->acct_tmp.foo);
 #include "tbl/acct_fields.h"
 #undef ACCT
diff --git a/bin/varnishd/cache/cache_esi_deliver.c b/bin/varnishd/cache/cache_esi_deliver.c
index 6c8a16f..518c5a9 100644
--- a/bin/varnishd/cache/cache_esi_deliver.c
+++ b/bin/varnishd/cache/cache_esi_deliver.c
@@ -91,9 +91,6 @@ ved_include(struct sess *sp, const char *src, const char *host)
 	/* Client content already taken care of */
 	http_Unset(sp->http, H_Content_Length);
 
-	sp->wrk->do_esi = 0;
-	sp->wrk->do_gzip = 0;
-	sp->wrk->do_gunzip = 0;
 	sp->wrk->do_stream = 0;
 
 	sxid = sp->xid;
diff --git a/bin/varnishd/cache/cache_esi_fetch.c b/bin/varnishd/cache/cache_esi_fetch.c
index e0765b6..aea28ed 100644
--- a/bin/varnishd/cache/cache_esi_fetch.c
+++ b/bin/varnishd/cache/cache_esi_fetch.c
@@ -304,10 +304,10 @@ vfp_esi_begin(struct worker *w, size_t estimate)
 	CHECK_OBJ_NOTNULL(w->busyobj, BUSYOBJ_MAGIC);
 
 	AZ(w->busyobj->vgz_rx);
-	if (w->busyobj->is_gzip && w->do_gunzip) {
+	if (w->busyobj->is_gzip && w->busyobj->do_gunzip) {
 		w->busyobj->vgz_rx = VGZ_NewUngzip(w, "U F E");
 		VEP_Init(w, NULL);
-	} else if (w->busyobj->is_gunzip && w->do_gzip) {
+	} else if (w->busyobj->is_gunzip && w->busyobj->do_gzip) {
 		ALLOC_OBJ(vef, VEF_MAGIC);
 		AN(vef);
 		vef->vgz = VGZ_NewGzip(w, "G F E");
@@ -340,9 +340,9 @@ vfp_esi_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
 	AZ(w->busyobj->fetch_failed);
 	AN(w->busyobj->vep);
 	assert(&w->busyobj->htc == htc);
-	if (w->busyobj->is_gzip && w->do_gunzip)
+	if (w->busyobj->is_gzip && w->busyobj->do_gunzip)
 		i = vfp_esi_bytes_gu(w, htc, bytes);
-	else if (w->busyobj->is_gunzip && w->do_gzip)
+	else if (w->busyobj->is_gunzip && w->busyobj->do_gzip)
 		i = vfp_esi_bytes_ug(w, htc, bytes);
 	else if (w->busyobj->is_gzip)
 		i = vfp_esi_bytes_gg(w, htc, bytes);
diff --git a/bin/varnishd/cache/cache_panic.c b/bin/varnishd/cache/cache_panic.c
index 85fcae0..718a686 100644
--- a/bin/varnishd/cache/cache_panic.c
+++ b/bin/varnishd/cache/cache_panic.c
@@ -213,6 +213,9 @@ pan_busyobj(const struct busyobj *bo)
 	VSB_printf(pan_vsp, "  busyobj = %p {\n", bo);
 	if (bo->is_gzip)	VSB_printf(pan_vsp, "    is_gzip\n");
 	if (bo->is_gunzip)	VSB_printf(pan_vsp, "    is_gunzip\n");
+	if (bo->do_gzip)	VSB_printf(pan_vsp, "    do_gzip\n");
+	if (bo->do_gunzip)	VSB_printf(pan_vsp, "    do_gunzip\n");
+	if (bo->do_esi)		VSB_printf(pan_vsp, "    do_esi\n");
 	VSB_printf(pan_vsp, "    bodystatus = %d,\n", bo->body_status);
 	VSB_printf(pan_vsp, "    },\n");
 }
@@ -256,9 +259,6 @@ pan_sess(const struct sess *sp)
 
 	VSB_printf(pan_vsp, "  flags = ");
 	if (sp->wrk->do_stream)	VSB_printf(pan_vsp, " do_stream");
-	if (sp->wrk->do_gzip)	VSB_printf(pan_vsp, " do_gzip");
-	if (sp->wrk->do_gunzip)	VSB_printf(pan_vsp, " do_gunzip");
-	if (sp->wrk->do_esi)	VSB_printf(pan_vsp, " do_esi");
 	if (sp->wrk->do_close)	VSB_printf(pan_vsp, " do_close");
 	VSB_printf(pan_vsp, "\n");
 	pan_busyobj(sp->wrk->busyobj);
diff --git a/bin/varnishd/cache/cache_vrt_var.c b/bin/varnishd/cache/cache_vrt_var.c
index b889d69..adf1017 100644
--- a/bin/varnishd/cache/cache_vrt_var.c
+++ b/bin/varnishd/cache/cache_vrt_var.c
@@ -194,9 +194,9 @@ VRT_r_##dir##_##onm(const struct sess *sp)				\
 	return (sp->wrk->field);					\
 }
 
-VBERESP(beresp, unsigned, do_esi, do_esi)
-VBERESP(beresp, unsigned, do_gzip, do_gzip)
-VBERESP(beresp, unsigned, do_gunzip, do_gunzip)
+VBERESP(beresp, unsigned, do_esi, busyobj->do_esi)
+VBERESP(beresp, unsigned, do_gzip, busyobj->do_gzip)
+VBERESP(beresp, unsigned, do_gunzip, busyobj->do_gunzip)
 VBERESP(beresp, unsigned, do_stream, do_stream)
 
 /*--------------------------------------------------------------------*/



From phk at varnish-cache.org  Wed Nov 30 07:10:47 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 30 Nov 2011 08:10:47 +0100
Subject: [master] 69f785e move do_stream from worker to busyobj
Message-ID: <E1RVeJb-0008P5-H4@project.varnish-software.com>

commit 69f785edcf843f5386444cad4d3a94c9b82ac303
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Nov 30 07:10:28 2011 +0000

    move do_stream from worker to busyobj

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 9a4e5f3..8491150 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -332,7 +332,6 @@ struct worker {
 	struct http		*beresp;
 	struct vbc		*vbc;
 
-	unsigned		do_stream;
 	unsigned		do_close;
 	char			*h_content_length;
 
@@ -508,6 +507,7 @@ struct busyobj {
 	unsigned		do_esi;
 	unsigned		do_gzip;
 	unsigned		do_gunzip;
+	unsigned		do_stream;
 };
 
 /* Object structure --------------------------------------------------*/
diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c
index 6acd218..a8138da 100644
--- a/bin/varnishd/cache/cache_center.c
+++ b/bin/varnishd/cache/cache_center.c
@@ -179,7 +179,7 @@ cnt_prepresp(struct sess *sp)
 	CHECK_OBJ_NOTNULL(wrk->obj, OBJECT_MAGIC);
 	CHECK_OBJ_NOTNULL(sp->vcl, VCL_CONF_MAGIC);
 
-	if (wrk->do_stream)
+	if (wrk->busyobj != NULL && wrk->busyobj->do_stream)
 		AssertObjCorePassOrBusy(wrk->obj->objcore);
 
 	wrk->res_mode = 0;
@@ -188,7 +188,7 @@ cnt_prepresp(struct sess *sp)
 		wrk->res_mode |= RES_LEN;
 
 	if (wrk->busyobj != NULL &&
-	    (wrk->h_content_length != NULL || !wrk->do_stream) &&
+	    (wrk->h_content_length != NULL || !wrk->busyobj->do_stream) &&
 	    !wrk->busyobj->do_gzip && !wrk->busyobj->do_gunzip)
 		wrk->res_mode |= RES_LEN;
 
@@ -214,7 +214,8 @@ cnt_prepresp(struct sess *sp)
 	}
 
 	if (!(wrk->res_mode & (RES_LEN|RES_CHUNKED|RES_EOF))) {
-		if (wrk->obj->len == 0 && !wrk->do_stream)
+		if (wrk->obj->len == 0 &&
+		    (wrk->busyobj == NULL || !wrk->busyobj->do_stream))
 			/*
 			 * If the object is empty, neither ESI nor GUNZIP
 			 * can make it any different size
@@ -246,7 +247,7 @@ cnt_prepresp(struct sess *sp)
 	case VCL_RET_RESTART:
 		if (sp->restarts >= cache_param->max_restarts)
 			break;
-		if (wrk->do_stream) {
+		if (wrk->busyobj->do_stream) {
 			VDI_CloseFd(wrk);
 			HSH_Drop(wrk);
 		} else {
@@ -264,7 +265,7 @@ cnt_prepresp(struct sess *sp)
 	default:
 		WRONG("Illegal action in vcl_deliver{}");
 	}
-	if (wrk->do_stream) {
+	if (wrk->busyobj != NULL && wrk->busyobj->do_stream) {
 		AssertObjCorePassOrBusy(wrk->obj->objcore);
 		sp->step = STP_STREAMBODY;
 	} else {
@@ -343,8 +344,6 @@ cnt_done(struct sess *sp)
 
 	wrk->busyobj = NULL;
 
-	wrk->do_stream = 0;
-
 	SES_Charge(sp);
 
 	/* If we did an ESI include, don't mess up our state */
@@ -458,8 +457,6 @@ cnt_error(struct sess *sp)
 	wrk = sp->wrk;
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 
-	wrk->do_stream = 0;
-
 	if (wrk->obj == NULL) {
 		HSH_Prealloc(sp);
 		New_BusyObj(wrk);
@@ -780,9 +777,9 @@ cnt_fetchbody(struct sess *sp)
 		wrk->busyobj->vfp = &vfp_testgzip;
 
 	if (wrk->busyobj->do_esi || sp->esi_level > 0)
-		wrk->do_stream = 0;
+		wrk->busyobj->do_stream = 0;
 	if (!sp->wantbody)
-		wrk->do_stream = 0;
+		wrk->busyobj->do_stream = 0;
 
 	l = http_EstimateWS(wrk->beresp,
 	    pass ? HTTPH_R_PASS : HTTPH_A_INS, &nhttp);
@@ -871,11 +868,11 @@ cnt_fetchbody(struct sess *sp)
 	if (wrk->obj->response == 200 &&
 	    sp->http->conds &&
 	    RFC2616_Do_Cond(sp))
-		wrk->do_stream = 0;
+		wrk->busyobj->do_stream = 0;
 
 	AssertObjCorePassOrBusy(wrk->obj->objcore);
 
-	if (wrk->do_stream) {
+	if (wrk->busyobj->do_stream) {
 		sp->step = STP_PREPRESP;
 		return (0);
 	}
@@ -1062,8 +1059,6 @@ cnt_hit(struct sess *sp)
 
 	assert(!(wrk->obj->objcore->flags & OC_F_PASS));
 
-	AZ(wrk->do_stream);
-
 	VCL_hit_method(sp);
 
 	if (sp->handling == VCL_RET_DELIVER) {
@@ -1468,9 +1463,6 @@ cnt_recv(struct sess *sp)
 		return (0);
 	}
 
-	/* Zap these, in case we came here through restart */
-	wrk->do_stream = 0;
-
 	if (cache_param->http_gzip_support &&
 	     (recv_handling != VCL_RET_PIPE) &&
 	     (recv_handling != VCL_RET_PASS)) {
@@ -1651,7 +1643,6 @@ CNT_Session(struct sess *sp)
 	    sp->step == STP_LOOKUP ||
 	    sp->step == STP_RECV);
 
-	AZ(wrk->do_stream);
 	AZ(wrk->obj);
 	AZ(wrk->objcore);
 
@@ -1705,7 +1696,6 @@ CNT_Session(struct sess *sp)
 	WSL_Flush(wrk, 0);
 	AZ(wrk->obj);
 	AZ(wrk->objcore);
-	AZ(wrk->do_stream);
 #define ACCT(foo)	AZ(wrk->acct_tmp.foo);
 #include "tbl/acct_fields.h"
 #undef ACCT
diff --git a/bin/varnishd/cache/cache_esi_deliver.c b/bin/varnishd/cache/cache_esi_deliver.c
index 518c5a9..0d6b74c 100644
--- a/bin/varnishd/cache/cache_esi_deliver.c
+++ b/bin/varnishd/cache/cache_esi_deliver.c
@@ -91,8 +91,6 @@ ved_include(struct sess *sp, const char *src, const char *host)
 	/* Client content already taken care of */
 	http_Unset(sp->http, H_Content_Length);
 
-	sp->wrk->do_stream = 0;
-
 	sxid = sp->xid;
 	while (1) {
 		sp->wrk = w;
diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 15eeb9b..a198650 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -126,7 +126,7 @@ vfp_nop_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
 		st->len += wl;
 		w->busyobj->fetch_obj->len += wl;
 		bytes -= wl;
-		if (w->do_stream)
+		if (w->busyobj->do_stream)
 			RES_StreamPoll(w);
 	}
 	return (1);
@@ -601,7 +601,7 @@ FetchBody(struct worker *w, struct object *obj)
 		uu = 0;
 		VTAILQ_FOREACH(st, &obj->store, list)
 			uu += st->len;
-		if (w->do_stream)
+		if (w->busyobj->do_stream)
 			/* Streaming might have started freeing stuff */
 			assert (uu <= obj->len);
 
diff --git a/bin/varnishd/cache/cache_gzip.c b/bin/varnishd/cache/cache_gzip.c
index 4076081..cf07e1d 100644
--- a/bin/varnishd/cache/cache_gzip.c
+++ b/bin/varnishd/cache/cache_gzip.c
@@ -493,7 +493,7 @@ vfp_gunzip_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
 		if (i != VGZ_OK && i != VGZ_END)
 			return(FetchError(w, "Gunzip data error"));
 		w->busyobj->fetch_obj->len += dl;
-		if (w->do_stream)
+		if (w->busyobj->do_stream)
 			RES_StreamPoll(w);
 	}
 	assert(i == Z_OK || i == Z_STREAM_END);
@@ -569,7 +569,7 @@ vfp_gzip_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
 		i = VGZ_Gzip(vg, &dp, &dl, VGZ_NORMAL);
 		assert(i == Z_OK);
 		w->busyobj->fetch_obj->len += dl;
-		if (w->do_stream)
+		if (w->busyobj->do_stream)
 			RES_StreamPoll(w);
 	}
 	return (1);
@@ -597,7 +597,7 @@ vfp_gzip_end(struct worker *w)
 		i = VGZ_Gzip(vg, &dp, &dl, VGZ_FINISH);
 		w->busyobj->fetch_obj->len += dl;
 	} while (i != Z_STREAM_END);
-	if (w->do_stream)
+	if (w->busyobj->do_stream)
 		RES_StreamPoll(w);
 	VGZ_UpdateObj(vg, w->busyobj->fetch_obj);
 	if (VGZ_Destroy(&vg, -1) != VGZ_END)
@@ -655,7 +655,7 @@ vfp_testgzip_bytes(struct worker *w, struct http_conn *htc, ssize_t bytes)
 		VGZ_Ibuf(vg, st->ptr + st->len, wl);
 		st->len += wl;
 		w->busyobj->fetch_obj->len += wl;
-		if (w->do_stream)
+		if (w->busyobj->do_stream)
 			RES_StreamPoll(w);
 
 		while (!VGZ_IbufEmpty(vg)) {
diff --git a/bin/varnishd/cache/cache_panic.c b/bin/varnishd/cache/cache_panic.c
index 718a686..58fb8e4 100644
--- a/bin/varnishd/cache/cache_panic.c
+++ b/bin/varnishd/cache/cache_panic.c
@@ -216,6 +216,7 @@ pan_busyobj(const struct busyobj *bo)
 	if (bo->do_gzip)	VSB_printf(pan_vsp, "    do_gzip\n");
 	if (bo->do_gunzip)	VSB_printf(pan_vsp, "    do_gunzip\n");
 	if (bo->do_esi)		VSB_printf(pan_vsp, "    do_esi\n");
+	if (bo->do_stream)	VSB_printf(pan_vsp, "    do_stream\n");
 	VSB_printf(pan_vsp, "    bodystatus = %d,\n", bo->body_status);
 	VSB_printf(pan_vsp, "    },\n");
 }
@@ -258,7 +259,6 @@ pan_sess(const struct sess *sp)
 	    sp->restarts, sp->esi_level);
 
 	VSB_printf(pan_vsp, "  flags = ");
-	if (sp->wrk->do_stream)	VSB_printf(pan_vsp, " do_stream");
 	if (sp->wrk->do_close)	VSB_printf(pan_vsp, " do_close");
 	VSB_printf(pan_vsp, "\n");
 	pan_busyobj(sp->wrk->busyobj);
diff --git a/bin/varnishd/cache/cache_vrt_var.c b/bin/varnishd/cache/cache_vrt_var.c
index adf1017..0342de8 100644
--- a/bin/varnishd/cache/cache_vrt_var.c
+++ b/bin/varnishd/cache/cache_vrt_var.c
@@ -197,7 +197,7 @@ VRT_r_##dir##_##onm(const struct sess *sp)				\
 VBERESP(beresp, unsigned, do_esi, busyobj->do_esi)
 VBERESP(beresp, unsigned, do_gzip, busyobj->do_gzip)
 VBERESP(beresp, unsigned, do_gunzip, busyobj->do_gunzip)
-VBERESP(beresp, unsigned, do_stream, do_stream)
+VBERESP(beresp, unsigned, do_stream, busyobj->do_stream)
 
 /*--------------------------------------------------------------------*/
 



From phk at varnish-cache.org  Wed Nov 30 07:34:14 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 30 Nov 2011 08:34:14 +0100
Subject: [master] 0c18d87 Move do_close from worker to busyobj and rename it
	should_close to distinguish it from the electables (do_esi,
	do_gzip etc).
Message-ID: <E1RVegI-00058X-Dv@project.varnish-software.com>

commit 0c18d87646410a2468e97fb231c10a524c3586ef
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Nov 30 07:33:47 2011 +0000

    Move do_close from worker to busyobj and rename it should_close to
    distinguish it from the electables (do_esi, do_gzip etc).

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 8491150..819e701 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -332,7 +332,6 @@ struct worker {
 	struct http		*beresp;
 	struct vbc		*vbc;
 
-	unsigned		do_close;
 	char			*h_content_length;
 
 	/* Stream state */
@@ -504,6 +503,8 @@ struct busyobj {
 	enum body_status	body_status;
 	struct vef_priv		*vef_priv;
 
+	unsigned		should_close;
+
 	unsigned		do_esi;
 	unsigned		do_gzip;
 	unsigned		do_gunzip;
diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c
index a8138da..6bda81f 100644
--- a/bin/varnishd/cache/cache_center.c
+++ b/bin/varnishd/cache/cache_center.c
@@ -568,7 +568,7 @@ cnt_fetch(struct sess *sp)
 	AN(sp->director);
 	AZ(wrk->vbc);
 	AZ(wrk->h_content_length);
-	AZ(wrk->do_close);
+	AZ(wrk->busyobj->should_close);
 	AZ(wrk->storage_hint);
 
 	http_Setup(wrk->beresp, wrk->ws);
diff --git a/bin/varnishd/cache/cache_dir.c b/bin/varnishd/cache/cache_dir.c
index d4794f9..eba6183 100644
--- a/bin/varnishd/cache/cache_dir.c
+++ b/bin/varnishd/cache/cache_dir.c
@@ -61,7 +61,6 @@ VDI_CloseFd(struct worker *wrk)
 	wrk->vbc->backend = NULL;
 	VBE_ReleaseConn(wrk->vbc);
 	wrk->vbc = NULL;
-	wrk->do_close = 0;
 }
 
 /* Recycle a connection ----------------------------------------------*/
@@ -74,7 +73,6 @@ VDI_RecycleFd(struct worker *wrk)
 	CHECK_OBJ_NOTNULL(wrk->vbc, VBC_MAGIC);
 	CHECK_OBJ_NOTNULL(wrk->vbc->backend, BACKEND_MAGIC);
 	assert(wrk->vbc->fd >= 0);
-	AZ(wrk->do_close);
 
 	bp = wrk->vbc->backend;
 
diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index a198650..16509f0 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -589,7 +589,7 @@ FetchBody(struct worker *w, struct object *obj)
 	}
 	AZ(w->busyobj->fetch_failed);
 
-	if (cls == 0 && w->do_close)
+	if (cls == 0 && w->busyobj->should_close)
 		cls = 1;
 
 	WSLB(w, SLT_Length, "%u", obj->len);
diff --git a/bin/varnishd/cache/cache_panic.c b/bin/varnishd/cache/cache_panic.c
index 58fb8e4..0908b74 100644
--- a/bin/varnishd/cache/cache_panic.c
+++ b/bin/varnishd/cache/cache_panic.c
@@ -217,6 +217,7 @@ pan_busyobj(const struct busyobj *bo)
 	if (bo->do_gunzip)	VSB_printf(pan_vsp, "    do_gunzip\n");
 	if (bo->do_esi)		VSB_printf(pan_vsp, "    do_esi\n");
 	if (bo->do_stream)	VSB_printf(pan_vsp, "    do_stream\n");
+	if (bo->should_close)	VSB_printf(pan_vsp, "    should_close\n");
 	VSB_printf(pan_vsp, "    bodystatus = %d,\n", bo->body_status);
 	VSB_printf(pan_vsp, "    },\n");
 }
@@ -258,9 +259,6 @@ pan_sess(const struct sess *sp)
 	VSB_printf(pan_vsp, "  restarts = %d, esi_level = %d\n",
 	    sp->restarts, sp->esi_level);
 
-	VSB_printf(pan_vsp, "  flags = ");
-	if (sp->wrk->do_close)	VSB_printf(pan_vsp, " do_close");
-	VSB_printf(pan_vsp, "\n");
 	pan_busyobj(sp->wrk->busyobj);
 
 	pan_ws(sp->ws, 2);
diff --git a/bin/varnishd/cache/cache_rfc2616.c b/bin/varnishd/cache/cache_rfc2616.c
index ae9614d..c2a795a 100644
--- a/bin/varnishd/cache/cache_rfc2616.c
+++ b/bin/varnishd/cache/cache_rfc2616.c
@@ -186,11 +186,11 @@ RFC2616_Body(const struct sess *sp)
 	hp = sp->wrk->beresp;
 
 	if (hp->protover < 11 && !http_HdrIs(hp, H_Connection, "keep-alive"))
-		sp->wrk->do_close = 1;
+		sp->wrk->busyobj->should_close = 1;
 	else if (http_HdrIs(hp, H_Connection, "close"))
-		sp->wrk->do_close = 1;
+		sp->wrk->busyobj->should_close = 1;
 	else
-		sp->wrk->do_close = 0;
+		sp->wrk->busyobj->should_close = 0;
 
 	if (!strcasecmp(http_GetReq(sp->wrk->bereq), "head")) {
 		/*



From phk at varnish-cache.org  Wed Nov 30 07:52:38 2011
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 30 Nov 2011 08:52:38 +0100
Subject: [master] a2921da Move h_content_length from worker to busyobj
Message-ID: <E1RVey6-0007rO-Ca@project.varnish-software.com>

commit a2921da31f78a108322ce69bc5d476e09dbb29a6
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Nov 30 07:52:14 2011 +0000

    Move h_content_length from worker to busyobj

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 819e701..ff9dcea 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -332,8 +332,6 @@ struct worker {
 	struct http		*beresp;
 	struct vbc		*vbc;
 
-	char			*h_content_length;
-
 	/* Stream state */
 	struct stream_ctx	*sctx;
 
@@ -504,6 +502,7 @@ struct busyobj {
 	struct vef_priv		*vef_priv;
 
 	unsigned		should_close;
+	char			*h_content_length;
 
 	unsigned		do_esi;
 	unsigned		do_gzip;
diff --git a/bin/varnishd/cache/cache_center.c b/bin/varnishd/cache/cache_center.c
index 6bda81f..501293d 100644
--- a/bin/varnishd/cache/cache_center.c
+++ b/bin/varnishd/cache/cache_center.c
@@ -188,7 +188,8 @@ cnt_prepresp(struct sess *sp)
 		wrk->res_mode |= RES_LEN;
 
 	if (wrk->busyobj != NULL &&
-	    (wrk->h_content_length != NULL || !wrk->busyobj->do_stream) &&
+	    (wrk->busyobj->h_content_length != NULL ||
+	    !wrk->busyobj->do_stream) &&
 	    !wrk->busyobj->do_gzip && !wrk->busyobj->do_gunzip)
 		wrk->res_mode |= RES_LEN;
 
@@ -256,7 +257,6 @@ cnt_prepresp(struct sess *sp)
 		AZ(wrk->obj);
 		sp->restarts++;
 		sp->director = NULL;
-		wrk->h_content_length = NULL;
 		http_Setup(wrk->bereq, NULL);
 		http_Setup(wrk->beresp, NULL);
 		http_Setup(wrk->resp, NULL);
@@ -469,7 +469,6 @@ cnt_error(struct sess *sp)
 		if (wrk->obj == NULL) {
 			sp->doclose = "Out of objects";
 			sp->director = NULL;
-			wrk->h_content_length = NULL;
 			http_Setup(wrk->beresp, NULL);
 			http_Setup(wrk->bereq, NULL);
 			sp->step = STP_DONE;
@@ -567,7 +566,6 @@ cnt_fetch(struct sess *sp)
 
 	AN(sp->director);
 	AZ(wrk->vbc);
-	AZ(wrk->h_content_length);
 	AZ(wrk->busyobj->should_close);
 	AZ(wrk->storage_hint);
 
@@ -648,7 +646,6 @@ cnt_fetch(struct sess *sp)
 	}
 	http_Setup(wrk->bereq, NULL);
 	http_Setup(wrk->beresp, NULL);
-	wrk->h_content_length = NULL;
 	sp->director = NULL;
 	wrk->storage_hint = NULL;
 
@@ -880,8 +877,6 @@ cnt_fetchbody(struct sess *sp)
 	/* Use unmodified headers*/
 	i = FetchBody(wrk, wrk->obj);
 
-	wrk->h_content_length = NULL;
-
 	http_Setup(wrk->bereq, NULL);
 	http_Setup(wrk->beresp, NULL);
 	wrk->busyobj->vfp = NULL;
@@ -950,8 +945,6 @@ cnt_streambody(struct sess *sp)
 
 	i = FetchBody(wrk, wrk->obj);
 
-	wrk->h_content_length = NULL;
-
 	http_Setup(wrk->bereq, NULL);
 	http_Setup(wrk->beresp, NULL);
 	wrk->busyobj->vfp = NULL;
diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 16509f0..38a2cf3 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -525,7 +525,7 @@ FetchBody(struct worker *w, struct object *obj)
 		mklen = 1;
 		break;
 	case BS_LENGTH:
-		cl = fetch_number( w->h_content_length, 10);
+		cl = fetch_number( w->busyobj->h_content_length, 10);
 		w->busyobj->vfp->begin(w, cl > 0 ? cl : 0);
 		cls = fetch_straight(w, htc, cl);
 		mklen = 1;
diff --git a/bin/varnishd/cache/cache_response.c b/bin/varnishd/cache/cache_response.c
index 39c0ee6..7c63d51 100644
--- a/bin/varnishd/cache/cache_response.c
+++ b/bin/varnishd/cache/cache_response.c
@@ -345,9 +345,9 @@ RES_StreamStart(struct sess *sp)
 		http_Unset(sp->wrk->resp, H_Content_Encoding);
 
 	if (!(sp->wrk->res_mode & RES_CHUNKED) &&
-	    sp->wrk->h_content_length != NULL)
+	    sp->wrk->busyobj->h_content_length != NULL)
 		http_PrintfHeader(sp->wrk, sp->vsl_id, sp->wrk->resp,
-		    "Content-Length: %s", sp->wrk->h_content_length);
+		    "Content-Length: %s", sp->wrk->busyobj->h_content_length);
 
 	sp->wrk->acct_tmp.hdrbytes +=
 	    http_Write(sp->wrk, sp->vsl_id, sp->wrk->resp, 1);
diff --git a/bin/varnishd/cache/cache_rfc2616.c b/bin/varnishd/cache/cache_rfc2616.c
index c2a795a..599ffbb 100644
--- a/bin/varnishd/cache/cache_rfc2616.c
+++ b/bin/varnishd/cache/cache_rfc2616.c
@@ -239,7 +239,8 @@ RFC2616_Body(const struct sess *sp)
 		return (BS_ERROR);
 	}
 
-	if (http_GetHdr(hp, H_Content_Length, &sp->wrk->h_content_length)) {
+	if (http_GetHdr(hp, H_Content_Length,
+	    &sp->wrk->busyobj->h_content_length)) {
 		sp->wrk->stats.fetch_length++;
 		return (BS_LENGTH);
 	}



From perbu at varnish-cache.org  Wed Nov 30 09:39:00 2011
From: perbu at varnish-cache.org (Per Buer)
Date: Wed, 30 Nov 2011 10:39:00 +0100
Subject: [master] 1dfe07e typo. Thanks to 张学彬
Message-ID: <E1RVgd2-0003Rf-Nx@project.varnish-software.com>

commit 1dfe07e0d7a78c9cf048ce3df9d22c22c739d3d8
Author: Per Buer <perbu at varnish-software.com>
Date:   Wed Nov 30 10:38:43 2011 +0100

    typo. Thanks to ???

diff --git a/doc/sphinx/reference/vcl.rst b/doc/sphinx/reference/vcl.rst
index 743d5f2..b74e691 100644
--- a/doc/sphinx/reference/vcl.rst
+++ b/doc/sphinx/reference/vcl.rst
@@ -498,7 +498,7 @@ vcl_pass
   client, but is not entered into the cache.  Subsequent requests sub?
   mitted over the same client connection are handled normally.
   
-  The vcl_recv subroutine may terminate with calling return() with one of
+  The vcl_pass subroutine may terminate with calling return() with one of
   the following keywords:
   
   error code [reason]



From perbu at varnish-cache.org  Wed Nov 30 09:39:00 2011
From: perbu at varnish-cache.org (Per Buer)
Date: Wed, 30 Nov 2011 10:39:00 +0100
Subject: [master] c54185f Merge branch 'master' of
	ssh://git.varnish-cache.org/git/varnish-cache
Message-ID: <E1RVgd2-0003Rl-Ss@project.varnish-software.com>

commit c54185fabd889c25331ddd404d40f3849828165e
Merge: 1dfe07e a2921da
Author: Per Buer <perbu at varnish-software.com>
Date:   Wed Nov 30 10:38:57 2011 +0100

    Merge branch 'master' of ssh://git.varnish-cache.org/git/varnish-cache



From lkarsten at varnish-cache.org  Wed Nov 30 14:01:46 2011
From: lkarsten at varnish-cache.org (Lasse Karstensen)
Date: Wed, 30 Nov 2011 15:01:46 +0100
Subject: [master] e62db2e Clean up examples a bit
Message-ID: <E1RVkjK-0007bC-KN@project.varnish-software.com>

commit e62db2ea30d6b36fac5ce645f1f0db2c90dddd51
Author: Lasse Karstensen <lasse at varnish-software.com>
Date:   Wed Nov 30 15:01:40 2011 +0100

    Clean up examples a bit

diff --git a/doc/sphinx/tutorial/vary.rst b/doc/sphinx/tutorial/vary.rst
index 4da6744..a31e449 100644
--- a/doc/sphinx/tutorial/vary.rst
+++ b/doc/sphinx/tutorial/vary.rst
@@ -14,11 +14,11 @@ the page encoded with the deflate encoding.
 The problem is that the Accept-Encoding field contains a lot of
 different encodings. If one browser sends::
 
-  Accept-Encodign: gzip,deflate
+  Accept-Encoding: gzip,deflate
 
 And another one sends::
 
-  Accept-Encoding:: deflate,gzip
+  Accept-Encoding: deflate,gzip
 
 Varnish will keep two variants of the page requested due to the
 different Accept-Encoding headers. Normalizing the accept-encoding



From lkarsten at varnish-cache.org  Wed Nov 30 15:00:49 2011
From: lkarsten at varnish-cache.org (Lasse Karstensen)
Date: Wed, 30 Nov 2011 16:00:49 +0100
Subject: [master] a1de487 Clean up examples a bit (#2)
Message-ID: <E1RVleT-0008OG-L4@project.varnish-software.com>

commit a1de487811a739388169d3b97b34ed6a413d2c00
Author: Lasse Karstensen <lasse at varnish-software.com>
Date:   Wed Nov 30 15:05:31 2011 +0100

    Clean up examples a bit (#2)

diff --git a/doc/sphinx/tutorial/vary.rst b/doc/sphinx/tutorial/vary.rst
index a31e449..ad7b48d 100644
--- a/doc/sphinx/tutorial/vary.rst
+++ b/doc/sphinx/tutorial/vary.rst
@@ -23,7 +23,7 @@ And another one sends::
 Varnish will keep two variants of the page requested due to the
 different Accept-Encoding headers. Normalizing the accept-encoding
 header will sure that you have as few variants as possible. The
-following VCL code will normalize the Accept-Encoding headers.::
+following VCL code will normalize the Accept-Encoding headers::
 
     if (req.http.Accept-Encoding) {
         if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
@@ -34,7 +34,7 @@ following VCL code will normalize the Accept-Encoding headers.::
         } elsif (req.http.Accept-Encoding ~ "deflate") {
             set req.http.Accept-Encoding = "deflate";
         } else {
-            # unkown algorithm
+            # unknown algorithm
             remove req.http.Accept-Encoding;
         }
     }



From lkarsten at varnish-cache.org  Wed Nov 30 15:00:57 2011
From: lkarsten at varnish-cache.org (Lasse Karstensen)
Date: Wed, 30 Nov 2011 16:00:57 +0100
Subject: [master] 4c4ba07 remove minor typos
Message-ID: <E1RVleb-0008Oa-08@project.varnish-software.com>

commit 4c4ba072982c4f18c4455f50fe77e3be1c8149a0
Author: Lasse Karstensen <lasse at varnish-software.com>
Date:   Wed Nov 30 15:07:59 2011 +0100

    remove minor typos

diff --git a/doc/sphinx/tutorial/cookies.rst b/doc/sphinx/tutorial/cookies.rst
index 2ee14cd..ab59e21 100644
--- a/doc/sphinx/tutorial/cookies.rst
+++ b/doc/sphinx/tutorial/cookies.rst
@@ -10,7 +10,7 @@ the backend.
 
 This can be overly conservative. A lot of sites use Google Analytics
 (GA) to analyze their traffic. GA sets a cookie to track you. This
-cookie is used by the client side java script and is therefore of no
+cookie is used by the client side javascript and is therefore of no
 interest to the server. 
 
 For a lot of web application it makes sense to completely disregard the
@@ -62,4 +62,4 @@ cookies named COOKIE1 and COOKIE2 and you can marvel at it::
   }
 
 The example is taken from the Varnish Wiki, where you can find other
-scary examples of what can be done i VCL.
+scary examples of what can be done in VCL.



From lkarsten at varnish-cache.org  Wed Nov 30 15:00:58 2011
From: lkarsten at varnish-cache.org (Lasse Karstensen)
Date: Wed, 30 Nov 2011 16:00:58 +0100
Subject: [master] 30eaac9 s/reponse/response/
Message-ID: <E1RVlec-0008P4-OU@project.varnish-software.com>

commit 30eaac997b61eeeb3e47778338a673086495135c
Author: Lasse Karstensen <lasse at varnish-software.com>
Date:   Wed Nov 30 15:23:37 2011 +0100

    s/reponse/response/

diff --git a/doc/sphinx/tutorial/increasing_your_hitrate.rst b/doc/sphinx/tutorial/increasing_your_hitrate.rst
index 437461a..31e32d0 100644
--- a/doc/sphinx/tutorial/increasing_your_hitrate.rst
+++ b/doc/sphinx/tutorial/increasing_your_hitrate.rst
@@ -94,7 +94,7 @@ googling "Live HTTP Headers".
 The role of HTTP Headers
 ~~~~~~~~~~~~~~~~~~~~~~~~
 
-Along with each HTTP request and reponse comes a bunch of headers
+Along with each HTTP request and response comes a bunch of headers
 carrying metadata. Varnish will look at these headers to determine if
 it is appropriate to cache the contents and how long Varnish can keep
 the content.