From phk at varnish-cache.org  Mon Feb  4 11:30:29 2013
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 04 Feb 2013 12:30:29 +0100
Subject: [master] 8a2fdce Only msync() on OS's without coherent buf/vm (aka:
 OpenBSD)
Message-ID: <E1U2KFp-0000Qc-Lq@project.varnish-software.com>

commit 8a2fdce41f76c161a130c722d63cbdafddf6337c
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 4 11:30:03 2013 +0000

    Only msync() on OS's without coherent buf/vm (aka: OpenBSD)
    
    Fixes #1198

diff --git a/bin/varnishd/mgt/mgt_shmem.c b/bin/varnishd/mgt/mgt_shmem.c
index bc59194..7707309 100644
--- a/bin/varnishd/mgt/mgt_shmem.c
+++ b/bin/varnishd/mgt/mgt_shmem.c
@@ -247,8 +247,10 @@ mgt_SHM_Create(void)
 		exit (-1);
 	}
 
+#ifdef OpenBSD
 	/* Commit changes, for OS's without coherent VM/buf */
 	AZ(msync(p, getpagesize(), MS_SYNC));
+#endif
 }
 
 /*--------------------------------------------------------------------



From tfheen at varnish-cache.org  Mon Feb  4 11:33:34 2013
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Mon, 04 Feb 2013 12:33:34 +0100
Subject: [master] 683cc92 varnishadm: Handle EINTR from poll
Message-ID: <E1U2KIo-0000ns-Gg@project.varnish-software.com>

commit 683cc929be04d8693e005eb298bc0c9d5eb13f7d
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Mon Feb 4 12:32:20 2013 +0100

    varnishadm: Handle EINTR from poll
    
    poll can return EINTR, so we need to handle that.
    
    Fixes: #1166

diff --git a/bin/varnishadm/varnishadm.c b/bin/varnishadm/varnishadm.c
index e353d7f..bfcdf9d 100644
--- a/bin/varnishadm/varnishadm.c
+++ b/bin/varnishadm/varnishadm.c
@@ -230,6 +230,9 @@ pass(int sock)
 	fds[1].events = POLLIN;
 	while (1) {
 		i = poll(fds, 2, -1);
+		if (i == -1 && errno == EINTR) {
+			continue;
+		}
 		assert(i > 0);
 		if (fds[0].revents & POLLIN) {
 			/* Get rid of the prompt, kinda hackish */



From tfheen at varnish-cache.org  Mon Feb  4 13:50:57 2013
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Mon, 04 Feb 2013 14:50:57 +0100
Subject: [master] 81006ea Handle Cache-control: (private|no-cache|no-store) by
 default
Message-ID: <E1U2MRl-00085p-GX@project.varnish-software.com>

commit 81006eafd6d4cd6f9481740a1d172e316a184d05
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Mon Feb 4 14:43:22 2013 +0100

    Handle Cache-control: (private|no-cache|no-store) by default
    
    As per discussion at the Varnish dev meeting, seeing Cache-control:
    no-cache (and friends) now disable caching by default.  As a twist, we
    respect Surrogate-control if that is present, in the default VCL.
    
    Fixes: 1124
    References: 477

diff --git a/bin/varnishd/default.vcl b/bin/varnishd/default.vcl
index bb78074..0e2031a 100644
--- a/bin/varnishd/default.vcl
+++ b/bin/varnishd/default.vcl
@@ -104,6 +104,9 @@ sub vcl_miss {
 sub vcl_fetch {
     if (beresp.ttl <= 0s ||
         beresp.http.Set-Cookie ||
+        beresp.http.Surrogate-control ~ "no-store" ||
+        (!beresp.http.Surrogate-Control &&
+          beresp.http.Cache-Control ~ "no-cache|no-store|private") ||
         beresp.http.Vary == "*") {
 		/*
 		 * Mark as "Hit-For-Pass" for the next 2 minutes



From tfheen at varnish-cache.org  Mon Feb  4 13:50:57 2013
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Mon, 04 Feb 2013 14:50:57 +0100
Subject: [master] 78936f6 varnishlog: if -m given, but no -c and -b,
 assume both
Message-ID: <E1U2MRl-00085k-FC@project.varnish-software.com>

commit 78936f6460cb8d6dc61d93d828f51c42018e411f
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Mon Feb 4 14:17:50 2013 +0100

    varnishlog: if -m given, but no -c and -b, assume both
    
    If we don't add -c and -b in there, varnishlog will print all
    transactions that aren't associated with a session.  That is likely to
    not be what the user wants.
    
    Fixes: #1071

diff --git a/bin/varnishlog/varnishlog.c b/bin/varnishlog/varnishlog.c
index 07fef9b..270855f 100644
--- a/bin/varnishlog/varnishlog.c
+++ b/bin/varnishlog/varnishlog.c
@@ -49,7 +49,7 @@
 
 #include "compat/daemon.h"
 
-static int	b_flag, c_flag;
+static int	b_flag = 0, c_flag = 0;
 
 /* Ordering-----------------------------------------------------------*/
 
@@ -329,6 +329,16 @@ main(int argc, char * const *argv)
 		}
 	}
 
+	/* If we're matching, we want either -b or -c, apply both if
+	 * none are given.  This prevents spurious noise in the log
+	 * output. */
+	if (b_flag == 0 && c_flag == 0 && m_flag) {
+		b_flag = 1;
+		AN(VSL_Arg(vd, 'b', NULL));
+		c_flag = 1;
+		AN(VSL_Arg(vd, 'c', NULL));
+	}
+
 	if (O_flag && m_flag)
 		usage();
 



From martin at varnish-cache.org  Tue Feb  5 12:25:32 2013
From: martin at varnish-cache.org (Martin Blix Grydeland)
Date: Tue, 05 Feb 2013 13:25:32 +0100
Subject: [3.0] 6990271 Add n_waitinglist counter decrements when freeing
 struct waitinglist.
Message-ID: <E1U2hae-0007za-QO@project.varnish-software.com>

commit 6990271f0b6aadb0c51d776a504294d9134b4238
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Tue Feb 5 13:06:15 2013 +0100

    Add n_waitinglist counter decrements when freeing struct waitinglist.
    
    Fixes: #1261

diff --git a/bin/varnishd/cache_hash.c b/bin/varnishd/cache_hash.c
index 8da00a4..ca31404 100644
--- a/bin/varnishd/cache_hash.c
+++ b/bin/varnishd/cache_hash.c
@@ -138,6 +138,7 @@ HSH_Cleanup(struct worker *w)
 	if (w->nwaitinglist != NULL) {
 		FREE_OBJ(w->nwaitinglist);
 		w->nwaitinglist = NULL;
+		w->stats.n_waitinglist--;
 	}
 	if (w->nhashpriv != NULL) {
 		/* XXX: If needed, add slinger method for this */
@@ -486,12 +487,13 @@ HSH_Lookup(struct sess *sp, struct objhead **poh)
  */
 
 static void
-hsh_rush(struct objhead *oh)
+hsh_rush(struct dstat *ds, struct objhead *oh)
 {
 	unsigned u;
 	struct sess *sp;
 	struct waitinglist *wl;
 
+	AN(ds);
 	CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC);
 	Lck_AssertHeld(&oh->mtx);
 	wl = oh->waitinglist;
@@ -516,6 +518,7 @@ hsh_rush(struct objhead *oh)
 	if (VTAILQ_EMPTY(&wl->list)) {
 		oh->waitinglist = NULL;
 		FREE_OBJ(wl);
+		ds->n_waitinglist--;
 	}
 }
 
@@ -636,7 +639,7 @@ HSH_Unbusy(const struct sess *sp)
 	sp->wrk->nbusyobj = oc->busyobj;
 	oc->busyobj = NULL;
 	if (oh->waitinglist != NULL)
-		hsh_rush(oh);
+		hsh_rush(&sp->wrk->stats, oh);
 	AN(oc->ban);
 	Lck_Unlock(&oh->mtx);
 	assert(oc_getobj(sp->wrk, oc) == o);
@@ -714,7 +717,7 @@ HSH_Deref(struct worker *w, struct objcore *oc, struct object **oo)
 		AN(oc->methods);
 	}
 	if (oh->waitinglist != NULL)
-		hsh_rush(oh);
+		hsh_rush(&w->stats, oh);
 	Lck_Unlock(&oh->mtx);
 	if (r != 0)
 		return (r);
diff --git a/bin/varnishtest/tests/r01261.vtc b/bin/varnishtest/tests/r01261.vtc
new file mode 100644
index 0000000..5b35b4a
--- /dev/null
+++ b/bin/varnishtest/tests/r01261.vtc
@@ -0,0 +1,29 @@
+varnishtest "#1261 - Test n_waitinglist counter"
+
+server s1 {
+	rxreq
+	sema r1 sync 2
+	delay 1
+	txresp
+} -start
+
+varnish v1 -arg "-p thread_pools=1 -w2,2,100" -vcl+backend {
+} -start
+
+client c1 {
+	txreq -url "/test1"
+	rxresp
+	expect resp.status == 200
+} -start
+
+client c2 {
+	txreq -url "/test1"
+	sema r1 sync 2
+	rxresp
+	expect resp.status == 200
+} -start
+
+client c1 -wait
+client c2 -wait
+
+varnish v1 -expect n_waitinglist == 2



From martin at varnish-cache.org  Wed Feb  6 10:15:32 2013
From: martin at varnish-cache.org (Martin Blix Grydeland)
Date: Wed, 06 Feb 2013 11:15:32 +0100
Subject: [3.0] ef5e9e0 Set TCP_NODELAY on the accepted sockets to disable
 Nagle.
Message-ID: <E1U322O-0004LW-AY@project.varnish-software.com>

commit ef5e9e0cc6a65ffc9f0057380f93cad678e5b0ef
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Mon Sep 17 12:57:18 2012 +0200

    Set TCP_NODELAY on the accepted sockets to disable Nagle.
    
    This should not have any impact on the "normal" deliveries, as these
    will as before be done through a single writev operation.
    
    For deliveries involving chunked encoding (e.g. ESI or gunzip), this
    should prevent an unnecessary delay where the Nagle algorithm kicks in
    on the end-chunk write. This should give higher throughput and better
    utilization of a single connection, especially for clients not doing
    pipelining.
    
    Conflicts:
    
    	bin/varnishd/cache_acceptor.c

diff --git a/bin/varnishd/cache_acceptor.c b/bin/varnishd/cache_acceptor.c
index c513e1a..5108a55 100644
--- a/bin/varnishd/cache_acceptor.c
+++ b/bin/varnishd/cache_acceptor.c
@@ -41,6 +41,9 @@
 #include <sys/types.h>
 #include <sys/socket.h>
 
+#include <netinet/in.h>
+#include <netinet/tcp.h>
+
 #include "vcli.h"
 #include "cli_priv.h"
 #include "cache.h"
@@ -91,7 +94,8 @@ static const struct linger linger = {
 	.l_onoff	=	0,
 };
 
-static unsigned char	need_sndtimeo, need_rcvtimeo, need_linger, need_test;
+static unsigned char	need_sndtimeo, need_rcvtimeo, need_linger, need_test,
+			need_tcpnodelay;
 
 int vca_pipes[2] = { -1, -1 };
 
@@ -101,7 +105,7 @@ sock_test(int fd)
 	struct linger lin;
 	struct timeval tv;
 	socklen_t l;
-	int i;
+	int i, tcp_nodelay;
 
 	l = sizeof lin;
 	i = getsockopt(fd, SOL_SOCKET, SO_LINGER, &lin, &l);
@@ -145,6 +149,16 @@ sock_test(int fd)
 	(void)need_rcvtimeo;
 #endif
 
+	l = sizeof tcp_nodelay;
+	i = getsockopt(fd, IPPROTO_TCP, TCP_NODELAY, &tcp_nodelay, &l);
+	if (i) {
+		VTCP_Assert(i);
+		return;
+	}
+	assert(l == sizeof tcp_nodelay);
+	if (!tcp_nodelay)
+		need_tcpnodelay = 1;
+
 	need_test = 0;
 }
 
@@ -158,6 +172,7 @@ VCA_Prep(struct sess *sp)
 {
 	char addr[VTCP_ADDRBUFSIZE];
 	char port[VTCP_PORTBUFSIZE];
+	int tcp_nodelay = 1;
 
 	VTCP_name(sp->sockaddr, sp->sockaddrlen,
 	    addr, sizeof addr, port, sizeof port);
@@ -189,6 +204,9 @@ VCA_Prep(struct sess *sp)
 		VTCP_Assert(setsockopt(sp->fd, SOL_SOCKET, SO_RCVTIMEO,
 		    &tv_rcvtimeo, sizeof tv_rcvtimeo));
 #endif
+	if (need_tcpnodelay)
+		VTCP_Assert(setsockopt(sp->fd, IPPROTO_TCP, TCP_NODELAY,
+		    &tcp_nodelay, sizeof tcp_nodelay));
 }
 
 /*--------------------------------------------------------------------*/
@@ -206,6 +224,7 @@ vca_acct(void *arg)
 #ifdef SO_SNDTIMEO_WORKS
 	double send_timeout = 0;
 #endif
+	int tcp_nodelay = 1;
 	int i;
 	struct pollfd *pfd;
 	struct listen_sock *ls;
@@ -225,6 +244,8 @@ vca_acct(void *arg)
 		AZ(listen(ls->sock, params->listen_depth));
 		AZ(setsockopt(ls->sock, SOL_SOCKET, SO_LINGER,
 		    &linger, sizeof linger));
+		AZ(setsockopt(ls->sock, IPPROTO_TCP, TCP_NODELAY,
+		    &tcp_nodelay, sizeof tcp_nodelay));
 		pfd[i].events = POLLIN;
 		pfd[i++].fd = ls->sock;
 	}



From phk at varnish-cache.org  Wed Feb  6 12:03:16 2013
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 06 Feb 2013 13:03:16 +0100
Subject: [master] 0159805 Rewrite the vmod-spec compiler before we start
 adding new and fancy features.
Message-ID: <E1U33ie-0007yw-BI@project.varnish-software.com>

commit 015980576c7603ba13f29c1b0000761945958cf6
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 6 12:02:39 2013 +0000

    Rewrite the vmod-spec compiler before we start adding new and fancy
    features.

diff --git a/lib/libvcl/vmodtool.py b/lib/libvcl/vmodtool.py
index d5681ee..284848f 100755
--- a/lib/libvcl/vmodtool.py
+++ b/lib/libvcl/vmodtool.py
@@ -37,6 +37,8 @@
 # as a string, suitable for inclusion in the C-source of the compile VCL
 # program.
 
+from __future__ import print_function
+
 import sys
 import re
 
@@ -64,59 +66,14 @@ ctypes = {
 
 #######################################################################
 
-initname = ""
-modname = "???"
-pstruct = ""
-pinit = ""
-tdl = ""
-plist = ""
-slist = ""
-
-def do_func(fname, rval, args, vargs):
-	global pstruct
-	global pinit
-	global plist
-	global slist
-	global tdl
-	#print(fname, rval, args)
-
-	# C argument list
-	cargs = "(struct req *"
-	for i in args:
-		cargs += ", " + i
-	cargs += ")"
-
-	# Prototypes for vmod implementation and interface typedef
-	proto = ctypes[rval] + " vmod_" + fname + cargs
-	sproto = ctypes[rval] + " td_" + modname + "_" + fname + cargs
-
-	# append to lists of prototypes
-	plist += proto + ";\n"
-	tdl += "typedef " + sproto + ";\n"
-
-	# Append to struct members
-	pstruct += "\ttd_" + modname + "_" + fname + "\t*" + fname + ";\n"
-
-	# Append to struct initializer
-	pinit += "\tvmod_" + fname + ",\n"
-
-	# Compose the vmod spec-string
-	s = modname + '.' + fname + "\\0"
-	s += "Vmod_Func_" + modname + "." + fname + "\\0"
-	s += rval + '\\0'
-	for i in vargs:
-		s +=  i + '\\0'
-	slist += '\t"' + s + '",\n'
-
-#######################################################################
+def file_header(fo):
+        fo.write("""/*
+ * NB:  This file is machine generated, DO NOT EDIT!
+ *
+ * Edit vmod.vcc and run vmod.py instead
+ */
 
-def partition(string, separator):
-	if (hasattr(string,"partition")):
-		return string.partition(separator)
-	i = string.find(separator)
-	if i >= 0:
-		return (string[:i],separator,string[i+len(separator):])
-	return (string, '', '')
+""")
 
 #######################################################################
 
@@ -125,147 +82,277 @@ def is_c_name(s):
 
 #######################################################################
 
-def parse_enum(tq):
-	assert tq[0] == '{'
-	assert tq[-1] == '}'
-	f = tq[1:-1].split(',')
-	s="ENUM\\0"
-	b=dict()
-	for i in f:
-		i = i.strip()
-		if not is_c_name(i):
-			raise Exception("Enum value '%s' is illegal" % i)
-		if i in b:
-			raise Exception("Duplicate Enum value '%s'" % i)
-		b[i] = True
-		s = s + i.strip() + '\\0'
-	return s
-
-#######################################################################
+class token(object):
+	def __init__(self, ln, ch, str):
+		self.ln = ln
+		self.ch = ch
+		self.str = str
+
+	def __repr__(self):
+		return "<@%d \"%s\">" % (self.ln, self.str)
+
+class vmod(object):
+	def __init__(self, nam):
+		if not is_c_name(nam):
+			raise Exception("Module name '%s' is illegal" % nam)
+		self.nam = nam
+		self.init = None
+		self.fini = None
+		self.funcs = list()
+
+	def set_init(self, nam):
+		if self.init != None:
+			raise Exception("Module %s already has Init" % self.nam)
+		if not is_c_name(nam):
+			raise Exception("Init name '%s' is illegal" % nam)
+		self.init = nam
+
+	def set_fini(self, nam):
+		if self.fini != None:
+			raise Exception("Module %s already has Fini" % self.nam)
+		if not is_c_name(nam):
+			raise Exception("Fini name '%s' is illegal" % nam)
+		self.fini = nam
+
+	def add_func(self, fn):
+		self.funcs.append(fn)
+
+	def c_proto(self, fo):
+		for f in self.funcs:
+			f.c_proto(fo)
+		if self.init != None:
+			fo.write("int " + self.init)
+			fo.write(
+			    "(struct vmod_priv *, const struct VCL_conf *);\n")
+		if self.fini != None:
+			fo.write("int " + self.fini)
+			fo.write(
+			    "(struct vmod_priv *, const struct VCL_conf *);\n")
+		fo.write("extern const void * const Vmod_Id;\n")
+
+	def c_typedefs(self, fo):
+		for f in self.funcs:
+			fo.write(f.c_typedefs(fo, self.nam) + "\n")
+
+	def c_vmod(self, fo):
+		fo.write('const char Vmod_Name[] = \"' + self.nam + '";\n')
+		fo.write("\n")
+
+		cs = self.c_struct()
+		fo.write("const " + cs + ' Vmod_Func = {\n')
+
+		for f in self.funcs:
+			fo.write("\tvmod_" + f.nam + ",\n")
+		if self.init != None:
+			fo.write("\t" + self.init + ",\n")
+		if self.fini != None:
+			fo.write("\t" + self.fini + ",\n")
+		fo.write("};\n")
+		fo.write("\n")
+		fo.write("const int Vmod_Len = sizeof(Vmod_Func);\n")
+		fo.write("\n")
+		fo.write("const char Vmod_Proto[] =\n")
+		for f in self.funcs:
+			fo.write('\t"' + f.c_typedefs(fo, self.nam) + '\\n"\n')
+		fo.write('\t"\\n"\n')
+		for i in (cs + " Vmod_Func_" + self.nam + ';').split("\n"):
+			fo.write('\n\t"' + i + '\\n"')
+		fo.write(";\n\n")
+
+		fo.write("const char * const Vmod_Spec[] = {\n")
+		for f in self.funcs:
+			fo.write('\t"' + f.strspec(self.nam) + '",\n')
+		if self.init != None:
+			fo.write(
+			    '\t"INIT\\0Vmod_Func_' + self.nam + '._init",\n')
+		if self.fini != None:
+			fo.write(
+			    '\t"FINI\\0Vmod_Func_' + self.nam + '._fini",\n')
+		fo.write("\t0\n")
+		fo.write("};\n")
+		fo.write("\n")
+		fo.write('const char Vmod_Varnish_ABI[] = VMOD_ABI_Version;\n')
+		fo.write("\n")
+		fo.write('const void * const Vmod_Id = &Vmod_Id;\n')
+
+	def c_struct(self):
+		s = 'struct Vmod_Func_' + self.nam + ' {\n'
+		for f in self.funcs:
+			s += '\ttd_' + self.nam + "_" + f.nam
+			s += "\t*" + f.nam + ";\n"
+		if self.init != None:
+			s += "\tvmod_init_f\t*_init;\n"
+		if self.fini != None:
+			s += "\tvmod_fini_f\t*_fini;\n"
+		s += '}'
+		return s
+	
+
+class func(object):
+	def __init__(self, nam, retval, al):
+		if not is_c_name(nam):
+			raise Exception("Func name '%s' is illegal" % nam)
+		if retval not in ctypes:
+			raise Exception(
+			    "Return type '%s' not a valid type" % retval)
+		self.nam = nam
+		self.al = al
+		self.retval = retval
+
+	def __repr__(self):
+		return "<FUNC %s %s>" % (self.retval, self.nam)
+
+	def c_proto(self, fo):
+		fo.write(ctypes[self.retval])
+		fo.write(" vmod_" + self.nam)
+		fo.write("(struct req *")
+		for a in self.al:
+			fo.write(", " + ctypes[a.typ])
+		fo.write(");\n")
+
+	def c_typedefs(self, fo, modname):
+		s = "typedef "
+		s += ctypes[self.retval]
+		s += " td_" + modname + "_" + self.nam
+		s += "(struct req *"
+		for a in self.al:
+			s += ", " + ctypes[a.typ]
+		s += ");"
+		return s
+
+	def strspec(self, modname):
+		s = modname + "." + self.nam
+		s += "\\0"
+		s += "Vmod_Func_" + modname + "." + self.nam + "\\0"
+		s += self.retval + "\\0"
+		for a in self.al:
+			s += a.strspec()
+		return s
+		
+
+class arg(object):
+	def __init__(self, typ, nam = None, det = None):
+		self.nam = nam
+		self.typ = typ
+		self.det = det
+
+	def __repr__(self):
+		return "<ARG %s %s %s>" % (self.nam, self.typ, str(self.det))
+
+	def strspec(self):
+		if self.det == None:
+			return self.typ + "\\0"
+		else:
+			return self.det
+		return "??"
 
 f = open(specfile, "r")
-
-def nextline():
-	while True:
-		l0 = f.readline()
-		if l0 == "":
-			return l0
-		l0 = re.sub("#.*$", "", l0)
-		l0 = re.sub("\s\s*", " ", l0.strip())
-		if l0 != "":
-			return l0
-
-while True:
-	l0 = nextline()
-	if l0 == "":
-		break;
-	l = partition(l0, " ")
-
-	if l[0] == "Module":
-		modname = l[2].strip();
-		if not is_c_name(modname):
-			raise Exception("Module name '%s' is illegal" % modname)
+tl = list()
+lines = list()
+ln = 0
+for l in f:
+	ln += 1
+	lines.append(l)
+	if l == "":
 		continue
-
-	if l[0] == "Init":
-		initname = l[2].strip();
-		if not is_c_name(initname):
-			raise Exception("Init name '%s' is illegal" % initname)
+	l = re.sub("[ \t]*#.*$", "", l)
+	l = re.sub("[ \t]*\n", "", l)
+	l = re.sub("([(){},])", r' \1 ', l)
+	if l == "":
 		continue
-
-	if l[0] != "Function":
-		raise Exception("Expected 'Function' line, got '%s'" % l[0])
-
-	# Find the return type of the function
-	l = partition(l[2].strip(), " ")
-	rt_type = l[0]
-	if rt_type not in ctypes:
-		raise Exception("Return type '%s' not a valid type" % rt_type)
-
-	# Find the function name
-	l = partition(l[2].strip(), "(")
-
-	fname = l[0].strip()
-	if not is_c_name(fname):
-		raise Exception("Function name '%s' is illegal" % fname)
-
-	if l[1] != '(':
-		raise Exception("Missing '('")
-
-	l = l[2]
-
-	while -1 == l.find(")"):
-		l1 = nextline()
-		if l1 == "":
-			raise Exception("End Of Input looking for ')'")
-		l = l + l1
-
-	if -1 != l.find("("):
-		raise Exception("Nesting trouble with '(...)' ")
-
-	if l[-1:] != ')':
-		raise Exception("Junk after ')'")
-
-	l = l[:-1]
-
-	args = list()
-	vargs = list()
-
-	for i in re.finditer("([A-Z_]+)\s*({[^}]+})?(,|$)", l):
-		at = i.group(1)
-		tq = i.group(2)
-		if at not in ctypes:
-			raise Exception(
-			    "Argument type '%s' not a valid type" % at)
-
-		args.append(ctypes[at])
-
-		if at == "ENUM":
-			if tq == None:
-				raise Exception(
-				    "Argument type '%s' needs qualifier {...}"
-				    % at)
-			at=parse_enum(tq)
-
-		elif tq != None:
-			raise Exception(
-			    "Argument type '%s' cannot be qualified with {...}"
-			    % at)
-
-		vargs.append(at)
-
-	do_func(fname, rt_type, args, vargs)
+	for j in l.split():
+		tl.append(token(ln, 0, j))
+f.close()
 
 #######################################################################
-def dumps(s):
+#
+#
+def parse_enum2(tl):
+	t = tl.pop(0)
+	if t.str != "{":
+		raise Exception("expected \"{\"")
+	s = "ENUM\\0"
 	while True:
-		l = partition(s, "\n")
-		if len(l[0]) == 0:
+		t = tl.pop(0)
+		if t.str == "}":
 			break
-		fc.write('\t"' + l[0] + '\\n"\n')
-		s = l[2]
+		s += t.str + "\\0"
+		if tl[0].str == ",":
+			tl.pop(0)
+		elif tl[0].str != "}":
+			raise Exception("Expceted \"}\" or \",\"")
+	s += "\\0"
+	return arg("ENUM", det=s)
 
 #######################################################################
+# The first thing in the file must be the Module declaration
+#
 
-if initname != "":
-	plist += "int " + initname
-	plist += "(struct vmod_priv *, const struct VCL_conf *);\n"
-	pstruct += "\tvmod_init_f\t*_init;\n"
-	pinit += "\t" + initname + ",\n"
-	slist += '\t"INIT\\0Vmod_Func_' + modname + '._init",\n'
+t = tl.pop(0)
+if t.str != "Module":
+	raise Exception("\"Module\" must be first in file")
+t = tl.pop(0)
+vmod = vmod(t.str)
 
 #######################################################################
+# Parse the rest of the file
+#
 
-def file_header(fo):
-        fo.write("""/*
- * NB:  This file is machine generated, DO NOT EDIT!
- *
- * Edit vmod.vcc and run vmod.py instead
- */
+while len(tl) > 0:
+	t = tl.pop(0)
 
-""")
+	if t.str == "Init":
+		t = tl.pop(0)
+		vmod.set_init(t.str)
+		continue
+
+	if t.str == "Fini":
+		t = tl.pop(0)
+		vmod.set_fini(t.str)
+		continue
+
+	if t.str == "Function":
+		al = list()
+		t = tl.pop(0)
+		rt_type = t.str
+		if rt_type not in ctypes:
+			raise Exception(
+			    "Return type '%s' not a valid type" % rt_type)
+
+		t = tl.pop(0)
+		fname = t.str
+		if not is_c_name(fname):
+			raise Exception("Function name '%s' is illegal" % fname)
+
+		t = tl.pop(0)
+		if t.str != "(":
+			raise Exception("Expected \"(\" got \"%s\"", t.str)
+
+		while True:
+			t = tl.pop(0)
+			if t.str == ")":
+				break
+			if t.str == "ENUM":
+				al.append(parse_enum2(tl))
+			elif t.str in ctypes:
+				al.append(arg(t.str))
+			else:
+				raise Exception("ARG? %s" % t.str)
+			if tl[0].str == ",":
+				tl.pop(0)
+			elif tl[0].str != ")":
+				raise Exception("Expceted \")\" or \",\"")
+		if t.str != ")":
+			raise Exception("End Of Input looking for ')'")
+		f = func(fname, rt_type, al)
+		vmod.add_func(f)
+		continue
+
+	raise Exception("Expected \"Init\", \"Fini\" or \"Function\"")
 
 #######################################################################
+# Parsing done, now process
+#
 
 fc = open("vcc_if.c", "w")
 fh = open("vcc_if.h", "w")
@@ -278,43 +365,20 @@ fh.write('struct VCL_conf;\n')
 fh.write('struct vmod_priv;\n')
 fh.write("\n");
 
-fh.write(plist)
-
+vmod.c_proto(fh)
 
-fc.write('#include "config.h"\n')
-fc.write('\n')
-fc.write('#include "vrt.h"\n')
-fc.write('#include "vcc_if.h"\n')
-fc.write('#include "vmod_abi.h"\n')
-fc.write("\n");
+fc.write("""#include "config.h"
 
-fc.write("\n");
+#include "vrt.h"
+#include "vcc_if.h"
+#include "vmod_abi.h"
 
-fc.write(tdl);
-fc.write("\n");
 
-fc.write('const char Vmod_Name[] = "' + modname + '";\n')
-
-fc.write("const struct Vmod_Func_" + modname + " {\n")
-fc.write(pstruct + "} Vmod_Func = {\n" + pinit + "};\n")
-fc.write("\n");
-
-fc.write("const int Vmod_Len = sizeof(Vmod_Func);\n")
-fc.write("\n");
-
-fc.write('const char Vmod_Proto[] =\n')
-dumps(tdl);
-fc.write('\t"\\n"\n')
-dumps("struct Vmod_Func_" + modname + " {\n")
-dumps(pstruct + "} Vmod_Func_" + modname + ";\n")
-fc.write('\t;\n')
-fc.write("\n");
-
-fc.write('const char * const Vmod_Spec[] = {\n' + slist + '\t0\n};\n')
-
-fc.write('const char Vmod_Varnish_ABI[] = VMOD_ABI_Version;\n')
-
-fh.write('extern const void * const Vmod_Id;\n')
-fc.write('const void * const Vmod_Id = &Vmod_Id;\n')
+""")
 
+vmod.c_typedefs(fc)
 fc.write("\n")
+vmod.c_vmod(fc)
+
+fc.close()
+fh.close()



From phk at varnish-cache.org  Wed Feb  6 14:32:47 2013
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 06 Feb 2013 15:32:47 +0100
Subject: [master] da62191 First compiler now recoginize what will become "VMOD
 objects" once I get the other compiler to cope too.
Message-ID: <E1U363L-0002SX-I1@project.varnish-software.com>

commit da621917058b23215cb47ee1ac933ef57e288f74
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 6 14:31:56 2013 +0000

    First compiler now recoginize what will become "VMOD objects"
    once I get the other compiler to cope too.
    
    This snapshot is committed to solicit feedback, so please do :-)

diff --git a/lib/libvcl/vcc_vmod.c b/lib/libvcl/vcc_vmod.c
index c3a2a83..839577d 100644
--- a/lib/libvcl/vcc_vmod.c
+++ b/lib/libvcl/vcc_vmod.c
@@ -167,7 +167,13 @@ vcc_ParseImport(struct vcc *tl)
 	Fh(tl, 0, "static struct vmod_priv vmod_priv_%.*s;\n", PF(mod));
 	for (; *spec != NULL; spec++) {
 		p = *spec;
-		if (!strcmp(p, "INIT")) {
+		if (!strcmp(p, "OBJ")) {
+			// Nothing yet
+		} else if (!strcmp(p, "METHOD")) {
+			// Nothing yet
+		} else if (!strcmp(p, "FINI")) {
+			// Nothing yet
+		} else if (!strcmp(p, "INIT")) {
 			p += strlen(p) + 1;
 			Fi(tl, 0, "\t%s(&vmod_priv_%.*s, &VCL_conf);\n",
 			    p, PF(mod));
diff --git a/lib/libvcl/vmodtool.py b/lib/libvcl/vmodtool.py
index 284848f..d1dfb68 100755
--- a/lib/libvcl/vmodtool.py
+++ b/lib/libvcl/vmodtool.py
@@ -91,6 +91,8 @@ class token(object):
 	def __repr__(self):
 		return "<@%d \"%s\">" % (self.ln, self.str)
 
+#######################################################################
+
 class vmod(object):
 	def __init__(self, nam):
 		if not is_c_name(nam):
@@ -99,6 +101,7 @@ class vmod(object):
 		self.init = None
 		self.fini = None
 		self.funcs = list()
+		self.objs = list()
 
 	def set_init(self, nam):
 		if self.init != None:
@@ -117,7 +120,12 @@ class vmod(object):
 	def add_func(self, fn):
 		self.funcs.append(fn)
 
+	def add_obj(self, fn):
+		self.objs.append(fn)
+
 	def c_proto(self, fo):
+		for o in self.objs:
+			o.c_proto(fo)
 		for f in self.funcs:
 			f.c_proto(fo)
 		if self.init != None:
@@ -130,105 +138,217 @@ class vmod(object):
 			    "(struct vmod_priv *, const struct VCL_conf *);\n")
 		fo.write("extern const void * const Vmod_Id;\n")
 
-	def c_typedefs(self, fo):
+	def c_typedefs_(self):
+		l = list()
+		for o in self.objs:
+			for t in o.c_typedefs(self.nam):
+				l.append(t)
+		l.append("")
+		l.append("/* Functions */")
 		for f in self.funcs:
-			fo.write(f.c_typedefs(fo, self.nam) + "\n")
+			l.append(f.c_typedef(self.nam))
+		return l
+
+	def c_typedefs(self, fo):
+		for i in self.c_typedefs_():
+			fo.write(i + "\n")
 
 	def c_vmod(self, fo):
 		fo.write('const char Vmod_Name[] = \"' + self.nam + '";\n')
 		fo.write("\n")
 
 		cs = self.c_struct()
-		fo.write("const " + cs + ' Vmod_Func = {\n')
+		fo.write("const " + cs + ' Vmod_Func = ')
+		fo.write(self.c_initializer())
+		fo.write("\n")
 
-		for f in self.funcs:
-			fo.write("\tvmod_" + f.nam + ",\n")
-		if self.init != None:
-			fo.write("\t" + self.init + ",\n")
-		if self.fini != None:
-			fo.write("\t" + self.fini + ",\n")
-		fo.write("};\n")
 		fo.write("\n")
 		fo.write("const int Vmod_Len = sizeof(Vmod_Func);\n")
 		fo.write("\n")
+
+
 		fo.write("const char Vmod_Proto[] =\n")
-		for f in self.funcs:
-			fo.write('\t"' + f.c_typedefs(fo, self.nam) + '\\n"\n')
+		for t in self.c_typedefs_():
+			fo.write('\t"' + t + '\\n"\n')
 		fo.write('\t"\\n"\n')
 		for i in (cs + " Vmod_Func_" + self.nam + ';').split("\n"):
 			fo.write('\n\t"' + i + '\\n"')
 		fo.write(";\n\n")
 
-		fo.write("const char * const Vmod_Spec[] = {\n")
-		for f in self.funcs:
-			fo.write('\t"' + f.strspec(self.nam) + '",\n')
-		if self.init != None:
-			fo.write(
-			    '\t"INIT\\0Vmod_Func_' + self.nam + '._init",\n')
-		if self.fini != None:
-			fo.write(
-			    '\t"FINI\\0Vmod_Func_' + self.nam + '._fini",\n')
-		fo.write("\t0\n")
-		fo.write("};\n")
+		fo.write(self.c_strspec())
+
 		fo.write("\n")
 		fo.write('const char Vmod_Varnish_ABI[] = VMOD_ABI_Version;\n')
 		fo.write("\n")
 		fo.write('const void * const Vmod_Id = &Vmod_Id;\n')
 
+	def c_initializer(self):
+		s = '{\n'
+		for o in self.objs:
+			s += o.c_initializer()
+
+		s += "\n\t/* Functions */\n"
+		for f in self.funcs:
+			s += f.c_initializer()
+
+		s += "\n\t/* Init/Fini */\n"
+		if self.init != None:
+			s += "\t" + self.init + ",\n"
+		if self.fini != None:
+			s += "\t" + self.fini + ",\n"
+		s += "};"
+
+		return s
+
 	def c_struct(self):
 		s = 'struct Vmod_Func_' + self.nam + ' {\n'
+		for o in self.objs:
+			s += o.c_struct(self.nam)
+
+		s += "\n\t/* Functions */\n"
 		for f in self.funcs:
-			s += '\ttd_' + self.nam + "_" + f.nam
-			s += "\t*" + f.nam + ";\n"
+			s += f.c_struct(self.nam)
+
+		s += "\n\t/* Init/Fini */\n"
 		if self.init != None:
 			s += "\tvmod_init_f\t*_init;\n"
 		if self.fini != None:
 			s += "\tvmod_fini_f\t*_fini;\n"
 		s += '}'
 		return s
-	
+
+	def c_strspec(self):
+		s = "const char * const Vmod_Spec[] = {\n"
+
+		for o in self.objs:
+			s += o.c_strspec(self.nam)
+
+		s += "\n\t/* Functions */\n"
+		for f in self.funcs:
+			s += '\t"' + f.c_strspec(self.nam) + '",\n'
+
+		s += "\n\t/* Init/Fini */\n"
+		if self.init != None:
+			s += '\t"INIT\\0Vmod_Func_' + self.nam + '._init",\n'
+		if self.fini != None:
+			s += '\t"FINI\\0Vmod_Func_' + self.nam + '._fini",\n'
+
+		s += "\t0\n"
+		s += "};\n"
+		return s
+
+#######################################################################
 
 class func(object):
 	def __init__(self, nam, retval, al):
-		if not is_c_name(nam):
-			raise Exception("Func name '%s' is illegal" % nam)
+		#if not is_c_name(nam):
+		#	raise Exception("Func name '%s' is illegal" % nam)
 		if retval not in ctypes:
 			raise Exception(
 			    "Return type '%s' not a valid type" % retval)
 		self.nam = nam
+		self.cnam = nam.replace(".", "__")
 		self.al = al
 		self.retval = retval
+		self.pfx = None
 
 	def __repr__(self):
 		return "<FUNC %s %s>" % (self.retval, self.nam)
 
+	def set_pfx(self, s):
+		self.pfx = s
+
 	def c_proto(self, fo):
 		fo.write(ctypes[self.retval])
-		fo.write(" vmod_" + self.nam)
+		fo.write(" vmod_" + self.cnam)
 		fo.write("(struct req *")
+		if self.pfx != None:
+			fo.write(self.pfx)
 		for a in self.al:
 			fo.write(", " + ctypes[a.typ])
 		fo.write(");\n")
 
-	def c_typedefs(self, fo, modname):
+	def c_typedef(self, modname):
 		s = "typedef "
 		s += ctypes[self.retval]
-		s += " td_" + modname + "_" + self.nam
+		s += " td_" + modname + "_" + self.cnam
 		s += "(struct req *"
+		if self.pfx != None:
+			s += self.pfx
 		for a in self.al:
 			s += ", " + ctypes[a.typ]
 		s += ");"
 		return s
 
-	def strspec(self, modname):
-		s = modname + "." + self.nam
+	def c_struct(self, modname):
+		s = '\ttd_' + modname + "_" + self.cnam
+		while len(s.expandtabs()) < 40:
+			s += "\t"
+		s += "*" + self.cnam + ";\n"
+		return s
+
+	def c_initializer(self):
+		return "\tvmod_" + self.cnam + ",\n"
+
+	def c_strspec(self, modnam):
+		s = modnam + "." + self.nam
 		s += "\\0"
-		s += "Vmod_Func_" + modname + "." + self.nam + "\\0"
+		s += "Vmod_Func_" + modnam + "." + self.cnam + "\\0"
 		s += self.retval + "\\0"
 		for a in self.al:
-			s += a.strspec()
+			s += a.c_strspec()
 		return s
 		
+#######################################################################
+
+class obj(object):
+	def __init__(self, nam):
+		self.nam = nam
+		self.init_fini = None
+		self.methods = list()
+
+	def set_init_fini(self, f):
+		self.init_fini = f
+
+	def add_method(self, m):
+		self.methods.append(m)
+
+	def c_typedefs(self, modname):
+		l = list()
+		l.append("/* Object " + self.nam + " */")
+		l.append(self.init_fini.c_typedef(modname) + "")
+		for m in self.methods:
+			l.append(m.c_typedef(modname) + "")
+		return l
+
+	def c_proto(self, fo):
+		fo.write("struct vmod_" + self.nam + ";\n")
+		self.init_fini.c_proto(fo)
+		for m in o.methods:
+			m.c_proto(fo)
+
+	def c_struct(self, modnam):
+		s = "\t/* Object " + self.nam + " */\n"
+		s += self.init_fini.c_struct(modnam)
+		for m in self.methods:
+			s += m.c_struct(modnam)
+		return s
+
+	def c_initializer(self):
+		s = "\t/* Object " + self.nam + " */\n"
+		s += self.init_fini.c_initializer()
+		for m in self.methods:
+			s += m.c_initializer()
+		return s
+
+	def c_strspec(self, modnam):
+		s = "\t/* Object " + self.nam + " */\n"
+		s += '\t"OBJ\\0' + self.init_fini.c_strspec(modnam) + '",\n'
+		for m in self.methods:
+			s += '\t"METHOD\\0' + m.c_strspec(modnam) + '",\n'
+		return s
+
+#######################################################################
 
 class arg(object):
 	def __init__(self, typ, nam = None, det = None):
@@ -239,13 +359,15 @@ class arg(object):
 	def __repr__(self):
 		return "<ARG %s %s %s>" % (self.nam, self.typ, str(self.det))
 
-	def strspec(self):
+	def c_strspec(self):
 		if self.det == None:
 			return self.typ + "\\0"
 		else:
 			return self.det
 		return "??"
 
+#######################################################################
+
 f = open(specfile, "r")
 tl = list()
 lines = list()
@@ -285,6 +407,70 @@ def parse_enum2(tl):
 	return arg("ENUM", det=s)
 
 #######################################################################
+#
+#
+
+def parse_func(tl, rt_type = None, obj=None):
+	al = list()
+	if rt_type == None:
+		t = tl.pop(0)
+		rt_type = t.str
+	if rt_type not in ctypes:
+		raise Exception(
+		    "Return type '%s' not a valid type" % rt_type)
+
+	t = tl.pop(0)
+	fname = t.str
+	if obj != None and fname[0] == "." and is_c_name(fname[1:]):
+		fname = obj + fname
+	elif not is_c_name(fname):
+		raise Exception("Function name '%s' is illegal" % fname)
+
+	t = tl.pop(0)
+	if t.str != "(":
+		raise Exception("Expected \"(\" got \"%s\"", t.str)
+
+	while True:
+		t = tl.pop(0)
+		if t.str == ")":
+			break
+		if t.str == "ENUM":
+			al.append(parse_enum2(tl))
+		elif t.str in ctypes:
+			al.append(arg(t.str))
+		else:
+			raise Exception("ARG? %s" % t.str)
+		if tl[0].str == ",":
+			tl.pop(0)
+		elif tl[0].str != ")":
+			raise Exception("Expceted \")\" or \",\"")
+	if t.str != ")":
+		raise Exception("End Of Input looking for ')'")
+	f = func(fname, rt_type, al)
+	return f
+
+#######################################################################
+#
+#
+
+def parse_obj(tl):
+	o = obj(tl[0].str)
+	f = parse_func(tl, "VOID")
+	f.set_pfx(", struct vmod_" + o.nam + " **")
+	o.set_init_fini(f)
+	t = tl.pop(0)
+	assert t.str == "{"
+	while True:
+		t = tl.pop(0)
+		if t.str == "}":
+			break
+		assert t.str == "Method"
+		f = parse_func(tl, obj=o.nam)
+		f.set_pfx(", struct vmod_" + o.nam + " *")
+		o.add_method(f)
+	return o
+
+#######################################################################
 # The first thing in the file must be the Module declaration
 #
 
@@ -304,51 +490,17 @@ while len(tl) > 0:
 	if t.str == "Init":
 		t = tl.pop(0)
 		vmod.set_init(t.str)
-		continue
-
-	if t.str == "Fini":
+	elif t.str == "Fini":
 		t = tl.pop(0)
 		vmod.set_fini(t.str)
-		continue
-
-	if t.str == "Function":
-		al = list()
-		t = tl.pop(0)
-		rt_type = t.str
-		if rt_type not in ctypes:
-			raise Exception(
-			    "Return type '%s' not a valid type" % rt_type)
-
-		t = tl.pop(0)
-		fname = t.str
-		if not is_c_name(fname):
-			raise Exception("Function name '%s' is illegal" % fname)
-
-		t = tl.pop(0)
-		if t.str != "(":
-			raise Exception("Expected \"(\" got \"%s\"", t.str)
-
-		while True:
-			t = tl.pop(0)
-			if t.str == ")":
-				break
-			if t.str == "ENUM":
-				al.append(parse_enum2(tl))
-			elif t.str in ctypes:
-				al.append(arg(t.str))
-			else:
-				raise Exception("ARG? %s" % t.str)
-			if tl[0].str == ",":
-				tl.pop(0)
-			elif tl[0].str != ")":
-				raise Exception("Expceted \")\" or \",\"")
-		if t.str != ")":
-			raise Exception("End Of Input looking for ')'")
-		f = func(fname, rt_type, al)
+	elif t.str == "Function":
+		f = parse_func(tl)
 		vmod.add_func(f)
-		continue
-
-	raise Exception("Expected \"Init\", \"Fini\" or \"Function\"")
+	elif t.str == "Object":
+		o = parse_obj(tl)
+		vmod.add_obj(o)
+	else:
+		raise Exception("Expected \"Init\", \"Fini\" or \"Function\"")
 
 #######################################################################
 # Parsing done, now process
diff --git a/lib/libvmod_debug/Makefile.am b/lib/libvmod_debug/Makefile.am
index 25b3f3f..7e2c849 100644
--- a/lib/libvmod_debug/Makefile.am
+++ b/lib/libvmod_debug/Makefile.am
@@ -14,7 +14,8 @@ noinst_LTLIBRARIES = libvmod_debug.la
 libvmod_debug_la_LDFLAGS = $(AM_LDFLAGS) -module -export-dynamic -avoid-version -shared -rpath /nowhere
 
 libvmod_debug_la_SOURCES = \
-	vmod_debug.c
+	vmod_debug.c \
+	vmod_debug_obj.c
 
 nodist_libvmod_debug_la_SOURCES = \
 	vcc_if.c \
diff --git a/lib/libvmod_debug/vmod.vcc b/lib/libvmod_debug/vmod.vcc
index 2e3c33e..653f168 100644
--- a/lib/libvmod_debug/vmod.vcc
+++ b/lib/libvmod_debug/vmod.vcc
@@ -31,3 +31,7 @@ Function VOID panic(STRING_LIST)
 Function STRING author(ENUM { phk, des, kristian, mithrandir })
 Function VOID test_priv_call(PRIV_CALL)
 Function VOID test_priv_vcl(PRIV_VCL)
+Object obj(STRING) {
+	Method STRING .foo(STRING)
+	Method TIME .date()
+}



From phk at varnish-cache.org  Wed Feb  6 14:34:40 2013
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Wed, 06 Feb 2013 15:34:40 +0100
Subject: [master] d5bd9b8 Forgot to git add this one.
Message-ID: <E1U365A-0002cs-D4@project.varnish-software.com>

commit d5bd9b8742869183ad629afc45894f5c0317e857
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 6 14:34:31 2013 +0000

    Forgot to git add this one.

diff --git a/lib/libvmod_debug/vmod_debug_obj.c b/lib/libvmod_debug/vmod_debug_obj.c
new file mode 100644
index 0000000..656de88
--- /dev/null
+++ b/lib/libvmod_debug/vmod_debug_obj.c
@@ -0,0 +1,82 @@
+/*-
+ * Copyright (c) 2013 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at FreeBSD.org>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include "config.h"
+
+#include <stdlib.h>
+
+#include "cache/cache.h"
+
+#include "vrt.h"
+#include "vcc_if.h"
+
+struct vmod_obj {
+	unsigned		magic;
+#define VMOD_OBJ_MAGIC		0xccbd9b77
+	int foobar;
+};
+
+VCL_VOID
+vmod_obj(struct req *req, struct vmod_obj **op, VCL_STRING s)
+{
+	struct vmod_obj *o;
+
+	(void)req;
+	(void)s;
+	AN(op);
+	if (*op == NULL) {
+		/* INIT */
+		ALLOC_OBJ(o, VMOD_OBJ_MAGIC);
+		*op = o;
+		o->foobar = 42;
+		AN(*op);
+	} else {
+		/* FINI */
+		FREE_OBJ(*op);
+		*op = NULL;
+	}
+}
+
+VCL_STRING
+vmod_obj__foo(struct req *req, struct vmod_obj *o, VCL_STRING s)
+{
+	(void)req;
+	(void)s;
+	CHECK_OBJ_NOTNULL(o, VMOD_OBJ_MAGIC);
+	assert(o->foobar == 42);
+	return ("BOO");
+}
+
+VCL_TIME
+vmod_obj__date(struct req *req, struct vmod_obj *o)
+{
+	(void)req;
+	CHECK_OBJ_NOTNULL(o, VMOD_OBJ_MAGIC);
+	assert(o->foobar == 42);
+	return (21.4);
+}



From phk at varnish-cache.org  Mon Feb 11 08:07:03 2013
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 11 Feb 2013 09:07:03 +0100
Subject: [master] 66b32cb prefix the vmod-obj struct with the vmod name,
 they are in global namespace for vmod.
Message-ID: <E1U4oPn-0007Z2-Or@project.varnish-software.com>

commit 66b32cbe0b4bd569865265440ade1a3d11913588
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 11 08:06:42 2013 +0000

    prefix the vmod-obj struct with the vmod name, they are in global
    namespace for vmod.

diff --git a/lib/libvcl/vmodtool.py b/lib/libvcl/vmodtool.py
index d1dfb68..3b71226 100755
--- a/lib/libvcl/vmodtool.py
+++ b/lib/libvcl/vmodtool.py
@@ -120,8 +120,9 @@ class vmod(object):
 	def add_func(self, fn):
 		self.funcs.append(fn)
 
-	def add_obj(self, fn):
-		self.objs.append(fn)
+	def add_obj(self, obj):
+		self.objs.append(obj)
+		obj.set_modnam(self.nam)
 
 	def c_proto(self, fo):
 		for o in self.objs:
@@ -307,22 +308,28 @@ class obj(object):
 		self.init_fini = None
 		self.methods = list()
 
+	def set_modnam(self, modnam):
+		self.st = "struct vmod_" + modnam + "_" + self.nam
+		self.init_fini.set_pfx(", " + self.st + " **")
+		for m in self.methods:
+			m.set_pfx(", " + self.st + " *")
+
 	def set_init_fini(self, f):
 		self.init_fini = f
 
 	def add_method(self, m):
 		self.methods.append(m)
 
-	def c_typedefs(self, modname):
+	def c_typedefs(self, modnam):
 		l = list()
 		l.append("/* Object " + self.nam + " */")
-		l.append(self.init_fini.c_typedef(modname) + "")
+		l.append(self.init_fini.c_typedef(modnam) + "")
 		for m in self.methods:
-			l.append(m.c_typedef(modname) + "")
+			l.append(m.c_typedef(modnam) + "")
 		return l
 
 	def c_proto(self, fo):
-		fo.write("struct vmod_" + self.nam + ";\n")
+		fo.write(self.st + ";\n")
 		self.init_fini.c_proto(fo)
 		for m in o.methods:
 			m.c_proto(fo)
@@ -343,7 +350,9 @@ class obj(object):
 
 	def c_strspec(self, modnam):
 		s = "\t/* Object " + self.nam + " */\n"
-		s += '\t"OBJ\\0' + self.init_fini.c_strspec(modnam) + '",\n'
+		s += '\t"OBJ\\0'
+		s += self.st + '\\0'
+		s += self.init_fini.c_strspec(modnam) + '",\n'
 		for m in self.methods:
 			s += '\t"METHOD\\0' + m.c_strspec(modnam) + '",\n'
 		return s
@@ -456,7 +465,6 @@ def parse_func(tl, rt_type = None, obj=None):
 def parse_obj(tl):
 	o = obj(tl[0].str)
 	f = parse_func(tl, "VOID")
-	f.set_pfx(", struct vmod_" + o.nam + " **")
 	o.set_init_fini(f)
 	t = tl.pop(0)
 	assert t.str == "{"
@@ -466,7 +474,6 @@ def parse_obj(tl):
 			break
 		assert t.str == "Method"
 		f = parse_func(tl, obj=o.nam)
-		f.set_pfx(", struct vmod_" + o.nam + " *")
 		o.add_method(f)
 	return o
 
diff --git a/lib/libvmod_debug/vmod_debug_obj.c b/lib/libvmod_debug/vmod_debug_obj.c
index 656de88..5ae2f3b 100644
--- a/lib/libvmod_debug/vmod_debug_obj.c
+++ b/lib/libvmod_debug/vmod_debug_obj.c
@@ -35,23 +35,23 @@
 #include "vrt.h"
 #include "vcc_if.h"
 
-struct vmod_obj {
+struct vmod_debug_obj {
 	unsigned		magic;
-#define VMOD_OBJ_MAGIC		0xccbd9b77
+#define VMOD_DEBUG_OBJ_MAGIC	0xccbd9b77
 	int foobar;
 };
 
 VCL_VOID
-vmod_obj(struct req *req, struct vmod_obj **op, VCL_STRING s)
+vmod_obj(struct req *req, struct vmod_debug_obj **op, VCL_STRING s)
 {
-	struct vmod_obj *o;
+	struct vmod_debug_obj *o;
 
 	(void)req;
 	(void)s;
 	AN(op);
 	if (*op == NULL) {
 		/* INIT */
-		ALLOC_OBJ(o, VMOD_OBJ_MAGIC);
+		ALLOC_OBJ(o, VMOD_DEBUG_OBJ_MAGIC);
 		*op = o;
 		o->foobar = 42;
 		AN(*op);
@@ -63,20 +63,20 @@ vmod_obj(struct req *req, struct vmod_obj **op, VCL_STRING s)
 }
 
 VCL_STRING
-vmod_obj__foo(struct req *req, struct vmod_obj *o, VCL_STRING s)
+vmod_obj__foo(struct req *req, struct vmod_debug_obj *o, VCL_STRING s)
 {
 	(void)req;
 	(void)s;
-	CHECK_OBJ_NOTNULL(o, VMOD_OBJ_MAGIC);
+	CHECK_OBJ_NOTNULL(o, VMOD_DEBUG_OBJ_MAGIC);
 	assert(o->foobar == 42);
 	return ("BOO");
 }
 
 VCL_TIME
-vmod_obj__date(struct req *req, struct vmod_obj *o)
+vmod_obj__date(struct req *req, struct vmod_debug_obj *o)
 {
 	(void)req;
-	CHECK_OBJ_NOTNULL(o, VMOD_OBJ_MAGIC);
+	CHECK_OBJ_NOTNULL(o, VMOD_DEBUG_OBJ_MAGIC);
 	assert(o->foobar == 42);
 	return (21.4);
 }



From phk at varnish-cache.org  Mon Feb 11 08:17:00 2013
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 11 Feb 2013 09:17:00 +0100
Subject: [master] 25c5715 Forward declare the vmod object struct
Message-ID: <E1U4oZQ-00088e-A7@project.varnish-software.com>

commit 25c5715a83d292e509edc8533c6820b524d120bf
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 11 08:16:44 2013 +0000

    Forward declare the vmod object struct

diff --git a/lib/libvcl/vmodtool.py b/lib/libvcl/vmodtool.py
index 3b71226..d5f5c30 100755
--- a/lib/libvcl/vmodtool.py
+++ b/lib/libvcl/vmodtool.py
@@ -323,6 +323,7 @@ class obj(object):
 	def c_typedefs(self, modnam):
 		l = list()
 		l.append("/* Object " + self.nam + " */")
+		l.append(self.st + ";")
 		l.append(self.init_fini.c_typedef(modnam) + "")
 		for m in self.methods:
 			l.append(m.c_typedef(modnam) + "")



From phk at varnish-cache.org  Mon Feb 11 10:27:29 2013
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 11 Feb 2013 11:27:29 +0100
Subject: [master] 8a73d95 This is a very shaky first prototype of VMOD objects.
Message-ID: <E1U4qbh-0000U9-Tr@project.varnish-software.com>

commit 8a73d9572edf80f092bd1c239f0dd1d7651a4876
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 11 10:26:49 2013 +0000

    This is a very shaky first prototype of VMOD objects.

diff --git a/include/tbl/symbol_kind.h b/include/tbl/symbol_kind.h
index 155d36e..3d426f9 100644
--- a/include/tbl/symbol_kind.h
+++ b/include/tbl/symbol_kind.h
@@ -28,14 +28,16 @@
  */
 
 /*lint -save -e525 -e539 */
-VCC_SYMB(NONE,		none,		"undefined")
-VCC_SYMB(VAR,		var,		"variable")
-VCC_SYMB(FUNC,		func,		"function")	/* VMOD function */
-VCC_SYMB(PROC,		proc,		"procedure")	/* VMOD procedure */
-VCC_SYMB(VMOD,		vmod,		"vmod")
-VCC_SYMB(ACL,		acl,		"acl")
-VCC_SYMB(SUB,		sub,		"sub")		/* VCL subroutine */
-VCC_SYMB(BACKEND,	backend,	"backend")
-VCC_SYMB(PROBE,		probe,		"probe")
-VCC_SYMB(WILDCARD,	wildcard,	"wildcard")
+VCC_SYMB(NONE,		none)
+VCC_SYMB(VAR,		var)
+VCC_SYMB(FUNC,		func)		/* VMOD function */
+VCC_SYMB(PROC,		proc)		/* VMOD procedure */
+VCC_SYMB(VMOD,		vmod)
+VCC_SYMB(ACL,		acl)
+VCC_SYMB(SUB,		sub)		/* VCL subroutine */
+VCC_SYMB(BACKEND,	backend)
+VCC_SYMB(PROBE,		probe)
+VCC_SYMB(WILDCARD,	wildcard)
+VCC_SYMB(OBJECT,	object)
+VCC_SYMB(METHOD,	method)
 /*lint -restore */
diff --git a/lib/libvcl/vcc_action.c b/lib/libvcl/vcc_action.c
index 1eec1c4..40b56f8 100644
--- a/lib/libvcl/vcc_action.c
+++ b/lib/libvcl/vcc_action.c
@@ -32,6 +32,8 @@
 
 #include "config.h"
 
+#include <string.h>
+
 #include "vcc_compile.h"
 
 /*--------------------------------------------------------------------*/
@@ -175,6 +177,76 @@ parse_unset(struct vcc *tl)
 /*--------------------------------------------------------------------*/
 
 static void
+parse_new(struct vcc *tl)
+{
+	struct symbol *sy1, *sy2, *sy3;
+	const char *p, *s_obj, *s_init, *s_struct, *s_fini;
+	char buf1[128];
+	char buf2[128];
+
+	vcc_NextToken(tl);
+	ExpectErr(tl, ID);
+	sy1 = VCC_FindSymbol(tl, tl->t, SYM_NONE);
+	XXXAZ(sy1);
+
+	sy1 = VCC_AddSymbolTok(tl, tl->t, SYM_NONE);	// XXX: NONE ?
+	XXXAN(sy1);
+	vcc_NextToken(tl);
+
+	ExpectErr(tl, '=');
+	vcc_NextToken(tl);
+
+	ExpectErr(tl, ID);
+	sy2 = VCC_FindSymbol(tl, tl->t, SYM_OBJECT);
+	XXXAN(sy2);
+
+	/*lint -save -e448 */
+	/* Split the first three args */
+	p = sy2->args;
+	s_obj = p;
+	p += strlen(p) + 1;
+	s_init = p;
+	while (p[0] != '\0' || p[1] != '\0')
+		p++;
+	p += 2;
+	s_struct = p;
+	p += strlen(p) + 1;
+	s_fini = p + strlen(p) + 1;
+	while (p[0] != '\0' || p[1] != '\0')
+		p++;
+	p += 2;
+
+	Fh(tl, 0, "static %s *%s;\n\n", s_struct, sy1->name);
+
+	vcc_NextToken(tl);
+
+	bprintf(buf1, ", &%s", sy1->name);
+	vcc_Eval_Func(tl, s_init, buf1, "ASDF", s_init + strlen(s_init) + 1);
+	Ff(tl, 0, "\t%s((struct req*)0, &%s);\n", s_fini, sy1->name);
+	ExpectErr(tl, ';');
+
+	bprintf(buf1, ", %s", sy1->name);
+	/* Split the methods from the args */
+	while (*p != '\0') {
+		p += strlen(s_obj);
+		bprintf(buf2, "%s%s", sy1->name, p);
+		sy3 = VCC_AddSymbolStr(tl, buf2, SYM_FUNC);
+		sy3->eval = vcc_Eval_SymFunc;
+		p += strlen(p) + 1;
+		sy3->cfunc = p;
+		p += strlen(p) + 1;
+		sy3->args = p;
+		sy3->extra = TlDup(tl, buf1);
+		while (p[0] != '\0' || p[1] != '\0')
+			p++;
+		p += 2;
+	}
+	/*lint -restore */
+}
+
+/*--------------------------------------------------------------------*/
+
+static void
 parse_ban(struct vcc *tl)
 {
 
@@ -313,6 +385,7 @@ static struct action_table {
 	{ "synthetic",		parse_synthetic, VCL_MET_ERROR },
 	{ "unset",		parse_unset },
 	{ "purge",		parse_purge, VCL_MET_MISS | VCL_MET_HIT },
+	{ "new",		parse_new, VCL_MET_INIT},
 	{ NULL,			NULL }
 };
 
diff --git a/lib/libvcl/vcc_compile.h b/lib/libvcl/vcc_compile.h
index 99a1169..162fef6 100644
--- a/lib/libvcl/vcc_compile.h
+++ b/lib/libvcl/vcc_compile.h
@@ -98,7 +98,7 @@ struct token {
 };
 
 enum symkind {
-#define VCC_SYMB(uu, ll, dd)	SYM_##uu,
+#define VCC_SYMB(uu, ll)	SYM_##uu,
 #include "tbl/symbol_kind.h"
 #undef VCC_SYMB
 };
@@ -131,6 +131,7 @@ struct symbol {
 
 	/* SYM_PROC, SYM_FUNC */
 	const char			*cfunc;
+	const char			*extra;
 	const char			*args;
 
 	/* SYM_VAR */
@@ -278,7 +279,9 @@ void vcc_Expr(struct vcc *tl, enum var_type typ);
 void vcc_Expr_Call(struct vcc *tl, const struct symbol *sym);
 void vcc_Expr_Init(struct vcc *tl);
 sym_expr_t vcc_Eval_Var;
-sym_expr_t vcc_Eval_Func;
+sym_expr_t vcc_Eval_SymFunc;
+void vcc_Eval_Func(struct vcc *tl, const char *cfunc, const char *extra,
+    const char *name, const char *args);
 sym_expr_t vcc_Eval_Backend;
 
 /* vcc_dir_dns.c */
diff --git a/lib/libvcl/vcc_expr.c b/lib/libvcl/vcc_expr.c
index 9e0f360..b6790aa 100644
--- a/lib/libvcl/vcc_expr.c
+++ b/lib/libvcl/vcc_expr.c
@@ -519,30 +519,31 @@ vcc_Eval_Var(struct vcc *tl, struct expr **e, const struct symbol *sym)
 /*--------------------------------------------------------------------
  */
 
-void
-vcc_Eval_Func(struct vcc *tl, struct expr **e, const struct symbol *sym)
+static void
+vcc_func(struct vcc *tl, struct expr **e, const char *cfunc,
+    const char *extra, const char *name, const char *args)
 {
 	const char *p, *r;
-	// const struct var *v;
 	struct expr *e1, *e2;
 	enum var_type fmt;
 	char buf[32];
 
-	assert(sym->kind == SYM_FUNC || sym->kind == SYM_PROC);
-	AN(sym->cfunc);
-	AN(sym->args);
-	SkipToken(tl, ID);
+	AN(cfunc);
+	AN(args);
+	AN(name);
 	SkipToken(tl, '(');
-	p = sym->args;
-	e2 = vcc_mk_expr(vcc_arg_type(&p), "%s(req\v+", sym->cfunc);
+	p = args;
+	if (extra == NULL)
+		extra = "";
+	e2 = vcc_mk_expr(vcc_arg_type(&p), "%s(req%s\v+", cfunc, extra);
 	while (*p != '\0') {
 		e1 = NULL;
 		fmt = vcc_arg_type(&p);
 		if (fmt == VOID && !strcmp(p, "PRIV_VCL")) {
-			r = strchr(sym->name, '.');
+			r = strchr(name, '.');
 			AN(r);
 			e1 = vcc_mk_expr(VOID, "&vmod_priv_%.*s",
-			    (int) (r - sym->name), sym->name);
+			    (int) (r - name), name);
 			p += strlen(p) + 1;
 		} else if (fmt == VOID && !strcmp(p, "PRIV_CALL")) {
 			bprintf(buf, "vmod_priv_%u", tl->nvmodpriv++);
@@ -573,7 +574,7 @@ vcc_Eval_Func(struct vcc *tl, struct expr **e, const struct symbol *sym)
 				p += strlen(p) + 1;
 			p++;
 			SkipToken(tl, ID);
-			if (*p != '\0')
+			if (*p != '\0')		/*lint !e448 */
 				SkipToken(tl, ',');
 		} else {
 			vcc_expr0(tl, &e1, fmt);
@@ -604,6 +605,42 @@ vcc_Eval_Func(struct vcc *tl, struct expr **e, const struct symbol *sym)
 }
 
 /*--------------------------------------------------------------------
+ */
+
+void
+vcc_Eval_Func(struct vcc *tl, const char *cfunc,
+    const char *extra, const char *name, const char *args)
+{
+	struct expr *e = NULL;
+	struct token *t1;
+
+	t1 = tl->t;
+	vcc_func(tl, &e, cfunc, extra, name, args);
+	if (!tl->err) {
+		vcc_expr_fmt(tl->fb, tl->indent, e);
+		VSB_cat(tl->fb, ";\n");
+	} else if (t1 != tl->t) {
+		vcc_ErrWhere2(tl, t1, tl->t);
+	}
+	vcc_delete_expr(e);
+}
+
+/*--------------------------------------------------------------------
+ */
+
+void
+vcc_Eval_SymFunc(struct vcc *tl, struct expr **e, const struct symbol *sym)
+{
+
+	assert(sym->kind == SYM_FUNC || sym->kind == SYM_PROC);
+	AN(sym->cfunc);
+	AN(sym->name);
+	AN(sym->args);
+	SkipToken(tl, ID);
+	vcc_func(tl, e, sym->cfunc, sym->extra, sym->name, sym->args);
+}
+
+/*--------------------------------------------------------------------
  * SYNTAX:
  *    Expr4:
  *	'(' Expr0 ')'
@@ -1170,7 +1207,7 @@ vcc_Expr_Call(struct vcc *tl, const struct symbol *sym)
 
 	t1 = tl->t;
 	e = NULL;
-	vcc_Eval_Func(tl, &e, sym);
+	vcc_Eval_SymFunc(tl, &e, sym);
 	if (!tl->err) {
 		vcc_expr_fmt(tl->fb, tl->indent, e);
 		VSB_cat(tl->fb, ";\n");
diff --git a/lib/libvcl/vcc_symb.c b/lib/libvcl/vcc_symb.c
index 691f6b8..c3d8a13 100644
--- a/lib/libvcl/vcc_symb.c
+++ b/lib/libvcl/vcc_symb.c
@@ -40,7 +40,7 @@ const char *
 VCC_SymKind(struct vcc *tl, const struct symbol *s)
 {
 	switch(s->kind) {
-#define VCC_SYMB(uu, ll, dd)	case SYM_##uu: return(dd);
+#define VCC_SYMB(uu, ll)	case SYM_##uu: return(#ll);
 #include "tbl/symbol_kind.h"
 #undef VCC_SYMB
 	default:
diff --git a/lib/libvcl/vcc_vmod.c b/lib/libvcl/vcc_vmod.c
index 839577d..42fc1b6 100644
--- a/lib/libvcl/vcc_vmod.c
+++ b/lib/libvcl/vcc_vmod.c
@@ -168,10 +168,12 @@ vcc_ParseImport(struct vcc *tl)
 	for (; *spec != NULL; spec++) {
 		p = *spec;
 		if (!strcmp(p, "OBJ")) {
-			// Nothing yet
-		} else if (!strcmp(p, "METHOD")) {
-			// Nothing yet
+			p += strlen(p) + 1;
+			sym = VCC_AddSymbolStr(tl, p, SYM_OBJECT);
+			XXXAN(sym);
+			sym->args = p;
 		} else if (!strcmp(p, "FINI")) {
+			p += strlen(p) + 1;
 			// Nothing yet
 		} else if (!strcmp(p, "INIT")) {
 			p += strlen(p) + 1;
@@ -181,7 +183,7 @@ vcc_ParseImport(struct vcc *tl)
 			sym = VCC_AddSymbolStr(tl, p, SYM_FUNC);
 			ERRCHK(tl);
 			AN(sym);
-			sym->eval = vcc_Eval_Func;
+			sym->eval = vcc_Eval_SymFunc;
 			p += strlen(p) + 1;
 			sym->cfunc = p;
 			p += strlen(p) + 1;
diff --git a/lib/libvcl/vmodtool.py b/lib/libvcl/vmodtool.py
index d5f5c30..4076597 100755
--- a/lib/libvcl/vmodtool.py
+++ b/lib/libvcl/vmodtool.py
@@ -248,7 +248,7 @@ class func(object):
 			raise Exception(
 			    "Return type '%s' not a valid type" % retval)
 		self.nam = nam
-		self.cnam = nam.replace(".", "__")
+		self.cnam = nam.replace(".", "_")
 		self.al = al
 		self.retval = retval
 		self.pfx = None
@@ -299,23 +299,28 @@ class func(object):
 		for a in self.al:
 			s += a.c_strspec()
 		return s
-		
+
 #######################################################################
 
 class obj(object):
 	def __init__(self, nam):
 		self.nam = nam
-		self.init_fini = None
+		self.init = None
+		self.fini = None
 		self.methods = list()
 
 	def set_modnam(self, modnam):
 		self.st = "struct vmod_" + modnam + "_" + self.nam
-		self.init_fini.set_pfx(", " + self.st + " **")
+		self.init.set_pfx(", " + self.st + " **")
+		self.fini.set_pfx(", " + self.st + " **")
 		for m in self.methods:
 			m.set_pfx(", " + self.st + " *")
 
-	def set_init_fini(self, f):
-		self.init_fini = f
+	def set_init(self, f):
+		self.init = f
+		self.fini = func(f.nam, "VOID", [])
+		self.init.cnam += "__init"
+		self.fini.cnam += "__fini"
 
 	def add_method(self, m):
 		self.methods.append(m)
@@ -324,38 +329,44 @@ class obj(object):
 		l = list()
 		l.append("/* Object " + self.nam + " */")
 		l.append(self.st + ";")
-		l.append(self.init_fini.c_typedef(modnam) + "")
+		l.append(self.init.c_typedef(modnam) + "")
+		l.append(self.fini.c_typedef(modnam) + "")
 		for m in self.methods:
 			l.append(m.c_typedef(modnam) + "")
 		return l
 
 	def c_proto(self, fo):
 		fo.write(self.st + ";\n")
-		self.init_fini.c_proto(fo)
+		self.init.c_proto(fo)
+		self.fini.c_proto(fo)
 		for m in o.methods:
 			m.c_proto(fo)
 
 	def c_struct(self, modnam):
 		s = "\t/* Object " + self.nam + " */\n"
-		s += self.init_fini.c_struct(modnam)
+		s += self.init.c_struct(modnam)
+		s += self.fini.c_struct(modnam)
 		for m in self.methods:
 			s += m.c_struct(modnam)
 		return s
 
 	def c_initializer(self):
 		s = "\t/* Object " + self.nam + " */\n"
-		s += self.init_fini.c_initializer()
+		s += self.init.c_initializer()
+		s += self.fini.c_initializer()
 		for m in self.methods:
 			s += m.c_initializer()
 		return s
 
 	def c_strspec(self, modnam):
 		s = "\t/* Object " + self.nam + " */\n"
-		s += '\t"OBJ\\0'
-		s += self.st + '\\0'
-		s += self.init_fini.c_strspec(modnam) + '",\n'
+		s += '\t"OBJ\\0"\n'
+		s += '\t\t"' + self.init.c_strspec(modnam) + '\\0"\n'
+		s += '\t\t"' + self.st + '\\0"\n'
+		s += '\t\t"' + self.fini.c_strspec(modnam) + '\\0"\n'
 		for m in self.methods:
-			s += '\t"METHOD\\0' + m.c_strspec(modnam) + '",\n'
+			s += '\t\t"' + m.c_strspec(modnam) + '\\0"\n'
+		s += '\t\t"\\0",\n'
 		return s
 
 #######################################################################
@@ -466,7 +477,7 @@ def parse_func(tl, rt_type = None, obj=None):
 def parse_obj(tl):
 	o = obj(tl[0].str)
 	f = parse_func(tl, "VOID")
-	o.set_init_fini(f)
+	o.set_init(f)
 	t = tl.pop(0)
 	assert t.str == "{"
 	while True:
diff --git a/lib/libvmod_debug/vmod_debug_obj.c b/lib/libvmod_debug/vmod_debug_obj.c
index 5ae2f3b..d44e634 100644
--- a/lib/libvmod_debug/vmod_debug_obj.c
+++ b/lib/libvmod_debug/vmod_debug_obj.c
@@ -42,28 +42,34 @@ struct vmod_debug_obj {
 };
 
 VCL_VOID
-vmod_obj(struct req *req, struct vmod_debug_obj **op, VCL_STRING s)
+vmod_obj__init(struct req *req, struct vmod_debug_obj **op, VCL_STRING s)
 {
 	struct vmod_debug_obj *o;
 
 	(void)req;
 	(void)s;
 	AN(op);
-	if (*op == NULL) {
-		/* INIT */
-		ALLOC_OBJ(o, VMOD_DEBUG_OBJ_MAGIC);
-		*op = o;
-		o->foobar = 42;
-		AN(*op);
-	} else {
-		/* FINI */
-		FREE_OBJ(*op);
-		*op = NULL;
-	}
+	AZ(*op);
+	ALLOC_OBJ(o, VMOD_DEBUG_OBJ_MAGIC);
+	AN(o);
+	*op = o;
+	o->foobar = 42;
+	AN(*op);
 }
 
-VCL_STRING
-vmod_obj__foo(struct req *req, struct vmod_debug_obj *o, VCL_STRING s)
+VCL_VOID
+vmod_obj__fini(struct req *req, struct vmod_debug_obj **op)
+{
+
+	(void)req;
+	AN(op);
+	AN(*op);
+	FREE_OBJ(*op);
+	*op = NULL;
+}
+
+VCL_STRING __match_proto__()
+vmod_obj_foo(struct req *req, struct vmod_debug_obj *o, VCL_STRING s)
 {
 	(void)req;
 	(void)s;
@@ -72,8 +78,8 @@ vmod_obj__foo(struct req *req, struct vmod_debug_obj *o, VCL_STRING s)
 	return ("BOO");
 }
 
-VCL_TIME
-vmod_obj__date(struct req *req, struct vmod_debug_obj *o)
+VCL_TIME __match_proto__()
+vmod_obj_date(struct req *req, struct vmod_debug_obj *o)
 {
 	(void)req;
 	CHECK_OBJ_NOTNULL(o, VMOD_DEBUG_OBJ_MAGIC);



From phk at varnish-cache.org  Mon Feb 11 12:43:31 2013
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 11 Feb 2013 13:43:31 +0100
Subject: [master] 45af976 Discontinue words as fixed tokens ('if',
 'else' etc) and use ID's instead.
Message-ID: <E1U4sjL-0001u1-1x@project.varnish-software.com>

commit 45af9764189ec55f0626a44ff64df7bef5d87d41
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 11 12:42:29 2013 +0000

    Discontinue words as fixed tokens ('if', 'else' etc) and use ID's instead.
    
    Fixes #1259

diff --git a/bin/varnishtest/tests/v00018.vtc b/bin/varnishtest/tests/v00018.vtc
index 6fe35fb..7bc0b05 100644
--- a/bin/varnishtest/tests/v00018.vtc
+++ b/bin/varnishtest/tests/v00018.vtc
@@ -84,9 +84,9 @@ varnish v1 -errvcl {Only http header variables can be unset.} {
 	sub vcl_fetch { unset beresp.do_gzip; }
 }
 
-varnish v1 -errvcl {Unknown token 'if' when looking for STRING} {
+varnish v1 -errvcl {Unknown token '<<' when looking for STRING} {
 	backend b { .host = "127.0.0.1"; }
-	sub vcl_recv { ban (if); }
+	sub vcl_recv { ban (<<); }
 }
 
 varnish v1 -errvcl {Expected an action, 'if', '{' or '}'} {
@@ -104,7 +104,7 @@ varnish v1 -errvcl {Expected an action, 'if', '{' or '}'} {
 	sub vcl_recv { kluf ; }
 }
 
-varnish v1 -errvcl {Unknown token 'if' when looking for STRING_LIST} {
+varnish v1 -errvcl {Unknown token '<<' when looking for STRING_LIST} {
 	backend b { .host = "127.0.0.1"; }
-	sub vcl_error { synthetic if "foo"; }
+	sub vcl_error { synthetic << "foo"; }
 }
diff --git a/lib/libvcl/generate.py b/lib/libvcl/generate.py
index 2064b82..a12f296 100755
--- a/lib/libvcl/generate.py
+++ b/lib/libvcl/generate.py
@@ -62,11 +62,6 @@ tokens = {
 	"T_MUL":	"*=",
 	"T_DIV":	"/=",
 	"T_NOMATCH":	"!~",
-	"T_INCLUDE":	"include",
-	"T_IF":		"if",
-	"T_ELSEIF":	"elseif",
-	"T_ELSIF":	"elsif",
-	"T_ELSE":	"else",
 
 	# Single char tokens, for convenience on one line
 	None:		"{}()*+-/%><=;!&.|~,",
diff --git a/lib/libvcl/vcc_compile.c b/lib/libvcl/vcc_compile.c
index 318c1b9..e585e67 100644
--- a/lib/libvcl/vcc_compile.c
+++ b/lib/libvcl/vcc_compile.c
@@ -435,7 +435,7 @@ vcc_resolve_includes(struct vcc *tl)
 	struct source *sp;
 
 	VTAILQ_FOREACH(t, &tl->tokens, list) {
-		if (t->tok != T_INCLUDE)
+		if (t->tok != ID || !vcc_IdIs(t, "include"))
 			continue;
 
 		t1 = VTAILQ_NEXT(t, list);
diff --git a/lib/libvcl/vcc_parse.c b/lib/libvcl/vcc_parse.c
index da50811..f656cbd 100644
--- a/lib/libvcl/vcc_parse.c
+++ b/lib/libvcl/vcc_parse.c
@@ -84,37 +84,48 @@ static void
 vcc_IfStmt(struct vcc *tl)
 {
 
-	SkipToken(tl, T_IF);
+	SkipToken(tl, ID);
 	Fb(tl, 1, "if ");
 	vcc_Conditional(tl);
 	ERRCHK(tl);
 	L(tl, vcc_Compound(tl));
 	ERRCHK(tl);
-	while (1) {
-		switch (tl->t->tok) {
-		case T_ELSE:
+	while (tl->t->tok == ID) {
+		if (vcc_IdIs(tl->t, "else")) {
 			vcc_NextToken(tl);
-			if (tl->t->tok != T_IF) {
+			if (tl->t->tok == '{') {
 				Fb(tl, 1, "else\n");
 				L(tl, vcc_Compound(tl));
 				ERRCHK(tl);
 				return;
 			}
-			/* FALLTHROUGH */
-		case T_ELSEIF:
-		case T_ELSIF:
+			if (tl->t->tok != ID || !vcc_IdIs(tl->t, "if")) {
+				VSB_printf(tl->sb,
+				    "'else' must be followed by 'if' or '{'\n");
+				vcc_ErrWhere(tl, tl->t);
+				return;
+			}
+			Fb(tl, 1, "else if ");
+			vcc_NextToken(tl);
+			vcc_Conditional(tl);
+			ERRCHK(tl);
+			L(tl, vcc_Compound(tl));
+			ERRCHK(tl);
+			
+		} else if (vcc_IdIs(tl->t, "elseif") ||
+		     vcc_IdIs(tl->t, "elsif") ||
+		     vcc_IdIs(tl->t, "elif")) {
 			Fb(tl, 1, "else if ");
 			vcc_NextToken(tl);
 			vcc_Conditional(tl);
 			ERRCHK(tl);
 			L(tl, vcc_Compound(tl));
 			ERRCHK(tl);
+		} else {
 			break;
-		default:
-			C(tl, ";");
-			return;
 		}
 	}
+	C(tl, ";");
 }
 
 /*--------------------------------------------------------------------
@@ -144,9 +155,6 @@ vcc_Compound(struct vcc *tl)
 		case '{':
 			vcc_Compound(tl);
 			break;
-		case T_IF:
-			vcc_IfStmt(tl);
-			break;
 		case '}':
 			vcc_NextToken(tl);
 			tl->indent -= INDENT;
@@ -170,11 +178,16 @@ vcc_Compound(struct vcc *tl)
 			tl->err = 1;
 			return;
 		case ID:
-			i = vcc_ParseAction(tl);
-			ERRCHK(tl);
-			if (i) {
-				SkipToken(tl, ';');
+			if (vcc_IdIs(tl->t, "if")) {
+				vcc_IfStmt(tl);
 				break;
+			} else {
+				i = vcc_ParseAction(tl);
+				ERRCHK(tl);
+				if (i) {
+					SkipToken(tl, ';');
+					break;
+				}
 			}
 			/* FALLTHROUGH */
 		default:



From phk at varnish-cache.org  Mon Feb 11 19:19:51 2013
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Mon, 11 Feb 2013 20:19:51 +0100
Subject: [master] ba9f21b Don't tilt jenkins.
Message-ID: <E1U4yut-0008Pr-B8@project.varnish-software.com>

commit ba9f21b089ab1eb831bcac55bd03697e61606ec8
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 11 19:19:32 2013 +0000

    Don't tilt jenkins.

diff --git a/lib/libvcl/vmodtool.py b/lib/libvcl/vmodtool.py
index 4076597..9447549 100755
--- a/lib/libvcl/vmodtool.py
+++ b/lib/libvcl/vmodtool.py
@@ -37,8 +37,6 @@
 # as a string, suitable for inclusion in the C-source of the compile VCL
 # program.
 
-from __future__ import print_function
-
 import sys
 import re
 



From phk at varnish-cache.org  Tue Feb 12 12:28:05 2013
From: phk at varnish-cache.org (Poul-Henning Kamp)
Date: Tue, 12 Feb 2013 13:28:05 +0100
Subject: [master] fb07bc7 If we are going to start the client automatically,
 test that the accept-sockets can actually be bound before we muck
 about with daemon() or SHM files.
Message-ID: <E1U5Exx-00076e-7H@project.varnish-software.com>

commit fb07bc720fb190f139d00f6c7e77e4487e6175bb
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 12 12:27:21 2013 +0000

    If we are going to start the client automatically, test that the
    accept-sockets can actually be bound before we muck about with
    daemon() or SHM files.
    
    Belived to be implicated in #1262

diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h
index b66e729..8f0fd3b 100644
--- a/bin/varnishd/mgt/mgt.h
+++ b/bin/varnishd/mgt/mgt.h
@@ -44,6 +44,8 @@ void MGT_Run(void);
 void mgt_stop_child(void);
 void mgt_got_fd(int fd);
 void MGT_Child_Cli_Fail(void);
+int MGT_open_sockets(void);
+void MGT_close_sockets(void);
 
 /* mgt_cli.c */
 
diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c
index 3def397..fbb10ad 100644
--- a/bin/varnishd/mgt/mgt_child.c
+++ b/bin/varnishd/mgt/mgt_child.c
@@ -222,13 +222,13 @@ mgt_child_inherit(int fd, const char *what)
  * (The child is priv-sep'ed, so it can't do it.)
  */
 
-static int
-open_sockets(void)
+int
+MGT_open_sockets(void)
 {
-	struct listen_sock *ls, *ls2;
+	struct listen_sock *ls;
 	int good = 0;
 
-	VTAILQ_FOREACH_SAFE(ls, &heritage.socks, list, ls2) {
+	VTAILQ_FOREACH(ls, &heritage.socks, list) {
 		if (ls->sock >= 0) {
 			good++;
 			continue;
@@ -248,8 +248,8 @@ open_sockets(void)
 
 /*--------------------------------------------------------------------*/
 
-static void
-close_sockets(void)
+void
+MGT_close_sockets(void)
 {
 	struct listen_sock *ls;
 
@@ -326,7 +326,7 @@ mgt_launch_child(struct cli *cli)
 	if (child_state != CH_STOPPED && child_state != CH_DIED)
 		return;
 
-	if (open_sockets() != 0) {
+	if (MGT_open_sockets() != 0) {
 		child_state = CH_STOPPED;
 		if (cli != NULL) {
 			VCLI_SetResult(cli, CLIS_CANT);
@@ -410,7 +410,7 @@ mgt_launch_child(struct cli *cli)
 	mgt_child_inherit(heritage.cli_out, NULL);
 	closex(&heritage.cli_out);
 
-	close_sockets();
+	MGT_close_sockets();
 
 	child_std_vlu = VLU_New(NULL, child_line, 0);
 	AN(child_std_vlu);
diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c
index b5829a9..b41133c 100644
--- a/bin/varnishd/mgt/mgt_main.c
+++ b/bin/varnishd/mgt/mgt_main.c
@@ -602,6 +602,15 @@ main(int argc, char * const *argv)
 	if (C_flag)
 		exit (0);
 
+	if (!d_flag) {
+		if (MGT_open_sockets()) {
+			fprintf(stderr,
+			    "Failed to open (any) accept sockets.\n");
+			exit(1);
+		}
+		MGT_close_sockets();
+	}
+
 	/* If no -s argument specified, process default -s argument */
 	if (!s_arg_given)
 		STV_Config(s_arg);



From tfheen at varnish-cache.org  Tue Feb 12 12:46:14 2013
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Tue, 12 Feb 2013 13:46:14 +0100
Subject: [3.0] 4ba5ac1 Typo
Message-ID: <E1U5FFW-0000NW-C4@project.varnish-software.com>

commit 4ba5ac11e0d68fee6626db9382afc097a683e556
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Tue Feb 12 13:45:17 2013 +0100

    Typo

diff --git a/doc/sphinx/tutorial/introduction.rst b/doc/sphinx/tutorial/introduction.rst
index 0d43623..a44dd73 100644
--- a/doc/sphinx/tutorial/introduction.rst
+++ b/doc/sphinx/tutorial/introduction.rst
@@ -29,9 +29,9 @@ response should be altered. You can read more about this in our
 tutorial.
 
 
-Supported plattforms
+Supported platforms
 ~~~~~~~~~~~~~~~~~~~~
 
 Varnish is written to run on modern versions of Linux and FreeBSD and
-the best experience is had on those plattforms. Thanks to our
+the best experience is had on those platforms. Thanks to our
 contributors it also runs on NetBSD, OpenBSD and OS X.



From tfheen at varnish-cache.org  Thu Feb 14 12:39:18 2013
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Thu, 14 Feb 2013 13:39:18 +0100
Subject: [master] e4d79ff Typo
Message-ID: <E1U5y5u-0007aa-AS@project.varnish-software.com>

commit e4d79ff8b97c82e27e04050a345bb1c46e974b51
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Tue Feb 12 13:45:17 2013 +0100

    Typo

diff --git a/doc/sphinx/tutorial/introduction.rst b/doc/sphinx/tutorial/introduction.rst
index 3968811..c0a3a92 100644
--- a/doc/sphinx/tutorial/introduction.rst
+++ b/doc/sphinx/tutorial/introduction.rst
@@ -26,11 +26,11 @@ In such a policy you can decide what content you want to serve, from
 where you want to get the content and how the request or response
 should be altered. 
 
-Supported plattforms
+Supported platforms
 --------------------
 
 Varnish is written to run on modern versions of Linux and FreeBSD and
-the best experience is had on those plattforms. Thanks to our
+the best experience is had on those platforms. Thanks to our
 contributors it also runs on NetBSD, OpenBSD and OS X.
 
 About the Varnish development process



From tfheen at varnish-cache.org  Thu Feb 14 12:39:18 2013
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Thu, 14 Feb 2013 13:39:18 +0100
Subject: [master] 007a908 Re-enable -b support for varnishncsa
Message-ID: <E1U5y5u-0007af-BX@project.varnish-software.com>

commit 007a90854f1154c3d79fe8d8f6ed4cafb17bd1d3
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Thu Feb 14 09:19:56 2013 +0100

    Re-enable -b support for varnishncsa
    
    This isn't really functional right now, due to the log record changes,
    but it will allow you to specify -b again and you can have a different
    format for backend requests than for client requests.

diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c
index e9359cd..309019d 100644
--- a/bin/varnishncsa/varnishncsa.c
+++ b/bin/varnishncsa/varnishncsa.c
@@ -117,7 +117,8 @@ static size_t nll;
 
 static int m_flag = 0;
 
-static const char *format;
+static const char *c_format;
+static const char *b_format;
 
 static int
 isprefix(const char *str, const char *prefix, const char *end,
@@ -279,6 +280,8 @@ collect_backend(struct logline *lp, enum VSL_tag_e tag, unsigned spec,
     const char *ptr, unsigned len)
 {
 	const char *end, *next, *split;
+	struct hdr *h;
+	size_t l;
 
 	assert(spec & VSL_S_BACKEND);
 	end = ptr + len;
@@ -346,38 +349,36 @@ collect_backend(struct logline *lp, enum VSL_tag_e tag, unsigned spec,
 		trimline(&lp->df_s, ptr, end);
 		break;
 
-	case SLT_BerespHeader:
-		if (!lp->active)
-			break;
-		if (isprefix(ptr, "content-length:", end, &next))
-			trimline(&lp->df_b, next, end);
-		else if (isprefix(ptr, "date:", end, &next) &&
-			 strptime(next, "%a, %d %b %Y %T", &lp->df_t) == NULL) {
-			clean_logline(lp);
-		}
-		break;
-
 	case SLT_BereqHeader:
+	case SLT_BerespHeader:
 		if (!lp->active)
 			break;
 		split = memchr(ptr, ':', len);
 		if (split == NULL)
 			break;
-		if (isprefix(ptr, "authorization:", end, &next) &&
-		    isprefix(next, "basic", end, &next)) {
-			trimline(&lp->df_u, next, end);
+		if (tag == SLT_BerespHeader) {
+			if (isprefix(ptr, "content-length:", end, &next))
+				trimline(&lp->df_b, next, end);
+			else if (isprefix(ptr, "date:", end, &next) &&
+				 strptime(next, "%a, %d %b %Y %T", &lp->df_t) == NULL) {
+				clean_logline(lp);
+			}
 		} else {
-			struct hdr *h;
-			size_t l;
-			h = calloc(1, sizeof(struct hdr));
-			AN(h);
-			AN(split);
-			l = strlen(split);
-			trimline(&h->key, ptr, split-1);
-			trimline(&h->value, split+1, split+l-1);
-			VTAILQ_INSERT_HEAD(&lp->req_headers, h, list);
+			if (isprefix(ptr, "authorization:", end, &next) &&
+			    isprefix(next, "basic", end, &next)) {
+				trimline(&lp->df_u, next, end);
+			}
 		}
-		break;
+		h = calloc(1, sizeof(struct hdr));
+		AN(h);
+		AN(split);
+		l = strlen(split);
+		trimline(&h->key, ptr, split-1);
+		trimline(&h->value, split+1, split+l-1);
+		if (tag == SLT_BereqHeader)
+			VTAILQ_INSERT_HEAD(&lp->req_headers, h, list);
+		else
+			VTAILQ_INSERT_HEAD(&lp->resp_headers, h, list);
 
 	case SLT_BackendReuse:
 	case SLT_BackendClose:
@@ -585,6 +586,7 @@ h_ncsa(void *priv, enum VSL_tag_e tag, unsigned fd,
 	char *q, tbuf[64];
 	const char *p;
 	struct vsb *os;
+	const char *format;
 
 	/* XXX: Ignore fd's outside 65536 */
 	if (fd >= 65536)
@@ -607,11 +609,12 @@ h_ncsa(void *priv, enum VSL_tag_e tag, unsigned fd,
 		assert(ll[fd] != NULL);
 	}
 	lp = ll[fd];
-
 	if (spec & VSL_S_BACKEND) {
 		collect_backend(lp, tag, spec, ptr, len);
+		format = b_format;
 	} else if (spec & VSL_S_CLIENT) {
 		collect_client(lp, tag, spec, ptr, len);
+		format = c_format;
 	} else {
 		/* huh? */
 		return (reopen);
@@ -863,38 +866,45 @@ int
 main(int argc, char *argv[])
 {
 	int c;
-	int a_flag = 0, D_flag = 0, format_flag = 0;
+	int a_flag = 0, D_flag = 0, format_flag = 0, b_flag = 0;
 	const char *P_arg = NULL;
 	const char *w_arg = NULL;
 	struct vpf_fh *pfh = NULL;
 	FILE *of;
-	format = "%h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\"";
+	c_format = "%h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\"";
+	b_format = "%h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\"";
 
 	vd = VSM_New();
 
-	while ((c = getopt(argc, argv, VSL_ARGS "aDP:Vw:fF:")) != -1) {
+	while ((c = getopt(argc, argv, VSL_ARGS "aDP:Vw:fF:B:C:")) != -1) {
 		switch (c) {
 		case 'a':
 			a_flag = 1;
 			break;
 		case 'f':
-			if (format_flag) {
-				fprintf(stderr,
-				    "-f and -F can not be combined\n");
-				exit(1);
-			}
-			format = "%{X-Forwarded-For}i %l %u %t \"%r\""
-			    " %s %b \"%{Referer}i\" \"%{User-agent}i\"";
-			format_flag = 1;
-			break;
 		case 'F':
 			if (format_flag) {
 				fprintf(stderr,
-				    "-f and -F can not be combined\n");
+				    "-f, -F, -B or -C can not be combined\n");
 				exit(1);
 			}
 			format_flag = 1;
-			format = optarg;
+			switch (c) {
+			case 'f':
+				b_format = c_format =
+					"%{X-Forwarded-For}i %l %u %t \"%r\""
+					" %s %b \"%{Referer}i\" \"%{User-agent}i\"";
+				break;
+			case 'F':
+				b_format = c_format = optarg;
+				break;
+			case 'B':
+				b_format = optarg;
+				break;
+			case 'C':
+				c_format = optarg;
+				break;
+			}
 			break;
 		case 'D':
 			D_flag = 1;
@@ -909,8 +919,10 @@ main(int argc, char *argv[])
 			w_arg = optarg;
 			break;
 		case 'b':
-			fprintf(stderr, "-b is not valid for varnishncsa\n");
-			exit(1);
+			b_flag = 1;
+			if (VSL_Arg(vd, c, optarg) > 0)
+				break;
+			usage();
 			break;
 		case 'i':
 			fprintf(stderr, "-i is not valid for varnishncsa\n");
@@ -933,7 +945,8 @@ main(int argc, char *argv[])
 		}
 	}
 
-	VSL_Arg(vd, 'c', optarg);
+	if (! b_flag)
+		VSL_Arg(vd, 'c', optarg);
 
 	if (VSM_Open(vd)) {
 		fprintf(stderr, "%s\n", VSM_Error(vd));



From martin at varnish-cache.org  Fri Feb 15 13:58:00 2013
From: martin at varnish-cache.org (Martin Blix Grydeland)
Date: Fri, 15 Feb 2013 14:58:00 +0100
Subject: [master] 86dac72 Keep track of persisted ban byte usage,
 and it's fragmentation.
Message-ID: <E1U6Lnc-0005Ge-BY@project.varnish-software.com>

commit 86dac72326eca5f5cc9bc57f676f1c67b5f625fe
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Fri Dec 14 12:17:47 2012 +0100

    Keep track of persisted ban byte usage, and it's fragmentation.

diff --git a/bin/varnishd/cache/cache_ban.c b/bin/varnishd/cache/cache_ban.c
index 35c520b..4cf3d1b 100644
--- a/bin/varnishd/cache/cache_ban.c
+++ b/bin/varnishd/cache/cache_ban.c
@@ -262,13 +262,16 @@ ban_equal(const uint8_t *bs1, const uint8_t *bs2)
 static void
 ban_mark_gone(struct ban *b)
 {
+	unsigned ln;
 
 	CHECK_OBJ_NOTNULL(b, BAN_MAGIC);
+	ln = ban_len(b->spec);
 	b->flags |= BAN_F_GONE;
 	b->spec[BANS_FLAGS] |= BANS_FLAG_GONE;
 	VWMB();
 	vbe32enc(b->spec + BANS_LENGTH, 0);
 	VSC_C_main->bans_gone++;
+	VSC_C_main->bans_persisted_fragmentation += ln - ban_len(b->spec);
 }
 
 /*--------------------------------------------------------------------
@@ -484,6 +487,7 @@ BAN_Insert(struct ban *b)
 	 * the stevedores has been opened, but before any new objects
 	 * can have entered the cache (thus no objects in the mean
 	 * time depending on ban_magic in the list) */
+	VSC_C_main->bans_persisted_bytes += ln;
 	if (b != ban_magic)
 		ban_info(BI_NEW, b->spec, ln); /* Notify stevedores */
 	Lck_Unlock(&ban_mtx);
@@ -584,17 +588,21 @@ ban_export(void)
 {
 	struct ban *b;
 	struct vsb vsb;
+	unsigned ln;
 
 	Lck_AssertHeld(&ban_mtx);
-	/* XXX: Use the ban entry size measurements to hit the target
-	 * and avoid multiple allocations */
-	AN(VSB_new(&vsb, NULL, 64 * VSC_C_main->bans, VSB_AUTOEXTEND));
+	ln = VSC_C_main->bans_persisted_bytes -
+	    VSC_C_main->bans_persisted_fragmentation;
+	AN(VSB_new(&vsb, NULL, ln, VSB_AUTOEXTEND));
 	VTAILQ_FOREACH_REVERSE(b, &ban_head, banhead_s, list) {
 		AZ(VSB_bcat(&vsb, b->spec, ban_len(b->spec)));
 	}
 	AZ(VSB_finish(&vsb));
+	assert(VSB_len(&vsb) == ln);
 	STV_BanExport((const uint8_t *)VSB_data(&vsb), VSB_len(&vsb));
 	VSB_delete(&vsb);
+	VSC_C_main->bans_persisted_bytes = ln;
+	VSC_C_main->bans_persisted_fragmentation = 0;
 }
 
 static void
@@ -665,6 +673,7 @@ ban_reload(const uint8_t *ban, unsigned len)
 		VTAILQ_INSERT_TAIL(&ban_head, b2, list);
 	else
 		VTAILQ_INSERT_BEFORE(b, b2, list);
+	VSC_C_main->bans_persisted_bytes += len;
 
 	/* Hunt down older duplicates */
 	for (b = VTAILQ_NEXT(b2, list); b != NULL; b = VTAILQ_NEXT(b, list)) {
@@ -930,6 +939,8 @@ ban_cleantail(void)
 			VSC_C_main->bans--;
 			VSC_C_main->bans_deleted++;
 			VTAILQ_REMOVE(&ban_head, b, list);
+			VSC_C_main->bans_persisted_fragmentation +=
+			    ban_len(b->spec);
 			ban_info(BI_DROP, b->spec, ban_len(b->spec));
 		} else {
 			b = NULL;
diff --git a/include/tbl/vsc_f_main.h b/include/tbl/vsc_f_main.h
index c266f50..f487471 100644
--- a/include/tbl/vsc_f_main.h
+++ b/include/tbl/vsc_f_main.h
@@ -505,6 +505,15 @@ VSC_F(bans_dups,		uint64_t, 0, 'c',
     "Bans superseded by other bans",
 	"Count of bans replaced by later identical bans."
 )
+VSC_F(bans_persisted_bytes,	uint64_t, 0, 'g',
+    "Bytes used by the persisted ban lists",
+        "Number of bytes used by the persisted ban lists."
+)
+VSC_F(bans_persisted_fragmentation,	uint64_t, 0, 'g',
+    "Extra bytes in persisted ban lists due to fragmentation",
+        "Number of extra bytes accumulated through dropped and"
+	" gone bans in the persistent ban lists."
+)
 
 /*--------------------------------------------------------------------*/
 



From martin at varnish-cache.org  Fri Feb 15 13:58:00 2013
From: martin at varnish-cache.org (Martin Blix Grydeland)
Date: Fri, 15 Feb 2013 14:58:00 +0100
Subject: [master] bf36a1c Fix the length recorded for truncated bans (it
 should be the length of the header, not 0).
Message-ID: <E1U6Lnc-0005Gj-Cw@project.varnish-software.com>

commit bf36a1c58be5b8fe651e1ec6cf586feb6a671fb2
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Fri Dec 14 16:32:39 2012 +0100

    Fix the length recorded for truncated bans (it should be the length of
    the header, not 0).
    
    Add a test case for reload of truncated bans.
    
    Fixes: #1266

diff --git a/bin/varnishd/cache/cache_ban.c b/bin/varnishd/cache/cache_ban.c
index 4cf3d1b..ed4c041 100644
--- a/bin/varnishd/cache/cache_ban.c
+++ b/bin/varnishd/cache/cache_ban.c
@@ -269,7 +269,7 @@ ban_mark_gone(struct ban *b)
 	b->flags |= BAN_F_GONE;
 	b->spec[BANS_FLAGS] |= BANS_FLAG_GONE;
 	VWMB();
-	vbe32enc(b->spec + BANS_LENGTH, 0);
+	vbe32enc(b->spec + BANS_LENGTH, BANS_HEAD_LEN);
 	VSC_C_main->bans_gone++;
 	VSC_C_main->bans_persisted_fragmentation += ln - ban_len(b->spec);
 }
diff --git a/bin/varnishtest/tests/r01266.vtc b/bin/varnishtest/tests/r01266.vtc
new file mode 100644
index 0000000..49f2c67
--- /dev/null
+++ b/bin/varnishtest/tests/r01266.vtc
@@ -0,0 +1,41 @@
+varnishtest "#1266 - Check persisted truncated gone bans"
+
+# Test that bans which has been gone'd, truncated and persisted works
+
+shell "rm -f ${tmpdir}/_.per1"
+
+server s1 {
+	rxreq
+	txresp -hdr "x-foo: foo"
+} -start
+
+varnish v1 \
+	-arg "-pfeature=+wait_silo" \
+	-arg "-pban_lurker_sleep=0.01" \
+	-storage "-sper1=persistent,${tmpdir}/_.per1,10m" \
+	-vcl+backend {
+	}
+varnish v1 -start
+
+# Add a ban that will (with lurker help) become a truncated gone ban last
+# in the list
+varnish v1 -cliok "ban obj.http.x-foo == bar"
+delay 1
+
+# Add an object that will point to our ban
+client c1 {
+	txreq
+	rxresp
+	expect resp.http.x-foo == "foo"
+} -run
+
+# Force a reload
+varnish v1 -stop
+varnish v1 -start
+
+# Check that our object is still there
+client c1 {
+	txreq
+	rxresp
+	expect resp.http.x-foo == "foo"
+} -run



From martin at varnish-cache.org  Fri Feb 15 13:58:00 2013
From: martin at varnish-cache.org (Martin Blix Grydeland)
Date: Fri, 15 Feb 2013 14:58:00 +0100
Subject: [master] d1c6ef5 Mark the ban_magic as gone also in the persisted ban
 spec, so it can be recognized as such on reload
Message-ID: <E1U6Lnc-0005Gp-E0@project.varnish-software.com>

commit d1c6ef5e91aff24a72d64542d77050ab0d1a2acf
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Fri Dec 14 17:33:04 2012 +0100

    Mark the ban_magic as gone also in the persisted ban spec, so it can
    be recognized as such on reload

diff --git a/bin/varnishd/cache/cache_ban.c b/bin/varnishd/cache/cache_ban.c
index ed4c041..eb9e179 100644
--- a/bin/varnishd/cache/cache_ban.c
+++ b/bin/varnishd/cache/cache_ban.c
@@ -265,6 +265,8 @@ ban_mark_gone(struct ban *b)
 	unsigned ln;
 
 	CHECK_OBJ_NOTNULL(b, BAN_MAGIC);
+	AN(b->spec);
+	AZ(b->flags & BAN_F_GONE);
 	ln = ban_len(b->spec);
 	b->flags |= BAN_F_GONE;
 	b->spec[BANS_FLAGS] |= BANS_FLAG_GONE;
@@ -1267,9 +1269,8 @@ BAN_Init(void)
 
 	ban_magic = BAN_New();
 	AN(ban_magic);
-	ban_magic->flags |= BAN_F_GONE;
-	VSC_C_main->bans_gone++;
 	AZ(BAN_Insert(ban_magic));
+	ban_mark_gone(ban_magic);
 }
 
 /*--------------------------------------------------------------------



From martin at varnish-cache.org  Fri Feb 15 13:58:00 2013
From: martin at varnish-cache.org (Martin Blix Grydeland)
Date: Fri, 15 Feb 2013 14:58:00 +0100
Subject: [master] 4fa4a80 Honor the persisted GONE flag on reload of bans,
 and keep the duplicate counters consistent on reloads.
Message-ID: <E1U6Lnc-0005Gx-FE@project.varnish-software.com>

commit 4fa4a80cc1bce3adb1da79f8024b5643ce5989b3
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Fri Dec 14 18:36:45 2012 +0100

    Honor the persisted GONE flag on reload of bans, and keep the
    duplicate counters consistent on reloads.
    
    Add a test case for this.
    
    Add a section on bans and offline silos in the docs.

diff --git a/bin/varnishd/cache/cache_ban.c b/bin/varnishd/cache/cache_ban.c
index eb9e179..2e1488f 100644
--- a/bin/varnishd/cache/cache_ban.c
+++ b/bin/varnishd/cache/cache_ban.c
@@ -634,7 +634,7 @@ static void
 ban_reload(const uint8_t *ban, unsigned len)
 {
 	struct ban *b, *b2;
-	int gone = 0;
+	int dup = 0;
 	double t0, t1, t2 = 9e99;
 
 	ASSERT_CLI();
@@ -651,11 +651,8 @@ ban_reload(const uint8_t *ban, unsigned len)
 			return;
 		if (t1 < t0)
 			break;
-		if (ban_equal(b->spec, ban)) {
-			gone |= BAN_F_GONE;
-			VSC_C_main->bans_dups++;
-			VSC_C_main->bans_gone++;
-		}
+		if (ban_equal(b->spec, ban))
+			dup = 1;
 	}
 
 	VSC_C_main->bans++;
@@ -666,11 +663,14 @@ ban_reload(const uint8_t *ban, unsigned len)
 	b2->spec = malloc(len);
 	AN(b2->spec);
 	memcpy(b2->spec, ban, len);
-	b2->flags |= gone;
 	if (ban[BANS_FLAGS] & BANS_FLAG_REQ) {
 		VSC_C_main->bans_req++;
 		b2->flags |= BAN_F_REQ;
 	}
+	if (dup)
+		VSC_C_main->bans_dups++;
+	if (dup || (ban[BANS_FLAGS] & BANS_FLAG_GONE))
+		ban_mark_gone(b2);
 	if (b == NULL)
 		VTAILQ_INSERT_TAIL(&ban_head, b2, list);
 	else
diff --git a/bin/varnishtest/tests/p00009.vtc b/bin/varnishtest/tests/p00009.vtc
new file mode 100644
index 0000000..36196d7
--- /dev/null
+++ b/bin/varnishtest/tests/p00009.vtc
@@ -0,0 +1,56 @@
+varnishtest "Check that reloaded bans with gone flag are really gone on restart"
+
+shell "rm -f ${tmpdir}/_.per[12]"
+
+server s1 {
+	rxreq
+	txresp -hdr "x-foo: foo"
+
+	accept
+	rxreq
+	txresp -hdr "x-foo: bar"
+} -start
+
+varnish v1 \
+	-arg "-pfeature=+wait_silo" \
+	-arg "-pban_lurker_sleep=0" \
+	-storage "-sper1=persistent,${tmpdir}/_.per1,10m -sper2=persistent,${tmpdir}/_.per2,10m" \
+	-vcl+backend {
+	}
+varnish v1 -start
+
+client c1 {
+	txreq
+	rxresp
+	expect resp.http.x-foo == "foo"
+} -run
+
+varnish v1 -cliok "ban req.url == /test"
+varnish v1 -cliok "ban req.url == /test"
+varnish v1 -cliok "ban.list"
+
+# Expect ban_magic plus the 2 we added
+varnish v1 -expect bans == 3
+
+# Expect 1 of the 2 added to be marked dup
+varnish v1 -expect bans_dups == 1
+
+# Expect ban_magic plus our 1 dup to be marked gone
+varnish v1 -expect bans_gone == 2
+
+# Restart 
+varnish v1 -stop
+varnish v1 -start
+varnish v1 -cliok "ban.list"
+
+# Check that our object is still there
+client c1 {
+	txreq
+	rxresp
+	expect resp.http.x-foo == "foo"
+} -run
+
+# Expect our duplicate
+varnish v1 -expect bans_dups == 1
+# One more than before restart due to the new ban_magic
+varnish v1 -expect bans_gone == 3
diff --git a/doc/sphinx/include/storage_backends.rst b/doc/sphinx/include/storage_backends.rst
index ec80601..fc646ba 100644
--- a/doc/sphinx/include/storage_backends.rst
+++ b/doc/sphinx/include/storage_backends.rst
@@ -105,6 +105,11 @@ open at any given point in time. Full silos are *sealed*. When Varnish
 starts after a shutdown it will discard the content of any silo that
 isn't sealed.
 
+Note that taking persistent silos offline and at the same time using
+bans can cause problems. This because bans added while the silo was
+offline will not be applied to the silo when it reenters the cache,
+and can make previously banned objects reappear.
+
 Transient Storage
 -----------------
       



From martin at varnish-cache.org  Fri Feb 15 14:12:07 2013
From: martin at varnish-cache.org (Martin Blix Grydeland)
Date: Fri, 15 Feb 2013 15:12:07 +0100
Subject: [master] f91fc4d Turn on SO_KEEPALIVE on all TCP connections.
Message-ID: <E1U6M1H-00016o-C4@project.varnish-software.com>

commit f91fc4d99dd7c2e98133a8a4d873114c9e5785c4
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Fri Feb 8 17:16:28 2013 +0100

    Turn on SO_KEEPALIVE on all TCP connections.
    
    This will help in determining remote hang up of the connection for
    situations where we still are not able to send any reply, but freeing
    the session will reduce resource overhead (e.g. when staying on
    waitinglists for extended periods).
    
    On platforms that support it also add runtime parameters to control
    the keep-alive packet settings through socket options. On platforms
    that don't support these socket options, the values must be set system
    wide.
    
    The Varnish runtime parameters will only be applied when they are less
    than the system default.

diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c
index 62209a5..43ebe6e 100644
--- a/bin/varnishd/cache/cache_acceptor.c
+++ b/bin/varnishd/cache/cache_acceptor.c
@@ -70,8 +70,26 @@ static const struct linger linger = {
 	.l_onoff	=	0,
 };
 
+/*
+ * We turn on keepalives by default to assist in detecting clients that have
+ * hung up on connections returning from waitinglists
+ */
+static const int keepalive = 1;
+
 static unsigned char	need_sndtimeo, need_rcvtimeo, need_linger, need_test,
 			need_tcpnodelay;
+static unsigned char	need_keepalive = 0;
+#ifdef HAVE_TCP_KEEP
+static unsigned char	need_ka_time = 0;
+static unsigned char	need_ka_probes = 0;
+static unsigned char	need_ka_intvl = 0;
+static int		ka_time_cur = 0;
+static int		ka_probes_cur = 0;
+static int		ka_intvl_cur = 0;
+static int		ka_time, ka_time_sys;
+static int		ka_probes, ka_probes_sys;
+static int		ka_intvl, ka_intvl_sys;
+#endif
 
 /*--------------------------------------------------------------------
  * Some kernels have bugs/limitations with respect to which options are
@@ -83,6 +101,10 @@ static void
 sock_test(int fd)
 {
 	struct linger lin;
+	int tka;
+#ifdef HAVE_TCP_KEEP
+	int tka_time, tka_probes, tka_intvl;
+#endif
 	struct timeval tv;
 	socklen_t l;
 	int i, tcp_nodelay;
@@ -97,6 +119,48 @@ sock_test(int fd)
 	if (memcmp(&lin, &linger, l))
 		need_linger = 1;
 
+	l = sizeof tka;
+	i = getsockopt(fd, SOL_SOCKET, SO_KEEPALIVE, &tka, &l);
+	if (i) {
+		VTCP_Assert(i);
+		return;
+	}
+	assert(l == sizeof tka);
+	if (tka != keepalive)
+		need_keepalive = 1;
+
+#ifdef HAVE_TCP_KEEP
+	l = sizeof tka_time;
+	i = getsockopt(fd, IPPROTO_TCP, TCP_KEEPIDLE, &tka_time, &l);
+	if (i) {
+		VTCP_Assert(i);
+		return;
+	}
+	assert(l == sizeof tka_time);
+	if (tka_time != ka_time_cur)
+		need_ka_time = 1;
+
+	l = sizeof tka_probes;
+	i = getsockopt(fd, IPPROTO_TCP, TCP_KEEPCNT, &tka_probes, &l);
+	if (i) {
+		VTCP_Assert(i);
+		return;
+	}
+	assert(l == sizeof tka_probes);
+	if (tka_probes != ka_probes_cur)
+		need_ka_probes = 1;
+
+	l = sizeof tka_intvl;
+	i = getsockopt(fd, IPPROTO_TCP, TCP_KEEPINTVL, &tka_intvl, &l);
+	if (i) {
+		VTCP_Assert(i);
+		return;
+	}
+	assert(l == sizeof tka_intvl);
+	if (tka_intvl != ka_intvl_cur)
+		need_ka_intvl = 1;
+#endif
+
 #ifdef SO_SNDTIMEO_WORKS
 	l = sizeof tv;
 	i = getsockopt(fd, SOL_SOCKET, SO_SNDTIMEO, &tv, &l);
@@ -281,6 +345,22 @@ VCA_SetupSess(struct worker *wrk, struct sess *sp)
 	if (need_linger)
 		VTCP_Assert(setsockopt(sp->fd, SOL_SOCKET, SO_LINGER,
 		    &linger, sizeof linger));
+	if (need_keepalive)
+		VTCP_Assert(setsockopt(sp->fd, SOL_SOCKET, SO_KEEPALIVE,
+		    &keepalive, sizeof keepalive));
+#ifdef HAVE_TCP_KEEP
+	AN(ka_time);
+	if (need_ka_time)
+		VTCP_Assert(setsockopt(sp->fd, IPPROTO_TCP, TCP_KEEPIDLE,
+			&ka_time_cur, sizeof ka_time_cur));
+	if (need_ka_probes)
+		VTCP_Assert(setsockopt(sp->fd, IPPROTO_TCP, TCP_KEEPCNT,
+			&ka_probes_cur, sizeof ka_probes_cur));
+	if (need_ka_intvl)
+		VTCP_Assert(setsockopt(sp->fd, IPPROTO_TCP, TCP_KEEPINTVL,
+			&ka_intvl_cur, sizeof ka_intvl_cur));
+#endif
+
 #ifdef SO_SNDTIMEO_WORKS
 	if (need_sndtimeo)
 		VTCP_Assert(setsockopt(sp->fd, SOL_SOCKET, SO_SNDTIMEO,
@@ -312,10 +392,19 @@ vca_acct(void *arg)
 	struct listen_sock *ls;
 	double t0, now;
 	int i;
+#ifdef HAVE_TCP_KEEP
+	socklen_t len;
+#endif
 
 	THR_SetName("cache-acceptor");
 	(void)arg;
 
+#ifdef HAVE_TCP_KEEP
+	ka_time = cache_param->tcp_keepalive_time;
+	ka_probes = cache_param->tcp_keepalive_probes;
+	ka_intvl = cache_param->tcp_keepalive_intvl;
+#endif
+
 	VTAILQ_FOREACH(ls, &heritage.socks, list) {
 		if (ls->sock < 0)
 			continue;
@@ -324,6 +413,50 @@ vca_acct(void *arg)
 		    &linger, sizeof linger));
 		AZ(setsockopt(ls->sock, IPPROTO_TCP, TCP_NODELAY,
 		    &tcp_nodelay, sizeof tcp_nodelay));
+		AZ(setsockopt(ls->sock, SOL_SOCKET, SO_KEEPALIVE,
+		    &keepalive, sizeof keepalive));
+#ifdef HAVE_TCP_KEEP
+		if (!ka_time_cur) {
+			len = sizeof ka_time_sys;
+			AZ(getsockopt(ls->sock, IPPROTO_TCP, TCP_KEEPIDLE,
+				&ka_time_sys, &len));
+			assert(len == sizeof ka_time_sys);
+			AN(ka_time_sys);
+			ka_time_cur = ka_time =
+			    (ka_time_sys < cache_param->tcp_keepalive_time ?
+				ka_time_sys : cache_param->tcp_keepalive_time);
+		}
+		AZ(setsockopt(ls->sock, IPPROTO_TCP, TCP_KEEPIDLE,
+		    &ka_time_cur, sizeof ka_time_cur));
+
+		if (!ka_probes_cur) {
+			len = sizeof ka_probes_sys;
+			AZ(getsockopt(ls->sock, IPPROTO_TCP, TCP_KEEPCNT,
+			    &ka_probes_sys, &len));
+			assert(len == sizeof ka_probes_sys);
+			AN(ka_probes_sys);
+			ka_probes_cur = ka_probes =
+			    (ka_probes_sys < cache_param->tcp_keepalive_probes ?
+				ka_probes_sys :
+				cache_param->tcp_keepalive_probes);
+		}
+		AZ(setsockopt(ls->sock, IPPROTO_TCP, TCP_KEEPCNT,
+		    &ka_probes_cur, sizeof ka_probes_cur));
+
+		if (!ka_intvl_cur) {
+			len = sizeof ka_intvl_sys;
+			AZ(getsockopt(ls->sock, IPPROTO_TCP, TCP_KEEPINTVL,
+			    &ka_intvl_sys, &len));
+			assert(len == sizeof ka_intvl_sys);
+			AN(ka_intvl_sys);
+			ka_intvl_cur = ka_intvl =
+			    (ka_intvl_sys < cache_param->tcp_keepalive_intvl ?
+				ka_intvl_sys :
+				cache_param->tcp_keepalive_intvl);
+		}
+		AZ(setsockopt(ls->sock, IPPROTO_TCP, TCP_KEEPINTVL,
+		    &ka_intvl_cur, sizeof ka_intvl_cur));
+#endif
 		if (cache_param->accept_filter) {
 			i = VTCP_filter_http(ls->sock);
 			if (i)
@@ -339,6 +472,35 @@ vca_acct(void *arg)
 	t0 = VTIM_real();
 	while (1) {
 		(void)sleep(1);
+#ifdef HAVE_TCP_KEEP
+		ka_time = (ka_time_sys < cache_param->tcp_keepalive_time ?
+		    ka_time_sys : cache_param->tcp_keepalive_time);
+		ka_probes = (ka_probes_sys < cache_param->tcp_keepalive_probes ?
+		    ka_probes_sys : cache_param->tcp_keepalive_probes);
+		ka_intvl = (ka_intvl_sys < cache_param->tcp_keepalive_intvl ?
+		    ka_intvl_sys : cache_param->tcp_keepalive_intvl);
+		if (ka_time_cur != ka_time ||
+		    ka_probes_cur != ka_probes ||
+		    ka_intvl_cur != ka_intvl) {
+			need_test = 1;
+			ka_time_cur = ka_time;
+			ka_probes_cur = ka_probes;
+			ka_intvl_cur = ka_intvl;
+			VTAILQ_FOREACH(ls, &heritage.socks, list) {
+				if (ls->sock < 0)
+					continue;
+				AZ(setsockopt(ls->sock, IPPROTO_TCP,
+				    TCP_KEEPIDLE,
+				    &ka_time_cur, sizeof ka_time_cur));
+				AZ(setsockopt(ls->sock, IPPROTO_TCP,
+				    TCP_KEEPCNT,
+				    &ka_probes_cur, sizeof ka_probes_cur));
+				AZ(setsockopt(ls->sock, IPPROTO_TCP,
+				    TCP_KEEPINTVL,
+				    &ka_intvl_cur, sizeof ka_intvl_cur));
+			}
+		}
+#endif
 #ifdef SO_SNDTIMEO_WORKS
 		if (cache_param->idle_send_timeout != send_timeout) {
 			need_test = 1;
diff --git a/bin/varnishd/common/params.h b/bin/varnishd/common/params.h
index a6e881b..ebeff0f 100644
--- a/bin/varnishd/common/params.h
+++ b/bin/varnishd/common/params.h
@@ -110,6 +110,11 @@ struct params {
 	unsigned		pipe_timeout;
 	unsigned		send_timeout;
 	unsigned		idle_send_timeout;
+#ifdef HAVE_TCP_KEEP
+	unsigned		tcp_keepalive_time;
+	unsigned		tcp_keepalive_probes;
+	unsigned		tcp_keepalive_intvl;
+#endif
 
 	/* Management hints */
 	unsigned		auto_restart;
diff --git a/bin/varnishd/mgt/mgt_param_tbl.c b/bin/varnishd/mgt/mgt_param_tbl.c
index 8601bae..b92c71b 100644
--- a/bin/varnishd/mgt/mgt_param_tbl.c
+++ b/bin/varnishd/mgt/mgt_param_tbl.c
@@ -205,6 +205,31 @@ const struct parspec mgt_parspec[] = {
 		"See setsockopt(2) under SO_SNDTIMEO for more information.",
 		DELAYED_EFFECT,
 		"60", "seconds" },
+#ifdef HAVE_TCP_KEEP
+	{ "tcp_keepalive_time", tweak_timeout, &mgt_param.tcp_keepalive_time,
+		1, 7200,
+		"The number of seconds a connection needs to be idle before "
+		"TCP begins sending out keep-alive probes. Note that this "
+		"setting will only take effect when it is less than the "
+		"system default.",
+		EXPERIMENTAL,
+		"600", "seconds" },
+	{ "tcp_keepalive_probes", tweak_uint, &mgt_param.tcp_keepalive_probes,
+		1, 100,
+		"The maximum number of TCP keep-alive probes to send before "
+		"giving up and killing the connection if no response is "
+		"obtained from the other end. Note that this setting will "
+		"only take effect when it is less than the system default.",
+		EXPERIMENTAL,
+		"5", "probes" },
+	{ "tcp_keepalive_intvl", tweak_timeout, &mgt_param.tcp_keepalive_intvl,
+		1, 100,
+		"The number of seconds between TCP keep-alive probes. Note "
+		"that this setting will only take effect when it is less than"
+		"the system default.",
+		EXPERIMENTAL,
+		"5", "seconds" },
+#endif
 	{ "auto_restart", tweak_bool, &mgt_param.auto_restart, 0, 0,
 		"Restart child process automatically if it dies.\n",
 		0,
diff --git a/configure.ac b/configure.ac
index a4cd8e8..76406d0 100644
--- a/configure.ac
+++ b/configure.ac
@@ -423,6 +423,38 @@ if test "$ac_cv_so_rcvtimeo_works" = no ||
 fi
 LIBS="${save_LIBS}"
 
+# Check if the OS supports TCP_KEEP(CNT|IDLE|INTVL) socket options
+save_LIBS="${LIBS}"
+LIBS="${LIBS} ${NET_LIBS}"
+AC_CACHE_CHECK([for TCP_KEEP(CNT|IDLE|INTVL) socket options],
+  [ac_cv_have_tcp_keep],
+  [AC_RUN_IFELSE(
+    [AC_LANG_PROGRAM([[
+#include <stdio.h>
+#include <sys/types.h>
+#include <sys/socket.h>
+#include <netinet/in.h>
+#include <netinet/tcp.h>
+    ]],[[
+int s = socket(AF_INET, SOCK_STREAM, 0);
+int i;
+i = 5;
+if (setsockopt(s, IPPROTO_TCP, TCP_KEEPCNT, &i, sizeof i))
+  return (1);
+if (setsockopt(s, IPPROTO_TCP, TCP_KEEPIDLE, &i, sizeof i))
+  return (1);
+if (setsockopt(s, IPPROTO_TCP, TCP_KEEPINTVL, &i, sizeof i))
+  return (1);
+return (0);
+    ]])],
+    [ac_cv_have_tcp_keep=yes],
+    [ac_cv_have_tcp_keep=no])
+  ])
+if test "$ac_cv_have_tcp_keep" = yes; then
+   AC_DEFINE([HAVE_TCP_KEEP], [1], [Define if OS supports TCP_KEEP* socket options])
+fi
+LIBS="${save_LIBS}"
+
 # Run-time directory
 VARNISH_STATE_DIR='${localstatedir}/varnish'
 AC_SUBST(VARNISH_STATE_DIR)
diff --git a/doc/sphinx/installation/platformnotes.rst b/doc/sphinx/installation/platformnotes.rst
index 3ad486c..048442c 100644
--- a/doc/sphinx/installation/platformnotes.rst
+++ b/doc/sphinx/installation/platformnotes.rst
@@ -35,3 +35,28 @@ Reduce the maximum stack size by running::
 
 in the Varnish startup script.
 
+TCP keep-alive configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+On some systems, Varnish is not able to set the TCP keep-alive values
+per socket, and therefor the tcp_keepalive_* Varnish runtime
+parameters are not available. On these platforms it can be benefitial
+to tune the system wide values for these in order to more reliably
+detect remote close for sessions spending long time on
+waitinglists. This will help free up resources faster.
+
+Systems to not support TCP keep-alive values per socket include:
+
+- Solaris releases prior to version 11
+- FreeBSD releases prior to version 9.1
+- OS X releases prior to Mountain Lion
+
+On platforms with the necessary socket options the defaults are set
+to:
+
+- tcp_keepalive_time = 600 seconds
+- tcp_keepalive_probes = 5
+- tcp_keepalive_intvl = 5 seconds
+
+Note that Varnish will only apply these run-time parameters so long as
+they are less than the system default value.



From martin at varnish-cache.org  Fri Feb 15 14:12:07 2013
From: martin at varnish-cache.org (Martin Blix Grydeland)
Date: Fri, 15 Feb 2013 15:12:07 +0100
Subject: [master] f6c4379 On return from waitinglist,
 do a poll() on the socket to see if the client has closed the
 connection and gone away. If so, release the session early.
Message-ID: <E1U6M1H-00016u-DX@project.varnish-software.com>

commit f6c4379a856987be89d4eb0b5ffe4fb0ba9cd242
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Fri Feb 8 17:19:49 2013 +0100

    On return from waitinglist, do a poll() on the socket to see if the
    client has closed the connection and gone away. If so, release the
    session early.
    
    Fixes: #1252

diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index d0d987e..db906ec 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -399,7 +399,7 @@ HSH_Lookup(struct req *req)
 		assert(oc->objhead == oh);
 		oc->refcnt++;
 		Lck_Unlock(&oh->mtx);
-		assert(hash->deref(oh));
+		assert(HSH_DerefObjHead(&wrk->stats, &oh));
 		o = oc_getobj(&wrk->stats, oc);
 		CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
 		if (!cache_param->obj_readonly && o->hits < INT_MAX)
@@ -694,13 +694,30 @@ HSH_Deref(struct dstat *ds, struct objcore *oc, struct object **oo)
 	if (oh != NULL) {
 		/* Drop our ref on the objhead */
 		assert(oh->refcnt > 0);
-		if (hash->deref(oh))
-			return (0);
-		HSH_DeleteObjHead(ds, oh);
+		(void)HSH_DerefObjHead(ds, &oh);
 	}
 	return (0);
 }
 
+int
+HSH_DerefObjHead(struct dstat *ds, struct objhead **poh)
+{
+	struct objhead *oh;
+	int r;
+
+	AN(ds);
+	AN(poh);
+	oh = *poh;
+	*poh = NULL;
+	CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC);
+
+	assert(oh->refcnt > 0);
+	r = hash->deref(oh);
+	if (!r)
+		HSH_DeleteObjHead(ds, oh);
+	return (r);
+}
+
 void
 HSH_Init(const struct hash_slinger *slinger)
 {
diff --git a/bin/varnishd/cache/cache_http1_fsm.c b/bin/varnishd/cache/cache_http1_fsm.c
index 2f99c6e..ecc3583 100644
--- a/bin/varnishd/cache/cache_http1_fsm.c
+++ b/bin/varnishd/cache/cache_http1_fsm.c
@@ -74,6 +74,7 @@
 #include <stdlib.h>
 
 #include "cache.h"
+#include "hash/hash_slinger.h"
 
 #include "vcl.h"
 #include "vct.h"
@@ -329,6 +330,19 @@ HTTP1_Session(struct worker *wrk, struct req *req)
 		return;
 	}
 
+	/*
+	 * Return from waitinglist. Check to see if the remote has left.
+	 */
+	if (req->req_step == R_STP_LOOKUP && VTCP_check_hup(sp->fd)) {
+		AN(req->hash_objhead);
+		(void)HSH_DerefObjHead(&wrk->stats, &req->hash_objhead);
+		AZ(req->hash_objhead);
+		SES_Close(sp, SC_REM_CLOSE);
+		sdr = http1_cleanup(sp, wrk, req);
+		assert(sdr == SESS_DONE_RET_GONE);
+		return;
+	}
+
 	if (sp->sess_step == S_STP_NEWREQ) {
 		HTTP1_Init(req->htc, req->ws, sp->fd, req->vsl,
 		    cache_param->http_req_size,
diff --git a/bin/varnishd/hash/hash_slinger.h b/bin/varnishd/hash/hash_slinger.h
index c385ea6..f90f592 100644
--- a/bin/varnishd/hash/hash_slinger.h
+++ b/bin/varnishd/hash/hash_slinger.h
@@ -98,6 +98,7 @@ struct objhead {
 void HSH_Unbusy(struct dstat *, struct objcore *);
 void HSH_Complete(struct objcore *oc);
 void HSH_DeleteObjHead(struct dstat *, struct objhead *oh);
+int HSH_DerefObjHead(struct dstat *, struct objhead **poh);
 int HSH_Deref(struct dstat *, struct objcore *oc, struct object **o);
 #endif /* VARNISH_CACHE_CHILD */
 
diff --git a/bin/varnishtest/tests.disabled/r01252.vtc b/bin/varnishtest/tests.disabled/r01252.vtc
new file mode 100644
index 0000000..c988da6
--- /dev/null
+++ b/bin/varnishtest/tests.disabled/r01252.vtc
@@ -0,0 +1,44 @@
+varnishtest "#1252 - Drop remote closed connections returning from waitinglists"
+
+# This test case is disabled because it will only pass on platforms
+# where the tcp_keepalive_* runtime arguments are available, and also
+# because it requires "-t 75" argument to varnishtest (remote closed
+# state will only be detected after FIN timeout has passed (60s))
+
+server s1 {
+	rxreq
+	expect req.http.X-Client == "1"
+	sema r1 sync 2
+	delay 75
+	close
+} -start
+
+server s2 {
+	rxreq
+	expect req.url == "/should/not/happen"
+	txresp
+} -start
+
+varnish v1 -arg "-p debug=+waitinglist -p tcp_keepalive_time=1s -p tcp_keepalive_probes=1 -p tcp_keepalive_intvl=1s -p first_byte_timeout=70" -vcl+backend {
+	sub vcl_recv {
+		if (req.http.x-client == "2") {
+			set req.backend = s2;
+		}
+	}
+} -start
+
+client c1 {
+	timeout 70
+	txreq -hdr "X-Client: 1"
+	rxresp
+	expect resp.status == 503
+} -start
+
+client c2 {
+	sema r1 sync 2
+	txreq -hdr "X-Client: 2"
+	delay 1
+} -start
+
+client c1 -wait
+client c2 -wait
diff --git a/include/vtcp.h b/include/vtcp.h
index 77f86ed..1594a4d 100644
--- a/include/vtcp.h
+++ b/include/vtcp.h
@@ -62,6 +62,7 @@ int VTCP_filter_http(int sock);
 int VTCP_blocking(int sock);
 int VTCP_nonblocking(int sock);
 int VTCP_linger(int sock, int linger);
+int VTCP_check_hup(int sock);
 
 #ifdef SOL_SOCKET
 int VTCP_port(const struct sockaddr_storage *addr);
diff --git a/lib/libvarnish/vtcp.c b/lib/libvarnish/vtcp.c
index 2c6dd3f..7227fb3 100644
--- a/lib/libvarnish/vtcp.c
+++ b/lib/libvarnish/vtcp.c
@@ -308,3 +308,22 @@ VTCP_linger(int sock, int linger)
 	VTCP_Assert(i);
 	return (i);
 }
+
+/*--------------------------------------------------------------------
+ * Do a poll to check for remote HUP
+ */
+
+int
+VTCP_check_hup(int sock)
+{
+	struct pollfd pfd;
+
+	assert(sock > 0);
+	pfd.fd = sock;
+	pfd.events = POLLOUT;
+	pfd.revents = 0;
+
+	if (poll(&pfd, 1, 0) == 1 && pfd.revents & POLLHUP)
+		return (1);
+        return (0);
+}



From martin at varnish-cache.org  Mon Feb 18 13:52:00 2013
From: martin at varnish-cache.org (Martin Blix Grydeland)
Date: Mon, 18 Feb 2013 14:52:00 +0100
Subject: [master] 3b38929 Rename variable dup -> duplicate to unbreak build
 (warning on global shadowing).
Message-ID: <E1U7R8S-0002Ag-HS@project.varnish-software.com>

commit 3b389292fe1a0acac516385495a230ee30387f61
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Mon Feb 18 14:51:06 2013 +0100

    Rename variable dup -> duplicate to unbreak build (warning on global
    shadowing).

diff --git a/bin/varnishd/cache/cache_ban.c b/bin/varnishd/cache/cache_ban.c
index 2e1488f..1229dfb 100644
--- a/bin/varnishd/cache/cache_ban.c
+++ b/bin/varnishd/cache/cache_ban.c
@@ -634,7 +634,7 @@ static void
 ban_reload(const uint8_t *ban, unsigned len)
 {
 	struct ban *b, *b2;
-	int dup = 0;
+	int duplicate = 0;
 	double t0, t1, t2 = 9e99;
 
 	ASSERT_CLI();
@@ -652,7 +652,7 @@ ban_reload(const uint8_t *ban, unsigned len)
 		if (t1 < t0)
 			break;
 		if (ban_equal(b->spec, ban))
-			dup = 1;
+			duplicate = 1;
 	}
 
 	VSC_C_main->bans++;
@@ -667,9 +667,9 @@ ban_reload(const uint8_t *ban, unsigned len)
 		VSC_C_main->bans_req++;
 		b2->flags |= BAN_F_REQ;
 	}
-	if (dup)
+	if (duplicate)
 		VSC_C_main->bans_dups++;
-	if (dup || (ban[BANS_FLAGS] & BANS_FLAG_GONE))
+	if (duplicate || (ban[BANS_FLAGS] & BANS_FLAG_GONE))
 		ban_mark_gone(b2);
 	if (b == NULL)
 		VTAILQ_INSERT_TAIL(&ban_head, b2, list);



From perbu at varnish-cache.org  Tue Feb 19 13:43:20 2013
From: perbu at varnish-cache.org (Per Buer)
Date: Tue, 19 Feb 2013 14:43:20 +0100
Subject: [master] 87cad02 rework tutorial according to new layout
Message-ID: <E1U7nTc-0002OC-V7@project.varnish-software.com>

commit 87cad02c2af022be899a622d1b3a26b385d05726
Author: Per Buer <perbu at varnish-software.com>
Date:   Tue Feb 19 14:00:03 2013 +0100

    rework tutorial according to new layout

diff --git a/doc/sphinx/tutorial/index.rst b/doc/sphinx/tutorial/index.rst
index 6444be9..bd42134 100644
--- a/doc/sphinx/tutorial/index.rst
+++ b/doc/sphinx/tutorial/index.rst
@@ -13,13 +13,12 @@ following points:
 
 Then we hand things over to the user guide. 
 
-Let's start off by including some of the  stuff we chucked out of the
-previous tutorial - now "users guide"
+.. Let's start off by including some of the  stuff we chucked out of the
+.. previous tutorial - now "users guide"
 
 .. toctree:: :maxdepth: 1
 
    introduction
-   web_accelerator
    starting_varnish
    putting_varnish_on_port_80
    backend_servers
diff --git a/doc/sphinx/tutorial/introduction.rst b/doc/sphinx/tutorial/introduction.rst
index c0a3a92..1f41b0f 100644
--- a/doc/sphinx/tutorial/introduction.rst
+++ b/doc/sphinx/tutorial/introduction.rst
@@ -4,15 +4,38 @@ What is Varnish?
 ----------------
 
 Varnish Cache is a web application accelerator. It can also be called
-a HTTP reverse proxy. The next chapter :ref:`tutorial-web-accelerator`
-will go into detail on what Varnish is.
+a HTTP reverse proxy. It sits in front of a web server, accepts HTTP
+requests and tries to answer those by looking them up in it's
+cache. If it can't answer the request from cache it will hand the
+request over to a backend server.
+
+You can say that Varnish Cache is a specialized key-value store that
+stores HTTP responses. These are looked up with HTTP requests. If a
+match is found the content is delivered.
+
+Varnish Cache will typically look up a response by inspecting the HTTP
+Host header together with the URL.  Varnish Cache maintains an index,
+a hash, with all the host+url combinations that are kept in the cache.
+
+Some times Varnish will refuse to store the content in it's
+cache. This might be because the HTTP reponse has metadata that
+disables cacheing or that there might be a cookie involved. Varnish,
+in the default configuration, will refuse to cache content when there
+are cookies involved because it has no idea if the content is derived
+from the cookie or not.
+
+All this behaviour can be changed using VCL. See the Users Guide for
+more information on how to do that.
+
 
 Performance
 ~~~~~~~~~~~
 
 Varnish has a modern architecture and is written with performance in
 mind.  It is usually bound by the speed of the network, effectivly
-turning performance into a non-issue.
+turning performance into a non-issue. You get to focus on how your web
+application work and you can allow yourself, to some degree, to care
+less about performance and scalability.
 
 Flexibility
 ~~~~~~~~~~~
@@ -51,3 +74,5 @@ The are two mailing lists available. One for user questions and one
 for development discussions. See varnish-cache.org/mailinglist for
 information and signup.  There is also a web forum on the same site.
 
+Now that you have a vague idea on what Varnish Cache is, let see if we
+can get it up and running.
diff --git a/doc/sphinx/tutorial/peculiarities.rst b/doc/sphinx/tutorial/peculiarities.rst
new file mode 100644
index 0000000..9e83e99
--- /dev/null
+++ b/doc/sphinx/tutorial/peculiarities.rst
@@ -0,0 +1,35 @@
+
+Peculiarities
+-------------
+
+There are a couple of things that are different with Varnish Cache, as
+opposed to other programs. One thing you've already seen - VCL. I'll
+just give you a very quick tour of the other pecularities.
+
+
+varnishadm
+~~~~~~~~~~
+
+Varnish Cache has an admin console. You can connect it it through the
+"varnishadm" command. In order to connect the user needs to be able to
+read /etc/varnish/secret in order to authenticate.
+
+Once you've started the console you can do quite a few operations on
+Varnish, like stopping and starting the cache process, load VCL,
+adjust the built in load balancer and invalidate cached content.
+
+It has a built in command "help" which will give you some hints on
+what it does.
+
+varnishlog
+~~~~~~~~~~
+
+Varnish does not log to disk. Instead it logs to a bit of memory. It
+is like a stream of logs. At any time you'll be able to connect to the
+stream and see what is going on. Varnish logs quite a bit of
+information. You can have a look at the logstream with the command
+``varnishlog``.
+
+
+
+
diff --git a/doc/sphinx/tutorial/putting_varnish_on_port_80.rst b/doc/sphinx/tutorial/putting_varnish_on_port_80.rst
index 73a80ff..05b8d17 100644
--- a/doc/sphinx/tutorial/putting_varnish_on_port_80.rst
+++ b/doc/sphinx/tutorial/putting_varnish_on_port_80.rst
@@ -3,23 +3,46 @@ Put Varnish on port 80
 ----------------------
 
 Until now we've been running with Varnish on a high port, for testing
-purposes. You should test your application and if it works OK we can
-switch, so Varnish will be running on port 80 and your web server on a
-high port.
+purposes. You probably don't want to do this in the long term so let's
+put Varnish on port 80.
 
-First we kill off varnishd::
+First we stop varnish::
 
-     # pkill varnishd
+     # service varnish stop
 
-and stop your web server. Edit the configuration for your web server
-and make it bind to port 8080 instead of 80. Now open the Varnish
-default.vcl and change the port of the *default* backend to 8080.
+Now we need to edit the configuration file that starts Varnish. 
 
-Start up your web server and then start varnish::
 
-      # varnishd -f /usr/local/etc/varnish/default.vcl -s malloc,1G -T 127.0.0.1:2000
+Debian/Ubuntu
+~~~~~~~~~~~~~
 
-Note that we've removed the -a option. Now Varnish, as its default
-setting dictates, will bind to the http port (80). Now everyone
-accessing your site will be accessing through Varnish.
+On Debian/Ubuntu this is /etc/default/varnish. In the file you'll find some text that looks like this::
+
+  DAEMON_OPTS="-a :6081 \
+               -T localhost:6082 \
+               -f /etc/varnish/default.vcl \
+               -S /etc/varnish/secret \
+               -s malloc,256m"
+
+Change it to::
+
+  DAEMON_OPTS="-a :80 \
+               -T localhost:6082 \
+               -f /etc/varnish/default.vcl \
+               -S /etc/varnish/secret \
+               -s malloc,256m"
+
+Red Hat EL / Centos
+~~~~~~~~~~~~~~~~~~~
+
+On Red Hat EL / Centos
+On Red Hat/Centos it is
+
+
+Restarting Varnish
+------------------
+
+Once the change is done. Restart varnish: ``service varnish
+restart``. Now everyone accessing your site will be accessing through
+Varnish.
 
diff --git a/doc/sphinx/tutorial/starting_varnish.rst b/doc/sphinx/tutorial/starting_varnish.rst
index fcdc37c..ed2ca12 100644
--- a/doc/sphinx/tutorial/starting_varnish.rst
+++ b/doc/sphinx/tutorial/starting_varnish.rst
@@ -1,35 +1,60 @@
 .. _tutorial-starting_varnish:
 
+
 Starting Varnish
 ----------------
 
-You might want to run ``pkill varnishd`` to make sure varnishd isn't
-already running. Become root and type:
-
-``# /usr/local/sbin/varnishd -f /usr/local/etc/varnish/default.vcl -s malloc,1G -a :80``
-
-I added a few options, lets go through them:
+This tutorial will assume that you are either running Ubuntu, Debian,
+Red Hat Enterprise Linux or Centos. Those of you running on other
+platforms might have to do some mental translation exercises in order
+to follow me. Since you're on a "weird" platform you're probably used
+to it. :-)
 
-``-f /usr/local/etc/varnish/default.vcl``
- The -f options specifies what configuration varnishd should use. If
- you are on a Linux system and have installed Varnish through packages
- the configuration files might reside in ``/etc/varnish``.
+I assume you have Varnish Cache installed.
 
-``-s malloc,1G``
- The -s options chooses the storage type Varnish should use for
- storing its content. I used the type *malloc*, which uses memory for
- storage. There are other backends as well, described in
- :ref:`user-guide-storage`. 1G specifies how much memory should be
- allocated - one gigabyte. 
+You start Varnish with ``service varnish start``.  This will start
+Varnish if it isn't already running.
 
 Now you have Varnish running. Let us make sure that it works
-properly. Use your browser to go to http://192.168.2.2/
+properly. Use your browser to go to http://127.0.0.1:6081/
 (obviously, you should replace the IP address with one on your own
-system) - you should now see your web application running there.
-
-There are many command line options available for Varnish. For a walk
-through the most important ones see :ref:`users-guide-command-line` or
-for a complete list see :ref:`ref-varnishd`. 
-
-Ignore that for the moment, we'll revisit that topic in the Users
-Guide :ref:`users-guide-increasing_your_hitrate`.
+system). 
+
+The default configuration will try to forward requests to a web
+service running on the same computer as Varnish was installed at,
+port 8080.
+
+If there is no web application being served up there Varnish will
+issue an error. Varnish Cache is very conservative about telling the
+world what is wrong so whenever something is amiss it will issue the
+same generic "Error 503 Service Unavilable".
+
+You might have a web application running on some other port or some
+other computer. Let's edit the configuration and make it point to
+something that actually works.
+
+Fire up your favorite editor and edit /etc/varnish/default.vcl. Most
+of it is commented out but there is some text that is not. It will
+probably look like this::
+
+  backend default {
+      .host = "127.0.0.1";
+      .port = "8080";
+  }
+
+We'll change it and make it point to something that works. Hopefully
+www.varnish-cache.org is up. Let's use that. Replace the text with::
+
+  backend default {
+      .host = "www.varnish-cache.org";
+      .port = "80";
+  }
+
+
+Now issue ``service varnish reload`` to make Varnish reload it's
+configuration. If that succeeded visit http://127.0.0.1:6081/ in your
+browser and you should see some directory listing. It works! The
+reason you're not seeing the Varnish official website is because your
+client isn't sending the appropriate Host: header in the request and
+it ends up showing a listing of the default webfolder on the machine
+usually serving up varnish-cache.org.
diff --git a/doc/sphinx/tutorial/web_accelerator.rst b/doc/sphinx/tutorial/web_accelerator.rst
deleted file mode 100644
index 28f8365..0000000
--- a/doc/sphinx/tutorial/web_accelerator.rst
+++ /dev/null
@@ -1,23 +0,0 @@
-.. _tutorial-web_accelerator:
-
-What is a web accelerator
--------------------------
-
-Really.XXX.
-
-
-The problem
------------
-
-Application servers  er slow. They do lots of different things and
-they can sometimes take seconds to complete a web page.
-
-The solution
-------------
-
-Enter Varnish. It keeps a copy of the web pages that pass through
-it. If it finds that it can reuse these later so the server doesn't
-have to regenerate these it speeds things up.
-
-
-



From perbu at varnish-cache.org  Tue Feb 19 13:43:21 2013
From: perbu at varnish-cache.org (Per Buer)
Date: Tue, 19 Feb 2013 14:43:21 +0100
Subject: [master] 002345f Move storage backend prosa into users guide. Rework
 cmd line and config
Message-ID: <E1U7nTd-0002OI-0Q@project.varnish-software.com>

commit 002345f0ac07813d5f044586a7f7bbf60a898a04
Author: Per Buer <perbu at varnish-software.com>
Date:   Tue Feb 19 14:43:52 2013 +0100

    Move storage backend prosa into users guide. Rework cmd line and config

diff --git a/doc/sphinx/include/storage_backends.rst b/doc/sphinx/include/storage_backends.rst
deleted file mode 100644
index fc646ba..0000000
--- a/doc/sphinx/include/storage_backends.rst
+++ /dev/null
@@ -1,118 +0,0 @@
-
-malloc
-~~~~~~
-
-syntax: malloc[,size]
-
-Malloc is a memory based backend. Each object will be allocated from
-memory. If your system runs low on memory swap will be used. Be aware
-that the size limitation only limits the actual storage and that
-approximately 1k of memory per object will be used for various
-internal structures.
-
-The size parameter specifies the maximum amount of memory varnishd
-will allocate.  The size is assumed to be in bytes, unless followed by
-one of the following suffixes:
-
-      K, k    The size is expressed in kibibytes.
-
-      M, m    The size is expressed in mebibytes.
-
-      G, g    The size is expressed in gibibytes.
-
-      T, t    The size is expressed in tebibytes.
-
-The default size is unlimited. 
-
-Mallocs performance is bound by memory speed so it is very fast. 
-
-file
-~~~~
-
-syntax: file[,path[,size[,granularity]]]
-
-The file backend stores objects in memory backed by a file on disk
-with mmap. This is the default storage backend and unless you specify
-another storage this one will used along with Transient storage.
-
-The path parameter specifies either the path to the backing file or
-the path to a directory in which varnishd will create the backing
-file.  The default is /tmp.
-
-The size parameter specifies the size of the backing file.  The size
-is assumed to be in bytes, unless fol? lowed by one of the following
-suffixes:
-
-      K, k    The size is expressed in kibibytes.
-
-      M, m    The size is expressed in mebibytes.
-
-      G, g    The size is expressed in gibibytes.
-
-      T, t    The size is expressed in tebibytes.
-
-      %       The size is expressed as a percentage of the free space on the
-              file system where it resides.
-
-The default size is 50%.
-
-If the backing file already exists, it will be truncated or expanded
-to the specified size.
-
-Note that if varnishd has to create or expand the file, it will not
-pre-allocate the added space, leading to fragmentation, which may
-adversely impact performance.  Pre-creating the storage file using
-dd(1) will reduce fragmentation to a minimum.
-
-The granularity parameter specifies the granularity of
-allocation.  All allocations are rounded up to this size.  The
-is assumed to be in bytes, unless followed by one of the
-suffixes described for size except for %.
-
-The default size is the VM page size.  The size should be reduced if
-you have many small objects.
-
-File performance is typically limited by the write speed of the
-device, and depending on use, the seek time.
-
-persistent (experimental)
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-syntax: persistent,path,size {experimental}
-
-Persistent storage. Varnish will store objects in a file in a manner
-that will secure the survival of *most* of the objects in the event of
-a planned or unplanned shutdown of Varnish.
-
-The path parameter specifies the path to the backing file. If
-the file doesn't exist Varnish will create it.
-
-The size parameter specifies the size of the backing file.  The
-size is assumed to be in bytes, unless followed by one of the
-following suffixes:
-
-      K, k    The size is expressed in kibibytes.
-
-      M, m    The size is expressed in mebibytes.
-
-      G, g    The size is expressed in gibibytes.
-
-      T, t    The size is expressed in tebibytes.
-
-Varnish will split the file into logical *silos* and write to the
-silos in the manner of a circular buffer. Only one silo will be kept
-open at any given point in time. Full silos are *sealed*. When Varnish
-starts after a shutdown it will discard the content of any silo that
-isn't sealed.
-
-Note that taking persistent silos offline and at the same time using
-bans can cause problems. This because bans added while the silo was
-offline will not be applied to the silo when it reenters the cache,
-and can make previously banned objects reappear.
-
-Transient Storage
------------------
-      
-If you name any of your storage backend "Transient" it will be
-used for transient (short lived) objects. By default Varnish
-would use an unlimited malloc backend for this.
diff --git a/doc/sphinx/reference/varnishd.rst b/doc/sphinx/reference/varnishd.rst
index 95b0097..bab04ad 100644
--- a/doc/sphinx/reference/varnishd.rst
+++ b/doc/sphinx/reference/varnishd.rst
@@ -157,7 +157,124 @@ Storage Types
 
 The following storage types are available:
 
-.. include:: ../include/storage_backends.rst
+malloc
+~~~~~~
+
+syntax: malloc[,size]
+
+Malloc is a memory based backend. Each object will be allocated from
+memory. If your system runs low on memory swap will be used. Be aware
+that the size limitation only limits the actual storage and that
+approximately 1k of memory per object will be used for various
+internal structures.
+
+The size parameter specifies the maximum amount of memory varnishd
+will allocate.  The size is assumed to be in bytes, unless followed by
+one of the following suffixes:
+
+      K, k    The size is expressed in kibibytes.
+
+      M, m    The size is expressed in mebibytes.
+
+      G, g    The size is expressed in gibibytes.
+
+      T, t    The size is expressed in tebibytes.
+
+The default size is unlimited. 
+
+Mallocs performance is bound by memory speed so it is very fast. 
+
+file
+~~~~
+
+syntax: file[,path[,size[,granularity]]]
+
+The file backend stores objects in memory backed by a file on disk
+with mmap. This is the default storage backend and unless you specify
+another storage this one will used along with Transient storage.
+
+The path parameter specifies either the path to the backing file or
+the path to a directory in which varnishd will create the backing
+file.  The default is /tmp.
+
+The size parameter specifies the size of the backing file.  The size
+is assumed to be in bytes, unless fol? lowed by one of the following
+suffixes:
+
+      K, k    The size is expressed in kibibytes.
+
+      M, m    The size is expressed in mebibytes.
+
+      G, g    The size is expressed in gibibytes.
+
+      T, t    The size is expressed in tebibytes.
+
+      %       The size is expressed as a percentage of the free space on the
+              file system where it resides.
+
+The default size is 50%.
+
+If the backing file already exists, it will be truncated or expanded
+to the specified size.
+
+Note that if varnishd has to create or expand the file, it will not
+pre-allocate the added space, leading to fragmentation, which may
+adversely impact performance.  Pre-creating the storage file using
+dd(1) will reduce fragmentation to a minimum.
+
+The granularity parameter specifies the granularity of
+allocation.  All allocations are rounded up to this size.  The
+is assumed to be in bytes, unless followed by one of the
+suffixes described for size except for %.
+
+The default size is the VM page size.  The size should be reduced if
+you have many small objects.
+
+File performance is typically limited by the write speed of the
+device, and depending on use, the seek time.
+
+persistent (experimental)
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+syntax: persistent,path,size {experimental}
+
+Persistent storage. Varnish will store objects in a file in a manner
+that will secure the survival of *most* of the objects in the event of
+a planned or unplanned shutdown of Varnish.
+
+The path parameter specifies the path to the backing file. If
+the file doesn't exist Varnish will create it.
+
+The size parameter specifies the size of the backing file.  The
+size is assumed to be in bytes, unless followed by one of the
+following suffixes:
+
+      K, k    The size is expressed in kibibytes.
+
+      M, m    The size is expressed in mebibytes.
+
+      G, g    The size is expressed in gibibytes.
+
+      T, t    The size is expressed in tebibytes.
+
+Varnish will split the file into logical *silos* and write to the
+silos in the manner of a circular buffer. Only one silo will be kept
+open at any given point in time. Full silos are *sealed*. When Varnish
+starts after a shutdown it will discard the content of any silo that
+isn't sealed.
+
+Note that taking persistent silos offline and at the same time using
+bans can cause problems. This because bans added while the silo was
+offline will not be applied to the silo when it reenters the cache,
+and can make previously banned objects reappear.
+
+Transient Storage
+-----------------
+      
+If you name any of your storage backend "Transient" it will be
+used for transient (short lived) objects. By default Varnish
+would use an unlimited malloc backend for this.
+
 
 Management Interface
 --------------------
diff --git a/doc/sphinx/tutorial/index.rst b/doc/sphinx/tutorial/index.rst
index bd42134..6b68b6a 100644
--- a/doc/sphinx/tutorial/index.rst
+++ b/doc/sphinx/tutorial/index.rst
@@ -22,5 +22,6 @@ Then we hand things over to the user guide.
    starting_varnish
    putting_varnish_on_port_80
    backend_servers
+   peculiarities.rst
    now_what
 
diff --git a/doc/sphinx/users-guide/command-line.rst b/doc/sphinx/users-guide/command-line.rst
index b91e4f0..2bf7e2d 100644
--- a/doc/sphinx/users-guide/command-line.rst
+++ b/doc/sphinx/users-guide/command-line.rst
@@ -1,43 +1,48 @@
 .. _users-guide-command-line:
 
-XXX: Total rewrite of this
-
-Command Line options
---------------------
-
-I assume varnishd is in your path. You might want to run ``pkill
-varnishd`` to make sure varnishd isn't running. 
-
-Become root and type:
-
-``# varnishd -f /usr/local/etc/varnish/default.vcl -s malloc,1G -T 127.0.0.1:2000 -a 0.0.0.0:8080``
-
-I added a few options, lets go through them:
-
-``-f /usr/local/etc/varnish/default.vcl``
- The -f options specifies what configuration varnishd should use.
-
-``-s malloc,1G``
- The -s options chooses the storage type Varnish should use for
- storing its content. I used the type *malloc*, which just uses memory
- for storage. There are other backends as well, described in 
- :ref:users-guide-storage. 1G specifies how much memory should be allocated 
- - one gigabyte. 
-
-``-T 127.0.0.1:2000``
- Varnish has a built-in text-based administration
- interface. Activating the interface makes Varnish manageble without
- stopping it. You can specify what interface the management interface
- should listen to. Make sure you don't expose the management interface
- to the world as you can easily gain root access to a system via the
- Varnish management interface. I recommend tieing it to localhost. If
- you have users on your system that you don't fully trust, use firewall
- rules to restrict access to the interface to root only.
-
-``-a 0.0.0.0:8080``
- I specify that I want Varnish to listen on port 8080 for incomming
- HTTP requests. For a production environment you would probably make
- Varnish listen on port 80, which is the default.
+Typical command line options
+----------------------------
+
+On a modern Linux distro the various options that are used when
+starting up Varnish are stored in /etc/default/varnish (Debian, Ubuntu) or
+/etc/sysconfig/varnish (Red Hat, Centos).
+
+There are quite a few options you can tweak but most of you will only
+need to change a few them.
+
+The typical command line options you want to change are:
+
+-a *listen_address*
+    What address should Varnish listen to. The default is to listen to
+    all IP adresses and stick to port 80. ":80" will ask Varnish to
+    listen to all adresses, both IPv4 and IPv6 and is probably a
+    sensible thing.
+ 
+-f *config file*
+     The -f options specifies what VCL file Varnish should use as the default.
+
+-s *storage options*
+
+     This is probably the most important one. The default is to use
+     the memory storage backend and to allocate a small amount of
+     memory. On a small site this might suffice. If you have dedicated
+     Varnish Cache server you most definitivly want to increase
+     the memory allocated or consider another backend. 
+     Please note that in addition to the memory allocated by the
+     storage engine itself Varnish also has internal data structures
+     that consume memory. More or less 1kb per object.  
+     See also :ref:`guide-storage`.
+
+-T *listen address*  
+     Varnish has a built-in text-based administration
+     interface. Activating the interface makes Varnish manageble
+     without stopping it. You can specify what interface the
+     management interface should listen to. Make sure you don't expose
+     the management interface to the world as you can easily gain root
+     access to a system via the Varnish management interface. I
+     recommend tieing it to localhost. If you have users on your
+     system that you don't fully trust, use firewall rules to restrict
+     access to the interface to root only.
 
 For a complete list of the command line parameters please see
 :ref:`ref-varnishd-options`.
diff --git a/doc/sphinx/users-guide/configuration.rst b/doc/sphinx/users-guide/configuration.rst
index e4f3e00..f49f99b 100644
--- a/doc/sphinx/users-guide/configuration.rst
+++ b/doc/sphinx/users-guide/configuration.rst
@@ -1,7 +1,12 @@
 Configuration
 =============
 
-This should deal with 
+This chapter deals with configuration of Varnish Cache. It will not
+cover the actual configuration policy that's written in VCL. 
+
+This guide assumes you're running Debian, Ubuntu, Red Hat or
+Centos. If you're on any other platform you're probably used to doing
+some mental translations.
 
 .. toctree::
    :maxdepth: 2
diff --git a/doc/sphinx/users-guide/storage-backends.rst b/doc/sphinx/users-guide/storage-backends.rst
index 7436837..2ce30c5 100644
--- a/doc/sphinx/users-guide/storage-backends.rst
+++ b/doc/sphinx/users-guide/storage-backends.rst
@@ -1,5 +1,7 @@
 
 
+.. _guide-storage:
+
 Storage backends
 ----------------
 
@@ -9,5 +11,120 @@ Intro
 Varnish has pluggable storage backends. It can store data in various
 backends which have different performance characteristics.
 
-.. include:: ../include/storage_backends.rst
+malloc
+~~~~~~
+
+syntax: malloc[,size]
+
+Malloc is a memory based backend. Each object will be allocated from
+memory. If your system runs low on memory swap will be used. Be aware
+that the size limitation only limits the actual storage and that
+approximately 1k of memory per object will be used for various
+internal structures.
+
+The size parameter specifies the maximum amount of memory varnishd
+will allocate.  The size is assumed to be in bytes, unless followed by
+one of the following suffixes:
+
+      K, k    The size is expressed in kibibytes.
+
+      M, m    The size is expressed in mebibytes.
+
+      G, g    The size is expressed in gibibytes.
+
+      T, t    The size is expressed in tebibytes.
+
+The default size is unlimited. 
+
+Mallocs performance is bound by memory speed so it is very fast. 
+
+file
+~~~~
+
+syntax: file[,path[,size[,granularity]]]
+
+The file backend stores objects in memory backed by a file on disk
+with mmap. This is the default storage backend and unless you specify
+another storage this one will used along with Transient storage.
+
+The path parameter specifies either the path to the backing file or
+the path to a directory in which varnishd will create the backing
+file.  The default is /tmp.
+
+The size parameter specifies the size of the backing file.  The size
+is assumed to be in bytes, unless fol? lowed by one of the following
+suffixes:
+
+      K, k    The size is expressed in kibibytes.
+
+      M, m    The size is expressed in mebibytes.
+
+      G, g    The size is expressed in gibibytes.
+
+      T, t    The size is expressed in tebibytes.
+
+      %       The size is expressed as a percentage of the free space on the
+              file system where it resides.
+
+The default size is 50%.
+
+If the backing file already exists, it will be truncated or expanded
+to the specified size.
+
+Note that if varnishd has to create or expand the file, it will not
+pre-allocate the added space, leading to fragmentation, which may
+adversely impact performance.  Pre-creating the storage file using
+dd(1) will reduce fragmentation to a minimum.
+
+The granularity parameter specifies the granularity of
+allocation.  All allocations are rounded up to this size.  The
+is assumed to be in bytes, unless followed by one of the
+suffixes described for size except for %.
+
+The default size is the VM page size.  The size should be reduced if
+you have many small objects.
+
+File performance is typically limited by the write speed of the
+device, and depending on use, the seek time.
+
+persistent (experimental)
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+syntax: persistent,path,size {experimental}
+
+Persistent storage. Varnish will store objects in a file in a manner
+that will secure the survival of *most* of the objects in the event of
+a planned or unplanned shutdown of Varnish.
+
+The path parameter specifies the path to the backing file. If
+the file doesn't exist Varnish will create it.
+
+The size parameter specifies the size of the backing file.  The
+size is assumed to be in bytes, unless followed by one of the
+following suffixes:
+
+      K, k    The size is expressed in kibibytes.
+
+      M, m    The size is expressed in mebibytes.
+
+      G, g    The size is expressed in gibibytes.
+
+      T, t    The size is expressed in tebibytes.
+
+Varnish will split the file into logical *silos* and write to the
+silos in the manner of a circular buffer. Only one silo will be kept
+open at any given point in time. Full silos are *sealed*. When Varnish
+starts after a shutdown it will discard the content of any silo that
+isn't sealed.
+
+Note that taking persistent silos offline and at the same time using
+bans can cause problems. This because bans added while the silo was
+offline will not be applied to the silo when it reenters the cache,
+and can make previously banned objects reappear.
 
+Transient Storage
+-----------------
+      
+If you name any of your storage backend "Transient" it will be
+used for transient (short lived) objects. By default Varnish
+would use an unlimited malloc backend for this.



From perbu at varnish-cache.org  Tue Feb 19 13:43:21 2013
From: perbu at varnish-cache.org (Per Buer)
Date: Tue, 19 Feb 2013 14:43:21 +0100
Subject: [master] dabfa1d Remove the storage backend descr
Message-ID: <E1U7nTd-0002OP-1g@project.varnish-software.com>

commit dabfa1d7d9db14aef64f6c14da2d5dea616b711a
Author: Per Buer <perbu at varnish-software.com>
Date:   Tue Feb 19 14:44:13 2013 +0100

    Remove the storage backend descr

diff --git a/doc/sphinx/reference/varnishd.rst b/doc/sphinx/reference/varnishd.rst
index bab04ad..a51039c 100644
--- a/doc/sphinx/reference/varnishd.rst
+++ b/doc/sphinx/reference/varnishd.rst
@@ -92,11 +92,15 @@ OPTIONS
 -S file     Path to a file containing a secret used for authorizing access to the management port.
 
 -s [name=]type[,options]
-            Use the specified storage backend.  See Storage Types for a list of supported storage
-            types.  This option can be used multiple times to specify multiple storage files. You
- 	    can name the different backends. Varnish will then reference that backend with the 
-	    given name in logs, statistics, etc. You will also be able to ask Varnish to use
-            these named backends specifically through VCL by setting backend.storage in vcl_fetch.
+            Use the specified storage backend. The storage backends can be one of the following:
+               * malloc[,size]
+               * file[,path[,size[,granularity]]]
+               * persistent,path,size
+
+            See Storage Types in the Users Guide for more information
+            on the various storage backends.  This option can be used
+            multiple times to specify multiple storage files. Names
+            are referenced in logs, vcl, statistics, etc.
 
 -T address[:port]
             Offer a management interface on the specified address and port.  See Management
@@ -152,130 +156,6 @@ critbit
   comparison to a more traditional B tree the critbit tree is almost
   completely lockless.
 
-Storage Types
--------------
-
-The following storage types are available:
-
-malloc
-~~~~~~
-
-syntax: malloc[,size]
-
-Malloc is a memory based backend. Each object will be allocated from
-memory. If your system runs low on memory swap will be used. Be aware
-that the size limitation only limits the actual storage and that
-approximately 1k of memory per object will be used for various
-internal structures.
-
-The size parameter specifies the maximum amount of memory varnishd
-will allocate.  The size is assumed to be in bytes, unless followed by
-one of the following suffixes:
-
-      K, k    The size is expressed in kibibytes.
-
-      M, m    The size is expressed in mebibytes.
-
-      G, g    The size is expressed in gibibytes.
-
-      T, t    The size is expressed in tebibytes.
-
-The default size is unlimited. 
-
-Mallocs performance is bound by memory speed so it is very fast. 
-
-file
-~~~~
-
-syntax: file[,path[,size[,granularity]]]
-
-The file backend stores objects in memory backed by a file on disk
-with mmap. This is the default storage backend and unless you specify
-another storage this one will used along with Transient storage.
-
-The path parameter specifies either the path to the backing file or
-the path to a directory in which varnishd will create the backing
-file.  The default is /tmp.
-
-The size parameter specifies the size of the backing file.  The size
-is assumed to be in bytes, unless fol? lowed by one of the following
-suffixes:
-
-      K, k    The size is expressed in kibibytes.
-
-      M, m    The size is expressed in mebibytes.
-
-      G, g    The size is expressed in gibibytes.
-
-      T, t    The size is expressed in tebibytes.
-
-      %       The size is expressed as a percentage of the free space on the
-              file system where it resides.
-
-The default size is 50%.
-
-If the backing file already exists, it will be truncated or expanded
-to the specified size.
-
-Note that if varnishd has to create or expand the file, it will not
-pre-allocate the added space, leading to fragmentation, which may
-adversely impact performance.  Pre-creating the storage file using
-dd(1) will reduce fragmentation to a minimum.
-
-The granularity parameter specifies the granularity of
-allocation.  All allocations are rounded up to this size.  The
-is assumed to be in bytes, unless followed by one of the
-suffixes described for size except for %.
-
-The default size is the VM page size.  The size should be reduced if
-you have many small objects.
-
-File performance is typically limited by the write speed of the
-device, and depending on use, the seek time.
-
-persistent (experimental)
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-syntax: persistent,path,size {experimental}
-
-Persistent storage. Varnish will store objects in a file in a manner
-that will secure the survival of *most* of the objects in the event of
-a planned or unplanned shutdown of Varnish.
-
-The path parameter specifies the path to the backing file. If
-the file doesn't exist Varnish will create it.
-
-The size parameter specifies the size of the backing file.  The
-size is assumed to be in bytes, unless followed by one of the
-following suffixes:
-
-      K, k    The size is expressed in kibibytes.
-
-      M, m    The size is expressed in mebibytes.
-
-      G, g    The size is expressed in gibibytes.
-
-      T, t    The size is expressed in tebibytes.
-
-Varnish will split the file into logical *silos* and write to the
-silos in the manner of a circular buffer. Only one silo will be kept
-open at any given point in time. Full silos are *sealed*. When Varnish
-starts after a shutdown it will discard the content of any silo that
-isn't sealed.
-
-Note that taking persistent silos offline and at the same time using
-bans can cause problems. This because bans added while the silo was
-offline will not be applied to the silo when it reenters the cache,
-and can make previously banned objects reappear.
-
-Transient Storage
------------------
-      
-If you name any of your storage backend "Transient" it will be
-used for transient (short lived) objects. By default Varnish
-would use an unlimited malloc backend for this.
-
-
 Management Interface
 --------------------
 



From perbu at varnish-cache.org  Tue Feb 19 13:50:34 2013
From: perbu at varnish-cache.org (Per Buer)
Date: Tue, 19 Feb 2013 14:50:34 +0100
Subject: [master] 73a4b5d More 'about this document' type of text
Message-ID: <E1U7nac-0004Ne-5L@project.varnish-software.com>

commit 73a4b5df98d1084d6d04df9c83f89071aa5c4f6f
Author: Per Buer <perbu at varnish-software.com>
Date:   Tue Feb 19 14:51:30 2013 +0100

    More 'about this document' type of text

diff --git a/doc/sphinx/tutorial/index.rst b/doc/sphinx/tutorial/index.rst
index 6b68b6a..c540f8e 100644
--- a/doc/sphinx/tutorial/index.rst
+++ b/doc/sphinx/tutorial/index.rst
@@ -4,17 +4,12 @@
 The Varnish Tutorial
 %%%%%%%%%%%%%%%%%%%%
 
-What is a tutorial, anyway?
+This documents the basic concept. What is Varnish and how does it
+work? How can I get it up and running. Most likely you would want to
+continue with the users guide (:ref:`users-guide-index`) afterwards.
 
-Scope: Lets start off by explaining the basic concept. Cover the
-following points:
-- what is a cache. How does Varnish work?
-- how do I start it.  
-
-Then we hand things over to the user guide. 
-
-.. Let's start off by including some of the  stuff we chucked out of the
-.. previous tutorial - now "users guide"
+If you're reading this on the web note the "Next topic" and "Previous
+topic" on the right side.
 
 .. toctree:: :maxdepth: 1
 



From perbu at varnish-cache.org  Tue Feb 19 15:47:18 2013
From: perbu at varnish-cache.org (Per Buer)
Date: Tue, 19 Feb 2013 16:47:18 +0100
Subject: [master] c0c52de Rework the storage text for clarity
Message-ID: <E1U7pPa-0000Nt-7v@project.varnish-software.com>

commit c0c52dee438e0fa80be42f67a40f8249cb42dff3
Author: Per Buer <perbu at varnish-software.com>
Date:   Tue Feb 19 16:47:51 2013 +0100

    Rework the storage text for clarity

diff --git a/doc/sphinx/users-guide/storage-backends.rst b/doc/sphinx/users-guide/storage-backends.rst
index 2ce30c5..acad31b 100644
--- a/doc/sphinx/users-guide/storage-backends.rst
+++ b/doc/sphinx/users-guide/storage-backends.rst
@@ -9,7 +9,10 @@ Intro
 ~~~~~
 
 Varnish has pluggable storage backends. It can store data in various
-backends which have different performance characteristics.
+backends which have different performance characteristics. The default
+configuration is to use the malloc backend with a limited size. For a
+serious Varnish deployment you probably need to adjust the storage
+settings.
 
 malloc
 ~~~~~~
@@ -17,9 +20,10 @@ malloc
 syntax: malloc[,size]
 
 Malloc is a memory based backend. Each object will be allocated from
-memory. If your system runs low on memory swap will be used. Be aware
-that the size limitation only limits the actual storage and that
-approximately 1k of memory per object will be used for various
+memory. If your system runs low on memory swap will be used. 
+
+Be aware that the size limitation only limits the actual storage and
+that approximately 1k of memory per object will be used for various
 internal structures.
 
 The size parameter specifies the maximum amount of memory varnishd
@@ -34,7 +38,8 @@ one of the following suffixes:
 
       T, t    The size is expressed in tebibytes.
 
-The default size is unlimited. 
+If no size is given the malloc backend will not limit how much memory
+it can allocate.
 
 Mallocs performance is bound by memory speed so it is very fast. 
 
@@ -66,15 +71,15 @@ suffixes:
       %       The size is expressed as a percentage of the free space on the
               file system where it resides.
 
-The default size is 50%.
+The default size is to use 50% of the space available on the device.
 
 If the backing file already exists, it will be truncated or expanded
 to the specified size.
 
 Note that if varnishd has to create or expand the file, it will not
 pre-allocate the added space, leading to fragmentation, which may
-adversely impact performance.  Pre-creating the storage file using
-dd(1) will reduce fragmentation to a minimum.
+adversely impact performance on rotating hard drives.  Pre-creating
+the storage file using dd(1) will reduce fragmentation to a minimum.
 
 The granularity parameter specifies the granularity of
 allocation.  All allocations are rounded up to this size.  The
@@ -128,3 +133,6 @@ Transient Storage
 If you name any of your storage backend "Transient" it will be
 used for transient (short lived) objects. By default Varnish
 would use an unlimited malloc backend for this.
+
+Varnish will consider an object short lived if the TTL is below the
+parameter "shortlived".



From perbu at varnish-cache.org  Tue Feb 19 15:47:18 2013
From: perbu at varnish-cache.org (Per Buer)
Date: Tue, 19 Feb 2013 16:47:18 +0100
Subject: [master] 67450d8 First attempt at params
Message-ID: <E1U7pPa-0000Ny-9b@project.varnish-software.com>

commit 67450d830e0563a043032c3f335c19976727f277
Author: Per Buer <perbu at varnish-software.com>
Date:   Tue Feb 19 16:48:08 2013 +0100

    First attempt at params

diff --git a/doc/sphinx/users-guide/params.rst b/doc/sphinx/users-guide/params.rst
index a0123ec..635fdcf 100644
--- a/doc/sphinx/users-guide/params.rst
+++ b/doc/sphinx/users-guide/params.rst
@@ -1,4 +1,27 @@
 
 
 Parameters
-----------
\ No newline at end of file
+----------
+
+Varnish Cache has a set of parameter that affect its behaviour and
+performance. Most of these parameters can be set on the Varnish
+command line (through varnishadm) useing the param.set keyword.
+
+Some parameters can, for security purposes be read only using the "-r"
+command line switch to varnishd.
+
+I don't recommend tweaking the parameters unless you're sure of what
+you're doing. We've worked hard to make the defaults sane and Varnish
+should be able to handle most workloads with the default settings.
+
+For a complete listing of all the parameters and a short descriptions
+type ``param.show`` in the CLI. To inspect a certain parameter and get a somewhat longer description on what it does and what the default is type ``param.show *param*``, like this::
+
+  varnish> param.show shortlived
+  200        
+  shortlived                  10.000000 [s]
+                              Default is 10.0
+                              Objects created with TTL shorter than this are
+                              always put in transient storage.
+
+



From perbu at varnish-cache.org  Tue Feb 19 15:52:18 2013
From: perbu at varnish-cache.org (Per Buer)
Date: Tue, 19 Feb 2013 16:52:18 +0100
Subject: [master] 4be996a rst syntax
Message-ID: <E1U7pUQ-0002GD-4v@project.varnish-software.com>

commit 4be996a307c7e34f6211667daa9c2b0bccb846ad
Author: Per Buer <perbu at varnish-software.com>
Date:   Tue Feb 19 16:53:16 2013 +0100

    rst syntax

diff --git a/doc/sphinx/users-guide/params.rst b/doc/sphinx/users-guide/params.rst
index 635fdcf..8985bdc 100644
--- a/doc/sphinx/users-guide/params.rst
+++ b/doc/sphinx/users-guide/params.rst
@@ -15,7 +15,9 @@ you're doing. We've worked hard to make the defaults sane and Varnish
 should be able to handle most workloads with the default settings.
 
 For a complete listing of all the parameters and a short descriptions
-type ``param.show`` in the CLI. To inspect a certain parameter and get a somewhat longer description on what it does and what the default is type ``param.show *param*``, like this::
+type ``param.show`` in the CLI. To inspect a certain parameter and get
+a somewhat longer description on what it does and what the default is
+type param.show and the name of the parameter, like this::
 
   varnish> param.show shortlived
   200        



From perbu at varnish-cache.org  Tue Feb 19 16:25:55 2013
From: perbu at varnish-cache.org (Per Buer)
Date: Tue, 19 Feb 2013 17:25:55 +0100
Subject: [master] 962f512 chapter on built in subs
Message-ID: <E1U7q0x-0006mO-1X@project.varnish-software.com>

commit 962f512a52e10b86523951836dc26142ce9e6084
Author: Per Buer <perbu at varnish-software.com>
Date:   Tue Feb 19 17:23:11 2013 +0100

    chapter on built in subs

diff --git a/doc/sphinx/users-guide/vcl-built-in-subs.rst b/doc/sphinx/users-guide/vcl-built-in-subs.rst
new file mode 100644
index 0000000..99c1641
--- /dev/null
+++ b/doc/sphinx/users-guide/vcl-built-in-subs.rst
@@ -0,0 +1,180 @@
+
+.. vcl-built-in-subs_
+
+Built in subroutines
+--------------------
+
+
+vcl_init
+~~~~~~~~
+
+ Called when VCL is loaded, before any requests pass through it.
+ Typically used to initialize VMODs.
+
+  return() values:
+
+  ok
+    Normal return, VCL continues loading.
+
+vcl_recv
+~~~~~~~~
+
+ Called at the beginning of a request, after the complete request has
+ been received and parsed.  Its purpose is to decide whether or not
+ to serve the request, how to do it, and, if applicable, which backend
+ to use.
+  
+ The vcl_recv subroutine may terminate with calling ``return()`` on one of
+ the following keywords:
+
+  error code [reason]
+    Return the specified error code to the client and abandon the request.
+
+  pass    
+    Switch to pass mode.  Control will eventually pass to vcl_pass.
+
+  pipe    
+    Switch to pipe mode.  Control will eventually pass to vcl_pipe.
+
+  lookup  
+    Look up the requested object in the cache.  Control will
+    eventually pass to vcl_hit or vcl_miss, depending on whether the
+    object is in the cache.  The ``bereq.method`` value will be set
+    to ``GET`` regardless of the value of ``req.method``.
+
+
+vcl_pipe
+~~~~~~~~
+
+ Called upon entering pipe mode.  In this mode, the request is passed
+ on to the backend, and any further data from either client or
+ backend is passed on unaltered until either end closes the
+ connection.
+  
+ The vcl_pipe subroutine may terminate with calling return() with one of
+ the following keywords:
+
+  error code [reason]
+    Return the specified error code to the client and abandon the request.
+
+  pipe
+    Proceed with pipe mode.
+
+vcl_pass
+~~~~~~~~
+
+ Called upon entering pass mode.  In this mode, the request is passed
+ on to the backend, and the backend's response is passed on to the
+ client, but is not entered into the cache.  Subsequent requests
+ submitted over the same client connection are handled normally.
+  
+ The vcl_pass subroutine may terminate with calling return() with one
+ of the following keywords:
+  
+  error code [reason]
+    Return the specified error code to the client and abandon the request.
+
+  pass
+    Proceed with pass mode.
+
+  restart
+    Restart the transaction. Increases the restart counter. If the number 
+    of restarts is higher than *max_restarts* varnish emits a guru meditation 
+    error.
+
+vcl_miss
+~~~~~~~~
+
+Called after a cache lookup if the requested document was not found in
+the cache.  Its purpose is to decide whether or not to attempt to
+retrieve the document from the backend, and which backend to use.
+  
+The vcl_miss subroutine may terminate with calling return() with one
+of the following keywords:
+
+  error code [reason]
+    Return the specified error code to the client and abandon the request.
+
+  pass
+    Switch to pass mode.  Control will eventually pass to vcl_pass.
+
+  fetch
+    Retrieve the requested object from the backend.  Control will
+    eventually pass to vcl_fetch.
+
+vcl_fetch
+~~~~~~~~~
+
+Called after a document has been successfully retrieved from the backend.
+  
+The vcl_fetch subroutine may terminate with calling return() with one
+of the following keywords:
+
+  deliver
+    Possibly insert the object into the cache, then deliver it to the
+    client.  Control will eventually pass to vcl_deliver.
+
+  error code [reason]
+    Return the specified error code to the client and abandon the request.
+
+  hit_for_pass 
+    Pass in fetch. Passes the object without caching it. This will
+    create a so-called hit_for_pass object which has the side effect
+    that the decision not to cache will be cached. This is to allow
+    would-be uncachable requests to be passed to the backend at the
+    same time. The same logic is not necessary in vcl_recv because
+    this happens before any potential queueing for an object takes
+    place.  Note that the TTL for the hit_for_pass object will be set
+    to what the current value of beresp.ttl is. Control will be
+    handled to vcl_deliver on the current request, but subsequent
+    requests will go directly to vcl_pass based on the hit_for_pass
+    object.
+
+  restart
+    Restart the transaction. Increases the restart counter. If the number 
+    of restarts is higher than *max_restarts* varnish emits a guru meditation 
+    error.
+
+vcl_deliver
+~~~~~~~~~~~
+
+Called before a cached object is delivered to the client.
+  
+The vcl_deliver subroutine may terminate with one of the following
+keywords:
+
+  deliver
+    Deliver the object to the client.
+
+  restart
+    Restart the transaction. Increases the restart counter. If the number 
+    of restarts is higher than *max_restarts* varnish emits a guru meditation 
+    error.
+
+vcl_error
+~~~~~~~~~
+
+Called when we hit an error, either explicitly or implicitly due to 
+backend or internal errors.
+
+The vcl_error subroutine may terminate by calling return with one of
+the following keywords:
+ 
+  deliver
+    Deliver the error object to the client.
+
+  restart
+    Restart the transaction. Increases the restart counter. If the number 
+    of restarts is higher than *max_restarts* varnish emits a guru meditation 
+    error.
+
+vcl_fini
+~~~~~~~~
+
+  Called when VCL is discarded only after all requests have exited the VCL.
+  Typically used to clean up VMODs.
+
+  return() values:
+
+  ok
+    Normal return, VCL will be discarded.



From perbu at varnish-cache.org  Tue Feb 19 16:25:55 2013
From: perbu at varnish-cache.org (Per Buer)
Date: Tue, 19 Feb 2013 17:25:55 +0100
Subject: [master] 160b2e0 Split the examples into sep files
Message-ID: <E1U7q0x-0006mU-2n@project.varnish-software.com>

commit 160b2e08c9f993454b667782587176abf0f04ecd
Author: Per Buer <perbu at varnish-software.com>
Date:   Tue Feb 19 17:24:00 2013 +0100

    Split the examples into sep files

diff --git a/doc/sphinx/users-guide/vcl-example1-manipulating-headers.rst b/doc/sphinx/users-guide/vcl-example1-manipulating-headers.rst
new file mode 100644
index 0000000..35dcb1e
--- /dev/null
+++ b/doc/sphinx/users-guide/vcl-example1-manipulating-headers.rst
@@ -0,0 +1,21 @@
+
+
+
+Manipulating request headers in VCL
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Lets say we want to remove the cookie for all objects in the /images
+directory of our web server::
+
+  sub vcl_recv {
+    if (req.url ~ "^/images") {
+      unset req.http.cookie;
+    }
+  }
+
+Now, when the request is handled to the backend server there will be
+no cookie header. The interesting line is the one with the
+if-statement. It matches the URL, taken from the request object, and
+matches it against the regular expression. Note the match operator. If
+it matches the Cookie: header of the request is unset (deleted). 
+
diff --git a/doc/sphinx/users-guide/vcl-example2-manipulating-responses.rst b/doc/sphinx/users-guide/vcl-example2-manipulating-responses.rst
new file mode 100644
index 0000000..7362789
--- /dev/null
+++ b/doc/sphinx/users-guide/vcl-example2-manipulating-responses.rst
@@ -0,0 +1,19 @@
+
+
+Altering the backend response
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Here we override the TTL of a object comming from the backend if it
+matches certain criteria::
+
+  sub vcl_fetch {
+     if (req.url ~ "\.(png|gif|jpg)$") {
+       unset beresp.http.set-cookie;
+       set beresp.ttl = 1h;
+    }
+  }
+
+.. XXX ref hit-for-pass
+
+We also remove any Set-Cookie headers in order to avoid a hit-for-pass
+object to be created.
diff --git a/doc/sphinx/users-guide/vcl-example3-acls.rst b/doc/sphinx/users-guide/vcl-example3-acls.rst
new file mode 100644
index 0000000..3d0226a
--- /dev/null
+++ b/doc/sphinx/users-guide/vcl-example3-acls.rst
@@ -0,0 +1,34 @@
+
+ACLs
+~~~~
+
+You create a named access control list with the *acl* keyword. You can match
+the IP address of the client against an ACL with the match operator.::
+
+  # Who is allowed to purge....
+  acl local {
+      "localhost";
+      "192.168.1.0"/24; /* and everyone on the local network */
+      ! "192.168.1.23"; /* except for the dialin router */
+  }
+  
+  sub vcl_recv {
+    if (req.method == "PURGE") {
+      if (client.ip ~ local) {
+         return(lookup);
+      }
+    } 
+  }
+  
+  sub vcl_hit {
+     if (req.method == "PURGE") {
+       set obj.ttl = 0s;
+       error 200 "Purged.";
+      }
+  }
+
+  sub vcl_miss {
+    if (req.method == "PURGE") {
+      error 404 "Not in cache.";
+    }
+  }
diff --git a/doc/sphinx/users-guide/vcl-examples.rst b/doc/sphinx/users-guide/vcl-examples.rst
index ea78b9b..fdd892a 100644
--- a/doc/sphinx/users-guide/vcl-examples.rst
+++ b/doc/sphinx/users-guide/vcl-examples.rst
@@ -1,65 +1,14 @@
-Example 1 - manipulating headers
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Lets say we want to remove the cookie for all objects in the /images
-directory of our web server::
+VCL Examples
+------------
 
-  sub vcl_recv {
-    if (req.url ~ "^/images") {
-      unset req.http.cookie;
-    }
-  }
+These are a short collection of examples that showcase some of the
+capabilites of the VCL language.
 
-Now, when the request is handled to the backend server there will be
-no cookie header. The interesting line is the one with the
-if-statement. It matches the URL, taken from the request object, and
-matches it against the regular expression. Note the match operator. If
-it matches the Cookie: header of the request is unset (deleted). 
+.. toctree::
 
-Example 2 - manipulating beresp
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+  vcl-example1-manipulating-headers
+  vcl-example2-manipulating-responses
+  vcl-example3-acls
 
-Here we override the TTL of a object comming from the backend if it
-matches certain criteria::
-
-  sub vcl_fetch {
-     if (req.url ~ "\.(png|gif|jpg)$") {
-       unset beresp.http.set-cookie;
-       set beresp.ttl = 1h;
-    }
-  }
-
-Example 3 - ACLs
-~~~~~~~~~~~~~~~~
-
-You create a named access control list with the *acl* keyword. You can match
-the IP address of the client against an ACL with the match operator.::
-
-  # Who is allowed to purge....
-  acl local {
-      "localhost";
-      "192.168.1.0"/24; /* and everyone on the local network */
-      ! "192.168.1.23"; /* except for the dialin router */
-  }
-  
-  sub vcl_recv {
-    if (req.method == "PURGE") {
-      if (client.ip ~ local) {
-         return(lookup);
-      }
-    } 
-  }
-  
-  sub vcl_hit {
-     if (req.method == "PURGE") {
-       set obj.ttl = 0s;
-       error 200 "Purged.";
-      }
-  }
-
-  sub vcl_miss {
-    if (req.method == "PURGE") {
-      error 404 "Not in cache.";
-    }
-  }
 



From perbu at varnish-cache.org  Tue Feb 19 16:25:55 2013
From: perbu at varnish-cache.org (Per Buer)
Date: Tue, 19 Feb 2013 17:25:55 +0100
Subject: [master] 54df9be Add strings to syntax doc
Message-ID: <E1U7q0x-0006mb-4H@project.varnish-software.com>

commit 54df9be7fadb293ddf259a0714750baa43728e76
Author: Per Buer <perbu at varnish-software.com>
Date:   Tue Feb 19 17:24:32 2013 +0100

    Add strings to syntax doc

diff --git a/doc/sphinx/users-guide/vcl-syntax.rst b/doc/sphinx/users-guide/vcl-syntax.rst
index c49ef91..95cb6c6 100644
--- a/doc/sphinx/users-guide/vcl-syntax.rst
+++ b/doc/sphinx/users-guide/vcl-syntax.rst
@@ -13,11 +13,37 @@ Note that VCL doesn't contain any loops or jump statements.
 Strings
 ~~~~~~~
 
+Basic strings are enclosed in " ... ", and may not contain newlines.
+
+Long strings are enclosed in {" ... "}. They may contain any character
+including ", newline and other control characters except for the NUL
+(0x00) character. If you really want NUL characters in a string there
+is a VMOD that makes it possible to create such strings.
 
 
 Access control lists (ACLs)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
+An ACL declaration creates and initializes a named access control list
+which can later be used to match client addresses::
+
+       acl local {
+         "localhost";         // myself
+         "192.0.2.0"/24;      // and everyone on the local network
+         ! "192.0.2.23";      // except for the dialin router
+       }
+
+If an ACL entry specifies a host name which Varnish is unable to
+resolve, it will match any address it is compared to.  Consequently,
+if it is preceded by a negation mark, it will reject any address it is
+compared to, which may not be what you intended.  If the entry is
+enclosed in parentheses, however, it will simply be ignored.
+
+To match an IP address against an ACL, simply use the match operator::
+
+       if (client.ip ~ local) {
+         return (pipe);
+       }
 
 Operators
 ~~~~~~~~~
@@ -43,3 +69,23 @@ down for, uhm, examples.
 ||
  Logical *or*
 
+
+Subroutines
+~~~~~~~~~~~
+
+A subroutine is used to group code for legibility or reusability:
+::
+  
+  sub pipe_if_local {
+    if (client.ip ~ local) {
+      return (pipe);
+    }
+  }
+
+Subroutines in VCL do not take arguments, nor do they return values.
+
+To call a subroutine, use the call keyword followed by the subroutine's name:
+
+call pipe_if_local;
+
+Varnish has quite a few built in subroutines that are called for each transaction as it flows through Varnish. See :ref:`vcl-built-in-subs`.



From perbu at varnish-cache.org  Tue Feb 19 16:25:55 2013
From: perbu at varnish-cache.org (Per Buer)
Date: Tue, 19 Feb 2013 17:25:55 +0100
Subject: [master] 3dd3f56 rework vars
Message-ID: <E1U7q0x-0006mj-5e@project.varnish-software.com>

commit 3dd3f56bbd883d3cc2415528293f6addf50f3c66
Author: Per Buer <perbu at varnish-software.com>
Date:   Tue Feb 19 17:24:56 2013 +0100

    rework vars

diff --git a/doc/sphinx/users-guide/vcl-variables.rst b/doc/sphinx/users-guide/vcl-variables.rst
index 10af0ab..7064bc0 100644
--- a/doc/sphinx/users-guide/vcl-variables.rst
+++ b/doc/sphinx/users-guide/vcl-variables.rst
@@ -8,16 +8,21 @@ object, stored in cache.
 
 In VCL you should know the following structures.
 
+.. XXX: Needs verification
+
 *req*
  The request object. When Varnish has received the request the req object is 
  created and populated. Most of the work you do in vcl_recv you 
  do on or with the req object.
 
+*bereq*
+ The backend request object. 
+
 *beresp*
  The backend respons object. It contains the headers of the object 
  comming from the backend. Most of the work you do in vcl_fetch you 
  do on the beresp object.
 
-*obj*
+*resp*
  The cached object. Mostly a read only object that resides in memory. 
- obj.ttl is writable, the rest is read only.
+ resp.ttl is writable, the rest is read only.



From perbu at varnish-cache.org  Tue Feb 19 16:25:55 2013
From: perbu at varnish-cache.org (Per Buer)
Date: Tue, 19 Feb 2013 17:25:55 +0100
Subject: [master] e230477 various fixes
Message-ID: <E1U7q0x-0006mu-7l@project.varnish-software.com>

commit e23047789522cbbdf27c652901fa3aca16250bcc
Author: Per Buer <perbu at varnish-software.com>
Date:   Tue Feb 19 17:25:18 2013 +0100

    various fixes

diff --git a/doc/sphinx/users-guide/vcl.rst b/doc/sphinx/users-guide/vcl.rst
index b36a43f..6dba5eb 100644
--- a/doc/sphinx/users-guide/vcl.rst
+++ b/doc/sphinx/users-guide/vcl.rst
@@ -10,9 +10,9 @@ Yes. Is great. Ja.
 
    vcl-intro
    vcl-syntax
+   vcl-built-in-subs
    vcl-variables
    vcl-actions
-   vcl-subs
    vcl-backends
    vcl-hashing
    vcl-saint-and-grace
@@ -20,4 +20,4 @@ Yes. Is great. Ja.
    vcl-examples
    websockets
    devicedetection
-   
\ No newline at end of file
+   



From perbu at varnish-cache.org  Tue Feb 19 16:25:55 2013
From: perbu at varnish-cache.org (Per Buer)
Date: Tue, 19 Feb 2013 17:25:55 +0100
Subject: [master] 214317a removed a .
Message-ID: <E1U7q0x-0006n7-9u@project.varnish-software.com>

commit 214317aef68b2fe424598ab6b8c88ab69cf64341
Author: Per Buer <perbu at varnish-software.com>
Date:   Tue Feb 19 17:25:47 2013 +0100

    removed a .

diff --git a/doc/sphinx/users-guide/vcl-hashing.rst b/doc/sphinx/users-guide/vcl-hashing.rst
index 10b2920..01b42d0 100644
--- a/doc/sphinx/users-guide/vcl-hashing.rst
+++ b/doc/sphinx/users-guide/vcl-hashing.rst
@@ -1,4 +1,3 @@
-
 Hashing
 -------
 
@@ -7,7 +6,7 @@ key to find the object again. In the default setup this key is
 calculated based on the content of the *Host* header or the IP adress
 of the server and the URL.
 
-Behold the default vcl.::
+Behold the default vcl::
 
  sub vcl_hash {
      hash_data(req.url);



From perbu at varnish-cache.org  Tue Feb 19 16:25:55 2013
From: perbu at varnish-cache.org (Per Buer)
Date: Tue, 19 Feb 2013 17:25:55 +0100
Subject: [master] fbc9e69 Removed the subs chapter.
Message-ID: <E1U7q0x-0006nH-BV@project.varnish-software.com>

commit fbc9e698825cd222319745ede2a4e307c3316427
Author: Per Buer <perbu at varnish-software.com>
Date:   Tue Feb 19 17:26:06 2013 +0100

    Removed the subs chapter.
    
    It's in it's own file now, based on the VCL manpage.

diff --git a/doc/sphinx/index.rst b/doc/sphinx/index.rst
index 9d948b4..1cc29c6 100644
--- a/doc/sphinx/index.rst
+++ b/doc/sphinx/index.rst
@@ -16,7 +16,7 @@ our tutorial - :ref:`tutorial-index`.
 Contents:
 
 .. toctree::
-   :maxdepth: 2
+   :maxdepth: 1
 
    installation/index.rst
    tutorial/index.rst
@@ -30,6 +30,3 @@ Indices and tables
 
 * :ref:`genindex`
 * :ref:`search`
-
-
-$Id$
diff --git a/doc/sphinx/users-guide/vcl-subs.rst b/doc/sphinx/users-guide/vcl-subs.rst
deleted file mode 100644
index d759b2a..0000000
--- a/doc/sphinx/users-guide/vcl-subs.rst
+++ /dev/null
@@ -1,25 +0,0 @@
-
-vcl_recv
-~~~~~~~~
-
-vcl_recv (yes, we're skimpy with characters, it's Unix) is called at
-the beginning of a request, after the complete request has been
-received and parsed.  Its purpose is to decide whether or not to serve
-the request, how to do it, and, if applicable, which backend to use.
-
-In vcl_recv you can also alter the request. Typically you can alter
-the cookies and add and remove request headers.
-
-Note that in vcl_recv only the request object, req is available.
-
-vcl_fetch
-~~~~~~~~~
-
-vcl_fetch is called *after* a document has been successfully retrieved
-from the backend. Normal tasks her are to alter the response headers,
-trigger ESI processing, try alternate backend servers in case the
-request failed.
-
-In vcl_fetch you still have the request object, req, available. There
-is also a *backend response*, beresp. beresp will contain the HTTP
-headers from the backend.



From perbu at varnish-cache.org  Tue Feb 19 16:50:09 2013
From: perbu at varnish-cache.org (Per Buer)
Date: Tue, 19 Feb 2013 17:50:09 +0100
Subject: [master] 47dc9d9 Merge intro / TOC
Message-ID: <E1U7qOP-0001sK-F8@project.varnish-software.com>

commit 47dc9d9eb8d700a2ae9276081dd234da50da754e
Author: Per Buer <perbu at varnish-software.com>
Date:   Tue Feb 19 17:48:06 2013 +0100

    Merge intro / TOC

diff --git a/doc/sphinx/users-guide/vcl-intro.rst b/doc/sphinx/users-guide/vcl-intro.rst
deleted file mode 100644
index 922310a..0000000
--- a/doc/sphinx/users-guide/vcl-intro.rst
+++ /dev/null
@@ -1,30 +0,0 @@
-Varnish Configuration Language - VCL
--------------------------------------
-
-Varnish has a great configuration system. Most other systems use
-configuration directives, where you basically turn on and off lots of
-switches. Varnish uses a domain specific language called Varnish
-Configuration Language, or VCL for short. Varnish translates this
-configuration into binary code which is then executed when requests
-arrive.
-
-The VCL files are divided into subroutines. The different subroutines
-are executed at different times. One is executed when we get the
-request, another when files are fetched from the backend server.
-
-Varnish will execute these subroutines of code at different stages of
-its work. Because it is code it is execute line by line precedence
-isn't a problem. At some point you call an action in this subroutine
-and then the execution of the subroutine stops.
-
-If you don't call an action in your subroutine and it reaches the end
-Varnish will execute some built in VCL code. You will see this VCL
-code commented out in default.vcl.
-
-99% of all the changes you'll need to do will be done in two of these
-subroutines. *vcl_recv* and *vcl_fetch*.
-
-
-.. _users-guide-vcl_fetch_actions:
-
-
diff --git a/doc/sphinx/users-guide/vcl.rst b/doc/sphinx/users-guide/vcl.rst
index 6dba5eb..031cf36 100644
--- a/doc/sphinx/users-guide/vcl.rst
+++ b/doc/sphinx/users-guide/vcl.rst
@@ -1,14 +1,34 @@
-VCL
----
+VCL - Varnish Configuration Language
+------------------------------------
 
+Varnish has a great configuration system. Most other systems use
+configuration directives, where you basically turn on and off lots of
+switches. Varnish uses a domain specific language called Varnish
+Configuration Language, or VCL for short. 
 
-Yes. Is great. Ja.
+Every inbound request flows through Varnish and you can influence how
+the request is being handled by altering the VCL code. You can direct
+certain requests to certains backends, you can alter the requests and
+the responses or have Varnish take various actions depending on
+arbitrary properties of the request or the response. This makes
+Varnish an extremly powerful HTTP processor, not just for caching.
 
+Varnish translates VCL into binary code which is then executed when
+requests arrive. The performance impact of VCL is negligible.
+
+The VCL files are organized into subroutines. The different subroutines
+are executed at different times. One is executed when we get the
+request, another when files are fetched from the backend server.
+
+If you don't call an action in your subroutine and it reaches the end
+Varnish will execute some built in VCL code. You will see this VCL
+code commented out in default.vcl that ships with Varnish Cache.
+
+.. _users-guide-vcl_fetch_actions:
 
 .. toctree::
    :maxdepth: 2
 
-   vcl-intro
    vcl-syntax
    vcl-built-in-subs
    vcl-variables



From perbu at varnish-cache.org  Tue Feb 19 16:50:09 2013
From: perbu at varnish-cache.org (Per Buer)
Date: Tue, 19 Feb 2013 17:50:09 +0100
Subject: [master] 46f4d0d rework hiarchy
Message-ID: <E1U7qOP-0001sP-Gm@project.varnish-software.com>

commit 46f4d0dbb195be32a40734abdf46a9b635e39c80
Author: Per Buer <perbu at varnish-software.com>
Date:   Tue Feb 19 17:51:07 2013 +0100

    rework hiarchy

diff --git a/doc/sphinx/users-guide/compression.rst b/doc/sphinx/users-guide/compression.rst
index 1e93384..0b78893 100644
--- a/doc/sphinx/users-guide/compression.rst
+++ b/doc/sphinx/users-guide/compression.rst
@@ -1,7 +1,7 @@
 .. _users-guide-compression:
 
 Compression
-~~~~~~~~~~~
+-----------
 
 New in Varnish 3.0 was native support for compression, using gzip
 encoding. *Before* 3.0, Varnish would never compress objects. 
diff --git a/doc/sphinx/users-guide/purging.rst b/doc/sphinx/users-guide/purging.rst
index 2402632..668cacf 100644
--- a/doc/sphinx/users-guide/purging.rst
+++ b/doc/sphinx/users-guide/purging.rst
@@ -15,7 +15,7 @@ banning and forced cache misses. First, let me explain the HTTP purges.
 
 
 HTTP Purges
------------
+~~~~~~~~~~~
 
 A *purge* is what happens when you pick out an object from the cache
 and discard it along with its variants. Usually a purge is invoked
@@ -75,7 +75,7 @@ And Varnish would then discard the front page. This will remove all
 variants as defined by Vary.
 
 Bans
-----
+~~~~
 
 There is another way to invalidate content: Bans. You can think of
 bans as a sort of a filter on objects already in the cache. You *ban*
@@ -164,7 +164,7 @@ be marked as Gone if it is a duplicate ban, but is still kept in the list
 for optimization purposes.
 
 Forcing a cache miss
---------------------
+~~~~~~~~~~~~~~~~~~~~
 
 The final way to invalidate an object is a method that allows you to
 refresh an object by forcing a hash miss for a single request. If you set
@@ -173,3 +173,4 @@ cache, thus forcing a fetch from the backend. This can in turn add the
 freshly fetched object to the cache, thus overriding the current one. The
 old object will stay in the cache until ttl expires or it is evicted by
 some other means.
+
diff --git a/doc/sphinx/users-guide/vary.rst b/doc/sphinx/users-guide/vary.rst
index 8b6fa2b..4f4f475 100644
--- a/doc/sphinx/users-guide/vary.rst
+++ b/doc/sphinx/users-guide/vary.rst
@@ -1,7 +1,7 @@
 .. _users-guide-vary:
 
-Vary
-~~~~
+HTTP Vary
+---------
 
 The Vary header is sent by the web server to indicate what makes a
 HTTP object Vary. This makes a lot of sense with headers like



From tfheen at varnish-cache.org  Wed Feb 20 08:17:56 2013
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Wed, 20 Feb 2013 09:17:56 +0100
Subject: [master] 52d8147 Make send_line static and drop prototype
Message-ID: <E1U84sG-0000ud-Vz@project.varnish-software.com>

commit 52d81477d1d7fb423770b7820a3bd86b65b636e8
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Wed Feb 20 09:06:34 2013 +0100

    Make send_line static and drop prototype

diff --git a/bin/varnishadm/varnishadm.c b/bin/varnishadm/varnishadm.c
index bfcdf9d..ff44924 100644
--- a/bin/varnishadm/varnishadm.c
+++ b/bin/varnishadm/varnishadm.c
@@ -68,8 +68,6 @@
 		exit(status); \
 	} while (0)
 
-void send_line(char *l);
-
 #else
 #define RL_EXIT(status) exit(status)
 #endif
@@ -185,7 +183,7 @@ do_args(int sock, int argc, char * const *argv)
  * to have a global variable.
  */
 static int _line_sock;
-void send_line(char *l)
+static void send_line(char *l)
 {
 	if (l) {
 		cli_write(_line_sock, l);



From tfheen at varnish-cache.org  Wed Feb 20 08:17:57 2013
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Wed, 20 Feb 2013 09:17:57 +0100
Subject: [master] e6a88da Add a bit of tab completion to varnishadm
Message-ID: <E1U84sH-0000ui-1E@project.varnish-software.com>

commit e6a88dabc9e7e1fdf5bbce23f2e44afb64c84b97
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Wed Feb 20 09:17:21 2013 +0100

    Add a bit of tab completion to varnishadm

diff --git a/bin/varnishadm/varnishadm.c b/bin/varnishadm/varnishadm.c
index ff44924..08adc79 100644
--- a/bin/varnishadm/varnishadm.c
+++ b/bin/varnishadm/varnishadm.c
@@ -193,6 +193,41 @@ static void send_line(char *l)
 		RL_EXIT(0);
 	}
 }
+
+static char *commands[256];
+static char *
+command_generator (const char *text, int state)
+{
+	static int list_index, len;
+	const char *name;
+
+	/* If this is a new word to complete, initialize now.  This
+	   includes saving the length of TEXT for efficiency, and
+	   initializing the index variable to 0. */
+	if (!state) {
+		list_index = 0;
+		len = strlen(text);
+	}
+
+	while ((name = commands[list_index]) != NULL) {
+		list_index++;
+		if (strncmp (name, text, len) == 0)
+			return (strdup(name));
+	}
+	/* If no names matched, then return NULL. */
+	return (NULL);
+}
+
+static char **
+varnishadm_completion (const char *text, int start, int end)
+{
+	char **matches;
+	(void)end;
+	matches = (char **)NULL;
+	if (start == 0)
+		matches = rl_completion_matches(text, command_generator);
+	return (matches);
+}
 #endif
 
 /*
@@ -219,13 +254,42 @@ pass(int sock)
 	} else {
 		rl_callback_handler_install("", send_line);
 	}
+	rl_attempted_completion_function = varnishadm_completion;
 #endif
 
-	cli_write(sock, "banner\n");
 	fds[0].fd = sock;
 	fds[0].events = POLLIN;
 	fds[1].fd = 0;
 	fds[1].events = POLLIN;
+
+	/* Grab the commands, for completion */
+	cli_write(sock, "help\n");
+	u = VCLI_ReadResult(fds[0].fd, &status, &answer, timeout);
+	if (!u) {
+		char *t, c[128];
+		if (status == CLIS_COMMS) {
+			RL_EXIT(0);
+		}
+		t = answer;
+
+		i = 0;
+		while (*t) {
+			if (sscanf(t, "%127s", c) == 1) {
+				commands[i++] = strdup(c);
+				while (*t != '\n' && *t != '\0')
+					t++;
+				if (*t == '\n')
+					t++;
+			} else {
+				/* what? */
+				fprintf(stderr, "Unknown command '%s' parsing "
+					"help output. Tab completion may be "
+					"broken\n", t);
+				break;
+			}
+		}
+	}
+	cli_write(sock, "banner\n");
 	while (1) {
 		i = poll(fds, 2, -1);
 		if (i == -1 && errno == EINTR) {



From tfheen at varnish-cache.org  Wed Feb 20 10:18:48 2013
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Wed, 20 Feb 2013 11:18:48 +0100
Subject: [master] 3be4a80 Change AM_CONFIG_HEADER to AC_CONFIG_HEADERS
Message-ID: <E1U86lE-0000oX-7k@project.varnish-software.com>

commit 3be4a80b7893807953ce067159d3bfab059f3bd1
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Wed Feb 20 11:17:16 2013 +0100

    Change AM_CONFIG_HEADER to AC_CONFIG_HEADERS
    
    Fixes #1258

diff --git a/configure.ac b/configure.ac
index 76406d0..597ae29 100644
--- a/configure.ac
+++ b/configure.ac
@@ -4,7 +4,7 @@ Copyright (c) 2006-2011 Varnish Software AS])
 AC_REVISION([$Id$])
 AC_INIT([Varnish], [trunk], [varnish-dev at varnish-cache.org])
 AC_CONFIG_SRCDIR(include/miniobj.h)
-AM_CONFIG_HEADER(config.h)
+AC_CONFIG_HEADERS([config.h])
 AC_CONFIG_MACRO_DIR([m4])
 AC_USE_SYSTEM_EXTENSIONS
 



From tfheen at varnish-cache.org  Fri Feb 22 12:01:09 2013
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Fri, 22 Feb 2013 13:01:09 +0100
Subject: [master] 4e1fbb6 ix Makefile to match change in 87cad02, 47dc9d,
 fbc9e69
Message-ID: <E1U8rJN-0003Iu-AX@project.varnish-software.com>

commit 4e1fbb6bad215ce9dc058f2ef5d5443655597194
Author: Tollef Fog Heen <tfheen at varnish-software.com>
Date:   Fri Feb 22 12:42:23 2013 +0100

    ix Makefile to match change in 87cad02, 47dc9d, fbc9e69
    
    foo

diff --git a/doc/sphinx/Makefile.am b/doc/sphinx/Makefile.am
index 8d9927d..8be9296 100644
--- a/doc/sphinx/Makefile.am
+++ b/doc/sphinx/Makefile.am
@@ -130,7 +130,6 @@ EXTRA_DIST = \
 	reference/vsm.rst \
 	tutorial/index.rst \
 	tutorial/introduction.rst \
-	tutorial/web_accelerator.rst \
 	tutorial/starting_varnish.rst \
 	tutorial/putting_varnish_on_port_80.rst \
 	tutorial/backend_servers.rst \
@@ -158,10 +157,8 @@ EXTRA_DIST = \
 	users-guide/vcl-examples.rst \
 	users-guide/vcl-hashing.rst \
 	users-guide/vcl-inline-c.rst \
-	users-guide/vcl-intro.rst \
 	users-guide/vcl.rst \
 	users-guide/vcl-saint-and-grace.rst \
-	users-guide/vcl-subs.rst \
 	users-guide/vcl-syntax.rst \
 	users-guide/vcl-variables.rst \
 	users-guide/websockets.rst



From tfheen at varnish-cache.org  Fri Feb 22 12:01:09 2013
From: tfheen at varnish-cache.org (Tollef Fog Heen)
Date: Fri, 22 Feb 2013 13:01:09 +0100
Subject: [master] 6a9302c Make std.collect() also work for resp.http and
 bereq.http.
Message-ID: <E1U8rJN-0003J0-Ct@project.varnish-software.com>

commit 6a9302c986762616288b612f261aa2d066bdc57c
Author: Dag Haavi Finstad <daghf at varnish-software.com>
Date:   Tue Feb 19 14:52:58 2013 +0100

    Make std.collect() also work for resp.http and bereq.http.
    
    Fixes: #1222

diff --git a/bin/varnishtest/tests/m00006.vtc b/bin/varnishtest/tests/m00006.vtc
index c35cc38..123d8fd 100644
--- a/bin/varnishtest/tests/m00006.vtc
+++ b/bin/varnishtest/tests/m00006.vtc
@@ -6,6 +6,10 @@ server s1 {
 	expect req.http.foo == "1, 2"
 	txresp -hdr "bar: a" -hdr "bar: b" -bodylen 1
 
+	rxreq
+	expect req.url == "/2"
+	expect req.http.baz == "1, 2"
+	txresp -hdr "qux: a" -hdr "qux: b" -bodylen 1
 } -start
 
 varnish v1 -vcl+backend {
@@ -14,9 +18,15 @@ varnish v1 -vcl+backend {
 	sub vcl_recv {
 		std.collect(req.http.foo);
 	}
+	sub vcl_miss {
+		std.collect(bereq.http.baz);
+	}
 	sub vcl_fetch {
 		std.collect(beresp.http.bar);
 	}
+	sub vcl_deliver {
+		std.collect(resp.http.qux);
+	}
 } -start
 
 client c1 {
@@ -25,6 +35,12 @@ client c1 {
 	expect resp.http.bar == "a, b"
 	expect resp.status == 200
 	expect resp.bodylen == 1
+
+	txreq -url "/2" -hdr "Baz: 1" -hdr "Baz: 2"
+	rxresp
+	expect resp.http.qux == "a, b"
+	expect resp.status == 200
+	expect resp.bodylen == 1
 } -run
 
 varnish v1 -errvcl {'beresp.http.bar': Not available in method 'vcl_recv'}  {
diff --git a/lib/libvmod_std/vmod_std.c b/lib/libvmod_std/vmod_std.c
index 86b2ad3..cbea944 100644
--- a/lib/libvmod_std/vmod_std.c
+++ b/lib/libvmod_std/vmod_std.c
@@ -175,6 +175,15 @@ vmod_collect(struct req *req, const struct gethdr_s *hdr)
 	CHECK_OBJ_NOTNULL(req, REQ_MAGIC);
 	if (hdr->where == HDR_REQ)
 		http_CollectHdr(req->http, hdr->what);
-	else if (hdr->where == HDR_BERESP && req->busyobj != NULL)
+	else if (hdr->where == HDR_BEREQ) {
+		AN(req->busyobj);
+		http_CollectHdr(req->busyobj->bereq, hdr->what);
+	}
+	else if (hdr->where == HDR_BERESP) {
+		AN(req->busyobj);
 		http_CollectHdr(req->busyobj->beresp, hdr->what);
+	}
+	else if (hdr->where == HDR_RESP) {
+		http_CollectHdr(req->resp, hdr->what);
+	}
 }



From daghf at varnish-cache.org  Mon Feb 25 10:12:45 2013
From: daghf at varnish-cache.org (Dag Haavi Finstad)
Date: Mon, 25 Feb 2013 11:12:45 +0100
Subject: [master] ebb6670 Set ulimit -u (RLIMIT_NPROC) to unlimited for RHEL
 init script.
Message-ID: <E1U9v37-0007Xn-8e@project.varnish-software.com>

commit ebb667015cc531c2cdc11c709b67c09e75467275
Author: Dag Haavi Finstad <daghf at varnish-software.com>
Date:   Mon Feb 25 11:07:39 2013 +0100

    Set ulimit -u (RLIMIT_NPROC) to unlimited for RHEL init script.

diff --git a/redhat/varnish.initrc b/redhat/varnish.initrc
index 19cc274..4afa6c5 100755
--- a/redhat/varnish.initrc
+++ b/redhat/varnish.initrc
@@ -56,6 +56,10 @@ start() {
 	# Varnish wants to lock shared memory log in memory. 
 	ulimit -l ${MEMLOCK:-82000}
 
+	# Maximum number of threads (default in CentOS is 1024, which
+	# is often too small for varnish)
+	ulimit -u ${NPROCS:-unlimited}
+
         # $DAEMON_OPTS is set in /etc/sysconfig/varnish. At least, one
         # has to set up a backend, or /tmp will be used, which is a bad idea.
 	if [ "$DAEMON_OPTS" = "" ]; then
diff --git a/redhat/varnish.sysconfig b/redhat/varnish.sysconfig
index d88712f..f5846c9 100644
--- a/redhat/varnish.sysconfig
+++ b/redhat/varnish.sysconfig
@@ -11,6 +11,9 @@ NFILES=131072
 # Default log size is 82MB + header
 MEMLOCK=82000
 
+# Maximum number of threads (for ulimit -u)
+NPROCS="unlimited"
+
 # Maximum size of corefile (for ulimit -c). Default in Fedora is 0
 # DAEMON_COREFILE_LIMIT="unlimited"
 



From perbu at varnish-cache.org  Wed Feb 27 19:11:09 2013
From: perbu at varnish-cache.org (Per Buer)
Date: Wed, 27 Feb 2013 20:11:09 +0100
Subject: [master] 649221b cleanup.
Message-ID: <E1UAmPF-00084y-Oi@project.varnish-software.com>

commit 649221ba7a5c6416783600d4d92df3140488993d
Author: Per Buer <per.buer at gmail.com>
Date:   Sat Nov 10 14:28:02 2012 +0100

    cleanup.

diff --git a/doc/sphinx/phk/spdy.rst b/doc/sphinx/phk/spdy.rst
index 327518a..5c5e51e 100644
--- a/doc/sphinx/phk/spdy.rst
+++ b/doc/sphinx/phk/spdy.rst
@@ -90,7 +90,7 @@ I already can hear some of you amazing VCL wizards say "Well,
 if you inline-C grab a refcount, then restart and ..." but lets
 be honest, that's not how it should look.
 
-You should be able to do something like:
+You should be able to do something like:::
 
 	if (req.proto == "SPDY" && req.url ~ "index.html") {
 		req.obj1 = lookup(backend1, "/main.css")



From perbu at varnish-cache.org  Wed Feb 27 19:11:09 2013
From: perbu at varnish-cache.org (Per Buer)
Date: Wed, 27 Feb 2013 20:11:09 +0100
Subject: [master] 22b7b85 Shorter desc
Message-ID: <E1UAmPF-000853-QQ@project.varnish-software.com>

commit 22b7b856de4929d1abbd744f044c90dd5ae17206
Author: Per Buer <per.buer at gmail.com>
Date:   Wed Feb 27 17:05:38 2013 +0100

    Shorter desc

diff --git a/doc/sphinx/reference/varnishd.rst b/doc/sphinx/reference/varnishd.rst
index 95b0097..79ff54a 100644
--- a/doc/sphinx/reference/varnishd.rst
+++ b/doc/sphinx/reference/varnishd.rst
@@ -157,7 +157,30 @@ Storage Types
 
 The following storage types are available:
 
-.. include:: ../include/storage_backends.rst
+malloc
+~~~~~~
+
+syntax: malloc[,size]
+
+malloc is a memory based backend.
+
+file
+~~~~
+
+syntax: file[,path[,size[,granularity]]]
+
+The file backend stores data in a file on disk. The file will be accessed using mmap.
+
+persistent (experimental)
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+syntax: persistent,path,size
+
+Persistent storage. Varnish will store objects in a file in a manner
+that will secure the survival of *most* of the objects in the event of
+a planned or unplanned shutdown of Varnish.
+
+
 
 Management Interface
 --------------------



From perbu at varnish-cache.org  Wed Feb 27 19:11:09 2013
From: perbu at varnish-cache.org (Per Buer)
Date: Wed, 27 Feb 2013 20:11:09 +0100
Subject: [master] 850f116 language cleanup
Message-ID: <E1UAmPF-000859-Rh@project.varnish-software.com>

commit 850f1165265895e959e3a2d732ea7da11d6026ea
Author: Per Buer <per.buer at gmail.com>
Date:   Wed Feb 27 17:07:17 2013 +0100

    language cleanup

diff --git a/doc/sphinx/users-guide/vcl-intro.rst b/doc/sphinx/users-guide/vcl-intro.rst
index 922310a..bc90411 100644
--- a/doc/sphinx/users-guide/vcl-intro.rst
+++ b/doc/sphinx/users-guide/vcl-intro.rst
@@ -13,18 +13,16 @@ are executed at different times. One is executed when we get the
 request, another when files are fetched from the backend server.
 
 Varnish will execute these subroutines of code at different stages of
-its work. Because it is code it is execute line by line precedence
-isn't a problem. At some point you call an action in this subroutine
-and then the execution of the subroutine stops.
+its work. At some point you call an action in this subroutine and then
+the execution of that subroutine stops.
 
 If you don't call an action in your subroutine and it reaches the end
-Varnish will execute some built in VCL code. You will see this VCL
+Varnish will execute the built in VCL code. You will see this VCL
 code commented out in default.vcl.
 
 99% of all the changes you'll need to do will be done in two of these
 subroutines. *vcl_recv* and *vcl_fetch*.
 
-
 .. _users-guide-vcl_fetch_actions:
 
 



From perbu at varnish-cache.org  Wed Feb 27 19:11:09 2013
From: perbu at varnish-cache.org (Per Buer)
Date: Wed, 27 Feb 2013 20:11:09 +0100
Subject: [master] 9ed7ae7 storage
Message-ID: <E1UAmPF-00085G-T9@project.varnish-software.com>

commit 9ed7ae73db16f011a230641ff50f17e46a036c2a
Author: Per Buer <per.buer at gmail.com>
Date:   Wed Feb 27 17:08:32 2013 +0100

    storage

diff --git a/doc/sphinx/users-guide/storage-backends.rst b/doc/sphinx/users-guide/storage-backends.rst
index 7436837..60476b7 100644
--- a/doc/sphinx/users-guide/storage-backends.rst
+++ b/doc/sphinx/users-guide/storage-backends.rst
@@ -1,13 +1,128 @@
 
-
 Storage backends
 ----------------
 
+
 Intro
 ~~~~~
 
-Varnish has pluggable storage backends. It can store data in various
-backends which have different performance characteristics.
+When you configure Varnish you need to give it some place to store
+data. Varnish has pluggable storage backends. It can store data in
+various backends which have different characteristics with regards to
+semantics and performance.
+
+malloc
+~~~~~~
+
+syntax: malloc[,size]
+
+Malloc is a memory based backend. Each object will be allocated from
+memory. If your system runs low on memory swap will be used. Be aware
+that the size limitation only limits the actual storage and that
+approximately 1k of memory per object will be used for various
+internal structures.
+
+The size parameter specifies the maximum amount of memory varnishd
+will allocate.  The size is assumed to be in bytes, unless followed by
+one of the following suffixes:
+
+      K, k    The size is expressed in kibibytes.
+
+      M, m    The size is expressed in mebibytes.
+
+      G, g    The size is expressed in gibibytes.
+
+      T, t    The size is expressed in tebibytes.
+
+The default size is unlimited. 
+
+Mallocs performance is bound by memory speed so it is very fast. If
+the dataset is bigger than what can fit in memory performance will
+depend on the operating system and how well it doesn paging. 
+
+file
+~~~~
+
+syntax: file[,path[,size[,granularity]]]
+
+The file backend stores objects in memory backed by a file on disk
+with mmap. This is the default storage backend and unless you specify
+another storage this one will used along with Transient storage.
+
+The path parameter specifies either the path to the backing file or
+the path to a directory in which varnishd will create the backing
+file.  The default is /tmp.
+
+The size parameter specifies the size of the backing file.  The size
+is assumed to be in bytes, unless followed by one of the following
+suffixes:
+
+      K, k    The size is expressed in kibibytes.
+
+      M, m    The size is expressed in mebibytes.
+
+      G, g    The size is expressed in gibibytes.
+
+      T, t    The size is expressed in tebibytes.
+
+      %       The size is expressed as a percentage of the free space on the
+              file system where it resides.
+
+The default size is 50%.
+
+If the backing file already exists, it will be truncated or expanded
+to the specified size.
+
+Note that if varnishd has to create or expand the file, it will not
+pre-allocate the added space, leading to fragmentation, which may
+adversely impact performance on rotating hard drives.  Pre-creating
+the storage file using dd(1) will reduce fragmentation to a minimum.
+
+The granularity parameter specifies the granularity of
+allocation.  All allocations are rounded up to this size.  The
+is assumed to be in bytes, unless followed by one of the
+suffixes described for size except for %.
+
+The default size is the VM page size.  The size should be reduced if
+you have many small objects.
+
+File performance is typically limited by the write speed of the
+device, and depending on use, the seek time.
+
+persistent (experimental)
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+syntax: persistent,path,size {experimental}
+
+Persistent storage. Varnish will store objects in a file in a manner
+that will secure the survival of *most* of the objects in the event of
+a planned or unplanned shutdown of Varnish.
+
+The path parameter specifies the path to the backing file. If
+the file doesn't exist Varnish will create it.
+
+The size parameter specifies the size of the backing file.  The
+size is assumed to be in bytes, unless followed by one of the
+following suffixes:
+
+      K, k    The size is expressed in kibibytes.
+
+      M, m    The size is expressed in mebibytes.
+
+      G, g    The size is expressed in gibibytes.
+
+      T, t    The size is expressed in tebibytes.
+
+Varnish will split the file into logical *silos* and write to the
+silos in the manner of a circular buffer. Only one silo will be kept
+open at any given point in time. Full silos are *sealed*. When Varnish
+starts after a shutdown it will discard the content of any silo that
+isn't sealed.
 
-.. include:: ../include/storage_backends.rst
+Transient Storage
+-----------------
+      
+If you name any of your storage backend "Transient" it will be
+used for transient (short lived) objects. By default Varnish
+would use an unlimited malloc backend for this.
 



From perbu at varnish-cache.org  Wed Feb 27 19:11:09 2013
From: perbu at varnish-cache.org (Per Buer)
Date: Wed, 27 Feb 2013 20:11:09 +0100
Subject: [master] 379704d merge
Message-ID: <E1UAmPF-00085U-V6@project.varnish-software.com>

commit 379704da8ab6b581978a62d6ffb43e63a565fe8b
Merge: 9ed7ae7 ebb6670
Author: Per Buer <per.buer at gmail.com>
Date:   Wed Feb 27 19:55:53 2013 +0100

    merge

diff --cc doc/sphinx/reference/varnishd.rst
index 79ff54a,a51039c..201c99a
--- a/doc/sphinx/reference/varnishd.rst
+++ b/doc/sphinx/reference/varnishd.rst
@@@ -152,36 -156,6 +156,35 @@@ critbi
    comparison to a more traditional B tree the critbit tree is almost
    completely lockless.
  
 +Storage Types
 +-------------
 +
 +The following storage types are available:
 +
 +malloc
 +~~~~~~
 +
 +syntax: malloc[,size]
 +
 +malloc is a memory based backend.
 +
 +file
 +~~~~
 +
 +syntax: file[,path[,size[,granularity]]]
 +
 +The file backend stores data in a file on disk. The file will be accessed using mmap.
 +
 +persistent (experimental)
 +~~~~~~~~~~~~~~~~~~~~~~~~~
 +
 +syntax: persistent,path,size
 +
 +Persistent storage. Varnish will store objects in a file in a manner
 +that will secure the survival of *most* of the objects in the event of
 +a planned or unplanned shutdown of Varnish.
 +
 +
- 
  Management Interface
  --------------------
  
diff --cc doc/sphinx/users-guide/storage-backends.rst
index 60476b7,acad31b..2bf2bd8
--- a/doc/sphinx/users-guide/storage-backends.rst
+++ b/doc/sphinx/users-guide/storage-backends.rst
@@@ -1,3 -1,6 +1,4 @@@
 -
 -
+ .. _guide-storage:
  
  Storage backends
  ----------------