From daghf at varnish-software.com  Thu Oct  1 09:21:35 2015
From: daghf at varnish-software.com (Dag Haavi Finstad)
Date: Thu, 01 Oct 2015 11:21:35 +0200
Subject: [master] 6d24fb2 "appearantly" > "apparently"
Message-ID: <E1Zha3T-0007VH-5W@project.varnish-software.com>


commit 6d24fb23bd200e0f7ae26f601f08a3594c6c3c6f
Author: Andrew Langhorn <andrew.langhorn at digital.cabinet-office.gov.uk>
Date:   Wed Sep 30 22:50:07 2015 +0100

    "appearantly" > "apparently"
    
    A very minor spelling fix for something I noticed whilst reading the source
    for CIDR notation inside ACLs.

diff --git a/lib/libvcc/vcc_acl.c b/lib/libvcc/vcc_acl.c
index 72dd304..b9722ce 100644
--- a/lib/libvcc/vcc_acl.c
+++ b/lib/libvcc/vcc_acl.c
@@ -241,7 +241,7 @@ vcc_acl_try_getaddrinfo(struct vcc *tl, struct acl_e *ae)
 /*--------------------------------------------------------------------
  * Ancient stupidity on the part of X/Open and other standards orgs
  * dictate that "192.168" be translated to 192.0.0.168.  Ever since
- * CIDR happened, "192.168/16" notation has been used, but appearantly
+ * CIDR happened, "192.168/16" notation has been used, but apparently
  * no API supports parsing this, so roll our own.
  */
 



From lkarsten at varnish-software.com  Thu Oct  1 11:19:03 2015
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 01 Oct 2015 13:19:03 +0200
Subject: [4.1] ee8045e Be more friendly to our repository software.
Message-ID: <E1Zhbt9-0003te-MZ@project.varnish-software.com>


commit ee8045e27a0fe8de78189c061a9d30edc7f73395
Author: Lasse Karstensen <lkarsten at varnish-software.com>
Date:   Thu Oct 1 13:19:02 2015 +0200

    Be more friendly to our repository software.

diff --git a/control b/control
index e67a508..43cccb4 100644
--- a/control
+++ b/control
@@ -1,5 +1,5 @@
 Source: varnish
-Section: web
+Section: varnish-4.1/web
 Priority: optional
 Maintainer: Varnish Package Maintainers <varnish-dist at varnish-cache.org>
 Build-Depends:
@@ -46,13 +46,13 @@ Description: state of the art, high-performance web accelerator
 
 Package: varnish-doc
 Architecture: all
-Section: doc
+Section: varnish-4.1/doc
 Depends: ${misc:Depends}, libjs-jquery, libjs-underscore
 Description: documentation for Varnish Cache
  This package contains HTML documentation for Varnish Cache.
 
 Package: libvarnishapi1
-Section: libs
+Section: varnish-4.1/libs
 Architecture: any
 Provides: libvarnish1
 Conflicts: libvarnish1
@@ -62,7 +62,7 @@ Description: shared libraries for Varnish
  Shared libraries for Varnish Cache HTTP accelerator.
 
 Package: libvarnishapi-dev
-Section: libdevel
+Section: varnish-4.1/libdevel
 Architecture: any
 Depends: ${shlibs:Depends}, ${misc:Depends}, libvarnishapi1 (= ${binary:Version}), python
 Provides: libvarnish-dev
@@ -71,7 +71,7 @@ Description: development files for Varnish
  Development files for Varnish Cache HTTP accelerator.
 
 Package: varnish-dbg
-Section: debug
+Section: varnish-4.1/debug
 Priority: extra
 Architecture: any
 Depends: varnish (= ${binary:Version}),



From fgsch at lodoss.net  Thu Oct  1 16:18:16 2015
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Thu, 01 Oct 2015 18:18:16 +0200
Subject: [master] 26fd51c Spelling
Message-ID: <E1ZhgYi-0007FE-LN@project.varnish-software.com>


commit 26fd51c5529db6e36a635c10e2253a20f0a80772
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Thu Oct 1 11:03:58 2015 +0100

    Spelling

diff --git a/doc/sphinx/reference/states.rst b/doc/sphinx/reference/states.rst
index 2f80f29..9effce1 100644
--- a/doc/sphinx/reference/states.rst
+++ b/doc/sphinx/reference/states.rst
@@ -24,7 +24,7 @@ The following graphs attempt to provide an overview over the
 processing states, their transitions and the most relevant functions
 in core code. They represent a compromise between usefulness for
 core/VMOD developers and administrators and are intended to serve as
-the reference basis for deriavtive work, such as more VCL-centric
+the reference basis for derivative work, such as more VCL-centric
 views.
 
 -----------



From fgsch at lodoss.net  Thu Oct  1 16:18:16 2015
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Thu, 01 Oct 2015 18:18:16 +0200
Subject: [master] 701e943 Correct argument
Message-ID: <E1ZhgYi-0007FK-MT@project.varnish-software.com>


commit 701e9438f9ed0d2d8f6b65a32dbd0ee52a3b69ea
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Thu Oct 1 17:05:26 2015 +0100

    Correct argument

diff --git a/lib/libvmod_std/vmod.vcc b/lib/libvmod_std/vmod.vcc
index 0002426..2e20825 100644
--- a/lib/libvmod_std/vmod.vcc
+++ b/lib/libvmod_std/vmod.vcc
@@ -65,7 +65,7 @@ Description
 	utilize it.
 Example
 	| if (req.url ~ ^/slow/) {
-	|	std.set_ip_tos(0x0);
+	|	std.set_ip_tos(0);
 	| }
 
 $Function REAL random(REAL lo, REAL hi)



From phk at FreeBSD.org  Fri Oct  2 09:23:47 2015
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 02 Oct 2015 11:23:47 +0200
Subject: [master] f68da1e Fail the VSB if vsnprintf(3) fails.
Message-ID: <E1ZhwZ9-0002rs-7S@project.varnish-software.com>


commit f68da1e503e39ce54645274fd9a81246160758f9
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Oct 2 09:23:33 2015 +0000

    Fail the VSB if vsnprintf(3) fails.

diff --git a/lib/libvarnish/vsb.c b/lib/libvarnish/vsb.c
index eba639e..63c196d 100644
--- a/lib/libvarnish/vsb.c
+++ b/lib/libvarnish/vsb.c
@@ -350,6 +350,10 @@ VSB_vprintf(struct vsb *s, const char *fmt, va_list ap)
 		va_copy(ap_copy, ap);
 		len = vsnprintf(&s->s_buf[s->s_len], VSB_FREESPACE(s) + 1,
 		    fmt, ap_copy);
+		if (len < 0) {
+			s->s_error = errno;
+			return (-1);
+		}
 		va_end(ap_copy);
 	} while (len > VSB_FREESPACE(s) &&
 	    VSB_extend(s, len - VSB_FREESPACE(s)) == 0);



From phk at FreeBSD.org  Mon Oct  5 12:32:55 2015
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 05 Oct 2015 14:32:55 +0200
Subject: [master] eef6eb6 Add explict expect for 206 status
Message-ID: <E1Zj4wp-0007cv-KZ@project.varnish-software.com>


commit eef6eb6828c3eb7c40741a3c372ce72c08d9d35f
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Oct 5 12:25:37 2015 +0000

    Add explict expect for 206 status

diff --git a/bin/varnishtest/tests/g00005.vtc b/bin/varnishtest/tests/g00005.vtc
index bc47a5d..7cb2664 100644
--- a/bin/varnishtest/tests/g00005.vtc
+++ b/bin/varnishtest/tests/g00005.vtc
@@ -27,6 +27,7 @@ client c1 {
 
 	txreq -hdr "Range: bytes=3-5"
 	rxresp
+	expect resp.status == 206
 	expect resp.http.content-encoding == "<undef>"
 	expect resp.bodylen == "3"
 	expect resp.body == "BAR"



From lkarsten at varnish-software.com  Sun Oct 11 08:06:31 2015
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Sun, 11 Oct 2015 10:06:31 +0200
Subject: [master] 4013504 Lint.
Message-ID: <E1ZlBeJ-00067N-9R@project.varnish-software.com>


commit 4013504d40654b382b7692c1dd99980e1129e4ef
Author: Lasse Karstensen <lkarsten at varnish-software.com>
Date:   Sun Oct 11 10:02:16 2015 +0200

    Lint.

diff --git a/lib/libvcc/generate.py b/lib/libvcc/generate.py
index 6d6915a..81e773c 100755
--- a/lib/libvcc/generate.py
+++ b/lib/libvcc/generate.py
@@ -674,7 +674,7 @@ sp_variables = [
 		'HTTP',
 		( 'deliver', 'synth'),
 		( ), """
-		The entire response HTTP data structure
+		The entire response HTTP data structure.
 		"""
 	),
 	('resp.proto',



From lkarsten at varnish-software.com  Sun Oct 11 08:06:31 2015
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Sun, 11 Oct 2015 10:06:31 +0200
Subject: [master] 2696bab Document that resp.status overwrites resp.reason.
Message-ID: <E1ZlBeJ-00067T-Ae@project.varnish-software.com>


commit 2696bab0f4fb495e8fe7acfc9fe7ad32dbeac85a
Author: Lasse Karstensen <lkarsten at varnish-software.com>
Date:   Sun Oct 11 10:02:21 2015 +0200

    Document that resp.status overwrites resp.reason.
    
    For those cases where you want to send a different reason
    message, but use the normal status code.
    
    Noticed by Kacper when doing 301/302 in vcl_synth().

diff --git a/lib/libvcc/generate.py b/lib/libvcc/generate.py
index 81e773c..062d180 100755
--- a/lib/libvcc/generate.py
+++ b/lib/libvcc/generate.py
@@ -689,6 +689,9 @@ sp_variables = [
 		( 'deliver', 'synth', ),
 		( 'deliver', 'synth', ), """
 		The HTTP status code that will be returned.
+
+                Assigning a HTTP standardized code to resp.status will also
+                set resp.reason to the corresponding status message.
 		"""
 	),
 	('resp.reason',



From fgsch at lodoss.net  Tue Oct 13 04:10:19 2015
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Tue, 13 Oct 2015 06:10:19 +0200
Subject: [master] 4fc12e1 Add vsl arguments to the comment
Message-ID: <E1Zlqup-0006lc-E6@project.varnish-software.com>


commit 4fc12e105bece687824f2812a0d9eb63194c8b73
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Tue Oct 13 04:40:35 2015 +0100

    Add vsl arguments to the comment

diff --git a/bin/varnishtest/vtc_logexp.c b/bin/varnishtest/vtc_logexp.c
index 520feb7..767f7e6 100644
--- a/bin/varnishtest/vtc_logexp.c
+++ b/bin/varnishtest/vtc_logexp.c
@@ -43,7 +43,7 @@
  *   -T <seconds>         Transaction end timeout
  *
  *
- * logexpect lN -v <id> [-g <grouping>] [-d 0|1] [-q query] {
+ * logexpect lN -v <id> [-g <grouping>] [-d 0|1] [-q query] [vsl arguments] {
  *    expect <skip> <vxid> <tag> <regex>
  * }
  *



From fgsch at lodoss.net  Tue Oct 13 04:10:19 2015
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Tue, 13 Oct 2015 06:10:19 +0200
Subject: [master] 28966bd Doc fixes
Message-ID: <E1Zlqup-0006li-FJ@project.varnish-software.com>


commit 28966bd1d3105c0ac9af55924cf95228498fab17
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Tue Oct 13 05:07:37 2015 +0100

    Doc fixes
    
    From minusf via github with minor tweaks.

diff --git a/doc/sphinx/users-guide/run_security.rst b/doc/sphinx/users-guide/run_security.rst
index 41c68a8..2beb4aa 100644
--- a/doc/sphinx/users-guide/run_security.rst
+++ b/doc/sphinx/users-guide/run_security.rst
@@ -57,11 +57,9 @@ much anything the kernel will accept::
 	-T '[fe80::1]:8082'
 
 The default is ``-T localhost:0`` which will pick a random
-port number, which `varnishadm(8)` can learn in the shared
+port number, which `varnishadm(8)` can learn from the shared
 memory.
 
-.. XXX:Me no understand sentence above, (8)? and learn in the shared memory? Stored and retrieved by varnishadm from th e shared memory? benc 
-
 By using a "localhost" address, you restrict CLI access
 to the local machine.
 
@@ -77,9 +75,9 @@ Alternatively you can bind the CLI port to a 'localhost' address,
 and give remote users access via a secure connection to the local
 machine, using ssh/VPN or similar.
 
-If you use `ssh` you can restrict which commands each user can execute to
-just `varnishadm`, or even to wrapper scripts around `varnishadm`, which
-only allow specific CLI commands.
+If you use `ssh` you can restrict which commands each user can execute
+to just `varnishadm`, or even use a wrapper scripts around `varnishadm`
+to allow specific CLI commands.
 
 It is also possible to configure `varnishd` for "reverse mode", using
 the '-M' argument.  In that case `varnishd` will attempt to open a
@@ -99,11 +97,9 @@ would be hard to prevent you getting CLI access, wouldn't it ?
 CLI interface authentication
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-By default the CLI interface is protected with a simple, yet
-strong "Pre Shared Key" authentication method, which do not provide
-secrecy (ie: The CLI commands and responses are not encrypted).
-
-.. XXX:Encryption instead of secrecy? benc
+By default the CLI interface is protected with a simple, yet powerful
+"Pre Shared Key" authentication method, which do not provide secrecy
+(ie: The CLI commands and responses are not encrypted).
 
 The way -S/PSK works is really simple:  During startup a file is
 created with a random content and the file is only accessible to
@@ -111,18 +107,14 @@ the user who started `varnishd` (or the superuser).
 
 To authenticate and use a CLI connection, you need to know the
 contents of that file, in order to answer the cryptographic
-challenge `varnishd` issues. 
-
-
-(XXX: xref to algo in refman)
-.. XXX:Dunno what this is? benc
+challenge `varnishd` issues, see :ref:`ref_psk_auth`.
 
 `varnishadm` uses all of this to restrict access, it will only function,
 provided it can read the secret file.
 
-If you want to allow other users, local or remote, to be able to access CLI connections, you must create your
-own secret file and make it possible for (only!) these users to
-read it.
+If you want to allow other users, local or remote, to be able to access
+CLI connections, you must create your own secret file and make it possible
+for (only!) these users to read it.
 
 A good way to create the secret file is::
 
@@ -168,7 +160,7 @@ Furthermore you may want to look at and lock down:
 :ref:`ref_param_vcc_unsafe_path`
 	Restrict VCL/VMODS to :ref:`ref_param_vcl_dir` and :ref:`ref_param_vmod_dir`
 
-:ref:`ref_param_vmod_dir` 
+:ref:`ref_param_vmod_dir`
         The directory where Varnish will will look
         for modules. This could potentially be used to load rouge
         modules into Varnish.
@@ -188,13 +180,9 @@ We do not currently have a way to restrict specific CLI commands
 to specific CLI connections. One way to get such an effect is to
 "wrap" all CLI access in pre-approved scripts which use `varnishadm(1)`
 
-.. XXX:what does the 1 stand for? benc
-
 to submit the sanitized CLI commands, and restrict a remote user
 to only those scripts, for instance using sshd(8)'s configuration.
 
-.. XXX:what does the 8 stand for? benc
-
 VCL programs
 ------------
 
@@ -214,9 +202,9 @@ lower the privilege of a child process...
 Inline-C is disabled by default starting with Varnish version 4, so unless
 you enable it, you don't have to worry about it.
 
-The parameters mentioned above can restrict the loading of VMODs to only 
-be loaded from a designated directory, restricting VCL wranglers
-to a pre-approved subset of VMODs.
+The parameters mentioned above can restrict the loading of VMODs to only
+be loaded from a designated directory, restricting VCL wranglers to a
+pre-approved subset of VMODs.
 
 If you do that, we are confident that your local system cannot be compromised
 from VCL code.



From fgsch at lodoss.net  Wed Oct 14 17:18:41 2015
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Wed, 14 Oct 2015 19:18:41 +0200
Subject: [master] dadd2ae This requires feature user_varnish as well
Message-ID: <E1ZmPhJ-0000Oe-FY@project.varnish-software.com>


commit dadd2ae08afa11dc4b66114fff8d3fbc9a4de177
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Wed Oct 14 09:58:57 2015 +0100

    This requires feature user_varnish as well

diff --git a/bin/varnishtest/tests/j00002.vtc b/bin/varnishtest/tests/j00002.vtc
index cb58437..1ab0ade 100644
--- a/bin/varnishtest/tests/j00002.vtc
+++ b/bin/varnishtest/tests/j00002.vtc
@@ -6,4 +6,7 @@ err_shell "unknown sub-argument" "${varnishd} -junix,bla=foo 2>&1"
 err_shell "user not found" "${varnishd} -junix,user=/// 2>&1"
 err_shell "user not found" "${varnishd} -junix,workuser=/// 2>&1"
 err_shell "group not found" "${varnishd} -junix,ccgroup=/// 2>&1"
+
+feature user_varnish
+
 err_shell "have different login groups" "${varnishd} -junix,workuser=root 2>&1"



From fgsch at lodoss.net  Wed Oct 14 17:18:41 2015
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Wed, 14 Oct 2015 19:18:41 +0200
Subject: [master] 8dd603f Expose accept_filter parameter to Linux
Message-ID: <E1ZmPhJ-0000Ok-Gp@project.varnish-software.com>


commit 8dd603f6de55a0f09641ee438a8d3dc342de7532
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Tue Oct 13 17:11:13 2015 +0100

    Expose accept_filter parameter to Linux
    
    Disabled by default. Prompted by Rafael Zalamena
    (rafaelfz at taghos dot com dot br) on varnish-dev.

diff --git a/include/tbl/params.h b/include/tbl/params.h
index d3a2982..c7f1479 100644
--- a/include/tbl/params.h
+++ b/include/tbl/params.h
@@ -30,21 +30,23 @@
 
 /*lint -save -e525 -e539 */
 
-#ifdef HAVE_ACCEPT_FILTERS
 PARAM(
 	/* name */	accept_filter,
 	/* typ */	bool,
 	/* min */	NULL,
 	/* max */	NULL,
+#if defined(HAVE_ACCEPT_FILTERS) || defined(__linux)
 	/* default */	"on",
+#else
+	/* default */	"off",
+#endif /* HAVE_ACCEPT_FILTERS || __linux */
 	/* units */	"bool",
 	/* flags */	MUST_RESTART,
 	/* s-text */
-	"Enable kernel accept-filters, (if available in the kernel).",
+	"Enable kernel accept-filters (if available in the kernel).",
 	/* l-text */	NULL,
 	/* func */	NULL
 )
-#endif /* HAVE_ACCEPT_FILTERS */
 
 PARAM(
 	/* name */	acceptor_sleep_decay,



From fgsch at lodoss.net  Wed Oct 14 21:36:10 2015
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Wed, 14 Oct 2015 23:36:10 +0200
Subject: [master] a5b9d6b Really expose accept_filter parameter to Linux
Message-ID: <E1ZmTiU-000725-K3@project.varnish-software.com>


commit a5b9d6b01cfc98fddfa0734bdb4f21a38bbfe801
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Wed Oct 14 22:33:26 2015 +0100

    Really expose accept_filter parameter to Linux
    
    This was missing on my previous commit.

diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c
index 9736ca9..a17351d 100644
--- a/bin/varnishd/cache/cache_acceptor.c
+++ b/bin/varnishd/cache/cache_acceptor.c
@@ -489,7 +489,6 @@ vca_acct(void *arg)
 		assert (ls->sock > 0);		// We know where stdin is
 		AZ(listen(ls->sock, cache_param->listen_depth));
 		vca_tcp_opt_set(ls->sock, 1);
-#ifdef HAVE_ACCEPT_FILTERS
 		if (cache_param->accept_filter) {
 			int i;
 			i = VTCP_filter_http(ls->sock);
@@ -498,7 +497,6 @@ vca_acct(void *arg)
 				    "Kernel filtering: sock=%d, ret=%d %s",
 				    ls->sock, i, strerror(errno));
 		}
-#endif /* HAVE_ACCEPT_FILTERS */
 	}
 
 	need_test = 1;



From fgsch at lodoss.net  Wed Oct 14 21:44:44 2015
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Wed, 14 Oct 2015 23:44:44 +0200
Subject: [master] 9a8e69f Add some tests using std.ip() with IPv6
Message-ID: <E1ZmTqm-00087j-4w@project.varnish-software.com>


commit 9a8e69f26b9981493c8c699bad63a34e7d0570d3
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Wed Oct 14 22:41:50 2015 +0100

    Add some tests using std.ip() with IPv6

diff --git a/bin/varnishtest/tests/m00011.vtc b/bin/varnishtest/tests/m00011.vtc
index 29c4261..bf96dd5 100644
--- a/bin/varnishtest/tests/m00011.vtc
+++ b/bin/varnishtest/tests/m00011.vtc
@@ -16,6 +16,8 @@ varnish v1 -arg "-Wpoll" -vcl+backend {
 		set resp.http.foo1 = std.ip("9.9.9.*", server.ip);
 		set resp.http.foo2 = std.ip("1.2.3.*", "127.0.0.2");
 		set resp.http.foo3 = std.ip("1.2.3.5", "127.0.0.3");
+		set resp.http.foo4 = std.ip("2001:db8::", "[::1]");
+		set resp.http.foo5 = std.ip("2001::db8::", "[::1]");
 	}
 } -start
 
@@ -27,6 +29,8 @@ client c1 {
 	expect resp.http.foo1 == "127.0.0.1"
 	expect resp.http.foo2 == "127.0.0.2"
 	expect resp.http.foo3 == "1.2.3.5"
+	expect resp.http.foo4 == "2001:db8::"
+	expect resp.http.foo5 == "::1"
 	txreq -url "/foo2"
 	rxresp
 	expect resp.bodylen == 2



From fgsch at lodoss.net  Wed Oct 14 22:19:38 2015
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Thu, 15 Oct 2015 00:19:38 +0200
Subject: [master] 405e771 Document vcl.state and correct existing mention
Message-ID: <E1ZmUOY-00045F-K0@project.varnish-software.com>


commit 405e7711387bd4429fda757b2909524cebe837a6
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Wed Oct 14 23:13:42 2015 +0100

    Document vcl.state and correct existing mention
    
    Partially addresses #1795.

diff --git a/doc/sphinx/reference/varnish-cli.rst b/doc/sphinx/reference/varnish-cli.rst
index 9a2d0de..934ff1e 100644
--- a/doc/sphinx/reference/varnish-cli.rst
+++ b/doc/sphinx/reference/varnish-cli.rst
@@ -119,6 +119,10 @@ vcl.list
 vcl.show [-v] <configname>
   Display the source code for the specified configuration.
 
+vcl.state <configname> <state>
+  Force the state of the specified configuration.
+  State is any of auto, warm or cold values.
+
 param.show [-l] [<param>]
   Show parameters and their values.
 
diff --git a/doc/sphinx/whats-new/changes.rst b/doc/sphinx/whats-new/changes.rst
index d9b0825..22e61a0 100644
--- a/doc/sphinx/whats-new/changes.rst
+++ b/doc/sphinx/whats-new/changes.rst
@@ -44,11 +44,12 @@ Output from `vcl.list`::
     available  auto/warm       0 62f5275f-a937-4df9-9fbb-c12336bdfdb8
 
 
-A single VCL's state can be chanced with the `vcl.state` call in
+A single VCL's state can be changed with the `vcl.state` call in
 ``varnishadm``::
 
-    vcl.state <configname> [auto|cold|warm]
-        Force the state of the named configuration.
+    vcl.state <configname> <state>
+        Force the state of the specified configuration.
+        State is any of auto, warm or cold values.
 
 Example::
 



From bzeeb-lists at lists.zabbadoz.net  Thu Oct 15 01:18:16 2015
From: bzeeb-lists at lists.zabbadoz.net (Bjoern A. Zeeb)
Date: Thu, 15 Oct 2015 01:18:16 +0000
Subject: [master] 9a8e69f Add some tests using std.ip() with IPv6
In-Reply-To: <E1ZmTqm-00087j-4w@project.varnish-software.com>
References: <E1ZmTqm-00087j-4w@project.varnish-software.com>
Message-ID: <6C1DDFC8-659A-4138-BBFE-B6A06278EA94@lists.zabbadoz.net>


> On 14 Oct 2015, at 21:44 , Federico G. Schwindt <fgsch at lodoss.net> wrote:
> 
> 
> commit 9a8e69f26b9981493c8c699bad63a34e7d0570d3
> Author: Federico G. Schwindt <fgsch at lodoss.net>
> Date:   Wed Oct 14 22:41:50 2015 +0100
> 
>    Add some tests using std.ip() with IPv6
> 
> diff --git a/bin/varnishtest/tests/m00011.vtc b/bin/varnishtest/tests/m00011.vtc
> index 29c4261..bf96dd5 100644
> --- a/bin/varnishtest/tests/m00011.vtc
> +++ b/bin/varnishtest/tests/m00011.vtc
> @@ -16,6 +16,8 @@ varnish v1 -arg "-Wpoll" -vcl+backend {
> 		set resp.http.foo1 = std.ip("9.9.9.*", server.ip);
> 		set resp.http.foo2 = std.ip("1.2.3.*", "127.0.0.2");
> 		set resp.http.foo3 = std.ip("1.2.3.5", "127.0.0.3");
> +		set resp.http.foo4 = std.ip("2001:db8::", "[::1]");
> +		set resp.http.foo5 = std.ip(?2001::db8::", "[::1]");

That second ne is not a documentation prefix anymore but a valid address assigned to people.  Not that anyone seems to care here for IPv4, but .. maybe one could use test or benchmark blocks properly.  There?s plenty of addresses (both legacy and IPv6) st aside for this.

/bz




From lkarsten at varnish-software.com  Thu Oct 15 13:40:31 2015
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 15 Oct 2015 15:40:31 +0200
Subject: [master] 8dd603f Expose accept_filter parameter to Linux
In-Reply-To: <E1ZmPhJ-0000Ok-Gp@project.varnish-software.com>
References: <E1ZmPhJ-0000Ok-Gp@project.varnish-software.com>
Message-ID: <20151015134029.GA24008@immer.varnish-software.com>

On Wed, Oct 14, 2015 at 07:18:41PM +0200, Federico G. Schwindt wrote:
> commit 8dd603f6de55a0f09641ee438a8d3dc342de7532
> Author: Federico G. Schwindt <fgsch at lodoss.net>
> Date:   Tue Oct 13 17:11:13 2015 +0100
> 
>     Expose accept_filter parameter to Linux
>     
>     Disabled by default. Prompted by Rafael Zalamena
>     (rafaelfz at taghos dot com dot br) on varnish-dev.
[cut]
> +#if defined(HAVE_ACCEPT_FILTERS) || defined(__linux)
>  	/* default */	"on",
> +#else
> +	/* default */	"off",
> +#endif /* HAVE_ACCEPT_FILTERS || __linux */

Isn't __linux always defined on linux, so "enabled by default"?


-- 
Lasse Karstensen
Varnish Software AS



From fgsch at lodoss.net  Thu Oct 15 20:27:11 2015
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Thu, 15 Oct 2015 22:27:11 +0200
Subject: [master] 006760c Use `all' for vars marked as all
Message-ID: <E1Zmp7H-0001CI-5P@project.varnish-software.com>


commit 006760c39c2f8457958e11196c27941bb00d8523
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Thu Oct 15 15:24:22 2015 +0100

    Use `all' for vars marked as all
    
    Previously it said `vcl_all' which doesn't exist.  Minor cosmetics while
    I'm here.

diff --git a/lib/libvcc/generate.py b/lib/libvcc/generate.py
index 062d180..f8aeb79 100755
--- a/lib/libvcc/generate.py
+++ b/lib/libvcc/generate.py
@@ -690,8 +690,8 @@ sp_variables = [
 		( 'deliver', 'synth', ), """
 		The HTTP status code that will be returned.
 
-                Assigning a HTTP standardized code to resp.status will also
-                set resp.reason to the corresponding status message.
+		Assigning a HTTP standardized code to resp.status will also
+		set resp.reason to the corresponding status message.
 		"""
 	),
 	('resp.reason',
@@ -1316,6 +1316,8 @@ def rst_where(fo, h, l):
 			ll.append(j)
 		elif j == "backend":
 			ll.append(j)
+		elif j == "all":
+			ll.append(j)
 		else:
 			ll.append("vcl_" + j)
 	for j in ll:



From phk at FreeBSD.org  Mon Oct 19 09:44:11 2015
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 19 Oct 2015 11:44:11 +0200
Subject: [master] 5793190 It is not kosher to have #ifdefs inside macros.
Message-ID: <E1Zo6zD-0001WJ-4p@project.varnish-software.com>


commit 57931908ca6f292f602883ca0872ebc6f9c27d56
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Oct 19 09:43:29 2015 +0000

    It is not kosher to have #ifdefs inside macros.

diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c
index 899b9ba..5ceb80d 100644
--- a/bin/varnishd/mgt/mgt_main.c
+++ b/bin/varnishd/mgt/mgt_main.c
@@ -395,6 +395,10 @@ init_params(struct cli *cli)
 		MCF_SetDefault("gzip_buffer", "4k");
 	}
 
+#if !defined(HAVE_ACCEPT_FILTERS) || defined(__linux)
+	MCF_SetDefault("accept_filter", "off");
+#endif
+
 	low = sysconf(_SC_THREAD_STACK_MIN);
 	bprintf(stackmin, "%jd", (intmax_t)low);
 	MCF_SetMinimum("thread_pool_stack", stackmin);
diff --git a/include/tbl/params.h b/include/tbl/params.h
index c7f1479..d2650f4 100644
--- a/include/tbl/params.h
+++ b/include/tbl/params.h
@@ -35,11 +35,7 @@ PARAM(
 	/* typ */	bool,
 	/* min */	NULL,
 	/* max */	NULL,
-#if defined(HAVE_ACCEPT_FILTERS) || defined(__linux)
 	/* default */	"on",
-#else
-	/* default */	"off",
-#endif /* HAVE_ACCEPT_FILTERS || __linux */
 	/* units */	"bool",
 	/* flags */	MUST_RESTART,
 	/* s-text */



From lkarsten at varnish-software.com  Mon Oct 19 15:19:17 2015
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Mon, 19 Oct 2015 17:19:17 +0200
Subject: [4.0] ef6f15e Allow invalid headers in 304 backend responses.
Message-ID: <E1ZoCDV-0001VO-E6@project.varnish-software.com>


commit ef6f15e5521829d60cb44233b8d5fe5ad29f3620
Author: Lasse Karstensen <lkarsten at varnish-software.com>
Date:   Mon Oct 19 17:01:20 2015 +0200

    Allow invalid headers in 304 backend responses.
    
    Allow the backend server to send headers lacking ":"/colon in
    them when responding to a conditional request yielding a 304 response.
    
    In master/4.1 such responses are aborted as invalid. The backend is
    clearly not feeling well.
    
    Since we've accepted it nicely for 200 responses so far in Varnish 4.0,
    continue that trend also for 304s.
    
    Fixes: #1598

diff --git a/bin/varnishd/cache/cache_http.c b/bin/varnishd/cache/cache_http.c
index 572d987..24e4444 100644
--- a/bin/varnishd/cache/cache_http.c
+++ b/bin/varnishd/cache/cache_http.c
@@ -805,7 +805,8 @@ http_Merge(const struct http *fm, struct http *to, int not_ce)
 	}
 	for (v = HTTP_HDR_FIRST; v < to->nhd; v++) {
 		p = strchr(to->hd[v].b, ':');
-		AN(p);
+		if (!p)
+			p = 0;
 		u = http_findhdr(fm, p - to->hd[v].b, to->hd[v].b);
 		if (u)
 			fm->hdf[u] &= ~HDF_MARKER;
diff --git a/bin/varnishtest/tests/r01598.vtc b/bin/varnishtest/tests/r01598.vtc
new file mode 100644
index 0000000..3264e9d
--- /dev/null
+++ b/bin/varnishtest/tests/r01598.vtc
@@ -0,0 +1,39 @@
+varnishtest "#1598 - Missing ':' in server response headers with backend IMS"
+
+server s1 {
+	rxreq
+	txresp -hdr "ETag: \"tag\"" -hdr "foo"
+
+	rxreq
+	expect req.http.if-none-match == "\"tag\""
+	txresp -status 304 -nolen \
+		-hdr "ETag: \"tag\"" \
+		-hdr "foo"
+} -start
+
+varnish v1 -vcl+backend {
+	sub vcl_backend_response {
+		set beresp.ttl = 1s;
+		set beresp.grace = 0s;
+		set beresp.keep = 60s;
+	}
+} -start
+
+varnish v1 -cliok "param.set debug +syncvsl"
+
+client c1 {
+	txreq
+	rxresp
+	expect resp.status == 200
+	# varnishtest doesn't accept the faulty header, and there
+	# is no "resp.headers" to do a regex match against.
+	expect resp.http.foo == <undef>
+} -run
+
+delay 1
+
+client c1 {
+	txreq
+	rxresp
+	expect resp.status == 200
+} -run
diff --git a/doc/changes.rst b/doc/changes.rst
index 2ecd3af..fcd83df 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -17,12 +17,14 @@ Bugs fixed
 .. _1691: https://www.varnish-cache.org/trac/ticket/1691
 .. _1688: https://www.varnish-cache.org/trac/ticket/1688
 .. _1602: https://www.varnish-cache.org/trac/ticket/1602
+.. _1598: https://www.varnish-cache.org/trac/ticket/1598
 
 - 1744_ - Update the users guide to for new -sfile syntax
 - 1742_ - Document varnishlog -w/-r with more details
 - 1691_ - Fail fetch on malformed Content-Length header
 - 1688_ - Add a VDP_pretend_gzip for use with synth bodies in ESI includes with gzip
 - 1602_ - Deal with known zero length objects properly when handling do_gzip/do_gunzip
+- 1598_ - Allow invalid headers in 304 backend responses.
 
 
 ============================================



From lkarsten at varnish-software.com  Mon Oct 19 15:48:39 2015
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Mon, 19 Oct 2015 17:48:39 +0200
Subject: [4.0] 6b2cc80 Skip headers on errors.
Message-ID: <E1ZoCfv-0007kW-OQ@project.varnish-software.com>


commit 6b2cc8091125bd505a3762cda338c3593bd3e720
Author: Lasse Karstensen <lkarsten at varnish-software.com>
Date:   Mon Oct 19 17:44:10 2015 +0200

    Skip headers on errors.
    
    This didn't make any sense and should never have been commited.

diff --git a/bin/varnishd/cache/cache_http.c b/bin/varnishd/cache/cache_http.c
index 24e4444..982af71 100644
--- a/bin/varnishd/cache/cache_http.c
+++ b/bin/varnishd/cache/cache_http.c
@@ -806,7 +806,7 @@ http_Merge(const struct http *fm, struct http *to, int not_ce)
 	for (v = HTTP_HDR_FIRST; v < to->nhd; v++) {
 		p = strchr(to->hd[v].b, ':');
 		if (!p)
-			p = 0;
+			continue;
 		u = http_findhdr(fm, p - to->hd[v].b, to->hd[v].b);
 		if (u)
 			fm->hdf[u] &= ~HDF_MARKER;



From perbu at varnish-software.com  Tue Oct 20 11:17:58 2015
From: perbu at varnish-software.com (Per Buer)
Date: Tue, 20 Oct 2015 13:17:58 +0200
Subject: [master] ed6dca6 backend threads fail tovcl_backend_error
Message-ID: <E1ZoUvW-0007dI-JK@project.varnish-software.com>


commit ed6dca6b9ced699591470ddae65d3fab862d1b73
Author: Per Buer <perbu at varnish-software.com>
Date:   Tue Oct 20 13:17:56 2015 +0200

    backend threads fail tovcl_backend_error

diff --git a/doc/sphinx/users-guide/vcl-built-in-subs.rst b/doc/sphinx/users-guide/vcl-built-in-subs.rst
index a1eba28..12d3377 100644
--- a/doc/sphinx/users-guide/vcl-built-in-subs.rst
+++ b/doc/sphinx/users-guide/vcl-built-in-subs.rst
@@ -268,8 +268,8 @@ The `vcl_backend_fetch` subroutine may terminate with calling
 
   ``abandon``
     Abandon the backend request. Unless the backend request was a
-    background fetch, control is passed to :ref:`vcl_synth` on the
-    client side with ``resp.status`` preset to 503.
+    background fetch, control is passed to :ref:`vcl_backend_error`
+    with ``resp.status`` preset to 503.
 
 .. _vcl_backend_response:
 



From phk at FreeBSD.org  Wed Oct 21 07:36:42 2015
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 21 Oct 2015 09:36:42 +0200
Subject: [master] 233b132 Polish the heritage structure a little bit,
 and don't pass the panic_string to syslog until we have washed it.
Message-ID: <E1Zonww-0003NS-Do@project.varnish-software.com>


commit 233b1328a937b243b87dadc858e612990cf66018
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Oct 21 07:36:05 2015 +0000

    Polish the heritage structure a little bit, and don't pass the
    panic_string to syslog until we have washed it.

diff --git a/bin/varnishd/cache/cache_vrt_var.c b/bin/varnishd/cache/cache_vrt_var.c
index c51007b..5dd890e 100644
--- a/bin/varnishd/cache/cache_vrt_var.c
+++ b/bin/varnishd/cache/cache_vrt_var.c
@@ -623,7 +623,7 @@ VRT_r_server_identity(VRT_CTX)
 {
 
 	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
-	if (heritage.identity[0] != '\0')
+	if (heritage.identity != NULL)
 		return (heritage.identity);
 	else
 		return (heritage.name);
diff --git a/bin/varnishd/common/heritage.h b/bin/varnishd/common/heritage.h
index 27c3a33..42e9ec7 100644
--- a/bin/varnishd/common/heritage.h
+++ b/bin/varnishd/common/heritage.h
@@ -56,7 +56,6 @@ struct heritage {
 
 	/* Sockets from which to accept connections */
 	struct listen_sock_head		socks;
-	unsigned			nsocks;
 
 	/* Hash method */
 	const struct hash_slinger	*hash;
@@ -66,7 +65,7 @@ struct heritage {
 	struct params			*param;
 
 	char				*name;
-	char                            identity[1024];
+	const char			*identity;
 
 	char				*panic_str;
 	ssize_t				panic_str_len;
diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c
index 1d2a7a3..9308fa3 100644
--- a/bin/varnishd/mgt/mgt_child.c
+++ b/bin/varnishd/mgt/mgt_child.c
@@ -125,18 +125,17 @@ mgt_panic_record(pid_t r)
 {
 	char time_str[30];
 
-	AN(heritage.panic_str[0]);
-	REPORT(LOG_ERR, "Child (%jd) Panic message:\n%s",
-	    (intmax_t)r, heritage.panic_str);
-
 	if (child_panic != NULL)
 		VSB_delete(child_panic);
 	child_panic = VSB_new_auto();
 	AN(child_panic);
 	VTIM_format(VTIM_real(), time_str);
-	VSB_printf(child_panic, "Last panic at: %s\n", time_str);
-	VSB_cat(child_panic, heritage.panic_str);
+	VSB_printf(child_panic, "Panic at: %s\n", time_str);
+	VSB_quote(child_panic, heritage.panic_str,
+	    strnlen(heritage.panic_str, heritage.panic_str_len), 0);
 	AZ(VSB_finish(child_panic));
+	REPORT(LOG_ERR, "Child (%jd) %s",
+	    (intmax_t)r, VSB_data(child_panic));
 }
 
 static void
diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c
index 5ceb80d..5dea3d4 100644
--- a/bin/varnishd/mgt/mgt_main.c
+++ b/bin/varnishd/mgt/mgt_main.c
@@ -424,9 +424,10 @@ identify(const char *i_arg)
 	strcpy(id, "varnishd");
 
 	if (i_arg != NULL) {
-		if (strlen(i_arg) + 1 > sizeof heritage.identity)
-			ARGV_ERR("Identity (-i) name too long.\n");
-		strcpy(heritage.identity, i_arg);
+		if (strlen(i_arg) + 1 > 1024)
+			ARGV_ERR("Identity (-i) name too long (max 1023).\n");
+		heritage.identity = strdup(i_arg);
+		AN(heritage.identity);
 		i = strlen(id);
 		id[i++] = '/';
 		for (; i < (sizeof(id) - 1L); i++) {



From phk at FreeBSD.org  Wed Oct 21 08:49:18 2015
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 21 Oct 2015 10:49:18 +0200
Subject: [master] 6544de0 Centralize mgt's include of common/params.h
Message-ID: <E1Zop5C-0006YA-Bo@project.varnish-software.com>


commit 6544de0b728152b2aae13989e8665b5e1240ab74
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Oct 21 08:42:32 2015 +0000

    Centralize mgt's include of common/params.h

diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h
index 18356dc..3b44172 100644
--- a/bin/varnishd/mgt/mgt.h
+++ b/bin/varnishd/mgt/mgt.h
@@ -31,6 +31,7 @@
 #include <stdint.h>
 
 #include "common/common.h"
+#include "common/params.h"
 
 struct cli;
 struct parspec;
diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c
index 9308fa3..0195245 100644
--- a/bin/varnishd/mgt/mgt_child.c
+++ b/bin/varnishd/mgt/mgt_child.c
@@ -45,7 +45,6 @@
 
 #include "mgt/mgt.h"
 #include "common/heritage.h"
-#include "common/params.h"
 
 #include "vbm.h"
 #include "vcli.h"
diff --git a/bin/varnishd/mgt/mgt_cli.c b/bin/varnishd/mgt/mgt_cli.c
index da3cf69..125a91d 100644
--- a/bin/varnishd/mgt/mgt_cli.c
+++ b/bin/varnishd/mgt/mgt_cli.c
@@ -45,7 +45,6 @@
 #include <unistd.h>
 
 #include "mgt/mgt.h"
-#include "common/params.h"
 
 #include "vcli.h"
 #include "vcli_common.h"
diff --git a/bin/varnishd/mgt/mgt_jail_solaris.c b/bin/varnishd/mgt/mgt_jail_solaris.c
index 16f3ccc..142a43c 100644
--- a/bin/varnishd/mgt/mgt_jail_solaris.c
+++ b/bin/varnishd/mgt/mgt_jail_solaris.c
@@ -216,7 +216,6 @@
 #include "mgt/mgt.h"
 
 #include "common/heritage.h"
-#include "common/params.h"
 
 #ifdef HAVE_PRIV_H
 #include <priv.h>
diff --git a/bin/varnishd/mgt/mgt_param.c b/bin/varnishd/mgt/mgt_param.c
index 1fd3f33..2a52709 100644
--- a/bin/varnishd/mgt/mgt_param.c
+++ b/bin/varnishd/mgt/mgt_param.c
@@ -36,7 +36,6 @@
 
 #include "mgt/mgt.h"
 #include "common/heritage.h"
-#include "common/params.h"
 
 #include "mgt/mgt_param.h"
 #include "vav.h"
diff --git a/bin/varnishd/mgt/mgt_param_bits.c b/bin/varnishd/mgt/mgt_param_bits.c
index 627c7a9..4713ac6 100644
--- a/bin/varnishd/mgt/mgt_param_bits.c
+++ b/bin/varnishd/mgt/mgt_param_bits.c
@@ -33,7 +33,6 @@
 #include <string.h>
 #include <stdio.h>
 
-#include "common/params.h"
 #include "mgt/mgt.h"
 #include "mgt/mgt_param.h"
 
diff --git a/bin/varnishd/mgt/mgt_param_tbl.c b/bin/varnishd/mgt/mgt_param_tbl.c
index 876b6d3..129300f 100644
--- a/bin/varnishd/mgt/mgt_param_tbl.c
+++ b/bin/varnishd/mgt/mgt_param_tbl.c
@@ -32,7 +32,6 @@
 #include <stdio.h>
 
 #include "mgt/mgt.h"
-#include "common/params.h"
 
 #include "mgt/mgt_param.h"
 
diff --git a/bin/varnishd/mgt/mgt_param_tcp.c b/bin/varnishd/mgt/mgt_param_tcp.c
index 12baf83..ab8f6d4 100644
--- a/bin/varnishd/mgt/mgt_param_tcp.c
+++ b/bin/varnishd/mgt/mgt_param_tcp.c
@@ -44,7 +44,6 @@
 #include <unistd.h>
 
 #include "mgt/mgt.h"
-#include "common/params.h"
 
 #include "vtcp.h"
 
diff --git a/bin/varnishd/mgt/mgt_param_tweak.c b/bin/varnishd/mgt/mgt_param_tweak.c
index b5f653c..00ad35c 100644
--- a/bin/varnishd/mgt/mgt_param_tweak.c
+++ b/bin/varnishd/mgt/mgt_param_tweak.c
@@ -40,7 +40,6 @@
 #include <unistd.h>
 
 #include "mgt/mgt.h"
-#include "common/params.h"
 
 #include "mgt/mgt_param.h"
 #include "vav.h"
diff --git a/bin/varnishd/mgt/mgt_pool.c b/bin/varnishd/mgt/mgt_pool.c
index 7affacd..c08a2af 100644
--- a/bin/varnishd/mgt/mgt_pool.c
+++ b/bin/varnishd/mgt/mgt_pool.c
@@ -45,7 +45,6 @@
 #include <stdio.h>
 
 #include "mgt/mgt.h"
-#include "common/params.h"
 
 #include "mgt/mgt_param.h"
 
diff --git a/bin/varnishd/mgt/mgt_sandbox.c b/bin/varnishd/mgt/mgt_sandbox.c
index 1f0962d..785014c 100644
--- a/bin/varnishd/mgt/mgt_sandbox.c
+++ b/bin/varnishd/mgt/mgt_sandbox.c
@@ -59,7 +59,6 @@
 #include <unistd.h>
 
 #include "mgt/mgt.h"
-#include "common/params.h"
 #include "mgt/mgt_param.h"
 
 #include <vsub.h>
diff --git a/bin/varnishd/mgt/mgt_shmem.c b/bin/varnishd/mgt/mgt_shmem.c
index 1afcfbf..9345d38 100644
--- a/bin/varnishd/mgt/mgt_shmem.c
+++ b/bin/varnishd/mgt/mgt_shmem.c
@@ -43,7 +43,6 @@
 
 #include "mgt/mgt.h"
 #include "common/heritage.h"
-#include "common/params.h"
 
 #include "flopen.h"
 #include "vsm_priv.h"
diff --git a/bin/varnishd/mgt/mgt_vcc.c b/bin/varnishd/mgt/mgt_vcc.c
index 8a72f20..d13432e 100644
--- a/bin/varnishd/mgt/mgt_vcc.c
+++ b/bin/varnishd/mgt/mgt_vcc.c
@@ -40,7 +40,6 @@
 #include <unistd.h>
 #include <sys/stat.h>
 
-#include "common/params.h"
 #include "mgt/mgt.h"
 
 #include "libvcc.h"
diff --git a/bin/varnishd/mgt/mgt_vcl.c b/bin/varnishd/mgt/mgt_vcl.c
index 7971346..bb298bb 100644
--- a/bin/varnishd/mgt/mgt_vcl.c
+++ b/bin/varnishd/mgt/mgt_vcl.c
@@ -39,7 +39,6 @@
 #include <sys/stat.h>
 
 #include "mgt/mgt.h"
-#include "common/params.h"
 
 #include "vcli.h"
 #include "vcli_priv.h"



From phk at FreeBSD.org  Wed Oct 21 08:49:18 2015
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 21 Oct 2015 10:49:18 +0200
Subject: [master] 6e5ffc0 Go over mgt process's complaining on stderr and
 syslog and try to make somewhat more predictable.
Message-ID: <E1Zop5C-0006YI-D5@project.varnish-software.com>


commit 6e5ffc081c569737e7ea8d1380f43b53b37d74a2
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Oct 21 08:48:24 2015 +0000

    Go over mgt process's complaining on stderr and syslog and try to
    make somewhat more predictable.
    
    Disable syslogging when we're running under varnishtest.

diff --git a/bin/varnishd/common/params.h b/bin/varnishd/common/params.h
index 3dbfe6a..44f2f2a 100644
--- a/bin/varnishd/common/params.h
+++ b/bin/varnishd/common/params.h
@@ -29,6 +29,11 @@
  * This file contains the heritage passed when mgt forks cache
  */
 
+#ifdef COMMON_PARAMS_H
+#error "Multiple includes of common/params.h"
+#endif
+#define COMMON_PARAMS_H
+
 #include <stdint.h>
 
 #include "vre.h"
diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h
index 3b44172..5fb12d4 100644
--- a/bin/varnishd/mgt/mgt.h
+++ b/bin/varnishd/mgt/mgt.h
@@ -123,6 +123,13 @@ struct choice {
 };
 const void *pick(const struct choice *cp, const char *which, const char *kind);
 
+extern const char C_ERR[];	// Things are not as they should be
+extern const char C_INFO[];	// Normal stuff, keep a record for later
+extern const char C_DEBUG[];	// More detail than you'd normally want
+extern const char C_SECURITY[];	// Security issues
+extern const char C_CLI[];	// CLI traffic between master and child
+void MGT_complain(const char *loud, const char *, ...) __v_printflike(2, 3);
+
 /* mgt_param.c */
 void MCF_InitParams(struct cli *);
 void MCF_CollectParams(void);
@@ -169,18 +176,20 @@ extern unsigned mgt_vcc_err_unref;
 extern unsigned mgt_vcc_allow_inline_c;
 extern unsigned mgt_vcc_unsafe_path;
 
-#define REPORT0(pri, fmt)				\
-	do {						\
-		fprintf(stderr, fmt "\n");		\
-		syslog(pri, fmt);			\
-	} while (0)
-
-#define REPORT(pri, fmt, ...)				\
-	do {						\
-		fprintf(stderr, fmt "\n", __VA_ARGS__);	\
-		syslog(pri, fmt, __VA_ARGS__);		\
-	} while (0)
-
 #if defined(PTHREAD_CANCELED) || defined(PTHREAD_MUTEX_DEFAULT)
 #error "Keep pthreads out of in manager process"
 #endif
+
+static inline int
+MGT_FEATURE(enum feature_bits x)
+{
+	return (mgt_param.feature_bits[(unsigned)x>>3] &
+	    (0x80U >> ((unsigned)x & 7)));
+}
+
+static inline int
+MGT_DO_DEBUG(enum debug_bits x)
+{
+	return (mgt_param.debug_bits[(unsigned)x>>3] &
+	    (0x80U >> ((unsigned)x & 7)));
+}
diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c
index 0195245..3784f13 100644
--- a/bin/varnishd/mgt/mgt_child.c
+++ b/bin/varnishd/mgt/mgt_child.c
@@ -86,20 +86,6 @@ static struct vlu	*child_std_vlu;
 static struct vsb *child_panic = NULL;
 static double mgt_uptime_t0 = 0.;
 
-/* XXX: Doesn't really belong here, but only place we use it */
-static inline int
-MGT_FEATURE(enum feature_bits x)
-{
-	return (mgt_param.feature_bits[(unsigned)x>>3] &
-	    (0x80U >> ((unsigned)x & 7)));
-}
-static inline int
-MGT_DO_DEBUG(enum debug_bits x)
-{
-	return (mgt_param.debug_bits[(unsigned)x>>3] &
-	    (0x80U >> ((unsigned)x & 7)));
-}
-
 static void mgt_reap_child(void);
 
 /*---------------------------------------------------------------------
@@ -133,7 +119,7 @@ mgt_panic_record(pid_t r)
 	VSB_quote(child_panic, heritage.panic_str,
 	    strnlen(heritage.panic_str, heritage.panic_str_len), 0);
 	AZ(VSB_finish(child_panic));
-	REPORT(LOG_ERR, "Child (%jd) %s",
+	MGT_complain(C_ERR, "Child (%jd) %s",
 	    (intmax_t)r, VSB_data(child_panic));
 }
 
@@ -237,7 +223,7 @@ child_line(void *priv, const char *p)
 {
 	(void)priv;
 
-	REPORT(LOG_NOTICE, "Child (%jd) said %s", (intmax_t)child_pid, p);
+	MGT_complain(C_INFO, "Child (%jd) said %s", (intmax_t)child_pid, p);
 	return (0);
 }
 
@@ -320,7 +306,7 @@ mgt_launch_child(struct cli *cli)
 			VCLI_SetResult(cli, CLIS_CANT);
 			return;
 		}
-		REPORT0(LOG_ERR,
+		MGT_complain(C_ERR,
 		    "Child start failed: could not open sockets");
 		return;
 	}
@@ -394,7 +380,7 @@ mgt_launch_child(struct cli *cli)
 		exit(0);
 	}
 	assert(pid > 1);
-	REPORT(LOG_NOTICE, "child (%jd) Started", (intmax_t)pid);
+	MGT_complain(C_DEBUG, "Child (%jd) Started", (intmax_t)pid);
 	VSC_C_mgt->child_start = ++static_VSC_C_mgt.child_start;
 
 	/* Close stuff the child got */
@@ -432,7 +418,8 @@ mgt_launch_child(struct cli *cli)
 	mgt_cli_start_child(child_cli_in, child_cli_out);
 	child_pid = pid;
 	if (mgt_push_vcls_and_start(&u, &p)) {
-		REPORT(LOG_ERR, "Pushing vcls failed:\n%s", p);
+		MGT_complain(C_ERR, "Child (%jd) Pushing vcls failed:\n%s",
+		    (intmax_t)child_pid, p);
 		free(p);
 		child_state = CH_RUNNING;
 		mgt_stop_child();
@@ -502,7 +489,8 @@ mgt_reap_child(void)
 	/* Compose obituary */
 	vsb = VSB_new_auto();
 	XXXAN(vsb);
-	VSB_printf(vsb, "Child (%ld) %s", (long)r, status ? "died" : "ended");
+	VSB_printf(vsb, "Child (%jd) %s", (intmax_t)r,
+	    status ? "died" : "ended");
 	if (WIFEXITED(status) && WEXITSTATUS(status)) {
 		VSB_printf(vsb, " status=%d", WEXITSTATUS(status));
 		exit_status |= 0x20;
@@ -524,7 +512,7 @@ mgt_reap_child(void)
 	}
 #endif
 	AZ(VSB_finish(vsb));
-	REPORT(LOG_INFO, "%s", VSB_data(vsb));
+	MGT_complain(status ? C_ERR : C_INFO, "%s", VSB_data(vsb));
 	VSB_delete(vsb);
 
 	/* Dispose of shared memory but evacuate panic messages first */
@@ -548,7 +536,7 @@ mgt_reap_child(void)
 
 	child_pid = -1;
 
-	REPORT0(LOG_DEBUG, "Child cleanup complete");
+	MGT_complain(C_DEBUG, "Child cleanup complete");
 
 	if (child_state == CH_DIED && mgt_param.auto_restart)
 		mgt_launch_child(NULL);
@@ -576,7 +564,7 @@ MGT_Child_Cli_Fail(void)
 		return;
 	if (child_pid < 0)
 		return;
-	REPORT(LOG_ERR, "Child (%jd) not responding to CLI, killing it.",
+	MGT_complain(C_ERR, "Child (%jd) not responding to CLI, killing it.",
 	    (intmax_t)child_pid);
 	if (MGT_FEATURE(FEATURE_NO_COREDUMP))
 		(void)kill(child_pid, SIGKILL);
@@ -599,7 +587,7 @@ mgt_stop_child(void)
 
 	child_state = CH_STOPPING;
 
-	REPORT0(LOG_DEBUG, "Stopping Child");
+	MGT_complain(C_DEBUG, "Stopping Child");
 
 	mgt_reap_child();
 }
@@ -646,7 +634,7 @@ mgt_sigint(const struct vev *e, int what)
 
 	(void)e;
 	(void)what;
-	REPORT0(LOG_ERR, "Manager got SIGINT");
+	MGT_complain(C_ERR, "Manager got SIGINT");
 	(void)fflush(stdout);
 	if (child_pid >= 0)
 		mgt_stop_child();
@@ -716,7 +704,7 @@ MGT_Run(void)
 	AZ(sigaction(SIGHUP, &sac, NULL));
 
 	if (!d_flag && !mgt_has_vcl())
-		REPORT0(LOG_ERR, "No VCL loaded yet");
+		MGT_complain(C_ERR, "No VCL loaded yet");
 	else if (!d_flag) {
 		mgt_launch_child(NULL);
 		if (child_state != CH_RUNNING) {
@@ -730,7 +718,7 @@ MGT_Run(void)
 
 	i = vev_schedule(mgt_evb);
 	if (i != 0)
-		REPORT(LOG_ERR, "vev_schedule() = %d", i);
+		MGT_complain(C_ERR, "vev_schedule() = %d", i);
 
-	REPORT0(LOG_ERR, "manager dies");
+	MGT_complain(C_INFO, "manager dies");
 }
diff --git a/bin/varnishd/mgt/mgt_cli.c b/bin/varnishd/mgt/mgt_cli.c
index 125a91d..fa021d6 100644
--- a/bin/varnishd/mgt/mgt_cli.c
+++ b/bin/varnishd/mgt/mgt_cli.c
@@ -41,7 +41,6 @@
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
-#include <syslog.h>
 #include <unistd.h>
 
 #include "mgt/mgt.h"
@@ -58,10 +57,6 @@
 
 #include "mgt_cli.h"
 
-#ifndef LOG_AUTHPRIV
-#  define LOG_AUTHPRIV	0
-#endif
-
 static int		cli_i = -1, cli_o = -1;
 static struct VCLS	*cls;
 static const char	*secret_file;
@@ -301,7 +296,7 @@ mcf_auth(struct cli *cli, const char *const *av, void *priv)
 	VCLI_AuthResponse(fd, cli->challenge, buf);
 	AZ(close(fd));
 	if (strcasecmp(buf, av[2])) {
-		syslog(LOG_WARNING|LOG_AUTHPRIV,
+		MGT_complain(C_SECURITY,
 		    "CLI Authentication failure from %s", cli->ident);
 		VCLI_SetResult(cli, CLIS_CLOSE);
 		return;
@@ -321,21 +316,20 @@ static struct cli_proto cli_auth[] = {
 };
 
 /*--------------------------------------------------------------------*/
+
 static void
 mgt_cli_cb_before(const struct cli *cli)
 {
 
-	if (mgt_param.syslog_cli_traffic)
-		syslog(LOG_NOTICE, "CLI %s Rd %s", cli->ident, cli->cmd);
+	MGT_complain(C_CLI, "CLI %s Rd %s", cli->ident, cli->cmd);
 }
 
 static void
 mgt_cli_cb_after(const struct cli *cli)
 {
 
-	if (mgt_param.syslog_cli_traffic)
-		syslog(LOG_NOTICE, "CLI %s Wr %03u %s",
-		    cli->ident, cli->result, VSB_data(cli->sb));
+	MGT_complain(C_CLI, "CLI %s Wr %03u %s",
+	    cli->ident, cli->result, VSB_data(cli->sb));
 }
 
 /*--------------------------------------------------------------------*/
@@ -610,7 +604,7 @@ Marg_connect(const struct vev *e, int what)
 
 	M_fd = VTCP_connected(M_fd);
 	if (M_fd < 0) {
-		syslog(LOG_INFO, "Could not connect to CLI-master: %m");
+		MGT_complain(C_INFO, "Could not connect to CLI-master: %m");
 		ma = VTAILQ_FIRST(&m_addr_list);
 		AN(ma);
 		VTAILQ_REMOVE(&m_addr_list, ma, list);
diff --git a/bin/varnishd/mgt/mgt_jail_solaris.c b/bin/varnishd/mgt/mgt_jail_solaris.c
index 142a43c..bb90598 100644
--- a/bin/varnishd/mgt/mgt_jail_solaris.c
+++ b/bin/varnishd/mgt/mgt_jail_solaris.c
@@ -210,7 +210,6 @@
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
-#include <syslog.h>
 #include <unistd.h>
 
 #include "mgt/mgt.h"
@@ -390,7 +389,7 @@ vjs_setup(enum jail_gen_e jge)
 	priv_set_t *priv_all;
 
 	if (! (priv_all = priv_allocset())) {
-		REPORT(LOG_ERR,
+		MGT_complain(C_SECURITY,
 		    "Solaris Jail warning: "
 		    " vjs_setup - priv_allocset failed: errno=%d (%s)",
 		    errno, strerror(errno));
@@ -423,7 +422,7 @@ vjs_privsep(enum jail_gen_e jge)
 		if (getuid() != mgt_param.uid)
 			XXXAZ(setuid(mgt_param.uid));
 	} else {
-		REPORT(LOG_INFO,
+		MGT_complain(C_SECURITY,
 		    "Privilege %s missing, will not change uid/gid",
 		    PRIV_PROC_SETID);
 	}
@@ -454,7 +453,7 @@ vjs_waive(enum jail_gen_e jge)
 	    !(inheritable = priv_allocset()) ||
 	    !(permitted = priv_allocset()) ||
 	    !(limited = priv_allocset())) {
-		REPORT(LOG_ERR,
+		MGT_complain(C_SECURITY,
 		    "Solaris Jail warning: "
 		    " vjs_waive - priv_allocset failed: errno=%d (%s)",
 		    errno, strerror(errno));
diff --git a/bin/varnishd/mgt/mgt_jail_unix.c b/bin/varnishd/mgt/mgt_jail_unix.c
index 3b3ce12..edb4653 100644
--- a/bin/varnishd/mgt/mgt_jail_unix.c
+++ b/bin/varnishd/mgt/mgt_jail_unix.c
@@ -43,7 +43,6 @@
 #include "mgt/mgt.h"
 
 #ifdef __linux__
-#include <syslog.h>
 #include <sys/prctl.h>
 #endif
 
@@ -231,7 +230,7 @@ vju_subproc(enum jail_subproc_e jse)
 	 * reenable them again.
 	 */
 	if (prctl(PR_SET_DUMPABLE, 1) != 0) {
-		REPORT0(LOG_INFO,
+		MGT_complain(C_INFO,
 		    "Could not set dumpable bit.  Core dumps turned off\n");
 	}
 #endif
diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c
index 5dea3d4..1073b8e 100644
--- a/bin/varnishd/mgt/mgt_main.c
+++ b/bin/varnishd/mgt/mgt_main.c
@@ -37,6 +37,7 @@
 #include <errno.h>
 #include <fcntl.h>
 #include <stdio.h>
+#include <stdarg.h>
 #include <stdlib.h>
 #include <string.h>
 #include <syslog.h>
@@ -120,6 +121,57 @@ build_vident(void)
 	}
 }
 
+/*--------------------------------------------------------------------
+ * 'Ello, I wish to register a complaint...
+ */
+
+#ifndef LOG_AUTHPRIV
+#  define LOG_AUTHPRIV 0
+#endif
+
+const char C_ERR[] = "Error:";
+const char C_INFO[] = "Info:";
+const char C_DEBUG[] = "Debug:";
+const char C_SECURITY[] = "Security:";
+const char C_CLI[] = "Cli:";
+
+void
+MGT_complain(const char *loud, const char *fmt, ...)
+{
+	va_list ap;
+	struct vsb *vsb;
+	int sf;
+
+	if (loud == C_CLI && !mgt_param.syslog_cli_traffic)
+		return;
+	vsb = VSB_new_auto();
+	AN(vsb);
+	va_start(ap, fmt);
+	VSB_vprintf(vsb, fmt, ap);
+	va_end(ap);
+	AZ(VSB_finish(vsb));
+
+	if (loud == C_ERR)
+		sf = LOG_ERR;
+	else if (loud == C_INFO)
+		sf = LOG_INFO;
+	else if (loud == C_DEBUG)
+		sf = LOG_DEBUG;
+	else if (loud == C_SECURITY)
+		sf = LOG_WARNING | LOG_AUTHPRIV;
+	else if (loud == C_CLI)
+		sf = LOG_INFO;
+	else
+		WRONG("Wrong complaint loudness");
+
+	if (loud != C_CLI)
+		fprintf(stderr, "%s %s\n", loud, VSB_data(vsb));
+
+	if (!MGT_DO_DEBUG(DBG_VTC_MODE))
+		syslog(sf, "%s", VSB_data(vsb));
+	VSB_delete(vsb);
+}
+
 /*--------------------------------------------------------------------*/
 
 const void *
@@ -755,9 +807,7 @@ main(int argc, char * const *argv)
 
 	assert(pfh == NULL || !VPF_Write(pfh));
 
-	if (d_flag)
-		fprintf(stderr, "Platform: %s\n", VSB_data(vident) + 1);
-	syslog(LOG_NOTICE, "Platform: %s\n", VSB_data(vident) + 1);
+	MGT_complain(C_DEBUG, "Platform: %s\n", VSB_data(vident) + 1);
 
 	mgt_pid = getpid();	/* daemon() changed this */
 
diff --git a/bin/varnishd/mgt/mgt_sandbox.c b/bin/varnishd/mgt/mgt_sandbox.c
index 785014c..2b89a38 100644
--- a/bin/varnishd/mgt/mgt_sandbox.c
+++ b/bin/varnishd/mgt/mgt_sandbox.c
@@ -54,7 +54,6 @@
 #include <grp.h>
 #include <stdio.h>
 #include <stdlib.h>
-#include <syslog.h>
 #include <string.h>
 #include <unistd.h>
 
@@ -236,7 +235,7 @@ mgt_sandbox_unix(enum sandbox_e who)
 	 * reenable them again.
 	 */
 	if (prctl(PR_SET_DUMPABLE, 1) != 0) {
-		REPORT0(LOG_INFO,
+		MGT_complain(C_INFO,
 		    "Could not set dumpable bit.  Core dumps turned off\n");
 	}
 #endif
@@ -277,10 +276,10 @@ mgt_sandbox_init(void)
 	subs = VSUB_run(sb, run_sandbox_test, NULL, "SANDBOX-test", 10);
 	VSB_delete(sb);
 	if (subs) {
-		REPORT0(LOG_INFO, "Warning: init of platform-specific sandbox "
-		    "failed - sandboxing disabled");
-		REPORT0(LOG_INFO, "Warning: Varnish might run with elevated "
-		    "privileges");
+		MGT_complain(C_SECURITY,
+		    "Platform-specific sandbox failed - sandboxing disabled");
+		MGT_complain(C_SECURITY,
+		    "Varnish runs with elevated privileges");
 		mgt_sandbox = mgt_sandbox_null;
 	}
 



From lkarsten at varnish-software.com  Wed Oct 21 15:13:39 2015
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Wed, 21 Oct 2015 17:13:39 +0200
Subject: [master] f358342 Add changelist from 4.1 branch.
Message-ID: <E1Zov59-0005Rq-U0@project.varnish-software.com>


commit f3583420d3bac8f9e6a2843694a67e7b66cd88f4
Author: Lasse Karstensen <lkarsten at varnish-software.com>
Date:   Wed Oct 21 17:13:14 2015 +0200

    Add changelist from 4.1 branch.

diff --git a/doc/changes.rst b/doc/changes.rst
index 745d1bc..6016ddc 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -1,9 +1,49 @@
+================================
+Varnish Cache 4.1.0 (2015-09-30)
+================================
+
+Changes since 4.1.0-beta1:
+
+- Documentation updates.
+- Stabilization fixes on testcase p00005.vtc.
+- Avoid compiler warning in zlib.
+- Bug 1792_: Avoid using fallocate() with -sfile on non-EXT4.
+
+.. _1792: https://www.varnish-cache.org/trac/ticket/1792
+
+
+======================================
+Varnish Cache 4.1.0-beta1 (2015-09-11)
+======================================
+
+Changes since 4.1.0-tp1:
+
+- Redhat packaging files are now separate from the normal tree.
+- Client workspace overflow should now result in a 500 response
+  instead of panic.
+- [varnishstat] -w option has been retired.
+- libvarnishapi release number is increased.
+- Body bytes sent on ESI subrequests with gzip are now counted correctly.
+- [vmod-std] Data type conversion functions now take additional fallback argument.
+
+Bugs fixed
+----------
+
+- 1777_ - Disable speculative Range handling on streaming transactions.
+- 1778_ - [varnishstat] Cast to integer to prevent negative values messing the statistics
+- 1781_ - Propagate gzip CRC upwards from nested ESI includes.
+- 1783_ - Align code with RFC7230 section 3.3.3 which allows POST without a body.
+
+.. _1777: https://www.varnish-cache.org/trac/ticket/1777
+.. _1778: https://www.varnish-cache.org/trac/ticket/1778
+.. _1781: https://www.varnish-cache.org/trac/ticket/1781
+.. _1783: https://www.varnish-cache.org/trac/ticket/1783
+
+
 ====================================
-Varnish Cache 4.1.0-dev (unreleased)
+Varnish Cache 4.1.0-tp1 (2015-07-08)
 ====================================
 
-This is a placeholder entry for the upcoming 4.1.0 prereleases.
-
 Changes between 4.0 and 4.1 are numerous. Please read the upgrade
 section in the documentation for a general overview.
 



From lkarsten at varnish-software.com  Wed Oct 21 15:15:44 2015
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Wed, 21 Oct 2015 17:15:44 +0200
Subject: [master] 1b1635a Remove 4.1 unreleased notice.
Message-ID: <E1Zov7A-00084M-Sr@project.varnish-software.com>


commit 1b1635a7db39483e5a72231921caa4917ce49d65
Author: Lasse Karstensen <lkarsten at varnish-software.com>
Date:   Wed Oct 21 17:15:37 2015 +0200

    Remove 4.1 unreleased notice.

diff --git a/doc/sphinx/whats-new/upgrading.rst b/doc/sphinx/whats-new/upgrading.rst
index e23c96d..49c4f08 100644
--- a/doc/sphinx/whats-new/upgrading.rst
+++ b/doc/sphinx/whats-new/upgrading.rst
@@ -1,8 +1,8 @@
 .. _whatsnew_upgrade41:
 
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-Upgrading to Varnish 4.1 (unreleased)
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%
+Upgrading to Varnish 4.1
+%%%%%%%%%%%%%%%%%%%%%%%%
 
 Changes to VCL
 ==============



From lkarsten at varnish-software.com  Wed Oct 21 15:16:52 2015
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Wed, 21 Oct 2015 17:16:52 +0200
Subject: [4.1] b4225ee Remove 4.1 unreleased notice.
Message-ID: <E1Zov8G-00024e-9J@project.varnish-software.com>


commit b4225ee0bff7580c7143674c55d0b9e696c7adce
Author: Lasse Karstensen <lkarsten at varnish-software.com>
Date:   Wed Oct 21 17:15:37 2015 +0200

    Remove 4.1 unreleased notice.

diff --git a/doc/sphinx/whats-new/upgrading.rst b/doc/sphinx/whats-new/upgrading.rst
index e23c96d..49c4f08 100644
--- a/doc/sphinx/whats-new/upgrading.rst
+++ b/doc/sphinx/whats-new/upgrading.rst
@@ -1,8 +1,8 @@
 .. _whatsnew_upgrade41:
 
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-Upgrading to Varnish 4.1 (unreleased)
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%
+Upgrading to Varnish 4.1
+%%%%%%%%%%%%%%%%%%%%%%%%
 
 Changes to VCL
 ==============



From fgsch at lodoss.net  Thu Oct 22 07:38:14 2015
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Thu, 22 Oct 2015 09:38:14 +0200
Subject: [master] e7c4e0b Revert "backend threads fail tovcl_backend_error"
Message-ID: <E1ZpARy-0007MS-6M@project.varnish-software.com>


commit e7c4e0b6e1d8a23c1757e556b951d4004e1e7bb1
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Thu Oct 22 08:35:58 2015 +0100

    Revert "backend threads fail tovcl_backend_error"
    
    This reverts commit ed6dca6b9ced699591470ddae65d3fab862d1b73.
    abandon in vcl_backend_* will take us to vcl_synth{}.

diff --git a/doc/sphinx/users-guide/vcl-built-in-subs.rst b/doc/sphinx/users-guide/vcl-built-in-subs.rst
index 12d3377..a1eba28 100644
--- a/doc/sphinx/users-guide/vcl-built-in-subs.rst
+++ b/doc/sphinx/users-guide/vcl-built-in-subs.rst
@@ -268,8 +268,8 @@ The `vcl_backend_fetch` subroutine may terminate with calling
 
   ``abandon``
     Abandon the backend request. Unless the backend request was a
-    background fetch, control is passed to :ref:`vcl_backend_error`
-    with ``resp.status`` preset to 503.
+    background fetch, control is passed to :ref:`vcl_synth` on the
+    client side with ``resp.status`` preset to 503.
 
 .. _vcl_backend_response:
 



From phk at FreeBSD.org  Thu Oct 22 15:40:38 2015
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 22 Oct 2015 17:40:38 +0200
Subject: [master] 30b5f02 Split ban_lurker into separate source file.
Message-ID: <E1ZpHyo-0002cj-IJ@project.varnish-software.com>


commit 30b5f02c527cb227737d0a26ed87af8237ba96a9
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Oct 22 05:44:51 2015 +0000

    Split ban_lurker into separate source file.
    
    No functional changes.

diff --git a/bin/varnishd/Makefile.am b/bin/varnishd/Makefile.am
index e4d2ab5..86906a2 100644
--- a/bin/varnishd/Makefile.am
+++ b/bin/varnishd/Makefile.am
@@ -15,6 +15,7 @@ varnishd_SOURCES = \
 	cache/cache_backend_probe.c \
 	cache/cache_backend_tcp.c \
 	cache/cache_ban.c \
+	cache/cache_ban_lurker.c \
 	cache/cache_busyobj.c \
 	cache/cache_cli.c \
 	cache/cache_deliver_proc.c \
@@ -99,6 +100,7 @@ varnishd_SOURCES = \
 
 noinst_HEADERS = \
 	builtin_vcl.h \
+	cache/cache_ban.h \
 	cache/cache_esi.h \
 	cache/cache_pool.h \
 	cache/cache_priv.h \
diff --git a/bin/varnishd/cache/cache_ban.c b/bin/varnishd/cache/cache_ban.c
index f63c8b4..016c77c 100644
--- a/bin/varnishd/cache/cache_ban.c
+++ b/bin/varnishd/cache/cache_ban.c
@@ -26,38 +26,6 @@
  * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
  * SUCH DAMAGE.
  *
- * Ban processing
- *
- * A ban consists of a number of conditions (or tests), all of which must be
- * satisfied.  Here are some potential bans we could support:
- *
- *	req.url == "/foo"
- *	req.url ~ ".iso" && obj.size > 10MB
- *	req.http.host ~ "web1.com" && obj.http.set-cookie ~ "USER=29293"
- *
- * We make the "&&" mandatory from the start, leaving the syntax space
- * for latter handling of "||" as well.
- *
- * Bans are compiled into bytestrings as follows:
- *	8 bytes	- double: timestamp		XXX: Byteorder ?
- *	4 bytes - be32: length
- *	1 byte - flags: 0x01: BAN_F_{REQ|OBJ|COMPLETED}
- *	N tests
- * A test have this form:
- *	1 byte - arg (see ban_vars.h col 3 "BANS_ARG_XXX")
- *	(n bytes) - http header name, canonical encoding
- *	lump - comparison arg
- *	1 byte - operation (BANS_OPER_)
- *	(lump) - compiled regexp
- * A lump is:
- *	4 bytes - be32: length
- *	n bytes - content
- *
- * In a perfect world, we should vector through VRE to get to PCRE,
- * but since we rely on PCRE's ability to encode the regexp into a
- * byte string, that would be a little bit artificial, so this is
- * the exception that confirms the rule.
- *
  */
 
 #include "config.h"
@@ -65,6 +33,7 @@
 #include <pcre.h>
 
 #include "cache.h"
+#include "cache_ban.h"
 
 #include "hash/hash_slinger.h"
 #include "vcli.h"
@@ -73,66 +42,13 @@
 #include "vmb.h"
 #include "vtim.h"
 
-/*--------------------------------------------------------------------
- * BAN string defines & magic markers
- */
-
-#define BANS_TIMESTAMP		0
-#define BANS_LENGTH		8
-#define BANS_FLAGS		12
-#define BANS_HEAD_LEN		16
-
-#define BANS_FLAG_REQ		(1<<0)
-#define BANS_FLAG_OBJ		(1<<1)
-#define BANS_FLAG_COMPLETED	(1<<2)
-#define BANS_FLAG_HTTP		(1<<3)
-#define BANS_FLAG_ERROR		(1<<4)
-
-#define BANS_OPER_EQ		0x10
-#define BANS_OPER_NEQ		0x11
-#define BANS_OPER_MATCH		0x12
-#define BANS_OPER_NMATCH	0x13
-
-#define BANS_ARG_URL		0x18
-#define BANS_ARG_REQHTTP	0x19
-#define BANS_ARG_OBJHTTP	0x1a
-#define BANS_ARG_OBJSTATUS	0x1b
-
-/*--------------------------------------------------------------------*/
-
-struct ban {
-	unsigned		magic;
-#define BAN_MAGIC		0x700b08ea
-	VTAILQ_ENTRY(ban)	list;
-	VTAILQ_ENTRY(ban)	l_list;
-	int			refcount;
-	unsigned		flags;		/* BANS_FLAG_* */
-
-	VTAILQ_HEAD(,objcore)	objcore;
-	struct vsb		*vsb;
-	uint8_t			*spec;
-};
-
-VTAILQ_HEAD(banhead_s,ban);
+struct lock ban_mtx;
+int ban_shutdown = 0;
+struct banhead_s ban_head = VTAILQ_HEAD_INITIALIZER(ban_head);
+struct ban * volatile ban_start;
 
-struct ban_test {
-	uint8_t			arg1;
-	const char		*arg1_spec;
-	uint8_t			oper;
-	const char		*arg2;
-	const void		*arg2_spec;
-};
-
-static void ban_info(enum baninfo event, const uint8_t *ban, unsigned len);
-static struct banhead_s ban_head = VTAILQ_HEAD_INITIALIZER(ban_head);
-static struct lock ban_mtx;
 static struct ban *ban_magic;
 static pthread_t ban_thread;
-static struct ban * volatile ban_start;
-static struct objcore oc_marker = { .magic = OBJCORE_MAGIC, };
-static bgthread_t ban_lurker;
-static unsigned ban_batch;
-static int ban_shutdown = 0;
 
 /*--------------------------------------------------------------------
  * Variables we can purge on
@@ -231,7 +147,7 @@ BAN_TailDeref(struct ban **bb)
  * Extract time and length from ban-spec
  */
 
-static double
+double
 ban_time(const uint8_t *banspec)
 {
 	double t;
@@ -241,7 +157,7 @@ ban_time(const uint8_t *banspec)
 	return (t);
 }
 
-static unsigned
+unsigned
 ban_len(const uint8_t *banspec)
 {
 	unsigned u;
@@ -264,7 +180,7 @@ ban_equal(const uint8_t *bs1, const uint8_t *bs2)
 	return (!memcmp(bs1 + BANS_LENGTH, bs2 + BANS_LENGTH, u - BANS_LENGTH));
 }
 
-static void
+void
 ban_mark_completed(struct ban *b)
 {
 	unsigned ln;
@@ -677,7 +593,7 @@ ban_export(void)
 	VSC_C_main->bans_persisted_fragmentation = 0;
 }
 
-static void
+void
 ban_info(enum baninfo event, const uint8_t *ban, unsigned len)
 {
 	if (STV_BanInfo(event, ban, len)) {
@@ -799,36 +715,10 @@ BAN_Time(const struct ban *b)
 }
 
 /*--------------------------------------------------------------------
- * All silos have read their bans, ready for action
- */
-
-void
-BAN_Compile(void)
-{
-
-	ASSERT_CLI();
-	AZ(ban_shutdown);
-
-	Lck_Lock(&ban_mtx);
-
-	/* Do late reporting of ban_magic */
-	AZ(STV_BanInfo(BI_NEW, ban_magic->spec, ban_len(ban_magic->spec)));
-
-	/* All bans have been read from all persistent stevedores. Export
-	   the compiled list */
-	ban_export();
-
-	Lck_Unlock(&ban_mtx);
-
-	ban_start = VTAILQ_FIRST(&ban_head);
-	WRK_BgThread(&ban_thread, "ban-lurker", ban_lurker, NULL);
-}
-
-/*--------------------------------------------------------------------
  * Evaluate ban-spec
  */
 
-static int
+int
 ban_evaluate(struct worker *wrk, const uint8_t *bs, struct objcore *oc,
     const struct http *reqhttp, unsigned *tests)
 {
@@ -971,227 +861,6 @@ BAN_CheckObject(struct worker *wrk, struct objcore *oc, struct req *req)
 	}
 }
 
-static void
-ban_cleantail(void)
-{
-	struct ban *b;
-
-	do {
-		Lck_Lock(&ban_mtx);
-		b = VTAILQ_LAST(&ban_head, banhead_s);
-		if (b != VTAILQ_FIRST(&ban_head) && b->refcount == 0) {
-			if (b->flags & BANS_FLAG_COMPLETED)
-				VSC_C_main->bans_completed--;
-			if (b->flags & BANS_FLAG_OBJ)
-				VSC_C_main->bans_obj--;
-			if (b->flags & BANS_FLAG_REQ)
-				VSC_C_main->bans_req--;
-			VSC_C_main->bans--;
-			VSC_C_main->bans_deleted++;
-			VTAILQ_REMOVE(&ban_head, b, list);
-			VSC_C_main->bans_persisted_fragmentation +=
-			    ban_len(b->spec);
-			ban_info(BI_DROP, b->spec, ban_len(b->spec));
-		} else {
-			b = NULL;
-		}
-		Lck_Unlock(&ban_mtx);
-		if (b != NULL)
-			BAN_Free(b);
-	} while (b != NULL);
-}
-
-/*--------------------------------------------------------------------
- * Our task here is somewhat tricky:  The canonical locking order is
- * objhead->mtx first, then ban_mtx, because that is the order which
- * makes most sense in HSH_Lookup(), but we come the other way.
- * We optimistically try to get them the other way, and get out of
- * the way if that fails, and retry again later.
- */
-
-static struct objcore *
-ban_lurker_getfirst(struct vsl_log *vsl, struct ban *bt)
-{
-	struct objhead *oh;
-	struct objcore *oc;
-
-	while (1) {
-		Lck_Lock(&ban_mtx);
-		oc = VTAILQ_FIRST(&bt->objcore);
-		CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-		if (oc == &oc_marker) {
-			VTAILQ_REMOVE(&bt->objcore, oc, ban_list);
-			Lck_Unlock(&ban_mtx);
-			return (NULL);
-		}
-		oh = oc->objhead;
-		CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC);
-		if (!Lck_Trylock(&oh->mtx)) {
-			if (oc->refcnt == 0) {
-				Lck_Unlock(&oh->mtx);
-			} else {
-				/*
-				 * We got the lock, and the oc is not being
-				 * dismantled under our feet, run with it...
-				 */
-				AZ(oc->flags & OC_F_BUSY);
-				oc->refcnt += 1;
-				VTAILQ_REMOVE(&bt->objcore, oc, ban_list);
-				VTAILQ_INSERT_TAIL(&bt->objcore, oc, ban_list);
-				Lck_Unlock(&oh->mtx);
-				Lck_Unlock(&ban_mtx);
-				break;
-			}
-		}
-
-		/* Try again, later */
-		Lck_Unlock(&ban_mtx);
-		VSC_C_main->bans_lurker_contention++;
-		VSL_Flush(vsl, 0);
-		VTIM_sleep(cache_param->ban_lurker_sleep);
-	}
-	return (oc);
-}
-
-static void
-ban_lurker_test_ban(struct worker *wrk, struct vsl_log *vsl, struct ban *bt,
-    struct banhead_s *obans)
-{
-	struct ban *bl, *bln;
-	struct objcore *oc;
-	unsigned tests;
-	int i;
-
-	/*
-	 * First see if there is anything to do, and if so, insert marker
-	 */
-	Lck_Lock(&ban_mtx);
-	oc = VTAILQ_FIRST(&bt->objcore);
-	if (oc != NULL)
-		VTAILQ_INSERT_TAIL(&bt->objcore, &oc_marker, ban_list);
-	Lck_Unlock(&ban_mtx);
-	if (oc == NULL)
-		return;
-
-	while (1) {
-		if (++ban_batch > cache_param->ban_lurker_batch) {
-			VTIM_sleep(cache_param->ban_lurker_sleep);
-			ban_batch = 0;
-		}
-		oc = ban_lurker_getfirst(vsl, bt);
-		if (oc == NULL)
-			return;
-		i = 0;
-		VTAILQ_FOREACH_REVERSE_SAFE(bl, obans, banhead_s, l_list, bln) {
-			if (bl->flags & BANS_FLAG_COMPLETED) {
-				/* Ban was overtaken by new (dup) ban */
-				VTAILQ_REMOVE(obans, bl, l_list);
-				continue;
-			}
-			tests = 0;
-			i = ban_evaluate(wrk, bl->spec, oc, NULL, &tests);
-			VSC_C_main->bans_lurker_tested++;
-			VSC_C_main->bans_lurker_tests_tested += tests;
-			if (i)
-				break;
-		}
-		if (i) {
-			VSLb(vsl, SLT_ExpBan, "%u banned by lurker",
-			    ObjGetXID(wrk, oc));
-
-			EXP_Rearm(oc, oc->exp.t_origin, 0, 0, 0);
-					// XXX ^ fake now
-			VSC_C_main->bans_lurker_obj_killed++;
-		}
-		(void)HSH_DerefObjCore(wrk, &oc);
-	}
-}
-
-/*--------------------------------------------------------------------
- * Ban lurker thread
- */
-
-static int
-ban_lurker_work(struct worker *wrk, struct vsl_log *vsl)
-{
-	struct ban *b, *bt;
-	struct banhead_s obans;
-	double d;
-	int i;
-
-	/* Make a list of the bans we can do something about */
-	VTAILQ_INIT(&obans);
-	Lck_Lock(&ban_mtx);
-	b = ban_start;
-	Lck_Unlock(&ban_mtx);
-	i = 0;
-	d = VTIM_real() - cache_param->ban_lurker_age;
-	while (b != NULL) {
-		if (b->flags & BANS_FLAG_COMPLETED) {
-			;
-		} else if (b->flags & BANS_FLAG_REQ) {
-			;
-		} else if (b == VTAILQ_LAST(&ban_head, banhead_s)) {
-			;
-		} else if (ban_time(b->spec) > d) {
-			;
-		} else {
-			VTAILQ_INSERT_TAIL(&obans, b, l_list);
-			i++;
-		}
-		b = VTAILQ_NEXT(b, list);
-	}
-	if (DO_DEBUG(DBG_LURKER))
-		VSLb(vsl, SLT_Debug, "lurker: %d actionable bans", i);
-	if (i == 0)
-		return (0);
-
-	/* Go though all the bans to test the objects */
-	VTAILQ_FOREACH_REVERSE(bt, &ban_head, banhead_s, list) {
-		if (bt == VTAILQ_LAST(&obans, banhead_s)) {
-			if (DO_DEBUG(DBG_LURKER))
-				VSLb(vsl, SLT_Debug,
-				    "Lurk bt completed %p", bt);
-			Lck_Lock(&ban_mtx);
-			/* We can be raced by a new ban */
-			if (!(bt->flags & BANS_FLAG_COMPLETED))
-				ban_mark_completed(bt);
-			Lck_Unlock(&ban_mtx);
-			VTAILQ_REMOVE(&obans, bt, l_list);
-			if (VTAILQ_EMPTY(&obans))
-				break;
-		}
-		if (DO_DEBUG(DBG_LURKER))
-			VSLb(vsl, SLT_Debug, "Lurk bt %p", bt);
-		ban_lurker_test_ban(wrk, vsl, bt, &obans);
-		if (VTAILQ_EMPTY(&obans))
-			break;
-	}
-	return (1);
-}
-
-static void * __match_proto__(bgthread_t)
-ban_lurker(struct worker *wrk, void *priv)
-{
-	struct vsl_log vsl;
-	volatile double d;
-
-	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
-	AZ(priv);
-
-	VSL_Setup(&vsl, NULL, 0);
-
-	while (!ban_shutdown) {
-		d = cache_param->ban_lurker_sleep;
-		if (d <= 0.0 || !ban_lurker_work(wrk, &vsl))
-			d = 0.609;	// Random, non-magic
-		ban_cleantail();
-		VTIM_sleep(d);
-	}
-	pthread_exit(0);
-	NEEDLESS_RETURN(NULL);
-}
-
 /*--------------------------------------------------------------------
  * CLI functions to add bans
  */
@@ -1326,6 +995,34 @@ static struct cli_proto ban_cmds[] = {
 	{ NULL }
 };
 
+/*--------------------------------------------------------------------
+ */
+
+void
+BAN_Compile(void)
+{
+
+	/* All bans have been read from all persistent stevedores. Export
+	 * the compiled list
+	 */
+
+	ASSERT_CLI();
+	AZ(ban_shutdown);
+
+	Lck_Lock(&ban_mtx);
+
+	/* Do late reporting of ban_magic */
+	AZ(STV_BanInfo(BI_NEW, ban_magic->spec, ban_len(ban_magic->spec)));
+
+	ban_export();
+
+	Lck_Unlock(&ban_mtx);
+
+	ban_start = VTAILQ_FIRST(&ban_head);
+	WRK_BgThread(&ban_thread, "ban-lurker", ban_lurker, NULL);
+}
+
+
 void
 BAN_Init(void)
 {
diff --git a/bin/varnishd/cache/cache_ban.h b/bin/varnishd/cache/cache_ban.h
new file mode 100644
index 0000000..fae2b80
--- /dev/null
+++ b/bin/varnishd/cache/cache_ban.h
@@ -0,0 +1,124 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2015 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * Ban processing
+ *
+ * A ban consists of a number of conditions (or tests), all of which must be
+ * satisfied.  Here are some potential bans we could support:
+ *
+ *	req.url == "/foo"
+ *	req.url ~ ".iso" && obj.size > 10MB
+ *	req.http.host ~ "web1.com" && obj.http.set-cookie ~ "USER=29293"
+ *
+ * We make the "&&" mandatory from the start, leaving the syntax space
+ * for latter handling of "||" as well.
+ *
+ * Bans are compiled into bytestrings as follows:
+ *	8 bytes	- double: timestamp		XXX: Byteorder ?
+ *	4 bytes - be32: length
+ *	1 byte - flags: 0x01: BAN_F_{REQ|OBJ|COMPLETED}
+ *	N tests
+ * A test have this form:
+ *	1 byte - arg (see ban_vars.h col 3 "BANS_ARG_XXX")
+ *	(n bytes) - http header name, canonical encoding
+ *	lump - comparison arg
+ *	1 byte - operation (BANS_OPER_)
+ *	(lump) - compiled regexp
+ * A lump is:
+ *	4 bytes - be32: length
+ *	n bytes - content
+ *
+ * In a perfect world, we should vector through VRE to get to PCRE,
+ * but since we rely on PCRE's ability to encode the regexp into a
+ * byte string, that would be a little bit artificial, so this is
+ * the exception that confirms the rule.
+ *
+ */
+
+/*--------------------------------------------------------------------
+ * BAN string defines & magic markers
+ */
+
+#define BANS_TIMESTAMP		0
+#define BANS_LENGTH		8
+#define BANS_FLAGS		12
+#define BANS_HEAD_LEN		16
+
+#define BANS_FLAG_REQ		(1<<0)
+#define BANS_FLAG_OBJ		(1<<1)
+#define BANS_FLAG_COMPLETED	(1<<2)
+#define BANS_FLAG_HTTP		(1<<3)
+#define BANS_FLAG_ERROR		(1<<4)
+
+#define BANS_OPER_EQ		0x10
+#define BANS_OPER_NEQ		0x11
+#define BANS_OPER_MATCH		0x12
+#define BANS_OPER_NMATCH	0x13
+
+#define BANS_ARG_URL		0x18
+#define BANS_ARG_REQHTTP	0x19
+#define BANS_ARG_OBJHTTP	0x1a
+#define BANS_ARG_OBJSTATUS	0x1b
+
+/*--------------------------------------------------------------------*/
+
+struct ban {
+	unsigned		magic;
+#define BAN_MAGIC		0x700b08ea
+	VTAILQ_ENTRY(ban)	list;
+	VTAILQ_ENTRY(ban)	l_list;
+	int			refcount;
+	unsigned		flags;		/* BANS_FLAG_* */
+
+	VTAILQ_HEAD(,objcore)	objcore;
+	struct vsb		*vsb;
+	uint8_t			*spec;
+};
+
+VTAILQ_HEAD(banhead_s,ban);
+
+struct ban_test {
+	uint8_t			arg1;
+	const char		*arg1_spec;
+	uint8_t			oper;
+	const char		*arg2;
+	const void		*arg2_spec;
+};
+
+bgthread_t ban_lurker;
+extern struct lock ban_mtx;
+extern int ban_shutdown;
+extern struct banhead_s ban_head;
+extern struct ban * volatile ban_start;
+
+void ban_mark_completed(struct ban *b);
+unsigned ban_len(const uint8_t *banspec);
+void ban_info(enum baninfo event, const uint8_t *ban, unsigned len);
+int ban_evaluate(struct worker *wrk, const uint8_t *bs, struct objcore *oc,
+    const struct http *reqhttp, unsigned *tests);
+double ban_time(const uint8_t *banspec);
diff --git a/bin/varnishd/cache/cache_ban_lurker.c b/bin/varnishd/cache/cache_ban_lurker.c
new file mode 100644
index 0000000..cfcdd26
--- /dev/null
+++ b/bin/varnishd/cache/cache_ban_lurker.c
@@ -0,0 +1,261 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2015 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ */
+
+#include "config.h"
+
+#include "cache.h"
+#include "cache_ban.h"
+
+#include "hash/hash_slinger.h"
+#include "vtim.h"
+
+static struct objcore oc_marker = { .magic = OBJCORE_MAGIC, };
+static unsigned ban_batch;
+
+static void
+ban_cleantail(void)
+{
+	struct ban *b;
+
+	do {
+		Lck_Lock(&ban_mtx);
+		b = VTAILQ_LAST(&ban_head, banhead_s);
+		if (b != VTAILQ_FIRST(&ban_head) && b->refcount == 0) {
+			if (b->flags & BANS_FLAG_COMPLETED)
+				VSC_C_main->bans_completed--;
+			if (b->flags & BANS_FLAG_OBJ)
+				VSC_C_main->bans_obj--;
+			if (b->flags & BANS_FLAG_REQ)
+				VSC_C_main->bans_req--;
+			VSC_C_main->bans--;
+			VSC_C_main->bans_deleted++;
+			VTAILQ_REMOVE(&ban_head, b, list);
+			VSC_C_main->bans_persisted_fragmentation +=
+			    ban_len(b->spec);
+			ban_info(BI_DROP, b->spec, ban_len(b->spec));
+		} else {
+			b = NULL;
+		}
+		Lck_Unlock(&ban_mtx);
+		if (b != NULL)
+			BAN_Free(b);
+	} while (b != NULL);
+}
+
+/*--------------------------------------------------------------------
+ * Our task here is somewhat tricky:  The canonical locking order is
+ * objhead->mtx first, then ban_mtx, because that is the order which
+ * makes most sense in HSH_Lookup(), but we come the other way.
+ * We optimistically try to get them the other way, and get out of
+ * the way if that fails, and retry again later.
+ */
+
+static struct objcore *
+ban_lurker_getfirst(struct vsl_log *vsl, struct ban *bt)
+{
+	struct objhead *oh;
+	struct objcore *oc;
+
+	while (1) {
+		Lck_Lock(&ban_mtx);
+		oc = VTAILQ_FIRST(&bt->objcore);
+		CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+		if (oc == &oc_marker) {
+			VTAILQ_REMOVE(&bt->objcore, oc, ban_list);
+			Lck_Unlock(&ban_mtx);
+			return (NULL);
+		}
+		oh = oc->objhead;
+		CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC);
+		if (!Lck_Trylock(&oh->mtx)) {
+			if (oc->refcnt == 0) {
+				Lck_Unlock(&oh->mtx);
+			} else {
+				/*
+				 * We got the lock, and the oc is not being
+				 * dismantled under our feet, run with it...
+				 */
+				AZ(oc->flags & OC_F_BUSY);
+				oc->refcnt += 1;
+				VTAILQ_REMOVE(&bt->objcore, oc, ban_list);
+				VTAILQ_INSERT_TAIL(&bt->objcore, oc, ban_list);
+				Lck_Unlock(&oh->mtx);
+				Lck_Unlock(&ban_mtx);
+				break;
+			}
+		}
+
+		/* Try again, later */
+		Lck_Unlock(&ban_mtx);
+		VSC_C_main->bans_lurker_contention++;
+		VSL_Flush(vsl, 0);
+		VTIM_sleep(cache_param->ban_lurker_sleep);
+	}
+	return (oc);
+}
+
+static void
+ban_lurker_test_ban(struct worker *wrk, struct vsl_log *vsl, struct ban *bt,
+    struct banhead_s *obans)
+{
+	struct ban *bl, *bln;
+	struct objcore *oc;
+	unsigned tests;
+	int i;
+
+	/*
+	 * First see if there is anything to do, and if so, insert marker
+	 */
+	Lck_Lock(&ban_mtx);
+	oc = VTAILQ_FIRST(&bt->objcore);
+	if (oc != NULL)
+		VTAILQ_INSERT_TAIL(&bt->objcore, &oc_marker, ban_list);
+	Lck_Unlock(&ban_mtx);
+	if (oc == NULL)
+		return;
+
+	while (1) {
+		if (++ban_batch > cache_param->ban_lurker_batch) {
+			VTIM_sleep(cache_param->ban_lurker_sleep);
+			ban_batch = 0;
+		}
+		oc = ban_lurker_getfirst(vsl, bt);
+		if (oc == NULL)
+			return;
+		i = 0;
+		VTAILQ_FOREACH_REVERSE_SAFE(bl, obans, banhead_s, l_list, bln) {
+			if (bl->flags & BANS_FLAG_COMPLETED) {
+				/* Ban was overtaken by new (dup) ban */
+				VTAILQ_REMOVE(obans, bl, l_list);
+				continue;
+			}
+			tests = 0;
+			i = ban_evaluate(wrk, bl->spec, oc, NULL, &tests);
+			VSC_C_main->bans_lurker_tested++;
+			VSC_C_main->bans_lurker_tests_tested += tests;
+			if (i)
+				break;
+		}
+		if (i) {
+			VSLb(vsl, SLT_ExpBan, "%u banned by lurker",
+			    ObjGetXID(wrk, oc));
+
+			EXP_Rearm(oc, oc->exp.t_origin, 0, 0, 0);
+					// XXX ^ fake now
+			VSC_C_main->bans_lurker_obj_killed++;
+		}
+		(void)HSH_DerefObjCore(wrk, &oc);
+	}
+}
+
+/*--------------------------------------------------------------------
+ * Ban lurker thread
+ */
+
+static int
+ban_lurker_work(struct worker *wrk, struct vsl_log *vsl)
+{
+	struct ban *b, *bt;
+	struct banhead_s obans;
+	double d;
+	int i;
+
+	/* Make a list of the bans we can do something about */
+	VTAILQ_INIT(&obans);
+	Lck_Lock(&ban_mtx);
+	b = ban_start;
+	Lck_Unlock(&ban_mtx);
+	i = 0;
+	d = VTIM_real() - cache_param->ban_lurker_age;
+	while (b != NULL) {
+		if (b->flags & BANS_FLAG_COMPLETED) {
+			;
+		} else if (b->flags & BANS_FLAG_REQ) {
+			;
+		} else if (b == VTAILQ_LAST(&ban_head, banhead_s)) {
+			;
+		} else if (ban_time(b->spec) > d) {
+			;
+		} else {
+			VTAILQ_INSERT_TAIL(&obans, b, l_list);
+			i++;
+		}
+		b = VTAILQ_NEXT(b, list);
+	}
+	if (DO_DEBUG(DBG_LURKER))
+		VSLb(vsl, SLT_Debug, "lurker: %d actionable bans", i);
+	if (i == 0)
+		return (0);
+
+	/* Go though all the bans to test the objects */
+	VTAILQ_FOREACH_REVERSE(bt, &ban_head, banhead_s, list) {
+		if (bt == VTAILQ_LAST(&obans, banhead_s)) {
+			if (DO_DEBUG(DBG_LURKER))
+				VSLb(vsl, SLT_Debug,
+				    "Lurk bt completed %p", bt);
+			Lck_Lock(&ban_mtx);
+			/* We can be raced by a new ban */
+			if (!(bt->flags & BANS_FLAG_COMPLETED))
+				ban_mark_completed(bt);
+			Lck_Unlock(&ban_mtx);
+			VTAILQ_REMOVE(&obans, bt, l_list);
+			if (VTAILQ_EMPTY(&obans))
+				break;
+		}
+		if (DO_DEBUG(DBG_LURKER))
+			VSLb(vsl, SLT_Debug, "Lurk bt %p", bt);
+		ban_lurker_test_ban(wrk, vsl, bt, &obans);
+		if (VTAILQ_EMPTY(&obans))
+			break;
+	}
+	return (1);
+}
+
+void * __match_proto__(bgthread_t)
+ban_lurker(struct worker *wrk, void *priv)
+{
+	struct vsl_log vsl;
+	volatile double d;
+
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
+	AZ(priv);
+
+	VSL_Setup(&vsl, NULL, 0);
+
+	while (!ban_shutdown) {
+		d = cache_param->ban_lurker_sleep;
+		if (d <= 0.0 || !ban_lurker_work(wrk, &vsl))
+			d = 0.609;	// Random, non-magic
+		ban_cleantail();
+		VTIM_sleep(d);
+	}
+	pthread_exit(0);
+	NEEDLESS_RETURN(NULL);
+}



From phk at FreeBSD.org  Thu Oct 22 15:40:38 2015
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 22 Oct 2015 17:40:38 +0200
Subject: [master] 54a38c1 struct ban_test is only used in cache_ban.c
Message-ID: <E1ZpHyo-0002cq-Jn@project.varnish-software.com>


commit 54a38c1d44b0ef1e4b0746f90bf771924348e668
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Oct 22 05:47:28 2015 +0000

    struct ban_test is only used in cache_ban.c

diff --git a/bin/varnishd/cache/cache_ban.c b/bin/varnishd/cache/cache_ban.c
index 016c77c..8d6337c 100644
--- a/bin/varnishd/cache/cache_ban.c
+++ b/bin/varnishd/cache/cache_ban.c
@@ -50,6 +50,14 @@ struct ban * volatile ban_start;
 static struct ban *ban_magic;
 static pthread_t ban_thread;
 
+struct ban_test {
+	uint8_t			arg1;
+	const char		*arg1_spec;
+	uint8_t			oper;
+	const char		*arg2;
+	const void		*arg2_spec;
+};
+
 /*--------------------------------------------------------------------
  * Variables we can purge on
  */
diff --git a/bin/varnishd/cache/cache_ban.h b/bin/varnishd/cache/cache_ban.h
index fae2b80..ce0ae9e 100644
--- a/bin/varnishd/cache/cache_ban.h
+++ b/bin/varnishd/cache/cache_ban.h
@@ -102,14 +102,6 @@ struct ban {
 
 VTAILQ_HEAD(banhead_s,ban);
 
-struct ban_test {
-	uint8_t			arg1;
-	const char		*arg1_spec;
-	uint8_t			oper;
-	const char		*arg2;
-	const void		*arg2_spec;
-};
-
 bgthread_t ban_lurker;
 extern struct lock ban_mtx;
 extern int ban_shutdown;



From phk at FreeBSD.org  Thu Oct 22 15:40:38 2015
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 22 Oct 2015 17:40:38 +0200
Subject: [master] 756fbe2 BAN_Compile() is private stuff.
Message-ID: <E1ZpHyo-0002cz-Kw@project.varnish-software.com>


commit 756fbe2301a5cee50a2344cb68fcb0bbf6c43ce2
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Oct 22 07:40:28 2015 +0000

    BAN_Compile() is private stuff.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 742f89e..65ed081 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -665,7 +665,6 @@ void BAN_DestroyObj(struct objcore *oc);
 int BAN_CheckObject(struct worker *, struct objcore *, struct req *);
 void BAN_Reload(const uint8_t *ban, unsigned len);
 struct ban *BAN_TailRef(void);
-void BAN_Compile(void);
 struct ban *BAN_RefBan(struct objcore *oc, double t0, const struct ban *tail);
 void BAN_TailDeref(struct ban **ban);
 double BAN_Time(const struct ban *ban);
diff --git a/bin/varnishd/cache/cache_priv.h b/bin/varnishd/cache/cache_priv.h
index 7d69cfa..2423afd 100644
--- a/bin/varnishd/cache/cache_priv.h
+++ b/bin/varnishd/cache/cache_priv.h
@@ -59,6 +59,7 @@ void VBP_Init(void);
 
 /* cache_ban.c */
 void BAN_Init(void);
+void BAN_Compile(void);
 void BAN_Shutdown(void);
 
 /* cache_busyobj.c */



From phk at FreeBSD.org  Thu Oct 22 15:40:38 2015
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 22 Oct 2015 17:40:38 +0200
Subject: [master] aaff00e Convert the (persistent) stevedores tail reference
 into a proper "hold" on the ban lurker not starting.
Message-ID: <E1ZpHyo-0002d8-MP@project.varnish-software.com>


commit aaff00e84c5885a10c835c60ddceb4836bf12f66
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Oct 22 08:07:14 2015 +0000

    Convert the (persistent) stevedores tail reference into a proper "hold"
    on the ban lurker not starting.
    
    Move cache_hash's access to bans into cache_priv.h

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 65ed081..11c8be1 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -655,18 +655,19 @@ struct sess {
 typedef enum htc_status_e htc_complete_f(struct http_conn *);
 
 /* cache_ban.c */
+
+/* for constructing bans */
 struct ban *BAN_New(void);
 int BAN_AddTest(struct ban *, const char *, const char *, const char *);
 void BAN_Free(struct ban *b);
 char *BAN_Insert(struct ban *b);
 void BAN_Free_Errormsg(char *);
-void BAN_NewObjCore(struct objcore *oc);
-void BAN_DestroyObj(struct objcore *oc);
-int BAN_CheckObject(struct worker *, struct objcore *, struct req *);
+
+/* for stevedoes resurrecting bans */
+void BAN_Hold(void);
+void BAN_Release(void);
 void BAN_Reload(const uint8_t *ban, unsigned len);
-struct ban *BAN_TailRef(void);
-struct ban *BAN_RefBan(struct objcore *oc, double t0, const struct ban *tail);
-void BAN_TailDeref(struct ban **ban);
+struct ban *BAN_RefBan(struct objcore *oc, double t0);
 double BAN_Time(const struct ban *ban);
 
 /* cache_busyobj.c */
diff --git a/bin/varnishd/cache/cache_ban.c b/bin/varnishd/cache/cache_ban.c
index 8d6337c..e5e03e6 100644
--- a/bin/varnishd/cache/cache_ban.c
+++ b/bin/varnishd/cache/cache_ban.c
@@ -43,12 +43,13 @@
 #include "vtim.h"
 
 struct lock ban_mtx;
-int ban_shutdown = 0;
+int ban_shutdown;
 struct banhead_s ban_head = VTAILQ_HEAD_INITIALIZER(ban_head);
 struct ban * volatile ban_start;
 
 static struct ban *ban_magic;
 static pthread_t ban_thread;
+static int ban_holds;
 
 struct ban_test {
 	uint8_t			arg1;
@@ -121,34 +122,31 @@ BAN_Free(struct ban *b)
 }
 
 /*--------------------------------------------------------------------
- * Get & Release a tail reference, used to hold the list stable for
- * traversals etc.
+ * Get/release holds which prevent the ban_lurker from starting.
+ * Holds are held while stevedores load zombie objects.
  */
 
-struct ban *
-BAN_TailRef(void)
+void
+BAN_Hold(void)
 {
-	struct ban *b;
 
-	ASSERT_CLI();
 	Lck_Lock(&ban_mtx);
-	b = VTAILQ_LAST(&ban_head, banhead_s);
-	AN(b);
-	b->refcount++;
+	/* Once holds are released, we allow no more */
+	assert(ban_holds > 0);
+	ban_holds++;
 	Lck_Unlock(&ban_mtx);
-	return (b);
 }
 
 void
-BAN_TailDeref(struct ban **bb)
+BAN_Release(void)
 {
-	struct ban *b;
 
-	b = *bb;
-	*bb = NULL;
 	Lck_Lock(&ban_mtx);
-	b->refcount--;
+	assert(ban_holds > 0);
+	ban_holds--;
 	Lck_Unlock(&ban_mtx);
+	if (ban_holds == 0)
+		WRK_BgThread(&ban_thread, "ban-lurker", ban_lurker, NULL);
 }
 
 /*--------------------------------------------------------------------
@@ -549,11 +547,11 @@ BAN_DestroyObj(struct objcore *oc)
 
 /*--------------------------------------------------------------------
  * Find and/or Grab a reference to an objects ban based on timestamp
- * Assume we hold a TailRef, so list traversal is safe.
+ * Assume we have a BAN_Hold, so list traversal is safe.
  */
 
 struct ban *
-BAN_RefBan(struct objcore *oc, double t0, const struct ban *tail)
+BAN_RefBan(struct objcore *oc, double t0)
 {
 	struct ban *b;
 	double t1 = 0;
@@ -562,12 +560,11 @@ BAN_RefBan(struct objcore *oc, double t0, const struct ban *tail)
 		t1 = ban_time(b->spec);
 		if (t1 <= t0)
 			break;
-		if (b == tail)
-			break;
 	}
 	AN(b);
 	assert(t1 == t0);
 	Lck_Lock(&ban_mtx);
+	assert(ban_holds > 0);
 	b->refcount++;
 	VTAILQ_INSERT_TAIL(&b->objcore, oc, ban_list);
 	Lck_Unlock(&ban_mtx);
@@ -966,7 +963,10 @@ ccf_ban_list(struct cli *cli, const char * const *av, void *priv)
 	(void)priv;
 
 	/* Get a reference so we are safe to traverse the list */
-	bl = BAN_TailRef();
+	Lck_Lock(&ban_mtx);
+	bl = VTAILQ_LAST(&ban_head, banhead_s);
+	bl->refcount++;
+	Lck_Unlock(&ban_mtx);
 
 	VCLI_Out(cli, "Present bans:\n");
 	VTAILQ_FOREACH(b, &ban_head, list) {
@@ -994,7 +994,9 @@ ccf_ban_list(struct cli *cli, const char * const *av, void *priv)
 		}
 	}
 
-	BAN_TailDeref(&bl);
+	Lck_Lock(&ban_mtx);
+	bl->refcount--;
+	Lck_Unlock(&ban_mtx);
 }
 
 static struct cli_proto ban_cmds[] = {
@@ -1010,7 +1012,8 @@ void
 BAN_Compile(void)
 {
 
-	/* All bans have been read from all persistent stevedores. Export
+	/*
+	 * All bans have been read from all persistent stevedores. Export
 	 * the compiled list
 	 */
 
@@ -1027,10 +1030,9 @@ BAN_Compile(void)
 	Lck_Unlock(&ban_mtx);
 
 	ban_start = VTAILQ_FIRST(&ban_head);
-	WRK_BgThread(&ban_thread, "ban-lurker", ban_lurker, NULL);
+	BAN_Release();
 }
 
-
 void
 BAN_Init(void)
 {
@@ -1043,6 +1045,7 @@ BAN_Init(void)
 	AZ(BAN_Insert(ban_magic));
 	Lck_Lock(&ban_mtx);
 	ban_mark_completed(ban_magic);
+	ban_holds = 1;
 	Lck_Unlock(&ban_mtx);
 }
 
diff --git a/bin/varnishd/cache/cache_priv.h b/bin/varnishd/cache/cache_priv.h
index 2423afd..483d041 100644
--- a/bin/varnishd/cache/cache_priv.h
+++ b/bin/varnishd/cache/cache_priv.h
@@ -57,11 +57,19 @@ void VBE_Poll(void);
 /* cache_backend_poll.c */
 void VBP_Init(void);
 
-/* cache_ban.c */
+/* == cache_ban.c == */
+
+/* From cache_main.c */
 void BAN_Init(void);
 void BAN_Compile(void);
 void BAN_Shutdown(void);
 
+/* From cache_hash.c */
+void BAN_NewObjCore(struct objcore *oc);
+void BAN_DestroyObj(struct objcore *oc);
+int BAN_CheckObject(struct worker *, struct objcore *, struct req *);
+
+
 /* cache_busyobj.c */
 void VBO_Init(void);
 
diff --git a/bin/varnishd/storage/storage_persistent.c b/bin/varnishd/storage/storage_persistent.c
index 2679675..b87fece 100644
--- a/bin/varnishd/storage/storage_persistent.c
+++ b/bin/varnishd/storage/storage_persistent.c
@@ -287,8 +287,7 @@ smp_thread(struct worker *wrk, void *priv)
 			smp_load_seg(wrk, sc, sg);
 
 	sc->flags |= SMP_SC_LOADED;
-	BAN_TailDeref(&sc->tailban);
-	AZ(sc->tailban);
+	BAN_Release();
 	printf("Silo completely loaded\n");
 
 	/* Housekeeping loop */
@@ -358,8 +357,7 @@ smp_open(const struct stevedore *st)
 	 * has loaded all objects, so we can be sure that all of our
 	 * proto-bans survive until then.
 	 */
-	sc->tailban = BAN_TailRef();
-	AN(sc->tailban);
+	BAN_Hold();
 
 	/* XXX: save segments to ensure consistency between seg1 & seg2 ? */
 
diff --git a/bin/varnishd/storage/storage_persistent.h b/bin/varnishd/storage/storage_persistent.h
index 8dac5e2..3a1d4f2 100644
--- a/bin/varnishd/storage/storage_persistent.h
+++ b/bin/varnishd/storage/storage_persistent.h
@@ -253,8 +253,6 @@ struct smp_sc {
 	struct smp_signspace	seg1;
 	struct smp_signspace	seg2;
 
-	struct ban		*tailban;
-
 	struct lock		mtx;
 
 	/* Cleaner metrics */
diff --git a/bin/varnishd/storage/storage_persistent_silo.c b/bin/varnishd/storage/storage_persistent_silo.c
index 45c8a66..ab604d8 100644
--- a/bin/varnishd/storage/storage_persistent_silo.c
+++ b/bin/varnishd/storage/storage_persistent_silo.c
@@ -163,7 +163,7 @@ smp_load_seg(struct worker *wrk, const struct smp_sc *sc,
 		oc->stobj->stevedore = sc->parent;
 		smp_init_oc(oc, sg, no);
 		oc->stobj->priv2 |= NEED_FIXUP;
-		oc->ban = BAN_RefBan(oc, so->ban, sc->tailban);
+		oc->ban = BAN_RefBan(oc, so->ban);
 		HSH_Insert(wrk, so->hash, oc);
 		oc->exp = so->exp;
 		sg->nobj++;



From phk at FreeBSD.org  Thu Oct 22 15:40:38 2015
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 22 Oct 2015 17:40:38 +0200
Subject: [master] 9339f1a Split BAN construction code into a separate source
 file
Message-ID: <E1ZpHyo-0002dI-No@project.varnish-software.com>


commit 9339f1aea31eba131e3dc116458b8a225b43cd23
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Oct 22 08:53:53 2015 +0000

    Split BAN construction code into a separate source file

diff --git a/bin/varnishd/Makefile.am b/bin/varnishd/Makefile.am
index 86906a2..d252c1b 100644
--- a/bin/varnishd/Makefile.am
+++ b/bin/varnishd/Makefile.am
@@ -15,6 +15,7 @@ varnishd_SOURCES = \
 	cache/cache_backend_probe.c \
 	cache/cache_backend_tcp.c \
 	cache/cache_ban.c \
+	cache/cache_ban_build.c \
 	cache/cache_ban_lurker.c \
 	cache/cache_busyobj.c \
 	cache/cache_cli.c \
diff --git a/bin/varnishd/cache/cache_ban.c b/bin/varnishd/cache/cache_ban.c
index e5e03e6..0247d44 100644
--- a/bin/varnishd/cache/cache_ban.c
+++ b/bin/varnishd/cache/cache_ban.c
@@ -40,14 +40,13 @@
 #include "vcli_priv.h"
 #include "vend.h"
 #include "vmb.h"
-#include "vtim.h"
 
 struct lock ban_mtx;
 int ban_shutdown;
 struct banhead_s ban_head = VTAILQ_HEAD_INITIALIZER(ban_head);
 struct ban * volatile ban_start;
+struct ban *ban_magic;
 
-static struct ban *ban_magic;
 static pthread_t ban_thread;
 static int ban_holds;
 
@@ -60,21 +59,6 @@ struct ban_test {
 };
 
 /*--------------------------------------------------------------------
- * Variables we can purge on
- */
-
-static const struct pvar {
-	const char		*name;
-	unsigned		flag;
-	uint8_t			tag;
-} pvars[] = {
-#define PVAR(a, b, c)	{ (a), (b), (c) },
-#include "tbl/ban_vars.h"
-#undef PVAR
-	{ 0, 0, 0}
-};
-
-/*--------------------------------------------------------------------
  * Storage handling of bans
  */
 
@@ -172,7 +156,7 @@ ban_len(const uint8_t *banspec)
 	return (u);
 }
 
-static int
+int
 ban_equal(const uint8_t *bs1, const uint8_t *bs2)
 {
 	unsigned u;
@@ -209,19 +193,6 @@ ban_mark_completed(struct ban *b)
  * Access a lump of bytes in a ban test spec
  */
 
-static void
-ban_add_lump(const struct ban *b, const void *p, uint32_t len)
-{
-	uint8_t buf[sizeof len];
-
-	buf[0] = 0xff;
-	while (VSB_len(b->vsb) & PALGN)
-		VSB_bcat(b->vsb, buf, 1);
-	vbe32enc(buf, len);
-	VSB_bcat(b->vsb, buf, sizeof buf);
-	VSB_bcat(b->vsb, p, len);
-}
-
 static const void *
 ban_get_lump(const uint8_t **bs)
 {
@@ -258,256 +229,6 @@ ban_iter(const uint8_t **bs, struct ban_test *bt)
 }
 
 /*--------------------------------------------------------------------
- */
-
-static int
-ban_error(struct ban *b, const char *fmt, ...)
-{
-	va_list ap;
-
-	CHECK_OBJ_NOTNULL(b, BAN_MAGIC);
-	AN(b->vsb);
-
-	/* First error is sticky */
-	if (!(b->flags & BANS_FLAG_ERROR)) {
-		b->flags |= BANS_FLAG_ERROR;
-
-		/* Record the error message in the vsb */
-		VSB_clear(b->vsb);
-		va_start(ap, fmt);
-		(void)VSB_vprintf(b->vsb, fmt, ap);
-		va_end(ap);
-	}
-	return (-1);
-}
-
-/*--------------------------------------------------------------------
- * Parse and add a http argument specification
- * Output something which HTTP_GetHdr understands
- */
-
-static void
-ban_parse_http(const struct ban *b, const char *a1)
-{
-	int l;
-
-	l = strlen(a1) + 1;
-	assert(l <= 127);
-	VSB_putc(b->vsb, (char)l);
-	VSB_cat(b->vsb, a1);
-	VSB_putc(b->vsb, ':');
-	VSB_putc(b->vsb, '\0');
-}
-
-/*--------------------------------------------------------------------
- * Parse and add a ban test specification
- */
-
-static int
-ban_parse_regexp(struct ban *b, const char *a3)
-{
-	const char *error;
-	int erroroffset, rc;
-	size_t sz;
-	pcre *re;
-
-	re = pcre_compile(a3, 0, &error, &erroroffset, NULL);
-	if (re == NULL)
-		return (ban_error(b, "Regex compile error: %s", error));
-	rc = pcre_fullinfo(re, NULL, PCRE_INFO_SIZE, &sz);
-	AZ(rc);
-	ban_add_lump(b, re, sz);
-	pcre_free(re);
-	return (0);
-}
-
-/*--------------------------------------------------------------------
- * Add a (and'ed) test-condition to a ban
- */
-
-int
-BAN_AddTest(struct ban *b, const char *a1, const char *a2, const char *a3)
-{
-	const struct pvar *pv;
-	int i;
-
-	CHECK_OBJ_NOTNULL(b, BAN_MAGIC);
-	AN(b->vsb);
-	AN(a1);
-	AN(a2);
-	AN(a3);
-
-	if (b->flags & BANS_FLAG_ERROR)
-		return (-1);
-
-	for (pv = pvars; pv->name != NULL; pv++)
-		if (!strncmp(a1, pv->name, strlen(pv->name)))
-			break;
-
-	if (pv->name == NULL)
-		return (ban_error(b,
-		    "Unknown or unsupported field \"%s\"", a1));
-
-	b->flags |= pv->flag;
-
-	VSB_putc(b->vsb, pv->tag);
-	if (pv->flag & BANS_FLAG_HTTP)
-		ban_parse_http(b, a1 + strlen(pv->name));
-
-	ban_add_lump(b, a3, strlen(a3) + 1);
-	if (!strcmp(a2, "~")) {
-		VSB_putc(b->vsb, BANS_OPER_MATCH);
-		i = ban_parse_regexp(b, a3);
-		if (i)
-			return (i);
-	} else if (!strcmp(a2, "!~")) {
-		VSB_putc(b->vsb, BANS_OPER_NMATCH);
-		i = ban_parse_regexp(b, a3);
-		if (i)
-			return (i);
-	} else if (!strcmp(a2, "==")) {
-		VSB_putc(b->vsb, BANS_OPER_EQ);
-	} else if (!strcmp(a2, "!=")) {
-		VSB_putc(b->vsb, BANS_OPER_NEQ);
-	} else {
-		return (ban_error(b,
-		    "expected conditional (~, !~, == or !=) got \"%s\"", a2));
-	}
-	return (0);
-}
-
-/*--------------------------------------------------------------------
- * We maintain ban_start as a pointer to the first element of the list
- * as a separate variable from the VTAILQ, to avoid depending on the
- * internals of the VTAILQ macros.  We tacitly assume that a pointer
- * write is always atomic in doing so.
- *
- * Returns:
- *   0: Ban successfully inserted
- *  -1: Ban not inserted due to shutdown in progress. The ban has been
- *      deleted.
- */
-
-static char ban_error_nomem[] = "Could not get memory";
-
-static char *
-ban_ins_error(const char *p)
-{
-	char *r = NULL;
-
-	if (p != NULL)
-		r = strdup(p);
-	if (r == NULL)
-		r = ban_error_nomem;
-	return (r);
-}
-
-void
-BAN_Free_Errormsg(char *p)
-{
-	if (p != ban_error_nomem)
-		free(p);
-}
-
-char *
-BAN_Insert(struct ban *b)
-{
-	struct ban  *bi, *be;
-	ssize_t ln;
-	double t0;
-	char *p;
-
-	CHECK_OBJ_NOTNULL(b, BAN_MAGIC);
-	AN(b->vsb);
-
-	if (ban_shutdown) {
-		BAN_Free(b);
-		return (ban_ins_error("Shutting down"));
-	}
-
-	AZ(VSB_finish(b->vsb));
-	ln = VSB_len(b->vsb);
-	assert(ln >= 0);
-
-	if (b->flags & BANS_FLAG_ERROR) {
-		p = ban_ins_error(VSB_data(b->vsb));
-		BAN_Free(b);
-		return (p);
-	}
-
-	b->spec = malloc(ln + BANS_HEAD_LEN);
-	if (b->spec == NULL) {
-		BAN_Free(b);
-		return (ban_ins_error(NULL));
-	}
-
-	memset(b->spec, 0, BANS_HEAD_LEN);
-	t0 = VTIM_real();
-	memcpy(b->spec + BANS_TIMESTAMP, &t0, sizeof t0);
-	b->spec[BANS_FLAGS] = b->flags & 0xff;
-	memcpy(b->spec + BANS_HEAD_LEN, VSB_data(b->vsb), ln);
-	ln += BANS_HEAD_LEN;
-	vbe32enc(b->spec + BANS_LENGTH, ln);
-
-	VSB_delete(b->vsb);
-	b->vsb = NULL;
-
-	Lck_Lock(&ban_mtx);
-	if (ban_shutdown) {
-		/* Check again, we might have raced */
-		Lck_Unlock(&ban_mtx);
-		BAN_Free(b);
-		return (ban_ins_error("Shutting down"));
-	}
-	VTAILQ_INSERT_HEAD(&ban_head, b, list);
-	ban_start = b;
-	VSC_C_main->bans++;
-	VSC_C_main->bans_added++;
-	if (b->flags & BANS_FLAG_OBJ)
-		VSC_C_main->bans_obj++;
-	if (b->flags & BANS_FLAG_REQ)
-		VSC_C_main->bans_req++;
-
-	be = VTAILQ_LAST(&ban_head, banhead_s);
-	if (cache_param->ban_dups && be != b)
-		be->refcount++;
-	else
-		be = NULL;
-
-	/* ban_magic is magic, and needs to be inserted early to give
-	 * a handle to grab a ref on. We don't report it here as the
-	 * stevedores will not be opened and ready to accept it
-	 * yet. Instead it is reported on BAN_Compile, which is after
-	 * the stevedores has been opened, but before any new objects
-	 * can have entered the cache (thus no objects in the mean
-	 * time depending on ban_magic in the list) */
-	VSC_C_main->bans_persisted_bytes += ln;
-	if (b != ban_magic)
-		ban_info(BI_NEW, b->spec, ln); /* Notify stevedores */
-	Lck_Unlock(&ban_mtx);
-
-	if (be == NULL)
-		return (NULL);
-
-	/* Hunt down duplicates, and mark them as completed */
-	bi = b;
-	Lck_Lock(&ban_mtx);
-	while (!ban_shutdown && bi != be) {
-		bi = VTAILQ_NEXT(bi, list);
-		if (bi->flags & BANS_FLAG_COMPLETED)
-			continue;
-		if (!ban_equal(b->spec, bi->spec))
-			continue;
-		ban_mark_completed(bi);
-		VSC_C_main->bans_dups++;
-	}
-	be->refcount--;
-	Lck_Unlock(&ban_mtx);
-
-	return (NULL);
-}
-
-/*--------------------------------------------------------------------
  * A new object is created, grab a reference to the newest ban
  */
 
diff --git a/bin/varnishd/cache/cache_ban.h b/bin/varnishd/cache/cache_ban.h
index ce0ae9e..755b2f4 100644
--- a/bin/varnishd/cache/cache_ban.h
+++ b/bin/varnishd/cache/cache_ban.h
@@ -107,6 +107,7 @@ extern struct lock ban_mtx;
 extern int ban_shutdown;
 extern struct banhead_s ban_head;
 extern struct ban * volatile ban_start;
+extern struct ban *ban_magic;
 
 void ban_mark_completed(struct ban *b);
 unsigned ban_len(const uint8_t *banspec);
@@ -114,3 +115,4 @@ void ban_info(enum baninfo event, const uint8_t *ban, unsigned len);
 int ban_evaluate(struct worker *wrk, const uint8_t *bs, struct objcore *oc,
     const struct http *reqhttp, unsigned *tests);
 double ban_time(const uint8_t *banspec);
+int ban_equal(const uint8_t *bs1, const uint8_t *bs2);
diff --git a/bin/varnishd/cache/cache_ban_build.c b/bin/varnishd/cache/cache_ban_build.c
new file mode 100644
index 0000000..ee2f0e7
--- /dev/null
+++ b/bin/varnishd/cache/cache_ban_build.c
@@ -0,0 +1,317 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2015 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ */
+
+#include "config.h"
+
+#include <pcre.h>
+
+#include "cache.h"
+#include "cache_ban.h"
+
+#include "vend.h"
+#include "vtim.h"
+
+/*--------------------------------------------------------------------
+ * Variables we can purge on
+ */
+
+static const struct pvar {
+	const char		*name;
+	unsigned		flag;
+	uint8_t			tag;
+} pvars[] = {
+#define PVAR(a, b, c)	{ (a), (b), (c) },
+#include "tbl/ban_vars.h"
+#undef PVAR
+	{ 0, 0, 0}
+};
+
+static void
+ban_add_lump(const struct ban *b, const void *p, uint32_t len)
+{
+	uint8_t buf[sizeof len];
+
+	buf[0] = 0xff;
+	while (VSB_len(b->vsb) & PALGN)
+		VSB_bcat(b->vsb, buf, 1);
+	vbe32enc(buf, len);
+	VSB_bcat(b->vsb, buf, sizeof buf);
+	VSB_bcat(b->vsb, p, len);
+}
+
+/*--------------------------------------------------------------------
+ */
+
+static int
+ban_error(struct ban *b, const char *fmt, ...)
+{
+	va_list ap;
+
+	CHECK_OBJ_NOTNULL(b, BAN_MAGIC);
+	AN(b->vsb);
+
+	/* First error is sticky */
+	if (!(b->flags & BANS_FLAG_ERROR)) {
+		b->flags |= BANS_FLAG_ERROR;
+
+		/* Record the error message in the vsb */
+		VSB_clear(b->vsb);
+		va_start(ap, fmt);
+		(void)VSB_vprintf(b->vsb, fmt, ap);
+		va_end(ap);
+	}
+	return (-1);
+}
+
+/*--------------------------------------------------------------------
+ * Parse and add a http argument specification
+ * Output something which HTTP_GetHdr understands
+ */
+
+static void
+ban_parse_http(const struct ban *b, const char *a1)
+{
+	int l;
+
+	l = strlen(a1) + 1;
+	assert(l <= 127);
+	VSB_putc(b->vsb, (char)l);
+	VSB_cat(b->vsb, a1);
+	VSB_putc(b->vsb, ':');
+	VSB_putc(b->vsb, '\0');
+}
+
+/*--------------------------------------------------------------------
+ * Parse and add a ban test specification
+ */
+
+static int
+ban_parse_regexp(struct ban *b, const char *a3)
+{
+	const char *error;
+	int erroroffset, rc;
+	size_t sz;
+	pcre *re;
+
+	re = pcre_compile(a3, 0, &error, &erroroffset, NULL);
+	if (re == NULL)
+		return (ban_error(b, "Regex compile error: %s", error));
+	rc = pcre_fullinfo(re, NULL, PCRE_INFO_SIZE, &sz);
+	AZ(rc);
+	ban_add_lump(b, re, sz);
+	pcre_free(re);
+	return (0);
+}
+
+/*--------------------------------------------------------------------
+ * Add a (and'ed) test-condition to a ban
+ */
+
+int
+BAN_AddTest(struct ban *b, const char *a1, const char *a2, const char *a3)
+{
+	const struct pvar *pv;
+	int i;
+
+	CHECK_OBJ_NOTNULL(b, BAN_MAGIC);
+	AN(b->vsb);
+	AN(a1);
+	AN(a2);
+	AN(a3);
+
+	if (b->flags & BANS_FLAG_ERROR)
+		return (-1);
+
+	for (pv = pvars; pv->name != NULL; pv++)
+		if (!strncmp(a1, pv->name, strlen(pv->name)))
+			break;
+
+	if (pv->name == NULL)
+		return (ban_error(b,
+		    "Unknown or unsupported field \"%s\"", a1));
+
+	b->flags |= pv->flag;
+
+	VSB_putc(b->vsb, pv->tag);
+	if (pv->flag & BANS_FLAG_HTTP)
+		ban_parse_http(b, a1 + strlen(pv->name));
+
+	ban_add_lump(b, a3, strlen(a3) + 1);
+	if (!strcmp(a2, "~")) {
+		VSB_putc(b->vsb, BANS_OPER_MATCH);
+		i = ban_parse_regexp(b, a3);
+		if (i)
+			return (i);
+	} else if (!strcmp(a2, "!~")) {
+		VSB_putc(b->vsb, BANS_OPER_NMATCH);
+		i = ban_parse_regexp(b, a3);
+		if (i)
+			return (i);
+	} else if (!strcmp(a2, "==")) {
+		VSB_putc(b->vsb, BANS_OPER_EQ);
+	} else if (!strcmp(a2, "!=")) {
+		VSB_putc(b->vsb, BANS_OPER_NEQ);
+	} else {
+		return (ban_error(b,
+		    "expected conditional (~, !~, == or !=) got \"%s\"", a2));
+	}
+	return (0);
+}
+
+/*--------------------------------------------------------------------
+ * We maintain ban_start as a pointer to the first element of the list
+ * as a separate variable from the VTAILQ, to avoid depending on the
+ * internals of the VTAILQ macros.  We tacitly assume that a pointer
+ * write is always atomic in doing so.
+ *
+ * Returns:
+ *   0: Ban successfully inserted
+ *  -1: Ban not inserted due to shutdown in progress. The ban has been
+ *      deleted.
+ */
+
+static char ban_error_nomem[] = "Could not get memory";
+
+static char *
+ban_ins_error(const char *p)
+{
+	char *r = NULL;
+
+	if (p != NULL)
+		r = strdup(p);
+	if (r == NULL)
+		r = ban_error_nomem;
+	return (r);
+}
+
+void
+BAN_Free_Errormsg(char *p)
+{
+	if (p != ban_error_nomem)
+		free(p);
+}
+
+char *
+BAN_Insert(struct ban *b)
+{
+	struct ban  *bi, *be;
+	ssize_t ln;
+	double t0;
+	char *p;
+
+	CHECK_OBJ_NOTNULL(b, BAN_MAGIC);
+	AN(b->vsb);
+
+	if (ban_shutdown) {
+		BAN_Free(b);
+		return (ban_ins_error("Shutting down"));
+	}
+
+	AZ(VSB_finish(b->vsb));
+	ln = VSB_len(b->vsb);
+	assert(ln >= 0);
+
+	if (b->flags & BANS_FLAG_ERROR) {
+		p = ban_ins_error(VSB_data(b->vsb));
+		BAN_Free(b);
+		return (p);
+	}
+
+	b->spec = malloc(ln + BANS_HEAD_LEN);
+	if (b->spec == NULL) {
+		BAN_Free(b);
+		return (ban_ins_error(NULL));
+	}
+
+	memset(b->spec, 0, BANS_HEAD_LEN);
+	t0 = VTIM_real();
+	memcpy(b->spec + BANS_TIMESTAMP, &t0, sizeof t0);
+	b->spec[BANS_FLAGS] = b->flags & 0xff;
+	memcpy(b->spec + BANS_HEAD_LEN, VSB_data(b->vsb), ln);
+	ln += BANS_HEAD_LEN;
+	vbe32enc(b->spec + BANS_LENGTH, ln);
+
+	VSB_delete(b->vsb);
+	b->vsb = NULL;
+
+	Lck_Lock(&ban_mtx);
+	if (ban_shutdown) {
+		/* Check again, we might have raced */
+		Lck_Unlock(&ban_mtx);
+		BAN_Free(b);
+		return (ban_ins_error("Shutting down"));
+	}
+	VTAILQ_INSERT_HEAD(&ban_head, b, list);
+	ban_start = b;
+	VSC_C_main->bans++;
+	VSC_C_main->bans_added++;
+	if (b->flags & BANS_FLAG_OBJ)
+		VSC_C_main->bans_obj++;
+	if (b->flags & BANS_FLAG_REQ)
+		VSC_C_main->bans_req++;
+
+	be = VTAILQ_LAST(&ban_head, banhead_s);
+	if (cache_param->ban_dups && be != b)
+		be->refcount++;
+	else
+		be = NULL;
+
+	/* ban_magic is magic, and needs to be inserted early to give
+	 * a handle to grab a ref on. We don't report it here as the
+	 * stevedores will not be opened and ready to accept it
+	 * yet. Instead it is reported on BAN_Compile, which is after
+	 * the stevedores has been opened, but before any new objects
+	 * can have entered the cache (thus no objects in the mean
+	 * time depending on ban_magic in the list) */
+	VSC_C_main->bans_persisted_bytes += ln;
+	if (b != ban_magic)
+		ban_info(BI_NEW, b->spec, ln); /* Notify stevedores */
+	Lck_Unlock(&ban_mtx);
+
+	if (be == NULL)
+		return (NULL);
+
+	/* Hunt down duplicates, and mark them as completed */
+	bi = b;
+	Lck_Lock(&ban_mtx);
+	while (!ban_shutdown && bi != be) {
+		bi = VTAILQ_NEXT(bi, list);
+		if (bi->flags & BANS_FLAG_COMPLETED)
+			continue;
+		if (!ban_equal(b->spec, bi->spec))
+			continue;
+		ban_mark_completed(bi);
+		VSC_C_main->bans_dups++;
+	}
+	be->refcount--;
+	Lck_Unlock(&ban_mtx);
+
+	return (NULL);
+}



From phk at FreeBSD.org  Thu Oct 22 15:40:38 2015
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 22 Oct 2015 17:40:38 +0200
Subject: [master] c4c8f2f Split the datastructures we use to construct bans
 (ban_proto) from the data structure we use to store and operate on the ban
 (ban).
Message-ID: <E1ZpHyo-0002dS-Ov@project.varnish-software.com>


commit c4c8f2fae16fe888b9745442e708ad6dfcc91d93
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Oct 22 15:39:51 2015 +0000

    Split the datastructures we use to construct bans (ban_proto)
    from the data structure we use to store and operate on the ban (ban).

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 11c8be1..93e5de4 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -102,6 +102,7 @@ enum {
 
 struct VSC_C_lck;
 struct ban;
+struct ban_proto;
 struct backend;
 struct busyobj;
 struct cli;
@@ -657,11 +658,11 @@ typedef enum htc_status_e htc_complete_f(struct http_conn *);
 /* cache_ban.c */
 
 /* for constructing bans */
-struct ban *BAN_New(void);
-int BAN_AddTest(struct ban *, const char *, const char *, const char *);
-void BAN_Free(struct ban *b);
-char *BAN_Insert(struct ban *b);
-void BAN_Free_Errormsg(char *);
+struct ban_proto *BAN_Build(void);
+const char *BAN_AddTest(struct ban_proto *,
+    const char *, const char *, const char *);
+const char *BAN_Commit(struct ban_proto *b);
+void BAN_Abandon(struct ban_proto *b);
 
 /* for stevedoes resurrecting bans */
 void BAN_Hold(void);
diff --git a/bin/varnishd/cache/cache_ban.c b/bin/varnishd/cache/cache_ban.c
index 0247d44..95ca144 100644
--- a/bin/varnishd/cache/cache_ban.c
+++ b/bin/varnishd/cache/cache_ban.c
@@ -45,7 +45,6 @@ struct lock ban_mtx;
 int ban_shutdown;
 struct banhead_s ban_head = VTAILQ_HEAD_INITIALIZER(ban_head);
 struct ban * volatile ban_start;
-struct ban *ban_magic;
 
 static pthread_t ban_thread;
 static int ban_holds;
@@ -73,23 +72,6 @@ ban_alloc(void)
 	return (b);
 }
 
-struct ban *
-BAN_New(void)
-{
-	struct ban *b;
-
-	b = ban_alloc();
-	if (b != NULL) {
-		b->vsb = VSB_new_auto();
-		if (b->vsb == NULL) {
-			FREE_OBJ(b);
-			return (NULL);
-		}
-		VTAILQ_INIT(&b->objcore);
-	}
-	return (b);
-}
-
 void
 BAN_Free(struct ban *b)
 {
@@ -98,8 +80,6 @@ BAN_Free(struct ban *b)
 	AZ(b->refcount);
 	assert(VTAILQ_EMPTY(&b->objcore));
 
-	if (b->vsb != NULL)
-		VSB_delete(b->vsb);
 	if (b->spec != NULL)
 		free(b->spec);
 	FREE_OBJ(b);
@@ -595,8 +575,8 @@ static void
 ccf_ban(struct cli *cli, const char * const *av, void *priv)
 {
 	int narg, i;
-	struct ban *b;
-	char *p;
+	struct ban_proto *bp;
+	const char *err = NULL;
 
 	(void)priv;
 
@@ -616,19 +596,24 @@ ccf_ban(struct cli *cli, const char * const *av, void *priv)
 		}
 	}
 
-	b = BAN_New();
-	if (b == NULL) {
+	bp = BAN_Build();
+	if (bp == NULL) {
 		VCLI_Out(cli, "Out of Memory");
 		VCLI_SetResult(cli, CLIS_CANT);
 		return;
 	}
-	for (i = 0; i < narg; i += 4)
-		if (BAN_AddTest(b, av[i + 2], av[i + 3], av[i + 4]))
+	for (i = 0; i < narg; i += 4) {
+		err = BAN_AddTest(bp, av[i + 2], av[i + 3], av[i + 4]);
+		if (err)
 			break;
-	p = BAN_Insert(b);
-	if (p != NULL) {
-		VCLI_Out(cli, "%s", p);
-		BAN_Free_Errormsg(p);
+	}
+
+	if (err == NULL)
+		err = BAN_Commit(bp);
+
+	if (err != NULL) {
+		VCLI_Out(cli, "%s", err);
+		BAN_Abandon(bp);
 		VCLI_SetResult(cli, CLIS_PARAM);
 	}
 }
@@ -679,6 +664,7 @@ static void
 ccf_ban_list(struct cli *cli, const char * const *av, void *priv)
 {
 	struct ban *b, *bl;
+	int64_t o;
 
 	(void)av;
 	(void)priv;
@@ -691,14 +677,14 @@ ccf_ban_list(struct cli *cli, const char * const *av, void *priv)
 
 	VCLI_Out(cli, "Present bans:\n");
 	VTAILQ_FOREACH(b, &ban_head, list) {
-		VCLI_Out(cli, "%10.6f %5u %s", ban_time(b->spec),
-		    bl == b ? b->refcount - 1 : b->refcount,
+		o = bl == b ? 1 : 0;
+		VCLI_Out(cli, "%10.6f %5ju %s", ban_time(b->spec),
+		    (intmax_t)b->refcount - o,
 		    b->flags & BANS_FLAG_COMPLETED ? "C" : " ");
 		if (DO_DEBUG(DBG_LURKER)) {
-			VCLI_Out(cli, "%s%s%s %p ",
+			VCLI_Out(cli, "%s%s %p ",
 			    b->flags & BANS_FLAG_REQ ? "R" : "-",
 			    b->flags & BANS_FLAG_OBJ ? "O" : "-",
-			    b->flags & BANS_FLAG_ERROR ? "E" : "-",
 			    b);
 		}
 		VCLI_Out(cli, "  ");
@@ -732,6 +718,7 @@ static struct cli_proto ban_cmds[] = {
 void
 BAN_Compile(void)
 {
+	struct ban *b;
 
 	/*
 	 * All bans have been read from all persistent stevedores. Export
@@ -743,8 +730,9 @@ BAN_Compile(void)
 
 	Lck_Lock(&ban_mtx);
 
-	/* Do late reporting of ban_magic */
-	AZ(STV_BanInfo(BI_NEW, ban_magic->spec, ban_len(ban_magic->spec)));
+	/* Report the place-holder ban */
+	b = VTAILQ_FIRST(&ban_head);
+	AZ(STV_BanInfo(BI_NEW, b->spec, ban_len(b->spec)));
 
 	ban_export();
 
@@ -757,16 +745,19 @@ BAN_Compile(void)
 void
 BAN_Init(void)
 {
+	struct ban_proto *bp;
 
 	Lck_New(&ban_mtx, lck_ban);
 	CLI_AddFuncs(ban_cmds);
 
-	ban_magic = BAN_New();
-	AN(ban_magic);
-	AZ(BAN_Insert(ban_magic));
-	Lck_Lock(&ban_mtx);
-	ban_mark_completed(ban_magic);
 	ban_holds = 1;
+
+	/* Add a placeholder ban */
+	bp = BAN_Build();
+	AN(bp);
+	AZ(BAN_Commit(bp));
+	Lck_Lock(&ban_mtx);
+	ban_mark_completed(VTAILQ_FIRST(&ban_head));
 	Lck_Unlock(&ban_mtx);
 }
 
diff --git a/bin/varnishd/cache/cache_ban.h b/bin/varnishd/cache/cache_ban.h
index 755b2f4..047d0f9 100644
--- a/bin/varnishd/cache/cache_ban.h
+++ b/bin/varnishd/cache/cache_ban.h
@@ -73,7 +73,6 @@
 #define BANS_FLAG_OBJ		(1<<1)
 #define BANS_FLAG_COMPLETED	(1<<2)
 #define BANS_FLAG_HTTP		(1<<3)
-#define BANS_FLAG_ERROR		(1<<4)
 
 #define BANS_OPER_EQ		0x10
 #define BANS_OPER_NEQ		0x11
@@ -90,13 +89,12 @@
 struct ban {
 	unsigned		magic;
 #define BAN_MAGIC		0x700b08ea
+	unsigned		flags;		/* BANS_FLAG_* */
 	VTAILQ_ENTRY(ban)	list;
 	VTAILQ_ENTRY(ban)	l_list;
-	int			refcount;
-	unsigned		flags;		/* BANS_FLAG_* */
+	int64_t			refcount;
 
 	VTAILQ_HEAD(,objcore)	objcore;
-	struct vsb		*vsb;
 	uint8_t			*spec;
 };
 
@@ -107,7 +105,6 @@ extern struct lock ban_mtx;
 extern int ban_shutdown;
 extern struct banhead_s ban_head;
 extern struct ban * volatile ban_start;
-extern struct ban *ban_magic;
 
 void ban_mark_completed(struct ban *b);
 unsigned ban_len(const uint8_t *banspec);
@@ -116,3 +113,4 @@ int ban_evaluate(struct worker *wrk, const uint8_t *bs, struct objcore *oc,
     const struct http *reqhttp, unsigned *tests);
 double ban_time(const uint8_t *banspec);
 int ban_equal(const uint8_t *bs1, const uint8_t *bs2);
+void BAN_Free(struct ban *b);
diff --git a/bin/varnishd/cache/cache_ban_build.c b/bin/varnishd/cache/cache_ban_build.c
index ee2f0e7..eb10c30 100644
--- a/bin/varnishd/cache/cache_ban_build.c
+++ b/bin/varnishd/cache/cache_ban_build.c
@@ -38,6 +38,15 @@
 #include "vend.h"
 #include "vtim.h"
 
+struct ban_proto {
+	unsigned		magic;
+#define BAN_PROTO_MAGIC		0xd8adc494
+	unsigned		flags;		/* BANS_FLAG_* */
+
+	struct vsb		*vsb;
+	char			*err;
+};
+
 /*--------------------------------------------------------------------
  * Variables we can purge on
  */
@@ -53,41 +62,83 @@ static const struct pvar {
 	{ 0, 0, 0}
 };
 
+/*--------------------------------------------------------------------
+ */
+
+static char ban_build_err_no_mem[] = "No Memory";
+
+/*--------------------------------------------------------------------
+ */
+
+struct ban_proto *
+BAN_Build(void)
+{
+	struct ban_proto *bp;
+
+	ALLOC_OBJ(bp, BAN_PROTO_MAGIC);
+	if (bp == NULL)
+		return (bp);
+	bp->vsb = VSB_new_auto();
+	if (bp->vsb == NULL) {
+		FREE_OBJ(bp);
+		return (NULL);
+	}
+	return (bp);
+}
+
+void
+BAN_Abandon(struct ban_proto *bp)
+{
+
+	CHECK_OBJ_NOTNULL(bp, BAN_PROTO_MAGIC);
+	VSB_delete(bp->vsb);
+	if (bp->err != NULL && bp->err != ban_build_err_no_mem)
+		REPLACE(bp->err, NULL);
+	FREE_OBJ(bp);
+}
+
+/*--------------------------------------------------------------------
+ */
+
 static void
-ban_add_lump(const struct ban *b, const void *p, uint32_t len)
+ban_add_lump(const struct ban_proto *bp, const void *p, uint32_t len)
 {
 	uint8_t buf[sizeof len];
 
 	buf[0] = 0xff;
-	while (VSB_len(b->vsb) & PALGN)
-		VSB_bcat(b->vsb, buf, 1);
+	while (VSB_len(bp->vsb) & PALGN)
+		VSB_bcat(bp->vsb, buf, 1);
 	vbe32enc(buf, len);
-	VSB_bcat(b->vsb, buf, sizeof buf);
-	VSB_bcat(b->vsb, p, len);
+	VSB_bcat(bp->vsb, buf, sizeof buf);
+	VSB_bcat(bp->vsb, p, len);
 }
 
 /*--------------------------------------------------------------------
  */
 
-static int
-ban_error(struct ban *b, const char *fmt, ...)
+static const char *
+ban_error(struct ban_proto *bp, const char *fmt, ...)
 {
 	va_list ap;
 
-	CHECK_OBJ_NOTNULL(b, BAN_MAGIC);
-	AN(b->vsb);
+	CHECK_OBJ_NOTNULL(bp, BAN_PROTO_MAGIC);
+	AN(bp->vsb);
 
 	/* First error is sticky */
-	if (!(b->flags & BANS_FLAG_ERROR)) {
-		b->flags |= BANS_FLAG_ERROR;
-
-		/* Record the error message in the vsb */
-		VSB_clear(b->vsb);
-		va_start(ap, fmt);
-		(void)VSB_vprintf(b->vsb, fmt, ap);
-		va_end(ap);
+	if (bp->err == NULL) {
+		if (fmt == ban_build_err_no_mem) {
+			bp->err = ban_build_err_no_mem;
+		} else {
+			/* Record the error message in the vsb */
+			VSB_clear(bp->vsb);
+			va_start(ap, fmt);
+			(void)VSB_vprintf(bp->vsb, fmt, ap);
+			va_end(ap);
+			AZ(VSB_finish(bp->vsb));
+			bp->err = VSB_data(bp->vsb);
+		}
 	}
-	return (-1);
+	return (bp->err);
 }
 
 /*--------------------------------------------------------------------
@@ -96,24 +147,24 @@ ban_error(struct ban *b, const char *fmt, ...)
  */
 
 static void
-ban_parse_http(const struct ban *b, const char *a1)
+ban_parse_http(const struct ban_proto *bp, const char *a1)
 {
 	int l;
 
 	l = strlen(a1) + 1;
 	assert(l <= 127);
-	VSB_putc(b->vsb, (char)l);
-	VSB_cat(b->vsb, a1);
-	VSB_putc(b->vsb, ':');
-	VSB_putc(b->vsb, '\0');
+	VSB_putc(bp->vsb, (char)l);
+	VSB_cat(bp->vsb, a1);
+	VSB_putc(bp->vsb, ':');
+	VSB_putc(bp->vsb, '\0');
 }
 
 /*--------------------------------------------------------------------
  * Parse and add a ban test specification
  */
 
-static int
-ban_parse_regexp(struct ban *b, const char *a3)
+static const char *
+ban_parse_regexp(struct ban_proto *bp, const char *a3)
 {
 	const char *error;
 	int erroroffset, rc;
@@ -122,10 +173,10 @@ ban_parse_regexp(struct ban *b, const char *a3)
 
 	re = pcre_compile(a3, 0, &error, &erroroffset, NULL);
 	if (re == NULL)
-		return (ban_error(b, "Regex compile error: %s", error));
+		return (ban_error(bp, "Regex compile error: %s", error));
 	rc = pcre_fullinfo(re, NULL, PCRE_INFO_SIZE, &sz);
 	AZ(rc);
-	ban_add_lump(b, re, sz);
+	ban_add_lump(bp, re, sz);
 	pcre_free(re);
 	return (0);
 }
@@ -134,55 +185,56 @@ ban_parse_regexp(struct ban *b, const char *a3)
  * Add a (and'ed) test-condition to a ban
  */
 
-int
-BAN_AddTest(struct ban *b, const char *a1, const char *a2, const char *a3)
+const char *
+BAN_AddTest(struct ban_proto *bp,
+    const char *a1, const char *a2, const char *a3)
 {
 	const struct pvar *pv;
-	int i;
+	const char *err;
 
-	CHECK_OBJ_NOTNULL(b, BAN_MAGIC);
-	AN(b->vsb);
+	CHECK_OBJ_NOTNULL(bp, BAN_PROTO_MAGIC);
+	AN(bp->vsb);
 	AN(a1);
 	AN(a2);
 	AN(a3);
 
-	if (b->flags & BANS_FLAG_ERROR)
-		return (-1);
+	if (bp->err != NULL)
+		return (bp->err);
 
 	for (pv = pvars; pv->name != NULL; pv++)
 		if (!strncmp(a1, pv->name, strlen(pv->name)))
 			break;
 
 	if (pv->name == NULL)
-		return (ban_error(b,
+		return (ban_error(bp,
 		    "Unknown or unsupported field \"%s\"", a1));
 
-	b->flags |= pv->flag;
+	bp->flags |= pv->flag;
 
-	VSB_putc(b->vsb, pv->tag);
+	VSB_putc(bp->vsb, pv->tag);
 	if (pv->flag & BANS_FLAG_HTTP)
-		ban_parse_http(b, a1 + strlen(pv->name));
+		ban_parse_http(bp, a1 + strlen(pv->name));
 
-	ban_add_lump(b, a3, strlen(a3) + 1);
+	ban_add_lump(bp, a3, strlen(a3) + 1);
 	if (!strcmp(a2, "~")) {
-		VSB_putc(b->vsb, BANS_OPER_MATCH);
-		i = ban_parse_regexp(b, a3);
-		if (i)
-			return (i);
+		VSB_putc(bp->vsb, BANS_OPER_MATCH);
+		err = ban_parse_regexp(bp, a3);
+		if (err)
+			return (err);
 	} else if (!strcmp(a2, "!~")) {
-		VSB_putc(b->vsb, BANS_OPER_NMATCH);
-		i = ban_parse_regexp(b, a3);
-		if (i)
-			return (i);
+		VSB_putc(bp->vsb, BANS_OPER_NMATCH);
+		err = ban_parse_regexp(bp, a3);
+		if (err)
+			return (err);
 	} else if (!strcmp(a2, "==")) {
-		VSB_putc(b->vsb, BANS_OPER_EQ);
+		VSB_putc(bp->vsb, BANS_OPER_EQ);
 	} else if (!strcmp(a2, "!=")) {
-		VSB_putc(b->vsb, BANS_OPER_NEQ);
+		VSB_putc(bp->vsb, BANS_OPER_NEQ);
 	} else {
-		return (ban_error(b,
+		return (ban_error(bp,
 		    "expected conditional (~, !~, == or !=) got \"%s\"", a2));
 	}
-	return (0);
+	return (NULL);
 }
 
 /*--------------------------------------------------------------------
@@ -197,121 +249,80 @@ BAN_AddTest(struct ban *b, const char *a1, const char *a2, const char *a3)
  *      deleted.
  */
 
-static char ban_error_nomem[] = "Could not get memory";
-
-static char *
-ban_ins_error(const char *p)
+const char *
+BAN_Commit(struct ban_proto *bp)
 {
-	char *r = NULL;
-
-	if (p != NULL)
-		r = strdup(p);
-	if (r == NULL)
-		r = ban_error_nomem;
-	return (r);
-}
-
-void
-BAN_Free_Errormsg(char *p)
-{
-	if (p != ban_error_nomem)
-		free(p);
-}
-
-char *
-BAN_Insert(struct ban *b)
-{
-	struct ban  *bi, *be;
+	struct ban  *b, *bi;
 	ssize_t ln;
 	double t0;
-	char *p;
 
-	CHECK_OBJ_NOTNULL(b, BAN_MAGIC);
-	AN(b->vsb);
+	CHECK_OBJ_NOTNULL(bp, BAN_PROTO_MAGIC);
+	AN(bp->vsb);
 
-	if (ban_shutdown) {
-		BAN_Free(b);
-		return (ban_ins_error("Shutting down"));
-	}
+	if (ban_shutdown)
+		return (ban_error(bp, "Shutting down"));
 
-	AZ(VSB_finish(b->vsb));
-	ln = VSB_len(b->vsb);
+	AZ(VSB_finish(bp->vsb));
+	ln = VSB_len(bp->vsb);
 	assert(ln >= 0);
 
-	if (b->flags & BANS_FLAG_ERROR) {
-		p = ban_ins_error(VSB_data(b->vsb));
-		BAN_Free(b);
-		return (p);
-	}
+	ALLOC_OBJ(b, BAN_MAGIC);
+	if (b == NULL)
+		return (ban_error(bp, ban_build_err_no_mem));
+	VTAILQ_INIT(&b->objcore);
 
 	b->spec = malloc(ln + BANS_HEAD_LEN);
 	if (b->spec == NULL) {
-		BAN_Free(b);
-		return (ban_ins_error(NULL));
+		free(b);
+		return (ban_error(bp, ban_build_err_no_mem));
 	}
 
+	b->flags = bp->flags;
+
 	memset(b->spec, 0, BANS_HEAD_LEN);
 	t0 = VTIM_real();
 	memcpy(b->spec + BANS_TIMESTAMP, &t0, sizeof t0);
 	b->spec[BANS_FLAGS] = b->flags & 0xff;
-	memcpy(b->spec + BANS_HEAD_LEN, VSB_data(b->vsb), ln);
+	memcpy(b->spec + BANS_HEAD_LEN, VSB_data(bp->vsb), ln);
 	ln += BANS_HEAD_LEN;
 	vbe32enc(b->spec + BANS_LENGTH, ln);
 
-	VSB_delete(b->vsb);
-	b->vsb = NULL;
-
 	Lck_Lock(&ban_mtx);
 	if (ban_shutdown) {
-		/* Check again, we might have raced */
+		/* We could have raced a shutdown */
 		Lck_Unlock(&ban_mtx);
 		BAN_Free(b);
-		return (ban_ins_error("Shutting down"));
+		return (ban_error(bp, "Shutting down"));
 	}
+	bi = VTAILQ_FIRST(&ban_head);
 	VTAILQ_INSERT_HEAD(&ban_head, b, list);
 	ban_start = b;
+
 	VSC_C_main->bans++;
 	VSC_C_main->bans_added++;
+	VSC_C_main->bans_persisted_bytes += ln;
+
 	if (b->flags & BANS_FLAG_OBJ)
 		VSC_C_main->bans_obj++;
 	if (b->flags & BANS_FLAG_REQ)
 		VSC_C_main->bans_req++;
 
-	be = VTAILQ_LAST(&ban_head, banhead_s);
-	if (cache_param->ban_dups && be != b)
-		be->refcount++;
-	else
-		be = NULL;
-
-	/* ban_magic is magic, and needs to be inserted early to give
-	 * a handle to grab a ref on. We don't report it here as the
-	 * stevedores will not be opened and ready to accept it
-	 * yet. Instead it is reported on BAN_Compile, which is after
-	 * the stevedores has been opened, but before any new objects
-	 * can have entered the cache (thus no objects in the mean
-	 * time depending on ban_magic in the list) */
-	VSC_C_main->bans_persisted_bytes += ln;
-	if (b != ban_magic)
-		ban_info(BI_NEW, b->spec, ln); /* Notify stevedores */
-	Lck_Unlock(&ban_mtx);
-
-	if (be == NULL)
-		return (NULL);
-
-	/* Hunt down duplicates, and mark them as completed */
-	bi = b;
-	Lck_Lock(&ban_mtx);
-	while (!ban_shutdown && bi != be) {
-		bi = VTAILQ_NEXT(bi, list);
-		if (bi->flags & BANS_FLAG_COMPLETED)
-			continue;
-		if (!ban_equal(b->spec, bi->spec))
-			continue;
-		ban_mark_completed(bi);
-		VSC_C_main->bans_dups++;
+	if (bi != NULL)
+		ban_info(BI_NEW, b->spec, ln);	/* Notify stevedores */
+
+	if (cache_param->ban_dups) {
+		/* Hunt down duplicates, and mark them as completed */
+		for (bi = VTAILQ_NEXT(b, list); bi != NULL;
+		    bi = VTAILQ_NEXT(bi, list)) {
+			if (!(bi->flags & BANS_FLAG_COMPLETED) &&
+			    ban_equal(b->spec, bi->spec)) {
+				ban_mark_completed(bi);
+				VSC_C_main->bans_dups++;
+			}
+		}
 	}
-	be->refcount--;
 	Lck_Unlock(&ban_mtx);
 
+	BAN_Abandon(bp);
 	return (NULL);
 }
diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c
index 642e754..09441e0 100644
--- a/bin/varnishd/cache/cache_vrt.c
+++ b/bin/varnishd/cache/cache_vrt.c
@@ -395,15 +395,16 @@ VRT_ban_string(VRT_CTX, const char *str)
 {
 	char *a1, *a2, *a3;
 	char **av;
-	struct ban *b;
+	struct ban_proto *bp;
+	const char *err;
 	int i;
 
 	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
 	AN(ctx->vsl);
 	AN(str);
 
-	b = BAN_New();
-	if (b == NULL) {
+	bp = BAN_Build();
+	if (bp == NULL) {
 		VSLb(ctx->vsl, SLT_VCL_Error, "ban(): Out of Memory");
 		return;
 	}
@@ -412,7 +413,7 @@ VRT_ban_string(VRT_CTX, const char *str)
 	if (av[0] != NULL) {
 		VSLb(ctx->vsl, SLT_VCL_Error, "ban(): %s", av[0]);
 		VAV_Free(av);
-		BAN_Free(b);
+		BAN_Abandon(bp);
 		return;
 	}
 	for (i = 0; ;) {
@@ -434,13 +435,17 @@ VRT_ban_string(VRT_CTX, const char *str)
 			    "ban(): Expected second operand.");
 			break;
 		}
-		if (BAN_AddTest(b, a1, a2, a3) || av[++i] == NULL) {
-			a1 = BAN_Insert(b);
-			if (a1 != NULL) {
-				VSLb(ctx->vsl, SLT_VCL_Error,
-				    "ban(): %s", a1);
-				BAN_Free_Errormsg(a1);
-			}
+		err = BAN_AddTest(bp, a1, a2, a3);
+		if (err) {
+			VSLb(ctx->vsl, SLT_VCL_Error, "ban(): %s", err);
+			break;
+		}
+		if (av[++i] == NULL) {
+			err = BAN_Commit(bp);
+			if (err == NULL)
+				bp = NULL;
+			else
+				VSLb(ctx->vsl, SLT_VCL_Error, "ban(): %s", err);
 			break;
 		}
 		if (strcmp(av[i], "&&")) {
@@ -450,6 +455,8 @@ VRT_ban_string(VRT_CTX, const char *str)
 			break;
 		}
 	}
+	if (bp != NULL)
+		BAN_Abandon(bp);
 	VAV_Free(av);
 }
 
diff --git a/bin/varnishd/flint.lnt b/bin/varnishd/flint.lnt
index 5904b1e..38453d1 100644
--- a/bin/varnishd/flint.lnt
+++ b/bin/varnishd/flint.lnt
@@ -106,6 +106,7 @@
 
 -emacro(527, NEEDLESS_RETURN)	// unreachable code
 
+-sem(BAN_Free, custodial(1))
 -sem(EXP_Inject, custodial(1))
 -sem(HSH_Insert, custodial(3))
 -sem(WS_Init, custodial(2))



From phk at FreeBSD.org  Thu Oct 22 15:57:44 2015
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 22 Oct 2015 17:57:44 +0200
Subject: [master] a420b7f Sigh... almost all fun in C gets outlawed these
 days...
Message-ID: <E1ZpIFM-00043P-QB@project.varnish-software.com>


commit a420b7fea11ee823535c1031823428bf2ed97209
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Oct 22 15:57:14 2015 +0000

    Sigh... almost all fun in C gets outlawed these days...

diff --git a/bin/varnishd/cache/cache_ban_build.c b/bin/varnishd/cache/cache_ban_build.c
index eb10c30..bec9062 100644
--- a/bin/varnishd/cache/cache_ban_build.c
+++ b/bin/varnishd/cache/cache_ban_build.c
@@ -93,7 +93,7 @@ BAN_Abandon(struct ban_proto *bp)
 	CHECK_OBJ_NOTNULL(bp, BAN_PROTO_MAGIC);
 	VSB_delete(bp->vsb);
 	if (bp->err != NULL && bp->err != ban_build_err_no_mem)
-		REPLACE(bp->err, NULL);
+		free(bp->err);
 	FREE_OBJ(bp);
 }
 



From phk at FreeBSD.org  Thu Oct 22 18:01:06 2015
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 22 Oct 2015 20:01:06 +0200
Subject: [master] 6ca1aef Don't double-free.
Message-ID: <E1ZpKAk-0004Jt-PS@project.varnish-software.com>


commit 6ca1aef16add038f8427ed142434e760d1b3c600
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Oct 22 18:00:41 2015 +0000

    Don't double-free.
    
    Amazingly JEMALLOC didn't spot this one.

diff --git a/bin/varnishd/cache/cache_ban_build.c b/bin/varnishd/cache/cache_ban_build.c
index bec9062..b807c9a 100644
--- a/bin/varnishd/cache/cache_ban_build.c
+++ b/bin/varnishd/cache/cache_ban_build.c
@@ -92,8 +92,6 @@ BAN_Abandon(struct ban_proto *bp)
 
 	CHECK_OBJ_NOTNULL(bp, BAN_PROTO_MAGIC);
 	VSB_delete(bp->vsb);
-	if (bp->err != NULL && bp->err != ban_build_err_no_mem)
-		free(bp->err);
 	FREE_OBJ(bp);
 }
 



From phk at FreeBSD.org  Thu Oct 22 19:32:39 2015
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 22 Oct 2015 21:32:39 +0200
Subject: [master] 3670ad6 Make sure complaints from (je)malloc aren't lost,
 at least on FreeBSD.
Message-ID: <E1ZpLbL-0002Gf-KH@project.varnish-software.com>


commit 3670ad6dafb563c04400ebb42fc260c6f11cab17
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Oct 22 19:31:28 2015 +0000

    Make sure complaints from (je)malloc aren't lost, at least on FreeBSD.

diff --git a/bin/varnishd/cache/cache_main.c b/bin/varnishd/cache/cache_main.c
index e47f685..cc5471c 100644
--- a/bin/varnishd/cache/cache_main.c
+++ b/bin/varnishd/cache/cache_main.c
@@ -188,6 +188,16 @@ static struct cli_proto debug_cmds[] = {
  * XXX: Think more about which order we start things
  */
 
+#ifdef __FreeBSD__
+static void
+child_malloc_fail(void *p, const char *s)
+{
+	VSL(SLT_Error, 0, "MALLOC ERROR: %s (%p)", s, p);
+	fprintf(stderr, "MALLOC ERROR: %s (%p)\n", s, p);
+	WRONG("Malloc Error");
+}
+#endif
+
 void
 child_main(void)
 {
@@ -195,6 +205,9 @@ child_main(void)
 	setbuf(stdout, NULL);
 	setbuf(stderr, NULL);
 	printf("Child starts\n");
+#ifdef __FreeBSD__
+	malloc_message = child_malloc_fail;
+#endif
 
 	cache_param = heritage.param;
 



From phk at FreeBSD.org  Thu Oct 22 22:49:02 2015
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 23 Oct 2015 00:49:02 +0200
Subject: [master] 467f44b Make the ban lurker sleep on a condvar so other code
 can wake it up.
Message-ID: <E1ZpOfO-0006n3-RQ@project.varnish-software.com>


commit 467f44b2afff70c177e8ad099816ae304cfddd36
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Oct 22 20:34:18 2015 +0000

    Make the ban lurker sleep on a condvar so other code can wake it up.

diff --git a/bin/varnishd/cache/cache_ban.c b/bin/varnishd/cache/cache_ban.c
index 95ca144..6d66999 100644
--- a/bin/varnishd/cache/cache_ban.c
+++ b/bin/varnishd/cache/cache_ban.c
@@ -756,6 +756,7 @@ BAN_Init(void)
 	bp = BAN_Build();
 	AN(bp);
 	AZ(BAN_Commit(bp));
+	AZ(pthread_cond_init(&ban_lurker_cond, NULL));
 	Lck_Lock(&ban_mtx);
 	ban_mark_completed(VTAILQ_FIRST(&ban_head));
 	Lck_Unlock(&ban_mtx);
@@ -774,9 +775,8 @@ BAN_Shutdown(void)
 {
 	void *status;
 
-	Lck_Lock(&ban_mtx);
 	ban_shutdown = 1;
-	Lck_Unlock(&ban_mtx);
+	ban_kick_lurker();
 
 	AZ(pthread_join(ban_thread, &status));
 	AZ(status);
diff --git a/bin/varnishd/cache/cache_ban.h b/bin/varnishd/cache/cache_ban.h
index 047d0f9..100acee 100644
--- a/bin/varnishd/cache/cache_ban.h
+++ b/bin/varnishd/cache/cache_ban.h
@@ -105,6 +105,7 @@ extern struct lock ban_mtx;
 extern int ban_shutdown;
 extern struct banhead_s ban_head;
 extern struct ban * volatile ban_start;
+extern pthread_cond_t	ban_lurker_cond;
 
 void ban_mark_completed(struct ban *b);
 unsigned ban_len(const uint8_t *banspec);
@@ -114,3 +115,4 @@ int ban_evaluate(struct worker *wrk, const uint8_t *bs, struct objcore *oc,
 double ban_time(const uint8_t *banspec);
 int ban_equal(const uint8_t *bs1, const uint8_t *bs2);
 void BAN_Free(struct ban *b);
+void ban_kick_lurker(void);
diff --git a/bin/varnishd/cache/cache_ban_lurker.c b/bin/varnishd/cache/cache_ban_lurker.c
index cfcdd26..821d8c8 100644
--- a/bin/varnishd/cache/cache_ban_lurker.c
+++ b/bin/varnishd/cache/cache_ban_lurker.c
@@ -38,6 +38,18 @@
 
 static struct objcore oc_marker = { .magic = OBJCORE_MAGIC, };
 static unsigned ban_batch;
+static unsigned ban_generation;
+
+pthread_cond_t	ban_lurker_cond;
+
+void
+ban_kick_lurker(void)
+{
+	Lck_Lock(&ban_mtx);
+	ban_generation++;
+	AZ(pthread_cond_signal(&ban_lurker_cond));
+	Lck_Unlock(&ban_mtx);
+}
 
 static void
 ban_cleantail(void)
@@ -254,7 +266,10 @@ ban_lurker(struct worker *wrk, void *priv)
 		if (d <= 0.0 || !ban_lurker_work(wrk, &vsl))
 			d = 0.609;	// Random, non-magic
 		ban_cleantail();
-		VTIM_sleep(d);
+		d += VTIM_real();
+		Lck_Lock(&ban_mtx);
+		(void)Lck_CondWait(&ban_lurker_cond, &ban_mtx, d);
+		Lck_Unlock(&ban_mtx);
 	}
 	pthread_exit(0);
 	NEEDLESS_RETURN(NULL);



From phk at FreeBSD.org  Thu Oct 22 22:49:02 2015
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 23 Oct 2015 00:49:02 +0200
Subject: [master] 128fb32 Kick the ban_lurker into action whenever we do
 something that makes work for it. Have it sleep as long as possible,
 rather than hot-polling for work to do.
Message-ID: <E1ZpOfO-0006n9-Sj@project.varnish-software.com>


commit 128fb323827894042561c7d62286a7e471cacf27
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Oct 22 22:17:34 2015 +0000

    Kick the ban_lurker into action whenever we do something that
    makes work for it.  Have it sleep as long as possible, rather
    than hot-polling for work to do.

diff --git a/bin/varnishd/cache/cache_ban.c b/bin/varnishd/cache/cache_ban.c
index 6d66999..3bf9073 100644
--- a/bin/varnishd/cache/cache_ban.c
+++ b/bin/varnishd/cache/cache_ban.c
@@ -552,6 +552,9 @@ BAN_CheckObject(struct worker *wrk, struct objcore *oc, struct req *req)
 		b0->refcount++;
 	}
 
+	if (oc->ban->refcount == 0 && VTAILQ_NEXT(oc->ban, list) == NULL)
+		ban_kick_lurker();
+
 	Lck_Unlock(&ban_mtx);
 
 	if (b == oc->ban) {	/* not banned */
@@ -703,6 +706,7 @@ ccf_ban_list(struct cli *cli, const char * const *av, void *priv)
 
 	Lck_Lock(&ban_mtx);
 	bl->refcount--;
+	ban_kick_lurker();	// XXX: Mostly for testcase b00009.vtc
 	Lck_Unlock(&ban_mtx);
 }
 
@@ -775,8 +779,10 @@ BAN_Shutdown(void)
 {
 	void *status;
 
+	Lck_Lock(&ban_mtx);
 	ban_shutdown = 1;
 	ban_kick_lurker();
+	Lck_Unlock(&ban_mtx);
 
 	AZ(pthread_join(ban_thread, &status));
 	AZ(status);
diff --git a/bin/varnishd/cache/cache_ban_build.c b/bin/varnishd/cache/cache_ban_build.c
index b807c9a..801774f 100644
--- a/bin/varnishd/cache/cache_ban_build.c
+++ b/bin/varnishd/cache/cache_ban_build.c
@@ -319,6 +319,8 @@ BAN_Commit(struct ban_proto *bp)
 			}
 		}
 	}
+	if (!(b->flags & BANS_FLAG_REQ))
+		ban_kick_lurker();
 	Lck_Unlock(&ban_mtx);
 
 	BAN_Abandon(bp);
diff --git a/bin/varnishd/cache/cache_ban_lurker.c b/bin/varnishd/cache/cache_ban_lurker.c
index 821d8c8..352b219 100644
--- a/bin/varnishd/cache/cache_ban_lurker.c
+++ b/bin/varnishd/cache/cache_ban_lurker.c
@@ -45,10 +45,10 @@ pthread_cond_t	ban_lurker_cond;
 void
 ban_kick_lurker(void)
 {
-	Lck_Lock(&ban_mtx);
+
+	Lck_AssertHeld(&ban_mtx);
 	ban_generation++;
 	AZ(pthread_cond_signal(&ban_lurker_cond));
-	Lck_Unlock(&ban_mtx);
 }
 
 static void
@@ -191,14 +191,17 @@ ban_lurker_test_ban(struct worker *wrk, struct vsl_log *vsl, struct ban *bt,
  * Ban lurker thread
  */
 
-static int
+static double
 ban_lurker_work(struct worker *wrk, struct vsl_log *vsl)
 {
 	struct ban *b, *bt;
 	struct banhead_s obans;
-	double d;
+	double d, dt, n;
 	int i;
 
+	dt = 49.62;		// Random, non-magic
+	if (cache_param->ban_lurker_sleep == 0)
+		return (dt);
 	/* Make a list of the bans we can do something about */
 	VTAILQ_INIT(&obans);
 	Lck_Lock(&ban_mtx);
@@ -206,26 +209,27 @@ ban_lurker_work(struct worker *wrk, struct vsl_log *vsl)
 	Lck_Unlock(&ban_mtx);
 	i = 0;
 	d = VTIM_real() - cache_param->ban_lurker_age;
-	while (b != NULL) {
-		if (b->flags & BANS_FLAG_COMPLETED) {
-			;
-		} else if (b->flags & BANS_FLAG_REQ) {
-			;
-		} else if (b == VTAILQ_LAST(&ban_head, banhead_s)) {
-			;
-		} else if (ban_time(b->spec) > d) {
-			;
-		} else {
+	for (; b != NULL; b = VTAILQ_NEXT(b, list)) {
+		if (b->flags & BANS_FLAG_COMPLETED)
+			continue;
+		if (b->flags & BANS_FLAG_REQ)
+			continue;
+		if (b == VTAILQ_LAST(&ban_head, banhead_s))
+			continue;	// XXX: why ?
+		n = ban_time(b->spec) - d;
+		if (n < 0) {
 			VTAILQ_INSERT_TAIL(&obans, b, l_list);
 			i++;
+		} else if (n < dt) {
+			dt = n;
 		}
-		b = VTAILQ_NEXT(b, list);
 	}
 	if (DO_DEBUG(DBG_LURKER))
-		VSLb(vsl, SLT_Debug, "lurker: %d actionable bans", i);
+		VSLb(vsl, SLT_Debug, "lurker: %d actionable bans, dt = %lf", i, dt);
 	if (i == 0)
-		return (0);
+		return (dt);
 
+	dt = cache_param->ban_lurker_sleep;
 	/* Go though all the bans to test the objects */
 	VTAILQ_FOREACH_REVERSE(bt, &ban_head, banhead_s, list) {
 		if (bt == VTAILQ_LAST(&obans, banhead_s)) {
@@ -247,7 +251,7 @@ ban_lurker_work(struct worker *wrk, struct vsl_log *vsl)
 		if (VTAILQ_EMPTY(&obans))
 			break;
 	}
-	return (1);
+	return (dt);
 }
 
 void * __match_proto__(bgthread_t)
@@ -255,6 +259,7 @@ ban_lurker(struct worker *wrk, void *priv)
 {
 	struct vsl_log vsl;
 	volatile double d;
+	unsigned gen = ban_generation + 1;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	AZ(priv);
@@ -262,13 +267,17 @@ ban_lurker(struct worker *wrk, void *priv)
 	VSL_Setup(&vsl, NULL, 0);
 
 	while (!ban_shutdown) {
-		d = cache_param->ban_lurker_sleep;
-		if (d <= 0.0 || !ban_lurker_work(wrk, &vsl))
-			d = 0.609;	// Random, non-magic
+		d = ban_lurker_work(wrk, &vsl);
+		if (DO_DEBUG(DBG_LURKER))
+			VSLb(&vsl, SLT_Debug, "lurker: sleep = %lf", d);
 		ban_cleantail();
 		d += VTIM_real();
 		Lck_Lock(&ban_mtx);
-		(void)Lck_CondWait(&ban_lurker_cond, &ban_mtx, d);
+		if (gen == ban_generation) {
+			(void)Lck_CondWait(&ban_lurker_cond, &ban_mtx, d);
+			ban_batch = 0;
+		}
+		gen = ban_generation;
 		Lck_Unlock(&ban_mtx);
 	}
 	pthread_exit(0);
diff --git a/bin/varnishtest/tests/c00049.vtc b/bin/varnishtest/tests/c00049.vtc
index 40f7b94..8d10e68 100644
--- a/bin/varnishtest/tests/c00049.vtc
+++ b/bin/varnishtest/tests/c00049.vtc
@@ -40,6 +40,7 @@ varnish v1 -vcl+backend {} -start
 varnish v1 -cliok "param.set ban_lurker_age 0"
 varnish v1 -cliok "param.set ban_lurker_sleep 0"
 varnish v1 -cliok "param.set debug +lurker"
+varnish v1 -cliok "param.set debug +syncvsl"
 
 
 client c1 {
@@ -114,6 +115,8 @@ varnish v1 -expect bans_dups == 0
 
 varnish v1 -cliok "param.set ban_lurker_sleep .01"
 
+varnish v1 -cliok "ban.list"
+
 delay 2
 
 varnish v1 -cliok "ban.list"
diff --git a/bin/varnishtest/tests/p00009.vtc b/bin/varnishtest/tests/p00009.vtc
index 6ec15d4..21d65a6 100644
--- a/bin/varnishtest/tests/p00009.vtc
+++ b/bin/varnishtest/tests/p00009.vtc
@@ -53,5 +53,4 @@ client c1 {
 
 # Expect our duplicate
 varnish v1 -expect bans_dups == 1
-# One more than before restart due to the new ban_magic
-varnish v1 -expect bans_completed == 3
+varnish v1 -expect bans_completed == 1
diff --git a/bin/varnishtest/tests/r01030.vtc b/bin/varnishtest/tests/r01030.vtc
index 302b918..3f7343a 100644
--- a/bin/varnishtest/tests/r01030.vtc
+++ b/bin/varnishtest/tests/r01030.vtc
@@ -40,9 +40,9 @@ client c1 {
 } -run
 
 #delay 0.1
-varnish v1 -expect bans_lurker_tests_tested == 0
+#varnish v1 -expect bans_lurker_tests_tested == 0
 
-delay 1.5
+#delay 1.5
 varnish v1 -expect bans_lurker_tests_tested >= 1
 
 varnish v1 -cliok "param.set ban_lurker_sleep 5.01"
@@ -58,7 +58,7 @@ client c2 {
 } -run
 
 #delay 0.1
-varnish v1 -expect bans_lurker_tests_tested == 1
+#varnish v1 -expect bans_lurker_tests_tested == 1
 
-delay 1.1
+#delay 1.1
 varnish v1 -expect bans_lurker_tests_tested == 2
diff --git a/include/tbl/params.h b/include/tbl/params.h
index d2650f4..1968df4 100644
--- a/include/tbl/params.h
+++ b/include/tbl/params.h
@@ -135,10 +135,10 @@ PARAM(
 	/* units */	"seconds",
 	/* flags */	0,
 	/* s-text */
-	"The ban lurker only process bans when they are this old.  "
-	"When a ban is added, the most frequently hit objects will "
-	"get tested against it as part of object lookup.  This parameter "
-	"prevents the ban-lurker from kicking in, until the rush is over.",
+	"The ban lurker will ignore bans until they are this old.  "
+	"When a ban is added, the active traffic will be tested against it "
+	"as part of object lookup.  This parameter "
+	"holds the ban-lurker off, until the rush is over.",
 	/* l-text */	"",
 	/* func */	NULL
 )
@@ -152,9 +152,9 @@ PARAM(
 	/* units */	NULL,
 	/* flags */	0,
 	/* s-text */
-	"The ban lurker slees ${ban_lurker_sleep} after examining this "
-	"many objects.  Use this to pace the ban-lurker if it eats too "
-	"many resources.",
+	"The ban lurker sleeps ${ban_lurker_sleep} after examining this "
+	"many objects."
+	"  Use this to pace the ban-lurker if it eats too many resources.",
 	/* l-text */	"",
 	/* func */	NULL
 )
@@ -169,7 +169,8 @@ PARAM(
 	/* flags */	0,
 	/* s-text */
 	"How long the ban lurker sleeps after examining ${ban_lurker_batch} "
-	"objects.\n"
+	"objects."
+	"  Use this to pace the ban-lurker if it eats too many resources.\n"
 	"A value of zero will disable the ban lurker entirely.",
 	/* l-text */	"",
 	/* func */	NULL



From fgsch at lodoss.net  Fri Oct 23 04:23:13 2015
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Fri, 23 Oct 2015 06:23:13 +0200
Subject: [master] 159fc1c Do not force python3
Message-ID: <E1ZpTso-0004Jn-05@project.varnish-software.com>


commit 159fc1cd5e9a3594f4dda781d7db43527ccba620
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Thu Oct 22 18:51:29 2015 +0100

    Do not force python3
    
    Use `python' as with the other Python scripts.

diff --git a/lib/libvcc/generate.py b/lib/libvcc/generate.py
index f8aeb79..7a7413f 100755
--- a/lib/libvcc/generate.py
+++ b/lib/libvcc/generate.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python3
+#!/usr/bin/env python
 #-
 # Copyright (c) 2006 Verdens Gang AS
 # Copyright (c) 2006-2015 Varnish Software AS



From fgsch at lodoss.net  Fri Oct 23 04:23:14 2015
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Fri, 23 Oct 2015 06:23:14 +0200
Subject: [master] da39283 First stab at fixing some corner case
Message-ID: <E1ZpTso-0004Jt-1C@project.varnish-software.com>


commit da3928356730f1aa99b873ad1ce2c1dec3009069
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Fri Oct 23 05:10:20 2015 +0100

    First stab at fixing some corner case
    
    Found by ingvar while packaging 4.1.0.

diff --git a/bin/varnishtest/tests/v00017.vtc b/bin/varnishtest/tests/v00017.vtc
index b0d6c20..e00ad4f 100644
--- a/bin/varnishtest/tests/v00017.vtc
+++ b/bin/varnishtest/tests/v00017.vtc
@@ -36,9 +36,9 @@ varnish v1 -errvcl {DNS lookup(...com): } {
 	sub vcl_recv { if (client.ip ~ a) { return(pass); } }
 }
 
-varnish v1 -errvcl {DNS lookup(10.1.2.): } {
+varnish v1 -errvcl {DNS lookup(10.1..2): } {
 	backend b { .host = "127.0.0.1"; }
-	acl a { "10.1.2."; }
+	acl a { "10.1..2"; }
 	sub vcl_recv { if (client.ip ~ a) { return(pass); } }
 }
 



From daghf at varnish-software.com  Sun Oct 25 19:22:10 2015
From: daghf at varnish-software.com (Dag Haavi Finstad)
Date: Sun, 25 Oct 2015 20:22:10 +0100
Subject: [master] 6b283ff Add a note to clarify the event function's priv
 object.
Message-ID: <E1ZqQrq-0002kK-K5@project.varnish-software.com>


commit 6b283ff1cbc3c3a2b01a7cef477003ccc9fe80a8
Author: Dag Haavi Finstad <daghf at varnish-software.com>
Date:   Sun Oct 25 20:22:05 2015 +0100

    Add a note to clarify the event function's priv object.

diff --git a/doc/sphinx/reference/vmod.rst b/doc/sphinx/reference/vmod.rst
index 697fc6a..91953b0 100644
--- a/doc/sphinx/reference/vmod.rst
+++ b/doc/sphinx/reference/vmod.rst
@@ -302,7 +302,8 @@ The VCL compiler supports the following private pointers:
 * ``PRIV_VCL`` "per vcl" private pointers are useful for such global
   state that applies to all calls in this VCL, for instance flags that
   determine if regular expressions are case-sensitive in this vmod or
-  similar.
+  similar. The ``PRIV_VCL`` object is the same object that is passed
+  to the VMOD's event function.
 
 The way it works in the vmod code, is that a ``struct vmod_priv *`` is
 passed to the functions where one of the ``PRIV_*`` argument types is



From fgsch at lodoss.net  Sun Oct 25 23:21:25 2015
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Mon, 26 Oct 2015 00:21:25 +0100
Subject: [master] 54471f2 Update RFC reference
Message-ID: <E1ZqUbN-0006sC-GH@project.varnish-software.com>


commit 54471f2a86ae7234a31c97c0adb5e70a1b28ee38
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Sun Oct 25 23:16:28 2015 +0000

    Update RFC reference
    
    Fixes #1809.

diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 8ee198b..8e0a70b 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -342,7 +342,7 @@ vbf_stp_startfetch(struct worker *wrk, struct busyobj *bo)
 	} else if (http_IsStatus(bo->beresp, 204)) {
 		/*
 		 * 204 is "No Content", obviously don't expect a body.
-		 * [RFC2616 10.2.5 p60]
+		 * [RFC7230 3.3.1 p28 and 3.3.2 p30]
 		 */
 		wrk->stats->fetch_204++;
 		if (http_GetHdr(bo->beresp, H_Content_Length, NULL) ||



From guillaume at varnish-software.com  Mon Oct 26 10:52:39 2015
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Mon, 26 Oct 2015 11:52:39 +0100
Subject: [master] 5830d18 Remove unnecessary check
Message-ID: <E1ZqfOJ-0000Mn-UU@project.varnish-software.com>


commit 5830d189014d2625212ddc7b39b3d0c494d41c3e
Author: Guillaume Quintard <guillaume at varnish-software.com>
Date:   Mon Oct 26 10:59:05 2015 +0100

    Remove unnecessary check

diff --git a/include/miniobj.h b/include/miniobj.h
index 3de4091..8b4f043 100644
--- a/include/miniobj.h
+++ b/include/miniobj.h
@@ -61,8 +61,7 @@
 
 #define REPLACE(ptr, val)						\
 	do {								\
-		if ((ptr) != NULL)					\
-			free(ptr);					\
+		free(ptr);						\
 		if ((val) != NULL) {					\
 			ptr = strdup(val);				\
 			AN((ptr));					\



From fgsch at lodoss.net  Mon Oct 26 15:52:05 2015
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Mon, 26 Oct 2015 16:52:05 +0100
Subject: [master] b8f55d9 Require -w when running in daemon mode
Message-ID: <E1Zqk45-0002hm-Tc@project.varnish-software.com>


commit b8f55d90d085da59b69ca5185818ea83e284552b
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Mon Oct 26 13:50:15 2015 +0000

    Require -w when running in daemon mode

diff --git a/bin/varnishlog/varnishlog.c b/bin/varnishlog/varnishlog.c
index 3b5061f..8e600b5 100644
--- a/bin/varnishlog/varnishlog.c
+++ b/bin/varnishlog/varnishlog.c
@@ -145,6 +145,9 @@ main(int argc, char * const *argv)
 	if (optind != argc)
 		usage(1);
 
+	if (VUT.D_opt && !LOG.w_arg)
+		VUT_Error(1, "Missing -w option");
+
 	/* Setup output */
 	if (LOG.A_opt || !LOG.w_arg)
 		VUT.dispatch_f = VSL_PrintTransactions;
diff --git a/bin/varnishlog/varnishlog_options.h b/bin/varnishlog/varnishlog_options.h
index 187ac57..ed62abd 100644
--- a/bin/varnishlog/varnishlog_options.h
+++ b/bin/varnishlog/varnishlog_options.h
@@ -49,7 +49,8 @@
 	    " receives a SIGHUP the file will be reopened allowing"	\
 	    " the old one to be rotated away. The file can then be"	\
 	    " read by varnishlog and other tools with the -r option,"   \
-	    " unless the -A option was specified."			\
+	    " unless the -A option was specified. This option is"	\
+	    " required when running in daemon mode."			\
 	)
 
 LOG_OPT_a
diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c
index 77e6354..2ee2137 100644
--- a/bin/varnishncsa/varnishncsa.c
+++ b/bin/varnishncsa/varnishncsa.c
@@ -972,6 +972,9 @@ main(int argc, char * const *argv)
 	if (optind != argc)
 		usage(1);
 
+	if (VUT.D_opt && !CTX.w_arg)
+		VUT_Error(1, "Missing -w option");
+
 	/* Check for valid grouping mode */
 	assert(VUT.g_arg < VSL_g__MAX);
 	if (VUT.g_arg != VSL_g_vxid && VUT.g_arg != VSL_g_request)
diff --git a/bin/varnishncsa/varnishncsa_options.h b/bin/varnishncsa/varnishncsa_options.h
index edb5a39..87075cc 100644
--- a/bin/varnishncsa/varnishncsa_options.h
+++ b/bin/varnishncsa/varnishncsa_options.h
@@ -51,7 +51,8 @@
 	    "Redirect output to file. The file will be overwritten"	\
 	    " unless the -a option was specified. If the application"	\
 	    " receives a SIGHUP the file will be reopened allowing"	\
-	    " the old one to be rotated away."				\
+	    " the old one to be rotated away. This option is required"	\
+	    " when running in daemon mode."				\
 	)
 
 NCSA_OPT_a



From fgsch at lodoss.net  Mon Oct 26 15:52:05 2015
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Mon, 26 Oct 2015 16:52:05 +0100
Subject: [master] b214d48 Handle terminal disconnections correctly
Message-ID: <E1Zqk45-0002hs-Uw@project.varnish-software.com>


commit b214d4806ed9f63a8bf95db29c176f1da86be13b
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Mon Oct 26 13:54:29 2015 +0000

    Handle terminal disconnections correctly
    
    Change SIGHUP handling depending on whether we're running in daemon mode
    or foreground. The former will continue rotating the logs, the latter
    will abort the loop and die gracefully.
    
    Fixes #1794.

diff --git a/bin/varnishhist/varnishhist.c b/bin/varnishhist/varnishhist.c
index aec72e9..9eab9bf 100644
--- a/bin/varnishhist/varnishhist.c
+++ b/bin/varnishhist/varnishhist.c
@@ -291,6 +291,12 @@ accumulate(struct VSL_data *vsl, struct VSL_transaction * const pt[],
 	return (0);
 }
 
+static int __match_proto__(VUT_cb_f)
+sighup(void)
+{
+	return (1);
+}
+
 static void *
 do_curses(void *arg)
 {
@@ -465,6 +471,7 @@ main(int argc, char **argv)
 	}
 	VUT.dispatch_f = &accumulate;
 	VUT.dispatch_priv = NULL;
+	VUT.sighup_f = sighup;
 	VUT_Main();
 	end_of_file = 1;
 	AZ(pthread_join(thr, NULL));
diff --git a/bin/varnishlog/varnishlog.c b/bin/varnishlog/varnishlog.c
index 8e600b5..bc4a915 100644
--- a/bin/varnishlog/varnishlog.c
+++ b/bin/varnishlog/varnishlog.c
@@ -110,6 +110,12 @@ flushout(void)
 	return (0);
 }
 
+static int __match_proto__(VUT_cb_f)
+sighup(void)
+{
+	return (1);
+}
+
 int
 main(int argc, char * const *argv)
 {
@@ -153,10 +159,12 @@ main(int argc, char * const *argv)
 		VUT.dispatch_f = VSL_PrintTransactions;
 	else
 		VUT.dispatch_f = VSL_WriteTransactions;
+	VUT.sighup_f = sighup;
 	if (LOG.w_arg) {
 		openout(LOG.a_opt);
 		AN(LOG.fo);
-		VUT.sighup_f = rotateout;
+		if (VUT.D_opt)
+			VUT.sighup_f = rotateout;
 	} else
 		LOG.fo = stdout;
 	VUT.idle_f = flushout;
diff --git a/bin/varnishlog/varnishlog_options.h b/bin/varnishlog/varnishlog_options.h
index ed62abd..7b23a6c 100644
--- a/bin/varnishlog/varnishlog_options.h
+++ b/bin/varnishlog/varnishlog_options.h
@@ -46,11 +46,11 @@
 	VOPT("w:", "[-w filename]", "Output filename",			\
 	    "Redirect output to file. The file will be overwritten"	\
 	    " unless the -a option was specified. If the application"	\
-	    " receives a SIGHUP the file will be reopened allowing"	\
-	    " the old one to be rotated away. The file can then be"	\
-	    " read by varnishlog and other tools with the -r option,"   \
-	    " unless the -A option was specified. This option is"	\
-	    " required when running in daemon mode."			\
+	    " receives a SIGHUP in daemon mode the file will be "	\
+	    " reopened allowing the old one to be rotated away. The"	\
+	    " file can then be read by varnishlog and other tools with"	\
+	    " the -r option, unless the -A option was specified. This"	\
+	    " option is required when running in daemon mode."		\
 	)
 
 LOG_OPT_a
diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c
index 2ee2137..b19b679 100644
--- a/bin/varnishncsa/varnishncsa.c
+++ b/bin/varnishncsa/varnishncsa.c
@@ -927,6 +927,12 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[],
 	return (0);
 }
 
+static int __match_proto__(VUT_cb_f)
+sighup(void)
+{
+	return (1);
+}
+
 int
 main(int argc, char * const *argv)
 {
@@ -991,10 +997,12 @@ main(int argc, char * const *argv)
 	/* Setup output */
 	VUT.dispatch_f = &dispatch_f;
 	VUT.dispatch_priv = NULL;
+	VUT.sighup_f = sighup;
 	if (CTX.w_arg) {
 		openout(CTX.a_opt);
 		AN(CTX.fo);
-		VUT.sighup_f = &rotateout;
+		if (VUT.D_opt)
+			VUT.sighup_f = &rotateout;
 	} else
 		CTX.fo = stdout;
 	VUT.idle_f = &flushout;
diff --git a/bin/varnishncsa/varnishncsa_options.h b/bin/varnishncsa/varnishncsa_options.h
index 87075cc..291a1c7 100644
--- a/bin/varnishncsa/varnishncsa_options.h
+++ b/bin/varnishncsa/varnishncsa_options.h
@@ -41,7 +41,7 @@
 	)
 
 #define NCSA_OPT_g							\
-	VOPT("g:", "[-g <request|vxid>]", "Grouping mode (default: vxid)",		\
+	VOPT("g:", "[-g <request|vxid>]", "Grouping mode (default: vxid)", \
 	    "The grouping of the log records. The default is to group"	\
 	    " by vxid."							\
 	)
@@ -50,9 +50,9 @@
 	VOPT("w:", "[-w filename]", "Output filename",			\
 	    "Redirect output to file. The file will be overwritten"	\
 	    " unless the -a option was specified. If the application"	\
-	    " receives a SIGHUP the file will be reopened allowing"	\
-	    " the old one to be rotated away. This option is required"	\
-	    " when running in daemon mode."				\
+	    " receives a SIGHUP in daemon mode the file will be"	\
+	    " reopened allowing the old one to be rotated away. This"	\
+	    " option is required when running in daemon mode."		\
 	)
 
 NCSA_OPT_a
diff --git a/bin/varnishtest/tests/u00001.vtc b/bin/varnishtest/tests/u00001.vtc
index 8f35294..1a1c039 100644
--- a/bin/varnishtest/tests/u00001.vtc
+++ b/bin/varnishtest/tests/u00001.vtc
@@ -12,7 +12,7 @@ server s1 {
 
 varnish v1 -vcl+backend "" -start
 
-process p1 "exec ${varnishncsa} -n ${v1_name} -w ${tmpdir}/ncsa.log -F %s" -start
+shell "${varnishncsa} -D -P ${tmpdir}/ncsa.pid -n ${v1_name} -w ${tmpdir}/ncsa.log -F %s"
 
 # give varnishncsa enough time to open the VSM
 delay 2
@@ -27,7 +27,7 @@ delay 2
 
 # rotate logs
 shell "mv ${tmpdir}/ncsa.log ${tmpdir}/ncsa.old.log >/dev/null 2>&1"
-process p1 -kill "HUP"
+shell "kill -HUP `cat ${tmpdir}/ncsa.pid`"
 
 client c1 {
 	txreq -url "/bar"
@@ -37,7 +37,7 @@ client c1 {
 # give varnishncsa enough time to write
 delay 2
 
-process p1 -stop
+shell "kill `cat ${tmpdir}/ncsa.pid`"
 
 shell "grep 200 ${tmpdir}/ncsa.old.log >/dev/null"
 shell "grep 404 ${tmpdir}/ncsa.log >/dev/null"
diff --git a/bin/varnishtop/varnishtop.c b/bin/varnishtop/varnishtop.c
index 3ab9009..f56b9a7 100644
--- a/bin/varnishtop/varnishtop.c
+++ b/bin/varnishtop/varnishtop.c
@@ -180,6 +180,12 @@ accumulate(struct VSL_data *vsl, struct VSL_transaction * const pt[],
 	return (0);
 }
 
+static int __match_proto__(VUT_cb_f)
+sighup(void)
+{
+	return (1);
+}
+
 static void
 update(int p)
 {
@@ -362,6 +368,7 @@ main(int argc, char **argv)
 	}
 	VUT.dispatch_f = &accumulate;
 	VUT.dispatch_priv = NULL;
+	VUT.sighup_f = sighup;
 	VUT_Main();
 	end_of_file = 1;
 	if (once)



From fgsch at lodoss.net  Mon Oct 26 16:46:52 2015
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Mon, 26 Oct 2015 17:46:52 +0100
Subject: [master] e70fe44 Fix building on FreeBSD < 10.0
Message-ID: <E1Zqkv6-0006xX-C3@project.varnish-software.com>


commit e70fe44a67408c8deb95f87fe8ca2826bc0d2f10
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Mon Oct 26 16:46:18 2015 +0000

    Fix building on FreeBSD < 10.0

diff --git a/bin/varnishd/cache/cache_main.c b/bin/varnishd/cache/cache_main.c
index cc5471c..83de0cd 100644
--- a/bin/varnishd/cache/cache_main.c
+++ b/bin/varnishd/cache/cache_main.c
@@ -188,7 +188,7 @@ static struct cli_proto debug_cmds[] = {
  * XXX: Think more about which order we start things
  */
 
-#ifdef __FreeBSD__
+#if defined(__FreeBSD__) && __FreeBSD__version >= 1000000
 static void
 child_malloc_fail(void *p, const char *s)
 {
@@ -205,7 +205,7 @@ child_main(void)
 	setbuf(stdout, NULL);
 	setbuf(stderr, NULL);
 	printf("Child starts\n");
-#ifdef __FreeBSD__
+#if defined(__FreeBSD__) && __FreeBSD__version >= 1000000
 	malloc_message = child_malloc_fail;
 #endif
 



From martin at varnish-software.com  Tue Oct 27 14:28:41 2015
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Tue, 27 Oct 2015 15:28:41 +0100
Subject: [master] 86b3f9f Log proxy related messages on the session,
 not on the request.
Message-ID: <E1Zr5Ev-0000Zk-DN@project.varnish-software.com>


commit 86b3f9feca54af49ce194f5d6bed9e4dcfa54971
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Fri Oct 23 15:51:58 2015 +0200

    Log proxy related messages on the session, not on the request.
    
    The proxy log records were attempted to be logged on the request' log
    buffer, which at that point in time has not yet been set up and does
    not posess a VXID. This caused VXID==0 log records to be inserted in
    the beginning of the log for the first request on the session when
    using proxy protocol, and subsequently the VSL to fail picking them
    out as valid request log transactions.
    
    Fix by logging the PROXY handling related messages on the session, in
    which they belong.
    
    Fixes: #1804

diff --git a/bin/varnishd/proxy/cache_proxy_proto.c b/bin/varnishd/proxy/cache_proxy_proto.c
index 433581e..954beef 100644
--- a/bin/varnishd/proxy/cache_proxy_proto.c
+++ b/bin/varnishd/proxy/cache_proxy_proto.c
@@ -62,8 +62,9 @@ vpx_proto1(const struct worker *wrk, struct req *req)
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(req, REQ_MAGIC);
+	CHECK_OBJ_NOTNULL(req->sp, SESS_MAGIC);
 
-	VSL(SLT_Debug, req->sp->fd, "PROXY1");
+	VSL(SLT_Debug, req->sp->vxid, "PROXY1");
 
 	q = strchr(req->htc->rxbuf_b, '\r');
 	if (q == NULL)
@@ -80,7 +81,7 @@ vpx_proto1(const struct worker *wrk, struct req *req)
 	for (i = 0; i < 5; i++) {
 		p = strchr(p, ' ');
 		if (p == NULL) {
-			VSLb(req->vsl, SLT_ProxyGarbage,
+			VSL(SLT_ProxyGarbage, req->sp->vxid,
 			    "PROXY1: Too few fields");
 			return (-1);
 		}
@@ -89,7 +90,7 @@ vpx_proto1(const struct worker *wrk, struct req *req)
 	}
 
 	if (strchr(p, ' ')) {
-		VSLb(req->vsl, SLT_ProxyGarbage,
+		VSL(SLT_ProxyGarbage, req->sp->vxid,
 		    "PROXY1: Too many fields");
 		return (-1);
 	}
@@ -102,7 +103,7 @@ vpx_proto1(const struct worker *wrk, struct req *req)
 	else if (!strcmp(fld[0], "TCP6"))
 		pfam = AF_INET6;
 	else {
-		VSLb(req->vsl, SLT_ProxyGarbage,
+		VSL(SLT_ProxyGarbage, req->sp->vxid,
 		    "PROXY1: Wrong TCP[46] field");
 		return (-1);
 	}
@@ -113,14 +114,14 @@ vpx_proto1(const struct worker *wrk, struct req *req)
 
 	i = getaddrinfo(fld[1], fld[3], &hints, &res);
 	if (i != 0) {
-		VSLb(req->vsl, SLT_ProxyGarbage,
+		VSL(SLT_ProxyGarbage, req->sp->vxid,
 		    "PROXY1: Cannot resolve source address (%s)",
 		    gai_strerror(i));
 		return (-1);
 	}
 	AZ(res->ai_next);
 	if (res->ai_family != pfam) {
-		VSLb(req->vsl, SLT_ProxyGarbage,
+		VSL(SLT_ProxyGarbage, req->sp->vxid,
 		    "PROXY1: %s got wrong protocol (%d)",
 		    fld[0], res->ai_family);
 		freeaddrinfo(res);
@@ -134,14 +135,14 @@ vpx_proto1(const struct worker *wrk, struct req *req)
 
 	i = getaddrinfo(fld[2], fld[4], &hints, &res);
 	if (i != 0) {
-		VSLb(req->vsl, SLT_ProxyGarbage,
+		VSL(SLT_ProxyGarbage, req->sp->vxid,
 		    "PROXY1: Cannot resolve destination address (%s)",
 		    gai_strerror(i));
 		return (-1);
 	}
 	AZ(res->ai_next);
 	if (res->ai_family != pfam) {
-		VSLb(req->vsl, SLT_ProxyGarbage,
+		VSL(SLT_ProxyGarbage, req->sp->vxid,
 		    "PROXY1: %s got wrong protocol (%d)",
 		    fld[0], res->ai_family);
 		freeaddrinfo(res);
@@ -151,7 +152,7 @@ vpx_proto1(const struct worker *wrk, struct req *req)
 	AN(VSA_Build(sa, res->ai_addr, res->ai_addrlen));
 	freeaddrinfo(res);
 
-	VSLb(req->vsl, SLT_Proxy, "1 %s %s %s %s",
+	VSL(SLT_Proxy, req->sp->vxid, "1 %s %s %s %s",
 	    fld[1], fld[3], fld[2], fld[4]);
 	req->htc->pipeline_b = q;
 	return (0);
@@ -182,6 +183,7 @@ vpx_proto2(const struct worker *wrk, struct req *req)
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(req, REQ_MAGIC);
+	CHECK_OBJ_NOTNULL(req->sp, SESS_MAGIC);
 
 	assert(req->htc->rxbuf_e - req->htc->rxbuf_b >= 16L);
 	l = vbe16dec(req->htc->rxbuf_b + 14);
@@ -191,7 +193,7 @@ vpx_proto2(const struct worker *wrk, struct req *req)
 
 	/* Version @12 top half */
 	if ((p[12] >> 4) != 2) {
-		VSLb(req->vsl, SLT_ProxyGarbage,
+		VSL(SLT_ProxyGarbage, req->sp->vxid,
 		    "PROXY2: bad version (%d)", p[12] >> 4);
 		return (-1);
 	}
@@ -205,7 +207,7 @@ vpx_proto2(const struct worker *wrk, struct req *req)
 		/* Proxied connection */
 		break;
 	default:
-		VSLb(req->vsl, SLT_ProxyGarbage,
+		VSL(SLT_ProxyGarbage, req->sp->vxid,
 		    "PROXY2: bad command (%d)", p[12] & 0x0f);
 		return (-1);
 	}
@@ -214,14 +216,14 @@ vpx_proto2(const struct worker *wrk, struct req *req)
 	switch(p[13]) {
 	case 0x00:
 		/* UNSPEC|UNSPEC, ignore proxy header */
-		VSLb(req->vsl, SLT_ProxyGarbage,
+		VSL(SLT_ProxyGarbage, req->sp->vxid,
 		    "PROXY2: Ignoring UNSPEC|UNSPEC addresses");
 		return (0);
 	case 0x11:
 		/* IPv4|TCP */
 		pfam = AF_INET;
 		if (l < 12) {
-			VSLb(req->vsl, SLT_ProxyGarbage,
+			VSL(SLT_ProxyGarbage, req->sp->vxid,
 			    "PROXY2: Ignoring short IPv4 addresses (%d)", l);
 			return (0);
 		}
@@ -230,14 +232,14 @@ vpx_proto2(const struct worker *wrk, struct req *req)
 		/* IPv6|TCP */
 		pfam = AF_INET6;
 		if (l < 36) {
-			VSLb(req->vsl, SLT_ProxyGarbage,
+			VSL(SLT_ProxyGarbage, req->sp->vxid,
 			    "PROXY2: Ignoring short IPv6 addresses (%d)", l);
 			return (0);
 		}
 		break;
 	default:
 		/* Ignore proxy header */
-		VSLb(req->vsl, SLT_ProxyGarbage,
+		VSL(SLT_ProxyGarbage, req->sp->vxid,
 		    "PROXY2: Ignoring unsupported protocol (0x%02x)", p[13]);
 		return (0);
 	}
@@ -286,7 +288,7 @@ vpx_proto2(const struct worker *wrk, struct req *req)
 	SES_Set_String_Attr(req->sp, SA_CLIENT_IP, hb);
 	SES_Set_String_Attr(req->sp, SA_CLIENT_PORT, pb);
 
-	VSLb(req->vsl, SLT_Proxy, "2 %s %s %s %s", hb, pb, ha, pa);
+	VSL(SLT_Proxy, req->sp->vxid, "2 %s %s %s %s", hb, pb, ha, pa);
 	return (0);
 }
 
diff --git a/bin/varnishtest/tests/r01804.vtc b/bin/varnishtest/tests/r01804.vtc
new file mode 100644
index 0000000..c6705eb
--- /dev/null
+++ b/bin/varnishtest/tests/r01804.vtc
@@ -0,0 +1,39 @@
+varnishtest "#1804: varnishapi transaction grouping fails for PROXY"
+
+server s1 {
+	rxreq
+	txresp
+} -start
+
+
+varnish v1 -proto "PROXY" -vcl+backend {
+} -start
+
+logexpect l1 -v v1 -d 0 -g session {
+	expect * *	Begin	{^sess .* PROXY$}
+	expect * =	Proxy	{^1 }
+	expect * *	Begin	{^req}
+	expect * *	Begin	{^sess .* PROXY$}
+	expect * =	Proxy	{^2 }
+	expect * *	Begin	{^req}
+} -start
+
+client c1 {
+	send "PROXY TCP4 1.2.3.4 5.6.7.8 1234 5678\r\n"
+	txreq
+	rxresp
+} -run
+
+client c2 {
+	# good IPv4
+	sendhex "0d 0a 0d 0a 00 0d 0a 51 55 49 54 0a"
+	sendhex "21 11 00 0c"
+	sendhex "01 02 03 04"
+	sendhex "05 06 07 08"
+	sendhex "09 0a"
+	sendhex "0b 0c"
+	txreq
+	rxresp
+} -run
+
+logexpect l1 -wait



From martin at varnish-software.com  Tue Oct 27 14:37:29 2015
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Tue, 27 Oct 2015 15:37:29 +0100
Subject: [master] 43e192c Remove some debug log records in the proxy v1 code
Message-ID: <E1Zr5NR-0001M3-Iv@project.varnish-software.com>


commit 43e192cac762cae99b2a25bc4bb7604ca57fa3d7
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Tue Oct 27 15:37:03 2015 +0100

    Remove some debug log records in the proxy v1 code

diff --git a/bin/varnishd/proxy/cache_proxy_proto.c b/bin/varnishd/proxy/cache_proxy_proto.c
index 954beef..604d796 100644
--- a/bin/varnishd/proxy/cache_proxy_proto.c
+++ b/bin/varnishd/proxy/cache_proxy_proto.c
@@ -64,8 +64,6 @@ vpx_proto1(const struct worker *wrk, struct req *req)
 	CHECK_OBJ_NOTNULL(req, REQ_MAGIC);
 	CHECK_OBJ_NOTNULL(req->sp, SESS_MAGIC);
 
-	VSL(SLT_Debug, req->sp->vxid, "PROXY1");
-
 	q = strchr(req->htc->rxbuf_b, '\r');
 	if (q == NULL)
 		return (-1);
@@ -95,9 +93,6 @@ vpx_proto1(const struct worker *wrk, struct req *req)
 		return (-1);
 	}
 
-	VSL(SLT_Debug, req->sp->fd, "PROXY1 <%s> <%s> <%s> <%s> <%s>",
-	    fld[0], fld[1], fld[2], fld[3], fld[4]);
-
 	if (!strcmp(fld[0], "TCP4"))
 		pfam = AF_INET;
 	else if (!strcmp(fld[0], "TCP6"))



From phk at FreeBSD.org  Fri Oct 30 10:27:20 2015
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 30 Oct 2015 11:27:20 +0100
Subject: [master] e06d5dc Don't attempt to allocate a V1L from the workspace
 if it is overflowed.
Message-ID: <E1Zs6u0-0003bw-Bb@project.varnish-software.com>


commit e06d5dc0804ce7e81ccee7a5b95459c5564a196a
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Oct 30 10:26:29 2015 +0000

    Don't attempt to allocate a V1L from the workspace if it is overflowed.
    
    Fixes:	#1796
    Also triggered by:	The Wemm-Field

diff --git a/bin/varnishd/http1/cache_http1_deliver.c b/bin/varnishd/http1/cache_http1_deliver.c
index 997d28b..0135ed8 100644
--- a/bin/varnishd/http1/cache_http1_deliver.c
+++ b/bin/varnishd/http1/cache_http1_deliver.c
@@ -112,10 +112,12 @@ V1D_Deliver(struct req *req, struct busyobj *bo, int sendbody)
 	if (sendbody && req->resp_len != 0)
 		VDP_push(req, v1d_bytes, NULL, 1);
 
+	AZ(req->wrk->v1l);
 	V1L_Reserve(req->wrk, req->ws, &req->sp->fd, req->vsl, req->t_prev);
 
 	if (WS_Overflowed(req->ws)) {
 		v1d_error(req, "workspace_client overflow");
+		AZ(req->wrk->v1l);
 		return;
 	}
 
@@ -134,5 +136,6 @@ V1D_Deliver(struct req *req, struct busyobj *bo, int sendbody)
 
 	if ((V1L_FlushRelease(req->wrk) || ois != OIS_DONE) && req->sp->fd >= 0)
 		SES_Close(req->sp, SC_REM_CLOSE);
+	AZ(req->wrk->v1l);
 	VDP_close(req);
 }
diff --git a/bin/varnishd/http1/cache_http1_line.c b/bin/varnishd/http1/cache_http1_line.c
index 9c49fca..9f13543 100644
--- a/bin/varnishd/http1/cache_http1_line.c
+++ b/bin/varnishd/http1/cache_http1_line.c
@@ -78,6 +78,8 @@ V1L_Reserve(struct worker *wrk, struct ws *ws, int *fd, struct vsl_log *vsl,
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	AZ(wrk->v1l);
 
+	if (WS_Overflowed(ws))
+		return;
 	res = WS_Snapshot(ws);
 	v1l = WS_Alloc(ws, sizeof *v1l);
 	if (v1l == NULL)



From phk at FreeBSD.org  Fri Oct 30 14:26:19 2015
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 30 Oct 2015 15:26:19 +0100
Subject: [master] 165f191 So one of those strange cornercases in HTTP/1
Message-ID: <E1ZsAdH-0008Qf-13@project.varnish-software.com>


commit 165f191d68c2637629b6dd4303293c328745acae
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Oct 30 14:22:11 2015 +0000

    So one of those strange cornercases in HTTP/1
    
    If we send the backend a HTTP/1.0 request, and it doesn't have a
    Content-Length, it cannot use Chunked and must fall back to EOF.
    
    However, the protocol field in the response tells us what version
    backend *could* have used, not what it *does* use.
    
    So we can get a response with HTTP/1.1 and EOF, following HTTP/1.0
    semantics - because we asked for it.
    
    Most sensible backends avoid this, either by buffering and creation
    of a C-L or, smartly, returning "HTTP/1.0", even though that
    is strictly speaking against the apocrphal texts.
    
    Anyway, now we cope...
    
    Fixes: 	#1810

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 93e5de4..e7700d0 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -805,7 +805,8 @@ enum sess_close http_DoConnection(struct http *hp);
 
 htc_complete_f HTTP1_Complete;
 uint16_t HTTP1_DissectRequest(struct http_conn *, struct http *);
-uint16_t HTTP1_DissectResponse(struct http_conn *, struct http *);
+uint16_t HTTP1_DissectResponse(struct http_conn *, struct http *resp,
+    const struct http *req);
 unsigned HTTP1_Write(const struct worker *w, const struct http *hp, const int*);
 
 #define HTTPH(a, b, c) extern char b[];
diff --git a/bin/varnishd/http1/cache_http1_fetch.c b/bin/varnishd/http1/cache_http1_fetch.c
index 1b6ceb4..a69fb92 100644
--- a/bin/varnishd/http1/cache_http1_fetch.c
+++ b/bin/varnishd/http1/cache_http1_fetch.c
@@ -193,7 +193,7 @@ V1F_FetchRespHdr(struct busyobj *bo)
 
 	hp = bo->beresp;
 
-	i = HTTP1_DissectResponse(htc, hp);
+	i = HTTP1_DissectResponse(htc, hp, bo->bereq);
 	bo->acct.beresp_hdrbytes += htc->rxbuf_e - htc->rxbuf_b;
 	if (i) {
 		VSLb(bo->vsl, SLT_FetchError, "http format error");
diff --git a/bin/varnishd/http1/cache_http1_proto.c b/bin/varnishd/http1/cache_http1_proto.c
index 8d08278..deb191d 100644
--- a/bin/varnishd/http1/cache_http1_proto.c
+++ b/bin/varnishd/http1/cache_http1_proto.c
@@ -321,15 +321,15 @@ http1_body_status(const struct http *hp, struct http_conn *htc)
 
 /*--------------------------------------------------------------------*/
 
-static void
-http1_proto_ver(struct http *hp)
+static int8_t
+http1_proto_ver(const struct http *hp)
 {
 	if (!strcasecmp(hp->hd[HTTP_HDR_PROTO].b, "HTTP/1.0"))
-		hp->protover = 10;
+		return (10);
 	else if (!strcasecmp(hp->hd[HTTP_HDR_PROTO].b, "HTTP/1.1"))
-		hp->protover = 11;
+		return (11);
 	else
-		hp->protover = 0;
+		return (0);
 }
 
 /*--------------------------------------------------------------------*/
@@ -347,7 +347,7 @@ HTTP1_DissectRequest(struct http_conn *htc, struct http *hp)
 	retval = http1_splitline(hp, htc, HTTP1_Req);
 	if (retval != 0)
 		return (retval);
-	http1_proto_ver(hp);
+	hp->protover = http1_proto_ver(hp);
 	if (hp->protover == 0)
 		return (400);
 
@@ -391,22 +391,28 @@ HTTP1_DissectRequest(struct http_conn *htc, struct http *hp)
 /*--------------------------------------------------------------------*/
 
 uint16_t
-HTTP1_DissectResponse(struct http_conn *htc, struct http *hp)
+HTTP1_DissectResponse(struct http_conn *htc, struct http *hp,
+    const struct http *req)
 {
 	uint16_t retval = 0;
 	const char *p;
+	int8_t rv;
 
 
 	CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC);
 	CHECK_OBJ_NOTNULL(hp, HTTP_MAGIC);
+	CHECK_OBJ_NOTNULL(req, HTTP_MAGIC);
 
 	if (http1_splitline(hp, htc, HTTP1_Resp))
 		retval = 503;
 
 	if (retval == 0) {
-		http1_proto_ver(hp);
+		hp->protover = http1_proto_ver(hp);
 		if (hp->protover == 0)
 			retval = 503;
+		rv = http1_proto_ver(req);
+		if (hp->protover > rv)
+			hp->protover = rv;
 	}
 
 	if (retval == 0 && Tlen(hp->hd[HTTP_HDR_STATUS]) != 3)
diff --git a/bin/varnishtest/tests/r01810.vtc b/bin/varnishtest/tests/r01810.vtc
new file mode 100644
index 0000000..77e84db
--- /dev/null
+++ b/bin/varnishtest/tests/r01810.vtc
@@ -0,0 +1,21 @@
+varnishtest "POST HTTP/1.0 response"
+
+server s1 {
+	non-fatal
+	rxreq
+	txresp -proto HTTP/1.1 -nolen -hdr "Connection: close"
+	send "Hello World\n"
+	delay .4
+} -start
+
+varnish v1 -vcl+backend { 
+	sub vcl_backend_fetch {
+		set bereq.proto = "HTTP/1.0";
+	}
+} -start
+
+client c1 {
+	txreq -req POST -hdr "Content-Length: 0"
+	rxresp
+	expect resp.bodylen == 12
+} -run



From fgsch at lodoss.net  Fri Oct 30 17:54:20 2015
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Fri, 30 Oct 2015 18:54:20 +0100
Subject: [master] 00bcc3e Convert arguments to strings as well
Message-ID: <E1ZsDsa-0003Sr-VO@project.varnish-software.com>


commit 00bcc3eacbf0beb56667c2f82723e89abfac3d00
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Thu Oct 1 11:06:07 2015 +0100

    Convert arguments to strings as well
    
    This is consistent with what we do in other cases.

diff --git a/bin/varnishtest/tests/d00003.vtc b/bin/varnishtest/tests/d00003.vtc
index 0c95d61..56b3035 100644
--- a/bin/varnishtest/tests/d00003.vtc
+++ b/bin/varnishtest/tests/d00003.vtc
@@ -5,6 +5,8 @@ server s1 {
 	txresp -hdr "Foo: 1" -body "1"
 	rxreq
 	txresp -hdr "Foo: 3" -body "3"
+	rxreq
+	txresp -hdr "Foo: 9" -body "9"
 } -start
 
 server s2 {
@@ -28,19 +30,18 @@ varnish v1 -vcl+backend {
 	}
 
 	sub vcl_recv {
-		return(pass);
-	}
-
-	sub vcl_backend_fetch {
-		if (bereq.url == "/nohdr") {
-			set bereq.backend = h1.backend(bereq.http.Void);
-		} else if (bereq.url == "/emptystring") {
-			set bereq.backend = h1.backend("");
-		} else if (bereq.url == "/13") {
-			set bereq.backend = h1.backend(bereq.http.Void + "" + bereq.url);
+		if (req.url == "/nohdr") {
+			set req.backend_hint = h1.backend(req.http.Void);
+		} else if (req.url == "/emptystring") {
+			set req.backend_hint = h1.backend("");
+		} else if (req.url == "/13") {
+			set req.backend_hint = h1.backend(req.http.Void + "" + req.url);
+		} else if (req.url == "/ip") {
+			set req.backend_hint = h1.backend(client.ip);
 		} else {
-			set bereq.backend = h1.backend(bereq.url);
+			set req.backend_hint = h1.backend(req.url);
 		}
+		return (pass);
 	}
 } -start
 
@@ -68,4 +69,8 @@ client c1 {
 	txreq -url /nohdr
 	rxresp
 	expect resp.http.foo == "8"
+
+	txreq -url /ip
+	rxresp
+	expect resp.http.foo == "9"
 } -run
diff --git a/lib/libvcc/vcc_expr.c b/lib/libvcc/vcc_expr.c
index 294d12a..b498a53 100644
--- a/lib/libvcc/vcc_expr.c
+++ b/lib/libvcc/vcc_expr.c
@@ -626,6 +626,9 @@ vcc_do_arg(struct vcc *tl, struct func_arg *fa)
 	} else {
 		vcc_expr0(tl, &e2, fa->type);
 		ERRCHK(tl);
+		if (e2->fmt != fa->type &&
+		    (fa->type == STRING || fa->type == STRING_LIST))
+			vcc_expr_tostring(tl, &e2, fa->type);
 		if (e2->fmt != fa->type) {
 			VSB_printf(tl->sb, "Wrong argument type.");
 			VSB_printf(tl->sb, "  Expected %s.",