From noreply at github.com  Thu Jun  1 20:03:06 2017
From: noreply at github.com (GitHub)
Date: Thu, 01 Jun 2017 22:03:06 +0200
Subject: [master] 54c59ac If pipe'ing, unset any existing Connection headers
Message-ID: <E1dGWJG-0007tc-JX@project.varnish-software.com>


commit 54c59ac87dc64ca502e50b6f15fcb4a88baa5e8e
Author: Federico G. Schwindt <fgsch at users.noreply.github.com>
Date:   Thu Jun 1 21:02:05 2017 +0100

    If pipe'ing, unset any existing Connection headers
    
    Otherwise the backend might see multiple Connection entries.
    Fixes #2337.

diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c
index cdd443f..1bbc552 100644
--- a/bin/varnishd/cache/cache_req_fsm.c
+++ b/bin/varnishd/cache/cache_req_fsm.c
@@ -676,7 +676,7 @@ cnt_pipe(struct worker *wrk, struct req *req)
 	HTTP_Setup(bo->bereq, bo->ws, bo->vsl, SLT_BereqMethod);
 	http_FilterReq(bo->bereq, req->http, 0);	// XXX: 0 ?
 	http_PrintfHeader(bo->bereq, "X-Varnish: %u", VXID(req->vsl->wid));
-	http_SetHeader(bo->bereq, "Connection: close");
+	http_ForceHeader(bo->bereq, H_Connection, "close");
 
 	if (req->want100cont) {
 		http_SetHeader(bo->bereq, "Expect: 100-continue");
diff --git a/bin/varnishtest/tests/b00001.vtc b/bin/varnishtest/tests/b00001.vtc
index d48b9a0..b45ddb5 100644
--- a/bin/varnishtest/tests/b00001.vtc
+++ b/bin/varnishtest/tests/b00001.vtc
@@ -2,17 +2,30 @@ varnishtest "Check that a pipe transaction works"
 
 server s1 -repeat 1 {
 	rxreq
-	txresp -body "012345\n"
+	expect req.http.connection == "close"
+	txresp
+	rxreq
+	expect req.http.connection == "keep-alive"
+	txresp
 } -start
 
 varnish v1 -vcl+backend {
 	sub vcl_recv {
 		return(pipe);
 	}
+
+	sub vcl_pipe {
+		if (req.url == "/2") {
+			set bereq.http.connection = req.http.connection;
+		}
+	}
 } -start
 
 client c1 {
-	txreq -url "/"
+	txreq -url /1 -hdr "Connection: keep-alive"
+	rxresp
+	expect resp.status == 200
+	txreq -url /2 -hdr "Connection: keep-alive"
 	rxresp
 	expect resp.status == 200
 } -run



From hermunn at varnish-software.com  Tue Jun  6 08:29:06 2017
From: hermunn at varnish-software.com (Pål Hermunn Johansen)
Date: Tue, 06 Jun 2017 10:29:06 +0200
Subject: [4.1] 768a853 Only root can jail
Message-ID: <E1dI9rO-0003q9-96@project.varnish-software.com>


commit 768a8536c04f25e3097763a6f5ca310c8d55a78b
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Mon May 15 17:46:46 2017 +0200

    Only root can jail
    
    Fixes #2329

diff --git a/doc/sphinx/reference/varnishd.rst b/doc/sphinx/reference/varnishd.rst
index 4da80a7..4506a63 100644
--- a/doc/sphinx/reference/varnishd.rst
+++ b/doc/sphinx/reference/varnishd.rst
@@ -233,8 +233,8 @@ specific options. Available jails are:
 
 -j <unix[,user=`user`][,ccgroup=`group`][,workuser=`user`]>
 
-  Default on all other platforms when `varnishd` is either started with
-  an effective uid of 0 ("as root") or as user ``varnish``.
+  Default on all other platforms when `varnishd` is started with an
+  effective uid of 0 ("as root").
 
   With the ``unix`` jail mechanism activated, varnish will switch to
   an alternative user for subprocesses and change the effective uid of



From phk at FreeBSD.org  Tue Jun  6 08:55:06 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 06 Jun 2017 10:55:06 +0200
Subject: [master] 8c1c32f The -i argument should not control our process name
 in syslog.
Message-ID: <E1dIAGY-0004ug-Mg@project.varnish-software.com>


commit 8c1c32f94ada259afe1e8ade7eb5fad37873af75
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Jun 6 08:54:20 2017 +0000

    The -i argument should not control our process name in syslog.
    
    But it should be available via VSM.

diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c
index 85113f8..a9ab422 100644
--- a/bin/varnishd/mgt/mgt_main.c
+++ b/bin/varnishd/mgt/mgt_main.c
@@ -290,37 +290,6 @@ init_params(struct cli *cli)
 	MCF_InitParams(cli);
 }
 
-
-/*--------------------------------------------------------------------*/
-
-static void
-identify(const char *i_arg)
-{
-	char id[17], *p;
-	int i;
-
-	strcpy(id, "varnishd");
-
-	if (i_arg != NULL) {
-		if (strlen(i_arg) + 1 > 1024)
-			ARGV_ERR("Identity (-i) name too long (max 1023).\n");
-		heritage.identity = strdup(i_arg);
-		AN(heritage.identity);
-		i = strlen(id);
-		id[i++] = '/';
-		for (; i < (sizeof(id) - 1L); i++) {
-			if (!isalnum(*i_arg))
-				break;
-			id[i] = *i_arg++;
-		}
-		id[i] = '\0';
-	}
-	p = strdup(id);
-	AN(p);
-
-	openlog(p, LOG_PID, LOG_LOCAL0);
-}
-
 static void
 mgt_tests(void)
 {
@@ -781,7 +750,11 @@ main(int argc, char * const *argv)
 	setproctitle("Varnish-Mgr %s", heritage.name);
 #endif
 
-	identify(i_arg);
+	if (i_arg == NULL || *i_arg == '\0')
+		i_arg = "varnishd";
+	heritage.identity = i_arg;
+
+	openlog("varnishd", LOG_PID, LOG_LOCAL0);
 
 	if (VJ_make_workdir(dirname))
 		ARGV_ERR("Cannot create working directory (%s): %s\n",
@@ -861,6 +834,8 @@ main(int argc, char * const *argv)
 
 	mgt_SHM_Init();
 
+	mgt_SHM_static_alloc(i_arg, strlen(i_arg) + 1L, "Arg", "-i", "");
+
 	if (M_arg != NULL)
 		mgt_cli_master(M_arg);
 	if (T_arg != NULL)



From hermunn at varnish-software.com  Tue Jun  6 09:08:06 2017
From: hermunn at varnish-software.com (Pål Hermunn Johansen)
Date: Tue, 06 Jun 2017 11:08:06 +0200
Subject: [4.1] 8b86c61 Corrected grammar for install.rst
Message-ID: <E1dIAT8-0005Tw-1S@project.varnish-software.com>


commit 8b86c61751a18f26cf2ab927d85845ae4af37eb6
Author: Akshay Ranganath <rakshay2 at gmail.com>
Date:   Fri May 26 13:33:37 2017 -0700

    Corrected grammar for install.rst
    
    In the section related to make test, the sentences were a bit hard incorrect. I've corrected it.

diff --git a/doc/sphinx/installation/install.rst b/doc/sphinx/installation/install.rst
index f7e6492..eb876dc 100644
--- a/doc/sphinx/installation/install.rst
+++ b/doc/sphinx/installation/install.rst
@@ -143,11 +143,11 @@ tea while it runs, it usually takes a couple of minutes::
 
 	make check
 
-Don't worry if one or two tests fail, some of the tests are a
-bit too timing sensitive (Please tell us which so we can fix them.) but
-if a lot of them fails, and in particular if the `b00000.vtc` test
-fails, something is horribly wrong, and you will get nowhere without
-figuring out what.
+Don't worry if one or two tests fail. Some of the tests are a
+bit too timing sensitive (Please tell us which so we can fix them). 
+However, if a lot of them fail, and in particular if the `b00000.vtc` test
+fails, something is horribly wrong. You will get nowhere without
+figuring this one out.
 
 Installing
 ----------



From hermunn at varnish-software.com  Tue Jun  6 09:36:06 2017
From: hermunn at varnish-software.com (Pål Hermunn Johansen)
Date: Tue, 06 Jun 2017 11:36:06 +0200
Subject: [4.1] 6f43c0e If pipe'ing, unset any existing Connection headers
Message-ID: <E1dIAuE-0006bk-BH@project.varnish-software.com>


commit 6f43c0e9974ee61ff17a3a6bcc8487d88041a284
Author: Federico G. Schwindt <fgsch at users.noreply.github.com>
Date:   Thu Jun 1 21:02:05 2017 +0100

    If pipe'ing, unset any existing Connection headers
    
    Otherwise the backend might see multiple Connection entries.
    Fixes #2337.

diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c
index f114284..ca6e316 100644
--- a/bin/varnishd/cache/cache_req_fsm.c
+++ b/bin/varnishd/cache/cache_req_fsm.c
@@ -584,7 +584,7 @@ cnt_pipe(struct worker *wrk, struct req *req)
 	HTTP_Setup(bo->bereq, bo->ws, bo->vsl, SLT_BereqMethod);
 	http_FilterReq(bo->bereq, req->http, 0);	// XXX: 0 ?
 	http_PrintfHeader(bo->bereq, "X-Varnish: %u", VXID(req->vsl->wid));
-	http_SetHeader(bo->bereq, "Connection: close");
+	http_ForceHeader(bo->bereq, H_Connection, "close");
 
 	VCL_pipe_method(req->vcl, wrk, req, bo, NULL);
 
diff --git a/bin/varnishtest/tests/b00001.vtc b/bin/varnishtest/tests/b00001.vtc
index d48b9a0..b45ddb5 100644
--- a/bin/varnishtest/tests/b00001.vtc
+++ b/bin/varnishtest/tests/b00001.vtc
@@ -2,17 +2,30 @@ varnishtest "Check that a pipe transaction works"
 
 server s1 -repeat 1 {
 	rxreq
-	txresp -body "012345\n"
+	expect req.http.connection == "close"
+	txresp
+	rxreq
+	expect req.http.connection == "keep-alive"
+	txresp
 } -start
 
 varnish v1 -vcl+backend {
 	sub vcl_recv {
 		return(pipe);
 	}
+
+	sub vcl_pipe {
+		if (req.url == "/2") {
+			set bereq.http.connection = req.http.connection;
+		}
+	}
 } -start
 
 client c1 {
-	txreq -url "/"
+	txreq -url /1 -hdr "Connection: keep-alive"
+	rxresp
+	expect resp.status == 200
+	txreq -url /2 -hdr "Connection: keep-alive"
 	rxresp
 	expect resp.status == 200
 } -run



From phk at FreeBSD.org  Tue Jun  6 10:07:05 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 06 Jun 2017 12:07:05 +0200
Subject: [master] 54bb1bb Append -i argument to setproctitle() if given.
Message-ID: <E1dIBOD-0007tN-Pz@project.varnish-software.com>


commit 54bb1bbd3ecaa790199f0a78af33886e063ba535
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Jun 6 10:04:42 2017 +0000

    Append -i argument to setproctitle() if given.
    
    Return "varnishd" for server.identity if no -i argument is given.
    
    Wrap setproctitle() in mgt_util.c

diff --git a/bin/varnishd/cache/cache_vrt_var.c b/bin/varnishd/cache/cache_vrt_var.c
index 5c7a7b1..ec85e64 100644
--- a/bin/varnishd/cache/cache_vrt_var.c
+++ b/bin/varnishd/cache/cache_vrt_var.c
@@ -679,7 +679,7 @@ VRT_r_server_identity(VRT_CTX)
 	if (heritage.identity != NULL)
 		return (heritage.identity);
 	else
-		return (heritage.name);
+		return ("varnishd");
 }
 
 const char*
diff --git a/bin/varnishd/common/heritage.h b/bin/varnishd/common/heritage.h
index 055b240..3955690 100644
--- a/bin/varnishd/common/heritage.h
+++ b/bin/varnishd/common/heritage.h
@@ -75,7 +75,6 @@ struct heritage {
 
 	struct params			*param;
 
-	char				*name;
 	const char			*identity;
 
 	char				*panic_str;
diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h
index 5d81642..8cb2490 100644
--- a/bin/varnishd/mgt/mgt.h
+++ b/bin/varnishd/mgt/mgt.h
@@ -174,6 +174,7 @@ void mgt_SHM_Size_Adjust(void);
 void MCF_TcpParams(void);
 
 /* mgt_util.c */
+void mgt_ProcTitle(const char *comp);
 void mgt_DumpRstVsl(void);
 struct vsb *mgt_BuildVident(void);
 void MGT_Complain(const char *, const char *, ...) __v_printflike(2, 3);
diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c
index ed398b7..183cc5b 100644
--- a/bin/varnishd/mgt/mgt_child.c
+++ b/bin/varnishd/mgt/mgt_child.c
@@ -347,10 +347,7 @@ mgt_launch_child(struct cli *cli)
 			if (close(i) == 0)
 				VFIL_null_fd(i);
 		}
-#ifdef HAVE_SETPROCTITLE
-		setproctitle("Varnish-Chld %s", heritage.name);
-#endif
-
+		mgt_ProcTitle("Child");
 		if (mgt_param.sigsegv_handler) {
 			memset(&sa, 0, sizeof sa);
 			sa.sa_sigaction = child_sigsegv_handler;
diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c
index a9ab422..3918608 100644
--- a/bin/varnishd/mgt/mgt_main.c
+++ b/bin/varnishd/mgt/mgt_main.c
@@ -743,17 +743,16 @@ main(int argc, char * const *argv)
 		VJ_master(JAIL_MASTER_LOW);
 	}
 
-	if (VIN_n_Arg(n_arg, &heritage.name, &dirname) != 0)
+	if (VIN_n_Arg(n_arg, &p, &dirname) != 0)
 		ARGV_ERR("Invalid instance (-n) name: %s\n", strerror(errno));
-
-#ifdef HAVE_SETPROCTITLE
-	setproctitle("Varnish-Mgr %s", heritage.name);
-#endif
+	free(p);
 
 	if (i_arg == NULL || *i_arg == '\0')
 		i_arg = "varnishd";
 	heritage.identity = i_arg;
 
+	mgt_ProcTitle("Mgr");
+
 	openlog("varnishd", LOG_PID, LOG_LOCAL0);
 
 	if (VJ_make_workdir(dirname))
diff --git a/bin/varnishd/mgt/mgt_util.c b/bin/varnishd/mgt/mgt_util.c
index 9e63ef7..5730ae5 100644
--- a/bin/varnishd/mgt/mgt_util.c
+++ b/bin/varnishd/mgt/mgt_util.c
@@ -37,9 +37,26 @@
 #include <stdlib.h>
 #include <string.h>
 #include <syslog.h>
+#include <unistd.h>
 
 #include "mgt/mgt.h"
 
+#include "common/heritage.h"
+
+/*--------------------------------------------------------------------*/
+
+void
+mgt_ProcTitle(const char *comp)
+{
+#ifdef HAVE_SETPROCTITLE
+	if (strcmp(heritage.identity, "varnishd"))
+		setproctitle("Varnish-%s -i %s", comp, heritage.identity);
+	else
+		setproctitle("Varnish-%s", comp);
+#else
+	(void)comp;
+#endif
+}
 
 /*--------------------------------------------------------------------*/
 



From phk at FreeBSD.org  Tue Jun  6 10:38:06 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 06 Jun 2017 12:38:06 +0200
Subject: [master] 3c00c88 Get rid of VUT.name and ask VSM_Name() directly for
 the name when we need it.
Message-ID: <E1dIBsE-0000nm-Db@project.varnish-software.com>


commit 3c00c88b652b6adc477e5fa9d3e77fc6bd4c2c48
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Jun 6 10:20:12 2017 +0000

    Get rid of VUT.name and ask VSM_Name() directly for the name when we need it.

diff --git a/bin/varnishhist/varnishhist.c b/bin/varnishhist/varnishhist.c
index d82b1ce..bc38a2c 100644
--- a/bin/varnishhist/varnishhist.c
+++ b/bin/varnishhist/varnishhist.c
@@ -45,6 +45,7 @@
 #include "vdef.h"
 #include "vcurses.h"
 #include "vapi/vsl.h"
+#include "vapi/vsm.h"
 #include "vapi/voptget.h"
 #include "vas.h"
 #include "vut.h"
@@ -152,7 +153,7 @@ update(void)
 	if (end_of_file)
 		mvprintw(0, 0, "%*s", COLS - 1, "EOF");
 	else
-		mvprintw(0, 0, "%*s", COLS - 1, VUT.name);
+		mvprintw(0, 0, "%*s", COLS - 1, VSM_Name(VUT.vsm));
 
 	/* count our flock */
 	for (i = 0; i < n; ++i)
diff --git a/bin/varnishtop/varnishtop.c b/bin/varnishtop/varnishtop.c
index 1911dd3..ca4df6d 100644
--- a/bin/varnishtop/varnishtop.c
+++ b/bin/varnishtop/varnishtop.c
@@ -44,6 +44,7 @@
 
 #include "vcurses.h"
 #include "vapi/vsl.h"
+#include "vapi/vsm.h"
 #include "vapi/voptget.h"
 #include "vas.h"
 #include "vdef.h"
@@ -188,6 +189,7 @@ update(int p)
 	double t = 0;
 	static time_t last = 0;
 	static unsigned n;
+	const char *q;
 	time_t now;
 
 	now = time(NULL);
@@ -199,11 +201,12 @@ update(int p)
 	if (n < p)
 		n++;
 	AC(erase());
+	q = VSM_Name(VUT.vsm);
+	len = COLS - strlen(q);
 	if (end_of_file)
-		AC(mvprintw(0, (COLS - strlen(VUT.name)) - (1 + 6), "%s (EOF)",
-			VUT.name));
+		AC(mvprintw(0, len - (1 + 6), "%s (EOF)", q));
 	else
-		AC(mvprintw(0, (COLS - strlen(VUT.name)) - 1, "%s", VUT.name));
+		AC(mvprintw(0, len - 1, "%s", q));
 	AC(mvprintw(0, 0, "list length %u", ntop));
 	for (tp = VRB_MIN(t_order, &h_order); tp != NULL; tp = tp2) {
 		tp2 = VRB_NEXT(t_order, &h_order, tp);
diff --git a/include/vut.h b/include/vut.h
index 1e1942c..fb9b873 100644
--- a/include/vut.h
+++ b/include/vut.h
@@ -35,7 +35,6 @@ typedef int VUT_cb_f(void);
 
 struct VUT {
 	const char	*progname;
-	char		*name;
 
 	/* Options */
 	int		d_opt;
diff --git a/lib/libvarnishapi/vut.c b/lib/libvarnishapi/vut.c
index 83a08da..f9faa10 100644
--- a/lib/libvarnishapi/vut.c
+++ b/lib/libvarnishapi/vut.c
@@ -215,7 +215,6 @@ VUT_Init(const char *progname, int argc, char * const *argv,
 		exit(vut_options(voc));
 
 	VUT.progname = progname;
-	REPLACE(VUT.name, "");
 	VUT.g_arg = VSL_g_vxid;
 	AZ(VUT.vsl);
 	VUT.vsl = VSL_New();
@@ -248,7 +247,6 @@ VUT_Setup(void)
 
 	/* Setup input */
 	if (VUT.r_arg) {
-		REPLACE(VUT.name, VUT.r_arg);
 		c = VSL_CursorFile(VUT.vsl, VUT.r_arg, 0);
 		if (c == NULL)
 			VUT_Error(1, "%s", VSL_Error(VUT.vsl));
@@ -257,7 +255,6 @@ VUT_Setup(void)
 		AN(VUT.vsm);
 		if (VUT.n_arg && VSM_n_Arg(VUT.vsm, VUT.n_arg) <= 0)
 			VUT_Error(1, "%s", VSM_Error(VUT.vsm));
-		REPLACE(VUT.name, VSM_Name(VUT.vsm));
 		t_start = NAN;
 		c = NULL;
 		while (1) {
@@ -330,7 +327,6 @@ VUT_Fini(void)
 	free(VUT.n_arg);
 	free(VUT.r_arg);
 	free(VUT.P_arg);
-	free(VUT.name);
 
 	vut_vpf_remove();
 	AZ(VUT.pfh);



From phk at FreeBSD.org  Tue Jun  6 10:38:06 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 06 Jun 2017 12:38:06 +0200
Subject: [master] 0f32c87 Make VSM_Name() return the -i argument to varnishd.
Message-ID: <E1dIBsE-0000ns-Fi@project.varnish-software.com>


commit 0f32c873b2b77ab910072976925631ab7040305f
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Jun 6 10:36:44 2017 +0000

    Make VSM_Name() return the -i argument to varnishd.

diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c
index 3918608..df1fbda 100644
--- a/bin/varnishd/mgt/mgt_main.c
+++ b/bin/varnishd/mgt/mgt_main.c
@@ -743,9 +743,8 @@ main(int argc, char * const *argv)
 		VJ_master(JAIL_MASTER_LOW);
 	}
 
-	if (VIN_n_Arg(n_arg, &p, &dirname) != 0)
+	if (VIN_n_Arg(n_arg, &dirname) != 0)
 		ARGV_ERR("Invalid instance (-n) name: %s\n", strerror(errno));
-	free(p);
 
 	if (i_arg == NULL || *i_arg == '\0')
 		i_arg = "varnishd";
diff --git a/include/vapi/vsm.h b/include/vapi/vsm.h
index 52d8167..0739c96 100644
--- a/include/vapi/vsm.h
+++ b/include/vapi/vsm.h
@@ -100,7 +100,7 @@ int VSM_n_Arg(struct VSM_data *vd, const char *n_arg);
 
 const char *VSM_Name(const struct VSM_data *vd);
 	/*
-	 * Return the instance name.
+	 * Return the instance name (-i argument to varnishd)
 	 */
 
 int VSM_Open(struct VSM_data *vd);
diff --git a/include/vin.h b/include/vin.h
index e0fe073..c594980 100644
--- a/include/vin.h
+++ b/include/vin.h
@@ -32,5 +32,5 @@
 #define VIN_H_INCLUDED
 
 /* This function lives in both libvarnish and libvarnishapi */
-int VIN_n_Arg(const char *n_arg, char **name, char **dir);
+int VIN_n_Arg(const char *n_arg, char **dir);
 #endif
diff --git a/lib/libvarnish/vin.c b/lib/libvarnish/vin.c
index fecc554..78ab952 100644
--- a/lib/libvarnish/vin.c
+++ b/lib/libvarnish/vin.c
@@ -44,7 +44,7 @@
 #include "vin.h"
 
 int
-VIN_n_Arg(const char *n_arg, char **name, char **dir)
+VIN_n_Arg(const char *n_arg, char **dir)
 {
 	char nm[PATH_MAX];
 	char dn[PATH_MAX];
@@ -83,18 +83,10 @@ VIN_n_Arg(const char *n_arg, char **name, char **dir)
 
 	strcat(dn, "/");
 
-	if (name != NULL) {
-		*name = strdup(nm);
-		if (*name == NULL)
-			return (-1);
-	}
 	if (dir != NULL) {
 		*dir = strdup(dn);
-		if (*dir == NULL) {
-			if (name != NULL)
-				free(*name);
+		if (*dir == NULL) 
 			return (-1);
-		}
 	}
 	return (0);
 }
diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c
index 83bab97..35551d4 100644
--- a/lib/libvarnishapi/vsm.c
+++ b/lib/libvarnishapi/vsm.c
@@ -70,7 +70,6 @@ VSM_New(void)
 	if (vd == NULL)
 		return (vd);
 
-	REPLACE(vd->name, "");
 	vd->vsm_fd = -1;
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
@@ -129,7 +128,6 @@ VSM_ResetError(struct VSM_data *vd)
 int
 VSM_n_Arg(struct VSM_data *vd, const char *arg)
 {
-	char *name = NULL;
 	char *dname = NULL;
 	struct vsb *vsb;
 
@@ -137,22 +135,19 @@ VSM_n_Arg(struct VSM_data *vd, const char *arg)
 
 	if (vd->head)
 		return (vsm_diag(vd, "VSM_n_Arg: Already open"));
-	if (VIN_n_Arg(arg, &name, &dname))
+	if (VIN_n_Arg(arg, &dname))
 		return (vsm_diag(vd, "Invalid instance name: %s",
 		    strerror(errno)));
-	AN(name);
 	AN(dname);
 	vsb = VSB_new_auto();
 	AN(vsb);
 	VSB_printf(vsb, "%s%s", dname, VSM_FILENAME);
 	AZ(VSB_finish(vsb));
 
-	REPLACE(vd->name, name);
 	REPLACE(vd->dname, dname);
 	REPLACE(vd->iname, VSB_data(vsb));
 
 	VSB_destroy(&vsb);
-	free(name);
 	free(dname);
 
 	return (1);
@@ -163,10 +158,12 @@ VSM_n_Arg(struct VSM_data *vd, const char *arg)
 const char *
 VSM_Name(const struct VSM_data *vd)
 {
+	struct VSM_fantom vt;
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-
-	return (vd->name);
+	if (VSM_Get(vd, &vt, "Arg", "-i", ""))
+		return (vt.b);
+	return ("");
 }
 
 /*--------------------------------------------------------------------*/
@@ -181,7 +178,6 @@ VSM_Delete(struct VSM_data *vd)
 	if (vd->vsc != NULL)
 		VSC_Delete(vd);
 	VSM_ResetError(vd);
-	free(vd->name);
 	free(vd->dname);
 	FREE_OBJ(vd);
 }
diff --git a/lib/libvarnishapi/vsm_api.h b/lib/libvarnishapi/vsm_api.h
index e357a8f..e9402b0 100644
--- a/lib/libvarnishapi/vsm_api.h
+++ b/lib/libvarnishapi/vsm_api.h
@@ -37,7 +37,6 @@ struct VSM_data {
 
 	struct vsb		*diag;
 
-	char			*name;
 	char			*dname;
 	char			*iname;
 



From phk at FreeBSD.org  Tue Jun  6 12:37:05 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 06 Jun 2017 14:37:05 +0200
Subject: [master] 4327745 Start documenting API breakage for next release.
Message-ID: <E1dIDjN-0005g9-Cd@project.varnish-software.com>


commit 43277451f61c6c8fb362e9c0c721953325c47afb
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Jun 6 11:33:35 2017 +0000

    Start documenting API breakage for next release.

diff --git a/doc/changes.rst b/doc/changes.rst
index 41d83d0..478202e 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -1,3 +1,13 @@
+=============================
+Varnish Cache Trunk (ongoing)
+=============================
+
+* On platforms with setproctitle(3), use the -i argument rather than -n
+
+* VSM_Name() returns the -i argument name, rather than the -n name.
+
+* VUT.name is goine, use VSM_Name(VUT.vsm)
+
 ================================
 Varnish Cache 5.1.2 (2017-04-07)
 ================================



From phk at FreeBSD.org  Tue Jun  6 12:37:05 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 06 Jun 2017 14:37:05 +0200
Subject: [master] 52c58d8 Update description of -n and -i arguments and where
 they end up.
Message-ID: <E1dIDjN-0005gI-E9@project.varnish-software.com>


commit 52c58d8643e3aee5fa4aa15e06bcddae1f63077b
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Jun 6 12:35:47 2017 +0000

    Update description of -n and -i arguments and where they end up.

diff --git a/doc/sphinx/reference/varnishd.rst b/doc/sphinx/reference/varnishd.rst
index 56a90c4..0064580 100644
--- a/doc/sphinx/reference/varnishd.rst
+++ b/doc/sphinx/reference/varnishd.rst
@@ -72,11 +72,10 @@ Basic options
 
 -n name
 
-  Specify the name for this instance.  Amongst other things, this name
-  is used to construct the name of the directory in which `varnishd`
-  keeps temporary files and persistent state. If the specified name
-  begins with a forward slash, it is interpreted as the absolute path
-  to the directory which should be used for this purpose.
+  Specify the name for this instance.  This name is used to construct
+  the name of the directory in which `varnishd` keeps temporary files
+  and persistent state. If the specified name begins with a forward slash,
+  it is interpreted as the absolute path to the directory.
 
 Documentation options
 ---------------------
@@ -135,7 +134,8 @@ Operations options
 -i identity
 
   Specify the identity of the Varnish server. This can be accessed
-  using ``server.identity`` from VCL.
+  using ``server.identity`` from VCL and with VSM_Name() from
+  utilities.
 
 -I clifile
 
diff --git a/lib/libvcc/generate.py b/lib/libvcc/generate.py
index fac8ee4..1a3d5da 100755
--- a/lib/libvcc/generate.py
+++ b/lib/libvcc/generate.py
@@ -216,8 +216,7 @@ sp_variables = [
 		(), """
 		The identity of the server, as set by the -i
 		parameter.  If the -i parameter is not passed to varnishd,
-		server.identity will be set to the name of the instance, as
-		specified by the -n parameter.
+		server.identity will be set to "varnishd".
 		"""
 	),
 	('req',



From phk at FreeBSD.org  Tue Jun  6 14:54:06 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 06 Jun 2017 16:54:06 +0200
Subject: [master] 941087b On futher reflection: gethostname(3) is a better -i
 default than "varnishd"
Message-ID: <E1dIFry-0002p8-LP@project.varnish-software.com>


commit 941087b494da96afdd80c3865950fb5b51f0a304
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Jun 6 14:52:57 2017 +0000

    On futher reflection:  gethostname(3) is a better -i default than "varnishd"

diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h
index 8cb2490..07b8c61 100644
--- a/bin/varnishd/mgt/mgt.h
+++ b/bin/varnishd/mgt/mgt.h
@@ -174,6 +174,7 @@ void mgt_SHM_Size_Adjust(void);
 void MCF_TcpParams(void);
 
 /* mgt_util.c */
+char *mgt_HostName(void);
 void mgt_ProcTitle(const char *comp);
 void mgt_DumpRstVsl(void);
 struct vsb *mgt_BuildVident(void);
diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c
index df1fbda..4c78e39 100644
--- a/bin/varnishd/mgt/mgt_main.c
+++ b/bin/varnishd/mgt/mgt_main.c
@@ -747,7 +747,7 @@ main(int argc, char * const *argv)
 		ARGV_ERR("Invalid instance (-n) name: %s\n", strerror(errno));
 
 	if (i_arg == NULL || *i_arg == '\0')
-		i_arg = "varnishd";
+		i_arg = mgt_HostName();
 	heritage.identity = i_arg;
 
 	mgt_ProcTitle("Mgr");
diff --git a/bin/varnishd/mgt/mgt_util.c b/bin/varnishd/mgt/mgt_util.c
index 5730ae5..15bee9f 100644
--- a/bin/varnishd/mgt/mgt_util.c
+++ b/bin/varnishd/mgt/mgt_util.c
@@ -45,6 +45,20 @@
 
 /*--------------------------------------------------------------------*/
 
+char *
+mgt_HostName(void)
+{
+	char *p;
+	char buf[1024];
+
+	AZ(gethostname(buf, sizeof buf));
+	p = strdup(buf);
+	AN(p);
+	return (p);
+}
+
+/*--------------------------------------------------------------------*/
+
 void
 mgt_ProcTitle(const char *comp)
 {
diff --git a/doc/changes.rst b/doc/changes.rst
index 478202e..eef34c2 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -2,6 +2,8 @@
 Varnish Cache Trunk (ongoing)
 =============================
 
+* Default the -i argument to gethostname(3)
+
 * On platforms with setproctitle(3), use the -i argument rather than -n
 
 * VSM_Name() returns the -i argument name, rather than the -n name.
diff --git a/doc/sphinx/reference/varnishd.rst b/doc/sphinx/reference/varnishd.rst
index 0064580..1d63dbe 100644
--- a/doc/sphinx/reference/varnishd.rst
+++ b/doc/sphinx/reference/varnishd.rst
@@ -135,7 +135,7 @@ Operations options
 
   Specify the identity of the Varnish server. This can be accessed
   using ``server.identity`` from VCL and with VSM_Name() from
-  utilities.
+  utilities.  If not specified the output of gethostname(3) is used.
 
 -I clifile
 
diff --git a/lib/libvarnish/vin.c b/lib/libvarnish/vin.c
index 78ab952..f2a3119 100644
--- a/lib/libvarnish/vin.c
+++ b/lib/libvarnish/vin.c
@@ -85,7 +85,7 @@ VIN_n_Arg(const char *n_arg, char **dir)
 
 	if (dir != NULL) {
 		*dir = strdup(dn);
-		if (*dir == NULL) 
+		if (*dir == NULL)
 			return (-1);
 	}
 	return (0);
diff --git a/lib/libvcc/generate.py b/lib/libvcc/generate.py
index 1a3d5da..94275d4 100755
--- a/lib/libvcc/generate.py
+++ b/lib/libvcc/generate.py
@@ -216,7 +216,7 @@ sp_variables = [
 		(), """
 		The identity of the server, as set by the -i
 		parameter.  If the -i parameter is not passed to varnishd,
-		server.identity will be set to "varnishd".
+		server.identity will be set to the hostname of the machine.
 		"""
 	),
 	('req',



From fgsch at lodoss.net  Wed Jun  7 08:38:06 2017
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Wed, 07 Jun 2017 10:38:06 +0200
Subject: [master] 9f33508 Fix crash under MacOS while investigating #2332
Message-ID: <E1dIWTe-000457-Fh@project.varnish-software.com>


commit 9f335089c5d2b8b8a45e662dd8f3d8b6f41dec71
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Sun Jun 4 16:28:16 2017 +0100

    Fix crash under MacOS while investigating #2332
    
    MacOS will return EINVAL under e.g. setsockopt if the connection was
    reset.

diff --git a/lib/libvarnish/vtcp.c b/lib/libvarnish/vtcp.c
index 941a90b..2a4c25f 100644
--- a/lib/libvarnish/vtcp.c
+++ b/lib/libvarnish/vtcp.c
@@ -577,6 +577,13 @@ VTCP_Check(int a)
 	 */
 	if (errno == EINVAL || errno == ETIMEDOUT || errno == EPIPE)
 		return (1);
+#elif defined (__APPLE__)
+	/*
+	 * MacOS returns EINVAL if the other end unexpectedly reset
+	 * the connection.
+	 */
+	if (errno == EINVAL)
+		return (1);
 #endif
 	return (0);
 }



From phk at FreeBSD.org  Wed Jun  7 11:37:05 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 07 Jun 2017 13:37:05 +0200
Subject: [master] ac6f9ca Convert from enum to erzatz-but-type-safe-enum
Message-ID: <E1dIZGr-0002sk-Pn@project.varnish-software.com>


commit ac6f9ca1b7a30f2d5cbad813e488592a40c380ef
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Jun 7 08:47:10 2017 +0000

    Convert from enum to erzatz-but-type-safe-enum

diff --git a/include/vapi/vsm.h b/include/vapi/vsm.h
index 0739c96..bd26bc9 100644
--- a/include/vapi/vsm.h
+++ b/include/vapi/vsm.h
@@ -153,13 +153,15 @@ int VSM__itern(const struct VSM_data *vd, struct VSM_fantom *vf);
 	 * vd = "struct VSM_data *"
 	 */
 
-enum VSM_valid_e {
-	VSM_invalid,
-	VSM_valid,
-	VSM_similar,
+struct vsm_valid {
+	const char *name;
 };
 
-enum VSM_valid_e VSM_StillValid(const struct VSM_data *vd,
+extern const struct vsm_valid VSM_invalid[];
+extern const struct vsm_valid VSM_valid[];
+extern const struct vsm_valid VSM_similar[];
+
+const struct vsm_valid *VSM_StillValid(const struct VSM_data *vd,
     struct VSM_fantom *vf);
 	/*
 	 * Check the validity of a previously looked up VSM_fantom.
diff --git a/lib/libvarnishapi/libvarnishapi.map b/lib/libvarnishapi/libvarnishapi.map
index 14c4418..7d76489 100644
--- a/lib/libvarnishapi/libvarnishapi.map
+++ b/lib/libvarnishapi/libvarnishapi.map
@@ -179,4 +179,7 @@ LIBVARNISHAPI_1.6 {
 LIBVARNISHAPI_1.7 {
   global:
 	VSC_ChangeLevel;
+	VSM_invalid;
+	VSM_valid;
+	VSM_similar;
 } LIBVARNISHAPI_1.0;
diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c
index 35551d4..5835014 100644
--- a/lib/libvarnishapi/vsm.c
+++ b/lib/libvarnishapi/vsm.c
@@ -58,6 +58,9 @@
 #define MAP_HASSEMAPHORE 0 /* XXX Linux */
 #endif
 
+const struct vsm_valid VSM_invalid[1] = {{"invalid"}};
+const struct vsm_valid VSM_valid[1] = {{"valid"}};
+const struct vsm_valid VSM_similar[1] = {{"similar"}};
 
 /*--------------------------------------------------------------------*/
 
@@ -390,7 +393,7 @@ VSM__itern(const struct VSM_data *vd, struct VSM_fantom *vf)
 
 /*--------------------------------------------------------------------*/
 
-enum VSM_valid_e
+const struct vsm_valid *
 VSM_StillValid(const struct VSM_data *vd, struct VSM_fantom *vf)
 {
 	struct VSM_fantom f2;



From phk at FreeBSD.org  Wed Jun  7 11:37:05 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 07 Jun 2017 13:37:05 +0200
Subject: [master] dec41ee Isolate VSC from VSM
Message-ID: <E1dIZGr-0002sq-RI@project.varnish-software.com>


commit dec41ee7062634b58cd0c82185ff0ce2bfe31ec5
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Jun 7 11:35:51 2017 +0000

    Isolate VSC from VSM

diff --git a/lib/libvarnishapi/Makefile.am b/lib/libvarnishapi/Makefile.am
index 0e5f34c..54f1884 100644
--- a/lib/libvarnishapi/Makefile.am
+++ b/lib/libvarnishapi/Makefile.am
@@ -14,7 +14,7 @@ libvarnishapi_la_LDFLAGS = $(AM_LDFLAGS) -version-info 1:6:0
 libvarnishapi_la_SOURCES = \
 	vjsn.c \
 	vjsn.h \
-	vsm_api.h \
+	vsc_priv.h \
 	vsl_api.h \
 	vxp.h \
 	vxp_tokens.h \
diff --git a/lib/libvarnishapi/vsc.c b/lib/libvarnishapi/vsc.c
index c42f501..37334f2 100644
--- a/lib/libvarnishapi/vsc.c
+++ b/lib/libvarnishapi/vsc.c
@@ -49,7 +49,7 @@
 #include "vapi/vsc.h"
 #include "vapi/vsm.h"
 
-#include "vsm_api.h"
+#include "vsc_priv.h"
 
 struct vsc_vf {
 	unsigned		magic;
@@ -111,18 +111,20 @@ static const size_t nlevels =
 static struct vsc *
 vsc_setup(struct VSM_data *vd)
 {
+	struct vsc *vsc;
 
-	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	if (vd->vsc == NULL) {
-		ALLOC_OBJ(vd->vsc, VSC_MAGIC);
-		AN(vd->vsc);
-		VTAILQ_INIT(&vd->vsc->vf_list);
-		VTAILQ_INIT(&vd->vsc->pt_list);
-		VTAILQ_INIT(&vd->vsc->sf_list_include);
-		VTAILQ_INIT(&vd->vsc->sf_list_exclude);
+	vsc = VSM_GetVSC(vd);
+	if (vsc == NULL) {
+		ALLOC_OBJ(vsc, VSC_MAGIC);
+		AN(vsc);
+		VTAILQ_INIT(&vsc->vf_list);
+		VTAILQ_INIT(&vsc->pt_list);
+		VTAILQ_INIT(&vsc->sf_list_include);
+		VTAILQ_INIT(&vsc->sf_list_exclude);
+		VSM_SetVSC(vd, vsc);
 	}
-	CHECK_OBJ_NOTNULL(vd->vsc, VSC_MAGIC);
-	return (vd->vsc);
+	CHECK_OBJ_NOTNULL(vsc, VSC_MAGIC);
+	return (vsc);
 }
 
 /*--------------------------------------------------------------------*/
@@ -168,13 +170,9 @@ vsc_delete_sf_list(struct vsc_sf_head *head)
 }
 
 void
-VSC_Delete(struct VSM_data *vd)
+VSC_Delete(struct vsc *vsc)
 {
-	struct vsc *vsc;
 
-	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	vsc = vd->vsc;
-	vd->vsc = NULL;
 	CHECK_OBJ_NOTNULL(vsc, VSC_MAGIC);
 	vsc_delete_sf_list(&vsc->sf_list_include);
 	vsc_delete_sf_list(&vsc->sf_list_exclude);
@@ -343,7 +341,7 @@ vsc_build_pt_list(struct VSM_data *vd)
 		vve = vjsn_child(vf->vjsn->value, "elem");
 		AN(vve);
 		VTAILQ_FOREACH(vv, &vve->children, list) {
-			vdsc = calloc(sizeof *vd, 1);
+			vdsc = calloc(sizeof *vdsc, 1);
 			AN(vdsc);
 
 #define DOF(n, k)						\
diff --git a/lib/libvarnishapi/vsc_priv.h b/lib/libvarnishapi/vsc_priv.h
new file mode 100644
index 0000000..363f6cc
--- /dev/null
+++ b/lib/libvarnishapi/vsc_priv.h
@@ -0,0 +1,36 @@
+/*-
+ * Copyright (c) 2006 Verdens Gang AS
+ * Copyright (c) 2006-2015 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ */
+
+struct vsc;
+struct VSM_data;
+
+void VSM_SetVSC(struct VSM_data *, struct vsc *);
+struct vsc *VSM_GetVSC(const struct VSM_data *);
+void VSC_Delete(struct vsc *);
diff --git a/lib/libvarnishapi/vsl_cursor.c b/lib/libvarnishapi/vsl_cursor.c
index c90f93d..1c9b2ba 100644
--- a/lib/libvarnishapi/vsl_cursor.c
+++ b/lib/libvarnishapi/vsl_cursor.c
@@ -54,7 +54,6 @@
 #include "vapi/vsm.h"
 
 #include "vsl_api.h"
-#include "vsm_api.h"
 
 struct vslc_vsm {
 	unsigned			magic;
@@ -125,7 +124,6 @@ vslc_vsm_next(const struct VSL_cursor *cursor)
 
 	CAST_OBJ_NOTNULL(c, cursor->priv_data, VSLC_VSM_MAGIC);
 	assert(&c->cursor == cursor);
-	CHECK_OBJ_NOTNULL(c->vsm, VSM_MAGIC);
 
 	while (1) {
 		i = vslc_vsm_check(&c->cursor, &c->next);
@@ -248,7 +246,6 @@ VSL_CursorVSM(struct VSL_data *vsl, struct VSM_data *vsm, unsigned options)
 	int i;
 
 	CHECK_OBJ_NOTNULL(vsl, VSL_MAGIC);
-	CHECK_OBJ_NOTNULL(vsm, VSM_MAGIC);
 
 	if (!VSM_Get(vsm, &vf, VSL_CLASS, "", "")) {
 		(void)vsl_diag(vsl,
diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c
index 5835014..b7b96b2 100644
--- a/lib/libvarnishapi/vsm.c
+++ b/lib/libvarnishapi/vsm.c
@@ -48,11 +48,11 @@
 #include "vin.h"
 #include "vsb.h"
 #include "vsm_priv.h"
+#include "vsc_priv.h"
 #include "vtim.h"
 
 #include "vapi/vsm.h"
 
-#include "vsm_api.h"
 
 #ifndef MAP_HASSEMAPHORE
 #define MAP_HASSEMAPHORE 0 /* XXX Linux */
@@ -64,6 +64,32 @@ const struct vsm_valid VSM_similar[1] = {{"similar"}};
 
 /*--------------------------------------------------------------------*/
 
+struct vsc;
+
+struct VSM_data {
+	unsigned		magic;
+#define VSM_MAGIC		0x6e3bd69b
+
+	struct vsb		*diag;
+
+	char			*dname;
+	char			*iname;
+
+	struct stat		fstat;
+
+	int			vsm_fd;
+	struct VSM_head		*head;
+	char			*b;
+	char			*e;
+
+	uint64_t		age_ok;
+	double			t_ok;
+
+	struct vsc		*vsc;
+};
+
+/*--------------------------------------------------------------------*/
+
 struct VSM_data *
 VSM_New(void)
 {
@@ -81,7 +107,25 @@ VSM_New(void)
 
 /*--------------------------------------------------------------------*/
 
-int
+void
+VSM_SetVSC(struct VSM_data *vd, struct vsc *vsc)
+{
+	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
+
+	vd->vsc = vsc;
+}
+
+struct vsc *
+VSM_GetVSC(const struct VSM_data *vd)
+{
+	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
+
+	return (vd->vsc);
+}
+
+/*--------------------------------------------------------------------*/
+
+static int
 vsm_diag(struct VSM_data *vd, const char *fmt, ...)
 {
 	va_list ap;
@@ -179,7 +223,7 @@ VSM_Delete(struct VSM_data *vd)
 
 	VSM_Close(vd);
 	if (vd->vsc != NULL)
-		VSC_Delete(vd);
+		VSC_Delete(vd->vsc);
 	VSM_ResetError(vd);
 	free(vd->dname);
 	FREE_OBJ(vd);
diff --git a/lib/libvarnishapi/vsm_api.h b/lib/libvarnishapi/vsm_api.h
deleted file mode 100644
index e9402b0..0000000
--- a/lib/libvarnishapi/vsm_api.h
+++ /dev/null
@@ -1,58 +0,0 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2015 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- */
-
-struct vsc;
-struct vsb;
-
-struct VSM_data {
-	unsigned		magic;
-#define VSM_MAGIC		0x6e3bd69b
-
-	struct vsb		*diag;
-
-	char			*dname;
-	char			*iname;
-
-	struct stat		fstat;
-
-	int			vsm_fd;
-	struct VSM_head		*head;
-	char			*b;
-	char			*e;
-
-	uint64_t		age_ok;
-	double			t_ok;
-
-	struct vsc		*vsc;
-};
-
-int vsm_diag(struct VSM_data *vd, const char *fmt, ...)
-    __v_printflike(2, 3);
-void VSC_Delete(struct VSM_data *vd);



From dridi.boukelmoune at gmail.com  Wed Jun  7 14:44:05 2017
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Wed, 07 Jun 2017 16:44:05 +0200
Subject: [master] 3302009 s/vc/vbc/
Message-ID: <E1dIcBp-00022z-KI@project.varnish-software.com>


commit 330200927616962c9486a3c7846e69dea2f859dc
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Wed Jun 7 16:33:07 2017 +0200

    s/vc/vbc/
    
    This is the only place where a `struct vbc *` pointer isn't called `vbc`
    and this helps me get a better picture.

diff --git a/bin/varnishd/cache/cache_backend.c b/bin/varnishd/cache/cache_backend.c
index d9759d8..4e75b2f 100644
--- a/bin/varnishd/cache/cache_backend.c
+++ b/bin/varnishd/cache/cache_backend.c
@@ -64,7 +64,7 @@
 static struct vbc *
 vbe_dir_getfd(struct worker *wrk, struct backend *bp, struct busyobj *bo)
 {
-	struct vbc *vc;
+	struct vbc *vbc;
 	double tmod;
 	char abuf1[VTCP_ADDRBUFSIZE], abuf2[VTCP_ADDRBUFSIZE];
 	char pbuf1[VTCP_PORTBUFSIZE], pbuf2[VTCP_PORTBUFSIZE];
@@ -94,16 +94,16 @@ vbe_dir_getfd(struct worker *wrk, struct backend *bp, struct busyobj *bo)
 	bo->htc->doclose = SC_NULL;
 
 	FIND_TMO(connect_timeout, tmod, bo, bp);
-	vc = VBT_Get(bp->tcp_pool, tmod, bp, wrk);
-	if (vc == NULL) {
+	vbc = VBT_Get(bp->tcp_pool, tmod, bp, wrk);
+	if (vbc == NULL) {
 		// XXX: Per backend stats ?
 		VSC_C_main->backend_fail++;
 		bo->htc = NULL;
 		return (NULL);
 	}
 
-	assert(vc->fd >= 0);
-	AN(vc->addr);
+	assert(vbc->fd >= 0);
+	AN(vbc->addr);
 
 	Lck_Lock(&bp->mtx);
 	bp->n_conn++;
@@ -112,21 +112,21 @@ vbe_dir_getfd(struct worker *wrk, struct backend *bp, struct busyobj *bo)
 	Lck_Unlock(&bp->mtx);
 
 	if (bp->proxy_header != 0)
-		VPX_Send_Proxy(vc->fd, bp->proxy_header, bo->sp);
+		VPX_Send_Proxy(vbc->fd, bp->proxy_header, bo->sp);
 
-	VTCP_myname(vc->fd, abuf1, sizeof abuf1, pbuf1, sizeof pbuf1);
-	VTCP_hisname(vc->fd, abuf2, sizeof abuf2, pbuf2, sizeof pbuf2);
+	VTCP_myname(vbc->fd, abuf1, sizeof abuf1, pbuf1, sizeof pbuf1);
+	VTCP_hisname(vbc->fd, abuf2, sizeof abuf2, pbuf2, sizeof pbuf2);
 	VSLb(bo->vsl, SLT_BackendOpen, "%d %s %s %s %s %s",
-	    vc->fd, bp->display_name, abuf2, pbuf2, abuf1, pbuf1);
+	    vbc->fd, bp->display_name, abuf2, pbuf2, abuf1, pbuf1);
 
 	INIT_OBJ(bo->htc, HTTP_CONN_MAGIC);
-	bo->htc->priv = vc;
-	bo->htc->rfd = &vc->fd;
+	bo->htc->priv = vbc;
+	bo->htc->rfd = &vbc->fd;
 	FIND_TMO(first_byte_timeout,
 	    bo->htc->first_byte_timeout, bo, bp);
 	FIND_TMO(between_bytes_timeout,
 	    bo->htc->between_bytes_timeout, bo, bp);
-	return (vc);
+	return (vbc);
 }
 
 static unsigned __match_proto__(vdi_healthy_f)



From dridi.boukelmoune at gmail.com  Wed Jun  7 14:47:05 2017
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Wed, 07 Jun 2017 16:47:05 +0200
Subject: [master] 8d5b148 Reduce noise when running the test suite
Message-ID: <E1dIcEj-0002CU-QF@project.varnish-software.com>


commit 8d5b1483bbe936c10075efa7b4671caff4b50e15
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Wed Jun 7 16:45:04 2017 +0200

    Reduce noise when running the test suite

diff --git a/bin/varnishtest/Makefile.am b/bin/varnishtest/Makefile.am
index 3710cf9..5ffbf79 100644
--- a/bin/varnishtest/Makefile.am
+++ b/bin/varnishtest/Makefile.am
@@ -9,7 +9,7 @@ check: check-am check-local
 check-am: check-local
 # See if list of checks have changed, recheck
 check-local:
-	mkdir -p tests ; \
+	@mkdir -p tests ; \
 	LC_ALL=C; \
 	if [ "$$(cd $(srcdir) && echo tests/*.vtc)" != "@VTC_TESTS@" ]; then \
 		cd $(top_builddir) && ./config.status --recheck ; \



From phk at FreeBSD.org  Thu Jun  8 13:01:06 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 08 Jun 2017 15:01:06 +0200
Subject: [master] a93c455 Put the VSC->type (ie: MAIN,
 VBE etc.) into the ->ident field.
Message-ID: <E1dIx3i-0005qF-GJ@project.varnish-software.com>


commit a93c4555ef664ce2835cb43d4cccddd188a27ecf
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Jun 8 12:54:45 2017 +0000

    Put the VSC->type (ie: MAIN, VBE etc.) into the ->ident field.
    
    Also add back the "formats" property for VSC counters.

diff --git a/bin/varnishd/main.vsc b/bin/varnishd/main.vsc
index c903c28..97ca0a2 100644
--- a/bin/varnishd/main.vsc
+++ b/bin/varnishd/main.vsc
@@ -16,6 +16,7 @@
 
 .. varnish_vsc:: uptime
 	:oneliner:	Child process uptime
+	:format:	duration
 
 	How long the child process has been running.
 
diff --git a/bin/varnishd/mempool.vsc b/bin/varnishd/mempool.vsc
index 2890863..0f333cc 100644
--- a/bin/varnishd/mempool.vsc
+++ b/bin/varnishd/mempool.vsc
@@ -24,12 +24,14 @@
 .. varnish_vsc:: sz_wanted
 	:type:	gauge
 	:level:	debug
+	:format: bytes
 	:oneliner:	Size requested
 
 
 .. varnish_vsc:: sz_actual
 	:type:	gauge
 	:level:	debug
+	:format: bytes
 	:oneliner:	Size allocated
 
 
diff --git a/bin/varnishd/sma.vsc b/bin/varnishd/sma.vsc
index 9b4aadd..5756bec 100644
--- a/bin/varnishd/sma.vsc
+++ b/bin/varnishd/sma.vsc
@@ -26,6 +26,7 @@
 .. varnish_vsc:: c_bytes
 	:type:	counter
 	:level:	info
+	:format: bytes
 	:oneliner:	Bytes allocated
 
 	Number of total bytes allocated by this storage.
@@ -33,6 +34,7 @@
 .. varnish_vsc:: c_freed
 	:type:	counter
 	:level:	info
+	:format: bytes
 	:oneliner:	Bytes freed
 
 	Number of total bytes returned to this storage.
@@ -47,6 +49,7 @@
 .. varnish_vsc:: g_bytes
 	:type:	gauge
 	:level:	info
+	:format: bytes
 	:oneliner:	Bytes outstanding
 
 	Number of bytes allocated from the storage.
@@ -54,6 +57,7 @@
 .. varnish_vsc:: g_space
 	:type:	gauge
 	:level:	info
+	:format: bytes
 	:oneliner:	Bytes available
 
 	Number of bytes left in the storage.
diff --git a/bin/varnishd/smf.vsc b/bin/varnishd/smf.vsc
index 806b2a6..3ba2f3f 100644
--- a/bin/varnishd/smf.vsc
+++ b/bin/varnishd/smf.vsc
@@ -26,6 +26,7 @@
 .. varnish_vsc:: c_bytes
 	:type:	counter
 	:level:	info
+	:format: bytes
 	:oneliner:	Bytes allocated
 
 	Number of total bytes allocated by this storage.
@@ -33,6 +34,7 @@
 .. varnish_vsc:: c_freed
 	:type:	counter
 	:level:	info
+	:format: bytes
 	:oneliner:	Bytes freed
 
 	Number of total bytes returned to this storage.
@@ -47,6 +49,7 @@
 .. varnish_vsc:: g_bytes
 	:type:	gauge
 	:level:	info
+	:format: bytes
 	:oneliner:	Bytes outstanding
 
 	Number of bytes allocated from the storage.
@@ -54,6 +57,7 @@
 .. varnish_vsc:: g_space
 	:type:	gauge
 	:level:	info
+	:format: bytes
 	:oneliner:	Bytes available
 
 	Number of bytes left in the storage.
diff --git a/bin/varnishd/vbe.vsc b/bin/varnishd/vbe.vsc
index a84acbb..6ce7ee1 100644
--- a/bin/varnishd/vbe.vsc
+++ b/bin/varnishd/vbe.vsc
@@ -17,6 +17,7 @@
 .. varnish_vsc:: bereq_hdrbytes
 	:type:	counter
 	:level:	info
+	:format: bytes
 	:oneliner:	Request header bytes
 
 	Total backend request header bytes sent
@@ -24,6 +25,7 @@
 .. varnish_vsc:: bereq_bodybytes
 	:type:	counter
 	:level:	info
+	:format: bytes
 	:oneliner:	Request body bytes
 
 	Total backend request body bytes sent
@@ -31,6 +33,7 @@
 .. varnish_vsc:: beresp_hdrbytes
 	:type:	counter
 	:level:	info
+	:format: bytes
 	:oneliner:	Response header bytes
 
 	Total backend response header bytes received
@@ -38,6 +41,7 @@
 .. varnish_vsc:: beresp_bodybytes
 	:type:	counter
 	:level:	info
+	:format: bytes
 	:oneliner:	Response body bytes
 
 	Total backend response body bytes received
@@ -45,6 +49,7 @@
 .. varnish_vsc:: pipe_hdrbytes
 	:type:	counter
 	:level:	info
+	:format: bytes
 	:oneliner:	Pipe request header bytes
 
 	Total request bytes sent for piped sessions
@@ -52,6 +57,7 @@
 .. varnish_vsc:: pipe_out
 	:type:	counter
 	:level:	info
+	:format: bytes
 	:oneliner:	Piped bytes to backend
 
 	Total number of bytes forwarded to backend in pipe sessions
@@ -59,6 +65,7 @@
 .. varnish_vsc:: pipe_in
 	:type:	counter
 	:level:	info
+	:format: bytes
 	:oneliner:	Piped bytes from backend
 
 	Total number of bytes forwarded from backend in pipe sessions
diff --git a/bin/varnishstat/varnishstat.c b/bin/varnishstat/varnishstat.c
index 2323b2b..7f6bd33 100644
--- a/bin/varnishstat/varnishstat.c
+++ b/bin/varnishstat/varnishstat.c
@@ -66,11 +66,7 @@ do_xml_cb(void *priv, const struct VSC_point * const pt)
 	sec = pt->section;
 
 	printf("\t<stat>\n");
-	if (strcmp(sec->type, ""))
-		printf("\t\t<type>%s</type>\n", sec->type);
-	if (strcmp(sec->ident, ""))
-		printf("\t\t<ident>%s</ident>\n", sec->ident);
-	printf("\t\t<name>%s</name>\n", pt->desc->name);
+	printf("\t\t<name>%s.%s</name>\n", sec->ident, pt->desc->name);
 	printf("\t\t<value>%ju</value>\n", (uintmax_t)val);
 	printf("\t\t<flag>%c</flag>\n", pt->desc->semantics);
 	printf("\t\t<format>%c</format>\n", pt->desc->format);
@@ -118,18 +114,12 @@ do_json_cb(void *priv, const struct VSC_point * const pt)
 
 	printf("  \"");
 	/* build the JSON key name.  */
-	if (sec->type[0])
-		printf("%s.", sec->type);
 	if (sec->ident[0])
 		printf("%s.", sec->ident);
 	printf("%s\": {\n", pt->desc->name);
 	printf("    \"description\": \"%s\",\n", pt->desc->sdesc);
 
-	if (strcmp(sec->type, ""))
-		printf("    \"type\": \"%s\", ", sec->type);
-	if (strcmp(sec->ident, ""))
-		printf("\"ident\": \"%s\", ", sec->ident);
-	printf("\"flag\": \"%c\", ", pt->desc->semantics);
+	printf("    \"flag\": \"%c\", ", pt->desc->semantics);
 	printf("\"format\": \"%c\",\n", pt->desc->format);
 	printf("    \"value\": %ju", (uintmax_t)val);
 	printf("\n  }");
@@ -177,7 +167,7 @@ do_once_cb_first(void *priv, const struct VSC_point * const pt)
 	op = priv;
 	AZ(strcmp(pt->desc->ctype, "uint64_t"));
 	sec = pt->section;
-	if (strcmp(sec->type, "MAIN") || strcmp(pt->desc->name, "uptime"))
+	if (strcmp(sec->ident, "MAIN") || strcmp(pt->desc->name, "uptime"))
 		return (0);
 	val = *(const volatile uint64_t*)pt->ptr;
 	op->up = (double)val;
@@ -199,8 +189,6 @@ do_once_cb(void *priv, const struct VSC_point * const pt)
 	val = *(const volatile uint64_t*)pt->ptr;
 	sec = pt->section;
 	i = 0;
-	if (strcmp(sec->type, ""))
-		i += printf("%s.", sec->type);
 	if (strcmp(sec->ident, ""))
 		i += printf("%s.", sec->ident);
 	i += printf("%s", pt->desc->name);
@@ -244,8 +232,6 @@ do_list_cb(void *priv, const struct VSC_point * const pt)
 
 	sec = pt->section;
 	i = 0;
-	if (strcmp(sec->type, ""))
-		i += printf("%s.", sec->type);
 	if (strcmp(sec->ident, ""))
 		i += printf("%s.", sec->ident);
 	i += printf("%s", pt->desc->name);
diff --git a/bin/varnishstat/varnishstat_curses.c b/bin/varnishstat/varnishstat_curses.c
index 6d3d142..97995dd 100644
--- a/bin/varnishstat/varnishstat_curses.c
+++ b/bin/varnishstat/varnishstat_curses.c
@@ -276,8 +276,7 @@ build_pt_list_cb(void *priv, const struct VSC_point *vpt)
 	CAST_OBJ_NOTNULL(pt_priv, priv, PT_PRIV_MAGIC);
 
 	AZ(strcmp(vpt->desc->ctype, "uint64_t"));
-	bprintf(buf, "%s.%s.%s", vpt->section->type,
-	    vpt->section->ident, vpt->desc->name);
+	bprintf(buf, "%s.%s", vpt->section->ident, vpt->desc->name);
 
 	if (!strcmp(buf, "MGT..uptime"))
 		mgt_uptime = vpt->ptr;
@@ -313,12 +312,7 @@ build_pt_list_cb(void *priv, const struct VSC_point *vpt)
 	pt->key = strdup(buf);
 	AN(pt->key);
 
-	if (*vpt->section->type != '\0')
-		bprintf(buf, "%s.%s", vpt->section->type, vpt->desc->name);
-	else if (*vpt->section->ident != '\0')
-		bprintf(buf, "%s.%s", vpt->section->ident, vpt->desc->name);
-	else
-		bprintf(buf, "%s", vpt->desc->name);
+	bprintf(buf, "%s.%s", vpt->section->ident, vpt->desc->name);
 	REPLACE(pt->name, buf);
 	AN(pt->name);
 
diff --git a/bin/varnishtest/tests/u00005.vtc b/bin/varnishtest/tests/u00005.vtc
index 4b068d4..2653854 100644
--- a/bin/varnishtest/tests/u00005.vtc
+++ b/bin/varnishtest/tests/u00005.vtc
@@ -45,7 +45,7 @@ shell -err -expect "Cannot open /nonexistent/_.vsm" \
 	"varnishstat -n /nonexistent -t 1"
 shell -expect "MAIN.uptime" \
 	"varnishstat -n ${v1_name} -1"
-shell -expect "<type>MAIN</type>" \
+shell -expect "<name>MAIN.uptime</name>" \
 	"varnishstat -n ${v1_name} -x"
 shell -match {"MAIN.uptime":} \
 	"varnishstat -n ${v1_name} -j"
diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c
index c385ffc..82d8444 100644
--- a/bin/varnishtest/vtc_varnish.c
+++ b/bin/varnishtest/vtc_varnish.c
@@ -822,11 +822,7 @@ do_stat_dump_cb(void *priv, const struct VSC_point * const pt)
 		return (0);
 	u = *(const volatile uint64_t*)pt->ptr;
 
-	strcpy(buf, pt->section->type);
-	if (pt->section->ident[0] != '\0')
-		bprintf(buf, "%s.%s", pt->section->ident, pt->desc->name);
-	else
-		bprintf(buf, "MAIN.%s", pt->desc->name);
+	bprintf(buf, "%s.%s", pt->section->ident, pt->desc->name);
 
 	if (strcmp(dp->arg, "*")) {
 		if (fnmatch(dp->arg, buf, 0))
@@ -858,7 +854,6 @@ varnish_vsc(const struct varnish *v, const char *arg)
  */
 
 struct stat_priv {
-	char target_type[256];
 	char target_ident[256];
 	char target_name[256];
 	uintmax_t val;
@@ -873,8 +868,6 @@ do_stat_cb(void *priv, const struct VSC_point * const pt)
 	if (pt == NULL)
 		return(0);
 
-	if (strcmp(pt->section->type, sp->target_type))
-		return(0);
 	if (strcmp(pt->section->ident, sp->target_ident))
 		return(0);
 	if (strcmp(pt->desc->name, sp->target_name))
@@ -895,7 +888,6 @@ varnish_expect(const struct varnish *v, char * const *av)
 	int good;
 	char *r;
 	char *p;
-	char *q;
 	int i, not = 0;
 	struct stat_priv sp;
 
@@ -908,22 +900,13 @@ varnish_expect(const struct varnish *v, char * const *av)
 		AN(av[1]);
 		AN(av[2]);
 	}
-	p = strchr(r, '.');
+	p = strrchr(r, '.');
 	if (p == NULL) {
-		strcpy(sp.target_type, "MAIN");
-		sp.target_ident[0] = '\0';
+		strcpy(sp.target_ident, "MAIN");
 		bprintf(sp.target_name, "%s", r);
 	} else {
-		bprintf(sp.target_type, "%.*s", (int)(p - r), r);
-		p++;
-		q = strrchr(p, '.');
-		if (q == NULL) {
-			sp.target_ident[0] = '\0';
-			bprintf(sp.target_name, "%s", p);
-		} else {
-			bprintf(sp.target_ident, "%.*s", (int)(q - p), p);
-			bprintf(sp.target_name, "%s", q + 1);
-		}
+		bprintf(sp.target_ident, "%.*s", (int)(p - r), r);
+		bprintf(sp.target_name, "%s", p + 1);
 	}
 
 	sp.val = 0;
diff --git a/include/vapi/vsc.h b/include/vapi/vsc.h
index 3f96f09..c53d388 100644
--- a/include/vapi/vsc.h
+++ b/include/vapi/vsc.h
@@ -85,7 +85,6 @@ struct VSC_level_desc {
 };
 
 struct VSC_section {
-	char *type;
 	char *ident;
 };
 
diff --git a/lib/libvarnishapi/vsc.c b/lib/libvarnishapi/vsc.c
index 37334f2..339b588 100644
--- a/lib/libvarnishapi/vsc.c
+++ b/lib/libvarnishapi/vsc.c
@@ -249,12 +249,19 @@ static struct vsc_vf *
 vsc_add_vf(struct vsc *vsc, const struct VSM_fantom *fantom, int order)
 {
 	struct vsc_vf *vf, *vf2;
+	struct vsb *vsb;
 
 	ALLOC_OBJ(vf, VSC_VF_MAGIC);
 	AN(vf);
 	vf->fantom = *fantom;
-	REPLACE(vf->section.type, vf->fantom.type);
-	REPLACE(vf->section.ident, vf->fantom.ident);
+	vsb = VSB_new_auto();
+	AN(vsb);
+	VSB_printf(vsb, "%s", vf->fantom.type);
+	if (*vf->fantom.ident != '\0')
+		VSB_printf(vsb, ".%s", vf->fantom.ident);
+	AZ(VSB_finish(vsb));
+	REPLACE(vf->section.ident, VSB_data(vsb));
+	VSB_destroy(&vsb);
 	vf->order = order;
 
 	VTAILQ_FOREACH(vf2, &vsc->vf_list, list) {
@@ -355,7 +362,38 @@ vsc_build_pt_list(struct VSM_data *vd)
 			DOF(sdesc, "oneliner");
 			DOF(ldesc, "docs");
 #undef DOF
+			vt = vjsn_child(vv, "type");
+			AN(vt);
+			assert(vt->type == VJSN_STRING);
+
+			if (!strcmp(vt->value, "counter")) {
+				vdsc->semantics = 'c';
+			} else if (!strcmp(vt->value, "gauge")) {
+				vdsc->semantics = 'g';
+			} else if (!strcmp(vt->value, "bitmap")) {
+				vdsc->semantics = 'b';
+			} else {
+				vdsc->semantics = '?';
+			}
+
+			vt = vjsn_child(vv, "format");
+			AN(vt);
+			assert(vt->type == VJSN_STRING);
+
+			if (!strcmp(vt->value, "integer")) {
+				vdsc->format = 'i';
+			} else if (!strcmp(vt->value, "bytes")) {
+				vdsc->format = 'B';
+			} else if (!strcmp(vt->value, "bitmap")) {
+				vdsc->format = 'b';
+			} else if (!strcmp(vt->value, "duration")) {
+				vdsc->format = 'd';
+			} else {
+				vdsc->format = '?';
+			}
+
 			vdsc->level = &level_info;
+
 			vt = vjsn_child(vv, "index");
 			AN(vt);
 			vsc_add_pt(vsc,
@@ -373,8 +411,6 @@ vsc_filter_match_pt(struct vsb *vsb, const struct vsc_sf *sf, const
     struct vsc_pt *pt)
 {
 	VSB_clear(vsb);
-	if (strcmp(pt->point.section->type, ""))
-		VSB_printf(vsb, "%s.", pt->point.section->type);
 	if (strcmp(pt->point.section->ident, ""))
 		VSB_printf(vsb, "%s.", pt->point.section->ident);
 	VSB_printf(vsb, "%s", pt->point.desc->name);
diff --git a/lib/libvcc/vsctool.py b/lib/libvcc/vsctool.py
index 00a1714..93c7d2d 100644
--- a/lib/libvcc/vsctool.py
+++ b/lib/libvcc/vsctool.py
@@ -43,12 +43,14 @@ import collections
 TYPES = [ "counter", "gauge", "bitmap" ]
 CTYPES = [ "uint64_t" ]
 LEVELS = [ "info", "diag", "debug" ]
+FORMATS = [ "integer", "bytes", "duration" ]
 
 PARAMS = {
 	"type":		["counter", TYPES],
 	"ctype":	["uint64_t", CTYPES],
 	"level":	["info", LEVELS],
 	"oneliner":	True,
+	"format":	[ "integer", FORMATS],
 }
 
 def gzip_str(s):



From phk at FreeBSD.org  Thu Jun  8 13:42:05 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 08 Jun 2017 15:42:05 +0200
Subject: [master] ef56a0d Rename struct "VSM_data" to just "vsm",
 to definitively break the API.
Message-ID: <E1dIxhN-0007is-TV@project.varnish-software.com>


commit ef56a0d60eb862ca624c80306abfb6b5b6d141d6
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Jun 8 13:41:06 2017 +0000

    Rename struct "VSM_data" to just "vsm", to definitively break the API.

diff --git a/bin/varnishadm/varnishadm.c b/bin/varnishadm/varnishadm.c
index efca881..7e0c15b 100644
--- a/bin/varnishadm/varnishadm.c
+++ b/bin/varnishadm/varnishadm.c
@@ -393,7 +393,7 @@ n_arg_sock(const char *n_arg)
 {
 	char *T_arg = NULL, *T_start = NULL;
 	char *S_arg = NULL;
-	struct VSM_data *vsm;
+	struct vsm *vsm;
 	char *p;
 	int sock;
 	struct VSM_fantom vt;
diff --git a/bin/varnishstat/varnishstat.c b/bin/varnishstat/varnishstat.c
index 7f6bd33..1290408 100644
--- a/bin/varnishstat/varnishstat.c
+++ b/bin/varnishstat/varnishstat.c
@@ -76,7 +76,7 @@ do_xml_cb(void *priv, const struct VSC_point * const pt)
 }
 
 static void
-do_xml(struct VSM_data *vd)
+do_xml(struct vsm *vd)
 {
 	char time_stamp[20];
 	time_t now;
@@ -130,7 +130,7 @@ do_json_cb(void *priv, const struct VSC_point * const pt)
 }
 
 static void
-do_json(struct VSM_data *vd)
+do_json(struct vsm *vd)
 {
 	char time_stamp[20];
 	time_t now;
@@ -206,7 +206,7 @@ do_once_cb(void *priv, const struct VSC_point * const pt)
 }
 
 static void
-do_once(struct VSM_data *vd)
+do_once(struct vsm *vd)
 {
 	struct once_priv op;
 
@@ -242,7 +242,7 @@ do_list_cb(void *priv, const struct VSC_point * const pt)
 }
 
 static void
-list_fields(struct VSM_data *vd)
+list_fields(struct vsm *vd)
 {
 	printf("Varnishstat -f option fields:\n");
 	printf("Field name                     Description\n");
@@ -268,7 +268,7 @@ usage(int status)
 int
 main(int argc, char * const *argv)
 {
-	struct VSM_data *vd;
+	struct vsm *vd;
 	double t_arg = 5.0, t_start = NAN;
 	int once = 0, xml = 0, json = 0, f_list = 0, curses = 0;
 	signed char opt;
diff --git a/bin/varnishstat/varnishstat.h b/bin/varnishstat/varnishstat.h
index 18d3def..54fd0ba 100644
--- a/bin/varnishstat/varnishstat.h
+++ b/bin/varnishstat/varnishstat.h
@@ -35,4 +35,4 @@
 #include "vas.h"
 #include "vcs.h"
 
-void do_curses(struct VSM_data *vd, double delay);
+void do_curses(struct vsm *vd, double delay);
diff --git a/bin/varnishstat/varnishstat_curses.c b/bin/varnishstat/varnishstat_curses.c
index 97995dd..f3b7449 100644
--- a/bin/varnishstat/varnishstat_curses.c
+++ b/bin/varnishstat/varnishstat_curses.c
@@ -334,7 +334,7 @@ build_pt_list_cb(void *priv, const struct VSC_point *vpt)
 }
 
 static void
-build_pt_list(struct VSM_data *vd, struct VSM_fantom *fantom)
+build_pt_list(struct vsm *vd, struct VSM_fantom *fantom)
 {
 	struct pt_priv pt_priv;
 	int i;
@@ -1065,7 +1065,7 @@ handle_keypress(int ch)
 }
 
 void
-do_curses(struct VSM_data *vd, double delay)
+do_curses(struct vsm *vd, double delay)
 {
 	struct pollfd pollfd;
 	long t;
diff --git a/bin/varnishtest/vtc_logexp.c b/bin/varnishtest/vtc_logexp.c
index 45e856e..6b79933 100644
--- a/bin/varnishtest/vtc_logexp.c
+++ b/bin/varnishtest/vtc_logexp.c
@@ -155,7 +155,7 @@ struct logexp {
 	int				g_arg;
 	char				*query;
 
-	struct VSM_data			*vsm;
+	struct vsm			*vsm;
 	struct vsb			*n_arg;
 	struct VSL_data			*vsl;
 	struct VSLQ			*vslq;
diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c
index 82d8444..4cf7edd 100644
--- a/bin/varnishtest/vtc_varnish.c
+++ b/bin/varnishtest/vtc_varnish.c
@@ -76,7 +76,7 @@ struct varnish {
 	char			*jail;
 	char			*proto;
 
-	struct VSM_data		*vd;		/* vsc use */
+	struct vsm		*vd;		/* vsc use */
 
 	unsigned		vsl_tag_count[256];
 
@@ -197,7 +197,7 @@ varnishlog_thread(void *priv)
 {
 	struct varnish *v;
 	struct VSL_data *vsl;
-	struct VSM_data *vsm;
+	struct vsm *vsm;
 	struct VSL_cursor *c;
 	enum VSL_tag_e tag;
 	uint32_t vxid;
diff --git a/include/vapi/vsc.h b/include/vapi/vsc.h
index c53d388..5b08446 100644
--- a/include/vapi/vsc.h
+++ b/include/vapi/vsc.h
@@ -37,14 +37,14 @@
 
 #include "vapi/vsc_int.h"
 
-struct VSM_data;
+struct vsm;
 struct VSM_fantom;
 
 /*---------------------------------------------------------------------
  * VSC level access functions
  */
 
-int VSC_Arg(struct VSM_data *vd, int arg, const char *opt);
+int VSC_Arg(struct vsm *vd, int arg, const char *opt);
 	/*
 	 * Handle standard stat-presenter arguments
 	 * Return:
@@ -53,7 +53,7 @@ int VSC_Arg(struct VSM_data *vd, int arg, const char *opt);
 	 *	 1 Handled.
 	 */
 
-void *VSC_Get(const struct VSM_data *vd, struct VSM_fantom *fantom,
+void *VSC_Get(const struct vsm *vd, struct VSM_fantom *fantom,
     const char *type, const char *ident);
 	/*
 	 * Looks up the given VSC type and identifier. If fantom is
@@ -61,7 +61,7 @@ void *VSC_Get(const struct VSM_data *vd, struct VSM_fantom *fantom,
 	 * VSM_StillValid.
 	 *
 	 * Arguments:
-	 *	vd:	The VSM_data context
+	 *	vd:	The vsm context
 	 *	fantom: Pointer to a fantom. Can be NULL.
 	 *	type:	The type of the counter segment
 	 *	ident:	The identifier of the counter segment
@@ -107,7 +107,7 @@ struct VSC_point {
 
 typedef int VSC_iter_f(void *priv, const struct VSC_point *const pt);
 
-int VSC_Iter(struct VSM_data *vd, struct VSM_fantom *fantom, VSC_iter_f *func,
+int VSC_Iter(struct vsm *vd, struct VSM_fantom *fantom, VSC_iter_f *func,
     void *priv);
 	/*
 	 * Iterate over all statistics counters, calling "func" for
@@ -126,7 +126,7 @@ int VSC_Iter(struct VSM_data *vd, struct VSM_fantom *fantom, VSC_iter_f *func,
 	 * change (child restart, allocations/deallocations)
 	 *
 	 * Arguments:
-	 *	    vd: The VSM_data context
+	 *	    vd: The vsm context
 	 *	fantom: Pointer to a fantom. Can be NULL.
 	 *	  func: The callback function
 	 *	  priv: Passed as argument to func
diff --git a/include/vapi/vsl.h b/include/vapi/vsl.h
index 03d98d7..caa3443 100644
--- a/include/vapi/vsl.h
+++ b/include/vapi/vsl.h
@@ -38,7 +38,7 @@
 
 #include "vapi/vsl_int.h"
 
-struct VSM_data;
+struct vsm;
 
 /*
  * enum VSL_tag_e enumerates the SHM log tags, where the identifiers are
@@ -266,7 +266,7 @@ void VSL_ResetError(struct VSL_data *vsl);
 #define VSL_COPT_TAIL		(1 << 0)
 #define VSL_COPT_BATCH		(1 << 1)
 #define VSL_COPT_TAILSTOP	(1 << 2)
-struct VSL_cursor *VSL_CursorVSM(struct VSL_data *vsl, struct VSM_data *vsm,
+struct VSL_cursor *VSL_CursorVSM(struct VSL_data *vsl, struct vsm *vsm,
     unsigned options);
        /*
 	* Set the cursor pointed to by cursor up as a raw cursor in the
diff --git a/include/vapi/vsm.h b/include/vapi/vsm.h
index bd26bc9..4037c50 100644
--- a/include/vapi/vsm.h
+++ b/include/vapi/vsm.h
@@ -37,7 +37,7 @@
 #define VAPI_VSM_H_INCLUDED
 
 struct VSM_chunk;
-struct VSM_data;
+struct vsm;
 
 /*
  * This structure is used to reference a VSM chunk
@@ -59,36 +59,36 @@ struct VSM_fantom {
  * VSM level access functions
  */
 
-struct VSM_data *VSM_New(void);
+struct vsm *VSM_New(void);
 	/*
 	 * Allocate and initialize a VSL_data handle structure.
 	 * This is the first thing you will have to do, always.
-	 * You can have multiple active VSM_data handles at the same time
+	 * You can have multiple active vsm handles at the same time
 	 * referencing the same or different shared memory files.
 	 * Returns:
 	 *	Pointer to usable VSL_data handle.
 	 *	NULL: malloc failed.
 	 */
 
-void VSM_Delete(struct VSM_data *vd);
+void VSM_Delete(struct vsm *vd);
 	/*
 	 * Close and deallocate all storage and mappings.
 	 * (including any VSC and VSL "sub-classes")
 	 */
 
-const char *VSM_Error(const struct VSM_data *vd);
+const char *VSM_Error(const struct vsm *vd);
 	/*
 	 * Return the latest error message.
 	 */
 
-void VSM_ResetError(struct VSM_data *vd);
+void VSM_ResetError(struct vsm *vd);
 	/*
 	 * Reset any error message.
 	 */
 
 #define VSM_n_USAGE	"[-n varnish_name]"
 
-int VSM_n_Arg(struct VSM_data *vd, const char *n_arg);
+int VSM_n_Arg(struct vsm *vd, const char *n_arg);
 	/*
 	 * Configure which varnishd instance to access.
 	 * Uses hostname if n_arg is NULL or "".
@@ -98,12 +98,12 @@ int VSM_n_Arg(struct VSM_data *vd, const char *n_arg);
 	 *	 <0 on failure, VSM_Error() returns diagnostic string
 	 */
 
-const char *VSM_Name(const struct VSM_data *vd);
+const char *VSM_Name(const struct vsm *vd);
 	/*
 	 * Return the instance name (-i argument to varnishd)
 	 */
 
-int VSM_Open(struct VSM_data *vd);
+int VSM_Open(struct vsm *vd);
 	/*
 	 * Attempt to open and map the VSM file.
 	 *
@@ -112,7 +112,7 @@ int VSM_Open(struct VSM_data *vd);
 	 *	<0 on failure, VSM_Error() returns diagnostic string
 	 */
 
-int VSM_IsOpen(const struct VSM_data *vd);
+int VSM_IsOpen(const struct vsm *vd);
 	/*
 	 * Check if the VSM is open.
 	 *
@@ -121,7 +121,7 @@ int VSM_IsOpen(const struct VSM_data *vd);
 	 *       0: Is closed
 	 */
 
-int VSM_Abandoned(struct VSM_data *vd);
+int VSM_Abandoned(struct vsm *vd);
 	/*
 	 * Find out if the VSM file has been abandoned or closed and should
 	 * be reopened.  This function calls stat(2) and should only be
@@ -133,7 +133,7 @@ int VSM_Abandoned(struct VSM_data *vd);
 	 *	1  VSM abandoned.
 	 */
 
-void VSM_Close(struct VSM_data *vd);
+void VSM_Close(struct vsm *vd);
 	/*
 	 * Close and unmap shared memory, if open. Any reference to
 	 * previously returned memory areas will cause segmentation
@@ -142,15 +142,15 @@ void VSM_Close(struct VSM_data *vd);
 	 */
 
 
-void VSM__iter0(const struct VSM_data *vd, struct VSM_fantom *vf);
-int VSM__itern(const struct VSM_data *vd, struct VSM_fantom *vf);
+void VSM__iter0(const struct vsm *vd, struct VSM_fantom *vf);
+int VSM__itern(const struct vsm *vd, struct VSM_fantom *vf);
 
 #define VSM_FOREACH(vf, vd) \
     for (VSM__iter0((vd), (vf)); VSM__itern((vd), (vf));)
 	/*
 	 * Iterate over all chunks in shared memory
 	 * vf = "struct VSM_fantom *"
-	 * vd = "struct VSM_data *"
+	 * vd = "struct vsm *"
 	 */
 
 struct vsm_valid {
@@ -161,7 +161,7 @@ extern const struct vsm_valid VSM_invalid[];
 extern const struct vsm_valid VSM_valid[];
 extern const struct vsm_valid VSM_similar[];
 
-const struct vsm_valid *VSM_StillValid(const struct VSM_data *vd,
+const struct vsm_valid *VSM_StillValid(const struct vsm *vd,
     struct VSM_fantom *vf);
 	/*
 	 * Check the validity of a previously looked up VSM_fantom.
@@ -193,7 +193,7 @@ const struct vsm_valid *VSM_StillValid(const struct VSM_data *vd,
 	 *   VSM_similar: a fantom with same dimensions exist in same position.
 	 */
 
-int VSM_Get(const struct VSM_data *vd, struct VSM_fantom *vf,
+int VSM_Get(const struct vsm *vd, struct VSM_fantom *vf,
     const char *class, const char *type, const char *ident);
 	/*
 	 * Find a chunk, produce fantom for it.
diff --git a/include/vut.h b/include/vut.h
index fb9b873..981524e 100644
--- a/include/vut.h
+++ b/include/vut.h
@@ -49,7 +49,7 @@ struct VUT {
 
 	/* State */
 	struct VSL_data	*vsl;
-	struct VSM_data	*vsm;
+	struct vsm	*vsm;
 	struct VSLQ	*vslq;
 	struct vpf_fh	*pfh;
 	int		sighup;
diff --git a/lib/libvarnishapi/vsc.c b/lib/libvarnishapi/vsc.c
index 339b588..3e6bf0b 100644
--- a/lib/libvarnishapi/vsc.c
+++ b/lib/libvarnishapi/vsc.c
@@ -109,7 +109,7 @@ static const size_t nlevels =
 /*--------------------------------------------------------------------*/
 
 static struct vsc *
-vsc_setup(struct VSM_data *vd)
+vsc_setup(struct vsm *vd)
 {
 	struct vsc *vsc;
 
@@ -184,7 +184,7 @@ VSC_Delete(struct vsc *vsc)
 /*--------------------------------------------------------------------*/
 
 static int
-vsc_f_arg(struct VSM_data *vd, const char *opt)
+vsc_f_arg(struct vsm *vd, const char *opt)
 {
 	struct vsc *vsc = vsc_setup(vd);
 	struct vsc_sf *sf;
@@ -215,7 +215,7 @@ vsc_f_arg(struct VSM_data *vd, const char *opt)
 /*--------------------------------------------------------------------*/
 
 int
-VSC_Arg(struct VSM_data *vd, int arg, const char *opt)
+VSC_Arg(struct vsm *vd, int arg, const char *opt)
 {
 
 	switch (arg) {
@@ -230,7 +230,7 @@ VSC_Arg(struct VSM_data *vd, int arg, const char *opt)
  */
 
 void *
-VSC_Get(const struct VSM_data *vd, struct VSM_fantom *fantom, const char *type,
+VSC_Get(const struct vsm *vd, struct VSM_fantom *fantom, const char *type,
     const char *ident)
 {
 	struct VSM_fantom f2 = VSM_FANTOM_NULL;
@@ -297,7 +297,7 @@ vsc_add_pt(struct vsc *vsc, const volatile void *ptr,
  */
 
 static void
-vsc_build_vf_list(struct VSM_data *vd)
+vsc_build_vf_list(struct vsm *vd)
 {
 	uint64_t u;
 	struct vsc *vsc = vsc_setup(vd);
@@ -335,7 +335,7 @@ vsc_build_vf_list(struct VSM_data *vd)
 }
 
 static void
-vsc_build_pt_list(struct VSM_data *vd)
+vsc_build_pt_list(struct vsm *vd)
 {
 	struct vsc *vsc = vsc_setup(vd);
 	struct vsc_vf *vf;
@@ -419,7 +419,7 @@ vsc_filter_match_pt(struct vsb *vsb, const struct vsc_sf *sf, const
 }
 
 static void
-vsc_filter_pt_list(struct VSM_data *vd)
+vsc_filter_pt_list(struct vsm *vd)
 {
 	struct vsc *vsc = vsc_setup(vd);
 	struct vsc_pt_head tmplist;
@@ -475,7 +475,7 @@ vsc_filter_pt_list(struct VSM_data *vd)
  */
 
 int
-VSC_Iter(struct VSM_data *vd, struct VSM_fantom *fantom, VSC_iter_f *func,
+VSC_Iter(struct vsm *vd, struct VSM_fantom *fantom, VSC_iter_f *func,
     void *priv)
 {
 	struct vsc *vsc = vsc_setup(vd);
diff --git a/lib/libvarnishapi/vsc_priv.h b/lib/libvarnishapi/vsc_priv.h
index 363f6cc..48e24c6 100644
--- a/lib/libvarnishapi/vsc_priv.h
+++ b/lib/libvarnishapi/vsc_priv.h
@@ -29,8 +29,8 @@
  */
 
 struct vsc;
-struct VSM_data;
+struct vsm;
 
-void VSM_SetVSC(struct VSM_data *, struct vsc *);
-struct vsc *VSM_GetVSC(const struct VSM_data *);
+void VSM_SetVSC(struct vsm *, struct vsc *);
+struct vsc *VSM_GetVSC(const struct vsm *);
 void VSC_Delete(struct vsc *);
diff --git a/lib/libvarnishapi/vsl_cursor.c b/lib/libvarnishapi/vsl_cursor.c
index 1c9b2ba..eb223a2 100644
--- a/lib/libvarnishapi/vsl_cursor.c
+++ b/lib/libvarnishapi/vsl_cursor.c
@@ -63,7 +63,7 @@ struct vslc_vsm {
 
 	unsigned			options;
 
-	struct VSM_data			*vsm;
+	struct vsm			*vsm;
 	struct VSM_fantom		vf;
 
 	const struct VSL_head		*head;
@@ -238,7 +238,7 @@ static const struct vslc_tbl vslc_vsm_tbl = {
 };
 
 struct VSL_cursor *
-VSL_CursorVSM(struct VSL_data *vsl, struct VSM_data *vsm, unsigned options)
+VSL_CursorVSM(struct VSL_data *vsl, struct vsm *vsm, unsigned options)
 {
 	struct vslc_vsm *c;
 	struct VSM_fantom vf;
diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c
index b7b96b2..4740b5e 100644
--- a/lib/libvarnishapi/vsm.c
+++ b/lib/libvarnishapi/vsm.c
@@ -66,7 +66,7 @@ const struct vsm_valid VSM_similar[1] = {{"similar"}};
 
 struct vsc;
 
-struct VSM_data {
+struct vsm {
 	unsigned		magic;
 #define VSM_MAGIC		0x6e3bd69b
 
@@ -90,10 +90,10 @@ struct VSM_data {
 
 /*--------------------------------------------------------------------*/
 
-struct VSM_data *
+struct vsm *
 VSM_New(void)
 {
-	struct VSM_data *vd;
+	struct vsm *vd;
 
 	ALLOC_OBJ(vd, VSM_MAGIC);
 	if (vd == NULL)
@@ -108,7 +108,7 @@ VSM_New(void)
 /*--------------------------------------------------------------------*/
 
 void
-VSM_SetVSC(struct VSM_data *vd, struct vsc *vsc)
+VSM_SetVSC(struct vsm *vd, struct vsc *vsc)
 {
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
 
@@ -116,7 +116,7 @@ VSM_SetVSC(struct VSM_data *vd, struct vsc *vsc)
 }
 
 struct vsc *
-VSM_GetVSC(const struct VSM_data *vd)
+VSM_GetVSC(const struct vsm *vd)
 {
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
 
@@ -126,7 +126,7 @@ VSM_GetVSC(const struct VSM_data *vd)
 /*--------------------------------------------------------------------*/
 
 static int
-vsm_diag(struct VSM_data *vd, const char *fmt, ...)
+vsm_diag(struct vsm *vd, const char *fmt, ...)
 {
 	va_list ap;
 
@@ -146,7 +146,7 @@ vsm_diag(struct VSM_data *vd, const char *fmt, ...)
 /*--------------------------------------------------------------------*/
 
 const char *
-VSM_Error(const struct VSM_data *vd)
+VSM_Error(const struct vsm *vd)
 {
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
@@ -160,7 +160,7 @@ VSM_Error(const struct VSM_data *vd)
 /*--------------------------------------------------------------------*/
 
 void
-VSM_ResetError(struct VSM_data *vd)
+VSM_ResetError(struct vsm *vd)
 {
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
@@ -173,7 +173,7 @@ VSM_ResetError(struct VSM_data *vd)
 /*--------------------------------------------------------------------*/
 
 int
-VSM_n_Arg(struct VSM_data *vd, const char *arg)
+VSM_n_Arg(struct vsm *vd, const char *arg)
 {
 	char *dname = NULL;
 	struct vsb *vsb;
@@ -203,7 +203,7 @@ VSM_n_Arg(struct VSM_data *vd, const char *arg)
 /*--------------------------------------------------------------------*/
 
 const char *
-VSM_Name(const struct VSM_data *vd)
+VSM_Name(const struct vsm *vd)
 {
 	struct VSM_fantom vt;
 
@@ -216,7 +216,7 @@ VSM_Name(const struct VSM_data *vd)
 /*--------------------------------------------------------------------*/
 
 void
-VSM_Delete(struct VSM_data *vd)
+VSM_Delete(struct vsm *vd)
 {
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
@@ -241,7 +241,7 @@ VSM_Delete(struct VSM_data *vd)
 /*--------------------------------------------------------------------*/
 
 int
-VSM_Open(struct VSM_data *vd)
+VSM_Open(struct vsm *vd)
 {
 	int i;
 	struct VSM_head slh;
@@ -310,7 +310,7 @@ VSM_Open(struct VSM_data *vd)
 /*--------------------------------------------------------------------*/
 
 int
-VSM_IsOpen(const struct VSM_data *vd)
+VSM_IsOpen(const struct vsm *vd)
 {
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
@@ -320,7 +320,7 @@ VSM_IsOpen(const struct VSM_data *vd)
 /*--------------------------------------------------------------------*/
 
 void
-VSM_Close(struct VSM_data *vd)
+VSM_Close(struct vsm *vd)
 {
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
@@ -339,7 +339,7 @@ VSM_Close(struct VSM_data *vd)
 /*--------------------------------------------------------------------*/
 
 int
-VSM_Abandoned(struct VSM_data *vd)
+VSM_Abandoned(struct vsm *vd)
 {
 	struct stat st;
 	double now;
@@ -376,7 +376,7 @@ VSM_Abandoned(struct VSM_data *vd)
 /*--------------------------------------------------------------------*/
 
 void
-VSM__iter0(const struct VSM_data *vd, struct VSM_fantom *vf)
+VSM__iter0(const struct vsm *vd, struct VSM_fantom *vf)
 {
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
@@ -386,7 +386,7 @@ VSM__iter0(const struct VSM_data *vd, struct VSM_fantom *vf)
 }
 
 int
-VSM__itern(const struct VSM_data *vd, struct VSM_fantom *vf)
+VSM__itern(const struct vsm *vd, struct VSM_fantom *vf)
 {
 	struct VSM_chunk *c = NULL;
 
@@ -438,7 +438,7 @@ VSM__itern(const struct VSM_data *vd, struct VSM_fantom *vf)
 /*--------------------------------------------------------------------*/
 
 const struct vsm_valid *
-VSM_StillValid(const struct VSM_data *vd, struct VSM_fantom *vf)
+VSM_StillValid(const struct vsm *vd, struct VSM_fantom *vf)
 {
 	struct VSM_fantom f2;
 
@@ -468,7 +468,7 @@ VSM_StillValid(const struct VSM_data *vd, struct VSM_fantom *vf)
 }
 
 int
-VSM_Get(const struct VSM_data *vd, struct VSM_fantom *vf,
+VSM_Get(const struct vsm *vd, struct VSM_fantom *vf,
     const char *class, const char *type, const char *ident)
 {
 



From phk at FreeBSD.org  Thu Jun  8 13:59:06 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 08 Jun 2017 15:59:06 +0200
Subject: [master] 53234f7 Make mapping of VSM segments an explicit request,
 so that a VSL-only program will not need to map in all the VSC
 segments.
Message-ID: <E1dIxxq-0008PB-3t@project.varnish-software.com>


commit 53234f7d3de0b9d1fb9d852ca8da472ccecea7af
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Jun 8 13:57:42 2017 +0000

    Make mapping of VSM segments an explicit request, so that a VSL-only
    program will not need to map in all the VSC segments.

diff --git a/bin/varnishadm/varnishadm.c b/bin/varnishadm/varnishadm.c
index 7e0c15b..465f2d6 100644
--- a/bin/varnishadm/varnishadm.c
+++ b/bin/varnishadm/varnishadm.c
@@ -416,10 +416,12 @@ n_arg_sock(const char *n_arg)
 		VSM_Delete(vsm);
 		return (-1);
 	}
+	AZ(VSM_Map(vsm, &vt));
 	AN(vt.b);
 	T_start = T_arg = strdup(vt.b);
 
 	if (VSM_Get(vsm, &vt, "Arg", "-S", "")) {
+		AZ(VSM_Map(vsm, &vt));
 		AN(vt.b);
 		S_arg = strdup(vt.b);
 	}
diff --git a/include/vapi/vsc.h b/include/vapi/vsc.h
index 5b08446..01427f4 100644
--- a/include/vapi/vsc.h
+++ b/include/vapi/vsc.h
@@ -53,7 +53,7 @@ int VSC_Arg(struct vsm *vd, int arg, const char *opt);
 	 *	 1 Handled.
 	 */
 
-void *VSC_Get(const struct vsm *vd, struct VSM_fantom *fantom,
+void *VSC_Get(struct vsm *vd, struct VSM_fantom *fantom,
     const char *type, const char *ident);
 	/*
 	 * Looks up the given VSC type and identifier. If fantom is
diff --git a/include/vapi/vsm.h b/include/vapi/vsm.h
index 4037c50..d556b4c 100644
--- a/include/vapi/vsm.h
+++ b/include/vapi/vsm.h
@@ -153,6 +153,8 @@ int VSM__itern(const struct vsm *vd, struct VSM_fantom *vf);
 	 * vd = "struct vsm *"
 	 */
 
+int VSM_Map(struct vsm *vd, struct VSM_fantom *vf);
+
 struct vsm_valid {
 	const char *name;
 };
diff --git a/lib/libvarnishapi/libvarnishapi.map b/lib/libvarnishapi/libvarnishapi.map
index 7d76489..01c39f7 100644
--- a/lib/libvarnishapi/libvarnishapi.map
+++ b/lib/libvarnishapi/libvarnishapi.map
@@ -182,4 +182,5 @@ LIBVARNISHAPI_1.7 {
 	VSM_invalid;
 	VSM_valid;
 	VSM_similar;
+	VSM_Map;
 } LIBVARNISHAPI_1.0;
diff --git a/lib/libvarnishapi/vsc.c b/lib/libvarnishapi/vsc.c
index 3e6bf0b..7a85443 100644
--- a/lib/libvarnishapi/vsc.c
+++ b/lib/libvarnishapi/vsc.c
@@ -230,7 +230,7 @@ VSC_Arg(struct vsm *vd, int arg, const char *opt)
  */
 
 void *
-VSC_Get(const struct vsm *vd, struct VSM_fantom *fantom, const char *type,
+VSC_Get(struct vsm *vd, struct VSM_fantom *fantom, const char *type,
     const char *ident)
 {
 	struct VSM_fantom f2 = VSM_FANTOM_NULL;
@@ -240,6 +240,8 @@ VSC_Get(const struct vsm *vd, struct VSM_fantom *fantom, const char *type,
 	if (VSM_invalid == VSM_StillValid(vd, fantom) &&
 	    !VSM_Get(vd, fantom, VSC_CLASS, type, ident))
 		return (NULL);
+	AZ(VSM_Map(vd, fantom));
+	AN(fantom->b);
 	return ((void*)((char*)fantom->b + 8));
 }
 
@@ -313,6 +315,7 @@ vsc_build_vf_list(struct vsm *vd)
 	VSM_FOREACH(&vsc->iter_fantom, vd) {
 		if (strcmp(vsc->iter_fantom.class, VSC_CLASS))
 			continue;
+		AZ(VSM_Map(vd, &vsc->iter_fantom));
 		u = vbe64dec(vsc->iter_fantom.b);
 		assert(u > 0);
 		p = (char*)vsc->iter_fantom.b + 8 + u;
diff --git a/lib/libvarnishapi/vsl_cursor.c b/lib/libvarnishapi/vsl_cursor.c
index eb223a2..b193e44 100644
--- a/lib/libvarnishapi/vsl_cursor.c
+++ b/lib/libvarnishapi/vsl_cursor.c
@@ -252,6 +252,8 @@ VSL_CursorVSM(struct VSL_data *vsl, struct vsm *vsm, unsigned options)
 		    "No VSL chunk found (child not started ?)");
 		return (NULL);
 	}
+	AZ(VSM_Map(vsm, &vf));
+	AN(vf.b);
 
 	head = vf.b;
 	if (memcmp(head->marker, VSL_HEAD_MARKER, sizeof head->marker)) {
diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c
index 4740b5e..1e31533 100644
--- a/lib/libvarnishapi/vsm.c
+++ b/lib/libvarnishapi/vsm.c
@@ -437,6 +437,18 @@ VSM__itern(const struct vsm *vd, struct VSM_fantom *vf)
 
 /*--------------------------------------------------------------------*/
 
+int
+VSM_Map(struct vsm *vd, struct VSM_fantom *vf)
+{
+
+	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
+	AN(vf);
+	AN(vf->b);
+	return (0);
+}
+
+/*--------------------------------------------------------------------*/
+
 const struct vsm_valid *
 VSM_StillValid(const struct vsm *vd, struct VSM_fantom *vf)
 {



From dridi at varni.sh  Thu Jun  8 14:19:01 2017
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Thu, 8 Jun 2017 16:19:01 +0200
Subject: [master] ef56a0d Rename struct "VSM_data" to just "vsm", to
 definitively break the API.
In-Reply-To: <E1dIxhN-0007is-TV@project.varnish-software.com>
References: <E1dIxhN-0007is-TV@project.varnish-software.com>
Message-ID: <CABoVN9B6s5L7GL9Myfa67Vg9HtSX5W3t4Fm9y-KHYH0mY4QBZA@mail.gmail.com>

On Thu, Jun 8, 2017 at 3:42 PM, Poul-Henning Kamp <phk at freebsd.org> wrote:
>
> commit ef56a0d60eb862ca624c80306abfb6b5b6d141d6
> Author: Poul-Henning Kamp <phk at FreeBSD.org>
> Date:   Thu Jun 8 13:41:06 2017 +0000
>
>     Rename struct "VSM_data" to just "vsm", to definitively break the API.

#define VSM_data vsm

Yours Trolly



From phk at phk.freebsd.dk  Thu Jun  8 14:23:42 2017
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 08 Jun 2017 14:23:42 +0000
Subject: [master] ef56a0d Rename struct "VSM_data" to just "vsm",
 to definitively break the API.
In-Reply-To: <CABoVN9B6s5L7GL9Myfa67Vg9HtSX5W3t4Fm9y-KHYH0mY4QBZA@mail.gmail.com>
References: <E1dIxhN-0007is-TV@project.varnish-software.com>
 <CABoVN9B6s5L7GL9Myfa67Vg9HtSX5W3t4Fm9y-KHYH0mY4QBZA@mail.gmail.com>
Message-ID: <53409.1496931822@critter.freebsd.dk>

--------
In message <CABoVN9B6s5L7GL9Myfa67Vg9HtSX5W3t4Fm9y-KHYH0mY4QBZA at mail.gmail.com>
, Dridi Boukelmoune writes:
>On Thu, Jun 8, 2017 at 3:42 PM, Poul-Henning Kamp <phk at freebsd.org> wrote:
>>
>> commit ef56a0d60eb862ca624c80306abfb6b5b6d141d6
>> Author: Poul-Henning Kamp <phk at FreeBSD.org>
>> Date:   Thu Jun 8 13:41:06 2017 +0000
>>
>>     Rename struct "VSM_data" to just "vsm", to definitively break the API.
>
>#define VSM_data vsm

It may quench your compiler, but it won't make your code run...

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From dridi at varni.sh  Thu Jun  8 14:27:16 2017
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Thu, 8 Jun 2017 16:27:16 +0200
Subject: [master] ef56a0d Rename struct "VSM_data" to just "vsm", to
 definitively break the API.
In-Reply-To: <53409.1496931822@critter.freebsd.dk>
References: <E1dIxhN-0007is-TV@project.varnish-software.com>
 <CABoVN9B6s5L7GL9Myfa67Vg9HtSX5W3t4Fm9y-KHYH0mY4QBZA@mail.gmail.com>
 <53409.1496931822@critter.freebsd.dk>
Message-ID: <CABoVN9C8i4tfzDz0JgHEeL=wUSJvpWyMg22_h9+CXaGQQ-15SQ@mail.gmail.com>

>>>     Rename struct "VSM_data" to just "vsm", to definitively break the API.
>>
>>#define VSM_data vsm
>
> It may quench your compiler, but it won't make your code run...

I just couldn't resist the joke :)

Dridi



From phk at FreeBSD.org  Fri Jun  9 05:43:05 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 09 Jun 2017 07:43:05 +0200
Subject: [master] 53ca9ec Remove VSC_Get(),
 it is not useful and it was never exposed in the .map file anyway.
Message-ID: <E1dJChN-0004fL-C1@project.varnish-software.com>


commit 53ca9ec51feb49c3476b017f68f7c2603294b9fc
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Jun 9 05:41:29 2017 +0000

    Remove VSC_Get(), it is not useful and it was never exposed in the
    .map file anyway.

diff --git a/include/vapi/vsc.h b/include/vapi/vsc.h
index 01427f4..3d74531 100644
--- a/include/vapi/vsc.h
+++ b/include/vapi/vsc.h
@@ -53,24 +53,6 @@ int VSC_Arg(struct vsm *vd, int arg, const char *opt);
 	 *	 1 Handled.
 	 */
 
-void *VSC_Get(struct vsm *vd, struct VSM_fantom *fantom,
-    const char *type, const char *ident);
-	/*
-	 * Looks up the given VSC type and identifier. If fantom is
-	 * non-NULL, it can if successful later be used with
-	 * VSM_StillValid.
-	 *
-	 * Arguments:
-	 *	vd:	The vsm context
-	 *	fantom: Pointer to a fantom. Can be NULL.
-	 *	type:	The type of the counter segment
-	 *	ident:	The identifier of the counter segment
-	 *
-	 * Return values:
-	 *  NULL:	Failure
-	 *  non-NULL:	A void pointer to the stats structure.
-	 */
-
 struct VSC_level_desc;
 struct VSC_type_desc;
 struct VSC_section;
diff --git a/lib/libvarnishapi/vsc.c b/lib/libvarnishapi/vsc.c
index 7a85443..0706a39 100644
--- a/lib/libvarnishapi/vsc.c
+++ b/lib/libvarnishapi/vsc.c
@@ -226,25 +226,6 @@ VSC_Arg(struct vsm *vd, int arg, const char *opt)
 	}
 }
 
-/*--------------------------------------------------------------------
- */
-
-void *
-VSC_Get(struct vsm *vd, struct VSM_fantom *fantom, const char *type,
-    const char *ident)
-{
-	struct VSM_fantom f2 = VSM_FANTOM_NULL;
-
-	if (fantom == NULL)
-		fantom = &f2;
-	if (VSM_invalid == VSM_StillValid(vd, fantom) &&
-	    !VSM_Get(vd, fantom, VSC_CLASS, type, ident))
-		return (NULL);
-	AZ(VSM_Map(vd, fantom));
-	AN(fantom->b);
-	return ((void*)((char*)fantom->b + 8));
-}
-
 /*--------------------------------------------------------------------*/
 
 static struct vsc_vf *



From phk at FreeBSD.org  Fri Jun  9 05:58:06 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 09 Jun 2017 07:58:06 +0200
Subject: [master] ac92267 Remove the 'type' argument from VSM_Get()
Message-ID: <E1dJCvu-0005HW-1P@project.varnish-software.com>


commit ac9226747b4f37a048a42b9ca764e6816ee0b32d
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Jun 9 05:57:02 2017 +0000

    Remove the 'type' argument from VSM_Get()

diff --git a/bin/varnishadm/varnishadm.c b/bin/varnishadm/varnishadm.c
index 465f2d6..f9177d0 100644
--- a/bin/varnishadm/varnishadm.c
+++ b/bin/varnishadm/varnishadm.c
@@ -411,7 +411,7 @@ n_arg_sock(const char *n_arg)
 		return (-1);
 	}
 
-	if (!VSM_Get(vsm, &vt, "Arg", "-T", "")) {
+	if (!VSM_Get(vsm, &vt, "Arg", "-T")) {
 		fprintf(stderr, "No -T arg in shared memory\n");
 		VSM_Delete(vsm);
 		return (-1);
@@ -420,7 +420,7 @@ n_arg_sock(const char *n_arg)
 	AN(vt.b);
 	T_start = T_arg = strdup(vt.b);
 
-	if (VSM_Get(vsm, &vt, "Arg", "-S", "")) {
+	if (VSM_Get(vsm, &vt, "Arg", "-S")) {
 		AZ(VSM_Map(vsm, &vt));
 		AN(vt.b);
 		S_arg = strdup(vt.b);
diff --git a/bin/varnishd/mgt/mgt_cli.c b/bin/varnishd/mgt/mgt_cli.c
index 1eb3ce8..3694263 100644
--- a/bin/varnishd/mgt/mgt_cli.c
+++ b/bin/varnishd/mgt/mgt_cli.c
@@ -503,7 +503,7 @@ mgt_cli_secret(const char *S_arg)
 	char buf[BUFSIZ];
 
 	/* Save in shmem */
-	mgt_SHM_static_alloc(S_arg, strlen(S_arg) + 1L, "Arg", "-S", "");
+	mgt_SHM_static_alloc(S_arg, strlen(S_arg) + 1L, "Arg", "", "-S");
 
 	VJ_master(JAIL_MASTER_FILE);
 	fd = open(S_arg, O_RDONLY);
@@ -573,7 +573,7 @@ mgt_cli_telnet(const char *T_arg)
 	if (VSB_len(vsb) == 0)
 		ARGV_ERR("-T %s could not be listened on.\n", T_arg);
 	/* Save in shmem */
-	mgt_SHM_static_alloc(VSB_data(vsb), VSB_len(vsb) + 1, "Arg", "-T", "");
+	mgt_SHM_static_alloc(VSB_data(vsb), VSB_len(vsb) + 1, "Arg", "", "-T");
 	VSB_destroy(&vsb);
 }
 
diff --git a/include/vapi/vsm.h b/include/vapi/vsm.h
index d556b4c..3345371 100644
--- a/include/vapi/vsm.h
+++ b/include/vapi/vsm.h
@@ -196,11 +196,11 @@ const struct vsm_valid *VSM_StillValid(const struct vsm *vd,
 	 */
 
 int VSM_Get(const struct vsm *vd, struct VSM_fantom *vf,
-    const char *class, const char *type, const char *ident);
+    const char *class, const char *ident);
 	/*
 	 * Find a chunk, produce fantom for it.
 	 * Returns zero on failure.
-	 * class is mandatory, type and ident optional.
+	 * class is mandatory, ident optional.
 	 */
 
 #endif /* VAPI_VSM_H_INCLUDED */
diff --git a/lib/libvarnishapi/vsl_cursor.c b/lib/libvarnishapi/vsl_cursor.c
index b193e44..b393fe8 100644
--- a/lib/libvarnishapi/vsl_cursor.c
+++ b/lib/libvarnishapi/vsl_cursor.c
@@ -247,7 +247,7 @@ VSL_CursorVSM(struct VSL_data *vsl, struct vsm *vsm, unsigned options)
 
 	CHECK_OBJ_NOTNULL(vsl, VSL_MAGIC);
 
-	if (!VSM_Get(vsm, &vf, VSL_CLASS, "", "")) {
+	if (!VSM_Get(vsm, &vf, VSL_CLASS, NULL)) {
 		(void)vsl_diag(vsl,
 		    "No VSL chunk found (child not started ?)");
 		return (NULL);
diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c
index 1e31533..ad03d0c 100644
--- a/lib/libvarnishapi/vsm.c
+++ b/lib/libvarnishapi/vsm.c
@@ -208,7 +208,7 @@ VSM_Name(const struct vsm *vd)
 	struct VSM_fantom vt;
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	if (VSM_Get(vd, &vt, "Arg", "-i", ""))
+	if (VSM_Get(vd, &vt, "Arg", "-i"))
 		return (vt.b);
 	return ("");
 }
@@ -481,15 +481,13 @@ VSM_StillValid(const struct vsm *vd, struct VSM_fantom *vf)
 
 int
 VSM_Get(const struct vsm *vd, struct VSM_fantom *vf,
-    const char *class, const char *type, const char *ident)
+    const char *class, const char *ident)
 {
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
 	VSM_FOREACH(vf, vd) {
 		if (strcmp(vf->class, class))
 			continue;
-		if (type != NULL && strcmp(vf->type, type))
-			continue;
 		if (ident != NULL && strcmp(vf->ident, ident))
 			continue;
 		return (1);



From phk at FreeBSD.org  Fri Jun  9 08:13:06 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 09 Jun 2017 10:13:06 +0200
Subject: [master] 4950af8 Stop using VSM_Abandoned() in varnishtest,
 monitor ("Arg", "-i") fanton instead.
Message-ID: <E1dJF2Y-0002NW-4K@project.varnish-software.com>


commit 4950af854afdafb5539e85626bdb476164347557
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Jun 9 08:11:37 2017 +0000

    Stop using VSM_Abandoned() in varnishtest, monitor ("Arg","-i") fanton instead.

diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c
index 4c78e39..e8cd0df 100644
--- a/bin/varnishd/mgt/mgt_main.c
+++ b/bin/varnishd/mgt/mgt_main.c
@@ -832,7 +832,7 @@ main(int argc, char * const *argv)
 
 	mgt_SHM_Init();
 
-	mgt_SHM_static_alloc(i_arg, strlen(i_arg) + 1L, "Arg", "-i", "");
+	mgt_SHM_static_alloc(i_arg, strlen(i_arg) + 1L, "Arg", "-i", "-i");
 
 	if (M_arg != NULL)
 		mgt_cli_master(M_arg);
diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c
index 4cf7edd..e075120 100644
--- a/bin/varnishtest/vtc_varnish.c
+++ b/bin/varnishtest/vtc_varnish.c
@@ -77,6 +77,7 @@ struct varnish {
 	char			*proto;
 
 	struct vsm		*vd;		/* vsc use */
+	struct VSM_fantom	mgt_arg_i;
 
 	unsigned		vsl_tag_count[256];
 
@@ -525,6 +526,7 @@ varnish_launch(struct varnish *v)
 
 	(void)VSM_n_Arg(v->vd, v->workdir);
 	AZ(VSM_Open(v->vd));
+	assert(VSM_Get(v->vd, &v->mgt_arg_i, "Arg", "-i") > 0);
 }
 
 /**********************************************************************
@@ -834,14 +836,14 @@ do_stat_dump_cb(void *priv, const struct VSC_point * const pt)
 }
 
 static void
-varnish_vsc(const struct varnish *v, const char *arg)
+varnish_vsc(struct varnish *v, const char *arg)
 {
 	struct dump_priv dp;
 
 	memset(&dp, 0, sizeof dp);
 	dp.v = v;
 	dp.arg = arg;
-	if (VSM_Abandoned(v->vd)) {
+	if (VSM_StillValid(v->vd, &v->mgt_arg_i) != VSM_valid) {
 		VSM_Close(v->vd);
 		VSM_Open(v->vd);
 	}
@@ -882,7 +884,7 @@ do_stat_cb(void *priv, const struct VSC_point * const pt)
  */
 
 static void
-varnish_expect(const struct varnish *v, char * const *av)
+varnish_expect(struct varnish *v, char * const *av)
 {
 	uint64_t ref;
 	int good;
@@ -914,7 +916,7 @@ varnish_expect(const struct varnish *v, char * const *av)
 	ref = 0;
 	good = 0;
 	for (i = 0; i < 10; i++, (void)usleep(100000)) {
-		if (VSM_Abandoned(v->vd)) {
+		if (VSM_StillValid(v->vd, &v->mgt_arg_i) != VSM_valid) {
 			VSM_Close(v->vd);
 			good = VSM_Open(v->vd);
 		}



From phk at FreeBSD.org  Fri Jun  9 08:38:06 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 09 Jun 2017 10:38:06 +0200
Subject: [master] 7efdc74 Fix the exponential hitrate
Message-ID: <E1dJFQk-0003OY-D2@project.varnish-software.com>


commit 7efdc7465d20abad95f722b9931a62b9308e55a0
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Jun 9 08:27:02 2017 +0000

    Fix the exponential hitrate

diff --git a/bin/varnishstat/varnishstat_curses.c b/bin/varnishstat/varnishstat_curses.c
index f3b7449..49df731 100644
--- a/bin/varnishstat/varnishstat_curses.c
+++ b/bin/varnishstat/varnishstat_curses.c
@@ -104,8 +104,6 @@ static int n_ptarray = 0;
 static struct pt **ptarray = NULL;
 static const volatile uint64_t *mgt_uptime;
 static const volatile uint64_t *main_uptime;
-static const volatile uint64_t *main_hit;
-static const volatile uint64_t *main_miss;
 static const volatile uint64_t *main_cache_hit;
 static const volatile uint64_t *main_cache_miss;
 
@@ -278,17 +276,13 @@ build_pt_list_cb(void *priv, const struct VSC_point *vpt)
 	AZ(strcmp(vpt->desc->ctype, "uint64_t"));
 	bprintf(buf, "%s.%s", vpt->section->ident, vpt->desc->name);
 
-	if (!strcmp(buf, "MGT..uptime"))
+	if (!strcmp(buf, "MGT.uptime"))
 		mgt_uptime = vpt->ptr;
-	if (!strcmp(buf, "MAIN..uptime"))
+	if (!strcmp(buf, "MAIN.uptime"))
 		main_uptime = vpt->ptr;
-	if (!strcmp(buf, "MAIN..hit"))
-		main_hit = vpt->ptr;
-	if (!strcmp(buf, "MAIN..miss"))
-		main_miss = vpt->ptr;
-	if (!strcmp(buf, "MAIN..cache_hit"))
+	if (!strcmp(buf, "MAIN.cache_hit"))
 		main_cache_hit = vpt->ptr;
-	if (!strcmp(buf, "MAIN..cache_miss"))
+	if (!strcmp(buf, "MAIN.cache_miss"))
 		main_cache_miss = vpt->ptr;
 
 	VTAILQ_FOREACH(pt, &ptlist, list) {
@@ -352,8 +346,6 @@ build_pt_list(struct vsm *vd, struct VSM_fantom *fantom)
 
 	mgt_uptime = NULL;
 	main_uptime = NULL;
-	main_hit = NULL;
-	main_miss = NULL;
 	main_cache_hit = NULL;
 	main_cache_miss = NULL;
 
@@ -421,11 +413,11 @@ sample_hitrate(void)
 	double hr, mr, ratio;
 	uint64_t hit, miss;
 
-	if (main_hit == NULL)
+	if (main_cache_hit == NULL)
 		return;
 
-	hit = *main_hit;
-	miss = *main_miss;
+	hit = *main_cache_hit;
+	miss = *main_cache_miss;
 	hr = hit - hitrate.lhit;
 	mr = miss - hitrate.lmiss;
 	hitrate.lhit = hit;



From phk at FreeBSD.org  Fri Jun  9 09:04:05 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 09 Jun 2017 11:04:05 +0200
Subject: [master] f319ed5 Now that we have defined the unit of VSC to be 8
 bytes, simplify.
Message-ID: <E1dJFpt-0004Rd-N7@project.varnish-software.com>


commit f319ed53b5f2bd39b3eb26d5b3e9da518e7e513c
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Jun 9 09:03:08 2017 +0000

    Now that we have defined the unit of VSC to be 8 bytes, simplify.

diff --git a/bin/varnishstat/varnishstat_curses.c b/bin/varnishstat/varnishstat_curses.c
index 49df731..c517bc6 100644
--- a/bin/varnishstat/varnishstat_curses.c
+++ b/bin/varnishstat/varnishstat_curses.c
@@ -75,11 +75,7 @@ struct pt {
 
 	const struct VSC_point	*vpt;
 
-	char			*key;
 	char			*name;
-	int			semantics;
-	int			format;
-	const volatile uint64_t	*ptr;
 
 	char			seen;
 
@@ -287,8 +283,8 @@ build_pt_list_cb(void *priv, const struct VSC_point *vpt)
 
 	VTAILQ_FOREACH(pt, &ptlist, list) {
 		CHECK_OBJ_NOTNULL(pt, PT_MAGIC);
-		AN(pt->key);
-		if (strcmp(buf, pt->key))
+		AN(pt->name);
+		if (strcmp(buf, pt->name))
 			continue;
 		VTAILQ_REMOVE(&ptlist, pt, list);
 		AN(n_ptlist);
@@ -303,19 +299,12 @@ build_pt_list_cb(void *priv, const struct VSC_point *vpt)
 	ALLOC_OBJ(pt, PT_MAGIC);
 	AN(pt);
 
-	pt->key = strdup(buf);
-	AN(pt->key);
-
-	bprintf(buf, "%s.%s", vpt->section->ident, vpt->desc->name);
 	REPLACE(pt->name, buf);
 	AN(pt->name);
 
 	pt->vpt = vpt;
 
-	pt->ptr = vpt->ptr;
-	pt->last = *pt->ptr;
-	pt->semantics = vpt->desc->semantics;
-	pt->format = vpt->desc->format;
+	pt->last = *pt->vpt->ptr;
 
 	pt->ma_10.nmax = 10;
 	pt->ma_100.nmax = 100;
@@ -369,18 +358,20 @@ static void
 sample_points(void)
 {
 	struct pt *pt;
+	uint64_t v;
 
 	VTAILQ_FOREACH(pt, &ptlist, list) {
 		AN(pt->vpt);
-		AN(pt->ptr);
-		if (*pt->ptr == 0 && !pt->seen)
+		AN(pt->vpt->ptr);
+		v = *pt->vpt->ptr;
+		if (v == 0 && !pt->seen)
 			continue;
 		if (!pt->seen) {
 			pt->seen = 1;
 			rebuild = 1;
 		}
 		pt->last = pt->cur;
-		pt->cur = *pt->ptr;
+		pt->cur = v;
 		pt->t_last = pt->t_cur;
 		pt->t_cur = VTIM_mono();
 
@@ -388,12 +379,12 @@ sample_points(void)
 			pt->chg = ((int64_t)pt->cur - (int64_t)pt->last) /
 			    (pt->t_cur - pt->t_last);
 
-		if (pt->semantics == 'g') {
+		if (pt->vpt->desc->semantics == 'g') {
 			pt->avg = 0.;
 			update_ma(&pt->ma_10, (int64_t)pt->cur);
 			update_ma(&pt->ma_100, (int64_t)pt->cur);
 			update_ma(&pt->ma_1000, (int64_t)pt->cur);
-		} else if (pt->semantics == 'c') {
+		} else if (pt->vpt->desc->semantics == 'c') {
 			if (main_uptime != NULL && *main_uptime)
 				pt->avg = pt->cur / *main_uptime;
 			else
@@ -790,7 +781,7 @@ draw_line_bitmap(WINDOW *w, int y, int x, int X, const struct pt *pt)
 
 	AN(w);
 	AN(pt);
-	assert(pt->format == 'b');
+	assert(pt->vpt->desc->format == 'b');
 
 	col = 0;
 	while (col < COL_LAST) {
@@ -866,7 +857,7 @@ draw_line(WINDOW *w, int y, const struct pt *pt)
 		mvwprintw(w, y, x, "%.*s", colw_name, pt->name);
 	x += colw_name;
 
-	switch (pt->format) {
+	switch (pt->vpt->desc->format) {
 	case 'b':
 		draw_line_bitmap(w, y, x, X, pt);
 		break;
diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c
index e075120..7014b8a 100644
--- a/bin/varnishtest/vtc_varnish.c
+++ b/bin/varnishtest/vtc_varnish.c
@@ -822,7 +822,7 @@ do_stat_dump_cb(void *priv, const struct VSC_point * const pt)
 
 	if (strcmp(pt->desc->ctype, "uint64_t"))
 		return (0);
-	u = *(const volatile uint64_t*)pt->ptr;
+	u = *pt->ptr;
 
 	bprintf(buf, "%s.%s", pt->section->ident, pt->desc->name);
 
@@ -876,7 +876,7 @@ do_stat_cb(void *priv, const struct VSC_point * const pt)
 		return(0);
 
 	AZ(strcmp(pt->desc->ctype, "uint64_t"));
-	sp->val = *(const volatile uint64_t*)pt->ptr;
+	sp->val = *pt->ptr;
 	return (1);
 }
 
diff --git a/include/vapi/vsc.h b/include/vapi/vsc.h
index 3d74531..a3d847d 100644
--- a/include/vapi/vsc.h
+++ b/include/vapi/vsc.h
@@ -83,7 +83,7 @@ struct VSC_desc {
 
 struct VSC_point {
 	const struct VSC_desc *desc;	/* point description		*/
-	const volatile void *ptr;	/* field value			*/
+	const volatile uint64_t *ptr;	/* field value			*/
 	const struct VSC_section *section;
 };
 



From phk at FreeBSD.org  Sat Jun 10 22:57:05 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Sun, 11 Jun 2017 00:57:05 +0200
Subject: [master] 975b409 Hardcode the timestamp in the gzip header to get
 reproducible builds.
Message-ID: <E1dJpJZ-0004EO-VO@project.varnish-software.com>


commit 975b40926179165766bbe92eee8b019edbc31b90
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sat Jun 10 22:56:06 2017 +0000

    Hardcode the timestamp in the gzip header to get reproducible builds.

diff --git a/lib/libvcc/vsctool.py b/lib/libvcc/vsctool.py
index 93c7d2d..0b38843 100644
--- a/lib/libvcc/vsctool.py
+++ b/lib/libvcc/vsctool.py
@@ -55,7 +55,7 @@ PARAMS = {
 
 def gzip_str(s):
 	out = StringIO.StringIO()
-	gzip.GzipFile(fileobj=out, mode="w").write(s)
+	gzip.GzipFile(fileobj=out, mode="w", mtime=0x12bfd58).write(s)
 	return out.getvalue()
 
 def genhdr(fo, name):



From nils.goroll at uplex.de  Sun Jun 11 13:02:06 2017
From: nils.goroll at uplex.de (Nils Goroll)
Date: Sun, 11 Jun 2017 15:02:06 +0200
Subject: [master] 57a3042 add some memory barriers in vsc
Message-ID: <E1dK2VK-0004tY-OC@project.varnish-software.com>


commit 57a3042d7ffeeba18eebdb8d717b7bceee93c534
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Sun Jun 11 15:00:45 2017 +0200

    add some memory barriers in vsc
    
    not sure if they improve the situation, just checking

diff --git a/lib/libvarnishapi/vsc.c b/lib/libvarnishapi/vsc.c
index 0706a39..1efb4c4 100644
--- a/lib/libvarnishapi/vsc.c
+++ b/lib/libvarnishapi/vsc.c
@@ -45,6 +45,7 @@
 #include "vjsn.h"
 #include "vsb.h"
 #include "vend.h"
+#include "vmb.h"
 
 #include "vapi/vsc.h"
 #include "vapi/vsm.h"
@@ -272,6 +273,7 @@ vsc_add_pt(struct vsc *vsc, const volatile void *ptr,
 	pt->point.desc = desc;
 	pt->point.ptr = ptr;
 	pt->point.section = &vf->section;
+	VWMB();
 
 	VTAILQ_INSERT_TAIL(&vsc->pt_list, pt, list);
 }
@@ -476,6 +478,7 @@ VSC_Iter(struct vsm *vd, struct VSM_fantom *fantom, VSC_iter_f *func,
 	if (fantom != NULL)
 		*fantom = vsc->iter_fantom;
 	VTAILQ_FOREACH(pt, &vsc->pt_list, list) {
+		VRMB();
 		i = func(priv, &pt->point);
 		if (i)
 			return (i);
diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c
index ad03d0c..7368c2d 100644
--- a/lib/libvarnishapi/vsm.c
+++ b/lib/libvarnishapi/vsm.c
@@ -50,6 +50,7 @@
 #include "vsm_priv.h"
 #include "vsc_priv.h"
 #include "vtim.h"
+#include "vmb.h"
 
 #include "vapi/vsm.h"
 
@@ -112,6 +113,7 @@ VSM_SetVSC(struct vsm *vd, struct vsc *vsc)
 {
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
 
+	VWMB();
 	vd->vsc = vsc;
 }
 
@@ -120,6 +122,7 @@ VSM_GetVSC(const struct vsm *vd)
 {
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
 
+	VRMB();
 	return (vd->vsc);
 }
 



From phk at FreeBSD.org  Sun Jun 11 20:39:05 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Sun, 11 Jun 2017 22:39:05 +0200
Subject: [master] 92f3d07 Try adding a memory barrier on the varnishd side,
 and use the JSON offset as canary on the API side.
Message-ID: <E1dK9dZ-0006KA-Ew@project.varnish-software.com>


commit 92f3d075bc9bb561492d522ffb9e17d8018e0f19
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Jun 11 20:37:57 2017 +0000

    Try adding a memory barrier on the varnishd side, and use the JSON offset
    as canary on the API side.

diff --git a/bin/varnishd/common/common_vsc.c b/bin/varnishd/common/common_vsc.c
index 6157c8c..00a1ab0 100644
--- a/bin/varnishd/common/common_vsc.c
+++ b/bin/varnishd/common/common_vsc.c
@@ -39,6 +39,7 @@
 
 #include "vend.h"
 #include "vgz.h"
+#include "vmb.h"
 #include "vapi/vsc_int.h"
 
 /*--------------------------------------------------------------------*/
@@ -79,7 +80,6 @@ VSC_Alloc(const char *nm, size_t sd,
 	AN(p);
 
 	memset(p, 0, sd);
-	vbe64enc(p, sd);
 
 	memset(&vz, 0, sizeof vz);
 	assert(Z_OK == inflateInit2(&vz, 31));
@@ -94,6 +94,8 @@ VSC_Alloc(const char *nm, size_t sd,
 	vsg->seg = p;
 	vsg->ptr = p + 8;
 	VTAILQ_INSERT_TAIL(&vsc_seglist, vsg, list);
+	VWMB();
+	vbe64enc(p, sd);
 	return (p + 8);
 }
 
diff --git a/lib/libvarnishapi/vsc.c b/lib/libvarnishapi/vsc.c
index 1efb4c4..79b431e 100644
--- a/lib/libvarnishapi/vsc.c
+++ b/lib/libvarnishapi/vsc.c
@@ -37,6 +37,7 @@
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
+#include <unistd.h>
 
 #include "vdef.h"
 #include "vas.h"
@@ -300,6 +301,11 @@ vsc_build_vf_list(struct vsm *vd)
 			continue;
 		AZ(VSM_Map(vd, &vsc->iter_fantom));
 		u = vbe64dec(vsc->iter_fantom.b);
+		if (u == 0) {
+			VRMB();
+			usleep(100000);
+			u = vbe64dec(vsc->iter_fantom.b);
+		}
 		assert(u > 0);
 		p = (char*)vsc->iter_fantom.b + 8 + u;
 		vj = vjsn_parse(p, &e);



From phk at FreeBSD.org  Sun Jun 11 21:46:05 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Sun, 11 Jun 2017 23:46:05 +0200
Subject: [master] fe465df Do a NULL check on the VSC pointer
Message-ID: <E1dKAgP-0000vO-LJ@project.varnish-software.com>


commit fe465df7f94e355eb10d38a033460528846b662f
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Jun 11 21:45:29 2017 +0000

    Do a NULL check on the VSC pointer

diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c
index 7014b8a..114713e 100644
--- a/bin/varnishtest/vtc_varnish.c
+++ b/bin/varnishtest/vtc_varnish.c
@@ -876,6 +876,7 @@ do_stat_cb(void *priv, const struct VSC_point * const pt)
 		return(0);
 
 	AZ(strcmp(pt->desc->ctype, "uint64_t"));
+	AN(pt->ptr);
 	sp->val = *pt->ptr;
 	return (1);
 }



From phk at FreeBSD.org  Sun Jun 11 21:52:06 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Sun, 11 Jun 2017 23:52:06 +0200
Subject: [master] 2c0fed9 Check VSM_Open() return value.
Message-ID: <E1dKAmE-0001Ao-CG@project.varnish-software.com>


commit 2c0fed9ab5ebf235a8143e5dee806b057a4fd281
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Jun 11 21:51:22 2017 +0000

    Check VSM_Open() return value.

diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c
index 114713e..7e41d07 100644
--- a/bin/varnishtest/vtc_varnish.c
+++ b/bin/varnishtest/vtc_varnish.c
@@ -668,7 +668,7 @@ varnish_wait(struct varnish *v)
 
 	if (!vtc_error) {
 		/* Do a backend.list to log if child is still running */
-		varnish_ask_cli(v, "backend.list", &resp);
+		(void)varnish_ask_cli(v, "backend.list", &resp);
 	}
 
 	/* Then stop it */
@@ -845,7 +845,9 @@ varnish_vsc(struct varnish *v, const char *arg)
 	dp.arg = arg;
 	if (VSM_StillValid(v->vd, &v->mgt_arg_i) != VSM_valid) {
 		VSM_Close(v->vd);
-		VSM_Open(v->vd);
+		if (VSM_Open(v->vd) < 0)
+			vtc_fatal(v->vl, "Could not open VSM (%s)",
+			    VSM_Error(v->vd));
 	}
 
 	(void)VSC_Iter(v->vd, NULL, do_stat_dump_cb, &dp);



From fgsch at lodoss.net  Mon Jun 12 02:49:20 2017
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Mon, 12 Jun 2017 04:49:20 +0200
Subject: [4.1] 865cb36 Polish
Message-ID: <E1dKFPs-0004g6-HB@project.varnish-software.com>


commit 865cb36d93f4ec65cd6beff6235227a1ce3ef347
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Sun Nov 27 22:00:51 2016 +0000

    Polish

diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c
index 1e8fa3a..a4cf8be 100644
--- a/bin/varnishncsa/varnishncsa.c
+++ b/bin/varnishncsa/varnishncsa.c
@@ -35,8 +35,6 @@
  * See doc/sphinx/reference/varnishncsa.rst for the supported format
  * specifiers.
  *
- * Note: %r is "%m http://%{Host}i%U%q %H"
- *
  */
 
 #include "config.h"
@@ -69,6 +67,7 @@
 
 #define TIME_FMT "[%d/%b/%Y:%T %z]"
 #define FORMAT "%h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\""
+
 static const char progname[] = "varnishncsa";
 
 struct format;
@@ -178,7 +177,8 @@ openout(int append)
 	AN(CTX.w_arg);
 	CTX.fo = fopen(CTX.w_arg, append ? "a" : "w");
 	if (CTX.fo == NULL)
-		VUT_Error(1, "Can't open output file (%s)", strerror(errno));
+		VUT_Error(1, "Can't open output file (%s)",
+		    strerror(errno));
 }
 
 static int __match_proto__(VUT_cb_f)
@@ -343,7 +343,8 @@ format_time(const struct format *format)
 
 	switch (format->time_type) {
 	case 'D':
-		AZ(VSB_printf(CTX.vsb, "%d", (int)((t_end - t_start) * 1e6)));
+		AZ(VSB_printf(CTX.vsb, "%d",
+		    (int)((t_end - t_start) * 1e6)));
 		break;
 	case 't':
 		AN(format->time_fmt);
@@ -394,7 +395,7 @@ format_auth(const struct format *format)
 		if (format->string == NULL)
 			return (-1);
 		AZ(vsb_esc_cat(CTX.vsb, format->string,
-			format->string + strlen(format->string)));
+		    format->string + strlen(format->string)));
 		return (0);
 	}
 	q = strchr(buf, ':');
@@ -758,7 +759,8 @@ parse_format(const char *format)
 			while (*q && *q != '}')
 				q++;
 			if (!*q)
-				VUT_Error(1, "Unmatched bracket at: %s", p - 2);
+				VUT_Error(1, "Unmatched bracket at: %s",
+				    p - 2);
 			assert(q - p < sizeof buf - 1);
 			strncpy(buf, p, q - p);
 			buf[q - p] = '\0';
@@ -779,13 +781,15 @@ parse_format(const char *format)
 				parse_x_format(buf);
 				break;
 			default:
-				VUT_Error(1, "Unknown format specifier at: %s",
+				VUT_Error(1,
+				    "Unknown format specifier at: %s",
 				    p - 2);
 			}
 			p = q;
 			break;
 		default:
-			VUT_Error(1, "Unknown format specifier at: %s", p - 1);
+			VUT_Error(1, "Unknown format specifier at: %s",
+			    p - 1);
 		}
 	}
 
@@ -800,19 +804,18 @@ parse_format(const char *format)
 }
 
 static int
-isprefix(const char *str, const char *prefix, const char *end,
-    const char **next)
+isprefix(const char *prefix, const char *b, const char *e, const char **next)
 {
 	size_t len;
 
 	len = strlen(prefix);
-	if (end - str < len || strncasecmp(str, prefix, len))
+	if (e - b < len || strncasecmp(b, prefix, len))
 		return (0);
-	str += len;
+	b += len;
 	if (next) {
-		while (str < end && *str && *str == ' ')
-			++str;
-		*next = str;
+		while (b < e && *b && *b == ' ')
+			b++;
+		*next = b;
 	}
 	return (1);
 }
@@ -888,7 +891,7 @@ process_hdr(const struct watch_head *head, const char *b, const char *e)
 	struct watch *w;
 
 	VTAILQ_FOREACH(w, head, list) {
-		if (strncasecmp(b, w->key, w->keylen))
+		if (e - b < w->keylen || strncasecmp(b, w->key, w->keylen))
 			continue;
 		frag_line(1, b + w->keylen, e, &w->frag);
 	}
@@ -988,30 +991,32 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[],
 				break;
 			case (SLT_Timestamp + BACKEND_MARKER):
 			case SLT_Timestamp:
-				if (isprefix(b, "Start:", e, &p)) {
+				if (isprefix("Start:", b, e, &p)) {
 					frag_fields(0, p, e, 1,
 					    &CTX.frag[F_tstart], 0, NULL);
 
-				} else if (isprefix(b, "Resp:", e, &p) ||
-					isprefix(b, "PipeSess:", e, &p) ||
-					isprefix(b, "BerespBody:", e, &p)) {
+				} else if (isprefix("Resp:", b, e, &p) ||
+				    isprefix("PipeSess:", b, e, &p) ||
+				    isprefix("BerespBody:", b, e, &p)) {
 					frag_fields(0, p, e, 1,
 					    &CTX.frag[F_tend], 0, NULL);
 
-				} else if (isprefix(b, "Process:", e, &p) ||
-					isprefix(b, "Pipe:", e, &p) ||
-					isprefix(b, "Beresp:", e, &p)) {
+				} else if (isprefix("Process:", b, e, &p) ||
+				    isprefix("Pipe:", b, e, &p) ||
+				    isprefix("Beresp:", b, e, &p)) {
 					frag_fields(0, p, e, 2,
 					    &CTX.frag[F_ttfb], 0, NULL);
 				}
 				break;
 			case (SLT_BereqHeader + BACKEND_MARKER):
 			case SLT_ReqHeader:
-				if (isprefix(b, "Host:", e, &p))
-					frag_line(0, p, e, &CTX.frag[F_host]);
-				else if (isprefix(b, "Authorization:", e, &p) &&
-				    isprefix(p, "basic ", e, &p))
-					frag_line(0, p, e, &CTX.frag[F_auth]);
+				if (isprefix("Authorization:", b, e, &p) &&
+				    isprefix("basic ", p, e, &p))
+					frag_line(0, p, e,
+					    &CTX.frag[F_auth]);
+				else if (isprefix("Host:", b, e, &p))
+					frag_line(0, p, e,
+					    &CTX.frag[F_host]);
 				break;
 			case (SLT_VCL_call + BACKEND_MARKER):
 				break;
@@ -1039,16 +1044,16 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[],
 			case (SLT_VCL_return + BACKEND_MARKER):
 				break;
 			case SLT_VCL_return:
-				if (!strcasecmp(b, "restart")) {
-					skip = 1;
-				} else if (!strcasecmp(b, "pipe")) {
+				if (!strcasecmp(b, "pipe")) {
 					CTX.hitmiss = "miss";
 					CTX.handling = "pipe";
-				}
+				} else if (!strcasecmp(b, "restart"))
+					skip = 1;
 				break;
 			default:
 				break;
 			}
+
 			if (tag == SLT_VCL_Log) {
 				VTAILQ_FOREACH(w, &CTX.watch_vcl_log, list) {
 					CHECK_OBJ_NOTNULL(w, WATCH_MAGIC);
@@ -1062,19 +1067,19 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[],
 						continue;
 					frag_line(0, p, e, &w->frag);
 				}
-			}
-			if ((tag == SLT_ReqHeader && CTX.c_opt)
-			    || (tag == SLT_BereqHeader && CTX.b_opt))
+			} else if ((tag == SLT_ReqHeader && CTX.c_opt) ||
+			    (tag == SLT_BereqHeader && CTX.b_opt)) {
 				process_hdr(&CTX.watch_reqhdr, b, e);
-			if ((tag == SLT_RespHeader && CTX.c_opt)
-			    || (tag == SLT_BerespHeader && CTX.b_opt))
+			} else if ((tag == SLT_RespHeader && CTX.c_opt) ||
+			    (tag == SLT_BerespHeader && CTX.b_opt))
 				process_hdr(&CTX.watch_resphdr, b, e);
 
 			VTAILQ_FOREACH(vslw, &CTX.watch_vsl, list) {
 				CHECK_OBJ_NOTNULL(vslw, VSL_WATCH_MAGIC);
 				if (tag == vslw->tag) {
 					if (vslw->idx == 0)
-						frag_line(0, b, e, &vslw->frag);
+						frag_line(0, b, e,
+						    &vslw->frag);
 					else
 						frag_fields(0, b, e,
 						    vslw->idx, &vslw->frag,
@@ -1108,7 +1113,8 @@ read_format(const char *formatfile)
 
 	fmtfile = fopen(formatfile, "r");
 	if (fmtfile == NULL)
-		VUT_Error(1, "Can't open format file (%s)", strerror(errno));
+		VUT_Error(1, "Can't open format file (%s)",
+		    strerror(errno));
 	fmtlen = getline(&fmt, &len, fmtfile);
 	if (fmtlen == -1) {
 		free(fmt);



From fgsch at lodoss.net  Mon Jun 12 02:49:20 2017
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Mon, 12 Jun 2017 04:49:20 +0200
Subject: [4.1] 156c72e Further cleanup
Message-ID: <E1dKFPs-0004gC-Ip@project.varnish-software.com>


commit 156c72e843500ef2ff9c087325a68b0c8e52cb6f
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Mon Nov 28 11:36:30 2016 +0000

    Further cleanup

diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c
index a4cf8be..7ecf475 100644
--- a/bin/varnishncsa/varnishncsa.c
+++ b/bin/varnishncsa/varnishncsa.c
@@ -1018,8 +1018,6 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[],
 					frag_line(0, p, e,
 					    &CTX.frag[F_host]);
 				break;
-			case (SLT_VCL_call + BACKEND_MARKER):
-				break;
 			case SLT_VCL_call:
 				if (!strcasecmp(b, "recv")) {
 					CTX.hitmiss = "-";
@@ -1041,8 +1039,6 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[],
 					CTX.handling = "synth";
 				}
 				break;
-			case (SLT_VCL_return + BACKEND_MARKER):
-				break;
 			case SLT_VCL_return:
 				if (!strcasecmp(b, "pipe")) {
 					CTX.hitmiss = "miss";
@@ -1050,14 +1046,12 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[],
 				} else if (!strcasecmp(b, "restart"))
 					skip = 1;
 				break;
-			default:
-				break;
-			}
-
-			if (tag == SLT_VCL_Log) {
+			case (SLT_VCL_Log + BACKEND_MARKER):
+			case SLT_VCL_Log:
 				VTAILQ_FOREACH(w, &CTX.watch_vcl_log, list) {
 					CHECK_OBJ_NOTNULL(w, WATCH_MAGIC);
-					if (strncmp(b, w->key, w->keylen))
+					if (e - b <= w->keylen ||
+					    strncmp(b, w->key, w->keylen))
 						continue;
 					p = b + w->keylen;
 					if (*p != ':')
@@ -1067,10 +1061,15 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[],
 						continue;
 					frag_line(0, p, e, &w->frag);
 				}
-			} else if ((tag == SLT_ReqHeader && CTX.c_opt) ||
-			    (tag == SLT_BereqHeader && CTX.b_opt)) {
+				break;
+			default:
+				break;
+			}
+
+			if ((tag == SLT_ReqHeader && CTX.c_opt) ||
+			    (tag == SLT_BereqHeader && CTX.b_opt))
 				process_hdr(&CTX.watch_reqhdr, b, e);
-			} else if ((tag == SLT_RespHeader && CTX.c_opt) ||
+			else if ((tag == SLT_RespHeader && CTX.c_opt) ||
 			    (tag == SLT_BerespHeader && CTX.b_opt))
 				process_hdr(&CTX.watch_resphdr, b, e);
 



From fgsch at lodoss.net  Mon Jun 12 02:49:20 2017
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Mon, 12 Jun 2017 04:49:20 +0200
Subject: [4.1] b091673 Rework the shell command,
 so that expected exit code and pattern to be found are options to
 "shell". Deprecate err_shell.
Message-ID: <E1dKFPs-0004gL-KF@project.varnish-software.com>


commit b0916736e6fe5211856b5cce673b94f26714a011
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Jan 15 12:35:52 2017 +0000

    Rework the shell command, so that expected exit code and pattern to
    be found are options to "shell".  Deprecate err_shell.
    
    Structure output to always be prefixed by "shell" since we're running
    at the top level.
    
    Also prefix commands with "exec 2>&1 ;" to join stdout+stderr by default.
    This still doesn't cover the shells stderr in case of bad shell syntax,
    which is annoying, but not worth fighting for right now.

diff --git a/bin/varnishtest/vtc.c b/bin/varnishtest/vtc.c
index 4d75686..bd97fb2 100644
--- a/bin/varnishtest/vtc.c
+++ b/bin/varnishtest/vtc.c
@@ -425,27 +425,111 @@ cmd_varnishtest(CMD_ARGS)
 	AZ(av[2]);
 }
 
-/**********************************************************************
- * Shell command execution
+/* SECTION: shell shell
+ *
+ * Pass the string given as argument to a shell. If you have multiple commands
+ * to run, you can use curly barces to describe a multi-lines script, eg::
+ *
+ *         shell {
+ *                 echo begin
+ *                 cat /etc/fstab
+ *                 echo end
+ *         }
+ *
+ * By default a zero exit code is expected.
+ *
+ * Notice that the commandstring is prefixed with "exec 2>&1;" to join
+ * stderr and stdout back to the varnishtest process.
+ *
+ * Optional arguments:
+ *
+ *	-err			- expect non-zero exit code
+ *
+ *	-exit N			- expect exit code N
+ *
+ *	-expect string		- expect str to be found in stdout+err
+ *
+ * The vtc will fail if the return code of the shell is not 0.
+ */
+/* SECTION: client-server.spec.shell shell
+ *
+ * Same as for the top-level shell.
  */
 
 static void
+cmd_shell_engine(struct vtclog *vl, int ok,
+    const char *cmd, const char *expect)
+{
+	struct vsb *vsb;
+	FILE *fp;
+	int r, c;
+
+	AN(vl);
+	AN(cmd);
+	vsb = VSB_new_auto();
+	AN(vsb);
+	VSB_printf(vsb, "exec 2>&1 ; %s", cmd);
+	AZ(VSB_finish(vsb));
+	vtc_dump(vl, 4, "shell_cmd", VSB_data(vsb), -1);
+	fp = popen(VSB_data(vsb), "r");
+	if (fp == NULL)
+		vtc_log(vl, 0, "popen fails: %s", strerror(errno));
+	VSB_clear(vsb);
+	do {
+		c = getc(fp);
+		if (c != EOF)
+			VSB_putc(vsb, c);
+	} while (c != EOF);
+	r = pclose(fp);
+	vtc_log(vl, 4, "shell_status = 0x%04x", WEXITSTATUS(r));
+	if (WIFSIGNALED(r))
+		vtc_log(vl, 4, "shell_signal = %d", WTERMSIG(r));
+	if (ok < 0 && !WEXITSTATUS(r))
+		vtc_log(vl, 0, "shell did not fail as expected");
+	else if (ok >= 0 && WEXITSTATUS(r) != ok) {
+		vtc_log(vl, 0,
+		    "shell_exit not as expected: got 0x%04x wanted 0x%04x",
+			WEXITSTATUS(r), ok);
+	}
+	AZ(VSB_finish(vsb));
+	vtc_dump(vl, 4, "shell_out", VSB_data(vsb), VSB_len(vsb));
+	if (expect != NULL) {
+		if (strstr(VSB_data(vsb), expect) == NULL)
+			vtc_log(vl, 0,
+			    "shell_expect not found: (\"%s\")", expect);
+		else
+			vtc_log(vl, 4, "shell_expect found");
+	}
+	VSB_destroy(&vsb);
+}
+
+
+static void
 cmd_shell(CMD_ARGS)
 {
 	(void)priv;
 	(void)cmd;
-	int r, s;
+	int n;
+	int ok = 0;
+	const char *expect = NULL;
 
 	if (av == NULL)
 		return;
-	AN(av[1]);
-	AZ(av[2]);
-	vtc_dump(vl, 4, "shell", av[1], -1);
-	r = system(av[1]);
-	s = WEXITSTATUS(r);
-	if (s != 0)
-		vtc_log(vl, 0, "CMD '%s' failed with status %d (%s)",
-		    av[1], s, strerror(errno));
+	for (n = 1; av[n] != NULL; n++) {
+		if (!strcmp(av[n], "-err")) {
+			ok = -1;
+		} else if (!strcmp(av[n], "-exit")) {
+			n += 1;
+			ok = atoi(av[n]);
+		} else if (!strcmp(av[n], "-expect")) {
+			n += 1;
+			expect = av[n];
+		} else {
+			break;
+		}
+	}
+	AN(av[n]);
+	cmd_shell_engine(vl, ok, av[n], expect);
 }
 
 /**********************************************************************
@@ -457,43 +541,15 @@ cmd_err_shell(CMD_ARGS)
 {
 	(void)priv;
 	(void)cmd;
-	struct vsb *vsb;
-	FILE *fp;
-	int r, c;
 
 	if (av == NULL)
 		return;
 	AN(av[1]);
 	AN(av[2]);
 	AZ(av[3]);
-	vsb = VSB_new_auto();
-	AN(vsb);
-	vtc_dump(vl, 4, "cmd", av[2], -1);
-	fp = popen(av[2], "r");
-	if (fp == NULL)
-		vtc_log(vl, 0, "popen fails: %s", strerror(errno));
-	do {
-		c = getc(fp);
-		if (c != EOF)
-			VSB_putc(vsb, c);
-	} while (c != EOF);
-	r = pclose(fp);
-	vtc_log(vl, 4, "Status = %d", WEXITSTATUS(r));
-	if (WIFSIGNALED(r))
-		vtc_log(vl, 4, "Signal = %d", WTERMSIG(r));
-	if (WEXITSTATUS(r) == 0) {
-		vtc_log(vl, 0,
-		    "expected error from shell");
-	}
-	AZ(VSB_finish(vsb));
-	vtc_dump(vl, 4, "stdout", VSB_data(vsb), VSB_len(vsb));
-	if (strstr(VSB_data(vsb), av[1]) == NULL)
-		vtc_log(vl, 0,
-		    "Did not find expected string: (\"%s\")", av[1]);
-	else
-		vtc_log(vl, 4,
-		    "Found expected string: (\"%s\")", av[1]);
-	VSB_delete(vsb);
+	vtc_log(vl, 1,
+	    "NOTICE: err_shell is deprecated, use 'shell -err -expect'");
+	cmd_shell_engine(vl, -1, av[2], av[1]);
 }
 
 /**********************************************************************



From fgsch at lodoss.net  Mon Jun 12 02:49:20 2017
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Mon, 12 Jun 2017 04:49:20 +0200
Subject: [4.1] 0dffccf Add regexp matching support to the shell command
Message-ID: <E1dKFPs-0004gU-La@project.varnish-software.com>


commit 0dffccfeac8c6f4986d7aa7008b698d81a534734
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Sat Jan 21 21:22:02 2017 +0000

    Add regexp matching support to the shell command
    
    Change some tests to use the new parameter and more coverage.

diff --git a/bin/varnishtest/tests/u00003.vtc b/bin/varnishtest/tests/u00003.vtc
new file mode 100644
index 0000000..6369e45
--- /dev/null
+++ b/bin/varnishtest/tests/u00003.vtc
@@ -0,0 +1,79 @@
+varnishtest "varnishncsa coverage"
+
+server s1 -repeat 2 {
+	rxreq
+	txresp
+} -start
+
+varnish v1 -vcl+backend {} -start
+
+shell "varnishncsa -n ${v1_name} -D -P ${tmpdir}/ncsa.pid -w ${tmpdir}/ncsa.log"
+
+shell -expect "Usage: varnishncsa <options>" \
+	"varnishncsa -h"
+shell -expect "Copyright (c) 2006 Verdens Gang AS" \
+	"varnishncsa -V"
+shell -err -expect "Missing -w option" \
+	{varnishncsa -D}
+shell -err -expect "Unknown format specifier at: %{foo}A" \
+	{varnishncsa -F "%{foo}A"}
+shell -err -expect "Unknown format specifier at: %A" \
+	{varnishncsa -F "%A"}
+shell -err -expect "Unknown formatting extension: foo" \
+	{varnishncsa -F "%{foo}x"}
+shell -err -expect "Usage: varnishncsa <options>" \
+	{varnishncsa extra}
+shell -err -expect "Missing tag in VSL:" \
+	{varnishncsa -F "%{VSL:}x"}
+shell -err -expect "Unknown log tag: nonexistent" \
+	{varnishncsa -F "%{VSL:nonexistent}x"}
+shell -err -expect "Tag not unique: Req" \
+	{varnishncsa -F "%{VSL:Req}x"}
+shell -err -expect "Unknown log tag: Begin[" \
+	{varnishncsa -F "%{VSL:Begin[}x"}
+shell -err -expect "Syntax error: VSL:Begin]" \
+	{varnishncsa -F "%{VSL:Begin]}x"}
+shell -err -expect "Unknown log tag: Begin[a" \
+	{varnishncsa -F "%{VSL:Begin[a}x"}
+shell -err -expect "Syntax error: VSL:Begin[a]" \
+	{varnishncsa -F "%{VSL:Begin[a]}x"}
+shell -err -expect "Syntax error. Field specifier must be positive: Begin[0]" \
+	{varnishncsa -F "%{VSL:Begin[0]}x"}
+shell -err -expect "Field specifier 999999999999 for the tag VSL:Begin[999999999999] is probably too high" \
+	{varnishncsa -F "%{VSL:Begin[999999999999]}x"}
+shell -err -expect "Can't open format file (No such file or directory)" \
+	{varnishncsa -f /nonexistent/file}
+shell -err -expect "Empty format file" \
+	{varnishncsa -f /dev/null}
+# In Linux and SunOS, getline(3) fails when the stream refers to a
+# directory but happily works in FreeBSD.
+#shell -err -expect "Can't read format from file (Is a directory)" \
+#	{varnishncsa -f ${tmpdir}}
+
+delay 1
+
+client c1 {
+	txreq -url /foo
+	rxresp
+} -run
+
+delay 1
+
+shell "mv ${tmpdir}/ncsa.log ${tmpdir}/ncsa.old.log >/dev/null 2>&1"
+shell "kill -HUP `cat ${tmpdir}/ncsa.pid`"
+
+client c1 {
+	txreq -url /bar
+	rxresp
+} -run
+
+delay 1
+
+shell "kill `cat ${tmpdir}/ncsa.pid`"
+
+shell "grep -q /foo ${tmpdir}/ncsa.old.log"
+shell "grep -q /bar ${tmpdir}/ncsa.log"
+
+shell {echo "%{VSL:Begin}x %{Varnish:vxid}x %D %T %{Varnish:handling}x %{%Z}t" >${tmpdir}/format}
+shell -match "req 1000 rxreq 1001 [0-9]{5} 0 miss [A-Z]{3,}" \
+	{varnishncsa -n ${v1_name} -d -f ${tmpdir}/format}
diff --git a/bin/varnishtest/tests/u00006.vtc b/bin/varnishtest/tests/u00006.vtc
new file mode 100644
index 0000000..cda5d73
--- /dev/null
+++ b/bin/varnishtest/tests/u00006.vtc
@@ -0,0 +1,73 @@
+varnishtest "varnishlog coverage"
+
+server s1 -repeat 2 {
+	rxreq
+	txresp
+} -start
+
+varnish v1 -vcl+backend {} -start
+
+process p1 {
+	exec varnishlog -n ${v1_name} -g raw -k 3 -w ${tmpdir}/vlog -A
+} -start
+shell {
+	exec varnishlog -n ${v1_name} -D -P ${tmpdir}/vlog.pid \
+	    -w ${tmpdir}/vlog.bin
+}
+
+shell -expect "Usage: varnishlog <options>" \
+	"varnishlog -h"
+shell -expect "Copyright (c) 2006 Verdens Gang AS" \
+	"varnishlog -V"
+shell -err -expect "Usage: varnishlog <options>" \
+	"varnishlog extra"
+shell -err -expect "Missing -w option" \
+	"varnishlog -D"
+shell -err -expect "-L: Range error" \
+	"varnishlog -L 0"
+shell -err -expect {-i: "foo" matches zero tags} \
+	"varnishlog -i foo"
+shell -err -expect {-i: "Resp" is ambiguous} \
+	"varnishlog -i Resp"
+shell -err -expect {-I: "foo" matches zero tags} \
+	"varnishlog -I foo:bar"
+shell -err -expect {-I: "Resp" is ambiguous} \
+	"varnishlog -I Resp:bar"
+shell -err -expect {-I: Regex error at position 4 (missing ))} \
+	{varnishlog -I "(foo"}
+shell -err -expect {-x: Syntax error in "**"} \
+	{varnishlog -x "**"}
+shell -err -expect {-X: Syntax error in "**"} \
+	{varnishlog -X "**:bar"}
+
+process p1 -wait
+shell {grep -q "0 CLI" ${tmpdir}/vlog}
+shell -match "0 CLI[ ]+- Wr 200 [0-9]+ PONG" \
+	{varnishlog -n ${v1_name} -d -g raw -X "Wr 200 [0-9]+ [^P]"}
+
+client c1 {
+	txreq -url /foo
+	rxresp
+} -run
+
+delay 1
+
+shell "mv ${tmpdir}/vlog.bin ${tmpdir}/vlog.bin~ >/dev/null 2>&1"
+shell "kill -HUP `cat ${tmpdir}/vlog.pid`"
+
+client c1 {
+	txreq -url /bar
+	rxresp
+} -run
+
+delay 1
+
+shell "kill `cat ${tmpdir}/vlog.pid`"
+
+shell -match {^\*[ ]+<< Request\s+>>[ ]+1001[ ]+
+-[ ]+1001 ReqURL[ ]+c /foo
+$} {
+	varnishlog -v -r ${tmpdir}/vlog.bin~ -i ReqURL -q "RespStatus == 200"
+}
+shell -match "-[ ]+BereqURL[ ]+/bar" \
+	"varnishlog -r ${tmpdir}/vlog.bin -x ReqURL"
diff --git a/bin/varnishtest/vtc.c b/bin/varnishtest/vtc.c
index bd97fb2..ba9a18a 100644
--- a/bin/varnishtest/vtc.c
+++ b/bin/varnishtest/vtc.c
@@ -427,8 +427,9 @@ cmd_varnishtest(CMD_ARGS)
 
 /* SECTION: shell shell
  *
- * Pass the string given as argument to a shell. If you have multiple commands
- * to run, you can use curly barces to describe a multi-lines script, eg::
+ * Pass the string given as argument to a shell. If you have multiple
+ * commands to run, you can use curly barces to describe a multi-lines
+ * script, eg::
  *
  *         shell {
  *                 echo begin
@@ -449,6 +450,7 @@ cmd_varnishtest(CMD_ARGS)
  *
  *	-expect string		- expect str to be found in stdout+err
  *
+ *	-match regexp		- expect regexp to match the stdout+err output
  * The vtc will fail if the return code of the shell is not 0.
  */
 /* SECTION: client-server.spec.shell shell
@@ -457,17 +459,26 @@ cmd_varnishtest(CMD_ARGS)
  */
 
 static void
-cmd_shell_engine(struct vtclog *vl, int ok,
-    const char *cmd, const char *expect)
+cmd_shell_engine(struct vtclog *vl, int ok, const char *cmd,
+    const char *expect, const char *re)
 {
 	struct vsb *vsb;
 	FILE *fp;
+	vre_t *vre = NULL;
+	const char *errptr;
 	int r, c;
+	int err;
 
 	AN(vl);
 	AN(cmd);
 	vsb = VSB_new_auto();
 	AN(vsb);
+	if (re != NULL) {
+		vre = VRE_compile(re, 0, &errptr, &err);
+		if (vre == NULL)
+			vtc_log(vl, 0, "shell_match invalid regexp (\"%s\")",
+			    re);
+	}
 	VSB_printf(vsb, "exec 2>&1 ; %s", cmd);
 	AZ(VSB_finish(vsb));
 	vtc_dump(vl, 4, "shell_cmd", VSB_data(vsb), -1);
@@ -499,6 +510,14 @@ cmd_shell_engine(struct vtclog *vl, int ok,
 			    "shell_expect not found: (\"%s\")", expect);
 		else
 			vtc_log(vl, 4, "shell_expect found");
+	} else if (vre != NULL) {
+		if (VRE_exec(vre, VSB_data(vsb), VSB_len(vsb), 0, 0,
+		    NULL, 0, NULL) < 1)
+			vtc_log(vl, 0,
+			    "shell_match failed: (\"%s\")", re);
+		else
+			vtc_log(vl, 4, "shell_match succeeded");
+		VRE_free(&vre);
 	}
 	VSB_destroy(&vsb);
 }
@@ -507,11 +526,13 @@ cmd_shell_engine(struct vtclog *vl, int ok,
 static void
 cmd_shell(CMD_ARGS)
 {
-	(void)priv;
-	(void)cmd;
+	const char *expect = NULL;
+	const char *re = NULL;
 	int n;
 	int ok = 0;
-	const char *expect = NULL;
+
+	(void)priv;
+	(void)cmd;
 
 	if (av == NULL)
 		return;
@@ -522,14 +543,23 @@ cmd_shell(CMD_ARGS)
 			n += 1;
 			ok = atoi(av[n]);
 		} else if (!strcmp(av[n], "-expect")) {
+			if (re != NULL)
+				vtc_log(vl, 0,
+				    "Cannot use -expect with -match");
 			n += 1;
 			expect = av[n];
+		} else if (!strcmp(av[n], "-match")) {
+			if (expect != NULL)
+				vtc_log(vl, 0,
+				    "Cannot use -match with -expect");
+			n += 1;
+			re = av[n];
 		} else {
 			break;
 		}
 	}
 	AN(av[n]);
-	cmd_shell_engine(vl, ok, av[n], expect);
+	cmd_shell_engine(vl, ok, av[n], expect, re);
 }
 
 /**********************************************************************
@@ -549,7 +579,7 @@ cmd_err_shell(CMD_ARGS)
 	AZ(av[3]);
 	vtc_log(vl, 1,
 	    "NOTICE: err_shell is deprecated, use 'shell -err -expect'");
-	cmd_shell_engine(vl, -1, av[2], av[1]);
+	cmd_shell_engine(vl, -1, av[2], av[1], NULL);
 }
 
 /**********************************************************************



From fgsch at lodoss.net  Mon Jun 12 02:49:20 2017
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Mon, 12 Jun 2017 04:49:20 +0200
Subject: [4.1] 1338e18 Shuffle things around in prep for upcoming work
Message-ID: <E1dKFPs-0004gd-Mq@project.varnish-software.com>


commit 1338e18bc948bf5e5fc89e49ec29571d0f770503
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Fri Apr 14 21:07:13 2017 +0900

    Shuffle things around in prep for upcoming work

diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c
index 7ecf475..a239adf 100644
--- a/bin/varnishncsa/varnishncsa.c
+++ b/bin/varnishncsa/varnishncsa.c
@@ -804,11 +804,10 @@ parse_format(const char *format)
 }
 
 static int
-isprefix(const char *prefix, const char *b, const char *e, const char **next)
+isprefix(const char *prefix, size_t len, const char *b,
+    const char *e, const char **next)
 {
-	size_t len;
-
-	len = strlen(prefix);
+	assert(len > 0);
 	if (e - b < len || strncasecmp(b, prefix, len))
 		return (0);
 	b += len;
@@ -897,6 +896,24 @@ process_hdr(const struct watch_head *head, const char *b, const char *e)
 	}
 }
 
+
+static void
+process_vsl(const struct vsl_watch_head *head, enum VSL_tag_e tag,
+    const char *b, const char *e)
+{
+	struct vsl_watch *w;
+
+	VTAILQ_FOREACH(w, head, list) {
+		CHECK_OBJ_NOTNULL(w, VSL_WATCH_MAGIC);
+		if (tag != w->tag)
+			continue;
+		if (w->idx == 0)
+			frag_line(0, b, e, &w->frag);
+		else
+			frag_fields(0, b, e, w->idx, &w->frag, 0, NULL);
+	}
+}
+
 static int __match_proto__(VSLQ_dispatch_f)
 dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[],
     void *priv)
@@ -905,7 +922,6 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[],
 	unsigned tag;
 	const char *b, *e, *p;
 	struct watch *w;
-	struct vsl_watch *vslw;
 	int i, skip, be_mark;
 	(void)vsl;
 	(void)priv;
@@ -991,32 +1007,34 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[],
 				break;
 			case (SLT_Timestamp + BACKEND_MARKER):
 			case SLT_Timestamp:
-				if (isprefix("Start:", b, e, &p)) {
+#define PREFIX(a, b, c, d)	isprefix(a, strlen(a), b, c, d)
+				if (PREFIX("Start:", b, e, &p)) {
 					frag_fields(0, p, e, 1,
 					    &CTX.frag[F_tstart], 0, NULL);
 
-				} else if (isprefix("Resp:", b, e, &p) ||
-				    isprefix("PipeSess:", b, e, &p) ||
-				    isprefix("BerespBody:", b, e, &p)) {
+				} else if (PREFIX("Resp:", b, e, &p) ||
+				    PREFIX("PipeSess:", b, e, &p) ||
+				    PREFIX("BerespBody:", b, e, &p)) {
 					frag_fields(0, p, e, 1,
 					    &CTX.frag[F_tend], 0, NULL);
 
-				} else if (isprefix("Process:", b, e, &p) ||
-				    isprefix("Pipe:", b, e, &p) ||
-				    isprefix("Beresp:", b, e, &p)) {
+				} else if (PREFIX("Process:", b, e, &p) ||
+				    PREFIX("Pipe:", b, e, &p) ||
+				    PREFIX("Beresp:", b, e, &p)) {
 					frag_fields(0, p, e, 2,
 					    &CTX.frag[F_ttfb], 0, NULL);
 				}
 				break;
 			case (SLT_BereqHeader + BACKEND_MARKER):
 			case SLT_ReqHeader:
-				if (isprefix("Authorization:", b, e, &p) &&
-				    isprefix("basic ", p, e, &p))
+				if (PREFIX("Authorization:", b, e, &p) &&
+				    PREFIX("basic ", p, e, &p))
 					frag_line(0, p, e,
 					    &CTX.frag[F_auth]);
-				else if (isprefix("Host:", b, e, &p))
+				else if (PREFIX("Host:", b, e, &p))
 					frag_line(0, p, e,
 					    &CTX.frag[F_host]);
+#undef PREFIX
 				break;
 			case SLT_VCL_call:
 				if (!strcasecmp(b, "recv")) {
@@ -1073,18 +1091,7 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[],
 			    (tag == SLT_BerespHeader && CTX.b_opt))
 				process_hdr(&CTX.watch_resphdr, b, e);
 
-			VTAILQ_FOREACH(vslw, &CTX.watch_vsl, list) {
-				CHECK_OBJ_NOTNULL(vslw, VSL_WATCH_MAGIC);
-				if (tag == vslw->tag) {
-					if (vslw->idx == 0)
-						frag_line(0, b, e,
-						    &vslw->frag);
-					else
-						frag_fields(0, b, e,
-						    vslw->idx, &vslw->frag,
-						    0, NULL);
-				}
-			}
+			process_vsl(&CTX.watch_vsl, tag, b ,e);
 		}
 		if (skip)
 			continue;



From fgsch at lodoss.net  Mon Jun 12 02:49:20 2017
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Mon, 12 Jun 2017 04:49:20 +0200
Subject: [4.1] ac4f336 Add record-prefix support to varnishncsa
Message-ID: <E1dKFPs-0004gm-OH@project.varnish-software.com>


commit ac4f3360d515e6ebc0a4d08cc3f307270f6b800e
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Fri Apr 14 17:54:27 2017 +0200

    Add record-prefix support to varnishncsa
    
    The parsing still requires some TLC but this does not make it any
    worse so let's get this in first.
    
    Prompted by and ideas from github::xcir via #2077.

diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c
index a239adf..c605141 100644
--- a/bin/varnishncsa/varnishncsa.c
+++ b/bin/varnishncsa/varnishncsa.c
@@ -130,6 +130,8 @@ struct vsl_watch {
 	VTAILQ_ENTRY(vsl_watch)	list;
 	enum VSL_tag_e		tag;
 	int			idx;
+	char			*prefix;
+	int			prefixlen;
 	struct fragment		frag;
 };
 VTAILQ_HEAD(vsl_watch_head, vsl_watch);
@@ -567,7 +569,7 @@ addf_hdr(struct watch_head *head, const char *key, const char *str)
 }
 
 static void
-addf_vsl(enum VSL_tag_e tag, long i)
+addf_vsl(enum VSL_tag_e tag, long i, const char *prefix)
 {
 	struct vsl_watch *w;
 
@@ -576,6 +578,10 @@ addf_vsl(enum VSL_tag_e tag, long i)
 	w->tag = tag;
 	assert(i <= INT_MAX);
 	w->idx = i;
+	if (prefix) {
+		assert(asprintf(&w->prefix, "%s:", prefix) > 0);
+		w->prefixlen = strlen(w->prefix);
+	}
 	VTAILQ_INSERT_TAIL(&CTX.watch_vsl, w, list);
 
 	addf_fragment(&w->frag, "-");
@@ -658,14 +664,23 @@ parse_x_format(char *buf)
 			*r = '\0';
 		} else
 			i = 0;
-		slt = VSL_Name2Tag(buf, -1);
+		r = buf;
+		while (r < e && *r != ':')
+			r++;
+		if (r != e) {
+			slt = VSL_Name2Tag(buf, r - buf);
+			r++;
+		} else {
+			slt = VSL_Name2Tag(buf, -1);
+			r = NULL;
+		}
 		if (slt == -2)
 			VUT_Error(1, "Tag not unique: %s", buf);
 		if (slt == -1)
 			VUT_Error(1, "Unknown log tag: %s", buf);
 		assert(slt >= 0);
 
-		addf_vsl(slt, i);
+		addf_vsl(slt, i, r);
 		return;
 	}
 	VUT_Error(1, "Unknown formatting extension: %s", buf);
@@ -902,15 +917,20 @@ process_vsl(const struct vsl_watch_head *head, enum VSL_tag_e tag,
     const char *b, const char *e)
 {
 	struct vsl_watch *w;
+	const char *p;
 
 	VTAILQ_FOREACH(w, head, list) {
 		CHECK_OBJ_NOTNULL(w, VSL_WATCH_MAGIC);
 		if (tag != w->tag)
 			continue;
+		p = b;
+		if (w->prefixlen > 0 &&
+		    !isprefix(w->prefix, w->prefixlen, b, e, &p))
+			continue;
 		if (w->idx == 0)
-			frag_line(0, b, e, &w->frag);
+			frag_line(0, p, e, &w->frag);
 		else
-			frag_fields(0, b, e, w->idx, &w->frag, 0, NULL);
+			frag_fields(0, p, e, w->idx, &w->frag, 0, NULL);
 	}
 }
 
@@ -1091,7 +1111,7 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[],
 			    (tag == SLT_BerespHeader && CTX.b_opt))
 				process_hdr(&CTX.watch_resphdr, b, e);
 
-			process_vsl(&CTX.watch_vsl, tag, b ,e);
+			process_vsl(&CTX.watch_vsl, tag, b, e);
 		}
 		if (skip)
 			continue;
diff --git a/bin/varnishtest/tests/u00003.vtc b/bin/varnishtest/tests/u00003.vtc
index 6369e45..9c43f62 100644
--- a/bin/varnishtest/tests/u00003.vtc
+++ b/bin/varnishtest/tests/u00003.vtc
@@ -9,6 +9,30 @@ varnish v1 -vcl+backend {} -start
 
 shell "varnishncsa -n ${v1_name} -D -P ${tmpdir}/ncsa.pid -w ${tmpdir}/ncsa.log"
 
+delay 1
+
+client c1 {
+	txreq -url /foo
+	rxresp
+} -run
+
+delay 1
+
+shell "mv ${tmpdir}/ncsa.log ${tmpdir}/ncsa.old.log"
+shell "kill -HUP `cat ${tmpdir}/ncsa.pid`"
+
+client c1 {
+	txreq -url /bar
+	rxresp
+} -run
+
+delay 1
+
+shell "kill `cat ${tmpdir}/ncsa.pid`"
+
+shell "grep -q /foo ${tmpdir}/ncsa.old.log"
+shell "grep -q /bar ${tmpdir}/ncsa.log"
+
 shell -expect "Usage: varnishncsa <options>" \
 	"varnishncsa -h"
 shell -expect "Copyright (c) 2006 Verdens Gang AS" \
@@ -50,30 +74,10 @@ shell -err -expect "Empty format file" \
 #shell -err -expect "Can't read format from file (Is a directory)" \
 #	{varnishncsa -f ${tmpdir}}
 
-delay 1
-
-client c1 {
-	txreq -url /foo
-	rxresp
-} -run
-
-delay 1
-
-shell "mv ${tmpdir}/ncsa.log ${tmpdir}/ncsa.old.log >/dev/null 2>&1"
-shell "kill -HUP `cat ${tmpdir}/ncsa.pid`"
-
-client c1 {
-	txreq -url /bar
-	rxresp
-} -run
-
-delay 1
-
-shell "kill `cat ${tmpdir}/ncsa.pid`"
-
-shell "grep -q /foo ${tmpdir}/ncsa.old.log"
-shell "grep -q /bar ${tmpdir}/ncsa.log"
-
-shell {echo "%{VSL:Begin}x %{Varnish:vxid}x %D %T %{Varnish:handling}x %{%Z}t" >${tmpdir}/format}
-shell -match "req 1000 rxreq 1001 [0-9]{5} 0 miss [A-Z]{3,}" \
+shell {echo "%{VSL:Begin}x %{Varnish:vxid}x %{VCL_Log:foo}x %D %T %{Varnish:handling}x %{%Z}t" >${tmpdir}/format}
+shell -match "^req 1000 rxreq 1001  [0-9]+ 0 miss [A-Z]{3,}" \
 	{varnishncsa -n ${v1_name} -d -f ${tmpdir}/format}
+shell -match "^bereq 1001 fetch 1002  [0-9]+ 0 - [A-Z]{3,}" \
+	{varnishncsa -n ${v1_name} -d -f ${tmpdir}/format -b}
+shell -match "\d{10}\.\d{6} \d\.\d{6} \d\.\d{6} - \d{10}\.\d{6}" \
+	{varnishncsa -n ${v1_name} -d -F "%{VSL:Timestamp:Resp}x %{VSL:Timestamp:foo}x %{VSL:Timestamp:Resp[2]}x"}
diff --git a/doc/sphinx/reference/varnishncsa.rst b/doc/sphinx/reference/varnishncsa.rst
index 3022c92..708989d 100644
--- a/doc/sphinx/reference/varnishncsa.rst
+++ b/doc/sphinx/reference/varnishncsa.rst
@@ -174,13 +174,22 @@ Supported formatters are:
   VCL_Log:key
     Output value set by std.log("key:value") in VCL.
 
-  VSL:tag[field]
-    The value of the VSL entry for the given tag. The field will,
-    if present, treat the log record as a white space separated list
-    of fields, and only the nth part of the record will be matched
-    against. Fields start counting at 1. Defaults to '-' when the tag
-    is not seen, or when the field is out of bounds. If a tag appears
-    multiple times in a single transaction, the first occurrence is used.
+  VSL:tag:record-prefix[field]
+    The value of the VSL entry for the given tag-record prefix-field
+    combination. Tag is mandatory, the other components are optional.
+
+    The record prefix will limit the matches to those records that
+    have this prefix as the first part of the record content followed
+    by a colon.
+
+    The field will, if present, treat the log record as a white
+    space separated list of fields, and only the nth part of the
+    record will be matched against. Fields start counting at 1.
+
+    Defaults to '-' when the tag is not seen, the record prefix
+    does not match or the field is out of bounds. If a tag appears
+    multiple times in a single transaction, the first occurrence
+    is used.
 
 SIGNALS
 =======
@@ -202,11 +211,15 @@ multiple times in a single transaction, the first occurrence is used.
 EXAMPLE
 =======
 
-Log the second field of the Begin tag, corresponding to the VXID of the
-parent transaction::
+Log the second field of the Begin record, corresponding to the VXID
+of the parent transaction::
 
   varnishncsa -F "%{VSL:Begin[2]}x"
 
+Log the entire Timestamp record associated with the processing length::
+
+  varnishncsa -F "%{VSL:Timestamp:Process}x"
+
 SEE ALSO
 ========
 



From fgsch at lodoss.net  Mon Jun 12 02:49:20 2017
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Mon, 12 Jun 2017 04:49:20 +0200
Subject: [4.1] cf5cd43 Polish
Message-ID: <E1dKFPs-0004gv-Pa@project.varnish-software.com>


commit cf5cd43d06074ec75ed0cca51179b6c65c664832
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Sat Apr 15 04:50:24 2017 +0100

    Polish

diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c
index c605141..fbb5470 100644
--- a/bin/varnishncsa/varnishncsa.c
+++ b/bin/varnishncsa/varnishncsa.c
@@ -827,7 +827,7 @@ isprefix(const char *prefix, size_t len, const char *b,
 		return (0);
 	b += len;
 	if (next) {
-		while (b < e && *b && *b == ' ')
+		while (b < e && *b == ' ')
 			b++;
 		*next = b;
 	}
@@ -891,7 +891,7 @@ frag_line(int force, const char *b, const char *e, struct fragment *f)
 		++b;
 
 	/* Skip trailing space */
-	while (e > b && isspace(*(e - 1)))
+	while (e > b && isspace(e[-1]))
 		--e;
 
 	f->gen = CTX.gen;
@@ -911,7 +911,6 @@ process_hdr(const struct watch_head *head, const char *b, const char *e)
 	}
 }
 
-
 static void
 process_vsl(const struct vsl_watch_head *head, enum VSL_tag_e tag,
     const char *b, const char *e)
@@ -1027,34 +1026,34 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[],
 				break;
 			case (SLT_Timestamp + BACKEND_MARKER):
 			case SLT_Timestamp:
-#define PREFIX(a, b, c, d)	isprefix(a, strlen(a), b, c, d)
-				if (PREFIX("Start:", b, e, &p)) {
+#define ISPREFIX(a, b, c, d)	isprefix(a, strlen(a), b, c, d)
+				if (ISPREFIX("Start:", b, e, &p)) {
 					frag_fields(0, p, e, 1,
 					    &CTX.frag[F_tstart], 0, NULL);
 
-				} else if (PREFIX("Resp:", b, e, &p) ||
-				    PREFIX("PipeSess:", b, e, &p) ||
-				    PREFIX("BerespBody:", b, e, &p)) {
+				} else if (ISPREFIX("Resp:", b, e, &p) ||
+				    ISPREFIX("PipeSess:", b, e, &p) ||
+				    ISPREFIX("BerespBody:", b, e, &p)) {
 					frag_fields(0, p, e, 1,
 					    &CTX.frag[F_tend], 0, NULL);
 
-				} else if (PREFIX("Process:", b, e, &p) ||
-				    PREFIX("Pipe:", b, e, &p) ||
-				    PREFIX("Beresp:", b, e, &p)) {
+				} else if (ISPREFIX("Process:", b, e, &p) ||
+				    ISPREFIX("Pipe:", b, e, &p) ||
+				    ISPREFIX("Beresp:", b, e, &p)) {
 					frag_fields(0, p, e, 2,
 					    &CTX.frag[F_ttfb], 0, NULL);
 				}
 				break;
 			case (SLT_BereqHeader + BACKEND_MARKER):
 			case SLT_ReqHeader:
-				if (PREFIX("Authorization:", b, e, &p) &&
-				    PREFIX("basic ", p, e, &p))
+				if (ISPREFIX("Authorization:", b, e, &p) &&
+				    ISPREFIX("basic ", p, e, &p))
 					frag_line(0, p, e,
 					    &CTX.frag[F_auth]);
-				else if (PREFIX("Host:", b, e, &p))
+				else if (ISPREFIX("Host:", b, e, &p))
 					frag_line(0, p, e,
 					    &CTX.frag[F_host]);
-#undef PREFIX
+#undef ISPREFIX
 				break;
 			case SLT_VCL_call:
 				if (!strcasecmp(b, "recv")) {



From fgsch at lodoss.net  Mon Jun 12 02:49:20 2017
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Mon, 12 Jun 2017 04:49:20 +0200
Subject: [4.1] 524b9dc More cleanup
Message-ID: <E1dKFPs-0004h4-Qu@project.varnish-software.com>


commit 524b9dcb659dd33d51f09496d119a46530f5c170
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Sun Apr 16 14:38:06 2017 +0100

    More cleanup

diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c
index fbb5470..905fb24 100644
--- a/bin/varnishncsa/varnishncsa.c
+++ b/bin/varnishncsa/varnishncsa.c
@@ -487,7 +487,7 @@ addf_int32(int32_t *i)
 }
 
 static void
-addf_time(char type, const char *fmt, const char *str)
+addf_time(char type, const char *fmt)
 {
 	struct format *f;
 
@@ -499,10 +499,6 @@ addf_time(char type, const char *fmt, const char *str)
 		f->time_fmt = strdup(fmt);
 		AN(f->time_fmt);
 	}
-	if (str != NULL) {
-		f->string = strdup(str);
-		AN(f->string);
-	}
 	VTAILQ_INSERT_TAIL(&CTX.format, f, list);
 }
 
@@ -526,8 +522,7 @@ addf_vcl_log(const char *key, const char *str)
 	AN(key);
 	ALLOC_OBJ(w, WATCH_MAGIC);
 	AN(w);
-	w->key = strdup(key);
-	AN(w->key);
+	assert(asprintf(&w->key, "%s:", key) > 0);
 	w->keylen = strlen(w->key);
 	VTAILQ_INSERT_TAIL(&CTX.watch_vcl_log, w, list);
 
@@ -727,7 +722,7 @@ parse_format(const char *format)
 			addf_fragment(&CTX.frag[F_b], "-");
 			break;
 		case 'D':	/* Float request time */
-			addf_time(*p, NULL, NULL);
+			addf_time(*p, NULL);
 			break;
 		case 'h':	/* Client host name / IP Address */
 			addf_fragment(&CTX.frag[F_h], "-");
@@ -757,10 +752,10 @@ parse_format(const char *format)
 			addf_fragment(&CTX.frag[F_s], "-");
 			break;
 		case 't':	/* strftime */
-			addf_time(*p, TIME_FMT, NULL);
+			addf_time(*p, TIME_FMT);
 			break;
 		case 'T':	/* Int request time */
-			addf_time(*p, NULL, NULL);
+			addf_time(*p, NULL);
 			break;
 		case 'u':	/* Remote user from auth */
 			addf_auth("-");
@@ -790,7 +785,7 @@ parse_format(const char *format)
 				addf_hdr(&CTX.watch_resphdr, buf, "-");
 				break;
 			case 't':
-				addf_time(*q, buf, NULL);
+				addf_time(*q, buf);
 				break;
 			case 'x':
 				parse_x_format(buf);
@@ -903,11 +898,12 @@ static void
 process_hdr(const struct watch_head *head, const char *b, const char *e)
 {
 	struct watch *w;
+	const char *p;
 
 	VTAILQ_FOREACH(w, head, list) {
-		if (e - b < w->keylen || strncasecmp(b, w->key, w->keylen))
+		if (!isprefix(w->key, w->keylen, b, e, &p))
 			continue;
-		frag_line(1, b + w->keylen, e, &w->frag);
+		frag_line(1, p, e, &w->frag);
 	}
 }
 
@@ -942,6 +938,7 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[],
 	const char *b, *e, *p;
 	struct watch *w;
 	int i, skip, be_mark;
+
 	(void)vsl;
 	(void)priv;
 
@@ -1087,15 +1084,10 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[],
 			case SLT_VCL_Log:
 				VTAILQ_FOREACH(w, &CTX.watch_vcl_log, list) {
 					CHECK_OBJ_NOTNULL(w, WATCH_MAGIC);
-					if (e - b <= w->keylen ||
+					if (e - b < w->keylen ||
 					    strncmp(b, w->key, w->keylen))
 						continue;
 					p = b + w->keylen;
-					if (*p != ':')
-						continue;
-					p++;
-					if (p > e)
-						continue;
 					frag_line(0, p, e, &w->frag);
 				}
 				break;



From fgsch at lodoss.net  Mon Jun 12 02:49:20 2017
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Mon, 12 Jun 2017 04:49:20 +0200
Subject: [4.1] 1543b34 Polish
Message-ID: <E1dKFPs-0004hD-SI@project.varnish-software.com>


commit 1543b34240d63fc99c5c74f7d42e70d01750fa42
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Apr 18 12:44:01 2017 +0200

    Polish

diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c
index 905fb24..53db8c8 100644
--- a/bin/varnishncsa/varnishncsa.c
+++ b/bin/varnishncsa/varnishncsa.c
@@ -118,7 +118,7 @@ struct watch {
 
 	VTAILQ_ENTRY(watch)	list;
 	char			*key;
-	unsigned		keylen;
+	int			keylen;
 	struct fragment		frag;
 };
 VTAILQ_HEAD(watch_head, watch);
@@ -522,8 +522,8 @@ addf_vcl_log(const char *key, const char *str)
 	AN(key);
 	ALLOC_OBJ(w, WATCH_MAGIC);
 	AN(w);
-	assert(asprintf(&w->key, "%s:", key) > 0);
-	w->keylen = strlen(w->key);
+	w->keylen = asprintf(&w->key, "%s:", key);
+	assert(w->keylen > 0);
 	VTAILQ_INSERT_TAIL(&CTX.watch_vcl_log, w, list);
 
 	ALLOC_OBJ(f, FORMAT_MAGIC);
@@ -574,8 +574,8 @@ addf_vsl(enum VSL_tag_e tag, long i, const char *prefix)
 	assert(i <= INT_MAX);
 	w->idx = i;
 	if (prefix) {
-		assert(asprintf(&w->prefix, "%s:", prefix) > 0);
-		w->prefixlen = strlen(w->prefix);
+		w->prefixlen = asprintf(&w->prefix, "%s:", prefix);
+		assert(w->prefixlen > 0);
 	}
 	VTAILQ_INSERT_TAIL(&CTX.watch_vsl, w, list);
 



From fgsch at lodoss.net  Mon Jun 12 02:49:20 2017
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Mon, 12 Jun 2017 04:49:20 +0200
Subject: [4.1] 9f16b5a More cleanup
Message-ID: <E1dKFPs-0004hM-Tg@project.varnish-software.com>


commit 9f16b5ad53d91a82885213520e5bd777ac11cddf
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Wed Apr 19 14:13:24 2017 +0100

    More cleanup

diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c
index 53db8c8..71ef79b 100644
--- a/bin/varnishncsa/varnishncsa.c
+++ b/bin/varnishncsa/varnishncsa.c
@@ -393,7 +393,7 @@ format_auth(const struct format *format)
 
 	if (CTX.frag[F_auth].gen != CTX.gen ||
 	    VB64_decode(buf, sizeof buf, CTX.frag[F_auth].b,
-		CTX.frag[F_auth].e)) {
+	    CTX.frag[F_auth].e)) {
 		if (format->string == NULL)
 			return (-1);
 		AZ(vsb_esc_cat(CTX.vsb, format->string,
@@ -514,7 +514,7 @@ addf_requestline(void)
 }
 
 static void
-addf_vcl_log(const char *key, const char *str)
+addf_vcl_log(const char *key)
 {
 	struct watch *w;
 	struct format *f;
@@ -530,15 +530,13 @@ addf_vcl_log(const char *key, const char *str)
 	AN(f);
 	f->func = &format_fragment;
 	f->frag = &w->frag;
-	if (str != NULL) {
-		f->string = strdup(str);
-		AN(f->string);
-	}
+	f->string = strdup("");
+	AN(f->string);
 	VTAILQ_INSERT_TAIL(&CTX.format, f, list);
 }
 
 static void
-addf_hdr(struct watch_head *head, const char *key, const char *str)
+addf_hdr(struct watch_head *head, const char *key)
 {
 	struct watch *w;
 	struct format *f;
@@ -547,19 +545,16 @@ addf_hdr(struct watch_head *head, const char *key, const char *str)
 	AN(key);
 	ALLOC_OBJ(w, WATCH_MAGIC);
 	AN(w);
-	w->key = strdup(key);
-	AN(w->key);
-	w->keylen = strlen(w->key);
+	w->keylen = asprintf(&w->key, "%s:", key);
+	assert(w->keylen > 0);
 	VTAILQ_INSERT_TAIL(head, w, list);
 
 	ALLOC_OBJ(f, FORMAT_MAGIC);
 	AN(f);
 	f->func = &format_fragment;
 	f->frag = &w->frag;
-	if (str != NULL) {
-		f->string = strdup(str);
-		AN(f->string);
-	}
+	f->string = strdup("-");
+	AN(f->string);
 	VTAILQ_INSERT_TAIL(&CTX.format, f, list);
 }
 
@@ -573,27 +568,24 @@ addf_vsl(enum VSL_tag_e tag, long i, const char *prefix)
 	w->tag = tag;
 	assert(i <= INT_MAX);
 	w->idx = i;
-	if (prefix) {
+	if (prefix != NULL) {
 		w->prefixlen = asprintf(&w->prefix, "%s:", prefix);
 		assert(w->prefixlen > 0);
 	}
 	VTAILQ_INSERT_TAIL(&CTX.watch_vsl, w, list);
-
 	addf_fragment(&w->frag, "-");
 }
 
 static void
-addf_auth(const char *str)
+addf_auth(void)
 {
 	struct format *f;
 
 	ALLOC_OBJ(f, FORMAT_MAGIC);
 	AN(f);
 	f->func = &format_auth;
-	if (str != NULL) {
-		f->string = strdup(str);
-		AN(f->string);
-	}
+	f->string = strdup("-");
+	AN(f->string);
 	VTAILQ_INSERT_TAIL(&CTX.format, f, list);
 }
 
@@ -625,7 +617,7 @@ parse_x_format(char *buf)
 		return;
 	}
 	if (!strncmp(buf, "VCL_Log:", 8)) {
-		addf_vcl_log(buf + 8, "");
+		addf_vcl_log(buf + 8);
 		return;
 	}
 	if (!strncmp(buf, "VSL:", 4)) {
@@ -758,7 +750,7 @@ parse_format(const char *format)
 			addf_time(*p, NULL);
 			break;
 		case 'u':	/* Remote user from auth */
-			addf_auth("-");
+			addf_auth();
 			break;
 		case 'U':	/* URL */
 			addf_fragment(&CTX.frag[F_U], "-");
@@ -777,12 +769,10 @@ parse_format(const char *format)
 			q++;
 			switch (*q) {
 			case 'i':
-				strcat(buf, ":");
-				addf_hdr(&CTX.watch_reqhdr, buf, "-");
+				addf_hdr(&CTX.watch_reqhdr, buf);
 				break;
 			case 'o':
-				strcat(buf, ":");
-				addf_hdr(&CTX.watch_resphdr, buf, "-");
+				addf_hdr(&CTX.watch_resphdr, buf);
 				break;
 			case 't':
 				addf_time(*q, buf);



From nils.goroll at uplex.de  Mon Jun 12 08:00:10 2017
From: nils.goroll at uplex.de (Nils Goroll)
Date: Mon, 12 Jun 2017 10:00:10 +0200
Subject: [master] 1bce362 Revert "add some memory barriers in vsc"
Message-ID: <E1dKKGg-0000rr-Jv@project.varnish-software.com>


commit 1bce36290e7f1867cd408560349ab94f438f3fdb
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Mon Jun 12 09:58:25 2017 +0200

    Revert "add some memory barriers in vsc"
    
    The problem must be elsewhere, vtest results show no improvement
    
    This reverts commit 57a3042d7ffeeba18eebdb8d717b7bceee93c534.

diff --git a/lib/libvarnishapi/vsc.c b/lib/libvarnishapi/vsc.c
index 79b431e..b5c11ad 100644
--- a/lib/libvarnishapi/vsc.c
+++ b/lib/libvarnishapi/vsc.c
@@ -46,7 +46,6 @@
 #include "vjsn.h"
 #include "vsb.h"
 #include "vend.h"
-#include "vmb.h"
 
 #include "vapi/vsc.h"
 #include "vapi/vsm.h"
@@ -274,7 +273,6 @@ vsc_add_pt(struct vsc *vsc, const volatile void *ptr,
 	pt->point.desc = desc;
 	pt->point.ptr = ptr;
 	pt->point.section = &vf->section;
-	VWMB();
 
 	VTAILQ_INSERT_TAIL(&vsc->pt_list, pt, list);
 }
@@ -484,7 +482,6 @@ VSC_Iter(struct vsm *vd, struct VSM_fantom *fantom, VSC_iter_f *func,
 	if (fantom != NULL)
 		*fantom = vsc->iter_fantom;
 	VTAILQ_FOREACH(pt, &vsc->pt_list, list) {
-		VRMB();
 		i = func(priv, &pt->point);
 		if (i)
 			return (i);
diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c
index 7368c2d..ad03d0c 100644
--- a/lib/libvarnishapi/vsm.c
+++ b/lib/libvarnishapi/vsm.c
@@ -50,7 +50,6 @@
 #include "vsm_priv.h"
 #include "vsc_priv.h"
 #include "vtim.h"
-#include "vmb.h"
 
 #include "vapi/vsm.h"
 
@@ -113,7 +112,6 @@ VSM_SetVSC(struct vsm *vd, struct vsc *vsc)
 {
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
 
-	VWMB();
 	vd->vsc = vsc;
 }
 
@@ -122,7 +120,6 @@ VSM_GetVSC(const struct vsm *vd)
 {
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
 
-	VRMB();
 	return (vd->vsc);
 }
 



From nils.goroll at uplex.de  Mon Jun 12 11:56:06 2017
From: nils.goroll at uplex.de (Nils Goroll)
Date: Mon, 12 Jun 2017 13:56:06 +0200
Subject: [master] 514f4db oops - still need the include for other membars
Message-ID: <E1dKNx0-0001xE-9m@project.varnish-software.com>


commit 514f4db8abddd692f7e1ae7aeda009a794b02d90
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Mon Jun 12 13:54:04 2017 +0200

    oops - still need the include for other membars
    
    ... which were added in the meantime

diff --git a/lib/libvarnishapi/vsc.c b/lib/libvarnishapi/vsc.c
index b5c11ad..4cc91f3 100644
--- a/lib/libvarnishapi/vsc.c
+++ b/lib/libvarnishapi/vsc.c
@@ -46,6 +46,7 @@
 #include "vjsn.h"
 #include "vsb.h"
 #include "vend.h"
+#include "vmb.h"
 
 #include "vapi/vsc.h"
 #include "vapi/vsm.h"



From nils.goroll at uplex.de  Mon Jun 12 12:26:06 2017
From: nils.goroll at uplex.de (Nils Goroll)
Date: Mon, 12 Jun 2017 14:26:06 +0200
Subject: [master] 5d372d6 fix remaining stats details
Message-ID: <E1dKOQ2-0003Bj-7j@project.varnish-software.com>


commit 5d372d6fd8d659a912f8957ff68d8b4adfdce241
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Mon Jun 12 14:23:51 2017 +0200

    fix remaining stats details
    
    Most of the work was done in a93c4555ef664ce2835cb43d4cccddd188a27ecf, this
    commit fixes remaining regressions compared with 5.1.2 release
    
    Fixes #2340

diff --git a/bin/varnishd/main.vsc b/bin/varnishd/main.vsc
index 97ca0a2..962e4c8 100644
--- a/bin/varnishd/main.vsc
+++ b/bin/varnishd/main.vsc
@@ -327,36 +327,44 @@
 
 .. varnish_vsc:: s_req_hdrbytes
 	:oneliner:	Request header bytes
+	:format:	bytes
 
 	Total request header bytes received
 
 .. varnish_vsc:: s_req_bodybytes
 	:oneliner:	Request body bytes
+	:format:	bytes
 
 	Total request body bytes received
 
 .. varnish_vsc:: s_resp_hdrbytes
 	:oneliner:	Response header bytes
+	:format:	bytes
 
 	Total response header bytes transmitted
 
 .. varnish_vsc:: s_resp_bodybytes
 	:oneliner:	Response body bytes
+	:format:	bytes
 
 	Total response body bytes transmitted
+	:format:	bytes
 
 .. varnish_vsc:: s_pipe_hdrbytes
 	:oneliner:	Pipe request header bytes
+	:format:	bytes
 
 	Total request bytes received for piped sessions
 
 .. varnish_vsc:: s_pipe_in
 	:oneliner:	Piped bytes from client
+	:format:	bytes
 
 	Total number of bytes forwarded from clients in pipe sessions
 
 .. varnish_vsc:: s_pipe_out
 	:oneliner:	Piped bytes to client
+	:format:	bytes
 
 	Total number of bytes forwarded to clients in pipe sessions
 
@@ -636,6 +644,7 @@
 
 .. varnish_vsc:: bans_persisted_bytes
 	:type:	gauge
+	:format:	bytes
 	:level:	diag
 	:oneliner:	Bytes used by the persisted ban lists
 
@@ -643,6 +652,7 @@
 
 .. varnish_vsc:: bans_persisted_fragmentation
 	:type:	gauge
+	:format:	bytes
 	:level:	diag
 	:oneliner:	Extra bytes in persisted ban lists due to fragmentation
 
@@ -718,6 +728,7 @@
 
 .. varnish_vsc:: vsm_free
 	:type:	gauge
+	:format:	bytes
 	:level:	diag
 	:oneliner:	Free VSM space
 
@@ -726,6 +737,7 @@
 
 .. varnish_vsc:: vsm_used
 	:type:	gauge
+	:format:	bytes
 	:level:	diag
 	:oneliner:	Used VSM space
 
@@ -734,6 +746,7 @@
 
 .. varnish_vsc:: vsm_cooling
 	:type:	gauge
+	:format:	bytes
 	:level:	debug
 	:oneliner:	Cooling VSM space
 
@@ -743,6 +756,7 @@
 
 .. varnish_vsc:: vsm_overflow
 	:type:	gauge
+	:format:	bytes
 	:level:	diag
 	:oneliner:	Overflow VSM space
 
@@ -753,6 +767,7 @@
 
 .. varnish_vsc:: vsm_overflowed
 	:level:	diag
+	:format:	bytes
 	:oneliner:	Overflowed VSM space
 
 	Total number of bytes which did not fit in the shared memory used
diff --git a/bin/varnishd/mgt.vsc b/bin/varnishd/mgt.vsc
index 6aee001..d766e99 100644
--- a/bin/varnishd/mgt.vsc
+++ b/bin/varnishd/mgt.vsc
@@ -11,6 +11,7 @@
 
 .. varnish_vsc:: uptime
 	:type:	counter
+	:format:	duration
 	:level:	info
 	:oneliner:	Management process uptime
 
diff --git a/bin/varnishd/vbe.vsc b/bin/varnishd/vbe.vsc
index 6ce7ee1..fee72bf 100644
--- a/bin/varnishd/vbe.vsc
+++ b/bin/varnishd/vbe.vsc
@@ -11,6 +11,7 @@
 
 .. varnish_vsc:: happy
 	:type:	bitmap
+	:format:	bitmap
 	:level:	info
 	:oneliner:	Happy health probes
 
diff --git a/lib/libvcc/vsctool.py b/lib/libvcc/vsctool.py
index 0b38843..2d39f7b 100644
--- a/lib/libvcc/vsctool.py
+++ b/lib/libvcc/vsctool.py
@@ -43,7 +43,7 @@ import collections
 TYPES = [ "counter", "gauge", "bitmap" ]
 CTYPES = [ "uint64_t" ]
 LEVELS = [ "info", "diag", "debug" ]
-FORMATS = [ "integer", "bytes", "duration" ]
+FORMATS = [ "integer", "bytes", "bitmap", "duration" ]
 
 PARAMS = {
 	"type":		["counter", TYPES],



From phk at FreeBSD.org  Mon Jun 12 14:35:10 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 12 Jun 2017 16:35:10 +0200
Subject: [master] b9dc76c Try to force remapping to different addresses to
 expose bugs about that.
Message-ID: <E1dKQQw-0008O5-Es@project.varnish-software.com>


commit b9dc76cfdfaa7d336a6a84c1db80a3523256ab7a
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Jun 12 14:33:09 2017 +0000

    Try to force remapping to different addresses to expose bugs about that.

diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c
index ad03d0c..425e0f7 100644
--- a/lib/libvarnishapi/vsm.c
+++ b/lib/libvarnishapi/vsm.c
@@ -240,6 +240,8 @@ VSM_Delete(struct vsm *vd)
 
 /*--------------------------------------------------------------------*/
 
+static void *ppp;
+
 int
 VSM_Open(struct vsm *vd)
 {
@@ -291,13 +293,21 @@ VSM_Open(struct vsm *vd)
 		    vd->iname));
 	}
 
-	v = mmap(NULL, slh.shm_size,
+	v = mmap(ppp, slh.shm_size,
 	    PROT_READ, MAP_SHARED|MAP_HASSEMAPHORE, vd->vsm_fd, 0);
+	if (v == MAP_FAILED)
+		v = mmap(NULL, slh.shm_size,
+		    PROT_READ, MAP_SHARED|MAP_HASSEMAPHORE, vd->vsm_fd, 0);
 	if (v == MAP_FAILED) {
 		closefd(&vd->vsm_fd);
 		return (vsm_diag(vd, "Cannot mmap %s: %s",
 		    vd->iname, strerror(errno)));
 	}
+	/*
+	 * Force failure of client depends on remapping at same address. 
+	 */
+	ppp = (char*)v + getpagesize() * 1000;
+
 	vd->head = v;
 	vd->b = v;
 	vd->e = vd->b + slh.shm_size;



From phk at FreeBSD.org  Mon Jun 12 14:35:10 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 12 Jun 2017 16:35:10 +0200
Subject: [master] b262392 Force reload of VSC segments until we know what the
 remapping bug is or replace the code wholesale, whichever comes first.
Message-ID: <E1dKQQw-0008OB-Gr@project.varnish-software.com>


commit b2623920d6a4aac53163e9ec1e5af97303f80107
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Jun 12 14:33:27 2017 +0000

    Force reload of VSC segments until we know what the remapping bug is
    or replace the code wholesale, whichever comes first.

diff --git a/lib/libvarnishapi/vsc.c b/lib/libvarnishapi/vsc.c
index 4cc91f3..737af89 100644
--- a/lib/libvarnishapi/vsc.c
+++ b/lib/libvarnishapi/vsc.c
@@ -473,7 +473,8 @@ VSC_Iter(struct vsm *vd, struct VSM_fantom *fantom, VSC_iter_f *func,
 	struct vsc_pt *pt;
 	int i;
 
-	if (VSM_valid != VSM_StillValid(vd, &vsc->iter_fantom)) {
+	/* XXX: workaround: Force reload */
+	if (1 || VSM_valid != VSM_StillValid(vd, &vsc->iter_fantom)) {
 		/* Tell app that list will be nuked */
 		(void)func(priv, NULL);
 		vsc_build_vf_list(vd);



From phk at FreeBSD.org  Wed Jun 14 21:12:11 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 14 Jun 2017 23:12:11 +0200
Subject: [master] aa8f6a1 Copy&Paste error: Also check for lower-case 'http'
 in protocol.
Message-ID: <E1dLFaF-0007kF-EM@project.varnish-software.com>


commit aa8f6a1b9828677a35919fe447da7c2622064592
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Jun 14 21:11:00 2017 +0000

    Copy&Paste error:  Also check for lower-case 'http' in protocol.
    
    Spotted by:	@jambon69

diff --git a/bin/varnishd/cache/cache_http.c b/bin/varnishd/cache/cache_http.c
index 62d98bd..9be4805 100644
--- a/bin/varnishd/cache/cache_http.c
+++ b/bin/varnishd/cache/cache_http.c
@@ -213,9 +213,9 @@ http_Proto(struct http *to)
 	fm = to->hd[HTTP_HDR_PROTO].b;
 
 	if ((fm[0] == 'H' || fm[0] == 'h') &&
-	    (fm[1] == 'T' || fm[0] == 't') &&
-	    (fm[2] == 'T' || fm[0] == 't') &&
-	    (fm[3] == 'P' || fm[0] == 'p') &&
+	    (fm[1] == 'T' || fm[1] == 't') &&
+	    (fm[2] == 'T' || fm[2] == 't') &&
+	    (fm[3] == 'P' || fm[3] == 'p') &&
 	    fm[4] == '/' &&
 	    vct_isdigit(fm[5]) &&
 	    fm[6] == '.' &&



From hermunn at varnish-software.com  Thu Jun 15 12:08:11 2017
From: hermunn at varnish-software.com (Pål Hermunn Johansen)
Date: Thu, 15 Jun 2017 14:08:11 +0200
Subject: [4.1] 456ee02 add WS_Assert_Allocated: assert that ws->s <= a memory
 region < ws->f
Message-ID: <E1dLTZL-0002K6-AJ@project.varnish-software.com>


commit 456ee023f117b1122d1f44ad0cf18930440af22c
Author: Geoff Simmons <geoff at uplex.de>
Date:   Mon Feb 20 11:38:05 2017 +0100

    add WS_Assert_Allocated: assert that ws->s <= a memory region < ws->f
    
    Conflicts:
    	bin/varnishd/cache/cache.h
    	bin/varnishd/cache/cache_ws.c

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index c035703..e63fdf0 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -1047,6 +1047,7 @@ void *WS_Copy(struct ws *ws, const void *str, int len);
 char *WS_Snapshot(struct ws *ws);
 int WS_Overflowed(const struct ws *ws);
 void *WS_Printf(struct ws *ws, const char *fmt, ...) __v_printflike(2, 3);
+void WS_Assert_Allocated(const struct ws *ws, const void *ptr, ssize_t len);
 
 /* cache_rfc2616.c */
 void RFC2616_Ttl(struct busyobj *, double now);
diff --git a/bin/varnishd/cache/cache_ws.c b/bin/varnishd/cache/cache_ws.c
index f9402d3..3c44599 100644
--- a/bin/varnishd/cache/cache_ws.c
+++ b/bin/varnishd/cache/cache_ws.c
@@ -59,6 +59,17 @@ WS_Assert(const struct ws *ws)
 	assert(*ws->e == 0x15);
 }
 
+void
+WS_Assert_Allocated(const struct ws *ws, const void *ptr, ssize_t len)
+{
+	const char *p = ptr;
+
+	WS_Assert(ws);
+	if (len < 0)
+		len = strlen(p) + 1;
+	assert(p >= ws->s && (p + len) < ws->f);
+}
+
 /*
  * NB: The id must be max 3 char and lower-case.
  * (upper-case the first char to indicate overflow)
@@ -84,7 +95,6 @@ WS_Init(struct ws *ws, const char *id, void *space, unsigned len)
 	WS_Assert(ws);
 }
 
-
 void
 WS_MarkOverflow(struct ws *ws)
 {



From hermunn at varnish-software.com  Thu Jun 15 12:08:11 2017
From: hermunn at varnish-software.com (Pål Hermunn Johansen)
Date: Thu, 15 Jun 2017 14:08:11 +0200
Subject: [4.1] 68f15e0 Fix an off-by-one error in WS_Assert_Allocated().
Message-ID: <E1dLTZL-0002KC-CD@project.varnish-software.com>


commit 68f15e0e402958fcb7e509103912db33317ee7af
Author: Geoff Simmons <geoff at uplex.de>
Date:   Tue May 16 14:22:14 2017 +0200

    Fix an off-by-one error in WS_Assert_Allocated().

diff --git a/bin/varnishd/cache/cache_ws.c b/bin/varnishd/cache/cache_ws.c
index 3c44599..b0b3712 100644
--- a/bin/varnishd/cache/cache_ws.c
+++ b/bin/varnishd/cache/cache_ws.c
@@ -67,7 +67,7 @@ WS_Assert_Allocated(const struct ws *ws, const void *ptr, ssize_t len)
 	WS_Assert(ws);
 	if (len < 0)
 		len = strlen(p) + 1;
-	assert(p >= ws->s && (p + len) < ws->f);
+	assert(p >= ws->s && (p + len) <= ws->f);
 }
 
 /*



From martin at varnish-software.com  Thu Jun 15 12:45:06 2017
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Thu, 15 Jun 2017 14:45:06 +0200
Subject: [4.1] 365e605 Make param::nuke_limit a total count of nukes allowed
 for each object creation.
Message-ID: <E1dLU94-0003tF-I2@project.varnish-software.com>


commit 365e605627d2b383bcd440dcc0379ae503536a05
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 27 09:41:37 2017 +0000

    Make param::nuke_limit a total count of nukes allowed for each
    object creation.
    
    Fixes #1764
    
    Conflicts:
    	bin/varnishd/cache/cache_fetch.c
    	bin/varnishd/storage/stevedore.c
    	bin/varnishd/storage/storage.h
    	bin/varnishd/storage/storage_lru.c
    	bin/varnishd/storage/storage_persistent.c
    	bin/varnishd/storage/storage_simple.c

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index e63fdf0..7dfeff4 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -343,6 +343,7 @@ struct worker {
 	struct pool_task	task;
 
 	double			lastused;
+	int			strangelove;
 
 	struct v1l		*v1l;
 
diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c
index 5c27d07..8a7c46b 100644
--- a/bin/varnishd/cache/cache_expire.c
+++ b/bin/varnishd/cache/cache_expire.c
@@ -329,6 +329,10 @@ EXP_NukeOne(struct worker *wrk, struct lru *lru)
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
+	if (wrk->strangelove-- <= 0) {
+		VSLb(wrk->vsl, SLT_ExpKill, "LRU_Exhausted");
+		return (0);
+	}
 	/* Find the first currently unused object on the LRU.  */
 	Lck_Lock(&lru->mtx);
 	VTAILQ_FOREACH_SAFE(oc, &lru->lru_head, lru_list, oc2) {
diff --git a/bin/varnishd/cache/cache_obj.c b/bin/varnishd/cache/cache_obj.c
index edfd797..3c2eb01 100644
--- a/bin/varnishd/cache/cache_obj.c
+++ b/bin/varnishd/cache/cache_obj.c
@@ -239,7 +239,6 @@ objallocwithnuke(struct worker *wrk, const struct stevedore *stv,
     size_t size, int flags)
 {
 	struct storage *st = NULL;
-	unsigned fail;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(stv, STEVEDORE_MAGIC);
@@ -252,18 +251,11 @@ objallocwithnuke(struct worker *wrk, const struct stevedore *stv,
 
 	assert(size <= UINT_MAX);	/* field limit in struct storage */
 
-	for (fail = 0; fail <= cache_param->nuke_limit; fail++) {
-		/* try to allocate from it */
-		AN(stv->alloc);
+	AN(stv->alloc);
+	do {
 		st = STV_alloc(stv, size, flags);
-		if (st != NULL)
-			break;
+	} while (st == NULL && EXP_NukeOne(wrk, stv->lru) == 1);
 
-		/* no luck; try to free some space and keep trying */
-		if (fail < cache_param->nuke_limit &&
-		    EXP_NukeOne(wrk, stv->lru) == -1)
-			break;
-	}
 	CHECK_OBJ_ORNULL(st, STORAGE_MAGIC);
 	return (st);
 }
diff --git a/bin/varnishd/storage/stevedore.c b/bin/varnishd/storage/stevedore.c
index 4845dba..d53b845 100644
--- a/bin/varnishd/storage/stevedore.c
+++ b/bin/varnishd/storage/stevedore.c
@@ -259,13 +259,15 @@ STV_NewObject(struct objcore *oc, struct worker *wrk,
     const char *hint, unsigned wsl)
 {
 	struct stevedore *stv, *stv0;
-	int i, j;
+	int j;
 
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	assert(wsl > 0);
 
+	wrk->strangelove = cache_param->nuke_limit;
 	stv = stv0 = stv_pick_stevedore(wrk->vsl, &hint);
+	AN(stv);
 	AN(stv->allocobj);
 	j = stv->allocobj(stv, oc, wsl);
 	if (j == 0 && hint == NULL) {
@@ -275,15 +277,12 @@ STV_NewObject(struct objcore *oc, struct worker *wrk,
 			j = stv->allocobj(stv, oc, wsl);
 		} while (j == 0 && stv != stv0);
 	}
-	if (j == 0) {
+	while (j == 0) {
 		/* no luck; try to free some space and keep trying */
-		for (i = 0; j == 0 && i < cache_param->nuke_limit; i++) {
-			if (EXP_NukeOne(wrk, stv->lru) == -1)
-				break;
-			j = stv->allocobj(stv, oc, wsl);
-		}
+		if (EXP_NukeOne(wrk, stv->lru) != 1)
+			break;
+		j = stv->allocobj(stv, oc, wsl);
 	}
-
 	if (j == 0)
 		return (0);
 
diff --git a/bin/varnishtest/tests/r01764.vtc b/bin/varnishtest/tests/r01764.vtc
new file mode 100644
index 0000000..ba428ef
--- /dev/null
+++ b/bin/varnishtest/tests/r01764.vtc
@@ -0,0 +1,53 @@
+varnishtest "Test nuke_limit"
+
+server s1 {
+	# First consume (almost) all of the storage
+	rxreq
+	expect req.url == /url1
+	txresp -bodylen 200000
+
+	rxreq
+	expect req.url == /url2
+	txresp -bodylen 200000
+
+	rxreq
+	expect req.url == /url3
+	txresp -bodylen 200000
+
+	rxreq
+	expect req.url == /url4
+	txresp -bodylen 200000
+
+	rxreq
+	expect req.url == /url5
+	txresp -bodylen 1000000
+} -start
+
+varnish v1  -arg "-smalloc,1M" -arg "-p nuke_limit=1" -vcl+backend {
+	sub vcl_backend_response {
+		set beresp.do_stream = false;
+	}
+} -start
+
+
+client c1 {
+	txreq -url /url1
+	rxresp
+	expect resp.status == 200
+
+	txreq -url /url2
+	rxresp
+	expect resp.status == 200
+
+	txreq -url /url3
+	rxresp
+	expect resp.status == 200
+
+	txreq -url /url4
+	rxresp
+	expect resp.status == 200
+
+	txreq -url /url5
+	rxresp
+	expect resp.status == 503
+} -run



From martin at varnish-software.com  Thu Jun 15 12:45:06 2017
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Thu, 15 Jun 2017 14:45:06 +0200
Subject: [4.1] 365adbf Add 1764 to the changelog
Message-ID: <E1dLU94-0003tL-Kw@project.varnish-software.com>


commit 365adbfefe6f823c6c92481240677e11c983d70b
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Thu Jun 15 14:40:07 2017 +0200

    Add 1764 to the changelog

diff --git a/doc/changes.rst b/doc/changes.rst
index 83b93ab..34fff9a 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -1,3 +1,13 @@
+======================================
+Varnish Cache 4.1.7-beta1 (unreleased)
+======================================
+
+Bugs fixed
+----------
+* 1764_ - Correctly honor nuke_limit parameter
+
+.. _1764: https://github.com/varnishcache/varnish-cache/issues/1764
+
 ================================
 Varnish Cache 4.1.6 (2017-04-26)
 ================================



From martin at varnish-software.com  Thu Jun 15 12:45:06 2017
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Thu, 15 Jun 2017 14:45:06 +0200
Subject: [4.1] ba37d62 Switch to globbing on the whole string
Message-ID: <E1dLU94-0003tU-Mj@project.varnish-software.com>


commit ba37d626977d7ed3607e531c0b78ab46dc79b125
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Sun Sep 11 03:09:54 2016 +0100

    Switch to globbing on the whole string
    
    Fixes #2022
    
    Conflicts:
    	bin/varnishstat/varnishstat_options.h
    
    Backport to 4.1

diff --git a/doc/sphinx/reference/varnishstat.rst b/doc/sphinx/reference/varnishstat.rst
index a0619ae..d576b68 100644
--- a/doc/sphinx/reference/varnishstat.rst
+++ b/doc/sphinx/reference/varnishstat.rst
@@ -34,12 +34,10 @@ The following options are available:
 	the statistics to stdout.
 
 -f
-	Field inclusion glob. A field glob consists of three parts,
-        type, ident and name, where ident is optional. Each part can
-        contain a '*' character at the end to match a prefix. Use
-        backslash to escape characters. If the argument starts with
-        '^' it is used as an exclusion glob. Multiple -f arguments may
-        be given, and they will be applied in order.
+	Field inclusion glob.  Use backslash to escape characters. If
+	the argument starts with '^' it is used as an exclusion glob.
+	Multiple -f arguments may be given, and they will be applied
+	in order.
 
 -l
 	Lists the available fields to use with the -f option.
diff --git a/lib/libvarnishapi/vsc.c b/lib/libvarnishapi/vsc.c
index f051cea..6af266e 100644
--- a/lib/libvarnishapi/vsc.c
+++ b/lib/libvarnishapi/vsc.c
@@ -33,6 +33,7 @@
 #include <sys/types.h>
 #include <sys/stat.h>
 
+#include <fnmatch.h>
 #include <stdint.h>
 #include <stdio.h>
 #include <stdlib.h>
@@ -42,6 +43,7 @@
 #include "vas.h"
 #include "miniobj.h"
 #include "vqueue.h"
+#include "vsb.h"
 
 #include "vapi/vsc.h"
 #include "vapi/vsm.h"
@@ -75,14 +77,8 @@ struct vsc_sf {
 	unsigned		magic;
 #define VSC_SF_MAGIC		0x558478dd
 	VTAILQ_ENTRY(vsc_sf)	list;
-	int			flags;
-#define VSC_SF_EXCL		(1 << 0)
-#define VSC_SF_TY_WC		(1 << 1)
-#define VSC_SF_ID_WC		(1 << 2)
-#define VSC_SF_NM_WC		(1 << 3)
-	char			*type;
-	char			*ident;
-	char			*name;
+	char			*pattern;
+	unsigned		exclude;
 };
 
 struct vsc {
@@ -151,9 +147,7 @@ vsc_delete_sf_list(struct vsc *vsc)
 		sf = VTAILQ_FIRST(&vsc->sf_list);
 		CHECK_OBJ_NOTNULL(sf, VSC_SF_MAGIC);
 		VTAILQ_REMOVE(&vsc->sf_list, sf, list);
-		free(sf->type);
-		free(sf->ident);
-		free(sf->name);
+		free(sf->pattern);
 		FREE_OBJ(sf);
 	}
 }
@@ -180,105 +174,20 @@ vsc_f_arg(struct VSM_data *vd, const char *opt)
 {
 	struct vsc *vsc = vsc_setup(vd);
 	struct vsc_sf *sf;
-	const char *error = NULL;
-	const char *p, *q;
-	char *r;
-	int i;
-	int flags = 0;
-	char *parts[3];
 
 	AN(vd);
 	AN(opt);
 
-	if (opt[0] == '^') {
-		flags |= VSC_SF_EXCL;
-		opt++;
-	}
-
-	/* Split on '.' */
-	memset(parts, 0, sizeof parts);
-	for (i = 0, p = opt; *p != '\0'; i++) {
-		for (q = p; *q != '\0' && *q != '.'; q++)
-			if (*q == '\\')
-				q++;
-		if (i < 3) {
-			parts[i] = malloc(1 + q - p);
-			AN(parts[i]);
-			memcpy(parts[i], p, q - p);
-			parts[i][q - p] = '\0';
-			p = r = parts[i];
-
-			/* Unescape */
-			while (1) {
-				if (*p == '\\')
-					p++;
-				if (*p == '\0')
-					break;
-				*r++ = *p++;
-			}
-			*r = '\0';
-		}
-		p = q;
-		if (*p == '.')
-			p++;
-	}
-	if (i < 1 || i > 3) {
-		(void)vsm_diag(vd, "-f: Wrong number of elements");
-		for (i = 0; i < 3; i++)
-			free(parts[i]);
-		return (-1);
-	}
-
-	/* Set fields */
 	ALLOC_OBJ(sf, VSC_SF_MAGIC);
 	AN(sf);
-	sf->flags = flags;
-	AN(parts[0]);
-	sf->type = parts[0];
-	if (i == 2) {
-		AN(parts[1]);
-		sf->name = parts[1];
-	} else if (i == 3) {
-		AN(parts[1]);
-		sf->ident = parts[1];
-		AN(parts[2]);
-		sf->name = parts[2];
-	}
 
-	/* Check for wildcards */
-	if (sf->type != NULL) {
-		r = strchr(sf->type, '*');
-		if (r != NULL && r[1] == '\0') {
-			*r = '\0';
-			sf->flags |= VSC_SF_TY_WC;
-		} else if (r != NULL)
-			error = "-f: Wildcard not last";
-	}
-	if (sf->ident != NULL) {
-		r = strchr(sf->ident, '*');
-		if (r != NULL && r[1] == '\0') {
-			*r = '\0';
-			sf->flags |= VSC_SF_ID_WC;
-		} else if (r != NULL)
-			error = "-f: Wildcard not last";
-	}
-	if (sf->name != NULL) {
-		r = strchr(sf->name, '*');
-		if (r != NULL && r[1] == '\0') {
-			*r = '\0';
-			sf->flags |= VSC_SF_NM_WC;
-		} else if (r != NULL)
-			error = "-f: Wildcard not last";
+	if (opt[0] == '^') {
+		sf->exclude = 1;
+		opt++;
 	}
 
-	if (error != NULL) {
-		(void)vsm_diag(vd, "%s", error);
-		free(sf->type);
-		free(sf->ident);
-		free(sf->name);
-		FREE_OBJ(sf);
-		return (-1);
-	}
+	sf->pattern = strdup(opt);
+	AN(sf->pattern);
 
 	VTAILQ_INSERT_TAIL(&vsc->sf_list, sf, list);
 	return (1);
@@ -449,52 +358,43 @@ vsc_build_pt_list(struct VSM_data *vd)
 /*--------------------------------------------------------------------
  */
 
-static inline int
-iter_test(const char *s1, const char *s2, int wc)
-{
-
-	if (s1 == NULL)
-		return (0);
-	if (!wc)
-		return (strcmp(s1, s2));
-	for (; *s1 != '\0' && *s1 == *s2; s1++, s2++)
-		continue;
-	return (*s1 != '\0');
-}
-
 static void
 vsc_filter_pt_list(struct VSM_data *vd)
 {
 	struct vsc *vsc = vsc_setup(vd);
 	struct vsc_sf *sf;
 	struct vsc_pt *pt, *pt2;
-	VTAILQ_HEAD(, vsc_pt)	pt_list;
+	VTAILQ_HEAD(, vsc_pt) pt_list;
+	struct vsb *vsb;
 
 	if (VTAILQ_EMPTY(&vsc->sf_list))
 		return;
 
+	vsb = VSB_new_auto();
+	AN(vsb);
+
 	VTAILQ_INIT(&pt_list);
 	VTAILQ_FOREACH(sf, &vsc->sf_list, list) {
 		CHECK_OBJ_NOTNULL(sf, VSC_SF_MAGIC);
 		VTAILQ_FOREACH_SAFE(pt, &vsc->pt_list, list, pt2) {
 			CHECK_OBJ_NOTNULL(pt, VSC_PT_MAGIC);
-			if (iter_test(sf->type, pt->point.section->type,
-			    sf->flags & VSC_SF_TY_WC))
-				continue;
-			if (iter_test(sf->ident, pt->point.section->ident,
-			    sf->flags & VSC_SF_ID_WC))
-				continue;
-			if (iter_test(sf->name, pt->point.desc->name,
-			    sf->flags & VSC_SF_NM_WC))
+			VSB_clear(vsb);
+			VSB_printf(vsb, "%s.%s.%s",
+			    pt->point.section->type,
+			    pt->point.section->ident,
+			    pt->point.desc->name);
+			VSB_finish(vsb);
+			if (fnmatch(sf->pattern, VSB_data(vsb), 0))
 				continue;
 			VTAILQ_REMOVE(&vsc->pt_list, pt, list);
-			if (sf->flags & VSC_SF_EXCL) {
+			if (sf->exclude)
 				FREE_OBJ(pt);
-			} else {
+			else
 				VTAILQ_INSERT_TAIL(&pt_list, pt, list);
-			}
 		}
 	}
+
+	VSB_destroy(&vsb);
 	vsc_delete_pt_list(vsc);
 	VTAILQ_CONCAT(&vsc->pt_list, &pt_list, list);
 }



From martin at varnish-software.com  Thu Jun 15 12:45:06 2017
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Thu, 15 Jun 2017 14:45:06 +0200
Subject: [4.1] c8db16f Ident and type are optional
Message-ID: <E1dLU94-0003td-OX@project.varnish-software.com>


commit c8db16f0b8db8306d96c76970e941756c80c38df
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Wed Oct 26 14:51:57 2016 +0100

    Ident and type are optional
    
    Update the code to accommodate this.  Originally reported by Reinis
    Rozitis (r at roze.lv) on varnish-misc.
    
    Fixes #2118.
    
    Backport to 4.1

diff --git a/lib/libvarnishapi/vsc.c b/lib/libvarnishapi/vsc.c
index 6af266e..67da9d6 100644
--- a/lib/libvarnishapi/vsc.c
+++ b/lib/libvarnishapi/vsc.c
@@ -379,10 +379,13 @@ vsc_filter_pt_list(struct VSM_data *vd)
 		VTAILQ_FOREACH_SAFE(pt, &vsc->pt_list, list, pt2) {
 			CHECK_OBJ_NOTNULL(pt, VSC_PT_MAGIC);
 			VSB_clear(vsb);
-			VSB_printf(vsb, "%s.%s.%s",
-			    pt->point.section->type,
-			    pt->point.section->ident,
-			    pt->point.desc->name);
+			if (strcmp(pt->point.section->type, ""))
+				VSB_printf(vsb, "%s.",
+				    pt->point.section->type);
+			if (strcmp(pt->point.section->ident, ""))
+				VSB_printf(vsb, "%s.",
+				    pt->point.section->ident);
+			VSB_printf(vsb, "%s", pt->point.desc->name);
 			VSB_finish(vsb);
 			if (fnmatch(sf->pattern, VSB_data(vsb), 0))
 				continue;



From martin at varnish-software.com  Thu Jun 15 12:45:06 2017
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Thu, 15 Jun 2017 14:45:06 +0200
Subject: [4.1] 8482ced Rework -f option
Message-ID: <E1dLU94-0003tm-QS@project.varnish-software.com>


commit 8482ced92a4f2755d6882fbac07f149790873c3d
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Thu May 11 02:32:03 2017 +0100

    Rework -f option
    
    We now process the inclusive globs first, and then the exclusive
    ones.  The former are accumulative, while the latter will filter
    the output from the first.
    
    Discussed in detail with and final diff from @mbgrydeland.
    Tests tweaking by yours truly.
    
    Fixes #2320.
    
    Conflicts:
    	bin/varnishstat/varnishstat_options.h
    	bin/varnishtest/tests/u00005.vtc
    	lib/libvarnishapi/vsc.c

diff --git a/doc/sphinx/reference/varnishstat.rst b/doc/sphinx/reference/varnishstat.rst
index d576b68..f4aa12f 100644
--- a/doc/sphinx/reference/varnishstat.rst
+++ b/doc/sphinx/reference/varnishstat.rst
@@ -35,9 +35,9 @@ The following options are available:
 
 -f
 	Field inclusion glob.  Use backslash to escape characters. If
-	the argument starts with '^' it is used as an exclusion glob.
-	Multiple -f arguments may be given, and they will be applied
-	in order.
+	the argument starts with '^' it is used as an exclusive glob.
+	Multiple -f arguments may be given. Inclusive globs are
+	accumulative and are run before exclusive ones.
 
 -l
 	Lists the available fields to use with the -f option.
diff --git a/lib/libvarnishapi/vsc.c b/lib/libvarnishapi/vsc.c
index 67da9d6..cb718bd 100644
--- a/lib/libvarnishapi/vsc.c
+++ b/lib/libvarnishapi/vsc.c
@@ -72,22 +72,24 @@ struct vsc_pt {
 	VTAILQ_ENTRY(vsc_pt)	list;
 	struct VSC_point	point;
 };
+VTAILQ_HEAD(vsc_pt_head, vsc_pt);
 
 struct vsc_sf {
 	unsigned		magic;
 #define VSC_SF_MAGIC		0x558478dd
 	VTAILQ_ENTRY(vsc_sf)	list;
 	char			*pattern;
-	unsigned		exclude;
 };
+VTAILQ_HEAD(vsc_sf_head, vsc_sf);
 
 struct vsc {
 	unsigned		magic;
 #define VSC_MAGIC		0x3373554a
 
 	VTAILQ_HEAD(, vsc_vf)	vf_list;
-	VTAILQ_HEAD(, vsc_pt)	pt_list;
-	VTAILQ_HEAD(, vsc_sf)	sf_list;
+	struct vsc_pt_head	pt_list;
+	struct vsc_sf_head	sf_list_include;
+	struct vsc_sf_head	sf_list_exclude;
 	struct VSM_fantom	iter_fantom;
 };
 
@@ -104,7 +106,8 @@ vsc_setup(struct VSM_data *vd)
 		AN(vd->vsc);
 		VTAILQ_INIT(&vd->vsc->vf_list);
 		VTAILQ_INIT(&vd->vsc->pt_list);
-		VTAILQ_INIT(&vd->vsc->sf_list);
+		VTAILQ_INIT(&vd->vsc->sf_list_include);
+		VTAILQ_INIT(&vd->vsc->sf_list_exclude);
 	}
 	CHECK_OBJ_NOTNULL(vd->vsc, VSC_MAGIC);
 	return (vd->vsc);
@@ -126,27 +129,27 @@ vsc_delete_vf_list(struct vsc *vsc)
 }
 
 static void
-vsc_delete_pt_list(struct vsc *vsc)
+vsc_delete_pt_list(struct vsc_pt_head *head)
 {
 	struct vsc_pt *pt;
 
-	while (!VTAILQ_EMPTY(&vsc->pt_list)) {
-		pt = VTAILQ_FIRST(&vsc->pt_list);
+	while (!VTAILQ_EMPTY(head)) {
+		pt = VTAILQ_FIRST(head);
 		CHECK_OBJ_NOTNULL(pt, VSC_PT_MAGIC);
-		VTAILQ_REMOVE(&vsc->pt_list, pt, list);
+		VTAILQ_REMOVE(head, pt, list);
 		FREE_OBJ(pt);
 	}
 }
 
 static void
-vsc_delete_sf_list(struct vsc *vsc)
+vsc_delete_sf_list(struct vsc_sf_head *head)
 {
 	struct vsc_sf *sf;
 
-	while (!VTAILQ_EMPTY(&vsc->sf_list)) {
-		sf = VTAILQ_FIRST(&vsc->sf_list);
+	while (!VTAILQ_EMPTY(head)) {
+		sf = VTAILQ_FIRST(head);
 		CHECK_OBJ_NOTNULL(sf, VSC_SF_MAGIC);
-		VTAILQ_REMOVE(&vsc->sf_list, sf, list);
+		VTAILQ_REMOVE(head, sf, list);
 		free(sf->pattern);
 		FREE_OBJ(sf);
 	}
@@ -161,8 +164,9 @@ VSC_Delete(struct VSM_data *vd)
 	vsc = vd->vsc;
 	vd->vsc = NULL;
 	CHECK_OBJ_NOTNULL(vsc, VSC_MAGIC);
-	vsc_delete_sf_list(vsc);
-	vsc_delete_pt_list(vsc);
+	vsc_delete_sf_list(&vsc->sf_list_include);
+	vsc_delete_sf_list(&vsc->sf_list_exclude);
+	vsc_delete_pt_list(&vsc->pt_list);
 	vsc_delete_vf_list(vsc);
 	FREE_OBJ(vsc);
 }
@@ -174,6 +178,7 @@ vsc_f_arg(struct VSM_data *vd, const char *opt)
 {
 	struct vsc *vsc = vsc_setup(vd);
 	struct vsc_sf *sf;
+	unsigned exclude = 0;
 
 	AN(vd);
 	AN(opt);
@@ -182,14 +187,18 @@ vsc_f_arg(struct VSM_data *vd, const char *opt)
 	AN(sf);
 
 	if (opt[0] == '^') {
-		sf->exclude = 1;
+		exclude = 1;
 		opt++;
 	}
 
 	sf->pattern = strdup(opt);
 	AN(sf->pattern);
 
-	VTAILQ_INSERT_TAIL(&vsc->sf_list, sf, list);
+	if (exclude)
+		VTAILQ_INSERT_TAIL(&vsc->sf_list_exclude, sf, list);
+	else
+		VTAILQ_INSERT_TAIL(&vsc->sf_list_include, sf, list);
+
 	return (1);
 }
 
@@ -318,7 +327,7 @@ vsc_build_vf_list(struct VSM_data *vd)
 {
 	struct vsc *vsc = vsc_setup(vd);
 
-	vsc_delete_pt_list(vsc);
+	vsc_delete_pt_list(&vsc->pt_list);
 	vsc_delete_vf_list(vsc);
 
 	VSM_FOREACH(&vsc->iter_fantom, vd) {
@@ -339,7 +348,7 @@ vsc_build_pt_list(struct VSM_data *vd)
 	struct vsc *vsc = vsc_setup(vd);
 	struct vsc_vf *vf;
 
-	vsc_delete_pt_list(vsc);
+	vsc_delete_pt_list(&vsc->pt_list);
 
 	VTAILQ_FOREACH(vf, &vsc->vf_list, list) {
 #define VSC_DO(U,l,t)						\
@@ -358,48 +367,71 @@ vsc_build_pt_list(struct VSM_data *vd)
 /*--------------------------------------------------------------------
  */
 
+static int
+vsc_filter_match_pt(struct vsb *vsb, const struct vsc_sf *sf, const
+    struct vsc_pt *pt)
+{
+	VSB_clear(vsb);
+	if (strcmp(pt->point.section->type, ""))
+		VSB_printf(vsb, "%s.", pt->point.section->type);
+	if (strcmp(pt->point.section->ident, ""))
+		VSB_printf(vsb, "%s.", pt->point.section->ident);
+	VSB_printf(vsb, "%s", pt->point.desc->name);
+	AZ(VSB_finish(vsb));
+	return (!fnmatch(sf->pattern, VSB_data(vsb), 0));
+}
+
 static void
 vsc_filter_pt_list(struct VSM_data *vd)
 {
 	struct vsc *vsc = vsc_setup(vd);
+	struct vsc_pt_head tmplist;
 	struct vsc_sf *sf;
 	struct vsc_pt *pt, *pt2;
-	VTAILQ_HEAD(, vsc_pt) pt_list;
 	struct vsb *vsb;
 
-	if (VTAILQ_EMPTY(&vsc->sf_list))
+	if (VTAILQ_EMPTY(&vsc->sf_list_include) &&
+	    VTAILQ_EMPTY(&vsc->sf_list_exclude))
 		return;
 
 	vsb = VSB_new_auto();
 	AN(vsb);
+	VTAILQ_INIT(&tmplist);
+
+	/* Include filters. Empty include filter list implies one that
+	 * matches everything. Points are sorted by the order of include
+	 * filter they match. */
+	if (!VTAILQ_EMPTY(&vsc->sf_list_include)) {
+		VTAILQ_FOREACH(sf, &vsc->sf_list_include, list) {
+			CHECK_OBJ_NOTNULL(sf, VSC_SF_MAGIC);
+			VTAILQ_FOREACH_SAFE(pt, &vsc->pt_list, list, pt2) {
+				CHECK_OBJ_NOTNULL(pt, VSC_PT_MAGIC);
+				if (vsc_filter_match_pt(vsb, sf, pt)) {
+					VTAILQ_REMOVE(&vsc->pt_list,
+					    pt, list);
+					VTAILQ_INSERT_TAIL(&tmplist,
+					    pt, list);
+				}
+			}
+		}
+		vsc_delete_pt_list(&vsc->pt_list);
+		VTAILQ_CONCAT(&vsc->pt_list, &tmplist, list);
+	}
 
-	VTAILQ_INIT(&pt_list);
-	VTAILQ_FOREACH(sf, &vsc->sf_list, list) {
+	/* Exclude filters */
+	VTAILQ_FOREACH(sf, &vsc->sf_list_exclude, list) {
 		CHECK_OBJ_NOTNULL(sf, VSC_SF_MAGIC);
 		VTAILQ_FOREACH_SAFE(pt, &vsc->pt_list, list, pt2) {
 			CHECK_OBJ_NOTNULL(pt, VSC_PT_MAGIC);
-			VSB_clear(vsb);
-			if (strcmp(pt->point.section->type, ""))
-				VSB_printf(vsb, "%s.",
-				    pt->point.section->type);
-			if (strcmp(pt->point.section->ident, ""))
-				VSB_printf(vsb, "%s.",
-				    pt->point.section->ident);
-			VSB_printf(vsb, "%s", pt->point.desc->name);
-			VSB_finish(vsb);
-			if (fnmatch(sf->pattern, VSB_data(vsb), 0))
-				continue;
-			VTAILQ_REMOVE(&vsc->pt_list, pt, list);
-			if (sf->exclude)
-				FREE_OBJ(pt);
-			else
-				VTAILQ_INSERT_TAIL(&pt_list, pt, list);
+			if (vsc_filter_match_pt(vsb, sf, pt)) {
+				VTAILQ_REMOVE(&vsc->pt_list, pt, list);
+				VTAILQ_INSERT_TAIL(&tmplist, pt, list);
+			}
 		}
 	}
+	vsc_delete_pt_list(&tmplist);
 
 	VSB_destroy(&vsb);
-	vsc_delete_pt_list(vsc);
-	VTAILQ_CONCAT(&vsc->pt_list, &pt_list, list);
 }
 
 /*--------------------------------------------------------------------



From martin at varnish-software.com  Thu Jun 15 12:45:06 2017
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Thu, 15 Jun 2017 14:45:06 +0200
Subject: [4.1] a2c072c Add 2320 to the changelog
Message-ID: <E1dLU94-0003tv-SP@project.varnish-software.com>


commit a2c072c2f1e664a8a5871b11dde9f463926aa3ca
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Thu Jun 15 14:41:57 2017 +0200

    Add 2320 to the changelog

diff --git a/doc/changes.rst b/doc/changes.rst
index 34fff9a..6bc4d92 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -5,8 +5,10 @@ Varnish Cache 4.1.7-beta1 (unreleased)
 Bugs fixed
 ----------
 * 1764_ - Correctly honor nuke_limit parameter
+* 2320_ - Rework and fix varnishstat counter filtering
 
 .. _1764: https://github.com/varnishcache/varnish-cache/issues/1764
+.. _2320: https://github.com/varnishcache/varnish-cache/issues/2320
 
 ================================
 Varnish Cache 4.1.6 (2017-04-26)



From martin at varnish-software.com  Thu Jun 15 13:44:05 2017
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Thu, 15 Jun 2017 15:44:05 +0200
Subject: [master] 2b824d2 Allow newlines in long strings to sendhex
Message-ID: <E1dLV49-0006NI-S0@project.varnish-software.com>


commit 2b824d2522438c04514bdf065bf4b7bf577cebe4
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Thu Jun 15 15:16:46 2017 +0200

    Allow newlines in long strings to sendhex

diff --git a/bin/varnishtest/vtc_subr.c b/bin/varnishtest/vtc_subr.c
index eb6223f..2c1439a 100644
--- a/bin/varnishtest/vtc_subr.c
+++ b/bin/varnishtest/vtc_subr.c
@@ -48,7 +48,7 @@ vtc_hex_to_bin(struct vtclog *vl, const char *arg)
 	vsb = VSB_new_auto();
 	AN(vsb);
 	for (; *arg != '\0'; arg++) {
-		if (vct_issp(*arg))
+		if (vct_issp(*arg) || *arg == '\n')
 			continue;
 		c = (uint8_t)*arg;
 		if (c >= '0' && c <= '9')



From martin at varnish-software.com  Thu Jun 15 13:44:05 2017
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Thu, 15 Jun 2017 15:44:05 +0200
Subject: [master] 82554c5 Remember to reset the workspace
Message-ID: <E1dLV49-0006NO-Tr@project.varnish-software.com>


commit 82554c5defd611d2cbbcaf73b8dc7ec6c6518734
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Thu Jun 15 15:23:28 2017 +0200

    Remember to reset the workspace
    
    Original test case by Federico G. Schwindt
    
    Fixes: #2219

diff --git a/bin/varnishd/proxy/cache_proxy_proto.c b/bin/varnishd/proxy/cache_proxy_proto.c
index 0f6b691..b9f1626 100644
--- a/bin/varnishd/proxy/cache_proxy_proto.c
+++ b/bin/varnishd/proxy/cache_proxy_proto.c
@@ -147,6 +147,7 @@ vpx_proto1(const struct worker *wrk, struct req *req)
 	VSL(SLT_Proxy, req->sp->vxid, "1 %s %s %s %s",
 	    fld[1], fld[3], fld[2], fld[4]);
 	HTC_RxPipeline(req->htc, q);
+	WS_Reset(req->htc->ws, 0);
 	return (0);
 }
 
@@ -181,6 +182,7 @@ vpx_proto2(const struct worker *wrk, struct req *req)
 	l = vbe16dec(req->htc->rxbuf_b + 14);
 	assert(req->htc->rxbuf_e - req->htc->rxbuf_b >= 16L + l);
 	HTC_RxPipeline(req->htc, req->htc->rxbuf_b + 16L + l);
+	WS_Reset(req->ws, 0);
 	p = (const void *)req->htc->rxbuf_b;
 
 	/* Version @12 top half */
diff --git a/bin/varnishtest/tests/r02219.vtc b/bin/varnishtest/tests/r02219.vtc
new file mode 100644
index 0000000..7187abc
--- /dev/null
+++ b/bin/varnishtest/tests/r02219.vtc
@@ -0,0 +1,61 @@
+varnishtest "#2219: PROXY tight workspace handling"
+
+server s1 {
+	rxreq
+	txresp
+	rxreq
+	txresp
+} -start
+
+varnish v1 -arg "-p workspace_client=9k" -proto PROXY -vcl+backend {
+	sub vcl_recv {
+		return (pass);
+	}
+} -start
+
+client c1 {
+	send "PROXY TCP4 127.0.0.1 127.0.0.1 1111 2222\r\nGET /AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA HTTP/1.1\r\n\r\n"
+	rxresp
+} -run
+
+client c2 {
+	# UNSPEC proto
+	sendhex {
+0d 0a 0d 0a 00 0d 0a 51 55 49 54 0a
+21 00 00 00
+47 45 54 20 2f
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42
+20 48 54 54 50 2f 31 2e 31 0d 0a 0d 0a
+}
+	rxresp
+} -run



From martin at varnish-software.com  Thu Jun 15 13:45:06 2017
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Thu, 15 Jun 2017 15:45:06 +0200
Subject: [4.1] 2b7c89e Allow newlines in long strings to sendhex
Message-ID: <E1dLV58-0006S7-OJ@project.varnish-software.com>


commit 2b7c89e253bbface2857a2ba9383a6d78ee8ef6b
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Thu Jun 15 15:25:41 2017 +0200

    Allow newlines in long strings to sendhex

diff --git a/bin/varnishtest/vtc_http.c b/bin/varnishtest/vtc_http.c
index 1503b7e..57d36b6 100644
--- a/bin/varnishtest/vtc_http.c
+++ b/bin/varnishtest/vtc_http.c
@@ -1129,7 +1129,7 @@ cmd_http_sendhex(CMD_ARGS)
 	AN(p);
 	q = av[1];
 	for (i = 0; i < l; i++) {
-		while (vct_issp(*q))
+		while (vct_issp(*q) || *q == '\n')
 			q++;
 		if (*q == '\0')
 			break;



From martin at varnish-software.com  Thu Jun 15 13:45:06 2017
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Thu, 15 Jun 2017 15:45:06 +0200
Subject: [4.1] 7fc8493 Remember to reset the workspace
Message-ID: <E1dLV58-0006SD-SL@project.varnish-software.com>


commit 7fc849377fa001d726a8f5dfe26926e3e1f779f9
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Thu Jun 15 15:26:13 2017 +0200

    Remember to reset the workspace
    
    Original test case by Federico G. Schwindt
    
    Fixes: #2219
    
    Conflicts:
    	bin/varnishd/proxy/cache_proxy_proto.c

diff --git a/bin/varnishd/proxy/cache_proxy_proto.c b/bin/varnishd/proxy/cache_proxy_proto.c
index 4799e4b..a45367b 100644
--- a/bin/varnishd/proxy/cache_proxy_proto.c
+++ b/bin/varnishd/proxy/cache_proxy_proto.c
@@ -150,6 +150,7 @@ vpx_proto1(const struct worker *wrk, struct req *req)
 	VSL(SLT_Proxy, req->sp->vxid, "1 %s %s %s %s",
 	    fld[1], fld[3], fld[2], fld[4]);
 	req->htc->pipeline_b = q;
+	WS_Reset(req->htc->ws, NULL);
 	return (0);
 }
 
@@ -184,6 +185,7 @@ vpx_proto2(const struct worker *wrk, struct req *req)
 	l = vbe16dec(req->htc->rxbuf_b + 14);
 	assert(req->htc->rxbuf_e - req->htc->rxbuf_b >= 16L + l);
 	req->htc->pipeline_b = req->htc->rxbuf_b + 16L + l;
+	WS_Reset(req->ws, NULL);
 	p = (const void *)req->htc->rxbuf_b;
 
 	/* Version @12 top half */
diff --git a/bin/varnishtest/tests/r02219.vtc b/bin/varnishtest/tests/r02219.vtc
new file mode 100644
index 0000000..7187abc
--- /dev/null
+++ b/bin/varnishtest/tests/r02219.vtc
@@ -0,0 +1,61 @@
+varnishtest "#2219: PROXY tight workspace handling"
+
+server s1 {
+	rxreq
+	txresp
+	rxreq
+	txresp
+} -start
+
+varnish v1 -arg "-p workspace_client=9k" -proto PROXY -vcl+backend {
+	sub vcl_recv {
+		return (pass);
+	}
+} -start
+
+client c1 {
+	send "PROXY TCP4 127.0.0.1 127.0.0.1 1111 2222\r\nGET /AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA HTTP/1.1\r\n\r\n"
+	rxresp
+} -run
+
+client c2 {
+	# UNSPEC proto
+	sendhex {
+0d 0a 0d 0a 00 0d 0a 51 55 49 54 0a
+21 00 00 00
+47 45 54 20 2f
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42
+42 42 42 42 42 42 42 42 42 42 42 42 42 42
+20 48 54 54 50 2f 31 2e 31 0d 0a 0d 0a
+}
+	rxresp
+} -run



From martin at varnish-software.com  Thu Jun 15 13:46:05 2017
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Thu, 15 Jun 2017 15:46:05 +0200
Subject: [4.1] e46ee48 add 2219 to changelog
Message-ID: <E1dLV65-0006V6-Hc@project.varnish-software.com>


commit e46ee484f668e5039d1058732ab03bc511d75636
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Thu Jun 15 15:45:26 2017 +0200

    add 2219 to changelog

diff --git a/doc/changes.rst b/doc/changes.rst
index 6bc4d92..8314419 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -6,9 +6,11 @@ Bugs fixed
 ----------
 * 1764_ - Correctly honor nuke_limit parameter
 * 2320_ - Rework and fix varnishstat counter filtering
+* 2219_ - Remember to reset workspace
 
 .. _1764: https://github.com/varnishcache/varnish-cache/issues/1764
 .. _2320: https://github.com/varnishcache/varnish-cache/issues/2320
+.. _2219: https://github.com/varnishcache/varnish-cache/issues/2219
 
 ================================
 Varnish Cache 4.1.6 (2017-04-26)



From hermunn at varnish-software.com  Thu Jun 15 13:59:10 2017
From: hermunn at varnish-software.com (Pål Hermunn Johansen)
Date: Thu, 15 Jun 2017 15:59:10 +0200
Subject: [4.1] db96f60 Document varnish
Message-ID: <E1dLVIk-00073i-Nn@project.varnish-software.com>


commit db96f60d2306158fd15f98b1a94827fc717baa46
Author: Guillaume Quintard <guillaume at varnish-software.com>
Date:   Wed Jun 1 12:12:37 2016 +0200

    Document varnish

diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c
index bbfea18..10542ce 100644
--- a/bin/varnishtest/vtc_varnish.c
+++ b/bin/varnishtest/vtc_varnish.c
@@ -915,9 +915,79 @@ varnish_expect(const struct varnish *v, char * const *av)
 		    av[0], (uintmax_t)sp.val, av[1], av[2], (uintmax_t)ref);
 	}
 }
-
-/**********************************************************************
- * Varnish server cmd dispatch
+/* SECTION: varnish varnish
+ *
+ * Define and interact with varnish instances.
+ *
+ * To define a Varnish server, you'll use this syntax::
+ *
+ *         varnish vNAME [-arg STRING] [-vcl STRING] [-vcl+backend STRING]
+ *	                 [-errvcl STRING STRING] [-jail STRING] [-proto PROXY]
+ *
+ * With:
+ *
+ * vNAME
+ *         Identify the Varnish server with a string, it must starts with 'v'.
+ *
+ * \-arg STRING
+ *         Pass an argument to varnishd, for example "-h simple_list".
+ *
+ * \-vcl STRING
+ *         Specify the VCL to load on this Varnish instance. You'll probably
+ *         want to use multi-lines strings for this ({...}).
+ *
+ * \-vcl+backend STRING
+ *         Do the exact same thing as -vcl, but adds the definition block of
+ *         known backends (ie. already defined).
+ *
+ * \-errvcl STRING1 STRING2
+ *         Load STRING2 as VCL, expecting it to fail, and Varnish to send an
+ *         error string matching STRING2
+ *
+ * \-jail STRING
+ *         Look at ``man varnishd`` (-j) for more information.
+ *
+ * \-proto PROXY
+ *         Have Varnish use the proxy protocol. Note that PROXY here is the
+ *         actual string.
+ *
+ * You can decide to start the Varnish instance and/or wait for several events::
+ *
+ *         varnish vNAME [-start] [-wait] [-wait-running] [-wait-stopped]
+ *
+ * \-start
+ *         Start Varnish in the background, immediately processing the rest of
+ *         the file.
+ *
+ * \-wait
+ *         Wait for that instance to terminate.
+ *
+ * \-wait-running
+ *         Wait for the Varnish child process to be started.
+ *
+ * \-wait-stopped
+ *         Wait for the Varnish child process to stop.
+ *
+ * \-cleanup
+ *         Once Varnish is stopped, clean everything after it. This is only used
+ *         in one test and you should never need it.
+ *
+ * Once Varnish is started, you can talk to it (as you would through
+ * ``varnishadm``) with these additional switches::
+ *
+ *         varnish vNAME [-cli STRING] [-cliok STRING] [-clierr STRING]
+ *                       [-expect STRING OP NUMBER]
+ *
+ * \-cli|-cliok|-clierr STRING
+ *         All three of these will send STRING to the CLI, the only difference
+ *         is what they expect the return code to be. -cli doesn't expect
+ *         anything, -cliok expects 200 and -clierr expects not 200.
+ *
+ * \-expect STRING OP NUMBER
+ *         Look into the VSM and make sure the counter identified by STRING has
+ *         a correct value. OP can be ==, >, >=, <, <=. For example::
+ *
+ *                 varnish v1 -expect SMA.s1.g_space > 1000000
  */
 
 void



From hermunn at varnish-software.com  Thu Jun 15 13:59:10 2017
From: hermunn at varnish-software.com (Pål Hermunn Johansen)
Date: Thu, 15 Jun 2017 15:59:10 +0200
Subject: [4.1] 89c13d3 Add -stop doc, and be more detailed
Message-ID: <E1dLVIk-00073o-Q2@project.varnish-software.com>


commit 89c13d39ac595220f58b89b8791e473391bf1965
Author: Guillaume Quintard <guillaume at varnish-software.com>
Date:   Mon Jun 6 11:24:09 2016 +0200

    Add -stop doc, and be more detailed

diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c
index 10542ce..1774b78 100644
--- a/bin/varnishtest/vtc_varnish.c
+++ b/bin/varnishtest/vtc_varnish.c
@@ -924,6 +924,10 @@ varnish_expect(const struct varnish *v, char * const *av)
  *         varnish vNAME [-arg STRING] [-vcl STRING] [-vcl+backend STRING]
  *	                 [-errvcl STRING STRING] [-jail STRING] [-proto PROXY]
  *
+ * The first ``varnish vNAME`` invocation will start the varnishd master
+ * process in the background, waiting for the ``-start`` switch to actually
+ * start the child.
+ *
  * With:
  *
  * vNAME
@@ -956,8 +960,10 @@ varnish_expect(const struct varnish *v, char * const *av)
  *         varnish vNAME [-start] [-wait] [-wait-running] [-wait-stopped]
  *
  * \-start
- *         Start Varnish in the background, immediately processing the rest of
- *         the file.
+ *         Start the child process.
+ *
+ * \-stop
+ *         Stop the child process.
  *
  * \-wait
  *         Wait for that instance to terminate.



From hermunn at varnish-software.com  Thu Jun 15 13:59:10 2017
From: hermunn at varnish-software.com (Pål Hermunn Johansen)
Date: Thu, 15 Jun 2017 15:59:10 +0200
Subject: [4.1] 9699d82 Add a vsl_catchup varnishtest function
Message-ID: <E1dLVIk-00073x-Rk@project.varnish-software.com>


commit 9699d82d97bb78c6307339d3b8443f72e4b34d9b
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Wed Mar 22 18:09:12 2017 +0100

    Add a vsl_catchup varnishtest function
    
    This enables waiting for the logging thread to idle to make sure all
    generated log is caught.
    
    Conflicts:
    	bin/varnishtest/vtc_varnish.c

diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c
index 1774b78..98c15cc 100644
--- a/bin/varnishtest/vtc_varnish.c
+++ b/bin/varnishtest/vtc_varnish.c
@@ -79,6 +79,7 @@ struct varnish {
 
 	unsigned		vsl_tag_count[256];
 
+	volatile int		vsl_rec;
 	volatile int		vsl_idle;
 };
 
@@ -186,6 +187,16 @@ wait_running(const struct varnish *v)
  * Varnishlog gatherer thread
  */
 
+static void
+vsl_catchup(const struct varnish *v)
+{
+	int vsl_idle;
+
+	vsl_idle = v->vsl_idle;
+	while (vsl_idle == v->vsl_idle)
+		VTIM_sleep(0.1);
+}
+
 static void *
 varnishlog_thread(void *priv)
 {
@@ -231,6 +242,7 @@ varnishlog_thread(void *priv)
 		if (i == 0) {
 			v->vsl_idle++;
 			/* Nothing to do but wait */
+			v->vsl_idle++;
 			VTIM_sleep(0.1);
 			continue;
 		} else if (i == -2) {
@@ -994,6 +1006,10 @@ varnish_expect(const struct varnish *v, char * const *av)
  *         a correct value. OP can be ==, >, >=, <, <=. For example::
  *
  *                 varnish v1 -expect SMA.s1.g_space > 1000000
+ *
+ * \-vsl_catchup
+ *         Wait until the logging thread has idled to make sure that all
+ *         the generated log is flushed
  */
 
 void
@@ -1130,6 +1146,10 @@ cmd_varnish(CMD_ARGS)
 			varnish_wait(v);
 			continue;
 		}
+		if (!strcmp(*av, "-vsl_catchup")) {
+			vsl_catchup(v);
+			continue;
+		}
 		vtc_log(v->vl, 0, "Unknown varnish argument: %s", *av);
 	}
 }



From hermunn at varnish-software.com  Thu Jun 15 13:59:10 2017
From: hermunn at varnish-software.com (Pål Hermunn Johansen)
Date: Thu, 15 Jun 2017 15:59:10 +0200
Subject: [4.1] ab4cb70 Make -vsl_catchup bail on test-failure or child death
Message-ID: <E1dLVIk-000746-TR@project.varnish-software.com>


commit ab4cb7074ad8d549a0c64082e4006e580e485e31
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Mar 23 20:08:51 2017 +0000

    Make -vsl_catchup bail on test-failure or child death

diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c
index 98c15cc..24677b3 100644
--- a/bin/varnishtest/vtc_varnish.c
+++ b/bin/varnishtest/vtc_varnish.c
@@ -193,7 +193,7 @@ vsl_catchup(const struct varnish *v)
 	int vsl_idle;
 
 	vsl_idle = v->vsl_idle;
-	while (vsl_idle == v->vsl_idle)
+	while (!vtc_error && v->pid && vsl_idle == v->vsl_idle)
 		VTIM_sleep(0.1);
 }
 



From hermunn at varnish-software.com  Thu Jun 15 14:42:10 2017
From: hermunn at varnish-software.com (Pål Hermunn Johansen)
Date: Thu, 15 Jun 2017 16:42:10 +0200
Subject: [4.1] 355a56f Update changelog
Message-ID: <E1dLVyM-0000Sf-FQ@project.varnish-software.com>


commit 355a56fb44436a9bf0c43a20a6c515ec44a84ad5
Author: P?l Hermunn Johansen <hermunn at varnish-software.com>
Date:   Thu Jun 15 16:15:32 2017 +0200

    Update changelog

diff --git a/doc/changes.rst b/doc/changes.rst
index 8314419..7f24282 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -2,15 +2,30 @@
 Varnish Cache 4.1.7-beta1 (unreleased)
 ======================================
 
+Changes since 4.1.6:
+
+* Add -vsl_catchup to varnishtest
+* Add record-prefix support to varnishncsa
+
 Bugs fixed
 ----------
 * 1764_ - Correctly honor nuke_limit parameter
-* 2320_ - Rework and fix varnishstat counter filtering
+* 2022_ - varnishstat -1 -f field inclusion glob doesn't allow VBE
+  backend fields
+* 2069_ - Health probes fail when HTTP response does not contain
+  reason phrase
+* 2118_ - "varnishstat -f MAIN.sess_conn -1" produces empty output
 * 2219_ - Remember to reset workspace
+* 2320_ - Rework and fix varnishstat counter filtering
+* 2329_ - Docfix: Only root can jail
 
 .. _1764: https://github.com/varnishcache/varnish-cache/issues/1764
-.. _2320: https://github.com/varnishcache/varnish-cache/issues/2320
+.. _2022: https://github.com/varnishcache/varnish-cache/issues/2022
+.. _2069: https://github.com/varnishcache/varnish-cache/issues/2069
+.. _2118: https://github.com/varnishcache/varnish-cache/issues/2118
 .. _2219: https://github.com/varnishcache/varnish-cache/issues/2219
+.. _2320: https://github.com/varnishcache/varnish-cache/issues/2320
+.. _2329: https://github.com/varnishcache/varnish-cache/issues/2329
 
 ================================
 Varnish Cache 4.1.6 (2017-04-26)



From hermunn at varnish-software.com  Thu Jun 15 15:12:06 2017
From: hermunn at varnish-software.com (Pål Hermunn Johansen)
Date: Thu, 15 Jun 2017 17:12:06 +0200
Subject: [4.1] 6556c56 Prepare for 4.1.7-beta1 release
Message-ID: <E1dLWRK-0001jZ-Fc@project.varnish-software.com>


commit 6556c563f804a7eae9f7dc45284616e517a6c9c3
Author: P?l Hermunn Johansen <hermunn at varnish-software.com>
Date:   Thu Jun 15 17:03:01 2017 +0200

    Prepare for 4.1.7-beta1 release

diff --git a/configure.ac b/configure.ac
index ecf8530..32bd840 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2,7 +2,7 @@ AC_PREREQ(2.59)
 AC_COPYRIGHT([Copyright (c) 2006 Verdens Gang AS
 Copyright (c) 2006-2016 Varnish Software])
 AC_REVISION([$Id$])
-AC_INIT([Varnish], [4.1.6], [varnish-dev at varnish-cache.org])
+AC_INIT([Varnish], [4.1.7-beta1], [varnish-dev at varnish-cache.org])
 AC_CONFIG_SRCDIR(include/miniobj.h)
 AC_CONFIG_HEADERS([config.h])
 AC_CONFIG_MACRO_DIR([m4])
diff --git a/doc/changes.rst b/doc/changes.rst
index 7f24282..cf1883c 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -1,5 +1,5 @@
 ======================================
-Varnish Cache 4.1.7-beta1 (unreleased)
+Varnish Cache 4.1.7-beta1 (2017-06-15)
 ======================================
 
 Changes since 4.1.6:



From phk at FreeBSD.org  Thu Jun 15 23:00:08 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 16 Jun 2017 01:00:08 +0200
Subject: [master] 20a3eee Eliminate the VSC-sections.
Message-ID: <E1dLdkG-00048R-7K@project.varnish-software.com>


commit 20a3eee64da44939c2271fe1ee2a54d566d66fee
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Jun 15 22:58:57 2017 +0000

    Eliminate the VSC-sections.

diff --git a/bin/varnishstat/varnishstat.c b/bin/varnishstat/varnishstat.c
index 1290408..23056e7 100644
--- a/bin/varnishstat/varnishstat.c
+++ b/bin/varnishstat/varnishstat.c
@@ -56,21 +56,19 @@ static int
 do_xml_cb(void *priv, const struct VSC_point * const pt)
 {
 	uint64_t val;
-	const struct VSC_section *sec;
 
 	(void)priv;
 	if (pt == NULL)
 		return (0);
-	AZ(strcmp(pt->desc->ctype, "uint64_t"));
+	AZ(strcmp(pt->ctype, "uint64_t"));
 	val = *(const volatile uint64_t*)pt->ptr;
-	sec = pt->section;
 
 	printf("\t<stat>\n");
-	printf("\t\t<name>%s.%s</name>\n", sec->ident, pt->desc->name);
+	printf("\t\t<name>%s</name>\n", pt->name);
 	printf("\t\t<value>%ju</value>\n", (uintmax_t)val);
-	printf("\t\t<flag>%c</flag>\n", pt->desc->semantics);
-	printf("\t\t<format>%c</format>\n", pt->desc->format);
-	printf("\t\t<description>%s</description>\n", pt->desc->sdesc);
+	printf("\t\t<flag>%c</flag>\n", pt->semantics);
+	printf("\t\t<format>%c</format>\n", pt->format);
+	printf("\t\t<description>%s</description>\n", pt->sdesc);
 	printf("\t</stat>\n");
 	return (0);
 }
@@ -97,15 +95,13 @@ do_json_cb(void *priv, const struct VSC_point * const pt)
 {
 	uint64_t val;
 	int *jp;
-	const struct VSC_section *sec;
 
 	if (pt == NULL)
 		return (0);
 
 	jp = priv;
-	AZ(strcmp(pt->desc->ctype, "uint64_t"));
+	AZ(strcmp(pt->ctype, "uint64_t"));
 	val = *(const volatile uint64_t*)pt->ptr;
-	sec = pt->section;
 
 	if (*jp)
 		*jp = 0;
@@ -114,13 +110,11 @@ do_json_cb(void *priv, const struct VSC_point * const pt)
 
 	printf("  \"");
 	/* build the JSON key name.  */
-	if (sec->ident[0])
-		printf("%s.", sec->ident);
-	printf("%s\": {\n", pt->desc->name);
-	printf("    \"description\": \"%s\",\n", pt->desc->sdesc);
+	printf("%s\": {\n", pt->name);
+	printf("    \"description\": \"%s\",\n", pt->sdesc);
 
-	printf("    \"flag\": \"%c\", ", pt->desc->semantics);
-	printf("\"format\": \"%c\",\n", pt->desc->format);
+	printf("    \"flag\": \"%c\", ", pt->semantics);
+	printf("\"format\": \"%c\",\n", pt->format);
 	printf("    \"value\": %ju", (uintmax_t)val);
 	printf("\n  }");
 
@@ -160,14 +154,12 @@ do_once_cb_first(void *priv, const struct VSC_point * const pt)
 {
 	struct once_priv *op;
 	uint64_t val;
-	const struct VSC_section *sec;
 
 	if (pt == NULL)
 		return (0);
 	op = priv;
-	AZ(strcmp(pt->desc->ctype, "uint64_t"));
-	sec = pt->section;
-	if (strcmp(sec->ident, "MAIN") || strcmp(pt->desc->name, "uptime"))
+	AZ(strcmp(pt->ctype, "uint64_t"));
+	if (strcmp(pt->name, "MAIN.uptime"))
 		return (0);
 	val = *(const volatile uint64_t*)pt->ptr;
 	op->up = (double)val;
@@ -180,28 +172,24 @@ do_once_cb(void *priv, const struct VSC_point * const pt)
 	struct once_priv *op;
 	uint64_t val;
 	int i;
-	const struct VSC_section *sec;
 
 	if (pt == NULL)
 		return (0);
 	op = priv;
-	AZ(strcmp(pt->desc->ctype, "uint64_t"));
+	AZ(strcmp(pt->ctype, "uint64_t"));
 	val = *(const volatile uint64_t*)pt->ptr;
-	sec = pt->section;
 	i = 0;
-	if (strcmp(sec->ident, ""))
-		i += printf("%s.", sec->ident);
-	i += printf("%s", pt->desc->name);
+	i += printf("%s", pt->name);
 	if (i >= op->pad)
 		op->pad = i + 1;
 	printf("%*.*s", op->pad - i, op->pad - i, "");
-	if (pt->desc->semantics == 'c')
+	if (pt->semantics == 'c')
 		printf("%12ju %12.2f %s\n",
 		    (uintmax_t)val, op->up ? val / op->up : 0,
-		    pt->desc->sdesc);
+		    pt->sdesc);
 	else
 		printf("%12ju %12s %s\n",
-		    (uintmax_t)val, ".  ", pt->desc->sdesc);
+		    (uintmax_t)val, ".  ", pt->sdesc);
 	return (0);
 }
 
@@ -223,21 +211,17 @@ static int
 do_list_cb(void *priv, const struct VSC_point * const pt)
 {
 	int i;
-	const struct VSC_section * sec;
 
 	(void)priv;
 
 	if (pt == NULL)
 		return (0);
 
-	sec = pt->section;
 	i = 0;
-	if (strcmp(sec->ident, ""))
-		i += printf("%s.", sec->ident);
-	i += printf("%s", pt->desc->name);
+	i += printf("%s", pt->name);
 	if (i < 30)
 		printf("%*s", i - 30, "");
-	printf(" %s\n", pt->desc->sdesc);
+	printf(" %s\n", pt->sdesc);
 	return (0);
 }
 
diff --git a/bin/varnishstat/varnishstat_curses.c b/bin/varnishstat/varnishstat_curses.c
index c517bc6..ab32eca 100644
--- a/bin/varnishstat/varnishstat_curses.c
+++ b/bin/varnishstat/varnishstat_curses.c
@@ -73,9 +73,7 @@ struct pt {
 #define PT_MAGIC		0x41698E4F
 	VTAILQ_ENTRY(pt)	list;
 
-	const struct VSC_point	*vpt;
-
-	char			*name;
+	struct VSC_point	*vpt;
 
 	char			seen;
 
@@ -210,7 +208,7 @@ build_pt_array(void)
 		CHECK_OBJ_NOTNULL(pt, PT_MAGIC);
 		if (!pt->seen && hide_unseen)
 			continue;
-		if (pt->vpt->desc->level > verbosity)
+		if (pt->vpt->level > verbosity)
 			continue;
 		assert(n_ptarray < n_ptlist);
 		ptarray[n_ptarray++] = pt;
@@ -240,7 +238,7 @@ delete_pt_list(void)
 		pt = VTAILQ_FIRST(&ptlist);
 		CHECK_OBJ_NOTNULL(pt, PT_MAGIC);
 		VTAILQ_REMOVE(&ptlist, pt, list);
-		free(pt->name);
+		VSC_Destroy_Point(&pt->vpt);
 		FREE_OBJ(pt);
 		i++;
 	}
@@ -262,34 +260,34 @@ build_pt_list_cb(void *priv, const struct VSC_point *vpt)
 {
 	struct pt_priv *pt_priv;
 	struct pt *pt;
-	char buf[128];
 
 	if (vpt == NULL)
 		return (0);
 
 	CAST_OBJ_NOTNULL(pt_priv, priv, PT_PRIV_MAGIC);
 
-	AZ(strcmp(vpt->desc->ctype, "uint64_t"));
-	bprintf(buf, "%s.%s", vpt->section->ident, vpt->desc->name);
+	AZ(strcmp(vpt->ctype, "uint64_t"));
 
-	if (!strcmp(buf, "MGT.uptime"))
+	if (!strcmp(vpt->name, "MGT.uptime"))
 		mgt_uptime = vpt->ptr;
-	if (!strcmp(buf, "MAIN.uptime"))
+	if (!strcmp(vpt->name, "MAIN.uptime"))
 		main_uptime = vpt->ptr;
-	if (!strcmp(buf, "MAIN.cache_hit"))
+	if (!strcmp(vpt->name, "MAIN.cache_hit"))
 		main_cache_hit = vpt->ptr;
-	if (!strcmp(buf, "MAIN.cache_miss"))
+	if (!strcmp(vpt->name, "MAIN.cache_miss"))
 		main_cache_miss = vpt->ptr;
 
 	VTAILQ_FOREACH(pt, &ptlist, list) {
 		CHECK_OBJ_NOTNULL(pt, PT_MAGIC);
-		AN(pt->name);
-		if (strcmp(buf, pt->name))
+		AN(pt->vpt->name);
+		if (strcmp(vpt->name, pt->vpt->name))
 			continue;
 		VTAILQ_REMOVE(&ptlist, pt, list);
 		AN(n_ptlist);
 		n_ptlist--;
-		pt->vpt = vpt;
+		VSC_Destroy_Point(&pt->vpt);
+		pt->vpt = VSC_Clone_Point(vpt);
+		AN(pt->vpt);
 		VTAILQ_INSERT_TAIL(&pt_priv->ptlist, pt, list);
 		pt_priv->n_ptlist++;
 		return (0);
@@ -299,10 +297,8 @@ build_pt_list_cb(void *priv, const struct VSC_point *vpt)
 	ALLOC_OBJ(pt, PT_MAGIC);
 	AN(pt);
 
-	REPLACE(pt->name, buf);
-	AN(pt->name);
-
-	pt->vpt = vpt;
+	pt->vpt = VSC_Clone_Point(vpt);
+	AN(pt->vpt);
 
 	pt->last = *pt->vpt->ptr;
 
@@ -379,12 +375,12 @@ sample_points(void)
 			pt->chg = ((int64_t)pt->cur - (int64_t)pt->last) /
 			    (pt->t_cur - pt->t_last);
 
-		if (pt->vpt->desc->semantics == 'g') {
+		if (pt->vpt->semantics == 'g') {
 			pt->avg = 0.;
 			update_ma(&pt->ma_10, (int64_t)pt->cur);
 			update_ma(&pt->ma_100, (int64_t)pt->cur);
 			update_ma(&pt->ma_1000, (int64_t)pt->cur);
-		} else if (pt->vpt->desc->semantics == 'c') {
+		} else if (pt->vpt->semantics == 'c') {
 			if (main_uptime != NULL && *main_uptime)
 				pt->avg = pt->cur / *main_uptime;
 			else
@@ -781,7 +777,7 @@ draw_line_bitmap(WINDOW *w, int y, int x, int X, const struct pt *pt)
 
 	AN(w);
 	AN(pt);
-	assert(pt->vpt->desc->format == 'b');
+	assert(pt->vpt->format == 'b');
 
 	col = 0;
 	while (col < COL_LAST) {
@@ -851,13 +847,13 @@ draw_line(WINDOW *w, int y, const struct pt *pt)
 	assert(colw_name >= COLW_NAME_MIN);
 	X = getmaxx(w);
 	x = 0;
-	if (strlen(pt->name) > colw_name)
-		mvwprintw(w, y, x, "%.*s...", colw_name - 3, pt->name);
+	if (strlen(pt->vpt->name) > colw_name)
+		mvwprintw(w, y, x, "%.*s...", colw_name - 3, pt->vpt->name);
 	else
-		mvwprintw(w, y, x, "%.*s", colw_name, pt->name);
+		mvwprintw(w, y, x, "%.*s", colw_name, pt->vpt->name);
 	x += colw_name;
 
-	switch (pt->vpt->desc->format) {
+	switch (pt->vpt->format) {
 	case 'b':
 		draw_line_bitmap(w, y, x, X, pt);
 		break;
@@ -926,7 +922,7 @@ draw_bar_b(void)
 		mvwprintw(w_bar_b, 0, x, "vvv");
 	x += 4;
 	if (current < n_ptarray - 1)
-		mvwprintw(w_bar_b, 0, x, "%s", ptarray[current]->name);
+		mvwprintw(w_bar_b, 0, x, "%s", ptarray[current]->vpt->name);
 
 	bprintf(buf, "%d-%d/%d", page_start + 1,
 	    page_start + l_points < n_ptarray ?
@@ -957,9 +953,9 @@ draw_info(void)
 	if (current < n_ptarray - 1) {
 		/* XXX: Word wrapping, and overflow handling? */
 		mvwprintw(w_info, 0, 0, "%s:",
-		    ptarray[current]->vpt->desc->sdesc);
+		    ptarray[current]->vpt->sdesc);
 		mvwprintw(w_info, 1, 0, "%s",
-		    ptarray[current]->vpt->desc->ldesc);
+		    ptarray[current]->vpt->ldesc);
 	}
 	wnoutrefresh(w_info);
 }
diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c
index 7e41d07..96046c2 100644
--- a/bin/varnishtest/vtc_varnish.c
+++ b/bin/varnishtest/vtc_varnish.c
@@ -813,25 +813,22 @@ do_stat_dump_cb(void *priv, const struct VSC_point * const pt)
 	const struct varnish *v;
 	struct dump_priv *dp;
 	uint64_t u;
-	char buf[1024];
 
 	if (pt == NULL)
 		return (0);
 	dp = priv;
 	v = dp->v;
 
-	if (strcmp(pt->desc->ctype, "uint64_t"))
+	if (strcmp(pt->ctype, "uint64_t"))
 		return (0);
 	u = *pt->ptr;
 
-	bprintf(buf, "%s.%s", pt->section->ident, pt->desc->name);
-
 	if (strcmp(dp->arg, "*")) {
-		if (fnmatch(dp->arg, buf, 0))
+		if (fnmatch(dp->arg, pt->name, 0))
 			return (0);
 	}
 
-	vtc_log(v->vl, 4, "VSC %s %ju",  buf, (uintmax_t)u);
+	vtc_log(v->vl, 4, "VSC %s %ju",  pt->name, (uintmax_t)u);
 	return (0);
 }
 
@@ -858,7 +855,6 @@ varnish_vsc(struct varnish *v, const char *arg)
  */
 
 struct stat_priv {
-	char target_ident[256];
 	char target_name[256];
 	uintmax_t val;
 	const struct varnish *v;
@@ -872,12 +868,10 @@ do_stat_cb(void *priv, const struct VSC_point * const pt)
 	if (pt == NULL)
 		return(0);
 
-	if (strcmp(pt->section->ident, sp->target_ident))
-		return(0);
-	if (strcmp(pt->desc->name, sp->target_name))
+	if (strcmp(pt->name, sp->target_name))
 		return(0);
 
-	AZ(strcmp(pt->desc->ctype, "uint64_t"));
+	AZ(strcmp(pt->ctype, "uint64_t"));
 	AN(pt->ptr);
 	sp->val = *pt->ptr;
 	return (1);
@@ -907,11 +901,9 @@ varnish_expect(struct varnish *v, char * const *av)
 	}
 	p = strrchr(r, '.');
 	if (p == NULL) {
-		strcpy(sp.target_ident, "MAIN");
-		bprintf(sp.target_name, "%s", r);
+		bprintf(sp.target_name, "MAIN.%s", r);
 	} else {
-		bprintf(sp.target_ident, "%.*s", (int)(p - r), r);
-		bprintf(sp.target_name, "%s", p + 1);
+		bprintf(sp.target_name, "%s", r);
 	}
 
 	sp.val = 0;
diff --git a/include/vapi/vsc.h b/include/vapi/vsc.h
index a3d847d..b46593e 100644
--- a/include/vapi/vsc.h
+++ b/include/vapi/vsc.h
@@ -56,7 +56,6 @@ int VSC_Arg(struct vsm *vd, int arg, const char *opt);
 struct VSC_level_desc;
 struct VSC_type_desc;
 struct VSC_section;
-struct VSC_desc;
 struct VSC_point;
 
 struct VSC_level_desc {
@@ -66,12 +65,8 @@ struct VSC_level_desc {
 	const char *ldesc;		/* long description */
 };
 
-struct VSC_section {
-	char *ident;
-};
-
-/* See include/tbl/vsc_fields.h for descriptions */
-struct VSC_desc {
+struct VSC_point {
+	const volatile uint64_t *ptr;	/* field value			*/
 	const char *name;		/* field name			*/
 	const char *ctype;		/* C-type			*/
 	int semantics;			/* semantics			*/
@@ -81,11 +76,9 @@ struct VSC_desc {
 	const char *ldesc;		/* long description		*/
 };
 
-struct VSC_point {
-	const struct VSC_desc *desc;	/* point description		*/
-	const volatile uint64_t *ptr;	/* field value			*/
-	const struct VSC_section *section;
-};
+struct VSC_point *VSC_Clone_Point(const struct VSC_point * const);
+
+void VSC_Destroy_Point(struct VSC_point **);
 
 typedef int VSC_iter_f(void *priv, const struct VSC_point *const pt);
 
diff --git a/lib/libvarnishapi/libvarnishapi.map b/lib/libvarnishapi/libvarnishapi.map
index 01c39f7..097be21 100644
--- a/lib/libvarnishapi/libvarnishapi.map
+++ b/lib/libvarnishapi/libvarnishapi.map
@@ -183,4 +183,6 @@ LIBVARNISHAPI_1.7 {
 	VSM_valid;
 	VSM_similar;
 	VSM_Map;
+	VSC_Destroy_Point;
+	VSC_Clone_Point;
 } LIBVARNISHAPI_1.0;
diff --git a/lib/libvarnishapi/vsc.c b/lib/libvarnishapi/vsc.c
index 737af89..298e9bb 100644
--- a/lib/libvarnishapi/vsc.c
+++ b/lib/libvarnishapi/vsc.c
@@ -58,7 +58,7 @@ struct vsc_vf {
 #define VSC_VF_MAGIC		0x516519f8
 	VTAILQ_ENTRY(vsc_vf)	list;
 	struct VSM_fantom	fantom;
-	struct VSC_section	section;
+	char			*ident;
 	struct vjsn		*vjsn;
 	int			order;
 };
@@ -68,6 +68,7 @@ struct vsc_pt {
 	unsigned		magic;
 #define VSC_PT_MAGIC		0xa4ff159a
 	VTAILQ_ENTRY(vsc_pt)	list;
+	char			*name;
 	struct VSC_point	point;
 };
 VTAILQ_HEAD(vsc_pt_head, vsc_pt);
@@ -110,6 +111,21 @@ static const size_t nlevels =
 
 /*--------------------------------------------------------------------*/
 
+struct VSC_point *
+VSC_Clone_Point(const struct VSC_point * const vp)
+{
+	return ((void*)(uintptr_t)vp);
+}
+
+void
+VSC_Destroy_Point(struct VSC_point **p)
+{
+	AN(p);
+	*p = NULL;
+}
+
+/*--------------------------------------------------------------------*/
+
 static struct vsc *
 vsc_setup(struct vsm *vd)
 {
@@ -153,6 +169,7 @@ vsc_delete_pt_list(struct vsc_pt_head *head)
 		pt = VTAILQ_FIRST(head);
 		CHECK_OBJ_NOTNULL(pt, VSC_PT_MAGIC);
 		VTAILQ_REMOVE(head, pt, list);
+		REPLACE(pt->name, NULL);
 		FREE_OBJ(pt);
 	}
 }
@@ -245,7 +262,7 @@ vsc_add_vf(struct vsc *vsc, const struct VSM_fantom *fantom, int order)
 	if (*vf->fantom.ident != '\0')
 		VSB_printf(vsb, ".%s", vf->fantom.ident);
 	AZ(VSB_finish(vsb));
-	REPLACE(vf->section.ident, VSB_data(vsb));
+	REPLACE(vf->ident, VSB_data(vsb));
 	VSB_destroy(&vsb);
 	vf->order = order;
 
@@ -260,24 +277,6 @@ vsc_add_vf(struct vsc *vsc, const struct VSM_fantom *fantom, int order)
 	return (vf);
 }
 
-/*lint -esym(528, vsc_add_pt) */
-/*lint -sem(vsc_add_pt, custodial(3)) */
-static void
-vsc_add_pt(struct vsc *vsc, const volatile void *ptr,
-    const struct VSC_desc *desc, const struct vsc_vf *vf)
-{
-	struct vsc_pt *pt;
-
-	ALLOC_OBJ(pt, VSC_PT_MAGIC);
-	AN(pt);
-
-	pt->point.desc = desc;
-	pt->point.ptr = ptr;
-	pt->point.section = &vf->section;
-
-	VTAILQ_INSERT_TAIL(&vsc->pt_list, pt, list);
-}
-
 /*--------------------------------------------------------------------
  */
 
@@ -331,24 +330,36 @@ vsc_build_pt_list(struct vsm *vd)
 	struct vsc *vsc = vsc_setup(vd);
 	struct vsc_vf *vf;
 	struct vjsn_val *vve, *vv, *vt;
-	struct VSC_desc *vdsc = NULL;
+	struct vsc_pt *pt;
+	struct vsb *vsb;
 
 	vsc_delete_pt_list(&vsc->pt_list);
+	vsb = VSB_new_auto();
+	AN(vsb);
 
 	VTAILQ_FOREACH(vf, &vsc->vf_list, list) {
 		vve = vjsn_child(vf->vjsn->value, "elem");
 		AN(vve);
 		VTAILQ_FOREACH(vv, &vve->children, list) {
-			vdsc = calloc(sizeof *vdsc, 1);
-			AN(vdsc);
+			ALLOC_OBJ(pt, VSC_PT_MAGIC);
+			AN(pt);
+
+			vt = vjsn_child(vv, "name");
+			AN(vt);
+			assert(vt->type == VJSN_STRING);
+
+			VSB_clear(vsb);
+			VSB_printf(vsb, "%s.%s", vf->ident, vt->value);
+			AZ(VSB_finish(vsb));
+			REPLACE(pt->name, VSB_data(vsb));
+			pt->point.name = pt->name;
 
 #define DOF(n, k)						\
 			vt = vjsn_child(vv, k);			\
 			AN(vt);					\
 			assert(vt->type == VJSN_STRING);	\
-			vdsc->n = vt->value;
+			pt->point.n = vt->value;
 
-			DOF(name,  "name");
 			DOF(ctype, "ctype");
 			DOF(sdesc, "oneliner");
 			DOF(ldesc, "docs");
@@ -358,13 +369,13 @@ vsc_build_pt_list(struct vsm *vd)
 			assert(vt->type == VJSN_STRING);
 
 			if (!strcmp(vt->value, "counter")) {
-				vdsc->semantics = 'c';
+				pt->point.semantics = 'c';
 			} else if (!strcmp(vt->value, "gauge")) {
-				vdsc->semantics = 'g';
+				pt->point.semantics = 'g';
 			} else if (!strcmp(vt->value, "bitmap")) {
-				vdsc->semantics = 'b';
+				pt->point.semantics = 'b';
 			} else {
-				vdsc->semantics = '?';
+				pt->point.semantics = '?';
 			}
 
 			vt = vjsn_child(vv, "format");
@@ -372,41 +383,38 @@ vsc_build_pt_list(struct vsm *vd)
 			assert(vt->type == VJSN_STRING);
 
 			if (!strcmp(vt->value, "integer")) {
-				vdsc->format = 'i';
+				pt->point.format = 'i';
 			} else if (!strcmp(vt->value, "bytes")) {
-				vdsc->format = 'B';
+				pt->point.format = 'B';
 			} else if (!strcmp(vt->value, "bitmap")) {
-				vdsc->format = 'b';
+				pt->point.format = 'b';
 			} else if (!strcmp(vt->value, "duration")) {
-				vdsc->format = 'd';
+				pt->point.format = 'd';
 			} else {
-				vdsc->format = '?';
+				pt->point.format = '?';
 			}
 
-			vdsc->level = &level_info;
+			pt->point.level = &level_info;
 
 			vt = vjsn_child(vv, "index");
 			AN(vt);
-			vsc_add_pt(vsc,
-			    (char*)vf->fantom.b + atoi(vt->value),
-			    vdsc, vf);
+
+			pt->point.ptr = (volatile void*)
+			    ((volatile char*)vf->fantom.b + atoi(vt->value));
+
+			VTAILQ_INSERT_TAIL(&vsc->pt_list, pt, list);
 		}
 	}
+	VSB_destroy(&vsb);
 }
 
 /*--------------------------------------------------------------------
  */
 
 static int
-vsc_filter_match_pt(struct vsb *vsb, const struct vsc_sf *sf, const
-    struct vsc_pt *pt)
+vsc_filter_match_pt(const struct vsc_sf *sf, const struct vsc_pt *pt)
 {
-	VSB_clear(vsb);
-	if (strcmp(pt->point.section->ident, ""))
-		VSB_printf(vsb, "%s.", pt->point.section->ident);
-	VSB_printf(vsb, "%s", pt->point.desc->name);
-	AZ(VSB_finish(vsb));
-	return (!fnmatch(sf->pattern, VSB_data(vsb), 0));
+	return (!fnmatch(sf->pattern, pt->point.name, 0));
 }
 
 static void
@@ -416,14 +424,11 @@ vsc_filter_pt_list(struct vsm *vd)
 	struct vsc_pt_head tmplist;
 	struct vsc_sf *sf;
 	struct vsc_pt *pt, *pt2;
-	struct vsb *vsb;
 
 	if (VTAILQ_EMPTY(&vsc->sf_list_include) &&
 	    VTAILQ_EMPTY(&vsc->sf_list_exclude))
 		return;
 
-	vsb = VSB_new_auto();
-	AN(vsb);
 	VTAILQ_INIT(&tmplist);
 
 	/* Include filters. Empty include filter list implies one that
@@ -434,7 +439,7 @@ vsc_filter_pt_list(struct vsm *vd)
 			CHECK_OBJ_NOTNULL(sf, VSC_SF_MAGIC);
 			VTAILQ_FOREACH_SAFE(pt, &vsc->pt_list, list, pt2) {
 				CHECK_OBJ_NOTNULL(pt, VSC_PT_MAGIC);
-				if (vsc_filter_match_pt(vsb, sf, pt)) {
+				if (vsc_filter_match_pt(sf, pt)) {
 					VTAILQ_REMOVE(&vsc->pt_list,
 					    pt, list);
 					VTAILQ_INSERT_TAIL(&tmplist,
@@ -451,15 +456,13 @@ vsc_filter_pt_list(struct vsm *vd)
 		CHECK_OBJ_NOTNULL(sf, VSC_SF_MAGIC);
 		VTAILQ_FOREACH_SAFE(pt, &vsc->pt_list, list, pt2) {
 			CHECK_OBJ_NOTNULL(pt, VSC_PT_MAGIC);
-			if (vsc_filter_match_pt(vsb, sf, pt)) {
+			if (vsc_filter_match_pt(sf, pt)) {
 				VTAILQ_REMOVE(&vsc->pt_list, pt, list);
 				VTAILQ_INSERT_TAIL(&tmplist, pt, list);
 			}
 		}
 	}
 	vsc_delete_pt_list(&tmplist);
-
-	VSB_destroy(&vsb);
 }
 
 /*--------------------------------------------------------------------
diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c
index 425e0f7..c4a7bd9 100644
--- a/lib/libvarnishapi/vsm.c
+++ b/lib/libvarnishapi/vsm.c
@@ -304,7 +304,7 @@ VSM_Open(struct vsm *vd)
 		    vd->iname, strerror(errno)));
 	}
 	/*
-	 * Force failure of client depends on remapping at same address. 
+	 * Force failure of client depends on remapping at same address.
 	 */
 	ppp = (char*)v + getpagesize() * 1000;
 



From phk at FreeBSD.org  Fri Jun 16 07:46:05 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 16 Jun 2017 09:46:05 +0200
Subject: [master] 674c8df Overhaul the VSC api.
Message-ID: <E1dLlxF-0000km-9Q@project.varnish-software.com>


commit 674c8df775c1564d1b97ee14c8ac0f5d492d387b
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Jun 16 07:43:17 2017 +0000

    Overhaul the VSC api.
    
    VSC_Iter() only provides instantaneous access to the point.  If you want
    to keep monitoring it (like varnishstat in curses mode) you must
    VSC_Clone_Point() the returned points

diff --git a/bin/varnishtest/tests/u00005.vtc b/bin/varnishtest/tests/u00005.vtc
index 2653854..47a036e 100644
--- a/bin/varnishtest/tests/u00005.vtc
+++ b/bin/varnishtest/tests/u00005.vtc
@@ -26,7 +26,7 @@ shell "cmp -s ${p1_out} ${p2_out}"
 
 shell -expect "MGT.uptime" \
 	"varnishstat -1 -n ${v1_name} -f ^MAIN*"
-shell -match "^MAIN" \
+shell -match "^MGT" \
 	"varnishstat -1 -n ${v1_name} -f ^foo"
 
 shell -expect "Usage: varnishstat <options>" \
diff --git a/include/vapi/vsc.h b/include/vapi/vsc.h
index b46593e..01ea236 100644
--- a/include/vapi/vsc.h
+++ b/include/vapi/vsc.h
@@ -54,8 +54,6 @@ int VSC_Arg(struct vsm *vd, int arg, const char *opt);
 	 */
 
 struct VSC_level_desc;
-struct VSC_type_desc;
-struct VSC_section;
 struct VSC_point;
 
 struct VSC_level_desc {
diff --git a/lib/libvarnishapi/vsc.c b/lib/libvarnishapi/vsc.c
index 298e9bb..a6935e4 100644
--- a/lib/libvarnishapi/vsc.c
+++ b/lib/libvarnishapi/vsc.c
@@ -53,26 +53,6 @@
 
 #include "vsc_priv.h"
 
-struct vsc_vf {
-	unsigned		magic;
-#define VSC_VF_MAGIC		0x516519f8
-	VTAILQ_ENTRY(vsc_vf)	list;
-	struct VSM_fantom	fantom;
-	char			*ident;
-	struct vjsn		*vjsn;
-	int			order;
-};
-VTAILQ_HEAD(vsc_vf_head, vsc_vf);
-
-struct vsc_pt {
-	unsigned		magic;
-#define VSC_PT_MAGIC		0xa4ff159a
-	VTAILQ_ENTRY(vsc_pt)	list;
-	char			*name;
-	struct VSC_point	point;
-};
-VTAILQ_HEAD(vsc_pt_head, vsc_pt);
-
 struct vsc_sf {
 	unsigned		magic;
 #define VSC_SF_MAGIC		0x558478dd
@@ -85,11 +65,8 @@ struct vsc {
 	unsigned		magic;
 #define VSC_MAGIC		0x3373554a
 
-	struct vsc_vf_head	vf_list;
-	struct vsc_pt_head	pt_list;
 	struct vsc_sf_head	sf_list_include;
 	struct vsc_sf_head	sf_list_exclude;
-	struct VSM_fantom	iter_fantom;
 };
 
 /*--------------------------------------------------------------------
@@ -114,13 +91,26 @@ static const size_t nlevels =
 struct VSC_point *
 VSC_Clone_Point(const struct VSC_point * const vp)
 {
-	return ((void*)(uintptr_t)vp);
+	struct VSC_point *pt;
+	char *p;
+
+	pt = calloc(sizeof *pt, 1);
+	AN(pt);
+	*pt = *vp;
+	p = strdup(pt->name); AN(p); pt->name = p;
+	p = strdup(pt->sdesc); AN(p); pt->sdesc = p;
+	p = strdup(pt->ldesc); AN(p); pt->ldesc = p;
+	return (pt);
 }
 
 void
 VSC_Destroy_Point(struct VSC_point **p)
 {
 	AN(p);
+	free(TRUST_ME((*p)->ldesc));
+	free(TRUST_ME((*p)->sdesc));
+	free(TRUST_ME((*p)->name));
+	free(*p);
 	*p = NULL;
 }
 
@@ -135,8 +125,6 @@ vsc_setup(struct vsm *vd)
 	if (vsc == NULL) {
 		ALLOC_OBJ(vsc, VSC_MAGIC);
 		AN(vsc);
-		VTAILQ_INIT(&vsc->vf_list);
-		VTAILQ_INIT(&vsc->pt_list);
 		VTAILQ_INIT(&vsc->sf_list_include);
 		VTAILQ_INIT(&vsc->sf_list_exclude);
 		VSM_SetVSC(vd, vsc);
@@ -148,33 +136,6 @@ vsc_setup(struct vsm *vd)
 /*--------------------------------------------------------------------*/
 
 static void
-vsc_delete_vf_list(struct vsc_vf_head *head)
-{
-	struct vsc_vf *vf;
-
-	while (!VTAILQ_EMPTY(head)) {
-		vf = VTAILQ_FIRST(head);
-		CHECK_OBJ_NOTNULL(vf, VSC_VF_MAGIC);
-		VTAILQ_REMOVE(head, vf, list);
-		FREE_OBJ(vf);
-	}
-}
-
-static void
-vsc_delete_pt_list(struct vsc_pt_head *head)
-{
-	struct vsc_pt *pt;
-
-	while (!VTAILQ_EMPTY(head)) {
-		pt = VTAILQ_FIRST(head);
-		CHECK_OBJ_NOTNULL(pt, VSC_PT_MAGIC);
-		VTAILQ_REMOVE(head, pt, list);
-		REPLACE(pt->name, NULL);
-		FREE_OBJ(pt);
-	}
-}
-
-static void
 vsc_delete_sf_list(struct vsc_sf_head *head)
 {
 	struct vsc_sf *sf;
@@ -195,8 +156,6 @@ VSC_Delete(struct vsc *vsc)
 	CHECK_OBJ_NOTNULL(vsc, VSC_MAGIC);
 	vsc_delete_sf_list(&vsc->sf_list_include);
 	vsc_delete_sf_list(&vsc->sf_list_exclude);
-	vsc_delete_pt_list(&vsc->pt_list);
-	vsc_delete_vf_list(&vsc->vf_list);
 	FREE_OBJ(vsc);
 }
 
@@ -245,224 +204,132 @@ VSC_Arg(struct vsm *vd, int arg, const char *opt)
 	}
 }
 
-/*--------------------------------------------------------------------*/
+/*--------------------------------------------------------------------
+ */
 
-static struct vsc_vf *
-vsc_add_vf(struct vsc *vsc, const struct VSM_fantom *fantom, int order)
+static int
+vsc_filter(struct vsm *vd, const char *nm)
 {
-	struct vsc_vf *vf, *vf2;
-	struct vsb *vsb;
-
-	ALLOC_OBJ(vf, VSC_VF_MAGIC);
-	AN(vf);
-	vf->fantom = *fantom;
-	vsb = VSB_new_auto();
-	AN(vsb);
-	VSB_printf(vsb, "%s", vf->fantom.type);
-	if (*vf->fantom.ident != '\0')
-		VSB_printf(vsb, ".%s", vf->fantom.ident);
-	AZ(VSB_finish(vsb));
-	REPLACE(vf->ident, VSB_data(vsb));
-	VSB_destroy(&vsb);
-	vf->order = order;
+	struct vsc *vsc = vsc_setup(vd);
+	struct vsc_sf *sf;
 
-	VTAILQ_FOREACH(vf2, &vsc->vf_list, list) {
-		if (vf->order < vf2->order)
-			break;
-	}
-	if (vf2 != NULL)
-		VTAILQ_INSERT_BEFORE(vf2, vf, list);
-	else
-		VTAILQ_INSERT_TAIL(&vsc->vf_list, vf, list);
-	return (vf);
+	VTAILQ_FOREACH(sf, &vsc->sf_list_exclude, list)
+		if (!fnmatch(sf->pattern, nm, 0))
+			return (1);
+	if (VTAILQ_EMPTY(&vsc->sf_list_include))
+		return (0);
+	VTAILQ_FOREACH(sf, &vsc->sf_list_include, list)
+		if (!fnmatch(sf->pattern, nm, 0))
+			return (0);
+	return (1);
 }
 
 /*--------------------------------------------------------------------
  */
 
-static void
-vsc_build_vf_list(struct vsm *vd)
+static int
+vsc_iter_elem(struct vsm *vd, const struct VSM_fantom *fantom,
+    const struct vjsn_val *vv, struct vsb *vsb, VSC_iter_f *func, void *priv)
 {
-	uint64_t u;
-	struct vsc *vsc = vsc_setup(vd);
-	const char *p;
-	const char *e;
-	struct vjsn *vj;
-	struct vjsn_val *vv;
-	struct vsc_vf *vf;
+	struct VSC_point	point;
+	struct vjsn_val *vt;
 
-	vsc_delete_pt_list(&vsc->pt_list);
-	vsc_delete_vf_list(&vsc->vf_list);
+	memset(&point, 0, sizeof point);
 
-	VSM_FOREACH(&vsc->iter_fantom, vd) {
-		if (strcmp(vsc->iter_fantom.class, VSC_CLASS))
-			continue;
-		AZ(VSM_Map(vd, &vsc->iter_fantom));
-		u = vbe64dec(vsc->iter_fantom.b);
-		if (u == 0) {
-			VRMB();
-			usleep(100000);
-			u = vbe64dec(vsc->iter_fantom.b);
-		}
-		assert(u > 0);
-		p = (char*)vsc->iter_fantom.b + 8 + u;
-		vj = vjsn_parse(p, &e);
-		if (e != NULL) {
-			fprintf(stderr, "%s\n", p);
-			fprintf(stderr, "JSON ERROR %s\n", e);
-			exit(2);
-		}
-		AN(vj);
-		vv = vjsn_child(vj->value, "order");
-		AN(vv);
-		assert(vv->type == VJSN_NUMBER);
-		vf = vsc_add_vf(vsc, &vsc->iter_fantom, atoi(vv->value));
-		AN(vf);
-		vf->vjsn = vj;
-		// vjsn_dump(vf->vjsn, stderr);
-		AZ(e);
-	}
-}
+	vt = vjsn_child(vv, "name");
+	AN(vt);
+	assert(vt->type == VJSN_STRING);
 
-static void
-vsc_build_pt_list(struct vsm *vd)
-{
-	struct vsc *vsc = vsc_setup(vd);
-	struct vsc_vf *vf;
-	struct vjsn_val *vve, *vv, *vt;
-	struct vsc_pt *pt;
-	struct vsb *vsb;
+	VSB_clear(vsb);
+	VSB_printf(vsb, "%s", fantom->type);
+	if (*fantom->ident)
+		VSB_printf(vsb, ".%s", fantom->ident);
+	VSB_printf(vsb, ".%s", vt->value);
+	AZ(VSB_finish(vsb));
 
-	vsc_delete_pt_list(&vsc->pt_list);
-	vsb = VSB_new_auto();
-	AN(vsb);
+	if (vsc_filter(vd, VSB_data(vsb)))
+		return (0);
+
+	point.name = VSB_data(vsb);
+
+#define DOF(n, k)				\
+	vt = vjsn_child(vv, k);			\
+	AN(vt);					\
+	assert(vt->type == VJSN_STRING);	\
+	point.n = vt->value;
 
-	VTAILQ_FOREACH(vf, &vsc->vf_list, list) {
-		vve = vjsn_child(vf->vjsn->value, "elem");
-		AN(vve);
-		VTAILQ_FOREACH(vv, &vve->children, list) {
-			ALLOC_OBJ(pt, VSC_PT_MAGIC);
-			AN(pt);
-
-			vt = vjsn_child(vv, "name");
-			AN(vt);
-			assert(vt->type == VJSN_STRING);
-
-			VSB_clear(vsb);
-			VSB_printf(vsb, "%s.%s", vf->ident, vt->value);
-			AZ(VSB_finish(vsb));
-			REPLACE(pt->name, VSB_data(vsb));
-			pt->point.name = pt->name;
-
-#define DOF(n, k)						\
-			vt = vjsn_child(vv, k);			\
-			AN(vt);					\
-			assert(vt->type == VJSN_STRING);	\
-			pt->point.n = vt->value;
-
-			DOF(ctype, "ctype");
-			DOF(sdesc, "oneliner");
-			DOF(ldesc, "docs");
+	DOF(ctype, "ctype");
+	DOF(sdesc, "oneliner");
+	DOF(ldesc, "docs");
 #undef DOF
-			vt = vjsn_child(vv, "type");
-			AN(vt);
-			assert(vt->type == VJSN_STRING);
-
-			if (!strcmp(vt->value, "counter")) {
-				pt->point.semantics = 'c';
-			} else if (!strcmp(vt->value, "gauge")) {
-				pt->point.semantics = 'g';
-			} else if (!strcmp(vt->value, "bitmap")) {
-				pt->point.semantics = 'b';
-			} else {
-				pt->point.semantics = '?';
-			}
-
-			vt = vjsn_child(vv, "format");
-			AN(vt);
-			assert(vt->type == VJSN_STRING);
-
-			if (!strcmp(vt->value, "integer")) {
-				pt->point.format = 'i';
-			} else if (!strcmp(vt->value, "bytes")) {
-				pt->point.format = 'B';
-			} else if (!strcmp(vt->value, "bitmap")) {
-				pt->point.format = 'b';
-			} else if (!strcmp(vt->value, "duration")) {
-				pt->point.format = 'd';
-			} else {
-				pt->point.format = '?';
-			}
-
-			pt->point.level = &level_info;
-
-			vt = vjsn_child(vv, "index");
-			AN(vt);
-
-			pt->point.ptr = (volatile void*)
-			    ((volatile char*)vf->fantom.b + atoi(vt->value));
-
-			VTAILQ_INSERT_TAIL(&vsc->pt_list, pt, list);
-		}
+	vt = vjsn_child(vv, "type");
+	AN(vt);
+	assert(vt->type == VJSN_STRING);
+
+	if (!strcmp(vt->value, "counter")) {
+		point.semantics = 'c';
+	} else if (!strcmp(vt->value, "gauge")) {
+		point.semantics = 'g';
+	} else if (!strcmp(vt->value, "bitmap")) {
+		point.semantics = 'b';
+	} else {
+		point.semantics = '?';
 	}
-	VSB_destroy(&vsb);
-}
 
-/*--------------------------------------------------------------------
- */
+	vt = vjsn_child(vv, "format");
+	AN(vt);
+	assert(vt->type == VJSN_STRING);
+
+	if (!strcmp(vt->value, "integer")) {
+		point.format = 'i';
+	} else if (!strcmp(vt->value, "bytes")) {
+		point.format = 'B';
+	} else if (!strcmp(vt->value, "bitmap")) {
+		point.format = 'b';
+	} else if (!strcmp(vt->value, "duration")) {
+		point.format = 'd';
+	} else {
+		point.format = '?';
+	}
 
-static int
-vsc_filter_match_pt(const struct vsc_sf *sf, const struct vsc_pt *pt)
-{
-	return (!fnmatch(sf->pattern, pt->point.name, 0));
+	point.level = &level_info;
+
+	vt = vjsn_child(vv, "index");
+	AN(vt);
+
+	point.ptr = (volatile void*)
+	    ((volatile char*)fantom->b + atoi(vt->value));
+
+	return (func(priv, &point));
 }
 
-static void
-vsc_filter_pt_list(struct vsm *vd)
+static int
+vsc_iter_fantom(struct vsm *vd, const struct VSM_fantom *fantom,
+    struct vsb *vsb, VSC_iter_f *func, void *priv)
 {
-	struct vsc *vsc = vsc_setup(vd);
-	struct vsc_pt_head tmplist;
-	struct vsc_sf *sf;
-	struct vsc_pt *pt, *pt2;
-
-	if (VTAILQ_EMPTY(&vsc->sf_list_include) &&
-	    VTAILQ_EMPTY(&vsc->sf_list_exclude))
-		return;
-
-	VTAILQ_INIT(&tmplist);
-
-	/* Include filters. Empty include filter list implies one that
-	 * matches everything. Points are sorted by the order of include
-	 * filter they match. */
-	if (!VTAILQ_EMPTY(&vsc->sf_list_include)) {
-		VTAILQ_FOREACH(sf, &vsc->sf_list_include, list) {
-			CHECK_OBJ_NOTNULL(sf, VSC_SF_MAGIC);
-			VTAILQ_FOREACH_SAFE(pt, &vsc->pt_list, list, pt2) {
-				CHECK_OBJ_NOTNULL(pt, VSC_PT_MAGIC);
-				if (vsc_filter_match_pt(sf, pt)) {
-					VTAILQ_REMOVE(&vsc->pt_list,
-					    pt, list);
-					VTAILQ_INSERT_TAIL(&tmplist,
-					    pt, list);
-				}
-			}
-		}
-		vsc_delete_pt_list(&vsc->pt_list);
-		VTAILQ_CONCAT(&vsc->pt_list, &tmplist, list);
+	int i = 0;
+	const char *p;
+	const char *e;
+	struct vjsn *vj;
+	struct vjsn_val *vv, *vve;
+
+	p = (char*)fantom->b + 8 + vbe64dec(fantom->b);
+	vj = vjsn_parse(p, &e);
+	if (e != NULL) {
+		fprintf(stderr, "%s\n", p);
+		fprintf(stderr, "JSON ERROR %s\n", e);
+		exit(2);
 	}
-
-	/* Exclude filters */
-	VTAILQ_FOREACH(sf, &vsc->sf_list_exclude, list) {
-		CHECK_OBJ_NOTNULL(sf, VSC_SF_MAGIC);
-		VTAILQ_FOREACH_SAFE(pt, &vsc->pt_list, list, pt2) {
-			CHECK_OBJ_NOTNULL(pt, VSC_PT_MAGIC);
-			if (vsc_filter_match_pt(sf, pt)) {
-				VTAILQ_REMOVE(&vsc->pt_list, pt, list);
-				VTAILQ_INSERT_TAIL(&tmplist, pt, list);
-			}
-		}
+	AN(vj);
+	vve = vjsn_child(vj->value, "elem");
+	AN(vve);
+	VTAILQ_FOREACH(vv, &vve->children, list) {
+		i = vsc_iter_elem(vd, fantom, vv, vsb, func, priv);
+		if (i)
+			break;
 	}
-	vsc_delete_pt_list(&tmplist);
+	// XXX: destroy vj
+	return (i);
 }
 
 /*--------------------------------------------------------------------
@@ -472,26 +339,33 @@ int
 VSC_Iter(struct vsm *vd, struct VSM_fantom *fantom, VSC_iter_f *func,
     void *priv)
 {
-	struct vsc *vsc = vsc_setup(vd);
-	struct vsc_pt *pt;
-	int i;
+	struct VSM_fantom	ifantom;
+	uint64_t u;
+	int i = 0;
+	struct vsb *vsb;
 
-	/* XXX: workaround: Force reload */
-	if (1 || VSM_valid != VSM_StillValid(vd, &vsc->iter_fantom)) {
-		/* Tell app that list will be nuked */
-		(void)func(priv, NULL);
-		vsc_build_vf_list(vd);
-		vsc_build_pt_list(vd);
-		vsc_filter_pt_list(vd);
-	}
-	if (fantom != NULL)
-		*fantom = vsc->iter_fantom;
-	VTAILQ_FOREACH(pt, &vsc->pt_list, list) {
-		i = func(priv, &pt->point);
+	vsb = VSB_new_auto();
+	AN(vsb);
+	VSM_FOREACH(&ifantom, vd) {
+		if (strcmp(ifantom.class, VSC_CLASS))
+			continue;
+		AZ(VSM_Map(vd, &ifantom));
+		u = vbe64dec(ifantom.b);
+		if (u == 0) {
+			VRMB();
+			usleep(100000);
+			u = vbe64dec(ifantom.b);
+		}
+		assert(u > 0);
+		if (fantom != NULL)
+			*fantom = ifantom;
+		i = vsc_iter_fantom(vd, &ifantom, vsb, func, priv);
+		// AZ(VSM_Unmap(vd, &ifantom));
 		if (i)
-			return (i);
+			break;
 	}
-	return (0);
+	VSB_destroy(&vsb);
+	return (i);
 }
 
 /*--------------------------------------------------------------------



From phk at FreeBSD.org  Fri Jun 16 21:11:05 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 16 Jun 2017 23:11:05 +0200
Subject: [master] 66d1732 Add VSM_Unmap()
Message-ID: <E1dLyWH-0000fs-Mn@project.varnish-software.com>


commit 66d1732100b069cb84f138d98b341189fbc215cd
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Jun 16 20:23:17 2017 +0000

    Add VSM_Unmap()

diff --git a/include/vapi/vsm.h b/include/vapi/vsm.h
index 3345371..4055b0a 100644
--- a/include/vapi/vsm.h
+++ b/include/vapi/vsm.h
@@ -154,6 +154,7 @@ int VSM__itern(const struct vsm *vd, struct VSM_fantom *vf);
 	 */
 
 int VSM_Map(struct vsm *vd, struct VSM_fantom *vf);
+int VSM_Unmap(struct vsm *vd, struct VSM_fantom *vf);
 
 struct vsm_valid {
 	const char *name;
diff --git a/lib/libvarnishapi/libvarnishapi.map b/lib/libvarnishapi/libvarnishapi.map
index 097be21..3a3fbe0 100644
--- a/lib/libvarnishapi/libvarnishapi.map
+++ b/lib/libvarnishapi/libvarnishapi.map
@@ -183,6 +183,7 @@ LIBVARNISHAPI_1.7 {
 	VSM_valid;
 	VSM_similar;
 	VSM_Map;
+	VSM_Unmap;
 	VSC_Destroy_Point;
 	VSC_Clone_Point;
 } LIBVARNISHAPI_1.0;
diff --git a/lib/libvarnishapi/vsc.c b/lib/libvarnishapi/vsc.c
index a6935e4..59f034b 100644
--- a/lib/libvarnishapi/vsc.c
+++ b/lib/libvarnishapi/vsc.c
@@ -360,7 +360,7 @@ VSC_Iter(struct vsm *vd, struct VSM_fantom *fantom, VSC_iter_f *func,
 		if (fantom != NULL)
 			*fantom = ifantom;
 		i = vsc_iter_fantom(vd, &ifantom, vsb, func, priv);
-		// AZ(VSM_Unmap(vd, &ifantom));
+		AZ(VSM_Unmap(vd, &ifantom));
 		if (i)
 			break;
 	}
diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c
index c4a7bd9..8efba40 100644
--- a/lib/libvarnishapi/vsm.c
+++ b/lib/libvarnishapi/vsm.c
@@ -459,6 +459,18 @@ VSM_Map(struct vsm *vd, struct VSM_fantom *vf)
 
 /*--------------------------------------------------------------------*/
 
+int
+VSM_Unmap(struct vsm *vd, struct VSM_fantom *vf)
+{
+
+	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
+	AN(vf);
+	AN(vf->b);
+	return (0);
+}
+
+/*--------------------------------------------------------------------*/
+
 const struct vsm_valid *
 VSM_StillValid(const struct vsm *vd, struct VSM_fantom *vf)
 {



From phk at FreeBSD.org  Mon Jun 19 05:57:06 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 19 Jun 2017 07:57:06 +0200
Subject: [master] b124cf4 Rename struct VSM_fantom to vsm_fantom to be
 consistent with the naming pattern which has evolved.
Message-ID: <E1dMpgQ-0003WR-9X@project.varnish-software.com>


commit b124cf41d8a99c52c9b556ff475fa220c2917f7c
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Jun 19 05:55:39 2017 +0000

    Rename struct VSM_fantom to vsm_fantom to be consistent with the
    naming pattern which has evolved.

diff --git a/bin/varnishadm/varnishadm.c b/bin/varnishadm/varnishadm.c
index f9177d0..b983c52 100644
--- a/bin/varnishadm/varnishadm.c
+++ b/bin/varnishadm/varnishadm.c
@@ -396,7 +396,7 @@ n_arg_sock(const char *n_arg)
 	struct vsm *vsm;
 	char *p;
 	int sock;
-	struct VSM_fantom vt;
+	struct vsm_fantom vt;
 
 	vsm = VSM_New();
 	AN(vsm);
diff --git a/bin/varnishstat/varnishstat_curses.c b/bin/varnishstat/varnishstat_curses.c
index ab32eca..791ce69 100644
--- a/bin/varnishstat/varnishstat_curses.c
+++ b/bin/varnishstat/varnishstat_curses.c
@@ -313,7 +313,7 @@ build_pt_list_cb(void *priv, const struct VSC_point *vpt)
 }
 
 static void
-build_pt_list(struct vsm *vd, struct VSM_fantom *fantom)
+build_pt_list(struct vsm *vd, struct vsm_fantom *fantom)
 {
 	struct pt_priv pt_priv;
 	int i;
@@ -1050,7 +1050,7 @@ do_curses(struct vsm *vd, double delay)
 	long t;
 	int ch;
 	double now;
-	struct VSM_fantom f_iter = VSM_FANTOM_NULL;
+	struct vsm_fantom f_iter = VSM_FANTOM_NULL;
 
 	interval = delay;
 
diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c
index 96046c2..1488ce6 100644
--- a/bin/varnishtest/vtc_varnish.c
+++ b/bin/varnishtest/vtc_varnish.c
@@ -77,7 +77,7 @@ struct varnish {
 	char			*proto;
 
 	struct vsm		*vd;		/* vsc use */
-	struct VSM_fantom	mgt_arg_i;
+	struct vsm_fantom	mgt_arg_i;
 
 	unsigned		vsl_tag_count[256];
 
diff --git a/include/vapi/vsc.h b/include/vapi/vsc.h
index 01ea236..022ccfc 100644
--- a/include/vapi/vsc.h
+++ b/include/vapi/vsc.h
@@ -38,7 +38,7 @@
 #include "vapi/vsc_int.h"
 
 struct vsm;
-struct VSM_fantom;
+struct vsm_fantom;
 
 /*---------------------------------------------------------------------
  * VSC level access functions
@@ -80,13 +80,13 @@ void VSC_Destroy_Point(struct VSC_point **);
 
 typedef int VSC_iter_f(void *priv, const struct VSC_point *const pt);
 
-int VSC_Iter(struct vsm *vd, struct VSM_fantom *fantom, VSC_iter_f *func,
+int VSC_Iter(struct vsm *vd, struct vsm_fantom *fantom, VSC_iter_f *func,
     void *priv);
 	/*
 	 * Iterate over all statistics counters, calling "func" for
 	 * each counter not suppressed by any "-f" arguments.
 	 *
-	 * fantom points to a struct VSM_fantom. If non-NULL, it can be
+	 * fantom points to a struct vsm_fantom. If non-NULL, it can be
 	 * used with VSM_StillValid to check the validity of the points
 	 * returned.
 	 *
diff --git a/include/vapi/vsm.h b/include/vapi/vsm.h
index 4055b0a..a463e9a 100644
--- a/include/vapi/vsm.h
+++ b/include/vapi/vsm.h
@@ -43,7 +43,7 @@ struct vsm;
  * This structure is used to reference a VSM chunk
  */
 
-struct VSM_fantom {
+struct vsm_fantom {
 	struct VSM_chunk	*chunk;
 	void			*b;		/* first byte of payload */
 	void			*e;		/* first byte past payload */
@@ -142,19 +142,19 @@ void VSM_Close(struct vsm *vd);
 	 */
 
 
-void VSM__iter0(const struct vsm *vd, struct VSM_fantom *vf);
-int VSM__itern(const struct vsm *vd, struct VSM_fantom *vf);
+void VSM__iter0(const struct vsm *vd, struct vsm_fantom *vf);
+int VSM__itern(const struct vsm *vd, struct vsm_fantom *vf);
 
 #define VSM_FOREACH(vf, vd) \
     for (VSM__iter0((vd), (vf)); VSM__itern((vd), (vf));)
 	/*
 	 * Iterate over all chunks in shared memory
-	 * vf = "struct VSM_fantom *"
+	 * vf = "struct vsm_fantom *"
 	 * vd = "struct vsm *"
 	 */
 
-int VSM_Map(struct vsm *vd, struct VSM_fantom *vf);
-int VSM_Unmap(struct vsm *vd, struct VSM_fantom *vf);
+int VSM_Map(struct vsm *vd, struct vsm_fantom *vf);
+int VSM_Unmap(struct vsm *vd, struct vsm_fantom *vf);
 
 struct vsm_valid {
 	const char *name;
@@ -165,9 +165,9 @@ extern const struct vsm_valid VSM_valid[];
 extern const struct vsm_valid VSM_similar[];
 
 const struct vsm_valid *VSM_StillValid(const struct vsm *vd,
-    struct VSM_fantom *vf);
+    struct vsm_fantom *vf);
 	/*
-	 * Check the validity of a previously looked up VSM_fantom.
+	 * Check the validity of a previously looked up vsm_fantom.
 	 *
 	 * VSM_invalid means that the SHM chunk this fantom points to does
 	 * not exist in the log file any longer. Using the fantom's
@@ -196,7 +196,7 @@ const struct vsm_valid *VSM_StillValid(const struct vsm *vd,
 	 *   VSM_similar: a fantom with same dimensions exist in same position.
 	 */
 
-int VSM_Get(const struct vsm *vd, struct VSM_fantom *vf,
+int VSM_Get(const struct vsm *vd, struct vsm_fantom *vf,
     const char *class, const char *ident);
 	/*
 	 * Find a chunk, produce fantom for it.
diff --git a/lib/libvarnishapi/vsc.c b/lib/libvarnishapi/vsc.c
index 59f034b..c4245b7 100644
--- a/lib/libvarnishapi/vsc.c
+++ b/lib/libvarnishapi/vsc.c
@@ -228,7 +228,7 @@ vsc_filter(struct vsm *vd, const char *nm)
  */
 
 static int
-vsc_iter_elem(struct vsm *vd, const struct VSM_fantom *fantom,
+vsc_iter_elem(struct vsm *vd, const struct vsm_fantom *fantom,
     const struct vjsn_val *vv, struct vsb *vsb, VSC_iter_f *func, void *priv)
 {
 	struct VSC_point	point;
@@ -304,7 +304,7 @@ vsc_iter_elem(struct vsm *vd, const struct VSM_fantom *fantom,
 }
 
 static int
-vsc_iter_fantom(struct vsm *vd, const struct VSM_fantom *fantom,
+vsc_iter_fantom(struct vsm *vd, const struct vsm_fantom *fantom,
     struct vsb *vsb, VSC_iter_f *func, void *priv)
 {
 	int i = 0;
@@ -336,10 +336,10 @@ vsc_iter_fantom(struct vsm *vd, const struct VSM_fantom *fantom,
  */
 
 int
-VSC_Iter(struct vsm *vd, struct VSM_fantom *fantom, VSC_iter_f *func,
+VSC_Iter(struct vsm *vd, struct vsm_fantom *fantom, VSC_iter_f *func,
     void *priv)
 {
-	struct VSM_fantom	ifantom;
+	struct vsm_fantom	ifantom;
 	uint64_t u;
 	int i = 0;
 	struct vsb *vsb;
diff --git a/lib/libvarnishapi/vsl_cursor.c b/lib/libvarnishapi/vsl_cursor.c
index b393fe8..c161abb 100644
--- a/lib/libvarnishapi/vsl_cursor.c
+++ b/lib/libvarnishapi/vsl_cursor.c
@@ -64,7 +64,7 @@ struct vslc_vsm {
 	unsigned			options;
 
 	struct vsm			*vsm;
-	struct VSM_fantom		vf;
+	struct vsm_fantom		vf;
 
 	const struct VSL_head		*head;
 	const uint32_t			*end;
@@ -241,7 +241,7 @@ struct VSL_cursor *
 VSL_CursorVSM(struct VSL_data *vsl, struct vsm *vsm, unsigned options)
 {
 	struct vslc_vsm *c;
-	struct VSM_fantom vf;
+	struct vsm_fantom vf;
 	struct VSL_head *head;
 	int i;
 
diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c
index 8efba40..8af47fd 100644
--- a/lib/libvarnishapi/vsm.c
+++ b/lib/libvarnishapi/vsm.c
@@ -205,7 +205,7 @@ VSM_n_Arg(struct vsm *vd, const char *arg)
 const char *
 VSM_Name(const struct vsm *vd)
 {
-	struct VSM_fantom vt;
+	struct vsm_fantom vt;
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
 	if (VSM_Get(vd, &vt, "Arg", "-i"))
@@ -386,7 +386,7 @@ VSM_Abandoned(struct vsm *vd)
 /*--------------------------------------------------------------------*/
 
 void
-VSM__iter0(const struct vsm *vd, struct VSM_fantom *vf)
+VSM__iter0(const struct vsm *vd, struct vsm_fantom *vf)
 {
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
@@ -396,7 +396,7 @@ VSM__iter0(const struct vsm *vd, struct VSM_fantom *vf)
 }
 
 int
-VSM__itern(const struct vsm *vd, struct VSM_fantom *vf)
+VSM__itern(const struct vsm *vd, struct vsm_fantom *vf)
 {
 	struct VSM_chunk *c = NULL;
 
@@ -448,7 +448,7 @@ VSM__itern(const struct vsm *vd, struct VSM_fantom *vf)
 /*--------------------------------------------------------------------*/
 
 int
-VSM_Map(struct vsm *vd, struct VSM_fantom *vf)
+VSM_Map(struct vsm *vd, struct vsm_fantom *vf)
 {
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
@@ -460,7 +460,7 @@ VSM_Map(struct vsm *vd, struct VSM_fantom *vf)
 /*--------------------------------------------------------------------*/
 
 int
-VSM_Unmap(struct vsm *vd, struct VSM_fantom *vf)
+VSM_Unmap(struct vsm *vd, struct vsm_fantom *vf)
 {
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
@@ -472,9 +472,9 @@ VSM_Unmap(struct vsm *vd, struct VSM_fantom *vf)
 /*--------------------------------------------------------------------*/
 
 const struct vsm_valid *
-VSM_StillValid(const struct vsm *vd, struct VSM_fantom *vf)
+VSM_StillValid(const struct vsm *vd, struct vsm_fantom *vf)
 {
-	struct VSM_fantom f2;
+	struct vsm_fantom f2;
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
 	AN(vf);
@@ -502,7 +502,7 @@ VSM_StillValid(const struct vsm *vd, struct VSM_fantom *vf)
 }
 
 int
-VSM_Get(const struct vsm *vd, struct VSM_fantom *vf,
+VSM_Get(const struct vsm *vd, struct vsm_fantom *vf,
     const char *class, const char *ident)
 {
 



From dridi.boukelmoune at gmail.com  Wed Jun 21 13:15:06 2017
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Wed, 21 Jun 2017 15:15:06 +0200
Subject: [master] 3abbe2b Universal failure for VRT_purge
Message-ID: <E1dNfTO-0004AE-Pu@project.varnish-software.com>


commit 3abbe2bcdfc671693656564e3dad196b9210370d
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Mon Jun 5 17:28:22 2017 +0200

    Universal failure for VRT_purge
    
    Coverage via a simple debug.purge() wrapper.

diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c
index c69914c..3e88844 100644
--- a/bin/varnishd/cache/cache_vrt.c
+++ b/bin/varnishd/cache/cache_vrt.c
@@ -539,11 +539,17 @@ VRT_purge(VRT_CTX, double ttl, double grace, double keep)
 {
 
 	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
+
+	if (ctx->method != VCL_MET_HIT && ctx->method != VCL_MET_MISS) {
+		VSLb(ctx->vsl, SLT_VCL_Error,
+		    "purge can only happen in vcl_hit{} or vcl_miss{}");
+		VRT_handling(ctx, VCL_RET_FAIL);
+		return;
+	}
+
 	CHECK_OBJ_NOTNULL(ctx->req, REQ_MAGIC);
 	CHECK_OBJ_NOTNULL(ctx->req->wrk, WORKER_MAGIC);
-	if (ctx->method == VCL_MET_HIT || ctx->method == VCL_MET_MISS)
-		HSH_Purge(ctx->req->wrk, ctx->req->objcore->objhead,
-		    ttl, grace, keep);
+	HSH_Purge(ctx->req->wrk, ctx->req->objcore->objhead, ttl, grace, keep);
 }
 
 /*--------------------------------------------------------------------
diff --git a/bin/varnishtest/tests/r02339.vtc b/bin/varnishtest/tests/r02339.vtc
new file mode 100644
index 0000000..d9e7fbe
--- /dev/null
+++ b/bin/varnishtest/tests/r02339.vtc
@@ -0,0 +1,183 @@
+varnishtest "VMOD purge"
+
+server s1 -repeat 12 {
+	rxreq
+	txresp
+} -start
+
+varnish v1 -arg "-p thread_pools=1" -vcl+backend {
+	import debug;
+
+	sub vcl_recv {
+		if (req.url == "recv") { debug.purge(); }
+		if (req.url == "pass") { return (pass); }
+		if (req.url == "purge") { return (purge); }
+		if (req.url == "synth") { return (synth(200)); }
+	}
+	sub vcl_hash {
+		if (req.url == "hash") { debug.purge(); }
+	}
+	sub vcl_miss {
+		if (req.url == "miss") { debug.purge(); }
+	}
+	sub vcl_hit {
+		if (req.url == "hit") { debug.purge(); }
+	}
+	sub vcl_purge {
+		if (req.url == "purge") { debug.purge(); }
+	}
+	sub vcl_pass {
+		if (req.url == "pass") { debug.purge(); }
+	}
+	sub vcl_deliver {
+		if (req.url == "deliver") { debug.purge(); }
+	}
+	sub vcl_synth {
+		if (req.url == "synth") { debug.purge(); }
+	}
+	sub vcl_backend_fetch {
+		if (bereq.url == "fetch") { debug.purge(); }
+		if (bereq.url == "error") {
+			set bereq.backend = debug.no_backend();
+		}
+	}
+	sub vcl_backend_error {
+		if (bereq.url == "error") { debug.purge(); }
+	}
+	sub vcl_backend_response {
+		if (bereq.url == "response") { debug.purge(); }
+	}
+} -start
+
+logexpect l0 -v v1 -g raw {
+	expect * 0 ExpKill	"EXP_Expired x=1002"
+} -start
+
+logexpect l1 -v v1 {
+	expect * 1002	Begin		"bereq 1001 fetch"
+
+	expect * 1003	VCL_call	HIT
+	expect * =	VCL_Log		PURGE
+	expect * =	VCL_call	DELIVER
+
+	expect * 1004	VCL_call	MISS
+	expect * =	VCL_Log		PURGE
+	expect * =	VCL_call	DELIVER
+
+	expect * 1007	VCL_call	RECV
+	expect * =	VCL_Log		PURGE
+	expect * =	VCL_Error	purge
+	expect * =	VCL_return	fail
+
+	expect * 1009	VCL_call	HASH
+	expect * =	VCL_Log		PURGE
+	expect * =	VCL_Error	purge
+	expect * =	VCL_return	fail
+
+	expect * 1011	VCL_call	PURGE
+	expect * =	VCL_Log		PURGE
+	expect * =	VCL_Error	purge
+
+	expect * 1013	VCL_call	PASS
+	expect * =	VCL_Log		PURGE
+	expect * =	VCL_Error	purge
+
+	expect * 1015	VCL_call	DELIVER
+	expect * =	VCL_Log		PURGE
+	expect * =	VCL_Error	purge
+
+	expect * 1018	VCL_call	SYNTH
+	expect * =	VCL_Log		PURGE
+	expect * =	VCL_Error	purge
+
+	expect * 1021	VCL_call	BACKEND_FETCH
+	expect * =	VCL_Log		PURGE
+	expect * =	VCL_Error	purge
+
+	expect * 1024	VCL_call	BACKEND_ERROR
+	expect * =	VCL_Log		PURGE
+	expect * =	VCL_Error	purge
+
+	expect * 1027	VCL_call	BACKEND_RESPONSE
+	expect * =	VCL_Log		PURGE
+	expect * =	VCL_Error	purge
+} -start
+
+client c1 {
+	txreq -url hit
+	rxresp
+	expect resp.status == 200
+
+	txreq -url hit
+	rxresp
+	expect resp.status == 200
+
+	txreq -url miss
+	rxresp
+	expect resp.status == 200
+} -run
+
+logexpect l0 -wait
+
+client c1 {
+	txreq -url recv
+	rxresp
+	expect resp.status == 503
+	expect_close
+} -run
+
+client c1 {
+	txreq -url hash
+	rxresp
+	expect resp.status == 503
+	expect_close
+} -run
+
+client c1 {
+	txreq -url purge
+	rxresp
+	expect resp.status == 503
+	expect_close
+} -run
+
+client c1 {
+	txreq -url pass
+	rxresp
+	expect resp.status == 503
+	expect_close
+} -run
+
+client c1 {
+	txreq -url deliver
+	rxresp
+	expect resp.status == 503
+	expect_close
+} -run
+
+client c1 {
+	txreq -url synth
+	expect_close
+} -run
+
+client c1 {
+	txreq -url fetch
+	rxresp
+	expect resp.status == 503
+	expect_close
+} -run
+
+client c1 {
+	txreq -url error
+	rxresp
+	expect resp.status == 503
+	expect_close
+} -run
+
+client c1 {
+	txreq -url response
+	rxresp
+	expect resp.status == 503
+	expect_close
+} -run
+
+logexpect l1 -wait
diff --git a/doc/changes.rst b/doc/changes.rst
index eef34c2..436b4e0 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -6,6 +6,9 @@ Varnish Cache Trunk (ongoing)
 
 * On platforms with setproctitle(3), use the -i argument rather than -n
 
+* VRT_purge fails a transaction if used outside of ``vcl_hit`` and
+  ``vcl_miss`` (#2339)
+
 * VSM_Name() returns the -i argument name, rather than the -n name.
 
 * VUT.name is goine, use VSM_Name(VUT.vsm)
diff --git a/include/vrt.h b/include/vrt.h
index b8377f3..3308634 100644
--- a/include/vrt.h
+++ b/include/vrt.h
@@ -41,6 +41,7 @@
  *
  * 6.1 (unreleased):
  *	http_CollectHdrSep added
+ *	VRT_purge modified (may fail a transaction)
  * 6.0 (2017-03-15):
  *	VRT_hit_for_pass added
  *	VRT_ipcmp added
diff --git a/lib/libvmod_debug/vmod.vcc b/lib/libvmod_debug/vmod.vcc
index b33d3af..9323488 100644
--- a/lib/libvmod_debug/vmod.vcc
+++ b/lib/libvmod_debug/vmod.vcc
@@ -219,3 +219,7 @@ Returns the size in bytes of a collection of C-datatypes.
 * 'z' = size_t
 * 'o' = off_t
 * 'j' = intmax_t
+
+$Function VOID purge(DURATION ttl = 0, DURATION grace = 0, DURATION keep = 0)
+
+Performs a purge, by default a hard purge where ttl, grace and keep are zero.
diff --git a/lib/libvmod_debug/vmod_debug.c b/lib/libvmod_debug/vmod_debug.c
index d22c1a1..d8e2910 100644
--- a/lib/libvmod_debug/vmod_debug.c
+++ b/lib/libvmod_debug/vmod_debug.c
@@ -572,3 +572,13 @@ vmod_typesize(VRT_CTX, VCL_STRING s)
 	}
 	return ((VCL_INT)i);
 }
+
+VCL_VOID
+vmod_purge(VRT_CTX, double ttl, double grace, double keep)
+{
+
+	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
+
+	VSLb(ctx->vsl, SLT_VCL_Log, "PURGE");
+	VRT_purge(ctx, ttl, grace, keep);
+}



From Marilyn at varnish-cache.org  Fri Jun 23 11:28:12 2017
From: Marilyn at varnish-cache.org (Marilyn)
Date: Fri, 23 Jun 2017 16:58:12 +0530
Subject: Emailing: 943577669.gif
Message-ID: <000c01d1666f$957c2840$c07478c0$@illerart1.com>

Your message is ready to be sent with the following file or link
attachments:

943577669.gif


Note: To protect against computer viruses, e-mail programs may prevent
sending or receiving certain types of file attachments.  Check your e-mail
security settings to determine how attachments are handled.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 943577669.gif.zip
Type: application/zip
Size: 7552 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-commit/attachments/20170623/042c88d4/attachment.zip>

From hermunn at varnish-software.com  Fri Jun 23 11:57:06 2017
From: hermunn at varnish-software.com (Pål Hermunn Johansen)
Date: Fri, 23 Jun 2017 13:57:06 +0200
Subject: [4.1] 26bb5f0 Prevent storage backends name collisions
Message-ID: <E1dOND0-0002qH-5J@project.varnish-software.com>


commit 26bb5f0252395ebd600948d5109214cb02359fd0
Author: P?l Hermunn Johansen <hermunn at varnish-software.com>
Date:   Thu Jun 15 12:03:39 2017 +0200

    Prevent storage backends name collisions
    
    This is a backport of 986763198 by Dridi Boukelmoune, but this patch
    is a bit different from the original.
    
    Now name collisions with transient will be reported in a sane way
    instead of producing an assert. Normal name collisions are also more
    readable.
    
    The test case is unmodified, thanks to resent back porting efforts in
    varnishtest.
    
    Fixes #2321
    
    Conflicts:
    	bin/varnishd/storage/mgt_stevedore.c

diff --git a/bin/varnishd/storage/mgt_stevedore.c b/bin/varnishd/storage/mgt_stevedore.c
index 109290f..247574c 100644
--- a/bin/varnishd/storage/mgt_stevedore.c
+++ b/bin/varnishd/storage/mgt_stevedore.c
@@ -116,6 +116,28 @@ static const struct choice STV_choice[] = {
 	{ NULL,		NULL }
 };
 
+static void
+stv_check_ident(const char *spec, const char *ident)
+{
+	struct stevedore *stv;
+	unsigned found = 0;
+
+	if (!strcmp(ident, TRANSIENT_STORAGE))
+		found = (stv_transient != NULL);
+	else {
+		VTAILQ_FOREACH(stv, &stv_stevedores, list) {
+			CHECK_OBJ(stv, STEVEDORE_MAGIC);
+			if (!strcmp(stv->ident, ident)) {
+				found = 1;
+				break;
+			}
+		}
+	}
+
+	if (found)
+		ARGV_ERR("(-s %s) '%s' is already defined\n", spec, ident);
+}
+
 void
 STV_Config(const char *spec)
 {
@@ -170,17 +192,12 @@ STV_Config(const char *spec)
 		bprintf(stv->ident, "%.*s", l, spec);
 	}
 
-	VTAILQ_FOREACH(stv2, &stv_stevedores, list) {
-		if (strcmp(stv2->ident, stv->ident))
-			continue;
-		ARGV_ERR("(-s%s=%s) already defined once\n",
-		    stv->ident, stv->name);
-	}
+	stv_check_ident(spec, stv->ident);
 
 	if (stv->init != NULL)
 		stv->init(stv, ac, av);
 	else if (ac != 0)
-		ARGV_ERR("(-s%s) too many arguments\n", stv->name);
+		ARGV_ERR("(-s %s) too many arguments\n", stv->name);
 
 	AN(stv->alloc);
 	if (stv->allocobj == NULL)
diff --git a/bin/varnishtest/tests/r02321.vtc b/bin/varnishtest/tests/r02321.vtc
new file mode 100644
index 0000000..33c7658
--- /dev/null
+++ b/bin/varnishtest/tests/r02321.vtc
@@ -0,0 +1,19 @@
+varnishtest "Storage name collisions"
+
+# intentional collision
+shell -err -expect "Error: (-s main=malloc,100m) 'main' is already defined" {
+	exec varnishd -a :0 -f '' -n ${tmpdir} \
+		-s main=malloc,10m -s main=malloc,100m
+}
+
+# pseudo-accidental collision
+shell -err -expect "Error: (-s s0=malloc,100m) 's0' is already defined" {
+	exec varnishd -a :0 -f '' -n ${tmpdir} \
+		-s malloc,10m -s s0=malloc,100m
+}
+
+# Transient collision
+shell -err -expect "Error: (-s Transient=malloc,100m) 'Transient' is already defined" {
+	exec varnishd -a :0 -f '' -n ${tmpdir} \
+		-s Transient=malloc,10m -s Transient=malloc,100m
+}



From daghf at varnish-software.com  Fri Jun 23 12:52:05 2017
From: daghf at varnish-software.com (Dag Haavi Finstad)
Date: Fri, 23 Jun 2017 14:52:05 +0200
Subject: [master] afb5c06 Don't assert on missing HTTP2-Settings header
Message-ID: <E1dOO4D-00055w-R1@project.varnish-software.com>


commit afb5c068957eee0920568ed1c73448fae91fab7e
Author: Dag Haavi Finstad <daghf at varnish-software.com>
Date:   Thu Jun 8 13:46:36 2017 +0200

    Don't assert on missing HTTP2-Settings header

diff --git a/bin/varnishd/http2/cache_http2_session.c b/bin/varnishd/http2/cache_http2_session.c
index 0e04a84..80f50aa 100644
--- a/bin/varnishd/http2/cache_http2_session.c
+++ b/bin/varnishd/http2/cache_http2_session.c
@@ -155,9 +155,9 @@ h2_b64url_settings(struct h2_sess *h2, struct req *req)
 	 * If there is trouble with this, we could reject the upgrade
 	 * but putting this on the H1 side is just plain wrong...
 	 */
-	AN(http_GetHdr(req->http, H_HTTP2_Settings, &p));
-	if (p == NULL)
+	if (!http_GetHdr(req->http, H_HTTP2_Settings, &p))
 		return (-1);
+	AN(p);
 	VSLb(req->vsl, SLT_Debug, "H2CS %s", p);
 
 	n = 0;
diff --git a/bin/varnishtest/tests/t02001.vtc b/bin/varnishtest/tests/t02001.vtc
index d744822..1da6fb3 100644
--- a/bin/varnishtest/tests/t02001.vtc
+++ b/bin/varnishtest/tests/t02001.vtc
@@ -139,3 +139,20 @@ varnish v1 -expect MEMPOOL.req0.live == 0
 varnish v1 -expect MEMPOOL.req1.live == 0
 varnish v1 -expect MEMPOOL.sess0.live == 0
 varnish v1 -expect MEMPOOL.sess1.live == 0
+
+
+client c1 {
+	# Missing HTTP2-Settings
+	send "GET /noupgrade HTTP/1.1\r\n"
+	send "Host: foo.bar\r\n"
+	send "Upgrade: h2c\r\n"
+	send "\r\n"
+	expect_close
+} -run
+
+varnish v1 -vsl_catchup
+
+varnish v1 -expect MEMPOOL.req0.live == 0
+varnish v1 -expect MEMPOOL.req1.live == 0
+varnish v1 -expect MEMPOOL.sess0.live == 0
+varnish v1 -expect MEMPOOL.sess1.live == 0



From hermunn at varnish-software.com  Fri Jun 23 14:56:05 2017
From: hermunn at varnish-software.com (Pål Hermunn Johansen)
Date: Fri, 23 Jun 2017 16:56:05 +0200
Subject: [4.1] 3912645 fix the std.getenv example
Message-ID: <E1dOQ0D-00022A-L5@project.varnish-software.com>


commit 3912645048b575242a0c655ccf53ca0894d931db
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Thu Mar 2 09:55:50 2017 +0100

    fix the std.getenv example

diff --git a/lib/libvmod_std/vmod.vcc b/lib/libvmod_std/vmod.vcc
index e7f1d6f..3c058d0 100644
--- a/lib/libvmod_std/vmod.vcc
+++ b/lib/libvmod_std/vmod.vcc
@@ -286,7 +286,7 @@ Description
 
 	See getenv(3)
 Example
-	| set req.http.My-Env = getenv("MY_ENV");
+	| set req.http.My-Env = std.getenv("MY_ENV");
 
 SEE ALSO
 ========



From dridi.boukelmoune at gmail.com  Mon Jun 26 15:38:06 2017
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Mon, 26 Jun 2017 17:38:06 +0200
Subject: [master] 085c6a9 Make varnishd -f honor the vcl_path
Message-ID: <E1dPW5W-0001RJ-Gb@project.varnish-software.com>


commit 085c6a9fc345875b18b853cc274648d2ed34e853
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Thu Jun 8 23:59:51 2017 +0200

    Make varnishd -f honor the vcl_path

diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c
index e8cd0df..31f9bf7 100644
--- a/bin/varnishd/mgt/mgt_main.c
+++ b/bin/varnishd/mgt/mgt_main.c
@@ -73,6 +73,9 @@ static char		Cn_arg[] = "/tmp/varnishd_C_XXXXXXX";
 static struct vpf_fh *pfh1 = NULL;
 static struct vpf_fh *pfh2 = NULL;
 
+static struct vfil_path *vcl_path = NULL;
+VTAILQ_HEAD(,f_arg) f_args = VTAILQ_HEAD_INITIALIZER(f_args);
+
 int optreset;	// Some has it, some doesn't.  Cheaper than auto*
 
 /*--------------------------------------------------------------------*/
@@ -442,6 +445,27 @@ struct f_arg {
 	VTAILQ_ENTRY(f_arg)	list;
 };
 
+static void
+mgt_f_read(const char *fn)
+{
+	struct f_arg *fa;
+	char *f, *fnp;
+
+	ALLOC_OBJ(fa, F_ARG_MAGIC);
+	AN(fa);
+	REPLACE(fa->farg, fn);
+	VFIL_setpath(&vcl_path, mgt_vcl_path);
+	if (VFIL_searchpath(vcl_path, NULL, &f, fn, &fnp) || f == NULL) {
+		ARGV_ERR("Cannot read -f file '%s' (%s)\n",
+		    fnp != NULL ? fnp : fn, strerror(errno));
+		free(fnp);
+	}
+	free(fa->farg);
+	fa->farg = fnp;
+	fa->src = f;
+	VTAILQ_INSERT_TAIL(&f_args, fa, list);
+}
+
 static const char opt_spec[] = "a:b:Cdf:Fh:i:I:j:l:M:n:P:p:r:S:s:T:t:VW:x:";
 
 int
@@ -473,7 +497,6 @@ main(int argc, char * const *argv)
 	struct vev *e;
 	struct f_arg *fa;
 	struct vsb *vsb;
-	VTAILQ_HEAD(,f_arg) f_args = VTAILQ_HEAD_INITIALIZER(f_args);
 	pid_t pid;
 
 	setbuf(stdout, NULL);
@@ -626,14 +649,7 @@ main(int argc, char * const *argv)
 				novcl = 1;
 				break;
 			}
-			ALLOC_OBJ(fa, F_ARG_MAGIC);
-			AN(fa);
-			REPLACE(fa->farg, optarg);
-			fa->src = VFIL_readfile(NULL, fa->farg, NULL);
-			if (fa->src == NULL)
-				ARGV_ERR("Cannot read -f file (%s): %s\n",
-				    fa->farg, strerror(errno));
-			VTAILQ_INSERT_TAIL(&f_args, fa, list);
+			mgt_f_read(optarg);
 			break;
 		case 'h':
 			h_arg = optarg;
diff --git a/bin/varnishtest/tests/r02342.vtc b/bin/varnishtest/tests/r02342.vtc
new file mode 100644
index 0000000..bb37b6a
--- /dev/null
+++ b/bin/varnishtest/tests/r02342.vtc
@@ -0,0 +1,20 @@
+varnishtest "honor vcl_path with varnishd -f"
+
+shell {
+	cat >unlikely_file_name.vcl <<-EOF
+	vcl 4.0;
+
+	backend be {
+		.host = "${bad_backend}";
+	}
+	EOF
+}
+
+shell -err -expect "Cannot read -f file 'unlikely_file_name.vcl'" {
+	varnishd -F -a :0 -n ${tmpdir}/tmp -f unlikely_file_name.vcl
+}
+
+varnish v1 -arg "-p vcl_path=${tmpdir} -f unlikely_file_name.vcl" -start
+
+# Even when loaded from a relative path, show an absolute one
+varnish v1 -cliexpect "VCL.SHOW .+ ${tmpdir}/unlikely_file_name.vcl" "vcl.show -v boot"



From dridi.boukelmoune at gmail.com  Mon Jun 26 15:38:06 2017
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Mon, 26 Jun 2017 17:38:06 +0200
Subject: [master] d0c6b12 Document varnishd -f with regards to vcl_path
Message-ID: <E1dPW5W-0001RP-IV@project.varnish-software.com>


commit d0c6b128c90018e61b9c36851233ed4e1c048c4b
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Wed Jun 21 14:35:30 2017 +0200

    Document varnishd -f with regards to vcl_path

diff --git a/doc/changes.rst b/doc/changes.rst
index 436b4e0..a40d0a3 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -11,7 +11,9 @@ Varnish Cache Trunk (ongoing)
 
 * VSM_Name() returns the -i argument name, rather than the -n name.
 
-* VUT.name is goine, use VSM_Name(VUT.vsm)
+* VUT.name is gone, use VSM_Name(VUT.vsm)
+
+* varnishd honors vcl_path (#2342)
 
 ================================
 Varnish Cache 5.1.2 (2017-04-07)
diff --git a/doc/sphinx/reference/varnishd.rst b/doc/sphinx/reference/varnishd.rst
index 1d63dbe..4261ce2 100644
--- a/doc/sphinx/reference/varnishd.rst
+++ b/doc/sphinx/reference/varnishd.rst
@@ -70,6 +70,13 @@ Basic options
   script (-I option) and the child process will be started if there
   is an active VCL at the end of the initialization.
 
+  When used with a relative file name, config is searched in the
+  ``vcl_path``. It is possible to set this path prior to using ``-f``
+  options with a ``-p`` option. During startup, `varnishd` doesn't
+  complain about unsafe VCL paths: unlike the `varnish-cli(7)` that
+  could later be accessed remotely, starting `varnishd` requires
+  local privileges.
+
 -n name
 
   Specify the name for this instance.  This name is used to construct



From nils.goroll at uplex.de  Mon Jun 26 17:09:05 2017
From: nils.goroll at uplex.de (Nils Goroll)
Date: Mon, 26 Jun 2017 19:09:05 +0200
Subject: [master] 6b6969a Don't panic on a null ban
Message-ID: <E1dPXVZ-0005Lh-R3@project.varnish-software.com>


commit 6b6969aa542023aed36b6de0a52745210fc702b5
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Mon Jun 26 18:01:33 2017 +0200

    Don't panic on a null ban
    
    like ban(req.http.non-existing-header)

diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c
index 3e88844..e1e521b 100644
--- a/bin/varnishd/cache/cache_vrt.c
+++ b/bin/varnishd/cache/cache_vrt.c
@@ -457,7 +457,11 @@ VRT_ban_string(VRT_CTX, const char *str)
 
 	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
 	AN(ctx->vsl);
-	AN(str);
+
+	if (str == NULL) {
+		VSLb(ctx->vsl, SLT_VCL_Error, "ban(): Null argument");
+		return;
+	}
 
 	bp = BAN_Build();
 	if (bp == NULL) {
diff --git a/bin/varnishtest/tests/v00011.vtc b/bin/varnishtest/tests/v00011.vtc
index 0f470ec..ea79b64 100644
--- a/bin/varnishtest/tests/v00011.vtc
+++ b/bin/varnishtest/tests/v00011.vtc
@@ -11,6 +11,7 @@ server s1 {
 varnish v1 -vcl+backend {
 	sub vcl_recv {
 		if (req.method == "PURGE") {
+			ban(req.http.doesntexist);
 			ban("");
 			ban("req.url");
 			ban("req.url //");
@@ -31,6 +32,7 @@ client c1 {
 } -run
 
 logexpect l1 -v v1 -d 1 -g vxid {
+	expect * 1004	VCL_Error {ban[(][)]: Null argument}
 	expect * 1004	VCL_Error {ban[(][)]: No ban conditions found[.]}
 	expect * 1004	VCL_Error {ban[(][)]: Expected comparison operator[.]}
 	expect * 1004	VCL_Error {ban[(][)]: Expected second operand[.]}



From nils.goroll at uplex.de  Mon Jun 26 17:09:05 2017
From: nils.goroll at uplex.de (Nils Goroll)
Date: Mon, 26 Jun 2017 19:09:05 +0200
Subject: [master] 132a0d0 Document current behavior for bans on non-existing
 fields
Message-ID: <E1dPXVZ-0005Ln-Sq@project.varnish-software.com>


commit 132a0d0685a1fd9a5a3e36ca6fd8f051a84b8f7d
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Mon Jun 26 19:02:17 2017 +0200

    Document current behavior for bans on non-existing fields

diff --git a/doc/sphinx/reference/vcl.rst b/doc/sphinx/reference/vcl.rst
index 075c345..cb50b6e 100644
--- a/doc/sphinx/reference/vcl.rst
+++ b/doc/sphinx/reference/vcl.rst
@@ -428,6 +428,11 @@ ban(STRING)
   Expressions can be chained using the *and* operator ``&&``. For *or*
   semantics, use several bans.
 
+  The unset *<field>* is not equal to any string, such that, for a
+  non-existing header, the operators ``==`` and ``~`` always evaluate
+  as false, while the operators ``!=`` and ``!~`` always evaluate as
+  true, respectively, for any value of *<arg>*.
+
 hash_data(input)
 ~~~~~~~~~~~~~~~~
 



From dridi.boukelmoune at gmail.com  Tue Jun 27 10:03:06 2017
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Tue, 27 Jun 2017 12:03:06 +0200
Subject: [master] 745a6e0 s/mcf_bad_vclname/mcf_invalid_vclname/
Message-ID: <E1dPnKs-0005Gm-6f@project.varnish-software.com>


commit 745a6e08905f05a5dda425b8aee20c12d7f7c6ef
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Jun 27 10:50:58 2017 +0200

    s/mcf_bad_vclname/mcf_invalid_vclname/

diff --git a/bin/varnishd/mgt/mgt_vcl.c b/bin/varnishd/mgt/mgt_vcl.c
index 59f2026..27ee0cb 100644
--- a/bin/varnishd/mgt/mgt_vcl.c
+++ b/bin/varnishd/mgt/mgt_vcl.c
@@ -114,7 +114,7 @@ mcf_vcl_byname(const char *name)
 }
 
 static int
-mcf_bad_vclname(struct cli *cli, const char *name)
+mcf_invalid_vclname(struct cli *cli, const char *name)
 {
 	const char *p;
 	int bad = 0;
@@ -145,7 +145,7 @@ mcf_find_vcl(struct cli *cli, const char *name)
 {
 	struct vclprog *vp;
 
-	if (mcf_bad_vclname(cli, name))
+	if (mcf_invalid_vclname(cli, name))
 		return (NULL);
 
 	vp = mcf_vcl_byname(name);
@@ -160,7 +160,7 @@ static int
 mcf_find_no_vcl(struct cli *cli, const char *name)
 {
 
-	if (mcf_bad_vclname(cli, name))
+	if (mcf_invalid_vclname(cli, name))
 		return (0);
 
 	if (mcf_vcl_byname(name) != NULL) {
@@ -791,9 +791,9 @@ mcf_vcl_label(struct cli *cli, const char * const *av, void *priv)
 	int i;
 
 	(void)priv;
-	if (mcf_bad_vclname(cli, av[2]))
+	if (mcf_invalid_vclname(cli, av[2]))
 		return;
-	if (mcf_bad_vclname(cli, av[3]))
+	if (mcf_invalid_vclname(cli, av[3]))
 		return;
 	vpt = mcf_find_vcl(cli, av[3]);
 	if (vpt == NULL)



From dridi.boukelmoune at gmail.com  Tue Jun 27 10:03:06 2017
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Tue, 27 Jun 2017 12:03:06 +0200
Subject: [master] e7ed02c Enforce proper symbol names for stevedores
Message-ID: <E1dPnKs-0005Gs-94@project.varnish-software.com>


commit e7ed02c9ae8d4b15053ae1b1e0c043d2326a13cd
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Thu Jun 22 10:34:07 2017 +0200

    Enforce proper symbol names for stevedores
    
    For that a new VCT_invalid_name function is here to complain about
    malformed symbol names. It is used for both storage backends and VCL
    names at the moment and will also be used for named listen addresses
    when it comes to that, and symbols in general for libvcc.
    
    Refs #2325
    Refs #2354

diff --git a/bin/varnishd/mgt/mgt_vcl.c b/bin/varnishd/mgt/mgt_vcl.c
index 27ee0cb..afe8532 100644
--- a/bin/varnishd/mgt/mgt_vcl.c
+++ b/bin/varnishd/mgt/mgt_vcl.c
@@ -116,28 +116,21 @@ mcf_vcl_byname(const char *name)
 static int
 mcf_invalid_vclname(struct cli *cli, const char *name)
 {
-	const char *p;
-	int bad = 0;
+	const char *bad;
 
 	AN(name);
-	p = name;
-	if (!vct_isalpha(*p))
-		bad = *p;
-	for (p++; bad == 0 && *p != '\0'; p++)
-		if (!vct_isalpha(*p) &&
-		    !vct_isdigit(*p) &&
-		    *p != '_' &&
-		    *p != '-')
-			bad = *p;
-	if (bad) {
+	bad = VCT_invalid_name(name, NULL);
+
+	if (bad != NULL) {
 		VCLI_SetResult(cli, CLIS_PARAM);
 		VCLI_Out(cli, "Illegal character in VCL name ");
-		if (bad > 0x20 && bad < 0x7f)
-			VCLI_Out(cli, "('%c')", bad);
+		if (*bad > 0x20 && *bad < 0x7f)
+			VCLI_Out(cli, "('%c')", *bad);
 		else
-			VCLI_Out(cli, "(0x%02x)", bad & 0xff);
+			VCLI_Out(cli, "(0x%02x)", *bad & 0xff);
+		return (-1);
 	}
-	return (bad);
+	return (0);
 }
 
 static struct vclprog *
diff --git a/bin/varnishd/storage/mgt_stevedore.c b/bin/varnishd/storage/mgt_stevedore.c
index b05ecf8..0b0dac2 100644
--- a/bin/varnishd/storage/mgt_stevedore.c
+++ b/bin/varnishd/storage/mgt_stevedore.c
@@ -42,6 +42,7 @@
 
 #include "storage/storage.h"
 #include "vav.h"
+#include "vct.h"
 
 static VTAILQ_HEAD(, stevedore) stevedores =
     VTAILQ_HEAD_INITIALIZER(stevedores);
@@ -189,6 +190,9 @@ STV_Config(const char *spec)
 	if (p == NULL)
 		bprintf(stv->ident, "s%u", seq++);
 	else {
+		if (VCT_invalid_name(spec, p) != NULL)
+			ARGV_ERR("invalid storage name (-s %s)\n", spec);
+		/* XXX: no need for truncation once VSM ident becomes dynamic */
 		l = p - spec;
 		if (l > sizeof stv->ident - 1)
 			l = sizeof stv->ident - 1;
diff --git a/bin/varnishtest/tests/r02325.vtc b/bin/varnishtest/tests/r02325.vtc
new file mode 100644
index 0000000..0c6ad4d
--- /dev/null
+++ b/bin/varnishtest/tests/r02325.vtc
@@ -0,0 +1,5 @@
+varnishtest "validate storage identifiers"
+
+shell -err -expect "Error: invalid storage name (-s ...=malloc)" {
+	varnishd -a :0 -n ${tmpdir} -F -f '' -s ...=malloc
+}
diff --git a/include/vct.h b/include/vct.h
index 5e1af85..86b30cf 100644
--- a/include/vct.h
+++ b/include/vct.h
@@ -44,6 +44,8 @@
 
 extern const uint16_t vct_typtab[256];
 
+const char *VCT_invalid_name(const char *b, const char *e);
+
 static inline int
 vct_is(int x, uint16_t y)
 {
diff --git a/lib/libvarnish/vct.c b/lib/libvarnish/vct.c
index 3ba4046..5610566 100644
--- a/lib/libvarnish/vct.c
+++ b/lib/libvarnish/vct.c
@@ -30,8 +30,11 @@
 
 #include "config.h"
 
+#include <stdlib.h>
 #include <stdint.h>
+#include <string.h>
 
+#include "vas.h"
 #include "vct.h"
 
 /* NB: VCT always operate in ASCII, don't replace 0x0d with \r etc. */
@@ -232,3 +235,25 @@ const uint16_t vct_typtab[256] = {
 	[0xfe]	=	VCT_XMLNAMESTART,
 	[0xff]	=	VCT_XMLNAMESTART,
 };
+
+const char *
+VCT_invalid_name(const char *b, const char *e)
+{
+
+	AN(b);
+	if (e == NULL)
+		e = strchr(b, '\0');
+	assert(b < e);
+
+	if (!vct_isalpha(*b))
+		return (b);
+
+	for (; b < e; b++)
+		if (!vct_isalpha(*b) &&
+		    !vct_isdigit(*b) &&
+		    *b != '_' &&
+		    *b != '-')
+			return (b);
+
+	return (NULL);
+}



From dridi.boukelmoune at gmail.com  Tue Jun 27 10:03:06 2017
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Tue, 27 Jun 2017 12:03:06 +0200
Subject: [master] 56f7ff4 New character types for identifiers and variables
Message-ID: <E1dPnKs-0005H1-As@project.varnish-software.com>


commit 56f7ff4d67ab9b58206f98888a8da4e08aaee051
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Jun 27 11:26:24 2017 +0200

    New character types for identifiers and variables
    
    The name are borrowed from libvcc.
    
    Refs #2354

diff --git a/include/vct.h b/include/vct.h
index 86b30cf..2893e6b 100644
--- a/include/vct.h
+++ b/include/vct.h
@@ -41,6 +41,8 @@
 #define VCT_XMLNAMESTART	(1<<7)
 #define VCT_XMLNAME		(1<<8)
 #define VCT_TCHAR		(1<<9)
+#define VCT_NAME		(1<<10)
+#define VCT_VAR			(1<<11)
 
 extern const uint16_t vct_typtab[256];
 
@@ -62,6 +64,9 @@ vct_is(int x, uint16_t y)
 #define vct_isalpha(x) vct_is(x, VCT_ALPHA)
 #define vct_issep(x) vct_is(x, VCT_SEPARATOR)
 #define vct_issepctl(x) vct_is(x, VCT_SEPARATOR | VCT_CTL)
+#define vct_isident1(x) vct_isalpha(x)
+#define vct_isident(x) vct_is(x, VCT_ALPHA | VCT_DIGIT | VCT_NAME)
+#define vct_isvar(x) vct_is(x, VCT_ALPHA | VCT_DIGIT | VCT_NAME | VCT_VAR)
 #define vct_isxmlnamestart(x) vct_is(x, VCT_XMLNAMESTART)
 #define vct_isxmlname(x) vct_is(x, VCT_XMLNAMESTART | VCT_XMLNAME)
 #define vct_istchar(x) vct_is(x, VCT_ALPHA | VCT_DIGIT | VCT_TCHAR)
diff --git a/lib/libvarnish/vct.c b/lib/libvarnish/vct.c
index 5610566..7d82dd3 100644
--- a/lib/libvarnish/vct.c
+++ b/lib/libvarnish/vct.c
@@ -88,8 +88,8 @@ const uint16_t vct_typtab[256] = {
 	[0x2a]  =	VCT_TCHAR,
 	[0x2b]  =	VCT_TCHAR,
 	[0x2c]  =	VCT_SEPARATOR,
-	[0x2d]	=	VCT_XMLNAME | VCT_TCHAR,
-	[0x2e]	=	VCT_XMLNAME | VCT_TCHAR,
+	[0x2d]	=	VCT_XMLNAME | VCT_TCHAR | VCT_NAME,
+	[0x2e]	=	VCT_XMLNAME | VCT_TCHAR | VCT_VAR,
 	[0x2f]  =	VCT_SEPARATOR,
 	[0x30]	=	VCT_DIGIT | VCT_HEX | VCT_XMLNAME,
 	[0x31]	=	VCT_DIGIT | VCT_HEX | VCT_XMLNAME,
@@ -138,7 +138,7 @@ const uint16_t vct_typtab[256] = {
 	[0x5c]  =	VCT_SEPARATOR,
 	[0x5d]  =	VCT_SEPARATOR,
 	[0x5e]  =	VCT_TCHAR,
-	[0x5f]	=	VCT_XMLNAMESTART | VCT_TCHAR,
+	[0x5f]	=	VCT_XMLNAMESTART | VCT_TCHAR | VCT_NAME,
 	[0x60]	=	VCT_TCHAR,
 	[0x61]	=	VCT_LOALPHA | VCT_HEX | VCT_XMLNAMESTART,
 	[0x62]	=	VCT_LOALPHA | VCT_HEX | VCT_XMLNAMESTART,
@@ -245,14 +245,11 @@ VCT_invalid_name(const char *b, const char *e)
 		e = strchr(b, '\0');
 	assert(b < e);
 
-	if (!vct_isalpha(*b))
+	if (!vct_isident1(*b))
 		return (b);
 
 	for (; b < e; b++)
-		if (!vct_isalpha(*b) &&
-		    !vct_isdigit(*b) &&
-		    *b != '_' &&
-		    *b != '-')
+		if (!vct_isident(*b))
 			return (b);
 
 	return (NULL);
diff --git a/lib/libvcc/vcc_compile.h b/lib/libvcc/vcc_compile.h
index 5de8a2a..2ab1a66 100644
--- a/lib/libvcc/vcc_compile.h
+++ b/lib/libvcc/vcc_compile.h
@@ -49,9 +49,6 @@ struct vsb;
 struct token;
 struct sockaddr_storage;
 
-#define isident1(c) (isalpha(c))
-#define isident(c) (isalpha(c) || isdigit(c) || (c) == '_' || (c) == '-')
-#define isvar(c) (isident(c) || (c) == '.')
 unsigned vcl_fixed_token(const char *p, const char **q);
 extern const char * const vcl_tnames[256];
 void vcl_output_lang_h(struct vsb *sb);
diff --git a/lib/libvcc/vcc_token.c b/lib/libvcc/vcc_token.c
index 3113722..c38ea32 100644
--- a/lib/libvcc/vcc_token.c
+++ b/lib/libvcc/vcc_token.c
@@ -36,6 +36,8 @@
 
 #include "vcc_compile.h"
 
+#include "vct.h"
+
 /*--------------------------------------------------------------------*/
 
 void
@@ -300,6 +302,7 @@ vcc_ExpectCid(struct vcc *tl, const char *what)
 
 	ExpectErr(tl, ID);
 	ERRCHK(tl);
+	/* XXX: too soon to use vct_invalid_name() */
 	for (q = tl->t->b; q < tl->t->e; q++) {
 		if (!isalnum(*q) && *q != '_') {
 			VSB_printf(tl->sb, "Name of %s, ", what);
@@ -488,9 +491,9 @@ vcc_Lexer(struct vcc *tl, struct source *sp)
 		}
 
 		/* Match Identifiers */
-		if (isident1(*p)) {
+		if (vct_isident1(*p)) {
 			for (q = p; q < sp->e; q++)
-				if (!isvar(*q))
+				if (!vct_isvar(*q))
 					break;
 			vcc_AddToken(tl, ID, p, q);
 			p = q;



From dridi.boukelmoune at gmail.com  Tue Jun 27 10:03:06 2017
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Tue, 27 Jun 2017 12:03:06 +0200
Subject: [master] 3b2c6d3 Move libvcc away from is* macros
Message-ID: <E1dPnKs-0005HA-Ce@project.varnish-software.com>


commit 3b2c6d33465484014cd324fd832713a475ccddde
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Jun 27 11:40:35 2017 +0200

    Move libvcc away from is* macros
    
    Refs #2354

diff --git a/include/vct.h b/include/vct.h
index 2893e6b..e1332d8 100644
--- a/include/vct.h
+++ b/include/vct.h
@@ -43,6 +43,8 @@
 #define VCT_TCHAR		(1<<9)
 #define VCT_NAME		(1<<10)
 #define VCT_VAR			(1<<11)
+#define VCT_VT			(1<<12)
+#define VCT_SPACE		(VCT_LWS | VCT_VT)
 
 extern const uint16_t vct_typtab[256];
 
@@ -60,8 +62,10 @@ vct_is(int x, uint16_t y)
 #define vct_ishex(x) vct_is(x, VCT_HEX)
 #define vct_islws(x) vct_is(x, VCT_LWS)
 #define vct_isctl(x) vct_is(x, VCT_CTL)
+#define vct_isspace(x) vct_is(x, VCT_SPACE)
 #define vct_isdigit(x) vct_is(x, VCT_DIGIT)
 #define vct_isalpha(x) vct_is(x, VCT_ALPHA)
+#define vct_isalnum(x) vct_is(x, VCT_ALPHA | VCT_DIGIT)
 #define vct_issep(x) vct_is(x, VCT_SEPARATOR)
 #define vct_issepctl(x) vct_is(x, VCT_SEPARATOR | VCT_CTL)
 #define vct_isident1(x) vct_isalpha(x)
diff --git a/lib/libvarnish/vct.c b/lib/libvarnish/vct.c
index 7d82dd3..d8e1118 100644
--- a/lib/libvarnish/vct.c
+++ b/lib/libvarnish/vct.c
@@ -54,7 +54,7 @@ const uint16_t vct_typtab[256] = {
 	[0x08]	=	VCT_CTL,
 	[0x09]	=	VCT_CTL | VCT_SP | VCT_SEPARATOR,
 	[0x0a]	=	VCT_CTL | VCT_CRLF,
-	[0x0b]	=	VCT_CTL,
+	[0x0b]	=	VCT_CTL | VCT_VT,
 	[0x0c]	=	VCT_CTL,
 	[0x0d]	=	VCT_CTL | VCT_CRLF,
 	[0x0e]	=	VCT_CTL,
diff --git a/lib/libvcc/vcc_token.c b/lib/libvcc/vcc_token.c
index c38ea32..a262ded 100644
--- a/lib/libvcc/vcc_token.c
+++ b/lib/libvcc/vcc_token.c
@@ -29,7 +29,6 @@
 
 #include "config.h"
 
-#include <ctype.h>
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
@@ -304,7 +303,7 @@ vcc_ExpectCid(struct vcc *tl, const char *what)
 	ERRCHK(tl);
 	/* XXX: too soon to use vct_invalid_name() */
 	for (q = tl->t->b; q < tl->t->e; q++) {
-		if (!isalnum(*q) && *q != '_') {
+		if (!vct_isalnum(*q) && *q != '_') {
 			VSB_printf(tl->sb, "Name of %s, ", what);
 			vcc_ErrToken(tl, tl->t);
 			VSB_printf(tl->sb,
@@ -371,7 +370,7 @@ vcc_Lexer(struct vcc *tl, struct source *sp)
 	for (p = sp->b; p < sp->e; ) {
 
 		/* Skip any whitespace */
-		if (isspace(*p)) {
+		if (vct_isspace(*p)) {
 			p++;
 			continue;
 		}
@@ -501,9 +500,9 @@ vcc_Lexer(struct vcc *tl, struct source *sp)
 		}
 
 		/* Match numbers { [0-9]+ } */
-		if (isdigit(*p)) {
+		if (vct_isdigit(*p)) {
 			for (q = p; q < sp->e; q++)
-				if (!isdigit(*q))
+				if (!vct_isdigit(*q))
 					break;
 			vcc_AddToken(tl, CNUM, p, q);
 			p = q;



From dridi.boukelmoune at gmail.com  Tue Jun 27 10:03:06 2017
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Tue, 27 Jun 2017 12:03:06 +0200
Subject: [master] ff4f115 GC unused includes
Message-ID: <E1dPnKs-0005HJ-EU@project.varnish-software.com>


commit ff4f115a9617962fbc9269791a80898622a95e01
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Jun 27 11:53:18 2017 +0200

    GC unused includes

diff --git a/lib/libvcc/generate.py b/lib/libvcc/generate.py
index 94275d4..015a9e6 100755
--- a/lib/libvcc/generate.py
+++ b/lib/libvcc/generate.py
@@ -1277,9 +1277,6 @@ fo.write("""
 
 #include "config.h"
 
-#include <ctype.h>
-#include <stdio.h>
-
 #include "vcc_compile.h"
 """)
 



From dridi.boukelmoune at gmail.com  Tue Jun 27 10:24:06 2017
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Tue, 27 Jun 2017 12:24:06 +0200
Subject: [master] 1a14e5e I was initially going for vct_name...
Message-ID: <E1dPnfC-0006C5-B5@project.varnish-software.com>


commit 1a14e5eef8283057095af3c1a68138e315688ca3
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Jun 27 12:12:00 2017 +0200

    I was initially going for vct_name...
    
    Refs #2354

diff --git a/include/vct.h b/include/vct.h
index e1332d8..7659d99 100644
--- a/include/vct.h
+++ b/include/vct.h
@@ -41,7 +41,8 @@
 #define VCT_XMLNAMESTART	(1<<7)
 #define VCT_XMLNAME		(1<<8)
 #define VCT_TCHAR		(1<<9)
-#define VCT_NAME		(1<<10)
+#define VCT_ID			(1<<10)
+#define VCT_IDENT		(VCT_ALPHA | VCT_DIGIT | VCT_ID)
 #define VCT_VAR			(1<<11)
 #define VCT_VT			(1<<12)
 #define VCT_SPACE		(VCT_LWS | VCT_VT)
@@ -69,8 +70,8 @@ vct_is(int x, uint16_t y)
 #define vct_issep(x) vct_is(x, VCT_SEPARATOR)
 #define vct_issepctl(x) vct_is(x, VCT_SEPARATOR | VCT_CTL)
 #define vct_isident1(x) vct_isalpha(x)
-#define vct_isident(x) vct_is(x, VCT_ALPHA | VCT_DIGIT | VCT_NAME)
-#define vct_isvar(x) vct_is(x, VCT_ALPHA | VCT_DIGIT | VCT_NAME | VCT_VAR)
+#define vct_isident(x) vct_is(x, VCT_IDENT)
+#define vct_isvar(x) vct_is(x, VCT_IDENT | VCT_VAR)
 #define vct_isxmlnamestart(x) vct_is(x, VCT_XMLNAMESTART)
 #define vct_isxmlname(x) vct_is(x, VCT_XMLNAMESTART | VCT_XMLNAME)
 #define vct_istchar(x) vct_is(x, VCT_ALPHA | VCT_DIGIT | VCT_TCHAR)
diff --git a/lib/libvarnish/vct.c b/lib/libvarnish/vct.c
index d8e1118..087a9ca 100644
--- a/lib/libvarnish/vct.c
+++ b/lib/libvarnish/vct.c
@@ -88,7 +88,7 @@ const uint16_t vct_typtab[256] = {
 	[0x2a]  =	VCT_TCHAR,
 	[0x2b]  =	VCT_TCHAR,
 	[0x2c]  =	VCT_SEPARATOR,
-	[0x2d]	=	VCT_XMLNAME | VCT_TCHAR | VCT_NAME,
+	[0x2d]	=	VCT_XMLNAME | VCT_TCHAR | VCT_ID,
 	[0x2e]	=	VCT_XMLNAME | VCT_TCHAR | VCT_VAR,
 	[0x2f]  =	VCT_SEPARATOR,
 	[0x30]	=	VCT_DIGIT | VCT_HEX | VCT_XMLNAME,
@@ -138,7 +138,7 @@ const uint16_t vct_typtab[256] = {
 	[0x5c]  =	VCT_SEPARATOR,
 	[0x5d]  =	VCT_SEPARATOR,
 	[0x5e]  =	VCT_TCHAR,
-	[0x5f]	=	VCT_XMLNAMESTART | VCT_TCHAR | VCT_NAME,
+	[0x5f]	=	VCT_XMLNAMESTART | VCT_TCHAR | VCT_ID,
 	[0x60]	=	VCT_TCHAR,
 	[0x61]	=	VCT_LOALPHA | VCT_HEX | VCT_XMLNAMESTART,
 	[0x62]	=	VCT_LOALPHA | VCT_HEX | VCT_XMLNAMESTART,



From dridi.boukelmoune at gmail.com  Tue Jun 27 12:14:06 2017
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Tue, 27 Jun 2017 14:14:06 +0200
Subject: [master] eef7871 ASCII ALL THE THINGS
Message-ID: <E1dPpNe-0002DC-5d@project.varnish-software.com>


commit eef7871e509d97c6376f3e808a75e5ac4fe1bf74
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Jun 27 14:13:20 2017 +0200

    ASCII ALL THE THINGS

diff --git a/include/vct.h b/include/vct.h
index 7659d99..24143a3 100644
--- a/include/vct.h
+++ b/include/vct.h
@@ -76,7 +76,7 @@ vct_is(int x, uint16_t y)
 #define vct_isxmlname(x) vct_is(x, VCT_XMLNAMESTART | VCT_XMLNAME)
 #define vct_istchar(x) vct_is(x, VCT_ALPHA | VCT_DIGIT | VCT_TCHAR)
 
-#define vct_iscrlf(p) (((p)[0] == '\r' && (p)[1] == '\n') || (p)[0] == '\n')
+#define vct_iscrlf(p) (((p)[0] == 0x0d && (p)[1] == 0x0a) || (p)[0] == 0x0a)
 
 /* NB: VCT always operate in ASCII, don't replace 0x0d with \r etc. */
 #define vct_skipcrlf(p) ((p)[0] == 0x0d && (p)[1] == 0x0a ? 2 : 1)



From dridi.boukelmoune at gmail.com  Tue Jun 27 13:02:06 2017
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Tue, 27 Jun 2017 15:02:06 +0200
Subject: [master] a0b34bd Collapse VCT checks
Message-ID: <E1dPq86-0004M0-Ah@project.varnish-software.com>


commit a0b34bd410447aa6082bfeffd55567b2c97b5649
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Jun 27 15:01:22 2017 +0200

    Collapse VCT checks

diff --git a/bin/varnishd/cache/cache_http.c b/bin/varnishd/cache/cache_http.c
index 9be4805..e1b9117 100644
--- a/bin/varnishd/cache/cache_http.c
+++ b/bin/varnishd/cache/cache_http.c
@@ -679,7 +679,7 @@ http_GetContentLength(const struct http *hp)
 	cl = 0;
 	if (!vct_isdigit(*b))
 		return (-2);
-	for (;vct_isdigit(*b); b++) {
+	for (; vct_isdigit(*b); b++) {
 		cll = cl;
 		cl *= 10;
 		cl += *b - '0';
diff --git a/lib/libvcc/vcc_var.c b/lib/libvcc/vcc_var.c
index 9e4c8cb..6428b86 100644
--- a/lib/libvcc/vcc_var.c
+++ b/lib/libvcc/vcc_var.c
@@ -66,7 +66,7 @@ vcc_Var_Wildcard(struct vcc *tl, struct symbol *parent,
 	AN(vsb);
 	VSB_printf(vsb, "&VGC_%s_", vh->rname);
 	for (p = b, u = 1; p < e; p++, u++)
-		if (vct_isalpha(*p) || vct_isdigit(*p))
+		if (vct_isalnum(*p))
 			VSB_putc(vsb, *p);
 		else
 			VSB_printf(vsb, "_%02x_", *p);



From martin at varnish-software.com  Tue Jun 27 14:07:06 2017
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Tue, 27 Jun 2017 16:07:06 +0200
Subject: [4.1] 11830c5 Add extra locking to protect the pools list and
 refcounts
Message-ID: <E1dPr90-0006z6-17@project.varnish-software.com>


commit 11830c5d1e5efa3a0d3e2dcb4da56429278e60a3
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Tue Jun 27 15:33:41 2017 +0200

    Add extra locking to protect the pools list and refcounts
    
    Probes currently running on a worker thread at the time they are
    deleted will delay the release of the refcount they hold on the TCP
    pool. Since this call will not be from the CLI thread we need locking
    to protect these datastructures.

diff --git a/bin/varnishd/cache/cache_backend_tcp.c b/bin/varnishd/cache/cache_backend_tcp.c
index 9f28d22..8e9b660 100644
--- a/bin/varnishd/cache/cache_backend_tcp.c
+++ b/bin/varnishd/cache/cache_backend_tcp.c
@@ -71,6 +71,7 @@ struct tcp_pool {
 
 };
 
+static struct lock		pools_mtx;
 static VTAILQ_HEAD(, tcp_pool)	pools = VTAILQ_HEAD_INITIALIZER(pools);
 
 /*--------------------------------------------------------------------
@@ -127,6 +128,7 @@ VBT_Ref(const struct suckaddr *ip4, const struct suckaddr *ip6)
 {
 	struct tcp_pool *tp;
 
+	Lck_Lock(&pools_mtx);
 	VTAILQ_FOREACH(tp, &pools, list) {
 		assert(tp->refcnt > 0);
 		if (ip4 == NULL) {
@@ -148,8 +150,10 @@ VBT_Ref(const struct suckaddr *ip4, const struct suckaddr *ip6)
 				continue;
 		}
 		tp->refcnt++;
+		Lck_Unlock(&pools_mtx);
 		return (tp);
 	}
+	Lck_Unlock(&pools_mtx);
 
 	ALLOC_OBJ(tp, TCP_POOL_MAGIC);
 	AN(tp);
@@ -161,10 +165,14 @@ VBT_Ref(const struct suckaddr *ip4, const struct suckaddr *ip6)
 	Lck_New(&tp->mtx, lck_backend_tcp);
 	VTAILQ_INIT(&tp->connlist);
 	VTAILQ_INIT(&tp->killlist);
-	VTAILQ_INSERT_HEAD(&pools, tp, list);
 	INIT_OBJ(&tp->waitfor, WAITFOR_MAGIC);
 	tp->waitfor.func = tcp_handle;
 	tp->waitfor.tmo = &cache_param->backend_idle_timeout;
+
+	Lck_Lock(&pools_mtx);
+	VTAILQ_INSERT_HEAD(&pools, tp, list);
+	Lck_Unlock(&pools_mtx);
+
 	return (tp);
 }
 
@@ -182,11 +190,16 @@ VBT_Rel(struct tcp_pool **tpp)
 	tp = *tpp;
 	*tpp = NULL;
 	CHECK_OBJ_NOTNULL(tp, TCP_POOL_MAGIC);
+	Lck_Lock(&pools_mtx);
 	assert(tp->refcnt > 0);
-	if (--tp->refcnt > 0)
+	if (--tp->refcnt > 0) {
+		Lck_Unlock(&pools_mtx);
 		return;
+	}
 	AZ(tp->n_used);
 	VTAILQ_REMOVE(&pools, tp, list);
+	Lck_Unlock(&pools_mtx);
+
 	free(tp->name);
 	free(tp->ip4);
 	free(tp->ip6);
@@ -408,3 +421,11 @@ VBT_Wait(struct worker *wrk, struct vbc *vbc)
 	vbc->cond = NULL;
 	Lck_Unlock(&tp->mtx);
 }
+
+/*--------------------------------------------------------------------*/
+
+void
+VBT_Init(void)
+{
+	Lck_New(&pools_mtx, lck_backend);
+}
diff --git a/bin/varnishd/cache/cache_main.c b/bin/varnishd/cache/cache_main.c
index 2609d7b..f0ebf83 100644
--- a/bin/varnishd/cache/cache_main.c
+++ b/bin/varnishd/cache/cache_main.c
@@ -233,6 +233,7 @@ child_main(void)
 	HTTP_Init();
 
 	VBO_Init();
+	VBT_Init();
 	VBP_Init();
 	VBE_InitCfg();
 	Pool_Init();
diff --git a/bin/varnishd/cache/cache_priv.h b/bin/varnishd/cache/cache_priv.h
index 9a7ba9f..521f569 100644
--- a/bin/varnishd/cache/cache_priv.h
+++ b/bin/varnishd/cache/cache_priv.h
@@ -54,6 +54,9 @@ void VCA_Shutdown(void);
 void VBE_InitCfg(void);
 void VBE_Poll(void);
 
+/* cache_backend_tcp.c */
+void VBT_Init(void);
+
 /* cache_backend_poll.c */
 void VBP_Init(void);
 



From hermunn at varnish-software.com  Wed Jun 28 08:17:05 2017
From: hermunn at varnish-software.com (Pål Hermunn Johansen)
Date: Wed, 28 Jun 2017 10:17:05 +0200
Subject: [4.1] 440e2f1 Don't panic on a null ban
Message-ID: <E1dQ89p-00014t-Ua@project.varnish-software.com>


commit 440e2f1a899c75537c5fac000293c2a8af9afc19
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Mon Jun 26 18:01:33 2017 +0200

    Don't panic on a null ban
    
    like ban(req.http.non-existing-header)

diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c
index a73489b..678d8aa 100644
--- a/bin/varnishd/cache/cache_vrt.c
+++ b/bin/varnishd/cache/cache_vrt.c
@@ -416,7 +416,11 @@ VRT_ban_string(VRT_CTX, const char *str)
 
 	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
 	AN(ctx->vsl);
-	AN(str);
+
+	if (str == NULL) {
+		VSLb(ctx->vsl, SLT_VCL_Error, "ban(): Null argument");
+		return;
+	}
 
 	bp = BAN_Build();
 	if (bp == NULL) {
diff --git a/bin/varnishtest/tests/v00011.vtc b/bin/varnishtest/tests/v00011.vtc
index 0f470ec..ea79b64 100644
--- a/bin/varnishtest/tests/v00011.vtc
+++ b/bin/varnishtest/tests/v00011.vtc
@@ -11,6 +11,7 @@ server s1 {
 varnish v1 -vcl+backend {
 	sub vcl_recv {
 		if (req.method == "PURGE") {
+			ban(req.http.doesntexist);
 			ban("");
 			ban("req.url");
 			ban("req.url //");
@@ -31,6 +32,7 @@ client c1 {
 } -run
 
 logexpect l1 -v v1 -d 1 -g vxid {
+	expect * 1004	VCL_Error {ban[(][)]: Null argument}
 	expect * 1004	VCL_Error {ban[(][)]: No ban conditions found[.]}
 	expect * 1004	VCL_Error {ban[(][)]: Expected comparison operator[.]}
 	expect * 1004	VCL_Error {ban[(][)]: Expected second operand[.]}



From hermunn at varnish-software.com  Wed Jun 28 08:58:06 2017
From: hermunn at varnish-software.com (Pål Hermunn Johansen)
Date: Wed, 28 Jun 2017 10:58:06 +0200
Subject: [4.1] 477c675 Update changelog
Message-ID: <E1dQ8nW-0002iu-6c@project.varnish-software.com>


commit 477c6757ce91f05a176380caa7b07ba8e6f4825f
Author: P?l Hermunn Johansen <hermunn at varnish-software.com>
Date:   Wed Jun 28 10:06:04 2017 +0200

    Update changelog

diff --git a/doc/changes.rst b/doc/changes.rst
index cf1883c..f06d40c 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -1,3 +1,19 @@
+================================
+Varnish Cache 4.1.7 (unreleased)
+================================
+
+Changes since 4.1.7-beta1:
+
+* Add extra locking to protect the pools list and refcounts
+* Don't panic on a null ban
+
+Bugs fixed
+----------
+
+* 2321_ - Prevent storage backends name collisions
+
+.. _2321: https://github.com/varnishcache/varnish-cache/issues/2321
+
 ======================================
 Varnish Cache 4.1.7-beta1 (2017-06-15)
 ======================================



From hermunn at varnish-software.com  Wed Jun 28 08:58:06 2017
From: hermunn at varnish-software.com (Pål Hermunn Johansen)
Date: Wed, 28 Jun 2017 10:58:06 +0200
Subject: [4.1] 48fa5da Prepare for 4.1.7 release
Message-ID: <E1dQ8nW-0002j0-8e@project.varnish-software.com>


commit 48fa5da61216210b98a0c1f9fc3461d2a883305f
Author: P?l Hermunn Johansen <hermunn at varnish-software.com>
Date:   Wed Jun 28 10:12:57 2017 +0200

    Prepare for 4.1.7 release

diff --git a/configure.ac b/configure.ac
index 32bd840..bf93943 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2,7 +2,7 @@ AC_PREREQ(2.59)
 AC_COPYRIGHT([Copyright (c) 2006 Verdens Gang AS
 Copyright (c) 2006-2016 Varnish Software])
 AC_REVISION([$Id$])
-AC_INIT([Varnish], [4.1.7-beta1], [varnish-dev at varnish-cache.org])
+AC_INIT([Varnish], [4.1.7], [varnish-dev at varnish-cache.org])
 AC_CONFIG_SRCDIR(include/miniobj.h)
 AC_CONFIG_HEADERS([config.h])
 AC_CONFIG_MACRO_DIR([m4])
diff --git a/doc/changes.rst b/doc/changes.rst
index f06d40c..dff047e 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -1,5 +1,5 @@
 ================================
-Varnish Cache 4.1.7 (unreleased)
+Varnish Cache 4.1.7 (2017-06-28)
 ================================
 
 Changes since 4.1.7-beta1:



From phk at FreeBSD.org  Wed Jun 28 19:42:05 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 28 Jun 2017 21:42:05 +0200
Subject: [master] 469d65e VSM_Delete() -> VSM_Destroy() to follow our current
 OO schema.
Message-ID: <E1dQIqj-0006sw-NI@project.varnish-software.com>


commit 469d65e03406817b3ef96b04f826fd2d093ac30f
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Jun 28 19:40:41 2017 +0000

    VSM_Delete() -> VSM_Destroy() to follow our current OO schema.

diff --git a/bin/varnishadm/varnishadm.c b/bin/varnishadm/varnishadm.c
index b983c52..52a6267 100644
--- a/bin/varnishadm/varnishadm.c
+++ b/bin/varnishadm/varnishadm.c
@@ -402,18 +402,18 @@ n_arg_sock(const char *n_arg)
 	AN(vsm);
 	if (VSM_n_Arg(vsm, n_arg) < 0) {
 		fprintf(stderr, "%s\n", VSM_Error(vsm));
-		VSM_Delete(vsm);
+		VSM_Destroy(&vsm);
 		return (-1);
 	}
 	if (VSM_Open(vsm)) {
 		fprintf(stderr, "%s\n", VSM_Error(vsm));
-		VSM_Delete(vsm);
+		VSM_Destroy(&vsm);
 		return (-1);
 	}
 
 	if (!VSM_Get(vsm, &vt, "Arg", "-T")) {
 		fprintf(stderr, "No -T arg in shared memory\n");
-		VSM_Delete(vsm);
+		VSM_Destroy(&vsm);
 		return (-1);
 	}
 	AZ(VSM_Map(vsm, &vt));
@@ -426,7 +426,7 @@ n_arg_sock(const char *n_arg)
 		S_arg = strdup(vt.b);
 	}
 
-	VSM_Delete(vsm);
+	VSM_Destroy(&vsm);
 
 	sock = -1;
 	while (*T_arg) {
diff --git a/bin/varnishtest/vtc_logexp.c b/bin/varnishtest/vtc_logexp.c
index 6b79933..6804772 100644
--- a/bin/varnishtest/vtc_logexp.c
+++ b/bin/varnishtest/vtc_logexp.c
@@ -192,7 +192,7 @@ logexp_delete(struct logexp *le)
 	logexp_delete_tests(le);
 	free(le->name);
 	free(le->query);
-	VSM_Delete(le->vsm);
+	VSM_Destroy(&le->vsm);
 	if (le->n_arg)
 		VSB_destroy(&le->n_arg);
 	FREE_OBJ(le);
diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c
index 1488ce6..b31a16f 100644
--- a/bin/varnishtest/vtc_varnish.c
+++ b/bin/varnishtest/vtc_varnish.c
@@ -276,7 +276,7 @@ varnishlog_thread(void *priv)
 	if (c)
 		VSL_DeleteCursor(c);
 	VSL_Delete(vsl);
-	VSM_Delete(vsm);
+	VSM_Destroy(&vsm);
 
 	return (NULL);
 }
@@ -335,7 +335,7 @@ varnish_delete(struct varnish *v)
 	free(v->name);
 	free(v->workdir);
 	if (v->vd != NULL)
-		VSM_Delete(v->vd);
+		VSM_Destroy(&v->vd);
 
 	/*
 	 * We do not delete the workdir, it may contain stuff people
diff --git a/include/vapi/vsm.h b/include/vapi/vsm.h
index a463e9a..37d25ef 100644
--- a/include/vapi/vsm.h
+++ b/include/vapi/vsm.h
@@ -70,7 +70,7 @@ struct vsm *VSM_New(void);
 	 *	NULL: malloc failed.
 	 */
 
-void VSM_Delete(struct vsm *vd);
+void VSM_Destroy(struct vsm **vd);
 	/*
 	 * Close and deallocate all storage and mappings.
 	 * (including any VSC and VSL "sub-classes")
diff --git a/lib/libvarnishapi/libvarnishapi.map b/lib/libvarnishapi/libvarnishapi.map
index 3a3fbe0..4ac8f19 100644
--- a/lib/libvarnishapi/libvarnishapi.map
+++ b/lib/libvarnishapi/libvarnishapi.map
@@ -182,6 +182,7 @@ LIBVARNISHAPI_1.7 {
 	VSM_invalid;
 	VSM_valid;
 	VSM_similar;
+	VSM_Destroy;
 	VSM_Map;
 	VSM_Unmap;
 	VSC_Destroy_Point;
diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c
index 8af47fd..e07f01b 100644
--- a/lib/libvarnishapi/vsm.c
+++ b/lib/libvarnishapi/vsm.c
@@ -216,8 +216,13 @@ VSM_Name(const struct vsm *vd)
 /*--------------------------------------------------------------------*/
 
 void
-VSM_Delete(struct vsm *vd)
+VSM_Destroy(struct vsm **vdp)
 {
+	struct vsm *vd;
+
+	AN(vdp);
+	vd = *vdp;
+	*vdp = NULL;
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
 
diff --git a/lib/libvarnishapi/vut.c b/lib/libvarnishapi/vut.c
index f9faa10..1d110f7 100644
--- a/lib/libvarnishapi/vut.c
+++ b/lib/libvarnishapi/vut.c
@@ -336,7 +336,7 @@ VUT_Fini(void)
 	if (VUT.vsl)
 		VSL_Delete(VUT.vsl);
 	if (VUT.vsm)
-		VSM_Delete(VUT.vsm);
+		VSM_Destroy(&VUT.vsm);
 
 	memset(&VUT, 0, sizeof VUT);
 }



From phk at FreeBSD.org  Wed Jun 28 21:20:07 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 28 Jun 2017 23:20:07 +0200
Subject: [master] ec1e6f4 Introduce VSM_Start() which takes care of waiting
 around for varnishd if that is desired.
Message-ID: <E1dQKNb-0002cJ-5h@project.varnish-software.com>


commit ec1e6f408508df6cd28040ec9fd24204a9f13374
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Jun 28 21:18:30 2017 +0000

    Introduce VSM_Start() which takes care of waiting around for varnishd
    if that is desired.

diff --git a/bin/varnishadm/varnishadm.c b/bin/varnishadm/varnishadm.c
index 52a6267..43cab6f 100644
--- a/bin/varnishadm/varnishadm.c
+++ b/bin/varnishadm/varnishadm.c
@@ -405,7 +405,7 @@ n_arg_sock(const char *n_arg)
 		VSM_Destroy(&vsm);
 		return (-1);
 	}
-	if (VSM_Open(vsm)) {
+	if (VSM_Start(vsm, 0, -1)) {
 		fprintf(stderr, "%s\n", VSM_Error(vsm));
 		VSM_Destroy(&vsm);
 		return (-1);
diff --git a/bin/varnishstat/varnishstat.c b/bin/varnishstat/varnishstat.c
index 23056e7..1cb5f94 100644
--- a/bin/varnishstat/varnishstat.c
+++ b/bin/varnishstat/varnishstat.c
@@ -253,7 +253,7 @@ int
 main(int argc, char * const *argv)
 {
 	struct vsm *vd;
-	double t_arg = 5.0, t_start = NAN;
+	double t_arg = 5.0;
 	int once = 0, xml = 0, json = 0, f_list = 0, curses = 0;
 	signed char opt;
 	int i;
@@ -308,34 +308,12 @@ main(int argc, char * const *argv)
 	if (!(xml || json || once || f_list))
 		curses = 1;
 
-	while (1) {
-		i = VSM_Open(vd);
-		if (!i)
-			break;
-		if (isnan(t_start) && t_arg > 0.) {
-			fprintf(stderr, "Can't open log -"
-			    " retrying for %.0f seconds\n", t_arg);
-			t_start = VTIM_real();
-		}
-		if (t_arg <= 0.)
-			break;
-		if (VTIM_real() - t_start > t_arg)
-			break;
-		VSM_ResetError(vd);
-		VTIM_sleep(0.5);
-	}
-
-	if (curses) {
-		if (i && t_arg >= 0.)
-			VUT_Error(1, "%s", VSM_Error(vd));
-		do_curses(vd, 1.0);
-		exit(0);
-	}
-
-	if (i)
+	if (VSM_Start(vd, t_arg, STDERR_FILENO))
 		VUT_Error(1, "%s", VSM_Error(vd));
 
-	if (xml)
+	if (curses)
+		do_curses(vd, 1.0);
+	else if (xml)
 		do_xml(vd);
 	else if (json)
 		do_json(vd);
diff --git a/bin/varnishtest/tests/u00005.vtc b/bin/varnishtest/tests/u00005.vtc
index 47a036e..5d91958 100644
--- a/bin/varnishtest/tests/u00005.vtc
+++ b/bin/varnishtest/tests/u00005.vtc
@@ -41,7 +41,7 @@ shell -err -expect "-t: Range error" \
 	"varnishstat -t -1"
 shell -err -expect "-t: Syntax error" \
 	"varnishstat -t foo"
-shell -err -expect "Cannot open /nonexistent/_.vsm" \
+shell -err -expect "Could not get hold of varnishd" \
 	"varnishstat -n /nonexistent -t 1"
 shell -expect "MAIN.uptime" \
 	"varnishstat -n ${v1_name} -1"
diff --git a/bin/varnishtest/vtc_logexp.c b/bin/varnishtest/vtc_logexp.c
index 6804772..1c5a696 100644
--- a/bin/varnishtest/vtc_logexp.c
+++ b/bin/varnishtest/vtc_logexp.c
@@ -142,6 +142,7 @@ struct logexp {
 	VTAILQ_ENTRY(logexp)		list;
 
 	char				*name;
+	char				*vname;
 	struct vtclog			*vl;
 	char				run;
 	VTAILQ_HEAD(,logexp_test)	tests;
@@ -156,7 +157,6 @@ struct logexp {
 	char				*query;
 
 	struct vsm			*vsm;
-	struct vsb			*n_arg;
 	struct VSL_data			*vsl;
 	struct VSLQ			*vslq;
 	pthread_t			tp;
@@ -193,15 +193,14 @@ logexp_delete(struct logexp *le)
 	free(le->name);
 	free(le->query);
 	VSM_Destroy(&le->vsm);
-	if (le->n_arg)
-		VSB_destroy(&le->n_arg);
 	FREE_OBJ(le);
 }
 
 static struct logexp *
-logexp_new(const char *name)
+logexp_new(const char *name, const char *varg)
 {
 	struct logexp *le;
+	struct vsb *vsb, *n_arg;
 
 	ALLOC_OBJ(le, LOGEXP_MAGIC);
 	AN(le);
@@ -217,6 +216,21 @@ logexp_new(const char *name)
 	AN(le->vsl);
 
 	VTAILQ_INSERT_TAIL(&logexps, le, list);
+
+	REPLACE(le->vname, varg);
+
+	vsb = VSB_new_auto();
+	AN(vsb);
+	AZ(VSB_printf(vsb, "${tmpdir}/%s", varg));
+	AZ(VSB_finish(vsb));
+	n_arg = macro_expand(le->vl, VSB_data(vsb));
+	VSB_destroy(&vsb);
+	if (n_arg == NULL)
+		vtc_fatal(le->vl, "-v argument problems");
+	if (VSM_n_Arg(le->vsm, VSB_data(n_arg)) <= 0)
+		vtc_fatal(le->vl, "-v argument error: %s",
+		    VSM_Error(le->vsm));
+	VSB_destroy(&n_arg);
 	return (le);
 }
 
@@ -376,13 +390,8 @@ logexp_start(struct logexp *le)
 	AN(le->vsl);
 	AZ(le->vslq);
 
-	if (le->n_arg == NULL)
-		vtc_fatal(le->vl, "-v argument not given");
-	if (VSM_n_Arg(le->vsm, VSB_data(le->n_arg)) <= 0)
-		vtc_fatal(le->vl, "-v argument error: %s",
-		    VSM_Error(le->vsm));
-	if (VSM_Open(le->vsm))
-		vtc_fatal(le->vl, "VSM_Open: %s", VSM_Error(le->vsm));
+	if (VSM_Start(le->vsm, 0, -1))
+		vtc_fatal(le->vl, "VSM_Start: %s", VSM_Error(le->vsm));
 	AN(le->vsl);
 	c = VSL_CursorVSM(le->vsl, le->vsm,
 	    (le->d_arg ? 0 : VSL_COPT_TAIL) | VSL_COPT_BATCH);
@@ -504,8 +513,6 @@ void
 cmd_logexpect(CMD_ARGS)
 {
 	struct logexp *le, *le2;
-	const char tmpdir[] = "${tmpdir}";
-	struct vsb *vsb;
 
 	(void)priv;
 	(void)cmd;
@@ -532,8 +539,12 @@ cmd_logexpect(CMD_ARGS)
 		if (!strcmp(le->name, av[0]))
 			break;
 	}
-	if (le == NULL)
-		le = logexp_new(av[0]);
+	if (le == NULL) {
+		if (strcmp(av[1], "-v") || av[2] == NULL)
+			vtc_fatal(vl, "new logexp lacks -v");
+		le = logexp_new(av[0], av[2]);
+		av += 2;
+	}
 	av++;
 
 	for (; *av != NULL; av++) {
@@ -556,18 +567,8 @@ cmd_logexpect(CMD_ARGS)
 		AZ(le->run);
 
 		if (!strcmp(*av, "-v")) {
-			if (av[1] == NULL)
-				vtc_fatal(le->vl, "Missing -v argument");
-			if (le->n_arg != NULL)
-				VSB_destroy(&le->n_arg);
-			vsb = VSB_new_auto();
-			AN(vsb);
-			AZ(VSB_printf(vsb, "%s/%s", tmpdir, av[1]));
-			AZ(VSB_finish(vsb));
-			le->n_arg = macro_expand(le->vl, VSB_data(vsb));
-			VSB_destroy(&vsb);
-			if (le->n_arg == NULL)
-				return;
+			if (av[1] == NULL || strcmp(av[1], le->vname))
+				vtc_fatal(le->vl, "-v argument cannot change");
 			av++;
 			continue;
 		}
diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c
index b31a16f..37aebdf 100644
--- a/bin/varnishtest/vtc_varnish.c
+++ b/bin/varnishtest/vtc_varnish.c
@@ -214,6 +214,9 @@ varnishlog_thread(void *priv)
 	AN(vsm);
 	(void)VSM_n_Arg(vsm, v->workdir);
 
+	if (VSM_Start(vsm, vtc_maxdur, -1))
+		vtc_fatal(v->vl, "vsm|%s", VSM_Error(vsm));
+
 	c = NULL;
 	opt = 0;
 	while (v->pid || c != NULL) {
diff --git a/include/vapi/vsm.h b/include/vapi/vsm.h
index 37d25ef..ce851b0 100644
--- a/include/vapi/vsm.h
+++ b/include/vapi/vsm.h
@@ -98,6 +98,8 @@ int VSM_n_Arg(struct vsm *vd, const char *n_arg);
 	 *	 <0 on failure, VSM_Error() returns diagnostic string
 	 */
 
+int VSM_Start(struct vsm *vd, double patience, int progress_fd);
+
 const char *VSM_Name(const struct vsm *vd);
 	/*
 	 * Return the instance name (-i argument to varnishd)
diff --git a/lib/libvarnishapi/libvarnishapi.map b/lib/libvarnishapi/libvarnishapi.map
index 4ac8f19..c860e7d 100644
--- a/lib/libvarnishapi/libvarnishapi.map
+++ b/lib/libvarnishapi/libvarnishapi.map
@@ -182,6 +182,7 @@ LIBVARNISHAPI_1.7 {
 	VSM_invalid;
 	VSM_valid;
 	VSM_similar;
+	VSM_Start;
 	VSM_Destroy;
 	VSM_Map;
 	VSM_Unmap;
diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c
index e07f01b..e43bd9d 100644
--- a/lib/libvarnishapi/vsm.c
+++ b/lib/libvarnishapi/vsm.c
@@ -85,6 +85,8 @@ struct vsm {
 	uint64_t		age_ok;
 	double			t_ok;
 
+	int			started;
+
 	struct vsc		*vsc;
 };
 
@@ -179,6 +181,7 @@ VSM_n_Arg(struct vsm *vd, const char *arg)
 	struct vsb *vsb;
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
+	AZ(vd->started);
 
 	if (vd->head)
 		return (vsm_diag(vd, "VSM_n_Arg: Already open"));
@@ -202,6 +205,37 @@ VSM_n_Arg(struct vsm *vd, const char *arg)
 
 /*--------------------------------------------------------------------*/
 
+int
+VSM_Start(struct vsm *vd, double patience, int progress)
+{
+	double t0;
+	int i, n = 0;
+
+	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
+	AN(vd->dname);
+	t0 = VTIM_mono();
+	while (1) {
+		i = VSM_Open(vd);
+		if (patience <= 0. || i == 0) {
+			if (progress >= 0 && n > 4)
+				(void)write(progress, "\n", 1);
+			vd->started = 1;
+			return (i);
+		}
+		if (t0 + patience < VTIM_mono()) {
+			if (progress >= 0 && n > 4)
+				(void)write(progress, "\n", 1);
+			return (vsm_diag(vd,
+			    "Could not get hold of varnishd, is it running?"));
+		}
+		if (progress >= 0 && !(++n % 4))
+			(void)write(progress, ".", 1);
+		VTIM_sleep(.25);
+	}
+}
+
+/*--------------------------------------------------------------------*/
+
 const char *
 VSM_Name(const struct vsm *vd)
 {
diff --git a/lib/libvarnishapi/vut.c b/lib/libvarnishapi/vut.c
index 1d110f7..8954b92 100644
--- a/lib/libvarnishapi/vut.c
+++ b/lib/libvarnishapi/vut.c
@@ -227,8 +227,6 @@ void
 VUT_Setup(void)
 {
 	struct VSL_cursor *c;
-	double t_start;
-	int i;
 
 	AN(VUT.vsl);
 	AZ(VUT.vsm);
@@ -255,45 +253,16 @@ VUT_Setup(void)
 		AN(VUT.vsm);
 		if (VUT.n_arg && VSM_n_Arg(VUT.vsm, VUT.n_arg) <= 0)
 			VUT_Error(1, "%s", VSM_Error(VUT.vsm));
-		t_start = NAN;
-		c = NULL;
-		while (1) {
-			i = VSM_Open(VUT.vsm);
-			if (!i)
-				c = VSL_CursorVSM(VUT.vsl, VUT.vsm,
-				    (VUT.d_opt ? VSL_COPT_TAILSTOP :
-					VSL_COPT_TAIL)
-				    | VSL_COPT_BATCH);
-			if (c)
-				break;
-
-			if (isnan(t_start) && VUT.t_arg > 0.) {
-				fprintf(stderr, "Cannot open log -"
-				    " retrying for %.0f seconds\n", VUT.t_arg);
-				t_start = VTIM_real();
-			}
-			VSM_Close(VUT.vsm);
-			if (VUT.t_arg <= 0.)
-				break;
-			if (VTIM_real() - t_start > VUT.t_arg)
-				break;
-
-			VSM_ResetError(VUT.vsm);
-			VSL_ResetError(VUT.vsl);
-			VTIM_sleep(0.5);
-		}
-
-		if (VUT.t_arg >= 0. && (i || !c)) {
-			if (i)
-				VUT_Error(1, "%s", VSM_Error(VUT.vsm));
-			else
-				VUT_Error(1, "%s", VSL_Error(VUT.vsl));
-		} else if (!isnan(t_start))
-			fprintf(stderr, "Log opened\n");
+		if (VSM_Start(VUT.vsm, VUT.t_arg, STDERR_FILENO))
+			VUT_Error(1, "VSM: %s", VSM_Error(VUT.vsm));
+		c = VSL_CursorVSM(VUT.vsl, VUT.vsm,
+		    (VUT.d_opt ? VSL_COPT_TAILSTOP : VSL_COPT_TAIL)
+		    | VSL_COPT_BATCH);
+		if (c == 0)
+			VUT_Error(1, "VSL: %s", VSL_Error(VUT.vsl));
 	}
 
-	if (c)
-		VSLQ_SetCursor(VUT.vslq, &c);
+	VSLQ_SetCursor(VUT.vslq, &c);
 	AZ(c);
 
 	/* Signal handlers */



From phk at FreeBSD.org  Thu Jun 29 04:41:05 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 29 Jun 2017 06:41:05 +0200
Subject: [master] bd4794e Remember to reset error buffer.
Message-ID: <E1dQRGL-0003rV-4u@project.varnish-software.com>


commit bd4794e60bd80a5b99d51db3834baaf3034eb20f
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Jun 29 04:39:52 2017 +0000

    Remember to reset error buffer.

diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c
index e43bd9d..9261645 100644
--- a/lib/libvarnishapi/vsm.c
+++ b/lib/libvarnishapi/vsm.c
@@ -215,6 +215,7 @@ VSM_Start(struct vsm *vd, double patience, int progress)
 	AN(vd->dname);
 	t0 = VTIM_mono();
 	while (1) {
+		VSM_ResetError(vd);
 		i = VSM_Open(vd);
 		if (patience <= 0. || i == 0) {
 			if (progress >= 0 && n > 4)



From phk at FreeBSD.org  Thu Jun 29 06:59:05 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 29 Jun 2017 08:59:05 +0200
Subject: [master] d3baf27 Try to find out why !FreeBSD hates VSM_Start()
Message-ID: <E1dQTPt-0000zl-KY@project.varnish-software.com>


commit d3baf2760c43b58b20100339d548f13997b11f5f
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Jun 29 06:58:17 2017 +0000

    Try to find out why !FreeBSD hates VSM_Start()

diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c
index 37aebdf..8fc5058 100644
--- a/bin/varnishtest/vtc_varnish.c
+++ b/bin/varnishtest/vtc_varnish.c
@@ -214,7 +214,7 @@ varnishlog_thread(void *priv)
 	AN(vsm);
 	(void)VSM_n_Arg(vsm, v->workdir);
 
-	if (VSM_Start(vsm, vtc_maxdur, -1))
+	if (VSM_Start(vsm, vtc_maxdur, 2))
 		vtc_fatal(v->vl, "vsm|%s", VSM_Error(vsm));
 
 	c = NULL;



From phk at FreeBSD.org  Thu Jun 29 08:38:05 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 29 Jun 2017 10:38:05 +0200
Subject: [master] 8bbbdfa Try to work around systems having unsigned char
 types.
Message-ID: <E1dQUxh-00053D-ON@project.varnish-software.com>


commit 8bbbdfa3f4661aa5149a5ebc4e80167b4465312e
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Jun 29 08:37:09 2017 +0000

    Try to work around systems having unsigned char types.

diff --git a/lib/libvarnishapi/vjsn.c b/lib/libvarnishapi/vjsn.c
index 01f10c2..6ae82fa 100644
--- a/lib/libvarnishapi/vjsn.c
+++ b/lib/libvarnishapi/vjsn.c
@@ -214,7 +214,7 @@ vjsn_string(struct vjsn *js)
 			js->err = "Unterminate string";
 			return (NULL);
 		}
-		if (*js->ptr >= 0x00 && *js->ptr <= 0x1f) {
+		if ((unsigned char)(*js->ptr) <= 0x1f) {
 			js->err = "unescaped control char in string";
 			return (NULL);
 		}



From phk at FreeBSD.org  Thu Jun 29 12:02:06 2017
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 29 Jun 2017 14:02:06 +0200
Subject: [master] 34487d1 Synchronously open the VSM for the varnishlog
Message-ID: <E1dQY98-000506-Bq@project.varnish-software.com>


commit 34487d143824d6b8f3be8255c5f29469b6ee67e5
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Jun 29 12:00:44 2017 +0000

    Synchronously open the VSM for the varnishlog

diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c
index 8fc5058..04e490c 100644
--- a/bin/varnishtest/vtc_varnish.c
+++ b/bin/varnishtest/vtc_varnish.c
@@ -77,6 +77,7 @@ struct varnish {
 	char			*proto;
 
 	struct vsm		*vd;		/* vsc use */
+	struct vsm		*vdl;		/* log use */
 	struct vsm_fantom	mgt_arg_i;
 
 	unsigned		vsl_tag_count[256];
@@ -210,12 +211,15 @@ varnishlog_thread(void *priv)
 
 	vsl = VSL_New();
 	AN(vsl);
-	vsm = VSM_New();
+	vsm = v->vdl;
+#if 0
+	//vsm = VSM_New();
 	AN(vsm);
 	(void)VSM_n_Arg(vsm, v->workdir);
 
 	if (VSM_Start(vsm, vtc_maxdur, 2))
 		vtc_fatal(v->vl, "vsm|%s", VSM_Error(vsm));
+#endif
 
 	c = NULL;
 	opt = 0;
@@ -279,7 +283,7 @@ varnishlog_thread(void *priv)
 	if (c)
 		VSL_DeleteCursor(c);
 	VSL_Delete(vsl);
-	VSM_Destroy(&vsm);
+	VSM_Destroy(&v->vdl);
 
 	return (NULL);
 }
@@ -402,8 +406,6 @@ varnish_launch(struct varnish *v)
 	const char *err;
 	char *r = NULL;
 
-	v->vd = VSM_New();
-
 	/* Create listener socket */
 	v->cli_fd = VTCP_listen_on("127.0.0.1:0", NULL, 1, &err);
 	if (err != NULL)
@@ -472,7 +474,6 @@ varnish_launch(struct varnish *v)
 	v->fds[2] = v->fds[3] = -1;
 	VSB_destroy(&vsb);
 	AZ(pthread_create(&v->tp, NULL, varnish_thread, v));
-	AZ(pthread_create(&v->tp_vsl, NULL, varnishlog_thread, v));
 
 	/* Wait for the varnish to call home */
 	memset(fd, 0, sizeof fd);
@@ -527,9 +528,15 @@ varnish_launch(struct varnish *v)
 		vtc_fatal(v->vl, "CLI auth command failed: %u %s", u, r);
 	free(r);
 
+	v->vd = VSM_New();
 	(void)VSM_n_Arg(v->vd, v->workdir);
-	AZ(VSM_Open(v->vd));
+	AZ(VSM_Start(v->vd, vtc_maxdur, -1));
 	assert(VSM_Get(v->vd, &v->mgt_arg_i, "Arg", "-i") > 0);
+
+	v->vdl = VSM_New();
+	(void)VSM_n_Arg(v->vdl, v->workdir);
+	AZ(VSM_Start(v->vdl, vtc_maxdur, -1));
+	AZ(pthread_create(&v->tp_vsl, NULL, varnishlog_thread, v));
 }
 
 /**********************************************************************



From nils.goroll at uplex.de  Fri Jun 30 09:16:06 2017
From: nils.goroll at uplex.de (Nils Goroll)
Date: Fri, 30 Jun 2017 11:16:06 +0200
Subject: [master] 27aa567 update VCL_trace usage example
Message-ID: <E1dQs22-0006gO-BC@project.varnish-software.com>


commit 27aa56736015ac021194f106e91985588a5b9d2c
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Fri Jun 30 11:15:07 2017 +0200

    update VCL_trace usage example

diff --git a/doc/sphinx/installation/bugs.rst b/doc/sphinx/installation/bugs.rst
index 091b7c7..2305c1c 100644
--- a/doc/sphinx/installation/bugs.rst
+++ b/doc/sphinx/installation/bugs.rst
@@ -150,9 +150,9 @@ than what you intended it to do. If it sounds like a bug that would
 have tripped up everybody else, take a moment to read through your
 VCL and see if it really does what you think it does.
 
-You can also try setting the ``vcl_trace`` parameter, that will generate log
-records with like and character number for each statement executed in your VCL
-program.
+You can also try setting the ``vsl_mask=+VCL_trace`` parameter (or use
+``varnishadm param.set vsl_mask +VCL_trace`` on a running instance),
+that will generate log records with like and character number for each
+statement executed in your VCL program.
 
 .. XXX:Example of the command perhaps? benc
-



From nils.goroll at uplex.de  Fri Jun 30 17:02:05 2017
From: nils.goroll at uplex.de (Nils Goroll)
Date: Fri, 30 Jun 2017 19:02:05 +0200
Subject: [master] b5dd75b noticed on a prestine instance that we require make ; )
Message-ID: <E1dQzIz-0000JL-W3@project.varnish-software.com>


commit b5dd75b5249e6e48721d477eb2f4ce749ab346af
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Fri Jun 30 19:01:14 2017 +0200

    noticed on a prestine instance that we require make ;)

diff --git a/doc/sphinx/installation/install.rst b/doc/sphinx/installation/install.rst
index 0596db5..5ab087f 100644
--- a/doc/sphinx/installation/install.rst
+++ b/doc/sphinx/installation/install.rst
@@ -78,6 +78,7 @@ installed. On a Debian or Ubuntu system these are:
 
 ..  grep-dctrl -n -sBuild-Depends -r ^ ../../../../varnish-cache-debian/control | tr -d '\n' | awk -F,\  '{ for (i = 0; ++i <= NF;) { sub (/ .*/, "", $i); print "* `" $i "`"; }}' | egrep -v '(debhelper)'
 
+* `make`
 * `automake`
 * `autotools-dev`
 * `libedit-dev`
@@ -101,6 +102,7 @@ packages installed:
 
 .. gawk '/^BuildRequires/ {print "* `" $2 "`"}' ../../../redhat/varnish.spec | sort | uniq | egrep -v '(systemd)'
 
+* `make`
 * `autoconf`
 * `automake`
 * `jemalloc-devel`



From nils.goroll at uplex.de  Fri Jun 30 17:59:05 2017
From: nils.goroll at uplex.de (Nils Goroll)
Date: Fri, 30 Jun 2017 19:59:05 +0200
Subject: [master] 01ef5b2 (struct vsm).dname gets initialized in VSM_Open
Message-ID: <E1dR0C9-0002k4-Tv@project.varnish-software.com>


commit 01ef5b28904b989ae7170a32205471163d984289
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Fri Jun 30 19:56:10 2017 +0200

    (struct vsm).dname gets initialized in VSM_Open
    
    Fixes #2359

diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c
index 9261645..a92a84c 100644
--- a/lib/libvarnishapi/vsm.c
+++ b/lib/libvarnishapi/vsm.c
@@ -212,11 +212,11 @@ VSM_Start(struct vsm *vd, double patience, int progress)
 	int i, n = 0;
 
 	CHECK_OBJ_NOTNULL(vd, VSM_MAGIC);
-	AN(vd->dname);
 	t0 = VTIM_mono();
 	while (1) {
 		VSM_ResetError(vd);
 		i = VSM_Open(vd);
+		AN(vd->dname);
 		if (patience <= 0. || i == 0) {
 			if (progress >= 0 && n > 4)
 				(void)write(progress, "\n", 1);